Published November 30, 2023 © GPL3+

Haptic Swim Assistant for Low Visibility Swimmers

Help low visibility swimmers avoid obstacles and know when they're approaching a wall with haptics, as well as provide safety features.

AdvancedFull instructions provided20 hours391

Grand Prize Swimming

Build2gether Inclusive Innovation Challenge

Haptic Swim Assistant for Low Visibility Swimmers

Things used in this project

Hardware components

Blues Notecard (Cellular)

Blues Notecarrier-F

DFRobot FireBeetle ESP32 IOT Microcontroller (Supports Wi-Fi & Bluetooth)

I used the FireBeetle esp32-e

DFRobot underwater ultrasonic sensor

DFRobot 6 DOF Sensor - MPU6050

Brown Dog Gadgets Solar Cockroach Vibrating Disc Motor

vibration motors for haptic feedback

Software apps and online services

Arduino IDE

Story

The Haptic Swim Assistant

This project is specifically inspired by the low-visibility swimmers involved in the Build2gether challenge. Through discussions in Discord, what I gleaned was that this would ideally be usable in any body of water without setup, and that feedback should be provided via haptics, which to me means the device should be a wearable. The basics of this are pretty straightforward, but our goal here is to blow this out of the water (pun very much intended). We'll be adding every convenience feature that actually makes sense to include as well as balancing cost-effectiveness and integrating the parts that will get the job done best.

Even as someone without low-visibility issues, I wouldn't exactly mind having an automatic warning system so I could just focus on swimming without worrying about concussing myself against a wall. If this can be just universally beneficial, it may as well also get some features that benefit swimmers in general. So, without further ado, here are the goals of the project:

ProjectGoals:

The device should be a convenient solution, allowing the user to simply put it on and enter any body of water without initial setup.
Full haptic warning systems for obstacles and walls, with increasing haptic feedback as the wearer approaches the object to convey distance as well as direction.
Provide a convenient, automatic safety system that detects issues and notifies an outside party via Blues Wireless in the case of an emergency.

Haptic Swim Assistant... ASSEMBLE!

The microcontroller I'm using is the FireBeetle 2 ESP32-E. It offers extra UART pins which are necessary for running our two ultrasonic sensors smoothly, it can handle a lot of code, and they're still cheap. For haptics, we can use coin vibration motors. We're not trying to shake the user silly; we just need a little buzz that can clearly convey that there is an obstacle ahead, and for that we only need these little buzzers. As for the ultrasonic sensors, I'm specifically using underwater ultrasonic sensors. These are made for applications exactly like this, so going this route made a lot more sense to me than trying to use normal ultrasonic sensors with a see-through enclosure (my initial thought). To be clear though, these sensors only work in water, so when you're testing them you need to put them in water to begin getting readings.

The ultrasonic sensors have to be in water to work

We're also adding a safety feature, which requires an accelerometer and Blues hardware. For this, I got the Blues North America Starter Kit, and we'll be using the Notecard and Notecarrier-F (as well as the Swan for testing the setup). To assemble this, you slot the notecard into the Notecarrier-F and connect the Molex antenna from LTE and GPS to MAIN and GPS, respectively.

The Notecarrier-F with the antenna and Notecard connected

To use these with an external microcontroller is surprisingly easy - just connect the SDA from the FireBeetle to the SDA, the SCL to SCL, and the GND to GND. The accelerometer is used to determine if the user is face-down for an extended period of time.

Here's a full pinout of the setup:

MPU6050 Sensor:

SDA (Data Line) → Connect to SDA of your microcontroller (MCU)
SCL (Clock Line) → Connect to SCL of your MCU
VCC → Connect to 3.3V
GND → Connect to Ground

Notecard Module:

Serial Communication (RX/TX) → Connect to corresponding RX/TX pins on MCU
Power and Ground → Connect to appropriate power source and Ground

HardwareSerial sensorLeft:

RX → Pin 16 of MCU
TX → Pin 17 of MCU

SoftwareSerial sensorRight:

RX → Pin 15 of MCU
TX → Pin 4 of MCU

Motor Pins:

motorPinLeft → Pin 18 of MCU
motorPinRight → Pin 19 of MCU

With that, the project should be

The Blues setup:

I'm happy to say, this part is pretty easy - I had a notably more difficult time getting the accelerometer to give me accurate values than I did in getting the microcontroller to get an alert routed to my phone via text, which is pretty cool.

You can find the quick start guide here. Step 1 is to create your project. You click your name in the top right, go to View Projects and click Create Project.

This gives you your project id that you'll need going forward.

We went over the hardware setup previously, but in getting up and running I did connect the Swan instead of the FireBeetle initially, which can be done by just inserting it onto the Notecarrier-F. Be sure to plug everything into your computer, including the Notecarrier-F. I also am using the recommended SWD programmer/debugger, found here. It's not required but I found things went smoothly while using it.

From the same quick start section, go to the Notecard Quickstart. This, you do need to go through to get your initial setup done, but it's pretty painless. You just get your Notecard connected and synced with your project and you'll be good to go for using it on the Arduino.

I do want to briefly note that the VS Code PlatformIO Extension was noted as the preferred setup, but I had no issues utilizing the Arduino IDE. Before starting with it, you need to install a couple of prerequisites here and here. If you forget to install them, the debugger output will specifically tell you that these are missing and where to get them.

To add the board (the Swan in this case), add this URL to your board manager:

https://github.com/stm32duino/BoardManagerFiles/raw/main/package_stmicroelectronics_index.json

Then install the following board:

With this, you should now see "Blues Wireless Boards" as an option under the STM32 lists, and the Swan can now work with your sketch.

As for the sketch, I'll also attach a simple starting sketch within this project that is designed to just send one test message to ensure that everything is working correctly.

From here, events will begin appearing in your Events tab as you send messages, but we need to actually do something with them. To accomplish this, click the Routes tab in your project, go to Create Route, and choose where you want it routed. In my particular case, I wanted text message alerts, so I utilized Twilio.

With that, we have everything we need to be able to send alert messages to our phone from the Haptic Swim Assistant device, which brings us to the code!

Code:

Starting with the basic functionality, we continuously check for distance with both underwater ultrasonic sensors and update what kind of haptic feedback to give accordingly. As noted, one sensor runs on hardware serial while the other runs on software serial, which I found gave a consistent and notably smoother result than putting both on software serial. To help the user get an actual sense of the distance, we increase both the frequency and intensity of the buzzing as we approach an object. Since one ultrasonic sensor faces slightly left and one faces slightly right, the user can then know if something is in front of them to one side or the other and how far away it is all based on just 2 vibration motors. When both buzzers go off, this means something is in front of the user, which will most commonly be the wall. The underwater ultrasonic sensors have a maximum distance of 6 meters, which we'll use as the starting point of when warnings begin. This begins with low power vibrations and less frequent feedback. By 1 meter, we provide full haptic feedback, which is full strength vibration with constant buzzing, since any wearer doing laps would need a good heads up on when to slow down to ensure they don't hit the wall. This can all be easily adjusted.

For our safety feature, we use an accelerometer and specifically look for when a user is face down with minimal movement for an extended period of time. In the 3d model of the enclosure's lid, you'll notice that it's rounded - this is to avoid false positives in this segment, since the enclosure then can't be left on and face down by accident. When a user is detected to be face down, we start a timer. When that timer reaches the first threshold, we begin haptic feedback. If the user remains face down, we send out an alert via Blues.

The Wearable Enclosure!

This brings us to the last part of the puzzle, the actual wearable. In the video, I made a point to make this out of materials anyone would have, but I do think that realistically people aren't going to want to be seen out and about wearing something like Tupperware even if it does have some logic to it. As such, let's put together something a bit nicer, shall we?

The non-3d modeled version of the haptic swim assistant

Thankfully, there's a solid starting place to ensure we get a nice result, so I want to credit this waterproof box and this adapter model for providing a surefire basis for this enclosure. You can find the latch and hinge models in the waterproof box link. For everything else, I'll be making modifications and will share those models within this project. As far as those modifications go, the top of the enclosure is rounded because it reduces the resistance as the user swims, but also because it is going to help us get false positives in our safety feature, as noted previously. This enclosure solution also offers a much easier and realistically feasible way to charge the power bank. The ultrasonic sensors will be able to neatly screw into place with the printed enclosure, but I would still recommend using some clear silicone sealant to ensure no water gets in.

The waterproof enclosure with places for each ultrasonic sensor to screw in

Conclusion

With that, we have our device fully online. It is something that the user can just start and use without any setup. It can warn the user with clear and intuitive feedback not just if there are obstacles, but also where they are and how far away they are. And, it can provide a safety net for the wearer by alerting others via text message if the user has been face down in the water for an extended period of time. Hopefully you enjoyed the read, and hopefully it helps someone out there.

Code

#include <HardwareSerial.h>
#include <Wire.h>
#include <Notecard.h>
#include <Adafruit_MPU6050.h>
#include <Adafruit_Sensor.h>
#include <SoftwareSerial.h>

#define PRODUCT_UID "<your project id here>"

Notecard notecard;
Adafruit_MPU6050 mpu;
unsigned char buffer_RTT[4] = { 0 };
uint8_t CS;
#define COM 0x55
HardwareSerial sensorLeft(2);  // Use UART2 for the first sensor
SoftwareSerial sensorRight(15, 4);

const int motorPinLeft = 18;
const int motorPinRight = 19;

int distanceLeft, distanceRight;

//Determined through collecting data with numerous tests -
const float thresholdXHigh = -0.35;  // Highest X
const float thresholdXLow = -2.25;   // Lowest X
const float thresholdYHigh = 0.3;    // Highest Y
const float thresholdYLow = -1.25;   // Lowest Y
const float thresholdZHigh = -8.4;   // Highest Z
const float thresholdZLow = -9.1;    // Lowest Z

const float thresholdRotXLow = -0.25;  // Example values, adjust as needed
const float thresholdRotXHigh = 0.25;
const float thresholdRotYLow = -0.25;
const float thresholdRotYHigh = 0.25;
const float thresholdRotZLow = -0.25;
const float thresholdRotZHigh = 0.25;

bool useFaceDownDetection = true;
bool faceDownHapticsRunning = false;
const unsigned long buzzDuration = 100;  // Duration of buzz in milliseconds
unsigned long nextBuzzLeft = 0;          //In millis
unsigned long nextBuzzRight = 0;
int intensityLeft = 0;
int intensityRight = 0;

// const unsigned long buzzDuration = 100;  // Duration of buzz in milliseconds
unsigned long buzzStartTimeLeft = 0;  // Tracks when buzzing starts for the left motor
unsigned long buzzStartTimeRight = 0;


void setup() {
  Serial.begin(115200);
  sensorLeft.begin(115200, SERIAL_8N1, 16, 17);  // Initialize Sensor1 on pins 16 (RX) and 17 (TX)
  sensorRight.begin(115200);

  Wire.begin();
  notecard.begin();

  pinMode(motorPinLeft, OUTPUT);
  pinMode(motorPinRight, OUTPUT);
  // mpu.initialize();
  if (!mpu.begin()) {
    Serial.println("Failed to find MPU6050 chip");
    useFaceDownDetection = false;
  }
  mpu.setAccelerometerRange(MPU6050_RANGE_8_G);
  mpu.setGyroRange(MPU6050_RANGE_500_DEG);
  mpu.setFilterBandwidth(MPU6050_BAND_21_HZ);

  Serial.println("Setup done");
}

void loop() {

  static unsigned long faceDownStartTime = 0;
  static bool faceDownAlertSent = false;

  distanceLeft = readSensor(sensorLeft, "left");
  if (distanceLeft > 0)
    updateHapticFeedback(distanceLeft, true);
  distanceRight = readSensor(sensorRight, "right");
  if (distanceRight > 0)
    updateHapticFeedback(distanceRight, false);
  provideHapticFeedback();

  if (useFaceDownDetection) {
    if (isFaceDown()) {
      if (faceDownStartTime == 0) {
        Serial.println("User is face down! Starting timer.");
        faceDownStartTime = millis();
        // Start different haptic feedback pattern for face down
      } else if ((millis() - faceDownStartTime > 60000) && !faceDownAlertSent) {
        faceDownHapticsRunning = true;

        analogWrite(motorPinLeft, 128);
        analogWrite(motorPinRight, 128);

        unsigned long timeElapsed = millis() - faceDownStartTime;
        unsigned long timeRemaining = 0;
        timeRemaining = 105000 - timeElapsed;
        Serial.print("User has been face down for ");
        Serial.print(timeElapsed);
        Serial.print(" ms. Time remaining before sending alert: ");
        Serial.println(timeRemaining);

        if (millis() - faceDownStartTime > 105000) {

          Serial.println("User is face down! Sending alert message!! ");
          sendMessageViaBlues("Haptic Swim Assistant Alert - USER IS FACE DOWN!!!!!");
          faceDownAlertSent = true;
        }
      } else if (!faceDownAlertSent) {
        Serial.print("User is currently face down ");
        Serial.print(millis() - faceDownStartTime);
        Serial.println(" milliseconds");
      }
    } else {
      if (faceDownHapticsRunning) {
        analogWrite(motorPinLeft, 0);  // Stop haptic feedback
        analogWrite(motorPinRight, 0);
      }
      faceDownStartTime = 0;
      faceDownAlertSent = false;
      faceDownHapticsRunning = false;
    }
  }
}

void sendMessageViaBlues(char *message) {

  J *req = notecard.newRequest("hub.set");
  if (req != NULL) {
    JAddStringToObject(req, "product", PRODUCT_UID);
    JAddStringToObject(req, "mode", "continuous");
    notecard.sendRequest(req);
  }

  req = notecard.newRequest("note.add");
  if (req != NULL) {
    JAddStringToObject(req, "file", "sensors.qo");
    JAddBoolToObject(req, "sync", true);
    J *body = JAddObjectToObject(req, "body");
    if (body) {
      JAddStringToObject(body, "message", message);
    }
    notecard.sendRequest(req);
  }
}

void provideHapticFeedback() {
  unsigned long currentTime = millis();

  // Handle left motor
  if (currentTime >= nextBuzzLeft && intensityLeft > 0 && buzzStartTimeLeft == 0) {
    Serial.println("Buzzing Left");
    analogWrite(motorPinLeft, intensityLeft);
    buzzStartTimeLeft = currentTime;  // Record start time of buzzing
    nextBuzzLeft = ULONG_MAX;         // Prevent immediate re-buzz
  } else if (buzzStartTimeLeft > 0 && (currentTime - buzzStartTimeLeft) >= buzzDuration) {
    Serial.println("Buzzing Left stopped");
    analogWrite(motorPinLeft, LOW);
    buzzStartTimeLeft = 0;  // Reset the start time
  }

  // Handle right motor
  if (currentTime >= nextBuzzRight && intensityRight > 0 && buzzStartTimeRight == 0) {
    analogWrite(motorPinRight, intensityRight);
    buzzStartTimeRight = currentTime;  // Record start time of buzzing
    nextBuzzRight = ULONG_MAX;         // Prevent immediate re-buzz
  } else if (buzzStartTimeRight > 0 && (currentTime - buzzStartTimeRight) >= buzzDuration) {
    analogWrite(motorPinRight, LOW);
    buzzStartTimeRight = 0;  // Reset the start time
  }
}

void updateHapticFeedback(long distance, bool isLeft) {
  unsigned long currentTime = millis();
  unsigned long nextBuzzTime;
  int intensity;
  if (distance > 600) {
    // No feedback if distance is more than 6 meters
    nextBuzzTime = ULONG_MAX;
    intensity = 0;
  } else if (distance <= 100) {
    // Full constant feedback at 1 meter or less
    nextBuzzTime = currentTime;
    intensity = 255;
  } else {
    // Gradually increase frequency and intensity as object gets closer
    long delayDuration = map(distance, 100, 600, 50, 500);
    nextBuzzTime = currentTime + delayDuration;
    intensity = map(distance, 100, 600, 255, 0);
  }

  if (isLeft) {
    if (nextBuzzLeft > nextBuzzTime || nextBuzzLeft <= 0) {
      nextBuzzLeft = nextBuzzTime;
    }
    intensityLeft = intensity;
  } else {
    if (nextBuzzRight > nextBuzzTime || nextBuzzRight <= 0) {
      nextBuzzRight = nextBuzzTime;
    }
    intensityRight = intensity;
  }
}

int readSensor(Stream &mySerial, String sensor) {
  mySerial.write(COM);
  delay(100);
  if (mySerial.available() > 0) {
    delay(4);
    if (mySerial.read() == 0xff) {
      buffer_RTT[0] = 0xff;
      for (int i = 1; i < 4; i++) {
        buffer_RTT[i] = mySerial.read();
      }
      CS = buffer_RTT[0] + buffer_RTT[1] + buffer_RTT[2];
      if (buffer_RTT[3] == CS) {
        int distance = (buffer_RTT[1] << 8) + buffer_RTT[2];
        return distance;
      }
    }
  }
  return 0;
}

bool isFaceDown() {
  sensors_event_t a, g, temp;
  mpu.getEvent(&a, &g, &temp);

  // Check if acceleration and rotation values are within thresholds
  bool xAccelInRange = (a.acceleration.x >= thresholdXLow) && (a.acceleration.x <= thresholdXHigh);
  bool yAccelInRange = (a.acceleration.y >= thresholdYLow) && (a.acceleration.y <= thresholdYHigh);
  bool zAccelInRange = (a.acceleration.z >= thresholdZLow) && (a.acceleration.z <= thresholdZHigh);

  bool xGyroInRange = (g.gyro.x >= thresholdRotXLow) && (g.gyro.x <= thresholdRotXHigh);
  bool yGyroInRange = (g.gyro.y >= thresholdRotYLow) && (g.gyro.y <= thresholdRotYHigh);
  bool zGyroInRange = (g.gyro.z >= thresholdRotZLow) && (g.gyro.z <= thresholdRotZHigh);

  // Combined check for face down
  bool isFaceDown = xAccelInRange && yAccelInRange && zAccelInRange && xGyroInRange && yGyroInRange && zGyroInRange;

  // Print only out-of-range values
  if (!xAccelInRange) Serial.println("Accel X out of range: " + String(a.acceleration.x));
  if (!yAccelInRange) Serial.println("Accel Y out of range: " + String(a.acceleration.y));
  if (!zAccelInRange) Serial.println("Accel Z out of range: " + String(a.acceleration.z));

  if (!xGyroInRange) Serial.println("Gyro X out of range: " + String(g.gyro.x));
  if (!yGyroInRange) Serial.println("Gyro Y out of range: " + String(g.gyro.y));
  if (!zGyroInRange) Serial.println("Gyro Z out of range: " + String(g.gyro.z));

  return isFaceDown;
}

#include <Wire.h>
#include <Notecard.h>

#define PRODUCT_UID "<put your project id here>"


Notecard notecard;

void setup() {
    Serial.begin(115200);
    while (!Serial);

    Wire.begin();
    notecard.begin();

    notecard.setDebugOutputStream(Serial);

    J *req = notecard.newRequest("hub.set");
    if (req != NULL) {
        JAddStringToObject(req, "product", PRODUCT_UID);
        JAddStringToObject(req, "mode", "continuous");
        notecard.sendRequest(req);
    }

    // Send a test message
    req = notecard.newRequest("note.add");
    if (req != NULL) {
        JAddStringToObject(req, "file", "sensors.qo");
        JAddBoolToObject(req, "sync", true);
        J *body = JAddObjectToObject(req, "body");
        if (body) {
            JAddStringToObject(body, "message", "Put in your test message here");
        }
        notecard.sendRequest(req);
    }
}

void loop() {
    // Do nothing here
}

Credits

donutsorelse

13 projects • 11 followers

I make different stuff every week of all kinds. Usually I make funny yet useful inventions.

Comments

Awards

Grand Prize Swimming

Build2gether Inclusive Innovation Challenge

Haptic Swim Assistant for Low Visibility Swimmers