Pothole Detection with Blues Wireless and Edge Impulse

Introduction

We've all been there; a morning rush to the office and suddenly you feel the harsh dip of your car and the resulting suspension strain. Your coffee spills a little, your patience is overly tested, and you curse the existence of potholes. Why are they so easily missable and dangerous but yet so prevalent? May it be from a lack of drainage, or recurrent freezing and thawing, accumulation of water causes weak spots under the road that ultimately deteriorate to potholes.

Ideally, your city's potholes are promptly detected and paved over. However, you're more than likely to spar with the same potholes on many trips to the office before it's brought to attention. American insurance company, AAA, estimates that Americans approximately paid 26.5 billion USD in repairs from pothole-related damage in 2021 (1). Another article estimates that the death toll in India from pothole-related accidents to be over 5000 (2). Given the consequences, ranging from minor inconveniences to serious injury to cars and drivers, potholes should dealt with as quickly as possible.

To this end, many cities offer a portal or a phone number with which a report could be made. However, exact coordinates can be difficult to manually acquire for the average person, especially while driving. Utilizing the Blues Wireless Notecard, this device conducts automatic detection of potholes to enable more prompt and accurate reporting. In real time, as you drive, this device senses potholes by proxy of your car's changes in acceleration, saves the according GPS location, and sends the data off to Notehub. As such, it may serve as a first step to a potential streamlining of early detection, reporting, and repairing of potholes.

Training with Edge Impulse

Edge Impulse was used to conduct data collection and train the pothole detection model. When a car comes into contact with a pothole, its z-axis acceleration decreases. Edge Impulse was thereby used to allow the average smartphone to be utilized as a portable accelerometer; enabling pothole identification via car motion data.

Edge Impulse includes a suite of different methods to collect and import data of all types. By selecting 'Connect a New Device', you are prompted with a QR code with which to pair a mobile device to Edge Impulse. You are then able to select what kind of data you would to capture.

Training data was collected from a vehicle driven on different road conditions. This included repeatedly driving over confirmed potholes. The smartphone was placed and relatively leveled at the centre of the car's dashboard. The length of data capture was adjusted based on the different environments. Data capture was shorter for confirmed potholes and longer for smooth driving. Potholes were hit repeatedly from different angles and vehicle sides to produce varied results. A major concern were false positives; slight bumps, such as ground irregularities and speed bumps, would trigger pothole classification. Calibration was therefore conducted by driving the vehicle on roads of different smoothness and uniformity (ie. debris, speed bumps, different weather conditions).

The Project Setup

The project setup was made pleasantly simple with the use of Blues Wireless 'Blues Starter Kit', which included the Notecarrier-F, cellular notecard, and Swan 3.0 with both antennae attached.

Additionally, the MPU (GY-521) must be connected such as below:

Notecarrier-F ------ MPU

GND <----------------> GND

F_3V3 <-------------> VCC

F_SDA <-------------> SDA

F_SCL <-------------> SCL

An optional open-top enclosure can be 3D printed for a neater design.

When the project is cloned from Github,

git clone git@github.com:michaelmalinowski/pothole-detection.git

You should add it to the platform IO project list by selecting 'Add Existing',

Once the project is added, you can perform different actions such as Building, Uploading, and Monitoring the code, as seen by the menu below.

For more detailed instructions on how to use the Swan 3.0, please refer to this link: https://dev.blues.io/quickstart/swan-quickstart/#using-the-vs-code-platformio-extension

Connection with NoteHub

Within the code, it is important to add your product ID here to allow Notehub connections.

#define PRODUCT_UID "<product_id>"

Optionally, location data of the pothole can be sent to a default file location `pothole-location.qo`, this can also be changed as such:

#define NOTEHUB_LOCATION_FILE "pothole-location.qo"

How It Works

The foundation of this device is the celluar note card, for its fundamental features of cellular connection and GPS functionalities.

Our device first syncs to the Notehub to verify a project UID and request local time, which is a critical component for location functionality.

J *req = NoteNewRequest("hub.sync");
NoteRequest(req);

In order to retrieve accurate GPS coordinates at any moment, the board's location mode must be set to 'Continuous'. The Notecard library offers complete functions for certain requests, such as setting the location mode:

bool success = NoteSetLocationMode("continuous", 0);

The next step is to activate the IMU. A library, Motion.h, has been created for convenient set up of the IMU. The library, by default, is set to interact with registers from the GY-521 MPU. The IMU is started and the acceleration sensitivity is determined with an integer ranging from 0 to 3. Briefly, 0 represents ±2 gravities (g), 1 represents ±4 g, 2 represents ±8 g, and 3 represents ±16 g.

motion.start(0);

Calibration is required with this accelerometer as the default value on a flat surface reads ±1000, which is a large discrepancy. You can determine your IMU's base readings by running:

motion.updateMotionData(print=true);

This will print the raw readings to the serial port and those values can be set respectively below:

motion.setAccelDrift(XaccelCorrection,YaccelCorrection,ZaccelCorrection);

The raw IMU data is read with the function and stored within the object Motion:

motion.updateMotionData();

And the acceleration data in gravities (g) is copied into an array with:

motion.getAccelDataInGravities(accelModifiedData);

The acceleration data in gravities (g) is used in the function:

hasHitPothole();

which returns a boolean. The Edge Impulse model is utilized in this function with a 200 millisecond (ms) time frame that allows twelve accelerometer reads to constitute a pothole classification. A minimum threshold of 90% accuracy has to be reached for a pothole to register. Upon pothole detection, a note is created locally using the NOTEHUB_LOCATION_FILE variable. Since continuous GPS and sync cannot be active, the function below constantly checks for a lack of motion for a set duration at set intervals. When the device is stable, a Notehub sync is triggered, saving notes on there.

bool hasStablized(float duration, float interval);

The image below represents pre- and post-pothole accelerometer data; in other words, milliseconds (ms) before hitting a pothole to the estimated recovery. The two disturbances (at approximately 32 and 224 ms; x-axis) represent the front and rear wheels of the vehicle driving over a pothole. The blue line indicates the sudden vehicle vertical drop (Z axis change in acceleration; accZ) with a quick rebound. The green line represents the sudden jolt on the side of the vehicle that had hit the pothole (Y axis change in acceleration; accY). As contact with a pothole slows the vehicle, the red line represents the acceleration opposing the direction of travel (X axis change in acceleration; accX).

The diagram below illustrates the aforementioned acceleration variables, whereby the colours match and correspond to Accelerometer data figure above.

Do More With Notehub

Using Notehub, we can analyze or forward the location notes collected. Location notes offer exact coordinates, which can be sent to your city's pothole reporting resources or be forwarded through a created route in Notehub.

The location data via the latter route can be sent to a database to be queried with any type of application, such as Google Maps. To demonstrate how the pothole data can be utilize with routes, the web application in the photo below displays all pothole locations on a map.

References

1. https://media.acg.aaa.com/aaa-offers-safety-tips-for-dealing-with-potholes.htm

2. https://www.indiatimes.com/trending/social-relevance/over-5000-people-died-by-potholes-accidents-between-2018-2020-577821.html