Surface Cracks Detection and Notification Systems with Blues

A 50-year-old bridge collapsed in Pittsburgh (Pennsylvania) on January 28, 2022. There is only one reason why a sturdy structure such as a concrete bridge could suddenly collapse: wear and tear.

Concrete structures generally start deteriorating after about 40 to 50 years. For this reason, overlooking signs of wear can result in severe accidents, which is why the inspection and repair of concrete structures are crucial for safeguarding our way of life. Cracks are one of the important criteria used for diagnosing the deterioration of concrete structures. Typically, a specialist would inspect such structures by checking for cracks visually, sketching the results of the inspection, and then preparing inspection data based on their findings. An inspection method like this is not only very time-consuming and costly but it also cannot accurately detect cracks.

In this project, a mobile surface crack detection system is built using machine learning. A pre-trained image classification model is fine-tuned using Transfer Learning with the Edge Impulse Studio and deployed to the Raspberry Pi 4 to detect surface cracks in real-time and also localize them. The detected cracks images and the GPS coordinates are uploaded to Google Drive using the Blues Notecard.

Why localization?

Why do we want to localize the detection using an image classification model? Can't we use the object detection model? Yes, we can use the object detection model but we would need to add bounding boxes to thousands of samples manually. Existing object detection models may not be a good choice to auto-annotate these cracks since they are trained on definite-shaped objects. Repurposing the classification model for localizing the detection saves a lot of effort and still would be able to identify the regions of interest.

How does it work?

The CNN (convolutional neural networks) with GAP (Global Average Pooling) layers that have been trained for a classification task can also be used for object localization. GAP is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding category of the classification task in the last MLP conv layer. A GAP-CNN not only tells us what object is contained in the image - it also tells us where the object is in the image, and through no additional work on our part! The localization is expressed as a heat map (class activation map) where the color-coding scheme identifies regions that are relatively important for the GAP-CNN to perform the object identification task.

Hardware Setup

We will be using a Raspberry Pi 4 with a High-Quality Camera module for image capture and running the ML model.

The Blues Notecard will be used for LTE cellular connectivity that would be connected to the Raspberry Pi 4 using a Notecarrier Pi (Raspberry Pi Hat).

We often need to monitor infrastructure in the field, and a mobile robot could be a good option. We will use a DFRobot Devastator tracked mobile robot chassis and a DFRobot Romeo V2, an Arduino-compatible motor driver development board. Also, we will be using a DFRobot ESP8266 WiFi Bee that can be connected to the DFRobot Romeo V2 XBee socket.

For this project, we will control the robot using an M5Stack's M5StickC Plus (ESP32) and a Joystick Unit, although it would be great if the robot could operate autonomously (maybe a good idea for a future project!).

Data collection

The datasets were downloaded from Mendeley Data (Concrete Crack Images for Classification). The dataset contains various concrete surfaces with and without cracks. The data is collected from multiple METU Campus Buildings. The dataset is divided into negative and positive crack images for image classification. Each class has 20, 000 images with a total of 40, 000 images with 227 x 227 pixels with RGB channels.

To differentiate crack and non-crack surface images from the other natural world scenes, 25, 000 randomly sampled images for 80 object categories from the COCO-Minitrain, a subset of the COCO train2017 dataset, were downloaded. The data can be accessed from the links below.

• Surface Crack Dataset:https://data.mendeley.com/datasets/5y9wdsg2zt/2
• COCO-Minitrain dataset: https://github.com/giddyyupp/coco-minitrain

Uploading data to Edge Impulse Studio

We need to create a new project to upload data to Edge Impulse Studio

The data is uploaded using the Edge Impulse CLI. Please follow the instructions to install the CLI here: https://docs.edgeimpulse.com/docs/cli-installation.

The downloaded images are labeled into 3 classes and saved into the directories with the label name.

• Positive - surface with crack
• Negative - surface without crack
• Unknown - images from the 80 objects

Execute the following commands to upload the images to the Edge Impulse Studio. The datasets are automatically split into training and testing datasets.

$ edge-impulse-uploader --category split  --label positive positive/*.jpg
$ edge-impulse-uploader --category split  --label negative negative/*.jpg
$ edge-impulse-uploader --category split  --label unknown  unknown/*.jpg

We can see the uploaded datasets on the Edge Impulse Studio's Data Acquisition page.

Training

Go to the Impulse Design > Create Impulse page, click Add a processing block, and then choose Image, which preprocesses and normalizes image data, and optionally reduces the color depth. Also, on the same page, click Add a learning block, and choose Transfer Learning (Images), which fine-tunes a pre-trained image classification model on the data. We are using a 160x160 image size. Now click on the Save Impulse button.

Next, go to the Impulse Design > Image page and set the Color depth parameter as RGB, and click the Save parameters button which redirects to another page where we should click on the Generate Feature button. It usually takes a couple of minutes to complete feature generation.

We can see the 2D visualization of the generated features in Feature Explorer.

Now go to the Impulse Design > Transfer Learning page and choose the Neural Network architecture. We are using the MobileNetV2 160x160 1.0 transfer learning model with the pre-trained weight provided by the Edge Impulse Studio.

The pre-trained model outputs the class prediction probabilities. To get the class activation map, we need to modify the model and make it a multi-output model. To customize the model, we need to switch to Keras (expert) mode.

We can modify the generated code in the text editor as shown below.

We will connect the 2nd last layer which is a GAP layer to the Dense layer (a layer that is deeply connected with its preceding layer) with 3 neurons ( 3 classes in our case). We will be using this Dense layer's weights for generating a class activation map later.

base_model = tf.keras.applications.MobileNetV2(
    input_shape = INPUT_SHAPE, alpha=1,
    weights = WEIGHTS_PATH
)
last_layer  = base_model.layers[-2].output
dense_layer = Dense(classes)
output_pred = Softmax(name="prediction")(dense_layer(last_layer))

For the class activation map, we need to calculate the dot product of the last convolutional block output and the final dense layers' weight. The Keras Dot layer does not broadcast the multiplier vector with the dynamic batch size so we can not use it. But we can take advantage of the Dense layer which internally does the dot product of the kernel weight with the input. There is a side effect in this approach, the Dense layer adds up bias weight to each dot product. However, this bias weight is very small and does not change the final normalized values of the class activation map so we can use it without any problems.

conv_layer  = base_model.layers[-4].output
reshape_layer = Reshape((conv_layer.shape[1] * conv_layer.shape[2] , -1))(conv_layer)
dot_output = dense_layer(reshape_layer)

We need to resample the dot product output to the same size as the input image (160x160) so that we can overlay the heat map. We use the UpSampling2D layer for this purpose.

transpose = Permute((2, 1))(dot_output)
reshape_2_layer = Reshape((-1, conv_layer.shape[1] , conv_layer.shape[2]))(transpose)
SIZE = (int(INPUT_SHAPE[1] / conv_layer.shape[2]), 
 int(INPUT_SHAPE[0] / conv_layer.shape[1]))
output_act_map = UpSampling2D(size=SIZE, interpolation="bilinear", data_format="channels_first", name="activation_map")(reshape_2_layer)
model = Model(inputs=base_model.inputs, outputs=[output_pred, output_act_map])

Also, we will be training the model from the last two convolutional blocks and freezing all layers before that.

TRAINABLE_START_IDX = -12
for layer in model.layers[:TRAINABLE_START_IDX]:
    layer.trainable = False

The modified network architecture after the last convolutional block is given below. This is a multi-output model where the first output provides the prediction class probabilities and the second output provides the class activation map.

The full modified training code is as follows.

import math
from pathlib import Path
import tensorflow as tf
from tensorflow.keras import Model
from tensorflow.keras.layers import Dense, UpSampling2D, Permute, Reshape, Softmax
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import categorical_crossentropy

sys.path.append('./resources/libraries')
import ei_tensorflow.training

WEIGHTS_PATH = './transfer-learning-weights/keras/mobilenet_v2_weights_tf_dim_ordering_tf_kernels_1.0_160.h5'

INPUT_SHAPE = (160, 160, 3)

base_model = tf.keras.applications.MobileNetV2(
    input_shape = INPUT_SHAPE, alpha=1,
    weights = WEIGHTS_PATH
)

last_layer  = base_model.layers[-2].output
dense_layer = Dense(classes)
output_pred = Softmax(name="prediction")(dense_layer(last_layer))


conv_layer  = base_model.layers[-4].output
reshape_layer = Reshape((conv_layer.shape[1] * conv_layer.shape[2] , -1))(conv_layer)

dot_output = dense_layer(reshape_layer)
transpose = Permute((2, 1))(dot_output)
reshape_2_layer = Reshape((-1, conv_layer.shape[1] , conv_layer.shape[2]))(transpose)

SIZE = (int(INPUT_SHAPE[1] / conv_layer.shape[2]), 
 int(INPUT_SHAPE[0] / conv_layer.shape[1]))

output_act_map = UpSampling2D(size=SIZE, interpolation="bilinear", data_format="channels_first", name="activation_map")(reshape_2_layer)

model = Model(inputs=base_model.inputs, outputs=[output_pred, output_act_map])

TRAINABLE_START_IDX = -12
for layer in model.layers[:TRAINABLE_START_IDX]:
    layer.trainable = False

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005),
      loss={'prediction': 'categorical_crossentropy', 'activation_map': None},
      metrics={'prediction': ['accuracy'], 'activation_map': [None]})
                
BATCH_SIZE = 32
EPOCHS=5

train_dataset = train_dataset.batch(BATCH_SIZE, drop_remainder=False)

validation_dataset = validation_dataset.batch(BATCH_SIZE, drop_remainder=False)

callbacks.append(BatchLoggerCallback(BATCH_SIZE, train_sample_count, epochs=EPOCHS))

model.fit(train_dataset, validation_data=validation_dataset, epochs=EPOCHS, verbose=2, callbacks=callbacks)

Now click the Start Training button and wait until the training is completed. We can see the Training output below. The quantized (int8) model has 99.6% accuracy which is pretty good.

Model Deployment

Currently, Edge Impulse for Linux SDK does not support a multi-output model so we will be using the compiled TensorFlow Lite runtime for inferencing. This interpreter-only package is a fraction of the size of the complete TensorFlow package and includes the bare minimum code required to run inferences with TensorFlow Lite. To accelerate the inferencing, the TFLite interpreter can be used with XNNPACK which is a highly optimized library of neural network inference operators for ARM, and other platforms. To enable XNNPACK for 64-bit Raspberry Pi OS, we need to build the TFLite Runtime Python package from the source. We will need to execute the following commands on a faster Debian/Ubuntu Linux machine with Docker to cross-compile and build the package.

$ git clone -b v2.16.1 https://github.com/tensorflow/tensorflow.git

Enable XNNPACK support in the tensorflow/lite/tools/pip_package/build_pip_package_with_cmake.sh file.

aarch64)
...
-DTFLITE_ENABLE_XNNPACK=ON \
...
;;

Execute the following command to build it.

$ make BASE_IMAGE=debian:bookworm PYTHON_VERSION=3.11 TENSORFLOW_TARGET=aarch64 docker-build

After building is completed successfully, copy the generated pip package (tensorflow/lite/tools/pip_package/gen/tflite_pip/python3.11/dist/tflite_runtime-2.16.1-cp311-cp311-linux_aarch64.whl) to the Raspberry Pi 4 and install it using the command below.

$ python3 -m pip install tflite_runtime-2.16.1-cp311-cp311-linux_aarch64.whl

Now We can download the quantized model from the Edge Impulse Studio Dashboard.

Set up Notehub

We need to set up Notehub, which is a cloud service that receives data from the Notecard and allows us to manage the device, and route that data to our own or 3rd party cloud apps and services. We can create a free account at https://notehub.io/sign-up and after successful login, we can create a new project.

We should copy the ProductUID which is used by Notehub to associate the Notecard to the project created.

Notecard Configuration

There are many options to configure a Notecard. Since we are using a Raspberry Pi 4, we can configure it using a Python script. Before we begin, make sure to install some prerequisites.

$ sudo apt install python3-periphery
$ pip3 install note-python

Please save the below code as configure_notecard.py.

from notecard import notecard
from periphery import I2C

productUID = 'com.xxx.xxx:surface_crack_detection'

try:
    card = notecard.OpenI2C(I2C('/dev/i2c-1'), 0, 0, debug=True)
    req = {
      'req': 'hub.set',
      'product': productUID,
      'mode': 'continuous',
       'sync': True
    }
    card.Transaction(req)
except Exception as e:
    print(e)

And execute the script as follows.

$ python3 configure_notecard.py

Resetting Notecard I2C communications.
Using transaction timeout of 30 seconds.
{"req":"hub.set","product":"com.xxx.xxx:surface_crack_detection","mode":"continuous","sync":true,"body":{"agent":"note-python","os_name":"cpython","os_platform":"linux","os_version":"3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]","os_family":"posix","req_interface":"i2c","req_port":0},"crc":"0000:7b8152f8"}
{'crc': '0000:A3A6BF43'}

Route Configuration

We will be uploading the detected cracks image data with GPS coordinates to Google Drive. We would need to create an account for the Google Cloud Platform and setup credentials by creating a new project and enabling Google Drive service for that project.

Now go to the Credentials > Create Credentials and create an OAuth Client ID. For the authorized redirect URIs, we can set it as http://127.0.0.1 for testing. After setting it up, we should copy the Client ID and the Client secrets.

Now we need to access the following URL using a web browser for authorization. In a production environment, we would need a dedicated domain name and web server to automate this process, but for testing purposes, we will do it manually.

https://accounts.google.com/o/oauth2/auth?client_id=<Client ID>&redirect_uri=http://127.0.0.1&scope=https://www.googleapis.com/auth/drive.file&response_type=code&include_granted_scopes=true&access_type=offline

The browser would redirect us to the following URL.

http://127.0.0.1/?code=<CODE>&scope=https://www.googleapis.com/auth/drive.file

We need to copy the CODE from the URL above and execute the following command to generate credentials that we will be using while making web request from the Blues Notecard. The generated access_token get expires in 60 mins so we will be using the refresh_token to regenerate it from the Notecard automatically.

$ curl -X POST https://oauth2.googleapis.com/token -d "code=<CODE>&client_id=<Client ID>&client_secret=<Client secrets>&redirect_uri=http://127.0.0.1&access_type=offline&grant_type=authorization_code"

{
  "access_token": “<access token>”,
  "expires_in": 3599,
  "refresh_token": “<refresh token>,
  "token_type": "Bearer"
}

Now we can create a route by clicking the Create Route link at the top right on the Notehub Route page. We will select the Proxy for Notecard Web Requests route.

We need to create two Routes as mentioned above, one for Google Drive OAuth2 access token generation and another for uploading data to Google Drive as shown in the images below. Please make note of the Alias column for the routes that we will be using in the code to refer to them.

Route 1:

Route 2:

Application

The application uses multithreading to use all available 4-cores on the Raspberry Pi 4 to achieve low latency and better FPS. The inferencing application is a multithreaded Python script that offloads the display task to a dedicated thread. The heatmap is overlaid with the input images whenever a crack is detected. Some test run results' images are given below.

The application uses Blues Notecard official Python library to perform requests and obtain responses from 3rd-party APIs and services with the Web API requests. The application detects the surface cracks and uploads the 160x160 pixels cracks image data with GPS coordinates to the Google Drive. Since sending large images consumes more time and incurs additional charges, we have opted to send a smaller image that adequately verifies the cracks in the infrastructure and can be precisely located with the corresponding location information. The Google Drive API allows to send metadata with image data in a single request as a multipart/related content-type. The access token can be sent either as a header or as part of the request URL. We chose the latter option so that we wouldn’t need to make another web request every 60 minutes to modify the Notehub proxy route headers column when the access token is changed. Also, we are using the Notecard binary data functionality to send complete multipart payload. The location data is sent as a description to the Google Drive.

Below is the complete code for the inferencing and sending image/location payload.

import sys
import os
import json
import time
import uuid
import cv2
import numpy as np
import traceback
import threading
import logging
import queue
import collections
import matplotlib.pyplot as plt
import geopy.distance

from matplotlib import cm
from periphery import I2C
from tflite_runtime.interpreter import Interpreter
from picamera2 import Picamera2
from urllib3.filepost import encode_multipart_formdata, choose_boundary
from urllib3.fields import RequestField
from notecard import notecard, binary_helpers
from Stream import StreamingOutput, StreamingHandler, StreamingServer

productUID = 'com.xxx.xxx:surface_crack_detection'
gdClientID = '<client-id>'
gdClientSecret = '<secret>'
gdRefreshToken = '<refresh-token>'
gdAccessToken = None
gdLastAccessTokenTime = 0
upload = True

def NotecardExceptionInfo(exception):
    s1 = '{}'.format(sys.exc_info()[-1].tb_lineno)
    s2 = exception.__class__.__name__
    return "line " + s1 + ": " + s2 + ": " + ' '.join(map(str, exception.args))

def configure_notecard(card):
    req = {
      'req': 'hub.set',
      'product': productUID,
      'mode': 'continuous'
    }
    try:
        card.Transaction(req)
    except Exception as exception:
        logging.error(f'Transaction error: {NotecardExceptionInfo(exception)}')
    time.sleep(1)

def get_notecard_location():
    req = {
        'req': 'card.location'
    }
    location = None

    try:
        rsp = card.Transaction(req)
        location = (rsp['lat'], rsp['lon'])
    except Exception as exception:
        logging.error(f'Transaction error: {NotecardExceptionInfo(exception)}')
    time.sleep(1)
    return location

def get_notecard_timestamp():
    req = {
        'req': 'card.time'
    }
    ts = None

    try:
        rsp = card.Transaction(req)
        ts = rsp['time']
    except Exception as exception:
        logging.error(f'Transaction error: {NotecardExceptionInfo(exception)}')
    time.sleep(1)
    return ts

def refresh_access_token():
    req = {
      'req': 'web.post',
      'route': 'GoogleDriveAccessToken',
      'body': {
        'client_id': gdClientID,
        'client_secret': gdClientSecret,
        'refresh_token': gdRefreshToken,
        'grant_type': 'refresh_token'
    }
}

access_token = None
try:
    rsp = card.Transaction(req)
    access_token = rsp['body']['access_token']
except Exception as exception:
    logging.error(f'Transaction error: {NotecardExceptionInfo(exception)}')
    time.sleep(1)
return access_token

def build_payload(filename, img_data, description):
    rf1 = RequestField(
      name = 'metadata',
      data = json.dumps({'name': filename, 'description': description}),
      headers = {'Content-Disposition': 'form-data; name="metadata"', 'Content-Type': 'application/json; charset=UTF-8'}
    )

    rf2 = RequestField(
      name = 'file',
      data = img_data,
      headers={'Content-Disposition': 'form-data; name="file"', 'Content-Type': 'image/jpeg'}
    )

    boundary = choose_boundary()
    payload, _ = encode_multipart_formdata((rf1, rf2), boundary)
    return payload, boundary

def upload_to_google_drive(gdAccessToken, payload, boundary):
    req = {
      'req': 'web.post',
      'route': 'GoogleDriveFileUpload',
      'name': f'/files?access_token={gdAccessToken}&uploadType=multipart',
      'content': f'multipart/related; boundary={boundary}',
      'binary': True
    }

    rsp = None  
    try:
        rsp = card.Transaction(req)
    except Exception as exception:
        logging.error(f'Transaction error: {NotecardExceptionInfo(exception)}')
        time.sleep(1)
    return rsp

def notecard_binary_upload(buf):
    location = get_notecard_location()
    global prev_location
    logging.info(f'{prev_location} {location}')
    
    if prev_location is not None:
        distance = geopy.distance.geodesic(prev_location, location).m
        logging.info(f'distance = {distance}')
    
    if distance < 10:
        logging.warning(f'Seems previous location, returning')
        return

    if get_notecard_timestamp() > gdLastAccessTokenTime + 3600:
        gdAccessToken = refresh_access_token()
    
    filename = f'img_{uuid.uuid4()}.jpg'
    description = f'lattitude:{location[0]}, longitude:{location[1]}'
    payload, boundary = build_payload(filename, buf, description)
    binary_helpers.binary_store_transmit(card, payload, 0)
    upload_to_google_drive(gdAccessToken, payload, boundary)
    binary_helpers.binary_store_reset(card)
    prev_location = location

def capture(queueIn):
    picam2 = Picamera2()
    picam2.configure(picam2.create_preview_configuration(main={"format": 'RGB888', "size": (640, 640)}))
    picam2.set_controls({"FrameRate": 40})
    picam2.start()

    while True:
        try:
            img = picam2.capture_array()
            img = cv2.resize(img, (width, height))
            img_scaled = ((img / 255.0).astype(np.float32) / input_scale) + input_zero_point
            input_data = np.expand_dims(img_scaled, axis=0).astype(input_details[0]["dtype"])
            if not queueIn.full():
                queueIn.put((img, input_data))
            logging.debug('Image Captured')
        except Exception as inst:
            logging.error("Exception", inst)
            logging.error(traceback.format_exc())
            break

def inferencing(interpreter, queueIn, queueOut):
    while True:
        start_time = time.time()
        try:
            if queueIn.empty():
                time.sleep(0.01)
                continue

        img, input_data = queueIn.get()
        interpreter.set_tensor(input_details[0]['index'], input_data)
        interpreter.invoke()
        output_0_tensor = interpreter.tensor(output_details[0]['index'])
        output_1_tensor = interpreter.tensor(output_details[1]['index'])
        output_1 = output_1_scale * ((output_1_tensor()).astype(np.float32) - output_1_zero_point)

        pred_class = np.argmax(np.squeeze(output_1))
        pred_score = np.squeeze(output_1)[pred_class]

        dp_out = None
        if pred_class == 1:
            dp_out = output_0_scale * (np.squeeze(output_0_tensor())[pred_class].astype(np.float32) - output_0_zero_point)
        
        if not queueOut.full():
            queueOut.put((img, pred_class, pred_score, dp_out))
        except Exception as inst:
            logging.error("Exception", inst)
            logging.error(traceback.format_exc())
            break

        logging.debug('Inferencing time: {:.3f}ms'.format((time.time() - start_time) * 1000))

def display(streamOut, queueOut):
    while True:
        if queueOut.empty():
            time.sleep(0.01)
            continue

        start_time = time.time()
        img, pred_class, pred_score, dp_out = queueOut.get()

        if pred_class == 1:
            label = 'Crack'
            color = (0, 0, 255)
            if dp_out is not None:
                heatmap = None
                heatmap = cv2.normalize(dp_out, heatmap, alpha=0, beta=255, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_8U)
                colormap = plt.get_cmap('jet')
                heatmap_img = cv2.addWeighted(img.astype(np.float32), 1.0, colormap(heatmap).astype(np.float32)[:,:,:3], 0.4, 0)
                heatmap_img = heatmap_img * 255.0
                heatmap_img = heatmap_img.astype(np.uint8)
        else:
            if pred_class == 0:
                label = 'No Crack'
                color = (0, 0, 0)
            else:
                label = 'Unknown'
                color = (255, 0, 0)

        logging.info(f'Prediction: {label}, Score: {pred_score}')
        _, buf = cv2.imencode('.jpg', img)
        
        if pred_class == 1 and pred_score > 0.55:
            _, heatmap_buf = cv2.imencode('.jpg', heatmap_img)
            streamOut.write(heatmap_buf)
            notecard_binary_upload(buf)
        else:
            streamOut.write(buf)

if __name__ == '__main__':
    log_fmt = "%(asctime)s: %(message)s"
    logging.basicConfig(format=log_fmt, level=logging.INFO, datefmt="%H:%M:%S")
    prev_location = None

    try:
        card = notecard.OpenI2C(I2C('/dev/i2c-1'), 0, 0, debug=True)
    except Exception as exception:
        raise Exception(f'Error: {NotecardExceptionInfo(exception)}')

    configure_notecard(card)
    binary_helpers.binary_store_reset(card)

    model_file = './model/quantized-model.lite'
    interpreter = Interpreter(model_path=model_file, num_threads=2)
    interpreter.allocate_tensors()

    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()
    height = input_details[0]['shape'][1]
    width = input_details[0]['shape'][2]

    input_scale, input_zero_point = input_details[0]['quantization']
    output_0_scale, output_0_zero_point = output_details[0]['quantization']
    output_1_scale, output_1_zero_point = output_details[1]['quantization']
    
    queueIn = queue.Queue(maxsize=1)
    queueOut = queue.Queue(maxsize=1)

    streamOut = StreamingOutput()
    StreamingHandler.set_stream_output(streamOut)
    StreamingHandler.set_page('Surface Crack Detection with Blues', 320, 320 )
    server = StreamingServer(('', 8889), StreamingHandler)

    t1 = threading.Thread(target=capture, args=(queueIn,), daemon=True)
    t2 = threading.Thread(target=inferencing, args=(interpreter, queueIn, queueOut), daemon=True)
    t3 = threading.Thread(target=display, args=(streamOut, queueOut), daemon=True)
    t1.start()
    t2.start()
    t3.start()
    
    try:
        logging.debug("Server started at 0.0.0.0:8889")
        server.serve_forever()
    except KeyboardInterrupt:
        doCapture = False

Driving the Robot

We need to install the Arduino IDE to build and upload sketches. There are 3 sketches to control the robot.

Sketch for driving the motors and should be uploaded to DFRobot Romeo V2:

#include <ArduinoJson.h>

int E1 = 5;  // M1 Speed Control
int E2 = 6;  // M2 Speed Control
int M1 = 7;  // M1 Direction Control
int M2 = 4;  // M1 Direction Control
bool motor_stopped = true;

void stop(void)
{
  digitalWrite(E1, 0);
  digitalWrite(M1, LOW);
  digitalWrite(E2, 0);
  digitalWrite(M2, LOW);
}

void forward(uint8_t a, uint8_t b)
{
  analogWrite (E1, a);
  digitalWrite(M1, HIGH);
  analogWrite (E2, b);
  digitalWrite(M2, HIGH);
}

void backward (uint8_t a, uint8_t b)
{
  analogWrite (E1, a);
  digitalWrite(M1, LOW);
  analogWrite (E2, b);
  digitalWrite(M2, LOW);
}

void left (uint8_t a, uint8_t b)
{
  analogWrite (E1, a);
  digitalWrite(M1, LOW);
  analogWrite (E2, b);
  digitalWrite(M2, HIGH);
}

void right (uint8_t a, uint8_t b)
{
  analogWrite (E1, a);
  digitalWrite(M1, HIGH);
  analogWrite (E2, b);
  digitalWrite(M2, LOW);
}

void setup(void)
{
  Serial.begin(115200);
  Serial1.begin(115200);
  pinMode(4, OUTPUT);
  pinMode(5, OUTPUT);
  pinMode(6, OUTPUT);
  pinMode(7, OUTPUT);
  digitalWrite(E1, LOW);
  digitalWrite(E2, LOW);
}

void loop(void)
{
  if (Serial1.available()) {
    JsonDocument doc;
    deserializeJson(doc, Serial1);
    uint8_t cmd = doc["cmd"];
    uint8_t speed = doc["speed"];
    uint8_t enable = doc["enable"];
    uint8_t speed_value = 0;

    if (speed == 0) {
      if (!motor_stopped) {
        stop();
        motor_stopped = true;
        speed_value = 0;
      }
    }

    if (speed == 1) {
      speed_value = 100;
    }

    if (speed == 2) {
      speed_value = 175;
    }
  
    if (speed == 3) {
      speed_value = 255;
    }

    if (enable == 0) {
      if (!motor_stopped) {
        stop();
        motor_stopped = true; 
      }
    }

    if (enable == 1) {  
      if (cmd == 1) {
        forward(speed_value, speed_value);
      }
      if (cmd == 2) {
        left(speed_value, speed_value);
      }
      if (cmd == 3) {
        backward(speed_value, speed_value);
      }
      if (cmd == 4) {
        right(speed_value, speed_value);
      }
      motor_stopped = false;
    }
  }
}

Remote Control

We will be using ESP-NOW to communicate the ESP32 to ESP8266. ESP-NOW is a wireless communication protocol defined by Espressif, which enables the direct peer-to-peer communication without the need of a router. We need to install the Arduino Core for ESP8266 and ESP32 to build and upload the following sketches.

• Sketch for the DFRobot ESP8266 WiFi Bee (receiver) that is mounted on the Romeo v2.

#include <ESP8266WiFi.h>
#include <espnow.h>
#include <ArduinoJson.h>

typedef struct {
  uint8_t cmd;
  uint8_t speed;
  uint8_t enable;
} Message;

Message msg;

void OnDataRecv(uint8_t * mac, uint8_t *incomingData, uint8_t len) {
  memcpy(&msg, incomingData, sizeof(msg));
  JsonDocument doc;
  doc["cmd"] = msg.cmd;
  doc["speed"] = msg.speed;
  doc["enable"] = msg.enable;
  serializeJson(doc, Serial);
}

void setup() {
  Serial.begin(115200);
  WiFi.mode(WIFI_STA);
  if (esp_now_init() != 0) {
    Serial.println("Error initializing ESP-NOW");
    return;
  }

  esp_now_set_self_role(ESP_NOW_ROLE_SLAVE);
  esp_now_register_recv_cb(OnDataRecv);
}

void loop() {
}

• Sketch for the M5StickC Plus (transmitter):

#include <esp_now.h>
#include <WiFi.h>
#include <M5StickCPlus.h>
#include <Wire.h>

#define I2C_ADDR 0x52
#define OFFSET 30

class DebounceButton
{
  private:
  unsigned long debounceDelay;
  unsigned long lastDebounceTime;
  int lastButtonState;
  
  public:
    DebounceButton(unsigned long debounceDelay, int lastButtonState) {
      this->debounceDelay = debounceDelay;
      this->lastButtonState = lastButtonState;
      lastDebounceTime = 0;
    }

    bool debounce(int reading) {
      if ((millis() - lastDebounceTime) > debounceDelay) {
        if (reading != lastButtonState) {
          lastDebounceTime = millis();
          return false;
        }
      }
      lastButtonState = reading;
      return true;
    }
};

DebounceButton button(100, 0);
uint8_t broadcastAddress[] = {0x84, 0xCC, 0xA8, 0xA6, 0xAB, 0x6c};

typedef struct {
uint8_t cmd;
uint8_t speed;
uint8_t enable;
} Message;

Message msg;
esp_now_peer_info_t peerInfo;
uint8_t packet_sent_status;
char *cmd_str[5] = {"Stop", "Forward", "Left", "Backward", "Right"};
char *speed_str[4] = {"0", "Slow", "Medium", "High"};

void OnDataSent(const uint8_t *mac_addr, esp_now_send_status_t status) {
  packet_sent_status = status;
}

void setup() {
  M5.begin();
  Wire.begin();
  M5.Lcd.setRotation(3);
  WiFi.mode(WIFI_STA);
  
  if (esp_now_init() != ESP_OK) {
    Serial.println("Error initializing ESP-NOW");
    return;
  }

  esp_now_register_send_cb(OnDataSent);
  memcpy(peerInfo.peer_addr, broadcastAddress, 6);
  peerInfo.channel = 0;
  peerInfo.encrypt = false;

  if (esp_now_add_peer(&peerInfo) != ESP_OK) {
    Serial.println("Failed to add peer");
    return;
  }
}

void loop() {
  Wire.requestFrom(I2C_ADDR, 3);
  if (Wire.available()) {
    uint8_t x_reading = Wire.read();
    uint8_t y_reading = Wire.read();
    uint8_t button_reading = Wire.read();
    float r = abs(sqrt(sq(x_reading - 127) + sq(y_reading - 127)));
    float theta = atan2(y_reading - 127, x_reading - 127) * (180.0 / PI);
    
    if (theta < 0) {
      theta += 360;
    }

    msg.cmd = 0x00;
    msg.speed = 0x00;

    if (r > 27 ) {
       if ( theta > 90 - OFFSET && theta < 90 + OFFSET) {
           msg.cmd = 0x01; // forward
       }

      if ( theta > 180 - OFFSET && theta < 180 + OFFSET) {
        msg.cmd = 0x02; // left
      }
      if ( theta > 270 - OFFSET && theta < 270 + OFFSET) {
        msg.cmd = 0x03; // backward
      }

      if ( (theta > 360 - OFFSET && theta <= 360 ) || (theta >= 0 && theta < 0 + OFFSET) ) {
        msg.cmd = 0x04; // right 
      }
    }

    if (r > 27 && r <= 60) {
      msg.speed = 0x01; // slow
    }
    if (r > 60 && r <= 100) {
      msg.speed = 0x02; // medium
    }
    if (r > 100) {
      msg.speed = 0x03; // high
    }
    if (false == button.debounce(button_reading)) {
      if ( button_reading == 1) {
        msg.enable = (msg.enable == 0x0) ? 0x1 : 0x0;
      }
    }

   M5.Lcd.setTextColor(TFT_WHITE, TFT_BLACK);
   M5.Lcd.setCursor(1, 10, 4);
   M5.Lcd.printf("Cmd: %-16s", cmd_str[msg.cmd]);
   M5.Lcd.setCursor(1, 40, 4);
   M5.Lcd.printf("Speed: %-18s", speed_str[msg.speed]);
   M5.Lcd.setCursor(1, 70, 4);
   M5.Lcd.printf("Enabled: %-6s\n", msg.enable ? "Yes" : "No");
   M5.Lcd.setCursor(1, 100, 4);
   M5.Lcd.printf("Status: ");
   
   if (packet_sent_status == ESP_NOW_SEND_SUCCESS) {
     M5.Lcd.setTextColor(TFT_GREEN, TFT_BLACK);
     M5.Lcd.printf("%-16s", "Success");
   } else {
     M5.Lcd.setTextColor(TFT_RED, TFT_BLACK);
     M5.Lcd.printf("%-16s", "Fail");
   }

   esp_err_t result = esp_now_send(broadcastAddress, (uint8_t *) &msg, sizeof(msg));
  }
  delay(50);
}

Live Demo

Conclusion

This project showcases a use case for the surface cracks detection that can be used for predictive maintenance. The project has the following key characteristics.

• Customize the pre-trained transfer learning model
• Demonstrate the use of a multi-output model
• Runtime heat-map visualization to localize the detected cracks.
• An affordable, low-powered, and reliable cellular network notification system

Please leave comments, and feel free to share any suggestions to enhance this project. Thank you for reading this far!