How to deploy CNN model on ESP32

This project implements a lightweight convolutional neural network (CNN) for handwritten digit recognition (0–9) on an ESP32 microcontroller as shown in figure below:

The system processes input data received via serial (UART) and displays results on a 16x2 LCD. Key components include:

• A PyTorch-trained CNN model for digit classification
• ESP32 firmware for performing inference on 100 float inputs

1. Requirements and Setup

This project uses ESP-IDF version 5.4.1 for the ESP32 and Python 3.9.21 with the following dependencies:

numpy==1.26.4
opencv-python==4.7.0
pyserial==3.5
torch==2.0.1
tourchvision==0.15.2

The MNIST dataset in JPG format can be downloaded from: MNIST Dataset.

2. System Overview

The following diagram illustrates the system workflow:

Workflow

• Environment Setup: Configure the Python environment and install the required packages.
• Model Training: Train the CNN model using the PyTorch framework. Refer to the training script in mnist_train/train.py.
• Weight Extraction: Export the trained model weights as a C header file for ESP32 compatibility.
• Model Testing: Load the header file into the testing code mnist_test/test.c to evaluate performance.
• ESP32 Deployment: Integrate the weights into the ESP-IDF project workspace, located in esp32_deploy/mnist_esp32, for inference on the ESP32.

Note:

• mnist_train/: Contains the PyTorch training script (train.py) for the CNN model.
• mnist_test/: Includes the C testing code (test.c) for model evaluation.
• esp32_deploy/: Houses the ESP-IDF project for deploying the model on the ESP32.

3. Results

Detailed results and implementation steps are documented in the README files within each respective directory. The expected outcome of the deployment is shown below: