AI Crash Reporting for ESP32 That Survives the Reboot

The Problem Nobody Talks About

You've been there. Your ESP32-based product has been running for 6 hours at a customer site when it crashes. The device reboots — because that's what you programmed it to do — and all you're left with is this in the logs:

Guru Meditation Error: Core 0 panic'd (LoadProhibited). Exception was unhandled.

JTAG isn't attached. You can't reproduce it on the bench. The customer is annoyed. You have no idea what happened.

This is the firmware debugging problem nobody has a good answer for. Commercial solutions like Memfault start at ~$300/mo with a sales call. So most firmware engineers do nothing — they wait for it to happen again and hope they can catch it.

I spent three weeks debugging one of these crashes — a device running at a customer site, no JTAG, couldn't reproduce it on the bench.The device was running for 4+ hours before hitting it. I eventually found it — a null pointer dereference in a sensor task triggered only by a specific sequence of brownout recovery events — but it took days of reading logs, adding debug prints, and reflashing.

After that I built Firmware Sentry: an Apache-2.0 SDK that captures the full fault state before the device resets, survives the reboot, sends the crash to the cloud, and uses Claude AI to write the root cause in plain English.

This post explains how it works technically and how to integrate it into your ESP32 project in under 30 minutes.

How It Works: The Core Insight

The key insight is timing. When an ESP32 crashes, there's a window — microseconds — between the fault and the reset where the CPU is still running in the panic handler. That window is enough to:

• Read all the fault registers (PC, SP, LR, EXCCAUSE, EXCVADDR)
• Get the current FreeRTOS task name
• Capture a snapshot of the stack
• Copy the last 10 breadcrumb events your app logged
• Write everything to RTC NOINIT memory

The last step is the trick. RTC_NOINIT_ATTR variables in ESP-IDF survive a software reset (SW_CPU_RESET). They're stored in RTC RAM which isn't zeroed by the C startup code on reboot. So when the device comes back up, the crash data is still there, waiting to be sent.

Device crashes
      ↓
Panic handler fires (14 µs)
      ↓ Captures PC, SP, LR, EXCCAUSE, task name, stack, breadcrumbs
      ↓ Writes to RTC_NOINIT_ATTR → survives reset
      ↓
Device reboots
      ↓
fs_has_pending_dump() → true
fs_send_pending() → HTTPS POST to cloud
      ↓
AI diagnosis appears in dashboard within 41 seconds

The Panic Handler Hook

The SDK hooks into ESP-IDF's panic handler using the GNU linker --wrap flag. This replaces esp_panic_handler at link time with our version — no ESP-IDF source patches neede

// fs_hal_esp32.c — called instead of esp_panic_handler
void __wrap_esp_panic_handler(panic_info_t *info)
{
    XtExcFrame *frame = (XtExcFrame *)info->frame;

    s_crash_pc     = panic_get_address(info->frame);
    s_crash_sp     = frame->a1;
    s_crash_a0     = frame->a0;           // return address (caller)
    s_crash_cause  = frame->exccause;     // EXCCAUSE: 24=null read, 25=null write
    s_crash_vaddr  = frame->excvaddr;     // the bad address
    s_crash_uptime = (uint32_t)(esp_timer_get_time() / 1000);
    s_crash_task_hwm = uxTaskGetStackHighWaterMark(NULL);

    // Get current task name
    const char *task = pcTaskGetName(NULL);
    strncpy(s_crash_task, task ? task : "unknown", sizeof(s_crash_task) - 1);

    // Capture stack snapshot (256 bytes from SP)
    memcpy(s_crash_stack, (void *)frame->a1,
           MIN(sizeof(s_crash_stack), 256));

    s_crash_magic = FS_MAGIC;  // marks dump as valid on next boot
    esp_restart();
}

The --wrap flag is added to the root CMakeLists.txt:

target_link_options(${CMAKE_PROJECT_NAME}.elf PRIVATE
    -Wl,--wrap=esp_panic_handler
)

Without this flag, the SDK installs but crashes are never captured.

RTC NOINIT: Why It Works

The crash data is stored in RTC_NOINIT_ATTR variables:

static RTC_NOINIT_ATTR uint32_t s_crash_magic;
static RTC_NOINIT_ATTR uint32_t s_crash_pc;
static RTC_NOINIT_ATTR uint32_t s_crash_cause;
static RTC_NOINIT_ATTR uint32_t s_crash_sp;
static RTC_NOINIT_ATTR uint32_t s_crash_a0;
static RTC_NOINIT_ATTR uint32_t s_crash_vaddr;
static RTC_NOINIT_ATTR uint32_t s_crash_uptime;
static RTC_NOINIT_ATTR char     s_crash_task[32];
static RTC_NOINIT_ATTR uint8_t  s_crash_stack[256];
static RTC_NOINIT_ATTR fs_event_t s_rtc_events[10];   // breadcrumbs
static RTC_NOINIT_ATTR int s_rtc_event_head;
static RTC_NOINIT_ATTR int s_rtc_event_count;

Total: ~601 bytes of RTC RAM used. ESP32 has 8KB available so this is fine.

On the next boot, we check the magic value:

bool fs_has_pending_dump(void) {
    return s_crash_magic == FS_MAGIC;
}

If the magic matches, a crash was captured and needs to be sent. After sending, we clear it:

static void hal_clear_crash(void) {
    s_crash_magic = 0;
}

One important note: this only works for software resets (watchdog, esp_restart()). A power-on reset (brownout, hard reset) clears RTC RAM — which is correct behaviour. A power cycle is not a crash.

Breadcrumbs: What Was the Device Doing?

The most useful feature for field debugging isn't the registers — it's knowing what the device was doing before it crashed.

// Call this throughout your application
fs_log_event("boot");
fs_log_event("wifi_connected");
fs_log_event("mqtt_subscribed");
fs_log_event("sensor_init_ok");
fs_log_event("processing_packet_1042");
// ... crash happens here

// Dashboard shows:
// boot → wifi_connected → mqtt_subscribed → sensor_init_ok → processing_packet_1042

The ring buffer holds 10 events of up to 47 characters each. It's stored in RTC_NOINIT_ATTR so it survives the reset alongside the crash registers. Thread-safe, ISR-safe, zero malloc.

The AI diagnosis receives the full breadcrumb trail and uses it to reason about what the device was doing when it crashed — this is often more useful than the registers alone.

DWARF Symbol Resolution: PC → Function Name

Raw program counter addresses like 0x400D2A4C aren't useful. What you want is sensor_read() at sensor.c:87.

The SDK captures the PC address. The cloud uses pyelftools to walk the DWARF debug tables in your ELF binary and resolve it to an exact function, file, and line number. You upload the ELF once per firmware build — the dashboard does the rest automatically for every crash.

0x400D2A4C → sensor_read()  sensor.c:87
0x400D1B20 → i2c_master_cmd_begin()  [callee]
0x400C8800 → app_main()  [caller]

Build with -g3 to get maximum symbol resolution:

# ESP-IDF
idf_build_set_property(COMPILE_OPTIONS "-g3" APPEND)

AI Diagnosis: What You Actually Get

Here's an example of what appears in the dashboard for a null pointer crash:

Root cause: i2c_handle is NULL — sensor_read() runs before i2c_init()
completes after a brown-out reset. The EXCCAUSE=24 (LoadProhibited)
at 0x400D2A4C confirms a null dereference. EXCVADDR=0x00000000
confirms the bad address.

The breadcrumb trail shows: boot → wifi_connected → sensor_init_started
→ sensor_poll_started. The crash occurs at +1842ms, immediately after
sensor polling begins — before init completes.

Fix: Add a null guard for i2c_handle in sensor_read() and ensure
sensor_init() completes before starting the poll task. Consider using
an event group or semaphore to signal init completion.

Cursor prompt (paste into Copilot/Cursor):
Fix sensor_read() in sensor.c: i2c_handle is NULL at line 87 when
called before i2c_init() completes after brownout. Add null guard
before line 87 and reorder task creation to wait for init semaphore.

The AI receives all captured registers, the DWARF-resolved function/file/line, the breadcrumb trail, and the fault classification. No JTAG. No reproduction. Just an answer.

Integration: 30 Minutes from Zero to First Crash

Step 1 — Sign up and create a device

• Sign up free at firmwaresentry.io (no card required)
• Dashboard → Groups → New Group → ESP32 → Developer mode
• Add Device → copy the Device API Key

Step 2 — Add the SDK

Option A — Component (recommended for ESP-IDF projects):

# Clone the SDK
git clone https://github.com/chinmay-02/firmware-sentry-sdk

# Copy the component into your project
cp -r firmware-sentry-sdk/components/firmware_sentry your_project/components/

Option B — Amalgamated (two files, drop anywhere):

Copy dist/firmware_sentry.h and dist/firmware_sentry.c into your main/ directory.

Step 3 — Update CMakeLists.txt

# main/CMakeLists.txt
idf_component_register(
    SRCS "main.c"
    INCLUDE_DIRS "."
    REQUIRES firmware_sentry nvs_flash esp_wifi esp_event esp_netif
)

# Root CMakeLists.txt — required for panic hook
target_link_options(${CMAKE_PROJECT_NAME}.elf PRIVATE
    -Wl,--wrap=esp_panic_handler
)

Step 4 — Integrate in app_main()

#include "firmware_sentry.h"

static void send_task(void *arg) {
    fs_send_pending();
    vTaskDelete(NULL);
}

void app_main(void)
{
    // NVS init — required before Firmware Sentry
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES ||
        ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
        nvs_flash_erase();
        nvs_flash_init();
    }

    // Init SDK — installs panic handler
    fs_core_init(&g_fs_hal_esp32, &g_fs_transport_https);

    // Provision with your device API key (first boot only)
    if (!fs_is_provisioned()) {
        fs_provision("YOUR_DEVICE_API_KEY");
    }

    // Connect WiFi (must be up before crash send)
    wifi_init();

    // Log boot event — appears in crash breadcrumb trail
    fs_log_event("boot");

    // Send crash from previous boot if one exists
    if (fs_has_pending_dump()) {
        xTaskCreate(send_task, "fs_send", 8192, NULL, 5, NULL);
        vTaskDelay(pdMS_TO_TICKS(6000));
    }

    // Your application starts here
    start_your_app();
}

Step 5 — Flash and trigger a test crash

bash

idf.py -p COM4 -b 115200 flash monitor

Trigger a crash anywhere in your code:

fs_log_event("about_to_crash");
volatile int *p = NULL;
*p = 0xDEAD;  // null pointer write — EXCCAUSE=25

The device crashes, reboots, sends the crash, and within ~41 seconds the AI diagnosis appears in your dashboard.

Zero-Touch Provisioning for Production Fleets

For production, you don't want to hardcode a device API key. The SDK supports zero-touch provisioning: every device self-registers on first boot using a shared Burn Secret compiled into the firmware.

// Same binary on every device
#define FS_BURN_SECRET "your_burn_secret_from_dashboard"

if (!fs_is_provisioned()) {
    fs_auto_provision(FS_BURN_SECRET);  // uses MAC address as unique device ID
}

Each device gets its own unique API key based on its hardware MAC address. Rate-limited at 50 registrations/minute per secret. The provisioning window can be time-limited and toggled off after factory runs.

This means one firmware binary, one flash step, zero per-device configuration.

What It Costs

The SDK is Apache-2.0 and free forever. The cloud is:

• Free tier — 10 devices, 50 AI diagnoses/month, no card required
• Starter — $59/mo, 10 devices, 500 AI diagnoses/month
• Growth — $249/mo, 100 devices, 2000 AI diagnoses/month

Duplicate crashes from the same fault are deduplicated — you're charged for unique fault signatures, not total crash count. A fleet of 100 devices all hitting the same bug counts as 1 AI call.

What's Next

The SDK currently supports:

• ✅ ESP32-WROOM-32 (ESP-IDF v5.x, verified on real hardware)
• ✅ STM32L476 / Cortex-M4 NUCLEO-L476RG (UART + gateway.py, verified)
• ⚠️ Zephyr RTOS (experimental, not hardware-validated)

On the roadmap: nRF52, RP2040, and more Cortex-M variants.

Links

• GitHub (Apache-2.0):github.com/chinmay-02/firmware-sentry-sdk
• Dashboard + free tier:firmwaresentry.io
• Live demo: firmwaresentry.io — press "Inject HardFault" on the homepage

If you're shipping firmware to the field and dreading the first crash report from a customer, give it a try. The free tier covers most evaluation and prototyping use cases with no card required.

Built by a firmware engineer who got tired of "Guru Meditation Error" and nothing else.