ALEC: 90% IoT Data Compression in 2KB RAM

Why I Built ALEC

Every IoT developer knows the pain: you've got thousands of sensors sending small messages, and bandwidth costs are eating your budget. "Just use gzip, " they say.

So you try it. And your 24-byte temperature reading becomes... 44 bytes.

Gzip made it worse.

This isn't a bug — it's fundamental. File compression algorithms like gzip, LZ4, and zstd were designed for large files, not tiny sensor messages. Their headers alone can exceed your entire payload.

After hitting this wall repeatedly in production deployments, I decided to build something better.

The Problem: Compression Overhead

Here's what happens when you compress a typical IoT message:

| Original | gzip | LZ4 | ALEC |
|----------|------|-----|------|
| 24 bytes | 44 bytes (+83%) | 33 bytes (+37%) | 4 bytes (-83%) |
| 64 bytes | 72 bytes (+12%) | 68 bytes (+6%) | 11 bytes (-83%) |
| 128 bytes | 98 bytes (-23%) | 104 bytes (-19%) | 19 bytes (-85%) |

Traditional compression only starts winning at ~100+ bytes. But most sensor readings are tiny!

The Solution: Adaptive Lazy Evolving Compression

ALEC takes a completely different approach, built from scratch for IoT constraints.

1. Lazy Compression

The encoder first sends its compression *decision*, allowing the decoder to prepare before data arrives. This eliminates synchronization overhead.

2. Evolving Context

Encoder and decoder share a dictionary that improves over time. After 100 messages, ALEC "knows" your sensor's patterns and compresses even more aggressively.

3. Priority System

Every message is tagged P1 (critical) to P5 (routine). Safety alerts skip the queue — they're never delayed by bulk telemetry.

How It Works

┌─────────────┐     Compressed      ┌─────────────┐
│   Sensor    │──────(4 bytes)─────▶│   Gateway   │
│   (ALEC     │                     │   (ALEC     │
│   Encoder)  │◀────Sync Context────│   Decoder)  │
└─────────────┘                     └─────────────┘
│                                   │
│ Original: 24 bytes                │ Restored: 24 bytes
▼                                   ▼
[Sensor HW]                      [Cloud/Storage]

The magic is in the **evolving shared context**. Both encoder and decoder maintain identical state. As messages flow, they learn:

• Common value ranges (temperature usually 15-30°C)
• Typical deltas (temperature changes <0.5°C between readings)
• Field patterns (always "temp=XX.X, hum=YY")

After the learning phase, ALEC transmits only the *differences* from expected values — often just a few bits.

Real-World Results

I've tested ALEC across different industries:

| Use Case | Compression | Impact |
|----------|-------------|--------|
| Agriculture (soil sensors) | 83% | Battery: 6 months → 3+ years |
| Manufacturing (vibration) | 97% | 2000 sensors, no network upgrade |
| Smart City (air quality) | 86% | 10x more sensors, same bandwidth |
| Satellite IoT | 89% | Message cost: $0.10 → $0.01 |

Hardware Requirements

ALEC is designed for constrained devices:

• RAM: 2KB minimum
• Flash: 8KB for the codec
• CPU: Any 32-bit processor

Tested on:

• ESP32 / ESP32-S3 / ESP32-C3
• STM32F4 / STM32L4
• Raspberry Pi (all models)
• Nordic nRF52840
• Any Linux system

Getting Started

Step 1: Install

Rust:

```toml
[dependencies]
alec-codec = "1.1"

Or with Cargo:

```bash
cargo install alec-codec

Step 2: Encode

```rust
use alec_codec::{Encoder, Decoder, Priority};
// Create encoder (sensor side)
let mut encoder = Encoder::new();
// Your sensor reading
let reading = b"temp=23.5,humidity=67,pressure=1013";
// Compress it
let compressed = encoder.encode(reading, Priority::P3).unwrap();
println!("Original: {} bytes", reading.len());   // 35 bytes
println!("Compressed: {} bytes", compressed.len()); // ~6 bytes
```

Step 3: Decode

```rust
// Create decoder (gateway side)
let mut decoder = Decoder::new();
// Decompress
let original = decoder.decode(&compressed).unwrap();
assert_eq!(original, reading); // Perfect reconstruction
```

Complete Example: ESP32 Weather Station

Here's a real-world example — an ESP32 sending compressed weather data:

```rust
use alec_codec::{Encoder, Priority};
fn main() {
let mut encoder = Encoder::new();
loop {
// Read sensors
let temp = read_temperature();
let humidity = read_humidity();
let pressure = read_pressure();
// Format message
let msg = format!("t={:.1},h={:.0},p={:.0}", temp, humidity, pressure);
// Compress with ALEC (35 bytes → 4 bytes)
let compressed = encoder.encode(msg.as_bytes(), Priority::P3).unwrap();
// Send via LoRaWAN — now fits in SF12!
lora_send(&compressed);
// Sleep 15 minutes
sleep(Duration::from_secs(900));
}
}
```

Result: A 35-byte message becomes 4 bytes. Fits easily in LoRaWAN SF12 with room to spare!

Run Your Own Benchmarks

Test ALEC against your real sensor data:

Install CLI tool

cargo install alec-codec

Run benchmark

alec benchmark --input your_sensor_data.csv

Sample output:

Dataset: sensor_readings.csv (10, 000 messages)

Average message size: 47 bytes

Results:

gzip: 52 bytes (+11%) ❌

lz4: 49 bytes (+4%) ❌

zstd: 44 bytes (-6%) ⚠️

ALEC: 8 bytes (-83%) ✅

When to Use ALEC

✅ Perfect for:

• Small, frequent sensor messages (<200 bytes)
• LoRaWAN, Satellite, NB-IoT, Sigfox
• Battery-powered devices
• High-volume deployments (1000+ sensors)

❌ Not needed for:

• Large file transfers (use gzip)
• One-off messages without patterns
• Unconstrained WiFi/Ethernet

What's Next?

ALEC is part of a larger ecosystem I'm building:

• ALEC Gateway — Multi-sensor orchestration with Prometheus/Grafana
• ALEC Complexity — Anomaly detection from compression patterns
• Reference designs — For ESP32, STM32, nRF52

The codec is open source (AGPL-3.0) with commercial licenses available for proprietary products.

Resources

• GitHub: https://github.com/zeekmartin/alec-codec
• Crate: https://crates.io/crates/alec-codec
• Website: https://alec-codec.com
• Paper: DOI 10.5281/zenodo.18351236

Questions? Reach me at david.martin@alec-codec.com

Contributions welcome! 🚀