I Ran a 55M-Parameter Speech-to-Text Model on an ESP32-S3

Offline speech-to-text on a microcontroller: Cactus Compute's Whistle model cut to fit 16 MB flash, with a from-scratch C engine. No cloud.

By Amal V S, Backend & AI Systems Architect · Mon Oct 26 2026

I spoke into a web page, and a microcontroller wrote down what I said. Not a Raspberry Pi, not a phone, and no cloud API: an ESP32-S3 board with 16 MB of flash ran a 55-million-parameter speech recognition model entirely on the chip. The laptop only records the audio and shows the result.

The model is Whistle, an open speech-to-text model from Cactus Compute. It does not fit on the board as shipped, and there is no official engine that runs it there. So I cut the model down to fit and wrote the inference engine from scratch in C. Here is how it works, what it costs in accuracy and speed, and how to run it yourself. All the code is on GitHub: alchemist123/STT-Model-ESP32.

• Board: ESP32-S3-DevKitC-1 N16R8, 2 cores at 240 MHz, 16 MB flash, 8 MB PSRAM

• Model: Whistle by Cactus Compute, 55.15M parameters, 2-bit weights, Apache-2.0

• On the chip: 14.5 MB, a 4-layer decoder, about 11.5% word error rate on clean speech

• Speed: about 3 to 4 times the length of the audio. My 6.4 s sentence took 20 s

• Cloud calls: zero

Watch it run: the real, unedited test

This is a screen recording of one real run, in real time. I connect the board over USB, speak for 6.4 seconds, and the chip spends 20 seconds working out the text. The browser tabs are blurred; everything else is exactly as it happened.

Prefer the vlog player? Watch it in my vlogs.

The text that came back: "I created a model with the busy and I put it on a ESP32." One word came out wrong. That is what an 11.5% word error rate looks like in practice. But every word in that line was produced by the microcontroller, from raw audio, on its own.

Why speech-to-text on a microcontroller matters

Most voice features today send your audio to a server. That needs a network connection, costs money per request, and means your voice leaves the device. Running speech recognition on the device itself fixes all three:

• Privacy: the audio never leaves the board.

• Works offline: no Wi-Fi, no API key, no outage.

• Cost: no per-minute speech API bill, on hardware that is cheap and draws little power.

Wake words and a handful of fixed commands have run on chips like this for years. This experiment is about something harder: open-vocabulary speech-to-text with a real neural model that writes whole sentences.

The model: Whistle by Cactus Compute

Whistle is an open speech-to-text model built for small devices (weights on Hugging Face, Apache-2.0). It has 55.15 million parameters, understands 7 languages (English, German, French, Spanish, Italian, Dutch and Polish), and takes 16 kHz audio of up to 30 seconds.

The reason it can be this small is that it was trained with quantization in the loop. Almost every weight is stored in just 2 bits, using a codebook format (CQ): each group of 128 weights is rotated with a Walsh-Hadamard transform, scaled, and replaced with an index into a tiny 4-entry codebook. The whole model is one 16.9 MB file.

Which leads to the problem: the board has 16 MB of flash, and part of it is needed for the bootloader and the app.

Making a 16.9 MB model fit in 16 MB of flash

After the bootloader and the firmware, the model gets a partition of exactly 15,073,280 bytes. Re-quantizing further was not an option, because the weights are already 2-bit and the model was trained that way. So I cut decoder layers instead.

Whistle supports this by design. Its decoder is a ladder: every depth from 2 layers up is a trained, valid model. The ladder keeps both end layers and then fills the widest gaps, so a 4-layer decoder uses layers 0, 3, 5 and 7, not simply the first four. The encoder always keeps all 8 blocks.

Depth 5 misses by about 45 KB, so depth 4 is the deepest that fits: 14.52 MB with 11.5% word error rate, against 9.7% for the full model. The cut is made by tools/slice_cact.py, which edits the .cact file directly: it drops the removed layers' tensors, renumbers the remaining ones, and rewrites the header. As a self-check, slicing at the original depth reproduces the input file byte for byte.

Writing the inference engine from scratch in C

Cactus publishes the model code (in JAX) and the file format, but not an engine that runs on a microcontroller. So everything that runs on the chip is new code: a portable C engine that builds on a Mac for testing and on the ESP32-S3 as an ESP-IDF component.

For each recording, the chip runs the whole pipeline:

1. Mel front end: a 512-point FFT turns raw audio into log-mel features, one frame every 10 ms.

2. Convolutional stem: shortens the audio 8 times, computed row by row so it never needs a huge buffer.

3. Encoder, 8 blocks: attention, convolution and Hadamard MLPs over the whole clip.

4. Decoder, 4 layers: writes one token at a time (greedy decoding with KV caches), looking back at the encoder output.

5. Tokenizer: turns token ids back into UTF-8 text.

The 14.5 MB model is never copied into RAM. It is memory-mapped from flash and read in place; only about 1.8 MB of small float tensors are unpacked into PSRAM. Loading the model takes about 0.28 seconds.

From 89 seconds to 10 seconds: making it fast enough

The first version that worked was correct but painfully slow: 88.9 seconds for a 2.25-second clip. These are the changes that made it usable:

• Integer math: int8 activations times int8 weight levels, added up in int32, instead of floating point.

• The S3 vector unit: the ee.vmulas.s8.accx instruction does 16 multiply-accumulates at once. Matrix-multiply time dropped from 16 s to 3.5 s.

• Reuse: 2-bit weights are unpacked once and reused across a tile of 16 audio frames.

• One rotation, shared: because the Hadamard matrix is symmetric, the engine rotates the activation once per group and reuses it for every output row, instead of rebuilding weights.

• Both cores: every matrix multiply, MLP and attention step is split across the two CPU cores.

• Small things that add up: a bigger 64 KB data cache, a fast exp, and table-based sigmoid and SiLU.

Now a 4.8-second clip takes 15.7 seconds, about 3.3 times real time. The remaining bottleneck is interesting: the decoder is limited by flash bandwidth, not compute. Every generated token reads about 4.5 MB of weights from flash at roughly 25 MB/s, which puts a floor of about 0.18 seconds on each token.

How I know the engine is correct

A from-scratch engine is easy to get subtly wrong, so it was checked against the original model at every stage:

• tools/make_golden.py runs the original JAX model layer by layer and saves every intermediate result.

• The C engine is compared stage by stage (mel, stem, each encoder block, each decoder layer, logits). They agree to int8 rounding noise.

• On 73 LibriSpeech clips, the engine scores 11.4 to 11.7% word error rate, against 11.46% for the original model at the same depth.

• On the chip, the same clips produce the same text as the Mac build.

Talking to the board: a web page over USB

The front end is one HTML file with no libraries. It uses Web Serial (Chrome, Edge or Brave) to talk to the board's native USB port. It records the mic, resamples to 16 kHz mono, and sends the audio in small frames. The chip answers with JSON lines: progress updates ("Understanding the speech 3/8", "Writing words 5") and finally the transcript.

As a small bonus, if you say a colour ("the sky is blue"), the board's RGB LED blinks that colour after it sends the text. On the firmware side, one core reads USB while the other runs recognition, and both cores share the heavy matrix work.

Run it yourself

You need an ESP32-S3 board with 16 MB flash and 8 MB PSRAM (N16R8), a USB-C data cable, Python 3.11, ESP-IDF 5.4 and Chrome, Edge or Brave.

1. Get the model and cut it to 4 decoder layers

curl -L -o whistle.cact https://huggingface.co/Cactus-Compute/whistle/resolve/main/whistle.cact
python3 -I tools/slice_cact.py whistle.cact model_d4.cact 4   # 14.52 MB

2. Flash the firmware and the model

source ~/esp/esp-idf/export.sh
cd firmware
idf.py -p /dev/cu.usbmodemNNN flash
esptool.py --chip esp32s3 -p /dev/cu.usbmodemNNN -b 921600 \
           write_flash 0x1A0000 ../model_d4.cact     # ~3 minutes

3. Open the page and talk

python3 -m http.server 8765 --directory web

Open http://localhost:8765, click Connect ESP32, wait for "Connected (whistle-1)", click Start talking, say a sentence (up to 15 s) and click again to stop. The README has a troubleshooting table for flashing and port issues.

Limits, and what comes next

This is an experiment, not a product. It is honest to say what it cannot do yet:

• Slow: about 3 to 4 times the length of the audio, with no streaming. You speak, then you wait.

• Small model: about 11.5% word errors on clean read speech, and much worse with noise (33% at 5 dB noise) or strong accents.

• Short clips: up to 15 seconds and 128 tokens per recording.

• English tested: the other 6 languages work through the selector but were not evaluated.

Ideas that could make it faster or better:

• Keep the hot decoder weights in PSRAM to get under the 0.18 s per token flash floor.

• Run attention and the Hadamard MLPs on the vector unit too.

• Prune the vocabulary to English only, which frees about 1 MB: enough for a 5-layer decoder.

• Add voice-activity detection and streaming, so text appears while you talk.

The takeaway

For years, "AI on a microcontroller" mostly meant spotting a wake word. Now a cheap board with no network can listen to a full sentence and write it down. It is slow and it makes mistakes, but the hard part is proven: a real speech-to-text model can live entirely on a microcontroller. Models trained for 2-bit weights, like Whistle, are what make this possible, and they will only get better.

If you build something with it, I would love to see it. Star the repo, open an issue, or reach out: github.com/alchemist123/STT-Model-ESP32.

FAQ

Can an ESP32 run speech-to-text without the cloud?

Yes. This project runs the Whistle model (55M parameters) fully on an ESP32-S3 with 16 MB flash and 8 MB PSRAM. The laptop only records audio and shows the text.

Which ESP32 board do I need?

An ESP32-S3 with 16 MB flash and 8 MB PSRAM, such as the ESP32-S3-DevKitC-1 N16R8. The S3's vector instructions and the PSRAM are both needed.

How accurate is it?

About 11.5% word error rate on clean LibriSpeech audio with the 4-layer decoder that fits on the board. The full model scores 9.7%, but it does not fit in 16 MB.

How fast is speech-to-text on the ESP32-S3?

About 3 to 4 times the length of the audio. A 4.8-second clip takes 15.7 seconds; in my recorded test, 6.4 seconds of speech took 20 seconds.

What is the Whistle model?

An open, Apache-2.0 speech-to-text model from Cactus Compute, designed for small devices. It uses 2-bit weights trained with quantization in the loop and supports 7 languages.

Credits: Whistle model and weights by Cactus Compute (Apache-2.0); this project reimplements inference for the ESP32-S3 and does not ship the model file. ESP-IDF by Espressif; test audio from LibriSpeech. Engine, firmware, web page and tools: STT-Model-ESP32. All measurements are from the project README.