yousanKoeAct-JP (Japanese speech straight to robot actions)
A 2.59M-parameter model that turns Japanese speech into action calls (Action JSON) with no transcript in between. It runs on a PC CPU and on an M5Stack CoreS3 (ESP32-S3) with no cloud.
Overview
A small model for voice control: say “turn your face 45 degrees to the right” in Japanese and it returns [{"name":"look","arguments":{"direction":"right","degrees":45}}]. Instead of chaining speech recognition and a language model, it goes from speech directly to Action JSON. For negations such as “actually, don’t turn off the light” and for speech that is not a request, it returns []. The demo runs inside the browser, and the audio never leaves it.
What I did
Starting from the speech encoder of Ralomi, my own speech recognition model, I handled the grammar-constrained decoder design, building the training data with synthetic speech, training, quantization for the ESP32-S3, the firmware, evaluation and release.
Technical notes
- A 0.54M decoder sits on top of Ralomi (2.05M), for 2.59M parameters in total. It covers the same 11 actions as JapaneseTinyAgentLM, with up to two calls per utterance.
- Against a two-stage pipeline (transcribe, then a language model; 5.20M) trained on the same data, exact match on the actions was 93.2% (pipeline: 83.6%), and 88.1% (pipeline: 71.1%) in heavy noise (SNR 0 dB). Both are averages over five training runs on 6,850 synthetic utterances in voices not seen in training.
- On the CoreS3 itself, the action comes out about 0.53 s (median) after you stop speaking, against about 1.5 s for the pipeline.
- All request audio used for evaluation is synthetic; it has not yet been checked with real speakers. Because ongoing human speech can trigger it by mistake, it is meant to be combined with a wake word or a button. The weights and code are Apache-2.0.
