Introduction
Deploying deep learning on microcontrollers (MCUs) requires fitting both the model weights and runtime activations within tens to hundreds of kilobytes of SRAM and flash. Existing approaches either use an interpreter with a pre-written kernel library (TFLite Micro, ExecuTorch) — incurring dispatch overhead and requiring framework dependencies — or combine a Python code generator with hand-written C kernels (MCUNet TinyEngine). None produces self-contained, dependency-free C code specialized to a given model.
We present te-comp, the first complete MLIR-based compiler that takes a
standard model file .tflite or .safetensors and emits plain
C source files implementing the full inference engine — kernels, memory
layout, weight arrays, and RTOS integration — with zero runtime dependencies.
The generated code compiles with any ARM cross-compiler and drops into an
existing firmware project in three lines of CMake.
te-comp introduces a custom te MLIR dialect with 33 quantized
neural network operations and implements five composable optimization passes:
flash-resident weights, in-place depthwise convolution, operator
specialization, static arena memory planning, and Q4 weight quantization for
transformer models.
On all 10 MCUNet benchmark models, te-comp uses 36% less SRAM and 15% less
flash than MCUNet TinyEngine, while generating complete,
self-contained C code rather than linking a pre-written kernel library.
Both systems use the same zero-copy I/O pattern: the caller writes input
directly into the engine’s arena buffer and reads output in-place, with no
intermediate staging buffers or memcpy.
Numerical verification against the TFLite reference interpreter on 16
int8 models using real test samples from Imagenette (ImageNet subset)
and MCUNet’s VWW minival shows bit-exact output on 4 models, 1 LSB error on 2 models,
and less than 9 LSB on the deepest architectures — with ground-truth label accuracy of 9/10 or 10/10
on all 10 MCUNet models (both VWW and ImageNet), matching TFLite
exactly on every model. te-comp also compiles an 8.3M-parameter GPT-Neo transformer
(TinyStories-v2) to run on a Cortex-M7 MCU with 512KB SRAM and 4MB flash via Q4 weight quantization, int4 KV cache, and vocabulary pruning —
a capability no existing MCU inference framework supports.
Demo Video
Please have a look at the demo video of the model in action on NXP S32K3x4EVB-T172 board.
The APIs
With te-comp, embedded SW engineers just drop the generated C in the project and call the generated APIs.
/* Auto-generated by te-comp
*
* Public inference API for: tinystories_v2_pruned
* tinystories_v2_pruned
*
* Generated by te-comp v0.1.0
*
* Zero-copy usage:
* inference_init(NULL);
* memcpy(inference_get_input(), sensor_data, 8);
* inference_run();
* const float *out = inference_get_output();
*
* This file is auto-generated. Do not edit manually.
*/
#ifndef TINYSTORIES_V2_PRUNED_INFERENCE_API_H
#define TINYSTORIES_V2_PRUNED_INFERENCE_API_H
#include <stdint.h>
#ifdef __cplusplus
extern "C" {
#endif
/* -----------------------------------------------------------------------
* Model constants
* ----------------------------------------------------------------------- */
#define TINYSTORIES_V2_PRUNED_INPUT_SHAPE {1, 1}
#define TINYSTORIES_V2_PRUNED_INPUT_SIZE_BYTES 8
#define TINYSTORIES_V2_PRUNED_OUTPUT_SHAPE {1, 1, 14632}
#define TINYSTORIES_V2_PRUNED_OUTPUT_SIZE_BYTES 58528
#define TINYSTORIES_V2_PRUNED_ARENA_SIZE_BYTES 59040
#define TINYSTORIES_V2_PRUNED_NUM_LAYERS 0
#define TINYSTORIES_V2_PRUNED_TOTAL_MACS 0
/* -----------------------------------------------------------------------
* Types
* ----------------------------------------------------------------------- */
typedef enum {
INFERENCE_STATUS_NOT_READY = 0, /* Engine not initialized */
INFERENCE_STATUS_IDLE = 1, /* Ready for input */
INFERENCE_STATUS_RUNNING = 2, /* Inference in progress */
INFERENCE_STATUS_ERROR = 3, /* Last inference failed */
} inference_status_t;
typedef struct {
uint32_t task_priority; /* Unused in bare-metal */
uint32_t stack_size_bytes; /* Unused in bare-metal */
} inference_config_t;
typedef struct {
const char *model_name;
const char *model_description;
uint32_t input_size_bytes;
uint32_t output_size_bytes;
uint32_t arena_size_bytes;
uint32_t weights_flash_bytes;
uint32_t peak_sram_bytes;
uint32_t num_layers;
uint32_t total_macs;
} inference_info_t;
/* -----------------------------------------------------------------------
* API
* ----------------------------------------------------------------------- */
/**
* Initialize inference engine.
* @param config Optional configuration (NULL for defaults).
* @return 0 on success, negative on error.
*/
int inference_init(const inference_config_t *config);
/**
* Get pointer to the input region inside the engine's arena.
* Write your sensor data here (8 bytes).
* The pointer is valid as long as the engine is initialized.
*/
int64_t *inference_get_input(void);
/**
* Get pointer to the output region inside the engine's arena.
* Valid after inference_run() returns 0.
* @return Pointer to 58528 bytes of output data.
*/
const float *inference_get_output(void);
/**
* Run inference (blocking).
* Reads from the input region and writes to the output region.
* @return 0 on success, negative on error.
*/
int inference_run(void);
/**
* Get current inference status.
*/
inference_status_t inference_get_status(void);
/**
* Get static model info (sizes, MACs, etc.).
*/
const inference_info_t *inference_get_info(void);
/**
* Get total number of completed inferences.
*/
uint32_t inference_get_count(void);
/**
* Tear down inference engine.
*/
void inference_deinit(void);
#ifdef __cplusplus
}
#endif
#endif /* TINYSTORIES_V2_PRUNED_INFERENCE_API_H */