Oreo mascot
Oreo
All documentation
engineering deep dive

From 0.5 FPS to native video.

Building responsive 24 FPS Gallery playback on an ESP32-S3 badge required more than a faster decoder. It meant redesigning the complete path from host encoding and flash storage to PSRAM, native Xtensa code, and the LCD bus.

ESP32-S3MicroPython 1.28RV565 v424 FPS320 × 240 RGB565

Gallery video initially looked like an image-decoding problem. In practice, it became a systems problem involving storage bandwidth, interpreter overhead, PSRAM fragmentation, display timing, input latency, and safe deployment.

The hardware budget

The ST7789-compatible panel is connected over 40 MHz SPI. A full frame contains:

320 × 240 × 2 bytes = 153,600 bytes

That transfer consumes about 30.7 ms. A 24 FPS stream gets 41.67 ms per frame, leaving approximately 11 ms for frame selection, decoding, scaling, UI state, and input handling.

41.67 ms

24 FPS deadline

30.7 ms

SPI transfer

≈11 ms

compute headroom

What failed first

Each RV565 generation isolated a different bottleneck. “Native decode” alone was not enough; the entire data path had to fit inside the deadline.

ApproachMeasured costResult
Python nearest-neighbour scaling~367 ms/frame2–3 FPS before LCD output
Persistent Deflate RGB565 stream~234 ms/frameNative inflate still missed budget
Flash-read indexed frame + C scale~40 ms before LCDFilesystem bandwidth became limiting
PSRAM indexed frame + C scale~5.2 msEnough time remains for display output
C scale + physical LCD transfer~39.0 ms25.6 FPS measured ceiling

The final pipeline

The workstation performs expensive colour analysis once. The badge executes only predictable operations: select, map, scale, and transfer.

RV565 v4 from source video to physical pixels

RV565 v4 format

V4 retains the 12-byte RV565 header, followed by fixed-size independently addressable frames.

RV5

Magic · 3 B

4

Version · 1 B

180

Width · LE16

135

Height · LE16

24

FPS · 1 B

240

Frames · LE16

RGB565 palette

256 colours · 512 bytes

8-bit index plane

180 × 135 · 24,300 bytes

frame_size = 512 + (180 × 135) = 24,812 bytes
payload    = 24,812 × 240       = 5,954,880 bytes
file_size  = payload + 12       = 5,954,892 bytes

Every frame has an adaptive 256-colour palette. The 512-byte overhead provides materially better colour than a fixed RGB332 palette. Dithering stays disabled so palette changes do not make static regions shimmer.

Fragmentation-safe preload

One 5.95 MB bytearray worked after reset but could fail after ordinary UI usage fragmented the heap. The final loader allocates 240 independent ~24 KB blocks, requiring only a small contiguous region for each allocation while preserving the same total data and quality.

Incremental, input-friendly preload state machine

Three frames load per update. The launcher regains control between chunks, polls buttons, and redraws the progress indicator. Leaving Gallery releases the list and all frame blocks for garbage collection.

Native code without new firmware

The firmware does not expose micropython.viper, but its ABI advertises MicroPython MPY version 6, native sub-version 3, and xtensawin. Gallery can therefore ship a dynamically loaded native module at apps/gallery/src/gallery_native.mpy without rebuilding firmware.

One playback frame through the OreoOS application lifecycle

The C kernel builds nearest-neighbour X/Y lookup tables, reads one palette index per output pixel, and writes the matching 16-bit value directly into Display._buf. There is no 153 KB intermediate image. A second native function applies the same fast scaling path to Gallery photos.

Playback timing and controls

Playback remains inside the normal OreoOS lifecycle. The launcher polls buttons before update and draw, so no second event loop or thread competes for input or display ownership.

Gallery video states and input transitions

If a tick is late, Gallery advances at most one frame and discards excess accumulated delay. It never tries to catch up with a burst of work—the behaviour that previously froze controls. With native expansion plus LCD output at about 39 ms, input latency remains near one video frame.

Deployment and compatibility

The deploy manifest includes Python and native MPY modules from app source trees, plus Gallery RV565 assets. A targeted replacement is:

python3 tools/deploy.py --override=gallery

The override removes the device-side Gallery tree before copying, preventing interrupted uploads from leaving partial media. Versions 1–3 remain readable for compatibility; V4 additionally requires the matching native module and validates its magic, dimensions, frame count, FPS, and exact payload size before playback.

Designed for this display, not every display.

RV565 v4 is intentionally not a general H.264 decoder. It optimizes for short 320 × 240 clips, predictable timing, responsive controls, good palette quality, and deployment without a custom firmware image. Its speed comes from assigning each job to the right resource: quantization on the host, storage in PSRAM, pixel loops in native Xtensa code, and control in MicroPython.