First Iteration

This version was used in the MAT EoY 2026. The system is functional at the DSP level — the reverb engine runs in C++/JUCE with 56 parameters — and gesture recognition via MediaPipe is integrated in the Webview layer.

Context

What if a musician could shape sound with their hands — not by pressing buttons or turning knobs, but through gesture? PostTalk detects hand postures in real time via MediaPipe and maps them to audio effect parameters. The interface disappears. The performer's physical presence becomes the control surface.

A key inspiration for this direction is the work of ROLI — their instruments reimagine the relationship between the performer's body and sound in ways that have shaped how I think about expressive control.

Technical Detail

• —DSP engine in C++17 / JUCE 8: 8-line FDN reverb with Hadamard matrix, early reflections, Schroeder allpass diffusion, shimmer (vibrato + pitch shift + chorus), input conditioning, stereo widener, and dry/wet mixer — 47 parameters total.
• —Svelte 5 UI runs inside a WebView. JS-to-C++ communication via URL-scheme interception; C++ pushes state back via evaluateJavascript() at ~30 Hz. The audio thread stays lock-free.
• —MediaPipe HandLandmarker runs in the WebView. Dual-hand index fingers map to continuous pitch modulation; single-hand index-up/down toggles a sustain gate.
• —Passes pluginval at strictness level 5 (VST3). Builds as VST3, AU, and Standalone.

Learnings

• —Gesture-to-parameter mapping needs careful calibration — too sensitive and it is unplayable, too coarse and it loses expressiveness.
• —Separating the UI thread from the DSP thread via message passing — not shared state — was the most important reliability decision. Audio dropouts break a performance.