First Iteration

This version was used in the MAT EoY 2026. The system is functional at the DSP level — the reverb engine runs in C++/JUCE with 56 parameters — and gesture recognition via MediaPipe is integrated in the Webview layer.

Context

LLMs changed human-computer interaction permanently. Natural language as an interface is a genuine leap. But it also narrows something. We have reduced the richness of human expression to text and voice, as if language were the only channel. Communication is posture, movement, proximity, sound, gesture, context. We have bodies, and our bodies carry meaning that language cannot fully encode.

PostTalk asks: what if a musician could shape sound with their hands — not by pressing buttons or turning knobs, but through gesture? The performer's hand postures, detected in real time via MediaPipe, control audio effect parameters. The interface disappears. The musician's physical presence becomes the control surface.

A key inspiration for this direction is the work of ROLI — their instruments reimagine the relationship between the performer's body and sound in ways that have shaped how I think about expressive control.

Technical Detail

• —DSP layer built in C++17 with JUCE 8. The reverb engine is an 8-line Feedback Delay Network with a Hadamard feedback matrix, plus Early Reflections (8-tap parallel delays with spin modulation), a 4-stage Schroeder allpass diffusion network, frequency-dependent decay via per-line crossover filtering, saturation, vibrato, pitch shifting, stereo widening, and an XY-controlled timbre filter — 56 parameters in total.
• —UI layer is a Svelte + TypeScript frontend delivered via WebBrowserComponent, running on the message thread separate from the DSP engine. All 56 parameters are bridged universally: JS → C++ via a custom URL scheme (juce://setparameter?name=decay&value=0.42) intercepted by the WebBrowserComponent; C++ → JS via evaluateJavascript() on the message thread, keeping the audio thread lock-free.
• —Hand gesture detection via MediaPipe HandLandmarker running inside the Webview. Dual-hand pitch control maps both index fingers pointing up/down to continuous frequency modulation. Single-hand index-up engages a sustain gate; index-down releases it. Gate state latches until the opposite gesture occurs.
• —Architecture insight: separating the UI thread from the DSP thread via message passing — not shared state — keeps audio processing deterministic while the interface stays responsive.

Learnings

• —Real-time computer vision in a live performance context has strict latency requirements. MediaPipe running in the Webview is fast enough, but the mapping between gesture and effect needs careful calibration — too sensitive and it is unplayable, too coarse and it loses expressiveness.
• —Thread separation is not just an architectural choice — it is a musical one. Audio dropouts break a performance. Keeping the DSP thread isolated from UI events was the most important reliability decision in the project.