Sound Examples

Claude Collider MCP + Rave example

Rave Midi example

Concept

RAVE — a variational autoencoder from IRCAM — learns a compressed latent representation of a sound corpus and reconstructs it through a decoder. The birds.ts model was trained on bird recordings: organic, textured sound collapsed into a 16-dimensional space. NeoChucao asks a simple question: what happens if a performer navigates that space directly, note by note, from a keyboard?

Each MIDI note-on maps to a coordinate in that latent space and feeds the RAVE decoder in real time via SuperCollider's nn.ar. Pitch maps to z0 — the highest-variance axis — velocity to z1, the mod wheel to z2 for timbre morphing, and a CC knob injects Gaussian randomness into z3–z15, giving the performer control over how deterministic or chaotic the instrument sounds. The hardest part was not the code but the mapping: a keyboard implies pitch steps, but latent space has no natural pitch axis — only variance.

Technical Detail

• —MIDI input is translated to a 16-element latent vector: note number → z0 (linearly mapped to [−2, +2] over the playable range), velocity → z1, mod wheel (CC 1) → z2, and CC 2 → noise_scale. z3–z15 are filled with session-fixed Gaussian draws per MIDI note, so each key produces a consistent timbral character within a session but differs across sessions.
• —The birds.ts TorchScript model (WassersteinRAVE, 16 latent dims, block size 2048 samples) is loaded into scsynth via the nn.ar SuperCollider extension. Each MIDI note spawns a new Synth node running NN(\birds, \decode).ar at audio rate. A Lag.kr of 0.05 s on each latent parameter smooths coordinate transitions and prevents clicks between notes.
• —The RAVE decoder's convolutional layers initialize from zero on every new Synth node, producing near-silence for 200–500 ms while the network fills up. An 800 ms ADSR attack envelope is used to hide this warm-up artefact — a constraint imposed by the model's architecture, not a musical choice.
• —Raw RAVE output passes through a three-stage signal chain: a Compander (threshold 0.4, ~3:1 ratio, 10 ms attack / 100 ms release) tames amplitude swells from the decoder's own dynamics; a three-voice chorus (DelayL at 15 ms ± 3 ms, LFOs at 0.15–0.22 Hz spaced 120° apart) masks timbral wobble from latent lag; FreeVerb2 adds room depth.

Iteration process

• —Python + Supriya: Replaces the SuperCollider-native approach with a Python-driven architecture: mido handles MIDI input, supriya communicates with scsynth over OSC, and a real-time 3D visualizer built in vispy renders the latent trajectory at 60 fps — a glowing point and fading trail moving through the (z0, z1, z2) space as the performer plays. What makes this version compelling is that it makes the geometry of the performance visible. Navigating a 16-dimensional learned space is abstract by nature; the visualization turns it into something you can watch and reason about. The tradeoff is complexity — the Python layer adds infrastructure that pulls attention away from the instrument itself, which is ultimately why the SuperCollider version became the primary one.
• —live-coding-rave: Takes a different approach entirely. Instead of a performer navigating the latent space from a keyboard, Claude does it through language. Using ClaudeCollider — an MCP server that lets Claude generate and execute SuperCollider code live — you describe what you want and Claude composes patterns that are routed through RAVE models in real time. The models run in parallel on separate buses (birds, marine mammals, or other). This version raises a question about authorship and control, which is one of the key component in the discussion about AI and Media Art. The latent space is still being navigated, but the navigator is a language model responding to human intent rather than a hand on a keyboard.

Learnings

• —The most important lesson was infrastructure versus musical focus. Python with Supriya worked technically but shifted development time toward OSC routing and async scheduling — away from musical decisions. Moving everything into a single SuperCollider file made calibration fast enough to happen in real time, which changed what the project could become.
• —The warm-up artefact — 200–500 ms of near-silence when the decoder's convolutional state initializes from zero — is architectural and unfixable by signal processing alone. Designing the instrument's ADSR around it was the right approach. Distinguishing between a musical decision and a model constraint turned out to matter more than any specific technical fix.
• —Using a latent space as an instrument is genuinely unpredictable. Some coordinates produce sounds that are striking and alive; others produce noise with no musical value. That instability is not a bug to fix — it is a property of what RAVE learned. The model was trained on birds, which are not musical in any conventional sense. Navigating that space with a keyboard takes time: time to map which regions are expressive, which are dead, and which are worth returning to. Audio encoders trained on non-musical material demand a different kind of performer patience.