1 · What it is
Movement Voltage is a hand tracker wired straight into a Eurorack modular synthesizer: a stereo camera watches one hand, taking four measurements off it that become four control voltages. No MIDI anywhere in between.
2 · Why I built it
The closest modules I already owned are a joystick, and a "knob recorder" which captures your hand moving a pot and plays that movement back as 0 to 5 volts. The former gives you continuous control on three axes (vertical, horizontal, twist) but snaps back to origin when you let go. The latter treats a hand movement as a modulation source, but only on playback. I wanted the amalgamation: direct control via body movement that is neither strictly ephemeral nor exclusively repeatable.
3 · The four channels
There are four channels, one per finger not including the thumb. Each "slider" is controlled by the distance from fingertip to thumb, scaled by the width of the hand, so that distance to camera is not a factor nor is size of user's hand.
Playing starts with about four seconds of setup, during which the screen prompts, counts down, and watches the hand move through its full range. Then it stops learning, freezing that range for the session.
A closed pinch reads zero and an open stretch reads one.
4 · The Pipeline Architecture
An OAK-D Lite watches in stereo and runs the landmark network on its own MyriadX chip. 21 hand landmark positions arrive at the Raspberry Pi 4 which computes four normalized distances (fingertips to thumb) and sends them out over USB to an FT232H which pipes via I²C to an MCP4728, where finally the 12-bit DAC outputs four voltages 0-5V.
Because the MCP4728 is rail-to-rail, a five-volt supply gets it driving its output the whole way to both rails. There's no op-amp after it: the converter's native output fits within Eurorack's expected input.
The question anyone who has built CV hardware asks first is latency. Camera to jack, there is inference on the MyriadX, a USB hop, the Pi's own loop, an I²C write and a panel redraw all in the path. 18-22 fps with a hand in frame means a new value cannot arrive more often than about every 50 ms, whatever happens downstream of it. The end-to-end latency has not been measured yet.
5 · Initial Build
Initially an ESP32 generating PWM was the plan. It existed because the parts needed for a DAC were in transit. Turning PWM into a voltage needs an RC filter per channel; once analog conditioning is in play anyway, the converter that just produces analog wins out.
I added a SPI panel to the RPi4 early on to see the landmark capture, distance estimation, and an OLED went on the converter card itself for the same reason one layer in.
The project touches everything, and I enjoyed doing it all. Camera and inference at one end, I²C timing and a hand-soldered board at the other, with Python and C++ as the glue.
6 · Powering it on
Nothing came out of the jacks. Then it half-worked, then software said fine and hardware said no. Then it worked while I watched it. Spooky action at a distance, surely.
The bus scan found the converter and then wouldn't write to it: the MCP4728 acknowledged its address and refused the data. I chased that as an I²C bus problem and put a scope on the ACK. When that went nowhere it became a USB permissions problem, then a firmware problem, then a bad solder joint, and the reflow that followed didn't clear it either. What it ended up being was an open in the 5V path, between the Eurorack header and the protoboard rail. Putting meter probes across the gap completed the circuit through the meter, so the converter's power LED went steady the same moment I measured it.
The diagnostic OLED redrew too slowly to read while playing. The FT232H has no I²C peripheral: pyftdi emulates I²C on the MPSSE engine, where the per-byte ACK forces a synchronous read-back. Every transaction ended with the host waiting on an IN packet that shipped only once the latency timer expired (15.6 ms of it, factory default). Dropping that timer to 1ms was worth just 14% decrease in frame write time alone.
An SSD1306 is addressed in eight-pixel-tall pages (8 rows to a page), so one byte is eight vertically stacked pixels of a single column within a page, and a vertical bar across the whole screen touches every page it spans: changing its height rewrites bytes across the whole stack. Turned sideways, each bar sits inside a single page as a contiguous run, and moving it dirties only the columns between the old end and the new one.
The other fix was to stop pushing entire frames and only push differences. The class keeps its own framebuffer and the last length of every bar, compares each new value against it, and writes only the columns that changed. That span write lights and clears in one pass, and a bar whose length has not changed is skipped before a transaction is opened.
Together those fixes took each frame from 371.6 ms to 36.8.
7 · Playing it
It engages the same attention guitar does — continuous, physical, no menu between the intent and the sound. The difference is that a fretted guitar is discrete and this isn't. There are no steps in it at all, akin to a theremin.
A hand is seldom truly still without great concentration. Combined with the slight jitter of the pipeline's processing, the signal arrives as a minute living variation.
The primary mode is explorative, where you move the fingers continuously to find sweet spots in a given module's current settings and place in the synth's signal chain. Another option is committing to a configuration and then retracting the hand from the focus cone of the camera (the code retains and relays the last processed input). Both are available in the same session and they serve different purposes to achieve different goals depending on the modular patch.
The first session lasted five or ten minutes and left me genuinely overwhelmed and exhausted from imagining the possibilities that had just opened up (in a good way). I shut it off and crashed onto an armchair to process what had happened.
8 · What I plan to do next
It runs at 26 to 30 fps with no hand in frame and 18 to 22 with one, against a camera that caps at 30. There is room for efficiency exploration now that the full system works as intended.
An alternate scheme is written but not plugged in or tested: the hand as an airplane — transX, transY, roll, pitch. No rudder, limited by wrist joint biomechanics. Same four 0–1 outputs so it drops into the identical pipeline.
Waht that would provide is four axes that are genuinely independent. Fingers are not strictly independent. A hand is a linkage, and ring and middle drag each other whether you want them to or not. Different musical control would be the result, which makes it a different instrument rather than the same one with better ergonomics.
More importantly, the ultimate goal is using the full body as a controller. The next milestone on that front is getting the system to work on two hands by upgrading the MV card with another MCP4728. I will need to design a breakout PCB adapter to wire into the power/gnd and signal/clock of the existing MCP4728, and then run four jumpers to four new jacks to carry the control voltage resulting from the second hand.
Full-body pose is where my deep muay thai background and beginner tai chi practice become truly relevant.