Wekinator Explained: Teaching a Computer to Turn Gestures Into Music
Wekinator is free, open-source software created by computer scientist Rebecca Fiebrink that lets musicians, artists and students build machine learning models by giving examples instead of writing complex code. You show it inputs, such as hand positions from a webcam or values from a sensor, and tell it what sound or control value each input should produce. After training, it maps new movements to sound in real time. Since its release around 2009, it has been used in workshops, university courses and experimental performances to create new digital instruments.
This guide explains how the tool works, why its approach matters, and what it teaches about AI in music more generally.
The idea: learning by demonstration
Traditional software requires you to program every rule. If you want raising your hand to make a note higher, you would write code that measures height and calculates pitch. Wekinator replaces that with demonstration. You raise your hand, play the high note, record a few examples, then lower your hand with a low note. The software learns the relationship.
This approach is called interactive machine learning, because the human trains, tests and adjusts the model in short cycles.
How a session works
Choose inputs
Inputs can come from a webcam, a game controller, a phone's motion sensors, a microphone or a microcontroller board. Wekinator receives them through a simple network message format called OSC.
Choose outputs
Outputs are numbers sent to a sound program, such as pitch, volume, filter brightness or which sample to trigger.
Record examples
You perform a gesture while setting the desired output, repeating with variations.
Train and play
With a click, the model trains. You move again and immediately hear what it learned. If it behaves oddly, you add examples and retrain.
Three types of mapping
- Classification: recognising distinct poses, for example open hand versus fist, to trigger different sounds.
- Regression: producing smooth continuous values, such as sliding pitch as an arm moves.
- Dynamic time warping: recognising a whole movement over time, like drawing a circle in the air.
Why musicians like it
The approach feels close to practising an instrument. You try something, listen, adjust and try again. Because training takes seconds, creative experimentation stays fluid. It is also accessible: people without programming backgrounds can design responsive instruments, which is why the tool appears in accessibility projects where performers with limited movement design controls that suit their own bodies.
Lessons about AI in general
Wekinator makes several abstract AI concepts tangible.
- Data shapes behaviour: the model only knows what your examples show.
- Too few examples make it unreliable; contradictory examples confuse it.
- Testing with real use reveals problems that look fine on paper.
- The human designer remains responsible for choices and for how the system is used.
These points apply to every AI music system, including large generative models and practice feedback engines.
A classroom project idea
Teachers can run a simple activity. Students pair up, connect a webcam to Wekinator and a basic synthesiser, then design a gesture instrument with three poses, each triggering one chord. Next, they compose a short piece for the instrument and perform it for the class. Finally, they compare it with playing the same chords on a keyboard: which felt more expressive, which was more accurate, and why? The discussion connects technology, performance and music theory in one lesson, and students usually discover that clear examples matter more than clever code.
Where the piano fits
Gesture instruments do not replace keyboard skills. The piano offers precise pitch, dynamic control and centuries of repertoire. Yet experimenting with Wekinator can deepen a pianist's awareness of movement: how arm weight shapes tone, how preparation before a leap matters, and how gestures communicate to an audience.
FAQ
Q: Is Wekinator free?
A: Yes. It is open-source software that anyone can download and use.
Q: Do I need programming skills?
A: Not for basic projects. Connecting new sensors may require some setup, but training works by example.
Q: What is interactive machine learning?
A: A process where a person repeatedly trains, tests and adjusts a model in quick cycles.
Q: Can it work with a piano?
A: Yes, it can take MIDI or audio features as inputs, though most projects use movement sensors or cameras.
Explore music and technology with TIGA ALPHA
TIGA ALPHA is an AI piano teaching app built by a Thai team, with lessons in Thai, English and Mandarin. Its CEO, Teacher Aum, has 20 years of experience in piano teaching, and that music teaching background shapes every lesson, while piano teaching with AI gives you feedback each time you play piano. Use it as an online piano lessons app for short daily sessions, follow it as a complete AI online piano course, or combine it with a piano course or piano learning course led by a teacher. Whether you want to learn piano from zero, learn piano online after a long break, learn music alongside your children or simply play music for fun, this online piano teaching application keeps your next step clear. New users get a 7-day free Premium trial, then 799 THB/month. Open the piano teaching app below and pick a first lesson that fits your week.
- Download and start: TIGA ALPHA landing page
- In-person lessons in Thailand: TIGA Studio
- About the company: tigaalpha.com
A good way to understand AI is to train a small model yourself and watch what it learns.