Hand Tracking Computer Control

I built a hand tracking system using MediaPipe in three different demonstrations.

Hand Tracking Visualization

The foundation is a debug tool that visualizes hand landmarks in real time, showing finger positions and FPS to help calibrate the camera and verify gesture recognition.

Virtual Theremin

The theremin was the world’s first electronic instrument, using capacitance to control pitch and volume without physical contact. This system maps hand positions to musical parameters.

• Right hand Y-axis controls pitch
• Volume controlled by either left hand Y-axis (two-hand mode) or right hand X-axis (one-hand mode)
• Real-time audio synthesis with multiple waveform types
• Pitch quantization to C Major scale for musical accuracy

Mouse Control System

A simple application, only yet allowing for left clicks and moving the cursor based on the distance between the thumb and index fingers. The system uses exponential smoothing to reduce cursor jitter and implements debouncing to prevent accidental clicks. Built with macOS integration using the Quartz framework for low-level mouse control.

Try It Yourself

Here’s a simple browser-based demo using MediaPipe’s hand tracking: