Camera-Gesture Mouse Controller

Overview

This project turns your webcam into a gesture-driven mouse. The system tracks the thumb tip to move the cursor and uses pinch gestures to perform mouse actions: left-drag (thumb+index), right-click (thumb+middle), and scrolling (thumb+ring). Motion is smoothed using an Exponential Moving Average, while pyautogui injects native OS mouse events. A safe preview mode and live overlay aid debugging and calibration.

System Architecture

      Capture & I/O: cv2.VideoCapture (DirectShow on Windows) for frames.
Hand Tracking: mediapipe.solutions.hands.Hands for 21 landmarks (1 hand).
Cursor Mapping: Normalized thumb tip → screen coords via MOVE_REGION and display size.
Smoothing: Exponential Moving Average (Ema2D) to stabilize jitter.
Gesture Engine: Pinch detection (distance thresholds + debouncing) → actions.
Mouse Backend: pyautogui for move, click, drag, and scroll.
UX Overlay: On-frame diagnostics with distances, states, and FPS.
Resilience: Safe preview mode when MediaPipe init or process fails.

    

Key Features

      🧠 Thumb-driven cursor: Uses landmark 4 with EMA smoothing.
🫳 Drag & Drop: Hold thumb+index pinch to drag; release to drop.
🖱️ Right-Click: thumb+middle pinch (debounced single fire).
🧭 Scroll Mode: thumb+ring to enter; move up/down to scroll.
📐 Adjustable region: MOVE_REGION constrains motion to a comfortable box.
🧯 Safe Preview: Fallback window if MediaPipe fails to initialize or crashes.
🔧 Configurable: Tweak thresholds, smoothing, speeds, and camera settings.

    

Core Algorithms

      EMA Smoothing: ema = α·x + (1-α)·ema per axis; SMOOTHING sets α.
Pinch Detection: Euclidean distance in normalized hand space, e.g. dist(thumb, index) < PINCH_THRESH.
Debounce: Require DEBOUNCE_FRAMES consecutive frames to trigger/hold actions.
Cursor Mapping: Normalize into MOVE_REGION then scale to screen_w × screen_h.
Scroll Delta: In scroll mode, dy from anchor maps to pyautogui.scroll() scaled by SCROLL_SENS.

    

Controls

      🎯 Move: Thumb tip (landmark 4)
🫳 Left Drag: Pinch thumb + index (hold)
🖱️ Right Click: Pinch thumb + middle (debounced)
🧭 Scroll: Pinch thumb + ring and move vertically
⎋ Quit: Press q
🛑 Failsafe: Move cursor to top-left corner

    

Configuration

      SMOOTHING (0..1): higher = smoother (but laggier).
PINCH_THRESH: lower = stricter gesture recognition.
DEBOUNCE_FRAMES: raise to reduce false triggers.
CURSOR_SPEED: scale cursor movement.
MOVE_REGION: normalized box to limit motion area.
TARGET_FPS and USE_MJPG: try for camera stability.

    

Dependencies

      mediapipe==0.10.14
protobuf==3.20.3
opencv-contrib-python
numpy
pyautogui

    

Troubleshooting

      cv2 not found: Wrong interpreter. Ensure the Python running the script matches where packages are installed.
Protobuf descriptor error: Pin protobuf==3.20.3 and reinstall mediapipe.
Camera busy: Close Zoom/Discord/OBS; check OS camera permissions; try a different CAM_INDEX.

    

How It Works (Flow)

1) Open camera (DirectShow on Windows) → 2) Detect hand landmarks → 3) Thumb tip normalized → 4) Smooth with EMA → 5) Map to screen → 6) Detect pinches (index/middle/ring) with debouncing → 7) Emit OS input via pyautogui → 8) Draw overlay & FPS.

Example

Future Improvements

🖐️ Two-hand mode (one for cursor, one for gestures).
🧪 On-screen calibration for pinch threshold and region.
🎛️ GUI to adjust parameters live.
🖼️ Multi-camera selection and FPS monitor.