CyberStrum

Description

Many people want to learn how to play the guitar. But in reality… Guitars are expensive There is little time to practice Chords are hard to memorize Fingers hurt, motivation fades A guitar gets bought… and then left unused I was one of those people. I wanted to learn guitar, but work and limited free time made it hard to practice consistently. In the end, I still couldn’t play. I also remember watching Air Guitar competitions years ago. People performed without real guitars, yet the energy and emotion were real. Now we are living in the age of AI. So I asked myself: “If I can’t play a real guitar yet, can I practice the feeling of playing first?” That question became the beginning of CyberStrum.

Step 1: Designed

CyberStrum is a learning framework

that helps beginners practice guitar without owning a guitar.

The core ideas are simple:

• Train the left hand to recognize chords
• Train the right hand to feel strumming rhythm
• Practice performance and coordination before sound
• Reduce cost, frustration, and barriers to entry

CyberStrum does not replace a real guitar.

It prepares you for one.

Step 2: System Overview

CyberStrum uses two main components:

1️⃣ HuskyLens – Chord Recognition (Left Hand)

• Uses HuskyLens AI Camera
• Trained to recognize hand gestures as guitar chords
• Detects which chord shape the left hand is forming
• Acts like a visual coach saying: “Yes, that’s the chord.”

👉 Perfect for beginners who still struggle with chord shapes.

2️⃣ Raspberry Pi Pico – Strumming Detection (Right Hand)

• Uses Raspberry Pi Pico
• Detects hand movement for up/down strumming
• Converts gestures into rhythm signals
• Sends timing information for sound or visual feedback

👉 Helps build rhythm and muscle memory.

Step 3: How CyberStrum Works

🎶 How CyberStrum Works

• The left hand forms a chord gesture
• HuskyLens recognizes the chord
• The right hand performs a strumming motion
• Pico detects the strum direction and timing
• The system combines chord + rhythm
• Feedback is shown as sound, visuals, or indicators

There are no strings,

but the body begins to understand how playing feels.

🎯 Why CyberStrum Is Great for Beginners

• No need to buy a guitar first
• No finger pain
• No pressure
• Can practice in short sessions
• Learning feels playful and engaging

Most importantly:

You don’t need to be good to start.

Step 4: HuskyLens 2 – AI Vision for Chord Recognition

HuskyLens 2 is an AI vision sensor that can recognize images, objects, and gestures without complex programming.

In CyberStrum, HuskyLens 2 is used to:

• Recognize left-hand chord shapes
• Identify finger positions as different guitar chords
• Provide instant feedback on whether a chord gesture is correct
• Act as a “visual guitar teacher” for beginners

Why HuskyLens 2?

• No need for heavy AI models or cloud processing
• Easy to train using buttons (no PC required)
• Works offline and in real time
• Perfect for beginners and rapid prototyping

👉 This helps users who cannot memorize or form guitar chords yet

to visually understand and practice chord shapes first.

If you’re interested in building CyberStrum or experimenting with AI-based hand gesture recognition, the HuskyLens 2 AI camera used in this project is available from the official DFRobot store at:

https://www.dfrobot.com/product-2995.html

Step 5: TOF200_TPS – Time-of-Flight Distance Sensor (Strumming Detection)

The TOF200_TPS is a Time-of-Flight (ToF) distance sensor.

It measures how far an object is from the sensor in real time.

In CyberStrum, it is used to:

• Detect hand distance and movement speed
• Identify strumming direction (hand moving closer or farther)
• Measure motion without physical contact
• Enable smooth and responsive gesture tracking

Why a ToF Sensor?

• Works without touching anything
• Accurate and fast response
• Less sensitive to lighting than cameras
• Ideal for gesture-based interaction

👉 This allows CyberStrum to detect natural air-strumming motions,

similar to how people perform Air Guitar.

Step 6: Setting Up the Hardware

In this step, we assemble the core hardware components of CyberStrum.

The goal is to mount, align, and connect each module so the system can correctly detect chords and strumming gestures.

🧩 Components Shown in the Image

From the image, the CyberStrum hardware consists of:

• HuskyLens 2 AI Camera
• Raspberry Pi Pico (with ToF200_TPS sensor mounted on a perfboard)
• 3D-printed mounting brackets (black parts)
• 3D-printed enclosure (green case)

Each part has a specific role in the system.

Mounting the HuskyLens 2 (Chord Detection)

• Place the HuskyLens 2 in a stable, front-facing position.
• The camera should face the left hand area where chord gestures will be performed.
• Use the 3D-printed black mounting brackets to:
• Fix the HuskyLens at a stable angle
• Prevent movement during hand gestures
• Make sure the camera lens is:
• Clearly visible
• Not blocked by the enclosure
• Positioned at a comfortable distance (about 20–40 cm from the hand)

👉 Proper alignment is important for reliable chord recognition.

Step 7: System Diagram

CyberStrum Hardware Connection Diagram

This diagram illustrates how the CyberStrum system is connected. The HukSky Lens2 detects hand gestures and finger positions, while the TOF_200S sensor measures the strumming motion. Both devices are connected through a USB hub to the PC, where the Python software processes the data and converts the user's movements into real-time guitar sound.

Step 8: Software

CyberStrum is an AI-powered air guitar system that allows users to practice guitar chords and strumming without a real guitar.

The system combines:

• AI hand gesture recognition (for chord detection)
• Motion sensing (for strumming detection)
• Machine learning (to improve accuracy over time)
• A real-time desktop dashboard for interaction and feedback

CyberStrum is designed for beginners who:

• Want to learn guitar but cannot afford one
• Have limited practice time
• Struggle with chord shapes
• Want to build muscle memory before using a real instrument

🧩 System Architecture

CyberStrum consists of three main subsystems:

• Chord Detection (Vision + AI)
• Strumming Detection (Gesture Sensor + Microcontroller)
• Desktop AI Dashboard (Python Application)

🎥 Chord Detection with HuskyLens

• HuskyLens detects hand landmarks (wrist and finger tips)
• Hand position data is published via MQTT
• The Python application subscribes to the MQTT topic
• Finger coordinates are parsed and normalized
• 18 geometric features are extracted from the hand

These features describe:

• Finger distances
• Finger spread
• Palm size ratio
• Hand curl characteristics

🤖 AI-Based Chord Recognition

CyberStrum uses a machine learning ensemble model to recognize guitar chords:

• Random Forest
• K-Nearest Neighbors (KNN)
• Support Vector Machine (SVM)

The models are combined using soft voting for higher accuracy.

Key AI features:

• Automatic feature scaling
• Confidence scoring
• Stability filtering across multiple frames
• Noise augmentation during training

The system only confirms a chord when:

• Confidence exceeds a threshold
• Detection is stable across multiple frames

🎓 Learning Mode (User-Trainable AI)

CyberStrum supports custom chord learning:

• User enters a chord name in the dashboard
• The system records multiple hand frames
• Features are averaged and stored
• The AI model is retrained automatically
• Learned chords are saved to disk

This allows:

• Personalized hand shapes
• Custom chords
• Continuous improvement over time