iLLume – Gesture-Based Smart Control for All

A touch-free, gesture-based switch system that makes everyday spaces more accessible, hygienic, and easy to use for everyone.

IntermediateWork in progress15 hours1

iLLume – Gesture-Based Smart Control for All

Things used in this project

Hardware components

Arduino UNO

ControlEverything.com 4-CHANNEL RELAY CONTROLLER FOR I2C

Camera (generic)

Jumper wires (generic)

XP Power HIGH VOLTAGE DC-DC: SIP SERIES

Software apps and online services

Arduino IDE

Microsoft VS Code

Hand tools and fabrication machines

3D Printer (generic)

Story

The Story

Our project is about creating a gesture-based, touch-free control system that allows users to operate devices like lights and appliances simply by using their fingers—no physical contact, no internet, and no complex setup required.

We decided to build this project after noticing everyday challenges faced by people, especially the elderly and physically challenged. Simple tasks like turning on a switch can be difficult when hands are full, mobility is limited, or hygiene is a concern. During the COVID-19 pandemic, this problem became even more serious, as touching public switches and shared surfaces posed a risk of infection and made people more aware of the need for contactless solutions.

To solve this, we developed a system that uses computer vision and microcontroller technology. A webcam captures the user’s hand, and the system detects how many fingers are being shown in real time. Based on this, a microcontroller (Arduino) activates the corresponding number of devices—like LEDs. For example, showing three fingers turns on three lights. The process is fast, simple, and intuitive.

This project demonstrates how technology can become more accessible, hygienic, and user-friendly, especially for people who struggle with traditional systems or complex smart devices

Python Code

import cv2
import mediapipe as mp
import serial
import time

# ==== SETUP SERIAL COMMUNICATION WITH ARDUINO ====  
try:
    arduino = serial.Serial('COM3', 9600)  # Replace with your port
    time.sleep(2)
    print("✅ Connected to Arduino on COM7")
except:
    print("❌ Could not connect to Arduino. Check the COM port.")
    arduino = None

# ==== MEDIAPIPE HAND TRACKING SETUP ====
mp_hands = mp.solutions.hands
mp_draw = mp.solutions.drawing_utils
hands = mp_hands.Hands(
    max_num_hands=1,
    min_detection_confidence=0.7,
    min_tracking_confidence=0.7
)

# ==== TRY DIFFERENT CAMERA INDEXES ====
print("🔍 Trying to access webcam...")

cap = None
for i in range(3):
    temp_cap = cv2.VideoCapture(i)
    if temp_cap.isOpened():
        cap = temp_cap
        print(f"✅ Camera opened successfully on index {i}")
        break
    else:
        print(f"❌ Camera not found on index {i}")
        temp_cap.release()

if cap is None or not cap.isOpened():
    print("🚫 No working webcam found. Exiting...")
    exit()

# ==== FUNCTION TO DETERMINE FINGER STATES ====
def get_finger_status(landmarks):
    finger_tips = [4, 8, 12, 16, 20]  # Thumb, Index, Middle, Ring, Pinky
    fingers = []

    # Thumb: compare x-axis
    if landmarks[finger_tips[0]].x < landmarks[finger_tips[0] - 1].x:
        fingers.append(1)
    else:
        fingers.append(0)

    # Other fingers: compare y-axis
    for tip in finger_tips[1:]:
        if landmarks[tip].y < landmarks[tip - 2].y:
            fingers.append(1)
        else:
            fingers.append(0)

    return ''.join(str(bit) for bit in fingers)  # e.g. '10101'

# ==== MAIN LOOP ====
while True:
    ret, frame = cap.read()
    if not ret:
        print("⚠ Failed to grab frame.")
        continue

    frame = cv2.flip(frame, 1)
    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    result = hands.process(rgb)

    if result.multi_hand_landmarks:
        for hand_landmarks in result.multi_hand_landmarks:
            mp_draw.draw_landmarks(frame, hand_landmarks, mp_hands.HAND_CONNECTIONS)

            command = get_finger_status(hand_landmarks.landmark)
            print("✋ Gesture Command:", command)

            # Send command to Arduino
            if arduino:
                arduino.write((command + "\n").encode())
                time.sleep(0.1)

    cv2.imshow("🤖 Hand Gesture Control", frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        print("👋 Exiting...")
        break

# ==== CLEANUP ====
cap.release()
cv2.destroyAllWindows()
if arduino:
    arduino.close()