AI Fruit Ninja with Grove Vision AI V2 and XIAO ESP32-S3

I have always enjoyed turning “recognition” into something more than recognition itself—transforming it into an interactive experience that can respond to people in real time. This time, I combined Grove Vision AI V2, XIAO ESP32-S3, and Processing to create an interactive Fruit Ninja-style project that users can control with hand gestures.

In most vision recognition systems, the model outputs nothing more than a box, a label, or a set of coordinates. But I felt that once a gesture has been recognized, it should not stop at simply producing a result. It should go one step further and become a game mechanic that users can immediately feel and interact with. That led me to think: what if Rock, Paper, and Scissors were each given their own unique visual powers? Would that be more interesting than gesture recognition alone?

I eventually chose the Fruit Ninja concept because it naturally fits a dynamic, lightweight, and highly responsive interactive game. “Paper” can push fruits away, “Rock” can attract them, and “Scissors” can slice them apart. Each gesture creates a direct and intuitive relationship with how fruits move on the screen. As a result, when users make a gesture, they are no longer simply sending an input to the system—they are directly controlling the world inside the screen.

I really love this feeling: the hand gesture in front of the camera is no longer isolated. Through the AI model, serial communication, and the visual logic inside Processing, it becomes a real-time interactive experience filled with feedback, emotion, and fun.

Project Principle

• Vision Recognition Layer

Grove Vision AI V2 is responsible for capturing images in front of the camera and running the SenseCraft AI gesture recognition model. The model detects gesture classes in real time while outputting the target box position, size, and class ID.

The three gestures are mapped as follow:

Paper = 0

Rock = 1

Scissors = 2

At this stage, Grove Vision AI V2 acts like a vision sensor. It “sees” the user’s gesture and outputs structured digital data in real time through the device log.

• Data Transmission Layer

The recognition data from Grove Vision AI V2 is sent through serial communication to the XIAO ESP32-S3. The S3 serves as a bridge and processing layer, organizing the recognition results into a format that Processing can easily read and forwarding the data stably to the visual system.

• Game Expression Layer

After receiving the data, Processing maps the gesture position and size to the corresponding area on the screen, then triggers different fruit interactions:

Paper: Pushes fruits away within the gesture rangeRock: Pulls nearby fruits to the center and spins them like a vortexScissors: Cuts fruits apart with juice splash effects

As fruits continue to spawn, the screen becomes a fully interactive gesture-controlled game scene.

Detailed Build Process

Step 1：Deploy the Gesture Recognition Model on Grove Vision AI V2

First, we need to deploy the SenseCraft AI gesture recognition model onto Grove Vision AI V2. It is recommended to choose a model that supports Rock, Paper, and Scissors, and make sure it is compatible with Grove Vision AI V2.

Open the SenseCraft AI platform, select the corresponding Gesture Recognition model, and complete deployment. Once finished, the device will begin outputting recognition results in real time, including gesture class, position, and bounding box size.

The most important point here is not just whether the recognition is correct, but also understanding the format and meaning of the output data, because the later Processing-side parsing depends entirely on this structure.

Step 2：Set Up the Software Environment

If this is your first time using Arduino IDE, it is recommended to follow a tutorial to install and configure the environment, as well as download the required libraries.

• Install Arduino IDE
• Install Grove Vision AI V2 and XIAO ESP32-S3 related libraries
• Install Processing IDE
• Configure the serial port environment

Although this step may seem basic, it is actually very important, because the later data flow depends heavily on a stable software environment.

Step 3：Hardware Connection

Prepare two USB data cables and connect Grove Vision AI V2 and XIAO ESP32-S3 to your computer.

If your serial communication requires direct pin connection, you can use Dupont jumper wires to establish UART communication. The exact wiring can be shown according to your actual hardware setup.

Step 4：Reformat Gesture Recognition Data

To make the data easier for Processing to use, we first convert the output from Grove Vision AI V2 into a cleaner unified format and forward it through XIAO ESP32-S3.

At this stage, upload data-processing code to the S3 using Arduino IDE so it can receive the recognition results from Grove Vision AI V2.

#include <Seeed_Arduino_SSCMA.h>
#ifdef ESP32
#include <HardwareSerial.h>
// Define two Serial devices mapped to the two internal UARTs
HardwareSerial atSerial(0);
#else
#define atSerial Serial1
#endif
SSCMA AI;
void setup()
{
Serial.begin
}(921600);
AI.begin(&atSerial);
void loop()
{
if (!AI.invoke(1,false,true))
{
Serial.println("invoke success");
Serial.print("perf: prepocess=");
Serial.print(AI.perf().prepocess);
Serial.print(", inference=");
Serial.print(AI.perf().inference);
Serial.print(", postpocess=");
Serial.println(AI.perf().postprocess);
for (int i = 0; i < AI.boxes().size(); i++)
{
Serial.print("Box[");
Serial.print(i);
Serial.print("] target=");
Serial.print(AI.boxes()[i].target);
Serial.print(", score=");
Serial.print(AI.boxes()[i].score);
Serial.print(", x=");
Serial.print(AI.boxes()[i].x);
Serial.print(", y=");
Serial.print(AI.boxes()[i].y);
Serial.print(", w=");
Serial.print(AI.boxes()[i].w);
Serial.print(", h=");
Serial.println(AI.boxes()[i].h);
}
for (int i = 0; i < AI.classes().size(); i++)
{
Serial.print("Class[");
Serial.print(i);
Serial.print("] target=");
Serial.print(AI.classes()[i].target);
Serial.print(", score=");
Serial.println(AI.classes()[i].score);
}
for (int i = 0; i < AI.points().size(); i++)
{
Serial.print("Point[");
Serial.print(i);
Serial.print("]: target=");
Serial.print(AI.points()[i].target);
Serial.print(", score=");
Serial.print(AI.points()[i].score);
Serial.print(", x=");
Serial.print(AI.points()[i].x);
Serial.print(", y=");
Serial.println(AI.points()[i].y);
}
for (int i = 0; i < AI.keypoints().size(); i++)
{
Serial.print("keypoint[");
Serial.print(i);
Serial.print("] target=");
Serial.print(AI.keypoints()[i].box.target);
Serial.print(", score=");
Serial.print(AI.keypoints()[i].box.score);
Serial.print(", box:[x=");
Serial.print(AI.keypoints()[i].box.x);
Serial.print(", y=");
Serial.print(AI.keypoints()[i].box.y);
Serial.print(", w=");
Serial.print(AI.keypoints()[i].box.w);
Serial.print(", h=");
Serial.print(AI.keypoints()[i].box.h);
Serial.print("], points:[");
for (int j = 0; j < AI.keypoints()[i].points.size(); j++)
{
Serial.print("[");
Serial.print(AI.keypoints()[i].points[j].x);
Serial.print(",");
Serial.print(AI.keypoints()[i].points[j].y);
Serial.print("],");
}
Serial.println("]");
}
if(!AI.last_image().isEmpty())
{
Serial.print("Last image:");
Serial.println(AI.last_image().c_str());
}
}
}

If everything works correctly, you should see real-time gesture data in the Serial Monitor, including gesture type, coordinates, and box dimensions.

Step 5：Parse Recognition Results in Processing

Next, we receive the data from XIAO ESP32-S3 in Processing and transform it into information usable by the visual system.

The key points here are:

• Handling single or multiple gesture targets
• Mapping coordinates to screen dimensions
• Using bounding box size to determine interaction range
• Assigning different gesture powers
• Maintaining stable frame rate and responsiveness
• Matching fruit physics precisely with gesture area

The core Processing code can be structured this way:

import processing.serial.*;
import java.util.*;
import java.util.regex.*;

Serial myPort;

final int BAUD = 115200;
final int PORT_INDEX = 0;   // 你的串口是 0

final float CAM_W = 320.0;
final float CAM_H = 240.0;

final int MAX_BOXES = 20;
final int MAX_FRUITS = 85;       // 提高水果总量上限
final int MAX_JUICE = 1400;
final int MAX_SLASHES = 120;
final int MAX_WAVES = 40;

final int GESTURE_KEEP_MS = 100;
final int FRUIT_HIT_GAP = 6;

ArrayList<GestureBox> currentBoxes = new ArrayList<GestureBox>();
ArrayList<GestureBox> pendingBoxes = new ArrayList<GestureBox>();

ArrayList<Fruit> fruits = new ArrayList<Fruit>();
ArrayList<JuiceParticle> juices = new ArrayList<JuiceParticle>();
ArrayList<SlashEffect> slashes = new ArrayList<SlashEffect>();
ArrayList<ImpactWave> waves = new ArrayList<ImpactWave>();

Pattern boxPattern = Pattern.compile(
  "Box\\[(\\d+)\\]\\s*target=(\\d+),\\s*score=(\\d+),\\s*x=(\\d+),\\s*y=(\\d+),\\s*w=(\\d+),\\s*h=(\\d+)"
);

int spawnTimer = 0;
int lastGestureMillis = 0;

void setup() {
  size(1280, 720);
  frameRate(60);
  smooth(8);
  background(0);
  colorMode(HSB, 360, 100, 100, 100);

  println(Serial.list());
  if (Serial.list().length > 0) {
    int idx = constrain(PORT_INDEX, 0, Serial.list().length - 1);
    myPort = new Serial(this, Serial.list()[idx], BAUD);
    myPort.bufferUntil('\n');
    println("Opened serial port: " + Serial.list()[idx]);
  } else {
    println("No serial ports found.");
  }
}

void draw() {
  noStroke();
  fill(0, 0, 0, 14);
  rect(0, 0, width, height);

  if (millis() - lastGestureMillis > GESTURE_KEEP_MS) {
    currentBoxes.clear();
  }

  emitFruitsFromBottom();
  applyGestureEffects();

  updateFruits();
  updateJuice();
  updateSlashes();
  updateWaves();

  drawFruits();
  drawWaves();
  drawSlashes();
  drawJuice();
  drawHUD();
}

void serialEvent(Serial p) {
  String line = p.readStringUntil('\n');
  if (line == null) return;

  line = trim(line);
  if (line.length() == 0) return;
  if (line.length() > 300) return;

  if (line.startsWith("Box")) {
    Matcher m = boxPattern.matcher(line);
    if (m.find()) {
      if (pendingBoxes.size() >= MAX_BOXES) return;

      GestureBox b = new GestureBox();
      b.index = int(m.group(1));
      b.target = int(m.group(2));
      b.score  = int(m.group(3));

      float x = float(m.group(4));
      float y = float(m.group(5));
      float w = float(m.group(6));
      float h = float(m.group(7));

      b.sx = map(x, 0, CAM_W, 0, width);
      b.sy = map(y, 0, CAM_H, 0, height);
      b.sw = map(w, 0, CAM_W, 0, width);
      b.sh = map(h, 0, CAM_H, 0, height);

      b.cx = b.sx + b.sw * 0.5;
      b.cy = b.sy + b.sh * 0.5;
      b.area = max(1, b.sw * b.sh);
      b.c = gestureColor(b.target, b.score);

      pendingBoxes.add(b);
    }
  }

  if (line.contains("invoke success")) {
    currentBoxes = new ArrayList<GestureBox>(pendingBoxes);
    pendingBoxes.clear();
    lastGestureMillis = millis();
  }
}

void emitFruitsFromBottom() {
  spawnTimer++;

  // 更快、更频繁地产生水果
  if (spawnTimer < 5) return;
  spawnTimer = 0;

  int spawnCount = 2;
  if (random(1) < 0.50) spawnCount++;
  if (random(1) < 0.25) spawnCount++;

  for (int i = 0; i < spawnCount; i++) {
    Fruit f = new Fruit();

    // 从底部一条区域随机“抛出”，更像水果忍者
    f.x = random(-40, width + 40);
    f.y = height + random(10, 80);

    f.size = random(34, 58);
    f.type = int(random(4));

    // 让水果以较大角度随机飞出：大部分向上，同时左右散乱
    float ang = random(radians(205), radians(335));
    float speed = random(9.0, 17.5);

    f.vx = cos(ang) * speed + random(-1.5, 1.5);
    f.vy = sin(ang) * speed + random(-1.0, 1.0);

    // 少量额外随机扰动，让轨迹更“扔出来”
    if (random(1) < 0.25) {
      f.vx *= random(0.75, 1.25);
      f.vy *= random(0.80, 1.15);
    }

    f.spin = random(-0.025, 0.025);
    f.rot = random(TWO_PI);

    fruits.add(f);
  }

  while (fruits.size() > MAX_FRUITS) {
    fruits.remove(0);
  }
}

void applyGestureEffects() {
  if (currentBoxes.size() == 0) return;

  for (GestureBox b : currentBoxes) {
    float depthFactor = gestureDepthFactor(b);
    float radius = gestureRadius(b, depthFactor);

    // 让“石头/布”本身也有明显冲击圈
    if (frameCount - b.lastWaveFrame > 4) {
      spawnImpactWave(b, depthFactor);
      b.lastWaveFrame = frameCount;
    }

    for (Fruit f : fruits) {
      if (!f.active) continue;
      if (f.sliced) continue;
      if (frameCount - f.lastGestureFrame < FRUIT_HIT_GAP) continue;

      if (b.target == 2) {
        // 剪刀：只影响手势框所在区域内的水果
        if (!fruitInScissorsArea(f, b)) continue;

        float d = dist(f.x, f.y, b.cx, b.cy);
        applySingleGestureToFruit(b, f, radius, depthFactor, d);
        f.lastGestureFrame = frameCount;
      } else {
        float d = dist(f.x, f.y, b.cx, b.cy);
        if (d > radius) continue;

        applySingleGestureToFruit(b, f, radius, depthFactor, d);
        f.lastGestureFrame = frameCount;
      }
    }
  }
}

boolean fruitInScissorsArea(Fruit f, GestureBox b) {
  float marginX = max(18, b.sw * 0.12);
  float marginY = max(18, b.sh * 0.12);

  return f.x >= b.sx - marginX &&
         f.x <= b.sx + b.sw + marginX &&
         f.y >= b.sy - marginY &&
         f.y <= b.sy + b.sh + marginY;
}

void applySingleGestureToFruit(GestureBox b, Fruit f, float radius, float depthFactor, float d) {
  if (d < 0.001) d = 0.001;

  float falloff = 1.0 - constrain(d / radius, 0, 1);
  float strength = gestureStrength(b, falloff, depthFactor);

  if (b.target == 1) {
    // 石头：更强的黑洞坍缩 + 漩涡
    float nx = (f.x - b.cx) / d;
    float ny = (f.y - b.cy) / d;

    f.vx -= nx * strength * 4.0;
    f.vy -= ny * strength * 4.0;

    float tx = -ny;
    float ty = nx;
    f.vx += tx * strength * 2.0;
    f.vy += ty * strength * 2.0;

    if (d < radius * 0.35) {
      f.vx *= 0.60;
      f.vy *= 0.60;
    }

    f.spin += random(-0.02, 0.02) * depthFactor;
    f.vy *= 0.995;
    f.vx *= 0.995;

  } else if (b.target == 0) {
    // 布：更强的爆散拍飞
    float nx = (f.x - b.cx) / d;
    float ny = (f.y - b.cy) / d;

    f.vx += nx * strength * 6.2;
    f.vy += ny * strength * 6.2;

    f.vx += random(-2.0, 2.0) * strength;
    f.vy += random(-2.0, 2.0) * strength;

    // 让拍飞更像“一巴掌”
    f.vy -= strength * 2.2;

    // 少量旋转和抖动
    f.spin += random(-0.02, 0.02) * depthFactor;
    f.vx *= 0.995;
    f.vy *= 0.995;

  } else if (b.target == 2) {
    // 剪刀：只切当前手势区域内的水果
    if (!f.sliced) {
      f.cut(b, depthFactor);
      spawnJuiceBurst(f, b, depthFactor);
      spawnSlashEffect(f, b, depthFactor);
    }
  }
}

float gestureDepthFactor(GestureBox b) {
  float minA = 700;
  float maxA = width * height * 0.14;
  float a = constrain(b.area, minA, maxA);
  float t = map(a, minA, maxA, 0, 1);

  // 手越靠近镜头，面积越大，影响越强
  return lerp(0.75, 2.55, t) * map(b.score, 0, 100, 0.92, 1.12);
}

float gestureRadius(GestureBox b, float depthFactor) {
  float base = max(b.sw, b.sh);
  return (base * 0.75 + 30) * depthFactor;
}

float gestureStrength(GestureBox b, float falloff, float depthFactor) {
  float scoreFactor = map(b.score, 0, 100, 0.85, 1.25);
  return falloff * depthFactor * scoreFactor * 1.35;
}

void spawnImpactWave(GestureBox b, float depthFactor) {
  ImpactWave w = new ImpactWave();
  w.x = b.cx;
  w.y = b.cy;
  w.target = b.target;

  float base = max(b.sw, b.sh);
  w.r = max(20, base * 0.35);
  w.maxR = base * (1.7 + depthFactor * 0.7);
  w.life = int(8 + depthFactor * 5);
  w.maxLife = w.life;

  if (b.target == 1) {
    w.c = color(225, 80, 100, 100);
  } else if (b.target == 0) {
    w.c = color(190, 70, 100, 100);
  } else {
    w.c = color(0, 0, 100, 100);
  }

  waves.add(w);

  while (waves.size() > MAX_WAVES) {
    waves.remove(0);
  }
}

void spawnSlashEffect(Fruit f, GestureBox b, float depthFactor) {
  SlashEffect s = new SlashEffect();
  s.x = f.x;
  s.y = f.y;

  float ang = atan2(f.y - b.cy, f.x - b.cx);
  s.angle = ang + HALF_PI;

  s.life = int(10 + depthFactor * 4);
  s.maxLife = s.life;

  s.length = f.size * (2.2 + depthFactor * 0.55);
  s.thickness = max(3.0, f.size * 0.12 * depthFactor);
  s.tint = fruitSlashTint(f.type);

  slashes.add(s);

  // 近镜头时再补一条更炸裂的刀光
  if (depthFactor > 1.55) {
    SlashEffect s2 = new SlashEffect();
    s2.x = f.x + random(-4, 4);
    s2.y = f.y + random(-4, 4);
    s2.angle = s.angle + random(-0.12, 0.12);
    s2.life = int(8 + depthFactor * 3);
    s2.maxLife = s2.life;
    s2.length = f.size * (1.7 + depthFactor * 0.42);
    s2.thickness = max(2.0, f.size * 0.08 * depthFactor);
    s2.tint = fruitSlashTint(f.type);
    slashes.add(s2);
  }

  while (slashes.size() > MAX_SLASHES) {
    slashes.remove(0);
  }
}

void spawnJuiceBurst(Fruit f, GestureBox b, float depthFactor) {
  int count;
  if (f.type == 0) {
    count = int(map(depthFactor, 0.70, 2.20, 12, 26));
  } else if (f.type == 1) {
    count = int(map(depthFactor, 0.70, 2.20, 10, 22));
  } else if (f.type == 2) {
    count = int(map(depthFactor, 0.70, 2.20, 14, 30));
  } else {
    count = int(map(depthFactor, 0.70, 2.20, 16, 34));
  }

  count = int(count * map(b.score, 0, 100, 0.9, 1.2));
  count = constrain(count, 10, 38);

  for (int i = 0; i < count; i++) {
    JuiceParticle j = new JuiceParticle();
    j.x = f.x;
    j.y = f.y;
    j.prevX = f.x;
    j.prevY = f.y;

    float ang = random(TWO_PI);
    float power = random(1.8, 7.0) * depthFactor;

    float biasX = (b.cx < f.x ? 1 : -1) * map(b.sw, 20, width, 0.3, 1.2);
    float biasY = (b.cy < f.y ? 1 : -1) * map(b.sh, 20, height, 0.3, 1.2);

    if (f.type == 1) {
      // 香蕉：条状飞溅
      j.vx = cos(ang) * power * 1.25 + biasX + random(-0.7, 0.7);
      j.vy = sin(ang) * power * 0.70 + biasY + random(-0.5, 0.5);
      j.size = random(1.8, 4.8) * depthFactor;
      j.life = random(18, 34);
      j.streak = true;
    } else if (f.type == 2) {
      // 菠萝：碎裂感更强
      j.vx = cos(ang) * power + biasX + random(-1.0, 1.0);
      j.vy = sin(ang) * power + biasY + random(-1.0, 1.0);
      j.size = random(1.8, 4.6) * depthFactor;
      j.life = random(20, 40);
      j.streak = random(1) < 0.35;
    } else if (f.type == 3) {
      // 西瓜：厚重、炸裂感更明显
      j.vx = cos(ang) * power * 1.10 + biasX + random(-0.8, 0.8);
      j.vy = sin(ang) * power * 1.10 + biasY + random(-0.8, 0.8);
      j.size = random(2.0, 5.2) * depthFactor;
      j.life = random(22, 44);
      j.streak = random(1) < 0.30;
    } else {
      // 苹果：圆润喷溅
      j.vx = cos(ang) * power + biasX + random(-0.8, 0.8);
      j.vy = sin(ang) * power + biasY + random(-0.8, 0.8);
      j.size = random(1.8, 4.5) * depthFactor;
      j.life = random(18, 36);
      j.streak = random(1) < 0.20;
    }

    j.c = fruitJuiceColor(f.type);
    juices.add(j);
  }

  while (juices.size() > MAX_JUICE) {
    juices.remove(0);
  }
}

void updateFruits() {
  for (int i = fruits.size() - 1; i >= 0; i--) {
    Fruit f = fruits.get(i);
    f.update();

    if (!f.active) {
      fruits.remove(i);
    }
  }
}

void updateJuice() {
  for (int i = juices.size() - 1; i >= 0; i--) {
    JuiceParticle j = juices.get(i);
    j.update();

    if (!j.active) {
      juices.remove(i);
    }
  }
}

void updateSlashes() {
  for (int i = slashes.size() - 1; i >= 0; i--) {
    SlashEffect s = slashes.get(i);
    s.update();

    if (!s.active) {
      slashes.remove(i);
    }
  }
}

void updateWaves() {
  for (int i = waves.size() - 1; i >= 0; i--) {
    ImpactWave w = waves.get(i);
    w.update();

    if (!w.active) {
      waves.remove(i);
    }
  }
}

void drawFruits() {
  blendMode(BLEND);
  for (Fruit f : fruits) {
    if (f.active) f.draw();
  }
}

void drawWaves() {
  blendMode(ADD);
  for (ImpactWave w : waves) {
    if (w.active) w.draw();
  }
  blendMode(BLEND);
}

void drawSlashes() {
  blendMode(ADD);
  for (SlashEffect s : slashes) {
    if (s.active) s.draw();
  }
  blendMode(BLEND);
}

void drawJuice() {
  blendMode(ADD);
  for (JuiceParticle j : juices) {
    if (j.active) j.draw();
  }
  blendMode(BLEND);
}

void drawHUD() {
  blendMode(BLEND);
  fill(0, 0, 100, 30);
  noStroke();
  textSize(14);
  textAlign(LEFT, TOP);
  text("Fruits: " + fruits.size() + "   Juice: " + juices.size() + "   Slashes: " + slashes.size() + "   Boxes: " + currentBoxes.size(), 12, 12);
}

color gestureColor(int target, int score) {
  float hue;
  if (target == 0) hue = 200;
  else if (target == 1) hue = 35;
  else hue = 320;

  float bri = map(score, 0, 100, 70, 100);
  return color(hue, 85, bri, 100);
}

color fruitColor(int type) {
  if (type == 0) return color(0, 85, 95, 100);      // 苹果
  if (type == 1) return color(48, 90, 95, 100);     // 香蕉
  if (type == 2) return color(95, 75, 90, 100);     // 菠萝
  return color(120, 80, 85, 100);                   // 西瓜
}

color fruitDarkColor(int type) {
  if (type == 0) return color(18, 70, 55, 100);
  if (type == 1) return color(52, 60, 60, 100);
  if (type == 2) return color(45, 70, 55, 100);
  return color(140, 70, 55, 100);
}

color fruitLeafColor(int type) {
  if (type == 0) return color(110, 65, 50, 100);
  if (type == 1) return color(100, 70, 45, 100);
  if (type == 2) return color(110, 75, 55, 100);
  return color(115, 75, 50, 100);
}

color fruitJuiceColor(int type) {
  if (type == 0) return color(0, 70, 100, 100);
  if (type == 1) return color(50, 35, 100, 100);
  if (type == 2) return color(50, 60, 100, 100);
  return color(110, 55, 100, 100);
}

color fruitSlashTint(int type) {
  if (type == 0) return color(0, 0, 100, 100);
  if (type == 1) return color(45, 20, 100, 100);
  if (type == 2) return color(60, 20, 100, 100);
  return color(120, 15, 100, 100);
}

class GestureBox {
  int index;
  int target;
  int score;

  float sx, sy, sw, sh;
  float cx, cy;
  float area;
  color c;
  int lastWaveFrame = -9999;
}

class Fruit {
  boolean active = true;

  float x, y;
  float px, py;
  float vx, vy;
  float rot, spin;
  float size;
  int type;

  boolean sliced = false;
  int sliceAge = 0;
  float cutAngle = 0;
  float sliceOpen = 0;
  PVector cutAxis = new PVector(1, 0);

  int lastGestureFrame = -9999;

  void update() {
    px = x;
    py = y;

    if (!sliced) {
      x += vx;
      y += vy;

      vx += (noise(x * 0.008, frameCount * 0.01) - 0.5) * 0.02;
      vy += 0.015;

      vx *= 0.996;
      vy *= 0.996;

      rot += spin;
      spin *= 0.99;
    } else {
      sliceAge++;

      x += vx;
      y += vy;

      float openTarget = min(size * 0.55, sliceAge * size * 0.025);
      sliceOpen = lerp(sliceOpen, openTarget, 0.28);

      vx *= 0.992;
      vy *= 0.992;
      vy += 0.03;

      x += cutAxis.x * 0.03;
      y += cutAxis.y * 0.03;

      if (sliceAge > 30) {
        active = false;
      }
    }

    if (y < -160 || x < -160 || x > width + 160 || y > height + 220) {
      active = false;
    }
  }

  void cut(GestureBox b, float depthFactor) {
    if (sliced) return;

    sliced = true;
    sliceAge = 0;
    sliceOpen = 0;

    float ang = atan2(y - b.cy, x - b.cx);
    cutAngle = ang + random(-0.10, 0.10);
    cutAxis = new PVector(cos(cutAngle), sin(cutAngle));

    float kick = map(depthFactor, 0.70, 2.20, 0.6, 1.9);
    vx += cutAxis.x * kick * 1.0;
    vy += cutAxis.y * kick * 1.0;
    spin += random(-0.03, 0.03) * depthFactor;
  }

  void draw() {
    pushMatrix();
    translate(x, y);

    if (!sliced) {
      rotate(rot);

      if (type == 0) drawApple(size);
      else if (type == 1) drawBanana(size);
      else if (type == 2) drawPineapple(size);
      else drawWatermelon(size);

    } else {
      rotate(cutAngle);

      if (type == 0) drawAppleSliced(size);
      else if (type == 1) drawBananaSliced(size);
      else if (type == 2) drawPineappleSliced(size);
      else drawWatermelonSliced(size);

      drawSlashGlow(size);
    }

    popMatrix();
  }

  void drawSlashGlow(float s) {
    float a = map(sliceAge, 0, 12, 100, 0);
    a = constrain(a, 0, 100);

    blendMode(ADD);

    stroke(0, 0, 100, a);
    strokeWeight(max(2, s * 0.14));
    line(-s * 0.95, 0, s * 0.95, 0);

    stroke(tintForSlash(), a * 0.55);
    strokeWeight(max(1, s * 0.06));
    line(-s * 1.05, 0, s * 1.05, 0);

    blendMode(BLEND);
  }

  color tintForSlash() {
    return fruitSlashTint(type);
  }

  void drawApple(float s) {
    noStroke();
    fill(fruitColor(type));
    ellipse(0, 0, s * 0.95, s * 0.95);

    fill(fruitDarkColor(type));
    ellipse(-s * 0.18, s * 0.05, s * 0.18, s * 0.14);

    stroke(fruitDarkColor(type));
    strokeWeight(max(1, s * 0.04));
    line(0, -s * 0.44, 0, -s * 0.62);

    noStroke();
    fill(fruitLeafColor(type));
    pushMatrix();
    translate(s * 0.1, -s * 0.5);
    rotate(-0.55);
    ellipse(0, 0, s * 0.26, s * 0.12);
    popMatrix();
  }

  void drawBanana(float s) {
    noStroke();
    fill(fruitColor(type));

    beginShape();
    for (int i = 0; i <= 24; i++) {
      float t = i / 24.0;
      float ang = lerp(-PI * 0.85, PI * 0.15, t);
      float r1 = s * 0.45;
      float x1 = cos(ang) * r1;
      float y1 = sin(ang) * r1 * 0.62 - s * 0.06;
      vertex(x1, y1);
    }
    for (int i = 24; i >= 0; i--) {
      float t = i / 24.0;
      float ang = lerp(-PI * 0.85, PI * 0.15, t);
      float r2 = s * 0.26;
      float x2 = cos(ang) * r2 + s * 0.09;
      float y2 = sin(ang) * r2 * 0.55 + s * 0.03;
      vertex(x2, y2);
    }
    endShape(CLOSE);

    fill(fruitLeafColor(type));
    ellipse(s * 0.34, -s * 0.18, s * 0.12, s * 0.08);
    ellipse(-s * 0.34, s * 0.16, s * 0.12, s * 0.08);
  }

  void drawPineapple(float s) {
    noStroke();
    fill(fruitColor(type));
    beginShape();
    vertex(0, -s * 0.48);
    vertex(s * 0.33, -s * 0.18);
    vertex(s * 0.42, s * 0.18);
    vertex(0, s * 0.5);
    vertex(-s * 0.42, s * 0.18);
    vertex(-s * 0.33, -s * 0.18);
    endShape(CLOSE);

    stroke(fruitDarkColor(type));
    strokeWeight(max(1, s * 0.03));
    for (int i = -2; i <= 2; i++) {
      line(-s * 0.22, i * s * 0.12, s * 0.22, i * s * 0.12);
    }
    for (int i = -2; i <= 2; i++) {
      line(i * s * 0.12, -s * 0.2, i * s * 0.06, s * 0.26);
      line(i * s * 0.12, -s * 0.2, -i * s * 0.06, s * 0.26);
    }

    noStroke();
    fill(fruitLeafColor(type));
    for (int i = 0; i < 5; i++) {
      pushMatrix();
      rotate(-0.9 + i * 0.45);
      translate(0, -s * 0.56);
      beginShape();
      vertex(0, 0);
      vertex(s * 0.12, -s * 0.28);
      vertex(s * 0.02, -s * 0.18);
      vertex(-s * 0.08, -s * 0.28);
      endShape(CLOSE);
      popMatrix();
    }
  }

  void drawWatermelon(float s) {
    noStroke();
    fill(fruitColor(type));
    arc(0, 0, s * 1.0, s * 1.0, PI, TWO_PI);

    fill(0, 0, 95, 100);
    arc(0, 0, s * 0.78, s * 0.78, PI, TWO_PI);

    fill(fruitDarkColor(type));
    arc(0, 0, s * 1.0, s * 1.0, PI, TWO_PI);
    fill(0, 0, 95, 100);
    arc(0, 0, s * 0.82, s * 0.82, PI, TWO_PI);

    stroke(20, 80, 70, 100);
    strokeWeight(max(1, s * 0.02));
    for (int i = -2; i <= 2; i++) {
      float xx = i * s * 0.12;
      line(xx, -s * 0.03, xx, s * 0.34);
    }

    fill(fruitLeafColor(type));
    ellipse(-s * 0.33, -s * 0.05, s * 0.08, s * 0.05);
  }

  void drawAppleSliced(float s) {
    float gap = min(sliceOpen, s * 0.42);
    float r = s * 0.48;

    pushMatrix();
    translate(-gap * 0.5, 0);
    noStroke();
    fill(0, 80, 85, 100);
    beginShape();
    vertex(0, -r);
    for (float a = HALF_PI; a <= 3 * HALF_PI + 0.001; a += PI / 18.0) {
      vertex(cos(a) * r * 0.98, sin(a) * r * 0.98);
    }
    vertex(0, r);
    endShape(CLOSE);

    fill(0, 10, 98, 100);
    beginShape();
    vertex(0, -r * 0.88);
    for (float a = HALF_PI; a <= 3 * HALF_PI + 0.001; a += PI / 20.0) {
      vertex(cos(a) * r * 0.82, sin(a) * r * 0.82);
    }
    vertex(0, r * 0.88);
    endShape(CLOSE);

    fill(20, 80, 20, 100);
    ellipse(-r * 0.18, -r * 0.12, s * 0.03, s * 0.06);
    ellipse(-r * 0.10, r * 0.05, s * 0.03, s * 0.06);
    popMatrix();

    pushMatrix();
    translate(gap * 0.5, 0);
    noStroke();
    fill(0, 80, 85, 100);
    beginShape();
    vertex(0, -r);
    for (float a = -HALF_PI; a <= HALF_PI + 0.001; a += PI / 18.0) {
      vertex(cos(a) * r * 0.98, sin(a) * r * 0.98);
    }
    vertex(0, r);
    endShape(CLOSE);

    fill(0, 10, 98, 100);
    beginShape();
    vertex(0, -r * 0.88);
    for (float a = -HALF_PI; a <= HALF_PI + 0.001; a += PI / 20.0) {
      vertex(cos(a) * r * 0.82, sin(a) * r * 0.82);
    }
    vertex(0, r * 0.88);
    endShape(CLOSE);

    fill(20, 80, 20, 100);
    ellipse(r * 0.12, -r * 0.06, s * 0.03, s * 0.06);
    ellipse(r * 0.18, r * 0.10, s * 0.03, s * 0.06);
    popMatrix();

    stroke(0, 0, 100, 70);
    strokeWeight(max(1, s * 0.04));
    line(0, -r * 0.95, 0, r * 0.95);
  }

  void drawBananaSliced(float s) {
    float gap = min(sliceOpen, s * 0.34);

    pushMatrix();
    translate(-gap * 0.5, 0);
    drawBananaSegment(s, 0.00, 0.50);
    popMatrix();

    pushMatrix();
    translate(gap * 0.5, 0);
    drawBananaSegment(s, 0.50, 1.00);
    popMatrix();

    stroke(0, 0, 100, 70);
    strokeWeight(max(1, s * 0.04));
    line(-s * 0.02, -s * 0.05, s * 0.02, s * 0.05);
  }

  void drawBananaSegment(float s, float t0, float t1) {
    noStroke();
    fill(50, 90, 95, 100);

    beginShape();
    for (int i = 0; i <= 12; i++) {
      float t = lerp(t0, t1, i / 12.0);
      float ang = lerp(-PI * 0.85, PI * 0.15, t);
      float r1 = s * 0.45;
      float x1 = cos(ang) * r1;
      float y1 = sin(ang) * r1 * 0.62 - s * 0.06;
      vertex(x1, y1);
    }
    for (int i = 12; i >= 0; i--) {
      float t = lerp(t0, t1, i / 12.0);
      float ang = lerp(-PI * 0.85, PI * 0.15, t);
      float r2 = s * 0.26;
      float x2 = cos(ang) * r2 + s * 0.09;
      float y2 = sin(ang) * r2 * 0.55 + s * 0.03;
      vertex(x2, y2);
    }
    endShape(CLOSE);

    fill(45, 35, 100, 100);
    beginShape();
    for (int i = 0; i <= 10; i++) {
      float t = lerp(t0, t1, i / 10.0);
      float ang = lerp(-PI * 0.85, PI * 0.15, t);
      float r1 = s * 0.38;
      float x1 = cos(ang) * r1;
      float y1 = sin(ang) * r1 * 0.58 - s * 0.03;
      vertex(x1, y1);
    }
    for (int i = 10; i >= 0; i--) {
      float t = lerp(t0, t1, i / 10.0);
      float ang = lerp(-PI * 0.85, PI * 0.15, t);
      float r2 = s * 0.20;
      float x2 = cos(ang) * r2 + s * 0.07;
      float y2 = sin(ang) * r2 * 0.52 + s * 0.02;
      vertex(x2, y2);
    }
    endShape(CLOSE);

    stroke(40, 20, 80, 60);
    strokeWeight(max(1, s * 0.018));
    line(-s * 0.18, -s * 0.02, s * 0.18, s * 0.03);
  }

  void drawPineappleSliced(float s) {
    float gap = min(sliceOpen, s * 0.34);
    float halfW = s * 0.42;
    float halfH = s * 0.50;

    pushMatrix();
    translate(-gap * 0.5, 0);
    noStroke();
    fill(95, 75, 90, 100);
    beginShape();
    vertex(0, -halfH);
    vertex(-halfW, 0);
    vertex(0, halfH);
    endShape(CLOSE);

    fill(50, 70, 95, 100);
    beginShape();
    vertex(0, -halfH * 0.86);
    vertex(-halfW * 0.82, 0);
    vertex(0, halfH * 0.86);
    endShape(CLOSE);

    stroke(35, 50, 55, 100);
    strokeWeight(max(1, s * 0.03));
    line(-halfW * 0.30, -halfH * 0.45, -halfW * 0.05, halfH * 0.42);
    line(-halfW * 0.08, -halfH * 0.36, -halfW * 0.36, halfH * 0.18);
    popMatrix();

    pushMatrix();
    translate(gap * 0.5, 0);
    noStroke();
    fill(95, 75, 90, 100);
    beginShape();
    vertex(0, -halfH);
    vertex(halfW, 0);
    vertex(0, halfH);
    endShape(CLOSE);

    fill(50, 70, 95, 100);
    beginShape();
    vertex(0, -halfH * 0.86);
    vertex(halfW * 0.82, 0);
    vertex(0, halfH * 0.86);
    endShape(CLOSE);

    stroke(35, 50, 55, 100);
    strokeWeight(max(1, s * 0.03));
    line(halfW * 0.05, -halfH * 0.45, halfW * 0.30, halfH * 0.42);
    line(halfW * 0.08, -halfH * 0.36, halfW * 0.36, halfH * 0.18);
    popMatrix();

    stroke(0, 0, 100, 65);
    strokeWeight(max(1, s * 0.035));
    line(0, -halfH * 0.96, 0, halfH * 0.96);
  }

  void drawWatermelonSliced(float s) {
    float gap = min(sliceOpen, s * 0.36);
    float r = s * 0.48;

    pushMatrix();
    translate(-gap * 0.5, 0);
    noStroke();
    fill(120, 80, 85, 100);
    beginShape();
    vertex(0, -r);
    for (float a = HALF_PI; a <= 3 * HALF_PI + 0.001; a += PI / 18.0) {
      vertex(cos(a) * r * 0.98, sin(a) * r * 0.98);
    }
    vertex(0, r);
    endShape(CLOSE);

    fill(0, 75, 95, 100);
    beginShape();
    vertex(0, -r * 0.88);
    for (float a = HALF_PI; a <= 3 * HALF_PI + 0.001; a += PI / 20.0) {
      vertex(cos(a) * r * 0.82, sin(a) * r * 0.82);
    }
    vertex(0, r * 0.88);
    endShape(CLOSE);

    stroke(0, 0, 15, 100);
    strokeWeight(max(1, s * 0.025));
    ellipse(-r * 0.20, -r * 0.10, s * 0.03, s * 0.07);
    ellipse(-r * 0.10, r * 0.06, s * 0.03, s * 0.07);
    popMatrix();

    pushMatrix();
    translate(gap * 0.5, 0);
    noStroke();
    fill(120, 80, 85, 100);
    beginShape();
    vertex(0, -r);
    for (float a = -HALF_PI; a <= HALF_PI + 0.001; a += PI / 18.0) {
      vertex(cos(a) * r * 0.98, sin(a) * r * 0.98);
    }
    vertex(0, r);
    endShape(CLOSE);

    fill(0, 75, 95, 100);
    beginShape();
    vertex(0, -r * 0.88);
    for (float a = -HALF_PI; a <= HALF_PI + 0.001; a += PI / 20.0) {
      vertex(cos(a) * r * 0.82, sin(a) * r * 0.82);
    }
    vertex(0, r * 0.88);
    endShape(CLOSE);

    stroke(0, 0, 15, 100);
    strokeWeight(max(1, s * 0.025));
    ellipse(r * 0.10, -r * 0.06, s * 0.03, s * 0.07);
    ellipse(r * 0.18, r * 0.10, s * 0.03, s * 0.07);
    popMatrix();

    stroke(0, 0, 100, 55);
    strokeWeight(max(1, s * 0.04));
    line(0, -r * 0.95, 0, r * 0.95);
  }
}

class JuiceParticle {
  boolean active = true;

  float x, y;
  float prevX, prevY;
  float vx, vy;
  float life;
  float size;
  color c;
  boolean streak = false;

  void update() {
    prevX = x;
    prevY = y;

    x += vx;
    y += vy;

    vx *= 0.98;
    vy *= 0.98;
    vy += 0.08;

    life -= 1.0;
    if (life <= 0 || x < -50 || x > width + 50 || y > height + 80) {
      active = false;
    }
  }

  void draw() {
    float a = map(life, 0, 55, 0, 90);
    a = constrain(a, 0, 90);

    stroke(c, a);
    if (streak) {
      strokeWeight(size * 1.2);
      line(prevX, prevY, x, y);
    } else {
      strokeWeight(size);
      line(prevX, prevY, x, y);
    }

    noStroke();
    fill(c, a);
    ellipse(x, y, size * 1.1, size * 1.1);
  }
}

class SlashEffect {
  float x, y;
  float angle;
  float life;
  float maxLife;
  float length;
  float thickness;
  color tint;
  boolean active = true;

  void update() {
    life -= 1.0;
    if (life <= 0) active = false;
  }

  void draw() {
    float t = life / maxLife;
    float a = 100 * t;

    pushMatrix();
    translate(x, y);
    rotate(angle);

    blendMode(ADD);

    stroke(0, 0, 100, a);
    strokeWeight(thickness * 2.0);
    line(-length * 0.55, 0, length * 0.55, 0);

    stroke(tint, a * 0.55);
    strokeWeight(thickness);
    line(-length * 0.48, 0, length * 0.48, 0);

    stroke(0, 0, 100, a * 0.55);
    strokeWeight(max(1, thickness * 0.35));
    line(-length * 0.64, -thickness * 0.15, length * 0.64, -thickness * 0.15);
    line(-length * 0.64, thickness * 0.15, length * 0.64, thickness * 0.15);

    blendMode(BLEND);
    popMatrix();
  }
}

class ImpactWave {
  float x, y;
  float r;
  float maxR;
  int life;
  int maxLife;
  int target;
  color c;
  boolean active = true;

  void update() {
    life--;
    r = lerp(r, maxR, 0.26);
    if (life <= 0) active = false;
  }

  void draw() {
    float a = map(life, 0, maxLife, 0, 100);
    a = constrain(a, 0, 100);

    pushMatrix();
    translate(x, y);

    blendMode(ADD);

    noFill();
    stroke(c, a);
    strokeWeight(max(2, maxR * 0.035));
    ellipse(0, 0, r * 2.0, r * 2.0);

    stroke(c, a * 0.6);
    strokeWeight(max(1, maxR * 0.016));
    for (int i = 0; i < 8; i++) {
      float ang = TWO_PI * i / 8.0 + frameCount * 0.01;
      float x1 = cos(ang) * r * 0.65;
      float y1 = sin(ang) * r * 0.65;
      float x2 = cos(ang) * r * 1.02;
      float y2 = sin(ang) * r * 1.02;
      line(x1, y1, x2, y2);
    }

    if (target == 1) {
      stroke(225, 80, 100, a * 0.35);
      strokeWeight(max(1, maxR * 0.012));
      ellipse(0, 0, r * 1.35, r * 1.35);
    } else if (target == 0) {
      stroke(190, 75, 100, a * 0.35);
      strokeWeight(max(1, maxR * 0.012));
      ellipse(0, 0, r * 1.45, r * 1.45);
    }

    blendMode(BLEND);
    popMatrix();
  }
}

If the Processing window does not align with the camera detection area, coordinate mapping is usually the first thing to check.

Step 6：Implement Fruit Generation Logic

Once Processing successfully receives gesture data, it can begin driving the fruit system.

In this project, I designed a continuous fruit spawning system so fruits constantly appear on screen, creating a dynamic “ready to slice” state. This makes gesture actions feel more meaningful and increases interactivity.

The key points of fruit generation are:

• Fruits spawn randomly from the screen
• Each fruit has its own movement path
• Fruit positions and states update continuously
• Gestures create strong visible reactions

However, I have to admit that the fruit launching motion is not yet realistic enough. It should feel more affected by gravity—for example, different fruit types could have different parabolic trajectories instead of all flying upward in the same way. There are also some small imperfections. If you have interesting ideas to make this project more polished, feel free to leave a comment and share your own creations.

Step 7：Implement the Three Gesture Effects

This is the most fun part of the entire project.

Paper: Push Effect

When the system detects “Paper, ” it applies an outward force to fruits within the gesture area, rapidly pushing them away like a slap.

Rock: Black Hole Attraction Effect

When “Rock” is detected, it creates a black-hole-like attraction center in the area, pulling fruits inward and making them spin around the center.

Scissors: Slice and Juice Splash Effect

When “Scissors” is detected, fruits inside the area are cut apart with splash particles, creating a satisfying impact effect.These three effects turn Rock, Paper, and Scissors from simple classification outputs into three actual powers that can reshape the world on the screen.

At this point, the system has transformed simple gesture recognition into a real-time interactive game with clear feedback and fun mechanics.

This project reminded me once again that AI vision does not have to be used only for “recognition.” It can also be used for play. When recognition results are no longer just labels, but rules that can directly change what happens on screen, the system becomes more than a technical demo—it becomes a genuine interactive experience.

I really enjoy the process of translating real-world movements into on-screen actions. A user’s hand gesture is no longer just an input—it becomes a real force inside the game world. It creates the feeling that every movement in front of the camera is quietly shaping the digital world.

If you also enjoy combining AI, hardware, and visual effects into creative projects, try building your own interactive game. Perhaps with your improvements, the fruits will be sliced even better than mine.

Conclusion

This project reminded me once again that AI vision does not have to be used only for “recognition.” It can also be used for play. When recognition results are no longer just labels, but rules that can directly change what happens on screen, the system becomes more than a technical demo—it becomes a genuine interactive experience.

I really enjoy the process of translating real-world movements into on-screen actions. A user’s hand gesture is no longer just an input—it becomes a real force inside the game world. It creates the feeling that every movement in front of the camera is quietly shaping the digital world.

If you also enjoy combining AI, hardware, and visual effects into creative projects, try building your own interactive game. Perhaps with your improvements, the fruits will be sliced even better than mine！