Target Seeking Spider Bot

Motivation

I have this Combat Creature Attacknid that I want to add face seeking capabilities. The idea is to find a familiar object, like a face. Center on the target and lock in and shoot darts! It would roam around the room until it finds a target. It would have a camera on top of it to get first person view.  Add drone like capabilities to the Spider bot, remotely control it. (Of course, I want some safety mechanism)

This is the next generation Spider Bot. Let's see how I made this happen

Description

Had to close my eyes just in case the dart starts shooting my face. I aimed it to my body. It's a fun project, especially figuring out the opencv and hacking attacknid. 

What you Need:

1. Special Edition Combat Creatures Attacknid. I have the bluetooth version of Attacknid, not sold anymore. Rare find.

http://www.combatcreatures.com/attacknids

2. Creator Ci20

3. USB Camera

If this project got you interested in opencv in python or nodeJS, feel free to add respect and follow me. Thanks.

Step 1: Update Debian Image on Ci20

Here are the instructions how to update the Debian Image in your Ci20

http://elinux.org/CI20_Distros

Step 2: Install Python and OpenCV

https://github.com/alduxvm/rpi-opencv

Your rpi must be connected to internet, and updated...

sudo apt-get updatesudo apt-get upgrade

Install the essentials:

sudo apt-get install python-wxgtk2.8 python-matplotlib python-opencv python-pip python-numpy

Step 3: Install NodeJS

I had trouble installing NodeJS using debian packages since Creator Ci20 is a MIPS architecture, you have to compile nodeJS from source.

https://www.vultr.com/docs/installing-node-js-from-source-on-ubuntu-14-04

Step 4: Use Wireshark to monitor bluetooth serial messages to Attacknid

The Attacknid I got is bluetooth-enabled. They don't have an API available and any documentation so I monitor the message coming from attacknid phone app to the spider bot.

http://stackoverflow.com/questions/23877761/sniffing-logging-your-own-android-bluetooth-traffic

https://www.nowsecure.com/blog/2014/02/07/bluetooth-packet-capture-on-android-4-4/

Here are the commands I was able to get

forward = "7100028080808001";
backward = "7100828080808002";
left = "7100808280808008";
right = "7100800280808004";
down = "7100808002808010";
up = "7100808082808020";
light = "7100808080800240";
fire = "7100808080808280";

Step 3: Create NodeJS program to control the attacknid

The Creator ci20 has bluetooth module so I can connect to Attacknid via bluetooth.

I used this code to control the Attacknid. To test the attacknid I used the bluetooth-serial-port package and the cylon / cylon keyboard package. The bluetooth serial port package to send commands to the Attacknid. The cylon package to map keyboard. This is already a fun project controlling the attacknid, creating an api for it.

https://github.com/rlyle78/nodebot-opencvpython/blob/master/keyboard.js

//keyboard.js
"use strict";
var Cylon = require("cylon");
var commands = {};
commands["forward"] = "7100028080808001";
commands["backward"] = "7100828080808002";
commands["left"] = "7100808280808008";
commands["right"] = "7100800280808004";
commands["down"] = "7100808002808010";
commands["up"] = "7100808082808020";
commands["light"] = "7100808080800240";
commands["fire"] = "7100808080808280";
//var selectedCommand = commands["forward"];
var btSerial = new (require('bluetooth-serial-port')).BluetoothSerialPort();
btSerial.on('found', function(address, name) {
    console.log(address);
    btSerial.findSerialPortChannel(address, function(channel) {
        btSerial.connect(address, channel, function() {
            console.log(address);
            console.log('connected');
            btSerial.on('data', function(buffer) {
              
                //console.log(buffer.toString('utf-8'));
            });
        }, function () {
            console.log('cannot connect');
        });
        // close the connection when you're ready
        btSerial.close();
    }, function() {
        console.log('found nothing');
    });
});
btSerial.inquire();
function SendCommand(command) {
    if (btSerial.isOpen()) {
        btSerial.write(new Buffer(command, 'hex'), function (err, bytesWritten) {
            if (err) console.log(err);
        });
    }
}
Cylon.robot({
  connections: {
    keyboard: { adaptor: "keyboard" }
  },
  devices: {
    keyboard: { driver: "keyboard" }
  },
  work: function(my) {
      my.keyboard.on('left', function (key) {
          console.log('left key pressed');
          SendCommand(commands['left']); });
      my.keyboard.on('right', function (key) {
          console.log('right key pressed');
          SendCommand(commands['right']);
      });
      my.keyboard.on('up', function (key) {
          console.log('up key pressed');
          SendCommand(commands['up']);
      });
      my.keyboard.on('down', function (key) {
          console.log('down key pressed');
          SendCommand(commands['down']);
      });
      my.keyboard.on('f', function (key) {
        SendCommand(commands['forward']);
      });
      my.keyboard.on('b', function (key) {
      SendCommand(commands['backward']);
      });
      my.keyboard.on('n', function (key) {
      SendCommand(commands['fire']);
      });
      my.keyboard.on('m', function (key) {
      SendCommand(commands['light']);
      });
}
}).start();

Step 4: Create a python program to control do face detection in opencv

I forked this code 

https://github.com/gr3gdev/node-python-opencv

Module NodeJS for python-opencv

This module allows to use the features of opencv via python for:

• motion detection detection
• facial recognition
• streaming a webcam

https://github.com/rlyle78/node-python-opencv

The webcam part is slow so I added some parts that I learned from this

https://github.com/alduxvm/rpi-opencv

The face detection demo runs faster on raspberry pi

https://github.com/alduxvm/rpi-opencv/blob/master/face-detection.py 

so I created a python server that would communicate with my node js app

https://github.com/rlyle78/node-python-opencv/blob/master/python/webcam_detect.py

import cv, imutils, base64, socket, sys, detect
import numpy as np
from os.path import isfile
MOVE_DETECTION = 'MOVDET'
FACE_DETECTION = 'FACDET'
FACE_RECOGNIZE = 'FACREC'
if __name__ == '__main__':
    if len(sys.argv) == 2:
        TCP_IP = '127.0.0.1'
        TCP_PORT = int(sys.argv[1])
        video = cv.CreateCameraCapture(0) #cv2.VideoCapture(0)
        width = 320 #leave None for auto-detection
        height = 240 #leave None for auto-detection
        cv.SetCaptureProperty(video,cv.CV_CAP_PROP_FRAME_WIDTH,width)
        cv.SetCaptureProperty(video,cv.CV_CAP_PROP_FRAME_HEIGHT,height)
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.bind((TCP_IP, TCP_PORT))
        s.listen(1)
        while 1:
            conn, addr = s.accept()
            data = conn.recv(16)
            try:
                if data == 'STOP':
                    break
                if data.startswith(FACE_DETECTION):
                    buffer = int(data[6:])
                    conn.send('BUFFER OK')
                    data = conn.recv(buffer)
                    print data
                    flux = data.split(' ')
                    while True:
                        image = cv.QueryFrame(video)
                        faces = detect.FaceDetection.findFrame(image, flux[0], float(flux[1]), int(flux[2]))
                        if (faces):
                            finds = '{"faces": ['
                            index = 0
                            for ((x, y, w, h), n) in faces:
                                index += 1
                                finds += '{"x": "%s", "y": "%s", "w": "%s", "h": "%s"}' % (x, y, w, h)
                                if index < len(faces):
                                    finds += ','
                            finds += ']}'
                            conn.sendall('%s' % finds)
                        if cv.WaitKey(10) >= 0:
                                break
            except:
                message = '%s' % sys.exc_value
                message = message.replace('\\', '/').replace('\n', '').replace('\r', '')
                conn.send('{"error": "%s"}' % message)
                conn.close()
                s.close()
                video.release()
                cv.destroyAllWindows()

then use this code for face detection

https://github.com/rlyle78/node-python-opencv/blob/master/python/detect.py

Step 5: Send messages from Python OpenCV to NodeJS Attacknid app via sockets

I created a Node JS app that would communicate with python OpenCV server. I started receiving coordinates and width of the detected face. I setup logic that would move Attacknid to left/right to focus on the subject. The closer the subject is, it starts shooting dart. 

var opencv = require('node-python-opencv');
var commands = {};
commands["forward"] = "7100028080808001";
commands["backward"] = "7100828080808002";
commands["left"] = "7100808280808008";
commands["right"] = "7100800280808004";
commands["down"] = "7100808002808010";
commands["up"] = "7100808082808020";
commands["light"] = "7100808080800240";
commands["fire"] = "7100808080808280";
var btSerial = new (require('bluetooth-serial-port')).BluetoothSerialPort();
btSerial.on('found', function(address, name) {
    console.log(address);
    btSerial.findSerialPortChannel(address, function(channel) {
        btSerial.connect(address, channel, function() {
            console.log(address);
            console.log('connected');
            btSerial.on('data', function(buffer) {
                //console.log(buffer);
            });
        }, function () {
            console.log('cannot connect');
        });
        // close the connection when you're ready
        btSerial.close();
    }, function() {
        console.log('found nothing');
    });
});
btSerial.inquire();
var sendCommand = function(command) {
    if (btSerial.isOpen()) {
        btSerial.write(new Buffer(command, 'hex'),
         function (err, bytesWritten) {
            if (err) console.log(err);});
    }
}
lastcommand = "";
setInterval(function () {
        //console.log(lastcommand);
        if (lastcommand != "") {
                console.log("sending " + lastcommand);
                sendCommand(lastcommand);
        }
},450);
var webcamdetector = new opencv.webcamdetector({
    port: 9010,
    useFastDetect : true
});
setTimeout(function () {
        console.log('ready');
    webcamdetector.findFaces({
            'haarcascade': 'face.xml',
            'scaleFactor': 1.2,
            'minNeighbors': 2
        }, function (data, err) {
            if (err)
                console.log("Error:" + err);
            else
            {
            // Return JSON object {faces: [{x: N, y: N, w: N, h: N}, ...]}
            console.log(JSON.stringify(data));
            if (data.faces[0])
            {
              var face = data.faces[0];
              var x = Number(face.x);
              var y = Number(face.y);
              var w = Number(face.w);
              var h = Number(face.h);
              console.log(x);
              if (w > 60 && h > 60) {
                      console.log('fire');
                      lastcommand = commands['fire'];
              }
              else if (x < 10)
                      lastcommand = commands['left'];
              else if (x > 90)
                      lastcommand = commands['right'];
              else
                      lastcommand = ""
            }
          }
        });
}, 5000);

Open 2 terminal windows

ci20@ci20:~$ python node_modules/node-python-opencv/python/webcam_detect.py 9010

in another window

ci20@ci20:~$ node docterdoom.js

Next Steps

What if it can recognize the face, look up data information about the target? With the power of the cloud, that's possible. That would be so awesome! Only shoot the right target to terminate.

If this project got you interested in using wireshark to monitor bluetooth commands or even working on opencv in python or nodeJS, feel free to add respect and follow me. Thanks.