AI Motion Camera Prototype

Problem Addressing

• Lots of people go to some live events and keep themselves busy on record and miss the event practically by being busy with the camera screen.

• People who are without DSLR expert or with kids can't take there whole family on the record.

• To shoot action moves when on move

These are the few as example where one would wish his camera to be self operating. There are cameras which can shoot pic based on subject condition or timing but can't follow the subject.

Solution

This project provides solution with AI based camera head. This camera head, and so the camera, moves horizontally along with the subject. For demo, with limited resources, camera movement is shown only in X axis, which can easily be updated let it move in 3D by spending little more bucks. My purpose here is to see and demonstrate things in working state. More importantly, as this camera is built upon Android Things OS, it can easily be configured for updates on additional features.

Solution in the nutshell

The camera is mounted over a motorised platform and focused to the subject. Based on subject movement, the camera base slides and rotates to keep the subject on focus (controlled by the application configuration by the user). The solution is based on an AI solution running on Android things based module (iMX or RPi). Solution includes Tensorflow and machine vision. The solution includes sensors and actuator components as well.

Camera Module

We will be using separate camera for vision processing. We are avoiding to use DSLR camera preview to obsolete any camera model dependency and so any camera device could be mounted on top of the camera head.

Quick rundown

• Person shows his face to the camera (camera module)

• Android face detection API identifies the face and provides the face boundary of the face in frame.

• From face boundary, centre of the face and so offset of face centre from the camera preview centre is calculate.

• And object tracking is triggered by passing preview image buffer, along with boundary box, to correlation filter library.

• Next successive preview images are kept sending to correlation filter. Here point to be noted that we are not tracking face, but the correlation. As face landmark could easily be unvisible if subject moves or turnarund or go far from camera. This is the critical scenario for how to track the object. And so by applying correlation filter.

• Correlation filter returns the tracker object coordinate. Coordinate centre offset is calculated from the preview centre. And offset is turned into stepper motor movement. As we are moving camera in X axis we will consider only X axis offset.

• Stepper motor rotation direction and speed is sent to ATMega328N which in turn drives the stepper motor via motor driver.

Face detection

The concept starts with mode, to detect face using android vision API. As we are not certain about face being always visible to camera, we won't go with face tracking. Instead, we will go with correlation tracker which is best suitable here.

FaceDetector detector = new FaceDetector.Builder(context)
        .setClassificationType(FaceDetector.NO_LANDMARKS)
        .setProminentFaceOnly(true)
        .setTrackingEnabled(true)
        .build();

detector.setProcessor(
 new MultiProcessor.Builder<>(new GraphicFaceTrackerFactory())
                .build()); 

Object Tracking

Instead of following object feature, correlation tracker implements correlation filter to track object within shown boundary box. Which means, once object is shown in a frame of a video, successive video frame are applied with filter to find bounded boxed image and used for tracking object in next frame, until and unless object is lost. We will be using dlib library which implements correlation tracker. Once object is identified at a location, camera will give offset feedback to stepper motor which then moves to take object to the center of the viewed image. So it is a closed loop between camera feedback and motor movement or motor steps in correspondence. Here we are implementing solution to move camera only in X axis or in Z axis. I am using movement in X axis with having setup rail.

dlib is available in C++ APIs and so we will implement JNI interface and will comple dlib, along with jni, with Android NDK. Folling are few interfaces to start with dlib call

extern "C"
JNIEXPORT

jboolean JNICALL
Java_com_example_abhishek_myapplication_MainActivity_initTrackingJNI(JNIEnv *env,
  jobject instance,
  jobject bitmap//,
  jintArray rectangle_) {
  ..
  ..

}

extern "C"
JNIEXPORT

jintArray JNICALL
Java_com_example_abhishek_myapplication_MainActivity_getPositionUpdateJNI(JNIEnv *env,
 jobject instance,
 jobject bitmap) {
  ..
  ..
}

extern "C"
JNIEXPORT

jboolean JNICALL
Java_com_example_abhishek_myapplication_MainActivity_stopTrackingJNI(JNIEnv *env,
 jobject instance) {
  ..
  ..
}

And make file to compile jni

LOCAL_PATH := $(call my-dir) 
# ======================================================= 
include $(CLEAR_VARS) 
LOCAL_MODULE := native-lib 
LOCAL_SRC_FILES += \ 
           ../../../MyApplication/app/src/main/cpp/native-lib.cpp 
LOCAL_LDLIBS += -lm -llog -ldl -lz -ljnigraphics 
LOCAL_CPPFLAGS += -fexceptions -frtti -std=c++11 
# import dlib 
LOCAL_STATIC_LIBRARIES += dlib 
### import miniglog 
ifeq ($(MINIGLOG_LIB_TYPE),SHARED) 
   LOCAL_SHARED_LIBRARIES += miniglog 
else 
   LOCAL_STATIC_LIBRARIES += miniglog 
endif 
include $(BUILD_STATIC_LIBRARY) 

And native AI for tracking object is as:

correlation_tracker tracker; 
tracker.start_track(img, centered_rect(point(x,y), dx, dy)); 
..
..
load_image(img, preview_image); 
tracker.update(img);
..
..
tracker.get_position(); // get the position shift update

And received position shift is compared with with preview screen centre

float offsetX = PREVIEW_WIDTH/2 - (dlibFace.getPosition().x 
                                + dlibFace.getWidth()/2);
float offsetY = PREVIEW_HEIGHT/2 - (dlibFace.getPosition().y 
                                 + dlibFace.getHeight()/2);

This offset is sent to ATmega328P over serial interface, we are using USB which we can change to over board pinout.

Stepper motor control

We will be using stepper motor and L298N to drive stepper motor. And controller is implemented at ATmega328P and we will be using Arduino Uno for this purpose. Android things continuously sends serial data for motor movement direction. No direction means no movement.

const int pin_x1 = 8;  //in1
const int pin_x2 = 9;  //in2
const int pin_x3 = 10; //in3
const int pin_x4 = 11; //in4

Stepper myStepper(stepsPerRevolution, 8,9,10,11);  
void slide() { 
 // move the camera base 
  Serial.println("clockwise"); 
 myStepper.step(step_count*clock_direction); 
 delay(delay_ms);
 ..
   

These pins are connected to L298N in1, in2, in3 and in4.

Out1 to out4 are connected to stepper motor, it is worth to see how internally these four wires are connected within stepper motor.

Hardware Setup

I do have 3D printer DIY. So, instead of buying the stuff, I did make use of its parts for building motion video camera platform. Did disassembled the 3D printer and taken out following need hardware. This actually I felt the best to build a prototype by having things in working state and gentle appearance of the complete project.

I used CNC stepper motor, SL42STH40 1684A. Which works well with the torque to move DSLR. We can alter stepper motor with considering torque.

Two iron rods to build rail and to move camera on top of it.

Used base to mount camera along on top of camera head.

Above image shows how rail would fit with camera base. As this is part of DIY 3D printer, this is obviously a good with alignment.

L298N is a very common motor driver for either stepper stepper, servo or DC motor. This motor has four out and so we can connect one stepper motor per L298N module. This motor driver module is configured for 12V power through jumper, as we are supplying 12V externally, needed for stepper motor.

Used power supply, S-180-12, which is of 180W and provides 12V of power and good enough to drive stepper motor. It can take load of 15A at 12V. Obviously, there are about five such stepper motors were used in 3D printer so, this rating has more room for us to upgrade our solution to all 3 dimensional movement of the camera using camera, which I will try later sometime if there is any motivation to do so.

Complete setup at working condition would look like this, except for the camera which I am holding to capture this image J

I did use Arduino Uno to drive the motor, as this will keep more pins open to give all three dimensional movement. X, Y and Z axis. For now I did it for only X axis as I wanted to see the one motor in working state.

Who can use it

• Sports lovers

• Stage speakers

• Event organizer to take live from different angles with multiple cameras

• Self employed freelancers for product demo

Conclusion

With Android things, a product built could be scaled up to integrate with additional electr-omechanical parts in robotics and merely firmware upgrade at software would make things much easier. Things could be remotely configured as well for any tuning (refer google firebase based camera application for configuration).