Control Your Computer In The Air Like Tony Stark!

Introduction

Most of you might have seen that Tony Stark controls his computer in the air , when I was younger I used to always love the way tony stark simplified the interaction between the physical world and digital world.

And I would try to replicate them in my projects. And now I am happy to share with you the steps on how to write an application so that you can control your computer in the air! :)

Materials Required

Leap Motion supports Python , Java , C++ , C# , JavaScript. In this tutorial we will be working with Java.

So before we start making the program we need to have a look at the Materials Required for this Build :

• Leap Motion Controller ( Buy it here )

• Eclipse IDE for Java Developers ( Download Here )

• Leap Motion Setup File ( Download Here )

• Leap Motion SDK ( Download Here )

• Java SE 7 or later ( Download Here ) ( Download JDK and JRE )

If you don't know how to setup eclipse, I recommend this Introduction Tutorial to Java Series : Watch it Here

Ok, Now let us move to the steps involved for creating the program.

Software Setup

Leap motion supports variety of Programming Languages, like C++, C#, Python, Java, JavaScript. In this Tutorial we will be programming in Java.

Initial Setup:

• Firstly go to https://www.leapmotion.com/setup and download the Leap Motion Setup file and then install the application. It installs everything to connect the leap motion controller to your computer.

• Next go to https://developer.leapmotion.com/ ( Developer Portal ) and create an account. Then download the Leap Motion SDK.

• Then extract all the files of the zip file to any folder of your choice.

Configuring the Classpath :

In this tutorial I will be using the Eclipse IDE for Java Developers.

1. Open Eclipse , then click on File > New > Java Project .

2. Give the Project a name , and make sure you set execution runtime environment JRE to Java SE 1.7 as shown in the pictures.

3. Then click on next , the open the Libraries Tab and select "Add External JARs" and navigate to the folder you previously extracted.

4. Then open the LeapSDK folder and go to the libs folder and select the LeapJava.jar file.

5. Next click on the triangular drop down button beside LeapJava.jar and then click on Native Library Location from the Drop Down Menu , then click on edit as shown in the picture.

6. Then click on External Folder and navigate to LeapSDK folder > lib > and select x64 or x86 folder based on your Operating System and click OK and then click on finish.

We now have LeapMotion setup with our project , now lets get to some coding!

Leap Motion Listener

Now that we are done importing the libraries , let us start working on the project.

Initial Setup:

1. First we need to create a new class in the source folder , for click on the triangular drop-down arrow beside your project folder.

2. Inside you project folder right click on the src folder go to new>and click on class to create a new Java Class.

3. Give a name to the class and click finish.

Now lets start with the code:

1. First we create a Leap Motion Listener: (Note: Import the necessary libraries) , the listened detects when the leap motion controller is connected to the computer.

package starkmouse; 
import java.io.IOException; 
import java.awt.AWTException; 
import java.awt.GraphicsDevice; 
import java.awt.GraphicsEnvironment; 
import java.awt.MouseInfo; 
import java.awt.Point; 
import java.awt.Robot; 
import java.awt.event.InputEvent; 
import java.awt.event.KeyEvent; 
import com.leapmotion.leap.*; 
import com.leapmotion.leap.Controller.PolicyFlag; 
public class leapmoues { 
	public static void main(String[] args) throws AWTException { 
		Controller controller = new Controller(); 
		controller.setPolicyFlags(PolicyFlag.POLICY_BACKGROUND_FRAMES);	 
SampleListener listener = new SampleListener(); 
		controller.addListener(listener); 
		// controller.enableGesture(Gesture.Type.TYPE_SCREEN_TAP); 
		// controller.enableGesture(Gesture.Type.TYPE_SWIPE); 
		controller.enableGesture(Gesture.Type.TYPE_CIRCLE);		 
System.out.println("Press Enter to quit..."); 
		try { 
			System.in.read(); 
		} catch (IOException e) { 
			e.printStackTrace(); 
		} 
		controller.removeListener(listener); 
	} 
} 
class SampleListener extends Listener { 
	boolean readyForControl = false; 
	int screenWidth; 
	int screenHeight; 
	boolean iBoxGet = false; 
	InteractionBox iBox = null; 
	Robot robot; 
	boolean isMoving = false; 
	boolean unGrip = false; 
	boolean wasFacingDown = true; 
	boolean wasInTabState = false; 
	boolean wasTabbing = false; 
	boolean justCircleGestured = false; 
	boolean isResizing = false; 
	public void onConnect(Controller controller) { 
		System.out.println("Connected"); 
		GraphicsDevice gd = GraphicsEnvironment.getLocalGraphicsEnvironment() 
				.getDefaultScreenDevice(); 
		screenWidth = gd.getDisplayMode().getWidth(); 
		screenHeight = gd.getDisplayMode().getHeight(); 
		System.out.println("Screen Resolution: X: " + screenWidth + ", H: " 
				+ screenHeight); 
		readyForControl = true; 
		try { 
			robot = new Robot(); 
			robot.setAutoDelay(5); 
		} catch (AWTException e) { 
			// TODO Auto-generated catch block 
			e.printStackTrace(); 
		} 
	} 

Frame and Gesture Data

Leap Motion Frame Data which includes the Gestures (Circle, Pinch, Palm Face Down and all other gestures).

Leap motion has two IR Camera and three IR Led's. The leap motion forms a 3D Collection of data using the three IR Led that put out an IR Dot pattern , the IR Camera's takes pictures known as frames. The frames get sent from the leap motion device to the computer and we can program it.

The leap motion analysis the pictures and the dot positioning using complex math and then compare the data of the two pictures taken by IR Camera and the use the two 2D representation to get a 3D Representation.

The leap motion device captures at 300 fps. We then program what happens to each of the frames sent to our computer.

Code:

public void onFrame(Controller controller) {	 
	Frame frame = controller.frame(); // The latest frame 
		// Frame previous = controller.frame(1); //The previous frame 
		// System.out.println("Frame available"); 
		if (!iBoxGet) { 
			iBox = frame.interactionBox(); 
			iBoxGet = true; 
			System.out.println("Interaction box set!"); 
		} 
		// Pointable furthestFront = frame.pointables().frontmost(); 
		Hand rightHand = frame.hands().rightmost(); 
		Vector palmV = rightHand.palmVelocity(); 
		// System.out.println("Velocity: X: " + palmV.getX() + ", Y: " + 
		// palmV.getY() 
		// + ", Z: " + palmV.getZ()); 
		Vector palmN = rightHand.palmNormal(); 
		// System.out.println("Normal: X: " + palmN.getX() + ", Y: " 
		// + palmN.getY() + ", Z: " + palmN.getZ()); 
	Point mouseLoc = MouseInfo.getPointerInfo().getLocation(); 
		int currentMouseX = mouseLoc.x; 
		int currentMouseY = mouseLoc.y; 
	if (readyForControl && rightHand.confidence() > .15) { 
			if (!isMoving && !wasInTabState && frame.hands().count() > 1) { 
				Hand leftHand = frame.hands().leftmost(); 
				if (leftHand.pinchStrength() > .8 
						&& rightHand.pinchStrength() > .8) { 
					if (!isResizing) { 
						System.out.println("Resizing..."); 
						robot.keyPress(KeyEvent.VK_ALT); 
						robot.keyPress(KeyEvent.VK_SPACE); 
						robot.keyRelease(KeyEvent.VK_SPACE); 
						robot.keyRelease(KeyEvent.VK_ALT); 
						robot.keyPress(KeyEvent.VK_S); 
						robot.keyRelease(KeyEvent.VK_S); 
						robot.keyPress(KeyEvent.VK_DOWN); 
						robot.keyPress(KeyEvent.VK_RIGHT); 
						robot.keyRelease(KeyEvent.VK_DOWN); 
						robot.keyRelease(KeyEvent.VK_RIGHT); 
						isResizing = true; 
					} 
				}else{ 
				if(isResizing){ 
					System.out.println("Resizing complete!"); 
					robot.mousePress(InputEvent.BUTTON1_MASK); 
					robot.mouseRelease(InputEvent.BUTTON1_MASK); 
						isResizing = false; 
					} 
				} 
			} 
		// System.out.println("Confidence: " + rightHand.confidence()); 
		if (rightHand.grabStrength() > .99 && !wasInTabState && !isResizing) { 
				if (!isMoving && palmN.getY() < .8) { 
					robot.keyPress(KeyEvent.VK_ALT); 
					robot.keyPress(KeyEvent.VK_SPACE); 
					robot.keyRelease(KeyEvent.VK_SPACE); 
					robot.keyRelease(KeyEvent.VK_ALT); 
					robot.keyPress(KeyEvent.VK_R); 
					robot.keyRelease(KeyEvent.VK_R); 
					robot.keyPress(KeyEvent.VK_ALT); 
					robot.keyPress(KeyEvent.VK_SPACE); 
					robot.keyRelease(KeyEvent.VK_SPACE); 
					robot.keyRelease(KeyEvent.VK_ALT); 
					robot.keyPress(KeyEvent.VK_M); 
					robot.keyRelease(KeyEvent.VK_M); 
					robot.keyPress(KeyEvent.VK_DOWN); 
					robot.keyRelease(KeyEvent.VK_DOWN); 
					isMoving = true; 
				} 
				// System.out.println(rightHand.grabStrength()); 
			}  
else { 
// System.out.println("Not grabbing"); 
if (isMoving) { 
robot.mousePress(InputEvent.BUTTON1_MASK); 
robot.mouseRelease(InputEvent.BUTTON1_MASK); 
isMoving = false; 
if (palmN.getX() != 0 && palmN.getY() != 0 && palmN.getZ() != 0) { 
if (palmN.getY() < -.1 && palmN.getZ() > -.8) { 
if (currentMouseY <= 8) { 
robot.keyPress(KeyEvent.VK_WINDOWS);							 
robot.keyPress(KeyEvent.VK_UP);								 
robot.keyRelease(KeyEvent.VK_WINDOWS);
robot.keyRelease(KeyEvent.VK_UP); 
							}  
else { 
if (screenWidth - currentMouseX <= 12) { 
robot.keyPress(KeyEvent.VK_WINDOWS); 
robot.keyPress(KeyEvent.VK_RIGHT); 
robot.keyRelease(KeyEvent.VK_WINDOWS); 
robot.keyRelease(KeyEvent.VK_RIGHT); 
				}
 else if (currentMouseX <= 12) { 
 robot.keyPress(KeyEvent.VK_WINDOWS); 
 robot.keyPress(KeyEvent.VK_LEFT); 
 robot.keyRelease(KeyEvent.VK_WINDOWS); 
 robot.keyRelease(KeyEvent.VK_LEFT); 
} 
} 
} 
else { 
System.out.println("Normal: X: " + palmN.getX() 
		+ ", Y: " + palmN.getY() + ", Z: " 
		+ palmN.getZ()); 
robot.keyPress(KeyEvent.VK_ALT); 
robot.keyPress(KeyEvent.VK_SPACE); 
robot.keyRelease(KeyEvent.VK_SPACE); 
robot.keyRelease(KeyEvent.VK_ALT); 
robot.keyPress(KeyEvent.VK_N); 
robot.keyRelease(KeyEvent.VK_N); 
} 
					} 
				} 
			} 
		if (!isMoving && !isResizing) { 
				if (palmN.getY() < -.8 && palmN.getZ() > -.5) { 
					wasFacingDown = true; 
					wasTabbing = false; 
					if (wasInTabState) { 
						robot.keyPress(KeyEvent.VK_ENTER); 
						robot.keyRelease(KeyEvent.VK_ENTER); 
						wasInTabState = false; 
					} 
				} else if (palmN.getY() >= .8 && wasFacingDown 
						&& !wasInTabState) { 
					System.out.println("Alt tabbing"); 
					wasFacingDown = false; 
					wasInTabState = true; 
				wasTabbing = false; 
					robot.keyPress(KeyEvent.VK_ALT); 
					robot.keyPress(KeyEvent.VK_CONTROL); 
					robot.keyPress(KeyEvent.VK_TAB); 
					robot.delay(100); 
					robot.keyRelease(KeyEvent.VK_TAB); 
					robot.keyRelease(KeyEvent.VK_CONTROL); 
					robot.keyRelease(KeyEvent.VK_ALT); 
					try { 
						Runtime.getRuntime().exec( 
								"cmd /c start " + "C:\\WindowSwitcher.lnk"); 
					} catch (IOException e) { 
						// TODO Auto-generated catch block 
						e.printStackTrace(); 
					} 
					robot.delay(300); 
				} else if (wasInTabState && !wasFacingDown && !wasTabbing 
						&& palmN.getY() < .45) {					wasTabbing = true; 
				} else if (wasInTabState && !wasFacingDown && wasTabbing 
						&& palmN.getY() > .75) { 
					robot.keyPress(KeyEvent.VK_TAB); 
					robot.keyRelease(KeyEvent.VK_TAB); 
					wasTabbing = false; 
				} 
			}			/* 
			 * if (!isMoving && !wasInTabState) { /* if(palmN.getZ() <= -.7 && 
			 * rightHand.grabStrength() < .1){ 
			 * System.out.println("Palm vertical velocity: " + 
			 * rightHand.palmVelocity().getY()); //float resultVerticalV = 
			 * Math.round(Math.abs(rightHand.palmVelocity().getY()) - 1); 
			 * //if(resultVerticalV > 0){ robot.mouseWheel((int) 
			 * Math.round(((rightHand.palmVelocity().getY()) / 500))); //} 
			 * }else{ 
			 */ 
		if (!isMoving && !wasInTabState && frame.gestures().count() > 0 
					&& frame.hands().count() == 1 && !isResizing) { 
				CircleGesture circleGesture = new CircleGesture(frame 
						.gestures().get(0)); 
				// System.out.println("Pinch strength: " + 
				// rightHand.pinchStrength()); 
				if (circleGesture.durationSeconds() > .5 && !justCircleGestured 
						&& rightHand.pinchStrength() > .8) { 
					System.out.println("Closed a window!"); 
					robot.keyPress(KeyEvent.VK_ALT); 
					robot.keyPress(KeyEvent.VK_F4); 
					robot.keyRelease(KeyEvent.VK_F4); 
					robot.keyRelease(KeyEvent.VK_ALT); 
					justCircleGestured = true; 
				} 
			} else { 
				justCircleGestured = false; 
			} 
		float xSpeed = (palmV.getX() / 6); 
			float ySpeed = (palmV.getY() / 6); 
			// System.out.println("xSpeed: " + xSpeed + ", ySpeed: " + ySpeed); 
			robot.mouseMove((int) (currentMouseX + xSpeed), 
					(int) (currentMouseY - ySpeed)); 
			// } 
		} 
	} 
}

Running the Program and Results

Now that we have completed programming the Leap Motion Controller we can run the program in eclipse.

As you have seen in the video our program can recognize various Gestures now it's your time to add more gestures with the help from the Leap Motion Programming Guide here .

Feel free to drop your comments and query in the comments section , I will be happy to respond to them :)

I'm also on Youtube, Facebook and Linkedin , feel free me to ping me in case of any queries. :)

My Personal Blog :http://www.geevegeorge.co.vu