Turn Your Robot Into A Unity Game Controller

What if your robot could become a game controller?

In this project, we'll connect a simple 2-DOF robotic arm to a Unity game and use the physical arm to control what happens in the game.

We'll use an ESP32-WROOM-32 and two 5k potentiometers to represent the two joints of the arm. The ESP32 reads the joint positions and sends them to Unity over USB serial.

Unity then uses forward kinematics to recreate the arm in the game.

Once everything is working, we'll turn the virtual arm into a simple game controller and add physical feedback. We'll also look at how ROS 2 can be added using ROS-TCP-Endpoint if you want to connect the game to a larger robotics system.

You don't need ROS 2 to follow the main tutorial.

What We're Building

The basic idea is pretty simple. You move the physical arm, the virtual arm follows, and the virtual arm can then interact with the game.

For this project, we're using two potentiometers instead of a complete robotic arm. This keeps the hardware simple while still giving us the same basic joint-position data that a real robot would provide.

How It Works

There are three main parts to the project.

• The ESP32 reads the physical joint positions and sends them to the computer.
• Unity receives those positions, works out where the virtual robot should be, and runs the game.
• ROS 2 is optional. If we add it later, it gives Unity a way to communicate with other robotics software.

1. Build the Physical Controller

We'll use two 5k potentiometers to represent the two joints of a 2-DOF arm, where Potentiometer 1 and Potentiometer 2 represent Joint 1 and Joint 2 respectively.

A real robot could use encoders or other position sensors instead. For this project, potentiometers are a cheap and easy way to generate the same kind of joint-position data.

The arm we're modelling is a simple 2-link system that looks something like this:

The first potentiometer gives us theta1, and the second gives us theta2.

2. Wire the Potentiometers

Each potentiometer has three connections: 3.3V, GND, and Wiper. We can connect these as shown in the schematic, using GPIO 34 and 35 for the Wipers.

Make sure the potentiometers are connected to 3.3 V, not 5 V.

3. Program the ESP32

The ESP32 has a pretty simple job. It reads the two potentiometers, converts the readings into angles, and sends those angles to Unity.

We'll send them as:

theta1,theta2

For example:

45.2,30.7

Here's the Arduino code:

const int joint1Pin = 34;
const int joint2Pin = 35;

const float minAngle = -90.0;
const float maxAngle = 90.0;

void setup()
{
    Serial.begin(115200);
}

void loop()
{
    int joint1Raw = analogRead(joint1Pin);
    int joint2Raw = analogRead(joint2Pin);

    float joint1Angle = map(
        joint1Raw,
        0,
        4095,
        minAngle,
        maxAngle
    );

    float joint2Angle = map(
        joint2Raw,
        0,
        4095,
        minAngle,
        maxAngle
    );

    Serial.print(joint1Angle);
    Serial.print(",");
    Serial.println(joint2Angle);

    delay(20);
}

The -90 to 90 degree range is just an example. The actual range you'll get depends on how your potentiometers are mounted, so you may need to adjust these values:

const float minAngle = -90.0;
const float maxAngle = 90.0;

4. Check the ESP32 Output

Upload the code and open the Arduino Serial Monitor.

Set the baud rate to 115200.

You should see something like:

-12.4,34.8
-11.9,35.2
-11.5,35.7
-10.8,36.1

Turn the first potentiometer and check that the first number changes.

Then turn the second one and check that the second number changes.

If that's working, the physical side is ready.

5. Create the Virtual Robot in Unity

Now we'll build a simple version of the same arm in Unity.

You don't need a complicated robot model. Two rectangular GameObjects are enough for this project.

The important part is getting the hierarchy and pivots right.

A simple hierarchy could be:

Robot
|
+-- Joint1
|    |
|    +-- Link1
|         |
|         +-- Joint2
|              |
|              +-- Link2
|
+-- EndEffector

The exact look is up to you. The important thing is that Link 1 rotates around Joint 1 and Link 2 rotates around Joint 2.

For example, we'll use:

L1 = 1.0 Unity unit
L2 = 1.0 Unity unit

The Unity dimensions don't have to match the physical arm exactly. We just need the same basic two-link geometry.

6. Connect Unity to the ESP32

When the ESP32 is connected over USB, Windows will give it a COM port.

Unity can read this port using C#.

Create a script called SerialReader.cs and assign it to a GameObject:

using System.IO.Ports;
using UnityEngine;

public class SerialReader : MonoBehaviour
{
    public string portName = "COM3";
    public int baudRate = 115200;

    private SerialPort serialPort;

    public float joint1Angle;
    public float joint2Angle;

    void Start()
    {
        serialPort = new SerialPort(portName, baudRate);
        serialPort.ReadTimeout = 50;
        serialPort.Open();
    }

    void Update()
    {
        if (!serialPort.IsOpen)
        {
            return;
        }

        try
        {
            string line = serialPort.ReadLine();
            string[] values = line.Split(',');

            if (values.Length == 2)
            {
                joint1Angle = float.Parse(values[0]);
                joint2Angle = float.Parse(values[1]);
            }
        }
        catch
        {
        }
    }

    void OnApplicationQuit()
    {
        if (serialPort != null && serialPort.IsOpen)
        {
            serialPort.Close();
        }
    }
}

Change:

COM3

to whatever COM port your ESP32 is using.

You can find this in Windows Device Manager or the Arduino IDE.

Make sure to close the Arduino Serial Monitor before running your Unity scene. Whenthe Serial Monitor is open, it claims ownership over the COM port, meaningUnity won't be able to access it.

7. Add Forward Kinematics

Now for the robotics part.

The ESP32 is giving us two joint angles, but Unity still needs to figure out where the end of the arm should be. That's where forward kinematics comes in.

For our two-link arm, the end-effector position is:

x = L1*cos(theta1) + L2*cos(theta1 + theta2)

y = L1*sin(theta1) + L2*sin(theta1 + theta2)

In simple terms:

We know the two joint angles, so we can calculate where the end of the arm should be.

Unity expects angles in radians when using its trigonometric functions, so we'll convert the incoming degrees first.

Create RobotKinematics.cs and assign to any GameObject:

using UnityEngine;

public class RobotKinematics : MonoBehaviour
{
    public SerialReader serialReader;

    public float link1Length = 1.0f;
    public float link2Length = 1.0f;

    public Transform link1;
    public Transform link2;
    public Transform endEffector;

    void Update()
    {
        float theta1 = serialReader.joint1Angle * Mathf.Deg2Rad;
        float theta2 = serialReader.joint2Angle * Mathf.Deg2Rad;

        float x = link1Length * Mathf.Cos(theta1)
                + link2Length * Mathf.Cos(theta1 + theta2);

        float y = link1Length * Mathf.Sin(theta1)
                + link2Length * Mathf.Sin(theta1 + theta2);

        endEffector.position = new Vector3(x, y, 0f);
    }
}

8. Make the Virtual Arm Follow

We also need to rotate the links themselves.

The first link follows theta1. The second follows theta1 + theta2.

Move the first potentiometer and the first joint should move. Move the second potentiometer and the second joint should move relative to the first.

Once this works, we've successfully connected a physical input device to a virtual robot.

9. Turn the Robot Into a Game Controller

This is where things get more interesting. Instead of just watching the virtual arm move, we can give it something to do. For example, Reach the Target.

Create a target inside the robot's workspace. The player needs to physically move the arm until the virtual end effector reaches it.

When the end effector reaches the target: Target reached -> +1 score -> Next target

10. Send Feedback Back to the Robot

So far, everything has been going from the robot to Unity: But we can also go the other way - from Unity to robot.

For example, when the player reaches a target, Unity could send TARGET_REACHED to the ESP32.

The ESP32 could then turn on an LED or activate another output, giving us a proper feedback loop:

This is useful for applications where the virtual environment needs to provide feedback to the person using the robot.

11. What If We Used a Real Robot?

The nice thing about this setup is that the potentiometers aren't really the important part. They're just providing the joint angles.

On a real robot, we could replace them with rotary encoders, magnetic encoders, motor position feedback, or other joint-position sensors.

12. Optional: Add ROS 2

While optional for this tutorial, our architecture allows for ROS to be integrated without compromising any other features. ROS becomes useful when you want Unity to communicate with other robotics software.

ROS-TCP-Endpoint is a freely available tool from Unity Technologies that provides a connection point between Unity and ROS 2. It can be cloned into a ROS workspace and run in a separate terminal whenever a ROS 2 node needs to exchange data with Unity. Once the connection is established, information such as the robot's joint angles can be sent between Unity and ROS 2.

A ROS 2 node can then subscribe to this data and use it for a range of applications. This could be as simple as displaying the joint values, or as complex as using them in a robot controller, AI system, or data-processing pipeline.

Unity can't open the serial port

Check that:

• The ESP32 is connected.
• You've selected the correct COM port.
• Unity is using 115200 baud.
• The Arduino Serial Monitor is closed.
• Another application isn't already using the port.

The values aren't changing

Check the Arduino Serial Monitor first. If the values aren't changing there, the problem is on the ESP32 or wiring side. If they're changing correctly, check the Unity COM port and serial reader.

The virtual arm moves incorrectly

Check:

• The potentiometer angle limits.
• The direction of each joint.
• The link lengths.
• The Unity pivot positions.
• That the second joint is using theta1 + theta2.

The physical and virtual arms don't line up

The physical and Unity coordinate systems may have different zero positions or directions.

You may need to add an offset:

theta1 = measuredAngle + offset1
theta2 = measuredAngle + offset2

This is normal when transferring measurements from a physical mechanism into a virtual environment.

Conclusion

We've connected a physical 2-DOF robot interface to Unity using an ESP32 and USB serial communication.

The physical joint angles are used to recreate a virtual robot with forward kinematics, allowing the physical robot to become a game controller.

We then added bidirectional communication so that the game can send feedback back to the physical system.

Finally, we showed how ROS 2 can be added when the project needs to communicate with a larger robotics system.

The two potentiometers can be replaced with real robot joint sensors, meaning the same basic interface can be taken from a simple prototype to a real robotic platform.

The basic idea is simple: connect your robot to Unity and make it part of the game.