Jumping Jerboa - Transfer Mouse/Keyboard To Other Computer

What exactly is it ?

A method to transfer mouse and keyboard actions from your own computer to another computer over USB connection using a raspberry pi pico and USB to TTL UART converter.

Why Name it Jumping Jerboa ?

Jerboas are hopping desert rodents and this project allows you to let your mouse and keyboard hop between different computers just like Jerboa. Hence the name Jumping Jerboa.

Where to use it ?

Such a functionality can be helpful in following use case scenarios :

• Working with 2 computers simultaneously, It becomes clumsy to switch between different keyboard and mouse. Having such a device like Jumping Jerboa which allows you to transfer mouse/keyboard between these on the fly is really comfortable.
• Other computer(may be laptop) has its mouse/keyboard broken, and you don't want to end up buying a new set of mouse and keyboard until it is repaired.
• Other computer (say android phone/tablet) has its screen broken, and you may want to access something urgently before arranging a repair. You can use this setup with android's USB OTG functionality (yes literally every android around supports this and you can connect mouse and keyboard to it).
• When setting up SBCs (Single Board Computers) which have only 1x USB port and that too micro USB like Raspberry Pi Zero 2 W, it would require a USB hub (may be a powered one) to interface both mouse and keyboard. Jumping Jerboa can easily help with such case with only a USB 2.0 to micro USB OTG cable.

Off course all of the use cases mentioned above require you to have Raspberry Pi Pico (or any other CircuitPython supported development board) and a USB to TTL converter. But many of us already have both.

Required Parts

• A CircuitPython supported development board with USB HID support. I have used Rpi Pico with rp2040, but any other board will work. You can find the supported boards on CircuitPython website (link at the end).
• Any of the TTL UART to USB converter. I have used CP2102, but other converters like PL2303, FT232, CH340 will work just fine because they all support 115200 baudrate. Make sure that the logic level of this converter matches the board's logic level (i.e., 3.3V logic level)
• A PC running Windows/Linux with python installed. Also, same should have numpy, pyserial and pygame installed.

Following is a summary of setup I have used, but higher versions should work just as fine :

1. Raspberry Pi Pico :
   - CircuitPython v9.2.7
   - adafruit_hid v6.1.5
   - ulab v6.5.3-2D
2. TTL UART to USB Converter
   - CP2102 based
   - 3.3V logic level
3. Local Computer
   - Windows 10 on Intel x64
   - Python v3.7.4
   - pygame v2.6.1 (SDL v2.28.4)
   - pyserial v3.4
   - numpy v1.20.2
4. Remote Computer
   - Android 4.4 Tablet, but can be anything like a Windows/Linux PC/Laptop.

How it works ?

Below is a brief description of system architecture. Don't worry, detailed explanation of each step is explained afterwards.

• A pygame window runs on local computer and captures all mouse and keyboard actions activity.
• This actions are mapped from pygame values to circuitpython values.
• A binary data packet is made for each of the action and sent to TTL UART to USB converter using pyserial.
• Raspberry Pi Pico receives these binary data packets, decodes them and performs same HID mouse and keyboard activity on remote computer.

Further explanation below dives into source code specifics, hence you can refer github repo simultaneously.

Step 1. Capture mouse and keyboard inputs on local computer using pygame

There are few considerations to be taken care of when capturing mouse and keyboard inputs :

a) When capture is active, the mouse and keyboard actions should be only effective on remote computer and should not interact with local computer. Meaning that mouse and keyboard should not allow any action on local computer when capture is active. For this, pygame allows to grab inputs completely using below functions :

pygame.event.set_grab(True)                # enable mouse grab
pygame.event.set_keyboard_grab(True)       # enable keyboard grab

b) mouse should allow infinite motion and report relative motion correctly. What it means is that mouse pointer should not stop moving when it reaches one of the edges on local computer, because at the same time mouse pointer might not have reached the same edge on remote computer. pygame allows this using below function, by making mouse pointer invisible and allowing infinite mouse motion in virtual input mode.

pygame.mouse.set_visible(False)        # make mouse pointer invisible

c) There should be easy way to switch between local and remote computers. For this Ctrl+Alt+G (G for Grab) is used as shortcut to enable/disable grab. Grab is disabled initially when launching the Jumping Jerboa and mouse/keyboard work normally on local computer. By selecting pygame window and pressing Ctrl+Alt+G will enable the grab and mouse/keyboard will work on remote computer only. Pressing Ctrl+Alt+G again will disable the grab and mouse/keyboard will be back to local computer usage. Please refer source code for understanding this correctly.

The Jumping Jerboa python script samples the pygame events at 100 Hz and looks for MOUSEMOTION, MOUSEBUTTONDOWN, MOUSEBUTTONUP, MOUSEWHEEL, KEYDOWN, KEYUP events. When these events are observed, they are sent further for mapping (if required) from pygame events to circuitpython hid commands.

Also, there are some limitations to grab. pygame allows capturing almost all key combinations on Windows, except Ctrl+Alt+Del. But on Linux, many more key combinations are not allowed to be captured. However, this doesn't render this project useless.

Step 2. Keyboard key and mouse button mapper

pygame and circuitpython refer to same keyboard key or mouse button using different integer values. For example, pygame refers to keyboard key 'A' using value 0x97, while circuitpython refers to the same with value 0x04. Hence, there must be a lookup table and lookup function which helps with key and button mapping.

Below mapping table and function has been implemented for keyboard keys :

# Look-Up Table (LUT),
# Maps Pygame keyboard keycodes to circuitpython adafruit hid keycodes
Key_Mapper_LUT = {
    pygame.K_a: CircPyKeycode.A,
    pygame.K_b: CircPyKeycode.B,
    pygame.K_c: CircPyKeycode.C,
    ...
    ...
}

# helper function to map keyboard keycodes from pygame to circuitpython
def map_keycode_pyg_to_cpy(PygKeyCode):
    # return 0, if unknown key mapping requested.
    return np.uint8(Key_Mapper_LUT.get(PygKeyCode, 0))

Similarly, mapping table for mouse buttons :

# Look-Up Table (LUT),
# Maps Pygame Mouse keycodes to circuitpython adafruit hid keycodes
Button_Mapper_LUT = {
    1 : CircPyMouse.LEFT_BUTTON,
    2 : CircPyMouse.MIDDLE_BUTTON,
    3 : CircPyMouse.RIGHT_BUTTON,
    4 : CircPyMouse.BACK_BUTTON,
    5 : CircPyMouse.FORWARD_BUTTON
}

# helper function to map mouse button codes from pygame to circuitpython
def map_buttoncode_pyg_to_cpy(PygButtonCode):
    # return 0, if unknown button mapping requested.
    return np.uint8(Button_Mapper_LUT.get(PygButtonCode, 0))

Note that such mapping is not required for mouse motion or wheel, because they are merely signed integer values based on magnitude and direction of motion/scroll.

Step 3. Simplified binary communication protocol

Since we need to send very basic information about mouse and keyboard actions, following simplified binary communication protocol is used to send different actions from local computer to Rpi pico.

• Binary data packet has fixed length of 6 bytes. So no need to send an additional byte for size of packet.
• Checksum is simply sum of Header and Payload bytes. It may overflow, but that's fine because same unsigned integer arithmetic will work the same way on receiver side too. Anyways, this is not a high integrity system.

|     0    |     1    |     2    |     3    |     4    |     5    | 
+----------+----------+----------+----------+----------+----------+
|   HDR1   |   HDR2   |   AXID   |   AXV1   |   AXV2   |   CSUM   |
+----------+----------+----------+----------+----------+----------+
|       Header        |           Payload              | Checksum |
|     (2 Bytes)       |          (3 Bytes)             | (1 Byte) |
+---------------------+--------------------------------+----------+

Where,
- HDR1 : Header byte 1
- HDR2 : Header byte 2
- AXID : Action ID
- AXV1 : Action Value 1
- AXV2 : Action Value 2
- CSUM : Checksum (sum simply!) of Header+Payload

Below is the detailed description for contents of binary data packet for all the different actions.

+----------+------+------+------+----------+--------------+---------+
| Action   | HDR1 | HDR2 | AXID | AXV1     | AXV2         | CSUM    |
+----------+------+------+------+----------+--------------+---------+
| Mouse    |      |      | 0xA0 | X motion | Y motion     |         |
| Move     |      |      |      | units    | units        |         |
|          |      |      |      | (int8)   | (int8)       |         |
+----------+      |      +------+----------+--------------+   HDR1  |
| Mouse    |      |      | 0xA1 | Button   | 0x1, press   | + HDR2  |
| Button   |      |      |      | index    | 0x2, release | + AXID  |
|          |      |      |      | (uint8)  |              | + AXV1  |
+----------+ 0x13 | 0x13 +------+----------+--------------+ + AXV2  |
| Mouse    |      |      | 0xA2 | Scroll   | 0x0          | (uint8) |
| Scroll   |      |      |      | lines    | (unused)     |         |
|          |      |      |      | (int8)   |              |         |
+----------+      |      +------+----------+--------------+         |
| Keyboard |      |      | 0xA3 | Key      | 0x1, press   |         |
| Key      |      |      |      | index    | 0x2, release |         |
|          |      |      |      | (uint8)  |              |         |
+----------+------+------+------+----------+--------------+---------+

Despite being shorter in length, this packet has a very high overhead of 50% (3 bytes of header+checksum out of 6 bytes total length). But that's fine as long as it serves its purpose.

below is function that encodes the mouse X and Y motion as a binary packet.

# send mouse move action to rpi pico
def send_mouse_move_action(Xmotion, Ymotion):
    # create an empty byte array for data packet
    DataPacket = bytearray(b'')
    DataPacket.append(DATA_PACKET_HEADER)       # add header
    DataPacket.append(DATA_PACKET_HEADER)       # add header
    DataPacket.append(ACTIONID_MOUSE_MOVE)      # add action id
    DataPacket.append(np.uint8(Xmotion))        # add action value 1
    DataPacket.append(np.uint8(Ymotion))        # add action value 2
    
    Sum = np.uint8(0x00)        # zero the sum
    for Data in DataPacket :
        # calculate the check(SUM!) by summing all the bytes
        # it will overflow the uint8_t, but that's fine.
        Sum = np.uint8(np.uint8(Sum) + np.uint8(Data))
    
    DataPacket.append(Sum)      # add checksum at the end of bytearray

    print(DataPacket)       # print data packet for debugging purpose
    try :
        # send the data packet to rpi pico
        serial_port.write(DataPacket)
    except IOError :
        # if issues sending, then exit.
        print("Serial Port seems to have disconnected")
        print("exiting")
        exit()

Is baudrate of 115200 enough ?

• At baudrate of 115200 (bits/second) and 8N1 (1 startbit + 8 databits + No parity (0 bit) + 1 stop bit = 10 bits for a byte to be transmitted)
• We can transmit upto 11520 bytes persecond ( (115200 bit/second) / (10 bit/byte) = 11520 byte/second ).
• This would be around 1920 packets persecond ((11520 byte/second)/(6 byte/packet) = 1920 packet/second)
• At 100Hz event sampling (in pygame), this results in bandwidth of 19.2 packets persample. ((1920 packet/second) / (100 sample/second) = 19.2 packet/sample)
• 19.2 packet/sample or say 19.2 mouse&keyboard event/sample is pretty high rate of inputs and our human hands cannot easily provide such motion to mouse and keyboard.

So, baudrate of 115200 is more than enough and would allow longer cable length for our TTL level UART signals.

Step 4. Binary packet Decoding (on pico)

Encoding binary packets is easier by just adding the bytes to array, but it is the decoding which is a bit difficult compared to encoding. Following will be the decoding logic of the state machine running on Rpi pico for the binary protocol described in Step 3. This state machine has been implemented using if..elseif..else control statement with couple of state variables.

Below is a shallow description of this state machine and it's states :

State HEADER1 : If received byte is 0x13, 
                    then buffer the byte, goto HEADER2 
                    and wait for next byte.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

State HEADER2 : If received byte is 0x13, 
                    then buffer the byte, goto PAYLOAD 
                    and wait for next byte.
                else, 
                    reset the buffer, go to HEADER1  
                    and wait for next byte.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

State PAYLOAD : Buffer 3 incoming bytes, 
                    then go to SUM and wait for next byte.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

State SUM     : Buffer received byte (it is received checksum)
                Calculate the Checksum of all the bytes received 
                before (it is calculated checksum).
                If received and calculated checksum match,
                    then process the action based on 
                    binary packet payload stored in buffer.
                Finally, reset the buffer, go to HEADER1 
                and wait for next byte.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

State UNKNOWN : Reset the buffer, go to HEADER1 
                and wait for next byte.

Step 5. Send USB HID commands to remote computer

Once a valid data packet is received and buffered as per described in Step 4, following code reads byte #2 (AXID) and calls required function based on what action is represented by the AXID.

def process_action_input():
    actionid = DataRxBuf[2]
    
    if (ACTIONID_MOUSE_MOVE == actionid):
        move_mouse_action()
    elif (ACTIONID_MOUSE_BUTTON_PRESS_RELEASE == actionid):
        mouse_button_action()
    elif (ACTIONID_MOUSE_SCROLL == actionid):
        mouse_scroll_action()
    elif (ACTIONID_KEYBOARD_KEY_PRESS_RELEASE == actionid):
        keyboard_key_action()
    else :
        print("unknown action")

And following is the code which sends the usb hid command for mouse motion by reading the required values of bytes 3 and 4 (ACV1 and ACV2).

def move_mouse_action():
    motionX = 0
    motionY = 0

    if (DataRxBuf[3] > 127) :           # check if it is negative motion
        motionX = DataRxBuf[3] - 256    # treat uint8 as int8
    else:                               # else positive motion
        motionX = DataRxBuf[3]

    if (DataRxBuf[4] > 127) :           # check if it is negative motion
        motionY = DataRxBuf[4] - 256    # treat uint8 as int8
    else:                               # else positive motion
        motionY = DataRxBuf[4]

    print("Ms Motion : " + str(motionX) + ", " + str(motionY))
    m.move(motionX, motionY)

Rpi Pico UART receive considerations

circuitpython busio APIs take away all the hardwork required to implement the low level code for receiving UART data asynchronously. But it requires your application code to configure UART correctly, so that application gets received data on time whenever required with minimum blocking, despite the fact that data can arrive any time.

Basically, we need to define 2 things correctly for better scheduling of circuitpython application on pico :

• How many bytes to waitfor and receiveat once - pygame sends data in chunks (packets) of 6 bytes length, hence 6 bytes at once can be read as per below code snippet.

data = uart.read(6)

• How much time to wait for if no data is received on UART, aka timeout - pygame can send data at 100Hz max, hence we can set timeout as 10 millisecond (0.01 second), refer below code snippet.

uart = busio.UART(board.GP0, board.GP1, baudrate=115200, timeout=0.010)

The reason behind setting these parameters is that the busio.UART.read() function will block the whole application until either required number of bytes have been received or timeout occurs without receiving any additional bytes.

Although our application does only a single thing only of decoding and acting on the actions, in other applications you may require multiple things to be done by same code.py. In such cases, the choice of timeout and number of bytes to read can significantly help reduce blocking.

You can get away with blocking by setting timeout as 0. but that will result in more calls to read() and hence increase overhead without doing any actual work. You can use circuitpython's asyncio for better scheduling.

Exception Handling (on local computer)

python supports exceptions, which is good thing but that means you must handle the exception correctly, otherwise your code will stop execution as soon as it encounters an exception. Using exception handler, we can address some of the exceptions without crashing the code.

For example, upon initializing the local computer script we may encounter exception if ascii art file is not found. This can be handled easily by simply printing that ascii art file was not found, but execution can be still continued because none of the functionalities depend on ascii art file.

ascii_art_file_path = "ascii_art.txt"

try:
    with open(ascii_art_file_path, 'r') as art :
        art_lines = art.readlines()

        for art_line in art_lines:
            print(art_line, end='')
        print("\r\n")

except Exception as e:
    print("Jumping Jerboa - ascii art not found")

Other exception would be opening serial port which doesn't exist, may be because the USB to TTL UART module is not connected or connected to a different COM port. In such case we can simply print the issue and exit the application because nothing can be done further without a valid serial adapter.

try:
    serial_port = serial.Serial(port=sys.argv[1], baudrate=115200, timeout=0.1)
except IOError :
    print("Cannot open serial port, following could be the reasons : ")
    print("- Serial converter is not connected, hence port doesn't exist")
    print("- Serial port name is incorrect")
    print("- Serial port is used by another application")
    print("exiting")
    exit()

Similarly, if USB to TTL UART adapter is removed in between the usage, then an exception will be raised upon writing data packet to serial port. This can be handled by printing the issue and exiting, because nothing can be done further.

try :
    serial_port.write(DataPacket)
except IOError :
    print("Serial Port seems to have disconnected")
    print("exiting")
    exit()

These are the expected exception, but there can be many other exceptions which may occur, and we can write a common handler enclosing the calling of whole main(), with printing the exception and exiting.

try:
    main()
except KeyboardInterrupt:
    print("Interrupted by Ctrl+C")
    print("exiting")
    exit()
except Exception as e:
    print("Uncaught Exception : ", end='')
    print(e)
    print("exiting")
    exit()

Note that these are overly simplified exception handlers, as our application is also very simple. But in more complex applications you might have multiple resources opened, in such cases all the resources must be released correctly to avoid any corruption.

Exception Handling (on raspberry pi pico)

We should write embedded applications in such a way that avoids any exceptions. This can be implemented by having measures like range checks, index check, initialization of variables before using, etc. However, there can still be case when an exception can happen.

In such case, we can write exception handler enclosing the entire while True and restart our script execution from beginning. So, user wouldn't even notice the exception and pico will be back in action within no time. This approach works well for us because our application is not safety critical, and hence restarting won't cause any issues. But this approach can be different for safety critical, high integrity or hard real-time system.

while True:
    try:
        data = uart.read(6)

        if data is not None :
            for byte in data:
                detect_action_input(byte)
        else :
            pass
    
    except Exception as e:
        print("Uncaught Exception : ", end='')
        print(e)
        print("Soft Reset")
        supervisor.reload()

Installation and setup

Please refer readme.md file from github repo.

Photo

Demo video

Limitations

Note that not all the keyboard and mouse actions can be captured. Ctrl+Alt+Del will not be captured when local computer is windows. Similar limitations apply for Linux local computer also.

A note on keylogging !

At the very core, jumping jerboa is basically a keylogger. But unlike keylogger malwares, jumping jerboa python scripts are different in following ways :

• Script is run by user, and doesn't launch automatically. User is in control whether to run script or not.
• Script will always show a GUI pygame window whenever it is executed. The keylogging operation is only active when pygame window is in focus.
• It is open source and distribution is also source code (the python scripts) and not binary distribution. You are welcome to and recommended to refer to source code.
• These scripts do not connect to internet, and whatever keys are logged (captured) are never stored anywhere. Rather, the key logs are just streamed instantaneously without any feedback/request from receiver hardware (rpi pico here).
• The script does not require installation, does not launch automatically, does not replicate itself or does not try to infect any other system. Deleting the script is equal as uninstalling.
• User still needs to install dependencies (python, pygame, pyserial, etc.) manually in order to make use of this script. The script will never install any dependencies.
• This is just a proof of concept which can help working with two computer systems simultaneously a bit easier.

Further possibilities

• Add receive functionality for local computer script and rpi pico should acknowledge/not acknowledge the serial packets based on their validity. Not getting acknowledge/not acknowledge can be considered rpi pico having issues communicating. This way we can also determine whether rpi pico is connected properly or not, as the jumper cables can become loose sometimes.
• Port this code to nRF52840 based development board to make use of BLE HID functionalities for wireless mouse and keyboard transfer. This shouldn't be much of a hard work considering circuitpython has excellent support for BLE HID too just like USB HID used in this project.

References

Following links will be helpful in understanding more.

• How to create lookup tables efficiently in python - labex.io
• pygame event grab - pygame docs
• pygame keyboard grab - pygame docs
• circuitpython downloads page - circuitpython website
• circuitpython libraries page - circuitpython website
• UART communication between two circuitpython boards - adafruit learn
• Usage of circuitpython busio read with timeout - circuitpython docs
• Ways of resetting circuitpython board - circuitpython essentials
• Basics of circuitpython HID mouse and keyboard - circuitpython essentials
• circuitpython HID library documentation - circuitpython docs
• Typing and keyboard control using the HID library (CircuitPython School) - Youtube video (Build with Prof. G.)
• Build a Mouse Glove! Controlling the Mouse with HID Libraries (CircuitPython School) - Youtube video (Build with Prof. G.) > This one is really cool !
• circuitpython UART asyncio (although not used in this project, but can be helpful in more complex projects) - github gist
• cooperative multitasking in circuitpython with asyncio (although not used in this project, but can be helpful in more complex projects) - adafruit learn