SCABO - Low cost yet AR-enabled cardboard toy kits

Overview

I am a bit hesitate to submit the project since at the beginning the main goal of the project was to target to the toy industry for children. However, sharing the project under the contest would help me and my friends to get your feedback.

This project demonstrates how to construct 12DOF cardboard robot to help people after-stroke a new way to play with cardboard through 2 steps.

First, cardboard robot can be assembled from scratch manually with 3D instructions. Secondly, AR platform allows users to play physical robots with 3D scenarios in mixed reality mode in their smart devices (Android/iOS)

Problem

What were the needs or pain points that you attended to and identified when you were solving problems faced by the Contest Masters?

Rehabilitation for patient after-stroke normally is time-consuming and energy-intensive process. Recently, many haptic interfaces and software have been introduced to support after-stroke person. The main advantage of the above systems is that gamification scenarios allowing patients have fun time with their families members. Therefore, I think that our project started more than 2 years ago with my friends would be suitable with the 2 mentioned factors for after-stroke person.

In terms of people with mobility impairment, I got one brother who got irreversible paralysis from Poliomyelitis. Through playing with him since I was a little kid, I understood the miracle happened to him only from Acupuncture treatments. He got difficulties in controlling his hands and his legs but he still can stand with legs can control his hands for basic tasks. However, he would love to play with toys like LEGO but it's too expensive for children in developing countries. Therefore, I do hope that the low cost cardboard toys can be suitable for children in different parts of the world.

Developing a Solution: Quadruped Cardboard Robot

First, the cardboard paper had been developed from scratch. It took me and my friend a lot of time to understand the cardboard material and how to combine it with electronics, servos. There are more than 20 versions of cardboard developed from the different cardboard materials. As you can see in the Fig. 1 the leg of the robot a bit long. It caused huge problems to the servos in the femur. The long leg version has been gradually improved to the shorter version in the Fig. 2. The great thing of the cardboard robot is that it is designed in modular approach. We can easily replace some parts of the robot with new cardboard parts.

Right after the prototype version, I de-engineer to unfold the robot back to cardboard paper sheet. Therefore, users can assemble their own cardboard robots with their friends or families.

Materials

• NO more screw or double tape
• 4x SG92 servo
• 1x MCU node ESP8286 or ESP32
• 8x SG90 servo
• 8x cardboard paper sheet
• 1x PCA Driver 9685 Servo Driver
• 1x Li-ion battery 2000 mAh
• 3x Jumper Wire

Fig. 3 demonstrated the main parts of the cardboard robot. It does NOT require more glue or screw to assemble the robot. The cardboard has 12 DOF with 3 servos for each leg. All servos can be attached to the cardboard parts without using glue or screw.

There are 3 jumper wires used to connect the MCU node with battery and the PCA driver 9685. Users can assemble their own cardboard from scratch with the 3D instruction manual demonstrated below.

3D Instruction - DIY experience

The 3D step-by-step user manual is implemented on Android/iOS devices. Users can view the instructions in 3D to assemble their own robot cardboard toy. The experience when you can assemble cardboard toys with your friends or family member is worth although it may take 2.5 hours to complete the toys.

BLE is utilized to for communication between the robot and Android/iOS devices. The robot can be moved and rotated by users. In addition, Inverse Kinematics algorithm allows the robot to have different motions such as happy, wonder, shaking body, etc. Fig.5 demonstrates the robot walking on the table in trot gait.

The robot can be also controlled by voice with your smart devices. Users can talk to the robot and learn scientific subjects with the robot by talking to the robot.

3D Marker-less Object Detection

In our project we utilized Mediapipe 3D Object Detection for detecting our cardboard robot. The cardboard robot position and orientation can be estimated at 30 FPS to integrate with virtual scenarios. Fig. 6 demonstrates the 3D object detection even the robot is out of view, the system still remembers the 3D position and orientation of the cardboard robot. The pose vector of the robot can be simplified as a 6D vector in Cartesian space.

Integrate with ARCore/ARKit and Sceneform/SceneKit

Given the 6D vector pose of the robot above, the main goal of this module is to integrate the vector into AR virtual scenarios. Collision detection and collision response algorithms have been implemented to detect the collision between the robot and the virtual characters.

Several basic games have been developed to play with cardboard robot using native SDK in Android( ARCore and Sceneform) and iOS (ARKit and SceneKit).

Virtual characters can attack the real cardboard robot and the cardboard robot can shoot the virtual characters as you see in the demonstration below.

USER STUDY

I and my friends had visited couple schools to demonstrate our cardboard toys. We had great time with them to assemble and play with the robot. We strongly believe that we can scale up the practices with your help - your schools in rural areas (please let me know). The cost-effective cardboard toys are our gifts to the schools. Please connect with us and tell us your request. We can deliver our cardboard toys to you.

FUTURE DEVELOPMENT

Several future work to the project may include:

• User study with actual patients in hospitals and people with disability: actual after-stroke patients in rehabilitation process to assemble with cardboard toys. I am not sure about the user study with people with disability but I think the moment they can control the robot with their voice or talking to the robot can be learned more to improve the user experience.
• Improve game play with the cardboard robot
• Improve the object detection module: the goal is to improve the object detection to many other toys. Zero-shot object detection or 3D object tracking algorithm can be implemented for many other toys.
• Open source project: I am sorry that I can not put the firmware and the robot design by the time of the contest. I just knew about hackster.io recently and I will do it very soon. Please stay tuned.