Project by Team 13, Accelerated Motion Dynamics:
- Yuanxi Chen: cubit010
- Guy Sandler: GuySandler
- Aaron Chen: achen18
AI-powered SO-101 Robot Arm using CV to press buttons for a Whack-A-Mole type game. Trained using the online software Hugging Face and Weights & Biases. Constructed for Hack Club's Prototype hackathon Dec. 19-21.
Additional information continues below this section (Submission Details), please enjoy viewing our project!
Our mission is to explore how imitation learning and computer vision can enable robotic manipulators to perform fast, reactive tasks in various environments. While our group used a simple model of a Whack-A-Mole game, the core problem we address--detecting visual signals, mapping them to spacial coordinates, executing time-sensitive actions with accuracy--applies to real-world applications galore. Through our limited and simplified enviroment, we have made a proof-of-concept demonstrating the viability of substituting learning-based control for hand-tuned code.
Possible Applications
- Pick & Place: Visual cue --> spatial target and action
- Assistive Robotics: Reacting to illuminated objects in everyday scenarios e.g. traffic lights
- Intuitive Human-Robot Interaction: Robot responds to human-readable, visual stimuli as opposed to code or complex interfaces
Scripting button presses, Control Logic?
Our group treats the Whack-A-Mole game as a learning problem, training the SO-101 robot arm entirely through demonstration. The robot learns to intepret dynamic visual stimuli and adjust motor trajectories based on camera feeds and motor telemetry.
Our group has creativly reframed the classic arcade Whack-A-Mole as a robotics benchmark test. A low-cost, visually intuitive setup that still demands sufficient speed, accuracy, and consistency. This innovation makes the system engaging and effective, additionally allowing humans and robots to engage in friendly competition on the same level.
Thus, our group designed the game system from scratch, wiring and soldering each component, 3D printing a custom case to hold our electronics and buttons, and writing custom Python software to power our game logic.
- Teleoperation: Robot was teleoperated to press illuminated buttons while recording camera data and motor trajectories
- Training: Neural network policy trained via imitation learning through Hugging Face and Weights & Biases, learning to map visual cues to motor actions without explicit programming.
- Deployment: Trained policy was deployed on the physical robot, where it autonomously and consistently pressed random lit buttons across repeated trials.
For further details, please see Link to TRAINING.md
Because the system is trained on demonstration rather than mapping spatial coordinates to specfic buttons, the model can be extrapolated to new/larger button layouts. No need to change the control logic, the similar lighting and interaction conditions should be within the scope of the model's performance.
Interactions with the robot are relativly simple. A terminal command is used to start the autonomous model, whereupon the robotic arm plays HackAMole for a set amount of time. In addition, another terminal command can be used for human teleoperation using the leader arm, encorportaing different camera angles for operator convenience.
- 4x MX-style keyswitch
- 1x Individually-addresable RGB Pebble Lights
- 1x ESP32 microcontroller
- 1x Breadboard
- 11x Jumper Wire
videoDemo.mov
Our initial concept was inspired by a Whack-A-Mole style game, designed to emphisize developing speed, accuracy, and repeatability in our AI-powered arm. This task provided clear and measureable results for smoother experimentation.
To begin, we soldered MX-style mechanical key switches and connected them to a ESP32 microcontroller, allowing us to detect presses and communicate input data to our system. A strip of individually-addressable LEDs was also wired to the ESP32.
For rapid iteration, the first version was mounted on a cardboard stand. Once concept testing was completed, a more robust design was modelled using the online CAD software Onshape and 3D printed using BambuLab A1 Minis.
Implements a simple game where the player must press a button indicated by a illuminated section of a LED strip.
Components
- Buttons: 4 MX-style switch inputs with internal pull-up resistors
- Addressable LED strip: Single strip divided into 4 sections, each mapped to a button
- Random Selection Logic: Choose a new LED section after each correct press, no repeat sections
- Input Handler: Continously poll button states, track active LED section and evaluate presses
- Gameplay Flow: Utilize timers to gradually speed up game tempo
Pressing the Correct Buttons
softwareCorrect.1.mp4
Pressing an Incorrect Button
softwareIncorrect.1.mp4
- Hugging Face Dataset: https://huggingface.co/datasets/cubit010/pusht_merged
- Hugging Face Model: https://huggingface.co/cubit010/act_wackamole
- Google Drive of all Images/Videos: https://drive.google.com/drive/folders/1TB7pIp2-6yzYLLt-cenjhjwB0DylGWsw
- Weights and Biases: https://wandb.ai/chenyuanxi5-hack-club/lerobot/runs/yloubmel