I recently gave my OpenClaw a real robot arm to play with. The results just about blew my own neural network.
The AI agent was able to configure the arm, use it to see and slowly grab things, and even train another AI model to pick up and place specific objects. And they say AGI is still a few years away! (I’m joking, it probably is).
The results have me convinced that we may be on the brink of a robotics breakthrough. Training and controlling robots used to require considerable skill. Today’s AI models can make it almost easy.
“AI-powered coding is super exciting because it has the potential to bridge the gap between conventional engineering methods, which are reliable but don’t generalize, and contemporary vision-language-action models, which generalize but are not yet reliable,” says Ken Goldberg, a roboticist at UC Berkeley who is exploring the approach.
I bought a prebuilt arm called a LeRobot 101. It’s part of an open-source project from HuggingFace that makes it relatively cheap to start building and experimenting with robotics.
The LeRobot comes with two arms: a controller arm that a person operates using a handle and a trigger, and a follower arm with a camera that replicates those movements. You can train an AI model by teleoperating the controller arm and having the model learn how to move the follower in response to what it sees on the camera.
Building With OpenClaw
Before using OpenClaw, I spent several hours trying to connect and calibrate the robot, at one point nearly breaking the motors by applying the wrong settings, which caused them to overheat.
Then, with help from OpenClaw and Codex, I was able to vibe code a simple program that closed the claw’s gripper when it spotted a red ball. In the terminal, Codex went through the tricky work of configuring the connections to the robot. Then, with my help, it calibrated the positions of its joints. It also wrote a Python script that used several libraries to identify and grip the ball in question. Vibe-coding isn’t perfect of course, and hallucinations can introduce bugs especially when working with different hardware, but the results were impressive.
A neat result, yes, but not exactly Terminator. Next I tried having OpenClaw help me train a model to control the arm. We experimented with a few different approaches, and OpenClaw was adept at guiding me through the process and checking the error rate of the model after each training run.
Code as Policy
The idea that AI-powered coding could offer a powerful new way to build robots was first highlighted in a research paper from 2022 that dubbed the approach “code as policy.” Since then, AI’s coding skills have advanced at a dizzying pace, and the code-as-policy method has gained traction in many labs.
Goldberg’s research group, together with researchers from Nvidia, Carnegie Mellon University, and Stanford, recently developed a new benchmark called CaP-X to measure the robot capabilities of coding models. Interestingly, CaP-X shows that the best model for programming robots isn’t Claude or ChatGPT but Gemini—perhaps because Google DeepMind has focused on training its models to be multimodal and make sense of the physical world. Along with the benchmark, the researchers created CaP-Gym, an environment that lets coding agents control both simulated and real robots. They also developed CaP-Agent0, an agentic framework that boosts the performance of coding models so much that they beat models trained to control a robot’s movements directly on some manipulation tasks.
Goldberg’s team is working with Nvidia to explore the potential of the code-as-policy approach. I spoke to Spencer Huang (none other than Jensen Huang’s son), who has been involved in organizing hackathons inside the company to let people try their hand at vibe coding robots. Huang is currently working on a research project with Goldberg that should make the code-as-policy approach compatible with more robot software tools.
“Nearly anyone can get into robotics, which is the true holy grail,” Huang tells me. Making it possible for people to control robots with spoken or typed commands, or by demonstrating an action, is the “critical unlock for robots in society,” he adds.
The post I Gave My OpenClaw Agent a Physical Body appeared first on Wired.
