Tesla shifted its Optimus training strategy — and it’s using a familiar playbook

Tesla is turning to a well-worn blueprint to train its humanoid robot.

The carmaker told employees in late June it would focus more on a vision-only approach for the Optimus program, insiders familiar with the team told Business Insider.

Tesla previously used motion capture suits and virtual reality headsets to record data from human operators and remotely operate the robot. Now, the automaker will primarily focus on recording videos of workers performing tasks to teach the robot how to do things like pick up an object or fold a t-shirt.

The company said moving away from motion capture suits and teleoperation would allow the team to scale data collection more quickly, according to the insiders.

The change marks a significant shift in Tesla’s robotics strategy, aligning Optimus with CEO Elon Musk’s long-standing belief that AI can master complex tasks using cameras. The automaker has long used a similar approach to train its autonomous driving software.

The change occurred shortly after Milan Kovac, the director of the Optimus program, stepped down. AI director Ashok Elluswamy has since taken charge of the program, the insiders said.

Teleoperation and motion capture are standard industry practices to train robots. Leading robotics company Boston Dynamics, for example, has used teleoperation to train its Atlas robot. During the training process, motion capture suits are worn by workers who perform various tasks; the data is then fed to the robot. Motion capture suits can also be worn to remotely control the robot.

It’s unclear whether Tesla will prioritize motion capture suits and teleoperation again in the future, or whether it will use the video data to build on the previously collected information.

Robert Griffin, a senior research scientist at the Institute for Human and Machine Cognition, told Business Insider that a large set of teleoperation data allows the robot to learn by physically interacting with its environment. It can be difficult to teach the robot to translate video data to the real world, he said.

“If you’re just using video data, you don’t have that directly,” he said.

In May, Tesla posted a video that appeared to show Optimus performing tasks based on video recordings.

Multiple fully Tesla-made Bots now walking around & learning about the real world 🤖 Join the Tesla AI team → https://t.co/dBhQqg1qya pic.twitter.com/3TZ2znxkfd

— Tesla Optimus (@Tesla_Optimus) May 16, 2023

Konstantinos Laskaris, a director working on Optimus hardware, appeared to acknowledge the new strategy in a May LinkedIn post. “It seems unbelievable, but our robot is learning new tasks directly from human videos!” Laskaris wrote.

Musk also said in May that the robot will eventually be able to learn how to perform tasks by watching YouTube videos.

The company briefly paused hiring for the Optimus team amid the transition, insiders with knowledge of the issue said. As of late August, Tesla’s careers page listed more than 50 Optimus-related roles.

A spokesperson for Tesla did not respond to a request for comment.

Folding t-shirts and picking up objects

Musk first announced Tesla’s plans to create a humanoid robot named Optimus in 2021. The billionaire has said the robot will eventually be able to handle tasks like factory work and caregiving.

Last year, the company was hiring for “data collection operators.” The roles have involved performing and recording basic household tasks. The job postings said the operators would be required to wear motion capture suits and virtual reality headsets for extended periods of time.

Up until late June, the project involved teleoperating Optimus and training the robot via the motion capture suits. Workers spent time troubleshooting issues with the suit and with Optimus, which limited the amount of data the team was able to collect, sources said.

Since the training shift, workers have focused on recording their own movements using a set of five cameras made in-house, insiders said. The cameras are attached to a helmet and a heavy backpack worn by the worker. The cameras point in all directions, giving the AI model data that orients it precisely in its environment, the insiders said.

Christian Hubicki, the director of the robotics lab for FAMU-FSU’s engineering school, said the various camera angles likely allow Tesla to collect more minute details, “like the location of joints and fingers,” and properly place the robot in space. The videos could also be a way of augmenting the previous data they collected using teleoperation, he said.

Workers receive specific instructions on how to perform the tasks for training, particularly when it comes to hand movement and ensuring the movements look as human as possible. They can spend months performing the same simple task, one person said.

Jonathan Aitken, a robotics expert with the University of Sheffield, told Business Insider that Tesla will likely need to figure out how to teach Optimus using actions that could apply to a variety of tasks.

“Working at this scale, they must have a generalized set of actions or else it would take forever to do all of them,” Aitken said.

He added that Tesla could employ a similar strategy to Physical Intelligence, a company that feeds robots vast amounts of demonstration data so they can learn transferable skills and apply them flexibly, rather than just memorizing how to perform a single task.

‘A very Tesla way of doing robotics’

The new strategy is in line with Tesla’s training process for its self-driving software.

While other autonomous vehicle companies use LiDAR and radar sensors to teach their software, Tesla primarily relies on a collection of cameras.

The company uses data collected from millions of Tesla owners, whose vehicles are equipped with between eight and nine cameras. Musk has said Tesla released its driver-assist software in China after training its AI system with publicly available videos of streets in Asia.

Musk acknowledged during an earnings call in January that “the training needs for Optimus humanoid robot, are probably at least ultimately 10x of what is needed for the car.”

“It’s a very Tesla way of doing robotics. No one else is trying to do this at the same scale,” Aitken said. “They’ll need as much data as they’ve used to train their cars,” he added.

Alan Fern, an AI and robotics expert at Oregon State University, said training Optimus will be an even more daunting project for Tesla than developing an autonomous vehicle.

“Driving is just one task,” he said. Learning primarily from videos “would require the robot to understand what’s going on in the video, and then have the skills to accomplish it,” he said. “There are some things it can learn by watching and others it will need to physically practice, whether in a simulator or the real world.”

Do you work for Tesla or have a tip? Contact this reporter via email at [email protected] or Signal at 248-894-6012. Use a personal email address, a nonwork device, and nonwork WiFi; here’s our guide to sharing information securely.

The post Tesla shifted its Optimus training strategy — and it’s using a familiar playbook appeared first on Business Insider.