Insider Brief
- Researchers led by the University of Hong Kong developed RoboDojo, a benchmark that evaluates physical AI across simulation and real-world robot tasks and found a wide performance gap between current models and humans.
- The platform combines 30 robot policies, 42 simulation tasks and 18 real-world tasks to test capabilities including generalization, memory, precision and multi-step execution.
- Developed with researchers from nearly 20 universities, RoboDojo is intended to provide a standardized and reproducible way to compare robot-learning systems across virtual and physical environments.
Researchers led by the University of Hong Kong have developed a new benchmark for testing physical AI systems across both simulation and real-world robot tasks, finding that even the strongest models remain far behind human performance.
The RoboDojo platform was developed by the Multimedia Laboratory at the University of Hong Kong, in collaboration with researchers from nearly 20 universities, including the University of California, Berkeley, and Tsinghua University, according to HKU. The project was co-initiated by Ping Luo, associate director of AI research and Ttech transfer with the HKU School of Computing and Data Science, and doctoral student Tianxing Chen.
The university pointed out that systems are often evaluated using different hardware, simulated environments and scoring methods, making it difficult to compare performance across models or determine whether results will carry over to real-world settings.
HKU said RoboDojo is intended to provide a reproducible way to compare robot-learning approaches in both virtual and physical environments rather than relying on isolated demonstrations or results generated under incompatible testing conditions.
“To the best of our knowledge, RoboDojo is the first Hong Kong-led benchmark to unify simulation and standardised real-robot evaluation,” noted Luo. “It moves embodied AI beyond impressive demonstrations towards progress that can be measured, compared and trusted.”
What is RoboDojo?
RoboDojo combines simulation testing, standardized physical-robot testing and evaluation of robot control policies within a single framework. According to HKU, the platform includes:
- 30 representative robot policies
- 42 simulation tasks
- 18 real-world robot tasks
The tests examine several capabilities considered important for physical AI, including a robot’s ability to generalize to new situations, remember information, carry out precise actions and complete tasks that require multiple steps over longer periods.
Physical AI and Human Performance Gap
Early results point to a large gap between current AI systems and human performance.
According to HKU, the top-performing AI model achieved a success rate of 8.8% on simulated tasks and 12.8% during real-world robot testing. Human experts, by comparison, recorded success rates of 76.03% in simulation and 100% on the physical tasks.
These results, the university added, highlight how a system that can complete a task under carefully selected conditions may struggle when environments, objects or sequences of actions change and that standardized benchmarks could make those weaknesses easier to identify by giving researchers a common set of tasks and measurements for comparing models.
The researchers are also making the benchmark available to the broader robotics community. According to HKU, RoboDojo’s open-source resources have been downloaded more than 100,000 times from Hugging Face since launch, while posts about the project received more than 100,000 views on X during its first week.
Featured image: Prof. Ping Luo is an Associate Director (AI Research and Tech Transfer) of School of Computing and Data Science (CDS), The University of Hong Kong. (Credit: The University of Hong Kong)