Parallelized GPU Simulation and Rendering

By scaling parallel environments with GPU simulation, MS-HAB achieves 4000 samples per second on a benchmark involving representative interaction with dynamic objects — 3× Habitat 2.0's implementation at similar GPU memory usage.

MS-HAB environments support realistic low-level control for successful grasping, manipulation, and interaction, while the Habitat 2.0 environments do not support such kind of low-level control.

This means MS-HAB is fast enough to support online training and efficient, extensive evaluation without sacrificing physical realism.

Extensive RL and IL Baselines with Whole-Body Control

To solve the HAB's long-horizon tasks (TidyHouse, PrepareGroceries, SetTable), MS-HAB chains individual skill policies (Pick, Place, Open, and Close). For each skill, MS-HAB provides extensive reinforcement learning (RL) and imitation learning (IL) baselines which use whole-body control, i.e. manipulation and navigation performed simultaneously.




We find that, despite signficant tuning, our baselines are unable to solve the MS-HAB tasks. In particular, individual subtask success rate (Pick, Place, Open, and Close) all leave significant room for improvement. This indicates our task is not yet saturated, and there is room for future work to improve performance.

Efficient, Controlled Data Generation at Scale

We develop a rule-based event labeling and trajectory categorization system to filter for specific demonstrations which match predefined criteria for robot behavior and safety. We provide these tools so users can generate data with custom requirements. We use this filtering system to generate a large vision-based robot dataset to train our IL policies.

Furthermore, we use the trajectory categorization system to group rollouts by success/failure modes. We provide these statistics in the paper appendix to provide the community with clearer insight on avenues for improvement beyond raw success rates.

Supporting Open Source Science

All environments, code, checkpoints, and datasets are open-sourced for the community to use. We will continue to support the MS-HAB benchmark with performance improvments, features, baselines, and tools. If you'd like to request a feature or contribute, please check out the GitHub repository!

Code Models Dataset
Whole-body low-level control under constraints in cluttered environments, long-horizon skill chaining, and scene-level rearrangement are challenging for current robot learning methods; we hope our benchmark and dataset aid the community in advancing these research areas.