The official dataset provided by robotwin2.0 does not contain depth maps

[2561681244]

Hello!
The official dataset provided by robotwin2.0 does not contain depth maps.

This is official link of RoboTwin2.0 dataset:
https://huggingface.co/datasets/TianxingChen/RoboTwin2.0/tree/main

This is data structure:

[monnv1]

I’m running into the same issue as well.

Has anyone found a fix, or confirmed whether this has already been resolved in the latest version?

Also, which dataset is recommended for Holobrain? If possible, could you share the correct link or address?

[AlanHuangT]

@2561681244 Hi, Please see similar issue #5

We re-genereta observations with camera depth for training and inference.

@monnv1 Since HoloBrain is based on SEM(https://arxiv.org/abs/2505.16196), we recommend using RoboTwin2.0 for all experiments.

[linxuewu]

Hi.
The best practice is to regenerate the data yourself and enable the depthoption in the config when generating data with RoboTwin.

If you do not want to generate new data, there are two ways to compensate:

1. Turn off with_depth and with_depth_loss in HoloBrain config.
2. Provide a fake depth map filled entirely with zeros.
   @monnv1 @2561681244

[2561681244]

Thanks!

[monnv1]

Thanks

[2561681244]

I took this method:Provide a fake depth map filled entirely with zeros. However, the robotic arm basically stays in place at inference.

[linxuewu]

Did you test it directly using the checkpoint we provided, or did you retrain the model with fake depth input?

I took this method:Provide a fake depth map filled entirely with zeros. However, the robotic arm basically stays in place at inference.

[2561681244]

Did you test it directly using the checkpoint we provided, or did you retrain the model with fake depth input?

I took this method:Provide a fake depth map filled entirely with zeros. However, the robotic arm basically stays in place at inference.

I used HoloBrain_v0.0_Qwen checkpoint as initialization and retrained the model with fake depth input .

[linxuewu]

That’s a bit strange—is the loss able to converge normally? @2561681244

[linxuewu]

Do all tasks stay in place during inference? How many steps are executed in a single inference pass? You could try increasing the number of execution steps to the maximum, for example, 64 steps. @2561681244

[2561681244]

That’s a bit strange—is the loss able to converge normally? @2561681244

Yeah!It converged.

[2561681244]

Do all tasks stay in place during inference? How many steps are executed in a single inference pass? You could try increasing the number of execution steps to the maximum, for example, 64 steps. @2561681244

When I increased the number of execution steps from 32 to 64,the problem still exists.

If possible, can I add you on WeChat？I can send generated videos to you.Thank you!

[linxuewu]