Component
data-management/viewer/ (frontend + backend)
Problem Statement
VLA models require per-episode natural language instructions that describe the task the robot should perform (e.g., "Pick up the box and place it on the conveyor belt"). The dataviewer annotation system currently supports task completeness ratings, trajectory quality scores, data quality flags, and anomaly markers — but has no field for language/task instructions.
The backend annotation model was extended to include LanguageInstructionAnnotation (branch feat/vla-twinvla-robotwin), but the frontend has no UI to create, view, or edit these annotations.
Without this capability:
- Operators cannot annotate existing demonstration datasets with language instructions for VLA fine-tuning
- Post-hoc language labeling of LeRobot datasets is not possible through the dataviewer
- Language instruction quality/consistency cannot be reviewed across episodes
- Paraphrase augmentation and subtask decomposition cannot be authored through the tool
Proposed Solution
Add a language instruction editing panel to the episode annotation view:
- Instruction text field — primary text input for the natural language instruction (required, 1-500 chars)
- Source selector — dropdown for
InstructionSource: human, template, vlm-generated
- Language selector — ISO language code (default:
en)
- Paraphrase list — editable list of alternative phrasings for instruction augmentation during training
- Subtask decomposition — ordered list of subtask instructions for hierarchical VLAs (π₀.₅, HiRobot)
- Bulk annotation — apply the same instruction to multiple episodes of the same task with one action
- Template library — save and reuse common instruction templates per dataset
Backend Model (Already Implemented)
class LanguageInstructionAnnotation(SanitizedModel):
instruction: str = Field(min_length=1, max_length=500)
source: InstructionSource # human | template | vlm-generated
language: str = Field(default="en", max_length=10)
paraphrases: list[str]
subtask_instructions: list[str]
The EpisodeAnnotation model already includes an optional language_instruction: LanguageInstructionAnnotation | None field.
Acceptance Criteria
Context
Component
data-management/viewer/(frontend + backend)Problem Statement
VLA models require per-episode natural language instructions that describe the task the robot should perform (e.g., "Pick up the box and place it on the conveyor belt"). The dataviewer annotation system currently supports task completeness ratings, trajectory quality scores, data quality flags, and anomaly markers — but has no field for language/task instructions.
The backend annotation model was extended to include
LanguageInstructionAnnotation(branchfeat/vla-twinvla-robotwin), but the frontend has no UI to create, view, or edit these annotations.Without this capability:
Proposed Solution
Add a language instruction editing panel to the episode annotation view:
InstructionSource: human, template, vlm-generateden)Backend Model (Already Implemented)
The
EpisodeAnnotationmodel already includes an optionallanguage_instruction: LanguageInstructionAnnotation | Nonefield.Acceptance Criteria
npm run testpasses with new component testsContext
data-management/viewer/backend/src/api/models/annotations.py(LanguageInstructionAnnotation)training/vla/(branchfeat/vla-twinvla-robotwin)