Skip to content

feat(dataviewer): add language/task instruction editing for VLA annotation #437

@akzaidi

Description

@akzaidi

Component

data-management/viewer/ (frontend + backend)

Problem Statement

VLA models require per-episode natural language instructions that describe the task the robot should perform (e.g., "Pick up the box and place it on the conveyor belt"). The dataviewer annotation system currently supports task completeness ratings, trajectory quality scores, data quality flags, and anomaly markers — but has no field for language/task instructions.

The backend annotation model was extended to include LanguageInstructionAnnotation (branch feat/vla-twinvla-robotwin), but the frontend has no UI to create, view, or edit these annotations.

Without this capability:

  1. Operators cannot annotate existing demonstration datasets with language instructions for VLA fine-tuning
  2. Post-hoc language labeling of LeRobot datasets is not possible through the dataviewer
  3. Language instruction quality/consistency cannot be reviewed across episodes
  4. Paraphrase augmentation and subtask decomposition cannot be authored through the tool

Proposed Solution

Add a language instruction editing panel to the episode annotation view:

  1. Instruction text field — primary text input for the natural language instruction (required, 1-500 chars)
  2. Source selector — dropdown for InstructionSource: human, template, vlm-generated
  3. Language selector — ISO language code (default: en)
  4. Paraphrase list — editable list of alternative phrasings for instruction augmentation during training
  5. Subtask decomposition — ordered list of subtask instructions for hierarchical VLAs (π₀.₅, HiRobot)
  6. Bulk annotation — apply the same instruction to multiple episodes of the same task with one action
  7. Template library — save and reuse common instruction templates per dataset

Backend Model (Already Implemented)

class LanguageInstructionAnnotation(SanitizedModel):
    instruction: str = Field(min_length=1, max_length=500)
    source: InstructionSource  # human | template | vlm-generated
    language: str = Field(default="en", max_length=10)
    paraphrases: list[str]
    subtask_instructions: list[str]

The EpisodeAnnotation model already includes an optional language_instruction: LanguageInstructionAnnotation | None field.

Acceptance Criteria

  • Episode annotation view includes a language instruction panel
  • Instruction text field validates 1-500 character range
  • Source type selectable (human, template, vlm-generated)
  • Paraphrases can be added, edited, and removed
  • Subtask instructions can be added, reordered, and removed
  • Bulk annotation applies the same instruction across selected episodes
  • Instruction data persists through the existing annotation save/load API
  • Annotation summary includes language instruction coverage statistics (annotated vs. unannotated episodes)
  • npm run test passes with new component tests

Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/srcSource code in src directoryenhancementNew feature or improvement request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions