Add Stirrup agent + GDPVal eval/RL environment (#1090)

## Summary

Adds a Stirrup-based agent + a GDPVal benchmark built on the NeMo-Gym
benchmark convention (`ng_prepare_benchmark` +
`ng_e2e_collect_rollouts`),
validated on the full 220-task GDPVal set in both rubric and comparison
scoring modes.

### Architecture

Split into three pieces, matching NeMo-Gym's server-type convention:

**Benchmark** — `benchmarks/gdpval/`
- `prepare.py` downloads `openai/gdpval` from HF →
`data/gdpval_benchmark.jsonl`
- `config.yaml` wires `gdpval_judge_model` + `gdpval_resources_server` +
  `gdpval_stirrup_agent`
- Entry point: `ng_e2e_collect_rollouts
+config_paths=[benchmarks/gdpval/config.yaml]`

**Resources server** — `resources_servers/gdpval/`
- Owns `verify()` and `aggregate_metrics()` with two modes via
`reward_mode`:
- `rubric` (default) — LLM-judge per-criterion score, reward in `[0, 1]`
  - `comparison` — pairwise vs `reference_deliverables_dir`, reward in
    `{0, 0.5, 1}`; `aggregate_metrics` reduces W/L/T → ELO anchored at
    `reference_elo` (default 1000)
- All scoring, pairwise comparison, Office→PDF preconvert live here.
  Multimodal judge path used whenever content blocks are available.

**Agent** — `responses_api_agents/stirrup_agent/`
- `StirrupAgentWrapper` is task-agnostic; task-specific logic in a
  `TaskStrategy` subclass (`GDPValTask`)
- `/run` executes the agent, persists deliverables, POSTs to the
resources
  server's `/verify`, returns the response. Agent is scoring-free.
- `aggregate_metrics` proxies to resources server so ELO extras flow
through;
`/verify` errors caught per-rollout so a single failure can't crash a
run
- Optional: Apptainer-backed `code_exec`, Tavily web-search

**Dependency:** `stirrup>=0.1.7` (Apache 2.0) declared as an extra of
the
stirrup_agent server, not of core.

## Validation (Ultra V3 SFT iter16k, full 220-task GDPVal,
num_repeats=2)

**Rubric mode** (n=440):
- mean/reward = **0.755**, pass@1 = 0.755, pass@2 = 0.821
- 56% of rollouts score ≥ 0.8
- Pre-refactor port-v4 baseline: 0.24 → **3.1× lift** (dominant
contributor:
  always-visual judge when content blocks are available)

**Comparison mode vs fork baseline** (4 trials per pairing, n=440):
- W/L/T = 147 / 208 / 77
- win_rate = 0.429
- **eval_elo = 950.6** (vs fork=1000; port-v3 historical=917 → +34 ELO)

## Running

```bash
ng_prepare_benchmark '+config_paths=[benchmarks/gdpval/config.yaml]'
ng_e2e_collect_rollouts \
  '+config_paths=[benchmarks/gdpval/config.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]' \
  '++split=benchmark' \
  '++output_jsonl_fpath=results/gdpval.jsonl' \
  "++gdpval_stirrup_agent.responses_api_agents.stirrup_agent.persist_deliverables_dir=$PWD/output/gdpval" \
  # ... policy_* overrides as usual

# Add for comparison mode:
  '++gdpval_resources_server.resources_servers.gdpval.reward_mode=comparison' \
  "++gdpval_resources_server.resources_servers.gdpval.reference_deliverables_dir=/path/to/reference"
```

## Test plan

- [x] `pytest resources_servers/gdpval/tests -x` — rubric + comparison
unit tests
- [x] 10-task rubric smoke (mean/reward 0.719)
- [x] Full 220-task rubric (mean/reward 0.755)
- [x] 10-task comparison smoke (eval_elo ~1017 on small sample)
- [x] Full 220-task comparison (eval_elo 950.6)

---------

Signed-off-by: Serge Panev <spanev@nvidia.com>

Author: Kh4L

benchmarks/gdpval/README.md

@@ -0,0 +1,68 @@
+# GDPVal benchmark
+
+[GDPVal](https://huggingface.co/datasets/openai/gdpval) — 220 professional
+knowledge-work tasks scored by an LLM judge against per-task rubrics. This
+benchmark wires the Stirrup-based agent (`responses_api_agents/stirrup_agent`)
+to the GDPVal resources server (`resources_servers/gdpval`).
+
+## Prepare data
+
+Downloads `openai/gdpval` from HuggingFace and writes
+`data/gdpval_benchmark.jsonl`:
+
+```bash
+ng_prepare_benchmark "+config_paths=[benchmarks/gdpval/config.yaml]"
+```
+
+## Run rubric mode (default)
+
+Each deliverable is scored 0–1 against the task rubric.
+
+```bash
+config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
+benchmarks/gdpval/config.yaml"
+ng_e2e_collect_rollouts \
+    "+config_paths=[${config_paths}]" \
+    ++output_jsonl_fpath=results/gdpval_rubric.jsonl \
+    ++split=benchmark \
+    ++policy_base_url=<vllm_base_url> \
+    ++policy_api_key=<vllm_api_key> \
+    ++policy_model_name=<served_model_name>
+```
+
+Required environment variables for the judge:
+
+- `JUDGE_API_KEY` — sk- key for the judge inference API (nvapi- keys 401 on
+  multimodal payloads)
+- `JUDGE_BASE_URL` — defaults to NVIDIA's internal inference API
+- `JUDGE_MODEL_NAME` — defaults to `gcp/google/gemini-3.1-pro-preview`
+- `HF_TOKEN` — for downloading reference files (avoids HF anonymous rate limits)
+
+## Run comparison mode (pairwise ELO vs. a reference model)
+
+Each deliverable is judged against a reference model's deliverable for the
+same `task_id`; aggregate metrics include ELO relative to a configurable
+anchor (default 1000).
+
+```bash
+ng_e2e_collect_rollouts \
+    "+config_paths=[${config_paths}]" \
+    ++output_jsonl_fpath=results/gdpval_compare.jsonl \
+    ++split=benchmark \
+    ++gdpval_resources_server.resources_servers.gdpval.reward_mode=comparison \
+    ++gdpval_resources_server.resources_servers.gdpval.reference_deliverables_dir=/path/to/reference/output
+```
+
+The reference directory must be laid out as
+`<reference_deliverables_dir>/task_<task_id>/` with `finish_params.json` and
+the deliverable files (the same layout the Stirrup agent persists).
+
+## Aggregate metrics
+
+After `ng_e2e_collect_rollouts` returns, the resources server's
+`/aggregate_metrics` endpoint emits headline scores in
+`results/<output>_metrics.json`:
+
+- Rubric mode: `mean/reward` (pass@1 equivalent)
+- Comparison mode: `comparison/wins`, `comparison/losses`, `comparison/ties`,
+  `comparison/win_rate`, `comparison/eval_elo`, `comparison/normalized_elo`

benchmarks/gdpval/__init__.py

@@ -0,0 +1,14 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

benchmarks/gdpval/config.yaml

@@ -0,0 +1,66 @@
+# GDPVal benchmark — Stirrup agent + GDPVal resources server.
+#
+# Run:
+#   ng_prepare_benchmark "+config_paths=[benchmarks/gdpval/config.yaml]"
+#   ng_e2e_collect_rollouts \
+#     "+config_paths=[responses_api_models/vllm_model/configs/vllm_model.yaml,benchmarks/gdpval/config.yaml]" \
+#     ++split=benchmark \
+#     ++output_jsonl_fpath=results/gdpval.jsonl
+#
+# Comparison mode (pairwise ELO vs a reference model's deliverables):
+#   ++gdpval_resources_server.resources_servers.gdpval.reward_mode=comparison \
+#   ++gdpval_resources_server.resources_servers.gdpval.reference_deliverables_dir=/path/to/fork
+
+# Judge model — proxy to NVIDIA inference API for Gemini 3.1 Pro.
+gdpval_judge_model:
+  responses_api_models:
+    openai_model:
+      entrypoint: app.py
+      openai_base_url: ${oc.env:JUDGE_BASE_URL,https://inference-api.nvidia.com/v1}
+      openai_api_key: ${oc.env:JUDGE_API_KEY,dummy}
+      openai_model: ${oc.env:JUDGE_MODEL_NAME,gcp/google/gemini-3.1-pro-preview}
+
+# GDPVal resources server (rubric scoring by default; switch to comparison via override).
+gdpval_resources_server:
+  resources_servers:
+    gdpval:
+      entrypoint: app.py
+      domain: other
+      verified: false
+      reward_mode: rubric
+      reference_deliverables_dir: null
+      num_comparison_trials: 4
+      reference_elo: 1000.0
+      preconvert_office_to_pdf: true
+      preconvert_max_concurrent: 1
+      judge_model_server:
+        type: responses_api_models
+        name: gdpval_judge_model
+      judge_responses_create_params_overrides: {}
+
+# Stirrup agent paired with the resources server above.
+gdpval_stirrup_agent:
+  responses_api_agents:
+    stirrup_agent:
+      entrypoint: app.py
+      task: gdpval
+      agent_max_turns: 100
+      concurrency: 32
+      temperature: 1.0
+      system_prompt_template: ${oc.env:SYSTEM_PROMPT_TEMPLATE,null}
+      user_prompt_template: ${oc.env:USER_PROMPT_TEMPLATE,null}
+      gdpval_container_path: ${oc.env:GDPVAL_CONTAINER_PATH,null}
+      persist_deliverables_dir: ${oc.env:PERSIST_DELIVERABLES_DIR,output/gdpval/deliverables}
+      resources_server:
+        type: resources_servers
+        name: gdpval_resources_server
+      model_server:
+        type: responses_api_models
+        name: policy_model
+      datasets:
+        - name: gdpval
+          type: benchmark
+          jsonl_fpath: benchmarks/gdpval/data/gdpval_benchmark.jsonl
+          prompt_config: null
+          prepare_script: benchmarks/gdpval/prepare.py
+          num_repeats: 2

benchmarks/gdpval/data/.gitignore

@@ -0,0 +1,6 @@
+*train.jsonl
+*validation.jsonl
+*benchmark.jsonl
+*train_prepare.jsonl
+*validation_prepare.jsonl
+*example_prepare.jsonl

benchmarks/gdpval/prepare.py

@@ -0,0 +1,70 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Prepare the GDPVal benchmark JSONL.
+
+Downloads the ``openai/gdpval`` HuggingFace dataset and converts it into the
+NeMo-Gym benchmark JSONL format: each row has ``responses_create_params`` (an
+empty input — the Stirrup agent builds the actual prompt from the top-level
+``prompt`` / ``sector`` / ``occupation`` fields) plus task metadata at the
+top level so the GDPVal resources server can pick them up via /verify.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+
+
+BENCHMARK_DIR = Path(__file__).parent
+DATA_DIR = BENCHMARK_DIR / "data"
+OUTPUT_FPATH = DATA_DIR / "gdpval_benchmark.jsonl"
+
+HF_DATASET = "openai/gdpval"
+HF_SPLIT = "train"
+
+
+def prepare() -> Path:
+    from datasets import load_dataset
+
+    DATA_DIR.mkdir(parents=True, exist_ok=True)
+    # Pass HF_TOKEN explicitly — ``load_dataset`` doesn't always pick it up
+    # from the env, and GDPVal's bucket aggressively rate-limits anonymous IPs.
+    hf_token = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN")
+    ds = load_dataset(HF_DATASET, split=HF_SPLIT, token=hf_token)
+
+    with OUTPUT_FPATH.open("w") as f:
+        for row in ds:
+            record = {
+                # Empty input: the Stirrup agent constructs the user prompt
+                # from the top-level ``prompt`` field at runtime.
+                "responses_create_params": {"input": []},
+                "task_id": row["task_id"],
+                "sector": row.get("sector", ""),
+                "occupation": row.get("occupation", ""),
+                "prompt": row["prompt"],
+                "reference_files": row.get("reference_files", []),
+                "reference_file_urls": row.get("reference_file_urls", []),
+                "rubric_json": row.get("rubric_json", {}),
+                "rubric_pretty": row.get("rubric_pretty", ""),
+            }
+            f.write(json.dumps(record) + "\n")
+
+    print(f"Wrote {len(ds)} tasks to {OUTPUT_FPATH}")
+    return OUTPUT_FPATH
+
+
+if __name__ == "__main__":
+    prepare()

resources_servers/gdpval/README.md

@@ -0,0 +1,22 @@
+# GDPVal resources server
+
+Scores deliverables produced by the Stirrup agent on the GDPVal benchmark.
+
+Two modes via `reward_mode` config:
+
+- `rubric` (default) — LLM judge scores each deliverable against a per-task
+  rubric, reward in `[0.0, 1.0]`.
+- `comparison` — pairwise judge compares eval deliverable vs. a reference
+  rollout (`reference_deliverables_dir` must be set), reward in
+  `{0.0, 0.5, 1.0}`. `aggregate_metrics` reduces to an ELO rating.
+
+Canonical entry point is the benchmark at `benchmarks/gdpval/`:
+
+```bash
+ng_prepare_benchmark "+config_paths=[benchmarks/gdpval/config.yaml]"
+ng_e2e_collect_rollouts \
+  "+config_paths=[responses_api_models/vllm_model/configs/vllm_model.yaml,benchmarks/gdpval/config.yaml]" \
+  ++split=benchmark
+```
+
+See `benchmarks/gdpval/README.md` for the full run recipe.

resources_servers/gdpval/__init__.py

@@ -0,0 +1,14 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.