Build the EvalOps Feedback Operating System

[haasonsaas]

Summary

Build a cross-repo feedback operating system that ingests PR review feedback, normalizes it into a shared ledger, clusters recurring issue classes, converts high-confidence classes into repo-local guardrails, and reports whether EvalOps repos are getting safer over time.

Current spine already shipped

• ci: emit review feedback ledger JSON #43: review feedback sentinel emits review-feedback-ledger.json
• ci: rank review feedback guardrail backlog #44: sentinel emits ranked guardrail backlog artifacts and supports larger backfills with pr_limit

Phases

1. Data spine

   • Capture review threads, top-level comments, review bodies, and CI/app feedback into one schema.
   • Preserve source links, repo/PR metadata, severity, author/app, state, path, and normalized class.

2. Backfill and taxonomy

   • Run 30-60 day backfills across EvalOps.
   • Cluster recurring classes such as runtime smoke gaps, workflow shell footguns, generated contract drift, release train drift, configuration safety, auth/security gaps, docs drift, and missing regression coverage.

3. Guardrail adapters

   • Platform: generated contract drift, SDK/changelog coverage, runtime evidence and replay/idempotency regressions.
   • Deploy: workflow shell safety, release-train desired-state drift, k8s/Terraform render invariants.
   • Maestro: upstream parity, MCP/prompt/tool contracts, replay/fuzz coverage.
   • .github: org-level reporting, duplicate avoidance, and backlog routing.

4. Automation loop

   • Open or update repo-scoped issues for recurring classes.
   • Link all originating review comments.
   • Generate acceptance criteria and guardrail suggestions.
   • Track whether the prevention PR landed.

5. Operator surface

   • Produce a weekly Slack or GitHub summary with top recurring classes, repos with rising feedback, newly prevented classes, stale unresolved feedback, and next guardrail candidates.

Acceptance criteria

• Every merged EvalOps PR with high+ unresolved meaningful feedback is discoverable from a ledger artifact.
• The ranked backlog produces stable JSON and markdown outputs from at least a 30-day backfill.
• At least 10 recurring classes have repo-local guardrails with tests and CI wiring.
• Platform, Deploy, Maestro, and .github consume either the ledger or derived backlog/report.
• Weekly reporting identifies the next guardrail candidates automatically.
• The top recurring issue classes show a measurable repeat-rate drop over a two-week window.

Immediate next slices

• Add scheduled 30-day backfill artifact publishing separate from the six-hour sentinel.
• Turn the current runtime-smoke-coverage finding from Platform #1676 into a repo-local regression or preflight guard.
• Turn the current workflow-shell-footgun finding from Deploy #2382 into a workflow guardrail or actionlint extension.
• Add duplicate-aware issue routing from guardrail backlog classes into repo-specific issues.

[haasonsaas]

Progress update for the first implementation slice:

• Merged ci: rank review feedback guardrail backlog #44 in evalops/.github: review-feedback sentinel now emits a ranked guardrail backlog artifact plus JSON.
• Merged evalops/deploy#2387 and #2390: workflow checksum-sync feedback is now enforced by parser-backed guardrails.
• Merged evalops/platform#1690: runtime receipt replay self-contention is fixed and covered by a one-connection Postgres pool regression.

The first live backlog candidates from the feedback ledger are now converted into repo-local guardrails. Next horizon slices should extend the sentinel from ranked discovery into issue/guardrail lifecycle tracking, then add broader guardrail classes for generated contract drift and release-train drift.

[haasonsaas]

Second implementation slice merged:

• Merged feat: track feedback guardrail lifecycle #47: the review-feedback sentinel now maintains one stable issue per ranked guardrail class with deterministic titles/bodies and acceptance criteria.
• Added review-feedback-guardrail-lifecycle.json so each run records whether class issues were created, updated, or reopened.
• The workflow uploads the lifecycle artifact alongside the ledger and backlog artifacts.

This moves the system from discovery/ranking into lifecycle tracking. The next slices should add stale/closed reconciliation for class issues and repo-local automation for generated-contract-drift/release-train-drift.

[haasonsaas]

Third implementation slice merged and reconciled:

• Merged fix: reconcile feedback guardrail classes #49: added parser/CLI contract drift and visual capture resilience feedback classes.
• Added stale guardrail-class issue reconciliation; classes no longer present in the active backlog are closed as closed_stale in lifecycle JSON.
• Ran a live 24h reconciliation from merged main: created [codex] Guardrail backlog: Parser and CLI contract drift (parser-cli-contract) #50 for parser-cli-contract and closed stale [codex] Guardrail backlog: Other feedback (other-feedback) #48 for other-feedback.

Current active guardrail-class tracker from the live backlog: #50 parser/CLI contract drift, sourced from evalops/deploy#2390 feedback.

[haasonsaas]

Update: shipped and validated the resolved-feedback lifecycle slice.

• Merged fix: keep resolved feedback guardrails closed #51 (3da1b572), which persists finding fingerprints in guardrail backlog JSON and guardrail issue bodies.
• The sentinel now returns already_closed for a CLOSED guardrail issue when the current backlog only contains already-recorded fingerprints, and reopens only on a new fingerprint.
• Refreshed [codex] Guardrail backlog: Parser and CLI contract drift (parser-cli-contract) #50 with fingerprints from main, closed it as already addressed by evalops/deploy#2390, then reran the live sentinel. It still sees the historical deploy#2390 review comment, but lifecycle output is now already_closed and [codex] Guardrail backlog: Parser and CLI contract drift (parser-cli-contract) #50 remains CLOSED.

This closes the class where historical fixed feedback kept reappearing as active backlog noise.

[haasonsaas]

Update: shipped the scheduled 30-day backfill slice.

• Merged ci: publish scheduled review feedback backfills #52 (581450c3), adding a separate weekly review-feedback-backfill.yml workflow.
• The workflow runs the feedback sweep in --dry-run mode with a default 720-hour lookback and uploads 90-day-retained ledger/backlog artifacts without mutating guardrail issues.
• Manually dispatched the merged workflow on main with since_hours=720, pr_limit=50, min_severity=high: run 25621389109 succeeded.
• Verified both uploaded artifacts exist: review-feedback-30d-ledger and review-feedback-30d-guardrail-backlog.

This covers the immediate slice for scheduled 30-day backfill artifact publishing separate from the six-hour sentinel.

[haasonsaas]

Update: shipped duplicate-aware repo-local guardrail routing.

• Merged feat: route feedback guardrails to source repos #53 (c78f6c74), adding repo-local guardrail candidate issue routing keyed by class plus per-repo finding fingerprints.
• The six-hour sentinel now passes --guardrail-repo-issues; the 30-day backfill remains dry-run only.
• Repo-local routing is suppressed when the org-level class result is already_closed, preventing already-burned-down historical feedback from opening repo tickets.
• Ran the merged sentinel path live over the current 24h window: lifecycle emitted already_closed for [codex] Guardrail backlog: Parser and CLI contract drift (parser-cli-contract) #50 and no open evalops/deploy repo-local parser/CLI candidate issue was created.

This covers the immediate duplicate-aware repo-specific issue routing slice while preserving the resolved-feedback noise controls.

[haasonsaas]

Update: shipped the weekly operator report slice.

• Merged feat: publish weekly feedback guardrail report #55 (ce75ed57), adding a weekly guardrail report generated from the 30-day backfill.
• The report summarizes top guardrail classes, repos with feedback, already-closed/suppressed classes, and next actions.
• The backfill workflow now publishes to one stable .github issue instead of creating a new report issue each run.
• Dispatched the merged workflow on main: run 25621645599 succeeded and uploaded both 30-day artifacts.
• Verified the stable report issue exists: [codex] Weekly review feedback guardrail report #56 [codex] Weekly review feedback guardrail report.

This covers the operator-surface slice from the feedback operating system tracker.

[haasonsaas]

Update: shipped repeat-rate trend and fixed false-empty workflow evidence.

• Merged feat: add feedback repeat-rate trend #57 (b4bd85ba): the weekly report now computes current-vs-previous 7-day repeat-rate metrics by feedback class from ledger merged_at timestamps.
• A live local 30-day dry-run rendered the current parser-cli-contract class as current=1, previous=0, change=new.
• Dispatching the merged backfill workflow exposed that Actions was using a repo-scoped fallback token and publishing empty ledgers.
• Merged ci: require org token for feedback sweeps #58 (36b9e479): sentinel/backfill now require secrets.EVALOPS_REVIEW_GUARD_TOKEN instead of falling back to github.token, so missing org-wide visibility fails loudly.
• Merged ci: skip feedback artifacts after preflight failure #59 (baa37508): artifact uploads are skipped when the token preflight prevents the sweep from running, leaving a clean actionable failure.
• Verified run 25621858887 now fails only at Require org review guard token and skips downstream sweep/artifact steps.

Current blocker: configure EVALOPS_REVIEW_GUARD_TOKEN in evalops/.github (or org/repo secrets) with org-wide PR read access. After that, rerun the 30-day backfill to update #56 with real repeat-rate data from Actions.

[haasonsaas]

Update: unblocked the real Actions backfill/report path and verified repeat-rate reporting from CI.

• Merged ci: accept existing org read token for feedback sweeps #60 (90b86530): sentinel/backfill now accept the existing EVALOPS_ORG_READ_TOKEN secret while still refusing repo-scoped github.token fallback.
• Dispatched the merged 30-day backfill workflow: run 25621917700 succeeded with the org token preflight passing.
• Verified uploaded artifacts grew from the false-empty run: review-feedback-30d-ledger is 1034 bytes and review-feedback-30d-guardrail-backlog is 1832 bytes.
• Verified [codex] Weekly review feedback guardrail report #56 now contains real data: source findings 1, ranked classes 1, and repeat-rate trend parser-cli-contract current=1 previous=0 delta=1 change=new.

This gives the feedback operating system an Actions-backed two-bucket repeat-rate signal instead of local-only evidence.