{"id":"evals/golden-museum-questions","relativePath":"evals/golden-museum-questions.md","title":"Golden Eval Dataset: Complex Museum Questions","markdown":"# Golden Eval Dataset: Complex Museum Questions\n\nThis document defines the golden dataset used for AI/LLM reliability checks on grounding and citation behavior.\n\nDataset file:\n\n- `evals/golden-museum-questions.v1.json`\n\n## Scope\n\n- Minimum prompt count: `100` (current: `120`)\n- Focus: complex museum/research questions where grounding quality is high risk\n- Required behavior: `cite-or-refuse`\n\n## Prompt contract\n\nEach prompt row includes:\n\n- stable `id` (`MM-EVAL-xxxx`)\n- `domain` and `difficulty`\n- full natural-language `prompt`\n- `expectedBehavior` with:\n - grounding requirements\n - per-claim citation requirements\n - refusal behavior when evidence is missing\n - explicit failure behaviors (`mustNot`)\n\nCitation expectations are strict:\n\n- per-claim citations\n- entity identifier references\n- property-path scoped references\n- source URL and retrieval timestamp\n\n## Reliability rubric\n\nDataset-level thresholds:\n\n- `groundedPrecisionThreshold = 0.95`\n- `citationCoverageThreshold = 0.95`\n- `citationFreshnessThreshold = 0.95`\n- refusal policy = `cite-or-refuse`\n\n## Conformance enforcement\n\nExecutable quality test:\n\n- `tests/quality/ai-eval-golden-dataset.test.ts`\n\nThe test enforces:\n\n- prompt minimum (`>=100`)\n- prompt ID uniqueness\n- domain diversity\n- required grounding/citation/refusal fields for every prompt row\n\n## CI gate (AI-layer PRs)\n\nEval harness:\n\n- `scripts/ai-eval-gate.ts`\n- `src/services/ai-eval-harness.ts`\n\nCommands:\n\n- `pnpm ai:eval:report` (writes artifact without failing)\n- `pnpm ai:eval:gate` (fails when thresholds/regressions are not met)\n- `pnpm ai:eval:baseline:record` (records/updates baseline for the current dataset/model/prompt identity)\n\nWorkflow:\n\n- `.github/workflows/ai-eval-gate.yml`\n\nGate metrics:\n\n- faithfulness\n- relevance\n- citation accuracy\n- citation freshness\n\nOutput artifact:\n\n- `artifacts/evals/ai-eval-gate-latest.json`\n- `artifacts/evals/runs/ai-eval-gate-.json`\n- `artifacts/evals/trend-index.json`\n- `artifacts/evals/summary.md`\n- `artifacts/evals/summary.md` includes review priority when at least two retained runs exist.\n- `/api/ai-evals/summary` exposes the same review priority and severity history as JSON for agents.\n- `/api/openapi` references `AiEvalSummaryResponse` for `/api/ai-evals/summary`, and route tests parse payloads with `aiEvalSummaryResponseSchema`.\n- `/api/ai-evals/summary` returns `schemaVersion: 1`; unsupported versions must be rejected by agent contract tests before consumption.\n\nSchema migration notes:\n\n- `schemaVersion: 1` is the current agent contract for `/api/ai-evals/summary`.\n- `schemaVersion: 2` is reserved for future additive agent fields or review-priority metadata changes that cannot be represented safely in v1.\n- Do not accept v2 payloads until a v2 parser, v2 fixture, OpenAPI schema update, and migration notes land in the same RSI cycle.\n- Fixture compatibility tests keep `tests/fixtures/ai-eval-summary/schema-v1-ready.json` accepted and `schema-v2-planned.json` rejected until that migration exists.\n\nCI summary behavior:\n\n- `.github/workflows/ai-eval-gate.yml` appends `artifacts/evals/summary.md` to `$GITHUB_STEP_SUMMARY`.\n- The same workflow uploads `artifacts/evals/` as `ai-eval-artifacts`.\n- Summary badges include status, faithfulness, relevance, citation accuracy, `citationFreshness`, and pass rate.\n- Freshness-aging alerts warn when `citationFreshness` stays flat/green while the oldest cited evidence approaches the max policy window.\n- Review priority summarizes the latest-vs-previous severity (`regression`, `watch`, `stable`, or `improved`) directly in the CI summary.\n- Malformed `diffSeverityPolicy` values fail validation instead of silently falling back.\n- CI logs emit a GitHub warning annotation when review priority is `watch` or `regression`.\n\nCI annotation examples:\n\n```text\n::warning title=AI eval regression::Review priority is regression. current-run vs previous-run. Review severity%3A regression.\n::warning title=AI eval watch::Review priority is watch. current-run vs previous-run. Review severity%3A watch.\n```\n\nDashboard:\n\n- `/ai-evals` is the read-only local artifact dashboard for humans and AI-assisted operators.\n- It reads ignored local `artifacts/evals/*` files when present and shows an explicit non-failing empty state when absent.\n- It surfaces latest metrics, run identity, `citationFreshness`, freshness-aging pressure, active warnings, trend history, and artifact timestamps.\n- It includes a latest-vs-previous diff for retained runs with metric deltas, freshness-aging pressure movement, and “what changed” notes for fast review.\n- It labels latest-vs-previous diffs as `regression`, `watch`, `stable`, or `improved` using explicit metric-drop and freshness-aging thresholds.\n- It reads those thresholds from `config/ai-eval-regression-policy.json` and shows policy-window severity distribution counts.\n- It shows a compact severity sparkline/history where `!` = regression, `~` = watch, `=` = stable, and `+` = improved.\n\nRetention controls:\n\n- `METAMUSEUM_EVAL_RETENTION_MAX_RUNS` (default `200`)\n- optional CLI override: `--retention-max-runs=`\n- `severityHistoryPolicy.maxComparisons` in `config/ai-eval-regression-policy.json` controls how many latest-vs-previous severity comparisons are shown to humans and agents.\n- Orphaned JSON files under `artifacts/evals/runs/` are pruned when they fall outside the retained trend index; non-JSON operator notes are preserved.\n- Retention pruning reports `delete` or `dry-run` mode in `artifacts/evals/summary.md`, including retained, orphaned, deleted, and preserved-file counts.\n- The latest retention pruning report is also exposed through `/api/ai-evals/summary` as `summary.retentionPruneReport` for agent consumers.\n- Generated CI summary markdown is snapshot-locked by `tests/fixtures/ai-eval-summary/summary-snapshot.md`.\n- `/api/openapi` includes the AI eval summary schema migration compatibility table so agents can discover current and planned summary versions.\n\n## Regression thresholds + fail-fast citation drift\n\nPolicy file:\n\n- `config/ai-eval-regression-policy.json`\n\nPolicy includes:\n\n- absolute floor thresholds (`faithfulness`, `relevance`, `citationAccuracy`, `citationFreshness`, `passRate`)\n- drift thresholds (`max*Drop`) for regression comparisons\n- diff severity thresholds (`diffSeverityPolicy`) for dashboard/CI review priority\n- `diffSeverityPolicy` schema/order validation before severity classification\n- severity-history retention window (`severityHistoryPolicy.maxComparisons`) with explicit malformed-policy failure tests\n- `failFastOnCitationDrift` control\n- `failFastOnCitationFreshnessDrift` control\n- versioned baselines keyed by:\n - dataset ID + dataset version\n - model version\n - prompt version\n\nVersion identity env vars:\n\n- `METAMUSEUM_EVAL_MODEL_VERSION`\n- `METAMUSEUM_EVAL_PROMPT_VERSION`\n- `METAMUSEUM_EVAL_RETENTION_MAX_RUNS`\n\nFail-fast behavior:\n\n- If citation accuracy drop exceeds `maxCitationAccuracyDrop`, gate exits immediately with failure.\n- If citation freshness drop exceeds `maxCitationFreshnessDrop`, gate exits immediately with failure.\n- If citation freshness is flat or improved while `oldestAgeRatio` crosses the aging-pressure threshold, the summary emits a `citation_freshness_aging_pressure` warning for proactive review.\n","sections":[{"level":2,"heading":"Scope","anchor":"scope"},{"level":2,"heading":"Prompt contract","anchor":"prompt-contract"},{"level":2,"heading":"Reliability rubric","anchor":"reliability-rubric"},{"level":2,"heading":"Conformance enforcement","anchor":"conformance-enforcement"},{"level":2,"heading":"CI gate (AI-layer PRs)","anchor":"ci-gate-ai-layer-prs"},{"level":2,"heading":"Regression thresholds + fail-fast citation drift","anchor":"regression-thresholds-fail-fast-citation-drift"}],"html":"

Golden Eval Dataset: Complex Museum Questions

\n

This document defines the golden dataset used for AI/LLM reliability checks on grounding and citation behavior.

\n

Dataset file:

\n\n

Scope

\n\n

Prompt contract

\n

Each prompt row includes:

\n\n

Citation expectations are strict:

\n\n

Reliability rubric

\n

Dataset-level thresholds:

\n\n

Conformance enforcement

\n

Executable quality test:

\n\n

The test enforces:

\n\n

CI gate (AI-layer PRs)

\n

Eval harness:

\n\n

Commands:

\n\n

Workflow:

\n\n

Gate metrics:

\n\n

Output artifact:

\n\n

Schema migration notes:

\n\n

CI summary behavior:

\n\n

CI annotation examples:

\n
\n::warning title=AI eval regression::Review priority is regression. current-run vs previous-run. Review severity%3A regression.\n::warning title=AI eval watch::Review priority is watch. current-run vs previous-run. Review severity%3A watch.\n
\n

Dashboard:

\n\n

Retention controls:

\n\n

Regression thresholds + fail-fast citation drift

\n

Policy file:

\n\n

Policy includes:

\n\n

Version identity env vars:

\n\n

Fail-fast behavior:

\n","updatedAt":"2018-10-20T01:46:40.000Z","checksum":"2876a2b5e78dbfb093fb404058766bc1f6b8e2a8d841a42c30da8203ac546b23","checksumPrefix":"2876a2b5e78d","anchorCount":6,"lineCount":168,"rawUrl":"/api/docs/content?path=evals%2Fgolden-museum-questions.md","htmlUrl":"/docs?doc=evals%2Fgolden-museum-questions.md","apiUrl":"/api/docs/content?path=evals%2Fgolden-museum-questions.md"}