MetaMuseum
Menu
← Documentation home

Canonical Markdown source · Oct 20, 2018

Golden Eval Dataset: Complex Museum Questions

evals/golden-museum-questions.md · 168 lines · SHA-256 2876a2b5e78d

This document defines the golden dataset used for AI/LLM reliability checks on grounding and citation behavior.

Dataset file:

  • `evals/golden-museum-questions.v1.json`

Scope

  • Minimum prompt count: `100` (current: `120`)
  • Focus: complex museum/research questions where grounding quality is high risk
  • Required behavior: `cite-or-refuse`

Prompt contract

Each prompt row includes:

  • stable `id` (`MM-EVAL-xxxx`)
  • `domain` and `difficulty`
  • full natural-language `prompt`
  • `expectedBehavior` with:
  • grounding requirements
  • per-claim citation requirements
  • refusal behavior when evidence is missing
  • explicit failure behaviors (`mustNot`)

Citation expectations are strict:

  • per-claim citations
  • entity identifier references
  • property-path scoped references
  • source URL and retrieval timestamp

Reliability rubric

Dataset-level thresholds:

  • `groundedPrecisionThreshold = 0.95`
  • `citationCoverageThreshold = 0.95`
  • `citationFreshnessThreshold = 0.95`
  • refusal policy = `cite-or-refuse`

Conformance enforcement

Executable quality test:

  • `tests/quality/ai-eval-golden-dataset.test.ts`

The test enforces:

  • prompt minimum (`>=100`)
  • prompt ID uniqueness
  • domain diversity
  • required grounding/citation/refusal fields for every prompt row

CI gate (AI-layer PRs)

Eval harness:

  • `scripts/ai-eval-gate.ts`
  • `src/services/ai-eval-harness.ts`

Commands:

  • `pnpm ai:eval:report` (writes artifact without failing)
  • `pnpm ai:eval:gate` (fails when thresholds/regressions are not met)
  • `pnpm ai:eval:baseline:record` (records/updates baseline for the current dataset/model/prompt identity)

Workflow:

  • `.github/workflows/ai-eval-gate.yml`

Gate metrics:

  • faithfulness
  • relevance
  • citation accuracy
  • citation freshness

Output artifact:

  • `artifacts/evals/ai-eval-gate-latest.json`
  • `artifacts/evals/runs/ai-eval-gate-<timestamp>.json`
  • `artifacts/evals/trend-index.json`
  • `artifacts/evals/summary.md`
  • `artifacts/evals/summary.md` includes review priority when at least two retained runs exist.
  • `/api/ai-evals/summary` exposes the same review priority and severity history as JSON for agents.
  • `/api/openapi` references `AiEvalSummaryResponse` for `/api/ai-evals/summary`, and route tests parse payloads with `aiEvalSummaryResponseSchema`.
  • `/api/ai-evals/summary` returns `schemaVersion: 1`; unsupported versions must be rejected by agent contract tests before consumption.

Schema migration notes:

  • `schemaVersion: 1` is the current agent contract for `/api/ai-evals/summary`.
  • `schemaVersion: 2` is reserved for future additive agent fields or review-priority metadata changes that cannot be represented safely in v1.
  • Do not accept v2 payloads until a v2 parser, v2 fixture, OpenAPI schema update, and migration notes land in the same RSI cycle.
  • Fixture compatibility tests keep `tests/fixtures/ai-eval-summary/schema-v1-ready.json` accepted and `schema-v2-planned.json` rejected until that migration exists.

CI summary behavior:

  • `.github/workflows/ai-eval-gate.yml` appends `artifacts/evals/summary.md` to `$GITHUB_STEP_SUMMARY`.
  • The same workflow uploads `artifacts/evals/` as `ai-eval-artifacts`.
  • Summary badges include status, faithfulness, relevance, citation accuracy, `citationFreshness`, and pass rate.
  • Freshness-aging alerts warn when `citationFreshness` stays flat/green while the oldest cited evidence approaches the max policy window.
  • Review priority summarizes the latest-vs-previous severity (`regression`, `watch`, `stable`, or `improved`) directly in the CI summary.
  • Malformed `diffSeverityPolicy` values fail validation instead of silently falling back.
  • CI logs emit a GitHub warning annotation when review priority is `watch` or `regression`.

CI annotation examples:


::warning title=AI eval regression::Review priority is regression. current-run vs previous-run. Review severity%3A regression.
::warning title=AI eval watch::Review priority is watch. current-run vs previous-run. Review severity%3A watch.

Dashboard:

  • `/ai-evals` is the read-only local artifact dashboard for humans and AI-assisted operators.
  • It reads ignored local `artifacts/evals/*` files when present and shows an explicit non-failing empty state when absent.
  • It surfaces latest metrics, run identity, `citationFreshness`, freshness-aging pressure, active warnings, trend history, and artifact timestamps.
  • It includes a latest-vs-previous diff for retained runs with metric deltas, freshness-aging pressure movement, and “what changed” notes for fast review.
  • It labels latest-vs-previous diffs as `regression`, `watch`, `stable`, or `improved` using explicit metric-drop and freshness-aging thresholds.
  • It reads those thresholds from `config/ai-eval-regression-policy.json` and shows policy-window severity distribution counts.
  • It shows a compact severity sparkline/history where `!` = regression, `~` = watch, `=` = stable, and `+` = improved.

Retention controls:

  • `METAMUSEUM_EVAL_RETENTION_MAX_RUNS` (default `200`)
  • optional CLI override: `--retention-max-runs=<n>`
  • `severityHistoryPolicy.maxComparisons` in `config/ai-eval-regression-policy.json` controls how many latest-vs-previous severity comparisons are shown to humans and agents.
  • Orphaned JSON files under `artifacts/evals/runs/` are pruned when they fall outside the retained trend index; non-JSON operator notes are preserved.
  • Retention pruning reports `delete` or `dry-run` mode in `artifacts/evals/summary.md`, including retained, orphaned, deleted, and preserved-file counts.
  • The latest retention pruning report is also exposed through `/api/ai-evals/summary` as `summary.retentionPruneReport` for agent consumers.
  • Generated CI summary markdown is snapshot-locked by `tests/fixtures/ai-eval-summary/summary-snapshot.md`.
  • `/api/openapi` includes the AI eval summary schema migration compatibility table so agents can discover current and planned summary versions.

Regression thresholds + fail-fast citation drift

Policy file:

  • `config/ai-eval-regression-policy.json`

Policy includes:

  • absolute floor thresholds (`faithfulness`, `relevance`, `citationAccuracy`, `citationFreshness`, `passRate`)
  • drift thresholds (`max*Drop`) for regression comparisons
  • diff severity thresholds (`diffSeverityPolicy`) for dashboard/CI review priority
  • `diffSeverityPolicy` schema/order validation before severity classification
  • severity-history retention window (`severityHistoryPolicy.maxComparisons`) with explicit malformed-policy failure tests
  • `failFastOnCitationDrift` control
  • `failFastOnCitationFreshnessDrift` control
  • versioned baselines keyed by:
  • dataset ID + dataset version
  • model version
  • prompt version

Version identity env vars:

  • `METAMUSEUM_EVAL_MODEL_VERSION`
  • `METAMUSEUM_EVAL_PROMPT_VERSION`
  • `METAMUSEUM_EVAL_RETENTION_MAX_RUNS`

Fail-fast behavior:

  • If citation accuracy drop exceeds `maxCitationAccuracyDrop`, gate exits immediately with failure.
  • If citation freshness drop exceeds `maxCitationFreshnessDrop`, gate exits immediately with failure.
  • If citation freshness is flat or improved while `oldestAgeRatio` crosses the aging-pressure threshold, the summary emits a `citation_freshness_aging_pressure` warning for proactive review.