Era C Exit-Gate Evidence Pack

This runbook turns the four Era C exit checks into automated, dated pass/fail evidence.

Scope

The evidence pack evaluates:

1. SOTA §20.4 p95 SLOs over a rolling 30-day window: cached Record, cold Record, keyword+facet search, whitelisted SPARQL, and IIIF tile serving.

1. Public-read uptime >= 99.9% over 30 days.

1. >= 3 declared external consumers of `/api/activity` within the adoption window.

1. SOTA §26 KPI thresholds.

Artifact schema

Canonical schemas:

• `docs/schemas/era-c-exit-gate-evidence.schema.json`
• `docs/schemas/review-goals.schema.json`

Latest and historical artifacts:

• `artifacts/exit-gate/era-c-exit-gate-latest.json`
• `artifacts/exit-gate/runs/era-c-exit-gate-<timestamp>.json`
• `artifacts/exit-gate/trend-index.json`
• `artifacts/long-term-evidence/long-term-evidence-runway-latest.json`
• `artifacts/long-term-evidence/runs/long-term-evidence-runway-<timestamp>.json`
• `artifacts/activity-syndication/activity-syndication-evidence-latest.json`
• `artifacts/activity-syndication/runs/activity-syndication-evidence-<timestamp>.json`
• `artifacts/review-goals/review-goals-latest.json`
• `artifacts/review-goals/review-goals-latest.md`
• `artifacts/review-goals/review-goals-<timestamp>.json`
• `artifacts/review-goals/review-goals-<timestamp>.md`

The trend index stores each check's pass/fail state plus compact `failedChecks` reasons so agents can identify the next blocking gate without opening every historical run artifact.

Top-level artifact status is intentionally three-state:

• `passed` — all four Era C exit checks meet policy.
• `blocked` — production exit is waiting on missing, insufficient, stale, or not-yet-external evidence. This is the expected state while collecting 30-day SLO/uptime samples, declared ActivityStreams consumers, or aggregate KPI exports.
• `failed` — measured evidence is present and below policy target, such as a p95 breach, uptime below 99.9% with enough samples, or evidence-backed KPI values below threshold.

`--check` exits non-zero for both `blocked` and `failed`; the distinction prevents missing evidence from being mistaken for a measured production-quality regression.

The long-term runway artifact is not a replacement for the exit gate. It is the operator-facing accumulation view for the 12+ month reliability/adoption work: it reports SLO sample count, distinct UTC observation days, public-read uptime sample depth, ignored placeholder consumer IDs, real declared external consumer IDs, and the next evidence needed before a 30-day production claim is defensible. Use `pnpm longterm:evidence:check` only when the runway is expected to be complete.

When the runway is `failed` because old samples are still inside the 30-day

window, run `pnpm longterm:evidence:maintenance`. It writes

`artifacts/long-term-evidence/long-term-evidence-maintenance-latest.json` and

`.md` with the failed SLO samples, failed uptime probe paths, passing distinct

days, per-failed-sample age-out dates, earliest failure-free window date,

daily evidence-ready date, and remaining external consumer count. The

maintenance report is read-only and must not be used to delete or launder

historical evidence.

Read the two runway dates separately:

failed or incomplete SLO/uptime samples leave the active policy window.

if every scheduled daily run passes.

• `failureFreeWindowReadyAt` / `blockingEvidenceAgesOutAt` says when retained
• `dailyEvidenceReadyAt` says when enough distinct passing UTC days can exist

The strict long-window claim can become green no earlier than the later of

those dates, and only if real external ActivityStreams and KPI/pilot evidence

are also satisfied where selected by review goals.

The ActivityStreams syndication artifact complements adoption telemetry by

requiring the same real declared external consumer IDs to have active external

HTTPS subscriptions covering `Create`, `Update`, and `Delete`. It prevents a

feed-read probe from being mistaken for durable callback workflow evidence.

The review-goals artifact is the 10/10 audit surface. It intentionally records

`external-evidence-required` while production SLO/uptime history, real external

ActivityStreams consumers, and paid-pilot evidence are still absent; it should

not be run with `--check` in nightly evidence collection until those blockers

are expected to be complete. Its JSON output is self-describing via the

`$schema` value in `docs/schemas/review-goals.schema.json`, and each goal lists

`evidenceArtifacts`, observed `evidenceArtifactStatuses`, and selected

`evidenceArtifactCheckStatuses` so operators can jump straight to the latest

launch, long-term, ActivityStreams, or pilot proof target and see whether that

artifact or goal-specific check is missing, present, blocked, failed, or

passing.

Supporting performance trend input:

• `artifacts/performance/k6-slo-summary.json`
• `artifacts/performance/k6-slo-trend.json`

New performance samples must include all five p95 metrics. Legacy three-metric samples remain in history, but they fail the all-SLO gate until replaced by 30 days of complete samples.

Exit-gate reports include `missingMetricsInWindow` plus per-sample `metricDetails` so partial k6 summaries are distinguishable from actual p95 threshold breaches.

Policy and inputs

Policy thresholds:

• `config/era-c-exit-gate-policy.json`
• Public-read uptime evidence must be recent: `uptime.maxSnapshotAgeHours` defaults to `48`, so stale or undated snapshots cannot satisfy the 99.9% gate.
• SOTA §26 KPI evidence must be recent: `kpis26.maxSnapshotAgeHours` defaults to `48`.
• AI query cost must come from real usage telemetry when `kpis26.requireAiQueryCostTelemetry` is `true`; fallback default cost values do not satisfy the KPI gate.
• AI query rows written by `/api/ai/query` include `costUsd`, `costCurrency`, `costSource`, and usage counts. The deterministic local planner records `costUsd: 0` with `costSource: "deterministic-local-planner"` because it makes no paid model call.

Metric inputs:

• Uptime snapshot: `monitoring/public-read-uptime.json`
• Exit-gate reports include the uptime `source` (`prometheus`, `probe`, or `unavailable`) and source `notes` so missing 30-day proof is actionable from the artifact alone.
• Uptime history: `monitoring/public-read-uptime-history.json`
• Probe-based samples retain per-path `checks` with URL, HTTP status, duration, deployment-protection flag, and error text, so a failed public-read uptime sample can be traced back to `/`, `/api/health`, `/api/activity`, or any custom `METAMUSEUM_UPTIME_PROBE_PATHS` route.
• KPI snapshot: `monitoring/kpis.json`
• Exit-gate reports include KPI `sources`, snapshot `notes`, and per-failed-metric details (`metric`, `source`, `reason`) so SOTA §26 blockers can be triaged from the evidence artifact.
• Optional aggregate KPI input: `monitoring/kpi-evidence.json` supplies production-like record-enrichment and reconciliation review counts before `pnpm monitoring:telemetry:sync`; see `docs/ops/kpi-evidence.md`.
• `aiQueryCostUsd` is calculated from `storage/ai-query-log.json`; run at least one representative `/api/ai/query` request in the target environment before telemetry sync so the source is no longer the fallback default.
• Activity adoption telemetry: `storage/activity-consumers.json`
• Declared consumer rows must preserve `class: "declared"` and `declaredId`; the exit-gate normalizer intentionally ignores derived fingerprints for adoption credit.
• Proof command: `pnpm activity:adoption:probe` records one partner-owned declared consumer and writes `artifacts/activity-adoption/activity-adoption-proof-latest.json`.
• Matrix proof command: `pnpm activity:adoption:matrix` records/checks three partner-owned declared consumers and writes `artifacts/activity-adoption/activity-adoption-matrix-latest.json`.
• Activity subscription telemetry: `storage/activity-subscriptions.json`
• Durable workflow proof command: `pnpm activity:syndication:evidence` ties the required declared consumer IDs to active external HTTPS callback subscriptions and writes `artifacts/activity-syndication/activity-syndication-evidence-latest.json`.

Activity adoption counts only declared external consumers (`x-linked-art-consumer-id`). Derived fingerprints remain useful diagnostics, but they do not satisfy the external Linked Art systems exit gate.

See `docs/ops/activity-adoption-proof.md` for partner hand-off instructions and placeholder-ID safeguards.

Commands

Refresh telemetry snapshots first:

pnpm monitoring:telemetry:sync

For manual public evidence refreshes, prefer the public helper so local

`.env.local` values cannot redirect uptime probes to `localhost`:

pnpm monitoring:telemetry:public

Generate evidence report (always writes artifact):

pnpm era-c:exit-gate:evidence

Generate evidence from a freshly probed public uptime snapshot:

pnpm era-c:exit-gate:public

Generate the long-term accumulation runway report:

pnpm longterm:evidence

Generate the read-only maintenance report for failed or incomplete windows:

pnpm longterm:evidence:maintenance

For manual public runway refreshes, regenerate the public telemetry and Era C

snapshot first; this command also publishes the maintenance report:

pnpm longterm:evidence:public

Generate the durable ActivityStreams syndication report:

pnpm activity:syndication:evidence

Scan canonical upstream records for real ActivityStreams tombstones:

pnpm activity:tombstone:scan

The tombstone scanner is provider-aware: it reports supported, conditional, skipped, and duplicate upstream targets across the stored corpus, while `Delete` evidence remains strict and is written only after a genuine upstream `404`/`410` appears.

Generate the 10/10 review-goals audit and enforce local gates:

pnpm review:goals:local

Enforce as a hard gate (non-zero exit on any failed check):

pnpm era-c:exit-gate:check

Nightly automation

Workflow:

• `.github/workflows/era-c-exit-gate-evidence.yml`

Nightly job does:

1. Run `pnpm k6:slo` against the deployed target when `METAMUSEUM_EVIDENCE_BASE_URL` and `METAMUSEUM_EVIDENCE_IIIF_TILE_URL` are configured; otherwise run local `pnpm k6:slo:ci` as a fallback.

1. Seed one deployed `/api/ai/query` request when `METAMUSEUM_EVIDENCE_BASE_URL` is configured so cost telemetry can flow into SOTA §26 KPI evidence.

1. Probe the declared `/api/activity` adoption matrix when `METAMUSEUM_ACTIVITY_CONSUMER_IDS` is configured; fall back to one declared consumer when only `METAMUSEUM_ACTIVITY_CONSUMER_ID` is configured.

1. Run `pnpm era-c:exit-gate:public` with production public-read probes.

1. Run `pnpm longterm:evidence` to publish a compact accumulation/runway artifact.

1. Run `pnpm longterm:evidence:maintenance` to publish the read-only repair report that names failed SLO/uptime samples, their age-out dates, clean passing days, daily evidence-ready date, failure-free window date, and remaining external-consumer count.

1. Run `pnpm activity:syndication:evidence` to publish durable consumer/subscription evidence.

1. Run `pnpm activity:tombstone:scan` to keep the real upstream `404`/`410` watch fresh without writing a fake `Delete` row.

1. Run `pnpm review:goals:local` to publish the 10/10 audit artifact and fail only when local commands/docs/tests are incomplete; expected external-evidence blockers remain visible in the artifact.

1. Commit compact rolling evidence inputs back to `main` when the workflow runs on `main`, including the k6 trend, uptime history, latest long-term runway, latest long-term maintenance report, latest exit-gate, latest syndication, latest tombstone scan, latest review-goals, and activity consumer/subscription files.

1. Upload exit-gate, long-term runway/maintenance, activity-syndication, review-goals, performance, activity-adoption, uptime, and KPI artifacts.

The workflow always uploads artifacts so failed checks still produce dated proof.

Only compact rolling inputs and latest summaries are committed; timestamped run

artifacts remain upload artifacts so the repository does not grow by one full

evidence packet per night. The committed allowlist includes the k6 trend,

public-read uptime history, latest runway, latest maintenance JSON/Markdown,

latest Era C exit-gate packet, latest review-goals packet, latest

ActivityStreams evidence, latest tombstone scan, and activity

consumer/subscription stores. The

workflow has `contents: write`, a single concurrency group, a main-branch guard,

and no push trigger, so the bot commit can accumulate the 30-day windows

without creating a workflow loop.

Deployed-target workflow variables:

• `METAMUSEUM_EVIDENCE_BASE_URL` — deployed app URL used as `BASE_URL` for k6, AI query seeding, and adoption proof.
• `METAMUSEUM_EVIDENCE_IIIF_TILE_URL` — CDN-backed IIIF tile used as `IIIF_TILE_URL` for the tile-serving p95 check.
• `METAMUSEUM_EVIDENCE_SPARQL_URL` — whitelisted SPARQL endpoint used as `SPARQL_URL` by `pnpm k6:slo`.
• `METAMUSEUM_EVIDENCE_SPARQL_QUERY` — optional whitelisted SPARQL query override for `pnpm k6:slo`.
• `METAMUSEUM_EVIDENCE_AI_QUERY` — optional representative query for the deployed `/api/ai/query` telemetry seed.
• `METAMUSEUM_ACTIVITY_CONSUMER_IDS` — comma-separated partner-owned declared consumer IDs for the full adoption matrix proof.
• `METAMUSEUM_ACTIVITY_CONSUMER_ID` — partner-owned declared consumer ID for the single-consumer adoption proof fallback.

Telemetry source configuration

Primary uptime source (recommended):

• `METAMUSEUM_UPTIME_PROMETHEUS_URL`
• `METAMUSEUM_UPTIME_PROMQL_AVAILABILITY`
• `METAMUSEUM_UPTIME_PROMQL_SAMPLE_COUNT`

GitHub Actions wiring:

• secret: `METAMUSEUM_UPTIME_PROMETHEUS_URL`
• repo/org var: `METAMUSEUM_UPTIME_PROMQL_AVAILABILITY`
• repo/org var: `METAMUSEUM_UPTIME_PROMQL_SAMPLE_COUNT`

Fallback uptime source:

• `METAMUSEUM_PUBLIC_READ_BASE_URL`
• `METAMUSEUM_UPTIME_PROBE_PATHS`
• `METAMUSEUM_UPTIME_PROBE_TIMEOUT_MS`

Manual public probe helper:

• `pnpm monitoring:telemetry:public` runs `scripts/monitoring-telemetry-sync.ts --probe-base-url=https://www.metamuseum.org --require-probe`.
• `pnpm era-c:exit-gate:public` runs the same forced public telemetry refresh before writing the Era C exit-gate artifact.

Window controls:

• `METAMUSEUM_UPTIME_WINDOW_DAYS` (default `30`)
• `METAMUSEUM_UPTIME_HISTORY_RETENTION_DAYS` (default `45`)