SOTA §26 KPI Evidence Input

This runbook defines the optional aggregate KPI evidence file consumed by `pnpm monitoring:telemetry:sync`.

Purpose

`monitoring/kpis.json` stays auto-generated. When `monitoring/kpi-evidence.json` is present, telemetry sync uses it for production-like SOTA §26 inputs that cannot be inferred reliably from local seed records:

• record enrichment share (`dataQualityEnrichedShare`)
• reconciliation auto-approval rate (`reconciliationAutoApproveRate`)
• reviewed reconciliation precision (`reconciliationPrecisionReviewed`)

If the evidence file is missing or invalid, telemetry sync falls back to record-state heuristics and does not manufacture passing values.

The generated file also includes `diagnostics` and `acceptanceRows`.

Diagnostics preview the three KPI metric values against the Era C thresholds,

name the next evidence needed for any gaps, and include a production-only

`capturePlan` with the exact fields and commands operators need for the next

export. The enrichment evidence needs also include a capped

`sampleRecordGaps[]` repair sample with record IDs, labels, current unique

recognized authority reference counts, and remaining authority URI gaps. Full

authority URLs and compact IDs such as `aat:...`, `ulan:...`, and `tgn:...`

are counted as references when they resolve to a recognized authority family.

`diagnostics.handoffSummary`

condenses that packet into ready-row

counts, the remaining metric blocker count, and the next command/evidence pair

an operator should run first. Acceptance rows do not replace the Era C KPI

threshold check; they only prove that the aggregate KPI inputs are

production-shaped enough to hand to telemetry. Strict readiness still requires

`pnpm era-c:exit-gate:public` to consume the export and pass the configured

thresholds.

Default Path

• Default: `monitoring/kpi-evidence.json`
• Override: `METAMUSEUM_KPI_EVIDENCE_PATH`

JSON Shape

{
  "version": 1,
  "updatedAt": "2026-06-10T00:00:00.000Z",
  "records": {
    "totalRecords": 10000,
    "enrichedRecords": 8300,
    "source": "warehouse materialized records"
  },
  "reconciliation": {
    "scoreDistribution": {
      "autoLink": 920,
      "weeklyDigest": 50,
      "humanReview": 30,
      "dropCandidate": 80
    },
    "reviewedAutoLinkedCandidates": 200,
    "acceptedAutoLinkedCandidates": 190,
    "reviewedPrecisionSource": "production curator review export",
    "source": "production reconciliation export"
  },
  "diagnostics": {
    "metricsPreview": {
      "dataQualityEnrichedShare": 0.83,
      "reconciliationAutoApproveRate": 0.92,
      "reconciliationPrecisionReviewed": 0.95
    },
    "evidenceNeeds": {
      "enrichment": {
        "requiredEnrichedRecords": 8300,
        "remainingEnrichedRecords": 0,
        "totalRecords": 10000,
        "enrichedRecords": 8300,
        "sampleRecordGaps": []
      }
    },
    "blockers": [],
    "capturePlan": {
      "productionOnly": true,
      "rows": [
        {
          "id": "record-enrichment-capture",
          "status": "ready"
        },
        {
          "id": "reconciliation-distribution-capture",
          "status": "ready"
        },
        {
          "id": "reviewed-precision-capture",
          "status": "ready"
        },
        {
          "id": "strict-era-c-refresh",
          "status": "ready"
        }
      ]
    },
    "handoffSummary": {
      "status": "ready-to-refresh",
      "readyAcceptanceRows": 5,
      "totalAcceptanceRows": 5,
      "readyCaptureRows": 4,
      "totalCaptureRows": 4,
      "metricBlockerCount": 0,
      "strictRefreshBlocked": false,
      "nextCommand": "pnpm monitoring:telemetry:public && pnpm era-c:exit-gate:public",
      "nextEvidenceNeeded": "Run public telemetry and Era C exit-gate evidence to consume the KPI export.",
      "nextMetricCommand": null,
      "nextMetricEvidenceNeeded": null
    }
  },
  "acceptanceRows": [
    {
      "id": "production-record-export",
      "status": "ready"
    },
    {
      "id": "reviewed-precision-counts",
      "status": "ready"
    }
  ]
}

Validation Rules

• `records.totalRecords` must be a positive integer.
• `records.enrichedRecords` must be an integer from `0` to `totalRecords`.
• Reconciliation considered candidates are `autoLink + weeklyDigest + humanReview`, or explicit `consideredCandidates`.
• Auto-linked candidates are `scoreDistribution.autoLink`, `autoLinkedCandidates`, or `autoApprovedCandidates`.
• Reviewed precision requires `acceptedAutoLinkedCandidates <= reviewedAutoLinkedCandidates <= autoLinkedCandidates`.
• Invalid sections are ignored independently so one bad source cannot create false-green KPI evidence.
• Strict KPI input acceptance requires production-labeled record and reconciliation sources, non-empty record and candidate denominators, and reviewed/accepted auto-link counts supplied together.
• Diagnostics compare the previewed KPI values against Era C defaults: `dataQualityEnrichedShare >= 0.8`, `reconciliationAutoApproveRate >= 0.9`, and `reconciliationPrecisionReviewed >= 0.95`.

Acceptance Rows

`pnpm monitoring:kpi-evidence:production` writes these handoff rows:

• `production-record-export`: production record export source plus non-zero record count.
• `record-enrichment-denominator`: non-zero denominator for `dataQualityEnrichedShare`.
• `production-reconciliation-export`: production reconciliation source plus real candidate decisions.
• `reviewed-precision-counts`: reviewed and accepted auto-linked candidate counts supplied together from real review evidence.
• `strict-era-c-refresh`: ready only when the previous KPI input rows are production-shaped; the final pass/fail still happens in `pnpm era-c:exit-gate:public`.

Diagnostics

`diagnostics.metricsPreview` mirrors the KPI values telemetry will derive from

the aggregate evidence file. `diagnostics.blockers[]` is empty only when those

previews meet the Era C KPI targets. Blockers currently use these IDs:

• `record-enrichment-gap`: production records do not yet meet the 0.8 enriched-share target.
• `reconciliation-decision-gap`: production reconciliation decisions are missing or below the 0.9 auto-approve target.
• `reviewed-precision-gap`: reviewed/accepted auto-link counts are missing or below the 0.95 precision target.
• `strict-era-c-refresh`: at least one KPI blocker remains before the public Era C refresh can pass.

`diagnostics.capturePlan.rows[]` turns those gaps into a capture checklist:

• `record-enrichment-capture`: production record id plus unique recognized authority URI/equivalent count export; current/local/demo/fixture/sample/test source labels do not count.
• `reconciliation-distribution-capture`: production `auto-link`, `weekly-digest`, `human-review`, and `drop-candidate` counts; dropped candidates are tracked but excluded from the considered-candidate denominator.
• `reviewed-precision-capture`: reviewed and accepted auto-link counts plus the production curator review source.
• `strict-era-c-refresh`: refresh `monitoring/kpis.json` and `artifacts/exit-gate/era-c-exit-gate-latest.json` only after the capture rows are ready.

`diagnostics.evidenceNeeds.enrichment.sampleRecordGaps[]` is a repair aid, not

a separate strict gate. It lists up to 10 under-enriched records sorted by the

largest remaining unique authority reference gap first. Regenerate it with

`pnpm monitoring:kpi-evidence:production` before treating the rows as production

cleanup evidence.

`diagnostics.handoffSummary` is the compact operator entry point. Its

`metricBlockerCount` counts only the KPI metric gaps, not the synthetic

`strict-era-c-refresh` blocker, so the handoff can say exactly how many input

rows still need real production evidence before the strict refresh can succeed.

Use `nextCommand` and `nextEvidenceNeeded` as the first remediation step when

the summary status is `needs-production-evidence`. When production source shape

and metric thresholds are different problems, `nextMetricCommand` and

`nextMetricEvidenceNeeded` point at the first KPI threshold blocker directly

while `nextCommand` remains the strict source-shape handoff.

Workflow

1. Export aggregate counts from production record materialization and curator review systems.

1. Write the aggregate JSON to `monitoring/kpi-evidence.json` or set `METAMUSEUM_KPI_EVIDENCE_PATH`.

1. Inspect `diagnostics.handoffSummary`; if it says `needs-production-evidence`, run its `nextCommand` and attach the named `nextEvidenceNeeded` first. If `nextMetricCommand` is present, capture that metric evidence in the same production export whenever possible.

1. Confirm every KPI `acceptanceRows[]` entry is `ready`.

1. Confirm every `diagnostics.capturePlan.rows[]` entry is `ready` and `diagnostics.blockers[]` is empty.

1. Run `pnpm monitoring:telemetry:public`.

1. Run `pnpm era-c:exit-gate:public`.

1. Inspect `monitoring/kpis.json` sources; evidence-backed sections cite the KPI evidence file and source label.

`pnpm monitoring:kpi-evidence:production` loads the same repo `.env*` files as

the launch evidence scripts before enforcing Postgres storage. Set

`METAMUSEUM_SKIP_ENV_FILES=1` only when supplying explicit shell/CI variables.