Meta Museum has two deliberately separate AI evaluation lanes. `pnpm ai:eval:gate` runs the 120-prompt golden dataset against local seed records to catch regressions. It is valuable technical evidence, but it cannot satisfy the A+ production requirement. `pnpm research:ai-production-eval` imports a separately frozen label manifest and genuine production trace bundle; it performs no model calls and fails closed when either is absent.
The optional `/research/evaluate-ai` collection candidate is documented in
`ai-question-collection.md`(ai-question-collection.md). It stays no-index and
renders no form until an exact contract-digest approval and separate runtime
switch both pass. API enforcement is independent of the page.
Freeze labels before scoring outputs
The dataset manifest must contain at least 50 unique production case IDs and unique question SHA-256 digests. Each case has a predeclared `shouldRefuse` label and an ordered list of relevant claim IDs. At least ten cases must be answerable and ten must require refusal. An independent label approver records a pseudonymous code, approval time after the freeze time, public HTTPS evidence reference, receipt digest, and independence from the later run operator. The freeze and completed approval must both strictly predate every production trace. A trace observed before or at either boundary invalidates the bundle, preventing expected claims or refusal labels from being fitted after model outputs are visible.
Every URL used as production evidence follows one strict public-host policy. Dataset
and question-export sources, approval records, run evidence, emitted citations, and
review receipts reject localhost, private or link-local IP ranges, `.local` and
`.invalid` names, and example/test/sandbox/demo/fixture hosts. A syntactically valid
HTTPS URL is not attributable production evidence by itself.
The production bundle must contain every frozen case exactly once and bind itself to the manifest's file SHA-256. Reordering or changing expected claims after observing an output fails alignment.
The bundle also carries the exact current Research Commons release digest. The
assembly command derives that digest from the canonical release manifest, the
evaluation command independently recomputes it, and the A+ scorer requires the
projected metric digest to equal the open-research digest being scored. A valid
evaluation from an older implementation cannot be combined with a newer release.
The binding digest is calculated from the canonical parsed manifest rather than
its raw whitespace. Pretty-printing an unchanged approved JSON file therefore
does not break the receipt, while any changed value does.
Required trace evidence
Every row retains only pseudonymous or hashed evidence:
- unique case ID and trace SHA-256;
- fresh production observation time, model version, and prompt version;
- actual refusal state and emitted stable claim IDs;
- each claim's HTTPS citation and an independent human supported/unsupported decision;
- measured latency;
- non-negative cost from a provider receipt or production instrumentation;
- independent reviewer code, review time, HTTPS evidence, and unique receipt digest.
Names, emails, phone numbers, addresses, contacts, and IP fields are rejected. A refused output cannot emit claims or citations, and every non-refused claim needs at least one reviewed citation. One bundle must use one model and prompt revision, at least two human reviewers, at least 25 reviewed citations, and traces no older than 35 days.
Metrics and thresholds
- Precision = accepted expected emitted claims / all emitted claims; minimum 0.90.
- Recall = emitted expected claims / all expected claims; minimum 0.90.
- Citation accuracy = human-supported citations / reviewed citations; minimum 0.95.
- Refusal accuracy = cases whose observed refusal state matches the frozen label / all cases; minimum 0.95.
- P95 latency uses the nearest-rank method and must be at most 10,000 ms.
- Mean cost uses every row's provider-receipt or instrumented production cost.
The evaluator calculates metrics from row-level evidence. Imported aggregate assertions are ignored. Passing metrics still do not establish scholarly novelty, approve a research finding, or authorize publication, outreach, email, payments, or a model run.
Operator sequence
- Run `pnpm research:ai-question-collection-readiness`. If its status is
`awaiting-human-approval`, review the exact contract and complete the blank
approval outside source control; rerun with
`-- --approval=<approved-collection-contract.json>`. A
`ready-for-runtime-activation` packet still does not authorize deployment or
enable collection. Only after separate authorized activation and genuine
explicitly consented traffic exists should log attestation begin.
- Export the operator-approved production `ai-query-log.json` and create a
separate version-1 attestation containing `environment: "production"`, an
attributable non-synthetic HTTPS `source`, the exact raw `logSha256`, a fresh
`exportedAt`, a pseudonymous `operatorCode`, and
`questionCollectionApproved: true`. Run
`pnpm research:ai-question-export -- --input=<ai-query-log.json> --attestation=<attestation.json> --output=<private-directory>`.
This explicit local transformer filters rows older than 35 days, rejects
unsuccessful, malformed, or contact-bearing rows, deduplicates normalized
questions, requires the exact `research-ai-evaluation-v1` consent envelope,
and replaces raw query IDs with deterministic pseudonyms. It
writes nothing unless 50 eligible questions remain and never treats its own
filtering as proof that the log is genuinely production data.
Before completing that attestation, run
`pnpm research:ai-question-attestation -- --input=<ai-query-log.json>`.
It emits only the exact raw receipt, aggregate exclusion counts, and an
attestation template whose environment, source, operator, and approval fields
remain null. The current local log has 2,497 legacy rows and zero eligible
questions because every row is malformed for the exact current evaluation-export contract;
this is local readiness evidence, not proof that any row is production traffic.
The candidate retains one privacy-safe outcome code per input row—eligible or
the applicable exclusion category—but no question, raw ID, timestamp, or contact
value. Verification reconstructs every aggregate count, blocker, readiness state,
blank attestation field, and authority boundary from those outcomes.
- Use the resulting private export—a version-1 JSON object
containing an attributable HTTPS `source` and `cases` with only `caseId` and
`question`. Run
`pnpm research:ai-manifest-prepare -- --input=<production-question-export.json>`.
The CLI computes and retains the exact raw-file SHA-256 and current
preparation time, but writes only normalized question hashes and blank label
fields. It never echoes or copies question text. Use `--prepared-at=<ISO>`
and `--output=<directory>` for a reproducible controlled run. The checked-in
configuration remains available for scheduled operation when no CLI input
is supplied.
The private status artifact retains a replay projection containing the source
receipt, normalized lengths, question/case hashes, and only valid pseudonymous
case IDs. It never retains question text or an invalid/private-like ID. Verification
reconstructs blockers, candidate count, readiness, and every blank worksheet row,
so outer-digest recomputation cannot forge labeling readiness.
- Have an independent human inspect the separately retained questions, fill
every refusal and relevant-claim label, freeze the manifest, and record the
approval receipt. Preparation is not approval.
- Run
`pnpm research:ai-production-run-kit -- --manifest=<approved-manifest.json> --operator=<RUN-CODE>`.
The CLI derives the canonical dataset digest and current preparation time,
then rejects a run operator matching the label approver. Use
`--prepared-at=<ISO>` and `--output=<directory>` for a controlled run;
configuration mode remains available for scheduling. This prepares blank
worksheets only and does not execute a model. The private status artifact
retains the exact approved manifest so verification can reconstruct dataset
alignment, operator separation, chronology, blockers, and every pending row.
The separately exported observation and review worksheets remain sanitized
and contain no question digest, approval reference, or approver code.
- Complete the generated observation worksheet only from genuine production
instrumentation. Give its cases to at least two independent human reviewers
using the balanced A/B lanes; the generator does not invent their identities
or decisions.
- Retain the raw log, attestation, manifest, and production export outside source control when they contain restricted evidence; expose only approved HTTPS receipts.
- Assemble the completed contracts with
`pnpm research:ai-production-evidence -- --manifest=<approved-manifest.json> --observations=<production-observations.json> --reviews=<independent-reviews.json> --evidence-source=<https-receipt-url> --evidence-sha256=<receipt-sha256> --operator=<RUN-CODE>`.
The CLI derives the canonical manifest identity and both local file receipts;
the external production evidence URL and receipt remain mandatory because
they cannot be truthfully inferred. Use `--evaluated-at=<ISO>` and
`--output=<directory>` for a controlled run.
The private assembly artifact retains the sanitized manifest, observations,
reviews, release digest, operator, and external receipt inputs. Verification
reruns every case join, citation decision, reviewer-separation, chronology,
cost, and alignment check at the recorded assembly time before accepting its
status or bundle. It still rejects prompt, question, output, content, contact,
and other direct-identifier fields.
- Run
`pnpm research:ai-production-eval -- --manifest=<approved-manifest.json> --bundle=<ready-production-bundle.json>`.
The scorer derives both receipts, rechecks exact frozen-label alignment, and
computes all metrics from the 50 retained rows. Use
`--evaluated-at=<ISO>` and `--output=<directory>` for deterministic evidence;
configuration mode remains available for scheduled scoring.
- Inspect `artifacts/research-a-plus/ai-evaluation/research-ai-production-evaluation-latest.json`.
- Only when status is `passed`, rerun `pnpm research:a-plus:readiness`; do not hand-enter substitute metrics.
Passed artifacts retain the complete independently approved dataset manifest and
normalized production bundle. Verification rechecks the manifest digest and
case alignment, then recomputes precision, recall, citation and refusal accuracy,
latency, cost, coverage, and reviewer independence from every trace. Rehashing an
edited aggregate metric cannot satisfy the production evaluation requirement.
The preceding run-kit verifier also rebuilds the complete kit from its retained
frozen manifest, canonical dataset digest, operator code, and preparation time.
Rehashed changes to readiness, blockers, case coverage, review lanes, or the
no-execution boundary therefore cannot advance evidence operations.
The assembly verifier likewise rebuilds blocked and ready artifacts from their
retained privacy-safe evidence. Rehashed blocker removal, row truncation, count
edits, or bundle substitution cannot advance the final evaluator.