Production AI research evaluation

Meta Museum has two deliberately separate AI evaluation lanes. `pnpm ai:eval:gate` runs the 120-prompt golden dataset against local seed records to catch regressions. It is valuable technical evidence, but it cannot satisfy the A+ production requirement. `pnpm research:ai-production-eval` imports a separately frozen label manifest and genuine production trace bundle; it performs no model calls and fails closed when either is absent.

The optional `/research/evaluate-ai` collection candidate is documented in

`ai-question-collection.md`(ai-question-collection.md). It stays no-index and

renders no form until an exact contract-digest approval and separate runtime

switch both pass. API enforcement is independent of the page.

Freeze labels before scoring outputs

The dataset manifest must contain at least 50 unique production case IDs and unique question SHA-256 digests. Each case has a predeclared `shouldRefuse` label and an ordered list of relevant claim IDs. At least ten cases must be answerable and ten must require refusal. An independent label approver records a pseudonymous code, approval time after the freeze time, public HTTPS evidence reference, receipt digest, and independence from the later run operator. The freeze and completed approval must both strictly predate every production trace. A trace observed before or at either boundary invalidates the bundle, preventing expected claims or refusal labels from being fitted after model outputs are visible.

Every URL used as production evidence follows one strict public-host policy. Dataset

and question-export sources, approval records, run evidence, emitted citations, and

review receipts reject localhost, private or link-local IP ranges, `.local` and

`.invalid` names, and example/test/sandbox/demo/fixture hosts. A syntactically valid

HTTPS URL is not attributable production evidence by itself.

The production bundle must contain every frozen case exactly once and bind itself to the manifest's file SHA-256. Reordering or changing expected claims after observing an output fails alignment.

The bundle also carries the exact current Research Commons release digest. The

assembly command derives that digest from the canonical release manifest, the

evaluation command independently recomputes it, and the A+ scorer requires the

projected metric digest to equal the open-research digest being scored. A valid

evaluation from an older implementation cannot be combined with a newer release.

The binding digest is calculated from the canonical parsed manifest rather than

its raw whitespace. Pretty-printing an unchanged approved JSON file therefore

does not break the receipt, while any changed value does.

Required trace evidence

Every row retains only pseudonymous or hashed evidence:

• unique case ID and trace SHA-256;
• fresh production observation time, model version, and prompt version;
• actual refusal state and emitted stable claim IDs;
• each claim's HTTPS citation and an independent human supported/unsupported decision;
• measured latency;
• non-negative cost from a provider receipt or production instrumentation;
• independent reviewer code, review time, HTTPS evidence, and unique receipt digest.

Names, emails, phone numbers, addresses, contacts, and IP fields are rejected. A refused output cannot emit claims or citations, and every non-refused claim needs at least one reviewed citation. One bundle must use one model and prompt revision, at least two human reviewers, at least 25 reviewed citations, and traces no older than 35 days.

Metrics and thresholds

• Precision = accepted expected emitted claims / all emitted claims; minimum 0.90.
• Recall = emitted expected claims / all expected claims; minimum 0.90.
• Citation accuracy = human-supported citations / reviewed citations; minimum 0.95.
• Refusal accuracy = cases whose observed refusal state matches the frozen label / all cases; minimum 0.95.
• P95 latency uses the nearest-rank method and must be at most 10,000 ms.
• Mean cost uses every row's provider-receipt or instrumented production cost.

The evaluator calculates metrics from row-level evidence. Imported aggregate assertions are ignored. Passing metrics still do not establish scholarly novelty, approve a research finding, or authorize publication, outreach, email, payments, or a model run.

Operator sequence

1. Run `pnpm research:ai-question-collection-readiness`. If its status is

`awaiting-human-approval`, review the exact contract and complete the blank

approval outside source control; rerun with

`-- --approval=<approved-collection-contract.json>`. A

`ready-for-runtime-activation` packet still does not authorize deployment or

enable collection. Only after separate authorized activation and genuine

explicitly consented traffic exists should log attestation begin.

1. Export the operator-approved production `ai-query-log.json` and create a

separate version-1 attestation containing `environment: "production"`, an

attributable non-synthetic HTTPS `source`, the exact raw `logSha256`, a fresh

`exportedAt`, a pseudonymous `operatorCode`, and

`questionCollectionApproved: true`. Run

`pnpm research:ai-question-export -- --input=<ai-query-log.json> --attestation=<attestation.json> --output=<private-directory>`.

This explicit local transformer filters rows older than 35 days, rejects

unsuccessful, malformed, or contact-bearing rows, deduplicates normalized

questions, requires the exact `research-ai-evaluation-v1` consent envelope,

and replaces raw query IDs with deterministic pseudonyms. It

writes nothing unless 50 eligible questions remain and never treats its own

filtering as proof that the log is genuinely production data.

Before completing that attestation, run

`pnpm research:ai-question-attestation -- --input=<ai-query-log.json>`.

It emits only the exact raw receipt, aggregate exclusion counts, and an

attestation template whose environment, source, operator, and approval fields

remain null. The current local log has 2,497 legacy rows and zero eligible

questions because every row is malformed for the exact current evaluation-export contract;

this is local readiness evidence, not proof that any row is production traffic.

The candidate retains one privacy-safe outcome code per input row—eligible or

the applicable exclusion category—but no question, raw ID, timestamp, or contact

value. Verification reconstructs every aggregate count, blocker, readiness state,

blank attestation field, and authority boundary from those outcomes.

1. Use the resulting private export—a version-1 JSON object

containing an attributable HTTPS `source` and `cases` with only `caseId` and

`question`. Run

`pnpm research:ai-manifest-prepare -- --input=<production-question-export.json>`.

The CLI computes and retains the exact raw-file SHA-256 and current

preparation time, but writes only normalized question hashes and blank label

fields. It never echoes or copies question text. Use `--prepared-at=<ISO>`

and `--output=<directory>` for a reproducible controlled run. The checked-in

configuration remains available for scheduled operation when no CLI input

is supplied.

The private status artifact retains a replay projection containing the source

receipt, normalized lengths, question/case hashes, and only valid pseudonymous

case IDs. It never retains question text or an invalid/private-like ID. Verification

reconstructs blockers, candidate count, readiness, and every blank worksheet row,

so outer-digest recomputation cannot forge labeling readiness.

1. Have an independent human inspect the separately retained questions, fill

every refusal and relevant-claim label, freeze the manifest, and record the

approval receipt. Preparation is not approval.

1. Run

`pnpm research:ai-production-run-kit -- --manifest=<approved-manifest.json> --operator=<RUN-CODE>`.

The CLI derives the canonical dataset digest and current preparation time,

then rejects a run operator matching the label approver. Use

`--prepared-at=<ISO>` and `--output=<directory>` for a controlled run;

configuration mode remains available for scheduling. This prepares blank

worksheets only and does not execute a model. The private status artifact

retains the exact approved manifest so verification can reconstruct dataset

alignment, operator separation, chronology, blockers, and every pending row.

The separately exported observation and review worksheets remain sanitized

and contain no question digest, approval reference, or approver code.

1. Complete the generated observation worksheet only from genuine production

instrumentation. Give its cases to at least two independent human reviewers

using the balanced A/B lanes; the generator does not invent their identities

or decisions.

1. Retain the raw log, attestation, manifest, and production export outside source control when they contain restricted evidence; expose only approved HTTPS receipts.

1. Assemble the completed contracts with

`pnpm research:ai-production-evidence -- --manifest=<approved-manifest.json> --observations=<production-observations.json> --reviews=<independent-reviews.json> --evidence-source=<https-receipt-url> --evidence-sha256=<receipt-sha256> --operator=<RUN-CODE>`.

The CLI derives the canonical manifest identity and both local file receipts;

the external production evidence URL and receipt remain mandatory because

they cannot be truthfully inferred. Use `--evaluated-at=<ISO>` and

`--output=<directory>` for a controlled run.

The private assembly artifact retains the sanitized manifest, observations,

reviews, release digest, operator, and external receipt inputs. Verification

reruns every case join, citation decision, reviewer-separation, chronology,

cost, and alignment check at the recorded assembly time before accepting its

status or bundle. It still rejects prompt, question, output, content, contact,

and other direct-identifier fields.

1. Run

`pnpm research:ai-production-eval -- --manifest=<approved-manifest.json> --bundle=<ready-production-bundle.json>`.

The scorer derives both receipts, rechecks exact frozen-label alignment, and

computes all metrics from the 50 retained rows. Use

`--evaluated-at=<ISO>` and `--output=<directory>` for deterministic evidence;

configuration mode remains available for scheduled scoring.

1. Inspect `artifacts/research-a-plus/ai-evaluation/research-ai-production-evaluation-latest.json`.

1. Only when status is `passed`, rerun `pnpm research:a-plus:readiness`; do not hand-enter substitute metrics.

Passed artifacts retain the complete independently approved dataset manifest and

normalized production bundle. Verification rechecks the manifest digest and

case alignment, then recomputes precision, recall, citation and refusal accuracy,

latency, cost, coverage, and reviewer independence from every trace. Rehashing an

edited aggregate metric cannot satisfy the production evaluation requirement.

The preceding run-kit verifier also rebuilds the complete kit from its retained

frozen manifest, canonical dataset digest, operator code, and preparation time.

Rehashed changes to readiness, blockers, case coverage, review lanes, or the

no-execution boundary therefore cannot advance evidence operations.

The assembly verifier likewise rebuilds blocked and ready artifacts from their

retained privacy-safe evidence. Rehashed blocker removal, row truncation, count

edits, or bundle substitution cannot advance the final evaluator.