Impact summary
`ai-agent-machine-evidence.ts` now owns fail-closed projection of verified public
trial receipts into the canonical AI-value audit. Calliope v10 and Clio evidence-
hook v6 are replayed against their exact fixtures and existing strict parsers
before the audit reports observations, mean model cost per system task, mean
latency, safety failures, and aggregate machine-gate state. Both scorecards move
from 0% to 60% completeness while quality lift and reviewer effort remain null;
their grade stays `unproven`. Runtime behavior, provider spend, deployment, and
publication authority are unchanged.
Ranked coupling and failure findings
- High — audit/readiness verification duplication: readiness and audit now
invoke the same strict parsers through separate orchestration paths; a future
receipt-version change could update one path but not the other.
- High — aggregate latency cannot be independently recomputed: privacy-safe
receipts retain min/mean/max but not individual timings, so the parser proves
ordering and ceilings rather than recalculating the mean from raw observations.
- Medium — 60% is measurement completeness, not value: machine gates can all
pass while blinded quality or reviewer effort fails; UI and docs must preserve
that distinction.
Actions
| Owner | Action and acceptance criteria | Validation |
|---|---|---|
| AI Reliability | Extract a versioned shared receipt registry used by readiness and audit; adding or superseding a receipt changes one registration only | Focused readiness, machine-evidence, and audit tests pass with no duplicated canonical path constants |
| AI Evaluation | Collect two genuine packet-bound responses for Calliope and Clio; populate quality and effort only through the strict aggregator | Run `pnpm audit:ai-agent-value:blind-review:aggregate` on each exact private packet/key and two distinct complete responses |
| Trust | Keep scorecard completeness visibly separate from proven benefit; no 60% machine row may change grade or authority | `node --import tsx --test --test-concurrency=1 tests/services/ai-agent-machine-evidence.test.ts tests/services/ai-agent-value-audit.test.ts` |
| Observability | If privacy review permits, add a bounded latency histogram or digest so aggregate mean can be replayed without retaining prompts, outputs, or reviewer data | Add a failing receipt-mutation test, update both trial parsers, and rerun provider-free replay tests |
Next-cycle hypothesis
A single versioned evidence registry can remove audit/readiness receipt-path drift
without weakening strict parser replay or changing any grade. Falsify by mutating
a fixture hash, cost, or registered path and running the focused readiness plus
machine-scorecard suites; every mutation must fail or remove evidence credit.