AI exam result · CURRENT
Five exams, five families — ten rollouts per lane
Observed outcomes from sealed evaluation records. Failed attempts and non-model errors remain visible.
Outcomes by AI exam
| AI exam | AI agent | Passed / attempts | Technical errors | Withheld | Invalid |
|---|---|---|---|---|---|
| Streaming items must equal what the full parse would have built | cli/codex | 10/10 | 0 | 0 | 0 |
| Answering questions from a document collection with outdated and distractor sources, and refusing when the evidence is insufficient | cli/codex | 10/10 | 0 | 0 | 0 |
| Updating a cross-session fact: answer from the newer authoritative roster, not stale carried memory | cli/codex | 10/10 | 0 | 0 | 0 |
| Harness recovery under a faulted tool loop | cli/codex | 10/10 | 0 | 0 | 0 |
| Correct one order in an admin application, through a real browser | cli/codex | 10/10 | 0 | 0 | 0 |
| Streaming items must equal what the full parse would have built | cli/claude-sonnet | 9/10 | 0 | 0 | 0 |
| Answering questions from a document collection with outdated and distractor sources, and refusing when the evidence is insufficient | cli/claude-sonnet | 1/10 | 0 | 0 | 0 |
| Updating a cross-session fact: answer from the newer authoritative roster, not stale carried memory | cli/claude-sonnet | 10/10 | 0 | 0 | 0 |
| Harness recovery under a faulted tool loop | cli/claude-sonnet | 10/10 | 0 | 0 | 0 |
| Correct one order in an admin application, through a real browser | cli/claude-sonnet | 10/10 | 0 | 0 | 0 |
| Streaming items must equal what the full parse would have built | codestral-latest | 0/10 | 0 | 0 | 0 |
| Answering questions from a document collection with outdated and distractor sources, and refusing when the evidence is insufficient | codestral-latest | 0/9 + 1 lane error | 1 | 0 | 0 |
| Updating a cross-session fact: answer from the newer authoritative roster, not stale carried memory | codestral-latest | 2/10 | 0 | 0 | 0 |
| Harness recovery under a faulted tool loop | codestral-latest | 0/10 | 0 | 0 | 0 |
| Correct one order in an admin application, through a real browser | codestral-latest | 1/9 + 1 lane error | 1 | 0 | 0 |
Technical evidence and detailed report
Detailed technical report (PDF) · Results and record digests (JSON) · Attempt counts and commitments (JSON)
These optional files provide the full audit detail behind this result.
fc-d89e429d2781