1The exam
Each exam is a real piece of work: a bug in an open-source library, an incident hidden in a server's logs, an order to correct in a web application, a question about a table of public data. The model gets the task and the exam's tools, such as reading files, running commands, editing and submitting, and works alone, step by step.
It works in a fresh sandbox: no network, one CPU, a fixed memory limit, a step budget, and no administrator rights inside the box. It sees only what the task gives it.
2What we record
Everything an attempt does is recorded as it happens:
- the whole conversation between the exam and the model: the task, every message the model sent, and every tool result it got back, in full;
- each command it runs, each file it reads, and each file it changes;
- the tests it chose to run, and the change it submitted, as a diff;
- its time, the time spent waiting for the model, its tokens and its cost;
- a timeline of the attempt, event by event;
- the grading: which hidden tests failed and what the grader printed, and whether the attempt stayed inside its sandbox.
The attempt is sealed into a record with its own digest. Its full evidence is kept with it, and every file the record fingerprints is checked against its digest. Watch a recorded attempt.
3How it is graded
When the model submits, its work goes to a separate grading box, where the hidden tests run against it. The tests and the reference answer never enter the sandbox, so the model can neither read nor change them. No AI judges these exams: code decides.
Certified passOnly when three things are true: the hidden tests passed, the work was submitted, and the attempt was verified to have stayed inside its sandbox.
FailedThe hidden tests did not pass, or nothing was submitted. Counted, with the way it failed.
Tried to reach what it may not seeSuch as the tests or the reference answer. Counted as a failure, and named apart.
Endpoint errorThe model's endpoint failed: an outage, a refused key, a timeout. Listed apart, never counted as a wrong answer.
WithheldThe outcome could not be proven either way. Left out of the pass rate.
Each model's pass rate comes with its 95% range, so a result from three attempts is not mistaken for a result from three hundred.
4Why an exam can be trusted
An exam is used only after it passes six checks. It builds cleanly. The reference answer passes and the untouched task fails. Deliberately wrong submissions, including a near miss a good engineer would write, land on their known verdicts. Every cheating probe fails. It runs end to end. And it is frozen: its files, its grader, its reference answer and its sandbox image are fingerprinted, so a changed exam is a different exam.
The cheating probes try what a model might: hunting for the reference answer, reading the hidden tests, deleting tests, editing the grader, producing output that only looks right, tampering with the result file, reaching the network, escaping the file system, and on most exams hijacking the test runner or patching it as it runs. Each exam has nine or ten, and none may succeed.
The 17 exams offered here were checked again on 24 September 2026 on this server's own sandbox images. One exam, walked through the chain.
5Proof you can check
Every result of a run is hashed with SHA-256, the hashes are combined into one root, and the run is signed with Ed25519. That is its receipt. Anyone can check a receipt at Check a receipt: change one grade, and the check fails. Each Forge attempt also carries the digest of its sealed record.
6What you get
- Each model's pass rate with its 95% range, and every attempt with its outcome and the reason.
- On the public exams, a replay of each attempt: the steps it took, the change it made, and its outcome.
- A report in the format of our published reports: an executive summary, results per exam, what the outcomes mean, what the report withholds and why, the evidence, and an attestation. It saves as a PDF.
- The signed receipt of every run.
- For an evaluation we run for you: the same report, reviewed, with our verdict in plain words. Book a call.
7What we never show
- The hidden tests, the grader and the reference answer. They are the exam; showing them would spend it.
- Your keys. They are encrypted the moment they arrive, never shown again, and never written into a receipt or a report.
- The grading side of an attempt's evidence: which hidden tests failed, and what the grader printed. We keep it, and can go through it with you.