Proof · Forge exams
Real tasks in a sandbox, graded by tests the model never sees
Forge exams put a model to work the way an agent works: it reads files, runs commands, edits code and submits. Each attempt runs in its own sandbox, and a hidden grader decides whether the work is right. How an exam is certified before it is offered — the six-stage chain, the sealed box and the attack probes — is on Forge exams; what we already tested is on What we tested.
What happens in one attempt
A fresh sandbox
The exam’s task tree is copied into a new Docker container with no network. The model sees only the task and the tools the exam allows, such as reading files, running a limited number of commands, editing and submitting.
The model works
VVDex sends each step to your model through its connection, with its own key, and passes the model’s tool calls back into the sandbox. The exam sets a step budget; running out of steps ends the attempt.
A hidden grader decides
When the model submits, a grader the model could not read runs in a separate grade box the model never reaches: hidden tests, property checks or a reference comparison. The model’s answer is never compared by another model.
Containment is checked
An attempt only counts as a pass if Forge verified the model stayed inside the task — it did not reach the reference solution, the grader or anything outside its sandbox. A detected breach counts as a fail.
The attempt is sealed
Forge writes a sealed record of the attempt with its own digest. VVDex keeps that record privately — it holds the exam’s internals — and puts its digest into the run’s signed receipt.
How each attempt is counted
- Certified pass — the hidden grader passed, the work was submitted, containment was verified. This is the only pass.
- Fail — the grader did not pass the work, or a containment breach was detected.
- Endpoint error — the model endpoint failed during the attempt. Shown on its own and left out of the pass rate: a broken connection is not a wrong answer.
- Withheld — Forge could not prove the outcome either way. Shown on its own and left out of the pass rate.
The pass rate of a run comes with a 95% Wilson interval. A few attempts give a wide interval; run more attempts for a narrower one.
Why an exam can be trusted
Before an exam is offered, Forge certifies it: the reference solution passes, doing nothing fails, and a set of cheating attempts — reading the grader, editing the tests, submitting without doing the work — is blocked. Each exam’s page in the app shows these checks. Where a check was not recorded, the page says “not recorded” instead of claiming it.
What stays private
The task text is shown to your model during the attempt, as with any exam. The reference solution, the hidden tests, the grader and the sealed record stay on VVDex’s servers. What you see for each attempt is its outcome, the steps and time it used, tokens and cost, and the digest of its sealed record.
Which models can take a Forge exam
Any model that answers the OpenAI chat-completions format with tool calls: a provider’s API, any OpenAI-compatible address, and a model on your own machine through the local connector. An app connected over HTTP cannot take a Forge exam, because the exam needs to call tools step by step.
Limits
A Forge attempt is a full agent session, so it has its own daily allowance, separate from single questions. On the Free plan: 20 attempts a day. The three public exams take up to 3 attempts per model in one run, and a workspace can start 3 such runs a day; every other exam runs on request (For teams). A run needs at least 3 scored attempts per model to be published.
What a Forge result does not tell you
A result says how a model did on this exam, in this sandbox, with this step budget, on that day. It is evidence for this kind of task, not a ranking for every use and not a certificate of fitness.