01 / AI EXAMS
Give AI agents
work worth testing.
Run defined tasks across software, browser, knowledge and other task families. Follow the actions, inspect the submission and understand the final verdict.
Explore AI examsAI exams · Multimodal annotation
Know what your AI
can actually do.
Build reliable exams for AI agents. Run real tasks, grade the work independently, and inspect the evidence behind each result.
Software, browser actions, knowledge, memory and multimodal annotation—all part of Forge, with task-specific grading and a shared standard of evidence.
This site publishes evidence. The working tools run locally in VVDex Studio.
Inside ForgeTASK → EVIDENCE
Real work.
Independent judgment.
A software fix. A browser task. A grounded answer. Every exam needs a reliable way to tell whether the work succeeded.
- 01Define the taskGive the agent a workspace, tools and a clear objective.
- 02Test the testCheck reference work, near misses and attempts to manipulate grading.
- 03Run the AI agentKeep the task separate from its private grader and reference.
- 04Inspect the resultReview actions, outcomes, failures and the evidence behind them.
AI exams
Real tasks. Certified evaluation environments. Capabilities
Exact exam measurements by model lane. Multimodal annotation
Video, image, text, structured data and audio. Model results
Observed outcomes and their limits.
Real tasks. Certified evaluation environments. Capabilities
Exact exam measurements by model lane. Multimodal annotation
Video, image, text, structured data and audio. Model results
Observed outcomes and their limits.
What you can do
Two kinds of work.
The same standard of proof.
Evaluate an agent’s ability to complete a task, or check the quality of an annotation. Keep the result connected to the work that produced it.
Give AI agents
02 / ANNOTATION QUALITY
Turn observations
into checkable data.
Inspect source media, label events and objects, validate submissions and compare annotation sets. Grade the result against a private reference.
Explore annotation qualityExplore the work
A broad range of tasks.
A clear route into each.
Start with a published AI exam or an annotation environment. Each explains the task, what is checked, and which evidence is available.
AI exams
Overview ↗Observed model results
Two campaigns.
The work behind the numbers.
These campaigns cover different fixed tasks and run conditions. Read them separately; neither is a universal model ranking.
AI exam campaign · September 1–23 MODEL LANES
Five kinds of work.
150 recorded attempts.
Software, grounded knowledge, memory, fault recovery and browser actions. Ten attempts per task for each model lane.
Recorded lanePassed / graded
Codex CLI50 / 50
Claude Sonnet CLI40 / 50
Codestral3 / 48
Two Codestral attempts ended in lane errors and are outside the graded denominator. These are the recorded September 1–2 results.
Inspect the AI exam campaignAnnotation campaign · September 6GPT-5.5 · CLI
Annotation quality.
Four fixed tasks.
Robot video
Passed
Failed
Other
Mean quality · 0–1
Annotation campaign · September 6
Four fixed tasks.
Forty retained attempts.
GPT-5.5 was evaluated on image, sampled-video, text and structured-data annotation. Audio was not evaluated.
Read the published report for task-level passes, failures, quality measurements and input limitations.
Read the annotation campaign →Read the evidence
A passed check means
something specific.
A certified environment is not a successful model run. Forge keeps those two claims separate, so you can see exactly what was verified.
Is the test trustworthy?
Environment certification checks that reference work passes, incorrect work fails and common attempts to manipulate grading are blocked.
Read certification evidenceHow did the model perform?
Campaign reports show observed attempts, passes, failures and limitations. Results belong to the specific model, task and run—not every possible use.
Read model resultsGo beyond the headline. Published reports link outcomes to recorded evidence. Where record bytes are public, you can verify their digest in your browser.
Verify evidence →VVDex Studio
The place to do the work.
Forge’s working tools run in the local Studio app: inspect an exam, run an AI agent, review source media and annotations, and examine independently graded results.
- Inspect the task
- Run the work
- Review the submission
- Grade independently
- Inspect the evidence
Built for work that can be checked.Understand the methodology
02
The certification chain
One real exam, walked through the chain that certified it. Select a stage; every string in its receipt was written by the machine and is reproduced character for character — A the Docker promotion receipt, B the factory log’s verbatim subprocess walk, C that wave’s freeze table.
Seed
okThe exam is built into a fresh workspace inside a network-none container.
"seed": { "ok": true }Afamily=public_oss_swe (active); entry=pytest; classification=public-ossB
Gold proof
isolatedUpstream’s own merged fix must pass the hidden grader; the untouched tree must fail it. Both, or the exam is worthless in one direction or the other.
"gold": { "ok": true, "promotionEligible": true, "isolated": true }Aisolated=True, gradeWithoutGold.ok=False, gradeWithGold.ok=True, leaks=[], graderVisibleToAgent=FalseB
Grader controls
distinguishedSeven deliberately wrong submissions, each of which has to land on a known verdict. The lookalike is the one that matters: a near-miss a competent engineer would actually write.
"grader": { "ok": true, "promotionEligible": true, "isolated": true, "controlsDistinguished": true }Auntouched F / gold P / incomplete F / lookalike F / plantedHook:conftest.py F / plantedReward F / goldAfterTamperAttempts PBhostGraderUnchanged=True, plantedRewardIgnored=True, plantedHooksIgnored=TrueB
Attack suite
no trivial exploitTen scripted probes go after the grade instead of the task — rewriting tests, planting hooks, reaching for the answer key. The reference solution has to still pass afterwards.
"attack": { "ok": true, "promotionEligible": true, "trivialExploits": [], "passedForPromotion": true, "goldStillPassesAfterSuite": true }A10 probes, trivialExploits=[], goldStillPassesAfterSuite=TrueB
Evaluation
eligibleThe exam is run end to end in the same container it was certified in, and the model-facing tree is scanned against the publication rule list.
"eval": { "ok": true, "promotionEligible": true }Ascan_publication.py on the model-facing tree plus the published graderB
Promoted
allOkEvery stage passed, so the exam is sealed against the image it was certified with. This is the point at which a score off it becomes quotable — against this freeze and no other.
"allOk": trueAfrozen agent files17CtreeSha256 (16)9999fb81b48354b3Cfreeze hash (16)e036a66feefc4c98Chidden-test lines190C
The mark is not on the exam. The mark is on the fact that the chain above ran and
every station answered. The freeze figures here are from the subprocess wave — it
records imageDigest: null honestly, and the Docker pass re-freezes.
03
The sealed box
gold never mounts in the agent box · network=none ·
graderVisibleToAgent=False
Agent CLIs (Claude Code, Codex) run on the host, not in the agent container: their containment is audited after each run from the CLI's own session store, and a lane that cannot be verified is withheld. Full rule: methodology · from a grader verdict to a certified result.
04
Families
Annotation · five certified fixtures
One family for reviewing and labeling multimodal data, with explicit task guidelines, ambiguity handling and deterministic quality checks. These fixtures extend Forge's coverage; their certification results are separate from the historical model campaigns.
Robotic video · Image objects · Text labeling · Structured data · Audio events
Public open-source SWE
Terminal operations
Algorithms, graded by property
Code quality
Security & permission boundaries
Tool use & MCP
Retrieval & grounding
Cross-session memory
Structured document work
Customer support
Data & SQL
Internal VVDex SWE
Harness evaluation
Browser & computer use
05
Ten probes, zero exploits
10 probes, trivialExploits=[], goldStillPassesAfterSuite=True
B
VVDex Studio · local workspace
Inspect the source.
Stand behind the result.
The working environment for Forge: review source media, prepare annotations, run AI evaluations and inspect the grade.
Studio runs locally. This website publishes selected evidence; it does not offer a hosted annotation editor or public Studio download.
AI exams
From task to recorded outcome.
Inspect the exam
Review the task, available tools, constraints and certification evidence before interpreting a model result.
Set the run conditions
Choose the model lane and record the intended attempts. Keep those conditions visible in the resulting evidence.
Run the task
Let the AI agent work in the exam environment while the private grader and reference remain separate.
Read the outcomes
Inspect passes, model failures and execution errors separately. Follow the recorded actions behind a result.
Compare and report
Compare results under their recorded conditions and preserve the evidence that supports the report.
Annotation
From source to a checked submission.
Inspect
Open the environment and review its source. Play or seek supported video and audio, inspect images, or read text and structured records.
Annotate
Prepare a structured submission with the labels, spans, geometry or time intervals required by the task.
Validate
Check the submission’s shape and consistency. Review validation feedback before submitting it for a grade.
Grade
Submit for independent grading against a private reference. Read the pass verdict and the applicable quality measurements separately.
Compare
Compare annotation sets and inspect disagreements. Agreement between submissions is a different question from correctness against the reference.
Discuss an evaluation or annotation workflow.
Get in touch07
Run your model against it
You bring the endpoint and the key; the exams and the grading are ours. The key is read from a file into the job, never from the environment and never from the command line, and it is scrubbed from every artifact a run produces — including the error text a rejecting gateway sends back. You get a written report and a durable record that attests to its own bytes.
What this is not
Task-specific results — published model campaign scores belong to their named exams and revisions; fixture certification alone says nothing about a model’s performance. Not a public exam list — zero listed, fail-closed, cleared one at a time. Not contamination-free — where a public fix pre-dates a model’s cutoff, that is recorded on the exam, not averaged into a headline. Not self-service — no sign-up, no dashboard, no billing.
Public by design · kept by design
Public: the methodology in full, the family list and counts, chosen campaign reports, the ledger verification recipe, and — once cleared — a listed exam’s title, family and badge. Kept: the factory, the scouts, the engine, the evidence chain behind a certification, the run history, and every customer’s results, which belong to the customer.