martinblech/xmltodict · issue 257

A real software-engineering task taken from a public open-source repository: the model is given the codebase and a written issue, and must change the code so that the behaviour described in the issue is fixed.

Source: martinblech/xmltodict · issue · license MIT. The hidden tests were written from the upstream fix; the model never sees them.

The failure mode this exposes is a fix that looks right and is not: a change that satisfies the visible description while breaking, or failing to cover, the contract the hidden tests check. The hidden tests were written from the upstream fix and are never shown to the model.

A written task statement plus the workspace it names. Recorded limits: 40 tool steps; tools list_files, read_file, write_file, edit_file, run_tests, submit. CLI temperature and stochastic seed controls are not established by this record; attempt indices are bookkeeping. The reference solution and the hidden tests are never mounted in the model's workspace.

# Streaming mode silently drops an element's text

Upstream report: https://github.com/martinblech/xmltodict/issues/257

`parse(..., item_depth=N, item_callback=...)` hands each element at depth `N`
to the callback instead of building the whole document. The callback is
supposed to receive what that element would have been worth in the full
document — but the streaming branch in `_DictSAXHandler.endElement` builds the
item itself, and it does less than the non-streaming branch does.

```python
>>> xmltodict.parse('<items><item id="1">hello</item></items>',
...                 item_depth=2, item_callback=show)
{'@id': '1'}                       # 'hello' is gone
>>> xmltodict.parse('<items><item id="1">hello</item></items>')
{'items': {'item': {'@id': '1', '#text': 'hello'}}}
```

The character data is only used when the item has no attributes and no
children; as soon as either is present it is discarded. And on the text-only
path the streamed value skips the handling the non-streaming path applies to
the same text — `strip_whitespace`, `force_cdata`, `cdata_key` and the
`postprocessor`.

Reported as "what happens to free text in the streaming API": text nodes appear
for some elements and are missing for others, with nothing in the API to
explain the difference.

Visible: the model may run the repository's own tests through the exam's run_tests action. Hidden: after submission, a separate test suite is executed in an isolated container (no network) against the submitted workspace. A pass requires every hidden test to pass.

Grading is one step of five. A certified pass additionally requires: verified containment, complete provenance (exam identity and revision fingerprint on the record), no integrity invariant fired (graded tree equals the submitted workspace), and no verdict-bearing harness suspicion. A grader pass alone is never presented as a result.

Every lane in this run was verified contained: wire lanes by construction (the provider sees only what the harness sends), CLI lanes by auditing the CLI's own session store for tool use, answer-key access, cross-session memory and network use after the run.

This run's self-check (API-failure share, identical-failure-across-providers, starvation, containment) fired no rule.

The certification chain for this exam revision is recorded in this package: reference solution passes at rate 1.0, baseline fails at rate 0.0.

Every rollout carries three separate outcomes (grader, integrity, certified), and the record is sealed with a digest over its canonical JSON. Changing any verdict-bearing field breaks verification.

Lane	Grader	Integrity	Certified outcome	Wall clock
cli/claude-sonnet	certified 9 of 10 graded rollouts, 90% (95% CI 60–98%)			10 rollouts
cli/codex	certified 10 of 10 graded rollouts, 100% (95% CI 72–100%)			10 rollouts
codestral-latest	certified 0 of 10 graded rollouts, 0% (95% CI 0–28%)			10 rollouts

• cli/claude-sonnet: 1 rollout(s) submitted an answer the hidden tests rejected; the names of the failing hidden assertions are withheld: they are the grader's own, and the grader stays private whoever owns the source under test
• codestral-latest: 10 rollout(s) submitted an answer the hidden tests rejected; the names of the failing hidden assertions are withheld: they are the grader's own, and the grader stays private whoever owns the source under test

The action sequence the harness recorded for each lane: tool and path only. Diff contents are not shown here on any exam. Public-source ownership may permit upstream source or task material and approved public-source diffs. It never declassifies VVDex grader, hidden-test, reference, answer-key or other evaluation internals.

cli:cli/claude-sonnet · 10 rollout(s) · 4–6 steps · most common shape: read_file ×2 → edit_file → run_tests → submit

• attempt index 0 · 5 steps · submitted_passed · read_file xmltodict.py → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 1 · 5 steps · submitted_passed · read_file xmltodict.py → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 2 · 5 steps · submitted_passed · read_file xmltodict.py → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 3 · 5 steps · submitted_passed · read_file xmltodict.py → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 4 · 5 steps · submitted_passed · read_file xmltodict.py → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 5 · 5 steps · submitted_passed · read_file xmltodict.py → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 6 · 5 steps · submitted_failed · read_file xmltodict.py → read_file xmltodict.py → invalid_tool_call (none) → run_tests → submit
• attempt index 7 · 4 steps · submitted_passed · read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 8 · 5 steps · submitted_passed · read_file xmltodict.py → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 9 · 6 steps · submitted_passed · read_file xmltodict.py → read_file xmltodict.py → invalid_tool_call (none) → edit_file xmltodict.py → run_tests → submit

cli:cli/codex · 10 rollout(s) · 6 steps · most common shape: list_files → read_file ×2 → edit_file → run_tests → submit

• attempt index 0 · 6 steps · submitted_passed · list_files . → read_file ISSUE.md → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 1 · 6 steps · submitted_passed · list_files . → read_file ISSUE.md → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 2 · 6 steps · submitted_passed · list_files . → read_file ISSUE.md → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 3 · 6 steps · submitted_passed · list_files . → read_file ISSUE.md → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 4 · 6 steps · submitted_passed · list_files . → read_file ISSUE.md → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 5 · 6 steps · submitted_passed · list_files . → read_file ISSUE.md → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 6 · 6 steps · submitted_passed · list_files . → read_file ISSUE.md → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 7 · 6 steps · submitted_passed · list_files . → read_file ISSUE.md → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 8 · 6 steps · submitted_passed · list_files . → read_file ISSUE.md → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 9 · 6 steps · submitted_passed · list_files . → read_file ISSUE.md → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit

mistral:codestral-latest · 10 rollout(s) · 5 steps · most common shape: list_files → read_file → edit_file → run_tests → submit

• attempt index 0 · 5 steps · submitted_failed · read_file ISSUE.md → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 1 · 5 steps · submitted_failed · list_files . → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 2 · 5 steps · submitted_failed · list_files . → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 3 · 5 steps · submitted_failed · list_files . → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 4 · 5 steps · submitted_failed · list_files . → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 5 · 5 steps · submitted_failed · list_files . → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 6 · 5 steps · submitted_failed · list_files . → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 7 · 5 steps · submitted_failed · list_files . → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 8 · 5 steps · submitted_failed · read_file ISSUE.md → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit
• attempt index 9 · 5 steps · submitted_failed · list_files . → read_file xmltodict.py → edit_file xmltodict.py → run_tests → submit

Third-party open source: the repository, the bug and the upstream fix are https://github.com/martinblech/xmltodict, issue https://github.com/martinblech/xmltodict/issues/257 under the MIT licence and are reproduced under it. VVDex owns the exam construction around that task — the hidden grader, the hidden tests, the attack probes and the private evidence — and that material is not published. Terms & ownership.

This exam's graded artefact is a change to a public upstream repository under the licence credited above, so the full report publishes a bounded, redacted excerpt of each rollout's own diff. Withheld here and everywhere: the contents of tool results, the hidden tests, the reference solution, host paths and keys.

Rule applied to this exam: public-source disclosure (ownership public_source declared by engine registry, upstream https://github.com/martinblech/xmltodict under MIT; family public_oss_swe): the submitted change is a patch against a public repository, so a redacted excerpt of it is published.

campaign fc-d89e429d2781
evaluation record fr-20260902-6616b188
record digest 138425da59aabc18aa208c339aa41919f387e1ecb80fb725cfed8f7d6ba78713
exam fingerprint aa41977120e6ecdddb5c4fc603e8ab23cb272726a86750119969ff4dfc90e297

The exam's own promotion chain — reference solution passes, baseline fails, grader controls distinguished, attack probes blocked, the runtime and isolation policy it ran under — is published as a machine-readable certification receipt. It states the same exam fingerprint as this page (aa41977120e6ecdddb5c4fc603e8ab23cb272726a86750119969ff4dfc90e297); that fingerprint is the join key, and a receipt stating a different one describes a different exam revision. Fields the recorded evidence does not substantiate read unavailable rather than a number.

This exam's chapter, and the campaign's per-exam matrix that places this result beside the other exams in the same sitting, are in the campaign report.

The record's canonical bytes are not published beside this page: the sealed record carries content the disclosure policy withholds (rollouts[].failedTests), and the canonical bytes are the sealed record verbatim; the record digest is published instead. The digest above is the digest of the sealed record as it exists in the Forge; a holder of that record — the customer who commissioned the run, or an auditor under agreement — recomputes it with shasum -a 256 on the canonical bytes, or with vvdex-env records verify, and reaches the value printed here. Exams whose graded artefact is public source do publish their bytes; the browser verifier works on those.

The engine's own check on a full record: vvdex-env records verify --report <evalId>.md. The campaign document is regenerated from records only: vvdex-env records campaign --workspace <ws>; a document that disagrees with its records fails generation.

• Each lane sat this exam 10 times, on independent seeds. Rates carry 95% Wilson intervals. Where two lanes' intervals overlap, this page does not rank them; where they do not, it says so in the results above.
• Lane errors (provider quota, rate limits) are recorded as lane facts and excluded from capability in both directions.
• Withheld cells are neither passes nor model failures.
• The certification chain for this exam revision is recorded in this package: reference solution passes at rate 1.0, baseline fails at rate 0.0.

• CERTIFIED PASS: the grader accepted the submission, containment was verified, provenance is complete and no integrity invariant fired.
• model fail: the submission failed the hidden tests under verified containment; a real, attributable model result.
• lane error: the provider or CLI lane failed (rate limit, quota, transport) before an attributable measurement existed; not a model result.
• withheld: the run cannot prove its own conditions (containment unproven, provenance incomplete, or a harness suspicion the record cannot clear); neither a pass nor a model failure.
• INVALID: an integrity invariant fired (isolation breach, evidence mismatch); the row is not a model result.
• harness suspect: the run's own self-check fired; each rule is classified as verdict-bearing or diagnostic, and verdict-bearing suspicion withholds the affected cells.