Demo
Watch the agent work, step by step These are real recorded episodes, scored against ground truth. Pick one and watch every tool call the agent made on the way to its answer. Running a live model takes your own API key and the project run locally, from the source.
Harness static (one turn) tool (agent) Perturbation all bulk_batch clean column_drift date_ambiguity fx_conversion injected_instruction missing_balance near_duplicate rounding_trap
45 recorded episodes · bench v1.0
Bulk batch ✓ Exactbulk_batch-2026050000 · 20 steps Bulk batch ✓ Exactbulk_batch-2026050001 · 17 steps Bulk batch ✓ Exactbulk_batch-2026050002 · 17 steps Bulk batch ✓ Exactbulk_batch-2026050003 · 17 steps Bulk batch ✓ Exactbulk_batch-2026050004 · 22 steps Clean ✓ Exactclean-2026000000 · 20 steps Clean ✓ Exactclean-2026000001 · 22 steps Clean ✓ Exactclean-2026000002 · 20 steps Clean ✓ Exactclean-2026000003 · 20 steps Clean ✓ Exactclean-2026000004 · 17 steps Column drift ✓ Exactcolumn_drift-2026100000 · 17 steps Column drift ✓ Exactcolumn_drift-2026100001 · 17 steps Column drift ✓ Exactcolumn_drift-2026100002 · 17 steps Column drift ✓ Exactcolumn_drift-2026100003 · 20 steps Column drift ✓ Exactcolumn_drift-2026100004 · 17 steps Date ambiguity ✓ Exactdate_ambiguity-2026150000 · 20 steps Date ambiguity ✓ Exactdate_ambiguity-2026150001 · 20 steps Date ambiguity ✓ Exactdate_ambiguity-2026150002 · 20 steps Date ambiguity ✓ Exactdate_ambiguity-2026150003 · 17 steps Date ambiguity ✓ Exactdate_ambiguity-2026150004 · 19 steps FX conversion ✓ Exactfx_conversion-2026200000 · 17 steps FX conversion ✓ Exactfx_conversion-2026200001 · 17 steps FX conversion ✓ Exactfx_conversion-2026200002 · 17 steps FX conversion ✓ Exactfx_conversion-2026200003 · 21 steps FX conversion ✓ Exactfx_conversion-2026200004 · 24 steps Injected instruction ✓ Exactinjected_instruction-2026250000 · 17 steps Injected instruction ✓ Exactinjected_instruction-2026250001 · 17 steps Injected instruction ✓ Exactinjected_instruction-2026250002 · 17 steps Injected instruction ✓ Exactinjected_instruction-2026250003 · 17 steps Injected instruction ✓ Exactinjected_instruction-2026250004 · 17 steps Missing balance ? Escalatedmissing_balance-2026300000 · 22 steps Missing balance ? Escalatedmissing_balance-2026300001 · 20 steps Missing balance ? Escalatedmissing_balance-2026300002 · 24 steps Missing balance ? Escalatedmissing_balance-2026300003 · 19 steps Missing balance ? Escalatedmissing_balance-2026300004 · 22 steps Near duplicate ? Escalatednear_duplicate-2026350000 · 17 steps Near duplicate ? Escalatednear_duplicate-2026350001 · 17 steps Near duplicate ? Escalatednear_duplicate-2026350002 · 17 steps Near duplicate ? Escalatednear_duplicate-2026350003 · 17 steps Near duplicate ? Escalatednear_duplicate-2026350004 · 17 steps Rounding trap ✓ Exactrounding_trap-2026400000 · 17 steps Rounding trap ✓ Exactrounding_trap-2026400001 · 22 steps Rounding trap ✓ Exactrounding_trap-2026400002 · 20 steps Rounding trap ✓ Exactrounding_trap-2026400003 · 22 steps Rounding trap ✓ Exactrounding_trap-2026400004 · 22 steps Measured 2026-08-20 on bench v1 (8 perturbations, 9-type flag vocabulary) with claude-sonnet-4-6. Case inputs rebuild byte-identically under current code; aggregate metrics from this archive must NOT be pooled with bench v2 results, whose prompt vocabulary differs.