a13n Harness on Terminal-Bench 4.0
A 62-task run with GPT-6.1 Sol. Observed pass rates and inference-cost estimates under the published Codex cache-hit assumptions.
We ran a13n Harness on 62 Terminal-Bench 4.0 tasks with GPT-6.1 Sol at maximum reasoning effort. a13n passed 36 tasks (58.06%), compared with 58.77% for Codex 0.159.0, recomputed from its published results on the same task set. The observed gap was about 0.7 percentage points.
On the 61 tasks with complete official usage records, estimated inference cost was $98.73 for a13n and $97.70 per Codex round. The a13n cost is an estimate under Codex’s per-task cache-hit assumptions. It does not measure a13n’s actual cache performance or billed spend. Evaluation report
Costs cover selected or final runs and exclude other attempts and machine costs.
Comparing the same 62 tasks#
| Condition | a13n Harness | Official Codex |
|---|---|---|
| Model and effort | GPT-6.1 Sol, max |
GPT-6.1 Sol, max |
| Task set | 62 of Terminal-Bench 4.0.0’s 66 tasks | The same 62 tasks |
| Scored outcomes | 36 passes and 26 failures | 181 successes across 308 scored records |
| Retained runs | One final outcome per task | Five final records per task |
| Build | Development build based on 4eb06e8, with local modifications |
Codex 0.159.0 |
Of the 310 official final records, 308 had valid scores. We excluded the two missing scores from Codex’s denominator while keeping all 62 task identities in scope. The unequal numbers of retained runs mean these observed results do not establish statistical equivalence.
What the cost estimate measures#
One of the five official runs for cumulative-layout-shift has no usage record. We removed that task from both sides of the cost comparison. On the remaining 61 tasks, a13n’s estimate is $1.03 higher per round, about 1.1%.
We kept a13n’s input and output token counts, applied the corresponding official task’s cache-hit rate, and summed the resulting fees. This holds the caching assumption constant, so the comparison reflects token use under those rates. The near-equal estimates support that conditional comparison; they do not establish equal observed cache performance or actual inference spend. Cost methodology
Across all 62 tasks, the figures are $106.88 estimated for a13n and $102.53 of known official cost per round. Official usage is incomplete on that wider set, so those totals do not have equal coverage.
Four tasks excluded by host resources#
Terminal-Bench 4.0 contains 66 tasks. Our host had 12 CPU threads, about 32 GiB of memory, and no NVIDIA GPU meeting the excluded tasks’ requirements. These four tasks were removed from both sides:
| Excluded task | Resource constraint |
|---|---|
fp8-rmsnorm-gemm |
Required GPU capability unavailable |
jax-speedrun-gpu |
Required GPU capability unavailable |
math-eval-grader |
Required GPU capability unavailable |
live-database-cutover |
Requires 16 CPUs; the host had 12 threads |
How the tasks ran#
Harbor 0.23.0 prepared each Docker task container. The real Harness UI App drove the model with native file and shell tools, connecting task-declared Skills and MCP services where needed. Each task’s verifier decided pass or fail. The evaluated a13n build included local modifications beyond base commit 4eb06e8. The public report does not publish the patch or a complete build manifest for those modifications. Checking out the base commit alone cannot recreate the evaluated build; an exact rerun needs that missing build information and the execution configuration.
The dataset was Terminal-Bench 4.0.0. Its resource, environment, instruction, and verifier changes make benchmark version part of the comparison. Terminal-Bench 4.0 release notes
Run a13n#
Use Harness UI to work through a task interactively, or follow the Harness documentation to embed an agent in your Python application.
The full evaluation report contains the results, exclusions, and accounting. For a different model and task set, see a13n Harness on FrontierHarness Eval with Kimi K3.