a13n Harness on FrontierHarness Eval with Kimi K3

We ran a13n Harness with Kimi K3 on all 30 FrontierHarness Eval tasks. It passed 21 tasks (70.0%). The reported median task cost was $0.41, and the reported total divided by passes was $2.93. Complete raw usage is available for 22 of the 30 tasks. Evaluation report

The result has an unresolved timing comparison: the public records do not establish equal DeepSWE time limits between this local run and the published baselines. We present the observed outcomes without assigning a leaderboard rank.

The frozen v1.0 set combines 21 tasks from Terminal-Bench 2.1 and nine from DeepSWE v1.1. a13n passed 15/21 and 6/9, respectively. This is a selected task set, distinct from the Terminal-Bench 4.0 evaluation in our other report. Frozen benchmark manifest

A timing question behind the score#

In the frozen baseline data, 20 DeepSWE failures have a duration of exactly 3,600 seconds; all other recorded baseline trials finish below that value. This pattern is consistent with a one-hour cutoff, but the durations alone do not establish which execution phases that cutoff covers. The upstream methodology says a cell watchdog records its configured limit. Baseline records · Timing definition

Three local a13n DeepSWE trials lasted longer than 3,600 seconds:

These local times include setup and verification on Apple Silicon with amd64 emulation. The public report does not provide the phase timings and deadline configuration needed to reconcile them with the baseline. Emulation may explain some overhead; it does not establish an equivalent agent time budget.

As a sensitivity check, counting the Arktype pass as a failure would give 20/30 (66.7%), equal to the frozen Codex pass count. Keeping the reported $61.4333 total unchanged would give $3.07 per pass. This is a hypothetical recount, not a rerun or a verified timeout correction. Resolving the deadline and phase timings is necessary before attributing the one-task difference to the harness.

Pass rate and two views of cost#

The chart uses the median cost per task, including successful and failed tasks. a13n’s reported median is $0.4137; the frozen Codex median is $0.2305. Eleven of the twelve published baseline configurations have a lower median than a13n’s reported value, although coverage and execution conditions differ.

The table keeps cost per pass alongside the median because it answers a different question: how the total reported cost, including failed tasks, is spread across the passes.

Most of the reported cost came from the nine DeepSWE tasks. The direction of the cost difference changes across the two subsets:

The baselines used Runta’s restored environments and Fireworks. a13n used local OrbStack, different runner versions, and the bailian-kimi-k3 route through an internal gateway. No matched control was run. These differences also affect speed, caching, and cost.

Three retained task records#

SCC bounded-memory spilling passed in about 3,065 seconds. a13n completed datacurve/scc-bounded-memory-spilling; none of the 12 configurations passed it in the frozen published results. The timing and environment qualifications above still apply. Per-task results

Merge-diff stayed a failure after a later retry passed. The original attempt exceeded the 900-second agent deadline, although a cleanup error masked the timeout in the raw runner status. We retained that first valid result and excluded the later passing retry.

Meriyah kept its first valid clean rerun. An earlier attempt obtained an upstream reference answer and was quarantined. After a setup-only failure, the clean rerun passed 48 of 49 fail-to-pass tests and all 51,469 pass-to-pass tests. The remaining failure involved for-of with using and of as the binding name. We retained the failure without selecting another quality retry.

The Meriyah rerun used an offline dependency cache, model-gateway-only agent egress, and a separate offline verifier. Other tasks kept their own task or suite policies. Selection and clean-rerun audit

How the reported costs are calculated#

We priced tokens with the frozen upstream kimi-k3-2026-08-20 table: $3.00 per million fresh-input or cache-write tokens, $0.30 per million cache-read tokens, and $15.00 per million output tokens. These are standardized evaluation rates, rather than an actual supplier invoice.

First-cold normalization charges cached reads in each task’s first included root generation at the fresh-input rate. It raises the reported total from $60.91 to $61.4332878. Dividing by 21 passes gives $2.9254 per pass; taking the median of the 30 reported task estimates gives $0.4137 per task.

The fee includes 1,110 of 1,119 retained request records. Nine requests returned no response or usage, so their charges are unknown. Earlier smoke tests, infrastructure-invalid attempts, quarantined attempts, superseded retries, and machine costs are excluded. Complete raw usage is available for 22/30 tasks; the published totals account for available usage across all 30. Detailed accounting

Run a13n#

Use Harness UI to work through a task interactively, or follow the Harness documentation to embed an agent in your Python application.

For GPT-6.1 Sol on a different benchmark version, read a13n Harness on Terminal-Bench 4.0.