In one paragraph
Agent evaluation covers the tools researchers and engineers use to find out whether an agent actually works. Some projects give agents real infrastructure to touch: aws-bench provisions live AWS accounts, while CommerceAgentBench spins up local replicas of commerce platforms. Others target narrower skill sets: faibench_Frontier_InfraBench focuses on ML systems work inside Docker, and jevbench scores decision models on calibration and cost per decision. eval fixes model and tasks to compare harnesses directly. What counts here: benchmarks, leaderboards, and research environments that measure how well agents or models do defined work.
The main approaches
Harness benchmarks
Benchmarks that hold model and tasks constant to isolate differences in the agent runtime itself. eval ran nine coding-agent harnesses on 30 identical tasks with one model, exposing 17.5x cost differences at similar pass rates.
Domain task benchmarks
Task suites grounded in a specific domain, with verifiers that check real state rather than text output. aws-bench evaluates agents on live AWS provisioning and misconfiguration tasks. CommerceAgentBench covers supplier analysis, product publishing, and logistics across 107 tasks on local mock commerce services. faibench_Frontier_InfraBench tests ML infrastructure work: CUDA kernels, distributed training, vLLM serving, and quantization, across 85 Docker-contained tasks. ceobench-src runs an LLM agent through 500 simulated startup days, scoring long-horizon planning under delayed consequences and partial observability.
Decision-model scoring
Leaderboards that measure structured classification and routing tasks on axes beyond accuracy. jevbench scores decision models on typed accuracy, probability calibration, latency, and cost per 1,000 decisions across six task families.
Research environments
Simulation platforms built for studying agent behavior rather than producing a single score. SwarmWorld runs societies of LLM agents in a 2-D world where artifacts persist across ticks and runs are deterministically replayable with counterfactual contributor removal. php_parkour is research code for vision-based humanoid parkour using RL distillation, sitting at the boundary of robotics policy evaluation.
Map of the theme
flowchart LR t["Agent evaluation"] t --> f1["Harness benchmarks"] t --> f2["Domain task benchmarks"] t --> f3["Decision-model scoring"] t --> f4["Research environments"] f1 --> r1["frontier-harness-eval/eval"] f2 --> r2["zlab-princeton/ceobench-src"] f2 --> r3["aws-bench/aws-bench"] f2 --> r4["Accio-org/CommerceAgentBench"] f2 --> r5["one2piece2hello/faibench_Frontier_InfraBench"] f3 --> r6["fstandhartinger/jevbench"] f4 --> r7["lamm-mit/SwarmWorld"] f4 --> r8["amazon-far/php_parkour"]
Where the new ideas are
ceobench-src includes a competitor agent that tracks an unreleased development bank and periodically raises customer quality expectations, creating adversarial drift inside a long-horizon business simulation.
faibench_Frontier_InfraBench uses a log-scale reward formula where matching the reference anchor scores 0: the agent must beat it to earn positive reward.
eval treats the harness itself as the variable under test, holding model, tasks, and runtime constant to produce harness-level cost and speed comparisons.
SwarmWorld adds counterfactual contributor-removal analysis, letting researchers measure how much a single agent contributed to a society’s technological trajectory by replaying runs with that agent removed.
jevbench reports cost per 1,000 decisions rather than per 1,000 tokens, aligning the unit of measurement to how decision models are actually deployed.
Side by side
| Repo | Approach | Real/live environment | Offline/reproducible scoring | Leaderboard | Multi-step / long-horizon tasks |
|---|---|---|---|---|---|
| zlab-princeton/ceobench-src | Domain task benchmarks | Not stated | Yes, encrypted SQLCipher ledger | Trajectory viewer at ceobench.com | Yes, 500 simulated days |
| aws-bench/aws-bench | Domain task benchmarks | Yes, live AWS accounts | Not stated | Not stated | Not stated |
| Accio-org/CommerceAgentBench | Domain task benchmarks | Not stated | Yes, host-side verifier on mock state | Not stated | Yes, 107 multi-step tasks |
| one2piece2hello/faibench_Frontier_InfraBench | Domain task benchmarks | Not stated | Yes, Docker-contained offline graders | Not stated | Yes, 20 long-horizon repo-level tasks |
| frontier-harness-eval/eval | Harness benchmarks | Not stated | Yes, deterministic verifiers | Yes, but new runs default to no rank | Not stated |
| lamm-mit/SwarmWorld | Research environments | Not stated | Yes, deterministically replayable | Not stated | Yes, multi-tick artifact persistence |
| fstandhartinger/jevbench | Decision-model scoring | Not stated | Not stated | Yes, live leaderboard | Not stated |
| amazon-far/php_parkour | Research environments | Not stated | Not stated | Not stated | Yes, long-horizon locomotion sequences |
How the idea moved
flowchart LR n1["started Mar 2026<br/>zlab-princeton/ceobench-src"] n2["started Jul 2026<br/>aws-bench/aws-bench"] n3["started Aug 2026<br/>Accio-org/CommerceAgentBench"] n4["started Aug 2026<br/>one2piece2hello/faibench_Frontier_InfraBench"] n5["started Aug 2026<br/>frontier-harness-eval/eval"] n6["started Sep 2026<br/>lamm-mit/SwarmWorld"] n7["started Sep 2026<br/>fstandhartinger/jevbench"] n8["started Sep 2026<br/>amazon-far/php_parkour"] n1 --> n2 --> n3 --> n4 --> n5 --> n6 --> n7 --> n8
- started Mar 2026 · ceobench-src · Adds: a 500-day startup simulation with an adversarial competitor agent, delayed consequences, and partially observable state, scoring long-horizon planning for LLM agents.
- started Jul 2026 · aws-bench · Adds: evaluation of AI agents on live, isolated AWS accounts across 134 curated tasks, scoring both diagnostic answers with an LLM judge and resource mutations against actual cloud state.
- started Aug 2026 · CommerceAgentBench · Adds: 107 long-horizon commerce tasks across CLI, browser, file, and API interfaces, each verified against actual state changes in isolated local mock services.
- started Aug 2026 · faibench_Frontier_InfraBench · Adds: 85 Docker-contained ML infrastructure tasks with offline graders, a log-scale reward that requires beating a reference anchor, and self-attesting baseline code.
- started Aug 2026 · eval · Adds: a method for isolating harness-level effects by holding model, tasks, and runtime constant across nine harnesses, producing cost and speed comparisons on the same 30-task set.
- started Sep 2026 · SwarmWorld · Adds: a deterministic multi-agent research environment where artifacts persist across simulation ticks and counterfactual contributor-removal lets researchers measure each agent’s causal effect on collective technological evolution.
- started Sep 2026 · jevbench · Adds: a leaderboard that scores decision models on four axes including probability calibration and cost per 1,000 decisions, across six structured classification and routing task families.
- started Sep 2026 · php_parkour · Adds: a three-stage pipeline for training and evaluating vision-based humanoid parkour policies, from offline motion matching through privileged-state RL to depth-sensor student distillation with sim2sim verification.
Easily confused
This theme might be confused with agent frameworks or training pipelines. The difference: projects here measure what an agent achieves on defined tasks, they do not build the agent or train it. php_parkour is the closest edge case: it trains a humanoid locomotion policy, but its sim2sim workflow evaluates the exported policy in MuJoCo, placing it at the boundary of evaluation and training. A reader might also confuse domain task benchmarks with integration test suites. The distinction is that these benchmarks produce scores intended for comparison across models or systems, not pass/fail gates for a single codebase.
Gaps nobody has filled
- No project measures agent safety or refusal behavior: all scoring focuses on task completion, accuracy, cost, or speed.
- No project provides a standardized inter-agent coordination score; SwarmWorld studies collective behavior but does not produce one.
- Systematic variation of task difficulty at run time is limited: ceobench-src includes a competitor that raises expectations over the simulation, but no project injects adversarial perturbations into task inputs.
- Cross-benchmark comparison is absent: no project provides a shared scoring protocol or meta-leaderboard that lets results from different suites be placed side by side.
- Human baseline scores are not provided for any task suite, making it hard to calibrate what a given pass rate means in practice.