What’s new here

Φ-Bench targets the ML infrastructure layer that LLMs themselves run on. Its 85 tasks cover kernel optimization, distributed training loops, vLLM serving, quantization, and MoE routing. Every task ships a public Dockerfile, so the environment is reproducible without network access during scoring. Reference solutions exist for 83 of the 85 tasks.

What it does

Φ-Bench organizes 85 tasks into three subsets: 55 kernel function completion tasks (kfc), 20 long-horizon repository-level tasks (lh), and 10 end-to-end system optimization tasks (e2e). Each task gives the agent a single instruction.md. The agent edits files inside the container, then scoring mounts tests/ read-only and runs test.sh. The reward formula for performance tasks is min(1, ln(speedup/ref_speedup)/ln(ref_speedup)), so matching the reference anchor scores 0. You must beat it.

The vendored upstream code under environment/repo/ is self-attesting: oracle.patch must apply forward onto it and fail in reverse, proving the tree is exactly the excavated baseline. scripts/verify_package.py runs this check across all 85 tasks.

The task runner (scripts/run_task.py) is single-file, stdlib-only, and runs on Python 3.8+. It handles build, agent invocation, scoring, and artifact collection. Supported agents include claude-code and codex; the Docker images ship no agent binary, so the CLI is injected at runtime.

Who it’s for

ML systems engineers and AI research teams who want to measure whether a frontier LLM or coding agent can work on the infrastructure stack: writing CUDA kernels, modifying vLLM internals, tuning distributed training. It is also useful for teams building coding agents who need a hard, domain-specific benchmark beyond general programming tasks.

Try it

Requires Docker 23+ with BuildKit (docker buildx build). GPU tasks need --gpus all; some tasks are CPU-only. Python 3.8+ for the task runner. No extra dependencies.

# Clone and verify the package
git clone https://github.com/one2piece2hello/faibench_Frontier_InfraBench
cd faibench_Frontier_InfraBench
python3 scripts/verify_package.py
 
# Run a single task manually
cd tasks/kfc/<dir>
docker buildx build -f environment/Dockerfile -t <docker_image from task.toml> .
docker run --rm --gpus all -it <image>
docker run --rm --gpus all -v "$PWD/tests:/tests:ro" <image> bash /tests/test.sh
cat /logs/verifier/reward.json
 
# Or use the task runner with an agent
python3 scripts/run_task.py --task tasks/kfc/<dir> --agent claude-code --model claude-opus-5 \
    --agent-bin /path/to/claude
 
# Sanity-check with the oracle (should hit reference score)
python3 scripts/run_task.py --task tasks/lh/<dir> --agent oracle

How mature is it

Created August 2026, last pushed September 2026. 66 stars, 3 forks, 3 contributors, 16 commits in the past 90 days. No formal releases. 2 open issues, no open PRs. Licensed Apache-2.0 for the benchmark’s own work; vendored upstream code retains its original licenses. The arXiv paper is listed as coming soon.