What’s new here

CEO-Bench drops a single agent into a 500-day business simulation and asks whether it can sustain a startup. Consequences are delayed and coupled: a pricing decision on day 50 may not hit revenue until day 120. The market is partially observable and noisy. LLM-driven simulators play the roles of social media, macro events, and enterprise customers, so the environment adapts. A rule-based competitor tracks an unreleased development bank and periodically raises customer quality expectations. All tunable constants (pricing, customer groups, competitor difficulty) live in a single config file.

What it does

The repo is the source for the CEO-Bench simulation engine and the bash-agent baseline. The agent gets a programmable interface to business databases, company management tools, and a social media feed. It must manage pricing, R&D spend, hiring, and marketing over 500 simulated days while a competitor tracks an unreleased development bank and periodically raises customer quality expectations.

All run state lands in an encrypted SQLCipher ledger (world.nmdb). After a run, you decrypt the ledger and query it directly with sqlcipher or use the analysis recipes in docs/analyze_trajectory.md. The trajectory viewer at ceobench.com shows published runs visually.

Three LLM roles are configured separately: the benchmarked agent model, a social/macro post simulator (Haiku 4.5 by default), and an enterprise customer simulator (Sonnet 4.5 by default). You can point each at Amazon Bedrock or the direct Anthropic API.

Who it’s for

Researchers measuring how frontier models handle long-horizon planning under uncertainty. Also useful for teams that want to stress-test a new model’s strategic reasoning before deploying it in a business-adjacent context. If you want to run a coding agent against a realistic, non-trivial environment, this is a concrete option.

Try it

You need API credentials for at least one provider (Anthropic direct or Amazon Bedrock). The simulator roles default to Claude Haiku 4.5 and Claude Sonnet 4.5.

For a quick agent evaluation, prompt any coding agent with terminal and internet access:

Download this, read instructions, and finish 500 day gameplay. https://github.com/zlab-princeton/run-ceobench

To run the bash-agent baseline from source:

uv sync

Create a .env at the repo root with your credentials, for example for Bedrock:

AWS_ACCESS_KEY_ID="..."
AWS_SECRET_ACCESS_KEY="..."
AWS_REGION="us-east-2"

Then:

uv run python -m saas_bench.agents.bash_agent.run_test \
    --model us.anthropic.claude-sonnet-4-6 \
    --provider bedrock \
    --reasoning-effort max \
    --seed 42 \
    --days 500 \
    --workspace bash_agent_runs

Output lands in bash_agent_runs/run_<id>/. To score and analyze, follow docs/analyze_trajectory.md.

How mature is it

82 stars, 20 forks, 2 contributors, no releases. 20 commits in the last 90 days; last push September 29, 2026. 2 open issues and 3 open pull requests. No license is declared.