LEADERBOARD

INTRODUCTION

DEPLOYMENT

DISCUSSION

CITATION

V1.0

← ASTRAEA GALLERY

PROCEDURAL AI BENCHMARK

Astraea Atomic

Atomic is an AI benchmark test that procedurally generates reasoning challenges, checks every answer instantly, and rates models on a scale with no ceiling. When AI gets better, the challenges get harder⁠, but the scale stays the same.

Quickstart

Leaderboard

01 Leaderboard

Atomic scores for frontier systems, each evaluated on freshly generated challenges. Every rating carries a 95% interval that includes the uncertainty in estimated difficulty.

LEADERBOARD — OCT 2026

ANCHOR 1,000 · UPDATED

MODEL

DEVELOPER

ATOMIC

SWITCHYARD

FORGE

95% CI

Opus 5.5

Anthropic

1,842

1,901

1,783

±21

Sonnet 5.5

Anthropic

1,768

1,812

1,724

±19

GPT 6 Astra

OpenAI

1,705

1,688

1,722

±23

GPT 6 Sol

OpenAI

1,611

1,655

1,567

±18

Gemini 3.5

Google

1,487

1,532

1,442

±20

Grok 4.7

xAI

1,322

1,298

1,346

±24

Ratings are uncapped and stay comparable across releases as the scale is never reset. The Atomic score is the equal-weighted mean of the two family ratings, and both components are published with it. All models are tested at their highest thinking level.

02 Introduction

Benchmarks are how AI progress is measured. But fixed test sets have a shelf life. Models saturate them, their questions leak into training data, and each new generation needs a new test⁠.

Astraea Atomic replaces the fixed test with a procedure: generate, verify, rate. Every evaluation draws fresh challenges from a calibrated generator, so there is no question set to memorize. Challenges come from two families. Switchyard asks for a switching network that produces an exact behavior within a wiring budget; Counterexample Forge asks for two structures that agree on every listed property yet differ on a specified one.

Each challenge is built solution-first, so a valid answer is guaranteed to exist, and an exact verifier checks every submission. Any valid construction passes⁠: no judge model, no rubric, no credit for sounding right.

Scores come from an Elo-style item response model. Every challenge carries an estimated difficulty on the same scale as the models, and a model’s rating in each family is the value that best explains its verified passes and failures. The Atomic score is the equal-weighted mean of the two family ratings.

Benchmarks are how AI progress is measured. But fixed test sets have a shelf life. Models saturate them, their questions leak into training data, and each new generation needs a new test⁠.

P(pass) = 1 / (1 + 10^((difficulty − rating) / 400))

Atomic = 0.5 × Switchyard + 0.5 × Forge

P(pass) = 1 / (1 + 10^((difficulty − rating) / 400))

Atomic = 0.5 × Switchyard + 0.5 × Forge

A rating equal to a challenge’s difficulty predicts an even chance of success, and every 400 points multiplies the odds by ten. The scale is anchored at 1,000, the average of a fixed reference panel, and is never reset, so ratings stay comparable across releases and have no maximum. A model rated 1,841 on Switchyard and 1,600 on Counterexample Forge scores 1,720. Both components are published alongside the Atomic score, so strength in one family can’t hide weakness in the other.

FIGURE 1 — ATOMIC SCORE BY RELEASE DATE

Each point is a model’s Atomic score at release, with its 95% interval. The solid line traces the best score to date; the dashed line marks the 1,000 anchor. With no maximum on the scale, the frontier can keep rising instead of flattening against a ceiling. There is no 100%.

03 Deployment

Atomic runs from your terminal. Test your own private model through a Python adapter, or a built-in Claude model, and get an experimental rating with its uncertainty.

1. Prepare your computer

PYTHON 3.10+

MACOS OR LINUX

UV ↗

DOCKER, RUNNING

ANTHROPIC API KEY WITH CREDIT

2. Start Atomic

uvx --from astraea-atomic==0.2.2 atomic evaluate

uvx --from astraea-atomic==0.2.2 atomic evaluate

Choose your own model or a built-in Claude model, then a run size and spending limit.

3. Connect your own model

# create the adapter template
uvx --from astraea-atomic==0.2.2 atomic init-adapter ./my-model

# once configured, run it directly
uvx --from astraea-atomic==0.2.2 atomic evaluate \
  --adapter ./my-model/adapter.json

# create the adapter template
uvx --from astraea-atomic==0.2.2 atomic init-adapter ./my-model

# once configured, run it directly
uvx --from astraea-atomic==0.2.2 atomic evaluate \
  --adapter ./my-model/adapter.json

Only needed for your own model, which stays on your infrastructure. Implement the template’s two Python functions, then enter ./my-model/adapter.json when prompted.

4. View your result

Atomic shows your Atomic score with a 95% confidence interval, and saves the run evidence locally. The default run tests 32 fresh puzzles against Sonnet 5.5, anchored at 1,000 points.

OPTIONAL

Try the free demo

uvx --from astraea-atomic==0.2.2 atomic demo

uvx --from astraea-atomic==0.2.2 atomic demo

No API key or Docker needed. It checks supplied tutorial answers and shows an unrated practice scorecard.

Prefer pip?

python3 -m venv .venv
source .venv/bin/activate
python -m pip install astraea-atomic==0.2.2
atomic evaluate

python3 -m venv .venv
source .venv/bin/activate
python -m pip install astraea-atomic==0.2.2
atomic evaluate

Native Windows is not supported in this release. The rating covers the Constraint Lock construction tasks under a fixed tool and resource policy; it is experimental and not a general intelligence rating. Your keys, adapter and results are never uploaded to Astraea.

04 Challenges

Models construct things⁠ instead of answering questions. Each family tests a different kind of construction.

FAMILY 01

Switchyard

Build a switching network that produces an exact behavior within a wiring budget.

TESTS

Combining rules

Planning

Efficient design

FAMILY 02

Counterexample Forge

Build two structures that agree on every listed property yet differ on a specified one.

TESTS

Structural insight

Finding exceptions

Interacting constraints

EXAMPLE CHALLENGES

Switchyard

DIFFICULTY

1,180

Three-switch parity

Route power through three switches so the output is on for switch states 001, 010, 100 and 111, and off for every other state.

GIVEN

3 switches · at most 4 junctions

CHECKS

All 8 switch states

Switchyard

DIFFICULTY

1,640

Two-bit address decoder

Using switches A and B, light exactly one of four lamps: the lamp numbered by the binary value AB.

GIVEN

2 switches · 4 lamps · at most 6 junctions

CHECKS

All 4 switch states · one lamp lit per state

Counterexample Forge

DIFFICULTY

1,320

Same degrees, different connectivity

Construct two graphs on six vertices with identical degree sequences and the same number of edges, where one is connected and the other is not.

GIVEN

Simple graphs · 6 vertices each · edges as vertex pairs

CHECKS

Degree sequences · edge counts · connectivity

Counterexample Forge

DIFFICULTY

1,780

Same spectrum, different structure

Construct two non-isomorphic graphs whose adjacency matrices have exactly the same eigenvalues.

GIVEN

Simple graphs · at most 5 vertices each

CHECKS

Characteristic polynomials match · graphs are non-isomorphic

These are intended emphases, not proven independent measures of intelligence. Both families involve search and constraint satisfaction, so we publish both scores.

05 Method

Deciding whether arbitrary requirements can be met can be as hard as meeting them. So we build backwards: generate a valid solution, derive the requirements from it, then hide it.

BUILDING A CHALLENGE — SWITCHYARD

Generate a valid routing network.

Compute its behavior for every switch combination.

Show the model only that behavior, plus a wiring budget the hidden network fits.

Verify the submitted network against every requirement.

The hidden network proves a solution exists. The model can submit any valid design⁠.

Counterexample Forge works the same way with pairs of graphs.

CALIBRATING DIFFICULTY

Generate sample challenges across settings.

Measure a fixed reference panel on them.

Fit a difficulty predictor and validate it on fresh challenges.

Predict each new challenge’s difficulty and carry that uncertainty into every reported rating.

We calibrate the generator.

A new random seed needs no new validation.

DOES A SOLUTION EXIST?

Guaranteed

Every challenge is built from a hidden, valid construction.

IS THE ANSWER CORRECT?

Exact

A verifier checks every requirement. Any valid construction passes.

HOW HARD WAS IT?

Estimated

Predicted from structure and measured performance, then reported with its uncertainty.

Status: the scorer computes family ratings and the Atomic score today. Procedural difficulty prediction and its uncertainty are in calibration.

06 Discussion

Future model performance

Fixed benchmarks saturate. Scores move from near zero to near perfect within a few model generations, and the test stops separating the strongest systems. Atomic has no top score to reach: when models clear the hardest challenges in the current range, the generator is extended to harder settings and calibrated against the same reference panel. Ratings keep rising on one scale instead of compressing against 100%.

Interpreting scores

Ratings are relative to a fixed reference panel whose average defines 1,000. Within a family, a 400-point gap means ten times the odds of passing the same challenge, so the difference between two models reads directly as an odds ratio. Because the scale is never reset, a score published today stays comparable with one published years from now.

Limitations

Difficulty is predicted rather than measured for each new challenge, and that uncertainty is carried into every reported interval; the predictor is still being calibrated. Both families test construction under constraint, so Atomic does not cover open-ended dialogue, knowledge recall or perception. A high Atomic score indicates strength at this kind of reasoning, not general ability.

07 Citation

If you use Astraea Atomic in your work, please cite:

@misc{astraea2026atomic,
  title  = {Astraea Atomic: A Procedural Benchmark with Exact Verification and an Uncapped Rating Scale},
  author = {{Astraea}},
  year   = {2026},
  note   = {Version 1.0},
  url    = {https://www.astraea.mx/atomic}
}

@misc{astraea2026atomic,
  title  = {Astraea Atomic: A Procedural Benchmark with Exact Verification and an Uncapped Rating Scale},
  author = {{Astraea}},
  year   = {2026},
  note   = {Version 1.0},
  url    = {https://www.astraea.mx/atomic}
}

As a benchmark that can’t be memorized or outgrown, Atomic contributes to our mission of preserving the future by keeping an honest measure of what AI can do.