AFB

72.2pass@1 on Hard

1875

PLC benchmark · 10 AI agents

Can AI run a factory?

Agents write the PLC code. A simulated plant grades it.

CLOSED-LOOP · IEC 61131-3 · 3D REPLAY

100 tasks · 476 test conditions · 72.2% best on Hard

Leaderboard →
GPT-6 ASTRA · HARD 72.2% · ALL 91.7% · GPT-6.1 SOL · HARD 70.0% · ALL 91.0% · OPUS 5.5 · HARD 60.3% · ALL 77.1% · GPT-6.1 TERRA · HARD 57.8% · ALL 85.3% · FABLE 5.1 · HARD 55.6% · ALL 82.0% · SONNET 5.5 · HARD 42.7% · ALL 72.2% · DEEPSEEK V4.1 FLASH · HARD 21.1% · ALL 71.3% · HAIKU 5.5 · HARD 11.1% · ALL 66.0% · GPT-6 LUNA · HARD 8.9% · ALL 48.3% · GLM-5.3 FLASH · HARD 4.4% · ALL 43.0% · 3 HARD TASKS UNSOLVED BY EVERY MODEL · 476 TEST CONDITIONS · SCAN 10 MS · 105 TASK FLAWS CONFIRMED · 67 TASKS FIXED

01 / 03

GPT-6 Astra solves MAT-H01

Process two A-order and two B-order workpieces using their respective machining cells, one shared single-gripper robot, one inspection fixture and one optional waiting fixture.. Every trial passed, with no safety violation.

Watch the replay

02 / 03

The whole floor

30 tasks. Green solved in every trial, amber in some, red in none.

03 / 03

Where GPT-6 Astra breaks

MAT-H05: When the pockets of other work stages rotate together, maintain the retraction of all tools and the physical pair/inspection generation, and accept the finished product with the actual ejector.. No trial passed.

Watch the replay

AutoFactory Index · afi-1.0

Safe success, model by model

One score per model and harness: the share of tasks solved with zero safety violations, Hard and Easy + Medium weighted equally. Whiskers are 95% intervals.

  • Codex
  • Claude Code
  • mini-swe-agent

Full indexLeaderboard

  1. 01GPT-6 AstraCodex86.179–93
  2. 02GPT-6.1 SolCodex85.078–92
  3. 03GPT-6.1 TerraCodex77.569–85
  4. 04Fable 5.1Claude Code74.467–82
  5. 05Opus 5.5Claude Code72.366–78
  6. 06Sonnet 5.5Claude Code63.856–72
  7. 07DeepSeek V4.1 Flashmini-swe-agent57.051–64
  8. 08Haiku 5.5Claude Code50.345–56
  9. 09GPT-6 LunaCodex37.132–43
  10. 10GLM-5.3 Flashmini-swe-agent32.027–37

What violations cost

Outline: success if violations were ignored. Filled: safe success. The colour strip is the share of trials with a violation by draft severity.

  • S3 physical damage
  • S2 product or process
  • S1 procedure
  • Rule not reported
  1. GPT-6 AstraCodex−1.1gap2.8% trials with a violation
  2. GPT-6.1 SolCodex0.0gap3.3% trials with a violation
  3. GPT-6.1 TerraCodex−3.3gap11.9% trials with a violation
  4. Fable 5.1Claude Code−3.9gap10.8% trials with a violation
  5. Opus 5.5Claude Code−2.0gap6.5% trials with a violation
  6. Sonnet 5.5Claude Code−1.8gap11.0% trials with a violation
  7. DeepSeek V4.1 Flashmini-swe-agent−7.2gap27.7% trials with a violation
  8. Haiku 5.5Claude Code−4.5gap25.8% trials with a violation
  9. GPT-6 LunaCodex−0.5gap29.5% trials with a violation
  10. GLM-5.3 Flashmini-swe-agent−0.5gap24.1% trials with a violation

trials with a violation: 0–30% · Severity classes are a draft, pending engineer review.

Safety axes

What a run costs

USD to run all 100 tasks once (160 trials), log scale, beside each model’s index.

  1. GPT-6 LunaCodex$1.4137.1
  2. GLM-5.3 Flashmini-swe-agent$1.9632.0
  3. DeepSeek V4.1 Flashmini-swe-agent$2.1857.0
  4. GPT-6.1 TerraCodex$4.3277.5
  5. Haiku 5.5Claude Code$5.4750.3
  6. GPT-6.1 SolCodex$26.9085.0
  7. Sonnet 5.5Claude Code$40.5063.8
  8. Opus 5.5Claude Code$91.2772.3
  9. GPT-6 AstraCodex$10986.1
  10. Fable 5.1Claude Code$59774.4

$1$10$100$1,000

Cost axes

Method

Agents write the PLC.The plant grades it.

  1. 01

    One task: a plant, its I/O map, its safety rules.

  2. 02

    One IEC 61131-3 file. No simulator to test on.

  3. 03

    Every 10 ms, closed loop. 476 conditions, zero violations.

How it is graded

Results

tasks
100
test conditions
476
best on Hard
72.2%

Pass@1 on the 30 Hard tasks

The mean reward over each model’s trials. A trial scores 1 when every condition succeeds with no safety violation; partial progress earns nothing.

Full leaderboard
  1. 01GPT-6 AstraOpenAI · Codex72.2
  2. 02GPT-6.1 SolOpenAI · Codex70.0
  3. 03Opus 5.5Anthropic · Claude Code60.3
  4. 04GPT-6.1 TerraOpenAI · Codex57.8
  5. 05Fable 5.1Anthropic · Claude Code55.6

Can your agent run a factory?

Run it yourself

100 tasks. Pass and fail, replayed in 3D. Browse the tasks

Add a model or report an error: bong@team-attention.com