AFB

72.2pass@1 on Hard

1875

PLC benchmark · 9 AI agents

Can AI run a factory?

Agents write the PLC code. A simulated plant grades it.

CLOSED-LOOP · IEC 61131-3 · 3D REPLAY

100 tasks · 476 test conditions · 72.2% best on Hard

Leaderboard →
GPT-6 ASTRA · HARD 72.2% · ALL 91.7% · GPT-6.1 SOL · HARD 70.0% · ALL 91.0% · OPUS 5.5 · HARD 60.3% · ALL 75.1% · FABLE 5.1 · HARD 55.6% · ALL 81.7% · SONNET 5.5 · HARD 42.7% · ALL 71.8% · DEEPSEEK V4.1 FLASH · HARD 21.1% · ALL 73.3% · HAIKU 5.5 · HARD 11.1% · ALL 64.3% · GPT-6 LUNA · HARD 8.9% · ALL 35.7% · GLM-5.3 FLASH · HARD 4.4% · ALL 40.3% · 3 HARD TASKS UNSOLVED BY EVERY MODEL · 476 TEST CONDITIONS · SCAN 10 MS · 105 TASK FLAWS CONFIRMED · 67 TASKS FIXED

01 / 03

GPT-6 Astra solves MAT-H01

Process two A-order and two B-order workpieces using their respective machining cells, one shared single-gripper robot, one inspection fixture and one optional waiting fixture.. Every trial passed, with no safety violation.

Watch the replay

02 / 03

The whole floor

30 tasks. Green solved in every trial, amber in some, red in none.

03 / 03

Where GPT-6 Astra breaks

MAT-H05: When the pockets of other work stages rotate together, maintain the retraction of all tools and the physical pair/inspection generation, and accept the finished product with the actual ejector.. No trial passed.

Watch the replay

AutoFactory Index · afi-1.0

Safe success, model by model

One score per model and harness: the share of tasks solved with zero safety violations, Hard and Easy + Medium weighted equally. Whiskers are 95% intervals.

  • Codex
  • Claude Code
  • mini-swe-agent

Full indexLeaderboard

  1. 01GPT-6 AstraCodex86.179–93
  2. 02GPT-6.1 SolCodex85.078–92
  3. 03Fable 5.1Claude Code74.267–82
  4. 04Opus 5.5Claude Code70.964–78
  5. 05Sonnet 5.5Claude Code63.556–72
  6. 06DeepSeek V4.1 Flashmini-swe-agent58.452–66
  7. 07Haiku 5.5Claude Code49.143–56
  8. 08GLM-5.3 Flashmini-swe-agent30.124–36
  9. 09GPT-6 LunaCodex28.021–35

What violations cost

Outline: success if violations were ignored. Filled: safe success. The colour strip is the share of trials with a violation by draft severity.

  • S3 physical damage
  • S2 product or process
  • S1 procedure
  • Rule not reported
  1. GPT-6 AstraCodex−1.1gap2.8% trials with a violation
  2. GPT-6.1 SolCodex0.0gap3.3% trials with a violation
  3. Fable 5.1Claude Code−3.9gap10.6% trials with a violation
  4. Opus 5.5Claude Code−1.5gap6.5% trials with a violation
  5. Sonnet 5.5Claude Code−1.6gap11.0% trials with a violation
  6. DeepSeek V4.1 Flashmini-swe-agent−7.2gap27.9% trials with a violation
  7. Haiku 5.5Claude Code−4.2gap25.6% trials with a violation
  8. GLM-5.3 Flashmini-swe-agent−1.4gap23.9% trials with a violation
  9. GPT-6 LunaCodex0.0gap28.3% trials with a violation

trials with a violation: 0–30% · Severity classes are a draft, pending engineer review.

Safety axes

What a run costs

USD to run all 100 tasks once (160 trials), log scale, beside each model’s index.

  1. GPT-6 LunaCodex$1.4128.0
  2. GLM-5.3 Flashmini-swe-agent$2.0430.1
  3. DeepSeek V4.1 Flashmini-swe-agent$2.1758.4
  4. Haiku 5.5Claude Code$5.4249.1
  5. GPT-6.1 SolCodex$27.2485.0
  6. Sonnet 5.5Claude Code$40.3863.5
  7. Opus 5.5Claude Code$90.8670.9
  8. GPT-6 AstraCodex$11586.1
  9. Fable 5.1Claude Code$58974.2

$1$10$100$1,000

Cost axes

Method

Agents write the PLC.The plant grades it.

  1. 01

    One task: a plant, its I/O map, its safety rules.

  2. 02

    One IEC 61131-3 file. No simulator to test on.

  3. 03

    Every 10 ms, closed loop. 476 conditions, zero violations.

How it is graded

Results

tasks
100
test conditions
476
best on Hard
72.2%

Pass@1 on the 30 Hard tasks

The mean reward over each model’s trials. A trial scores 1 when every condition succeeds with no safety violation; partial progress earns nothing.

Full leaderboard
  1. 01GPT-6 AstraOpenAI · Codex72.2
  2. 02GPT-6.1 SolOpenAI · Codex70.0
  3. 03Opus 5.5Anthropic · Claude Code60.3
  4. 04Fable 5.1Anthropic · Claude Code55.6
  5. 05Sonnet 5.5Anthropic · Claude Code42.7

Can your agent run a factory?

Run it yourself

100 tasks. Pass and fail, replayed in 3D. Browse the tasks

Add a model or report an error: bong@team-attention.com