Introducing RoboE-Bench 1.0

RoboE-Bench is a long-horizon benchmark that measures how well an agent can work as a robotics engineer on a real team.

We ran each agent on 97 tasks across five simulated robotics companies with robots in the field. Alloy passed every checkpoint on 7 of them and scored 26.4%. The next best harness scored 11.3%.

The SWE-bench of Robotics

SWE-bench¹ gave software agents a standard competency test: take a real issue from an open source repo, write a fix and pass the unit tests. It is the classic benchmark for frontier models to measure coding ability.

Robotics engineering has no standard like it. RoboE-Bench is built to be one.

We built it with the robotics engineers we work with every day, around their real work, where one issue can touch multiple disciplines, teams and surfaces, from firmware releases and field testing to deployment campaign reports for leadership.

What RoboE-Bench tests

RoboE-Bench tests how well an agent can work as a robotics engineer on a real team: across multi-week workflows, across fragmented context, and supporting its colleagues.

We mark performance across eight dimensions:

1. Fleet-wide pattern. RoboE-Bench runs at real fleet scale, with more than 120 GB of robot data in each org, far more than any model can hold in context. The agent has to find the right moment in a data stream among thousands of recordings, and look across the whole fleet, not just the robot in the ticket.
2. Cross-sensor. Reading telemetry, video, control commands, state estimates, GPS and event logs together to find the real cause, even when a colleague is sure it's something else.
3. Code or config link. Tying a behaviour to the build or parameter change behind it, such as a controller gain change that made position hold worse.
4. Regression check. Testing whether a fix or version change really changed anything.
5. Historical recall. Using earlier tickets, earlier recordings and its own earlier work, such as recognising that a new ticket repeats an old one.
6. Monitoring. When asked to keep an eye on an issue, setting up a scan that checks incoming recordings for regressions.
7. Org context. Following the team's process and conventions, and knowing who the people are.
8. Handoff. Traceable, clear answers: citing and linking the recordings and events behind each claim, and presenting results as clear charts instead of raw data. Closing the loop on every issue, including updating the ticket, following up with the right people and reporting to the right audience.

Methodology

We adapted TheAgentCompany² to evaluate agents on robotics engineering tasks. TheAgentCompany is an extensible benchmark from researchers at Carnegie Mellon for evaluating AI agents that interact with the world in similar ways to a digital worker. It tests them inside a simulated company, with colleagues who respond, real workplace tools and checkpoint scoring. It was built to be extended to new kinds of work, so we used it as designed and brought it to robotics.

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World TasksXu et al., Carnegie Mellon University. arXiv 2412.14161

1. Simulated colleaguesEngineers, leadership and business teams who reply in character
2. A company's real toolsSlack, Jira, wiki and robot data
3. Checkpoint scoringEvery task graded against checkpoints
4. Professional tasksRobotics engineering work, across multi week projects

Setup

RoboE-Bench runs across five simulated robotics companies at different stages of deployment. Together they cover these form factors:

• Multirotor drones
• Fixed-wing UAVs
• Dexterous arms in industrial settings
• Autonomous land and maritime vehicles

Each company is modelled on a small robotics company. Its employees include leadership, firmware, support engineers, robotics engineers and field technicians, along with marketing, human resources and procurement. We used what real customers ask us to identify the jobs and the tasks behind them, then built profiles of 49 workers who interact with the agent. Requests arrive the way they do on a busy team, with some context still to find.

Each company runs on the tools a real team uses: Jira for tickets, Confluence for docs and Slack for conversation, alongside its robot data. A single task can start from a Jira ticket, then ask the agent to find the right recording, check the team's history in Confluence, reach a root cause, update the ticket and hand off in Slack.

Each company comes with years of prior history: past tickets, team docs, earlier firmware builds and more than 5,000 recordings a year. Projects run for three to four weeks on top of that history. Tickets arrive at set times and new recordings land between them.

Data sources

Most of our benchmark data suite is proprietary. However, part of it is built on two public datasets with published fault labels and ground truth. That gives us a reference point.

• ALFA. Fixed-wing UAV flights with labelled control-surface and engine faults.³
• AURSAD. A UR3e screwdriving cell with labelled tightening faults.⁴

So agents can't recognise the public data from memory, we renamed the channels, offset the GPS positions and held some recordings out, then probed the agents to check they couldn't identify the source.

How we ran it

We ran every harness end to end, with no human input during a run. Simulated colleagues keep the work moving, and the company's clock moves on by itself.

1. Each run started from the same company brief, with equal access to all raw data sources and context.
2. Alloy ran on Claude Opus 5.5 and GPT-6.1 Sol. Claude Code ran on Claude Opus 5.5, Codex on GPT-6 Astra and GPT-6 Luna, DeepSeek Harness on DeepSeek V4 Pro, and Antigravity CLI on Gemini 4 Argon. Every model ran at medium reasoning effort, with no step limit.
3. Agents work through a total of 97 tasks across five companies.
4. New tickets, colleagues' replies and issues the agent proactively raises itself move the clock forward. Colleagues reveal facts by fixed rules in code, at temperature 0, so every agent faces the same person.
5. Each task ran 10 times, and we report the mean.

Scoring

Each run earns points on a rubric of about 700 checkpoints, across the eight dimensions RoboE-Bench tests.

Where we can, we check the record, such as which recording was opened, which fields were set and who was told. Where a judgment is needed, three fresh judges vote and the majority counts. We calibrated the judges until they agreed with each other more than 96% of the time.

What keeps it fair

Every agent saw the same tickets, company history, recordings and colleagues at the same moments, with no access to future data.

• Same instructions and access. Every agent gets the same company brief, the same tools and the same raw data.
• No peeking ahead. Before the runs, we probed 108 times for any way to see future data, and found no leaks. The companies are private and simulated, so no task can appear in a model's training data.
• Rubric kept hidden. Agents never see the grading rubric or the answer key.
• No answers given away. The company's history never states the solution outright. The agent has to work it out.
• Fair traps and red herrings. Every misleading clue has a real explanation the agent can find in the data.
• Consistency. We ran each task 10 times to reduce noise, and we report pass^10: the share of tasks an agent gets right on all 10 tries.

Frontier models ran in their makers' own coding agents, the strongest setup available off the shelf.

The frontier of this test

This test is hard by design. Agents can take a ticket through to a report. The hardest checkpoints ask for what comes next: a change to the hardware, software or firmware, and proof that the new build is better. That is the learning loop, and no agent closes it yet. Hard benchmarks start low. Devin launched on SWE-bench at 13.86%.⁵

Some checkpoints are hands-on work no agent can do yet, such as physically inspecting the robot or going to the site. We keep them in the total so the score measures the whole robotics engineering job. That means no agent can reach 100%.

Alloy's strengths lie in these particular areas:

• Ongoing monitoring. Alloy earned 63% of monitoring points, against 12% or less for every other agent.
• Regression checks. Testing whether a fix really fixed anything: 90%, against 68% for the next best.
• Shareability and handoff. Leaving work a teammate could pick up, such as a linked report, an updated ticket or a summary for leadership: 91%, against 49% or less.

In one task, an operator reported an issue on a Monday. Alloy flagged it, linked it to the Jira ticket and set up a scenario scanner, a background task that checks every incoming recording for the same issue. Two months later, when leadership asked for a report on it, Alloy built it instantly from every recording the scenario had already registered as a match for that anomaly type.

Some things trip up every agent, Alloy included. No agent linked a new ticket to the earlier one it repeated, and every agent lost points tying a behaviour to the firmware or parameter change behind it.

Part two

SOTA models inside Alloy

Models change every few weeks, and no single model wins on everything. So we test each one inside the same Alloy harness, on engineering jobs across real robot logs, telemetry and mission data. We move each job to the model that tests best, across 66 configurations so far.

Part two scores cover the full set of jobs, including jobs no model has solved yet. These results are from 4 November 2026. For every model and the latest scores, see the live benchmark.

What's next

The next versions go after the learning loop, with more companies, longer projects and more checkpoints on whether a change made the robot better. Every new model gets scored in part two as it releases, so Alloy improves as models improve.

Our goal is to open source RoboE-Bench, so any robotics team can test its own agents on it.

The methodology is written for your agent to read. Hand it over and put our numbers to the test.

References

1. Jimenez, C. E. et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. arXiv:2310.06770.
2. Xu, F. F. et al. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. 2024. arXiv:2412.14161.
3. Keipour, A., Mousaei, M. and Scherer, S. ALFA: A Dataset for UAV Fault and Anomaly Detection. The International Journal of Robotics Research, 40(2–3), 2021. arXiv:1907.06268. Dataset licensed CC BY 4.0.
4. Leporowski, B., Tola, D., Hansen, C. and Iosifidis, A. AURSAD: Universal Robot Screwdriving Anomaly Detection Dataset. 2021. arXiv:2102.01409. Dataset licensed CC BY 4.0.
5. Cognition. SWE-bench technical report. 15 March 2024. cognition.com.

Frequently asked questions

About Alloy

Alloy is the AI Robotics Engineer. It reviews every run without being asked, shows its work and remembers for the whole team. We set it up in Slack and Jira, live in week one, and we maintain it. Your team keeps its coding agents, because Claude Code, Codex and Cursor all connect to Alloy over MCP. Frontier robotics teams are resolving robot issues 10x faster. Read their stories.

See the benchmark