Real-SWE: プライベートで実世界のエンタープライズコードベースにおけるAIモデルのベンチマーキング

Real-SWE 1.1 のご紹介

リーダーボードタスク別分析効率性評価設定

リーダーボード

1. Rank 1［画像］

   Claude Opus 5.5

   Resolution rate: 56.80%, one standard error either side: 53.5% to 60.1%

2. Rank 2［画像］

   GPT 6 Astra

   Resolution rate: 56.00%, one standard error either side: 54.4% to 57.6%

3. Rank 3［画像］

   Claude Sonnet 5.5

   Resolution rate: 51.20%, one standard error either side: 49.0% to 53.4%

4. Rank 4［画像］

   GPT 6.1 Sol

   Resolution rate: 47.20%, one standard error either side: 44.8% to 49.6%

5. Rank 5［画像］

   Claude Fable 5.1

   Resolution rate: 45.60%, one standard error either side: 42.5% to 48.7%

6. Rank 6［画像］

   Muse Wonder 1.0

   Resolution rate: 42.40%, one standard error either side: 39.9% to 44.9%

7. Rank 7［画像］

   Gemini 4 Argon

   Resolution rate: 39.20%, one standard error either side: 35.9% to 42.5%

8. Rank 8［画像］

   DeepSeek V4.1 Flash

   Resolution rate: 38.40%, one standard error either side: 35.8% to 41.0%

9. Rank 9［画像］

   Hy4 preview

   Resolution rate: 33.60%, one standard error either side: 30.9% to 36.3%

10. Rank 10［画像］

    Grok 4.7

    Resolution rate: 28.00%, one standard error either side: 25.3% to 30.7%

11. Rank 11［画像］

    GLM-5.3

    Resolution rate: 26.40%, one standard error either side: 23.9% to 28.9%

12. Rank 12［画像］

    Qwen3.8-Max prime

    Resolution rate: 24.00%, one standard error either side: 21.2% to 26.8%

13. Rank 13［画像］

    Kimi K3

    Resolution rate: 11.20%, one standard error either side: 8.9% to 13.5%

14. Rank 14［画像］

    GPT 6 Luna

    Resolution rate: 10.40%, one standard error either side: 8.5% to 12.3%

15. Rank 15［画像］

    Inkling

    Resolution rate: 8.00%, one standard error either side: 6.5% to 9.5%

16. Rank 16［画像］

    Seed 2.1 Turbo

    Resolution rate: 5.60%, one standard error either side: 4.1% to 7.1%

17. Rank 17［画像］

    Nemotron 3 Ultra

    Resolution rate: 4.80%, one standard error either side: 3.4% to 6.2%

Resolution rate is equivalent to pass@1, averaged over five independent runs per task.All runs use the highest effort possible for each model with its harness.

Models in leaderboard order, with estimated cost per rollout

Model

Resolution rate

Estimated cost per rollout

1

［画像］Opus 5.5

56.80%

$13.95

2

［画像］GPT 6 Astra

56.00%

$3.78

3

［画像］Sonnet 5.5

51.20%

$10.09

4

［画像］GPT 6.1 Sol

47.20%

$0.79

5

［画像］Fable 5.1

45.60%

$14.24

6

［画像］Muse Wonder 1.0

42.40%

$3.76

7

［画像］Gemini 4 Argon

39.20%

$4.66

8

［画像］DeepSeek V4.1 Flash

38.40%

$0.19

9

［画像］Hy4 preview

33.60%

$1.13

10

［画像］Grok 4.7

28.00%

$2.66

11

［画像］GLM-5.3

26.40%

$3.35

12

［画像］Qwen3.8-Max prime

24.00%

$7.06

13

［画像］Kimi K3

11.20%

$2.31

14

［画像］GPT 6 Luna

10.40%

$0.08

15

［画像］Inkling

8.00%

$5.13

16

［画像］Seed 2.1 Turbo

5.60%

$3.03

17

［画像］Nemotron 3 Ultra

4.80%

$1.39

The dotted line connects the best observed trade-offs, not intermediate evaluations.

Introduction

Today we are releasing Real-SWE 1.1, a benchmark that evaluates frontier AI models on private, real-world, enterprise codebases. Each task comes from a private production codebase that we licensed from a real-world company. These are problems their engineers work on, with all the context and complexity that comes with an existing product. This release expands the benchmark to 25 tasks across TypeScript, Go, Python, Ruby and Java codebases and compares 17 models.

• Private codebases. Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet.
• Work with business consequences. Getting billing right, calculating taxes, migrating customers. Changes that affect how a business runs, often across multiple services.
• Company-specific complexity. Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what’s already there.

Can a coding agent actually do the work of a software engineer in the real world?

Expert-generated or synthetic tasks can be well designed, but they aren’t the verbatim, actual tasks that engineers in real companies need to do. Our tasks differ on two axes: the underlying coding artifact and specificity of the instruction. Both add complexities that challenge today’s frontier models.

We use native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation.

Real company tasks require company-specific context

Correct billing depends on business rules and external services

Fix invoice billing so each business charges the right tax and exempt customers aren't taxed.

View instructionHide instruction

Billing reopens on Monday and every invoice this service issues is coming out untaxed. Each business on the platform settles its tax a different way: some maintain a rate themselves, some want each invoice priced against the buyer's destination by our tax authority provider, and some collect nothing at all, while a customer we hold an exemption for is charged nothing whichever way its business is configured. Pricing a destination means going to the authority with both addresses, the priced lines and the product category that business sells under, on the sandbox or the production authority according to the account the business is on; an address the authority refuses must be reported without stopping the invoice. The rate, the tax and the gross belong on the issued invoice, and once an invoice is settled the sale is filed back to the authority under that invoice's number so the returns reconcile. Invoices between European parties show both sides' VAT registrations. The authority and ledger are available at TAX_JAR_URL, PROD_TAX_JAR_URL and INFLUX_URL.

Services in the sandbox

• TaxJar sandbox
• TaxJar production
• InfluxDB ledger
• NestJS service
• TypeScript

Agents work across code, infrastructure, and business tools

Tools and services across Real-SWE task environments. Each task exposes only the services its workflow needs.

• TypeScript
• Node.js
• Python
• Go
• Ruby
• Java
• NestJS
• Next.js
• React
• Django
• Flask
• Rails
• Spring Boot
• PostgreSQL
• SQLite
• Redis
• Gel
• InfluxDB
• Docker
• Kubernetes
• AWS emulator
• Firestore emulator
• Supabase stand-in
• Linear MCP
• TaxJar
• Stripe

Codebase Selection

We selected codebases through a rigorous screening process, focusing on real companies with substantial usage, strong engineering teams, and demanding production workloads. The tasks come from these codebases, including:

• A Luma/Partiful competitor with 200K+ users and a top 100 App Store ranking
• A consumer fintech platform processing 100K+ bank statements
• Enterprise AI sales platforms supporting complex business workflows

We prioritize code written to meet an actual user or business need over code written solely to create a benchmark task. Production engineering requires understanding existing architecture, preserving behavior that users rely on, and making changes within real operational constraints.

Brief instructions can require changes across many files

Our tasks describe the change needed, leaving agents to discover implementation details in the codebase and surrounding tools. Any behavior required by the verifier must be stated or reasonably discoverable. This leads to our prompts being slightly underspecified, shorter than DeepSWE and Terminal Bench, but specific enough to not omit instructions.

The work is cross-functional and complex: a single change can span multiple parts of the application. Agents must understand existing business logic and company coding patterns while keeping the surrounding system working.

Prompt length · median

A typical Real-SWE instruction is 1,208 characters, shorter than DeepSWE's and Terminal-Bench 3's but longer than FrontierCode's and FrontierSWE v2's.

• Real-SWE1,208 chars
• FrontierCode982 chars
• DeepSWE1,975 chars
• Terminal-Bench 31,584 chars
• FrontierSWE v2992 chars

Files edited by the reference solution · Real-SWE mean

On average, a Real-SWE reference solution edits 10.2 files.

• Real-SWE10.2
• FrontierCode6
• DeepSWE6

Real-SWE's files edited, which is the mean over its 25 tasks. FrontierCode and DeepSWE use Cognition's published comparison; FrontierCode's figure is its task description alone, without codebase guidelines. We measured instruction files from Terminal-Bench 3's 74 tasks, FrontierSWE v2's 34 tasks, and Real-SWE's 25 repository-backed tasks. Character counts are rounded to the nearest whole character. No comparable files-edited figure is included for Terminal-Bench 3 or FrontierSWE v2.

Every task is inspired or lifted verbatim from a private, real-world codebase. We find these types of tasks super interesting for three reasons:

1. Tasks on private codebases are natively out of distribution. These types of coding tasks are not available anywhere on the internet and are unlikely to have ever been trained on by any other ai model. 99% of tokens in real-world enterprises are hidden away from the frontier models.
2. These tasks are economically viable work. Each task here has a direct relationship to spend and was assigned to an engineer earning a salary. Most benchmarks test interesting, experimental capabilities that are often unlikely to be widespread in the real-world.
3. Company-specific engineering patterns matter. Does AI code match the bar of a real-world enterprise? Our results show us that we're far from that reality. Many enterprises care about code standards and patterns. We've found that today's models are weaker at understanding company coding patterns and frequently miss requirements or don't verify their assumptions.

Analysis

Here's an analysis of all the tasks from our benchmark. If you're interested in a sample of the tasks, request access here.

Unverified assumptions are the most common failure

Failures are grouped by observed submission behavior using the same taxonomy across models, following DeepSWE.

［画像］Opus 5.5

［画像］GPT 6 Astra

［画像］Sonnet 5.5

［画像］GPT 6.1 Sol

［画像］Fable 5.1

［画像］Muse Wonder 1.0

［画像］Gemini 4 Argon

［画像］DeepSeek V4.1 Flash

［画像］Hy4 preview

［画像］Grok 4.7

［画像］GLM-5.3

［画像］Qwen3.8-Max prime

［画像］Kimi K3

［画像］GPT 6 Luna

［画像］Inkling

［画像］Seed 2.1 Turbo

［画像］Nemotron 3 Ultra

Unverified assumptionMissed requirementMisunderstood taskIntegration errorWrong logicRegressionWrong file

Different models fail in different ways

Percentages are out of each model's failed runs, not all runs.

Share of each model's failed runs in each failure mode

Unverified assumption

Builds on a guess about the system instead of checking it in the workspace.

Missed requirement

Leaves out behavior the instruction requires.

Misunderstood task

Misreads the requested behavior, including how the stated requirements and rules fit together.

Integration error

Right idea, wired into the surrounding system incorrectly.

Wrong logic

Right requirement and inputs, wrong calculation or transformation.

Regression

Breaks existing behavior while making the change.

Wrong file

Puts the change in a different file or layer from the one the graded behavior runs through, such as a duplicate or unused path.

Efficiency & the Frontier

Estimated rollout costs range from $0.08 to $14.24

Effort per rollout, by task

mean per rollout, by task

Swipe the chart to see all tasks.

Select a model to hide or show its line.

Use the left and right arrow keys, Home and End to move between this model’s tasks. Escape closes the details.

View task valuesTask

• Opus 5.585k
• GPT 6 Astra11k
• Sonnet 5.5113k
• GPT 6.1 Sol15k
• Fable 5.163k
• Muse Wonder 1.026k
• Gemini 4 Argon64k
• DeepSeek V4.1 Flash48k
• Hy4 preview48k
• Grok 4.723k
• GLM-5.3185k
• Qwen3.8-Max prime36k
• Kimi K316k
• GPT 6 Luna25k
• Inkling21k
• Seed 2.1 Turbo30k
• Nemotron 3 Ultra33k

Output tokens per rollout by task

Evaluation Setup

Each agent was run in an isolated sandbox. All tasks are in Harbor format, and verifiers are injected at grading time. The verifiers are inspired by existing test suites in the codebase or use those tests verbatim.

Harness for each model