Artificial Analysis is an independent AI benchmarking company. Developers, researchers, and enterprises use its leaderboards to choose between models, inference providers, and hardware. Frontier labs give it access to models before they are public, and the numbers it publishes shape what the industry buys and builds.
In August 2026, Artificial Analysis launched Optima, which brings that same approach to private workloads: any enterprise can now build a custom benchmark for its own use case.
Benchmarking used to mean simply asking a model a list of questions and seeing how it performed. Today, it means giving an agent a task and letting it work: reading files, running shell commands, browsing the web, editing documents and writing code. Every agent needs its own isolated machine to work in, so no one agent's results can touch another. When a new model drops, Artificial Analysis runs its benchmarks, going from zero to thousands of tasks in a couple of seconds.
Every one of those machines, across the public benchmarks and Optima, is an E2B sandbox.
The challenge
Running benchmarks has changed a lot over the past few years. It used to be asking language models to complete Q&A questions. Now it's about asking agents to complete tasks.
Nothing can leak between runs
An agentic eval only means something if every model starts from the same clean environment. One run cannot see another's files, and no state can carry over from a previous eval. Many of the models under test are not public yet, which raises the bar.
Because we publish numbers the whole industry relies on to make real decisions, we can't risk any bleed between runs, one model's environment touching another's or state carrying over between evals. We're also frequently testing models before they're public, so we needed a provider we could trust to fully isolate and tear down each sandbox once it's done.
Sporadic by design
Artificial Analysis runs hundreds of benchmarks a day. A single question can require several sandboxes, depending on the task. Multiply that across the benchmark suite and the demand arrives in bursts.
Task length is just as uneven. The same task can take one model ten times as many turns as another. Some tasks finish in seconds, while others run for more than nine hours.
Build or buy
The main alternative to E2B was building it in-house: orchestrating containers at a scale of thousands of concurrent sandboxes. The team ruled it out for two reasons. Sandbox orchestration is not what Artificial Analysis is for, and scaling it would have been its own project.
That's not our specialty at Artificial Analysis. Being able to offload that expertise to those for whom it was their core focus was reassuring to us, and means that we don't need to have that expertise internally on our team. Our team can focus on things that make Artificial Analysis different, such as creating the best benchmarks out there to test AI.
The solution
One sandbox per task, then a fan-out
Every benchmark task starts in its own E2B sandbox. That sandbox runs the agent loop and gives the model a full machine: a shell, a file system, code execution, and the web. Artificial Analysis then spins up more sandboxes for the services the agent has to work with to finish the task. A single eval can fan out to hundreds of sandboxes.
Every sandbox is scoped to one task, and most are short-lived. Nothing persists between runs, so every model faces the same test and no leftover state can skew a score. It also means Artificial Analysis can re-run any task and know a change in score came from the model, not the environment.
Example: AA-Briefcase
AA-Briefcase tests agents on realistic knowledge work. Real tasks rarely arrive with the context attached, so the agent has to go find it. The sandbox holds a file system with Slack threads, email threads, and thousands of files, including messy Excel workbooks. The agent reads, views, and edits those files inside the sandbox to reach an answer.
Seconds to nine hours
E2B sandboxes scale up and down fast enough to absorb the bursts, and stay up for as long as a slow model needs.
Some of the tasks that are running within E2B sandboxes take seconds. Other times I've seen them take longer than nine hours. E2B sandboxes have proved themselves very flexible in supporting such large variances in duration.
Optima runs on the same foundation
Optima uses the same infrastructure for private workloads. An enterprise brings its own files, agent traces, or coding environment. Optima turns that into a benchmark, runs it across leading models on E2B sandboxes, and compares quality against cost per task and time per task.
Launch day proved E2B at scale for Artificial Analysis: a big marketing push, then thousands of people hitting Optima at once.
When we launched the product, we did a big marketing video and a marketing push. This is something that we've spent months putting together, distilling all of our expertise in how we made benchmarks. On the launch, we needed to have confidence that it would work, and E2B really delivered there. No issues handling the scale and the thousands of people that wanted to try it out as soon as we launched.
Custom templates for non-standard environments
Many agentic benchmarks drop the agent into an unfamiliar environment, such as a non-Debian Linux distribution like Alpine. Artificial Analysis uses E2B custom templates to build those images, and E2B worked with the team to add Alpine support.
Stirrup: E2B as the default backend
Stirrup is the open-source agent harness Artificial Analysis built for its evals. It runs the agent loop and gives the model context management, code execution, and file system access. It powers GDPval-AA and the agentic tasks in the Intelligence Index. E2B is a native backend and the default path, so anyone using Stirrup to reproduce Artificial Analysis results runs on the same infrastructure.
We liked E2B so much, we recommended it to our friends. We use it internally, and so we built native support into Stirrup, our agentic harness.
The results
- Hundreds of benchmarks and thousands of sandboxes a day
- Zero to thousands of tasks in a couple of seconds when a new model drops and the suite re-runs.
- Task durations from seconds to more than nine hours, on the same infrastructure.
- No in-house sandbox orchestration team, so that engineers can stay on benchmark design.
Being able to hand off compute sandboxes to E2B has been fantastic as we've scaled from going from tens of sandboxes and benchmarks a day to hundreds of benchmarks and many thousands of sandboxes. E2B's been great from the compute perspective, but also from the support perspective.
Looking ahead: a benchmark for every enterprise
Optima is set to become a much bigger workload than the public benchmarks. Public benchmarks are one suite for everyone. Optima runs a separate benchmark for every enterprise use case, and George expects that to grow compute needs by an order of magnitude or more. It also brings enterprise data into the sandbox, often sensitive files, so the sandboxes now have to meet each enterprise's security requirements too.
Optima uses context provided by enterprises, and so this can look like sensitive files. We need to be sure that our compute provider would be able to support the security requirements that we and these enterprises expect. This is something we looked into quite deeply, and we had a lot of confidence in E2B's security credentials.
For Optima, Artificial Analysis went straight to E2B. Every new model adds a run to the public suite, and every new enterprise adds a benchmark of its own. Public or private, each one starts in an E2B sandbox. The reach goes beyond Artificial Analysis too: AI companies and enterprises use Stirrup for their own internal agents, with E2B as the default path.