Firecracker vs QEMU

If you want to run untrusted code like code generated by LLMs you will quickly realize that you need a way to sandbox the code. You could of course run the code in a container, but that is generally not considered secure enough. You could also run the code in a traditional virtual machine, but that is probably overkill and too resource intensive for short-lived tasks. This is where Firecracker and QEMU come in.

Both Firecracker and QEMU are virtualization solutions that can be used to run untrusted code. They are both KVM-based and offer robust isolation, but they serve distinctly different needs and come with their own sets of tradeoffs.

Firecracker, developed by AWS, has gained significant attention for being lightweight, fast, and security-focused. It powers AWS Lambda's serverless infrastructure, handling tens of trillion function invocations monthly [16]. Its minimal footprint and rapid boot times make it particularly attractive for ephemeral workloads.

On the other hand, QEMU has established itself as the Swiss Army knife of virtualization. With its rich feature set and extensive hardware support, it powers everything from development environments to production servers. While it may not match Firecracker's lightning-fast boot times, QEMU offers unparalleled flexibility, supporting full system emulation, hardware acceleration, and a wide range of (sometimes legacy) virtual devices that modern development workflows depend on.

This comparison will go into both virtualization solutions, exploring their architectures, strengths, and limitations. So let's go and sandbox some untrusted code!

Virtualization Basics

Virtualization lets you run multiple isolated operating systems on a single physical machine. Modern CPUs include special instructions that make virtualization faster and more secure. If you're a bit rusty on the basics, here are the key components:

Virtual Machine (VM) is an isolated environment that acts like a complete computer, with its own CPU, memory, storage, and network resources. VMs run their own operating systems and can't directly interfere with each other.

Virtual Machine Monitor (VMM) or Hypervisor is the software layer that creates and manages VMs. It controls how VMs access the physical hardware and ensures they remain isolated from each other. VMMs come in two types:

• Type 1 hypervisors run directly on hardware (like VMware ESXi)
• Type 2 hypervisors run on top of an operating system (like VirtualBox)

KVM (Kernel-based Virtual Machine) is Linux's built-in hypervisor. When enabled, it turns the Linux kernel into a Type 1 hypervisor, letting you run VMs with near-native performance. Both QEMU and Firecracker use KVM for CPU virtualization.

QEMU

QEMU stands for Quick EMUlator. It's widely used in production environments to run virtual machines for web servers, databases, and development environments. It can run as a pure emulator to mimic different CPU architectures, or as a virtualizer using KVM for near-native performance [0].

QEMU handles common VM workloads like running Linux servers, Windows instances, and containerized applications. Developers use it to test software across different operating systems, run CI/CD pipelines, and deploy production services. It's particularly useful when you need to run multiple isolated environments on a single machine. When running with KVM, QEMU delegates CPU and memory management to the Linux kernel while handling I/O and device emulation itself. This split design lets QEMU support a wide range of devices while maintaining good performance for basic compute tasks.

What makes QEMU special?

• Extensive Platform Support: You can run almost anything on QEMU. From ancient s390x [17] systems to modern full-desktop environments, if you need it, QEMU probably supports it [5]
• Full Emulation and KVM Acceleration: Support for both emulation and hardware-accelerated KVM virtualization. [0]
• Ecosystem and Integration: QEMU's long history means there's a vast array of tools and integrations available.
• MicroVM Mode: QEMU MicroVMs [13], more similar to Firecracker. Less devices supported, faster boot times (4x as fast as normal QEMU, still 3x slower than Firecracker)

What are the tradeoffs?

With age and versatility comes a different set of tradeoffs:

• Complexity and Security: With nearly 2 million lines of C code [6], QEMU's attack surface is significantly larger, with a history of CVEs. [7]
• Performance: QEMU's full-featured nature means slower boot times and higher memory usage, which can be a bottleneck for ephemeral or lightweight workloads. [8][14]
• Developer Experience: The vast options (the manpage is over 47 pages long [11]) can be overwhelming, leading to a steeper learning curve.

Firecracker

Firecracker is a virtual machine monitor (VMM) built by AWS specifically for running serverless workloads. It's a stripped-down virtualization tool that focuses on speed and security rather than flexibility. Using KVM under the hood, Firecracker creates lightweight virtual machines called microVMs [1].

Unlike traditional VMs that might take several seconds to start, Firecracker VMs can boot up in as little as 125ms. AWS built it to power Lambda and Fargate [2], where they need to quickly spin up isolated environments for running customer code. Companies like E2B use Firecracker to run AI generated code securily in the cloud, while Fly.io uses it to run lightweight container-like VMs at the edge [4, 5].

Firecracker runs as a single process on your host machine and exposes a simple API for creating and managing VMs. Each microVM is minimal - it only includes the devices and functionality needed to run a basic Linux kernel. This minimalist approach helps keep the codebase small and reduces potential security vulnerabilities.

What makes Firecracker special?

• Small and Secure: Written in Rust with only 50k lines of code (as of 2020), significantly reducing the attack surface and overall complexity [12]
• Fast Boots: Starts VMs 3x faster than QEMU MicroVMs and 10x faster than normal QEMU emulation [8]
• Minimal Memory Usage: Each microVM has less than 5MB RAM overhead [9]
• Simple Architecture: Single process with a straightforward API, making it easy to manage and integrate [12]
• Resource Efficient: Designed for overcommitting hardware resources, perfect for serverless workloads

What are the tradeoffs?

• Limited Device Support: Minimal device emulation - no support for legacy hardware like floppy disks
• No GPU Support: Currently lacks PCIe support, making GPU passthrough impossible (work has been stopped for 2025) [10]
• Smaller Ecosystem: While growing with adopters like E2B and Fly.io, the ecosystem is still smaller than QEMU's [3, 4]

So what?

Of course, the decision is very nuanced and specific to your use case. In general:

• Choose Firecracker if you're looking for speed, security, and efficiency for serverless computing or edge computing scenarios with modern hardware support.
• Choose QEMU if you need versatility, support for a broader range of hardware, full system emulation, or if you're dealing with legacy systems or development environments requiring extensive device emulation.

Sources

0. QEMU
1. Firecracker
2. Firecracker Paper
3. E2B
4. Fly.io
5. QEMU Build Platforms
6. QEMU OpenHub
7. QEMU CVEs
8. Firecracker Boot Time
9. Firecracker Memory Footprint
10. Firecracker PCIe
11. QEMU manpage
12. Firecracker Github
13. QEMU MicroVMs
14. Performance analysis of KVM-based microVMs orchestrated by Firecracker and QEMU
15. Firecracker Credits
16. AWS Lambda
17. ‍S390x

Additional Reading

• ‍Memory Ballooning in Unikernels: QEMU, Firecracker, and oh my!‍
• Why we migrated from Firecracker to QEMU and the accompanying HN discussion‍
• The Firecracker virtual machine monitor‍
• 4000 Firecracker MicroVMs on one server