---
title: "Hugging Face TRL integration"
description: "Compute Python code rewards for TRL GRPO training in E2B sandboxes. Score your first programs without a GPU, then pass the same reward to GRPOTrainer."
url: "https://e2b.dev/integrations/trl"
---

# Hugging Face TRL

> Score the code your model writes during TRL GRPO training in E2B sandboxes. Generated programs run remotely, away from your training data and credentials.

Training & evals · [All integrations](https://e2b.dev/integrations.md)

## About this integration

- **Maintained by:** E2B
- **How it works:** Async reward function for GRPOTrainer
- **Package:** trl 1.14.2
- **Language:** Python
- **Upstream:** [huggingface/trl · ★ 20k](https://github.com/huggingface/trl)
- **Example:** [e2b-cookbook/trl-grpo-python](https://github.com/e2b-dev/e2b-cookbook/tree/main/examples/trl-grpo-python)

## Get started

Paste the prompt into your coding agent, or follow the steps yourself.

### Paste this into your coding agent

```text
Add E2B sandboxes to this TRL project.

1. Get the trl-grpo-python example from the E2B cookbook and install it with the project's package manager.
2. The reward pool needs the E2B_API_KEY environment variable. If it isn't set, ask me for my API key from https://console.e2b.dev/?tab=keys. Never hardcode it.
3. Follow the TRL guide at https://docs.e2b.dev/agents/trl and pass the pool's async code_reward to GRPOTrainer as reward_funcs, so generated programs run in E2B sandboxes. Close the pool in a finally block.
4. Run the reward smoke test once and show me the scores that came from the sandbox.
```

## Or set it up yourself

### 1. Get the example and set E2B_API_KEY in .env

```bash
git clone https://github.com/e2b-dev/e2b-cookbook.git
cd e2b-cookbook/examples/trl-grpo-python
uv sync --locked --python 3.12 --no-dev
cp .env.example .env
```

### 2. Score fixed programs without a GPU

```bash
uv run --locked --no-dev --env-file .env python reward_smoke.py
```

### 3. Pass the reward to GRPOTrainer

```python
import asyncio

from datasets import Dataset
from trl import GRPOTrainer

from rewards import RewardPool
from tasks import EVAL_TASKS, TRAIN_TASKS

pool = RewardPool(args.sandbox_workers, args.create_interval)
try:
    trainer = GRPOTrainer(
        model=args.model,
        args=config,
        reward_funcs=pool.code_reward,
        train_dataset=Dataset.from_list(TRAIN_TASKS),
        eval_dataset=Dataset.from_list(EVAL_TASKS),
    )
    trainer.train()
finally:
    asyncio.run(pool.close())
```

### 4. Check the baseline, then train

```bash
uv sync --locked --python 3.12 --extra training --no-dev
uv run --locked --extra training --no-dev --env-file .env python train.py \
  --model "$TRAIN_MODEL" --baseline-only
uv run --locked --extra training --no-dev --env-file .env python train.py \
  --model "$TRAIN_MODEL"
```

## What runs in the sandbox

Generation and weight updates stay on your training machine. E2B only runs the candidate programs.

- **Code:** Each generated program runs on a pooled worker sandbox, in a fresh isolated process for every test case.
- **Grading:** Expected outputs and the reward calculation stay on the host. Candidates only receive the test inputs.
- **Limits:** Two seconds per test, 64 KiB of output, and outbound internet disabled on every worker.
- **Lifetime:** Workers have a 600-second TTL, renewed while the pool lives. Close the pool in finally to kill them.

## More training & eval tools

Other RL and eval tools that run their rollouts on E2B.

- [Hugging Face OpenEnv](https://e2b.dev/integrations/openenv.md): Run OpenEnv environments like Terminus on E2B and get a first reward without a GPU.
- [Harbor / Terminal-Bench](https://e2b.dev/integrations/harbor.md): Run Harbor and Terminal-Bench trials on E2B and scale evals and rollouts to many parallel sandboxes.
- [DSPy RLMs](https://e2b.dev/integrations/dspy.md): Run DSPy recursive language model (RLM) code in E2B sandboxes.

## Links

- [Get API key](https://console.e2b.dev/?tab=keys)
- [Read the docs](https://docs.e2b.dev/agents/trl)
- [E2B documentation](https://docs.e2b.dev)
