Skip to main content

Training & evals

Hugging Face TRL

Score the code your model writes during TRL GRPO training in E2B sandboxes. Generated programs run remotely, away from your training data and credentials.

Maintained by
E2B
How it works
Async reward function for GRPOTrainer
Package
trl 1.14.2
Language
Python

Set it up

Get started

Paste the prompt into your coding agent, or follow the steps yourself.

Paste this into your coding agent

Add E2B sandboxes to this TRL project.

  1. 1.Get the trl-grpo-python example from the E2B cookbook and install it with the project's package manager.
  2. 2.The reward pool needs the E2B_API_KEY environment variable. If it isn't set, ask me for my API key from https://console.e2b.dev/?tab=keys. Never hardcode it.
  3. 3.Follow the TRL guide at https://docs.e2b.dev/agents/trl and pass the pool's async code_reward to GRPOTrainer as reward_funcs, so generated programs run in E2B sandboxes. Close the pool in a finally block.
  4. 4.Run the reward smoke test once and show me the scores that came from the sandbox.

Or set it up yourself

  1. 1. Get the example and set E2B_API_KEY in .env

    git clone https://github.com/e2b-dev/e2b-cookbook.git
    cd e2b-cookbook/examples/trl-grpo-python
    uv sync --locked --python 3.12 --no-dev
    cp .env.example .env
  2. 2. Score fixed programs without a GPU

    uv run --locked --no-dev --env-file .env python reward_smoke.py
  3. 3. Pass the reward to GRPOTrainer

    import asyncio
    
    from datasets import Dataset
    from trl import GRPOTrainer
    
    from rewards import RewardPool
    from tasks import EVAL_TASKS, TRAIN_TASKS
    
    pool = RewardPool(args.sandbox_workers, args.create_interval)
    try:
        trainer = GRPOTrainer(
            model=args.model,
            args=config,
            reward_funcs=pool.code_reward,
            train_dataset=Dataset.from_list(TRAIN_TASKS),
            eval_dataset=Dataset.from_list(EVAL_TASKS),
        )
        trainer.train()
    finally:
        asyncio.run(pool.close())
  4. 4. Check the baseline, then train

    uv sync --locked --python 3.12 --extra training --no-dev
    uv run --locked --extra training --no-dev --env-file .env python train.py \
      --model "$TRAIN_MODEL" --baseline-only
    uv run --locked --extra training --no-dev --env-file .env python train.py \
      --model "$TRAIN_MODEL"

Security

What runs in the sandbox

Generation and weight updates stay on your training machine. E2B only runs the candidate programs.

  • CodeEach generated program runs on a pooled worker sandbox, in a fresh isolated process for every test case.
  • GradingExpected outputs and the reward calculation stay on the host. Candidates only receive the test inputs.
  • LimitsTwo seconds per test, 64 KiB of output, and outbound internet disabled on every worker.
  • LifetimeWorkers have a 600-⁠second TTL, renewed while the pool lives. Close the pool in finally to kill them.

3B+ sandboxes started

Give every agent a machine