Commit f3363593f for llama.cpp
commit f3363593f457169cdfbdfe22ea399b1cc139ba18
Author: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Date: Fri Oct 16 11:52:18 2026 +0200
Add more benchmarks to llama-eval (#26174)
* llama-eval : add sandboxed execution for code suites
Adds a lightweight replacement for the Docker containers usually used to run
generated code, so a benchmark can be executed without a daemon, an image or
root.
A cached venv is provisioned per suite and reused; candidate programs run in a
throwaway working directory as their own process group, under the strongest
isolation the host offers:
bwrap read-only root, tmpfs /tmp and $HOME, no network
unshare no network, via --map-current-user
rlimit cpu/memory/file-size/process caps only
Candidate code must not run as uid 0, which is why "unshare -r" is a
last-resort tier that warns: root bypasses file permission bits, so a test
asserting that a write to a chmod 0444 file fails would instead see it succeed.
Two details worth noting for the bwrap tier. Only paths hidden by the $HOME
tmpfs are bound back, because bwrap cannot mount onto a symlink and
version-manager alias directories usually are one. And the venv's base
interpreter tree is resolved by walking the symlink chain, since a
version-managed interpreter is commonly reached through an unversioned alias
that startup resolves through.
Suites can declare provisioning steps, which run once at build time while the
network is still up, so test time never needs it.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
* llama-eval : add HumanEval and ClassEval code suites
Adds two execution-graded suites alongside the existing text-graded ones. The
model's reply is parsed for a python code block, combined with the task's own
unit tests, and run in the sandbox.
--dataset humaneval 164 function-level problems, scored pass@k
--dataset classeval 100 class-level tasks, scored at class level (feeding
pass@k) and method level, matching upstream
pass@k reuses the existing chunk mechanism: --n_cases at a multiple of the
dataset size gives that many samples per problem, aggregated with the unbiased
estimator from the Codex paper.
Program construction and result interpretation are dataset-owned hooks, because
the two suites differ: one HumanEval problem is a function judged by a single
assert, whereas one ClassEval task is a class judged by many independent test
methods, and reducing that to pass/fail would throw away partial credit. Suites
also declare their own package set, provisioning steps, interpreter and timeout.
The package set is derived from the dataset's own imports rather than a
requirements file. HumanEval turns out to need no third-party packages at all;
ClassEval needs thirteen, two of which must be declared because no import
mentions them -- lxml is selected by string in ClassEval_44, and scipy is a
gensim requirement.
Notes on ClassEval specifically:
* gensim publishes no wheel past cp313, so 3.13 is preferred; on 3.14+ the two
pure-Python APIs it uses are provided by a small shim instead.
* The three nltk corpora it needs are pre-seeded at venv build time. These are
the modern names -- the dataset asks for the since-renamed punkt and
averaged_perceptron_tagger.
* It defaults to a patched fork, ilintar/ClassEval, because three upstream
tests cannot be passed by anything: a hard-coded 2024 date checked against
datetime.now(), an assertion on the dataset author's own hostname, and an
exact float comparison. Reference solutions go from 96/100 to 100/100 classes
and 2182/2196 to 2196/2196 test methods. No skeleton is touched, so what the
model is asked to write is unchanged; --dataset-source selects the original.
--exec-seed (default 0) seeds the RNG before each test, so suites whose tests
draw random inputs score reproducibly. ClassEval_58 fails its own reference
solution about 30% of the time without it.
The simulator gains both suites, so the harness can be exercised without a
model.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
* llama-eval : fix silent isolation downgrade, generalise the sandbox runner
The isolation probe ran the interpreter without binding it back, so whenever
this tool is itself started from a venv under $HOME -- the usual case, since it
needs `datasets` -- bwrap's tmpfs hid the interpreter, the probe failed, and the
sandbox quietly fell back to a tier that does not hide $HOME at all. Nothing in
the output looked wrong when that happened, and the existing suites have been
running that way. The probe now uses the same interpreter and the same binds the
suite will actually get.
Two additions the agentic suite needs, neither of which changes the existing
suites:
* run_argv() runs an arbitrary command in a caller-owned directory that survives
the call. run() is now a thin wrapper over it, so HumanEval and ClassEval take
exactly the same path as before. A multi-step agent needs its edits to persist
across many invocations, and a suite for a language other than Python needs to
drive a linter and a test runner rather than a single source string.
* RLIMIT_AS is now per-sandbox, defaulting to what it was. It caps reserved
address space rather than resident memory, which is the wrong instrument for a
runtime that reserves a large region and commits little of it: node's
TypeScript type-stripping goes through WebAssembly, reserves gigabytes per
instance, and aborts at 4 GiB with an out-of-memory error that reads like a
fault in the code under test.
HumanEval stays at 164/164 and ClassEval at 100/100.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* llama-eval : add SACB, a multi-turn agentic coding suite
`--dataset agentic` is a different shape from the other code suites. Rather than
answering a single prompt, the model is given a small but real repository, a
narrow tool set and a bug report, drives its own multi-turn conversation, and is
graded by running a test suite it was never allowed to see.
The corpus is SACB (Simple Agent Coding Benchmark), 60 tasks over three
hand-written repositories in Python and TypeScript, hosted at ilintar/SACB.
Grading needs no container: a cached venv gives ruff and mypy for Python, and
node's own test runner plus tsc for TypeScript, so the whole dependency set is
four packages.
Three constraints are deliberate:
* No shell, and no way to run the tests. The only feedback channel is `lint`. A
syntax or type error can be driven out mechanically; a logic error has to be
reasoned about. A harness that lets an agent iterate against the test suite
measures something closer to search than to understanding.
* Two edit tools, one line-addressed and one content-addressed. Which one a
model reaches for, and whether it keeps line numbers straight after its own
earlier edit, is itself a signal.
* The tests never touch the working tree. They are held outside it and copied
into a throwaway copy only once the agent has stopped, so they can be neither
read nor edited. Every tool path is resolved and confinement-checked against
the repository root, so a symlink planted inside the tree cannot reach out of
it.
A task counts as resolved when every test that was failing passes and nothing
that already passed broke. Both sets are derived by running the tests against
the defective tree and against the reference fix, never written by hand, so a
task whose defect no test exercises cannot enter the corpus. Many tasks carry
several independent defects, so the fraction of target tests passed is reported
alongside, separating "fixed two of the three faults" from "changed nothing".
An episode that arrives with no tool call and no `finish` is nudged rather than
ended. Models routinely narrate their analysis mid-task, and treating that as
completion ends the episode with no edits and scores a reasoning failure that
never happened; about one episode in four exits that way.
Nothing here is installed unless this suite is selected, and then only for the
language being run: agentic_eval is imported lazily and each toolchain is keyed
per language.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* llama-eval : make the agentic hooks typed rather than discovered
`ty` flagged nine unresolved attributes, of which the seven in main() predate
the agentic suite: `EvalState.dataset` is Optional and was dereferenced without
narrowing.
* `loaded_dataset` turns the Optional into a value in one place, raising rather
than handing back None, so callers do not each have to narrow it.
* `is_agentic`, `prepare()` and `run()` are declared on BaseDataset. They were
reached through getattr with a default at three call sites, which no checker
can see through; `run()` now raises a clear error naming the suite instead of
failing as a missing attribute.
* The sandbox setup is one branch on `is_agentic` rather than two tests of
`grader_type`, which also drops a duplicated condition.
No behaviour change: HumanEval stays at 164/164, and `ty check` on
examples/llama-eval is clean.
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
* feat: API key
* fix reasoning and response length for agentic runs, add option to save entire transcripts
* llama-eval : fix ty unresolved-attribute on loaded_dataset.ae
Assisted-by: AI coding agent
* llama-eval : trim comments and remove non-ASCII from README
Assisted-by: AI coding agent
diff --git a/examples/llama-eval/README.md b/examples/llama-eval/README.md
index 3c5c35f78..1f0b5a09a 100644
--- a/examples/llama-eval/README.md
+++ b/examples/llama-eval/README.md
@@ -24,3 +24,267 @@ python3 llama-eval.py \
--dataset aime2025 --n_cases 240 \
--grader-type regex
```
+
+### Authentication
+
+A server started with `--api-key` (or any OpenAI-compatible endpoint behind a
+token) needs one here too. `--api-key` is sent as `Authorization: Bearer` on
+every eval request; it falls back to `$LLAMA_API_KEY`, which keeps the secret
+out of your shell history and off the process list.
+
+```bash
+# One key for every server
+python3 llama-eval.py --server http://localhost:8033 --api-key sk-... ...
+
+# From the environment
+export LLAMA_API_KEY=sk-...
+python3 llama-eval.py --server http://localhost:8033 ...
+
+# One key per server, in --server order
+python3 llama-eval.py \
+ --server http://server1:8033,http://server2:8033 \
+ --api-key sk-one,sk-two --threads 16,16 ...
+```
+
+The LLM grader authenticates too. It reuses the key of the server it runs on,
+so `--grader-type llm` without `--grader-server` needs nothing extra; a grader
+on a separate endpoint takes `--grader-api-key`.
+
+Endpoints on an opencode host additionally get an `x-opencode-session` header,
+generated fresh for every task. An agentic episode sends the same id on all of
+its turns, so opencode sees one session per task instead of one per request;
+other servers never see the header.
+
+## Agentic suite
+
+`--dataset agentic` is a different shape from the other code suites. It runs
+**SACB** (Simple Agent Coding Benchmark), hosted at
+[`ilintar/SACB`](https://huggingface.co/datasets/ilintar/SACB) -- 60 tasks over
+three hand-written repositories, in Python and TypeScript. Rather than
+answering one prompt, the model is given a small but real repository, a
+deliberately narrow tool set, and a bug report, and drives its own multi-turn
+conversation until it declares itself finished. It is then graded by running a
+test suite it was never allowed to see.
+
+```bash
+python3 llama-eval.py \
+ --server http://localhost:8033 --model my-model \
+ --dataset agentic --threads 1 --temperature 0
+```
+
+### The tools
+
+| tool | purpose |
+|---|---|
+| `list_files` | enumerate the tree |
+| `read_file` | read, with 1-based line numbers and optional line-range narrowing |
+| `search` | regular-expression search across the repository |
+| `edit_lines` | replace an inclusive line range |
+| `edit_replace` | replace an exact snippet, which must be unique |
+| `write_file` | write a whole file |
+| `lint` | run the language's linter and type checker |
+| `finish` | declare the change complete |
+
+Three constraints are deliberate:
+
+**There is no shell and no way to run the tests.** The only feedback channel is
+`lint`. A syntax or type error can be driven out mechanically; a logic error has
+to be reasoned about. A harness that lets an agent iterate against the test
+suite measures something closer to search than to understanding.
+
+**There are two edit tools**, one line-addressed and one content-addressed.
+Which one a model reaches for, and whether it keeps line numbers straight after
+its own earlier edit, is itself a signal.
+
+**The tests never touch the working tree.** They are held outside it and copied
+into a throwaway copy only after the agent has stopped, so they can be neither
+read nor edited. Every tool path is resolved and confinement-checked against the
+repository root, so a symlink planted inside the tree cannot reach outside it.
+
+### Scoring
+
+A task is **resolved** when every test that was failing now passes *and* nothing
+that was already passing broke. Both sets are derived by running the tests twice
+when the corpus is built -- once against the defective tree and once against the
+reference fix -- never hand-written, so a task whose defect no test exercises is
+rejected rather than shipped.
+
+Because many tasks contain several independent defects, the fraction of target
+tests passed is reported alongside, which separates "fixed two of the three
+faults" from "changed nothing". Process metrics are reported too: turns, tool
+calls, rejected calls, edits, and the peak conversation size.
+
+### Dependencies
+
+Nothing here is installed unless you select this suite, and then only for the
+language you actually run. `--agentic-lang python` provisions ruff and mypy;
+`--agentic-lang typescript` provisions the TypeScript compiler and uses node's
+own test runner. Neither touches the other, and no other suite touches either.
+
+### Useful flags
+
+| flag | meaning |
+|---|---|
+| `--agentic-lang` | restrict to one language, and provision only its toolchain |
+| `--agentic-max-turns` | give up after this many assistant turns (default 50) |
+| `--agentic-max-tokens` | cap generation per turn (default: no cap); bounds a model that falls into a repetition loop, but also cuts reasoning off mid-thought. Truncated turns are counted as `truncated_turns` |
+| `--agentic-context-limit` | abandon an episode once its context exceeds this many tokens |
+| `--agentic-transcripts DIR` | save each episode's full message history to `DIR/<task_id>.json`, plus a `DIR/<task_id>.html` showing every reasoning block, reply, tool call and tool result; the main report links to it |
+
+
+## Code suites (HumanEval, ClassEval)
+
+`--dataset humaneval` runs the 164 OpenAI HumanEval problems; `--dataset
+classeval` runs the 100 ClassEval class-generation tasks. Unlike the other
+suites these are not graded by matching text: the model's reply is parsed for a
+```` ```python ```` block, that code is combined with the task's own unit tests,
+and the result is **executed**.
+
+ClassEval asks for a whole class rather than one function, so it is scored the
+way upstream does - **class level** (every one of the task's tests passes, which
+is what feeds pass@k) and **method level** (what fraction of all 2196 test
+methods passed). The second number separates "generated nothing usable" from
+"got most of the class right", and both are printed at the end of a run.
+
+Give it more room to generate than HumanEval - a class with ten methods does not
+fit in a short budget:
+
+```bash
+python3 llama-eval.py \
+ --server http://localhost:8033 --model my-model \
+ --dataset classeval --threads 8 --n_predict 4096 --temperature 0
+```
+
+```bash
+# pass@1, greedy
+python3 llama-eval.py \
+ --server http://localhost:8033 --model my-model \
+ --dataset humaneval --threads 8 --n_predict 1024 --temperature 0
+
+# pass@10 -- each extra multiple of 164 is another sample per problem
+python3 llama-eval.py \
+ --server http://localhost:8033 --model my-model \
+ --dataset humaneval --n_cases 1640 --temperature 0.8 --top-p 0.95
+```
+
+`--grader-type exec` is the default for code suites, so neither it nor
+`--dataset-source` normally needs to be passed.
+
+### Execution sandbox - no Docker required
+
+Generated code is untrusted, but these suites need far less than a container to
+run safely. **The package set is derived from the dataset's own imports at
+startup**, not hardcoded and not taken from the upstream `requirements.txt` -
+which for ClassEval lists `torch`, `openai` and `transformers` that its *tests*
+never touch. So each suite installs exactly what it needs:
+
+| suite | third-party packages |
+|---|---|
+| HumanEval | **none** - every problem imports only stdlib (`typing, math, random, copy, string, collections, hashlib, re`) |
+| ClassEval | 13: `beautifulsoup4, gensim, lxml, netifaces, nltk, numpy, openpyxl, pandas, pillow, pypdf2, python-docx, reportlab, scipy` |
+
+Two of ClassEval's are *declared* rather than derived, because no import
+statement mentions them: `lxml` is selected by string
+(`BeautifulSoup(html, 'lxml')`, ClassEval_44 - without it that task silently
+drops from 23/23 to 7/23), and `scipy` is a hard gensim requirement. Note also
+that `PyPDF2` must **not** be modernised to `pypdf`: ClassEval_69's test imports
+`PdfFileReader`, a name pypdf removed.
+
+The venv is cached under `~/.cache/llama-eval/` (override with
+`LLAMA_EVAL_CACHE`) and rebuilt only when the package set, interpreter version
+or provisioning steps change; force one with `--rebuild-sandbox`. `uv` is used
+when available, which makes even the ClassEval venv build in seconds.
+
+**Interpreter selection.** A suite can prefer a different Python than the one
+running the tool, and declares it: ClassEval asks for 3.13, because gensim
+publishes no wheel past cp313 and its generated C no longer compiles on 3.14
+(it still uses `PyLongObject.ob_digit`). That interpreter is found automatically,
+including via `uv python`; `--sandbox-python` overrides it.
+
+3.14 is fully supported anyway. ClassEval uses exactly two gensim APIs
+(`utils.decode_htmlentities`, `matutils.unitvec`), both pure Python upstream, so
+on 3.14+ the suite drops gensim from the install set and provisions a small shim
+instead. Both paths score **100/100**.
+
+**Offline provisioning.** A suite that would otherwise fetch data at test time
+declares a provisioning step that runs once at venv build time, while the
+network is still up. ClassEval uses this to pre-seed exactly three nltk corpora
+(`punkt_tab`, `averaged_perceptron_tagger_eng`, `wordnet`, ~33 MB), so the tests
+themselves run with no network at all. Those are the *current* names - the
+dataset asks for `punkt` and `averaged_perceptron_tagger`, which nltk has since
+renamed, and seeding the legacy names too is not harmless: with a network
+available nltk considers them satisfied and re-fetches the modern ones anyway.
+The corpora are written *inside* the venv, not `~/nltk_data`, because `$HOME` is
+a tmpfs under the strongest isolation tier.
+
+Each candidate program runs in a throwaway directory as its own process group,
+under the strongest isolation the host offers:
+
+| tier | requires | isolates |
+|---|---|---|
+| `bwrap` | bubblewrap installed | read-only root, tmpfs `/tmp` and `$HOME`, **no network** |
+| `unshare` | `unshare --map-current-user` | **no network** |
+| `unshare-root` | any `unshare -r` | no network, but **runs as uid 0** - see below |
+| `rlimit` | always available | cpu/memory/file-size/process caps only |
+
+**Candidate code must not run as root**, which is why `unshare -r` is a
+last-resort tier that prints a warning. Root bypasses file permission bits, so a
+test asserting that writing to a `chmod 0444` file *fails* instead sees it
+succeed - ClassEval_50 scores 14/16 under uid 0 and 16/16 under uid 1000, with
+nothing in the output to suggest the sandbox caused it. Install bubblewrap.
+
+All tiers additionally cap CPU time, address space (4 GiB), file size and core
+dumps, and kill the whole process group on timeout (`--exec-timeout`, default
+15 s - raise it for ClassEval, whose tests do real work). Writes to the
+throwaway working directory are allowed, which is all the file-handling tasks in
+ClassEval need.
+
+Note that the weakest tier does not contain filesystem access. If you are
+scoring output from a model you do not trust, check the tier printed at startup.
+
+### Reproducibility
+
+Several tasks draw their test inputs from the *global, unseeded* RNG, so a
+verdict is not reproducible run to run - which adds noise exactly when you are
+comparing two models or two quantisations. `--exec-seed` (default `0`) seeds the
+RNG before each test; pass `--exec-seed -1` for the stock upstream behaviour.
+This does not change what the model is asked to compute.
+
+It matters more than it sounds. HumanEval/38, /50 and /53 are affected, and
+ClassEval_58 (duplicate mine placement) fails about 30% of the time on its own
+*reference* solution - measured here at 21/30 passes unseeded versus **30/30
+seeded**.
+
+### Dataset fidelity
+
+The canonical dataset runs **unmodified on current Python** - all 164 reference
+solutions pass on CPython 3.14 with no patches, so no forked or re-hosted copy
+is needed. `--dataset-source` accepts a local `.jsonl` or a HuggingFace repo id
+if you do want to point at a modified copy.
+
+HumanEval does contain a handful of long-standing defects where the *prompt*
+contradicts the graded test (notably HumanEval/47, /116 and /148). Those
+penalise a model that reads the docstring carefully, but correcting them makes
+the benchmark marginally easier and the resulting scores no longer comparable
+with published HumanEval numbers, so nothing is patched by default.
+
+ClassEval is different: the upstream data has defects that cap the achievable
+score below 100% for reasons unrelated to the model, so `--dataset classeval`
+defaults to a **patched fork**,
+[`ilintar/ClassEval`](https://huggingface.co/datasets/ilintar/ClassEval).
+Six edits over five tasks lift the reference solutions from 96/100 classes and
+2182/2196 test methods to **100/100 and 2196/2196**. Three repair tests nothing
+could pass:
+
+- **ClassEval_17** was a time bomb - a hard-coded 2024 date checked against
+ `datetime.now()`, unpassable since January 2024.
+- **ClassEval_48** asserted `get_hostname('0.0.0.0') == 'LAPTOP-2CS86KUM'`, the
+ dataset author's own machine name.
+- **ClassEval_31** compared floats with `assertEqual`, failing on a 4-ULP numpy
+ difference.
+
+The other three fix only the *reference* solutions (NumPy 2.0 removals in
+ClassEval_51, a 30%-flaky mine generator in ClassEval_58) and cannot affect a
+model's score. No `skeleton` was touched, so what the model is asked to write is
+byte-identical to upstream. Pass `--dataset-source FudanSELab/ClassEval` for the
+unpatched original.
diff --git a/examples/llama-eval/agentic_eval.py b/examples/llama-eval/agentic_eval.py
new file mode 100644
index 000000000..96629088a
--- /dev/null
+++ b/examples/llama-eval/agentic_eval.py
@@ -0,0 +1,918 @@
+#!/usr/bin/env python3
+"""Multi-turn agentic coding eval: tools over a real working tree.
+
+The existing code suites are single-shot -- one prompt, one reply, execute,
+grade. This one is not. The model is given a small but real repository and a
+deliberately narrow tool set, and has to find its way to the defect before it
+can fix it. That shape is what puts a run in the tens-of-thousands-of-tokens
+range without any artificial padding: the context grows because the model
+chose to read things.
+
+Three deliberate constraints:
+
+* No shell, and no way to run the tests. The only feedback channel is `lint`.
+ A syntax or type error can be driven out mechanically; a logic error has to
+ be reasoned about. Suites that let an agent iterate against the test suite
+ measure something closer to search than to understanding.
+* Two edit tools, one line-addressed and one content-addressed. Which one a
+ model reaches for, and whether it keeps line numbers straight after its own
+ earlier edit, is itself a signal.
+* Tests never touch the working tree. They are held outside it and copied in
+ after the agent has stopped, so they can be neither read nor edited.
+
+Everything here is imported lazily by llama-eval.py, and every dependency it
+needs is scoped to the language actually under test, so none of it is
+installed for anyone running the other suites.
+"""
+
+import html
+import json
+import os
+import re
+import shutil
+import tempfile
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Sequence, Tuple
+
+MAX_READ_LINES = 400
+MAX_TOOL_CHARS = 12000
+MAX_SEARCH_HITS = 60
+
+
+@dataclass
+class LangSpec:
+ """Everything language-specific: how to lint, how to test, what to install.
+
+ Kept as data so adding a language is a corpus concern rather than a harness
+ change, and so a run only ever provisions the toolchain for the language it
+ is actually evaluating.
+ """
+ name: str
+ exts: Tuple[str, ...]
+ packages: Tuple[str, ...] = ()
+ provision: Tuple[str, ...] = ()
+ env: Dict[str, str] = field(default_factory=dict)
+ address_space_limit: Optional[int] = 4 * 1024 * 1024 * 1024
+
+ def source_files(self, root: Path) -> List[Path]:
+ return sorted(p for p in root.rglob("*")
+ if p.is_file() and p.suffix in self.exts)
+
+
+PYTHON = LangSpec(
+ name="python",
+ exts=(".py",),
+ packages=("ruff==0.14.2", "mypy==1.18.2"),
+)
+
+TYPESCRIPT = LangSpec(
+ name="typescript",
+ exts=(".ts",),
+ address_space_limit=16 * 1024 * 1024 * 1024,
+ provision=(
+ "import os, subprocess, shutil, sys\n"
+ "npm = shutil.which('npm')\n"
+ "if not npm: sys.exit('npm not found on PATH; needed for the typescript suite')\n"
+ "dest = os.path.join(os.environ['SANDBOX_VENV'], 'node')\n"
+ "os.makedirs(dest, exist_ok=True)\n"
+ "r = subprocess.run([npm, 'install', '--silent', '--no-audit', '--no-fund',\n"
+ " '--prefix', dest, 'typescript@5.9.3'],\n"
+ " capture_output=True, text=True)\n"
+ "sys.exit(0 if r.returncode == 0 else (r.stderr or r.stdout)[-2000:])\n",
+ ),
+)
+
+LANGS: Dict[str, LangSpec] = {"python": PYTHON, "typescript": TYPESCRIPT}
+
+
+class PathEscape(Exception):
+ """A tool was handed a path outside the repository."""
+
+
+class Workspace:
+ """A private, writable copy of one task's repository.
+
+ Every tool path is resolved through here. Confinement is checked on the
+ fully-resolved path, so a symlink planted inside the tree cannot be used to
+ read or write outside it.
+ """
+
+ def __init__(self, root: Path):
+ self.root = root.resolve()
+ self.escape_attempts = 0
+
+ def resolve(self, rel: str) -> Path:
+ if rel is None or not str(rel).strip():
+ raise PathEscape("no path given")
+ raw = Path(str(rel))
+ base = raw if raw.is_absolute() else (self.root / raw)
+ candidate = base.resolve()
+ if candidate != self.root and self.root not in candidate.parents:
+ self.escape_attempts += 1
+ raise PathEscape(f"path {rel!r} is outside the repository")
+ return candidate
+
+ def rel(self, path: Path) -> str:
+ return str(path.relative_to(self.root))
+
+ def files(self) -> List[Path]:
+ return sorted(p for p in self.root.rglob("*")
+ if p.is_file() and "__pycache__" not in p.parts)
+
+ def snapshot(self) -> Dict[str, str]:
+ out = {}
+ for p in self.files():
+ try:
+ out[self.rel(p)] = p.read_text()
+ except (UnicodeDecodeError, OSError):
+ pass
+ return out
+
+
+def _clip(text: str, limit: int = MAX_TOOL_CHARS) -> str:
+ if len(text) <= limit:
+ return text
+ return text[:limit] + f"\n... [truncated, {len(text) - limit} more characters]"
+
+
+def _numbered(lines: Sequence[str], start: int) -> str:
+ width = len(str(start + len(lines) - 1))
+ return "\n".join(f"{start + i:>{width}}\t{ln}" for i, ln in enumerate(lines))
+
+
+def tool_schemas() -> List[Dict[str, Any]]:
+ """OpenAI-format function schemas for the whole tool set."""
+ def fn(name, desc, props, required=()):
+ return {"type": "function", "function": {
+ "name": name, "description": desc,
+ "parameters": {"type": "object", "properties": props,
+ "required": list(required)}}}
+
+ return [
+ fn("list_files", "List files in the repository.",
+ {"path": {"type": "string",
+ "description": "Directory relative to the repository root. "
+ "Omit for the whole tree."}}),
+ fn("read_file",
+ "Read a file. Output is prefixed with 1-based line numbers. Give "
+ "start_line/end_line to read only part of a large file.",
+ {"path": {"type": "string", "description": "File path relative to the repository root."},
+ "start_line": {"type": "integer", "description": "First line to read, 1-based, inclusive."},
+ "end_line": {"type": "integer", "description": "Last line to read, 1-based, inclusive."}},
+ ["path"]),
+ fn("search",
+ "Search the repository with a regular expression. Returns matching "
+ "lines with their file and line number.",
+ {"pattern": {"type": "string", "description": "Python regular expression."},
+ "path": {"type": "string", "description": "Limit the search to this subdirectory."},
+ "glob": {"type": "string", "description": "Limit to files matching this glob, e.g. '*.py'."}},
+ ["pattern"]),
+ fn("edit_lines",
+ "Replace an inclusive 1-based line range with new text. Line numbers "
+ "refer to the file as it is now, so re-read it after any edit that "
+ "changed the number of lines.",
+ {"path": {"type": "string", "description": "File path relative to the repository root."},
+ "start_line": {"type": "integer", "description": "First line to replace, 1-based, inclusive."},
+ "end_line": {"type": "integer", "description": "Last line to replace, 1-based, inclusive."},
+ "new_text": {"type": "string", "description": "Replacement text. Use an empty string to delete the range."}},
+ ["path", "start_line", "end_line", "new_text"]),
+ fn("edit_replace",
+ "Replace an exact snippet of text. The snippet must occur exactly "
+ "once in the file unless replace_all is set.",
+ {"path": {"type": "string", "description": "File path relative to the repository root."},
+ "old_text": {"type": "string", "description": "Exact text to find, including indentation."},
+ "new_text": {"type": "string", "description": "Replacement text."},
+ "replace_all": {"type": "boolean", "description": "Replace every occurrence instead of requiring exactly one."}},
+ ["path", "old_text", "new_text"]),
+ fn("write_file",
+ "Write a whole file, creating or overwriting it. Prefer the edit "
+ "tools for changes to existing files.",
+ {"path": {"type": "string", "description": "File path relative to the repository root."},
+ "content": {"type": "string", "description": "Full new file contents."}},
+ ["path", "content"]),
+ fn("lint",
+ "Run the language's linter and type checker over the repository and "
+ "return the diagnostics. This is the only feedback available; the "
+ "test suite cannot be run.",
+ {"path": {"type": "string", "description": "Limit to this file or directory."}}),
+ fn("finish",
+ "Declare the task complete and end the session. This is the only "
+ "way to end it; call this once the change is made.",
+ {"summary": {"type": "string", "description": "Brief description of the change made."}}),
+ ]
+
+
+class ToolBox:
+ """Executes tool calls against one workspace. Returns plain text results.
+
+ Failures are reported as ordinary text rather than raised, because how a
+ model recovers from a rejected edit is part of what is being measured.
+ """
+
+ def __init__(self, ws: Workspace, linter=None):
+ self.ws = ws
+ self.linter = linter
+ self.calls = 0
+ self.errors = 0
+ self.edits = 0
+ self.finished = False
+ self.finish_summary: Optional[str] = None
+ self.per_tool: Dict[str, int] = {}
+
+ def dispatch(self, name: str, args: Dict[str, Any]) -> str:
+ self.calls += 1
+ self.per_tool[name] = self.per_tool.get(name, 0) + 1
+ handler = getattr(self, f"_t_{name}", None)
+ if handler is None:
+ result = f"Error: unknown tool {name!r}."
+ else:
+ try:
+ result = handler(args)
+ except PathEscape as e:
+ result = f"Error: {e}"
+ except KeyError as e:
+ result = f"Error: missing required argument {e}."
+ except Exception as e:
+ result = f"Error: {type(e).__name__}: {e}"
+ if result.startswith("Error:"):
+ self.errors += 1
+ return _clip(result)
+
+ def _t_list_files(self, a):
+ base = self.ws.resolve(a.get("path") or ".")
+ if not base.exists():
+ return f"Error: {a.get('path')!r} does not exist."
+ paths = [p for p in self.ws.files() if p == base or base in p.parents]
+ if not paths:
+ return "(no files)"
+ return "\n".join(f"{self.ws.rel(p)}\t{p.stat().st_size} bytes" for p in paths)
+
+ def _t_read_file(self, a):
+ path = self.ws.resolve(a["path"])
+ if not path.is_file():
+ return f"Error: {a['path']!r} is not a file."
+ lines = path.read_text().splitlines()
+ start = max(1, int(a.get("start_line") or 1))
+ end = int(a.get("end_line") or len(lines))
+ end = min(end, len(lines))
+ if start > len(lines):
+ return f"Error: {a['path']!r} has only {len(lines)} lines."
+ window = lines[start - 1:end]
+ note = ""
+ if len(window) > MAX_READ_LINES:
+ window = window[:MAX_READ_LINES]
+ note = (f"\n... [showing {MAX_READ_LINES} of {end - start + 1} requested "
+ f"lines; re-read with start_line/end_line for the rest]")
+ header = f"{a['path']} ({len(lines)} lines total)\n"
+ return header + _numbered(window, start) + note
+
+ def _t_search(self, a):
+ try:
+ rx = re.compile(a["pattern"])
+ except re.error as e:
+ return f"Error: bad regular expression: {e}"
+ base = self.ws.resolve(a.get("path") or ".")
+ glob = a.get("glob")
+ hits, scanned = [], 0
+ for p in self.ws.files():
+ if not (p == base or base in p.parents):
+ continue
+ if glob and not p.match(glob):
+ continue
+ try:
+ text = p.read_text()
+ except (UnicodeDecodeError, OSError):
+ continue
+ scanned += 1
+ for i, line in enumerate(text.splitlines(), 1):
+ if rx.search(line):
+ hits.append(f"{self.ws.rel(p)}:{i}:{line.strip()[:200]}")
+ if len(hits) >= MAX_SEARCH_HITS:
+ return ("\n".join(hits) +
+ f"\n... [stopped at {MAX_SEARCH_HITS} matches; narrow the pattern]")
+ if not hits:
+ return f"No matches for {a['pattern']!r} in {scanned} file(s)."
+ return "\n".join(hits)
+
+ def _t_edit_lines(self, a):
+ path = self.ws.resolve(a["path"])
+ if not path.is_file():
+ return f"Error: {a['path']!r} is not a file."
+ lines = path.read_text().splitlines()
+ start, end = int(a["start_line"]), int(a["end_line"])
+ if start < 1 or start > len(lines):
+ return (f"Error: start_line {start} out of range; "
+ f"{a['path']!r} has {len(lines)} lines.")
+ if end < start or end > len(lines):
+ return (f"Error: end_line {end} out of range; must be between "
+ f"{start} and {len(lines)}.")
+ new = str(a["new_text"]).splitlines()
+ path.write_text("\n".join(lines[:start - 1] + new + lines[end:]) + "\n")
+ self.edits += 1
+ delta = len(new) - (end - start + 1)
+ return (f"Replaced lines {start}-{end} of {a['path']} "
+ f"({end - start + 1} -> {len(new)} lines, {delta:+d}). "
+ f"Later line numbers have shifted." if delta else
+ f"Replaced lines {start}-{end} of {a['path']}.")
+
+ def _t_edit_replace(self, a):
+ path = self.ws.resolve(a["path"])
+ if not path.is_file():
+ return f"Error: {a['path']!r} is not a file."
+ text = path.read_text()
+ old, new = str(a["old_text"]), str(a["new_text"])
+ if not old:
+ return "Error: old_text must not be empty."
+ n = text.count(old)
+ if n == 0:
+ return (f"Error: old_text not found in {a['path']}. It must match "
+ f"exactly, including indentation.")
+ if n > 1 and not a.get("replace_all"):
+ return (f"Error: old_text occurs {n} times in {a['path']}. Include "
+ f"more surrounding context to make it unique, or set "
+ f"replace_all.")
+ path.write_text(text.replace(old, new))
+ self.edits += 1
+ return f"Replaced {n} occurrence(s) in {a['path']}."
+
+ def _t_write_file(self, a):
+ path = self.ws.resolve(a["path"])
+ path.parent.mkdir(parents=True, exist_ok=True)
+ existed = path.is_file()
+ path.write_text(str(a["content"]))
+ self.edits += 1
+ return f"{'Overwrote' if existed else 'Created'} {a['path']}."
+
+ def _t_lint(self, a):
+ if self.linter is None:
+ return "Error: no linter is configured for this task."
+ target = a.get("path")
+ if target:
+ self.ws.resolve(target)
+ return self.linter(target)
+
+ def _t_finish(self, a):
+ self.finished = True
+ self.finish_summary = a.get("summary")
+ return "Task marked complete."
+
+
+PY_TEST_RUNNER = r'''
+import json, os, random, sys, unittest
+sys.path.insert(0, os.getcwd())
+random.seed(0)
+
+class Collect(unittest.TestResult):
+ def __init__(self):
+ super().__init__()
+ self.outcomes = {}
+ def addSuccess(self, t): self.outcomes[t.id()] = "pass"
+ def addFailure(self, t, e): self.outcomes[t.id()] = "fail"
+ def addError(self, t, e): self.outcomes[t.id()] = "error"
+ def addSkip(self, t, r): self.outcomes[t.id()] = "skip"
+ def addExpectedFailure(self, t, e): self.outcomes[t.id()] = "pass"
+ def addUnexpectedSuccess(self, t): self.outcomes[t.id()] = "fail"
+
+def flatten(s):
+ for t in s:
+ if isinstance(t, unittest.TestSuite):
+ yield from flatten(t)
+ else:
+ yield t
+
+out = {"tests": {}}
+try:
+ suite = unittest.TestLoader().discover(os.environ.get("TEST_DIR", "tests"),
+ top_level_dir=".")
+ res = Collect()
+ for t in flatten(suite):
+ tid = t.id()
+ try:
+ t.run(res)
+ except Exception:
+ pass
+ res.outcomes.setdefault(tid, "error")
+ out["tests"] = res.outcomes
+except Exception as e:
+ out["error"] = f"{type(e).__name__}: {e}"
+
+print("###RESULT###")
+print(json.dumps(out))
+'''
+
+
+def _node_roots() -> List[Path]:
+ """Directories that must stay visible for node to start under bwrap."""
+ node = shutil.which("node")
+ if not node:
+ return []
+ real = Path(node).resolve()
+ return [Path(node).parent.parent, real.parent.parent]
+
+
+class Runner:
+ """Runs the linter and the hidden tests for one language, in the sandbox."""
+
+ def __init__(self, sandbox, lang: LangSpec, timeout: float = 90.0):
+ self.sandbox = sandbox
+ self.lang = lang
+ self.timeout = timeout
+
+ def _binds(self) -> List[Path]:
+ return _node_roots() if self.lang.name == "typescript" else []
+
+ def _env(self) -> Dict[str, str]:
+ if self.lang.name != "typescript":
+ return {}
+ node = shutil.which("node")
+ return {"PATH": f"{Path(node).parent}:/usr/bin:/bin"} if node else {}
+
+ def lint(self, workdir: Path, target: Optional[str] = None) -> str:
+ if self.lang.name == "python":
+ return self._lint_python(workdir, target)
+ return self._lint_typescript(workdir, target)
+
+ def _lint_python(self, workdir: Path, target: Optional[str]) -> str:
+ venv = self.sandbox.venv_dir
+ tgt = target or "."
+ chunks = []
+ r = self.sandbox.run_argv(
+ [str(venv / "bin" / "ruff"), "check", "--no-cache",
+ "--output-format", "concise", tgt],
+ workdir, timeout=self.timeout)
+ text = (r.stdout or r.stderr).strip()
+ if text and "All checks passed" not in text:
+ chunks.append(text)
+ r = self.sandbox.run_argv(
+ [str(venv / "bin" / "mypy"), "--no-color-output", "--no-error-summary",
+ "--cache-dir", "/dev/null", "--ignore-missing-imports", tgt],
+ workdir, timeout=self.timeout)
+ text = (r.stdout or r.stderr).strip()
+ if text and not text.startswith("Success:"):
+ chunks.append(text)
+ return "\n".join(chunks) if chunks else "No problems found."
+
+ def _lint_typescript(self, workdir: Path, target: Optional[str]) -> str:
+ tsc = self.sandbox.venv_dir / "node" / "node_modules" / "typescript" / "bin" / "tsc"
+ files = [str(p.relative_to(workdir)) for p in sorted(workdir.rglob("*.ts"))
+ if "node_modules" not in p.parts]
+ if target:
+ t = (workdir / target).resolve()
+ files = [f for f in files if (workdir / f).resolve() == t
+ or t in (workdir / f).resolve().parents]
+ if not files:
+ return "No TypeScript files to check."
+ r = self.sandbox.run_argv(
+ ["node", str(tsc), "--noEmit", "--strict", "--target", "es2022",
+ "--module", "esnext", "--moduleResolution", "bundler",
+ "--allowImportingTsExtensions", *files],
+ workdir, timeout=self.timeout, env=self._env(), ro_binds=self._binds())
+ out = (r.stdout or "") + (r.stderr or "")
+ return out.strip() or "No problems found."
+
+ def test(self, workdir: Path) -> Dict[str, Any]:
+ if self.lang.name == "python":
+ return self._test_python(workdir)
+ return self._test_typescript(workdir)
+
+ def _test_python(self, workdir: Path) -> Dict[str, Any]:
+ runner = workdir / "__run_tests.py"
+ runner.write_text(PY_TEST_RUNNER)
+ r = self.sandbox.run_argv(
+ [str(self.sandbox.python), "-I", "-B", str(runner)],
+ workdir, timeout=self.timeout, env={"TEST_DIR": "tests"})
+ return self._parse_json_result(r)
+
+ def _parse_json_result(self, r) -> Dict[str, Any]:
+ if r.status == "timeout":
+ return {"tests": {}, "error": "timeout"}
+ marker = "###RESULT###"
+ if marker not in (r.stdout or ""):
+ return {"tests": {},
+ "error": f"no result: {(r.stderr or r.stdout or '')[-400:]}"}
+ try:
+ return json.loads(r.stdout.split(marker, 1)[1].strip())
+ except Exception as e:
+ return {"tests": {}, "error": f"unparseable result: {e}"}
+
+ def _test_typescript(self, workdir: Path) -> Dict[str, Any]:
+ tests = sorted(p for p in (workdir / "tests").rglob("*.ts")) \
+ if (workdir / "tests").is_dir() else []
+ if not tests:
+ return {"tests": {}, "error": "no test files"}
+ r = self.sandbox.run_argv(
+ ["node", "--test", "--experimental-strip-types",
+ "--test-reporter=tap",
+ *[str(p.relative_to(workdir)) for p in tests]],
+ workdir, timeout=self.timeout, env=self._env(), ro_binds=self._binds())
+ return self._parse_tap(r)
+
+ @staticmethod
+ def _parse_tap(r) -> Dict[str, Any]:
+ if r.status == "timeout":
+ return {"tests": {}, "error": "timeout"}
+ text = (r.stdout or "") + "\n" + (r.stderr or "")
+ outcomes: Dict[str, str] = {}
+ for line in text.splitlines():
+ m = re.match(r"\s*(not )?ok\s+\d+\s*-\s*(.+?)\s*$", line)
+ if not m:
+ continue
+ name = m.group(2).strip()
+ if name.endswith(" # SKIP") or " # SKIP" in name:
+ outcomes[name.split(" # ")[0]] = "skip"
+ continue
+ verdict = "fail" if m.group(1) else "pass"
+ if outcomes.get(name) != "fail":
+ outcomes[name] = verdict
+ if not outcomes:
+ return {"tests": {},
+ "error": f"no TAP output: {text.strip()[-400:]}"}
+ return {"tests": outcomes}
+
+
+SYSTEM_PROMPT = """You are an expert {lang} engineer working in an existing repository.
+
+Resolve the user's request by editing files with the tools provided.
+
+How this session works:
+- You are working autonomously. There is no human watching; nobody will answer
+ questions, approve a plan, or read progress reports.
+- Every reply you send must contain at least one tool call until the task is
+ done. A reply that contains only text does nothing: no work happens and the
+ turn is wasted. Do not stop to describe what you are about to do; do it.
+- The session ends only when you call `finish`. Call it once the change is
+ made, and not before. Only the files as you leave them are evaluated.
+
+Important constraints:
+- You cannot run the code, the tests, or any shell command. The only feedback
+ available is the `lint` tool, which reports syntax and type problems.
+- The repository already has a test suite that will be run against your work
+ after you finish, but you cannot see it or run it. Do not create test files.
+- Read enough of the code to understand it before editing. A change that only
+ makes the symptom go away will usually fail the tests.
+- Make the smallest correct change. Do not reformat or refactor unrelated code.
+
+Keep calling tools until the change is complete, then call `finish`."""
+
+
+@dataclass
+class AgenticTask:
+ task_id: str
+ lang: str
+ category: str
+ difficulty: int
+ instruction: str
+ files: Dict[str, str]
+ tests: Dict[str, str]
+ gold: Dict[str, str] = field(default_factory=dict)
+ fail_to_pass: List[str] = field(default_factory=list)
+ pass_to_pass: List[str] = field(default_factory=list)
+
+ @classmethod
+ def from_record(cls, rec: Dict[str, Any]) -> "AgenticTask":
+ def as_map(v):
+ return v if isinstance(v, dict) else json.loads(v or "{}")
+ def as_list(v):
+ return v if isinstance(v, list) else json.loads(v or "[]")
+ return cls(
+ task_id=rec["task_id"], lang=rec["lang"], category=rec["category"],
+ difficulty=int(rec.get("difficulty", 3)),
+ instruction=rec["instruction"],
+ files=as_map(rec["files"]), tests=as_map(rec["tests"]),
+ gold=as_map(rec.get("gold")),
+ fail_to_pass=as_list(rec.get("fail_to_pass")),
+ pass_to_pass=as_list(rec.get("pass_to_pass")),
+ )
+
+ @property
+ def spec(self) -> LangSpec:
+ return LANGS[self.lang]
+
+
+def materialise(files: Dict[str, str], dest: Path) -> Path:
+ """Write a file map out as a real directory tree."""
+ dest.mkdir(parents=True, exist_ok=True)
+ for rel, content in files.items():
+ p = dest / rel
+ p.parent.mkdir(parents=True, exist_ok=True)
+ p.write_text(content)
+ return dest
+
+
+def repo_overview(files: Dict[str, str]) -> str:
+ """A compact tree, so a run is not lost merely to not finding the files."""
+ lines = []
+ for rel in sorted(files):
+ n = files[rel].count("\n") + 1
+ lines.append(f" {rel} ({n} lines)")
+ return "\n".join(lines)
+
+
+def user_prompt(task: AgenticTask) -> str:
+ return (f"Repository files:\n{repo_overview(task.files)}\n\n"
+ f"Task:\n{task.instruction}")
+
+
+@dataclass
+class EpisodeResult:
+ stop_reason: str = "unknown"
+ turns: int = 0
+ tool_calls: int = 0
+ tool_errors: int = 0
+ edits: int = 0
+ escape_attempts: int = 0
+ finished: bool = False
+ bad_tool_json: int = 0
+ nudges: int = 0
+ prompt_tokens: int = 0
+ completion_tokens: int = 0
+ peak_context: int = 0
+ truncated_turns: int = 0
+ per_tool: Dict[str, int] = field(default_factory=dict)
+ error: Optional[str] = None
+
+
+NUDGE = ("Your last reply contained no tool call, so nothing happened. There is "
+ "no human in this session to respond to text. If your change is "
+ "complete, call `finish`. Otherwise continue working by calling the "
+ "tools; do not reply with text alone.")
+
+
+def run_episode(task: AgenticTask, toolbox: ToolBox, chat,
+ max_turns: int = 50,
+ context_limit: Optional[int] = None,
+ max_nudges: int = 3) -> Tuple[EpisodeResult, List[Dict]]:
+ """Drive one multi-turn tool-calling episode.
+
+ `chat(messages, tools) -> response dict` is supplied by the caller so this
+ module stays free of any HTTP concerns.
+ """
+ tools = tool_schemas()
+ messages: List[Dict[str, Any]] = [
+ {"role": "system", "content": SYSTEM_PROMPT.format(lang=task.spec.name)},
+ {"role": "user", "content": user_prompt(task)},
+ ]
+ res = EpisodeResult()
+
+ for turn in range(max_turns):
+ res.turns = turn + 1
+ try:
+ reply = chat(messages, tools)
+ except Exception as e:
+ res.stop_reason, res.error = "request_failed", f"{type(e).__name__}: {e}"
+ break
+
+ usage = reply.get("usage") or {}
+ pt = int(usage.get("prompt_tokens") or 0)
+ ct = int(usage.get("completion_tokens") or 0)
+ res.prompt_tokens += pt
+ res.completion_tokens += ct
+ res.peak_context = max(res.peak_context, pt + ct)
+
+ choices = reply.get("choices") or []
+ if not choices:
+ res.stop_reason, res.error = "no_choices", json.dumps(reply)[:300]
+ break
+ msg = choices[0].get("message") or {}
+ if choices[0].get("finish_reason") == "length":
+ res.truncated_turns += 1
+ calls = msg.get("tool_calls") or []
+
+ reasoning = msg.get("reasoning_content")
+ messages.append({
+ "role": "assistant",
+ "content": msg.get("content") or "",
+ **({"reasoning_content": reasoning} if reasoning else {}),
+ **({"tool_calls": calls} if calls else {}),
+ })
+
+ if not calls:
+ if toolbox.finished:
+ res.stop_reason = "finished"
+ break
+ if res.nudges >= max_nudges:
+ res.stop_reason = "no_tool_calls"
+ break
+ res.nudges += 1
+ messages.append({"role": "user", "content": NUDGE})
+ continue
+
+ for call in calls:
+ fn = call.get("function") or {}
+ name = fn.get("name") or ""
+ raw = fn.get("arguments")
+ if isinstance(raw, dict):
+ args = raw
+ else:
+ try:
+ args = json.loads(raw or "{}")
+ except (json.JSONDecodeError, TypeError):
+ res.bad_tool_json += 1
+ args = None
+ output = (toolbox.dispatch(name, args) if args is not None else
+ f"Error: arguments for {name!r} were not valid JSON.")
+ messages.append({"role": "tool",
+ "tool_call_id": call.get("id") or f"call_{res.tool_calls}",
+ "content": output})
+
+ if toolbox.finished:
+ res.stop_reason = "finished"
+ break
+ if context_limit and res.peak_context > context_limit:
+ res.stop_reason = "context_limit"
+ break
+ else:
+ res.stop_reason = "max_turns"
+
+ res.tool_calls = toolbox.calls
+ res.tool_errors = toolbox.errors
+ res.edits = toolbox.edits
+ res.escape_attempts = toolbox.ws.escape_attempts
+ res.finished = toolbox.finished
+ res.per_tool = dict(toolbox.per_tool)
+ return res, messages
+
+
+def run_tests_against(files: Dict[str, str], task: AgenticTask, runner: Runner,
+ workroot: Path, label: str) -> Dict[str, str]:
+ """Materialise a tree, drop the hidden tests in, and run them.
+
+ The tests are only ever written into this throwaway copy, never into the
+ tree the agent worked in, so there is no window in which they could have
+ been read or edited.
+ """
+ dest = workroot / f"grade-{label}"
+ shutil.rmtree(dest, ignore_errors=True)
+ materialise(files, dest)
+ materialise(task.tests, dest)
+ return runner.test(dest).get("tests", {})
+
+
+def score(task: AgenticTask, outcomes: Dict[str, str]) -> Dict[str, Any]:
+ """Resolved means every target test passes and nothing previously passing broke."""
+ def passing(tid):
+ return outcomes.get(tid) == "pass"
+
+ f2p_ok = [t for t in task.fail_to_pass if passing(t)]
+ p2p_ok = [t for t in task.pass_to_pass if passing(t)]
+ regressions = [t for t in task.pass_to_pass if not passing(t)]
+ resolved = (len(f2p_ok) == len(task.fail_to_pass)
+ and not regressions and bool(task.fail_to_pass))
+ return {
+ "resolved": resolved,
+ "f2p_passed": len(f2p_ok), "f2p_total": len(task.fail_to_pass),
+ "p2p_passed": len(p2p_ok), "p2p_total": len(task.pass_to_pass),
+ "regressions": regressions[:10],
+ "n_regressions": len(regressions),
+ "partial": (len(f2p_ok) / len(task.fail_to_pass)) if task.fail_to_pass else 0.0,
+ }
+
+
+def make_sandbox(lang_name: str, sandbox_cls, cache_root=None, base_python=None):
+ """A sandbox carrying only the toolchain for `lang_name`.
+
+ Keyed per language rather than per suite: running the Python tasks must
+ never provision node, and vice versa.
+ """
+ lang = LANGS[lang_name]
+ return sandbox_cls(
+ name=f"agentic-{lang_name}",
+ packages=list(lang.packages),
+ provision=list(lang.provision),
+ env=dict(lang.env),
+ address_space_limit=lang.address_space_limit,
+ cache_root=cache_root,
+ base_python=base_python,
+ )
+
+
+def run_task(task: AgenticTask, chat, sandbox, workroot: Optional[Path] = None,
+ max_turns: int = 50, context_limit: Optional[int] = None,
+ test_timeout: float = 90.0) -> Dict[str, Any]:
+ """One full attempt: fresh workspace, agent episode, then hidden tests."""
+ root = Path(tempfile.mkdtemp(prefix=f"agentic-{task.task_id}-",
+ dir=str(workroot) if workroot else None))
+ try:
+ ws_root = root / "repo"
+ materialise(task.files, ws_root)
+ ws = Workspace(ws_root)
+ runner = Runner(sandbox, task.spec, timeout=test_timeout)
+ toolbox = ToolBox(ws, linter=lambda t: runner.lint(ws_root, t))
+
+ episode, messages = run_episode(task, toolbox, chat,
+ max_turns=max_turns,
+ context_limit=context_limit)
+ final = ws.snapshot()
+ changed = sorted(k for k in final
+ if final[k] != task.files.get(k)) + \
+ sorted(k for k in task.files if k not in final)
+
+ outcomes = run_tests_against(final, task, runner, root, "final")
+ result = score(task, outcomes)
+ result.update({
+ "task_id": task.task_id, "lang": task.lang,
+ "category": task.category, "difficulty": task.difficulty,
+ "changed_files": changed,
+ "episode": {k: v for k, v in vars(episode).items()},
+ "messages": messages,
+ })
+ return result
+ finally:
+ shutil.rmtree(root, ignore_errors=True)
+
+
+TRANSCRIPT_CSS = """
+body { font-family: system-ui, sans-serif; margin: 0; padding: 16px; background: #fff; color: #222; max-width: 1100px; }
+h1 { font-size: 16px; margin: 0 0 4px; }
+.meta { font-size: 13px; color: #555; font-family: 'SF Mono', 'Menlo', 'Consolas', monospace; margin-bottom: 12px; white-space: pre-wrap; }
+.msg { border: 1px solid #e1e4e8; border-left-width: 4px; margin: 8px 0; padding: 6px 10px; background: #fafbfc; }
+.msg .role { font-size: 12px; font-weight: 600; color: #555; margin-bottom: 4px; }
+.system { border-left-color: #888; }
+.user { border-left-color: #0969da; }
+.nudge { border-left-color: #9a6700; }
+.assistant { border-left-color: #1a7f37; background: #fff; }
+.tool { border-left-color: #8250df; }
+.tool.err { border-left-color: #cf222e; }
+pre { font-family: 'SF Mono', 'Menlo', 'Consolas', monospace; font-size: 12px; background: #fff; border: 1px solid #e1e4e8; padding: 6px 8px; margin: 4px 0; white-space: pre-wrap; word-wrap: break-word; }
+.reasoning pre { color: #57606a; background: #f6f8fa; }
+.call { font-size: 12px; font-weight: 600; color: #8250df; margin-top: 6px; }
+details > summary { cursor: pointer; font-size: 12px; color: #555; }
+.empty { font-size: 12px; color: #888; font-style: italic; }
+"""
+
+
+def _pre(text: str) -> str:
+ return f"<pre>{html.escape(text)}</pre>"
+
+
+def _render_call(call: Dict[str, Any]) -> str:
+ fn = call.get("function") or {}
+ raw = fn.get("arguments")
+ try:
+ args = raw if isinstance(raw, dict) else json.loads(raw or "{}")
+ shown = json.dumps(args, indent=2, ensure_ascii=False)
+ except (json.JSONDecodeError, TypeError):
+ shown = f"(invalid JSON)\n{raw}"
+ return (f'<div class="call">→ {html.escape(fn.get("name") or "?")} '
+ f'<span style="font-weight:400;color:#888">{html.escape(call.get("id") or "")}</span></div>'
+ + _pre(shown))
+
+
+def render_transcript_html(messages: List[Dict[str, Any]], title: str,
+ meta: str = "") -> str:
+ """One page per episode: every turn, with reasoning, tool calls and results."""
+ names = {}
+ for m in messages:
+ for c in m.get("tool_calls") or []:
+ names[c.get("id")] = (c.get("function") or {}).get("name") or "?"
+
+ parts = []
+ turn = 0
+ for m in messages:
+ role = m.get("role") or "?"
+ content = m.get("content") or ""
+ if role == "assistant":
+ turn += 1
+ body = ""
+ if m.get("reasoning_content"):
+ body += ('<details class="reasoning" open><summary>reasoning '
+ f'({len(m["reasoning_content"])} chars)</summary>'
+ + _pre(m["reasoning_content"]) + "</details>")
+ body += _pre(content) if content.strip() else ""
+ body += "".join(_render_call(c) for c in m.get("tool_calls") or [])
+ if not body:
+ body = '<div class="empty">(empty reply)</div>'
+ parts.append(f'<div class="msg assistant"><div class="role">assistant '
+ f'· turn {turn}</div>{body}</div>')
+ elif role == "tool":
+ name = names.get(m.get("tool_call_id"), "?")
+ err = " err" if content.startswith("Error") else ""
+ parts.append(f'<div class="msg tool{err}"><div class="role">tool result '
+ f'· {html.escape(name)}</div>{_pre(content)}</div>')
+ elif role == "system":
+ parts.append('<div class="msg system"><details><summary>system prompt'
+ f'</summary>{_pre(content)}</details></div>')
+ else:
+ cls = "nudge" if content == NUDGE else role
+ label = "user (nudge)" if content == NUDGE else role
+ parts.append(f'<div class="msg {html.escape(cls)}"><div class="role">'
+ f'{html.escape(label)}</div>{_pre(content)}</div>')
+
+ return (f'<!DOCTYPE html>\n<html>\n<head>\n<meta charset="UTF-8">\n'
+ f'<title>{html.escape(title)}</title>\n<style>{TRANSCRIPT_CSS}</style>\n'
+ f'</head>\n<body>\n<h1>{html.escape(title)}</h1>\n'
+ f'<div class="meta">{html.escape(meta)}</div>\n'
+ + "\n".join(parts) + "\n</body>\n</html>\n")
+
+
+def write_transcript(messages: List[Dict[str, Any]], dest: Path, title: str,
+ meta: str = "") -> Path:
+ """Write `<dest>.json` (the raw message history) and `<dest>.html`."""
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ json_path = dest.with_name(dest.name + ".json")
+ with open(json_path, "w") as f:
+ json.dump(messages, f, indent=2, ensure_ascii=False)
+ with open(dest.with_name(dest.name + ".html"), "w") as f:
+ f.write(render_transcript_html(messages, title, meta))
+ return json_path
diff --git a/examples/llama-eval/eval_sandbox.py b/examples/llama-eval/eval_sandbox.py
new file mode 100644
index 000000000..b13bc9d71
--- /dev/null
+++ b/examples/llama-eval/eval_sandbox.py
@@ -0,0 +1,488 @@
+#!/usr/bin/env python3
+"""Lightweight sandboxed execution for code-generation evals (HumanEval, ClassEval).
+
+Replaces the usual Docker-per-suite setup with a cached venv plus whatever
+process isolation the host can give us without root:
+
+ tier 1 bubblewrap read-only root, tmpfs /tmp and $HOME, no network
+ tier 2 unshare -rn no network (home still visible)
+ tier 3 rlimits only last resort, still capped on cpu/memory/procs/files
+
+Every tier also applies rlimits and runs the candidate program in a throwaway
+working directory as its own process group, so a runaway solution is killed
+along with anything it spawned.
+"""
+
+import json
+import os
+import re
+import resource
+import shutil
+import signal
+import subprocess
+import sys
+import tempfile
+import time
+import venv
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Dict, List, Optional, Sequence
+
+DEFAULT_CACHE_ROOT = Path(
+ os.environ.get("LLAMA_EVAL_CACHE", Path.home() / ".cache" / "llama-eval")
+)
+
+LIMIT_AS_BYTES = 4 * 1024 * 1024 * 1024
+LIMIT_CPU_SECONDS_SLACK = 5
+LIMIT_NPROC = 256
+LIMIT_FSIZE_BYTES = 64 * 1024 * 1024
+
+
+@dataclass
+class ExecResult:
+ passed: bool
+ status: str
+ stdout: str = ""
+ stderr: str = ""
+ returncode: Optional[int] = None
+ duration: float = 0.0
+
+ def summary(self, max_chars: int = 400) -> str:
+ """Short human-readable reason, for grader logs."""
+ if self.passed:
+ return "ok"
+ if self.status == "timeout":
+ return "timeout"
+ tail = (self.stderr or self.stdout).strip()
+ if not tail:
+ return f"{self.status} (rc={self.returncode})"
+ lines = [ln for ln in tail.splitlines() if ln.strip()]
+ reason = lines[-1] if lines else tail
+ return reason[:max_chars]
+
+
+def resolve_python(spec: Optional[str]) -> Optional[str]:
+ """Turn '3.13' or a path into a usable interpreter, or None if unavailable.
+
+ Suites can need an older interpreter than the one running this tool, because
+ a dependency has no wheel (or no working source build) for the newest
+ CPython. Accepts an explicit path, a bare version, or an executable name.
+ """
+ if not spec:
+ return None
+
+ candidate = Path(spec).expanduser()
+ if candidate.exists() and candidate.is_file():
+ return str(candidate)
+
+ uv = shutil.which("uv")
+ if uv and re.fullmatch(r"\d+\.\d+(\.\d+)?", spec):
+ try:
+ r = subprocess.run([uv, "python", "find", spec],
+ capture_output=True, text=True, timeout=120)
+ path = r.stdout.strip()
+ if r.returncode == 0 and path and Path(path).exists():
+ return path
+ except Exception:
+ pass
+
+ return shutil.which(spec) or shutil.which(f"python{spec}")
+
+
+def _detect_isolation(probe_python: Optional[str] = None,
+ ro_binds: Sequence[Path] = ()) -> str:
+ """Pick the strongest isolation tier this host supports, once.
+
+ The probe must run the *same* interpreter the suite will, with the same
+ binds. Probing a bare sys.executable silently downgrades the tier whenever
+ this tool is itself run from a venv under $HOME -- which is the usual case,
+ since it needs `datasets`. bwrap tmpfs-mounts the home directory, the
+ interpreter disappears, the probe fails, and the sandbox falls back to a
+ tier that does not hide $HOME at all. Nothing about the results looks wrong
+ when that happens, so it has to be got right here.
+ """
+ py = probe_python or sys.executable
+ probe = "import sys; sys.exit(0)"
+
+ if shutil.which("bwrap"):
+ try:
+ r = subprocess.run(
+ _bwrap_argv(Path.cwd(), [py, "-I", "-c", probe], ro_binds=ro_binds),
+ capture_output=True, timeout=20,
+ )
+ if r.returncode == 0:
+ return "bwrap"
+ except Exception:
+ pass
+
+ if shutil.which("unshare"):
+ for tier, argv in (
+ ("unshare", ["unshare", "--user", "--map-current-user", "--net"]),
+ ("unshare-root", ["unshare", "-rn"]),
+ ):
+ try:
+ r = subprocess.run(
+ [*argv, py, "-I", "-c", probe],
+ capture_output=True, timeout=20,
+ )
+ if r.returncode == 0:
+ return tier
+ except Exception:
+ continue
+
+ return "rlimit"
+
+
+UNSHARE_ARGV = {
+ "unshare": ["unshare", "--user", "--map-current-user", "--net"],
+ "unshare-root": ["unshare", "-rn"],
+}
+
+
+def _under(path: Path, parent: str) -> bool:
+ return bool(parent) and (str(path) == parent
+ or str(path).startswith(parent.rstrip("/") + os.sep))
+
+
+def _bwrap_argv(workdir: Path, inner: Sequence[str],
+ ro_binds: Sequence[Path] = ()) -> List[str]:
+ home = os.path.expanduser("~")
+ argv = [
+ "bwrap",
+ "--ro-bind", "/", "/",
+ "--dev", "/dev",
+ "--proc", "/proc",
+ "--tmpfs", "/tmp",
+ ]
+ if home and home != "/":
+ argv += ["--tmpfs", home]
+ for path in ro_binds:
+ if _under(path, home):
+ argv += ["--ro-bind", str(path), str(path)]
+ argv += [
+ "--bind", str(workdir), str(workdir),
+ "--chdir", str(workdir),
+ "--unshare-all",
+ "--die-with-parent",
+ "--new-session",
+ "--",
+ ]
+ return argv + list(inner)
+
+
+def _nproc_ceiling() -> int:
+ """A process cap that bounds a fork bomb without tripping on existing load.
+
+ RLIMIT_NPROC is enforced per real UID across the whole system, not per
+ process tree, so it has to sit above whatever the user is already running.
+ """
+ try:
+ current = len(os.listdir("/proc"))
+ except OSError:
+ current = 512
+ return max(current * 2, 1024)
+
+
+def _apply_rlimits(timeout: float, with_nproc: bool,
+ as_bytes: Optional[int] = LIMIT_AS_BYTES):
+ def _preexec():
+ os.setsid()
+ cpu = int(timeout) + LIMIT_CPU_SECONDS_SLACK
+ limits = [
+ (resource.RLIMIT_CPU, (cpu, cpu)),
+ (resource.RLIMIT_FSIZE, (LIMIT_FSIZE_BYTES, LIMIT_FSIZE_BYTES)),
+ (resource.RLIMIT_CORE, (0, 0)),
+ ]
+ if as_bytes:
+ limits.append((resource.RLIMIT_AS, (as_bytes, as_bytes)))
+ if with_nproc:
+ ceiling = _nproc_ceiling()
+ limits.append((resource.RLIMIT_NPROC, (ceiling, ceiling)))
+ for res, lim in limits:
+ try:
+ resource.setrlimit(res, lim)
+ except (ValueError, OSError):
+ pass
+ return _preexec
+
+
+class Sandbox:
+ """A cached venv plus an isolated runner for untrusted candidate programs."""
+
+ def __init__(
+ self,
+ name: str,
+ packages: Optional[Sequence[str]] = None,
+ cache_root: Optional[Path] = None,
+ base_python: Optional[str] = None,
+ isolation: Optional[str] = None,
+ provision: Optional[Sequence[str]] = None,
+ env: Optional[Dict[str, str]] = None,
+ address_space_limit: Optional[int] = LIMIT_AS_BYTES,
+ ):
+ self.name = name
+ self.packages = sorted(set(packages or []))
+ self.provision = list(provision or [])
+ self.env = dict(env or {})
+ # None drops the RLIMIT_AS cap entirely -- see _apply_rlimits
+ self.address_space_limit = address_space_limit
+ self.cache_root = Path(cache_root or DEFAULT_CACHE_ROOT)
+ self.base_python = base_python or sys.executable
+ self.venv_dir = self.cache_root / f"{name}-venv"
+ self._python: Optional[Path] = None
+ self._isolation = isolation
+ self._base_prefix: List[Path] = []
+
+ @property
+ def python(self) -> Path:
+ if self._python is None:
+ raise RuntimeError("Sandbox.ensure() must be called first")
+ return self._python
+
+ @property
+ def isolation(self) -> str:
+ if self._isolation is None:
+ if self._python:
+ py = str(self._python)
+ roots = [self.venv_dir, *(self._base_prefix or [])]
+ else:
+ py = sys.executable
+ roots = self._detect_base_prefix(Path(py))
+ self._isolation = _detect_isolation(py, roots)
+ return self._isolation
+
+ def _stamp_path(self) -> Path:
+ return self.venv_dir / ".llama-eval-stamp.json"
+
+ def _stamp_matches(self) -> bool:
+ try:
+ stamp = json.loads(self._stamp_path().read_text())
+ except Exception:
+ return False
+ return (
+ stamp.get("packages") == self.packages
+ and stamp.get("py") == self._base_version()
+ and stamp.get("provision") == self.provision
+ )
+
+ def _resolved_env(self) -> Dict[str, str]:
+ return {k: v.replace("{venv}", str(self.venv_dir)) for k, v in self.env.items()}
+
+ def _base_version(self) -> str:
+ r = subprocess.run(
+ [self.base_python, "-c",
+ "import sys;print('.'.join(map(str,sys.version_info[:3])))"],
+ capture_output=True, text=True,
+ )
+ return r.stdout.strip()
+
+ def ensure(self, force: bool = False, quiet: bool = False) -> Path:
+ """Create/refresh the venv. Idempotent and cheap when already valid."""
+ py = self.venv_dir / "bin" / "python"
+ if force and self.venv_dir.exists():
+ shutil.rmtree(self.venv_dir)
+
+ if py.exists() and self._stamp_matches():
+ self._python = py
+ self._base_prefix = self._detect_base_prefix(py)
+ return py
+
+ if not quiet:
+ pkgs = ", ".join(self.packages) if self.packages else "stdlib only"
+ print(f"[sandbox] provisioning venv {self.venv_dir} ({pkgs})")
+
+ self.cache_root.mkdir(parents=True, exist_ok=True)
+ if self.venv_dir.exists():
+ shutil.rmtree(self.venv_dir)
+
+ uv = shutil.which("uv")
+ if uv:
+ subprocess.run(
+ [uv, "venv", "--python", self.base_python, str(self.venv_dir)],
+ check=True, capture_output=True,
+ )
+ elif Path(self.base_python).resolve() == Path(sys.executable).resolve():
+ venv.EnvBuilder(with_pip=bool(self.packages), clear=True).create(self.venv_dir)
+ else:
+ subprocess.run(
+ [self.base_python, "-m", "venv", str(self.venv_dir)],
+ check=True, capture_output=True,
+ )
+
+ if self.packages:
+ if uv:
+ cmd = [uv, "pip", "install", "--python", str(py), *self.packages]
+ else:
+ cmd = [str(py), "-m", "pip", "install", "--quiet",
+ "--disable-pip-version-check", *self.packages]
+ r = subprocess.run(cmd, capture_output=True, text=True)
+ if r.returncode != 0:
+ raise RuntimeError(
+ f"failed to install {len(self.packages)} package(s) into "
+ f"{self.venv_dir}:\n{(r.stderr or r.stdout)[-3000:]}"
+ )
+
+ for i, snippet in enumerate(self.provision, 1):
+ if not quiet:
+ print(f"[sandbox] provisioning step {i}/{len(self.provision)}")
+ prov_env = dict(os.environ)
+ prov_env["SANDBOX_VENV"] = str(self.venv_dir)
+ prov_env.update(self._resolved_env())
+ r = subprocess.run([str(py), "-c", snippet], env=prov_env,
+ capture_output=True, text=True, timeout=1800)
+ if r.returncode != 0:
+ raise RuntimeError(
+ f"provisioning step {i} failed:\n{(r.stderr or r.stdout)[-3000:]}"
+ )
+
+ self._stamp_path().write_text(json.dumps({
+ "packages": self.packages,
+ "py": self._base_version(),
+ "provision": self.provision,
+ }))
+ self._python = py
+ self._base_prefix = self._detect_base_prefix(py)
+ if not quiet:
+ print(f"[sandbox] ready ({self.isolation} isolation)")
+ if self.isolation == "unshare-root":
+ print("Warning: this host's unshare cannot keep the caller's uid, so "
+ "candidate code runs as root inside the namespace. Tests that "
+ "assert a permission denial will not see one. Install "
+ "bubblewrap (bwrap) for correct results.")
+ return py
+
+ @staticmethod
+ def _detect_base_prefix(py: Path) -> List[Path]:
+ """Every directory that must stay visible for the venv to start.
+
+ A venv is only symlinks into its base interpreter, so the whole install
+ tree has to survive the tmpfs that hides $HOME -- and binding just the
+ resolved tree is not enough. Version-managed interpreters (uv, pyenv)
+ are commonly reached through an unversioned alias symlink
+ (cpython-3.13-... -> cpython-3.13.14-...), and startup resolves through
+ the alias path, so both spellings need to be mounted. Walk the chain
+ and keep every install root it passes through.
+ """
+ roots: List[Path] = []
+
+ def add(path: Path):
+ if path.exists() and path not in roots:
+ roots.append(path)
+
+ cur, hops = py, 0
+ while hops < 16:
+ hops += 1
+ add(cur.parent.parent)
+ if cur.is_symlink():
+ target = Path(os.readlink(cur))
+ cur = target if target.is_absolute() else (cur.parent / target)
+ else:
+ break
+
+ try:
+ r = subprocess.run([str(py), "-c", "import sys; print(sys.base_prefix)"],
+ capture_output=True, text=True, timeout=60)
+ if r.returncode == 0 and r.stdout.strip():
+ add(Path(r.stdout.strip()))
+ except Exception:
+ pass
+
+ return [p for p in roots if str(p) not in ("/", "/usr", "/usr/local")]
+
+ def _isolate(self, argv: Sequence[str], workdir: Path,
+ ro_binds: Sequence[Path] = ()) -> List[str]:
+ """Wrap a command in the strongest isolation tier this host supports."""
+ if self.isolation == "bwrap":
+ return _bwrap_argv(
+ workdir, argv,
+ ro_binds=[self.venv_dir, *(self._base_prefix or []), *ro_binds],
+ )
+ if self.isolation in UNSHARE_ARGV:
+ return [*UNSHARE_ARGV[self.isolation], *argv]
+ return list(argv)
+
+ def _base_env(self, workdir: Path) -> Dict[str, str]:
+ env = {
+ "PATH": "/usr/bin:/bin",
+ "HOME": str(workdir),
+ "TMPDIR": str(workdir),
+ "LC_ALL": "C.UTF-8",
+ "PYTHONHASHSEED": "0",
+ "PYTHONDONTWRITEBYTECODE": "1",
+ "OPENBLAS_NUM_THREADS": "1",
+ "OMP_NUM_THREADS": "1",
+ }
+ env.update(self._resolved_env())
+ return env
+
+ def run_argv(self, argv: Sequence[str], workdir: Path,
+ timeout: float = 15.0, env: Optional[Dict[str, str]] = None,
+ ro_binds: Sequence[Path] = ()) -> ExecResult:
+ """Run an arbitrary command in `workdir`, which stays writable.
+
+ Unlike run(), the caller owns the directory and it survives the call.
+ That is what a multi-step agent needs: its edits have to persist across
+ many tool invocations before anything is graded. Taking an argv rather
+ than a source string is what lets a suite drive a linter or a test
+ runner for a language other than Python.
+ """
+ run_env = self._base_env(workdir)
+ if env:
+ run_env.update(env)
+
+ started = time.monotonic()
+ try:
+ proc = subprocess.Popen(
+ self._isolate(argv, workdir, ro_binds),
+ cwd=str(workdir),
+ stdin=subprocess.DEVNULL,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ text=True,
+ errors="replace",
+ env=run_env,
+ preexec_fn=_apply_rlimits(timeout,
+ with_nproc=self.isolation == "rlimit",
+ as_bytes=self.address_space_limit),
+ )
+ except Exception as e:
+ return ExecResult(False, "error", "", f"{type(e).__name__}: {e}")
+
+ try:
+ out, err = proc.communicate(timeout=timeout)
+ except subprocess.TimeoutExpired:
+ _kill_tree(proc)
+ out, err = proc.communicate()
+ return ExecResult(False, "timeout", out or "", err or "",
+ None, time.monotonic() - started)
+
+ duration = time.monotonic() - started
+ if proc.returncode == 0:
+ return ExecResult(True, "ok", out, err, 0, duration)
+ return ExecResult(False, "failed", out, err, proc.returncode, duration)
+
+ def run(self, code: str, timeout: float = 15.0,
+ env: Optional[Dict[str, str]] = None) -> ExecResult:
+ """Run `code` to completion in the sandbox. Exit 0 == passed."""
+ workdir = Path(tempfile.mkdtemp(prefix=f"{self.name}-", dir="/tmp"))
+ try:
+ prog = workdir / "candidate.py"
+ prog.write_text(code)
+ return self.run_argv([str(self.python), "-I", "-B", str(prog)],
+ workdir, timeout=timeout, env=env)
+ except Exception as e:
+ return ExecResult(False, "error", "", f"{type(e).__name__}: {e}")
+ finally:
+ shutil.rmtree(workdir, ignore_errors=True)
+
+
+def _kill_tree(proc: subprocess.Popen):
+ """Kill the candidate and everything it spawned."""
+ for sig in (signal.SIGKILL,):
+ try:
+ os.killpg(os.getpgid(proc.pid), sig)
+ except (ProcessLookupError, PermissionError):
+ try:
+ proc.kill()
+ except ProcessLookupError:
+ pass
diff --git a/examples/llama-eval/llama-eval.py b/examples/llama-eval/llama-eval.py
index 61bdbddd8..3ed68e3b5 100755
--- a/examples/llama-eval/llama-eval.py
+++ b/examples/llama-eval/llama-eval.py
@@ -1,6 +1,7 @@
#!/usr/bin/env python3
import argparse
+import ast
import json
import os
import re
@@ -8,23 +9,113 @@ import subprocess
import sys
import threading
import time
+import uuid
from abc import ABC, abstractmethod
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, asdict, field
from pathlib import Path
from queue import Queue
from typing import Dict, List, Optional, Any, Tuple
+from urllib.parse import urlparse
import requests
from tqdm import tqdm
import random
from math import sqrt
+from eval_sandbox import Sandbox, resolve_python
+
@dataclass
class ServerConfig:
url: str
threads: int
name: str = ""
+ api_key: Optional[str] = None
+
+ def is_opencode(self) -> bool:
+ """Whether this endpoint is an opencode gateway rather than a llama-server."""
+ return "opencode" in (urlparse(self.url).hostname or "")
+
+ def headers(self, session_id: Optional[str] = None) -> Dict[str, str]:
+ headers = auth_headers(self.api_key)
+ if session_id and self.is_opencode():
+ headers["x-opencode-session"] = session_id
+ return headers
+
+
+def auth_headers(api_key: Optional[str]) -> Dict[str, str]:
+ """Standard request headers, carrying a bearer token when one is configured."""
+ headers = {"Content-Type": "application/json"}
+ if api_key:
+ headers["Authorization"] = f"Bearer {api_key}"
+ return headers
+
+
+SERVER_ERROR = "server_error"
+
+RETRYABLE_HTTP = {408, 409, 425, 429}
+
+
+class ServerError(Exception):
+ """A request that still failed, or came back empty, after every retry."""
+
+
+def null_reply_reason(result: Any) -> Optional[str]:
+ """Why a chat completion carries no usable reply, or None when it does."""
+ if not isinstance(result, dict):
+ return "reply is not a JSON object"
+ choices = result.get("choices")
+ if not choices:
+ return "reply has no choices"
+ msg = choices[0].get("message")
+ if not msg:
+ return "reply has no message"
+ if not (msg.get("content") or msg.get("tool_calls") or msg.get("reasoning_content")):
+ return "reply is empty"
+ return None
+
+
+def post_with_retry(url: str, headers: Dict[str, str], data: Dict[str, Any],
+ retries: int, retry_delay: float,
+ timeout: Optional[float] = None) -> Dict[str, Any]:
+ """POST a chat completion, resending on errors and null replies.
+
+ The wait before each resend doubles, starting at `retry_delay` seconds.
+ Raises ServerError once `retries` resends have all failed.
+ """
+ reason = ""
+ attempts = 0
+ for attempt in range(retries + 1):
+ if attempt:
+ wait = retry_delay * 2 ** (attempt - 1)
+ print(f" [{url}] {reason}; retry {attempt}/{retries} in {wait:.0f}s",
+ file=sys.stderr)
+ time.sleep(wait)
+ attempts += 1
+ try:
+ response = requests.post(url, headers=headers, json=data, timeout=timeout)
+ except requests.RequestException as e:
+ reason = f"{type(e).__name__}: {e}"
+ continue
+ if response.status_code >= 400:
+ reason = f"HTTP {response.status_code}: {response.text[:300]}"
+ if response.status_code < 500 and response.status_code not in RETRYABLE_HTTP:
+ break
+ continue
+ try:
+ result = response.json()
+ except ValueError:
+ reason = f"invalid JSON: {response.text[:300]}"
+ continue
+ reason = null_reply_reason(result) or ""
+ if not reason:
+ return result
+ raise ServerError(f"{reason} (after {attempts} attempt{'s' if attempts > 1 else ''})")
+
+
+def new_session_id(task_id: str) -> str:
+ """A fresh session id for one task, unique across tasks, runs and resumes."""
+ return f"{task_id}-{uuid.uuid4().hex}"
def wilson_interval(correct: int, total: int, z: float = 1.96) -> Tuple[float, float]:
"""Wilson score confidence interval for a proportion."""
@@ -106,13 +197,118 @@ A) {A}
B) {B}
C) {C}
D) {D}
+""",
+ "humaneval": """Complete the following Python function.
+
+```python
+{question}
+```
+
+Reply with the complete function in a single ```python code block, including the \
+signature and any imports it needs. Do not include tests, example usage, or explanation.
+""",
+ "classeval": """Complete the following Python class.
+
+```python
+{question}
+```
+
+Reply with the complete class in a single ```python code block: keep the class name \
+and every method signature exactly as given, implement every method body, and include \
+any imports the class needs. Do not include tests, example usage, or explanation.
""",
}
+# Suites graded by running the generated code instead of matching its text.
+CODE_SUITES = {"humaneval", "classeval", "agentic"}
+
+AGENTIC_SUITES = {"agentic"}
+
+# Import name -> pip name, for the cases where they differ. Anything not listed is assumed to install under the name it imports as.
+PIP_NAMES = {
+ "bs4": "beautifulsoup4",
+ "PIL": "pillow",
+ "docx": "python-docx",
+ "fitz": "pymupdf",
+ "sklearn": "scikit-learn",
+ "yaml": "PyYAML",
+ "cv2": "opencv-python-headless",
+ "PyPDF2": "pypdf2",
+ "Levenshtein": "levenshtein",
+ "dateutil": "python-dateutil",
+ "attr": "attrs",
+ "OpenSSL": "pyOpenSSL",
+ "serial": "pyserial",
+ "Crypto": "pycryptodome",
+}
+
+DATASET_SOURCE: Optional[str] = None
+
+# Extra construction options for the agentic suite, set from the CLI.
+AGENTIC_OPTS: Dict[str, Any] = {}
+
class BaseDataset(ABC):
questions: List[Dict]
+ is_agentic: bool = False
+
+ def prepare(self, base_python: Optional[str] = None, force: bool = False) -> None:
+ """Provision whatever the suite needs before any task runs.
+
+ Single-shot suites are provisioned from required_packages() instead, so
+ this is a no-op for them.
+ """
+
+ def run(self, index: int, chat) -> Dict[str, Any]:
+ """Drive one episode to completion and return its scored result."""
+ raise NotImplementedError(
+ f"{type(self).__name__} does not drive its own conversation")
+
+ def required_packages(self) -> List[str]:
+ """Third-party pip packages the suite's tests need."""
+ return []
+
+ def provision_steps(self) -> List[str]:
+ """Python snippets run once at venv build time, with network access."""
+ return []
+
+ def preferred_python(self) -> Optional[str]:
+ """Interpreter the suite's dependencies need, if not the current one."""
+ return None
+
+ def configure_python(self, python: str):
+ """Told which interpreter the sandbox will use, before it is built.
+
+ Lets a suite adapt its package set to what that version can actually
+ install.
+ """
+
+ def default_exec_timeout(self) -> float:
+ """Per-task wall-clock limit that suits this suite's workload."""
+ return 15.0
+
+ def sandbox_env(self) -> Dict[str, str]:
+ """Extra env for every run; '{venv}' expands to the venv path."""
+ return {}
+
+ def extract(self, response: str, question: Dict) -> str:
+ """Pull the candidate code out of a chat reply."""
+ return extract_code(response)
+
+ def build_program(self, question: Dict, completion: str,
+ seed: Optional[int] = None) -> str:
+ """Assemble the program whose exit status decides pass/fail."""
+ raise NotImplementedError
+
+ def interpret(self, result: Any, question: Dict) -> Tuple[bool, Dict[str, Any]]:
+ """Turn an ExecResult into (passed, detail-for-the-log)."""
+ return result.passed, {
+ "status": result.status,
+ "reason": result.summary(),
+ "duration": round(result.duration, 3),
+ }
+
@abstractmethod
def get_question(self, index: int) -> Dict:
pass
@@ -175,6 +371,13 @@ class EvalState:
self.total_time: float = 0.0
self._lock = threading.Lock()
+ @property
+ def loaded_dataset(self) -> BaseDataset:
+ """The dataset, once loaded. Raises rather than handing back None."""
+ if self.dataset is None:
+ raise RuntimeError("dataset not loaded; call load_dataset() first")
+ return self.dataset
+
def load_dataset(self, seed: int = 1234):
if self.dataset_type == "aime":
self.dataset = AimeDataset()
@@ -186,6 +389,12 @@ class EvalState:
self.dataset = Gsm8kDataset()
elif self.dataset_type == "gpqa":
self.dataset = GpqaDataset(variant="diamond", seed=seed)
+ elif self.dataset_type == "humaneval":
+ self.dataset = HumanEvalDataset(source=DATASET_SOURCE)
+ elif self.dataset_type == "classeval":
+ self.dataset = ClassEvalDataset(source=DATASET_SOURCE)
+ elif self.dataset_type == "agentic":
+ self.dataset = AgenticDataset(source=DATASET_SOURCE, **AGENTIC_OPTS)
else:
raise ValueError(f"Unknown dataset type: {self.dataset_type}")
@@ -263,31 +472,55 @@ class EvalState:
self.correct = sum(1 for c in self.task_states.get("cases", {}).values() if c.get("correct", False))
+ def _display_pair(self, task_state: TaskState) -> Tuple[str, str]:
+ """The (expected, answer) columns, kept narrow for code suites.
+
+ There the gold value is a whole JSON test harness and the answer is a
+ whole function, so show the task id and the failure cause instead.
+ """
+ if self.dataset_type not in CODE_SUITES:
+ return (
+ task_state.expected,
+ task_state.answer if task_state.answer else "N/A",
+ )
+ try:
+ expected = json.loads(task_state.expected)["task_id"]
+ except (ValueError, KeyError, TypeError):
+ expected = self.dataset_type
+ reason = (task_state.grader_log or {}).get("reason", "")
+ if task_state.correct:
+ reason = "ok"
+ return expected, (reason[:28] or "N/A")
+
def print_progress(self, task_state: TaskState, total_tasks: int, n_correct: int = 0):
- display_answer = task_state.answer if task_state.answer else "N/A"
+ display_expected, display_answer = self._display_pair(task_state)
display_tokens = str(task_state.tokens) if task_state.tokens is not None else "N/A"
display_tps = f"{task_state.tps_gen:.1f}" if task_state.tps_gen is not None else "N/A"
display_t_gen = f"{task_state.t_gen_ms/1000:.1f}" if task_state.t_gen_ms is not None else "N/A"
display_server = task_state.server_name if task_state.server_name else "N/A"
success_ratio = n_correct / self.processed if self.processed > 0 else 0.0
- first_line = task_state.question_text.split('\n')[0]
+ first_line = next(
+ (ln for ln in task_state.question_text.split('\n') if ln.strip()), ""
+ )
truncated_question = first_line[:43]
if len(first_line) > 43:
truncated_question += "..."
else:
truncated_question = truncated_question.ljust(43) + "..."
- print(f"{self.processed:3}/{total_tasks:3} {task_state.task_id:<20} {self.dataset_type.upper()} {truncated_question:<40} {task_state.expected:<10} {display_answer:<10} {display_tokens:<6} {display_tps:<6} {display_t_gen:<8} {'✓' if task_state.correct else '✗'} [{n_correct:3}/{self.processed:3}, {success_ratio:.3f}] {display_server}")
+ print(f"{self.processed:3}/{total_tasks:3} {task_state.task_id:<20} {self.dataset_type.upper()} {truncated_question:<40} {display_expected:<16} {display_answer:<28} {display_tokens:<6} {display_tps:<6} {display_t_gen:<8} {'✓' if task_state.correct else ('⚠' if task_state.status == SERVER_ERROR else '✗')} [{n_correct:3}/{self.processed:3}, {success_ratio:.3f}] {display_server}")
+
+ def n_server_errors(self) -> int:
+ cases = self.task_states.get("cases", {})
+ return sum(1 for c in cases.values() if c.get("status") == SERVER_ERROR)
+
+ def print_server_errors(self):
+ n = self.n_server_errors()
+ if n:
+ print(f"Server errors: {n} (not counted in accuracy; run with --resume to retry them)")
def print_summary(self):
- if self.total == 0:
- print(f"\n{'='*60}")
- print(f"Results: 0/0 correct (0.0%)")
- print(f"{'='*60}")
- else:
- ci_lower, ci_upper = self.accuracy_ci()
- print(f"\n{'='*60}")
- print(f"Results: {self.correct}/{self.total} correct ({self.correct/self.total*100:.1f}%) [{ci_lower*100:.1f}%, {ci_upper*100:.1f}%]")
- print(f"{'='*60}")
+ print()
+ self.print_existing_summary()
def dump(self):
with self._lock:
@@ -347,7 +580,8 @@ class EvalState:
completed = {tid: c for tid, c in cases.items() if c.get("status") == "ok"}
n_correct = sum(1 for c in completed.values() if c.get("correct", False))
n_incorrect = len(completed) - n_correct
- n_pending = len(tasks_to_save) - len(completed)
+ n_server_errors = sum(1 for c in cases.values() if c.get("status") == SERVER_ERROR)
+ n_pending = len(tasks_to_save) - len(completed) - n_server_errors
accuracy = n_correct / len(completed) * 100 if completed else 0.0
ci_lower, ci_upper = wilson_interval(n_correct, len(completed)) if completed else (0.0, 1.0)
@@ -374,6 +608,9 @@ class EvalState:
elif status == "pending":
status_class = "pending"
status_text = "–"
+ elif status == SERVER_ERROR:
+ status_class = "server-error"
+ status_text = "⚠"
else:
status_class = "error"
status_text = "!"
@@ -392,6 +629,11 @@ class EvalState:
escaped_reasoning = self._escape_html(reasoning_content)
grader_log_str = self._escape_html(json.dumps(grader_log, indent=2))
escaped_server = self._escape_html(server_name)
+ transcript = grader_log.get("transcript") if isinstance(grader_log, dict) else None
+ transcript_link = (
+ f'<b>Transcript</b> <a href="{self._escape_html(transcript[:-5])}.html">'
+ f'{self._escape_html(transcript[:-5])}.html</a><br>'
+ if transcript and transcript.endswith(".json") else "")
answer_class = status_class if status == "ok" else ""
rows.append(f"""<tr class="task-row" onclick="toggleDetails('{task_id}')">
@@ -407,6 +649,7 @@ class EvalState:
<tr id="details-{task_id}" class="details-row">
<td colspan="8">
<div class="details-content">
+ {transcript_link}
<b>Prompt</b><pre>{escaped_prompt}</pre>
<b>Response</b><pre>{escaped_response}</pre>
{f'<b>Reasoning</b><pre>{escaped_reasoning}</pre>' if escaped_reasoning else ''}
@@ -483,6 +726,7 @@ class EvalState:
.incorrect {{ color: #cf222e; }}
.pending {{ color: #888; }}
.error {{ color: #9a6700; }}
+ .server-error {{ color: #8250df; }}
.details-row {{ display: none; }}
.details-row.open {{ display: table-row; }}
.details-content {{ padding: 8px 16px; background: #f6f8fa; font-size: 12px; }}
@@ -511,6 +755,7 @@ class EvalState:
<div class="label">Accuracy</div><div class="value"><b>{accuracy:.1f}%</b> [{ci_lower*100:.1f}%, {ci_upper*100:.1f}%]</div>
<div class="label">Correct</div><div class="value"><span class="correct">{n_correct}</span> / {len(completed)}</div>
<div class="label">Pending</div><div class="value">{n_pending}</div>
+ <div class="label">Server errors</div><div class="value"><span class="server-error">{n_server_errors}</span></div>
<div class="label">Time</div><div class="value">{self.total_time:.1f}s</div>
<div class="label">Sampling</div><div class="value">{sampling_str}</div>
</div>
@@ -680,6 +925,7 @@ class EvalState:
print(f"{'='*60}")
print(f"Results: {correct}/{total} correct ({correct/total*100:.1f}%) [{ci_lower*100:.1f}%, {ci_upper*100:.1f}%]")
print(f"{'='*60}")
+ self.print_server_errors()
def accuracy_ci(self) -> Tuple[float, float]:
"""Compute Wilson score confidence interval from completed cases."""
@@ -974,6 +1220,526 @@ class GpqaDataset(BaseDataset):
D=question["shuffled_answers"][3]
)
+FENCE_RE = re.compile(r"```[ \t]*([A-Za-z0-9_+-]*)[ \t]*\n(.*?)(?:```|\Z)", re.DOTALL)
+
+RESULT_MARKER = "__LLAMA_EVAL__"
+
+GENSIM_SHIM = {
+ "__init__.py": (
+ '"""Minimal stand-in for the two gensim entry points ClassEval uses."""\n'
+ "from . import utils, matutils # noqa: F401\n"
+ ),
+ "utils.py": (
+ "import re, html\n"
+ "RE_ENTITY = re.compile(r'&(#?)([xX]?)(\\w{1,8});')\n"
+ "def decode_htmlentities(text):\n"
+ " return RE_ENTITY.sub(lambda m: html.unescape(m.group(0)), text)\n"
+ ),
+ "matutils.py": (
+ "import numpy as np\n"
+ "def unitvec(vec, norm='l2', return_norm=False):\n"
+ " v = np.asarray(vec, dtype=float)\n"
+ " n = float(np.sum(np.abs(v))) if norm == 'l1' "
+ "else float(np.sqrt(np.sum(v ** 2)))\n"
+ " if n == 0.0:\n"
+ " return (v, 1.0) if return_norm else v\n"
+ " out = v / n\n"
+ " return (out, n) if return_norm else out\n"
+ ),
+}
+
+UNITTEST_PROLOGUE = '''
+import unittest as _ut_pre
+_ut_pre.main = lambda *a, **k: None
+'''
+
+UNITTEST_RUNNER = f'''
+
+if True:
+ import json as _json, os as _os, unittest as _ut
+ _loader = _ut.TestLoader()
+ _suite = _ut.TestSuite()
+ for _c in [v for v in list(globals().values())
+ if isinstance(v, type) and issubclass(v, _ut.TestCase)
+ and v is not _ut.TestCase]:
+ _suite.addTests(_loader.loadTestsFromTestCase(_c))
+ with open(_os.devnull, "w") as _null:
+ _res = _ut.TextTestRunner(stream=_null, verbosity=0).run(_suite)
+ _bad = {{t.id() for t, _ in _res.failures}} | {{t.id() for t, _ in _res.errors}}
+ _total = _res.testsRun
+ print("{RESULT_MARKER}" + _json.dumps({{
+ "total": _total,
+ "passed": _total - len(_bad),
+ "failed": sorted(_bad),
+ "first_error": (_res.failures + _res.errors)[0][1].strip().splitlines()[-1]
+ if (_res.failures or _res.errors) else None,
+ }}))
+'''
+
+
+def parse_test_report(stdout: str) -> Optional[Dict[str, Any]]:
+ """Read the JSON line emitted by UNITTEST_RUNNER, if the program got that far."""
+ for line in reversed(stdout.splitlines()):
+ if line.startswith(RESULT_MARKER):
+ try:
+ return json.loads(line[len(RESULT_MARKER):])
+ except ValueError:
+ return None
+ return None
+
+
+def extract_code(response: str, entry_point: Optional[str] = None,
+ class_name: Optional[str] = None) -> str:
+ """Pull the candidate program out of a chat reply.
+
+ Prefers a fenced block that actually defines the thing being asked for,
+ since models like to emit a usage example or a test in a second block.
+ """
+ if not response:
+ return ""
+
+ if class_name:
+ wanted = rf"^\s*class\s+{re.escape(class_name)}\s*[:\(]"
+ elif entry_point:
+ wanted = rf"^\s*(?:async\s+)?def\s+{re.escape(entry_point)}\s*\("
+ else:
+ wanted = None
+
+ blocks = [(lang.lower(), body) for lang, body in FENCE_RE.findall(response)]
+ if blocks:
+ py = [b for lang, b in blocks if lang in ("python", "py", "python3", "")]
+ candidates = py or [b for _, b in blocks]
+ if wanted:
+ defining = [b for b in candidates if re.search(wanted, b, re.M)]
+ if defining:
+ return defining[-1].strip("\n")
+ return candidates[0].strip("\n")
+
+ return response.strip("\n")
+
+
+def _import_preamble(prompt: str) -> str:
+ """The import lines from the original stub.
+
+ A chat model often returns just the function and drops the imports that the
+ stub had above it; replaying them keeps that from being scored as a failure.
+ """
+ lines = []
+ for line in prompt.splitlines():
+ stripped = line.strip()
+ if stripped.startswith(("import ", "from ")) and not line.startswith((" ", "\t")):
+ lines.append(stripped)
+ return "\n".join(lines)
+
+
+def build_program(question: Dict, completion: str, seed: Optional[int] = 0) -> str:
+ """Assemble the full program that decides pass/fail for one problem."""
+ entry_point = question["entry_point"]
+ code = completion
+
+ if not re.search(rf"^\s*(?:async\s+)?def\s+{re.escape(entry_point)}\s*\(", code, re.M):
+ body = completion if completion.startswith((" ", "\t")) else \
+ "\n".join(" " + ln for ln in completion.splitlines())
+ code = question["prompt"] + "\n" + body
+
+ prologue = f"import random; random.seed({seed})" if seed is not None else ""
+
+ return "\n\n".join(part for part in [
+ _import_preamble(question["prompt"]),
+ code,
+ question["test"],
+ prologue,
+ f"check({entry_point})",
+ ] if part)
+
+
+def pass_at_k(n: int, c: int, k: int) -> float:
+ """Unbiased pass@k estimator from the Codex paper (arXiv:2107.03374).
+
+ n samples drawn, c of them correct. Computed as 1 - C(n-c,k)/C(n,k) via a
+ product so it stays stable for large n.
+ """
+ if n - c < k:
+ return 1.0
+ prod = 1.0
+ for i in range(n - c + 1, n + 1):
+ prod *= 1.0 - k / i
+ return 1.0 - prod
+
+
+class HumanEvalDataset(BaseDataset):
+ """OpenAI HumanEval, 164 hand-written Python problems.
+
+ Every problem in this suite imports only from the standard library, so the
+ execution venv needs no pip installs at all -- see `required_packages`.
+ """
+
+ HF_REPO = "openai/openai_humaneval"
+
+ def __init__(self, source: Optional[str] = None, split: str = "test"):
+ self.source = source or self.HF_REPO
+ self.split = split
+ self.questions: List[Dict] = []
+ self._load_dataset()
+
+ def _load_dataset(self):
+ print(f"Loading HumanEval dataset (source: {self.source})...")
+
+ local = Path(self.source)
+ if local.exists():
+ rows = [json.loads(ln) for ln in local.read_text().splitlines() if ln.strip()]
+ else:
+ from datasets import load_dataset
+ rows = [dict(r) for r in load_dataset(self.source, split=self.split)]
+
+ required = {"task_id", "prompt", "entry_point", "test"}
+ for row in rows:
+ missing = required - row.keys()
+ if missing:
+ raise ValueError(
+ f"{self.source}: record {row.get('task_id', '?')} is missing {sorted(missing)}"
+ )
+ row["dataset_type"] = "humaneval"
+ self.questions.append(row)
+
+ print(f"HumanEval dataset loaded: {len(self.questions)} problems")
+
+ def required_packages(self) -> List[str]:
+ """Third-party pip packages this suite needs. Derived, not hardcoded."""
+ return sorted(third_party_imports(
+ "\n".join(q["prompt"] + "\n" + q["test"] for q in self.questions)
+ ))
+
+ def extract(self, response: str, question: Dict) -> str:
+ return extract_code(response, question.get("entry_point"))
+
+ def build_program(self, question: Dict, completion: str,
+ seed: Optional[int] = None) -> str:
+ return build_program(question, completion, seed=seed)
+
+ def get_question(self, index: int) -> Dict:
+ return self.questions[index]
+
+ def get_question_text(self, question: Dict) -> str:
+ return question["prompt"]
+
+ def get_answer(self, question: Dict) -> str:
+ return json.dumps({
+ "task_id": question["task_id"],
+ "entry_point": question["entry_point"],
+ "prompt": question["prompt"],
+ "test": question["test"],
+ })
+
+ def get_prompt(self, question: Dict) -> str:
+ return TEMPLATE_REGISTRY["humaneval"].format(
+ question=self.get_question_text(question).rstrip(),
+ )
+
+
+class ClassEvalDataset(BaseDataset):
+ """ClassEval: 100 class-level Python generation tasks.
+
+ A task is a whole class judged by several independent unittest classes, so
+ it is scored twice -- class level (every test passes) and method level
+ (what fraction of tests passed), matching the upstream protocol.
+ """
+
+ HF_REPO = "ilintar/ClassEval"
+
+ def __init__(self, source: Optional[str] = None):
+ self.source = source or self.HF_REPO
+ self.questions: List[Dict] = []
+ self._gensim_shim = False
+ self._load_dataset()
+
+ def _load_dataset(self):
+ print(f"Loading ClassEval dataset (source: {self.source})...")
+
+ local = Path(self.source)
+ if local.exists():
+ text = local.read_text()
+ rows = (json.loads(text) if local.suffix == ".json"
+ else [json.loads(ln) for ln in text.splitlines() if ln.strip()])
+ else:
+ from datasets import load_dataset
+ rows = [dict(r) for r in load_dataset(self.source, split="test")]
+
+ required = {"task_id", "skeleton", "test", "class_name"}
+ for row in rows:
+ missing = required - row.keys()
+ if missing:
+ raise ValueError(
+ f"{self.source}: record {row.get('task_id', '?')} is missing {sorted(missing)}"
+ )
+ row["dataset_type"] = "classeval"
+ row["n_tests"] = _count_test_methods(row["test"])
+ self.questions.append(row)
+
+ print(f"ClassEval dataset loaded: {len(self.questions)} classes, "
+ f"{sum(q['n_tests'] for q in self.questions)} test methods")
+
+ EXTRA_PACKAGES = [
+ "lxml",
+ "scipy",
+ ]
+
+ def configure_python(self, python: str):
+ """gensim stops at cp313, so on 3.14+ swap it for a small local shim."""
+ try:
+ r = subprocess.run(
+ [python, "-c", "import sys; print('%d.%d' % sys.version_info[:2])"],
+ capture_output=True, text=True, timeout=60,
+ )
+ major, minor = (int(x) for x in r.stdout.strip().split("."))
+ except Exception:
+ return
+ self._gensim_shim = (major, minor) >= (3, 14)
+
+ def required_packages(self) -> List[str]:
+ blob = "\n\n".join(
+ "\n\n".join(_as_text(q.get(f)) for f in
+ ("skeleton", "test", "solution_code", "import_statement"))
+ for q in self.questions
+ )
+ found = {PIP_NAMES.get(m, m) for m in third_party_imports(blob)}
+ found |= set(self.EXTRA_PACKAGES)
+ if self._gensim_shim:
+ found.discard("gensim")
+ return sorted(found)
+
+ NLTK_CORPORA = ("punkt_tab", "averaged_perceptron_tagger_eng", "wordnet")
+
+ def provision_steps(self) -> List[str]:
+ steps = []
+
+ if any("nltk" in _as_text(q.get("skeleton")) + _as_text(q.get("test"))
+ for q in self.questions):
+ steps.append(
+ "import os, nltk\n"
+ "target = os.path.join(os.environ['SANDBOX_VENV'], 'nltk_data')\n"
+ "os.makedirs(target, exist_ok=True)\n"
+ f"for corpus in {self.NLTK_CORPORA!r}:\n"
+ " nltk.download(corpus, download_dir=target, quiet=True)\n"
+ )
+
+ if self._gensim_shim:
+ steps.append(
+ "import os, sysconfig\n"
+ "pkg = os.path.join(sysconfig.get_paths()['purelib'], 'gensim')\n"
+ "os.makedirs(pkg, exist_ok=True)\n"
+ f"files = {GENSIM_SHIM!r}\n"
+ "for name, body in files.items():\n"
+ " open(os.path.join(pkg, name), 'w').write(body)\n"
+ "import gensim.utils, gensim.matutils\n"
+ )
+
+ return steps
+
+ def sandbox_env(self) -> Dict[str, str]:
+ return {"NLTK_DATA": "{venv}/nltk_data"}
+
+ def preferred_python(self) -> Optional[str]:
+ return "3.13"
+
+ def default_exec_timeout(self) -> float:
+ return 30.0
+
+ def extract(self, response: str, question: Dict) -> str:
+ return extract_code(response, class_name=question.get("class_name"))
+
+ def build_program(self, question: Dict, completion: str,
+ seed: Optional[int] = None) -> str:
+ prologue = f"import random; random.seed({seed})" if seed is not None else ""
+ return "\n\n".join(part for part in [
+ _as_text(question.get("import_statement")),
+ UNITTEST_PROLOGUE,
+ completion,
+ question["test"],
+ prologue,
+ UNITTEST_RUNNER,
+ ] if part)
+
+ def interpret(self, result: Any, question: Dict) -> Tuple[bool, Dict[str, Any]]:
+ expected = question.get("n_tests", 0)
+
+ report = parse_test_report(result.stdout)
+ if report is None:
+ return False, {
+ "status": result.status if result.status != "ok" else "no-report",
+ "reason": result.summary(),
+ "tests_total": expected,
+ "tests_passed": 0,
+ "duration": round(result.duration, 3),
+ }
+
+ total, passed = report.get("total", 0) or expected, report.get("passed", 0)
+ ok = total > 0 and passed == total
+ return ok, {
+ "status": "ok" if ok else "failed",
+ "reason": "ok" if ok else (report.get("first_error")
+ or f"{passed}/{total} tests passed")[:200],
+ "tests_total": total,
+ "tests_passed": passed,
+ "duration": round(result.duration, 3),
+ }
+
+ def get_question(self, index: int) -> Dict:
+ return self.questions[index]
+
+ def get_question_text(self, question: Dict) -> str:
+ return question["skeleton"]
+
+ def get_answer(self, question: Dict) -> str:
+ return json.dumps({
+ "task_id": question["task_id"],
+ "class_name": question["class_name"],
+ "import_statement": _as_text(question.get("import_statement")),
+ "test": question["test"],
+ "n_tests": question["n_tests"],
+ })
+
+ def get_prompt(self, question: Dict) -> str:
+ return TEMPLATE_REGISTRY["classeval"].format(
+ question=self.get_question_text(question).rstrip(),
+ )
+
+
+class AgenticDataset(BaseDataset):
+ """Multi-turn tool-driven repair tasks over a small real repository.
+
+ The other code suites answer one prompt. Here the model is handed a tool
+ set and drives its own conversation, so the Processor delegates the whole
+ episode rather than making a single request.
+
+ Everything language-specific is imported lazily and provisioned per
+ language, so selecting the Python tasks never installs a node toolchain --
+ and running any other suite installs none of it.
+ """
+
+ HF_REPO = "ilintar/SACB"
+ is_agentic = True
+ ae: Any
+
+ def __init__(self, source: Optional[str] = None, lang: Optional[str] = None,
+ max_turns: int = 50, context_limit: Optional[int] = None,
+ max_tokens: Optional[int] = None):
+ self.source = source or self.HF_REPO
+ self.lang_filter = lang
+ self.max_turns = max_turns
+ self.context_limit = context_limit
+ self.max_tokens = max_tokens
+ self._sandboxes: Dict[str, Any] = {}
+ self._load_dataset()
+
+ def _load_dataset(self):
+ import agentic_eval as ae
+ self.ae = ae
+
+ print(f"Loading agentic dataset (source: {self.source})...")
+ path = Path(self.source)
+ if path.exists():
+ rows = [json.loads(line) for line in path.read_text().splitlines()
+ if line.strip()]
+ else:
+ from datasets import load_dataset
+ rows = [dict(r) for r in load_dataset(self.source, split="test")]
+
+ if self.lang_filter:
+ rows = [r for r in rows if r.get("lang") == self.lang_filter]
+ if not rows:
+ raise ValueError(
+ f"no agentic tasks in {self.source}"
+ + (f" for language {self.lang_filter!r}" if self.lang_filter else ""))
+
+ self.tasks = [ae.AgenticTask.from_record(r) for r in rows]
+ self.languages = sorted({t.lang for t in self.tasks})
+ print(f"Agentic dataset loaded: {len(self.tasks)} tasks "
+ f"({', '.join(self.languages)})")
+
+ def __len__(self) -> int:
+ return len(self.tasks)
+
+ def get_question(self, index: int) -> Dict:
+ return {"index": index}
+
+ def get_question_text(self, question: Dict) -> str:
+ task = self.tasks[question["index"]]
+ first = task.instruction.strip().splitlines()[0]
+ return f"[{task.task_id}] {first}"
+
+ def get_answer(self, question: Dict) -> str:
+ task = self.tasks[question["index"]]
+ return json.dumps({"task_id": task.task_id, "index": question["index"]})
+
+ def get_prompt(self, question: Dict) -> str:
+ return self.ae.user_prompt(self.tasks[question["index"]])
+
+ def default_exec_timeout(self) -> float:
+ return 90.0
+
+ def required_packages(self) -> List[str]:
+ return []
+
+ def prepare(self, base_python: Optional[str] = None, force: bool = False):
+ """Provision one sandbox per language present in the selected tasks."""
+ for lang in self.languages:
+ sandbox = self.ae.make_sandbox(lang, Sandbox, base_python=base_python)
+ sandbox.ensure(force=force)
+ self._sandboxes[lang] = sandbox
+ return self._sandboxes
+
+ def run(self, index: int, chat) -> Dict[str, Any]:
+ task = self.tasks[index]
+ return self.ae.run_task(
+ task, chat, self._sandboxes[task.lang],
+ max_turns=self.max_turns, context_limit=self.context_limit,
+ test_timeout=self.default_exec_timeout())
+
+
+def _as_text(value: Any) -> str:
+ """Some dataset fields arrive as a list of lines rather than a blob."""
+ if not value:
+ return ""
+ if isinstance(value, (list, tuple)):
+ return "\n".join(str(v) for v in value)
+ return str(value)
+
+
+def _count_test_methods(test_code: str) -> int:
+ """How many `def test_*` a task's test suite defines."""
+ try:
+ tree = ast.parse(test_code)
+ except SyntaxError:
+ return len(re.findall(r"^\s*def\s+test\w*\s*\(", test_code, re.M))
+ return sum(
+ 1
+ for node in ast.walk(tree)
+ if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
+ and node.name.startswith("test")
+ )
+
+
+def third_party_imports(source: str) -> set:
+ """Top-level modules imported by `source` that are not in the stdlib."""
+ import ast
+
+ mods = set()
+ for chunk in source.split("\n\n"):
+ try:
+ tree = ast.parse(chunk)
+ except SyntaxError:
+ continue
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Import):
+ mods.update(a.name.split(".")[0] for a in node.names)
+ elif isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
+ mods.add(node.module.split(".")[0])
+
+ stdlib = set(sys.stdlib_module_names)
+ return {m for m in mods if m and m not in stdlib}
+
+
class Grader:
def __init__(
self,
@@ -981,14 +1747,29 @@ class Grader:
grader_script: Optional[str] = None,
grader_model_name: Optional[str] = None,
grader_server_url: str = "",
- dataset_type: str = "aime"
+ grader_api_key: Optional[str] = None,
+ dataset_type: str = "aime",
+ sandbox: Optional[Sandbox] = None,
+ exec_timeout: float = 15.0,
+ exec_seed: Optional[int] = 0,
+ dataset: Optional[BaseDataset] = None,
):
self.grader_type = grader_type
self.grader_script = grader_script
self.grader_model_name = grader_model_name
self.grader_server_url = grader_server_url
+ self.grader_api_key = grader_api_key
self.dataset_type = dataset_type
+ self.sandbox = sandbox
+ self.exec_timeout = exec_timeout
+ self.exec_seed = exec_seed
+ self.dataset = dataset
self.pattern = self._get_pattern()
+ self._local = threading.local()
+
+ @property
+ def last_exec(self) -> Dict[str, Any]:
+ return getattr(self._local, "exec_info", {})
def _get_pattern(self) -> Optional[str]:
if self.grader_type == "regex":
@@ -1077,7 +1858,7 @@ When extracting the answer, provide only the extracted answer itself, nothing el
Please provide only the extracted answer, nothing else. If there is no clear answer that can be extracted from the response, reply with 'no answer'."""
url = f"{self.grader_server_url}/v1/chat/completions"
- headers = {"Content-Type": "application/json"}
+ headers = auth_headers(self.grader_api_key)
data = {
"model": self.grader_model_name,
"messages": [
@@ -1102,6 +1883,28 @@ Please provide only the extracted answer, nothing else. If there is no clear ans
lines = response.split('\n')
return '\n'.join(lines[-max_lines:]) if len(lines) > max_lines else response
+ def _grade_exec(self, gold: str, pred: str) -> Tuple[bool, Optional[str]]:
+ """Grade by running the generated code against the suite's own tests.
+
+ `gold` is the JSON harness produced by the dataset's get_answer(); the
+ dataset owns how the program is assembled and how its result is read.
+ """
+ if self.sandbox is None or self.dataset is None:
+ raise ValueError("exec grader requires a sandbox and a dataset")
+
+ question = json.loads(gold)
+ completion = self.dataset.extract(pred, question)
+ if not completion.strip():
+ self._local.exec_info = {"status": "empty", "reason": "no code in reply"}
+ return False, None
+
+ program = self.dataset.build_program(question, completion, seed=self.exec_seed)
+ result = self.sandbox.run(program, timeout=self.exec_timeout)
+ passed, detail = self.dataset.interpret(result, question)
+
+ self._local.exec_info = detail
+ return passed, completion
+
def grade(self, gold: str, pred: str, problem: str = "") -> Tuple[bool, Optional[str]]:
"""Grade the response"""
if self.grader_type == "regex":
@@ -1110,6 +1913,8 @@ Please provide only the extracted answer, nothing else. If there is no clear ans
return self._grade_cli(gold, pred)
elif self.grader_type == "llm":
return self._grade_llm(gold, pred, problem)
+ elif self.grader_type == "exec":
+ return self._grade_exec(gold, pred)
else:
raise ValueError(f"Unknown grader type: {self.grader_type}")
@@ -1119,18 +1924,24 @@ class Processor:
server_configs: List[ServerConfig],
grader: Grader,
model_name: Optional[str] = None,
- n_predict: int = -1
+ n_predict: int = -1,
+ retries: int = 5,
+ retry_delay: float = 10.0,
+ transcript_dir: Optional[Path] = None,
):
self.server_configs = server_configs
self.grader = grader
self.model_name = model_name
self.n_predict = n_predict
+ self.retries = retries
+ self.retry_delay = retry_delay
+ self.transcript_dir = transcript_dir
@staticmethod
def _check_server(server_config: ServerConfig) -> List[str]:
url = f"{server_config.url}/v1/models"
try:
- response = requests.get(url)
+ response = requests.get(url, headers=server_config.headers())
response.raise_for_status()
models = [m["id"] for m in response.json().get("data", [])]
return models
@@ -1139,10 +1950,11 @@ class Processor:
sys.exit(1)
def _make_request(
- self, server_config: ServerConfig, eval_state: EvalState, prompt: str
+ self, server_config: ServerConfig, eval_state: EvalState, prompt: str,
+ session_id: Optional[str] = None
) -> Tuple[Dict[str, Any], int, Optional[float], Optional[float], str]:
url = f"{server_config.url}/v1/chat/completions"
- headers = {"Content-Type": "application/json"}
+ headers = server_config.headers(session_id)
data = {
"model": self.model_name if self.model_name else "llama",
"messages": [{"role": "user", "content": prompt}],
@@ -1156,10 +1968,10 @@ class Processor:
data["top_p"] = eval_state.sampling_config["top_p"]
if eval_state.sampling_config.get("min_p") is not None:
data["min_p"] = eval_state.sampling_config["min_p"]
+ if eval_state.sampling_config.get("reasoning_effort") is not None:
+ data["reasoning_effort"] = eval_state.sampling_config["reasoning_effort"]
- response = requests.post(url, headers=headers, json=data)
- response.raise_for_status()
- result = response.json()
+ result = post_with_retry(url, headers, data, self.retries, self.retry_delay)
tokens = result.get("usage", {}).get("completion_tokens", 0)
timings = result.get("timings", {})
tps_gen = timings.get("predicted_per_second") if timings else None
@@ -1187,8 +1999,14 @@ class Processor:
problem_idx=problem_idx,
)
+ session_id = new_session_id(task_id)
+
+ if eval_state.loaded_dataset.is_agentic:
+ return self._process_agentic_case(
+ server_config, eval_state, i, task_id, task_state, session_id)
+
try:
- response, tokens, tps_gen, t_gen_ms, finish_reason = self._make_request(server_config, eval_state, prompt)
+ response, tokens, tps_gen, t_gen_ms, finish_reason = self._make_request(server_config, eval_state, prompt, session_id)
result = response["choices"][0]["message"]["content"]
reasoning_content = response["choices"][0].get("message", {}).get("reasoning_content")
task_state.response = result
@@ -1208,15 +2026,20 @@ class Processor:
eval_state.dump()
return task_state
- result_truncated = self.grader._truncate_response(result, max_lines=10)
- is_correct, answer = self.grader.grade(expected, result_truncated, prompt)
+ if self.grader.grader_type == "exec":
+ graded_text = result
+ else:
+ graded_text = self.grader._truncate_response(result, max_lines=10)
+ is_correct, answer = self.grader.grade(expected, graded_text, prompt)
grader_log = {
- "pred": result_truncated,
+ "pred": graded_text,
"grader_type": self.grader.grader_type
}
if self.grader.grader_type == "regex" and self.grader.pattern:
grader_log["pattern"] = self.grader.pattern
+ if self.grader.grader_type == "exec":
+ grader_log.update(self.grader.last_exec)
task_state.correct = is_correct
task_state.answer = answer
@@ -1232,11 +2055,136 @@ class Processor:
eval_state.dump()
+ except ServerError as e:
+ task_state.status = SERVER_ERROR
+ task_state.grader_log = {"error": str(e)}
+ eval_state.add_result(
+ task_id, prompt, expected, None, None, task_state.grader_log,
+ False, SERVER_ERROR, None, None, None, None, server_config.name,
+ chunk_idx, problem_idx,
+ )
+ eval_state.dump()
except Exception as e:
task_state.status = f"error: {str(e)}"
return task_state
+ def _agentic_chat(self, server_config: ServerConfig, eval_state: EvalState,
+ session_id: Optional[str] = None):
+ """A chat callable for agentic_eval: messages + tools in, response out."""
+ url = f"{server_config.url}/v1/chat/completions"
+
+ def chat(messages, tools):
+ data = {
+ "model": self.model_name if self.model_name else "llama",
+ "messages": messages,
+ "tools": tools,
+ "tool_choice": "auto",
+ }
+ per_turn = getattr(eval_state.dataset, "max_tokens", None)
+ if self.n_predict and self.n_predict > 0:
+ data["max_tokens"] = self.n_predict
+ elif per_turn:
+ data["max_tokens"] = per_turn
+ for key in ("temperature", "top_k", "top_p", "min_p", "reasoning_effort"):
+ value = eval_state.sampling_config.get(key)
+ if value is not None:
+ data[key] = value
+ return post_with_retry(url, server_config.headers(session_id), data,
+ self.retries, self.retry_delay, timeout=1800)
+
+ return chat
+
+ def _process_agentic_case(
+ self, server_config: ServerConfig, eval_state: EvalState, i: int,
+ task_id: str, task_state: TaskState, session_id: Optional[str] = None
+ ) -> TaskState:
+ """Run one whole tool-driven episode and grade the tree it leaves behind."""
+ try:
+ result = eval_state.loaded_dataset.run(
+ i, self._agentic_chat(server_config, eval_state, session_id))
+ except Exception as e:
+ task_state.status = f"error: {type(e).__name__}: {e}"
+ eval_state.add_result(
+ task_id, task_state.prompt, task_state.expected,
+ task_state.status, None,
+ {"error": f"{type(e).__name__}: {e}", "resolved": False},
+ False, task_state.status, 0, None, None, None,
+ server_config.name, task_state.chunk_idx, i,
+ )
+ eval_state.dump()
+ return task_state
+
+ episode = result.get("episode", {})
+ messages = result.pop("messages", None)
+ if messages is not None and self.transcript_dir is not None:
+ self._save_transcript(eval_state, task_id, messages, result)
+
+ if episode.get("stop_reason") in ("request_failed", "no_choices"):
+ task_state.status = SERVER_ERROR
+ task_state.grader_log = result
+ task_state.response = f"server error on turn {episode.get('turns')}: {episode.get('error')}"
+ eval_state.add_result(
+ task_id, task_state.prompt, task_state.expected, task_state.response,
+ None, result, False, SERVER_ERROR,
+ episode.get("completion_tokens", 0), None, None, None,
+ server_config.name, task_state.chunk_idx, i,
+ )
+ eval_state.dump()
+ return task_state
+
+ task_state.tokens = episode.get("completion_tokens", 0)
+ task_state.correct = bool(result["resolved"])
+ task_state.answer = ", ".join(result.get("changed_files", [])) or None
+ task_state.grader_log = result
+ task_state.status = "ok"
+ task_state.response = (
+ f"stop={episode.get('stop_reason')} turns={episode.get('turns')} "
+ f"calls={episode.get('tool_calls')} edits={episode.get('edits')} "
+ f"errors={episode.get('tool_errors')} nudges={episode.get('nudges')} truncated={episode.get('truncated_turns')} "
+ f"peak_ctx={episode.get('peak_context')}\n"
+ f"changed: {task_state.answer or '(nothing)'}"
+ )
+
+ eval_state.add_result(
+ task_id, task_state.prompt, task_state.expected, task_state.response,
+ task_state.answer, result, task_state.correct, "ok",
+ task_state.tokens, None, None, None, server_config.name,
+ task_state.chunk_idx, i,
+ )
+ eval_state.dump()
+ return task_state
+
+ def _save_transcript(self, eval_state: EvalState, task_id: str,
+ messages: List[Dict[str, Any]], result: Dict[str, Any]):
+ """Write the episode's messages as <task_id>.json plus a readable .html."""
+ assert self.transcript_dir is not None
+ ep = result.get("episode", {})
+ verdict = "RESOLVED" if result.get("resolved") else "not resolved"
+ meta = (
+ f"{verdict} f2p {result.get('f2p_passed')}/{result.get('f2p_total')} "
+ f"p2p {result.get('p2p_passed')}/{result.get('p2p_total')}\n"
+ f"stop={ep.get('stop_reason')} turns={ep.get('turns')} "
+ f"calls={ep.get('tool_calls')} edits={ep.get('edits')} "
+ f"errors={ep.get('tool_errors')} nudges={ep.get('nudges')} truncated={ep.get('truncated_turns')} "
+ f"prompt_tokens={ep.get('prompt_tokens')} "
+ f"completion_tokens={ep.get('completion_tokens')} "
+ f"peak_ctx={ep.get('peak_context')}\n"
+ f"changed: {', '.join(result.get('changed_files') or []) or '(nothing)'}"
+ + (f"\nerror: {ep.get('error')}" if ep.get("error") else "")
+ )
+ title = f"{task_id} ({result.get('task_id')})"
+ dataset = eval_state.loaded_dataset
+ assert isinstance(dataset, AgenticDataset)
+ try:
+ path = dataset.ae.write_transcript(
+ messages, self.transcript_dir / task_id, title, meta)
+ except OSError as e:
+ print(f"Warning: could not write transcript for {task_id}: {e}")
+ return
+ base = Path(eval_state.output_file).resolve().parent
+ result["transcript"] = os.path.relpath(path.resolve(), base)
+
@staticmethod
def _worker(
server_config: ServerConfig,
@@ -1275,7 +2223,7 @@ class Processor:
print(f" {i+1}. {sc.name} — {sc.url} ({sc.threads} threads) [{models_str}]")
print(f"Model: {self.model_name}")
print(f"Grader: {self.grader.grader_type}")
- print(f"Sampling: temp={eval_state.sampling_config.get('temperature', 'skip')}, top-k={eval_state.sampling_config.get('top_k', 'skip')}, top-p={eval_state.sampling_config.get('top_p', 'skip')}, min-p={eval_state.sampling_config.get('min_p', 'skip')}")
+ print(f"Sampling: temp={eval_state.sampling_config.get('temperature', 'skip')}, top-k={eval_state.sampling_config.get('top_k', 'skip')}, top-p={eval_state.sampling_config.get('top_p', 'skip')}, min-p={eval_state.sampling_config.get('min_p', 'skip')}, reasoning-effort={eval_state.sampling_config.get('reasoning_effort', 'skip')}")
print()
# Shared task queue: all workers compete for tasks
@@ -1342,6 +2290,79 @@ class Processor:
eval_state.print_summary()
eval_state.dump()
+def preflight_sandbox(sandbox: Sandbox):
+ """Catch host policies that would fail problems for non-model reasons.
+
+ A FIPS-enforcing OpenSSL makes hashlib.md5() raise, which would sink
+ HumanEval/162 no matter what the model generated.
+ """
+ probe = sandbox.run("import hashlib; hashlib.md5(b'x').hexdigest()", timeout=15)
+ if not probe.passed:
+ print(f"Warning: md5 is unavailable in the execution venv "
+ f"({probe.summary(120)}). Problems that hash with md5 will fail "
+ f"regardless of the model -- this is a host policy, not a model result.")
+
+
+def print_pass_at_k(eval_state: "EvalState"):
+ """Report pass@k plus a breakdown of how the failures failed.
+
+ Each chunk is one full pass over the suite, so running --n_cases at a
+ multiple of the dataset size gives n samples per problem for free.
+ """
+ cases = eval_state.task_states.get("cases", {})
+ if not cases:
+ return
+
+ by_problem: Dict[int, List[bool]] = {}
+ reasons: Dict[str, int] = {}
+ for case in cases.values():
+ by_problem.setdefault(case.get("problem_idx", 0), []).append(
+ bool(case.get("correct", False))
+ )
+ if not case.get("correct", False):
+ status = (case.get("grader_log") or {}).get("status", case.get("status", "?"))
+ reasons[status] = reasons.get(status, 0) + 1
+
+ n = min(len(v) for v in by_problem.values())
+ n_problems = len(by_problem)
+
+ print(f"\n{'=' * 60}")
+ print(f" {eval_state.dataset_type}: {n_problems} problems, {n} sample(s) each")
+ print(f"{'=' * 60}")
+
+ for k in sorted({1, 5, 10, 100, n}):
+ if k > n:
+ continue
+ score = sum(
+ pass_at_k(len(v), sum(v), k) for v in by_problem.values()
+ ) / n_problems
+ print(f" pass@{k:<4d} {score:.4f}")
+
+ if n == 1:
+ solved = sum(1 for v in by_problem.values() if v[0])
+ lo, hi = wilson_interval(solved, n_problems)
+ print(f" {solved}/{n_problems} solved 95% CI [{lo:.4f}, {hi:.4f}]")
+
+ t_total = t_passed = 0
+ for case in cases.values():
+ log = case.get("grader_log") or {}
+ if "tests_total" in log:
+ t_total += log["tests_total"]
+ else:
+ try:
+ t_total += json.loads(case.get("expected") or "{}").get("n_tests", 0)
+ except ValueError:
+ pass
+ t_passed += log.get("tests_passed", 0)
+ if t_total:
+ print(f"\n test methods passed {t_passed}/{t_total} ({t_passed / t_total:.4f})")
+
+ if reasons:
+ print("\n failures by cause:")
+ for status, count in sorted(reasons.items(), key=lambda kv: -kv[1]):
+ print(f" {status:<12s} {count}")
+
+
def main():
parser = argparse.ArgumentParser(
description="Simplified evaluation tool for llama.cpp"
@@ -1358,13 +2379,28 @@ def main():
default="",
help="Comma-separated display names for servers (default: use URLs)"
)
+ parser.add_argument(
+ "--api-key",
+ type=str,
+ default=None,
+ help="Bearer token for the eval servers: one key for all of them, or a "
+ "comma-separated key per --server (default: $LLAMA_API_KEY, else none)"
+ )
parser.add_argument(
"--dataset",
type=str,
default="aime",
- choices=["aime", "aime2025", "aime2026", "gsm8k", "gpqa"],
+ choices=["aime", "aime2025", "aime2026", "gsm8k", "gpqa", "humaneval",
+ "classeval", "agentic"],
help="Dataset type (default: aime)"
)
+ parser.add_argument(
+ "--dataset-source",
+ type=str,
+ default=None,
+ help="Override the dataset location: a local .jsonl path or a HuggingFace "
+ "repo id (e.g. a patched HumanEval). Default: the suite's canonical repo"
+ )
parser.add_argument(
"--n_cases",
type=int,
@@ -1407,6 +2443,25 @@ def main():
default=None,
help="Min P sampling (default: not passed)"
)
+ parser.add_argument(
+ "--retries",
+ type=int,
+ default=5,
+ help="Times to resend a request that errors or comes back empty before "
+ "recording the case as a server error (default: 5)"
+ )
+ parser.add_argument(
+ "--retry-delay",
+ type=float,
+ default=10.0,
+ help="Seconds to wait before the first resend; doubles on each further one (default: 10)"
+ )
+ parser.add_argument(
+ "--reasoning-effort",
+ type=str,
+ default=None,
+ help="Reasoning effort sent as reasoning_effort with every request, e.g. low/medium/high (default: not passed)"
+ )
parser.add_argument(
"--threads",
type=str,
@@ -1433,9 +2488,37 @@ def main():
parser.add_argument(
"--grader-type",
type=str,
- default="llm",
- choices=["regex", "cli", "llm"],
- help="Grader type: regex, cli, or llm (default: llm)"
+ default=None,
+ choices=["regex", "cli", "llm", "exec"],
+ help="Grader type: regex, cli, llm, or exec (default: exec for code suites, "
+ "llm otherwise). 'exec' runs the generated code against the suite tests "
+ "in a sandboxed venv"
+ )
+ parser.add_argument(
+ "--exec-timeout",
+ type=float,
+ default=None,
+ help="Per-problem wall-clock limit for the exec grader, in seconds "
+ "(default: per-suite, 15s for humaneval, 30s for classeval)"
+ )
+ parser.add_argument(
+ "--exec-seed",
+ type=int,
+ default=0,
+ help="Seed the global RNG before running each test, so suites whose tests "
+ "draw random inputs score reproducibly (default: 0). Use -1 to leave "
+ "the RNG unseeded, matching the stock upstream harness"
+ )
+ parser.add_argument(
+ "--sandbox-python",
+ type=str,
+ default=None,
+ help="Base interpreter used to build the execution venv (default: this one)"
+ )
+ parser.add_argument(
+ "--rebuild-sandbox",
+ action="store_true",
+ help="Discard and re-provision the cached execution venv"
)
parser.add_argument(
"--grader-script",
@@ -1455,6 +2538,51 @@ def main():
default="",
help="Model name for LLM grader (default: same as main model)"
)
+ parser.add_argument(
+ "--grader-api-key",
+ type=str,
+ default=None,
+ help="Bearer token for the LLM grader server (default: the first "
+ "--api-key when the grader runs on the main server, else none)"
+ )
+ parser.add_argument(
+ "--agentic-lang",
+ type=str,
+ default=None,
+ choices=["python", "typescript"],
+ help="Restrict the agentic suite to one language. Only that language's "
+ "toolchain is then provisioned."
+ )
+ parser.add_argument(
+ "--agentic-max-turns",
+ type=int,
+ default=50,
+ help="Maximum assistant turns per agentic episode (default: 50)"
+ )
+ parser.add_argument(
+ "--agentic-max-tokens",
+ type=int,
+ default=None,
+ help="Cap on generated tokens per agentic turn (default: no cap). "
+ "Bounds a model that falls into a repetition loop, but also cuts "
+ "off long reasoning; truncated turns are counted in the results."
+ )
+ parser.add_argument(
+ "--agentic-context-limit",
+ type=int,
+ default=None,
+ help="Abandon an agentic episode once its context exceeds this many "
+ "tokens (default: no limit)"
+ )
+ parser.add_argument(
+ "--agentic-transcripts",
+ type=Path,
+ default=None,
+ metavar="DIR",
+ help="Save each agentic episode's full message history to "
+ "DIR/<task_id>.json, with a readable DIR/<task_id>.html showing "
+ "every reasoning block, reply, tool call and tool result"
+ )
parser.add_argument(
"--resume",
action="store_true",
@@ -1463,6 +2591,23 @@ def main():
args = parser.parse_args()
+ global DATASET_SOURCE, AGENTIC_OPTS
+ DATASET_SOURCE = args.dataset_source
+ AGENTIC_OPTS = {
+ "lang": args.agentic_lang,
+ "max_turns": args.agentic_max_turns,
+ "context_limit": args.agentic_context_limit,
+ "max_tokens": args.agentic_max_tokens,
+ }
+
+ if args.grader_type is None:
+ args.grader_type = "exec" if args.dataset in CODE_SUITES else "llm"
+
+ if args.grader_type == "exec" and args.dataset not in CODE_SUITES:
+ print(f"Error: --grader-type exec is only meaningful for code suites "
+ f"({', '.join(sorted(CODE_SUITES))}), not '{args.dataset}'")
+ sys.exit(1)
+
# Parse server URLs and thread counts
server_urls = [u.strip() for u in args.server.split(",") if u.strip()]
thread_counts = [int(t.strip()) for t in args.threads.split(",") if t.strip()]
@@ -1480,11 +2625,30 @@ def main():
else:
server_names = server_urls # fallback to URLs
+ api_key_arg = args.api_key if args.api_key is not None else os.environ.get("LLAMA_API_KEY")
+ if api_key_arg:
+ api_keys = [k.strip() for k in api_key_arg.split(",")]
+ if len(api_keys) == 1:
+ api_keys = api_keys * len(server_urls)
+ elif len(api_keys) != len(server_urls):
+ print(f"Error: --api-key ({len(api_keys)} keys) must be a single key or "
+ f"match --server ({len(server_urls)} URLs)")
+ sys.exit(1)
+ else:
+ api_keys = [None] * len(server_urls)
+
server_configs = [
- ServerConfig(url=url, threads=threads, name=name)
- for url, threads, name in zip(server_urls, thread_counts, server_names)
+ ServerConfig(url=url, threads=threads, name=name, api_key=key)
+ for url, threads, name, key in zip(server_urls, thread_counts, server_names, api_keys)
]
+ if args.grader_api_key is not None:
+ grader_api_key = args.grader_api_key
+ else:
+ grader_url = args.grader_server or server_configs[0].url
+ grader_api_key = next(
+ (s.api_key for s in server_configs if s.url == grader_url), None)
+
if args.dataset == "gpqa" and args.grader_type != "llm":
print("Error: GPQA dataset requires --grader-type llm")
parser.print_help()
@@ -1529,6 +2693,7 @@ def main():
grader_script=args.grader_script,
grader_model_name=grader_model_name,
grader_server_url=grader_server_url,
+ grader_api_key=grader_api_key,
dataset_type=eval_state.dataset_type
)
resume = True
@@ -1548,6 +2713,7 @@ def main():
grader_script=args.grader_script,
grader_model_name=grader_model_name,
grader_server_url=grader_server_url,
+ grader_api_key=grader_api_key,
dataset_type=args.dataset
)
@@ -1563,6 +2729,8 @@ def main():
sampling_config["top_p"] = args.top_p
if args.min_p is not None:
sampling_config["min_p"] = args.min_p
+ if args.reasoning_effort is not None:
+ sampling_config["reasoning_effort"] = args.reasoning_effort
eval_state = EvalState(
dataset_type=args.dataset,
@@ -1577,14 +2745,53 @@ def main():
eval_state.print_all_tasks()
+ if args.grader_type == "exec":
+ dataset = eval_state.loaded_dataset
+
+ if dataset.is_agentic:
+ dataset.prepare(base_python=args.sandbox_python,
+ force=args.rebuild_sandbox)
+ else:
+ base_python = args.sandbox_python
+ want = dataset.preferred_python()
+ if not base_python and want:
+ base_python = resolve_python(want)
+ if base_python:
+ print(f"[sandbox] using Python {want} for "
+ f"{eval_state.dataset_type} ({base_python})")
+
+ dataset.configure_python(base_python or sys.executable)
+
+ sandbox = Sandbox(
+ name=eval_state.dataset_type,
+ packages=dataset.required_packages(),
+ provision=dataset.provision_steps(),
+ env=dataset.sandbox_env(),
+ base_python=base_python,
+ )
+ sandbox.ensure(force=args.rebuild_sandbox)
+ preflight_sandbox(sandbox)
+ grader.sandbox = sandbox
+ grader.dataset = dataset
+ grader.exec_timeout = (args.exec_timeout if args.exec_timeout is not None
+ else dataset.default_exec_timeout())
+ grader.exec_seed = None if args.exec_seed < 0 else args.exec_seed
+
processor = Processor(
server_configs=server_configs,
grader=grader,
model_name=args.model,
- n_predict=args.n_predict
+ n_predict=args.n_predict,
+ retries=args.retries,
+ retry_delay=args.retry_delay,
+ transcript_dir=args.agentic_transcripts,
)
processor.evaluate(eval_state, verbose=args.verbose, resume=resume)
+
+ if args.grader_type == "exec":
+ print_pass_at_k(eval_state)
+
print(f"\nEval state dumped to {args.output}")
if __name__ == "__main__":
diff --git a/examples/llama-eval/llama-server-simulator.py b/examples/llama-eval/llama-server-simulator.py
index e64ba8933..dbd46d67f 100755
--- a/examples/llama-eval/llama-server-simulator.py
+++ b/examples/llama-eval/llama-server-simulator.py
@@ -13,8 +13,6 @@ from typing import Dict, List, Optional
from dataclasses import dataclass
from pathlib import Path
-import datasets
-
# Set cache directory for HuggingFace datasets
cache_dir = Path.home() / ".cache" / "huggingface" / "datasets"
cache_dir.mkdir(parents=True, exist_ok=True)
@@ -76,6 +74,8 @@ class AimeDataset:
return question.get("problem", question.get("question", ""))
def _load_dataset(self):
+ import datasets
+
if self.dataset_type == "aime":
print(f"Loading AIME dataset (split: {self.split})...")
cache_path = Path.home() / ".cache" / "huggingface" / "datasets" / "AI-MO___aimo-validation-aime" / "default" / "0.0.0"
@@ -164,6 +164,58 @@ class AimeDataset:
return str(normalized) if normalized is not None else answer
return str(answer)
+class CodeDataset:
+ """Stand-in for a code model: replays canonical solutions as chat replies.
+
+ Handles both code suites: HumanEval keys off the function stub, ClassEval
+ off the class skeleton.
+ """
+
+ DEFAULT_SOURCE = {
+ "humaneval": "openai/openai_humaneval",
+ "classeval": "FudanSELab/ClassEval",
+ }
+
+ def __init__(self, dataset_type: str, source: Optional[str] = None):
+ self.dataset_type = dataset_type
+ self.source = source or self.DEFAULT_SOURCE[dataset_type]
+ self.questions: List[Dict] = []
+ self._load_dataset()
+
+ def _load_dataset(self):
+ print(f"Loading {self.dataset_type} dataset (source: {self.source})...")
+ local = Path(self.source)
+ if local.exists():
+ text = local.read_text()
+ if local.suffix == ".json":
+ rows = json.loads(text)
+ else:
+ rows = [json.loads(ln) for ln in text.splitlines() if ln.strip()]
+ else:
+ import datasets
+ rows = [dict(r) for r in datasets.load_dataset(self.source, split="test")]
+ self.questions = rows
+ print(f"{self.dataset_type} dataset loaded: {len(self.questions)} questions")
+
+ def _stub_field(self) -> str:
+ return "skeleton" if self.dataset_type == "classeval" else "prompt"
+
+ def _get_question_text(self, question: Dict) -> str:
+ return question[self._stub_field()]
+
+ def find_question(self, request_text: str) -> Optional[Dict]:
+ for question in self.questions:
+ stub = question[self._stub_field()].rstrip()
+ if stub and stub in request_text:
+ return question
+ return None
+
+ def get_answer(self, question: Dict) -> str:
+ if self.dataset_type == "classeval":
+ return question["solution_code"]
+ return question["prompt"] + question["canonical_solution"]
+
+
class Simulator:
def __init__(
self,
@@ -171,12 +223,17 @@ class Simulator:
host: str = "localhost",
success_rate: float = 0.8,
dataset_split: str = "train",
- dataset_type: str = "aime"
+ dataset_type: str = "aime",
+ dataset_source: Optional[str] = None
):
self.port = port
self.host = host
self.success_rate = success_rate
- self.dataset = AimeDataset(dataset_split, dataset_type)
+ self.dataset_type = dataset_type
+ if dataset_type in ("humaneval", "classeval"):
+ self.dataset = CodeDataset(dataset_type, dataset_source)
+ else:
+ self.dataset = AimeDataset(dataset_split, dataset_type)
self.eval_state = EvalState(
id=dataset_type,
tasks=[dataset_type],
@@ -196,6 +253,12 @@ class Simulator:
else:
response_text = self._generate_wrong_answer(question)
+ if self.dataset_type in ("humaneval", "classeval"):
+ response_text = (
+ f"Here is the implementation:\n\n```python\n{response_text}\n```\n\n"
+ "Let me know if you want it optimised further."
+ )
+
comp_tokens = random.randint(10000, 60000)
tps_gen = random.uniform(90.0, 110.0)
t_gen_ms = comp_tokens / tps_gen * 1000
@@ -229,6 +292,12 @@ class Simulator:
def _generate_wrong_answer(self, question: Dict) -> str:
expected_answer = self.dataset.get_answer(question)
+ if self.dataset_type == "classeval":
+ return re.sub(r"(\n\s+def [^\n]+:\n)", r"\1 return None\n",
+ question["skeleton"])
+ if self.dataset_type == "humaneval":
+ return question["prompt"] + " return None\n"
+
if expected_answer.isdigit():
wrong_answer = str(int(expected_answer) + 1)
else:
@@ -335,7 +404,7 @@ def main():
"--dataset",
type=str,
default="aime",
- choices=["aime", "aime2025"],
+ choices=["aime", "aime2025", "humaneval", "classeval"],
help="Dataset type (default: aime)"
)
parser.add_argument(
@@ -344,6 +413,12 @@ def main():
default="train",
help="AIME dataset split to use (default: train)"
)
+ parser.add_argument(
+ "--dataset-source",
+ type=str,
+ default=None,
+ help="Override the dataset location (local .jsonl path or HuggingFace repo id)"
+ )
args = parser.parse_args()
@@ -353,7 +428,8 @@ def main():
host=args.host,
success_rate=args.success_rate,
dataset_split=args.dataset_split,
- dataset_type=args.dataset
+ dataset_type=args.dataset,
+ dataset_source=args.dataset_source
)
server = HTTPServer((args.host, args.port), RequestHandler)