Evaluation Output

Eval logs are saved to the logs/ directory by default.

A local eval runs through inspect's eval_set over --log-dir (--run-config delegates to inspect's plain eval and never resumes): rerunning an identical command reuses its completed log, resumes a crashed one, and runs only the samples still missing. Identity is the task name, task args, model and limits, not environment code, so after changing an environment, or to collect an independent rollout, point --log-dir at a fresh directory or pass a new --run-name. Expansion selections such as --all get a date-stamped default run name, so resuming one across midnight needs an explicit --run-name. A selection may list several task files; each is its own eval in the directory, and a combination listed in several files keeps its first file's sample. A run that completed under --no-fail-on-error is complete, errored samples included; a rerun reuses them rather than retrying them. With upload enabled, a rerun uploads the reused log as a new run.

By default, ct run eval exits non-zero and does not convert or upload the run if Inspect ends with an error or cancellation. Pass --no-fail-on-error only when sample errors should be retained without failing the command.

# Custom output location
uv run ct run eval --log-dir my-logs/ --run-name experiment1

Eval files use Inspect's .eval format and can be viewed with:

# Open in Inspect viewer
inspect view logs/my-eval.eval

# Convert a specific sample to one trajectory file
uv run ct traj make logs/my-eval.eval --sample-id <sample_id> --save-path ./trajectories/traj.json

# Batch convert multiple evals
uv run ct runs make logs/attack_eval/example.eval --save-dir ./trajectories/