Reviewing agent runs

How invited specialists audit agent trajectories, scores, failures, and policy-sensitive cases, and turn findings into better tasks and rubrics.

Audit and review is the core Expert Review work type. Your organization points its specialists at agent runs from Environments and scored results from Evaluation, and you judge whether the agent, the score, and the task itself hold up under expert scrutiny.

What you review

Item	What you check
Trajectories	Step-by-step agent runs: did each action make sense for the goal, and did the agent follow required procedure (confirmations, escalations, checks)?
Scores	Whether the assigned reward or grade matches what actually happened in the run
Failures	Why a run failed: agent error, ambiguous task, broken verifier, or environment issue
Policy-sensitive cases	Runs flagged for safety, privacy, or domain-policy concerns that need a specialist’s call
Task quality	Whether the task and its success criteria are unambiguous, checkable, and realistic for the domain

How to review a run

1. Read the brief first

   Every queue has a project brief with the rubric and scope. Review against that rubric, not personal preference.

2. Walk the trajectory

   Follow the run step by step against the task goal. Note where the agent diverged, skipped a required check, or acted on invented facts.

3. Check the score against the evidence

   Confirm the verifier outcome and reward match the run. A passing score on a run that violated domain procedure is a finding.

4. Classify the failure

   Separate agent mistakes from task or verifier defects. “The task was ambiguous” and “the agent was wrong” lead to different fixes.

5. Write the finding

   Record your judgment with a short rationale tied to specific steps in the run. Flag anything out of your specialty for reassignment instead of guessing.

What a good finding looks like

• Names the decisive step or span: “step 7 placed the order without the required confirmation turn”
• States the domain rule it violates or satisfies, in terms another specialist can verify
• Distinguishes agent error from task or scoring error
• Recommends a concrete fix when the defect is in the task or rubric: tighten a success criterion, add a failure mode, split an ambiguous goal

Vague findings (“seemed off”, “score feels harsh”) do not move tasks or rubrics. Falsifiable ones do.

Where findings go

Review findings feed back into the rest of the suite:

• Task and success-criteria fixes land in your org’s Environments task banks
• Rubric corrections and adjudicated hard cases tighten Evaluation datasets and human-review workflows
• Confirmed failures become regression cases so the same defect is caught automatically next time

Related

• RLHF evaluations: judging model outputs for preference and safety signal
• Domain writing for RL environments: authoring the tasks agents are graded on
• Policies: judgment must be your own; no external LLMs on task content