# Agent Traces and Agent Evaluation

The trace displays, the twenty evaluation schemes, and how they bind. It ships with the `potato-skill` skill:

```
/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato
```

---

The largest family in Potato and the one least like text annotation: about
twenty schema types, eight trace displays, and sixty example projects
(57 under `examples/agent-traces/`, 3 under `examples/agent-testing/`). Its
binding model fails silently when you get it wrong.

## How a trace scheme finds its steps

Almost every scheme in this family reads a **list off the item** and renders one
card per entry. The key it reads is configurable and defaulted:

| Key | Default | Read by |
|---|---|---|
| `steps_key` | `steps` | `gui_trajectory`, `tool_call_review`, `process_reward`, `failure_attribution`, `agent_scorecard`, `trajectory_eval`, `trajectory_edit`, `multimodal_reasoning`, `emergent_behavior`, `handoff_review`, `agent_interaction_graph` |
| `turns_key` | varies | `consensus_tracking`, `context_attribution` |
| `agent_key` | varies | the multi-agent schemes, for who is speaking |
| `step_text_key` | varies | which field of a step holds the text to show |

**Almost nothing in this family has a required field.** `rubric_eval` needs
`criteria`; the rest need only `name` and `description`. So a scheme pointed at
a key your data does not have renders an empty card list, `validate --strict`
passes, and the boot log says nothing. If a trace widget comes up blank, check
the key name before anything else:

```bash
python -c "import json;d=json.load(open('data/traces.json'));print(list(d[0]))"
```

Answers are stored as a hidden-input JSON list keyed by step `index`, and only
judged steps are saved — the list is **not** positional, so do not read it as
one entry per step.

## Routing from what they said

| They said | Reach for |
|---|---|
| "was each step right" | `trajectory_eval` (taxonomy + severity) or `tool_call_review` (right tool / args / ordering) |
| "where did it first go wrong" | `process_reward` in `first_error` mode |
| "label every step for PRM training" | `process_reward` in `per_step` mode |
| "fix the bad steps, don't just score them" | `trajectory_edit` — produces SFT/DPO data |
| "which agent caused the failure" | `failure_attribution` |
| "rate each agent and the team" | `agent_scorecard` |
| "did the handoff lose something" | `handoff_review` |
| "are they colluding / groupthinking / drifting" | `emergent_behavior` |
| "who used the shared resource when" | `tool_contention` |
| "how does the conversation use earlier context" | `context_attribution` |
| "did they actually reach consensus" | `consensus_tracking` |
| "draw me the interaction graph" | `agent_interaction_graph` |
| "did the click land on the right button" | `gui_trajectory` |
| "score it against a rubric" | `rubric_eval` (needs `criteria`) |
| "keep or drop this trace" | `triage` |
| "is this generated video physically plausible" | `rollout_evaluation` |
| "does our judge agree with our humans" | `judge_alignment`, `judge_calibration` |
| "compare two models' answers" | `arena`, or a `pairwise` / `bws` scheme |
| "traces arrive from production" | `trace_ingestion` (`/api/traces/*`) + `automation` + `triage` |
| "judge it while it runs" | `live_agent`, `live_coding_agent`, `agent_proxy` |

## Picking the display

The scheme decides the question; the display decides what the annotator sees.
They are chosen separately, and several schemes render their own cards and need
no display field at all.

| Display | For | Span target |
|---|---|---|
| `agent_trace` | Generic steps with type badges | yes, per step |
| `coding_trace` | Diffs, terminal blocks, a file tree | yes |
| `cot_trace` | Long chain-of-thought with a progress rail | no |
| `eval_trace` | Three panes: reasoning, calls, final answer | yes |
| `web_agent_trace` | Screenshots with SVG overlays | no |
| `multi_agent_discussion` | Several agents, colour-coded, filterable | yes |
| `conversation_tree` | Branching, collapsible | no |
| `interactive_chat` | Live chat, then the trace | yes |
| `live_agent`, `live_coding_agent` | Streaming, with intervention controls | no |

"Highlight the sentence where the reasoning broke" therefore works on
`eval_trace` and `coding_trace` and **not** on `cot_trace` or `web_agent_trace`.
If the researcher wants spans over a long chain of thought, that is
`cot_trace` for reading plus a separate `span` scheme over the text field, or
`process_reward` if what they actually want is per-step judgements.

Ask the registry rather than the flag:
`display_registry.get_span_target_capable_types()` returns twelve;
`list_displays()`'s `supports_span_target` says nine, because `pdf`,
`spreadsheet` and `agent_trace` anchor spans their own way and are missing from
it.

## Quirks worth knowing before you build

**`gui_trajectory` coordinates default to normalized.** Each step may carry
`x`/`y` for the grounding marker, read as 0..1 **unless** you set
`coord_space: pixels`. A trace exported in pixels with the default setting is
read as fractions, so a click at 420 becomes `left: 42000%` and the marker
leaves the picture entirely. The card's own coordinate readout is the tell —
`click @ 42000%, 12000%` beside a screenshot 640 pixels wide.

**`gui_trajectory` takes the screenshot from `screenshot`, not
`screenshot_url`.** The step key is `screenshot`, falling back to `image`, and
`screenshot_key` on the scheme renames it. `screenshot_url` is what the live
agent runner *writes* into the steps it produces, so a hand-authored trace that
copies that spelling gets "no screenshot" on every card while the file itself
serves fine over `/screenshots/`. Check the route with `curl` before you suspect
the file; if it answers 200, the key is what is wrong.

**`multimodal_reasoning` infers the step type from the keys, and every type
falls back to `content`.** An image step reads `image`, `image_url`, `url`,
`src`, then `content`; a tool step's arguments read `args`, `arguments`,
`tool_call.args`, `input`, then `content`, and its name reads `tool`, `name`,
`tool_call.name`, then the literal string `tool`; a text step reads `text`,
`content`, `reasoning`. So a trace keyed uniformly on `content` renders
correctly *as long as each step carries an explicit `type`* — the image draws
and the tool call prints.

What uniform keying breaks is the type inference. Without `type`, the type comes
from the presence of `image`/`image_url` or `tool`/`tool_calls`/`tool_call`, and
a step that has neither is text. Drop `type` from a `content`-only trace and
every card is labelled TEXT, with the image's URL printed as a line of prose
rather than fetched. `type_key` names the field holding the type; the per-step
content keys are not configurable.

**Screenshots come from a second static root.** Trace data that sets
`screenshot_url: screenshots/step_000.png` is served from `<task_dir>/screenshots/`,
which is separate from `media_directory` and not configurable. Miss it and every
step image and filmstrip thumbnail 404s.

**`process_reward` has two modes and they are different tasks.**
`first_error` (the default) asks for the first step that goes wrong and cascades
from there; `per_step` labels every step. `allow_neutral` is only meaningful in
`per_step`, because the cascade has no place for it. `inline_with_trace` puts the
controls inside the trace cards instead of beside them, and `ai_prelabel` seeds
suggestions the annotator corrects.

**`rubric_eval` is the only one with a required field:** `criteria`.

**`tool_contention` needs times on its calls.** A call is `{agent, tool, start,
end, resource}` and the seconds are what make a contention: same resource,
overlapping interval. Feed it `{agent, tool, resource}` and every call sits at
0-0 and can never overlap; from 2.8.2-11 it says the times are missing rather
than reporting a clean trace. Self-pairs are skipped from the same release, so
`coder <-> coder` cards no longer appear.

**The display reads different keys from the schemes.** Both halves of the page
take the step list, and they disagree about what a step is called: the schemes
take `agent_key` and `steps_key`, while `multi_agent_discussion`, `agent_trace`,
`cot_trace` and `eval_trace` take `speaker_key` and `text_key`, defaulting to
`speaker` and `text`. Steps shaped `{index, agent, text}` therefore render
correctly in the questions and as a column of grey "U" avatars with no names in
the trace above them. Set `speaker_key: agent` in `display_options`. (`list_displays()` did not admit
the option existed until 2.8.2-11; on an older checkout, read it off the
renderer class.)

**`eval_trace`'s third pane needs an answer-shaped speaker.** The Final Answer
pane matches a step whose speaker or tool name reads like an answer — `final
answer`, `send_message`, `respond`, `response`, `finish`, `submit`, `conclusion`
— and otherwise falls back to the last action in the trace. A step whose speaker
is the obvious `final` matches none of those, so the pane shows the last tool
call and the agent's real answer is rendered as an observation hanging off it.
Name the last step `Agent (Final Answer)` or similar.

**`process_reward` needs `step_text_key` when the steps are dicts.** Pair it
with `cot_segmentation` — the intended pairing, and what
`examples/agent-traces/cot-process-reward/config.yaml` ships — and each step is
a dict of `{index, text, type, char_start, char_end}`. Set `step_text_key: text`
and the widget shows the prose; leave it at its default and the annotator is
asked to judge the serialized dict. `cot_segmentation` also stamps every step it
makes `type: observation`, which is what the `cot_trace` display badges them.

**A linked act saves without its link.** In `consensus_tracking` and
`context_attribution`, an act named in `linked_acts` asks for a second click on
the turn it refers to. Skip that click and the act is stored with no `ref`, Next
advances, and the only sign is a badge reading `agreement` where a complete one
reads `agreement -> #1`. The widget marks the card `ct-awaiting-ref` and nothing
consults it. If the reference is the point of the study, count the ref-less acts
at export before you count anything else.

**`required` on these now counts the cells, from 2.8.2-11.** Each widget writes
what is still missing onto its own form and the banner beside Next reads it, so
a scorecard covering four agents on two dimensions plus two team dimensions
blocks with "Score each agent and the team as a whole (1 of 10 scored)" instead
of passing on the first click. Verified on that config. The banner carries
`role="alert"`, so a screen reader gets it too.

The media widgets were not part of that change: `video_annotation`,
`temporal_grounding` and `tiered_annotation` still treat one mark as an answer.
`modalities.md` has the detail.

**`trajectory_eval`'s score ignores everything but severity.** Mark all five
steps `incorrect`, give each an error type, and skip the severity, and the
widget shows `Score: 100 / 100` and stores `"score":100` — a flawless number on
a trajectory the annotator condemned at every step. A `correct` step with a
stray severity, meanwhile, is deducted. If you export `score`, make severity
mandatory in the instructions and recompute from `steps` before you believe it.

**`trajectory_edit`'s `require_reason_on_edit` does not require a reason.** It
decides whether the reason field is drawn, nothing more: an edit with
`"reason": ""` saves and Next advances. Read `reason` at export and treat blank
as unexplained.

**`code_review` stores empty comments.** *+ Add Comment* appends a row
immediately and the row persists whether or not anything is typed into it —
`{"category":"bug","file":"","line":null,"text":""}`. Drop rows with empty
`text` before counting review comments, and expect `"verdict": ""` from an
annotator who rated files and never picked one.

## World-model rollouts

`rollout_evaluation` is the odd one out and worth reading the module docstring
for. Several videos of the same scenario — real recording, model rollouts,
optionally a counterfactual under an intervention — on one frame-locked
timeline.

Its design decision is worth repeating to a researcher who asks for a
plausibility scale: **the primitive is a break-point, not a score.** Rating a
generated video 3/5 "produces a number that cannot be checked, cannot be
localised, and cannot be used to fix anything." A frame index plus a violation
type can be checked against the tensor and compared between annotators.

Three layers, three questions: `violations` (where does it break, and why),
`preference` (which rollout is better), `counterfactual` (is the divergence
plausible *given* the intervention). The third is the one that separates a world
model from a video generator: a beautiful continuation that ignores the
intervention is a failure no plausibility rating detects.

Three defaults that are on unless you turn them off: `blind` (hide generator
names), `shuffle` (permute panel order per annotator), and `require_clean` —
which refuses a submission that leaves a stream neither marked nor explicitly
cleared, because "no marks" otherwise conflates "watched it, found nothing" with
"never got to it".

## Blinding and the DOM

`rollout_evaluation` blinds server-side: under blinding the labels are
positional and the server chose the positions. **Every other scheme does not.**
The whole data row is serialized into the page — `data-instance-fields` on the
display container and a `data-instance-json` block — so a field you leave out of
`instance_display` is still in the DOM and still readable by anyone who looks.

To withhold something from annotators, withhold it from the data file. Omitting
it from the display is presentation, not access control.

## Judges and automatic evaluators

Sixteen evaluators are registered and run without a human:

```python
from potato.evaluators.registry import get_supported_evaluators
get_supported_evaluators()
# exact_match, contains, regex_match, json_valid, json_schema_match,
# edit_distance, embedding_similarity, tool_use, tool_call_accuracy,
# trajectory_match, llm_trajectory_judge, agent_as_judge, rubric_dag,
# groundedness, answer_relevance, context_relevance
```

Use them where the answer is checkable and save the annotators for what is not.
The human side of the same question is `judge_alignment` (score items with a
judge, compare against the humans, track agreement) and `judge_calibration`
(auto-label a sample, have humans re-label the same items). If a researcher
says "we want to know if we can trust the judge", that is the pair — not another
rating scheme.

## Live agents

`live_agent` and `live_coding_agent` run an agent inside the annotation page so
annotators judge it as it happens; `agent_proxy` makes an agent the subject of
annotation.

`live_coding_agent` runs the agent's tool calls inside a container. Two defaults
decide whether the session can do anything: `sandbox_image` is `python:3.12-slim`
and `sandbox_network` is `none`. Together they mean the agent has whatever that
image ships and no way to install more — asked to run the tests, it gets
`pytest: not found` and an exit code of 127, and burns its turns looking for a
runner. Build or name an image that already carries the tooling the task needs:
with `pytest` in the image the same session runs the tests, edits the file and
re-runs them to green. `working_dir` is copied into a per-session workspace under
`.potato-sandboxes/`, so the repository you point at is not the one the agent
edits; check the copy, not the original, when you want to see what it did.

The task box the annotator sees is seeded from the field the display is bound to.
Bind it to the field holding the task text, not to the one naming the repository,
or the box arrives holding the repository name and the annotator has to retype
the task before starting the agent. Which directory the agent gets is `working_dir`'s
job, so the key is free to carry the prompt.

`live_agent` drives a real browser, so it needs `playwright` and a browser
installed (`pip install playwright && playwright install chromium`). Its
`endpoint_type` takes `anthropic_vision`, `ollama_vision` or `openai_vision` —
the last is what an OpenAI-compatible server such as vLLM needs, and the shipped
example's comment names only the first two. The system prompt has to ask for an
`action` that is an **object**, `{"type": "click", ...}`, because that is what
the runner expects; a prompt that asks for a bare string gets one back and the
session ends at step 0. These open control surfaces, so treat them the way you would treat
anything that executes: do not expose them on a deployment that anyone with the
URL can register for, and check `deploying.md` before handing out a link.

## Verifying one

The failure mode is an empty card list, and an empty card list looks like a
short trace.

1. **Count what rendered against what is in the data.** If the item has eleven
   steps, eleven cards should exist. This is the check that catches a wrong
   `steps_key`, and nothing else does.
2. **Judge one step, navigate away and back.** Answers are stored keyed by
   `index` with only-judged steps saved; a restore bug shows up as answers
   attaching to the wrong card.
3. **Read `user_state.json`** and confirm the stored JSON list has the shape the
   export expects.
4. **For `gui_trajectory`, look at where the marker landed.** That is the whole
   point of the scheme, and it is the thing `coord_space` silently breaks.