Agent Traces and Agent Evaluation¶

The trace displays, the twenty evaluation schemes, and how they bind. It ships with the potato-skill skill:

/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato

The largest family in Potato and the one least like text annotation: about twenty schema types, eight trace displays, and sixty example projects (57 under examples/agent-traces/, 3 under examples/agent-testing/). Its binding model fails silently when you get it wrong.

How a trace scheme finds its steps¶

Almost every scheme in this family reads a list off the item and renders one card per entry. The key it reads is configurable and defaulted:

Key	Default	Read by
`steps_key`	`steps`	`gui_trajectory`, `tool_call_review`, `process_reward`, `failure_attribution`, `agent_scorecard`, `trajectory_eval`, `trajectory_edit`, `multimodal_reasoning`, `emergent_behavior`, `handoff_review`, `agent_interaction_graph`
`turns_key`	varies	`consensus_tracking`, `context_attribution`
`agent_key`	varies	the multi-agent schemes, for who is speaking
`step_text_key`	varies	which field of a step holds the text to show

Almost nothing in this family has a required field. rubric_eval needs criteria; the rest need only name and description. So a scheme pointed at a key your data does not have renders an empty card list, validate --strict passes, and the boot log says nothing. If a trace widget comes up blank, check the key name before anything else:

python -c "import json;d=json.load(open('data/traces.json'));print(list(d[0]))"

Answers are stored as a hidden-input JSON list keyed by step index, and only judged steps are saved — the list is not positional, so do not read it as one entry per step.

Routing from what they said¶

They said	Reach for
"was each step right"	`trajectory_eval` (taxonomy + severity) or `tool_call_review` (right tool / args / ordering)
"where did it first go wrong"	`process_reward` in `first_error` mode
"label every step for PRM training"	`process_reward` in `per_step` mode
"fix the bad steps, don't just score them"	`trajectory_edit` — produces SFT/DPO data
"which agent caused the failure"	`failure_attribution`
"rate each agent and the team"	`agent_scorecard`
"did the handoff lose something"	`handoff_review`
"are they colluding / groupthinking / drifting"	`emergent_behavior`
"who used the shared resource when"	`tool_contention`
"how does the conversation use earlier context"	`context_attribution`
"did they actually reach consensus"	`consensus_tracking`
"draw me the interaction graph"	`agent_interaction_graph`
"did the click land on the right button"	`gui_trajectory`
"score it against a rubric"	`rubric_eval` (needs `criteria`)
"keep or drop this trace"	`triage`
"is this generated video physically plausible"	`rollout_evaluation`
"does our judge agree with our humans"	`judge_alignment`, `judge_calibration`
"compare two models' answers"	`arena`, or a `pairwise` / `bws` scheme
"traces arrive from production"	`trace_ingestion` (`/api/traces/*`) + `automation` + `triage`
"judge it while it runs"	`live_agent`, `live_coding_agent`, `agent_proxy`

Picking the display¶

The scheme decides the question; the display decides what the annotator sees. They are chosen separately, and several schemes render their own cards and need no display field at all.

Display	For	Span target
`agent_trace`	Generic steps with type badges	yes, per step
`coding_trace`	Diffs, terminal blocks, a file tree	yes
`cot_trace`	Long chain-of-thought with a progress rail	no
`eval_trace`	Three panes: reasoning, calls, final answer	yes
`web_agent_trace`	Screenshots with SVG overlays	no
`multi_agent_discussion`	Several agents, colour-coded, filterable	yes
`conversation_tree`	Branching, collapsible	no
`interactive_chat`	Live chat, then the trace	yes
`live_agent`, `live_coding_agent`	Streaming, with intervention controls	no

"Highlight the sentence where the reasoning broke" therefore works on eval_trace and coding_trace and not on cot_trace or web_agent_trace. If the researcher wants spans over a long chain of thought, that is cot_trace for reading plus a separate span scheme over the text field, or process_reward if what they actually want is per-step judgements.

Ask the registry rather than the flag: display_registry.get_span_target_capable_types() returns twelve; list_displays()'s supports_span_target says nine, because pdf, spreadsheet and agent_trace anchor spans their own way and are missing from it.

Quirks worth knowing before you build¶

gui_trajectory coordinates default to normalized. Each step may carry x/y for the grounding marker, read as 0..1 unless you set coord_space: pixels. A trace exported in pixels with the default setting is read as fractions, so a click at 420 becomes left: 42000% and the marker leaves the picture entirely. The card's own coordinate readout is the tell — click @ 42000%, 12000% beside a screenshot 640 pixels wide.

gui_trajectory takes the screenshot from screenshot, not screenshot_url. The step key is screenshot, falling back to image, and screenshot_key on the scheme renames it. screenshot_url is what the live agent runner writes into the steps it produces, so a hand-authored trace that copies that spelling gets "no screenshot" on every card while the file itself serves fine over /screenshots/. Check the route with curl before you suspect the file; if it answers 200, the key is what is wrong.

multimodal_reasoning infers the step type from the keys, and every type falls back to content. An image step reads image, image_url, url, src, then content; a tool step's arguments read args, arguments, tool_call.args, input, then content, and its name reads tool, name, tool_call.name, then the literal string tool; a text step reads text, content, reasoning. So a trace keyed uniformly on content renders correctly as long as each step carries an explicit type — the image draws and the tool call prints.

What uniform keying breaks is the type inference. Without type, the type comes from the presence of image/image_url or tool/tool_calls/tool_call, and a step that has neither is text. Drop type from a content-only trace and every card is labelled TEXT, with the image's URL printed as a line of prose rather than fetched. type_key names the field holding the type; the per-step content keys are not configurable.

Screenshots come from a second static root. Trace data that sets screenshot_url: screenshots/step_000.png is served from <task_dir>/screenshots/, which is separate from media_directory and not configurable. Miss it and every step image and filmstrip thumbnail 404s.

process_reward has two modes and they are different tasks. first_error (the default) asks for the first step that goes wrong and cascades from there; per_step labels every step. allow_neutral is only meaningful in per_step, because the cascade has no place for it. inline_with_trace puts the controls inside the trace cards instead of beside them, and ai_prelabel seeds suggestions the annotator corrects.

rubric_eval is the only one with a required field: criteria.

tool_contention needs times on its calls. A call is {agent, tool, start, end, resource} and the seconds are what make a contention: same resource, overlapping interval. Feed it {agent, tool, resource} and every call sits at 0-0 and can never overlap; from 2.8.2-11 it says the times are missing rather than reporting a clean trace. Self-pairs are skipped from the same release, so coder <-> coder cards no longer appear.

The display reads different keys from the schemes. Both halves of the page take the step list, and they disagree about what a step is called: the schemes take agent_key and steps_key, while multi_agent_discussion, agent_trace, cot_trace and eval_trace take speaker_key and text_key, defaulting to speaker and text. Steps shaped {index, agent, text} therefore render correctly in the questions and as a column of grey "U" avatars with no names in the trace above them. Set speaker_key: agent in display_options. (list_displays() did not admit the option existed until 2.8.2-11; on an older checkout, read it off the renderer class.)

eval_trace's third pane needs an answer-shaped speaker. The Final Answer pane matches a step whose speaker or tool name reads like an answer — final answer, send_message, respond, response, finish, submit, conclusion — and otherwise falls back to the last action in the trace. A step whose speaker is the obvious final matches none of those, so the pane shows the last tool call and the agent's real answer is rendered as an observation hanging off it. Name the last step Agent (Final Answer) or similar.

process_reward needs step_text_key when the steps are dicts. Pair it with cot_segmentation — the intended pairing, and what examples/agent-traces/cot-process-reward/config.yaml ships — and each step is a dict of {index, text, type, char_start, char_end}. Set step_text_key: text and the widget shows the prose; leave it at its default and the annotator is asked to judge the serialized dict. cot_segmentation also stamps every step it makes type: observation, which is what the cot_trace display badges them.

A linked act saves without its link. In consensus_tracking and context_attribution, an act named in linked_acts asks for a second click on the turn it refers to. Skip that click and the act is stored with no ref, Next advances, and the only sign is a badge reading agreement where a complete one reads agreement -> #1. The widget marks the card ct-awaiting-ref and nothing consults it. If the reference is the point of the study, count the ref-less acts at export before you count anything else.

required on these now counts the cells, from 2.8.2-11. Each widget writes what is still missing onto its own form and the banner beside Next reads it, so a scorecard covering four agents on two dimensions plus two team dimensions blocks with "Score each agent and the team as a whole (1 of 10 scored)" instead of passing on the first click. Verified on that config. The banner carries role="alert", so a screen reader gets it too.

The media widgets were not part of that change: video_annotation, temporal_grounding and tiered_annotation still treat one mark as an answer. modalities.md has the detail.

trajectory_eval's score ignores everything but severity. Mark all five steps incorrect, give each an error type, and skip the severity, and the widget shows Score: 100 / 100 and stores "score":100 — a flawless number on a trajectory the annotator condemned at every step. A correct step with a stray severity, meanwhile, is deducted. If you export score, make severity mandatory in the instructions and recompute from steps before you believe it.

trajectory_edit's require_reason_on_edit does not require a reason. It decides whether the reason field is drawn, nothing more: an edit with "reason": "" saves and Next advances. Read reason at export and treat blank as unexplained.

code_review stores empty comments. + Add Comment appends a row immediately and the row persists whether or not anything is typed into it — {"category":"bug","file":"","line":null,"text":""}. Drop rows with empty text before counting review comments, and expect "verdict": "" from an annotator who rated files and never picked one.

World-model rollouts¶

rollout_evaluation is the odd one out and worth reading the module docstring for. Several videos of the same scenario — real recording, model rollouts, optionally a counterfactual under an intervention — on one frame-locked timeline.

Its design decision is worth repeating to a researcher who asks for a plausibility scale: the primitive is a break-point, not a score. Rating a generated video 3/5 "produces a number that cannot be checked, cannot be localised, and cannot be used to fix anything." A frame index plus a violation type can be checked against the tensor and compared between annotators.

Three layers, three questions: violations (where does it break, and why), preference (which rollout is better), counterfactual (is the divergence plausible given the intervention). The third is the one that separates a world model from a video generator: a beautiful continuation that ignores the intervention is a failure no plausibility rating detects.

Three defaults that are on unless you turn them off: blind (hide generator names), shuffle (permute panel order per annotator), and require_clean — which refuses a submission that leaves a stream neither marked nor explicitly cleared, because "no marks" otherwise conflates "watched it, found nothing" with "never got to it".

Blinding and the DOM¶

rollout_evaluation blinds server-side: under blinding the labels are positional and the server chose the positions. Every other scheme does not. The whole data row is serialized into the page — data-instance-fields on the display container and a data-instance-json block — so a field you leave out of instance_display is still in the DOM and still readable by anyone who looks.

To withhold something from annotators, withhold it from the data file. Omitting it from the display is presentation, not access control.

Judges and automatic evaluators¶

Sixteen evaluators are registered and run without a human:

from potato.evaluators.registry import get_supported_evaluators
get_supported_evaluators()
# exact_match, contains, regex_match, json_valid, json_schema_match,
# edit_distance, embedding_similarity, tool_use, tool_call_accuracy,
# trajectory_match, llm_trajectory_judge, agent_as_judge, rubric_dag,
# groundedness, answer_relevance, context_relevance

Use them where the answer is checkable and save the annotators for what is not. The human side of the same question is judge_alignment (score items with a judge, compare against the humans, track agreement) and judge_calibration (auto-label a sample, have humans re-label the same items). If a researcher says "we want to know if we can trust the judge", that is the pair — not another rating scheme.

Live agents¶

live_agent and live_coding_agent run an agent inside the annotation page so annotators judge it as it happens; agent_proxy makes an agent the subject of annotation.

live_coding_agent runs the agent's tool calls inside a container. Two defaults decide whether the session can do anything: sandbox_image is python:3.12-slim and sandbox_network is none. Together they mean the agent has whatever that image ships and no way to install more — asked to run the tests, it gets pytest: not found and an exit code of 127, and burns its turns looking for a runner. Build or name an image that already carries the tooling the task needs: with pytest in the image the same session runs the tests, edits the file and re-runs them to green. working_dir is copied into a per-session workspace under .potato-sandboxes/, so the repository you point at is not the one the agent edits; check the copy, not the original, when you want to see what it did.

The task box the annotator sees is seeded from the field the display is bound to. Bind it to the field holding the task text, not to the one naming the repository, or the box arrives holding the repository name and the annotator has to retype the task before starting the agent. Which directory the agent gets is working_dir's job, so the key is free to carry the prompt.

live_agent drives a real browser, so it needs playwright and a browser installed (pip install playwright && playwright install chromium). Its endpoint_type takes anthropic_vision, ollama_vision or openai_vision — the last is what an OpenAI-compatible server such as vLLM needs, and the shipped example's comment names only the first two. The system prompt has to ask for an action that is an object, {"type": "click", ...}, because that is what the runner expects; a prompt that asks for a bare string gets one back and the session ends at step 0. These open control surfaces, so treat them the way you would treat anything that executes: do not expose them on a deployment that anyone with the URL can register for, and check deploying.md before handing out a link.

Verifying one¶

The failure mode is an empty card list, and an empty card list looks like a short trace.

Count what rendered against what is in the data. If the item has eleven steps, eleven cards should exist. This is the check that catches a wrong steps_key, and nothing else does.
Judge one step, navigate away and back. Answers are stored keyed by index with only-judged steps saved; a restore bug shows up as answers attaching to the wrong card.
Read user_state.json and confirm the stored JSON list has the shape the export expects.
For gui_trajectory, look at where the marker landed. That is the whole point of the scheme, and it is the thing coord_space silently breaks.