Building the Annotation Interface¶

instance_display, the display types, and what a screenshot can prove. It ships with the potato-tasks skill:

/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-tasks@potato

The page an annotator sees has two halves: the item on top, controlled by instance_display, and the form below it, controlled by annotation_schemes. The navbar, progress indicator, Next button and keyboard hints come for free and are worth leaving alone.

The item: `instance_display`¶

Without it, Potato renders the field named by item_properties.text_key as plain text. That is the right answer for a text classification task and the wrong one for everything else.

instance_display:
  fields:
    - key: body               # required: the field in your data file
      type: text              # required: from the display registry
      label: "Comment"        # optional heading above it
      span_target: true       # optional: spans may be drawn on this field
      display_options: {}     # optional, and type-specific
  layout:
    direction: vertical       # or horizontal
    gap: "20px"               # a CSS length, as a string
  resizable: true             # default true

fields is required and cannot be empty. Each entry needs key and type. An unknown type fails validation with the valid list in the message.

The 24 display types¶

Type	For
`text`	Plain text
`html`	Rich content (sanitized)
`code`	Source with syntax highlighting
`document`	DOCX, Markdown and similar
`pdf`	PDF via PDF.js
`spreadsheet`	Tables, annotated by row or cell
`image`	Images, optionally zoomable
`gallery`	Several images with captions
`depth_map`	Depth data with windowing and a colormap
`video`	Video player
`audio`	Audio player
`audio_dialogue`	Interview or podcast turns with per-turn playback
`dialogue`	Conversation turns, optionally threaded
`conversation_tree`	Branching conversations, collapsible
`multi_agent_discussion`	Several agents, colour-coded, filterable
`pairwise`	Two things side by side
`agent_trace`	Agent steps with type badges
`coding_trace`	Coding-agent steps with diffs and terminal blocks
`cot_trace`	Long chain-of-thought with a progress rail
`eval_trace`	Three panes: reasoning, calls, final answer
`web_agent_trace`	Screenshots with SVG overlays
`interactive_chat`	Live chat, then the trace
`live_agent`	Live agent viewer with controls
`live_coding_agent`	Live coding agent with intervention

display_options differ per type, and from 2.8.2-11 a key the display does not accept is a validation error that names the alternatives:

instance_display.fields[0].display_options.no_such_option is not an option for
display type 'text', so it would be ignored silently. Accepted options:
collapsed_by_default, collapsible, key, max_height, preserve_whitespace.

Before that release nothing looked inside display_options at all: a misspelled option passed validate --strict with "OK — no issues found" and the page came up wrong with no error anywhere. If your Potato predates it, spell-check these by hand — that is where an hour goes.

To see what a display takes:

from potato.server_utils.displays.registry import display_registry
[d for d in display_registry.list_displays() if d["name"] == "multi_agent_discussion"]
# optional_fields: ['collapse_environment', 'show_addressees', 'show_legend',
#  'show_turn_numbers', 'speaker_key', 'text_key', 'thread_replies']

A display that reads a list has its own idea of the field names. The conversation and trace displays (dialogue, multi_agent_discussion, agent_trace, cot_trace, eval_trace) take speaker_key and text_key, defaulting to speaker and text; gallery takes url_key and caption_key. Point one at turns shaped {index, agent, text} and every turn renders under a grey "U" avatar with no name, no colour and no legend, while the annotation schemes above and below it read the same list correctly, because those take agent_key. Set speaker_key: agent and the page is right. Nothing warns.

pairwise sits side by side at its defaults from Potato 2.8.2: cell_width defaults to auto, which splits the row evenly and subtracts each cell's share of the gap. Measured at 1280px, both cells 398px wide on the same line.

Before that the default was a literal 50% in a flex row with a 16px gap and flex-wrap: wrap, and two halves plus a gap never fit, so the second cell wrapped onto its own line at every width — a comparison display that never compared. On an older build, set it under half:

- {key: summaries, type: pairwise, label: Two summaries,
   display_options: {show_labels: true, cell_width: 48%}}

Geometry schemes need a display field for their media¶

A geometry scheme (image_annotation, spatial_annotation, region_caption, video_annotation …) draws on a bitmap it does not fetch. source_field names the data key; the canvas takes its image from the <img> that instance_display renders on the page. Remove the image field from instance_display and the canvas comes up empty — no <img> in the DOM at all — even though the form still carries data-source-field and the media URL is still in the HTML.

So an image annotation task needs both halves:

instance_display:
  fields:
    - {key: image, type: image, label: Screenshot}
annotation_schemes:
  - annotation_type: image_annotation
    name: regions
    description: Draw a box around each problem.
    source_field: image
    tools: [bbox]                # required; the error at boot names it
    labels: [Broken, Confusing]

and the consequence is that the image renders twice: once as the display field, once inside the canvas. That looks like a misconfiguration and is not. If the duplicate is distracting, hide the display copy rather than deleting it:

.display-field.display-type-image { position: absolute; left: -100000px; }
.display-field-resizable:has(> .display-field.display-type-image) { height: 0 !important; }

(Observed on a task built from this pack; the CSS is one working answer, not the only one.)

`span_target`¶

A span scheme works without instance_display at all — the default text rendering supplies the wrapper spans are measured against. span_target is how you name the anchor field once you define instance_display, and then exactly one contract applies:

display_registry.get_span_target_capable_types()

Twelve types accept it. Nine anchor spans by character offset in the standard .text-content wrapper; pdf, spreadsheet and agent_trace anchor their own way — into the PDF.js text layer, per cell, and per step. Setting span_target on anything else fails validation and the error lists what is allowed.

With more than one span-target field, name the field on the scheme so spans land in the right place.

The form: `annotation_schemes`¶

Three keys are required on every scheme regardless of type: annotation_type, name, description. description is the question text the annotator reads, so write it as a question.

Nine more work on every type:

Key	Effect
`label_requirement`	`{required: true}`, or bare `true`
`required`	Shorthand for the above
`display_logic`	Show this scheme only under a condition
`layout`	Where this scheme sits in the form grid — not how its choices are arranged. See below
`humanize_labels`	On by default. Turns `not_relevant` into "Not relevant" — and also title-cases labels that were already prose, giving "Information Or Advice". Set `false` for prose labels
`session_level`	One answer per session rather than per item
`turn_level`, `turn_binding`, `turn_label`	Bind the scheme to conversation turns

Everything else is type-specific and listed in annotation-types.md.

Quote `Yes` and `No` in a label list¶

labels: [Yes, No]        # parses as [True, False]
labels: ['Yes', 'No']    # what you meant

YAML reads Yes, No, On, Off, True and False — in any capitalisation — as booleans, so the most common label pair in annotation reaches Potato as two Python bools. The scheme then does not render at all: identifier_utils accepts a label that is a string or a dict and raises on anything else.

validate --strict catches it from Potato 2.8.2 on, and names the cause:

annotation_schemes[0].labels[0] came through as the boolean True. YAML reads
Yes/No, On/Off, Y/N or True/False in any capitalisation as a boolean [...]
Quote it: labels: ["Yes", "No"].

On any earlier build --strict said OK — no issues found — it checked that labels was a list and never looked at the elements — and the scheme simply did not render, with one line in the boot log as the only signal:

[ERROR] potato.server_utils.schemas.identifier_utils: Failed to generate layout
        for schema 'q1': Invalid label format: True

If that scheme was required, every Next after it failed with 400 /annotate — an annotator stuck on item one with no visible question to answer. grep "Invalid label format" server.log after any edit to a label list, on a version whose validator you have not tested.

Quoting is the fix and it costs nothing, so quote every short label. The dict form ({name: 'No', key_value: '2'}) is also safe, because the check that rejects a bool accepts a dict with a name.

`select`, `constant_sum` and `ranking`: fixed in 2.8.2-10, broken before it¶

Three widgets used to answer for the annotator or lose the answer they gave. All three are fixed as of v2.8.2-10-ge54fdc26, and all three were silent, so if you are on an earlier build the symptom is in the data rather than on the screen.

A select emitted one <option> per label and no placeholder, so the browser preselected the first one and syncAnnotationsFromDOM stored it. Register, press Next three times touching nothing, and three items came back annotated with the first label. required: true could never fail either, because a select always had a value. It now opens on a disabled -- select one -- and three empty Nexts store nothing.

constant_sum drew a live "Allocated: 40 / 100" counter and let Next through anyway, so a distribution question came back holding 40 points. Requiredness now reads the declared total and blocks with (40 of 100 allocated) in the message. soft_label had the same gap by a different route — min_per_label floors stopping the other sliders absorbing a drag — and the same check catches it.

ranking wrote its config order into the hidden input on every render and flagged it data-modified, so arriving at an item counted as ranking it, and returning to one overwrote the real ranking with the default. It now starts empty, and an order survives next-then-previous.

To check a build you did not test yourself, all three take about a minute: fresh user, three Nexts touching nothing, then read instance_id_to_label_to_value — it should be {}. Then rank something, advance, come back, and read it again.

range_slider had the milder version: both thumbs at a default band the server did not have, so an annotator who agreed with what was on screen recorded nothing. It now opens at the ends in grey, reading "No range set", with both hidden inputs empty and no data-modified until the first drag. (pairwise used to pre-select its tie tile the same way; that went in 2.8.2.)

`required` on a composite widget means "touched", not "answered"¶

A scheme that answers through a hidden JSON input — agent_scorecard, handoff_review, failure_attribution, consensus_tracking, tool_call_review, image_annotation, most of the trace family — is checked for requiredness by testing that input for a non-empty string. One click anywhere inside satisfies the whole scheme.

Measured on 2.8.2-10, with all three schemes on the page marked required: true. Answer nothing and Next is blocked, naming all three. Then pick a responsible agent, flag one of five handoffs, click one of ten scorecard cells, and Next goes through with:

scorecard = {"agents":{"planner":{"did its own job":1}},"team":{},"milestones":{}}
attribution = {"responsible_agent":"coder","decisive_step":null,"reason":""}

Four agents times two dimensions plus two team dimensions is ten questions, and the check is satisfied by one. constant_sum is the exception, because its budget rule was added by hand; nothing generalises it to declared cells.

So required on these buys you "the annotator interacted with it" and not much else. If completeness matters, say the count in the question text ("score all four"), and check it at export — the stored JSON has the shape you need, one key per cell, so a per-item count is a few lines of pandas rather than an eyeball pass.

Free text is one line unless you say otherwise¶

annotation_type: text renders <input type="text">. A question phrased "in a sentence" or "explain why" gets a single-line box, and annotators type a paragraph into it. Two ways to get a real textarea:

  - annotation_type: text
    name: why
    description: Why?
    multiline: true          # -> <textarea>
    rows: 4

    textarea: {"on": true, rows: 4, cols: 60}    # the older form; same result

Either way the input is named <scheme>:::text_box, so a selector that assumes textarea[name=…] breaks on the default. Use [name='<scheme>:::text_box'].

Keyboard shortcuts¶

key_value per label, on any type that has labels:

labels:
  - name: "Yes"
    key_value: y
  - name: "No"
    key_value: n

The key is key_value. key_binding looks right, validates, renders, and does nothing — an unrecognized key inside a label is not checked, not even by --strict. Confirm the shortcuts exist rather than assuming: potato preview prints a keybinding count per scheme, and --format json gives you the actual list. An empty list where you expected shortcuts means the key name is wrong.

sequential_key_binding: true on the scheme numbers the labels 1, 2, 3… instead, which is often what you want and cannot be misspelled per label.

Add shortcuts whenever there are nine or fewer choices. An annotator doing 500 items notices nothing else as much. horizontal_key_bindings: true lays the hints out in a row.

Conflicts are real, and only `preview` reports them¶

Two schemes both declaring key_value: '1':

$ potato validate config.yaml --strict     -> OK — no issues found.
$ potato preview config.yaml               -> KEYBINDING CONFLICTS:
                                                Key '1' used by both 'first:One' and 'second:Alpha'
$ grep -i conflict server.log              -> nothing

On the live page, pressing 1 sets the first scheme and leaves the second untouched. The second scheme's shortcut is silently dead. Nothing warns at startup — a claim to the contrary was in an earlier version of this file and is wrong.

Two behaviours to keep apart:

sequential_key_binding schemes are allocated distinct keys. Three schemes all using it come out as Q W E R T, then A S D F G H J, then N Y. Preview lists these as conflicts on 1, 2, 3 because it reports the raw claim rather than the allocation. Those are not real.
Explicit key_value is honoured as written and never reallocated. Two schemes claiming the same character collide for real, and the loser is silent. Mixing explicit key_value with sequential_key_binding elsewhere is the usual way to get one.

A scheme with its own tool keys — image_annotation reserves keys for the bbox and polygon tools — can have a label key and a tool key fire together on the same press. That generator does warn:

WARNING image_annotation: Keybinding conflict … key 'r' is bound to bbox tool and
also to label 'Confusing order'. Both will fire.

So there are two conflict detectors with different coverage and neither is the startup log. Press the key on a running page. It is the only check that covers both.

Conditional questions: `display_logic`¶

- annotation_type: radio            # the gate
  name: has_anecdote
  description: Does this post tell a personal story?
  labels: ["Yes", "No"]
  humanize_labels: false

- annotation_type: span
  name: anecdote_span
  description: Highlight the part that is the anecdote.
  labels: [Anecdote]
  display_logic:
    show_when:
      - schema: has_anecdote
        operator: equals
        value: "Yes"

Both schemes have to be in the same config. A show_when naming a scheme that is not there is a startup error, not a silent no-op -- which is the one good outcome in this area.

show_when is a non-empty list; conditions are ANDed. Operators: equals, not_equals, contains, not_contains, matches, empty, not_empty, gt, gte, lt, lte, in_range, not_in_range, length_gt, length_lt, length_in_range. empty and not_empty take no value; the range operators take [min, max].

schema must name a real scheme; a reference to a scheme that does not exist, or a cycle, fails validation.

not_equals is true on a gate nobody has answered yet. An unanswered scheme reads as undefined, and undefined is not equal to anything, so a follow-up gated with not_equals is on screen from the moment the page loads -- which is the opposite of what the block was added for. Gate on the positive values with equals, or AND a not_empty condition in front:

display_logic:
  show_when:
    - {schema: misleading, operator: not_empty}
    - {schema: misleading, operator: not_equals, value: "No"}

A tile scheme can be gated on, but check it on the page. pairwise, bws, ranking and triage answer by clicking a tile and writing to a hidden input, which the gate reads through a different path from the one radios and checkboxes use. Measured on Potato 2.8.2: a {schema: <triage>, operator: equals, value: accept} gate hides its dependent at load, shows it on accept and hides it again on reject. Before 2.8.2 it fired in neither direction. If the version is not yours to choose, drive the gate once in a browser before you design a page around it.

Two things to watch either way. A gate on a scheme inside a phase page takes the DOM fallback rather than the live answer map, and that path requires the widget to have marked its input data-modified -- triage does not. And what a tile scheme stores is not what it shows, so write the condition against the stored value (see below).

A tile scheme's stored value is not its label. pairwise records A, B and tie; labels: and tie_label: are what the annotator reads, not what lands in the data. Written as value: "Summary A" a condition could never have matched even if the gate worked, and the same names are what you filter on at analysis time.

A required span scheme is enforced server-side only: the client cannot see whether a span was drawn, so Next posts, the server answers 400 Required annotation(s) not completed: <scheme>, and the annotator gets a small corner toast naming the internal scheme name rather than the question. Requiring a span is fine; just make sure every item is one where drawing a span makes sense, quality-control items included.

A hidden scheme may still be required — the server skips requiredness checks for schemes their own display_logic is hiding. Answers already given to a scheme that later becomes hidden are kept and saved, so a respondent who changes the gate answer leaves values behind on the hidden follow-up. Tell the researcher: it is a data-cleaning rule, not a bug.

A conditional scheme is invisible to a screenshot. The render captures the page as it loads, so a scheme behind display_logic is not in the picture and its absence is not evidence of anything. Verify it another way: comment out the display_logic block, screenshot, put it back. Otherwise you are shipping the most important widget in the task unseen. This is a real failure mode: a task whose whole point is highlighting a span, verified with a screenshot in which the span widget does not appear.

Arranging the questions: `layout`¶

Without a layout block every scheme stacks full-width in config order, and a task with a dozen questions is a dozen screens of scrolling. Potato has a form grid with collapsible groups, and almost nothing points at it.

annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "What sentiment does this carry?"
    labels: [Positive, Negative]
  - annotation_type: multiselect
    name: topic
    description: "Which topics apply?"
    labels: [Politics, Sport]
  - annotation_type: text
    name: notes
    description: "Anything else worth recording?"
layout:
  grid:
    columns: 3                 # 1-6; the base grid
    gap: "0.75rem"
  breakpoints:
    mobile: 480                # one column below this
    tablet: 768                # spans reduced below this
  groups:
    - id: primary
      title: Primary classification
      description: Required for every item
      schemas: [sentiment, topic]
    - id: details
      title: Additional details
      collapsible: true
      collapsed_default: false
      schemas: [notes]

Every name in layout.groups.schemas must be a scheme in the same config — that is one of the few nested blocks where a wrong name fails at boot rather than being ignored.

Per scheme, layout says how much of that grid it takes:

- annotation_type: text
  name: notes
  description: Anything else?
  layout:
    columns: 3                 # span the full 3-column grid
    order: 9                   # explicit position
    min_width: "200px"
    max_width: "400px"
    align_self: start

columns is 1–6, rows is 1–4, both clamped rather than rejected. A working model is examples/advanced/grid-layout/; the full key list is in docs/configuration/form_layout.md and in config-keys-nested.md.

Validation is real here, unlike most nested blocks: a layout.grid.columns outside 1–6, a non-string layout.grid.gap, a duplicate group id, an empty layout.groups.schemas, or a group naming a scheme that does not exist all fail at boot with a message naming the problem.

Two things to know before using groups:

collapsed_default: true is a decision to have the question skipped. Use it only for questions most items genuinely do not need.
A slider in a narrow column is unreadable. The tick labels and the value bubble collide. Give slider, range_slider, soft_label and semantic_differential a columns: 2 span or more, or leave them full width.

Form density¶

No config key sets this, and it is easy to let a task grow into something slow to annotate one reasonable question at a time. What to reach for, in order:

One scheme per judgment, always. designing-a-task.md covers why.
display_logic so follow-ups appear only when they apply. A question that is not shown costs nothing.
A purpose-built type where the questions are really one question repeated: ten ratings on the same scale is one multirate rather than ten likert schemes.
layout.grid to put short questions side by side. The five schemes in examples/advanced/grid-layout/ come out in three rows rather than five.
layout.groups to give the rest headings, so the page reads as three sections rather than twelve questions.
Split the task if it is still long. Two passes over the corpus with five questions each cost the same annotator time as one pass with fifteen, and each pass has an annotator holding fewer definitions in their head at once.

Question count is a poor proxy for how long an item takes; how far the annotator scrolls is a better one, and you can measure it. Screenshot full_page=True and read the page height against a 900px viewport. Anything past two screens is worth restructuring.

Page-level appearance¶

Key	Effect
`annotation_instructions`	Collapsible instructions banner; text or a path
`header_file`	HTML rendered above the item
`custom_footer_html`	HTML on every page
`header_logo`	Logo in the navbar
`base_css`	An extra stylesheet
`task_layout`	Custom HTML for the form area; the escape hatch
`layout`	The form grid and groups — see Arranging the questions above
`hide_navbar`	Hide the top bar
`jumping_to_id_disabled`	Remove the jump-to-item control
`horizontal_key_bindings`	Lay shortcut hints out horizontally
`highlight_linebreaks`	Make line breaks visible in item text
`keyword_highlights_file`	Highlight given keywords in the item
`ui`, `ui_config`	Interface toggles
`ui_language`	Interface language code

Reach for task_layout last. Hand-written layout HTML stops tracking changes to the schema generators, and most of what people want it for is instance_display.layout plus scheme order.

Checking it¶

potato preview config.yaml                        # what the config declares
potato preview config.yaml --screenshot out.png   # what it looks like

The screenshot boots the task in a real browser and reports every uncaught exception, console.error and failed request. Most annotation UI is built by JavaScript after the HTML arrives, so validation cannot see any of it.

It captures the viewport, not the page. The PNG is 1280x900 — on a task with an instructions banner that is the banner and part of the item, and none of the questions. To see the whole page, drive a browser against a running server and screenshot with full_page=True: running-a-task.md has the driver.

Some of what it reports is not yours. Every phase page of a healthy task logs GET /api/current_instance 404 and two [SpanManager] errors, because a phase page has no instance. Filter those before reacting.

Look at the PNG. A clean exit means nothing threw, not that the interface is usable. Check: every question present, nothing overlapping, labels readable, the item rendered rather than showing a raw path or a broken image, choices not running off the edge.

What the screenshot cannot show you, and you have to check another way:

anything behind display_logic (see above)
anything that appears after an interaction: a span being drawn, a canvas tool, a media player mid-playback
phase pages: pass --phase consent (also instructions, training, poststudy)
how it behaves after navigating away and back, which is where saved state goes wrong

Common UI failures¶

Symptom	Cause
The item shows a file path instead of an image	No `instance_display`, so the path rendered as text
A span scheme with nothing to highlight	No field has `span_target: true`
`span_target is set but type X does not support span annotation`	Wrong display type; the error lists the twelve that work
A scheme is missing from the page	It is behind `display_logic` and the condition is not met
Choices run off the right edge	Many labels in a horizontal `layout`; use vertical
A keyboard shortcut does nothing	The key is `key_value`, not `key_binding`. Check with `potato preview` — an empty keybinding list means the label key is misspelled
A shortcut works but the wrong label lights up	Two labels claimed the same key; the conflict warned at startup
Labels read "Information Or Advice"	`humanize_labels` is on by default; set it `false`
A feature you configured does nothing	Typo'd key. Unknown keys only warn — run `potato validate --strict`

Building the Annotation Interface¶

The item: instance_display¶

The 24 display types¶

Geometry schemes need a display field for their media¶

span_target¶

The form: annotation_schemes¶

Quote Yes and No in a label list¶

select, constant_sum and ranking: fixed in 2.8.2-10, broken before it¶

required on a composite widget means "touched", not "answered"¶