` — the task name lives in the navbar as plain text — so there is no top-level landmark for heading navigation. Custom canvas widgets are a separate question from the form; check those yourself if your annotators rely on assistive technology. ### The 24 display types | Type | For | |---|---| | `text` | Plain text | | `html` | Rich content (sanitized) | | `code` | Source with syntax highlighting | | `document` | DOCX, Markdown and similar — files, not a string field. See below | | `pdf` | PDF via PDF.js | | `spreadsheet` | Tables, annotated by row or cell | | `image` | Images, optionally zoomable | | `gallery` | Several images with captions; from 2.9.0, videos too | | `depth_map` | Depth data with windowing and a colormap | | `video` | Video player | | `audio` | Audio player | | `audio_dialogue` | Interview or podcast turns; one recording, per-turn play buttons that seek into it | | `dialogue` | Conversation turns, optionally threaded | | `conversation_tree` | Branching conversations, collapsible | | `multi_agent_discussion` | Several agents, colour-coded, filterable | | `pairwise` | Two things side by side | | `agent_trace` | Agent steps with type badges | | `coding_trace` | Coding-agent steps with diffs and terminal blocks | | `cot_trace` | Long chain-of-thought with a progress rail | | `eval_trace` | Three panes: reasoning, calls, final answer | | `web_agent_trace` | Screenshots with SVG overlays | | `interactive_chat` | Live chat, then the trace | | `live_agent` | Live agent viewer with controls | | `live_coding_agent` | Live coding agent with intervention | `display_options` differ per type, and from 2.8.2-11 a key the display does not accept is a validation error that names the alternatives: ``` instance_display.fields[0].display_options.no_such_option is not an option for display type 'text', so it would be ignored silently. Accepted options: collapsed_by_default, collapsible, key, max_height, preserve_whitespace. ``` Before that release nothing looked inside `display_options` at all: a misspelled option passed `validate --strict` with "OK — no issues found" and the page came up wrong with no error anywhere. If your Potato predates it, spell-check these by hand — that is where an hour goes. To see what a display takes: ```python from potato.server_utils.displays.registry import display_registry [d for d in display_registry.list_displays() if d["name"] == "multi_agent_discussion"] # optional_fields: ['collapse_environment', 'show_addressees', 'show_legend', # 'show_turn_numbers', 'speaker_key', 'text_key', 'thread_replies'] ``` **A display that reads a list has its own idea of the field names.** The conversation and trace displays (`dialogue`, `multi_agent_discussion`, `agent_trace`, `cot_trace`, `eval_trace`) take `speaker_key` and `text_key`, defaulting to `speaker` and `text`; `gallery` takes `url_key` and `caption_key`. Point one at turns shaped `{index, agent, text}` and the turns render with no speaker at all — just the text, in a `dialogue-turn` div with an empty `speaker-0` class and no name anywhere — while the annotation schemes above and below it read the same list correctly, because those take `agent_key`. Set `speaker_key: agent` and the names appear, with a colour per speaker. Nothing warns, and a reader skimming the page sees a conversation rather than an error. **`spreadsheet` takes `headers` or `columns`, and both work.** Give it `{headers: [...], rows: [[...]]}`, `{columns: [...], rows: [[...]]}`, or a plain list of row dicts, in which case the headers come from the first row's keys. I put the first two side by side on one page and they render identically, header row included. **`pairwise` takes its labels from `display_options`, not from the item.** `labels: ["Shift supervisor", "Automated report"]` names the two cells for the whole study; a `label` carried inside the data is ignored and the cells read "A" and "B". The data itself is `{left: ..., right: ...}` or a two-element list. **`pairwise` sits side by side at its defaults** from Potato 2.8.2: `cell_width` defaults to `auto`, which splits the row evenly and subtracts each cell's share of the gap. Measured at 1280px, both cells 398px wide on the same line. Before that the default was a literal `50%` in a flex row with a 16px gap and `flex-wrap: wrap`, and two halves plus a gap never fit, so the second cell wrapped onto its own line at every width — a comparison display that never compared. On an older build, set it under half: ```yaml - {key: summaries, type: pairwise, label: Two summaries, display_options: {show_labels: true, cell_width: 48%}} ``` **`document` expects a file that Potato ingested, not a field in your JSON.** The renderer reads `{rendered_html, metadata, text}` produced by the format pipeline (`data_directory`, `format_handling`). Point it at a plain string and that string is emitted as the field's HTML: Markdown is not rendered and `**Speaker:**` shows its asterisks, which setting `format_handling: {enabled: true, default_format: markdown}` does not change — that block renders markup in a field the pipeline already produced, not a raw string you supplied. Paragraph breaks do survive: the wrapper carries `document-preserve-structure` and `white-space: pre-wrap` whatever you set, so `preserve_structure` is a no-op because it is always on rather than because there is nothing to act on. `show_outline` draws nothing. `style_theme` reaches the DOM as a `document-theme-` class and takes `minimal` or `print` besides the default; a fourth value is refused at boot with all three named. It is a working span target either way, and spans anchor to the raw field value, newlines included. For a transcript you are shipping in a data file, `text` is the display you want. ### Geometry schemes need a display field for their media A geometry scheme (`image_annotation`, `spatial_annotation`, `region_caption`, `video_annotation` …) draws on a bitmap it does **not** fetch. `source_field` names the data key; the canvas takes its image from the `` that `instance_display` renders on the page. Remove the image field from `instance_display` and the canvas comes up empty — no `` in the DOM at all — even though the form still carries `data-source-field` and the media URL is still in the HTML. So an image annotation task needs both halves: ```yaml instance_display: fields: - {key: image, type: image, label: Screenshot} annotation_schemes: - annotation_type: image_annotation name: regions description: Draw a box around each problem. source_field: image tools: [bbox] # required; the error at boot names it labels: [Broken, Confusing] ``` and the consequence is that **the image renders twice**: once as the display field, once inside the canvas. That looks like a misconfiguration and is not. If the duplicate is distracting, hide the display copy rather than deleting it: ```css .display-field.display-type-image { position: absolute; left: -100000px; } .display-field-resizable:has(> .display-field.display-type-image) { height: 0 !important; } ``` (Observed on a task built from this pack; the CSS is one working answer, not the only one.) ### `span_target` A `span` scheme works without `instance_display` at all — the default text rendering supplies the wrapper spans are measured against. `span_target` is how you name the anchor field *once you define `instance_display`*, and then exactly one contract applies: ```python display_registry.get_span_target_capable_types() ``` Twelve types accept it. Nine anchor spans by character offset in the standard `.text-content` wrapper; `pdf`, `spreadsheet` and `agent_trace` anchor their own way — into the PDF.js text layer, per cell, and per step. Setting `span_target` on anything else fails validation and the error lists what is allowed. **`target_field` is about schemes, not fields.** The field a span lands in comes from where the drag was, always: I put three span-target fields on one page with a single scheme naming none of them, and all three spans recorded against the right field with the right offsets. So you do not need it to make a span land correctly. What it does is say which field a scheme *owns*, which is how a drag is routed to the right scheme when several span schemes share a page. Name it when you have more than one span **scheme**, not when you have more than one span-target **field**. It does not restrict: a scheme pinned to one field still records elsewhere, deliberately, because the alternative is an author pinning their only scheme to one field and silently losing every drag in the others. **A geometry scheme finds its media in `instance_display` if it has to.** `image_annotation` looks at `text_key` first, then falls back to the rendered image on the page, so a task with an `image` display field works without any extra wiring. Name `source_field` on the scheme anyway when the item carries more than one image, since the fallback takes the first one it finds; the console says which it used. **A span's offsets index the field as it was rendered, not as the data file holds it.** On a plain text field those are the same string. On a `dialogue` field they are not: the anchor is the turns joined as `"{speaker}: {text}"`, one per line, so an offset of 145 counts the speaker labels and their separators too. Reconstruct that exact string before slicing a dialogue span, or read the span's `text` from the export instead of computing it. It follows that **a display option which changes the rendering changes the anchor**. Turning on `show_turn_numbers` puts a `[1] ` in front of every turn and moves every offset after it. Settle the display options for a span-target field before anyone annotates against it, and treat changing them afterwards the way you would treat editing the item text. **Several span schemes can share one field, and their spans are independent.** Two schemes anchored to the same `target_field` — one marking entities, one marking which sentence is first-hand — draw over each other on the page and are stored separately, each span carrying its own `schema` and `target_field`. Use that instead of folding two questions into one label set. A span may also enclose one already drawn, in the same scheme or another, and both are kept. Spans do not live with the labels on disk. They are their own top-level key in `user_state.json`, `instance_id_to_span_to_value`, so a scheme that looks empty in `instance_id_to_label_to_value` may be fully annotated — see `getting-the-data-out.md` for the exported shape. ## The form: `annotation_schemes` Three keys are required on every scheme regardless of type: `annotation_type`, `name`, `description`. `description` is the question text the annotator reads, so write it as a question. **`name` is the storage key, so it has to be unique within the phase.** Since 2.8.2 two schemes sharing a name are refused, and the message says why: they also share one HTML input group, so answering the second clears the first and only one answer is saved. It applies across types: a `radio` and a `multiselect` sharing one name collide the same way, because the collision is the key rather than the widget. Names differing only in case are two names, and the same name on a phase page and in the annotation phase is fine, since those store separately. Nine more work on every type: | Key | Effect | |---|---| | `label_requirement` | `{required: true}`, or bare `true` | | `required` | Shorthand for the above | | `display_logic` | Show this scheme only under a condition | | `layout` | Where this scheme sits in the form grid — **not** how its choices are arranged. See below | | `humanize_labels` | **On by default.** Turns `not_relevant` into "Not relevant" — and also title-cases labels that were already prose, giving "Information **Or** Advice". Set `false` for prose labels | | `session_level` | One answer per session rather than per item | | `turn_level`, `turn_binding`, `turn_label` | Bind the scheme to conversation turns | Everything else is type-specific and listed in `annotation-types.md`. An `instance_display` field's own options — the ones `describe_display_type` lists as optional for that type — can be written flat on the field or nested under `display_options`, and the nested form wins when both are given. Only keys the type actually declares are picked up flat, so a misspelling is still silently ignored: check the rendered page, not the config. ### Quote `Yes` and `No` in a label list ```yaml labels: [Yes, No] # parses as [True, False] labels: ['Yes', 'No'] # what you meant ``` YAML reads `Yes`, `No`, `On`, `Off`, `True` and `False` — in any capitalisation — as booleans, so the most common label pair in annotation reaches Potato as two Python bools. The scheme then **does not render at all**: `identifier_utils` accepts a label that is a string or a dict and raises on anything else. `validate --strict` catches it from Potato 2.8.2 on, and names the cause: ``` annotation_schemes[0].labels[0] came through as the boolean True. YAML reads Yes/No, On/Off, Y/N or True/False in any capitalisation as a boolean [...] Quote it: labels: ["Yes", "No"]. ``` On any earlier build `--strict` said `OK — no issues found` — it checked that `labels` was a list and never looked at the elements — and the scheme simply **did not render**, with one line in the boot log as the only signal: ``` [ERROR] potato.server_utils.schemas.identifier_utils: Failed to generate layout for schema 'q1': Invalid label format: True ``` If that scheme was `required`, every `Next` after it failed with `400 /annotate` — an annotator stuck on item one with no visible question to answer. `grep "Invalid label format" server.log` after any edit to a label list, on a version whose validator you have not tested. Quoting is the fix and it costs nothing, so quote every short label. The dict form (`{name: 'No', key_value: '2'}`) is also safe, because the check that rejects a bool accepts a dict with a `name`. ### `select`, `constant_sum` and `ranking`: fixed in 2.8.2-10, broken before it Three widgets used to answer for the annotator or lose the answer they gave. All three are fixed as of `v2.8.2-10-ge54fdc26`, and all three were silent, so if you are on an earlier build the symptom is in the data rather than on the screen. A `select` emitted one `` per label and no placeholder, so the browser preselected the first one and `syncAnnotationsFromDOM` stored it. Register, press Next three times touching nothing, and three items came back annotated with the first label. `required: true` could never fail either, because a select always had a value. It now opens on a disabled `-- select one --` and three empty Nexts store nothing. `constant_sum` drew a live "Allocated: 40 / 100" counter and let Next through anyway, so a distribution question came back holding 40 points. Requiredness now reads the declared total and blocks with `(40 of 100 allocated)` in the message. `soft_label` had the same gap by a different route — `min_per_label` floors stopping the other sliders absorbing a drag — and the same check catches it. `ranking` wrote its config order into the hidden input on every render and flagged it `data-modified`, so arriving at an item counted as ranking it, and returning to one overwrote the real ranking with the default. It now starts empty, and an order survives next-then-previous. To check a build you did not test yourself, all three take about a minute: fresh user, three Nexts touching nothing, then read `instance_id_to_label_to_value` — it should be `{}`. Then rank something, advance, come back, and read it again. `range_slider` had the milder version: both thumbs at a default band the server did not have, so an annotator who agreed with what was on screen recorded nothing. It now opens at the ends in grey, reading "No range set", with both hidden inputs empty and no `data-modified` until the first drag. (`pairwise` used to pre-select its tie tile the same way; that went in 2.8.2.) ### `required` on a composite widget means "touched", not "answered" A scheme that answers through a hidden JSON input — `agent_scorecard`, `handoff_review`, `failure_attribution`, `consensus_tracking`, `tool_call_review`, `image_annotation`, most of the trace family — is checked for requiredness by testing that input for a non-empty string. One click anywhere inside satisfies the whole scheme. Measured on 2.8.2-10, with all three schemes on the page marked `required: true`. Answer nothing and Next is blocked, naming all three. Then pick a responsible agent, flag one of five handoffs, click one of ten scorecard cells, and Next goes through with: ``` scorecard = {"agents":{"planner":{"did its own job":1}},"team":{},"milestones":{}} attribution = {"responsible_agent":"coder","decisive_step":null,"reason":""} ``` Four agents times two dimensions plus two team dimensions is ten questions, and the check is satisfied by one. `constant_sum` is the exception, because its budget rule was added by hand; nothing generalises it to declared cells. So `required` on these buys you "the annotator interacted with it" and not much else. If completeness matters, say the count in the question text ("score all four"), and check it at export — the stored JSON has the shape you need, one key per cell, so a per-item count is a few lines of pandas rather than an eyeball pass. ### Free text is one line unless you say otherwise `annotation_type: text` renders ``. A question phrased "in a sentence" or "explain why" gets a single-line box, and annotators type a paragraph into it. Two ways to get a real textarea: ```yaml - annotation_type: text name: why description: Why? multiline: true # -> rows: 4 ``` ```yaml textarea: {"on": true, rows: 4, cols: 60} # the older form; same result ``` Either way the input is named `<scheme>:::text_box`, so a selector that assumes `textarea[name=…]` breaks on the default. Use `[name='<scheme>:::text_box']`. ### Keyboard shortcuts `key_value` per label, on any type that has labels: ```yaml labels: - name: "Yes" key_value: y - name: "No" key_value: n ``` **The key is `key_value`.** `key_binding` looks right, validates, renders, and does nothing — an unrecognized key inside a label is not checked, not even by `--strict`. Confirm the shortcuts exist rather than assuming: `potato preview` prints a keybinding count per scheme, and `--format json` gives you the actual list. An empty list where you expected shortcuts means the key name is wrong. `sequential_key_binding: true` on the scheme numbers the labels 1, 2, 3… instead, which is often what you want and cannot be misspelled per label. Add shortcuts whenever there are nine or fewer choices. An annotator doing 500 items notices nothing else as much. `horizontal_key_bindings: true` lays the hints out in a row. ### Conflicts are real, and only `preview` reports them Two schemes both declaring `key_value: '1'`: ``` $ potato validate config.yaml --strict -> OK — no issues found. $ potato preview config.yaml -> KEYBINDING CONFLICTS: Key '1' used by both 'first:One' and 'second:Alpha' $ grep -i conflict server.log -> nothing ``` On the live page, pressing `1` sets the **first** scheme and leaves the second untouched. The second scheme's shortcut is silently dead. Nothing warns at startup — a claim to the contrary was in an earlier version of this file and is wrong. Two behaviours to keep apart: - **`sequential_key_binding` schemes are allocated distinct keys.** Three schemes all using it come out as `Q W E R T`, then `A S D F G H J`, then `N Y`. Preview lists these as conflicts on `1`, `2`, `3` because it reports the raw claim rather than the allocation. Those are not real. - **Explicit `key_value` is honoured as written and never reallocated.** Two schemes claiming the same character collide for real, and the loser is silent. Mixing explicit `key_value` with `sequential_key_binding` elsewhere is the usual way to get one. A scheme with its own tool keys — `image_annotation` reserves keys for the bbox and polygon tools — can have a label key and a tool key fire together on the same press. That generator does warn: ``` WARNING image_annotation: Keybinding conflict … key 'r' is bound to bbox tool and also to label 'Confusing order'. Both will fire. ``` So there are two conflict detectors with different coverage and neither is the startup log. **Press the key on a running page.** It is the only check that covers both. ### Conditional questions: `display_logic` ```yaml - annotation_type: radio # the gate name: has_anecdote description: Does this post tell a personal story? labels: ["Yes", "No"] humanize_labels: false - annotation_type: span name: anecdote_span description: Highlight the part that is the anecdote. labels: [Anecdote] display_logic: show_when: - schema: has_anecdote operator: equals value: "Yes" ``` Both schemes have to be in the same config. A `show_when` naming a scheme that is not there is a startup error, not a silent no-op -- which is the one good outcome in this area. `show_when` is a non-empty list; conditions are ANDed. Operators: `equals`, `not_equals`, `contains`, `not_contains`, `matches`, `empty`, `not_empty`, `gt`, `gte`, `lt`, `lte`, `in_range`, `not_in_range`, `length_gt`, `length_lt`, `length_in_range`. `empty` and `not_empty` take no value; the range operators take `[min, max]`. `schema` must name a real scheme; a reference to a scheme that does not exist, or a cycle, fails validation. **`not_equals` is true on a gate nobody has answered yet.** An unanswered scheme reads as undefined, and undefined is not equal to anything, so a follow-up gated with `not_equals` is on screen from the moment the page loads -- which is the opposite of what the block was added for. Gate on the positive values with `equals`, or AND a `not_empty` condition in front: ```yaml display_logic: show_when: - {schema: misleading, operator: not_empty} - {schema: misleading, operator: not_equals, value: "No"} ``` **A tile scheme can be gated on, but check it on the page.** `pairwise`, `bws`, `ranking` and `triage` answer by clicking a tile and writing to a hidden input, which the gate reads through a different path from the one radios and checkboxes use. Measured on Potato 2.8.2: a `{schema: <triage>, operator: equals, value: accept}` gate hides its dependent at load, shows it on *accept* and hides it again on *reject*. Before 2.8.2 it fired in neither direction. If the version is not yours to choose, drive the gate once in a browser before you design a page around it. Two things to watch either way. A gate on a scheme inside a **phase page** takes the DOM fallback rather than the live answer map, and that path requires the widget to have marked its input `data-modified` -- `triage` does not. And what a tile scheme stores is not what it shows, so write the condition against the stored value (see below). **A tile scheme's stored value is not its label.** `pairwise` records `A`, `B` and `tie`; `labels:` and `tie_label:` are what the annotator reads, not what lands in the data. Written as `value: "Summary A"` a condition could never have matched even if the gate worked, and the same names are what you filter on at analysis time. A required **span** scheme is enforced server-side only: the client cannot see whether a span was drawn, so `Next` posts, the server answers `400 Required annotation(s) not completed: <scheme>`, and the annotator gets a small corner toast naming the internal scheme name rather than the question. Requiring a span is fine; just make sure every item is one where drawing a span makes sense, quality-control items included. A hidden scheme may still be `required` — the server skips requiredness checks for schemes their own `display_logic` is hiding. Answers already given to a scheme that later becomes hidden are kept and saved, so a respondent who changes the gate answer leaves values behind on the hidden follow-up. Tell the researcher: it is a data-cleaning rule, not a bug. **A conditional scheme is invisible to a screenshot.** The render captures the page as it loads, so a scheme behind `display_logic` is not in the picture and its absence is not evidence of anything. Verify it another way: comment out the `display_logic` block, screenshot, put it back. Otherwise you are shipping the most important widget in the task unseen. This is a real failure mode: a task whose whole point is highlighting a span, verified with a screenshot in which the span widget does not appear. ## Arranging the questions: `layout` Without a `layout` block every scheme stacks full-width in config order, and a task with a dozen questions is a dozen screens of scrolling. Potato has a form grid with collapsible groups, and almost nothing points at it. ```yaml annotation_schemes: - annotation_type: radio name: sentiment description: "What sentiment does this carry?" labels: [Positive, Negative] - annotation_type: multiselect name: topic description: "Which topics apply?" labels: [Politics, Sport] - annotation_type: text name: notes description: "Anything else worth recording?" layout: grid: columns: 3 # 1-6; the base grid gap: "0.75rem" breakpoints: mobile: 480 # one column below this tablet: 768 # spans reduced below this groups: - id: primary title: Primary classification description: Required for every item schemas: [sentiment, topic] - id: details title: Additional details collapsible: true collapsed_default: false schemas: [notes] ``` Every name in `layout.groups.schemas` must be a scheme in the same config — that is one of the few nested blocks where a wrong name fails at boot rather than being ignored. Per scheme, `layout` says how much of that grid it takes: ```yaml - annotation_type: text name: notes description: Anything else? layout: columns: 3 # span the full 3-column grid order: 9 # explicit position min_width: "200px" max_width: "400px" align_self: start ``` `columns` is 1–6, `rows` is 1–4, both clamped rather than rejected. A working model is `examples/advanced/grid-layout/`; the full key list is in `docs/configuration/form_layout.md` and in `config-keys-nested.md`. **Validation is real here**, unlike most nested blocks: a `layout.grid.columns` outside 1–6, a non-string `layout.grid.gap`, a duplicate group id, an empty `layout.groups.schemas`, or a group naming a scheme that does not exist all fail at boot with a message naming the problem. Three things to know before using groups: - **`collapsed_default: true` is a decision to have the question skipped.** Use it only for questions most items genuinely do not need. - **A slider in a narrow column is unreadable.** The tick labels and the value bubble collide. Give `slider`, `range_slider`, `soft_label` and `semantic_differential` a `columns: 2` span or more, or leave them full width. - **A grouped form goes to one column at `tablet`, not at `mobile`.** The container's column count follows the `tablet` threshold, so `breakpoints: {tablet: 768}` — the default — means a grouped form is a single column in any window narrower than 768px, which is what a split screen usually is. Measured on a three-column, three-group form: at a 640px viewport with `tablet: 700` every group computed one track; the same page with `tablet: 300` computed `164px 164px 164px`. An ungrouped form holds its grid down to `mobile`. If your annotators work narrow, design for the column and treat the grid as a bonus. Per-scheme spans are reduced separately from the container, at widths that are not all yours to set, so read the computed span rather than the config when the window is not wide: ```js [...document.querySelectorAll('.annotation-form')] .map(f => f.dataset.schemaName + ' ' + f.dataset.gridColumns + ' -> ' + getComputedStyle(f).gridColumn) ``` A mistyped `layout` block validates, so confirm the attribute exists at all before debugging the width: ```js [...document.querySelectorAll('.annotation-form')] .map(f => f.dataset.schemaName + ' -> ' + (f.dataset.gridColumns || 'MISSING')) ``` ## Form density No config key sets this, and it is easy to let a task grow into something slow to annotate one reasonable question at a time. What to reach for, in order: 1. **One scheme per judgment**, always. `designing-a-task.md` covers why. 2. **`display_logic`** so follow-ups appear only when they apply. A question that is not shown costs nothing. 3. **A purpose-built type** where the questions are really one question repeated: ten ratings on the same scale is one `multirate` rather than ten `likert` schemes. 4. **`layout.grid`** to put short questions side by side. The five schemes in `examples/advanced/grid-layout/` come out in three rows rather than five. 5. **`layout.groups`** to give the rest headings, so the page reads as three sections rather than twelve questions. 6. **Split the task** if it is still long. Two passes over the corpus with five questions each cost the same annotator time as one pass with fifteen, and each pass has an annotator holding fewer definitions in their head at once. Question count is a poor proxy for how long an item takes; how far the annotator scrolls is a better one, and you can measure it. Screenshot `full_page=True` and read the page height against a 900px viewport. Anything past two screens is worth restructuring. ## Page-level appearance | Key | Effect | |---|---| | `annotation_instructions` | Collapsible instructions banner; text or a path | | `header_file` | **Inert.** The header template path is no longer configurable and nothing reads the key; `validate` reports it | | `custom_footer_html` | HTML on every page | | `header_logo` | Logo in the navbar | | `base_css` | An extra stylesheet | | `task_layout` | Custom HTML for the form area; the escape hatch | | `layout` | The form grid and groups — see *Arranging the questions* above | | `hide_navbar` | Hide the top bar | | `jumping_to_id_disabled` | Remove the jump-to-item control | The navbar's two arrows are **previous and next _unannotated_**, not previous and next. On a stretch the annotator has already answered they do nothing and say nothing, which reads as a dead button; the jump-to-item box beside them is how you get back to an item that is already done. `jumping_to_id_disabled` removes that box and with it the only way back. | `horizontal_key_bindings` | Lay shortcut hints out horizontally | | `highlight_linebreaks` | Make line breaks visible in item text | | `keyword_highlights_file` | Highlight given keywords in the item | | `ui`, `ui_config` | Interface toggles | | `ui_language` | Interface language code | Two of those carry a contract the key name does not: **`keyword_highlights_file` reads 6 shapes.** The documented one is a CSV or TSV with a `keyword` header column; `label`, `schema` and `color` are optional beside it, matched by name in any order, and `keyword`/`word`/`pattern`/`term` are accepted spellings. ``` keyword,label,color latch,Defect,#ffcc00 Ridge*,Place, ``` One keyword per line also works, as do JSON, JSONL and YAML holding a list of strings, a list of objects, or a `{keyword: label}` map. The sixth shape is a headerless delimited file, read positionally as keyword, label, schema; it warns and tells you to add a header. A `*` matches any run of word characters. The boot log names the shape it read and the count — `Loaded 3 keyword highlight patterns from … (read as delimited with a header (keyword, label, color))` — so read that line rather than the page. A header the loader does not recognise is not an error: the file is read positionally and the header row itself becomes a keyword. **Keyword highlighting covers `text_key` plus every field with `span_target: true`**, and nothing else — a field with no span target has nowhere to draw. Each match carries its own `target_field`, and `/api/keyword_highlights/<instance_id>` reports `fields_scanned`, which is the way to check that the field you care about was one of them. **`list_as_text` takes three sub-keys**: `text_list_prefix_type` (`alphabet`, `number`, `bullet` or `none`), `horizontal`, and `alternating_shading`. A bare `true` turns it on with the defaults. Reach for `task_layout` last. Hand-written layout HTML stops tracking changes to the schema generators, and most of what people want it for is `instance_display.layout` plus scheme order. ## Checking it ```bash potato preview config.yaml # what the config declares potato preview config.yaml --screenshot out.png # what it looks like ``` The screenshot boots the task in a real browser and reports every uncaught exception, `console.error` and failed request. Most annotation UI is built by JavaScript after the HTML arrives, so validation cannot see any of it. **It captures the viewport, not the page.** The PNG is 1280x900 — on a task with an instructions banner that is the banner and part of the item, and none of the questions. To see the whole page, drive a browser against a running server and screenshot with `full_page=True`: `running-a-task.md` has the driver. Some of what it reports is not yours. Every phase page of a healthy task logs `GET /api/current_instance 404` and two `[SpanManager]` errors, because a phase page has no instance. Filter those before reacting. **Look at the PNG.** A clean exit means nothing threw, not that the interface is usable. Check: every question present, nothing overlapping, labels readable, the item rendered rather than showing a raw path or a broken image, choices not running off the edge. What the screenshot cannot show you, and you have to check another way: - anything behind `display_logic` (see above) - anything that appears after an interaction: a span being drawn, a canvas tool, a media player mid-playback - phase pages: pass `--phase consent` (also `instructions`, `training`, `poststudy`) - how it behaves after navigating away and back, which is where saved state goes wrong ## Common UI failures | Symptom | Cause | |---|---| | The item shows a file path instead of an image | No `instance_display`, so the path rendered as text | | A span scheme with nothing to highlight | No field has `span_target: true` | | `span_target is set but type X does not support span annotation` | Wrong display type; the error lists the twelve that work | | A scheme is missing from the page | It is behind `display_logic` and the condition is not met | | Choices run off the right edge | Many labels in a horizontal `layout`; use vertical | | A keyboard shortcut does nothing | The key is `key_value`, not `key_binding`. Check with `potato preview` — an empty keybinding list means the label key is misspelled | | A shortcut works but the wrong label lights up | Two labels claimed the same key; the conflict warned at startup | | Labels read "Information Or Advice" | `humanize_labels` is on by default; set it `false` | | A feature you configured does nothing | Typo'd key. Unknown keys only warn — run `potato validate --strict` | <!-- https://davidjurgens.github.io/potato-skill/evaluating-the-ui/ --> # Evaluating and Iterating on the Interface How to judge a rendered annotation page and what to change. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- A config that validates tells you the server will start. It tells you nothing about whether a person can do the work. That question only has one answer: render the page, look at it, and change something. ## The loop ```bash potato preview config.yaml --screenshot shot-01.png ``` **Before you start: the PNG is 1280x900 of the top of one page, not the whole page.** Half the checklist below is about things further down. Get a full-height render by driving a browser against a running server (`running-a-task.md`), and treat `--screenshot` as the fast smoke test rather than the visual check. Then, in order: 1. **Read the error lines first.** Uncaught exceptions and `console.error` mean part of the interface never initialized. Nothing else you notice matters until those are gone — except the three Potato logs on every phase page (`/api/current_instance` 404 and two `[SpanManager]` lines), which are present on a healthy task. 2. **Look at the PNG.** Not the exit code. A clean exit means nothing threw. 3. **Make one change.** 4. **Re-render to a new filename** — `shot-02.png`, not the same name. You need the pair to see what your change did. Keep going until the checklist below passes. Three or four rounds is normal for anything with a custom layout. ## What to look for Work down the list. Each line is a failure that has shipped in a real task. **The item** - Is it there at all? A raw file path where an image should be means no `instance_display`. - Is it readable — not clipped, not scrolled off, not one line of a long document with no way to see the rest? - If it is media: does the player have controls, and are they inside the frame? **The questions** - Is every scheme present? Count them against the config. A missing one is usually behind `display_logic` (see below) or failed to generate. - Is any of them empty — a heading with no inputs under it? - Are the labels readable, or truncated with an ellipsis? - Do radio buttons and checkboxes actually appear? An invisible input control is the classic CSS collision, and the label text still renders, so the question looks fine at a glance. - Do tick marks, scale points and their labels line up, or do the labels overlap each other at the ends? - For a grid or matrix scheme: does it use the full width, or is it squeezed into a narrow column with horizontal scroll? **The page** - Is anything shifted right, overlapping, or overflowing its container? - Is the Next button visible without scrolling past the fold? An annotator hits it several hundred times. - Does the whole page scroll horizontally? It should not. **The work itself** - Count the interactions to complete one item: clicks, scrolls, keystrokes. Multiply by the number of items. That number is the study's cost. - Do the schemes have keyboard shortcuts where they could? - Is anything ordered so that the annotator has to scroll up and back down? ## The states a screenshot will not show you The render captures the page as it loads. Four things live outside that, and each has to be checked another way. **Conditional schemes.** Anything behind `display_logic` is absent from the initial render, and its absence proves nothing. Comment out the `display_logic` block, render, look, put it back. Do this every time — the widget behind the condition is often the whole point of the task. **Anything that appears after an interaction.** A span highlight, a drawn box, a playing video, an expanded section, minting a code, saving a note. Render to confirm the control is there, then drive it if you have a browser tool. Without one there is still more evidence than the PNG. Run the server once in the background and read what it did: - the startup log says which subsystems initialized and what they loaded — codebook labels, detected cases, how many instances got indexed - `project.sqlite` in the task directory holds the codebook, cases, memos and the search index, and can be queried directly - `annotation_output/<user>/user_state.json` shows what a scripted pass actually stored That distinguishes "the feature is configured and initialized" from "the button is on the page", which is most of the distance. Say in your handover which of the two you established. **Other phases.** `--phase consent`, `--phase instructions`, `--phase training`, `--phase poststudy`. Only those four plus `annotation`: the preview CLI refuses `prestudy` even though the server supports it as a phase. **A phase-page render exits 1 on a healthy task.** Phase pages have no instance, so the span layer's `/api/current_instance` 404 and two `[SpanManager]` console errors are always reported, and `--screenshot` counts them. Judge the PNG and the error list, not `$?`. See `phases-and-pages.md`. Each is a page an annotator sees, and the training phase renders real schemes, so it can break in ways the annotation page does not. Only render a phase the task actually has. `--phase X` parks an annotator in that phase, so if the task never configures one, every route redirects toward a page that does not exist and the render fails with a redirect loop. `annotation_instructions` is the banner on the annotation page, not an instructions phase — a task can have the banner and no phase. **State across navigation.** Answers are saved and restored on a full page reload. If you have a browser tool, annotate, click Next, click Previous, and check the answer is still shown — visually, not just in a hidden input. Do not test this with a page refresh: browsers restore form state across refresh on their own, so a refresh-based check passes even when the server never stored anything. ### Driving more than one annotator Three things about the browser rather than about Potato, each of which looks like a Potato bug the first time. **Log out between annotators.** `/register` and `/auth` are no-ops while a session exists — the server logs `User already logged in with username: X, redirecting to annotate` and answers 200, so the second annotator's work lands under the first one's name and the second user never appears in `annotation_output/`. Hit `/logout` first, every time. **Two servers on the same host share one session.** Cookies are not scoped by port, so logging into the study on 8001 logs you out of the one on 8000. Reach one of them on 127.0.0.1 and the other by name, and they stay separate. **CSS transitions do not run in a background tab.** A sidebar or a modal that slides in will report the class that opens it applied and its transform still at the closed value, which reads exactly like a broken stylesheet. Settle it before measuring: ```js el.getAnimations().forEach(a => a.finish()) ``` ## Changing the interface Reach for these in order. Stop at the first one that fixes it. | Want | Use | |---|---| | Fields side by side rather than stacked | `instance_display.layout.direction: horizontal` | | More or less space between fields | `instance_display.layout.gap` | | Choices in a row rather than a column | `layout` on the scheme | | Shortcut hints along one line | `horizontal_key_bindings: true` | | Standing instructions at the top | `annotation_instructions` | | Something above every item | `header_file` | | A logo | `header_logo` | | Colours, spacing, fonts | `base_css` | | More room — no navbar, no jump control | `hide_navbar`, `jumping_to_id_disabled` | | An arrangement none of the above reaches | `task_layout` | `task_layout` is last for a reason: hand-written form HTML stops tracking changes to the schema generators, so a scheme that gains a feature later will not gain it here. Most requests that sound like they need it are `instance_display.layout` plus scheme order. With `base_css`, change what you meant to change. A rule that targets a bare element selector will reach the navbar and the admin pages too. ## Iterating without going in circles **One change per render.** Two changes and a difference you cannot attribute is how a session turns into twenty screenshots and no conclusions. **Keep the screenshots numbered.** They are the record of what you tried. **Write down what you were fixing.** "shot-03: moved confidence above the span so the page stops reflowing" beats three unlabelled PNGs. **Know when to stop.** The interface is done when: no console errors, every scheme visible and usable, nothing overlapping or clipped, the Next button reachable without scrolling, and shortcuts on anything that can take them. Past that you are decorating, and the researcher has opinions you do not have access to. Hand it over and say what you would change next. **Running the task writes to the project directory.** The first server start creates `project.sqlite` and seeds it — for a codebook task, the labels in your config *become* the project codebook. Change the labels afterwards and the codebook still holds the old ones until `potato codebook config.yaml` re-syncs. Worth knowing before you verify a config you are still editing. **When a change does nothing**, suspect the key before the CSS. Unrecognized config keys only warn, so a typo is silently ignored and the feature you thought you enabled is off. Run `potato validate --strict`. ## Reporting what you saw A handover that says "renders correctly" is not evidence. Say what you checked: > Rendered at 1280×900. Three schemes present, one behind `display_logic` > (verified separately by disabling the condition — screenshot `shot-04.png`). > No console errors. Keyboard shortcuts on the yes/no question. The consent and > instructions phases render (`shot-05`, `shot-06`). > > Not checked: span highlighting behaviour after a drag, and whether answers > survive navigating away and back — neither is visible in a static render. The second paragraph is the part that makes the first one worth reading. <!-- https://davidjurgens.github.io/potato-skill/modalities/ --> # Images, Video, Audio and 3D Routing a medium to its display type, its scheme, and its quirks. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Everything here is a display type plus a scheme type plus one or two quirks that are silent when you get them wrong. The generated reference (`annotation-types.md`) has the field lists and a worked example for every type; this file is the routing and the traps. Agent traces and agent evaluation are their own family — `agent-traces.md`. ## Routing from what they brought | They have | Display field | Scheme | Watch | |---|---|---|---| | Images, one per item | `image` | `image_annotation` | `tools` **and** `labels` are required; the image renders twice | | Very large images, microscopy, maps | `image` + `viewer`/`tiles` on the scheme | `image_annotation` | Deep zoom drives the transform; never compute it yourself | | Several images per item | `gallery` | a classification scheme, or `image_annotation` per field | `gallery` cannot be a `span_target` | | Depth or range data | `depth_map` | classification, or `spatial_annotation` | Windowing and colormap are display options, not data | | A region and a phrase for it | `image` | `image_annotation` **+** `region_caption` | Without the canvas scheme there is nothing to describe; agreement is over *matched* regions | | "Did the model point at the right thing" | `image` | `image_annotation` **+** `grounding_eval` | Same pairing; `region_type: point` scores differently from boxes | | Video, judge the whole clip | `video` | any classification scheme | Nothing special | | Several videos per item — camera views of one action, model rollouts side by side | one `video` field per clip | any classification scheme, or `rubric_eval` | **`gallery` only from Potato 2.9.0**, and only when each url ends in `.mp4`, `.m4v`, `.webm`, `.ogv` or `.mov`. Any other url, and every url before 2.9.0, goes into an `<img>` and renders as a broken image with no warning. Rendered on 2.9.0: three `.mp4` urls gave three `<video>`, an extensionless one an `<img>` | | Video, mark moments or objects | *(none needed)* | `video_annotation` | Five modes, and `labels` is required in every one | | "Find the interval this sentence describes" | *(none needed)* | `temporal_grounding` | Reads `video_key`/`events_key`, **not** `source_field` | | Audio, judge the whole clip | `audio` | any classification scheme | Nothing special | | Audio, mark regions | *(none needed)* | `audio_annotation` | Use `mode: label`. The two modes that ask questions per region do not render | | ASR/TTS output against a reference | *(none needed)* | `speech_transcript` | Segments come from **`segments_key`**. `turns_key`, `speaker_key` and `text_key` are in the key reference and the generator reads none of them — name your field with `turns` and the widget renders zero cards, silently | | ELAN-style tiers over audio or video | *(none needed)* | `tiered_annotation` | `media_type` defaults to `audio` — set it for video | | A podcast or interview, turn by turn | `audio_dialogue` | spans, ratings, links | **One recording plus per-turn `start`/`end`.** The play buttons seek into that one file; they do not each load their own | | A voice agent, one recording per utterance | **concatenate to one recording first** | `speech_transcript`, `voice_interaction` | No display plays a file per turn. `audio_dialogue` and `speech_transcript` both take one source — the item's `audio_key`, else the first turn's url — and the per-turn controls carry timestamps into it. Join the utterances, write global `start`/`end`, and both work properly | | A chat log or conversation | `dialogue` | anything, often `turn_level` | Span target; threading is a display option | | A branching conversation | `conversation_tree` | `tree_annotation` | **Not** a span target | | Several agents talking | `multi_agent_discussion` | `failure_attribution`, ratings | Span target; see `agent-traces.md` | | Point clouds, LiDAR, 3D scans | the cloud path, **unlabelled** | `spatial_annotation` | It reads the path off the page, not the item; `lod` defaults on, which changes what `max_points` does | | Robot episodes, teleop logs | *(none needed)* | `episode_annotation` | Four layers; pick the ones you need | ## Where the widget looks for its data **This is the mistake to design against.** There are at least six conventions, and picking the wrong one gives you an empty widget with nothing in the log. | Key | Types that read it | |---|---| | `source_field` | `image_annotation`, `video_annotation`, `audio_annotation`, `spatial_annotation`, `episode_annotation`, `tiered_annotation`, `text_edit` | | `video_key`, `events_key` | `temporal_grounding` | | `audio_key`, `segments_key`, `turns_key`, `speaker_key`, `text_key` | `speech_transcript`, `voice_interaction`, `tiered_annotation` | | `screenshot_key`, `steps_key` | `gui_trajectory` | | `steps_key`, `agent_key` | the multi-agent and trajectory families | | `image_key`, `rows_key`, `cols_key` | `table_grid` | | `episode_field` | `episode_annotation` (alongside `source_field`) | | `items_key` | `pairwise` — the data field holding the list of things to compare | | `caption_field`, `expressions_field`, `predictions_field` | `grounding_eval` | | `video_path` | the `video` *scheme*, which is not the `video` display type | | `target_field` | `span` | | `source_field` | `error_span` — name the field holding the text to mark, or it falls back to `text_key` | | `question_field`, `passage_field` | `extractive_qa` | | `items_field` | `card_sort` | | `profiles_field` | `conjoint` | | `span_schema` | `coreference`, `event_annotation`, `span_link` | Check before you write it: ```python from potato.server_utils.schemas.registry import schema_registry s = schema_registry.get("temporal_grounding") sorted(set(s.required_fields) | set(s.optional_fields)) ``` **The key alone is enough to get the media to the widget**, as of Potato 0fec4013. I ran one study with three schemes and no `instance_display` block, so the page carried zero `[data-field-key]` elements, and measured what each widget actually got: ``` audio_annotation source_field: clip <audio> src=/media/mono44.wav decodes: 5.000s, mono, 48 kHz video_annotation source_field: vid <video> src=/media/h264.mp4 image_annotation source_field: image_url plain.png fetched 200, painted onto canvas-boxes ``` Older checkouts need the display block for the media to arrive at all, so if you are pinned to one, declare it. Declare it anyway. Not for the URL, but because the `instance_display` path is the one that preserves your text as written (the default path collapses runs of spaces), gives the field a label, and decides the order things appear in: ```yaml instance_display: fields: - key: clip # the same name source_field points at type: audio ``` Two names collide. **`video` is both an annotation type and a display type**, with different required fields — the scheme wants `video_path`, the display field wants `key`. It is also the one annotation type with no example config anywhere, so there is nothing to copy. For anything beyond "play this clip", you want `video_annotation`. ## Serving the media Two static roots, and neither of them is `data_files`. ```yaml media_directory: media # default; served at /media/<path> ``` - **`/media/...`** ← `media_directory` under the task directory. Reference it from your data as `/media/clip_01.mp4`. Path traversal is blocked. - **`/screenshots/...`** ← a `screenshots/` directory under `task_dir`, separate and not configurable. Agent-trace data setting `screenshot_url` to `screenshots/step_000.png` is served from here. Miss it and every step image 404s. Remote URLs work, with the limits below, and suit a corpus you do not want in the bundle: `potato deploy` ships the whole task directory. On an `image_annotation` scheme, three tools need CORS headers from the image host and the rest do not. From a host that sends none (a local server with no headers, or `images.cocodataset.org`), boxes, polygons, ellipses, points, freeform lines and brush strokes all draw, save, and are redrawn when the annotator comes back. Region fill, the SAM magic wand and the text prompt have to read the image's pixels, and the browser refuses that for such a host. Each one says so and does nothing. Fill shows "Fill cannot read this image: its host does not allow it (no CORS headers)" and saves no mask; the other two show "The browser will not let the model read this image: its host sends no CORS headers." A task that needs any of those three should copy the images into `media_directory` and reference them as `/media/...`, or use a host that sends `Access-Control-Allow-Origin`. Before 2.11 the drawing area did not load at all from such a host, and read "Failed to load image. Check the URL or CORS settings." A `data:` URI in the field works as well, and is worth knowing about for a handful of small images you would rather not ship as files. A path that leaves `media_directory` fails in one of two ways, and only one of them tells you. `../outside/far.png` is refused by the traversal check and logged — `image path traversal blocked: ../outside/far.png` — so the log answers the question. An absolute filesystem path is not traversal, so it is handed to the browser as though it were a URL and 404s, with nothing written to the log. Both paint the same broken-image icon on the page, which is the only signal the annotator gets. Check the log before concluding the file is missing. ## Images and CV Both halves or nothing: ```yaml instance_display: fields: - {key: image, type: image, label: Screenshot} annotation_schemes: - annotation_type: image_annotation name: regions description: Draw a box around each problem. source_field: image tools: [bbox] # required labels: [Broken, Confusing] # required ``` `source_field` names the data key, but the canvas takes its bitmap from the `<img>` that `instance_display` renders. No display field, no `<img>`, empty canvas, and no error. The consequence is that the image renders twice, which is correct; `building-the-ui.md` has the CSS to hide the display copy. **`region_caption` and `grounding_eval` are not drawing schemes.** Each owns what hangs off a region — the description, or the phrase-to-region binding — and needs an `image_annotation` scheme in the same config to draw on. On their own they validate, boot clean, and render a widget nobody can use: `region_caption` shows "Draw a region on the image, then describe it" over a list that stays empty for the life of the study, and `grounding_eval` lists the phrases with only *Not present in the image* available, so every answer it can record is `{"regions":{},"absent":[...]}`. Put the canvas scheme first: ```yaml - annotation_type: image_annotation name: region description: Draw a region around each thing you describe. source_field: image tools: [bbox, polygon] labels: [{name: referent, color: '#6e56cf'}] - annotation_type: region_caption name: captions description: Describe each region you drew. ``` `min_length` on `region_caption` is dead — it reaches the browser and nothing reads it, so a three-character caption counts as described. `require_all` warns once and lets the second Next through. If a short caption is not acceptable, say so in the instructions and check it in the export. **The canvas is bigger than the picture.** A 640x420 image sits centred in an 831x600 canvas, so there is a margin around it that still takes a drag. A box drawn wholly in the margin is discarded silently. A box that *starts* in the margin is stored with coordinates outside `[0, 1]`, `{"x": -0.046, "y": -0.0007, ...}`, and shown as "at -5%, 0%". Clamp on the way out, or drop boxes with a negative corner, before anything computes IoU against them. **Which is also why a practice round cannot show the image.** Phase pages do not render `instance_display`, so a geometry scheme on a `training` phase has no `<img>` and paints "Failed to load image" instead. `quality-control.md` has the detail and the two ways round it. Decide it before writing the training file, because the config validates and the server boots clean either way. **Deep zoom.** `viewer` and `tiles` switch to a tiled viewer for images too big to send whole. The viewer owns the transform and the canvas draws in image pixels; anything that recomputes the mapping itself will be wrong at every zoom level but the first. **Pointing is not grounding with a small box.** `grounding_eval` takes `region_type` ∈ `box`, `polygon`, `mask`, `point`. The default is `box`. The scoring is not interchangeable, and the module says why: "A point has no area. Every IoU against it is 0, so scoring points the way boxes are scored reports total failure for a model that is pointing perfectly." Points are scored as a hit rate over regions. If the researcher is evaluating a pointing model, set `region_type: point` and report the hit rate; do not quote it beside an IoU as though they were the same number. The expressions come from the `expressions` field of the item, and `expressions_field` renames it. Point it at the wrong key and the widget says so on the page — "This item has no referring expressions. The schema reads them from the 'expressions' field; set `expressions_field` if they live somewhere else" — which is the only reason that mistake costs a minute rather than an afternoon. **`region_type` and `tools` are different vocabularies, and the UI labels do not match either.** `region_type: point` is right, but the drawing tool that places a point is `landmark`, and the button it renders says "Point" — so `tools: [point]` is the natural guess and is refused at boot, with the twelve real tool names listed. A mask comes from `brush`, `fill` or `sam` rather than from a tool called `mask`. **A painted mask keeps the image's own resolution, not the canvas's.** The buffer behind a `brush` stroke is tiled at the source image's native size, so zooming, fitting or resizing the drawing surface does not move a stored mask. I painted one stroke on a 640x400 image and got the same bounding box, to the pixel, from the painted canvas, the in-memory buffer, the persisted annotation, `mask_png` and `coco`. `grounding_eval` also distinguishes three states per expression — answered with a region, answered as *absent*, and not answered — because "no region" otherwise conflates "there is no referent" with "I did not get to this one", and those support opposite conclusions about a model that also produced nothing. ## Video ```yaml - annotation_type: video_annotation name: actions description: Mark each action and label it. source_field: clip mode: segment # segment | frame | keyframe | tracking | combined labels: [Reach, Grasp, Release] video_fps: 30 ``` `mode` defaults to `segment`. **`labels` is required in all five modes** — the generator raises without it. (It was declared optional in the registry until this pack was written, so `--strict` passed a config that could not render; it is required now.) `mode: combined` takes the same `segment_schemes` list as the audio widget and renders it the same way. It was unimplemented before Potato 2.8.2 -- the key reached the browser and nothing read it -- so check the version before planning a task around per-segment questions on video. "For each action, rate the quality" has to be a separate scheme over the whole clip, or a second pass. `video_fps` is what frame numbers are computed against. Wrong value, wrong frame indices in the export, and nothing complains. **`min_segments` is decoration.** It validates as an integer, reaches the browser as `minSegments`, and is read by nobody: a scheme with `min_segments: 2` and `required: true` is satisfied by one segment, and Next goes through. On the video and audio widgets `required` means "the annotator made one mark", not "the annotator finished". If a study needs two segments per clip, say so in the instructions and check it in the export — `segments` is a list, so the count is one `len()` away. **`temporal_grounding` is a different task and a different config shape.** It takes `video_key` and `events_key` off the item and scores the annotator's interval against a predicted one with live IoU. Writing `source_field` there gets you a player with no video. `required: true` on it means one keystroke in one box. An item declaring two events, with a start typed into the first and nothing else, saves `{"events":{"0":{"start":1.5}}}` and advances: an interval with no end, one event of two, and the study calls the item done. Check for `end` on every event before you trust the IoU column. ## Audio and speech `audio_annotation` renders a Peaks.js waveform. Its `mode` decides what else is required, and the requirement is conditional so `--strict` cannot see it: | `mode` | Also required | For | |---|---|---| | `label` (default) | `labels` | Mark regions and label them | | `questions` | `segment_schemes` | Mark regions and answer questions about each | | `both` | `labels` **and** `segment_schemes` | Both | **`questions` and `both` need `segment_schemes`**, a list of whole schemes asked once per region. The validator requires the key for those two modes and checks each sub-scheme against its own type's rules, so a `likert` in there still needs `min_label` and `max_label`. ```yaml - annotation_type: audio_annotation name: interruptions description: Mark each overlap, then answer for it. source_field: clip mode: questions labels: - {name: Clinician cuts in, key_value: '1'} - {name: Patient cuts in, key_value: '2'} segment_schemes: - annotation_type: radio name: who_started description: Who started talking first at this overlap? labels: [Clinician, Patient] ``` The server renders each sub-scheme once into a hidden `<template>` and the client clones it per region, so any annotation type works inside a region. The clones carry `data-segment-schema` instead of `schema`, and their ids are suffixed `__seg_<region id>` -- which is what keeps a region's answer from being read as a top-level answer for a scheme that does not exist. Regions land in `instance_id_to_label_to_value` under `<scheme>:::_data`, as a JSON string of `{id, start_time, end_time, label, annotations}` per region, with the per-region answers keyed by sub-scheme name inside `annotations`. Verified on Potato 2.8.2 (`v2.8.2-9-g19ce0041`); on earlier builds the panel said "Segment annotation questions will appear here" and every region saved an empty `annotations`. If you inherit an older checkout, put the per-region distinction in `labels` and ask everything else once for the whole clip. **Zoom in and zoom out do nothing on either waveform widget.** Both `audio_annotation` and `video_annotation` call `view.getZoom()`, which the bundled Peaks build does not have, so the button raises an uncaught `view.getZoom is not a function` and the view does not move. *Fit* works, since it is the one of the three that never asks for the current zoom. Plan segment work at whatever resolution the whole clip gives you, or keep the clips short. **The gesture is right-click drag, not drag.** A left-drag on the waveform moves the playhead, so a checker that drags the way it would on a canvas creates nothing and reports no error. `[` and `]` then Enter is the keyboard route. The widget's own Help panel says both; nothing else does. `waveform` defaults on, `spectrogram` defaults off; turning the spectrogram on is the right call for anything phonetic and costs render time on long files. `speech_transcript` scores ASR or TTS against an aligned reference, with per-segment error tags plus a correction. It reads `audio_key`, `segments_key`, `turns_key`, `speaker_key` and `text_key`. `tiered_annotation` is the ELAN/Praat shape: named tiers stacked over one timeline. It requires `source_field` and `tiers`, and **`media_type` defaults to `audio`** — a video task that forgets to set it gets an audio element and no picture. Three things about it on 2.8.2-11: - Its Peaks.js wiring throws on every load (`zoomview.on is not a function`, `tiered-annotation.js:1073`) and the failure is swallowed as a console warning. Everything after that line in the initialiser is skipped, so double-click on the waveform does not seek, clicking the overview does not navigate, the initial zoom and auto-scroll are never set, and nothing refits on resize. Segment click and drag still work, which is why it looks fine. - **`transcript_field` labels every seeded turn with the tier's first label.** It carries the turn's real `speaker` in a field nothing displays, so a two-speaker call arrives as four utterances all labelled "Caller", two of which are the agent. The contradiction (`"label":"Caller", "speaker":"agent"`) is what persists once the annotator edits anything. - **There is no gesture to relabel an existing annotation.** The label buttons set the label for the *next* one drawn; the keyboard offers play, step, Delete and Escape. Correcting a seeded label means deleting it and drawing it again. If the labels matter more than the boundaries, seed nothing and let annotators draw. ## Dialogue and podcasts Four displays, and the difference is the shape of the conversation: | Display | Shape | Span target | |---|---|---| | `dialogue` | A flat or reply-threaded log | yes | | `audio_dialogue` | Interview or podcast turns, each with a play button that seeks into the one recording | yes | | `conversation_tree` | Branching, collapsible | no | | `multi_agent_discussion` | Several agents, colour-coded, filterable | yes | For per-turn questions rather than one judgment for the whole thing, use `turn_level: true` on the scheme; for one judgment over a whole session, use `sessions` and `session_level: true`. Both are in `modes-and-subsystems.md`. **`list_displays()` under-reports span-target support.** Its `supports_span_target` flag is true for nine types, but twelve accept `span_target` — `pdf`, `spreadsheet` and `agent_trace` anchor spans their own way and are missing from the flag. Ask `display_registry.get_span_target_capable_types()`, which is what the validator uses. (The same table under-reported `optional_fields` until 2.8.2-11; it is accurate now, and `test_doc_counts.py` fails if it drifts again.) ## 3D and embodied ```yaml - annotation_type: spatial_annotation name: objects description: Box each object in the scan. source_field: cloud tools: [cuboid] labels: [Vehicle, Pedestrian] ``` `tools` and `labels` are required, as with images. **The viewer finds the cloud on the page, not in the item.** `source_field` names the key, and then `pc-viewer.js` looks for a rendered element whose `data-field-key` matches and reads its *text*. So the cloud path has to be either the item's `text_key` (what the bundled example does) or a display field — and that field must have **no `label:`**, because the label ends up in the same text node and the path is rejected for containing whitespace. All three of these produce "No point cloud for this item: the "point_cloud" field is empty. Check item_properties in the config", which is not where the problem is: ```yaml # dead: the cloud is in the data and nowhere on the page instance_display: {fields: [{key: notes, type: text, label: Notes}]} # dead: the label is in the way instance_display: {fields: [{key: cloud, type: text, label: Cloud}]} # works: 9,091 points, all loaded instance_display: {fields: [{key: cloud, type: text}]} ``` Boxes come back in **metres in the sensor frame**, not normalized like image annotation: `{"center": [35.47, -7.58, -0.90], "size": [15.71, 1.46, 1.70], "rotation": [0, 0, 0, 1]}`. **`lod` defaults to `True`**, and that changes the meaning of the neighbouring keys: under LOD the cloud loads as an octree and `point_budget` / `max_loaded_nodes` govern what is in memory, while **`max_points` is used only when `lod: false`**. Setting `max_points` on a default config and expecting a cap does nothing at all. `mpr` and `slab_thickness` switch to slab views for medical-style data. `rollout_evaluation` puts several generated videos on one clock, with violations, preference and counterfactual layers. It is the one media widget that already reports its own shortfall ("0 of 3 panels answered, still to do: A, B, C"), so `required` on it means what it says. Its status line claims "3 rollouts, 0.00 s" whatever the clips are, because the duration is only learned from the browser after that sentence is written; the clock and frame counter under the panels are right. Use WebM: the widget says so itself when handed MP4. `episode_annotation` is the embodied one: synchronized video streams plus time-series lanes. Its four `layers` (`phases`, `outcome`, `reward`, `instruction`) are all on by default, and each answers a different question, so name the ones you want rather than asking annotators for all four on every episode. `series_shown` picks which sensor lanes to draw; omit it and you get all of them, which on a real robot log is unreadable. ## Verifying a non-text task `potato preview --screenshot` is close to useless here. It renders the viewport of one page, and a canvas that has not loaded its bitmap, a waveform that has not decoded, and a video that has not buffered all look the same as ones that have. What actually proves it: 1. **Boot and read the log.** A geometry or media generator that failed renders a heading with no inputs, and the startup log names the scheme. 2. **Open it and interact.** Draw one box, drag one segment, play one turn. The media pipeline is what breaks, and only a real gesture exercises it. 3. **Read `user_state.json`.** Geometry lands in `instance_id_to_span_to_value` and the scheme's `_data` inputs. If the shape there is not what you expected, the export will not be either. 4. **Use a format the test browser can play.** The browser tests in this repo use WebM fixtures for video; a clip that plays in your desktop player is not evidence that headless Chromium will decode it. `running-a-task.md` has the driver. `scripts/walk_task.py` will answer a media task's classification questions but cannot draw a box for you. <!-- https://davidjurgens.github.io/potato-skill/agent-traces/ --> # Agent Traces and Agent Evaluation The trace displays, the twenty evaluation schemes, and how they bind. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- The largest family in Potato and the one least like text annotation: about twenty schema types, eight trace displays, and sixty example projects (57 under `examples/agent-traces/`, 3 under `examples/agent-testing/`). Its binding model fails silently when you get it wrong. ## How a trace scheme finds its steps Almost every scheme in this family reads a **list off the item** and renders one card per entry. The key it reads is configurable and defaulted: | Key | Default | Read by | |---|---|---| | `steps_key` | `steps` | `gui_trajectory`, `tool_call_review`, `process_reward`, `failure_attribution`, `agent_scorecard`, `trajectory_eval`, `trajectory_edit`, `multimodal_reasoning`, `emergent_behavior`, `handoff_review`, `agent_interaction_graph` | | `turns_key` | varies | `consensus_tracking`, `context_attribution` | | `agent_key` | varies | the multi-agent schemes, for who is speaking | | `step_text_key` | varies | which field of a step holds the text to show | **Almost nothing in this family has a required field.** `rubric_eval` needs `criteria`; the rest need only `name` and `description`. So a scheme pointed at a key your data does not have renders an empty card list, `validate --strict` passes, and the boot log says nothing. If a trace widget comes up blank, check the key name before anything else: ```bash python -c "import json;d=json.load(open('data/traces.json'));print(list(d[0]))" ``` Answers are stored as a hidden-input JSON list keyed by step `index`, and only judged steps are saved — the list is **not** positional, so do not read it as one entry per step. ## Routing from what they said | They said | Reach for | |---|---| | "was each step right" | `trajectory_eval` (taxonomy + severity) or `tool_call_review` (right tool / args / ordering) | | "where did it first go wrong" | `process_reward` in `first_error` mode | | "label every step for PRM training" | `process_reward` in `per_step` mode | | "fix the bad steps, don't just score them" | `trajectory_edit` — produces SFT/DPO data | | "which agent caused the failure" | `failure_attribution` | | "rate each agent and the team" | `agent_scorecard` | | "did the handoff lose something" | `handoff_review` | | "are they colluding / groupthinking / drifting" | `emergent_behavior` | | "who used the shared resource when" | `tool_contention` | | "how does the conversation use earlier context" | `context_attribution` | | "did they actually reach consensus" | `consensus_tracking` | | "draw me the interaction graph" | `agent_interaction_graph` | | "did the click land on the right button" | `gui_trajectory` | | "score it against a rubric" | `rubric_eval` (needs `criteria`) | | "keep or drop this trace" | `triage` | | "is this generated video physically plausible" | `rollout_evaluation` | | "does our judge agree with our humans" | `judge_alignment`, `judge_calibration` | | "compare two models' answers" | `arena`, or a `pairwise` / `bws` scheme | | "traces arrive from production" | `trace_ingestion` (`/api/traces/*`) + `automation` + `triage` | | "judge it while it runs" | `live_agent`, `live_coding_agent`, `agent_proxy` | ## Picking the display The scheme decides the question; the display decides what the annotator sees. They are chosen separately, and several schemes render their own cards and need no display field at all. | Display | For | Span target | |---|---|---| | `agent_trace` | Generic steps with type badges | yes, per step | | `coding_trace` | Diffs, terminal blocks, a file tree | yes | | `cot_trace` | Long chain-of-thought with a progress rail | no | | `eval_trace` | Three panes: reasoning, calls, final answer | yes | | `web_agent_trace` | Screenshots with SVG overlays | no | | `multi_agent_discussion` | Several agents, colour-coded, filterable | yes | | `conversation_tree` | Branching, collapsible | no | | `interactive_chat` | Live chat, then the trace | yes | | `live_agent`, `live_coding_agent` | Streaming, with intervention controls | no | "Highlight the sentence where the reasoning broke" therefore works on `eval_trace` and `coding_trace` and **not** on `cot_trace` or `web_agent_trace`. If the researcher wants spans over a long chain of thought, that is `cot_trace` for reading plus a separate `span` scheme over the text field, or `process_reward` if what they actually want is per-step judgements. Ask the registry rather than the flag: `display_registry.get_span_target_capable_types()` returns twelve; `list_displays()`'s `supports_span_target` says nine, because `pdf`, `spreadsheet` and `agent_trace` anchor spans their own way and are missing from it. ## Quirks worth knowing before you build **`gui_trajectory` coordinates default to normalized.** Each step may carry `x`/`y` for the grounding marker, read as 0..1 **unless** you set `coord_space: pixels`. A trace exported in pixels with the default setting is read as fractions, so a click at 420 becomes `left: 42000%` and the marker leaves the picture entirely. The card's own coordinate readout is the tell — `click @ 42000%, 12000%` beside a screenshot 640 pixels wide. **`gui_trajectory` takes the screenshot from `screenshot`, not `screenshot_url`.** The step key is `screenshot`, falling back to `image`, and `screenshot_key` on the scheme renames it. `screenshot_url` is what the live agent runner *writes* into the steps it produces, so a hand-authored trace that copies that spelling gets "no screenshot" on every card while the file itself serves fine over `/screenshots/`. Check the route with `curl` before you suspect the file; if it answers 200, the key is what is wrong. **`multimodal_reasoning` infers the step type from the keys, and every type falls back to `content`.** An image step reads `image`, `image_url`, `url`, `src`, then `content`; a tool step's arguments read `args`, `arguments`, `tool_call.args`, `input`, then `content`, and its name reads `tool`, `name`, `tool_call.name`, then the literal string `tool`; a text step reads `text`, `content`, `reasoning`. So a trace keyed uniformly on `content` renders correctly *as long as each step carries an explicit `type`* — the image draws and the tool call prints. What uniform keying breaks is the type inference. Without `type`, the type comes from the presence of `image`/`image_url` or `tool`/`tool_calls`/`tool_call`, and a step that has neither is text. Drop `type` from a `content`-only trace and every card is labelled TEXT, with the image's URL printed as a line of prose rather than fetched. `type_key` names the field holding the type; the per-step content keys are not configurable. **Screenshots come from a second static root.** Trace data that sets `screenshot_url: screenshots/step_000.png` is served from `<task_dir>/screenshots/`, which is separate from `media_directory` and not configurable. Miss it and every step image and filmstrip thumbnail 404s. **`process_reward` has two modes and they are different tasks.** `first_error` (the default) asks for the first step that goes wrong and cascades from there; `per_step` labels every step. `allow_neutral` is only meaningful in `per_step`, because the cascade has no place for it. `inline_with_trace` puts the controls inside the trace cards instead of beside them, and `ai_prelabel` seeds suggestions the annotator corrects. **`rubric_eval` is the only one with a required field:** `criteria`. **`tool_contention` needs times on its calls.** A call is `{agent, tool, start, end, resource}` and the seconds are what make a contention: same resource, overlapping interval. Feed it `{agent, tool, resource}` and every call sits at 0-0 and can never overlap; from 2.8.2-11 it says the times are missing rather than reporting a clean trace. Self-pairs are skipped from the same release, so `coder <-> coder` cards no longer appear. **The display reads different keys from the schemes.** Both halves of the page take the step list, and they disagree about what a step is called: the schemes take `agent_key` and `steps_key`, while `multi_agent_discussion`, `agent_trace`, `cot_trace` and `eval_trace` take `speaker_key` and `text_key`, defaulting to `speaker` and `text`. Steps shaped `{index, agent, text}` therefore render correctly in the questions and as a column of grey "U" avatars with no names in the trace above them. Set `speaker_key: agent` in `display_options`. (`list_displays()` did not admit the option existed until 2.8.2-11; on an older checkout, read it off the renderer class.) **`eval_trace`'s third pane needs an answer-shaped speaker.** The Final Answer pane matches a step whose speaker or tool name reads like an answer — `final answer`, `send_message`, `respond`, `response`, `finish`, `submit`, `conclusion` — and otherwise falls back to the last action in the trace. A step whose speaker is the obvious `final` matches none of those, so the pane shows the last tool call and the agent's real answer is rendered as an observation hanging off it. Name the last step `Agent (Final Answer)` or similar. **`process_reward` needs `step_text_key` when the steps are dicts.** Pair it with `cot_segmentation` — the intended pairing, and what `examples/agent-traces/cot-process-reward/config.yaml` ships — and each step is a dict of `{index, text, type, char_start, char_end}`. Set `step_text_key: text` and the widget shows the prose; leave it at its default and the annotator is asked to judge the serialized dict. `cot_segmentation` also stamps every step it makes `type: observation`, which is what the `cot_trace` display badges them. **A linked act saves without its link.** In `consensus_tracking` and `context_attribution`, an act named in `linked_acts` asks for a second click on the turn it refers to. Skip that click and the act is stored with no `ref`, Next advances, and the only sign is a badge reading `agreement` where a complete one reads `agreement -> #1`. The widget marks the card `ct-awaiting-ref` and nothing consults it. If the reference is the point of the study, count the ref-less acts at export before you count anything else. **`required` on these now counts the cells, from 2.8.2-11.** Each widget writes what is still missing onto its own form and the banner beside Next reads it, so a scorecard covering four agents on two dimensions plus two team dimensions blocks with "Score each agent and the team as a whole (1 of 10 scored)" instead of passing on the first click. Verified on that config. The banner carries `role="alert"`, so a screen reader gets it too. The media widgets were not part of that change: `video_annotation`, `temporal_grounding` and `tiered_annotation` still treat one mark as an answer. `modalities.md` has the detail. **`trajectory_eval`'s score ignores everything but severity.** Mark all five steps `incorrect`, give each an error type, and skip the severity, and the widget shows `Score: 100 / 100` and stores `"score":100` — a flawless number on a trajectory the annotator condemned at every step. A `correct` step with a stray severity, meanwhile, is deducted. If you export `score`, make severity mandatory in the instructions and recompute from `steps` before you believe it. **`trajectory_edit`'s `require_reason_on_edit` does not require a reason.** It decides whether the reason field is drawn, nothing more: an edit with `"reason": ""` saves and Next advances. Read `reason` at export and treat blank as unexplained. **`code_review` stores empty comments.** *+ Add Comment* appends a row immediately and the row persists whether or not anything is typed into it — `{"category":"bug","file":"","line":null,"text":""}`. Drop rows with empty `text` before counting review comments, and expect `"verdict": ""` from an annotator who rated files and never picked one. ## World-model rollouts `rollout_evaluation` is the odd one out and worth reading the module docstring for. Several videos of the same scenario — real recording, model rollouts, optionally a counterfactual under an intervention — on one frame-locked timeline. Its design decision is worth repeating to a researcher who asks for a plausibility scale: **the primitive is a break-point, not a score.** Rating a generated video 3/5 "produces a number that cannot be checked, cannot be localised, and cannot be used to fix anything." A frame index plus a violation type can be checked against the tensor and compared between annotators. Three layers, three questions: `violations` (where does it break, and why), `preference` (which rollout is better), `counterfactual` (is the divergence plausible *given* the intervention). The third is the one that separates a world model from a video generator: a beautiful continuation that ignores the intervention is a failure no plausibility rating detects. Three defaults that are on unless you turn them off: `blind` (hide generator names), `shuffle` (permute panel order per annotator), and `require_clean` — which refuses a submission that leaves a stream neither marked nor explicitly cleared, because "no marks" otherwise conflates "watched it, found nothing" with "never got to it". ## Blinding and the DOM `rollout_evaluation` blinds server-side: under blinding the labels are positional and the server chose the positions. **Every other scheme does not.** The whole data row is serialized into the page — `data-instance-fields` on the display container and a `data-instance-json` block — so a field you leave out of `instance_display` is still in the DOM and still readable by anyone who looks. To withhold something from annotators, withhold it from the data file. Omitting it from the display is presentation, not access control. ## Judges and automatic evaluators Sixteen evaluators are registered and run without a human: ```python from potato.evaluators.registry import get_supported_evaluators get_supported_evaluators() # exact_match, contains, regex_match, json_valid, json_schema_match, # edit_distance, embedding_similarity, tool_use, tool_call_accuracy, # trajectory_match, llm_trajectory_judge, agent_as_judge, rubric_dag, # groundedness, answer_relevance, context_relevance ``` Use them where the answer is checkable and save the annotators for what is not. The human side of the same question is `judge_alignment` (score items with a judge, compare against the humans, track agreement) and `judge_calibration` (auto-label a sample, have humans re-label the same items). If a researcher says "we want to know if we can trust the judge", that is the pair — not another rating scheme. ## Live agents `live_agent` and `live_coding_agent` run an agent inside the annotation page so annotators judge it as it happens; `agent_proxy` makes an agent the subject of annotation. `live_coding_agent` runs the agent's tool calls inside a container. Two defaults decide whether the session can do anything: `sandbox_image` is `python:3.12-slim` and `sandbox_network` is `none`. Together they mean the agent has whatever that image ships and no way to install more — asked to run the tests, it gets `pytest: not found` and an exit code of 127, and burns its turns looking for a runner. Build or name an image that already carries the tooling the task needs: with `pytest` in the image the same session runs the tests, edits the file and re-runs them to green. `working_dir` is copied into a per-session workspace under `.potato-sandboxes/`, so the repository you point at is not the one the agent edits; check the copy, not the original, when you want to see what it did. The task box the annotator sees is seeded from the field the display is bound to. Bind it to the field holding the task text, not to the one naming the repository, or the box arrives holding the repository name and the annotator has to retype the task before starting the agent. Which directory the agent gets is `working_dir`'s job, so the key is free to carry the prompt. `live_agent` drives a real browser, so it needs `playwright` and a browser installed (`pip install playwright && playwright install chromium`). Its `endpoint_type` takes `anthropic_vision`, `ollama_vision` or `openai_vision` — the last is what an OpenAI-compatible server such as vLLM needs, and the shipped example's comment names only the first two. The system prompt has to ask for an `action` that is an **object**, `{"type": "click", ...}`, because that is what the runner expects; a prompt that asks for a bare string gets one back and the session ends at step 0. These open control surfaces, so treat them the way you would treat anything that executes: do not expose them on a deployment that anyone with the URL can register for, and check `deploying.md` before handing out a link. ## Verifying one The failure mode is an empty card list, and an empty card list looks like a short trace. 1. **Count what rendered against what is in the data.** If the item has eleven steps, eleven cards should exist. This is the check that catches a wrong `steps_key`, and nothing else does. 2. **Judge one step, navigate away and back.** Answers are stored keyed by `index` with only-judged steps saved; a restore bug shows up as answers attaching to the wrong card. 3. **Read `user_state.json`** and confirm the stored JSON list has the shape the export expects. 4. **For `gui_trajectory`, look at where the marker landed.** That is the whole point of the scheme, and it is the thing `coord_space` silently breaks. <!-- https://davidjurgens.github.io/potato-skill/phases-and-pages/ --> # Phases and Pages Consent, instructions, training, surveys, and the order they run in. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Everything a researcher describes as "they should see X first" and "at the end I want to ask them Y" lives here. None of it is in the generated key reference: `phases` and `surveyflow` are documented as `object` with no sub-keys, and `potato validate --strict` does not check inside either of them. Every fact below was established by running the server and reading the startup log. ## Which of these you actually need Start from none of them. **7 of Potato's 218 bundled examples declare a `phases` block at all**; the other 207 open on the first item. A phase is a page an annotator has to get through before doing any work, and every one you add costs attrition and setup time, so add them because someone asked or because the protocol requires it — not because a study "should" have them. | Phase | Add it when | Skip it when | |---|---|---| | `consent` | Real participants, real data, and an IRB or ethics protocol that requires recorded consent — or you are paying strangers on a crowd platform | Prototyping, pilots with colleagues, your own data, any run whose output is a screenshot. A consent form on a prototype is theatre, and it is the wrong wording anyway | | `instructions` | The rules do not fit in `annotation_instructions`, or annotators need worked examples before the first item | The task is one question with obvious labels. `annotation_instructions` renders on every page and is read more than a page nobody returns to | | `training` | Labels need calibrating, the budget is large enough that a bad annotator is expensive, or you want a pass/fail gate | Fewer than a few hundred items, or the annotators are the researchers. Training needs a graded item file per scheme, which is real work | | `prestudy` | You are screening — language, expertise, a qualification question that decides who continues | You already know who is annotating | | `poststudy` | Someone named a question they want answered at the end | Always, otherwise. **Do not add a post-study questionnaire nobody asked for.** It is the phase most often added on reflex, it is where the identifiable answers accumulate, and it makes the export two files instead of one (`getting-the-data-out.md`) | Two things follow that are easy to get backwards. **Consent is a protocol decision, not a config decision.** If the study needs consent, the wording comes from the approved protocol and you should be asking for it rather than drafting it — `asking-the-experimenter.md` treats it as one of the questions worth interrupting for. A placeholder consent page that ships is worse than no consent page, because it looks like the study has ethics coverage. **A phase you add for the pilot tends to stay.** Adding `poststudy` later is a config edit and a page file; removing one after annotators have answered it means you now hold survey responses you did not need. Default to adding phases late. ## The six phase types ```python consent | instructions | training | annotation | poststudy | prestudy ``` That is the complete list. Anything else fails at boot with `Failed to load phase 'x': Unknown phase: x` — **and `potato validate --strict` passes it**. A config naming fourteen invented phase types validates clean and then loses fourteen phases at startup. There is no `survey` phase, no `demographics`, no `debrief`, no `end`, no `done`. A post-study questionnaire is a `poststudy` phase whose page happens to contain survey questions. Screening before the task is `prestudy`. **`order` is honoured as written**, including where you put `prestudy`. Walked in a browser, this order ```yaml phases: order: [consent, prestudy, instructions, training, annotation, poststudy] ``` serves consent, then the screening page, then the instructions page, then the training questions, in exactly that sequence. A phase with required questions gates the next one — the annotator cannot advance past it unanswered — which is what "only people who answer the screening questions get through" means in practice. **Phase pages record; they do not route.** There is no mechanism for screening someone *out* on the basis of what they answered, and no documented way to feed a survey answer into assignment. The nearest thing is `category_assignment`, which routes by item category and can gate on a qualification earned in **training**, not on a survey response. If a researcher says "only screen-reader users get through", tell them the phase collects the answer and someone has to act on it — do not imply the phase enforces it. Note also that the **top-level `prestudy:` key is dead code** — the key documentation says so outright ("the only code that reads it is an unreferenced method"). Screening is a `prestudy` *phase*, not the `prestudy` block. ## Two ways to write it The names in `order` are yours, not the type names, so a type can appear more than once. Two `instructions` phases called `intro_a` and `intro_b` serve in that order with a Previous button between them, which is the way to split long instructions across pages rather than one wall of text. **The exception is `annotation`: a second one is never served.** The annotation flow owns the item queue and ends the study when it is empty, so a study declaring `first_pass` and `second_pass` runs the first, reaches the last item and goes straight to the completion page — I drove three items and watched progress go 0/3, 1/3, 2/3, done, with the second phase's title never appearing. Since 2.8.2 `validate` warns and names both phases, and `--strict` refuses; the warning fires on the bare name `annotation` in `order` as well as on a named block, since `["annotation", "second_pass"]` is the same mistake spelled differently. For a second pass over the same items, look at adjudication or a second annotator instead. **Mapping form** — an `order` list plus one entry per phase, keyed by name: ```yaml phases: order: [consent, instructions, training, annotation, poststudy] consent: type: consent title: Consent to take part file: surveyflow/consent.jsonl instructions: type: instructions title: About this study file: surveyflow/instructions.jsonl training: type: training title: Practice annotation: type: annotation poststudy: type: poststudy title: Your own news habits file: surveyflow/news_habits.jsonl ``` **List form** — the same thing with `name` inside each entry, no `order`: ```yaml phases: - {name: consent, type: consent, title: Consent, file: surveyflow/consent.jsonl} - {name: annotation, type: annotation} - {name: poststudy, type: poststudy, title: Wrap up, file: surveyflow/end.jsonl} ``` Both work. Mapping form reads better once any phase has more than three keys. **Naming a phase in `order` without defining it fails `--strict`**, and is worth letting fail rather than downgrading: ``` OTHER WARNINGS: - Phase 'consent' is listed in phases.order but no 'phases.consent' block defines it. It will be dropped from the phase order. FAILED — 1 warning(s); --strict treats these as errors. ``` The phase is then genuinely dropped — the boot log says `dropping it from the phase order` and the study runs without it. So the failure mode is quiet rather than loud: a consent step you meant to have, and a study that consents nobody. `--strict` is the check that catches it, which is the argument for running `--strict` rather than plain `validate`. Define every phase you name. If you want a phase gone, take it out of `order`. The annotator never sees it and the config still validates. If a phase you configured is not appearing, this is the first line to grep for. ## Every non-annotation phase needs a page ``` ERROR: Phase consent requires 'instrument', 'instruments', or 'file' to specify its annotation schemes ``` - `file:` — a path to a `.jsonl` page (below). Use it for anything you wrote yourself. - `instrument:` / `instruments:` — take a **string** (or list of strings) naming a built-in instrument from the library below. Handing `instrument` a list of scheme dicts fails with `unhashable type: 'list'`, which is the error you get if you assume it is an inline scheme list. The two combine: instrument questions are laid out first, then the file's. Note that `instrument` and `instruments` do **not** — the singular wins and the list is ignored, silently. Use `instruments:` for one or many and forget the singular exists. ## The instrument library Potato ships 55 validated questionnaires, so a post-study survey that asks for a standard measure should almost never be hand-written into a page file. Naming one writes the whole questionnaire, wording and response options included, into the phase. ```yaml phases: order: [annotation, poststudy] poststudy: type: poststudy instruments: [tipi, who-5] ``` That boots to a page of fifteen questions — ten from `tipi`, five from `who-5` — with no page file anywhere. Eight categories: | Category | Examples | |---|---| | `personality` | `bfi-2` (60 items), `tipi` (10), `nfc`, `gse` | | `mental_health` | `phq-9`, `gad-7`, `k6`, `ces-d`, `pss-10`, `who-5` | | `affect` | `panas`, `iri` | | `social` | `rse`, `ucla-loneliness`, `mos-ss`, `macarthur-ladder` | | `attitudes` | `mfq`, `sdo-7`, `rwa`, `trust-ess`, `political-efficacy` | | `response_style` | `mc-sds` — social desirability, for detecting it in the rest | | `short_forms` | `bfi-10`, `mini-ipip`, `phq-2`, `gad-2`, `pss-4` | | `demographics` | batteries lifted from ANES, GSS, ESS, WVS, MIDUS, IPUMS and the ACS | The registry is the authority, and it carries the item count and a source URL for each one: ```python from potato.survey_instruments import list_instruments, get_categories get_categories() # category -> instrument ids list_instruments("mental_health") # id, name, items, description, url ``` Two things to tell a researcher before they pick one. **Long instruments are long**: `bfi-2` is 60 questions and `ztpi` is 56, a bigger ask than the annotation task in most studies. The `short_forms` category exists for that reason. And **a fifteen-question page runs about two and a half screens**, so the submit button ends up below the fold. `check_ui.py --config config.yaml --phase poststudy` reports the exact position — use `--phase` rather than the ordinary walk, which starts at the landing page and never gets past annotation on a corpus of any size. `psychometrics` is a different thing entirely despite the neighbouring name: it fits an item-response model over the *annotation labels* to estimate annotator ability and item difficulty. It has nothing to do with questionnaires. `training` and `annotation` need no page: training reads `training.data_file` and annotation reads `annotation_schemes`. ## The page file format A phase page is JSONL, **one annotation scheme per line**, using exactly the scheme syntax from `annotation-types.md`: ```jsonl {"annotation_type":"pure_display","name":"consent_text","allow_html":true,"description":"<h3>Consent</h3><p>…</p>"} {"annotation_type":"radio","name":"consent_agree","description":"Do you agree to take part?","labels":["I agree","I do not agree"],"humanize_labels":false,"label_requirement":{"required":true}} ``` Notes that cost time to find: - **Prose goes in a `pure_display` scheme**, in its `description`, with `allow_html: true`. There is no `text:` key on a phase that renders anything — `title` renders, `text` is ignored. - `pure_display` logs `WARNING: No labels provided for pure display content` on every render. It is noise; the content renders. - Phase-page prose lands inside a `<legend>`, which the theme sets to `font-weight: 600`. A page of it reads as entirely bold. Fix with `base_css`: ```css form.pure-display legend { font-weight: 400; } form.pure-display legend b, form.pure-display legend h3 { font-weight: 650; } ``` `base_css` is inlined into a `<style>` tag rather than served as a file, so the path only has to be readable at boot. - Anything with `label_requirement: {required: true}` is enforced on the phase page the same way it is on the annotation page. - Keys inside these lines are **not validated at all**. A misspelled `annotation_type` here fails at boot; a misspelled option key is ignored. ## Phase-level schemes replace the top-level list, and you cannot mix ```yaml phases: poststudy: type: poststudy annotation_schemes: [...] # with top-level annotation_schemes present ``` fails validation: ``` Configuration has both top-level 'annotation_schemes' and phase-level 'annotation_schemes' in: poststudy. Use only one location to avoid confusion. ``` This is all-or-nothing across the config. Defining a survey inline on one phase forces every phase, including `annotation`, to carry its own scheme list. For a normal study — one question set for the items, different questions on the survey pages — use `file:` for the phase pages and keep the top-level `annotation_schemes` for annotation. Reach for phase-level schemes only when you genuinely want different questions per phase and are willing to write them all out. ## `annotation_instructions` is not the instructions phase `annotation_instructions` is the collapsible banner at the top of **every** page, including consent, training and post-study. It is a standing reminder, not a page. Two consequences: 1. Long label definitions in it push the item and the questions below the fold on every single item. Put the definitions on an `instructions` phase and keep the banner to a few lines. 2. It appears above the consent text, which looks like a mistake to an annotator reading it for the first time. Word it so it survives that context. ### Rendering a phase page `potato preview --phase X --screenshot out.png` renders the page for phase X. Two things about it: **It accepts only five names.** `--phase prestudy` is refused: ``` Error: phase must be one of ('consent', 'instructions', 'training', 'annotation', 'poststudy'), got 'nonsense' ``` `prestudy` is a valid phase for the *server* and not for the *preview CLI*, so a screening page cannot be rendered this way. Walk it in a browser instead. **It exits 1 on a healthy phase page.** Every phase page fires Potato's own `/api/current_instance` 404 and two `[SpanManager]` console errors, and `--screenshot` counts those as things to report: ``` $ potato preview config.yaml --phase consent --screenshot consent.png ; echo $? console.error: [SpanManager] Failed to get server instance ID during initialization HTTP 404: http://127.0.0.1:9080/api/current_instance 1 $ potato preview config.yaml --screenshot annotation.png ; echo $? Rendered cleanly — no browser errors. 0 ``` Both configs are fine. Do not gate on the exit code for a phase page, and do not go looking for the bug — look at the PNG and check the error list contains nothing beyond those three lines. Only pass a phase the task actually has, or every route redirects toward a page that does not exist and the render fails with a redirect loop. ## `surveyflow` versus `phases` `surveyflow` is the older way to attach pages before and after annotation: ```yaml surveyflow: "on": true order: [pre_annotation, post_annotation] pre_annotation: [surveyflow/consent.jsonl, surveyflow/instructions.jsonl] post_annotation: [surveyflow/news_habits.jsonl] sample_status: [] ``` Quote the `"on"` key. Bare `on:` is a YAML boolean, and the loader warns: ``` `on:` at surveyflow was read as the boolean True … quote the key ("on":) to be explicit. ``` Prefer `phases`. It names the phase types explicitly, it controls order in one place, and it is what the training phase and `--debug-phase` key off. A directory called `surveyflow/` holding the page files is a convention worth keeping even when the `surveyflow:` block is absent — that is where the pages in the example above live. ## Checking the workflow actually assembled Validation cannot see any of this. Boot the server and read the log: ```bash grep -E "phase|Loaded [0-9]+ (training|attention|gold)" server.log ``` A healthy boot is **silent about phases**. Any of these means a phase is missing from the annotator's walk: | Line | Meaning | |---|---| | `Phase 'x' in order but not defined` | You listed it and never described it | | `Failed to load phase 'x': Unknown phase: y` | `type:` is not one of the six | | `Phase x requires 'instrument', 'instruments', or 'file'` | No page attached | | `Failed to load phase 'x': unhashable type: 'list'` | `instrument:` got a list of schemes | Then walk it in a browser. A phase can load cleanly and still be unreachable because of what precedes it. See `running-a-task.md`. <!-- https://davidjurgens.github.io/potato-skill/quality-control/ --> # Attention Checks and Gold Standards The quality-control ladder, and what each rung costs. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Four blocks, three of which need a side file whose format is documented nowhere and whose error messages mostly do not name the missing field. All formats below were recovered by booting the server against candidate files and reading the log. **The failure mode to design against:** every one of these can be enabled, validate clean under `--strict`, log a warning nobody reads, and run the entire study with the feature off. Two of them do not even log an error. After any change here, boot the server and check the count. The second failure mode is subtler and worse: the feature runs, scores everyone, and reports a number that is not what you think it is. Read "Where the results live" below before you act on an accuracy figure. ```bash grep -E "Loaded [0-9]+ (training instances|attention check items|gold standard items)" server.log ``` | Feature | Healthy line | If it says 0 | |---|---|---| | training | `Loaded 3 training instances` | Boot **fails** with an exception. Loud. | | attention checks | `Loaded 2 attention check items` | `WARNING … missing required fields` then silence. Study runs unchecked. | | gold standards | `Loaded 2 gold standard items` | `ERROR: Gold standards are enabled but NOT RUNNING`. Names the missing field. | ## Training A qualification round of worked examples, marked as the annotator answers, before they touch real items. It is the only quality-control feature that improves the annotations rather than only measuring them. ```yaml training: enabled: true data_file: data/training.json allow_retry: true feedback: show_explanations: true passing_criteria: min_correct: 3 ``` Recognized sub-keys: `enabled`, `data_file`, `allow_retry`, `feedback`, `passing_criteria`, `failure_action`, `annotation_schemes`. That is all seven. At this level `min_accuracy`, `max_attempts`, `num_items`, `show_feedback`, `threshold` and every other plausible name is rejected by `--strict`, which is how you find the real ones. Inside the two nested blocks an unknown key is a warning, not an error. `passing_criteria` takes `min_correct`, `require_all_correct`, `max_mistakes`, `max_mistakes_per_question` and `max_attempts`; `feedback` takes `show_explanations` and `allow_retry`. Anything else passes validation and logs `Unknown key training.feedback.enabled is ignored (known: allow_retry, show_explanations)`, so `min_accuracy: 0.6` under `passing_criteria` validates and does nothing. The pass mark is `min_correct`, a count of correct answers rather than a share; a value that is not a positive integer is a fatal error. `failure_action` is `move_to_done` (default) or `repeat_training`, and any other value is a fatal error. `annotation_schemes` lets the practice questions use different schemes from the real task; omit it and training uses the task's own. `feedback` and `passing_criteria` must be dictionaries; a bare `true` or `0.6` is a fatal error. Also add the phase, or the block does nothing: ```yaml phases: order: [consent, training, annotation] consent: type: consent title: Consent to take part file: surveyflow/consent.jsonl training: type: training title: Practice annotation: type: annotation ``` Define every phase you name in `order`. Naming one you do not define is a `--strict` failure, and the phase is dropped from the order rather than silently half-present. See `phases-and-pages.md`. ### The training data file An **object**, not an array, with a `training_instances` key: ```json { "training_instances": [ { "id": "train01", "text": "The energy regulator fined the utility £2 million on Monday …", "correct_answers": { "tone": "Neutral", "responsibility": "A company or developer", "misleading": "No" }, "explanation": "Wire copy. Every sentence is attributed reporting, so …" } ] } ``` **Required per instance: `id`, `text`, `correct_answers`.** Nothing else. Keys in `correct_answers` are scheme `name`s; values are the label strings exactly as they appear in `labels:` (the stored value, not the humanized display form). Three ways this goes wrong, with the exact message: | Message | Cause | |---|---| | `Training data must be a JSON object` | You wrote a JSON array | | `Training data must contain 'training_instances' field` | Right shape, wrong key | | `Training instance missing required fields: {…}` | Dumps the whole instance and **never names the field**. It is one of `id`, `text`, `correct_answers`. | Notes: - `explanation` is shown with the mark. Write it — a wrong answer with no explanation teaches nothing, and this is the one place in the task where you can correct a misreading before it contaminates 500 items. - The training page renders the question text only. Any context an annotator needs (a headline, a speaker, an outlet) must be folded into that string; the display fields from `instance_display` are not used here. From Potato 2.8.2 a training instance keeps all its fields and the question comes from the project's `text_key`, so `{"text": ...}` and `{"caption": ...}` both work where `text_key: caption`. Earlier builds kept six keys and dropped the rest. - **A practice round still cannot show media.** With no `instance_display` there is no `<img>` for an `image_annotation` or other geometry scheme to take its bitmap from, and `source_field` does not save it: the instance fields the client would read are not serialized onto a phase page. The trainee gets an empty canvas, the console says `[ImageAnnotation] No image URL found! Check that the instance data contains an image URL`, and the config validates clean. `check_ui.py` flags it as "a geometry scheme on a page with no image on it". Potato ships no example combining an image display with a training phase, which is why nothing catches it. Either drop `training` on a media task, or use it only for the classification schemes and say in the `explanation` that the drawing tool is practised on the first real item. (Before 2.8.2 this failed more loudly and more confusingly: the canvas read "Failed to load image. Check the URL or CORS settings." and the console showed a 404 for a URL made out of the instance's own prose.) - Grading is per scheme against `correct_answers`. Schemes you omit are not graded. Span schemes are not graded. There is no way to express a gold span here. - Progress shows as `Question 1 / 3` with a running `Correct: n · Mistakes: n`, and a correct answer advances with `Correct! Moving to next question.` - State lands in `annotation_output/<user>/user_state.json` under `training_state`: per-question `correct`/`attempts`, plus `passed`. ## Attention checks Items with an obvious right answer, mixed into the stream, to catch someone who stopped reading halfway through. ```yaml attention_checks: enabled: true frequency: 3 # every third item; use probability OR frequency, never both items_file: data/attention.json min_response_time: 8 # a faster answer fails the check failure_handling: warn_threshold: 2 # warn from the second failure warn_message: "Please read each item before answering." block_threshold: 5 # block at the fifth ``` `failure_handling` reads four keys and no others: `warn_threshold`, `warn_message`, `block_threshold`, `block_message`. There is no `action` key. A sub-key it does not know is a warning rather than an error, so a config naming one validates without `--strict` and does nothing. The thresholds then fall back to 2 and 5 (`quality_control.py:375`). Setting both `frequency` and `probability` is a fatal error. `min_response_time` fails a check answered faster than the threshold, whatever the content was. Set it well under a plausible reading time — the point is to catch a straight-through click, not to punish someone who recognised the item — and remember it is one signal among several: `/admin/api/suspicious_activity` and the behavioral analytics see patterns a per-item threshold cannot. ### The attention item file A plain JSON array. **Required per item: `id` and `expected_answer`.** ```json [ { "id": "check01", "text": "This item is a check that you are still reading carefully. It is not a real article. Highlight this sentence and mark it Factual reporting. For the tone question, select Neutral …", "expected_answer": {"tone": "Neutral", "responsibility": "No one", "misleading": "No"} } ] ``` Give the items whatever display fields your `instance_display` reads (`headline`, `body`, …) as well. They render as a normal item and look broken without them. Two traps worth designing around: - **Instruct every required scheme. Span schemes included.** A check item that says "select Neutral" but says nothing about the highlight leaves the annotator stuck: `Next` returns `400 Required annotation(s) not completed: <scheme>` and the only feedback is a small toast naming the internal scheme name. - **Attention checks consume item slots.** They are inserted into the annotator's ordering, so with a per-annotator cap equal to the item count they displace real items — annotators finish having never seen the last two. See `assignment-and-agreement.md`. ## Gold standards Items with known labels used to score annotators continuously, rather than once in training. Use them when annotators are paid strangers. ```yaml gold_standards: enabled: true items_file: data/gold.json mode: mixed # training | mixed | separate frequency: 5 accuracy: min_threshold: 0.7 evaluation_count: 10 auto_promote: enabled: false ``` **`mode` takes three values and `mixed` is the one that is implemented.** `separate` behaves identically to it and says so at boot — `gold_standards.mode 'separate' behaves as 'mixed': 3 gold items are injected into the annotation stream every 2 items` — and `training` serves nothing at all, because no code path delivers gold during the training phase. That last one fails `--strict` and warns twice at boot, naming the count that will never be served. Grade a training phase with its own practice questions instead. **`accuracy` and `auto_promote` must both be dictionaries**, even though the key documentation types them `number|object` and `boolean|object`. `accuracy: 0.7` and `auto_promote: false` both fail with `must be a dictionary`. Their documented sub-keys are `min_threshold` / `evaluation_count` and `min_annotators` / `agreement_threshold` — and nothing checks those, because validation stops one level in. A wrong sub-key here validates clean and does nothing. **The accuracy threshold reports to you, not to the annotator.** Crossing it logs `<user> is below the gold standard accuracy threshold: 0.50 over 2 evaluated items, threshold 0.90` and puts the annotator in `gold_standards.below_accuracy_threshold` on the admin route. It does not block them and there is no setting that does — attention checks block, gold standards never do. If you need underperformers stopped rather than counted, that is `attention_checks.failure_handling` or your own procedure. ### How many gold items an annotator sees `frequency` sets the spacing, but **the file governs the count**: every annotator is served every item in `items_file`, once. Two items in the file with `frequency: 4` over eight real items gives two insertions per annotator, not `8 / 4`. Size the file to how many checks you want each person to get. They are injected on top of the per-annotator quota rather than inside it, so there is nothing to pad (see `assignment-and-agreement.md`). ### The gold item file A JSON array. **Required per item: `gold_label`** (plus `id` and the display fields). Both shapes load: ```json [ {"id": "g1", "text": "…", "gold_label": {"scheme_name": "label"}}, {"id": "g2", "text": "…", "gold_label": "label"} ] ``` Use the dict form on any task with more than one scheme. Two things about gold scoring worth telling a researcher before they read the number: - **A drawn answer can be a gold answer.** Boxes, polygons, masks and points are graded by overlap at 0.5 IoU, not by equality, so "did they find the thing I planted" is a question gold standards can now answer. Write the shape in the form the client stores — normalized fractional coordinates nested under `coordinates`, not pixels: ```json {"id": "gold_dog", "image_url": "…", "gold_label": {"region": [{"type": "bbox", "label": "subject", "coordinates": {"x": 0.18, "y": 0.20, "width": 0.43, "height": 0.40}}]}} ``` On an `image_annotation` task with that item, an annotator drawing the same box scored 1.0 and one drawing a box in the opposite corner scored 0.0. Every gold shape must be matched by a user shape of the same label, with no extras — boxing the whole image does not count as finding the object. Spans are stored elsewhere and I have not tested them; assume a gold span does not work. - **Both scores are keyed by (user, item)**, so re-saving an answer replaces the earlier result rather than adding one. The denominator is the number of gold items the annotator was served, and an annotator who changes their mind three times still counts once. This is the one file in the pack with a good error message: `Gold standard item skipped, missing gold_label: {…}`, followed by `Gold standards are enabled but NOT RUNNING: no usable items were loaded`. Copy that standard when judging the others: if the log does not say the feature is running, assume it is not. ### Where the results live Neither feature tells the annotator anything by default and neither writes a number into the output. Gold standards are silent unless you ask otherwise: ```yaml gold_standards: feedback: show_correct_answer: true show_explanation: true ``` Both default to false, and both off means the annotator never learns how they scored. That is the right choice when the gold items are also the measurement, and the wrong one when they are meant to teach. I have not driven a study with the block on, so I am claiming only that the key is recognised and read, not what the annotator sees. The scores live on the admin API: ```bash curl -s -o /dev/null localhost:8000/admin # makes the key file exist K=$(cat admin_api_key.txt) curl -H "X-API-Key: $K" localhost:8000/admin/api/quality_control ``` which reports pass/fail counts and per-user and per-item breakdowns for both. Results are written to disk as they are recorded and read back at boot, so they survive a restart along with the block list. Pull that route at handover the way you would pull the export — it is the only place the pass rates are assembled, and it is not in the export. The raw answers are also in `annotation_output/<user>/user_state.json` under the check and gold item ids, so the scores can be recomputed by hand against the items files if you ever need to show your working. ## Seeding answers with `pre_annotation` Not quality control in the sense the rest of this file is — it is the same manager, and it is the block a researcher means by "we already have model labels, have people correct them". ```yaml pre_annotation: enabled: true field: predictions # the field on each item holding the labels allow_modification: true show_confidence: true highlight_low_confidence: 0.7 ``` The named field is a dict on the item, keyed by scheme name: ```json {"id": "t01", "body": "The export button spins forever ...", "predictions": {"category": "Bug"}} ``` A field that is not a dict is skipped with a warning naming the item; a scheme name that is not in the config is skipped silently. Seeds are applied only when the user has **no** annotation for that item yet, so a returning annotator sees their own answers rather than the model's. I ran this with plain labels. `show_confidence` and `highlight_low_confidence` need per-prediction confidences in the same field, and I did not supply any, so take those two keys as documented rather than driven. **A seed the annotator never looks at is saved as their answer.** Measured: an item seeded `category: Bug`, with the annotator answering only the other required scheme and pressing Next, stored `[{"schema": "category", "name": "Bug"}, "Bug"]` in their `user_state.json`. The export cannot distinguish it from a label they chose, because nothing records that it came from the model. Two consequences worth stating out loud to the researcher: - Agreement between two annotators over seeded items measures the seeds, not the people. Seed one arm of the study or neither. - If the point is to measure whether people accept model labels, the thing you need — did they change it — is in the behavioural log rather than the annotations, or in `export_include_annotation_changes`. `display_logic` reads seeded answers at load: a scheme gated on `category equals Bug` is visible on first paint for an item seeded that way, and the behavioural log records `display_logic_change ... "reason": "Conditions met"`. ## Adjudication A queue where a named person resolves disagreements into one final label. Use it when you need a single label per item rather than a distribution. ```yaml adjudication: enabled: true adjudicator_users: [alice] agreement_threshold: 0.8 ``` `adjudicator_users` must be a non-empty list of usernames — the validator says so outright. `agreement_threshold` is the other recognized key; `mode`, `strategy`, `queue`, `auto_resolve` and `adjudicators` are all rejected. A named adjudicator works at **`/adjudicate`**, which is not linked from the annotation page: a queue of items that fell below the threshold, each showing its agreement percentage and every annotator's answer per scheme, plus a decision form with confidence, an error taxonomy and a guideline-update flag. Items enter the queue on disagreement rather than on reaching the annotator cap, so an item two of three annotators disagree about is adjudicable before the third arrives. Tell the adjudicator the URL; `/annotate` shows them the completion page instead. Decisions are written to `annotation_output/adjudication/decisions.json` as they are made: ```json {"instance_id": "p01", "adjudicator_id": "adj@x.com", "label_decisions": {"decision": "Include", "quality": 3}, "source": {"decision": "adjudicator"}, "confidence": "medium", "notes": "...", "error_taxonomy": [], "time_spent_ms": 39553} ``` **They are not in `csv`, `jsonl` or `parquet`** — those carry the annotators' answers, so a study run for the resolved labels would export the disagreements and leave the resolution behind. Ask for the `adjudication` format, which writes `adjudicated.csv` (one row per instance and scheme, with `source`, `adjudicator_id`, `confidence` and a timestamp) beside `adjudication_log.jsonl` (the full decision record, notes and error taxonomy included). Request it explicitly: it is not in the default set, and `export_annotation_format` has to name it. Before you design a workflow around adjudication, **open `/adjudicate` on a real item and check that the schemes you care about have a control there.** The panel shows every scheme's annotator responses above a decision form; a composite scheme is decided by adopting one annotator's whole answer rather than by re-entering it, and `label_decisions` records only what had an input. ## `require_fully_annotated` ```yaml require_fully_annotated: true ``` Free, and nearly always right. Combined with `label_requirement: {required: true}` per scheme it stops an annotator advancing past a half-answered item. Know what it does at the boundary: the client checks radios, checkboxes and text boxes and shows `Please answer the required questions: <question text>` inline. A **required span scheme** is not checked client-side — the server refuses the navigation with `400 Required annotation(s) not completed: <scheme name>` and the annotator gets a small corner toast naming the internal scheme name, not the question. Requiring a span is fine; just make sure every item, attention checks included, is one where drawing a span makes sense. ## `text_as_image` ```yaml text_as_image: true ``` Offer it when a researcher is worried that annotators will paste items into a chatbot. The server renders each item's text to a PNG and blanks the words everywhere the page could leak them. On a classification task in 2.10.0 the item's sentence was absent from the `/annotate` HTML, and `/api/current_instance` returned the item with only its `id`. It makes copying more work, not impossible: an annotator can retype the text or run OCR on a screenshot. Say that when you offer it. It costs accessibility. `potato validate` warns that a screen reader gets nothing from the picture, and a study that has to stay accessible should leave it off. It refuses anything that still shows or selects the text. With a span scheme, `potato validate` and `potato start` both exit 1. `pocket.enabled: true` and `rooms.enabled: true` each fail `potato validate` the same way, with a message naming the conflict: ```text text_as_image is incompatible with these annotation schemes: span. Each one anchors its annotations to character offsets in the instance text, and text_as_image removes that text from the page, so an annotator would have nothing to select. Turn off text_as_image, or use a scheme that does not select text. ``` Two more are refused by `potato validate`. An `audio_annotation` scheme whose `source_field` is not the `text_key` field shows the item text in a box beside the player, which the picture does not replace. A scheme whose own field attribute names the text field reads a field the feature blanks: `text_edit` with `source_field: text` is refused, with a message naming the scheme and the attribute, and the same scheme pointed at an `mt_output` field validates. Potato's `docs/advanced/text_as_image.md` has the full list, including video and tiered schemes. Ask about anti-copying before you choose span annotation, because the two cannot be combined. ## Choosing among them Add in this order, and stop when the risk is covered. Each one costs annotator time. 1. `require_fully_annotated: true`. Free. 2. Agreement. Automatic once items overlap; nothing to configure. 3. `training`, when the label scheme needs explaining. The only one that improves the data. 4. `attention_checks`. Cheap to author, and catches clicking-through. 5. `gold_standards`, when annotators are paid strangers and you need a running accuracy per person. Costs someone writing correct answers. 6. `adjudication`, when you need one final label per item. <!-- https://davidjurgens.github.io/potato-skill/assignment-and-agreement/ --> # Assignment and Agreement Who annotates what, and which agreement measure follows from it. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Two questions that look separate and are not: the number of annotators per item decides whether an agreement number exists at all, and the per-annotator quota decides whether the items you care about ever reach three annotators. ## The keys | Key | What it does | Default | |---|---|---| | `num_annotators_per_item` | Annotators per item; an item retires when it has this many. **Set this one.** | unset: no limit (`0` too) | | `min_annotators_per_instance` | Deprecated spelling of the key above, read as the cap with a warning | unset | | `max_annotations_per_item` | Older spelling of the same cap; `-1` unlimited | -1 | | `max_annotations_per_user` | **Cap on dataset items per annotator**, unless `per_annotator_quota` overrides it | the item count | | `per_annotator_quota` | Per-annotator caps: `default`, `by_user`, `by_user_role`. Beats the key above | unset | | `automatic_assignment.instance_per_annotator` | Nothing. See below. | — | | `assignment_strategy` | Order items are handed out in | `fixed_order` | | `random_seed` | Makes the ordering reproducible | unset | The three are one setting. With none of them set there is no cap: on 2.11.0, five annotators of a two-item study were all served the first item. Two of them with the same number validate clean, and two with different numbers are refused: ``` 'max_annotations_per_item' and 'num_annotators_per_item' are both set with conflicting values (5 vs 3). Drop 'max_annotations_per_item' — 'num_annotators_per_item' is the canonical key. ``` `min_annotators_per_instance` used to be stored at load and read nowhere (checked against Potato `be457e7b`), so a study that set it collected as many annotators as it got. From 2.11 it is read as the cap. Set to 2 on its own, the first two annotators got the first item, the third got the second, and the fourth and fifth got nothing. Validation warns: `min_annotators_per_instance: 2 is deprecated and has been read as num_annotators_per_item: 2`. Rename it. (The key of the same name under `icl_labeling.example_selection` is a different setting, and that one is read.) #### `num_annotators_per_item.adaptive` `num_annotators_per_item` also takes a mapping, and the `adaptive` block under it reopens an item when the first annotators disagree: ```yaml num_annotators_per_item: default: 2 adaptive: enabled: true disagreement_threshold: 0.5 boost_to: 4 ``` Measured on 8 items with two annotators who disagreed on `x0`-`x3` and agreed on `x4`-`x7`. The server logged four lines and only four: ``` Adaptive boost: x0 raised to 4 annotators (current=2) ``` The third and fourth annotators through the door were each served exactly `x0 x1 x2 x3`; the agreed half had retired at 2; the fifth got nothing. So the budget goes where the label is actually contested, and a study that would have cost 8 x 4 = 32 annotations cost 24. Worth knowing before you reach for it: this is a cap being raised, so the arithmetic in the section above stops holding. `items x num_annotators_per_item` is no longer the ceiling on how much work exists — with `boost_to: 4` on a `default: 2` study the true ceiling is twice what `default` suggests, and how close you get depends on how much your annotators disagree, which is the thing you do not know yet. Recruit against `boost_to`, not against `default`. #### `instance_per_annotator` Measured on 10 items, reading the assignment the server actually built: | Config | Items served to one annotator | |---|---| | nothing set | 10 — all of them | | `max_annotations_per_user: 5` | 5 | | `automatic_assignment.instance_per_annotator: 5` | 10 — **no effect** | | `num_annotators_per_item: 3` | 10 — no effect on the quota | `instance_per_annotator` is not read anywhere in the server, and `automatic_assignment` does not validate its sub-keys, so it produces no warning and no behaviour. Use `max_annotations_per_user`, or `per_annotator_quota` if different annotators need different amounts. Left unset, the cap is the number of items loaded, so every annotator is offered the whole corpus. `-1` is explicit unlimited, and is what a dynamic data source needs for items added after boot to be assignable at all. #### Giving different annotators different amounts `per_annotator_quota` caps annotators individually. Potato resolves one annotator's cap in this order, taking the first that matches: 1. `per_annotator_quota.by_user[<user>]` 2. `per_annotator_quota.by_user_role[<their role>]`, where the role comes from the `user_roles` map 3. `per_annotator_quota.default` 4. `max_annotations_per_user` So `default` displaces `max_annotations_per_user` rather than sitting beside it: set both and the global is never read, and nothing warns. Without `default`, an annotator matching neither override falls through to the global, which is how the two keys are meant to combine. Measured on 24 items with `default: 5`, `by_user: {alice: 3}`, `by_user_role: {expert: 7}`, `user_roles: {alice: expert, bob: expert, carol: novice}`, and `max_annotations_per_user: 20`: | Annotator | Served | Why | |---|---|---| | alice | 3 | `by_user` wins over her `expert` role | | bob | 7 | `expert` role | | carol | 5 | role `novice` has no entry, so `default` | | dave | 5 | no role, so `default` | Nobody got 20. An annotator who reaches their cap is shown the completion page, the same as finishing the corpus. A role named in `user_roles` with no matching `by_user_role` entry is not an error and does not warn -- that annotator silently gets `default`, which is carol above. ## Quality-control items do not consume the quota Attention checks and gold items are injected into the annotator's ordering as they work, rather than drawn from the pool at assignment time. They do not count against `max_annotations_per_user`, so enabling them does not shorten the corpus each annotator sees. On a six-article study with checks every third item, both annotators are served all six articles plus the checks: ``` ordering = [n001, n002, n003, check01, n004, n005, n006] ``` You do not need to pad the quota to make room for them. If you set the quota explicitly, set it to the number of **dataset items** an annotator should see. **Verify with a count across every annotator, not one.** Per-annotator ordering looks plausible in isolation; what matters is whether each item reached the number of annotators you asked for: ```python import json, glob, collections c = collections.Counter() for f in glob.glob("annotation_output/*/user_state.json"): c.update(json.load(open(f))["instance_id_to_label_to_value"]) print(sorted(c.items())) # every real item should hit num_annotators_per_item ``` ## More annotators than work `num_annotators_per_item` times the item count is the ceiling on how much work exists. Once it is used up, the next person to register gets a page of its own: > **There is no work left for you** > > Every item in this study already has as many annotators as it needs, so > nothing could be assigned to you. You have not annotated anything, and nothing > has been recorded against your account. > > If you were asked to take part, tell whoever sent you the link -- they can > raise the number of annotators per item or add more data. It offers a **Check again** button, and says that reloading will give them work if more data arrives, which is true of a dynamic data source. The log carries a matching warning naming the cause and the fix: ``` No items could be assigned to carol: every item already has its full complement of annotators (num_annotators_per_item=1). Showing the no-work page rather than the completion page. Raise num_annotators_per_item or add data to give them work. ``` No `annotation_output/<user>/` directory is created for them. Three items at `num_annotators_per_item: 1` does this to the fourth person through the door, and on a pilot that person is usually you with a fresh login. Before recruiting, multiply: `items x num_annotators_per_item` is the number of annotator-items you have to hand out, and `max_annotations_per_user` divides it into people. ### Holds and annotations An annotator who is *given* an item holds it, and the hold counts against the item's cap so that people spread out across the corpus instead of piling onto its first item. A hold is not a submission, though, and somebody who takes items and closes the tab would otherwise strand them. So when the cap would send an arriving annotator away with nothing, Potato relaxes it and hands out items that another annotator holds but has not submitted, logging: ``` Assigned 4 item(s) to bob that another annotator holds but has not submitted. Enable instance_reclaim to hand those back automatically instead. ``` **This means an over-subscribed study can collect more annotations per item than `num_annotators_per_item`, and nothing warns.** Measured on 4 items at `num_annotators_per_item: 1`, with no per-user cap: the first annotator was given all four and submitted nothing, the second was given the same four, and when both then answered item 1 it ended up with two annotations under a cap of one. If you pay per judgement, that is real money, and an agreement number computed over it is not the design you configured. Size the pool and it does not happen. The same study at 8 items with `max_annotations_per_user: 4` gave the first annotator items 1-4 and the second items 5-8, no overlap, and the relaxation never fired. The rule is the same multiplication as above: keep `annotators x their quota` at or under `items x num_annotators_per_item`. ### Taking abandoned items back `instance_reclaim` is the machinery the log line points at, and it is off by default: ```yaml instance_reclaim: enabled: true timeout_hours: 0.001 # fractional hours are fine; this is 3.6 seconds ``` An annotator who was given items and never submitted loses them after `timeout_hours`, and they go back into the pool. Items they *did* annotate are never reclaimed. Measured with two annotators over 4 items: the second one's arrival reclaimed all four from the first, and the log named each one. Two things to know before relying on it. It runs **when somebody asks for an assignment**, not on a timer -- it is called from the assignment pass, so on a quiet study nothing is reclaimed until the next annotator arrives. And it does not stop the relaxation above: an annotator whose items were reclaimed, arriving back to a study with nothing free, is handed items the *other* annotator now holds, and the log prints its "Enable instance_reclaim" advice even though `instance_reclaim` is enabled. ## Giving cohorts different items and different questions `batch_assignment` pins named annotators to named items, and `scheme_sets` gives each cohort its own questions. They are usually wanted together. ```yaml assignment_strategy: batch # without this, the groups do not pick items annotation_schemes: - annotation_type: radio name: verdict description: Is this acceptable? labels: ["Yes", "No"] - annotation_type: radio name: severity description: How severe? labels: ["Low", "High"] scheme_sets: quick: - verdict detailed: - verdict - severity batch_assignment: groups: - name: cohort_a annotators: [alice, bob] instances: ["1", "2", "3", "4"] schemes: detailed - name: cohort_b annotators: [carol] instances: ["5", "6", "7", "8"] schemes: quick ``` Measured on that config: alice and bob were served items 1-4 with both questions, carol items 5-8 with only `verdict`. **`assignment_strategy: batch` is what makes the groups pick items.** Leave it out and the study still validates, still boots, and each cohort still gets its own questions -- but every annotator is served every item, because the group's item list is only consulted by the batch strategy. The working half is the problem: cohorts visibly getting different questions reads as proof the config took effect. On the same config without that one line, carol was served items 1 through 5. A `schemes` binding is a `scheme_sets` name, a single scheme's `name`, or an inline list; anything else is refused at load. An annotator in no group falls back to the global `annotation_schemes` -- but only for questions. Under `assignment_strategy: batch` they are served **no items at all**, and get the no-work page telling them every item has its full complement of annotators. That is the wrong reason, and the log says the same: on an unlimited `num_annotators_per_item` it reports every item as already having as many annotators as it needs. If you use `batch_assignment` with a fixed roster, anybody who arrives outside it is turned away with a misleading explanation, so list every annotator you expect. The keys accept several spellings. `annotators` or `users`; `instances`, `items` or `instance_ids`; `instances_file`, `items_file`, `instance_ids_file`, `data_file`, `input_data_file` or `input_file` to read the ids from a file instead; `max_annotators`, `annotator_count` or `slots` for a per-group cap. ## Choosing the number of annotators | Situation | Number | |---|---| | Near-mechanical labels, or prototyping, nobody reporting agreement | 1 | | Any judgement involved and an agreement number will be reported | 3 | | Genuinely subjective constructs, or per-annotator reliability is the object of study | 5+ | Three is the smallest number where a majority can break a tie. More annotators on fewer items beats fewer annotators on more items whenever you do not yet know the labels are learnable — find that out on 100 items before spending the budget on 10,000. This is the researcher's decision, not yours. Propose a number with the reason and say what it costs. ## Order `assignment_strategy` takes: `random`, `fixed_order`, `active_learning`, `llm_confidence`, `max_diversity`, `least_annotated`, `category_based`, `diversity_clustering`, `batch`, `priority`, `model_review`, `psychometric`. `random` with a `random_seed` is right unless there is a reason: - `fixed_order` when items are a narrative and order carries meaning - `least_annotated` when finishing every item matters more than annotator experience — the one to reach for when the quota trap above is biting and you cannot raise the cap - `active_learning` when a model is in the loop and the point is to spend annotator time where it changes something. It needs `assignment_strategy: active_learning`. With `active_learning.enabled: true` on its own the classifier is still fitted and its ordering serves nobody; the reorder log line and `ordering_is_served` in `/admin/active-learning/stats` both name which case you are in. `llm_confidence` sits beside it in the list, is not implemented, and assigns at random ## The response format decides the metric Potato classifies each scheme and computes what fits. You do not choose the metric separately, which makes this a measurement decision rather than a cosmetic one. | Kind | Schemes | What it computes | |---|---|---| | nominal | `radio`, `select`, `triage`, `conjoint`, `multiselect` capped at 1 | pairwise Cohen's κ, Fleiss' κ, nominal α | | ordinal | `likert`, `confidence`, `vas`, `range_slider`, `semantic_differential` | linear and quadratic weighted κ, Spearman ρ, ordinal α | | continuous | `slider`, `number` | Pearson r, MAE, RMSE, interval α, ICC(2,k) | | multilabel | `multiselect`, `hierarchical_multiselect`, `card_sort` | mean Jaccard, MASI α | | matrix | `multirate`, `constant_sum`, `soft_label`, `bws` | the ordinal or continuous measures per row, plus a pooled block over every (item, row) pair | | ranking | `ranking` | Kendall's τ, Spearman footrule | | span | `span`, `error_span`, `coreference`, `span_link`, `event_annotation`, `extractive_qa`, `tree_annotation` | token-level κ, exact and partial span F1, Krippendorff's αU, γ | | geometry | boxes, polygons, points | matched IoU, detection F1, chance-corrected σ, ks, detection α | | | | *(the only way to score a geometry task against a known answer — `gold_label` cannot hold a box)* | | free text | `text`, `text_edit` | nothing | `pairwise` does not follow its type name. It renders three widgets and stores three shapes, so `mode: scale` is scored continuous, `mode: multi_dimension` as a matrix, and the binary default as nominal. Read the kind out of the report. **`mean_matched_iou` averages over matched pairs, not over items.** A region one annotator drew and the other did not is not a zero in it; it is absent from it. Six matched pairs on a study where twelve of twenty regions never matched still reported 0.674, the mean of those six. That number answers "when both of them drew the same thing, how well did the boxes line up" and nothing else. What the disagreement about *whether* a thing is there costs you is in `detection_f1` and `mean_object_count_diff`; quote all three or none. **Region-caption agreement anchors on one annotator, so it is asymmetric.** The first annotator by sorted id defines the units, and everyone else is matched onto them at `match_iou`. Regions only the *second* annotator drew are never units and their captions are never counted, so `n_captions` is the anchor's captions plus one per match, not the number of captions in the data. Twelve anchor regions with six matches gave `n_captions: 18` over twenty stored captions. Swapping who annotates first changes the units, and the module says as much rather than hiding it. `agreement_distance` defaults to `token`, which scores a correct paraphrase as complete disagreement; `embedding` is the one that does not, and it falls back to `token` with a note in the report if `sentence-transformers` is missing. Consequences worth stating to a researcher up front: - **A 1–5 rating stored as a `radio` is scored as unordered**, so "1 vs 2" counts the same as "1 vs 5" and the agreement number is badly understated. Store ordered judgements as `likert`. - **Span agreement tolerates disagreement about boundaries.** Annotators asked to mark sentences will not agree on where a sentence starts; token-level and partial-overlap measures handle that, whereas splitting the text into sentence rows and computing per-sentence κ requires the units to match exactly. - **Free text has no metric.** If a judgement is going to be counted, make it a scale. If it is genuinely prose, plan to read it and say so. Enable the admin-side computation with: ```yaml num_annotators_per_item: 3 agreement_metrics: enabled: true ``` `agreement_metrics` recognizes `enabled` and nothing else — `metrics`, `measures`, `types`, `min_annotators` and `by_scheme` are all rejected by `--strict`. The measures follow from the schema kinds; there is nothing to select. ### Reading the agreement report Nothing is computed at boot, and nothing is linked from the annotation page. **Agreement appears once an item reaches its full annotator cap**, not once it has two annotators. An item sitting at 2 of `num_annotators_per_item: 3` is not scored. Measured on a six-item study: two annotators finished and every coefficient was null, the third finished and the whole report populated. The report tells you which of the two it is. Alongside `n_overlap_items` it returns `n_items_below_cap`, and an `items` block giving each item its cap, the annotators it has, and how many answered each scheme: ```json {"n_overlap_items": 6, "n_items_below_cap": 0, "items": {"p01": {"annotators": ["r1@x.com", "r2@x.com", "r3@x.com"], "cap": 3, "schemas": {"decision": {"n_annotators": 3}, "topics": {"n_annotators": 2}}}}} ``` Read those two before the coefficients. A study waiting on its third annotator and a study nobody has touched both produce nulls, and `n_items_below_cap` is what separates them. The per-item `schemas` counts also show a scheme an annotator skipped, which is the usual reason one scheme is null while the rest of the report is fine. ```bash cat admin_api_key.txt # written into task_dir on the first admin request curl -H "X-API-Key: $(cat admin_api_key.txt)" localhost:8000/admin/iaa ``` The file does not exist at boot. It appears the first time anything asks for an admin route, so the honest sequence is to make one unauthenticated request, let it be refused, and then read the key the refusal caused to be written. `/admin/iaa` returns, per scheme, the `kind` it inferred and the metrics that follow from it, plus `n_overlap_items`. Confirming the `kind` matters more than the number: it is where you find out a rating you meant as ordinal is being scored as nominal. Driven over two studies, eight items each, the kinds came back as: ``` sentiment radio nominal alpha_nominal, fleiss_kappa, pairwise_cohen_kappa severity likert ordinal alpha_ordinal, spearman_rho, weighted_kappa_linear, weighted_kappa_quadratic confidence confidence ordinal the same four issues multiselect multilabel alpha_masi, mean_jaccard rating slider continuous alpha_interval, icc_2_k, mae, pearson_r, rmse reasons constant_sum matrix one block per row, plus `pooled` and `n_rows` design soft_label matrix the same evidence span span token_level_kappa, span_f1_exact, span_f1_partial, krippendorff_alpha_u, gamma_mathet boxes image_annotation geometry detection_f1, mean_agreement, mean_matched_iou, mean_object_count_diff, detection_alpha ``` A `matrix` schema is scored twice over: once per sub-question, and once pooled over every (item, row) pair. Read the per-row block. A rating grid usually has one row everybody agrees on and one that carries the disagreement, and pooling averages the two together. On a three-row `constant_sum` over six items and three annotators, per-row interval α ran 0.86 to 1.0 against a pooled 0.94. Two things to read off that. **The `kind` follows the declared `annotation_type`, not the widget that rendered.** A `likert` given an explicit `labels:` list renders as a radio group — the boot log says "Complex labels detected, using radio layout" — and is still scored ordinal, which is what you wanted. And **a free-text scheme is absent from the report entirely** rather than present with empty metrics, so do not read a missing scheme as a failure. `n_items` is per scheme and counts only items with more than one annotator, so it is normally below `n_overlap_items`. A scheme that reports `n_annotators: 0` and all-null metrics has no item two people both answered — for spans that is easy to hit by accident, because one annotator forgetting to select the label leaves nothing to compare. Without the header, every admin JSON route answers `403 {"error":"Admin access required"}` — which reads like a broken config if you do not know the key exists. `/admin` itself serves HTML with no key. An empty report on a fresh task means "no overlapping annotations yet". ## Norming a codebook in a live room `rooms` is a live multiplayer session at `/rooms`. It belongs in this file rather than in a feature list because its output is annotations and its headline number is Krippendorff's alpha, computed twice. ```yaml rooms: enabled: true schema: sarcasm # the radio or likert scheme members vote on persist_votes: true who_can_create: any ``` Members join, vote blind on the current item, the host reveals, everyone argues, anyone may change their vote, the host advances. Three kinds of room: | `room_type` | Items come from | Who votes | |---|---|---| | `norming` | the `item_ids` the creator passes, or a random sample | every active member | | `huddle` | items the team already disagrees on, found live | every active member | | `shadow` | as for norming | the host; everyone else joins as an observer | Only the host reveals, advances and closes — a member who asks gets `Only the host can do that` — and observers cannot vote. The blind phase is genuinely blind on the room page: before the reveal, a member's poll of the room state and of the event stream carries who has voted and not what they voted. **Set `rooms.schema` rather than leaving it to the autopick.** With the key absent, the vote target is the first `radio` or `likert` scheme in the config, which moves when you reorder the file. `potato validate` covers both ways of getting it wrong, and `--strict` fails on either: nothing to vote on at all warns that "rooms will be disabled and /rooms will return 404", and a `rooms.schema` naming a scheme that does not exist warns that "rooms will refuse to open" and lists the names it did find. That second one is the dangerous shape, because a misspelling boots looking healthy and only fails when somebody tries to open a room. ### What a room writes into each member's annotations With `persist_votes: true` (the default), each member's vote is written into their own annotations for the room's schema, at the moment **the host advances past a revealed item, or closes the room on one**. An item nobody revealed is marked skipped and writes nothing, so a room abandoned mid-discussion leaves that item's votes only in the log. Two properties to say out loud to a researcher before they run one: - **What lands is the vote after the discussion, not the blind one.** A member who voted `Sincere`, watched the room go the other way and changed to `Sarcastic` has `Sarcastic` in her annotations. Her blind vote survives only in `<output_annotation_dir>/rooms/room-<id>.jsonl`. A study that needs independent judgements cannot source them from a room. - **The write replaces that member's existing answer for the room's schema, and touches no other schema.** Measured on a config carrying `sarcasm` and a `text` scheme called `notes`: the room overwrote both members' `sarcasm` answer for the item and left both `notes` entries as they were. ### The two alphas The room state carries a `metrics` block that computes nominal Krippendorff's alpha over revealed items twice, on the blind votes and on the current ones. The gap between them is the room's norming lift. Three annotators, two revealed items, one of whom changed her vote after both reveals: ``` blind_alpha -0.111 final_alpha 1.000 alpha_lift 1.111 total_changes 2 toward_majority 2 ``` `toward_majority` counts a post-reveal change that landed on the majority of *everyone else*: the changer's own vote is excluded from the count, so a two-person room can record conformity instead of sitting at a permanent tie. Both alphas are `null` until two items have been revealed. A room one item in reports `n_revealed: 1` and three nulls, which is undefined rather than broken. **In a huddle those two numbers do not mean what they mean in a norming room.** A huddle seeds itself from the team's live disagreements and shows every member's original annotation, by name, as context, before anyone in the room has voted. Measured: a member's state during the voting phase, no votes cast, already carried ``` seed_votes {"frank@example.com": "Sincere", "gina@example.com": "Sarcastic"} ``` That is what a huddle is for, and it is not blind, so `blind_alpha` there is agreement among people who are looking at each other's answers. Quote the lift out of a norming room. Out of a huddle, quote `final_alpha` and the conformity counts and say where they came from. **A room raises the study's own agreement score, and the report cannot see why.** A room-written vote registers its annotator against the item like any other annotation, so a room can carry an item to its cap and put it into the agreement report above. Measured: two annotators who voted `Sarcastic` and `Sincere` blind, argued, and converged left `/admin/iaa` reporting `percent_agreement: 1.0` for that item, with nothing anywhere in the report to say they had been in a room together. Kappa and alpha came back `null`, with a note that every annotator gave the same label, because that one item was all the report had. Norm on items you are willing to spend, and keep them out of the set you report agreement on. ## Related - `quality-control.md` — the checks that consume the quota - `designing-a-task.md` — the seven decisions, including how many annotators - `running-a-task.md` — reading `instance_id_ordering` to see what was assigned <!-- https://davidjurgens.github.io/potato-skill/modes-and-subsystems/ --> # Modes and Subsystems The whole-task modes a brief can be asking for without naming. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- A task is a config plus a data file, and about a quarter of what Potato can do fits that description. The rest is whole modes, surfaces and background systems that a researcher describes without naming — and the expensive mistake is not an invented type name, which validation catches, but hand-assembling something that already exists, which validates cleanly, renders fine and is worse than the real thing. This is the routing table. Match the brief, then read the block's sub-keys in `config-keys-nested.md` and `get_key_doc("<block>.<key>")` for the detail. ## The seven clusters Every capability decision belongs to one of seven, and each is owned by one file. Use this two ways. Reading a brief, walk the clusters and ask which ones it touches. The ones it never mentions are where a researcher is most likely to be missing something they would want. Running the interview in `interviewing.md`, these are the menu, offered as symptoms rather than by name. | Cluster | The question behind it | Owned by | |---|---|---| | **Material** | What is in front of the annotator? | `modalities.md`, `agent-traces.md`; **What they brought** below | | **The answer** | What kind of answer, and how precise? | `designing-a-task.md` — see below | | **Unit** | What counts as one thing? | `designing-a-task.md`; `sessions` and `cases` below | | **People** | Who annotates, and who sees which items? | `assignment-and-agreement.md`; **Different modes of working** below | | **Trust** | How will you know the answers are good? | `quality-control.md`, `assignment-and-agreement.md`; **Measurement and quality** below | | **Model help** | Should a model be involved at all? | **Models in the loop** below | | **Output** | What comes out, and where does it run? | `getting-the-data-out.md`, `deploying.md`; **Data in and out** below | ### The answer itself This cluster is the one with no rows here, because `designing-a-task.md` owns the response-format tables and duplicating them is how they drift. Go there — but know what is in them, because this is the cluster researchers most reliably miss. They ask for `radio` and `likert` and stop, and Potato has a whole family for answers that a scale cannot hold: | They said | Reach for | |---|---| | "it isn't one label — it's mostly A, a bit B" | `soft_label`, rather than throwing the ambiguity away | | "an acceptable range, not a point" | `range_slider` | | "warm/cold, weak/strong, active/passive" | `semantic_differential` | | "the parts have to add up" | `constant_sum` | | "how sure were they" | `confidence`, with `target_schema` naming the judgment it qualifies | | "which of these two is better" | `pairwise`, `bws`, `ranking`, `conjoint` | ## Routing from what the researcher said ### Different modes of working | They said | Reach for | Notes | |---|---|---| | "codes I'll invent as I read", "notes on passages", "in the participant's own words" | `qda_mode` | Moves memos, codebook and case grouping together. Do **not** build it out of a `multiselect` you plan to keep editing | | "a shared codebook", "the label set will change" | `codebook` + scheme-level `codebook: true` | Edited at `/codebook`. The first boot seeds it from the config; later config edits need `potato codebook config.yaml` | | "a participant", "an interview", "count codes per person not per excerpt" | `cases` | The unit of analysis becomes the case | | "it's just me", "I want to find my own inconsistencies" | `solo_mode` | LLM labels, you review where it is least sure, prompt is refined from your corrections | | "several people discussing an item together", "a norming session" | `rooms` | Live multiplayer at `/rooms`. What it writes back is the post-discussion vote — `assignment-and-agreement.md` | | "one final label per item, not a distribution" | `adjudication` | Needs `adjudicator_users` | | "a reviewer signs off", "a queue with states" | `review_workflow` | Kanban at `/admin/review` | | "go as fast as possible", "just keyboard" | `review_mode` | Advances by itself once the item is complete | | "score the whole conversation, not each turn" | `sessions` + `session_level: true` on the scheme | Warns at validate if no scheme opts in | | "on their phone" | `pocket` | PWA at `/pocket` with an offline queue. Seven answer types work there and one other scheme takes the task off the phone — `designing-a-task.md` | | "annotators should pick their own items" | `search.annotator_claim` | Admin search is always on; claiming is the opt-in | ### What they brought Route on the medium before the question — the display type, the scheme type and the key each one reads are all decided by it. Full tables in `modalities.md`; agent traces are their own family, in `agent-traces.md`. | They have | Display + scheme | The trap | |---|---|---| | Images to mark up | `image` display + `image_annotation` | Needs `tools` **and** `labels`; the canvas takes its bitmap from the display field, so removing it gives an empty canvas | | Very large images | `image` + `viewer`/`tiles` | The viewer owns the transform; never recompute it | | Video, mark moments or track objects | `video_annotation` (no display field needed) | Five modes and `labels` is required in all of them | | Video, find the interval for a phrase | `temporal_grounding` | Reads `video_key`/`events_key`, not `source_field` | | Audio regions | `audio_annotation` | `mode: label` for one label per region; `questions` or `both` to ask whole schemes per region, which then need `segment_schemes` | | ASR/TTS against a reference | `speech_transcript` | Its own key names throughout | | Tiers over audio or video, ELAN-style | `tiered_annotation` | `media_type` defaults to `audio` | | A podcast or interview | `audio_dialogue` display | Per-turn playback is built in; it is a span target | | A chat log | `dialogue` display + `turn_level` schemes | `conversation_tree` is **not** a span target | | Point clouds or LiDAR | `spatial_annotation` | `lod` defaults on, and `max_points` only applies when it is off | | Robot episodes | `episode_annotation` | Four layers, all on by default | | Generated video from a world model | `rollout_evaluation` | Break-points, not a plausibility score | | Agent traces of any kind | `agent-traces.md` | Nearly nothing is a required field, so a wrong `steps_key` renders an empty list | ### Models in the loop | They said | Reach for | |---|---| | "suggest a label", "pre-fill and let them correct" | `ai_support` — see `model-assistance.md`, which is where the keys actually live | | "label everything with a model first", "we can only afford to label a sample", "how good are the model's labels" | `labeling_engine` with `assignment_strategy: engine`, which tests the machine labels against a blind human sample — see `labeling-engine.md`. `pre_annotation` and `icl_labeling` still work and print a note pointing at the engine | | "use the labels we already have as examples" | `icl_labeling` | | "show me where the model is unsure" | `active_learning`, `llm_confidence` assignment | | "does the judge agree with our humans" | `judge_alignment`, `judge_calibration` | | "compare two models' answers" | `arena`, or a `pairwise` / `bws` scheme | | "let them ask a model while annotating" | `chat_support` — a sidebar, not per-scheme buttons; `model-assistance.md` | | "judge the agent while it runs" | `live_agent`, `live_coding_agent`, `agent_proxy` | | "we have traces coming out of production" | `trace_ingestion` (`/api/traces/*`), `automation`, `triage` | ### Measurement and quality | They said | Reach for | |---|---| | "how much do they agree" | `num_annotators_per_item` + `agreement_metrics` — see `assignment-and-agreement.md` | | "who is reliable", "some annotators are better" | `mace` (competence from disagreement alone), `psychometrics` (live IRT) | | "catch people clicking through" | `attention_checks`, `gold_standards` — see `quality-control.md` | | "I planted known problems, did they find them" **on a geometry or span task** | Not supported. `gold_label` holds a label, never a box or an offset. Score it with geometry agreement between annotators instead, and say so | | "the majority is sometimes wrong" | `truth_serum` (surprisingly-popular scoring) | | "what changes their mind", "contrast sets" | `boundary_probing` | | "how did they produce that text", "did they paste it" | `keystroke_logging` | | "how did they draw that", "are they rubber-stamping the AI" | `annotation_telemetry` | | "have them talk through it" | `thinkaloud` | ### Data in and out | They said | Reach for | |---|---| | "the data is in a spreadsheet / S3 / HuggingFace / a database" | `data_sources`, `data_directory` (+ `watch_data_directory`) | | "new data keeps arriving" | `data_sources` polling, `watch_data_directory`, `automation` | | "it's a big corpus" | `partial_loading`, `item_store: {backend: paged}` | | "show me the corpus shape" | `corpus_map`, `embedding_visualization`, `diversity_ordering` | | "find the ones like this", "a group that keeps updating", "what is even in here" | `curation` — the catalog at `/admin/catalog`, below | | "publish the dataset" | `publish`, `dataset_metadata` | | "pay people on Prolific / MTurk" | `crowdsourcing` (+ `completion_code`, `auto_redirect_on_completion`) | | "different groups see different items" | `batch_assignment` + `scheme_sets`, or `category_assignment` | | "people abandon assignments" | `instance_reclaim` | | "let annotators see their own progress" | `annotator_dashboard` | ### The `when` grammar `triage.rules`, `automation.rules` and `review_workflow.routing` all filter with the same `when` condition, evaluated against the item's data dict: ```yaml triage: enabled: true rules: - when: {field: status, equals: error} priority: 100 badge: "agent errored" - when: {field: metadata.score, lt: 0.5} priority: 60 signal_field: judge_score # read when no rule matched invert_signal: true # lower is worse ``` One condition takes `field` plus exactly one operator: `equals`, `in`, `contains`, `exists`, `lt`, `lte`, `gt`, `gte`. Field paths may be dotted (`metadata.score`). String comparison for `equals` and `in` is case-insensitive; `contains` tests membership in a list field or a substring of a string one; a field the item does not carry never matches anything but `exists: false`. `when` also accepts a list of conditions, all of which must hold. `automation` rules add `sample_rate` (0–1, hashed on item id so the decision is stable) and `actions`, which are dicts rather than names: ```yaml automation: enabled: true rules: - name: "Route errors to review" when: {field: status, equals: error} sample_rate: 1.0 actions: - {type: add_to_queue, priority: 200, reason: "agent errored"} - {type: add_to_dataset, dataset: errors-to-fix} - {type: notify, message: "New error trace"} ``` The seven action types are `add_to_queue`, `add_to_dataset`, `notify`, `enroll_review`, `run_evaluator`, `fire_webhook` and `refresh_topics`; the last three run on a background worker. `add_to_dataset` needs `datasets` enabled. `/admin/automation/status` reports `items_processed`, `rules_fired` and a count per action, which is how you find out whether a rule ever matched. ## What "enabling" a block costs Most blocks activate on `enabled: true` and nothing else. Two in this list refuse to start without more: ``` ai_support.endpoint_type is required when ai_support is enabled adjudication.adjudicator_users must be a non-empty list of usernames ``` One warns rather than failing, which is the pattern to expect elsewhere: ``` sessions.enabled is true but no annotation scheme sets session_level: true — the /sessions page will have no questions. ``` Verified by enabling 28 blocks at once and reading what the validator said. The rest accepted a bare `enabled: true`, which means only that the config loads — **not** that the feature has anything to work with. Boot the server and look for the subsystem's own initialization line before believing it is live. ## Pages each block adds Turning a block on usually adds a page. None of them are linked from the annotation page, so tell the researcher they exist: | Path | From | |---|---| | `/progress` | `annotator_dashboard` | | `/codebook` | `codebook`, `qda_mode` | | `/sessions` | `sessions` | | `/rooms` | `rooms` | | `/corpus` | `corpus_map` | | `/admin/catalog` | `curation` | | `/datasets/admin` | `datasets` (`/datasets` itself is not a route) | | `/pocket` | `pocket` (phones auto-route here) | | `/adjudicate` | `adjudication` | | `/admin/review` | `review_workflow` | | `/admin/arena` | `arena` | | `/psychometrics/dashboard` | `psychometrics` | | `/truth_serum/dashboard` | `truth_serum` | | a MACE tab inside `/admin` | `mace` (`/admin/api/mace/overview` for the JSON) | | `/admin/automation` | `automation`; `/status` and `/outcomes` beside it return JSON | | `/judge_calibration/admin` | `judge_calibration`; `/status`, `/progress`, `/run`, `/report` beside it | | `/api/mcp/manifest` | `mcp`, once an agent token exists | | `/admin/...` | agreement, exports, progress — always present | **`/admin` is the only admin page a browser can open cold.** It presents a key box; every other `/admin/*` page and both dashboards answer `403 {"error":"Admin authentication required"}` until the key has been entered there once, so a deep link handed to a colleague looks broken. The dashboards also sit at their own prefixes rather than under `/admin`: `/admin/psychometrics` is a 404, `/psychometrics/dashboard` is the page. ## Scheme-level keys These are keys on an `annotation_schemes` entry, not top-level blocks, and nothing validates them (see `config-keys-nested.md`): | Key | Effect | |---|---| | `display_logic` | Show a scheme only when another scheme has a given answer | | `label_requirement` / `required` | Enforce an answer | | `session_level` | One answer per session rather than per item | | `turn_level`, `turn_binding`, `turn_label` | Bind a scheme to conversation turns | | `codebook: true` | This scheme's labels come from the shared project codebook | | `target_field` | Which `instance_display` field a span scheme anchors to | | `has_free_response` | Adds an "other, please specify" input to a radio/multiselect | | `option_randomization` | Randomize option order per annotator | | `sequential_key_binding` | Number the labels 1,2,3… instead of naming keys | | `dynamic_options`, `dynamic_options_field` | Options come from the item, not the config | ### What `codebook: true` carries over The first boot seeds the codebook from the labels of **every** scheme that sets `codebook: true`, and how you wrote those labels decides what the codebook contains. Bare strings give it names and nothing else, so the tray shows colourless codes reading "No content yet" while the annotation chips still look right. The form looks configured, and everything the researcher wrote about each code is on the floor. The dict form carries three things: ```yaml annotation_schemes: - annotation_type: radio name: stance description: What stance does it take? codebook: true labels: - name: Supportive color: "#2e7d32" description: The text endorses the proposal, including qualified endorsement. - name: Opposed color: "#c62828" tooltip: The text argues against the proposal on any grounds. - name: Neutral color: "#616161" ``` `name` becomes the code, `color` its colour in the tray and on the chips, and `description` its definition. `tooltip` is accepted as the same thing, so a config already using it for hover text needs no change. A label with neither gets a code with no definition rather than an invented one -- `Neutral` above. Seeding is a first-run bootstrap gated on the codebook being empty, so a label added to the config later does not appear: run `potato codebook config.yaml`. That gate is per codebook rather than per scheme, and it is applied after every codebook-backed scheme has contributed, so a label declared only on a second scheme is seeded too. ## Deciding what to annotate: the catalog `curation` builds an embedding index over the loaded items and serves `/admin/catalog`. It is the answer to "which of these thousands should we label", which is a question the annotation config itself cannot ask. ```yaml curation: enabled: true model_name: all-MiniLM-L6-v2 # the default embed_on_ingest: true # off by default; costs boot time and memory ``` Everything below is an admin JSON route under `/admin/catalog/api/`, so it needs the `X-API-Key` header. **The catalog routes reject an unauthenticated request without generating the key file**, unlike `/admin/iaa`, so make the first unauthenticated request somewhere else and read `admin_api_key.txt` after that. Build the index once with `POST api/build`, which answers `{"indexed": 12}`. Then, on twelve short complaint texts: - **`POST api/search`** takes a `query` and returns instance ids by cosine similarity. "my train never showed up and nobody told us anything" put the two train items at 0.64 and 0.59 and everything else below 0.22. **The parameter is `top_k`, not `k`** — `k` is accepted and ignored, so you get the default ten results back, which looks like the model failing to discriminate. - **`POST api/slices`** saves a named query as a dynamic slice, and `GET api/slices/<name>/resolve` returns what currently matches. A slice named `transport-complaints` over "delayed or cancelled travel" at `threshold: 0.2` resolved to six ids. Pick the threshold by resolving and reading the list; 0.2 is loose enough to pull in neighbouring complaints. - **`POST api/topics/refresh`** clusters the index, and `GET api/topics/<name>/members` lists each cluster. On twelve items written as six near-duplicate pairs it recovered all six pairs exactly. Clusters are named `topic-1`, `topic-2` and so on with empty descriptions unless something supplies a label, so read the members rather than the name. - **`GET api/duplicates`** is **perceptual-hash duplicate detection over images**, not text. On a text-only project it returns zeros and says so in a `note` field: "No item could be perceptually hashed … This is NOT a finding of zero duplicates." Read the note before you quote the number. For text overlap, the topic clusters above are what you have. Slices and topics both have a `to_dataset` route, so a slice can become the thing you actually hand to annotators. ## When you cannot find the feature The routing above covers what the key documentation names. For anything else: ```python from potato.server_utils.examples_manifest import search_examples search_examples(query="qualitative") search_examples(config_key="gold_standards") search_examples(annotation_type="span", display_type="image") ``` Each entry records the annotation and display types a config uses, the features it switches on, and the command that runs it. Copying a config that is checked in CI beats assembling one from field lists — and if the examples are unavailable to you, the discovery method in `SKILL.md` is the fallback. <!-- https://davidjurgens.github.io/potato-skill/model-assistance/ --> # Model Assistance From a Self-Hosted Endpoint Pointing `ai_support` at your own vLLM or other OpenAI-compatible server. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- `ai_support` puts a row of assistant buttons beside each annotation scheme — Hint, Keyword, Rationale — and fills them from an LLM. This file covers pointing that at a server you run yourself: vLLM, SGLang, LM Studio, llama.cpp, anything speaking the OpenAI chat API. Commercial providers use the same keys and are not covered here, because I have not run one. The three endpoint types I drove against a live server all work. What is left to get wrong is the config: the keys sit one level deeper than they read, and the switch that decides whether any button appears is off by default and warns about nothing. ## Where the keys go Three levels, and only the first is guessable: ```yaml ai_support: enabled: true endpoint_type: openai_vision ai_config: base_url: "http://your-server:8001/v1" model: "google/gemma-4-12B-it-qat-w4a16-ct" temperature: 0.1 include: all: true ``` `endpoint_type` sits beside `ai_config`. Everything else — `base_url`, `model`, `api_key`, `max_tokens`, `temperature` — goes *inside* it. Written one level up, where it reads naturally, the keys are dropped and the endpoint falls back to its own default host. `validate --strict` catches that one: ``` Unrecognized config key 'ai_support.base_url'. This key will be ignored. ``` **`include.all` is off by default and nothing warns you.** Without it the config validates clean — `OK — no issues found` — the endpoint connects, the boot log says the endpoint is ready, and the page renders `<div class="ai-help none">` with no buttons in it. This is the single most likely reason a correctly configured `ai_support` shows nothing. `ai_config.include.special_include` is the per-page, per-scheme alternative, keyed page number then annotation id. `config-keys-nested.md` documents these, `include.all` included. Setting `base_url` makes `api_key` optional, on the validator as well as in the endpoint, so a local server needs no placeholder key. **One `ai_support` block serves every scheme, and each gets its own buttons and its own answer.** Three schemes on a page — a radio, a multiselect and a free text — gave three separate Hint/Keyword/Rationale rows, and each hint was about that scheme's question: the radio's discussed weighing the complaint against the resolution, the multiselect's walked its own five labels and said which phrases mapped to which, the text one described the summary it wanted. The suggestions landed on the right scheme too, highlighted with a sparkle — `Mixed` on the radio, and `Shipping`, `Build Quality` and `Support` but not `Battery` or `Price` on the multiselect. There is nothing per-scheme to configure. **The model's output is escaped before it reaches the page**, which matters because a hint is generated *from* the item: markup in your corpus comes back through the model. I put `<b>` and `<img>` in an item's text and the hint panel rendered neither — 607 characters of hint, no elements the renderer did not create. The same holds for the rationale panel. Markdown in the output is rendered rather than shown as asterisks; I have driven the escaping half against a live model and taken the markdown half from the fix, because the model did not happen to emit any on the runs I made. One rough edge left: on a narrow scheme the three buttons lose their labels and render as bare icons, so the same row reads differently depending on how wide its scheme box is. ### Keeping the endpoint out of the config `ai_support.ai_config_file` names a second YAML file, merged in at load. It is how the bundled examples keep API keys and lab-internal hostnames out of the committed config, and it is worth using for the same reason. The file is **flat**. Its keys go straight into `ai_config`, and `endpoint_type` is lifted out to sit beside it: ``` endpoint_type: openai_vision base_url: "http://your-server:8001/v1" model: "google/gemma-4-12B-it-qat-w4a16-ct" ``` Give that file its own `ai_config:` block and you get `ai_config.ai_config`, every real setting is invisible, and the boot log complains about a missing API key, which points at the wrong key entirely. If the named file does not exist, Potato warns and sets `enabled: false`. `validate --strict` fails on that warning, so the check catches a study whose whole point is model assistance running with the model switched off. Ship an `ai-config.yaml.example` beside the config for whoever clones it, as `examples/image/image-vllm-rationale/` does. ## Which endpoint_type to use `endpoint_type: vllm` for a vLLM server, `openai` for anything else OpenAI-compatible, `openai_vision` if items include images. Driven against a live vLLM 0.24.0 serving a 12B text+vision model, clicking every assistant in a browser and reading the response body: | `endpoint_type` | Text | Images | Constrains the output with | |---|---|---|---| | `vllm` | yes | no | `response_format: json_schema` | | `openai` | yes | no | `response_format: json_schema` | | `openai_vision` | yes | yes | `json_schema`, falling back to `json_object` then nothing if the server refuses | All three returned correctly shaped JSON for all three assistants, and `suggestive_choice` highlighted the label in the form. `openai_vision` is named for its capability, not its backend: its docstring is explicit that `base_url` is there for vLLM, SGLang, LM Studio, llama.cpp and LiteLLM, and it appends `/v1` if you leave it off. Pick it whenever any item is an image, because it is the only one of the three that sends one. On 2.11.1 the validator accepts `openai`, `openai_vision`, `anthropic`, `anthropic_vision`, `gemini`, `huggingface`, `ollama`, `ollama_vision`, `openrouter` (also spelt `open_router`; 2.11.0 refused both), `vllm`, and `yolo` for detection. Anything else fails with `ai_support.endpoint_type must be one of:` and that list. `sam` and `sam3` have endpoint classes in Potato's code but are refused here. I have run three of them. ## max_tokens Defaults to 800, which covers the multi-label formats. Rationale and Keyword both produce one entry per label, so a three-label scheme runs 300-500 tokens; the old default of 100 cut them off mid-object. Lower it, or give a scheme many labels, and from Potato 2.9.0 the log says so: ``` The model hit max_tokens (100) before finishing this response, so the reply is cut off and may parse into the wrong shape. Raise ai_config.max_tokens. ``` Worth knowing because truncation and a genuinely broken endpoint share a symptom: the salvage step returns whatever key-value pairs completed, so the renderer gets a plausible dict of the wrong type and the tooltip reads "No rationales available" either way. **From 2.9.0, all ten text and vision endpoints warn for the assistants**, on text and image items alike. Their chat methods warn too, with `this chat reply` where an assistant's warning says `this response`: on 2.9.0 only `anthropic`, `ollama`, `openai` and `vllm`, and on all ten from 2.9.1. Before 2.9.0 only `openai`, `openai_vision` and `vllm` warned, and never from chat. On `anthropic`, `anthropic_vision`, `gemini`, `huggingface`, `ollama`, `ollama_vision` and `openrouter` a cut-off reply reached the renderer with nothing in the log and produced the empty tooltip above, which is also what a broken endpoint produces. A quiet log is evidence the reply was complete only on 2.9.0 or later, and for chat on 2.9.0 only on `anthropic`, `ollama`, `openai` or `vllm`. ## Which field the model gets Two keys decide, and only one of them is the obvious one. `item_properties.text_key` names the field that goes to the model on the text path. It is the same key that decides plain-text rendering, span anchoring, training and search (`data-and-access.md`), so it is not free to point somewhere else just to change what the assistants read. `ai_support.image_key` names the field holding the picture, for an item that carries both. `item_properties.image_key` is the same key one level up, and `ai_support.image_key` wins: ```yaml item_properties: id_key: id text_key: body # the ticket text ai_support: enabled: true endpoint_type: openai_vision image_key: screenshot # the field holding the image URL ``` With both keys set the model receives both: the picture, and the text field named by `text_key`, in the same request. Without `image_key` a vision endpoint on a text-keyed item never receives an image, and nothing says so — the endpoint connects, the buttons render and the model answers about the half it was given. **Read the response body to find out what it was given.** The tooltip cannot tell you, and a model answering about the wrong half of the item is fluent about it — the hint reads like a hint, and the label it suggests is highlighted in the form either way. One `/api/get_ai_suggestion` call on one multimodal item settles which field reached the model; nothing in the config does. ## Which images the model can reach The AI path loads the image itself, server-side. From 2.8.2-29 a bare filename, or a `media/...` or `/media/...` path, under `media_directory` is read off disk and sent as base64. Anything else has to be an http(s) URL, and one resolving to a private, loopback or link-local address is refused. That refusal is SSRF protection on a URL that comes from your data file, and it means a full URL to your own Potato instance (`http://localhost:8000/media/cat.png`) never reaches the model, even though the same file written as `/media/cat.png` does. A `data:` URI is refused too: the page displays one (`modalities.md`), and the model never gets it. Annotators see the image in every one of these cases, so checking the page will not catch it. Write media paths in the data as `/media/<path>`, the form the page serves, and the model gets the same file. When the image cannot be loaded, the request goes to the model as text only, and the answer comes back in the same shape as an answer about the picture. From 2.9.0 the log says `so the model is being sent TEXT ONLY`, and by the source the `/api/get_ai_suggestion` response carries `image_attached: false`. Before 2.9.0 nothing said so. A separate detail: whether an item counts as an image at all is decided by sniffing the text for an extension (`.jpg`, `.png`, `.gif`, `.webp`, `.bmp`), a known image host, or the word "image" in an http URL. An extensionless URL from your own object store is read as text and the vision path never runs. ## The three text assistants Three, for text schemes, each with its own expected output shape: | Button | Produces | Where it lands | |---|---|---| | Hint | Guidance that avoids naming the answer | A tooltip, plus `suggestive_choice` highlighting that label in the form | | Keyword | Per-label keywords | Overlays on the item text, labelled — not the tooltip, which stays empty | | Rationale | One argument per label, for and against | A tooltip | Which appear is filtered per item by what the endpoint declares it can do, so an image item correctly drops Keyword while a text item keeps it, on the vision endpoints too. The prompts are per annotation type and live in `potato/ai/prompt/*.json`, one file per type — `radio`, `likert`, `multiselect`, `select`, `slider`, `number`, `span`, `text`, plus the image and video ones. A scheme whose type has no file there gets no assistants and the tooltip says the annotation type does not exist in the prompts. `ai_support.ai_config.model_module` and `annotation_path` override the schemas and the prompt files respectively, if the defaults do not fit the study. ## Letting them ask the model instead `chat_support` is a separate block with the same endpoint shape, and it puts a sidebar beside the form rather than buttons beside each scheme. Reach for it when the researcher says "let them ask a model while annotating" — the assistants answer one question about one scheme, the chat answers whatever the annotator types. ```yaml chat_support: enabled: true endpoint_type: openai ai_config: base_url: "http://your-server:8001/v1" model: "google/gemma-4-12B-it-qat-w4a16-ct" temperature: 0.3 max_tokens: 2000 ui: title: Ask the model sidebar_width: 420 max_history_per_instance: 50 ``` `endpoint_type` and `ai_config` sit at the same level as they do under `ai_support`, and `include.all` is not involved: the sidebar renders whenever the block is enabled. The boot log's line is `Chat endpoint initialized: <endpoint_type>`. **The sample sets `max_tokens` well above the 800 default, and should.** An assistant returns one entry per label, so the author knows roughly how long the reply will be. Chat length is set by whatever the annotator types. Asked a question covering three labels, this endpoint at 400 stopped on the bare heading `### 3. Unclear` with nothing under it; at 800 it reached the last section but not its closing sentence; at 2000 it finished. Those runs were on a build before 2.9.0, whose chat path never checked the finish reason, and nothing appeared in the log. From 2.9.0 this endpoint logs `before finishing this chat reply`, but the sidebar still shows the reply with nothing to mark it as cut off, because only its text reaches the browser, so the log is the only place a cut-off shows. A cap costs nothing unless the model reaches it. **It is given far more than the assistants are.** The default prompt carries the task name, the task description, every scheme's name and its labels, and the item's `text_key` field. Asked what it had been given, the model listed all eight scheme names in the study. It is *not* given the image, on any endpoint type. On a task whose item is a picture, the chat only ever sees the text field beside it. **The default prompt refuses to name a label.** Its closing line is an instruction not to tell the annotator which label to choose, only to help them reason. That is the right default for a study measuring human judgement and the wrong one for a study measuring how people use an assistant. Replace it with a `system_prompt` block holding a `template` string, which is `str.format`ed with `task_name`, `task_description`, `annotation_labels`, `instance_text` and `instance_id`. An unknown placeholder falls back to the default prompt with a warning rather than failing. ## The blocks past `ai_support` `ai_support` puts a button beside a scheme. Three other blocks put a model in the loop at other points. On 2.11.1 Potato documents their sub-keys (25 under `active_learning`, 23 under `icl_labeling`, 2 under `arena`), and `config-keys-nested.md` lists them. What follows is what I set and what the server then said it had. **`active_learning`** reorders the queue by what a classifier finds hard, and needs `assignment_strategy: active_learning` to have any effect on assignment. `classifier.name` and `vectorizer.name` take a short name or an import path: ```yaml assignment_strategy: active_learning active_learning: enabled: true schema_names: ["stance"] min_instances_for_training: 5 # refuses to fit below this update_frequency: 2 # refit every N new annotations query_strategy: uncertainty classifier: {name: sklearn.linear_model.LogisticRegression} vectorizer: {name: sklearn.feature_extraction.text.TfidfVectorizer} model_persistence: {enabled: true} # a dict, not a boolean ``` `{name: logistic}` and `{name: tfidf}` validate and train the same models as the paths above. The short classifier names are `logistic`, `random_forest` and `svm`, with a few spellings of each; the vectorizer ones are `tfidf`, `count` (or `bow`) and `sentence-transformers` (or `embeddings`), and from 2.11.1 `image` (or `clip`) for items that are image files. On 46 coloured images under `media_directory`, referenced as `/media/...`, `{name: image}` trained to 0.978 accuracy and warned about the two remote URLs it could not read. A name Potato cannot build fails validation and stops the boot, with the full list in the message: `active_learning.classifier.name: 'logreg' is not a classifier Potato can build. Use a short name (logistic, logistic_regression, ...)`. Two log lines tell you it is working, and they say different things. The first is `Trained classifier for schema <name> with N instances, accuracy: ...`, which proves a classifier was fitted. The second names the strategy, because a fitted classifier and a served ordering are separate questions: ``` Reordered 2 instances (2 ranked, 0 exploration); assignment_strategy: active_learning, so this is the order annotators get Reordered 2 instances (2 ranked, 0 exploration); assignment_strategy is 'fixed_order', NOT active_learning, so this ordering is computed and never served ``` `/admin/active-learning/stats` carries the same fact as `assignment_strategy` and `ordering_is_served`. Read one of those two rather than the training line: `active_learning.enabled: true` with the wrong strategy trains a classifier whose ordering reaches nobody, and until this pair existed the two cases were indistinguishable in the log, in the admin report and in `--strict`. **`icl_labeling`** builds few-shot prompts out of annotations you already have. Its sub-keys are **three levels deep**. On 2.11.1 `--strict` refuses a name it does not know at any level, and a real sub-key at the wrong level too: `min_agreement_threshold` written directly under `icl_labeling` fails with `Unrecognized config key 'icl_labeling.min_agreement_threshold'`. The layout: ```yaml icl_labeling: enabled: true example_selection: min_agreement_threshold: 0.5 min_annotators_per_instance: 2 # the default, and the usual reason for zero max_examples_per_schema: 6 refresh_interval_seconds: 60 llm_labeling: trigger_threshold: 3 batch_size: 5 batch_interval_seconds: 90 verification: {enabled: true, sample_rate: 0.2} persistence: {predictions_file: icl_predictions.json} ``` It needs **two annotators who agree** on an item before that item can become an example. On a one-annotator study it runs on schedule and reports `Refreshed high-confidence examples: 0 examples across 0 schemas` forever. Prove the keys landed by reading `example_refresh=` in the startup line back: it echoes `refresh_interval_seconds`, so if it still says 300 you are in the defaults. **`arena`** fans one prompt out to several models and keeps a leaderboard. Each entry is `endpoint_type` plus `model`, with optional `label`, `base_url` and `temperature`; the page is `/admin/arena` and it answers with per-model latency, an error field, and Elo, Bradley-Terry and win-rate-interval columns once somebody picks a winner. The same server can appear twice under different labels, which is the cheap way to compare decoding settings rather than models. ## Verifying one `validate --strict` will pass a config whose assistants never appear. Boot it and drive it instead: 1. **Read the boot log for the endpoint's own line.** What you want names the class that was actually constructed, and agrees with what you configured: ``` OpenAI Vision client initialized with model: <model> at http://<host>/v1 AI endpoint ready (OpenAIVisionEndpoint), vision: True ``` 2. **Confirm the buttons exist.** No buttons means `include.all`, nine times out of ten. The container renders as `<div class="ai-help none">` when there are none. 3. **Click one and read the network response**, not the tooltip. The tooltip says "No hint available" for a truncated answer, a wrongly-shaped answer and a real failure alike. `/api/get_ai_suggestion` shows you which: ``` {"hint":"...","suggestive_choice":"negative"} rendered {"response":"### Annotation Guidance..."} unconstrained; nothing renders {"label":...,"reasoning":...} truncated: raise max_tokens ``` For Keyword, look at the item text rather than the tooltip: it draws `.ai-keyword-overlay` spans titled `<label>: "<keyword>"`. 4. **Check the cost surface if the study is long.** `ai_budget.cap_usd` refuses a run projected to cross it, and `/admin/api/ai-cost` reports spend. Both are aimed at metered providers; against your own GPU they measure tokens rather than money. **The cap only binds a model Potato has a price for.** Prices come from a table of 59 rows compiled into `potato/ai/cost.py`, matched by the longest substring of the model name. That gives five states, and they are not equally loud: | State | Cap | Log | |---|---|---| | Its own row | binds | silent | | A dated snapshot of a row (`<model>-20260901`) | binds | silent | | Matched to a shorter row, so priced as a relative | binds, on that rate | warns, and names `ai_budget.prices` | | No row contains it | does not bind, run proceeds | warns | | Local endpoint | binds, at zero | silent | Measured on 2026-09-07 against Potato at `1f8dab21`, 500 items of about 100 words: an unpriced model projected 107,500 tokens and no dollar figure, and ran through a `cap_usd: 0.10` untouched -- ``` ai_budget.cap_usd is set but 'gemini-3.0-pro' has no price on record, so this run cannot be checked against it. Projected 107,500 tokens. ``` The third row of that table is the one that used to be silent, and it is the more dangerous of the two: an absent price gives you no number, while a price taken from a shorter row gives you a confident one. It runs in both directions -- an unlisted new generation picks up an older, cheaper rate and the cap lets spend through; an unlisted cheap variant picks up its expensive parent's rate and the cap refuses runs you could afford. **Fix it in the config rather than waiting for a release.** `ai_budget.prices` is merged over the built-in table by the same matching rule, so a family name or a dated snapshot both work: ```yaml ai_budget: cap_usd: 0.10 prices: gemini-3.0-pro: [1.0, 8.0] # USD per million tokens, input then output ``` Measured on the same run: without it the batch above was unpriced and ran; with it the batch was projected at $0.46 and refused. The warning stops too, because it was only ever saying the cap had nothing to bind. No rate is written down here, and none should be: the next repricing makes it wrong without making it look wrong. Ask at the moment you need it: ```bash python <skill-dir>/scripts/model_prices.py config.yaml ``` It finds every model the config names, including one that lives only in `ai_config_file`, fetches a catalogue when it runs, and puts the live rate next to what Potato would charge. It names the row the charge came from, whether that row was the model's own or a relative's, and whether the number came from your `ai_budget.prices` rather than the table. Run it before choosing a `cap_usd`, not after. Self-hosted endpoints are skipped rather than reported as unpriced, because zero is already the right answer for them. With no network it says so and falls back, which is a different sentence from "that model is not listed"; do not read one as the other. **A disagreement it reports is a question, not a verdict.** The catalogue is a router's published rate, which is not always the vendor's list price. Measured on 2026-09-07: OpenAI lists `gpt-5.6-sol` at $4/$20 per million and the catalogue publishes $2/$10, while `gpt-5.6-terra` and `gpt-5.6-luna` agree exactly across both. Potato was right about sol. Check the vendor's own page before you change a number on the strength of this. Watch `/admin/api/ai-cost` as well. The cap is checked against a projection, and the projection is the only thing the running total is made of. `cache_config.disk_cache` is worth turning on for anything with more than a few dozen items: without it, the same item re-queries the model on every page view. `cache_config.prefetch.warm_up_page_count` generates ahead so the first annotator is not waiting on the first token. ## What I have and have not verified Run against a Potato checkout at v2.8.2 plus the endpoint fixes that followed it (bullets naming 2.9.0 or 2.9.1 were run on that release, except where a bullet says "from the source"), and a vLLM 0.24.0 server on a 12B text+vision model, in headless Chromium, reading the response body and the rendered DOM: - the `ai_config` nesting, and what `--strict` says when you get it wrong - `include.all` deciding whether any button exists - `ai_config_file` merging flat, and a missing one failing `--strict` - all three text assistants on `vllm`, `openai` and `openai_vision`, including the keyword overlays - Hint on an `image_annotation` task through `openai_vision`, including the model reading the photograph correctly and suggesting a label - `api_key` being optional, in the validator and the endpoint, when `base_url` is set - `text_key` deciding what the model receives, both ways round, and the capability filter dropping Keyword for an image item and keeping it for a text one - `ai_support.image_key` resolving the picture on an item that carries text and an image, and the answer changing when it is set - `chat_support` end to end against the same server: the sidebar, a reply, the `ui` keys, and a custom `system_prompt.template` reaching the model - on a build before 2.9.0, the same chat question at `max_tokens` 400, 800 and 2000, reading each reply back from `/api/chat/send` and each server log: cut at a heading, cut before its closing sentence, complete — no truncation warning in any of the three - on 2.9.0, the `vllm` and `openai` chat methods against the same server at `max_tokens` 40 and 2000, called directly rather than through `/api/chat/send`: the `this chat reply` warning at 40 on both, nothing at 2000 - on 2.9.1, the `openai_vision`, `vllm` and `openai` chat methods called the same way: the `this chat reply` warning at 40 on all three, nothing at 2000 - which endpoint methods call the truncation check, from the source: on 2.9.0, `query` on all ten text and vision endpoints, `query_with_image` on the three vision ones, and `chat_query` on `anthropic`, `ollama`, `openai` and `vllm`; on 2.9.1, a `chat_query` of its own on all ten, each logging `this chat reply` - on 2.9.0, from the source, `chat_manager` sending the browser only the reply's text - on 2.9.0, the vision loader against a `media_directory` holding one PNG: `cat.png`, `media/cat.png` and `/media/cat.png` read off disk, `http://localhost:8000/media/cat.png` and a `data:` URI of the same PNG refused, and `http://127.0.0.1:8000/media/cat.png` sent through `resolve_vision_image` logging the `TEXT ONLY` warning; from the source, `get_ai_help` returning the `image_attached` stamp to the route unchanged; from Potato's history, the disk read arriving in `v2.8.2-29-g39173a45`, and the warning and the stamp later, in `2b24a056` - `judge_alignment.inline` rendering a judge's suggestion beside the item, and `/admin/judge-alignment` reporting agreement against the human labels - `active_learning` training and reordering a twenty-item queue, and the dotted sklearn paths above being the names it accepts - `active_learning` retraining as annotations arrive (six fits across 24 items at `update_frequency: 2`) and the reordered pool actually reaching the annotator — and the same log and the same `/admin/active-learning/stats` appearing on a `fixed_order` arm where it reaches nobody - `llm_confidence` assigning at random, on 24 items, with a fitted classifier available to it - `icl_labeling` collecting examples once a second annotator agreed — 0 across 0 on one annotator, 6 across 2 schemas on two - `arena` running two models against one prompt and recording a preference - the price table against a live catalogue, row by row, on 2026-09-07: 52 of the 59 rows name a model the catalogue still lists, and 51 of those agree with it to the cent. The one that does not is `gpt-5.6-sol`, where the vendor and the router publish different rates and the vendor's page backs the table. The 7 unresolvable rows are models the router no longer carries (`gemini-1.5-*`, `gpt-3.5-turbo-1106`, both `claude-haiku-3-5` spellings), never carried (`claude-mythos-5`), or carried only as a preview (`gemini-3.1-pro`). The table is accurate where it is confirmable; what it is not is complete, and an absent model is the case `ai_budget.prices` exists for - `cost.record_spend` having one caller outside the tests (`server_utils/judge_alignment.py`), which passes the projection with `estimated=True`. The `estimated` column anticipates measured figures and nothing writes one, so the running total is projections summing projections and cannot be checked against an invoice On 2.8.2 itself none of the three endpoint types worked against a local server: `vllm` sent a parameter vLLM ignores, `openai`'s correct answer was delivered to the browser labelled as an error, `openai_vision` never sent the schema, and the 100-token default truncated two of the three assistants. All four fail the same way — the model answers, the tokens are spent, and the annotator reads "No hint available" — so if you are on 2.8.2 or earlier, read the response body before you conclude anything about your config. Not run, and not claimed: every commercial provider; `ollama` and the rest of the endpoint list; `icl_labeling`, `judge_calibration`, `arena` and `solo_mode`, all of which take the same endpoint config and so inherit whatever is true of it above. <!-- https://davidjurgens.github.io/potato-skill/labeling-engine/ --> # Letting Models Label and People Certify The 2.11 labeling engine: blind samples, contracts, and what to ask first. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Potato 2.11 adds a labeling engine: models label every item, people label a blind random sample, and each contract in the config tests the machine labels against that sample. The machine labels a contract accepts come with a stated error bound, and the rest go to people. The researcher gets the dataset and a certificate saying what was tested, on how many human labels. It is configured in one `labeling_engine` block, and annotators are served its requests only under `assignment_strategy: engine`. Potato's own guide is `docs/labeling-engine/` in the checkout, five pages with every key; this page is what to decide with the researcher and what I saw when I ran it. ## Corpus size The engine saves human labels when the corpus is much larger than the human budget. On a small corpus it can sample most of the corpus and accept nothing, as it did on the 120-item run below. I ran it on 120 product reviews with Gemma 12B on vLLM as the only source and a `selective_risk` contract at 10% error and 0.9 confidence. Gemma was wrong on 8 of the 120 (6.7%), under the target, but the corpus was too small to test the contract and it never passed. The planner opened 107 calibration requests and 22 resolve requests, and every one of the 120 delivered labels came from a person. So ask how many items there are and how many human labels the researcher can pay for before proposing it. If every item will get a human label anyway, use an ordinary assignment strategy. ## What to ask the researcher | Question | Where the answer goes | |---|---| | "How often may an accepted machine label be wrong, and how sure do you need to be?" | `error` and `confidence` on a `selective_risk` contract | | "Do you need every label to be usable, or a rate, such as the share of toxic posts?" | `selective_risk` for labels; `ppi` for a prevalence with an interval | | "Should the model match your annotators rather than a fixed accuracy?" | `no_worse_than_human`, which needs `humans.overlap_k` of at least 3 | | "How many human labels can you pay for?" | `budgets.human_labels` | | "Which questions should the model answer?" | `labeling_engine.schemas`; free-text questions cannot be among them | The error target and the confidence are the researcher's call. They are what the certificate will claim, so do not pick them yourself. ## A minimal block ```yaml assignment_strategy: engine annotation_schemes: - annotation_type: radio name: sentiment description: "Overall sentiment?" labels: [positive, negative, neutral] labeling_engine: enabled: true sources: - id: gemma kind: llm endpoint_type: vllm model: google/gemma-4-12B-it-qat-w4a16-ct base_url: http://localhost:8001/v1 confidence: {method: logprob} contracts: - {id: accuracy, kind: selective_risk, schema: sentiment, error: 0.1, confidence: 0.9} - {id: prevalence, kind: ppi, schema: sentiment, confidence: 0.9} budgets: {human_labels: 200} ``` `logprob` confidence reads the probabilities of the model's one-letter answer. It needs a backend that returns logprobs, such as vLLM, and a scheme with at most 20 labels. `native` on an `llm` source is refused: `confidence.method native comes from decision models and detectors; an llm source uses verbalized, k_sample or logprob`. Contract keys are checked. A misspelt `error` names the key and the ones the kind reads: `labeling_engine.contracts[0] (selective_risk) has unknown keys: eror (known: confidence, error, id, kind, per_class, schema, tolerance)`. ## Free-text questions The engine labels only schemes it can score a machine answer on, and free text is not one of them. A contract on a free-text scheme is a validation error: `these contracts are on schemes the engine cannot score machine answers for: accuracy (note: text)`. A free-text scheme can still sit beside the engine's schemes, but under `assignment_strategy: engine` people see it only on the items the engine sends them, and `potato validate` says so: ``` assignment_strategy: engine serves only the engine's requests, and the engine does not label note, so people will answer them only on the items the engine sends them for the other schemes, not on every item. If every item needs them, use another assignment_strategy. ``` That is a warning, so `--strict` fails on it. If the researcher needs a comment on every item, the study cannot use `assignment_strategy: engine`. Tell them before building it. ## An engine with no contracts A machine label is accepted only when a contract has tested it. An engine with `contracts: []` accepts nothing. I enabled one with a `static` source holding the right answer for all 120 items. It validated clean under `--strict`, opened 62 calibration requests, and accepted no machine label. ## The budget check `potato validate` works out whether `budgets.human_labels` could certify each `selective_risk` contract at all: ``` budgets.human_labels 20 cannot certify accuracy (error 0.1, confidence 0.9) even with no machine errors: it needs at least 54 human labels (27 held out, the rest for choosing the threshold). Raise the budget or loosen the target. ``` That is the floor for a model that is never wrong. In the 120-item run, 124 answered requests were not enough for a model wrong on 6.7% of the items. `assignment_strategy: engine` with `labeling_engine.enabled: false` fails validation: `assignment_strategy: engine serves the labeling engine's requests, but labeling_engine.enabled is false.` ## Running it ```bash potato label run config.yaml # label every item, draw the blind sample potato start config.yaml -p 8000 # people answer the sample potato label status config.yaml # contracts, requests, what is still missing ``` `label run` made 120 model calls on 120 items, opened 62 calibration requests and accepted nothing: with no human labels yet, the contract read `insufficient`. Only one process drives a study. While the server runs, its planner holds the study's lock, and the command line refuses with exit 2: ``` server:<host>:<pid>:1 is driving this study. Certify through it instead (POST /api/engine/v1/certify, or the admin page's Certify button), or stop it first. ``` `status`, `certificate`, `export` and `calls` are read-only and work while the server runs. The planner certifies on its own as labels arrive, so a running study rarely needs an explicit certify. `POST /api/engine/v1/certify` with the admin key in `X-API-Key` worked, and the same route without the key returned 403. The admin page is `/admin/labeling-engine`. ## What annotators see Every calibration item showed "Label this one yourself. Suggestions are off for this item." above the form. Calibration requests are blind: the person does not see a machine answer, so their label is an independent check of it. ## Reading the result `potato label certificate config.yaml --format md` writes the certificate. A contract that has not passed says so in its statement: ``` Target, not established: at most 10.0% of accepted labels wrong, with probability at least 0.90 (overall). No machine label on this scheme is accepted until it is met (43 held-out labels, look 1). ``` When reporting to the researcher, quote the statement rather than the config: the config holds the target, and only a `met` statement reports a result. ## Getting the labels out `potato label export config.yaml --out DIR`, or the `labeling_engine` export format, writes `labels.jsonl`, `labels.csv`, the certificate as JSON, Markdown and HTML, a dataset card in `README.md`, and `model_calls.jsonl`. `labels.csv` has one row per item and scheme: ``` id,schema,label,label_origin,status,source,model,source_version,confidence,contract_version,request_ids r0001,sentiment,neutral,human,accepted,,,,,,[9] ``` The ordinary `csv` export still holds the annotators' answers. With the engine on it adds `engine_purpose` and `engine_request` columns, and the second one holds the request's `split`, `visibility` and `inclusion_prob`. Use them to tell the blind random sample from the other answers before anyone computes an accuracy from the human labels. ## Configs with the older keys `pre_annotation`, `icl_labeling` and `judge_calibration` still work in 2.11, and each prints a note pointing at the engine: ``` NOTE: pre_annotation still works, but the labeling engine now does this: use labeling_engine (a static source). `potato label migrate-config` writes the equivalent block. pre_annotation is planned for removal in 3.0. ``` `potato label migrate-config config.yaml --out new.yaml` writes a copy with a `labeling_engine` block. For `pre_annotation` it wrote a `static` source on the same field, `enabled: false`, and `contracts: []`. Enabling that block as it stands accepts nothing (see **An engine with no contracts**), so add the contract with the researcher before turning it on. ## What I have and have not verified Run on 2.11.0 (`eb284819`), from a checkout, against Gemma 12B on a vLLM server: - the 120-item study end to end: `label run`, the server's planner, three annotators labeling over HTTP from the items' true labels, `status`, `certificate`, `export`, and the `labeling_engine` and `csv` exports; - the writer-lock refusals of `label certify` and `label run`, `POST /api/engine/v1/certify` with and without the admin key, and the admin page; - every validation message quoted above, by `potato validate --strict` on a config built to trigger it; - `migrate-config` on a `pre_annotation` config, and that block enabled. Not verified here: decision-model sources (Laya, hosted Jev, OpenJev), the detector, judge, jury and distilled sources, the `no_worse_than_human`, `budget_first` and `distributional` contracts, verify requests, crowd recruiting on Prolific, Pocket Mode with the engine, and the Python SDK. <!-- https://davidjurgens.github.io/potato-skill/data-and-access/ --> # Data Files and Admin Access Input formats, side files, and reading results back out. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Where items come from, and who is allowed to annotate them. Nothing else in this pack covers either. ## Items ```yaml data_files: - data/articles.json item_properties: id_key: id text_key: body ``` `data_files` accepts JSON, JSONL, CSV and TSV. JSON may be an array of objects or one object per line. Every object needs the fields named by `id_key` and `text_key`. Several files merge in the order listed, and they do not have to carry the same fields — I loaded a three-item file and a two-item file whose objects had different extra keys and got five instances, the first file's items first. But **an id repeated across files stops the server dead**, and `validate` does not see it coming: ``` $ potato validate config.yaml --strict -> OK — no issues found. $ potato start config.yaml -> ValueError: Duplicate instance ID 'SHARED' found at item 2 ``` Failing rather than silently keeping one of the two is the right call, and the message names the id. Worth knowing before you split a corpus across files, because it is the one data problem that passes validation and then refuses to boot. | Key | Effect | |---|---| | `item_properties.id_key` | Field holding the unique identifier | | `item_properties.text_key` | Field shown to the annotator when there is no `instance_display` | | `item_properties.category_key` | Field holding a category, used by `category_based` assignment | | `item_properties.kwargs` | Extra per-item fields carried through to the display layer | **`text_key` still matters when you define `instance_display`.** It is what plain-text rendering falls back to, what span schemes anchor to by default, and what the training and search surfaces read. Point it at the field with the actual content, not at a title. **Every other field in the row is available to the display layer** by name. A row with `outlet`, `headline` and `body` can render all three as separate `instance_display.fields` without any of them being `text_key`. Fields you do not display are still stored, which is how you keep a condition label out of the annotator's view without dropping it from the data — but only `parquet` exports them. csv, tsv and jsonl carry `instance_id` and the answers and no item fields at all, so plan to hand the data file over with the export, or to use `parquet`. `getting-the-data-out.md` has the comparison. **Every path in the config has to resolve inside the project directory.** A `data_files` entry that climbs out with `../` is refused at boot with `ConfigSecurityError: Path '...' resolves to '...' which is outside the project directory`, and `potato validate` reports the same thing without booting. Copy shared data into the project, or move the config up to a directory that contains both. ### A directory instead of a list ```yaml data_directory: ./data/incoming data_directory_encoding: utf-8 watch_data_directory: true # rescan while running and pick up new files watch_poll_interval: 5.0 ``` It reads `.json`, `.jsonl`, `.csv` and `.tsv`, and skips everything else without comment. A directory of `.md` or `.txt` files loads nothing, and the only sign is `Loaded 0 instances from data_directory: ./incoming` at INFO in the boot log — the config validates, the server starts, and the first annotator gets an empty study. Count the instances in the log before handing out the URL. This is also not the way to get documents into a `document` field; see `building-the-ui.md`. ### Remote and live sources ```yaml # type: url, s3, huggingface, google_sheets, database, google_drive, # dropbox or file data_sources: - type: url url: https://example.org/items.json data_cache: enabled: true ttl_seconds: 3600 max_size_mb: 200 ``` Sources can be polled for new rows, which is how a task keeps ingesting while annotators work. Pair with `automation` (rules over incoming items) and `triage` (rank what arrives). Both are in `modes-and-subsystems.md`. ### Media ```yaml media_directory: media # default; served at /media/ ``` Put images, audio and video under it, and write the path **including the directory** in the data field: `media/photo.png`, not `photo.png`. The displays emit the field value as the URL; only `depth_map` prepends the directory itself, so a bare filename resolves against the site root and 404s. Measured across one page on 2.8.2-26: `image`, `gallery`, `video`, `audio`, `pdf`, `audio_dialogue` and `web_agent_trace` all need the prefix in the data, and all of them except `pdf` fail without a word on the page. Without an `instance_display` field of the right type, that path renders as **text** — the classic "the item shows a file path instead of an image". See `building-the-ui.md` for the display types. ### Big corpora ```yaml partial_loading: {...} # load lazily rather than reading everything at boot item_store: backend: paged # memory (default) or paged cache_size: 1000 path: .item_cache.sqlite # under the output directory when unset ``` An unknown `item_store.backend` warns and silently falls back to memory rather than refusing to start, so check the boot log if paging was the point. ## Output ```yaml output_annotation_dir: annotation_output/ export_annotation_format: [csv] # periodic auto-export; empty means none auto_export_interval: 60 # seconds export_include_phase_data: false # consent and survey answers; usually where the PII is export_include_annotation_changes: false ``` **There is one storage format and it is not configurable.** Annotations live at `annotation_output/<user>/user_state.json`, one file per annotator, whatever else the config says. Any other shape is an export, and there are two ways to ask for one: `export_annotation_format`, which writes it on a timer, and the admin export route, which writes it on demand. `output_annotation_format` is deprecated — the loader reads it as `export_annotation_format` and warns. Write the live key in anything you author. So what a researcher gets at the end of a study is a directory of per-annotator JSON, not a table. If they want one row per item, produce it: either set `export_annotation_format: [csv]`, or read the `user_state.json` files and write the table yourself. Decide which before handover and say so, because "where is my CSV" is the first question after the annotators finish. `annotation_output/` is rewritten wholesale by the server. Never hand-edit anything in it — `potato repair-annotations` exists for when something already did. Per-annotator state lives at `annotation_output/<user>/user_state.json` and is the fastest way to check what the server actually stored. Keys worth knowing: `instance_id_ordering`, `instance_id_to_label_to_value`, `instance_id_to_span_to_value`, `phase_to_page_to_label_to_value`, `training_state`, `max_assignments`. ## Who may log in ```yaml login: type: password # password | url_direct | none user_config: allow_all_users: true # anyone may register ``` | Key | Effect | |---|---| | `login.type` | `password` (register and sign in), `url_direct` (identity in a query parameter), `none` | | `login.url_argument` | The query parameter carrying the user id under `url_direct` | | `user_config.allow_all_users` | Default `true`. Anyone can register | | `user_config.users` | Allowlist, used when `allow_all_users: false` | | `require_password` / `require_no_password` | Force the choice regardless of `login.type` | | `authentication` | SSO and database-backed accounts | | `rbac` | Role assignments and SSO role mapping | `url_direct` is what crowdsourcing platforms want: the platform appends the worker id and the annotator never sees a login screen. ### Running on a crowd platform ```yaml login: {type: url_direct, url_argument: PROLIFIC_PID} crowdsourcing: {provider: prolific} completion_code: CODE-ABC123 auto_redirect_on_completion: true auto_redirect_delay: 3000 # milliseconds on the completion page first ``` `crowdsourcing.provider` takes `prolific`, `mturk`, `connect`, `sona`, `microworkers`, `clickworker`, `expert`, `generic` or `url_direct`, and the boot log confirms it — `Initialized crowdsourcing provider: prolific`. Set `login.url_argument` to whatever the platform appends: Prolific sends `PROLIFIC_PID`, so `?PROLIFIC_PID=worker777` seats that worker with no login screen and their id in the navbar. Naming a provider changes the completion page, which is the part worth seeing before you launch. With the redirect off it reads Thank You! You have completed the annotation task and your responses are saved. CODE-ABC123 Click to copy your completion code Return to Prolific — the code, a copy control, and a return link named after the provider. With `auto_redirect_on_completion: true` the worker is sent to the platform's completion URL instead, after `auto_redirect_delay`. I drove both: the redirect really does leave for the platform, so test with it off first unless you enjoy losing your browser session mid-check. The login page has two tabs, `login` and `register`, in one document — if you are driving it in a browser, switch with `switchTab('register')` before filling the register form. Both forms post `action` (`login` or `register`), `email` and `pass`; the fields are `#login-email`/`#login-pass` and `#register-email`/`#register-pass`. **The default authenticator keeps accounts in memory.** The boot log names the one you have: `Initialized UserAuthenticator with method: in_memory`. That method has no store behind it, so a username is only as durable as the process. `authentication` is the block that gives database-backed or SSO accounts; `secret_key` and `persist_sessions` cover the Flask session, which is a different thing. Decide this before you recruit rather than after the first restart. ## Serving it | Key | Effect | Default | |---|---|---| | `port` | Port; `-p` overrides | 8000 | | `host` | Interface to bind; `0.0.0.0` exposes beyond localhost | localhost | | `secret_key` | Flask session signing key, and the key that decides whether a login survives a restart: unset, every boot signs with a fresh random key and everyone is logged out | — | | `session_lifetime_days` | How long a session lasts. The login response carries it as an explicit cookie `Expires` | 2 | | `persist_sessions` | Asks for sessions to be kept across a restart; refuses to start without `secret_key` | false | | `admin_api_key` | Admin API key, sent as `X-API-Key`. Generated into `{task_dir}/admin_api_key.txt` when unset | — | **`debug: true` is for a server on your own machine, never for one an annotator can reach.** The same goes for `--debug` on the command line. On its own it still shows the login page. With `--debug-phase annotation` it signs every visitor in as one shared `debug_user` and drops them into the task, which is what makes it useful for checking the interface. It also opens the admin pages without a key to requests from the server's own machine, so an admin-auth check you run locally under debug proves nothing. On 2.11.1, with `--debug` on the default `0.0.0.0` bind, `/admin/api/overview` answered 200 on `localhost` and 403 on the machine's network address. 2.11.0 refused that bind outright ("Refusing to start: debug is enabled and the server is bound to 0.0.0.0 rather than loopback"), so on 2.11.0 pass `--host 127.0.0.1`. Binding `0.0.0.0` is how a researcher on the same network opens the task. Say so explicitly in a handover, along with the port, rather than leaving them to guess that localhost is not the only address. <!-- https://davidjurgens.github.io/potato-skill/importing-existing-work/ --> # Importing Existing Annotations, Transcripts and Corpora What to do when the researcher already has annotated data. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- The brief is not always "here is a corpus, build me a task". Often the researcher arrives with work already done: annotations from another tool, a folder of interview transcripts, a conversation corpus, a HuggingFace dataset. Potato converts all three, and building a config by hand around data that one of these commands would have shaped correctly is the most avoidable way to lose an afternoon. Ask what they have before you ask what they want to label. ## Annotations from another tool ```bash potato import --list-formats potato import -i instances_val.json -f coco -o ./project --image-url-prefix /media ``` 20 formats, which `--list-formats` prints in two groups. Image and video: `coco`, `cvat`, `labelme`, `labelbox`, `pascal_voc`, `via`, `darwin`, `cityscapes`, `kitti`, `mot`, `davis`, `openimages`, `webdataset`, `yolo` and `huggingface`, which also takes a HuggingFace Hub id through `--hf-dataset`. Text: `brat`, `conll`, `doccano`, `prodigy` and `qdpx`. The format is auto-detected when `-f` is omitted. What comes out is a runnable project: `config.yaml`, a data file and the annotations, rather than a converted file you still have to wrap. The decision that matters is what the imported annotations *are*: | | Without `--seed-user` | With `--seed-user NAME` | |---|---|---| | The annotations become | Pre-annotations | NAME's saved work | | An annotator sees | A starting point they can correct | Nothing; the item is already done | | Stored before someone saves | No | Yes | Pre-annotation is what a correction pass wants: existing boxes shown as a starting point, a human fixing them, the human's version stored. `--seed-user` **fabricates an annotator** and exists so import→export can be checked without opening every item by hand. Do not use it to load a gold set you intend real people to re-annotate. The items will read as finished. Two flags that lose information, both worth naming to the researcher rather than choosing quietly: - `--rle-as-polygon` traces RLE masks into editable polygons. Holes are dropped and the contour will not re-rasterize to the source mask. - `--extract-media DIR` is required for WebDataset shards. Without it the images stay inside the `.tar`, no web server can read them, and the canvas is blank. ## Transcripts and subtitles ```bash potato transcripts interviews/*.vtt -o data/interviews.json --dry-run potato transcripts interviews/ -r -o data/interviews.json \ --media-dir audio/ --emit-config ``` Reads Whisper, WhisperX and whisper.cpp JSON and TSV, SRT, WebVTT, SubStation Alpha, TTML, YouTube `json3`/`srv`, AWS Transcribe, Deepgram, AssemblyAI, Rev.ai, CTM, Praat TextGrid and ELAN EAF. Speaker labels and timings survive the conversion, which is what `audio_dialogue`, `speech_transcript` and `tiered_annotation` bind to. `--dry-run` reports the detected format and turn count per file and writes nothing. Run it first — a subtitle file that parses to one turn is usually a format guess gone wrong. `--emit-config` prints a config fragment for the file it just wrote, which is the fastest correct starting point for the item keys. `--media-dir` pairs each transcript with an audio or video file by basename, so per-turn playback works without you writing the paths. ## Conversation corpora ```bash potato convokit --list-corpora potato convokit reddit-corpus-small -o data/threads.json --unit conversation \ --context-mode ancestors --sample 200 --emit-config ``` `--unit` is the unit-of-annotation decision from `designing-a-task.md`, made at conversion time: `conversation` gives one item per thread, `utterance` one item per turn. It is the choice that is expensive to reverse, so confirm it rather than take the default. `--context-window`/`--context-after`/`--context-mode` decide how much surrounding conversation an utterance-level item carries, and `--sample`/`--seed` cut a reproducible subset before you have committed to anything. ## Four questions before importing 1. **Which tool wrote them?** Match it against `--list-formats` before writing any conversion yourself. 2. **Are these being corrected, or are they done?** That is the pre-annotation versus `--seed-user` question above, and it changes what the annotator sees. 3. **Where are the images or audio?** The import writes URLs. Files served by Potato go under `<task_dir>/media` with `--image-url-prefix /media`; everything else needs the absolute URL they are already hosted at. 4. **Is this the label set they want to keep?** An import reproduces the source scheme faithfully. Categories they meant to retire come across with the rest. <!-- https://davidjurgens.github.io/potato-skill/getting-the-data-out/ --> # Getting the Annotations Out The export formats, what a CSV holds, and where consent answers go. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- "Where is my CSV" is the first question after the annotators finish, and the answer surprises people: there isn't one until you make it. Decide the export before handover and say so, because the researcher who finds out at the end is the one who wanted COCO and got a directory of per-annotator JSON. ## Storage and export Annotations live at `annotation_output/<user>/user_state.json`, one file per annotator, and that is the only storage format. `output_annotation_format` looks like it changes that, and never did; it is deprecated now, and the loader reads it as `export_annotation_format`. Everything else is an export: a derived copy, produced on a timer or on demand, that you can regenerate and can safely delete. Three ways to produce one: ```yaml export_annotation_format: [csv, jsonl] # on a timer, while the study runs auto_export_interval: 60 # seconds; writes to annotation_output/exports/<fmt>/ ``` ```bash # on demand, against a running server curl -H "X-API-Key: $(cat admin_api_key.txt)" -H "Content-Type: application/json" \ -X POST -d '{"format":"coco"}' localhost:8000/admin/api/export ``` ```bash potato deploy pull config.yaml --dest ./collected # from a hosted task ``` The admin route writes to the **server's** disk and hands back the paths — it is not a download. On a host you cannot reach a shell on, `/admin/api/data/archive` streams the whole task directory as a gzipped tar instead. `deploying.md` has that path. **One scheme keeps its answers somewhere else.** `multi_document_event` writes the events themselves to `annotation_output/event_registry.json` — one registry for the whole study, not one file per annotator: ```json {"events": {"evt_f6ec517d3154": { "title": "Payments certificate outage", "slot_values": {"service": "Payments API", "cause": "expired certificate"}, "member_doc_ids": ["r01"], "evidence": [{"slot_name": "scope", "doc_id": "r01", "span_start": 0, "span_end": 30, "quoted_text": "Payments API returned 503s for", "created_by": "s4@x.com"}], "created_by": "s4@x.com"}}} ``` Two consequences to decide about before you recruit. The per-annotator side is missing entirely: on 2.8.2-11 the input holding which events a document belongs to carries no `annotation-input` class, so `user_state.json` stays `{}` for that scheme and the export has nothing in it. The registry file is the whole dataset. The registry is also shared. Annotator 2 opens a document and sees annotator 1's events already attached to it, `member_doc_ids` records no attribution, and there is no second opinion to compute agreement from. Treat this as one collaborative pass rather than a replicated one, and back the file up the way you would back up `user_state.json`. **A bad format name is silent until it matters.** `export_annotation_format` is not checked at load, so `[csvv]` validates clean under `--strict`, boots clean, and produces one runtime warning after the first save — by which point nobody is reading the log. Check the name against the list below. ## Which format 32 formats are registered. `GET /admin/api/export/formats` lists them with descriptions against the running server; `potato preview` does not. | They want | Format | |---|---| | A table, one row per annotator-item | `csv`, `tsv`, `jsonl` | | The same at corpus scale, for pandas or DuckDB | `parquet` | | To train an NER model | `conll_2003`, `conll_u` | | To train a detector or segmenter | `coco`, `yolo`, `pascal_voc`, `mask_png` | | To go back to the tool they came from | `cvat`, `labelme`, `darwin`, `cityscapes`, `kitti` | | Tracking ground truth | `mot`, `davis` | | Phonetics or linguistic tiers | `textgrid`, `eaf` | | A ConvoKit corpus with the labels attached | `convokit` | | Their qualitative codebook and quotes | `codebook`, `quotation_report` | | To open it in NVivo, MAXQDA or ATLAS.ti | `qdpx` | | Agent or coding-agent evaluation data | `agent_eval`, `coding_eval` | | Preference pairs and SFT targets from corrections | `trajectory_correction` | | Per-frame embodied episode labels | `episode_jsonl` | | How the text was typed, or how the boxes were drawn | `keystrokes`, `annotation_telemetry` | | The single resolved label per item, after adjudication | `adjudication` | | The labeling engine's labels, certificate and model-call log | `labeling_engine` (see `labeling-engine.md`) | | It published | `huggingface` (see `publish` in `modes-and-subsystems.md`) | **An item nobody annotated is not in the export.** I ran a three-image study, annotated two and advanced past the third, and every one of the eleven vision formats wrote two images. The third appears in none of them, not even as an entry with an empty annotation list. That matters for detector training, where images with no objects are the negatives, and it matters for anyone reconciling counts: a study of N items exports the number of items somebody actually marked, not N. Every format says so in its warnings and names the items, so read them — this is the one case where the count you get and the count you expected differ for a reason that is nobody's mistake. **Take masks from an export, not from `user_state.json`.** The stored form is an internal one: the mask lives in a `_data` label as a JSON string whose `rle` is row-major with `size` as `[height, width]`. COCO's RLE is column-major, so reading the raw file with pycocotools conventions gives a transposed mask that looks plausible and is wrong — a stroke of mine came back at [289,0,321,399] instead of [390,180,473,200]. The `coco` exporter does the conversion, so `annToMask` on its output is correct. **A composite scheme's answer arrives as a JSON string, one level in.** Types that record a structure rather than a label store the whole answer under a single `(schema, schema)` label, so the `jsonl` export nests it as a string. Measured on `code_review` and `text_edit`; the geometry schemes do the same thing through a `_data` label, as the mask note below describes: ```json {"instance_id": "t1", "user_id": "r@example.com", "labels": {"review": {"review": "{\"verdict\": \"request_changes\", \"comments\": [...]}"}}} ``` A consumer has to `json.loads` the inner value before it is data. Worth knowing before you write the analysis script, and worth checking on your own schemes: which of them land as a label and which land as a blob is decided by the type, not by anything in your config. The vision and linguistics formats are the reason to ask early: a researcher who says "we'll train a detector on this" wants `coco` or `yolo`, and a task designed without that in mind can produce geometry that does not survive the conversion. ## The CSV columns Columns are derived from what was **stored** rather than from what the config declares. One row per annotator per item, with a column per scheme-and-*label*: ``` instance_id,user_id,sentiment.Negative,sentiment.Positive,severity.Moderate,issues.Price r03,alice@x.com,Negative,,Moderate,Price r02,alice@x.com,,Positive,, ``` That is one column per label an annotator actually chose, not one column per scheme. A three-label `radio` becomes three sparse columns and there is no `sentiment` column anywhere in the file, so the first thing anyone does with a Potato CSV is coalesce them. Warn whoever receives it: a researcher who opens it expecting one column per question reads the file as broken. Driven on six schemes and three annotators, 23 rows came out as 17 columns. Three consequences: - A scheme nobody answered has no column, so an empty export is evidence about the annotations rather than about the exporter. - A label nobody chose has no column either, so the column set is not stable across two runs of the same study. - After a scheme is renamed mid-study, the columns carry the **old** name, and keep carrying it. See `after-annotators-start.md`. Spans arrive as one `<scheme>._spans` column holding a JSON array of `{schema, name, title, start, end, id, target_field, text}` objects — the offsets and the words they cover, sliced server-side, so the two cannot disagree: ``` [{"schema": "evidence", "name": "Evidence", "start": 23, "end": 39, "target_field": "body", "text": "price I expected"}] ``` **The item is not in the export.** csv, tsv and jsonl carry `instance_id`, `user_id` and the answers, and nothing else: not the annotated text, not the image URL, not the fields you never displayed. Verified on a study whose items carried `body`, `photo`, `sku` and `batch`. None of the four appears in any of the three. Whoever gets the CSV has labels keyed by `instance_id` and has to join back to the data file to find out what was labelled, so hand over the data file with it. `parquet` is the exception. ### The parquet export `parquet` writes **several** files rather than one, and reshapes the annotations: | | `csv` / `tsv` / `jsonl` | `parquet` | |---|---|---| | Files | `annotations.<ext>` | `annotations.parquet`, `items.parquet`, `spans.parquet` | | The items | absent | `items.parquet`, every field, displayed or not | | A `radio` scheme | one sparse column per label used | one column holding the chosen label | | A `multiselect` | one column per label used | one list column | | Spans | JSON blob in `<scheme>._spans` | `spans.parquet`, one row per span | Both families add a `phase_responses` file when `export_include_phase_data` is on, so a parquet handover is four files against csv's two. So the "keep a condition label out of the annotator's view without losing it" trick works, but only through `parquet` or by joining the data file back on `instance_id`. **Open the file before you hand it over.** A composite widget is one scheme holding several answers, and the column it produces depends on how the widget stored them. Both families agree, and the shape is worth knowing before the analyst asks: | Scheme | Stored as | Columns | |---|---|---| | `multirate`, `constant_sum`, `soft_label` | one entry per row or option | `reasons.Population`, `reasons.Intervention`, … | | `hierarchical_multiselect` | one entry, comma-joined | `topics.selected_labels` holding `Annotation,People,Experts` | | `ranking` | one entry, comma-joined in rank order | `priority.rank_order` holding `Cost,Agreement,Accuracy` | | `image_annotation` and the other geometry types | one entry, a JSON blob | `uibox._data` | The joined ones need splitting before anything can be counted. The hierarchical list also carries every ancestor of each leaf an annotator ticked, so a count of `Annotation` includes everyone who chose something inside it. ```python import pandas as pd print(pd.read_parquet("annotations.parquet").head()) ``` `head()` on the file you are about to send is the check. It costs a minute and it is the last point at which a column carrying the wrong thing is still cheap to fix. ## Consent and survey answers ```yaml export_include_phase_data: false # the default export_include_annotation_changes: false ``` Phase responses — consent, pre-study, post-study surveys — are excluded from exports unless you turn them on, and they are usually where the demographics and the free-text sit. That default is doing real work. Turning it on is a decision to move identifiable answers into a file people will pass around, so make it deliberately and say so in the handover. **They arrive as a second file, not as extra columns.** Turning the key on adds `phase_responses.csv` (or `.jsonl`) beside `annotations.csv`, long-format, one row per answer: ``` user_id,phase,page,sequence,schema,label_name,value alice@x.com,consent,consent,0,consent_agree,I agree,I agree alice@x.com,poststudy,poststudy,0,task_clarity,4,4 alice@x.com,poststudy,poststudy,1,comments,text_box,alice poststudy comment ``` "Here is the CSV" is two files once this is on, and the second one is the one carrying the demographics. Name that file explicitly when you hand the study over; a filename nobody was told about is one that gets left behind. The admin API's response tells you which way it went: `num_phase_responses` and `num_phase_responses_excluded` are both in `stats`, and one of them is always zero. `export_include_annotation_changes` adds the revision trail: every answer an annotator moved off, with timestamps. Useful for studying how people decide, much larger, and not what anyone means by "the data". ## Agreement The export holds answers. Agreement is computed on request, from the admin API: ```bash curl -H "X-API-Key: $(cat admin_api_key.txt)" localhost:8000/admin/iaa ``` Per scheme it reports the `kind` it inferred and the metrics that follow from it. The `kind` is inferred from the schema type, so a rating stored as a `radio` is scored as unordered — `assignment-and-agreement.md` has the table, and it is worth checking before quoting a number at anyone. <!-- https://davidjurgens.github.io/potato-skill/running-a-task/ --> # Running a Task Booting a server, reading the startup log, and handing it over. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- `potato validate` tells you the server will start. `potato preview --screenshot` tells you the first 900 pixels of one page did not throw. Neither tells you an annotator can get from the login screen to the end of the study, and most of what breaks a task lives in that gap. This file covers the three things the rest of the pack does not: how to get a server up and leave it up, how to read what it says while booting, and how to drive it like an annotator. ## Boot it early The startup log is the real validator. It is the only place that reports: - whether each phase assembled, or was skipped - how many training, attention-check and gold items actually loaded - which subsystems initialized - exceptions from side files that `validate` never opens Run it as soon as the config parses, well before the interface is finished. Every undocumented file format in this pack was recovered from these lines. ```bash grep -E "ERROR|Traceback|Loaded [0-9]+|phase|missing required" server.log ``` ## Backgrounding it `potato start` never returns. In a non-interactive session, run it detached and poll for readiness. Never in the foreground, and never behind a bare `sleep`. ```bash cd /path/to/project # must be the directory task_dir resolves to nohup potato start config.yaml -p 8421 > server.log 2>&1 & until curl -s -o /dev/null http://localhost:8421/; do sleep 2; done curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8421/ # 200 ``` Boot takes 10–20 seconds on a small task. The readiness lines in the log are `Serving Flask app 'potato.flask_server'` and `Running on …`. There is no `/health` endpoint on a plain `potato start`; a 200 on `/` is the check. For a throwaway boot — you want the log, not a server — start it, wait for the readiness line or the process to die, then kill the process group: ```python import subprocess, os, signal, time p = subprocess.Popen(["potato", "start", "config.yaml", "-p", "8951"], cwd=PROJ, stdout=open("/tmp/boot.log", "wb"), stderr=subprocess.STDOUT, start_new_session=True) t0 = time.time() while time.time() - t0 < 120: time.sleep(0.5) s = open("/tmp/boot.log", "rb").read().decode("utf8", "replace") if "Running on" in s or "Serving Flask" in s or p.poll() is not None: break os.killpg(os.getpgid(p.pid), signal.SIGKILL); p.wait() ``` `start_new_session=True` plus `killpg` matters: the server spawns children and a plain `kill` leaves the port held. Stop a named server with `pkill -f "potato start config.yaml -p 8421"` and confirm with `lsof -nP -iTCP:8421 -sTCP:LISTEN`. ## Running the server writes to the project directory | Path | What it is | Safe to delete? | |---|---|---| | `annotation_output/` | Annotations, per-user state, exports | Yes, **while stopped**. It is the data. | | `annotation_output/<user>/user_state.json` | One annotator's assignment, answers, spans, training state | Never edit by hand | | `project.sqlite` (+ `-wal`, `-shm`) | Codebook, cases, memos, search index | Yes, while stopped; it reseeds | | `layouts/task_layout_*.html` | Generated form HTML, cached by config hash | **Delete after editing schemes** if the page looks stale | | `server.log` | Whatever you redirected | Yes | Wiping `annotation_output/` **while the server runs** does not reset anything: item and user state are already in memory, and the next annotator to arrive gets the completion page with zero items. If a fresh task suddenly says "Thank You!" to a new account, this is why. Stop the server, wipe, start it again. Two consequences worth knowing before you verify a config you are still editing: - The first boot **seeds** `project.sqlite` from the config. For a codebook task the labels in your config become the project codebook, and later config edits do not change it until `potato codebook config.yaml` re-syncs. - Test annotators you create while checking the task are real annotators. They hold assignments and their answers count toward agreement. Wipe `annotation_output/` and restart before handing the task over, and say in the handover which accounts you left behind. ## Known console noise Potato logs these on every phase page (consent, instructions, training, post-study), because a phase page has no instance and the span layer asks for one anyway: ``` GET /api/current_instance 404 [SpanManager] Error fetching current instance ID: Failed to fetch current instance: 404 [SpanManager] Failed to get server instance ID during initialization ERROR potato.flask_server: Error getting instance text: 'null' ``` The 404 is the endpoint's own guard: it refuses to hand back an instance outside the annotation phase, which is right, and the span layer asks anyway. On the annotation page the same call returns 200. Before Potato 2.8.2 a `POST /api/track_annotation_change 400` joined this list, once per answer given on a phase page, and the behavioural trail for survey and training pages was lost. The three tracking endpoints now agree: a missing instance id becomes the `__phase_page__` sentinel. That 400 in your log dates the checkout. They appear on a task that is working correctly, and they have a consequence: **`potato preview --phase consent --screenshot` exits 1 on a healthy task**, while `--screenshot` on the annotation page exits 0 and says "rendered cleanly". Same config, both fine. Judge the PNG and the error list rather than `$?`. Filter them out before treating console errors as signal, and do not spend time chasing them. Real errors worth reacting to, seen from the client: | Client symptom | Meaning | |---|---| | `[NAV] Navigation failed: 400` | The server refused the save. Read the response body — it is JSON with `unsatisfied_schemas` | | `POST /annotate 400 {"status":"validation_error"}` | A required scheme is unanswered; a required span produces this with no inline message | | A scheme renders with no inputs under the heading | Generator failed for that scheme; the startup log names it | | A scheme is missing entirely | Same cause. `Invalid label format: True` in the log means a YAML boolean got into `labels` — see `building-the-ui.md` | ### Widgets that open a native dialog Eight files in Potato call `alert()`, and `error_span` is the one you will meet first: saving an error span without a severity pops "Please select a severity level." Playwright dismisses dialogs automatically, so under automation this reads as a button that did nothing — no error, no console line, no change on the page. If a widget silently refuses to accept input, check for an `alert()` in its schema module before assuming the widget is broken. Register a handler if you want to see them: ```python page.on("dialog", lambda d: (print("DIALOG:", d.message), d.dismiss())) ``` ## The admin surfaces They are not linked from the annotation page, and the JSON APIs are not open. ```bash cat admin_api_key.txt # generated into task_dir at first boot; the log says so curl -H "X-API-Key: $(cat admin_api_key.txt)" localhost:8000/admin/iaa ``` | Route | Auth | What it gives | |---|---|---| | `/admin` | none — serves HTML | The dashboard | | `/admin/iaa` | `X-API-Key` | Agreement per scheme: inferred `kind`, the metrics that follow, `n_overlap_items` | | `/admin/api/agreement` | `X-API-Key` | Also agreement: Cohen's κ, Fleiss' κ and Krippendorff's α per scheme, and an `overall` block averaging them | | `/admin/api/quality_control` | `X-API-Key` | Attention-check and gold-standard pass rates per user | Without the header every JSON route returns `403 {"error":"Admin access required"}`, which reads like a broken build if you do not know the key exists. The key is 43 characters, regenerated only if the file is deleted, and settable yourself with `admin_api_key` in the config. ## Driving it like an annotator A static render cannot show you a conditional question, a drawn span, a restored answer, or whether the workflow reaches its own last page. If a browser tool is available, use it against a running server. Playwright ships with the `[preview]` extra that `--screenshot` needs, so it is normally already installed. This is a working driver. It registers, walks the phases, screenshots each page **full height**, and reports console errors tagged by the phase they came from. ```python import sys from playwright.sync_api import sync_playwright BASE, SHOTS = "http://localhost:8421", "shots" errors, phase = [], ["boot"] def shot(page, tag): page.wait_for_timeout(800) page.screenshot(path=f"{SHOTS}/{tag}.png", full_page=True) # full_page, not the viewport def pick(page, name, value): # click an input by name+value page.evaluate("""([n,v])=>{const e=document.querySelector(`input[name="${n}"][value="${v}"]`); if(!e) throw new Error('missing '+n+'='+v); e.click();}""", [name, value]) with sync_playwright() as pw: b = pw.chromium.launch() page = b.new_context(viewport={"width": 1280, "height": 900}).new_page() page.on("console", lambda m: errors.append(f"[{phase[0]}] {m.text[:150]}") if m.type == "error" else None) page.on("pageerror", lambda e: errors.append(f"[{phase[0]}] {e}")) page.goto(BASE) page.evaluate("switchTab('register')") # login page has login/register tabs # NB: use a fresh username each run. A duplicate registration navigates to # /register, which has no switchTab(), so a re-run throws # "Cannot read properties of null (reading 'classList')" and leaves you # logged out on the sign-in page rather than on the task. for i in page.query_selector_all("form[action='/register'] input"): n = i.get_attribute("name") if n == "email": i.fill("checker@example.com") elif n == "pass": i.fill("pw12345") page.click("form[action='/register'] button[type=submit]") page.wait_for_load_state("networkidle") phase[0] = "consent"; shot(page, "10-consent") pick(page, "consent_agree", "I agree to take part") page.click("button:has-text('Next')"); page.wait_for_load_state("networkidle") ... ``` Selectors that are stable and worth knowing: | Selector | What it is | |---|---| | `#instance_id` | Hidden input holding the current item id. The cheapest proof you moved | | `#next-btn` | Next. There is **no** `#prev-btn` on the first item; Previous appears from item 2 | | `#go_to` + `#go-to-btn` | Jump to instance number N. Use this to navigate back | | `input[name='<scheme>'][value='<label>']` | Radio / likert option. `value` is the raw label, not the humanized one | | `input[name='span_label:::<scheme>'][value='<label>']` | Span label chip; it is a **checkbox** — `.check(force=True)` | | `[name='<scheme>:::text_box']` | Free-text scheme. **Not** `textarea[...]` — a `text` scheme is an `<input>` unless `multiline: true` | | `#span-overlays` | Present only on an ordinary text task. Its children are the drawn spans, and its presence is how you tell the two shapes apart | | `#text-content` | The span anchor on an ordinary text task — but it exists on a span-target task too, hidden. Check the overlay first | | `.span-target-field .text-content` | The per-field anchor once `instance_display` names span-target fields. Overlays sit inside it, as `.span-overlays-field > *` | | `form.annotation-form` | One per scheme; counting these counts the questions on the page | ### Drawing a span Spans are drawn as overlays over the text rather than by wrapping it, so "did a `<mark>` appear" is the wrong check. Count the overlay container's children, in whichever of the two shapes the task renders. An ordinary text task renders one `div#text-content` holding the item text, with `div#span-overlays` inside it. A task whose `instance_display` names span-target fields renders one `div#text-content-<field>.text-content` per field, each with a `div#span-overlays-<field>.span-overlays-field` inside it. **Do not tell the two apart by asking whether `#text-content` exists.** It exists on both. On a span-target task it is a `0×0` hidden div holding `text_key`, carried along beside the real per-field anchors, and Playwright refuses to scroll to it — `scroll_into_view_if_needed` times out with "element is not visible" after thirty seconds, which reads as a hung browser rather than a wrong selector. Anchor on the overlay container instead, which exists in one shape each: ```python anchor = ".text-content" if page.locator(".span-overlays-field").count() else "#text-content" ``` ```python page.locator("input[name='span_label:::sentence_type'][value='Factual reporting']").first.check(force=True) page.locator(anchor).first.scroll_into_view_if_needed() page.mouse.wheel(0, -160) # clear the sticky navbar, or the drag starts on it box = page.evaluate("""(sel)=>{ const c=document.querySelector(sel); const w=document.createTreeWalker(c, NodeFilter.SHOW_TEXT); let n=w.nextNode(); while (n && n.textContent.trim().length < 25) n=w.nextNode(); // skip turn numbers and speaker chips const t=n.textContent, stop=t.indexOf('.'); const r=document.createRange(); r.setStart(n,0); r.setEnd(n, stop>0?stop+1:40); const q=[...r.getClientRects()], a=q[0], z=q[q.length-1]; return {x1:a.left+2, y1:a.top+a.height/2, x2:z.right-2, y2:z.top+z.height/2};}""", anchor) page.mouse.move(box["x1"], box["y1"]); page.mouse.down() for k in range(1, 11): page.mouse.move(box["x1"] + (box["x2"]-box["x1"])*k/10, box["y1"], steps=2) page.mouse.up() ``` The `mouse.wheel` line is not optional. `scroll_into_view_if_needed` puts the text at y≈22, underneath the sticky navbar, and the drag then selects the header instead of the article. That looks like "spans are broken" and is not. The length filter in the tree walker is there for the same reason. A `dialogue` span target starts with the turn number and the speaker name as their own text nodes, so the first node the walker reaches is `[1]` — five pixels wide, and a drag across it selects nothing. **This generalizes to every synthetic drag, spans and canvases alike.** `page.mouse` works in viewport coordinates, so a target below the fold or under the navbar receives nothing: 1. `scrollIntoView({block: 'center'})` on the target element, 2. **re-read** `getBoundingClientRect()` after the scroll, 3. drag, 4. **count the result** — `#span-overlays > *` for spans (`.span-overlays-field > *` per field on a span-target task), the canvas's own "Annotations: N" readout for geometry. A drag that misses produces zero annotations, zero exceptions, zero console output and zero network traffic. It is indistinguishable from the feature being broken, and step 4 is the only thing that tells them apart. ### The four checks worth automating 1. **Conditional schemes appear.** Read the ancestor chain for `display:none`, not just the element: ```python page.evaluate("""s=>{let p=document.querySelector(s), hidden=false; while(p){ const cs=getComputedStyle(p); if(cs.display==='none'||cs.visibility==='hidden'){hidden=true;break;} p=p.parentElement; } return hidden;}""", "textarea[name='misleading_why:::text_box']") ``` A conditional input has a non-zero bounding box while hidden, so width and height prove nothing. 2. **Answers survive navigation.** Answer, `#next-btn`, then `#go_to` + `Go` back. Check the visual state — `input[name=x]:checked`, textarea `.value`, overlay count — not hidden inputs. **Never test this with `driver.refresh()`**: browsers restore form state across a refresh on their own, so a refresh-based check passes even when the server stored nothing. 3. **The workflow reaches its last page.** Answer every item and confirm the post-study page renders. This is where per-annotator quota bugs surface. 4. **The server really stored it.** Read `annotation_output/<user>/user_state.json`: ```python d = json.load(open(f"annotation_output/{user}/user_state.json")) d["instance_id_ordering"] # what they were assigned, in order d["instance_id_to_label_to_value"] # answers per item d["instance_id_to_span_to_value"] # spans with start/end offsets d["training_state"]["passed"] d["phase_to_page_to_label_to_value"] # consent and survey answers d["max_assignments"] # the quota that was actually applied ``` This is the difference between "the widget is on the page" and "the answer is in the file", and it is the claim a researcher cares about. ## What to say in the handover Distinguish what you established from what you assumed. A useful report reads: > Rendered at 1280 wide, full height: consent, instructions, a training question > with feedback, the annotation page, and the post-study survey (`shots/`). > Walked one annotator end to end: training marked 3/3, all six items plus two > attention checks annotated, answers and span offsets present in > `user_state.json`. Conditional follow-ups verified by driving the gate answer; > persistence verified by navigating away and back, not by refresh. > > Not checked: two annotators at once, the agreement pages with real overlap, and > whether an attention-check failure surfaces anywhere an admin would see it. The second paragraph is what makes the first one worth reading. Leave the server running only if that was asked for, and if you do, write down the URL, the port, the pid, how to stop it and how to start it again. A running server nobody can restart is a one-shot demo. <!-- https://davidjurgens.github.io/potato-skill/after-annotators-start/ --> # After Annotators Start Watching a live study, what is still safe to change, and what is not. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Everything in this pack up to here is about the state before anyone has answered anything. Once real answers exist the rules change: some edits are free, one whole class of edit is silently destructive, and the questions the researcher asks you stop being "does it render" and become "how is it going" and "can we still change this". ## Safe and unsafe edits | Change | Effect on work already done | |---|---| | Add items to the data file | Safe. Loaded at the next boot, offered to everyone, progress totals grow | | Edit instructions, descriptions, the banner | Safe. Text only; nothing is keyed to it | | Add or remove annotators | Safe | | Change `num_annotators_per_item`, quotas, assignment strategy | Safe, but it changes who gets what next — read `assignment-and-agreement.md` first | | Add labels to an existing scheme | Structurally safe, **methodologically not**. Everyone who finished chose from the old list | | Add a new scheme | Collects nothing from anyone who already finished. See below | | Rename a scheme | Silently discards the old answers from every report. See below | | Change a scheme's `annotation_type` | Same as a rename, plus the stored values may no longer be the right shape | The dividing line is that Potato tracks completion **per item**, never per question. An item that was annotated stays annotated whatever it was annotated with, so a question added afterwards is never put in front of the people who already worked through the corpus. ## Renaming a scheme mid-study Worth spelling out, because nothing errors and three separate surfaces report three different things. Given a task where three items were annotated under a scheme called `sentiment`, and the config is then edited to call it `polarity` and to add a second scheme `confidence`: - **The annotator** who returns is shown "Thank You! You have completed the annotation task." They never see either question. - **`/admin/api/overview`** reports `completion_percentage: 100.0`, `total_annotations: 3`, one completed user. The study looks finished and healthy. - **`/admin/api/agreement`** reports `polarity: items_count 0` and `confidence: items_count 0`. Every configured scheme has nothing. - **The CSV export** comes out with the column `sentiment.positive`, because exports are driven by what was stored rather than by the config. So the dashboard says done, agreement says empty, and the file says a scheme name the config does not contain. The boot log names it: ``` WARNING Saved annotations name 1 scheme(s) that annotation_schemes no longer defines: sentiment (3 answer(s)). Items already annotated stay annotated... ``` That warning is the whole safety net. Read the boot log after any edit to `annotation_schemes` on a task with data in it. **If you have to add or rename a question after people have started**, there are two honest options. Keep the old name and add the new question as a second scheme, and only later annotators answer it. Or start a fresh `output_annotation_dir` and re-run, and pay for the items twice. There is no migration command, and hand-editing `user_state.json` is the one thing this pack tells you never to do. ## Accounts do not survive a restart by default The default authentication backend is in-memory. Registrations are never written anywhere, so when the server stops, every annotator account goes with it — while their annotations stay safely in `annotation_output/`. The annotator comes back, cannot log in, re-registers under the same username, and is reattached to their saved work, which mostly hides the problem until someone picks a different username and starts the corpus again as a second person. ```yaml authentication: user_config_path: user_config.json # JSONL, one {username, password} per line ``` With that set, registration writes the account out with a salted hash and login survives a restart. Set it on anything an annotator will come back to. It matters most exactly where it is easiest to forget: `render` and `huggingface` restart containers on their own, so a hosted study without it loses its logins without anyone touching the server. ## Watching a live study Every JSON route below needs `X-API-Key`. The key file appears the first time an admin route is requested, not at boot — if you look for it immediately after starting the server it will not be there yet. ```bash curl -s -o /dev/null localhost:8000/admin # makes the key file exist K=$(cat admin_api_key.txt) curl -H "X-API-Key: $K" localhost:8000/admin/api/overview ``` | Question | Route | |---|---| | How far along is the study | `/admin/api/overview` | | Who has done what, and how fast | `/admin/api/annotators` | | Do they agree | `/admin/iaa` (prefer this), `/admin/api/agreement` | | Who is failing the attention checks | `/admin/api/quality_control` | | Is anyone clicking through | `/admin/api/suspicious_activity`, `/admin/annotation-integrity` | | Who is sitting on work they never finished | `/admin/api/stale_assignments` | | What is one annotator's state | `/admin/user_state/<user>` | | What happened to one item | `/admin/item_state/<item_id>` | | Give me the data | `/admin/api/export`, `/admin/api/data/archive` | `/admin/api/quality_control` answers `{"enabled": false, "message": "Quality control not configured"}` rather than erroring when there are no checks, which is a useful way to confirm from outside that the feature really is off. ## Fixing things while it runs | Situation | What to do | |---|---| | An annotator cannot log in | `POST /admin/reset_password` with `{username, new_password}`, or `POST /admin/create_reset_token` to hand them a link | | Someone holds items they abandoned | `GET /admin/api/stale_assignments`, then `POST /admin/api/reclaim_instance` with `{instance_id, username}`. Automatic reclaim is off unless `instance_reclaim` is configured | | One person should do more, or fewer, items | `POST /admin/api/user/<username>/set_instances` with `{max_instances}`; `-1` is unlimited | | Annotations were corrupted by the pre-2.7.2 single-select bug | `potato repair-annotations config.yaml` — dry run by default, `--apply` to write, and it backs up first | | Something hand-edited `annotation_output/` | The same command is the only supported repair, and it only knows about that one corruption | Wiping `annotation_output/` **while the server runs** resets nothing: item and user state are already in memory, and the next arrival gets the completion page. Stop, wipe, start. ## What to say at handover At handover, three facts they cannot work out for themselves: 1. **The admin pages exist and need a key**, and where the key file is. Without this every JSON route returns a 403 that reads like a broken build. 2. **The questions are frozen once people start.** If they are unsure about a label set, now is when it is free to change and next week it is not. 3. **What restarting costs** — nothing, if `authentication.user_config_path` is set; every login, if it is not. <!-- https://davidjurgens.github.io/potato-skill/deploying/ --> # Deploying a Task Putting a task somewhere annotators can reach it. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- `potato start` serves on one machine. Three ways past that, in increasing order of commitment: | Want | Use | |---|---| | A colleague on the same network | `potato start --host 0.0.0.0 -p 8000`, give them the LAN address | | Someone to look at it for an hour | `potato share config.yaml` — temporary public HTTPS | | Annotators working over days or weeks | `potato deploy up config.yaml --provider …` | | Other labs running their own copy | `potato deploy button config.yaml --target …` | `potato deploy` and `potato share` are listed under **other commands** in `potato --help`: `potato deploy --help`, `potato share --help`. ## A temporary public URL ```bash potato share config.yaml -p 8000 --backend cloudflared # or tailscale, ngrok ``` Serves the task and opens a tunnel to it. The URL dies when the command does, so it is for a demo or a pilot with three people, not for a study. It runs a preflight first and asks you to confirm the exposure; `--yes` skips the confirmation and `--skip-preflight` skips the assessment, which you should not do on anything with real annotators behind it. Same rule as `potato start`: it does not return. Background it. `potato deploy up --provider tunnel` exits 2 with a `REFUSED:` line telling you to run `potato share` instead. The tunnel target runs only under `share`. ## Hosted deployment ```bash potato deploy providers # what targets exist, which have credentials potato deploy check config.yaml --provider aws # preflight, changes nothing potato deploy build config.yaml --out ./bundle # assemble, stop potato deploy up config.yaml --provider aws --dry-run # the plan and the cost potato deploy up config.yaml --provider aws # provision potato deploy status config.yaml potato deploy logs config.yaml -f potato deploy pull config.yaml --dest ./collected # get the annotations back potato deploy destroy config.yaml potato deploy list config.yaml # deployments recorded for this config ``` The lifecycle is the same on every target, with two gaps: `potato deploy logs` does not work on `fly` (use `fly logs -a <app>`), and `aws-ecs` has logs but no SSH, because there is no server to log in to. The host runs the image for the Potato version you have installed: on 2.11.1, `up --dry-run` plans `docker.pull ghcr.io/davidjurgens/potato:2.11.1`. Up to 2.10.3 the default was `:latest`, so two `up`s of one study could run different releases; on those versions pass `--image ghcr.io/davidjurgens/potato:2.10.3`, or whichever version you tested the study on. `--image` picks any other tag. ### The fifteen targets `potato deploy providers` on 2.10.3: ``` aws AWS Lightsail: one VM, flat $12/mo for 2 GB with IPv4 and disk; the recommended AWS target aws-ec2 AWS EC2: a t4g.small VM with an Elastic IP, about $18/mo with its root disk aws-ecs AWS ECS Express Mode: managed HTTPS, no server, ~$45-70/mo; ephemeral disk, so it needs --backup digitalocean supports pull fly Fly.io: one Machine + volume on <app>.fly.dev, about $6/mo heroku Heroku container dyno from $7/mo; ephemeral disk, so it needs --backup (sustaining mode since 2026-02) hetzner Hetzner Cloud: a 4 GB VM for about €6/mo, the cheapest durable target huggingface ephemeral filesystem, supports pull linode Akamai/Linode: a 2 GB VM for $12/mo local local only, supports pull openstack OpenStack (Jetstream2, campus clouds): free with an allocation; real hostname on Jetstream2 railway Railway: image + volume on *.up.railway.app, usage-billed (~$10-20/mo) render ephemeral filesystem, supports pull tunnel temporary public URL; stops when `potato share` does vultr Vultr: a 2 GB VPS for $10/mo ``` Quote a researcher the price `--dry-run` prints. It comes from the planner's price table and includes the disk and the IPv4 address. Three extras pull in the provider SDKs: ```bash pip install 'potato-annotation[deploy]' # every target not named below pip install 'potato-annotation[deploy-aws]' # aws, aws-ec2, aws-ecs (boto3) pip install 'potato-annotation[deploy-openstack]' # openstack (openstacksdk) ``` `check` blocks a target whose extra is missing, and names the one to install: ``` ERROR [D021] the openstack target needs openstack, which is not installed -> pip install 'potato-annotation[deploy-openstack]' ``` ## Choosing a target The targets differ on who can reach the task, whether the annotations survive a restart, and what it costs. The rest is provisioning detail. Costs are the `--dry-run` figures at the default size. | Target | What it is | Annotations survive a restart | Cost | Also needs | |---|---|---|---|---| | `local` | Docker on this machine, `127.0.0.1` only | yes | free | Docker running | | `tunnel` | public until the command stops | yes, local disk | free | a tunnel binary | | `aws` | Lightsail VM | yes, on its disk; `--volume-gb` to outlive it | $12/mo | the `deploy-aws` extra | | `aws-ec2` | EC2 `t4g.small` | yes, as above | $18.31/mo | `deploy-aws`; `--subnet` if the region has no default VPC | | `aws-ecs` | ECS Express, a managed container | **no** — only what the backup holds | $47/mo | `deploy-aws`, and `--backup` | | `hetzner` | VM, 2 vCPU / 4 GB | yes | €5.99/mo, billed in euros | — | | `vultr` | VM, 1 vCPU / 2 GB | yes | $10/mo | your IP on the API key's access list | | `linode` | VM, 1 vCPU / 2 GB | yes | $12/mo | — | | `digitalocean` | VM, 2 vCPU / 2 GB | yes | $18/mo default, $6 smallest | — | | `openstack` | VM on Jetstream2 or a campus cloud | yes | your allocation's service units | `deploy-openstack`, a `clouds.yaml` | | `fly` | one Machine and a 1 GB volume | yes, on the volume | $5.85/mo | `--backup` if the project is over 512 KB | | `railway` | one service and a volume | yes, on the volume | usage-billed, $10–20/mo | `--backup`, always | | `heroku` | one Basic dyno | **no**; the dyno's disk is wiped at least daily | $7/mo | `--backup`, or `--demo` | | `render` (free) | web service; stops after 15 minutes idle | **no** | free | `--backup`, always | | `render --plan starter --volume-gb 1` | as above, with a disk | yes, on the disk | $7/mo + $0.25/GB | `--backup`, always | | `huggingface` | a Docker Space | **no**; the backup is always on | free to run | a write token, **and a paid HF plan to create the Space** | The lines that catch people out: - **`render`, `railway` and `aws-ecs` refuse `up` without `--backup`, and `--demo` does not change that.** These hosts run the published image and have no way to receive files, so the project is uploaded to the backup's storage and downloaded when the container starts. No backup means nowhere to put the project. `fly` refuses the same way once the bundle is over 512 KB. Heroku builds its own image, so `--demo` alone is accepted there. - **A free Render instance takes no disk.** `--volume-gb` without `--plan starter` or higher is refused. Without a disk the backup is the only copy of the annotations. - **A Docker Space needs a paid HuggingFace plan** (PRO personally, Team or Enterprise for an org). Restarting an existing Space does not. A free account also runs at most three Spaces, and one over the limit sits `PAUSED` and never wakes for a visitor. - **The $6 DigitalOcean droplet is too small.** `s-1vcpu-1gb` gets a warning in the plan: the image alone is ~840 MB and Potato's working set is numpy/pandas/scipy. `s-2vcpu-2gb` at $18 is the default for that reason. - **A VM without `--domain` gets a certificate for its IP address**, and those are valid for about six days rather than 90. They renew on their own; `potato deploy status` reports a renewal that fails. `openstack --cloud jetstream2` is the exception, because Jetstream2 gives every instance a DNS name. - **`fly` and `railway` delete the volume on `destroy`.** `--keep-data` cannot keep a Fly volume. Pull first. Where each one fits, from Potato's own install guide: an AWS account → `aws`; a US academic with an ACCESS allocation → `openstack --cloud jetstream2`, which costs no money; the cheapest durable VM → `hetzner`; nobody to look after a server → `fly` or `railway`. Heroku has been in sustaining-engineering mode since February 2026. It works, but use it only if the lab already has an account. Four questions decide it, and they are the researcher's to answer, not yours: | Ask | Because | |---|---| | How long does the study run? | An afternoon is `potato share`. Weeks of annotators is `deploy up`. | | Whose account and whose money? | Every public target except free Render bills someone, and `openstack` draws down an allocation. Do not create a billable resource on an assumption. | | Would losing the annotations end the study, or just cost a morning? | This is the whole ephemeral/durable choice, and it is much cheaper to answer now. | | Is anything in the data or the survey answers identifiable? | Decides whether a public URL with open sign-up is acceptable at all. | ## Backups and restore Any target can mirror what Potato collects to storage the researcher owns. On `aws-ecs`, `heroku`, `render` and `huggingface` that copy is the only one that outlives a restart. On a VM it is a second copy for no extra cost. ```bash potato deploy up config.yaml --provider heroku --backup hf --hf-token $HF_TOKEN potato deploy up config.yaml --provider heroku --backup s3 --s3-bucket my-bucket potato deploy up config.yaml --provider fly --backup s3 --s3-bucket my-bucket \ --s3-endpoint https://<account>.r2.cloudflarestorage.com # R2, B2, MinIO potato deploy up config.yaml --provider railway --backup hf,s3 --hf-token $HF_TOKEN \ --s3-bucket my-bucket --backup-minutes 10 ``` | Flag | Default | |---|---| | `--backup hf\|s3\|hf,s3` | none. Without it, `--hf-token` alone means `--backup hf` and `--s3-bucket` alone means `--backup s3` | | `--hf-token` | `HF_TOKEN`. Needs write access | | `--hf-backup-repo` | `<account>/<name>-annotations`, created private | | `--s3-bucket` | none; the bucket must already exist | | `--s3-prefix` | `potato/<name>` | | `--s3-region`, `--s3-endpoint` | boto3's defaults; the endpoint is for S3-compatible stores | | `--backup-minutes` | 5 | S3 credentials come from `POTATO_S3_ACCESS_KEY_ID` and `POTATO_S3_SECRET_ACCESS_KEY` and end up in the server's environment, where anyone with a shell on the host can read them. Use a key that reaches only this bucket. **What it copies:** the annotation output directory file by file (every `user_state.json`, plus `user_config.json`, which holds the accounts), and snapshots of `project.sqlite` and `datasets.sqlite` taken with SQLite's backup API rather than copied as live WAL files. Only changed files are uploaded each cycle. **Restore is automatic.** When the server starts and its output directory holds no `user_state.json`, it downloads the latest backup before it loads anything. It never overwrites: a disk that already holds annotations is treated as the authoritative copy. The log says which happened: ``` Restored project.sqlite from backup RESTORED 5 file(s) from the s3://probe-bucket/potato/probe backup into an empty task. This is expected after a restart on a host without a persistent disk. Loaded user data for 1 users ``` That is from a local run against an S3-compatible server: three items annotated, the task directory deleted, the server started again. The annotator logged in with the original password and landed on the fourth item. `potato deploy up` writes the `backup:` block into the bundled config, never into yours. On a server you run yourself, add it by hand: ```yaml backup: schedule_minutes: 5 restore_on_boot: true sinks: - type: huggingface repo_id: lab/pilot-annotations - type: s3 bucket: my-bucket prefix: potato/pilot region: us-east-1 ``` The older `huggingface_backup:` block still works and is read as one HuggingFace sink, **with `restore_on_boot` off**, so a config that relied on it backs up but does not restore. Move it to `backup:` if the host's disk does not last. On an S3-only backup, the project link the container downloads at start is a presigned URL valid for at most seven days. A host with a disk downloads it once per deploy. A host without one (`render` free, `aws-ecs`) downloads it on every restart, so a week later a restart fails until you run `up` again. The plan warns about this; use `--backup hf` on those hosts. ## Walking someone through a first deploy Six steps. The first three cost nothing and touch no provider, so run all three before asking anyone to make an account. **1. Find out what is already configured.** ```bash potato deploy providers # targets, and which have a credential potato deploy providers --verify # ask each one whether the token really works ``` Credentials are found without being configured for Potato: a HuggingFace token in the `huggingface_hub` cache, a single-context `~/.config/doctl/config.yaml`, the key `heroku login` writes to `~/.netrc`, an AWS profile or SSO session, and a `clouds.yaml` are all picked up. Often the answer to "which provider" is "the one they are already signed in to". `--verify` matters because an expired, read-only or newline-terminated token looks exactly like a good one until `up` is several resources deep. **2. Get a token, if the chosen target needs one.** Nothing is ever written to disk. The token is resolved per invocation from the environment, so a leaked project directory is not a leaked cloud account. | Provider | Credential | Notes | |---|---|---| | `aws`, `aws-ec2`, `aws-ecs` | the AWS chain: `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`, `AWS_PROFILE`, `~/.aws/`, SSO | `--aws-profile lab` picks a profile and is remembered for `status`, `pull` and `destroy`; `--region` defaults to `us-east-1` | | `digitalocean` | `DIGITALOCEAN_TOKEN` | read **and** write scope | | `fly` | `FLY_API_TOKEN` | `fly tokens create org`, or a personal token | | `heroku` | `HEROKU_API_KEY`, or `heroku login` | | | `hetzner` | `HCLOUD_TOKEN` | project token with read and write | | `huggingface` | `HF_TOKEN` | write access | | `linode` | `LINODE_TOKEN` | read/write on Linodes, Firewalls and Volumes | | `openstack` | `clouds.yaml` plus `--cloud <entry>`, or `OS_CLOUD` | Jetstream2: an application credential from Horizon, saved to `~/.config/openstack/clouds.yaml` | | `railway` | `RAILWAY_API_TOKEN` or `RAILWAY_TOKEN` | | | `render` | `RENDER_API_KEY` | | | `tunnel` | `NGROK_AUTHTOKEN` | ngrok only; cloudflared quick tunnels need no account | | `vultr` | `VULTR_API_KEY` | every request returns 401 until your IP is on the key's access list | | `local` | — | | For the token-based providers, `POTATO_DEPLOY_TOKEN_<PROVIDER>` overrides the variables above, and `--token` overrides that. `potato deploy up` without a token prints where to get one for the provider you asked for, so you can also just run it and read the error. **3. Run the preflight and the dry run.** ```bash potato deploy check config.yaml --provider aws potato deploy up config.yaml --provider aws --dry-run ``` `check` reports what the deployment exposes (next section). `--dry-run` is the other half: it builds the bundle, then prints the provisioning steps, the URL the task will end up at, and **the monthly cost**, without credentials and without touching the provider. ``` 14 step(s): 1. aws.identity confirm the credentials with sts:GetCallerIdentity 2. ssh.keygen generate an ed25519 deploy key, delivered by cloud-init 3. lightsail.bundles check that bundle small_3_0 is still sold 4. lightsail.instance create potato-deploy-probe (small_3_0, ubuntu_24_04) in us-east-1a 5. state.persist record the instance name before anything else can fail 6. wait.active poll until the instance is running 7. lightsail.static_ip allocate potato-deploy-probe-ip and attach it, so the address and its certificate survive a stop/start 8. lightsail.ports open 22/80/443 only; never 8000 9. wait.ssh poll until the host accepts an SSH connection 10. wait.cloud_init wait for docker install and image pull to finish 11. ssh.upload upload the bundle to /opt/potato/app 12. ssh.env write /opt/potato/potato.env at mode 0600 13. ssh.start systemctl start potato and caddy 14. wait.http poll https://<instance-ipv4>/health Result URL: https://<instance-ipv4> Estimated cost: $12.00/month WARNING: No --domain, so TLS uses a Let's Encrypt certificate for the instance's IP address. These are valid ~6 days rather than 90, so check `potato deploy status` if the study runs unattended. WARNING: No --volume-gb. Annotations live on the instance's own disk, which is destroyed with the instance. Pull before you destroy. ``` When `up` would refuse, the plan is followed by a `REFUSED:` line naming the reason and the flags that fix it, and the command exits 2 instead of 0. The dry run and the real `up` run the same check, before any provider is called: ``` 2. bundle.publish upload the project tarball to (nowhere; see REFUSED below); the container fetches it at start ... REFUSED: Render runs the published image and fetches your project into it at start, so the project has to be uploaded somewhere first. It goes to the backup's storage, so --demo is not an option here: pass --backup hf --hf-token <token> (recommended: the link never expires) or --backup s3 --s3-bucket <bucket>. On the free plan the backup is also the only copy of the annotations, since the instance stops after 15 minutes idle and loses its filesystem. ``` Show the researcher this output before spending their money. It is the only place the cost, the URL and the durability warnings appear together, and it reaches no provider. Building the bundle writes it under `.potato/bundle/<provider>/<name>/` in the task directory. That is gitignored in Potato's own tree, but it is a real directory of copied project files, so a dry run against four providers leaves four copies behind. **4. Deploy.** ```bash potato deploy up config.yaml --provider aws --volume-gb 10 ``` It prints the plan again and asks for confirmation; `--yes` skips the prompt and is for scripts, not for the first time. `--force` proceeds despite preflight *errors* and should never be used against a public host. If it fails partway, the deployment is still recorded — `state.persist` comes before anything expensive can fail — so `potato deploy status`, `logs` and `destroy` all work on a half-built deployment. Do not retry by hand-deleting resources in the provider's console. **5. Prove it works**, with the steps under **After it is up** below. A bundle that omitted a side file fails on the host having worked locally, so the walk-it-as-an-annotator step is not optional just because it passed at home. **6. Hand it over.** The URL, who can sign in, where the annotations live, how to pull them, and how to take it down. Say which of these steps you actually ran and which you only prepared. ## Flags by target `potato deploy up --help` lists them all on one screen without saying which target reads which. Grouped: | Flag | Read by | |---|---| | `--size` | every target with a machine size: a Lightsail bundle (`small_3_0`), an EC2 type (`t4g.small`; `t3.small` for x86), a Heroku dyno (`basic`, `standard-2x`), a Fly memory size in MB, a server type, or an OpenStack flavor | | `--volume-gb` | the VM targets, `fly` (default 1) and `render --plan starter`. Puts the task on a disk that outlives the instance | | `--acme-email` | the VM targets: the Let's Encrypt contact. Defaults to `git config user.email` | | `--domain` | the VM targets: point an A record at the IP first, and you get a 90-day certificate | | `--aws-profile`, `--region` | the three AWS targets (`--region` also picks a Fly or VM region) | | `--subnet` | `aws-ec2`, when the region has no default VPC | | `--cloud`, `--network`, `--os-image` | `openstack` | | `--heroku-registry` | `heroku`: build locally with Docker and push to Heroku's registry, for when the build on Heroku fails | | `--plan` | `render` (`free`, `starter`, `standard`) | | `--owner` | `huggingface` and `fly` (the organization) | The VM targets are `aws`, `aws-ec2`, `digitalocean`, `hetzner`, `vultr`, `linode` and `openstack`. All seven provision the same way: a deploy key made for that deployment alone, a firewall open on 22/80/443 and never 8000, Docker, Caddy in front, and Potato as a systemd service. `logs`, `pull` over SSH and `--volume-gb` work on all of them. Per-target traps: - **`hetzner`**: the default `cx` types are sold only in the European locations. In `ash` or `hil`, pass a `cpx` type with `--size`, or Potato stops before creating anything and lists where your type is sold. - **`openstack`**: `--cloud jetstream2` is a preset for the flavor (`m3.small`), the image (`Featured-Ubuntu24`) and a real hostname, `potato-<name>.<allocation>.projects.jetstream-cloud.org`. Any other `clouds.yaml` entry gets `m1.small`, `Ubuntu 24.04` and an IP certificate. An `m3.small` costs 2 service units an hour, about 17,500 a year if nobody destroys it. A project without an auto-allocated network needs `--network`. - **`aws-ecs`**: Potato switches the service to stop the old task before starting the new one, so two tasks never restore and back up the same study at once. A redeploy takes the task offline for a minute or two. If AWS changes the setting back, the next `up` refuses; `destroy` and `up` again, pulling first. The two IAM roles it creates are shared and left in place by `destroy`. - **`heroku`**: `--size eco` sleeps after 30 minutes idle and restores from the backup on every wake. `R14 (Memory quota exceeded)` in the logs means `--size standard-2x`. ## The preflight `potato deploy check` is the part of this worth learning. It never touches a provider, it takes a second, and it reports what the deployment exposes: ``` $ potato deploy check config.yaml --provider render WARNING [D003] Anyone who finds the URL can register and annotate. -> Set user_config.allow_all_users: false and list your annotators under user_config.users, or use authentication.method: oauth with at least one entry under authentication.providers. WARNING [D011] The render filesystem is ephemeral: annotations are lost when the host restarts or redeploys. -> Pass `potato deploy up` --backup hf (with --hf-token) or --backup s3 --s3-bucket <name> so the data is mirrored off the host and restored after a restart, or add a backup: block to the config. note [D012] No secret_key is set, so one will be generated and injected as POTATO_SECRET_KEY. Sessions survive restarts only because of it. Exposure: Reachable from the public internet: yes Sign-in: open — anyone with the URL can create an account Password required: yes Admin access: via the generated admin API key (injected as POTATO_ADMIN_API_KEY, not written into the bundle) Preflight: 0 error(s), 2 warning(s) PASS — safe to deploy ``` Exit codes: **0 for PASS, 2 for BLOCKED.** Errors block; warnings do not. The one that blocks most often: ``` ERROR [D002] debug is true. With debug_phase it signs every visitor in as one shared debug_user and skips them to that phase, and it opens the admin pages without a key to requests from the server's own machine. It is for a server on a developer's machine. -> Remove `debug: true` from the config before deploying. ``` If you turned on `debug` to check the interface, take it out. It is not a logging flag: it can sign every visitor in as the same user, and it lets the host's own machine into the admin pages without a key. Run `check` against the provider you actually intend to use. `local` reports no exposure and no durability problem, so a clean `check` with the default provider tells you nothing about hosting it. Add `--private` when the host will not be public (behind a VPN, on an internal network) so the assessment is not arguing about exposure you have already handled. ## What ships `potato deploy build` assembles a bundle and stops, which is the cheap way to see what would be uploaded: ``` Bundle: ./bundle 20 files, 152.6KB sha256 f70b22ad201b4e37 ``` Two things to know about it: - **It copies the whole task directory.** Screenshots in `shots/`, a stray `config_unsafe.yaml`, notes, scratch data — all of it goes. Clean the directory before building, and look at the file list rather than the count. - **`annotation_output/` is stripped** to an empty `.gitkeep`. Annotations you collected while testing do not ship. Verified by putting a `user_state.json` in there and rebuilding. The size matters on `fly`: a bundle up to 512 KB travels inside the Machine's configuration and needs no storage, and a larger one needs `--backup`. The bundle also carries the generated `layouts/`, so the host does not regenerate them. Delete `layouts/` and rebuild if you changed schemes and the deployed page looks stale. Secrets are **not** written into the bundle. The secret key and admin API key are generated and injected as `POTATO_SECRET_KEY` and `POTATO_ADMIN_API_KEY` environment variables at deploy time. Pass your own with `--env KEY=VALUE` and `--secret KEY=VALUE`. ## Deploy buttons `potato deploy button` writes the files a host reads to offer a one-click deploy of the task from its git repository, and prints the README badge. It is for other people running their own copy in their own accounts, not for hosting the study you run. ```bash potato deploy button config.yaml --target heroku --backup hf \ --hf-backup-repo lab/pilot-annotations --dry-run # print the files, write nothing ``` | `--target` | Writes | Needs | |---|---|---| | `heroku` | `app.json`, `heroku.yml`, `Dockerfile.potato` | `--backup` | | `render` | `render.yaml`, `Dockerfile.potato` | `--backup` | | `aws` | `potato-lightsail.cfn.yaml`, a CloudFormation stack for Lightsail | `--backup`, a public repository, and the template uploaded to S3 and passed back as `--template-url` | | `railway` | nothing; it prints the steps for publishing a template from the Railway dashboard | — | Without `--backup`, the heroku, render and aws targets exit 1 before writing anything. All of them also write `potato.deploy.yaml` beside your config, a copy with `debug: false`, `persist_sessions: true` and the backup block, and `.dockerignore` if the repository has none. Your `config.yaml` is not changed. No secrets are written. Read the warnings it prints before you commit. It lists anything that should not reach a public repository and is not gitignored (`.potato`, which holds the admin key, plus `annotation_output` and `project.sqlite`). With no `origin` remote, the badge carries a `<repo-url>` placeholder for you to fill in. Existing files are kept unless you pass `--force`. ## Behind a reverse proxy "We'll put it at example.edu/annotate" is a request for a path prefix, and Flask has to be told about it or every asset URL comes out rooted at `/`. Two switches, both environment variables rather than config keys: ```bash # the proxy sets headers (nginx: proxy_set_header X-Forwarded-Prefix /annotate) POTATO_PROXY_FIX=1 potato start config.yaml # the proxy cannot; name the prefix yourself POTATO_URL_PREFIX=/annotate potato start config.yaml ``` Both converge on the WSGI `SCRIPT_NAME`. `POTATO_PROXY_FIX` also reads `POTATO_PROXY_FIX_X_FOR`, `_X_PROTO`, `_X_HOST` and `_X_PREFIX`, for saying how many proxies deep the server sits. Potato's own `docs/deployment/reverse-proxy.md` is the upstream page. Driven on 2.8.2-26 behind a proxy mounting a task at `/annotate`: under either switch the annotation page renders styled and `url_prefix` reaches the page as `/annotate`. Walk the whole sign-in on the **proxied** URL before you hand it out, and do it in a browser: a prefix that reaches the stylesheet has not necessarily reached every form on the page. ## Before you expose anything The preflight will say all of this, but decide it before you type `up`: | Question | Where it is answered | |---|---| | Who can sign in? | `user_config.allow_all_users: false` plus `user_config.users`, or `authentication.method: oauth`. The default lets anyone with the URL create an account | | Is `debug` off? | It must be. It can sign everyone in as one user and opens admin to the host without a key | | Do sessions survive a restart? | `secret_key`, or let deploy generate one | | Where do annotations live if the host restarts? | A volume (`--volume-gb`), or `--backup` with restore on boot | | Who holds the admin key? | Generated per deployment; `potato deploy status` and the provider's env | | Is anything identifiable in the data or the survey answers? | `export_include_phase_data` defaults to false for a reason | ## Getting the data back ```bash potato deploy pull config.yaml --dest ./collected ``` Every provider supports pull. Do this **before** `destroy` — `destroy` refuses to run without a prior successful pull unless you pass `--force`, which is a deliberate guard rather than an annoyance. `--keep-data` on destroy keeps the provider-side volume where the provider allows it; on `fly` it cannot. `destroy` never touches the backup dataset or bucket. `pull --allow-empty` records a pull that returned nothing, for when you know the task collected nothing and want `destroy` to proceed. ## After it is up Deploying is not verifying. The same standard as `running-a-task.md` applies, and the failure modes are worse remotely because you cannot read the log by looking sideways: 1. `potato deploy status config.yaml`. It should say provisioned and running. 2. `potato deploy logs config.yaml --lines 200` (`fly logs -a <app>` on Fly) — the **startup log**, with the same lines you grep locally: `Loaded N training instances`, `Loaded N attention check items`, any phase errors, and on a host with a backup, `Backup: … every N minute(s)`. A feature that loaded locally can fail on the host if a side file was excluded from the bundle. 3. Open the URL and walk the study as an annotator: register, consent, one item, navigate away and back. 4. Run `potato deploy pull` once, early, and check the files are what you expect. Finding out that pull does not work after four weeks of annotation is the expensive version of this mistake. With a backup, open the dataset or bucket too and check a `user_state.json` has arrived. Then tell the researcher the URL, who can sign in, where the annotations live, how to pull them, and how to take it down. A deployment nobody can pull from or destroy is worse than a local server. ## What I have and have not verified Run on 2.11.1 (`cf0873a0`; the retagged `68880e8c` changes only wording), from a checkout, with Docker stopped: `up --dry-run` on `local`, `render` and `railway` plans the 2.11.1 image, which is published on ghcr.io; `check` with `debug: true` blocks on D002 with exit 2, quoted above; `check` on `openstack` without its SDK still blocks on D021. Run on 2.11.0 (`eb284819`), from a checkout, with Docker stopped: - `up --dry-run` on the fourteen targets other than `tunnel`. The three whose plans name the image, `local`, `render` and `railway`, plan the 2.11.0 image and none plans `latest`. The image tag `2.11.0` is published on ghcr.io; - `check` on the eleven VM and container targets. `openstack`, whose SDK was not installed, blocked on D021 with exit 2 and `ok: false` in `--json`; the others passed. Run on 2.10.3, installed from PyPI, with `--image ghcr.io/davidjurgens/potato:2.10.3`: - the whole lifecycle on `local`: `up`, `status` (`healthy True`), `logs`, a three-item annotator walk, `pull` (one annotator, `project.sqlite`, and the three answers in `user_state.json`) and `destroy`, which removed the container; - the provider list and credential detection, from `potato deploy providers`. Run on the re-cut 2.10.2 tag (`1a9235b1`). The only deploy change between it and 2.10.3 is the button command's HuggingFace account lookup, which was run again on the 2.10.3 release candidate: - `--dry-run` for all fifteen targets, which reaches no provider. That covers the plans, the costs above, the jetstream2 preset's flavor and hostname, and the `REFUSED:` exit 2 for heroku, aws-ecs, render and railway without a backup, for fly with a 1 MB bundle, for render with `--volume-gb` on the free plan, and for tunnel; - `check` output and exit codes, the `debug` block, and what `build` includes and strips; - the button command for all four targets with `--dry-run`, including its exit 1 without `--backup`. Run on 2.10.1 (`6b281231`) and not repeated: - the subcommands and flags, from `--help`. `tests/test_deploy_flags.py` still checks every flag this page names against the parser; - backup and restore on boot, end to end, with an S3 sink pointed at a local S3-compatible server. That run is quoted in **Backups and restore**. Read from the code but not run: that a real `up` reaches the same refusal check as the dry run, before calling the provider (`cmd_up` checks it ahead of the `--dry-run` branch, after the credential check); that the legacy `huggingface_backup:` block maps to a sink with `restore_on_boot` off; and that every other target takes the same default image as the three dry runs that print it (they all read `DEFAULT_IMAGE` in `potato/deploy/image.py`). The prices are the planner's own tables, not a quote from anyone's billing page. Not verified here: an actual `up` against any cloud provider, and therefore the provisioning, DNS, TLS, the HuggingFace sink and pull behaviour of a live host. Provisioning costs money and creates real resources, so do it deliberately and with the researcher's knowledge. In a handover, say which of these steps you ran and which you only prepared. <!-- https://davidjurgens.github.io/potato-skill/troubleshooting/ --> # When a Task Misbehaves Symptoms that validate clean and then do something else. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Ordered by where you are when it happens. Everything here was hit for real on potato-annotation 2.7.0. ## The command | Symptom | Cause | Fix | |---|---|---| | `potato --help` seems not to list `validate` or `preview` | They are under **other commands**, after the long `start` flag list | `potato validate --help`, `potato preview --help` | | A command never returns | `potato start` holds the foreground until killed | `nohup potato start … &` and poll for a 200 | | `validate --strict` prints "Re-run with --strict to fail on unknown keys" while you are running with `--strict` | Wrong message; the exit code is correct | Trust `$?`, not the sentence | | `--strict` reports nothing about unknown keys | Warnings are suppressed while there are hard errors | Fix the errors and run again | | Killing the server leaves the port held | It spawns children | `pkill -f "potato start <config> -p <port>"`, or `killpg` on the process group | | `timeout` not found (macOS) | Not installed | Poll in a loop, or use a Python wrapper | | `preview` prints `ERROR: Missing required field 'task_dir'` on a config `validate --strict` just passed | `preview` reads the raw file; the loader defaults `task_dir` to the config's own directory, so the server runs fine | Write `task_dir: .` and stop thinking about it. Every other relative path resolves against it anyway | ## Boot | Symptom | Cause | Fix | |---|---|---| | `ConfigSecurityError: Path … outside the project directory` | Running from the wrong directory, or a path escaping `task_dir` | `cd` to the directory `task_dir: .` refers to | | `Training data must be a JSON object` | The training file is an array | Wrap it: `{"training_instances": [...]}` | | `Training instance missing required fields: {…}` and it names none | Missing one of `id`, `text`, `correct_answers` | See `quality-control.md` | | `WARNING: Attention check item missing required fields` → `Loaded 0 attention check items` | Items lack `id` or `expected_answer`. **Boot still succeeds** | Fix the file; re-check the count | | `Gold standard item skipped, missing gold_label` | Exactly what it says | Add `gold_label` | | `Phase 'x' in order but not defined in phases config, skipping` | Listed in `order`, never described | Add the phase entry | | `Failed to load phase 'x': Unknown phase: y` | `type:` is not one of the six | `consent`, `instructions`, `training`, `annotation`, `poststudy`, `prestudy` | | `Phase x requires 'instrument', 'instruments', or 'file'` | Non-annotation phase with no page | `file: pages/x.jsonl` | | `Failed to load phase 'x': unhashable type: 'list'` | `instrument:` got a list of scheme dicts | It takes a **string**; use `file:` instead | | `` `on:` at X was read as the boolean True `` | YAML reads bare `on` as a boolean | Quote it: `"on": true` | | `Invalid label format: True` from a label list | Same trap in `labels:`. YAML reads bare `Yes`, `No`, `On`, `Off`, `Y`, `N` as booleans, so `labels: [Yes, No]` becomes `[True, False]` | Quote them: `labels: ["Yes", "No"]`. The scheme renders as a heading with no inputs until you do | | `ai_support.endpoint_type is required when ai_support is enabled` | Enabled without a model | Set `endpoint_type` | | `adjudication.adjudicator_users must be a non-empty list` | Enabled with no adjudicator | Name one | | `gold_standards.mode must be one of: training, mixed, separate` | Invented mode | Use one of the three | | `training.feedback must be a dictionary` / `passing_criteria` / `accuracy` | Passed a bare `true` or a number | `{show_explanations: true}`, `{min_correct: 3}` | | `attention_checks: specify either 'frequency' or 'probability', not both` | Both set | Pick one | | `Configuration has both top-level 'annotation_schemes' and phase-level 'annotation_schemes'` | Mixing the two | All-or-nothing; use `file:` for phase pages | ## The rendered page | Symptom | Cause | Fix | |---|---|---| | A scheme is missing from the screenshot | Behind `display_logic`; the initial render does not satisfy it | Comment out the condition, render, put it back — or drive the gate in a browser | | The item shows a file path instead of an image | No `instance_display` | Add one with the right display type | | A span scheme with nothing to highlight | No field has `span_target: true` | Set it, and `target_field` on the scheme | | `span_target is set but type X does not support span annotation` | Wrong display type | The error lists the twelve that accept it | | Labels read "Information Or Advice" | `humanize_labels` is on by default and title-cases prose labels | `humanize_labels: false` | | A keyboard shortcut does nothing | The key is `key_value`, not `key_binding`, and nothing validates it | `potato preview --format json` — an empty keybinding list means the label key is misspelled | | `preview` reports keybinding conflicts | Real for explicit `key_value`, spurious for `sequential_key_binding` (which the server reallocates) | Press the key on a live page. Nothing warns at startup | | A shortcut does nothing and nothing warns | Two schemes claimed the same `key_value`; the second one loses silently | `potato preview` is the only reporter | | A canvas scheme renders an empty drawing area | No `instance_display` field of that media type. `source_field` names the data key, it does not fetch the media | Add the display field; expect the media to render twice | | The image appears twice on an image task | Correct behaviour: display field plus canvas copy | Hide the display copy with `base_css` if it bothers you | | `No point cloud for this item: the "X" field is empty` on a field that is not empty | `spatial_annotation` reads the path off the rendered page, and a display `label:` ends up in the same text node, which disqualifies it | Display the field with no `label:`, or make the cloud path the item's `text_key` | | A `region_caption` panel that never lists a region, or a `grounding_eval` whose only answer is "not present" | Neither draws; both need an `image_annotation` scheme beside them | Add the canvas scheme — `modalities.md` has it | | `+ New event` on `multi_document_event` does nothing, forever | No top-level `event_template:` block, so `/corpus/api/event` 404s. Nothing appears on the page | Add `event_template: {enabled: true, slots: [...]}` | | The waveform zoom buttons do nothing | `view.getZoom` does not exist in the bundled Peaks build; both audio and video widgets call it | Nothing to configure. *Fit* works; the rest is broken on 2.8.2-11 | | A free-text answer arrives as one cramped line | `text` defaults to `<input>` | `multiline: true` | | A labelled Likert renders as radio buttons | `Complex labels detected … using radio layout` | Cosmetic. The metric keys off `annotation_type`; check `kind` in `/admin/iaa` | | Whole phase pages render bold | Phase prose sits in a `<legend>`, weight 600 | `base_css` with `form.pure-display legend { font-weight: 400 }` | | Item cards are enormous for one line of text | `instance_display.resizable` defaults on | `resizable: false` | | The questions are below the fold on every item | `annotation_instructions` is long and expanded by default | Move definitions to an `instructions` phase; keep the banner short | | Instructions banner appears on the consent page | It is page-level, on every page | Word it to survive that, or shorten it | | A grid scheme is squeezed into a narrow column | Auto layout | `layout` on the scheme, or `instance_display.layout` | ## Console and network | Symptom | Cause | |---|---| | `GET /api/current_instance 404` + `[SpanManager] Failed to get server instance ID` on consent/instructions/poststudy | Normal. Phase pages have no instance; the span layer asks anyway | | `GET /api/spans/null 404`, `POST /api/track_annotation_change 400` on phase pages | Same cause. Also benign | | `potato preview --phase consent --screenshot` exits 1 on a task that is fine | Same cause: those three lines are counted as things to report. The annotation page exits 0. Judge the PNG, not `$?` | | `--phase prestudy` refused: `phase must be one of (…)` | The preview CLI knows five phases; the server knows six. Walk a prestudy page in a browser | | `ERROR potato.flask_server: Error getting instance text: 'null'` | Same cause, server side | | `[NAV] Navigation failed: 400` | The server refused the save; body is JSON with `unsatisfied_schemas` | | `400 {"status":"validation_error","unsatisfied_schemas":["x"]}` after answering everything visible | A required **span** scheme with no span drawn. The client cannot validate spans, so the only feedback is a corner toast naming the internal scheme name | | A scheme renders as a heading with no inputs | Its generator failed; the startup log names it | ## Assignment and data | Symptom | Cause | Fix | |---|---|---| | Annotators never see the last items | `max_annotations_per_user` is set below the item count, or `per_annotator_quota` is capping them lower still | Raise whichever applies; `per_annotator_quota` wins, so raising the global alone changes nothing | | `instance_per_annotator` has no effect | It is not read; `automatic_assignment` sub-keys are unvalidated | Use `max_annotations_per_user`, or `per_annotator_quota` for per-annotator caps | | Agreement pages are empty | No overlapping annotations yet | Not a config problem | | Every admin JSON route returns `403 {"error":"Admin access required"}` | They need `X-API-Key` | `cat admin_api_key.txt` in `task_dir`; `/admin` HTML is open, the APIs are not | | One annotator got fewer items than the others | Usually a quota set below the item count | Count annotators per item across **all** `user_state.json` files, not one | | Wiping `annotation_output/` changed nothing, and new annotators get "Thank You!" immediately | State is in memory; the wipe happened while the server was running | Stop, wipe, start | | A new annotator gets "Thank You!" on a server that was never wiped | `items x num_annotators_per_item` is used up, so there is no work to hand out | Raise `num_annotators_per_item`, or add items. No log line marks this | | A re-run of a browser driver lands on the sign-in page | Duplicate registration navigates to `/register`, which has no `switchTab()` | Use a fresh username per run | | `gold_standards.auto_promote: false` / `accuracy: 0.7` rejected | Both must be dictionaries despite being typed `boolean\|object` and `number\|object` | `{enabled: false}`, `{min_threshold: 0.7}` | | Annotations look wrong after a restart | Something hand-edited `output_annotation_dir` | Never edit it; the server rewrites those files whole | | Edited schemes but the page looks the same | `layouts/task_layout_*.html` is cached by hash | `rm -rf layouts` and restart | | A codebook holds labels you already changed | The first boot seeded `project.sqlite` from the config | `potato codebook config.yaml` | | Test annotators show up in the researcher's data | They are real annotators holding real assignments | Wipe `annotation_output/` before handover | ## The form | Symptom | Cause | |---|---| | A `layout` group renders with fewer questions than it lists | Group membership used to match on `data-schema-name`, which 16 generators (`slider`, `span`, the canvas and media schemes, `multi_document_event`) never render. The missing ones sat loose above the group. Fixed; on an older build, put those schemes outside groups | | An item is "Labeled" before anyone answered | `slider`, `vas` and `soft_label` render at a starting position and the sync used to store it. Fixed; on an older build, treat a midpoint answer as possibly untouched, and check `annotation_telemetry` for whether anyone interacted | | A required `slider` refuses to let anyone past | The requiredness check wanted `data-modified`, which nothing set on a range input, so dragging it never satisfied the requirement. Fixed; on an older build, do not mark a slider scheme required | | A slider snaps to its minimum when the instance changes | `clearAllFormInputs` reset to the `min` attribute, and `starting_value` is not an attribute, so it never won. Fixed with the above | | Slider tick labels overlap and the value bubble covers them | The scheme is in a narrow grid column. Give it `layout: {columns: 2}` or leave it full width | | Two schemes both claim key `1` and only one fires | Explicit `key_value` is never reallocated. Only `preview` reports it | ## Verification traps | Trap | Why it lies | |---|---| | Testing persistence with a page refresh | Browsers restore form state across refresh; the check passes when the server stored nothing. Navigate away and back instead | | Checking a conditional input with width/height | Hidden inputs still have a bounding box. Walk the ancestor chain for `display:none` | | Treating `--screenshot` exit 0 as "the UI is fine" | It renders one page, viewport only, and phase-page console noise is filtered out of your attention | | Trusting `validate` on phases, surveyflow or scheme entries | It does not look inside them | | Trusting `validate` on the paths a config names | It checks them now, but at two severities. A missing `base_css`, `header_logo` or data file is an error; a missing `keyword_highlights_file` is a warning, because the server starts anyway with the feature off. `--strict` makes both fatal, so run it | | Driving a span drag after `scroll_into_view_if_needed` | Puts the text under the sticky navbar; the drag selects the header. Scroll up ~160px first | <!-- https://davidjurgens.github.io/potato-skill/reporting-upstream/ --> # Reporting a Bug or Asking for a Feature Telling a config mistake from a defect, and filing the ones that are real. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- When a task will not do what the researcher needs, the answer is one of three things, and only the third belongs on GitHub as a feature request. | What it is | How you know | What to do | |---|---|---| | A config mistake | `--strict` fails, or the boot log says `Loaded 0 <something>` | `troubleshooting.md`. Fix it yourself | | A bug in Potato | It does something other than what it documents | File a bug | | A gap | It does exactly what it documents, and that is not enough | File a feature request | The order matters. The first is by far the most common, it is the only one you can fix in the next minute, and a bug report that turns out to be a typo costs a maintainer more than it costs you. ## Before you call it a bug Three checks, none of which take long. **Run `potato validate config.yaml --strict`.** A misspelled key inside a block Potato treats as opaque is accepted, ignored, and never mentioned again, so the feature it was meant to configure simply does not happen. Without `--strict` that is a warning you will not read. This is the single most common cause of "the config is right and nothing happens". **Read the boot log.** `Loaded 0 training items`, `Loaded 0 attention checks` and their siblings mean the side file was found and rejected, usually for a missing required field. The count is the diagnosis. **Check what you measured, not what you inferred.** A rendered page tells you what an annotator sees and not what survives; `user_state.json` tells you what survives and not what they saw. Where the claim spans both — "the answer is not being saved" does — look at both before writing it down. Reading four strings out of a served page is not the same as four things having rendered, because the page also carries the JavaScript that would have rendered them. ## Before you ask for a feature Say what the closest existing thing is and why it does not fit. That sentence is what a maintainer acts on, and writing it is also how you find out the feature already exists, which happens often enough to be worth the two minutes. - `annotation-types.md` has every annotation type with its fields. - `config-keys.md` and `config-keys-nested.md` have every documented key. - `modalities.md` and `agent-traces.md` route a medium or a trace shape to the display and scheme that handle it. - `find_design.py --query "<what you are doing>"` searches published designs. - With the MCP server connected, `describe_annotation_type` and `describe_display_type` answer the same question against the live registry. A request that names the type it tried, quotes the field list, and says which field is missing is a different document from one that says Potato cannot do dialogue. ## The helper `scripts/report_issue.py` assembles the report. It collects the things every report needs and a reporter never has to hand, and it will not file anything on its own. ```bash python <skill-dir>/scripts/report_issue.py bug \ --title "Waveform never generates for media_directory audio" \ --observed "every POST to /api/waveform/generate returns use_client_fallback: true" \ --expected "a cached waveform, as audio_annotation documents" \ --repro "potato start config.yaml -p 8000, open an item, watch the network tab" \ --config config.yaml --log server.log --data data/items.json ``` ```bash python <skill-dir>/scripts/report_issue.py feature \ --title "Per-utterance audio in a dialogue display" \ --goal "annotate a voice-agent call where each turn is its own recording" \ --tried "audio_dialogue and speech_transcript both resolve one file per item" \ --needed "turn entries that each carry their own audio url" ``` It writes `potato-issue-<kind>.md`, prints the body, and prints a GitHub URL with the title and body already filled in. What it adds on top of what you typed: - **The version, asked of Potato rather than reconstructed.** If the installed Potato answers `potato --version`, that output goes into the report verbatim. Nothing outside Potato can work the answer out reliably: on one machine on 2026-09-09 the installed metadata said 2.7.0 while the source said 2.8.2, because dist-info is only rewritten on install; the same call answered 2.8.2 from inside the Potato repository, where a local `potato_annotation.egg-info` shadows the site-packages record; and `potato.__version__` was absent from every other directory, because a stale `site-packages/potato/` holding only `templates/` made `potato` resolve as a namespace package while each submodule still imported fine. Releases up to 2.7 have no `--version` -- it exits 2 with an argparse usage block -- and the helper then falls back to the metadata version plus the commit, which pip records for an editable install, and says in the report that the number may lag the source. - **Whether the config passes `--strict`**, with the output. This is the check above, done for you and shown to the reader. - **The data by shape** — field names and value types of the first record, with a count. Never the values, because those are someone's corpus. - **A duplicate search.** GitHub's issue search ANDs bare terms, so the obvious query matches nothing however many duplicates exist; the helper ORs the distinctive words and ranks what comes back by how much of your title each result shares. It says so when the search could not run, which is not the same answer as finding nothing. ## Redaction Configs hold `secret_key`, `ai_support.ai_config.api_key`, crowd credentials and annotator email addresses, and log tails hold bearer tokens. The helper removes them before anything is written: any value under a key named like a secret, anything shaped like an API key or a token, email addresses, and the home directory out of every path. It then prints what it removed and how often, because a redactor you cannot audit is worse than no redactor. It does **not** remove things that are not secrets but might still be yours: endpoint hostnames, task and file names, label vocabularies, item ids, and any prose you typed into `--observed`. Read the body before it goes anywhere. That is what the printed copy and the report file are for. ## Filing it Two paths, and both of them need the person whose study it is to say yes first. **By hand.** Open the prefilled URL. Nothing needs installing and the person sees the form before anything is public. **With `gh`.** Add `--submit --confirm`. Both flags are required, and the helper refuses to file a body still holding a `_(fill this in)_` placeholder. ```bash python <skill-dir>/scripts/report_issue.py bug \ --title "..." --observed "..." --expected "..." --repro "..." \ --config config.yaml --submit --confirm ``` It files under whichever account `gh` is logged in as, on a public repository, and it cannot be withdrawn. Show the body and get an answer before passing `--confirm`. `--label` needs write access on the repository, which a reporter usually does not have; the kind is in the title as `[bug]` or `[feature]` either way. ## What I have and have not verified Run against Potato 2.7.0 on 2026-09-09, with `gh` 2.85.0: - Redaction, on a config carrying `secret_key`, an `api_key`, an authorized-user list of email addresses, a bearer token in the log, and absolute paths under the home directory. None of the five appears in the report. - Both version branches. Against a Potato that answers `potato --version`, the report carries that output unchanged. Against one that does not -- a stub on `PATH` exiting 2 with a usage block, which is what 2.7 and earlier do -- the fallback gives the metadata version, the commit from pip's editable record, and a line saying the number can lag the source. - The duplicate search, both ways. A title sharing three words with an existing issue returned it; four titles with no duplicate returned nothing rather than filler. Rate-limited and offline both report themselves. - Both refusals: `--submit` without `--confirm`, and `--submit` on a body with placeholders. Each exits 2 and files nothing. - The URL falls back to the bare `/issues/new` when the body is too long to carry in a query string. **Not verified: the `gh issue create` call itself**, because verifying it means filing a real issue on a public repository. The flags it passes are the ones `gh issue create --help` documents, and the body goes through `--body-file` rather than an argument. Treat the first real `--submit` as the test. <!-- https://davidjurgens.github.io/potato-skill/interrogating-potato/ --> # Interrogating Potato Recovering an undocumented config block from the validator and the log. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Roughly forty config blocks take an object and document nothing about its contents, including `training`, `phases`, `surveyflow`, `quality_control` and `adjudication`. Guessing key names one at a time is slow, and a wrong guess is usually silent rather than loud. Every side-file format in this pack was recovered with the four methods below. Do this in a scratch directory with a two-item data file, not in the real project: booting writes `project.sqlite`, `layouts/` and `annotation_output/`. ## 1. The validator enumerates recognized sub-keys One level down, and no further. Put a batch of candidate names in the block and run `--strict`. Everything it *rejects* is definitely wrong, and it suggests near misses: ``` Unrecognized config key 'training.show_answers'. Did you mean: 'feedback'? Unrecognized config key 'training.passing_score'. Did you mean: 'passing_criteria'? ``` **An ignored key proves nothing on its own.** Validation descends two levels, so `training.feedback` is checked and `training.feedback.anything` is not. A plausible-looking sub-sub-key passes `--strict` and does nothing — that is how a wrong `gold_standards.accuracy.min_accuracy` survived into an earlier draft of this pack. At depth three or more, fall back to the key documentation (`config-keys-nested.md`, `get_key_doc`) and to method 3. Warnings are also suppressed while there are hard errors, so fix the errors first or you will think the block is clean. ## 2. Type errors name the shape `training.feedback must be a dictionary`, `gold_standards.mode must be one of: training, mixed, separate`. Feed it the wrong type on purpose. This is the fastest way to learn whether a key wants a scalar, a list or a dict, and several blocks accept only the dict form despite being documented as either. ## 3. The boot log on a bad side file How much it tells you depends on the loader. Gold standards says `missing gold_label`; training says `missing required fields` and dumps the instance without naming anything. When the message is unhelpful, delta-debug it. Put ~40 plausible keys on one item, confirm it loads, then drop half, boot, and keep the half that still works. Six boots gets you a minimal set. ```bash python <skill-dir>/scripts/boot_and_check.py config.yaml -p 8123 --json ``` reports the `Loaded N` counts as data, which makes that loop scriptable rather than a grep each time. ## 4. Errors that enumerate valid values `Unknown phase: x`, `annotation_type must be one of: …`, `display type X does not support span annotation`. Provoke them deliberately — an invalid value is a cheap way to get the complete valid set, and it is current by construction where a doc page may not be. ## Ask the package directly Before any of the above, the registries answer most questions without a server: ```python from potato.server_utils.config_key_docs import get_key_doc from potato.server_utils.schema_examples import example_scheme_for from potato.server_utils.examples_manifest import search_examples get_key_doc("attention_checks.frequency") example_scheme_for("bws") search_examples(annotation_type="span", display_type="image") ``` These read the same tables the server enforces, so they cannot be out of date with the running code the way a doc page can. <!-- https://davidjurgens.github.io/potato-skill/worked-example/ --> # A Worked Example One complete task, annotated, and what to change first. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Copying something that works beats assembling a config from field lists. Potato ships 218 examples for exactly this reason — but they are text-classification and schema demos, and none of them is a whole study: consent, instructions, a practice round, attention checks, the annotation itself, and a survey at the end. This is that skeleton. Every block below is the literal content of a file in `examples/advanced/full-study-skeleton/`, which validates under `--strict` and boots with `Loaded 2 training instances`, `Loaded 1 attention check items` and nothing about phases. A test in the Potato repo fails if this page and those files stop agreeing, so what you copy is what was run. Change the labels and the prose; keep the shapes. ``` project/ ├── config.yaml ├── data/ │ ├── items.json # the real items │ ├── training.json # practice, with model answers │ └── attention.json # attention checks ├── pages/ │ ├── consent.jsonl │ ├── instructions.jsonl │ └── poststudy.jsonl └── static/ └── study.css ``` Every file in that tree is below. A phase whose `file:` is *missing* does not stop the boot: the phase is dropped with one `ERROR` line and the study runs without it. A file that exists and is malformed does stop it — one entry without `annotation_type` and the server exits before it listens — so copy all three page files, not just the one you plan to edit. ## config.yaml ```yaml # yaml-language-server: $schema=https://potatoannotator.readthedocs.io/en/latest/schemas/potato-config.schema.json annotation_task_name: Full Study Skeleton task_dir: . output_annotation_dir: annotation_output/ data_files: [data/items.json] item_properties: id_key: id text_key: body instance_display: fields: - {key: title, type: text, label: Title} - {key: body, type: text, label: Item, span_target: true} layout: {direction: vertical, gap: 10px} resizable: false # otherwise one line of text gets a 130px card base_css: static/study.css annotation_instructions: >- <p><b>Reminder.</b> One line per rule, not the full definitions. Those live on the instructions page.</p> annotation_schemes: - annotation_type: span name: evidence description: Highlight the part that decides your answer. target_field: body # names the span_target field above labels: [Evidence, Counter-evidence] humanize_labels: false sequential_key_binding: true show_span_labels: true - annotation_type: radio name: category description: Which category is this? labels: [A, B, C, Unclear] # always have a "none" or "unclear" option humanize_labels: false sequential_key_binding: true label_requirement: {required: true} - annotation_type: likert # ordered judgement -> ordinal metric name: severity description: How severe is it? size: 3 min_label: Low max_label: High humanize_labels: false labels: [Low, Medium, High] # label every point, not just the ends sequential_key_binding: true label_requirement: {required: true} - annotation_type: text # follow-up, hidden until it applies name: why_unclear description: What made it unclear? multiline: true # without this a `text` scheme is a one-line <input> rows: 3 min_chars: 10 display_logic: show_when: - {schema: category, operator: equals, value: Unclear} num_annotators_per_item: 3 assignment_strategy: random random_seed: 1 max_annotations_per_user: 12 # every annotator sees all 12 items require_fully_annotated: true training: enabled: true data_file: data/training.json allow_retry: true feedback: {show_explanations: true} attention_checks: enabled: true frequency: 4 items_file: data/attention.json min_response_time: 5 # These four keys are the whole of `failure_handling`. It used to read # `{action: warn}` here, which is not one of them -- so it was dropped and # the thresholds silently fell back to these same defaults. failure_handling: warn_threshold: 2 warn_message: "Please read each item carefully before answering." block_threshold: 5 block_message: "Too many attention checks were missed." agreement_metrics: {enabled: true} phases: order: [consent, instructions, training, annotation, poststudy] consent: {type: consent, title: Consent, file: pages/consent.jsonl} instructions: {type: instructions, title: Instructions, file: pages/instructions.jsonl} training: {type: training, title: Practice} annotation: {type: annotation} poststudy: {type: poststudy, title: Last questions, file: pages/poststudy.jsonl} login: {type: password} user_config: {allow_all_users: true} ``` ## data/items.json Any of JSON array, JSONL, CSV or TSV. Fields beyond `id_key` and `text_key` are available to `instance_display` by name. ```json [ {"id": "c001", "source": "forum", "title": "Thread 1", "body": "Body text of item 1. It has two sentences so a span has somewhere to go."}, {"id": "c002", "source": "forum", "title": "Thread 2", "body": "Body text of item 2. It has two sentences so a span has somewhere to go."}, {"id": "c003", "source": "forum", "title": "Thread 3", "body": "Body text of item 3. It has two sentences so a span has somewhere to go."}, {"id": "c004", "source": "forum", "title": "Thread 4", "body": "Body text of item 4. It has two sentences so a span has somewhere to go."}, {"id": "c005", "source": "forum", "title": "Thread 5", "body": "Body text of item 5. It has two sentences so a span has somewhere to go."}, {"id": "c006", "source": "forum", "title": "Thread 6", "body": "Body text of item 6. It has two sentences so a span has somewhere to go."}, {"id": "c007", "source": "forum", "title": "Thread 7", "body": "Body text of item 7. It has two sentences so a span has somewhere to go."}, {"id": "c008", "source": "forum", "title": "Thread 8", "body": "Body text of item 8. It has two sentences so a span has somewhere to go."}, {"id": "c009", "source": "forum", "title": "Thread 9", "body": "Body text of item 9. It has two sentences so a span has somewhere to go."}, {"id": "c010", "source": "forum", "title": "Thread 10", "body": "Body text of item 10. It has two sentences so a span has somewhere to go."}, {"id": "c011", "source": "forum", "title": "Thread 11", "body": "Body text of item 11. It has two sentences so a span has somewhere to go."}, {"id": "c012", "source": "forum", "title": "Thread 12", "body": "Body text of item 12. It has two sentences so a span has somewhere to go."} ] ``` ## data/training.json An **object**, not an array. Each instance needs `id`, `text`, `correct_answers`. ```json { "training_instances": [ {"id": "t1", "text": "Worked example one. The model answer is A.", "correct_answers": {"category": "A", "severity": "Low"}, "explanation": "Why A is right, in a sentence the annotator can act on."}, {"id": "t2", "text": "Worked example two. Nothing here decides the category, so it is Unclear.", "correct_answers": {"category": "Unclear", "severity": "Low"}, "explanation": "Unclear is a real answer, not a way of skipping the item."} ] } ``` `correct_answers` keys are scheme names; values are the label strings exactly as written in `labels:`. Span schemes cannot be graded here. The training page renders `text` only, so fold any context the annotator needs into that string. ## data/attention.json An array. Each item needs `id` and `expected_answer`, plus whatever fields your `instance_display` renders. ```json [ {"id": "chk1", "title": "Attention check", "body": "This is a check, not a real item. Highlight this sentence and mark it Evidence, select A for category, and Low for severity.", "text": "This is a check, not a real item. Highlight this sentence and mark it Evidence, select A for category, and Low for severity.", "expected_answer": {"category": "A", "severity": "Low"}} ] ``` Instruct **every required scheme**, span schemes included. A check item that leaves a required span unanswered dead-ends the annotator on a 400 with only a corner toast for feedback. ## pages/consent.jsonl One annotation scheme per line, same syntax as `annotation_schemes`. Prose goes in a `pure_display` scheme with `allow_html: true`. ```jsonl {"annotation_type":"pure_display","name":"consent_text","allow_html":true,"description":"<h3>Consent</h3><p>Draft. Replace with the approved wording.</p>"} {"annotation_type":"radio","name":"consent_agree","description":"Do you agree to take part?","labels":["I agree","I do not agree"],"humanize_labels":false,"label_requirement":{"required":true}} ``` ## pages/instructions.jsonl The full definitions live here, not in `annotation_instructions` — that banner sits on top of every item and pushes the questions below the fold. ```jsonl {"annotation_type":"pure_display","name":"instructions_text","allow_html":true,"description":"<h3>What to do</h3><p>Read the item, highlight the part that decides your answer, then pick a category and a severity.</p><p><b>A.</b> Replace with the real definition.</p><p><b>B.</b> Replace with the real definition.</p><p><b>C.</b> Replace with the real definition.</p><p><b>Unclear.</b> Use it when the item genuinely does not decide. It is an answer, not a skip.</p>"} ``` ## pages/poststudy.jsonl A phase page can ask real questions, not just display prose. These land in `annotation_output/<user>/` like any other answer. ```jsonl {"annotation_type":"pure_display","name":"poststudy_text","allow_html":true,"description":"<h3>Last questions</h3><p>Two questions, then you are done.</p>"} {"annotation_type":"likert","name":"task_clarity","description":"How clear were the instructions?","size":5,"min_label":"Very unclear","max_label":"Very clear","label_requirement":{"required":true}} {"annotation_type":"text","name":"comments","description":"Anything that was hard to judge?","multiline":true,"rows":3} ``` ## static/study.css Phase-page prose lands in a `<legend>`, which the theme sets to weight 600, so consent and instructions render entirely bold without this. ```css form.pure-display legend { font-weight: 400; line-height: 1.55; } form.pure-display legend h3 { font-weight: 650; } form.pure-display legend b, form.pure-display legend strong { font-weight: 650; } ``` ## Check it ```bash potato validate config.yaml --strict python <skill-dir>/scripts/boot_and_check.py config.yaml -p 8000 ``` `boot_and_check.py` backgrounds the server, waits for a 200, and reports every feature that is configured but loaded nothing — which is the failure the log lines above exist to catch. By hand: ```bash nohup potato start config.yaml -p 8000 > server.log 2>&1 & until curl -s -o /dev/null http://localhost:8000/; do sleep 2; done grep -E "ERROR|Loaded [0-9]+|phase" server.log ``` Then walk it in a browser — `running-a-task.md`. ## What to change first | Want | Change | |---|---| | No practice round | Drop `training:` **and** `training` from `phases.order` | | No attention checks | Drop `attention_checks:` | | One annotator per item | `num_annotators_per_item: 1`, drop `agreement_metrics` | | Images, audio, PDFs, dialogues | `instance_display.fields[].type` — see `building-the-ui.md` | | Screening before the task | A `prestudy` phase with its own page file | | Different questions per cohort | `scheme_sets` + `batch_assignment` |