Modes and Subsystems

The whole-task modes a brief can be asking for without naming. It ships with the potato-skill skill:

/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato

---

A task is a config plus a data file, and about a quarter of what Potato can do fits that description. The rest is whole modes, surfaces and background systems that a researcher describes without naming — and the expensive mistake is not an invented type name, which validation catches, but hand-assembling something that already exists, which validates cleanly, renders fine and is worse than the real thing.

This is the routing table. Match the brief, then read the block's sub-keys in config-keys-nested.md and get_key_doc("<block>.<key>") for the detail.

The seven clusters

Every capability decision belongs to one of seven, and each is owned by one file. Use this two ways. Reading a brief, walk the clusters and ask which ones it touches. The ones it never mentions are where a researcher is most likely to be missing something they would want. Running the interview in interviewing.md, these are the menu, offered as symptoms rather than by name.

Cluster	The question behind it	Owned by
Material	What is in front of the annotator?	modalities.md, agent-traces.md; What they brought below
The answer	What kind of answer, and how precise?	designing-a-task.md — see below
Unit	What counts as one thing?	designing-a-task.md; sessions and cases below
People	Who annotates, and who sees which items?	assignment-and-agreement.md; Different modes of working below
Trust	How will you know the answers are good?	quality-control.md, assignment-and-agreement.md; Measurement and quality below
Model help	Should a model be involved at all?	Models in the loop below
Output	What comes out, and where does it run?	getting-the-data-out.md, deploying.md; Data in and out below

The answer itself

This cluster is the one with no rows here, because designing-a-task.md owns the response-format tables and duplicating them is how they drift. Go there — but know what is in them, because this is the cluster researchers most reliably miss. They ask for radio and likert and stop, and Potato has a whole family for answers that a scale cannot hold:

They said	Reach for
"it isn't one label — it's mostly A, a bit B"	soft_label, rather than throwing the ambiguity away
"an acceptable range, not a point"	range_slider
"warm/cold, weak/strong, active/passive"	semantic_differential
"the parts have to add up"	constant_sum
"how sure were they"	confidence, with target_schema naming the judgment it qualifies
"which of these two is better"	pairwise, bws, ranking, conjoint

Routing from what the researcher said

Different modes of working

They said	Reach for	Notes
"codes I'll invent as I read", "notes on passages", "in the participant's own words"	qda_mode	Moves memos, codebook and case grouping together. Do not build it out of a multiselect you plan to keep editing
"a shared codebook", "the label set will change"	codebook + scheme-level codebook: true	Edited at /codebook. The first boot seeds it from the config; later config edits need potato codebook config.yaml
"a participant", "an interview", "count codes per person not per excerpt"	cases	The unit of analysis becomes the case
"it's just me", "I want to find my own inconsistencies"	solo_mode	LLM labels, you review where it is least sure, prompt is refined from your corrections
"several people discussing an item together", "a norming session"	rooms	Live multiplayer at /rooms. What it writes back is the post-discussion vote — assignment-and-agreement.md
"one final label per item, not a distribution"	adjudication	Needs adjudicator_users
"a reviewer signs off", "a queue with states"	review_workflow	Kanban at /admin/review
"go as fast as possible", "just keyboard"	review_mode	Advances by itself once the item is complete
"score the whole conversation, not each turn"	sessions + session_level: true on the scheme	Warns at validate if no scheme opts in
"on their phone"	pocket	PWA at /pocket with an offline queue. Seven answer types work there and one other scheme takes the task off the phone — designing-a-task.md
"annotators should pick their own items"	search.annotator_claim	Admin search is always on; claiming is the opt-in

What they brought

Route on the medium before the question — the display type, the scheme type and the key each one reads are all decided by it. Full tables in modalities.md; agent traces are their own family, in agent-traces.md.

They have	Display + scheme	The trap
Images to mark up	image display + image_annotation	Needs tools and labels; the canvas takes its bitmap from the display field, so removing it gives an empty canvas
Very large images	image + viewer/tiles	The viewer owns the transform; never recompute it
Video, mark moments or track objects	video_annotation (no display field needed)	Five modes and labels is required in all of them
Video, find the interval for a phrase	temporal_grounding	Reads video_key/events_key, not source_field
Audio regions	audio_annotation	mode: label for one label per region; questions or both to ask whole schemes per region, which then need segment_schemes
ASR/TTS against a reference	speech_transcript	Its own key names throughout
Tiers over audio or video, ELAN-style	tiered_annotation	media_type defaults to audio
A podcast or interview	audio_dialogue display	Per-turn playback is built in; it is a span target
A chat log	dialogue display + turn_level schemes	conversation_tree is not a span target
Point clouds or LiDAR	spatial_annotation	lod defaults on, and max_points only applies when it is off
Robot episodes	episode_annotation	Four layers, all on by default
Generated video from a world model	rollout_evaluation	Break-points, not a plausibility score
Agent traces of any kind	agent-traces.md	Nearly nothing is a required field, so a wrong steps_key renders an empty list

Models in the loop

They said	Reach for
"suggest a label", "pre-fill and let them correct"	ai_support — see model-assistance.md, which is where the keys actually live
"label everything with a model first", "we can only afford to label a sample", "how good are the model's labels"	labeling_engine with assignment_strategy: engine, which tests the machine labels against a blind human sample — see labeling-engine.md. pre_annotation and icl_labeling still work and print a note pointing at the engine
"use the labels we already have as examples"	icl_labeling
"show me where the model is unsure"	active_learning, llm_confidence assignment
"does the judge agree with our humans"	judge_alignment, judge_calibration
"compare two models' answers"	arena, or a pairwise / bws scheme
"let them ask a model while annotating"	chat_support — a sidebar, not per-scheme buttons; model-assistance.md
"judge the agent while it runs"	live_agent, live_coding_agent, agent_proxy
"we have traces coming out of production"	trace_ingestion (/api/traces/*), automation, triage

Measurement and quality

They said	Reach for
"how much do they agree"	num_annotators_per_item + agreement_metrics — see assignment-and-agreement.md
"who is reliable", "some annotators are better"	mace (competence from disagreement alone), psychometrics (live IRT)
"catch people clicking through"	attention_checks, gold_standards — see quality-control.md
"I planted known problems, did they find them" on a geometry or span task	Not supported. gold_label holds a label, never a box or an offset. Score it with geometry agreement between annotators instead, and say so
"the majority is sometimes wrong"	truth_serum (surprisingly-popular scoring)
"what changes their mind", "contrast sets"	boundary_probing
"how did they produce that text", "did they paste it"	keystroke_logging
"how did they draw that", "are they rubber-stamping the AI"	annotation_telemetry
"have them talk through it"	thinkaloud

Data in and out

They said	Reach for
"the data is in a spreadsheet / S3 / HuggingFace / a database"	data_sources, data_directory (+ watch_data_directory)
"new data keeps arriving"	data_sources polling, watch_data_directory, automation
"it's a big corpus"	partial_loading, item_store: {backend: paged}
"show me the corpus shape"	corpus_map, embedding_visualization, diversity_ordering
"find the ones like this", "a group that keeps updating", "what is even in here"	curation — the catalog at /admin/catalog, below
"publish the dataset"	publish, dataset_metadata
"pay people on Prolific / MTurk"	crowdsourcing (+ completion_code, auto_redirect_on_completion)
"different groups see different items"	batch_assignment + scheme_sets, or category_assignment
"people abandon assignments"	instance_reclaim
"let annotators see their own progress"	annotator_dashboard

The when grammar

triage.rules, automation.rules and review_workflow.routing all filter with the same when condition, evaluated against the item's data dict:

triage:
  enabled: true
  rules:
    - when: {field: status, equals: error}
      priority: 100
      badge: "agent errored"
    - when: {field: metadata.score, lt: 0.5}
      priority: 60
  signal_field: judge_score      # read when no rule matched
  invert_signal: true            # lower is worse

One condition takes field plus exactly one operator: equals, in, contains, exists, lt, lte, gt, gte. Field paths may be dotted (metadata.score). String comparison for equals and in is case-insensitive; contains tests membership in a list field or a substring of a string one; a field the item does not carry never matches anything but exists: false. when also accepts a list of conditions, all of which must hold.

automation rules add sample_rate (0–1, hashed on item id so the decision is stable) and actions, which are dicts rather than names:

automation:
  enabled: true
  rules:
    - name: "Route errors to review"
      when: {field: status, equals: error}
      sample_rate: 1.0
      actions:
        - {type: add_to_queue, priority: 200, reason: "agent errored"}
        - {type: add_to_dataset, dataset: errors-to-fix}
        - {type: notify, message: "New error trace"}

The seven action types are add_to_queue, add_to_dataset, notify, enroll_review, run_evaluator, fire_webhook and refresh_topics; the last three run on a background worker. add_to_dataset needs datasets enabled. /admin/automation/status reports items_processed, rules_fired and a count per action, which is how you find out whether a rule ever matched.

What "enabling" a block costs

Most blocks activate on enabled: true and nothing else. Two in this list refuse to start without more:

ai_support.endpoint_type is required when ai_support is enabled
adjudication.adjudicator_users must be a non-empty list of usernames

One warns rather than failing, which is the pattern to expect elsewhere:

sessions.enabled is true but no annotation scheme sets session_level: true
  — the /sessions page will have no questions.

Verified by enabling 28 blocks at once and reading what the validator said. The rest accepted a bare enabled: true, which means only that the config loads — not that the feature has anything to work with. Boot the server and look for the subsystem's own initialization line before believing it is live.

Pages each block adds

Turning a block on usually adds a page. None of them are linked from the annotation page, so tell the researcher they exist:

Path	From
/progress	annotator_dashboard
/codebook	codebook, qda_mode
/sessions	sessions
/rooms	rooms
/corpus	corpus_map
/admin/catalog	curation
/datasets/admin	datasets (/datasets itself is not a route)
/pocket	pocket (phones auto-route here)
/adjudicate	adjudication
/admin/review	review_workflow
/admin/arena	arena
/psychometrics/dashboard	psychometrics
/truth_serum/dashboard	truth_serum
a MACE tab inside /admin	mace (/admin/api/mace/overview for the JSON)
/admin/automation	automation; /status and /outcomes beside it return JSON
/judge_calibration/admin	judge_calibration; /status, /progress, /run, /report beside it
/api/mcp/manifest	mcp, once an agent token exists
/admin/...	agreement, exports, progress — always present

/admin is the only admin page a browser can open cold. It presents a key box; every other /admin/* page and both dashboards answer 403 {"error":"Admin authentication required"} until the key has been entered there once, so a deep link handed to a colleague looks broken. The dashboards also sit at their own prefixes rather than under /admin: /admin/psychometrics is a 404, /psychometrics/dashboard is the page.

Scheme-level keys

These are keys on an annotation_schemes entry, not top-level blocks, and nothing validates them (see config-keys-nested.md):

Key	Effect
display_logic	Show a scheme only when another scheme has a given answer
label_requirement / required	Enforce an answer
session_level	One answer per session rather than per item
turn_level, turn_binding, turn_label	Bind a scheme to conversation turns
codebook: true	This scheme's labels come from the shared project codebook
target_field	Which instance_display field a span scheme anchors to
has_free_response	Adds an "other, please specify" input to a radio/multiselect
option_randomization	Randomize option order per annotator
sequential_key_binding	Number the labels 1,2,3… instead of naming keys
dynamic_options, dynamic_options_field	Options come from the item, not the config

What codebook: true carries over

The first boot seeds the codebook from the labels of every scheme that sets codebook: true, and how you wrote those labels decides what the codebook contains. Bare strings give it names and nothing else, so the tray shows colourless codes reading "No content yet" while the annotation chips still look right. The form looks configured, and everything the researcher wrote about each code is on the floor. The dict form carries three things:

annotation_schemes:
  - annotation_type: radio
    name: stance
    description: What stance does it take?
    codebook: true
    labels:
      - name: Supportive
        color: "#2e7d32"
        description: The text endorses the proposal, including qualified endorsement.
      - name: Opposed
        color: "#c62828"
        tooltip: The text argues against the proposal on any grounds.
      - name: Neutral
        color: "#616161"

name becomes the code, color its colour in the tray and on the chips, and description its definition. tooltip is accepted as the same thing, so a config already using it for hover text needs no change. A label with neither gets a code with no definition rather than an invented one -- Neutral above.

Seeding is a first-run bootstrap gated on the codebook being empty, so a label added to the config later does not appear: run potato codebook config.yaml. That gate is per codebook rather than per scheme, and it is applied after every codebook-backed scheme has contributed, so a label declared only on a second scheme is seeded too.

Deciding what to annotate: the catalog

curation builds an embedding index over the loaded items and serves /admin/catalog. It is the answer to "which of these thousands should we label", which is a question the annotation config itself cannot ask.

curation:
  enabled: true
  model_name: all-MiniLM-L6-v2   # the default
  embed_on_ingest: true          # off by default; costs boot time and memory

Everything below is an admin JSON route under /admin/catalog/api/, so it needs the X-API-Key header. The catalog routes reject an unauthenticated request without generating the key file, unlike /admin/iaa, so make the first unauthenticated request somewhere else and read admin_api_key.txt after that.

Build the index once with POST api/build, which answers {"indexed": 12}. Then, on twelve short complaint texts:

• POST api/search takes a query and returns instance ids by cosine similarity. "my train never showed up and nobody told us anything" put the two train items at 0.64 and 0.59 and everything else below 0.22. The parameter is top_k, not k — k is accepted and ignored, so you get the default ten results back, which looks like the model failing to discriminate.
• POST api/slices saves a named query as a dynamic slice, and GET api/slices/<name>/resolve returns what currently matches. A slice named transport-complaints over "delayed or cancelled travel" at threshold: 0.2 resolved to six ids. Pick the threshold by resolving and reading the list; 0.2 is loose enough to pull in neighbouring complaints.
• POST api/topics/refresh clusters the index, and GET api/topics/<name>/members lists each cluster. On twelve items written as six near-duplicate pairs it recovered all six pairs exactly. Clusters are named topic-1, topic-2 and so on with empty descriptions unless something supplies a label, so read the members rather than the name.
• GET api/duplicates is perceptual-hash duplicate detection over images, not text. On a text-only project it returns zeros and says so in a note field: "No item could be perceptually hashed … This is NOT a finding of zero duplicates." Read the note before you quote the number. For text overlap, the topic clusters above are what you have.

Slices and topics both have a to_dataset route, so a slice can become the thing you actually hand to annotators.

When you cannot find the feature

The routing above covers what the key documentation names. For anything else:

from potato.server_utils.examples_manifest import search_examples
search_examples(query="qualitative")
search_examples(config_key="gold_standards")
search_examples(annotation_type="span", display_type="image")

Each entry records the annotation and display types a config uses, the features it switches on, and the command that runs it. Copying a config that is checked in CI beats assembling one from field lists — and if the examples are unavailable to you, the discovery method in SKILL.md is the fallback.