Modes and Subsystems¶
The whole-task modes a brief can be asking for without naming. It ships with the potato-tasks skill:
/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-tasks@potato
A task is a config plus a data file, and about a quarter of what Potato can do fits that description. The rest is whole modes, surfaces and background systems that a researcher describes without naming — and the expensive mistake is not an invented type name, which validation catches, but hand-assembling something that already exists, which validates cleanly, renders fine and is worse than the real thing.
This is the routing table. Match the brief, then read the block's sub-keys in
config-keys-nested.md and get_key_doc("<block>.<key>") for the detail.
The seven clusters¶
Every capability decision belongs to one of seven, and each is owned by one file.
Use this two ways. Reading a brief, walk the clusters and ask which ones it
touches. The ones it never mentions are where a researcher is most likely to be
missing something they would want. Running the interview in interviewing.md,
these are the menu, offered as symptoms rather than by name.
| Cluster | The question behind it | Owned by |
|---|---|---|
| Material | What is in front of the annotator? | modalities.md, agent-traces.md; What they brought below |
| The answer | What kind of answer, and how precise? | designing-a-task.md — see below |
| Unit | What counts as one thing? | designing-a-task.md; sessions and cases below |
| People | Who annotates, and who sees which items? | assignment-and-agreement.md; Different modes of working below |
| Trust | How will you know the answers are good? | quality-control.md, assignment-and-agreement.md; Measurement and quality below |
| Model help | Should a model be involved at all? | Models in the loop below |
| Output | What comes out, and where does it run? | getting-the-data-out.md, deploying.md; Data in and out below |
The answer itself¶
This cluster is the one with no rows here, because designing-a-task.md owns the
response-format tables and duplicating them is how they drift. Go there — but
know what is in them, because this is the cluster researchers most reliably miss.
They ask for radio and likert and stop, and Potato has a whole family for
answers that a scale cannot hold:
| They said | Reach for |
|---|---|
| "it isn't one label — it's mostly A, a bit B" | soft_label, rather than throwing the ambiguity away |
| "an acceptable range, not a point" | range_slider |
| "warm/cold, weak/strong, active/passive" | semantic_differential |
| "the parts have to add up" | constant_sum |
| "how sure were they" | confidence, with target_schema naming the judgment it qualifies |
| "which of these two is better" | pairwise, bws, ranking, conjoint |
Routing from what the researcher said¶
Different modes of working¶
| They said | Reach for | Notes |
|---|---|---|
| "codes I'll invent as I read", "notes on passages", "in the participant's own words" | qda_mode |
Moves memos, codebook and case grouping together. Do not build it out of a multiselect you plan to keep editing |
| "a shared codebook", "the label set will change" | codebook + scheme-level codebook: true |
Edited at /codebook. The first boot seeds it from the config; later config edits need potato codebook config.yaml |
| "a participant", "an interview", "count codes per person not per excerpt" | cases |
The unit of analysis becomes the case |
| "it's just me", "I want to find my own inconsistencies" | solo_mode |
LLM labels, you review where it is least sure, prompt is refined from your corrections |
| "several people discussing an item together", "a norming session" | rooms |
Live multiplayer at /rooms over a shared event log |
| "one final label per item, not a distribution" | adjudication |
Needs adjudicator_users |
| "a reviewer signs off", "a queue with states" | review_workflow |
Kanban at /admin/review |
| "go as fast as possible", "just keyboard" | review_mode |
Advances by itself once the item is complete |
| "score the whole conversation, not each turn" | sessions + session_level: true on the scheme |
Warns at validate if no scheme opts in |
| "on their phone" | pocket |
PWA at /pocket with an offline queue |
| "annotators should pick their own items" | search.annotator_claim |
Admin search is always on; claiming is the opt-in |
What they brought¶
Route on the medium before the question — the display type, the scheme type and
the key each one reads are all decided by it. Full tables in modalities.md;
agent traces are their own family, in agent-traces.md.
| They have | Display + scheme | The trap |
|---|---|---|
| Images to mark up | image display + image_annotation |
Needs tools and labels; the canvas takes its bitmap from the display field, so removing it gives an empty canvas |
| Very large images | image + viewer/tiles |
The viewer owns the transform; never recompute it |
| Video, mark moments or track objects | video_annotation (no display field needed) |
Five modes and labels is required in all of them |
| Video, find the interval for a phrase | temporal_grounding |
Reads video_key/events_key, not source_field |
| Audio regions | audio_annotation |
mode: label for one label per region; questions or both to ask whole schemes per region, which then need segment_schemes |
| ASR/TTS against a reference | speech_transcript |
Its own key names throughout |
| Tiers over audio or video, ELAN-style | tiered_annotation |
media_type defaults to audio |
| A podcast or interview | audio_dialogue display |
Per-turn playback is built in; it is a span target |
| A chat log | dialogue display + turn_level schemes |
conversation_tree is not a span target |
| Point clouds or LiDAR | spatial_annotation |
lod defaults on, and max_points only applies when it is off |
| Robot episodes | episode_annotation |
Four layers, all on by default |
| Generated video from a world model | rollout_evaluation |
Break-points, not a plausibility score |
| Agent traces of any kind | agent-traces.md |
Nearly nothing is a required field, so a wrong steps_key renders an empty list |
Models in the loop¶
| They said | Reach for |
|---|---|
| "suggest a label", "pre-fill and let them correct" | ai_support — see model-assistance.md, which is where the keys actually live |
| "label everything with a model first" | llm_labeling, or pre_annotation for seeded answers |
| "use the labels we already have as examples" | icl_labeling |
| "show me where the model is unsure" | active_learning, llm_confidence assignment |
| "does the judge agree with our humans" | judge_alignment, judge_calibration |
| "compare two models' answers" | arena, or a pairwise / bws scheme |
| "let them ask a model while annotating" | chat_support (same endpoint config as ai_support) |
| "judge the agent while it runs" | live_agent, live_coding_agent, agent_proxy |
| "we have traces coming out of production" | trace_ingestion (/api/traces/*), automation, triage |
Measurement and quality¶
| They said | Reach for |
|---|---|
| "how much do they agree" | num_annotators_per_item + agreement_metrics — see assignment-and-agreement.md |
| "who is reliable", "some annotators are better" | mace (competence from disagreement alone), psychometrics (live IRT) |
| "catch people clicking through" | attention_checks, gold_standards — see quality-control.md |
| "I planted known problems, did they find them" on a geometry or span task | Not supported. gold_label holds a label, never a box or an offset. Score it with geometry agreement between annotators instead, and say so |
| "the majority is sometimes wrong" | truth_serum (surprisingly-popular scoring) |
| "what changes their mind", "contrast sets" | boundary_probing |
| "how did they produce that text", "did they paste it" | keystroke_logging |
| "how did they draw that", "are they rubber-stamping the AI" | annotation_telemetry |
| "have them talk through it" | thinkaloud |
Data in and out¶
| They said | Reach for |
|---|---|
| "the data is in a spreadsheet / S3 / HuggingFace / a database" | data_sources, data_directory (+ watch_data_directory) |
| "new data keeps arriving" | data_sources polling, watch_data_directory, automation |
| "it's a big corpus" | partial_loading, item_store: {backend: paged} |
| "show me the corpus shape" | corpus_map, embedding_visualization, diversity_ordering |
| "publish the dataset" | publish, dataset_metadata |
| "pay people on Prolific / MTurk" | crowdsourcing (+ completion_code, auto_redirect_on_completion) |
| "different groups see different items" | batch_assignment + scheme_sets, or category_assignment |
| "people abandon assignments" | instance_reclaim |
| "let annotators see their own progress" | annotator_dashboard |
What "enabling" a block costs¶
Most blocks activate on enabled: true and nothing else. Two in this list refuse
to start without more:
ai_support.endpoint_type is required when ai_support is enabled
adjudication.adjudicator_users must be a non-empty list of usernames
One warns rather than failing, which is the pattern to expect elsewhere:
sessions.enabled is true but no annotation scheme sets session_level: true
— the /sessions page will have no questions.
Verified by enabling 28 blocks at once and reading what the validator said. The
rest accepted a bare enabled: true, which means only that the config loads —
not that the feature has anything to work with. Boot the server and look for
the subsystem's own initialization line before believing it is live.
Pages each block adds¶
Turning a block on usually adds a page. None of them are linked from the annotation page, so tell the researcher they exist:
| Path | From |
|---|---|
/progress |
annotator_dashboard |
/codebook |
codebook, qda_mode |
/sessions |
sessions |
/rooms |
rooms |
/corpus |
corpus_map |
/datasets |
datasets |
/pocket |
pocket (phones auto-route here) |
/admin/review |
review_workflow |
/admin/... |
agreement, exports, progress — always present |
Scheme-level keys¶
These are keys on an annotation_schemes entry, not top-level blocks, and
nothing validates them (see config-keys-nested.md):
| Key | Effect |
|---|---|
display_logic |
Show a scheme only when another scheme has a given answer |
label_requirement / required |
Enforce an answer |
session_level |
One answer per session rather than per item |
turn_level, turn_binding, turn_label |
Bind a scheme to conversation turns |
codebook: true |
This scheme's labels come from the shared project codebook |
target_field |
Which instance_display field a span scheme anchors to |
has_free_response |
Adds an "other, please specify" input to a radio/multiselect |
option_randomization |
Randomize option order per annotator |
sequential_key_binding |
Number the labels 1,2,3… instead of naming keys |
dynamic_options, dynamic_options_field |
Options come from the item, not the config |
When you cannot find the feature¶
The routing above covers what the key documentation names. For anything else:
from potato.server_utils.examples_manifest import search_examples
search_examples(query="qualitative")
search_examples(config_key="gold_standards")
search_examples(annotation_type="span", display_type="image")
Each entry records the annotation and display types a config uses, the features
it switches on, and the command that runs it. Copying a config that is checked in
CI beats assembling one from field lists — and if the examples are unavailable to
you, the discovery method in SKILL.md is the fallback.