# Modes and Subsystems

The whole-task modes a brief can be asking for without naming. It ships with the `potato-skill` skill:

```
/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato
```

---

A task is a config plus a data file, and about a quarter of what Potato can do
fits that description. The rest is whole modes, surfaces and background systems
that a researcher describes without naming — and the expensive mistake is not an
invented type name, which validation catches, but hand-assembling something that
already exists, which validates cleanly, renders fine and is worse than the real
thing.

This is the routing table. Match the brief, then read the block's sub-keys in
`config-keys-nested.md` and `get_key_doc("<block>.<key>")` for the detail.

## The seven clusters

Every capability decision belongs to one of seven, and each is owned by one file.
Use this two ways. Reading a brief, walk the clusters and ask which ones it
touches. The ones it never mentions are where a researcher is most likely to be
missing something they would want. Running the interview in `interviewing.md`,
these are the menu, offered as symptoms rather than by name.

| Cluster | The question behind it | Owned by |
|---|---|---|
| **Material** | What is in front of the annotator? | `modalities.md`, `agent-traces.md`; **What they brought** below |
| **The answer** | What kind of answer, and how precise? | `designing-a-task.md` — see below |
| **Unit** | What counts as one thing? | `designing-a-task.md`; `sessions` and `cases` below |
| **People** | Who annotates, and who sees which items? | `assignment-and-agreement.md`; **Different modes of working** below |
| **Trust** | How will you know the answers are good? | `quality-control.md`, `assignment-and-agreement.md`; **Measurement and quality** below |
| **Model help** | Should a model be involved at all? | **Models in the loop** below |
| **Output** | What comes out, and where does it run? | `getting-the-data-out.md`, `deploying.md`; **Data in and out** below |

### The answer itself

This cluster is the one with no rows here, because `designing-a-task.md` owns the
response-format tables and duplicating them is how they drift. Go there — but
know what is in them, because this is the cluster researchers most reliably miss.
They ask for `radio` and `likert` and stop, and Potato has a whole family for
answers that a scale cannot hold:

| They said | Reach for |
|---|---|
| "it isn't one label — it's mostly A, a bit B" | `soft_label`, rather than throwing the ambiguity away |
| "an acceptable range, not a point" | `range_slider` |
| "warm/cold, weak/strong, active/passive" | `semantic_differential` |
| "the parts have to add up" | `constant_sum` |
| "how sure were they" | `confidence`, with `target_schema` naming the judgment it qualifies |
| "which of these two is better" | `pairwise`, `bws`, `ranking`, `conjoint` |

## Routing from what the researcher said

### Different modes of working

| They said | Reach for | Notes |
|---|---|---|
| "codes I'll invent as I read", "notes on passages", "in the participant's own words" | `qda_mode` | Moves memos, codebook and case grouping together. Do **not** build it out of a `multiselect` you plan to keep editing |
| "a shared codebook", "the label set will change" | `codebook` + scheme-level `codebook: true` | Edited at `/codebook`. The first boot seeds it from the config; later config edits need `potato codebook config.yaml` |
| "a participant", "an interview", "count codes per person not per excerpt" | `cases` | The unit of analysis becomes the case |
| "it's just me", "I want to find my own inconsistencies" | `solo_mode` | LLM labels, you review where it is least sure, prompt is refined from your corrections |
| "several people discussing an item together", "a norming session" | `rooms` | Live multiplayer at `/rooms`. What it writes back is the post-discussion vote — `assignment-and-agreement.md` |
| "one final label per item, not a distribution" | `adjudication` | Needs `adjudicator_users` |
| "a reviewer signs off", "a queue with states" | `review_workflow` | Kanban at `/admin/review` |
| "go as fast as possible", "just keyboard" | `review_mode` | Advances by itself once the item is complete |
| "score the whole conversation, not each turn" | `sessions` + `session_level: true` on the scheme | Warns at validate if no scheme opts in |
| "on their phone" | `pocket` | PWA at `/pocket` with an offline queue. Seven answer types work there and one other scheme takes the task off the phone — `designing-a-task.md` |
| "annotators should pick their own items" | `search.annotator_claim` | Admin search is always on; claiming is the opt-in |

### What they brought

Route on the medium before the question — the display type, the scheme type and
the key each one reads are all decided by it. Full tables in `modalities.md`;
agent traces are their own family, in `agent-traces.md`.

| They have | Display + scheme | The trap |
|---|---|---|
| Images to mark up | `image` display + `image_annotation` | Needs `tools` **and** `labels`; the canvas takes its bitmap from the display field, so removing it gives an empty canvas |
| Very large images | `image` + `viewer`/`tiles` | The viewer owns the transform; never recompute it |
| Video, mark moments or track objects | `video_annotation` (no display field needed) | Five modes and `labels` is required in all of them |
| Video, find the interval for a phrase | `temporal_grounding` | Reads `video_key`/`events_key`, not `source_field` |
| Audio regions | `audio_annotation` | `mode: label` for one label per region; `questions` or `both` to ask whole schemes per region, which then need `segment_schemes` |
| ASR/TTS against a reference | `speech_transcript` | Its own key names throughout |
| Tiers over audio or video, ELAN-style | `tiered_annotation` | `media_type` defaults to `audio` |
| A podcast or interview | `audio_dialogue` display | Per-turn playback is built in; it is a span target |
| A chat log | `dialogue` display + `turn_level` schemes | `conversation_tree` is **not** a span target |
| Point clouds or LiDAR | `spatial_annotation` | `lod` defaults on, and `max_points` only applies when it is off |
| Robot episodes | `episode_annotation` | Four layers, all on by default |
| Generated video from a world model | `rollout_evaluation` | Break-points, not a plausibility score |
| Agent traces of any kind | `agent-traces.md` | Nearly nothing is a required field, so a wrong `steps_key` renders an empty list |

### Models in the loop

| They said | Reach for |
|---|---|
| "suggest a label", "pre-fill and let them correct" | `ai_support` — see `model-assistance.md`, which is where the keys actually live |
| "label everything with a model first", "we can only afford to label a sample", "how good are the model's labels" | `labeling_engine` with `assignment_strategy: engine`, which tests the machine labels against a blind human sample — see `labeling-engine.md`. `pre_annotation` and `icl_labeling` still work and print a note pointing at the engine |
| "use the labels we already have as examples" | `icl_labeling` |
| "show me where the model is unsure" | `active_learning`, `llm_confidence` assignment |
| "does the judge agree with our humans" | `judge_alignment`, `judge_calibration` |
| "compare two models' answers" | `arena`, or a `pairwise` / `bws` scheme |
| "let them ask a model while annotating" | `chat_support` — a sidebar, not per-scheme buttons; `model-assistance.md` |
| "judge the agent while it runs" | `live_agent`, `live_coding_agent`, `agent_proxy` |
| "we have traces coming out of production" | `trace_ingestion` (`/api/traces/*`), `automation`, `triage` |

### Measurement and quality

| They said | Reach for |
|---|---|
| "how much do they agree" | `num_annotators_per_item` + `agreement_metrics` — see `assignment-and-agreement.md` |
| "who is reliable", "some annotators are better" | `mace` (competence from disagreement alone), `psychometrics` (live IRT) |
| "catch people clicking through" | `attention_checks`, `gold_standards` — see `quality-control.md` |
| "I planted known problems, did they find them" **on a geometry or span task** | Not supported. `gold_label` holds a label, never a box or an offset. Score it with geometry agreement between annotators instead, and say so |
| "the majority is sometimes wrong" | `truth_serum` (surprisingly-popular scoring) |
| "what changes their mind", "contrast sets" | `boundary_probing` |
| "how did they produce that text", "did they paste it" | `keystroke_logging` |
| "how did they draw that", "are they rubber-stamping the AI" | `annotation_telemetry` |
| "have them talk through it" | `thinkaloud` |

### Data in and out

| They said | Reach for |
|---|---|
| "the data is in a spreadsheet / S3 / HuggingFace / a database" | `data_sources`, `data_directory` (+ `watch_data_directory`) |
| "new data keeps arriving" | `data_sources` polling, `watch_data_directory`, `automation` |
| "it's a big corpus" | `partial_loading`, `item_store: {backend: paged}` |
| "show me the corpus shape" | `corpus_map`, `embedding_visualization`, `diversity_ordering` |
| "find the ones like this", "a group that keeps updating", "what is even in here" | `curation` — the catalog at `/admin/catalog`, below |
| "publish the dataset" | `publish`, `dataset_metadata` |
| "pay people on Prolific / MTurk" | `crowdsourcing` (+ `completion_code`, `auto_redirect_on_completion`) |
| "different groups see different items" | `batch_assignment` + `scheme_sets`, or `category_assignment` |
| "people abandon assignments" | `instance_reclaim` |
| "let annotators see their own progress" | `annotator_dashboard` |

### The `when` grammar

`triage.rules`, `automation.rules` and `review_workflow.routing` all filter with
the same `when` condition, evaluated against the item's data dict:

```yaml
triage:
  enabled: true
  rules:
    - when: {field: status, equals: error}
      priority: 100
      badge: "agent errored"
    - when: {field: metadata.score, lt: 0.5}
      priority: 60
  signal_field: judge_score      # read when no rule matched
  invert_signal: true            # lower is worse
```

One condition takes `field` plus exactly one operator: `equals`, `in`,
`contains`, `exists`, `lt`, `lte`, `gt`, `gte`. Field paths may be dotted
(`metadata.score`). String comparison for `equals` and `in` is
case-insensitive; `contains` tests membership in a list field or a substring of
a string one; a field the item does not carry never matches anything but
`exists: false`. `when` also accepts a list of conditions, all of which must
hold.

`automation` rules add `sample_rate` (0–1, hashed on item id so the decision is
stable) and `actions`, which are dicts rather than names:

```yaml
automation:
  enabled: true
  rules:
    - name: "Route errors to review"
      when: {field: status, equals: error}
      sample_rate: 1.0
      actions:
        - {type: add_to_queue, priority: 200, reason: "agent errored"}
        - {type: add_to_dataset, dataset: errors-to-fix}
        - {type: notify, message: "New error trace"}
```

The seven action types are `add_to_queue`, `add_to_dataset`, `notify`,
`enroll_review`, `run_evaluator`, `fire_webhook` and `refresh_topics`; the last
three run on a background worker. `add_to_dataset` needs `datasets` enabled.
`/admin/automation/status` reports `items_processed`, `rules_fired` and a count
per action, which is how you find out whether a rule ever matched.

## What "enabling" a block costs

Most blocks activate on `enabled: true` and nothing else. Two in this list refuse
to start without more:

```
ai_support.endpoint_type is required when ai_support is enabled
adjudication.adjudicator_users must be a non-empty list of usernames
```

One warns rather than failing, which is the pattern to expect elsewhere:

```
sessions.enabled is true but no annotation scheme sets session_level: true
  — the /sessions page will have no questions.
```

Verified by enabling 28 blocks at once and reading what the validator said. The
rest accepted a bare `enabled: true`, which means only that the config loads —
**not** that the feature has anything to work with. Boot the server and look for
the subsystem's own initialization line before believing it is live.

## Pages each block adds

Turning a block on usually adds a page. None of them are linked from the
annotation page, so tell the researcher they exist:

| Path | From |
|---|---|
| `/progress` | `annotator_dashboard` |
| `/codebook` | `codebook`, `qda_mode` |
| `/sessions` | `sessions` |
| `/rooms` | `rooms` |
| `/corpus` | `corpus_map` |
| `/admin/catalog` | `curation` |
| `/datasets/admin` | `datasets` (`/datasets` itself is not a route) |
| `/pocket` | `pocket` (phones auto-route here) |
| `/adjudicate` | `adjudication` |
| `/admin/review` | `review_workflow` |
| `/admin/arena` | `arena` |
| `/psychometrics/dashboard` | `psychometrics` |
| `/truth_serum/dashboard` | `truth_serum` |
| a MACE tab inside `/admin` | `mace` (`/admin/api/mace/overview` for the JSON) |
| `/admin/automation` | `automation`; `/status` and `/outcomes` beside it return JSON |
| `/judge_calibration/admin` | `judge_calibration`; `/status`, `/progress`, `/run`, `/report` beside it |
| `/api/mcp/manifest` | `mcp`, once an agent token exists |
| `/admin/...` | agreement, exports, progress — always present |

**`/admin` is the only admin page a browser can open cold.** It presents a key
box; every other `/admin/*` page and both dashboards answer
`403 {"error":"Admin authentication required"}` until the key has been entered
there once, so a deep link handed to a colleague looks broken. The dashboards
also sit at their own prefixes rather than under `/admin`:
`/admin/psychometrics` is a 404, `/psychometrics/dashboard` is the page.

## Scheme-level keys

These are keys on an `annotation_schemes` entry, not top-level blocks, and
nothing validates them (see `config-keys-nested.md`):

| Key | Effect |
|---|---|
| `display_logic` | Show a scheme only when another scheme has a given answer |
| `label_requirement` / `required` | Enforce an answer |
| `session_level` | One answer per session rather than per item |
| `turn_level`, `turn_binding`, `turn_label` | Bind a scheme to conversation turns |
| `codebook: true` | This scheme's labels come from the shared project codebook |
| `target_field` | Which `instance_display` field a span scheme anchors to |
| `has_free_response` | Adds an "other, please specify" input to a radio/multiselect |
| `option_randomization` | Randomize option order per annotator |
| `sequential_key_binding` | Number the labels 1,2,3… instead of naming keys |
| `dynamic_options`, `dynamic_options_field` | Options come from the item, not the config |

### What `codebook: true` carries over

The first boot seeds the codebook from the labels of **every** scheme that sets
`codebook: true`, and how you wrote those labels decides what the codebook
contains. Bare strings give it names and nothing else, so the tray shows
colourless codes reading "No content yet" while the annotation chips still look
right. The form looks configured, and everything the researcher wrote about each
code is on the floor. The dict form carries three things:

```yaml
annotation_schemes:
  - annotation_type: radio
    name: stance
    description: What stance does it take?
    codebook: true
    labels:
      - name: Supportive
        color: "#2e7d32"
        description: The text endorses the proposal, including qualified endorsement.
      - name: Opposed
        color: "#c62828"
        tooltip: The text argues against the proposal on any grounds.
      - name: Neutral
        color: "#616161"
```

`name` becomes the code, `color` its colour in the tray and on the chips, and
`description` its definition. `tooltip` is accepted as the same thing, so a
config already using it for hover text needs no change. A label with neither
gets a code with no definition rather than an invented one -- `Neutral` above.

Seeding is a first-run bootstrap gated on the codebook being empty, so a label
added to the config later does not appear: run `potato codebook config.yaml`.
That gate is per codebook rather than per scheme, and it is applied after every
codebook-backed scheme has contributed, so a label declared only on a second
scheme is seeded too.

## Deciding what to annotate: the catalog

`curation` builds an embedding index over the loaded items and serves
`/admin/catalog`. It is the answer to "which of these thousands should we
label", which is a question the annotation config itself cannot ask.

```yaml
curation:
  enabled: true
  model_name: all-MiniLM-L6-v2   # the default
  embed_on_ingest: true          # off by default; costs boot time and memory
```

Everything below is an admin JSON route under `/admin/catalog/api/`, so it needs
the `X-API-Key` header. **The catalog routes reject an unauthenticated request
without generating the key file**, unlike `/admin/iaa`, so make the first
unauthenticated request somewhere else and read `admin_api_key.txt` after that.

Build the index once with `POST api/build`, which answers `{"indexed": 12}`.
Then, on twelve short complaint texts:

- **`POST api/search`** takes a `query` and returns instance ids by cosine
  similarity. "my train never showed up and nobody told us anything" put the two
  train items at 0.64 and 0.59 and everything else below 0.22. **The parameter is
  `top_k`, not `k`** — `k` is accepted and ignored, so you get the default ten
  results back, which looks like the model failing to discriminate.
- **`POST api/slices`** saves a named query as a dynamic slice, and
  `GET api/slices/<name>/resolve` returns what currently matches. A slice named
  `transport-complaints` over "delayed or cancelled travel" at `threshold: 0.2`
  resolved to six ids. Pick the threshold by resolving and reading the list;
  0.2 is loose enough to pull in neighbouring complaints.
- **`POST api/topics/refresh`** clusters the index, and `GET api/topics/<name>/members`
  lists each cluster. On twelve items written as six near-duplicate pairs it
  recovered all six pairs exactly. Clusters are named `topic-1`, `topic-2` and so
  on with empty descriptions unless something supplies a label, so read the
  members rather than the name.
- **`GET api/duplicates`** is **perceptual-hash duplicate detection over images**,
  not text. On a text-only project it returns zeros and says so in a `note`
  field: "No item could be perceptually hashed … This is NOT a finding of zero
  duplicates." Read the note before you quote the number. For text overlap, the
  topic clusters above are what you have.

Slices and topics both have a `to_dataset` route, so a slice can become the thing
you actually hand to annotators.

## When you cannot find the feature

The routing above covers what the key documentation names. For anything else:

```python
from potato.server_utils.examples_manifest import search_examples
search_examples(query="qualitative")
search_examples(config_key="gold_standards")
search_examples(annotation_type="span", display_type="image")
```

Each entry records the annotation and display types a config uses, the features
it switches on, and the command that runs it. Copying a config that is checked in
CI beats assembling one from field lists — and if the examples are unavailable to
you, the discovery method in `SKILL.md` is the fallback.