# Assignment and Agreement Who annotates what, and which agreement measure follows from it. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Two questions that look separate and are not: the number of annotators per item decides whether an agreement number exists at all, and the per-annotator quota decides whether the items you care about ever reach three annotators. ## The keys | Key | What it does | Default | |---|---|---| | `num_annotators_per_item` | Annotators per item; an item retires when it has this many. **Set this one.** | unset: no limit (`0` too) | | `min_annotators_per_instance` | Deprecated spelling of the key above, read as the cap with a warning | unset | | `max_annotations_per_item` | Older spelling of the same cap; `-1` unlimited | -1 | | `max_annotations_per_user` | **Cap on dataset items per annotator**, unless `per_annotator_quota` overrides it | the item count | | `per_annotator_quota` | Per-annotator caps: `default`, `by_user`, `by_user_role`. Beats the key above | unset | | `automatic_assignment.instance_per_annotator` | Nothing. See below. | — | | `assignment_strategy` | Order items are handed out in | `fixed_order` | | `random_seed` | Makes the ordering reproducible | unset | The three are one setting. With none of them set there is no cap: on 2.11.0, five annotators of a two-item study were all served the first item. Two of them with the same number validate clean, and two with different numbers are refused: ``` 'max_annotations_per_item' and 'num_annotators_per_item' are both set with conflicting values (5 vs 3). Drop 'max_annotations_per_item' — 'num_annotators_per_item' is the canonical key. ``` `min_annotators_per_instance` used to be stored at load and read nowhere (checked against Potato `be457e7b`), so a study that set it collected as many annotators as it got. From 2.11 it is read as the cap. Set to 2 on its own, the first two annotators got the first item, the third got the second, and the fourth and fifth got nothing. Validation warns: `min_annotators_per_instance: 2 is deprecated and has been read as num_annotators_per_item: 2`. Rename it. (The key of the same name under `icl_labeling.example_selection` is a different setting, and that one is read.) #### `num_annotators_per_item.adaptive` `num_annotators_per_item` also takes a mapping, and the `adaptive` block under it reopens an item when the first annotators disagree: ```yaml num_annotators_per_item: default: 2 adaptive: enabled: true disagreement_threshold: 0.5 boost_to: 4 ``` Measured on 8 items with two annotators who disagreed on `x0`-`x3` and agreed on `x4`-`x7`. The server logged four lines and only four: ``` Adaptive boost: x0 raised to 4 annotators (current=2) ``` The third and fourth annotators through the door were each served exactly `x0 x1 x2 x3`; the agreed half had retired at 2; the fifth got nothing. So the budget goes where the label is actually contested, and a study that would have cost 8 x 4 = 32 annotations cost 24. Worth knowing before you reach for it: this is a cap being raised, so the arithmetic in the section above stops holding. `items x num_annotators_per_item` is no longer the ceiling on how much work exists — with `boost_to: 4` on a `default: 2` study the true ceiling is twice what `default` suggests, and how close you get depends on how much your annotators disagree, which is the thing you do not know yet. Recruit against `boost_to`, not against `default`. #### `instance_per_annotator` Measured on 10 items, reading the assignment the server actually built: | Config | Items served to one annotator | |---|---| | nothing set | 10 — all of them | | `max_annotations_per_user: 5` | 5 | | `automatic_assignment.instance_per_annotator: 5` | 10 — **no effect** | | `num_annotators_per_item: 3` | 10 — no effect on the quota | `instance_per_annotator` is not read anywhere in the server, and `automatic_assignment` does not validate its sub-keys, so it produces no warning and no behaviour. Use `max_annotations_per_user`, or `per_annotator_quota` if different annotators need different amounts. Left unset, the cap is the number of items loaded, so every annotator is offered the whole corpus. `-1` is explicit unlimited, and is what a dynamic data source needs for items added after boot to be assignable at all. #### Giving different annotators different amounts `per_annotator_quota` caps annotators individually. Potato resolves one annotator's cap in this order, taking the first that matches: 1. `per_annotator_quota.by_user[]` 2. `per_annotator_quota.by_user_role[]`, where the role comes from the `user_roles` map 3. `per_annotator_quota.default` 4. `max_annotations_per_user` So `default` displaces `max_annotations_per_user` rather than sitting beside it: set both and the global is never read, and nothing warns. Without `default`, an annotator matching neither override falls through to the global, which is how the two keys are meant to combine. Measured on 24 items with `default: 5`, `by_user: {alice: 3}`, `by_user_role: {expert: 7}`, `user_roles: {alice: expert, bob: expert, carol: novice}`, and `max_annotations_per_user: 20`: | Annotator | Served | Why | |---|---|---| | alice | 3 | `by_user` wins over her `expert` role | | bob | 7 | `expert` role | | carol | 5 | role `novice` has no entry, so `default` | | dave | 5 | no role, so `default` | Nobody got 20. An annotator who reaches their cap is shown the completion page, the same as finishing the corpus. A role named in `user_roles` with no matching `by_user_role` entry is not an error and does not warn -- that annotator silently gets `default`, which is carol above. ## Quality-control items do not consume the quota Attention checks and gold items are injected into the annotator's ordering as they work, rather than drawn from the pool at assignment time. They do not count against `max_annotations_per_user`, so enabling them does not shorten the corpus each annotator sees. On a six-article study with checks every third item, both annotators are served all six articles plus the checks: ``` ordering = [n001, n002, n003, check01, n004, n005, n006] ``` You do not need to pad the quota to make room for them. If you set the quota explicitly, set it to the number of **dataset items** an annotator should see. **Verify with a count across every annotator, not one.** Per-annotator ordering looks plausible in isolation; what matters is whether each item reached the number of annotators you asked for: ```python import json, glob, collections c = collections.Counter() for f in glob.glob("annotation_output/*/user_state.json"): c.update(json.load(open(f))["instance_id_to_label_to_value"]) print(sorted(c.items())) # every real item should hit num_annotators_per_item ``` ## More annotators than work `num_annotators_per_item` times the item count is the ceiling on how much work exists. Once it is used up, the next person to register gets a page of its own: > **There is no work left for you** > > Every item in this study already has as many annotators as it needs, so > nothing could be assigned to you. You have not annotated anything, and nothing > has been recorded against your account. > > If you were asked to take part, tell whoever sent you the link -- they can > raise the number of annotators per item or add more data. It offers a **Check again** button, and says that reloading will give them work if more data arrives, which is true of a dynamic data source. The log carries a matching warning naming the cause and the fix: ``` No items could be assigned to carol: every item already has its full complement of annotators (num_annotators_per_item=1). Showing the no-work page rather than the completion page. Raise num_annotators_per_item or add data to give them work. ``` No `annotation_output//` directory is created for them. Three items at `num_annotators_per_item: 1` does this to the fourth person through the door, and on a pilot that person is usually you with a fresh login. Before recruiting, multiply: `items x num_annotators_per_item` is the number of annotator-items you have to hand out, and `max_annotations_per_user` divides it into people. ### Holds and annotations An annotator who is *given* an item holds it, and the hold counts against the item's cap so that people spread out across the corpus instead of piling onto its first item. A hold is not a submission, though, and somebody who takes items and closes the tab would otherwise strand them. So when the cap would send an arriving annotator away with nothing, Potato relaxes it and hands out items that another annotator holds but has not submitted, logging: ``` Assigned 4 item(s) to bob that another annotator holds but has not submitted. Enable instance_reclaim to hand those back automatically instead. ``` **This means an over-subscribed study can collect more annotations per item than `num_annotators_per_item`, and nothing warns.** Measured on 4 items at `num_annotators_per_item: 1`, with no per-user cap: the first annotator was given all four and submitted nothing, the second was given the same four, and when both then answered item 1 it ended up with two annotations under a cap of one. If you pay per judgement, that is real money, and an agreement number computed over it is not the design you configured. Size the pool and it does not happen. The same study at 8 items with `max_annotations_per_user: 4` gave the first annotator items 1-4 and the second items 5-8, no overlap, and the relaxation never fired. The rule is the same multiplication as above: keep `annotators x their quota` at or under `items x num_annotators_per_item`. ### Taking abandoned items back `instance_reclaim` is the machinery the log line points at, and it is off by default: ```yaml instance_reclaim: enabled: true timeout_hours: 0.001 # fractional hours are fine; this is 3.6 seconds ``` An annotator who was given items and never submitted loses them after `timeout_hours`, and they go back into the pool. Items they *did* annotate are never reclaimed. Measured with two annotators over 4 items: the second one's arrival reclaimed all four from the first, and the log named each one. Two things to know before relying on it. It runs **when somebody asks for an assignment**, not on a timer -- it is called from the assignment pass, so on a quiet study nothing is reclaimed until the next annotator arrives. And it does not stop the relaxation above: an annotator whose items were reclaimed, arriving back to a study with nothing free, is handed items the *other* annotator now holds, and the log prints its "Enable instance_reclaim" advice even though `instance_reclaim` is enabled. ## Giving cohorts different items and different questions `batch_assignment` pins named annotators to named items, and `scheme_sets` gives each cohort its own questions. They are usually wanted together. ```yaml assignment_strategy: batch # without this, the groups do not pick items annotation_schemes: - annotation_type: radio name: verdict description: Is this acceptable? labels: ["Yes", "No"] - annotation_type: radio name: severity description: How severe? labels: ["Low", "High"] scheme_sets: quick: - verdict detailed: - verdict - severity batch_assignment: groups: - name: cohort_a annotators: [alice, bob] instances: ["1", "2", "3", "4"] schemes: detailed - name: cohort_b annotators: [carol] instances: ["5", "6", "7", "8"] schemes: quick ``` Measured on that config: alice and bob were served items 1-4 with both questions, carol items 5-8 with only `verdict`. **`assignment_strategy: batch` is what makes the groups pick items.** Leave it out and the study still validates, still boots, and each cohort still gets its own questions -- but every annotator is served every item, because the group's item list is only consulted by the batch strategy. The working half is the problem: cohorts visibly getting different questions reads as proof the config took effect. On the same config without that one line, carol was served items 1 through 5. A `schemes` binding is a `scheme_sets` name, a single scheme's `name`, or an inline list; anything else is refused at load. An annotator in no group falls back to the global `annotation_schemes` -- but only for questions. Under `assignment_strategy: batch` they are served **no items at all**, and get the no-work page telling them every item has its full complement of annotators. That is the wrong reason, and the log says the same: on an unlimited `num_annotators_per_item` it reports every item as already having as many annotators as it needs. If you use `batch_assignment` with a fixed roster, anybody who arrives outside it is turned away with a misleading explanation, so list every annotator you expect. The keys accept several spellings. `annotators` or `users`; `instances`, `items` or `instance_ids`; `instances_file`, `items_file`, `instance_ids_file`, `data_file`, `input_data_file` or `input_file` to read the ids from a file instead; `max_annotators`, `annotator_count` or `slots` for a per-group cap. ## Choosing the number of annotators | Situation | Number | |---|---| | Near-mechanical labels, or prototyping, nobody reporting agreement | 1 | | Any judgement involved and an agreement number will be reported | 3 | | Genuinely subjective constructs, or per-annotator reliability is the object of study | 5+ | Three is the smallest number where a majority can break a tie. More annotators on fewer items beats fewer annotators on more items whenever you do not yet know the labels are learnable — find that out on 100 items before spending the budget on 10,000. This is the researcher's decision, not yours. Propose a number with the reason and say what it costs. ## Order `assignment_strategy` takes: `random`, `fixed_order`, `active_learning`, `llm_confidence`, `max_diversity`, `least_annotated`, `category_based`, `diversity_clustering`, `batch`, `priority`, `model_review`, `psychometric`. `random` with a `random_seed` is right unless there is a reason: - `fixed_order` when items are a narrative and order carries meaning - `least_annotated` when finishing every item matters more than annotator experience — the one to reach for when the quota trap above is biting and you cannot raise the cap - `active_learning` when a model is in the loop and the point is to spend annotator time where it changes something. It needs `assignment_strategy: active_learning`. With `active_learning.enabled: true` on its own the classifier is still fitted and its ordering serves nobody; the reorder log line and `ordering_is_served` in `/admin/active-learning/stats` both name which case you are in. `llm_confidence` sits beside it in the list, is not implemented, and assigns at random ## The response format decides the metric Potato classifies each scheme and computes what fits. You do not choose the metric separately, which makes this a measurement decision rather than a cosmetic one. | Kind | Schemes | What it computes | |---|---|---| | nominal | `radio`, `select`, `triage`, `conjoint`, `multiselect` capped at 1 | pairwise Cohen's κ, Fleiss' κ, nominal α | | ordinal | `likert`, `confidence`, `vas`, `range_slider`, `semantic_differential` | linear and quadratic weighted κ, Spearman ρ, ordinal α | | continuous | `slider`, `number` | Pearson r, MAE, RMSE, interval α, ICC(2,k) | | multilabel | `multiselect`, `hierarchical_multiselect`, `card_sort` | mean Jaccard, MASI α | | matrix | `multirate`, `constant_sum`, `soft_label`, `bws` | the ordinal or continuous measures per row, plus a pooled block over every (item, row) pair | | ranking | `ranking` | Kendall's τ, Spearman footrule | | span | `span`, `error_span`, `coreference`, `span_link`, `event_annotation`, `extractive_qa`, `tree_annotation` | token-level κ, exact and partial span F1, Krippendorff's αU, γ | | geometry | boxes, polygons, points | matched IoU, detection F1, chance-corrected σ, ks, detection α | | | | *(the only way to score a geometry task against a known answer — `gold_label` cannot hold a box)* | | free text | `text`, `text_edit` | nothing | `pairwise` does not follow its type name. It renders three widgets and stores three shapes, so `mode: scale` is scored continuous, `mode: multi_dimension` as a matrix, and the binary default as nominal. Read the kind out of the report. **`mean_matched_iou` averages over matched pairs, not over items.** A region one annotator drew and the other did not is not a zero in it; it is absent from it. Six matched pairs on a study where twelve of twenty regions never matched still reported 0.674, the mean of those six. That number answers "when both of them drew the same thing, how well did the boxes line up" and nothing else. What the disagreement about *whether* a thing is there costs you is in `detection_f1` and `mean_object_count_diff`; quote all three or none. **Region-caption agreement anchors on one annotator, so it is asymmetric.** The first annotator by sorted id defines the units, and everyone else is matched onto them at `match_iou`. Regions only the *second* annotator drew are never units and their captions are never counted, so `n_captions` is the anchor's captions plus one per match, not the number of captions in the data. Twelve anchor regions with six matches gave `n_captions: 18` over twenty stored captions. Swapping who annotates first changes the units, and the module says as much rather than hiding it. `agreement_distance` defaults to `token`, which scores a correct paraphrase as complete disagreement; `embedding` is the one that does not, and it falls back to `token` with a note in the report if `sentence-transformers` is missing. Consequences worth stating to a researcher up front: - **A 1–5 rating stored as a `radio` is scored as unordered**, so "1 vs 2" counts the same as "1 vs 5" and the agreement number is badly understated. Store ordered judgements as `likert`. - **Span agreement tolerates disagreement about boundaries.** Annotators asked to mark sentences will not agree on where a sentence starts; token-level and partial-overlap measures handle that, whereas splitting the text into sentence rows and computing per-sentence κ requires the units to match exactly. - **Free text has no metric.** If a judgement is going to be counted, make it a scale. If it is genuinely prose, plan to read it and say so. Enable the admin-side computation with: ```yaml num_annotators_per_item: 3 agreement_metrics: enabled: true ``` `agreement_metrics` recognizes `enabled` and nothing else — `metrics`, `measures`, `types`, `min_annotators` and `by_scheme` are all rejected by `--strict`. The measures follow from the schema kinds; there is nothing to select. ### Reading the agreement report Nothing is computed at boot, and nothing is linked from the annotation page. **Agreement appears once an item reaches its full annotator cap**, not once it has two annotators. An item sitting at 2 of `num_annotators_per_item: 3` is not scored. Measured on a six-item study: two annotators finished and every coefficient was null, the third finished and the whole report populated. The report tells you which of the two it is. Alongside `n_overlap_items` it returns `n_items_below_cap`, and an `items` block giving each item its cap, the annotators it has, and how many answered each scheme: ```json {"n_overlap_items": 6, "n_items_below_cap": 0, "items": {"p01": {"annotators": ["r1@x.com", "r2@x.com", "r3@x.com"], "cap": 3, "schemas": {"decision": {"n_annotators": 3}, "topics": {"n_annotators": 2}}}}} ``` Read those two before the coefficients. A study waiting on its third annotator and a study nobody has touched both produce nulls, and `n_items_below_cap` is what separates them. The per-item `schemas` counts also show a scheme an annotator skipped, which is the usual reason one scheme is null while the rest of the report is fine. ```bash cat admin_api_key.txt # written into task_dir on the first admin request curl -H "X-API-Key: $(cat admin_api_key.txt)" localhost:8000/admin/iaa ``` The file does not exist at boot. It appears the first time anything asks for an admin route, so the honest sequence is to make one unauthenticated request, let it be refused, and then read the key the refusal caused to be written. `/admin/iaa` returns, per scheme, the `kind` it inferred and the metrics that follow from it, plus `n_overlap_items`. Confirming the `kind` matters more than the number: it is where you find out a rating you meant as ordinal is being scored as nominal. Driven over two studies, eight items each, the kinds came back as: ``` sentiment radio nominal alpha_nominal, fleiss_kappa, pairwise_cohen_kappa severity likert ordinal alpha_ordinal, spearman_rho, weighted_kappa_linear, weighted_kappa_quadratic confidence confidence ordinal the same four issues multiselect multilabel alpha_masi, mean_jaccard rating slider continuous alpha_interval, icc_2_k, mae, pearson_r, rmse reasons constant_sum matrix one block per row, plus `pooled` and `n_rows` design soft_label matrix the same evidence span span token_level_kappa, span_f1_exact, span_f1_partial, krippendorff_alpha_u, gamma_mathet boxes image_annotation geometry detection_f1, mean_agreement, mean_matched_iou, mean_object_count_diff, detection_alpha ``` A `matrix` schema is scored twice over: once per sub-question, and once pooled over every (item, row) pair. Read the per-row block. A rating grid usually has one row everybody agrees on and one that carries the disagreement, and pooling averages the two together. On a three-row `constant_sum` over six items and three annotators, per-row interval α ran 0.86 to 1.0 against a pooled 0.94. Two things to read off that. **The `kind` follows the declared `annotation_type`, not the widget that rendered.** A `likert` given an explicit `labels:` list renders as a radio group — the boot log says "Complex labels detected, using radio layout" — and is still scored ordinal, which is what you wanted. And **a free-text scheme is absent from the report entirely** rather than present with empty metrics, so do not read a missing scheme as a failure. `n_items` is per scheme and counts only items with more than one annotator, so it is normally below `n_overlap_items`. A scheme that reports `n_annotators: 0` and all-null metrics has no item two people both answered — for spans that is easy to hit by accident, because one annotator forgetting to select the label leaves nothing to compare. Without the header, every admin JSON route answers `403 {"error":"Admin access required"}` — which reads like a broken config if you do not know the key exists. `/admin` itself serves HTML with no key. An empty report on a fresh task means "no overlapping annotations yet". ## Norming a codebook in a live room `rooms` is a live multiplayer session at `/rooms`. It belongs in this file rather than in a feature list because its output is annotations and its headline number is Krippendorff's alpha, computed twice. ```yaml rooms: enabled: true schema: sarcasm # the radio or likert scheme members vote on persist_votes: true who_can_create: any ``` Members join, vote blind on the current item, the host reveals, everyone argues, anyone may change their vote, the host advances. Three kinds of room: | `room_type` | Items come from | Who votes | |---|---|---| | `norming` | the `item_ids` the creator passes, or a random sample | every active member | | `huddle` | items the team already disagrees on, found live | every active member | | `shadow` | as for norming | the host; everyone else joins as an observer | Only the host reveals, advances and closes — a member who asks gets `Only the host can do that` — and observers cannot vote. The blind phase is genuinely blind on the room page: before the reveal, a member's poll of the room state and of the event stream carries who has voted and not what they voted. **Set `rooms.schema` rather than leaving it to the autopick.** With the key absent, the vote target is the first `radio` or `likert` scheme in the config, which moves when you reorder the file. `potato validate` covers both ways of getting it wrong, and `--strict` fails on either: nothing to vote on at all warns that "rooms will be disabled and /rooms will return 404", and a `rooms.schema` naming a scheme that does not exist warns that "rooms will refuse to open" and lists the names it did find. That second one is the dangerous shape, because a misspelling boots looking healthy and only fails when somebody tries to open a room. ### What a room writes into each member's annotations With `persist_votes: true` (the default), each member's vote is written into their own annotations for the room's schema, at the moment **the host advances past a revealed item, or closes the room on one**. An item nobody revealed is marked skipped and writes nothing, so a room abandoned mid-discussion leaves that item's votes only in the log. Two properties to say out loud to a researcher before they run one: - **What lands is the vote after the discussion, not the blind one.** A member who voted `Sincere`, watched the room go the other way and changed to `Sarcastic` has `Sarcastic` in her annotations. Her blind vote survives only in `/rooms/room-.jsonl`. A study that needs independent judgements cannot source them from a room. - **The write replaces that member's existing answer for the room's schema, and touches no other schema.** Measured on a config carrying `sarcasm` and a `text` scheme called `notes`: the room overwrote both members' `sarcasm` answer for the item and left both `notes` entries as they were. ### The two alphas The room state carries a `metrics` block that computes nominal Krippendorff's alpha over revealed items twice, on the blind votes and on the current ones. The gap between them is the room's norming lift. Three annotators, two revealed items, one of whom changed her vote after both reveals: ``` blind_alpha -0.111 final_alpha 1.000 alpha_lift 1.111 total_changes 2 toward_majority 2 ``` `toward_majority` counts a post-reveal change that landed on the majority of *everyone else*: the changer's own vote is excluded from the count, so a two-person room can record conformity instead of sitting at a permanent tie. Both alphas are `null` until two items have been revealed. A room one item in reports `n_revealed: 1` and three nulls, which is undefined rather than broken. **In a huddle those two numbers do not mean what they mean in a norming room.** A huddle seeds itself from the team's live disagreements and shows every member's original annotation, by name, as context, before anyone in the room has voted. Measured: a member's state during the voting phase, no votes cast, already carried ``` seed_votes {"frank@example.com": "Sincere", "gina@example.com": "Sarcastic"} ``` That is what a huddle is for, and it is not blind, so `blind_alpha` there is agreement among people who are looking at each other's answers. Quote the lift out of a norming room. Out of a huddle, quote `final_alpha` and the conformity counts and say where they came from. **A room raises the study's own agreement score, and the report cannot see why.** A room-written vote registers its annotator against the item like any other annotation, so a room can carry an item to its cap and put it into the agreement report above. Measured: two annotators who voted `Sarcastic` and `Sincere` blind, argued, and converged left `/admin/iaa` reporting `percent_agreement: 1.0` for that item, with nothing anywhere in the report to say they had been in a room together. Kappa and alpha came back `null`, with a note that every annotator gave the same label, because that one item was all the report had. Norm on items you are willing to spend, and keep them out of the set you report agreement on. ## Related - `quality-control.md` — the checks that consume the quota - `designing-a-task.md` — the seven decisions, including how many annotators - `running-a-task.md` — reading `instance_id_ordering` to see what was assigned