Assignment and Agreement¶

Who annotates what, and which agreement measure follows from it. It ships with the potato-skill skill:

/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato

Two questions that look separate and are not: the number of annotators per item decides whether an agreement number exists at all, and the per-annotator quota decides whether the items you care about ever reach three annotators.

The keys¶

Key	What it does	Default
`num_annotators_per_item`	Annotators per item; an item retires when it has this many. Set this one.	unset: no limit (`0` too)
`min_annotators_per_instance`	Deprecated spelling of the key above, read as the cap with a warning	unset
`max_annotations_per_item`	Older spelling of the same cap; `-1` unlimited	-1
`max_annotations_per_user`	Cap on dataset items per annotator, unless `per_annotator_quota` overrides it	the item count
`per_annotator_quota`	Per-annotator caps: `default`, `by_user`, `by_user_role`. Beats the key above	unset
`automatic_assignment.instance_per_annotator`	Nothing. See below.	—
`assignment_strategy`	Order items are handed out in	`fixed_order`
`random_seed`	Makes the ordering reproducible	unset

The three are one setting. With none of them set there is no cap: on 2.11.0, five annotators of a two-item study were all served the first item. Two of them with the same number validate clean, and two with different numbers are refused:

'max_annotations_per_item' and 'num_annotators_per_item' are both set with
conflicting values (5 vs 3). Drop 'max_annotations_per_item' —
'num_annotators_per_item' is the canonical key.

min_annotators_per_instance used to be stored at load and read nowhere (checked against Potato be457e7b), so a study that set it collected as many annotators as it got. From 2.11 it is read as the cap. Set to 2 on its own, the first two annotators got the first item, the third got the second, and the fourth and fifth got nothing. Validation warns: min_annotators_per_instance: 2 is deprecated and has been read as num_annotators_per_item: 2. Rename it. (The key of the same name under icl_labeling.example_selection is a different setting, and that one is read.)

`num_annotators_per_item.adaptive`¶

num_annotators_per_item also takes a mapping, and the adaptive block under it reopens an item when the first annotators disagree:

num_annotators_per_item:
  default: 2
  adaptive:
    enabled: true
    disagreement_threshold: 0.5
    boost_to: 4

Measured on 8 items with two annotators who disagreed on x0-x3 and agreed on x4-x7. The server logged four lines and only four:

Adaptive boost: x0 raised to 4 annotators (current=2)

The third and fourth annotators through the door were each served exactly x0 x1 x2 x3; the agreed half had retired at 2; the fifth got nothing. So the budget goes where the label is actually contested, and a study that would have cost 8 x 4 = 32 annotations cost 24.

Worth knowing before you reach for it: this is a cap being raised, so the arithmetic in the section above stops holding. items x num_annotators_per_item is no longer the ceiling on how much work exists — with boost_to: 4 on a default: 2 study the true ceiling is twice what default suggests, and how close you get depends on how much your annotators disagree, which is the thing you do not know yet. Recruit against boost_to, not against default.

`instance_per_annotator`¶

Measured on 10 items, reading the assignment the server actually built:

Config	Items served to one annotator
nothing set	10 — all of them
`max_annotations_per_user: 5`	5
`automatic_assignment.instance_per_annotator: 5`	10 — no effect
`num_annotators_per_item: 3`	10 — no effect on the quota

instance_per_annotator is not read anywhere in the server, and automatic_assignment does not validate its sub-keys, so it produces no warning and no behaviour. Use max_annotations_per_user, or per_annotator_quota if different annotators need different amounts.

Left unset, the cap is the number of items loaded, so every annotator is offered the whole corpus. -1 is explicit unlimited, and is what a dynamic data source needs for items added after boot to be assignable at all.

Giving different annotators different amounts¶

per_annotator_quota caps annotators individually. Potato resolves one annotator's cap in this order, taking the first that matches:

per_annotator_quota.by_user[<user>]
per_annotator_quota.by_user_role[<their role>], where the role comes from the user_roles map
per_annotator_quota.default
max_annotations_per_user

So default displaces max_annotations_per_user rather than sitting beside it: set both and the global is never read, and nothing warns. Without default, an annotator matching neither override falls through to the global, which is how the two keys are meant to combine.

Measured on 24 items with default: 5, by_user: {alice: 3}, by_user_role: {expert: 7}, user_roles: {alice: expert, bob: expert, carol: novice}, and max_annotations_per_user: 20:

Annotator	Served	Why
alice	3	`by_user` wins over her `expert` role
bob	7	`expert` role
carol	5	role `novice` has no entry, so `default`
dave	5	no role, so `default`

Nobody got 20. An annotator who reaches their cap is shown the completion page, the same as finishing the corpus.

A role named in user_roles with no matching by_user_role entry is not an error and does not warn -- that annotator silently gets default, which is carol above.

Quality-control items do not consume the quota¶

Attention checks and gold items are injected into the annotator's ordering as they work, rather than drawn from the pool at assignment time. They do not count against max_annotations_per_user, so enabling them does not shorten the corpus each annotator sees. On a six-article study with checks every third item, both annotators are served all six articles plus the checks:

ordering = [n001, n002, n003, check01, n004, n005, n006]

You do not need to pad the quota to make room for them. If you set the quota explicitly, set it to the number of dataset items an annotator should see.

Verify with a count across every annotator, not one. Per-annotator ordering looks plausible in isolation; what matters is whether each item reached the number of annotators you asked for:

import json, glob, collections
c = collections.Counter()
for f in glob.glob("annotation_output/*/user_state.json"):
    c.update(json.load(open(f))["instance_id_to_label_to_value"])
print(sorted(c.items()))          # every real item should hit num_annotators_per_item

More annotators than work¶

num_annotators_per_item times the item count is the ceiling on how much work exists. Once it is used up, the next person to register gets a page of its own:

There is no work left for you

Every item in this study already has as many annotators as it needs, so nothing could be assigned to you. You have not annotated anything, and nothing has been recorded against your account.

If you were asked to take part, tell whoever sent you the link -- they can raise the number of annotators per item or add more data.

It offers a Check again button, and says that reloading will give them work if more data arrives, which is true of a dynamic data source. The log carries a matching warning naming the cause and the fix:

No items could be assigned to carol: every item already has its full complement
of annotators (num_annotators_per_item=1). Showing the no-work page rather than
the completion page. Raise num_annotators_per_item or add data to give them work.

No annotation_output/<user>/ directory is created for them. Three items at num_annotators_per_item: 1 does this to the fourth person through the door, and on a pilot that person is usually you with a fresh login.

Before recruiting, multiply: items x num_annotators_per_item is the number of annotator-items you have to hand out, and max_annotations_per_user divides it into people.

Holds and annotations¶

An annotator who is given an item holds it, and the hold counts against the item's cap so that people spread out across the corpus instead of piling onto its first item. A hold is not a submission, though, and somebody who takes items and closes the tab would otherwise strand them. So when the cap would send an arriving annotator away with nothing, Potato relaxes it and hands out items that another annotator holds but has not submitted, logging:

Assigned 4 item(s) to bob that another annotator holds but has not submitted.
Enable instance_reclaim to hand those back automatically instead.

This means an over-subscribed study can collect more annotations per item than num_annotators_per_item, and nothing warns. Measured on 4 items at num_annotators_per_item: 1, with no per-user cap: the first annotator was given all four and submitted nothing, the second was given the same four, and when both then answered item 1 it ended up with two annotations under a cap of one. If you pay per judgement, that is real money, and an agreement number computed over it is not the design you configured.

Size the pool and it does not happen. The same study at 8 items with max_annotations_per_user: 4 gave the first annotator items 1-4 and the second items 5-8, no overlap, and the relaxation never fired. The rule is the same multiplication as above: keep annotators x their quota at or under items x num_annotators_per_item.

Taking abandoned items back¶

instance_reclaim is the machinery the log line points at, and it is off by default:

instance_reclaim:
  enabled: true
  timeout_hours: 0.001    # fractional hours are fine; this is 3.6 seconds

An annotator who was given items and never submitted loses them after timeout_hours, and they go back into the pool. Items they did annotate are never reclaimed. Measured with two annotators over 4 items: the second one's arrival reclaimed all four from the first, and the log named each one.

Two things to know before relying on it. It runs when somebody asks for an assignment, not on a timer -- it is called from the assignment pass, so on a quiet study nothing is reclaimed until the next annotator arrives. And it does not stop the relaxation above: an annotator whose items were reclaimed, arriving back to a study with nothing free, is handed items the other annotator now holds, and the log prints its "Enable instance_reclaim" advice even though instance_reclaim is enabled.

Giving cohorts different items and different questions¶

batch_assignment pins named annotators to named items, and scheme_sets gives each cohort its own questions. They are usually wanted together.

assignment_strategy: batch          # without this, the groups do not pick items

annotation_schemes:
  - annotation_type: radio
    name: verdict
    description: Is this acceptable?
    labels: ["Yes", "No"]
  - annotation_type: radio
    name: severity
    description: How severe?
    labels: ["Low", "High"]

scheme_sets:
  quick:
    - verdict
  detailed:
    - verdict
    - severity

batch_assignment:
  groups:
    - name: cohort_a
      annotators: [alice, bob]
      instances: ["1", "2", "3", "4"]
      schemes: detailed
    - name: cohort_b
      annotators: [carol]
      instances: ["5", "6", "7", "8"]
      schemes: quick

Measured on that config: alice and bob were served items 1-4 with both questions, carol items 5-8 with only verdict.

assignment_strategy: batch is what makes the groups pick items. Leave it out and the study still validates, still boots, and each cohort still gets its own questions -- but every annotator is served every item, because the group's item list is only consulted by the batch strategy. The working half is the problem: cohorts visibly getting different questions reads as proof the config took effect. On the same config without that one line, carol was served items 1 through 5.

A schemes binding is a scheme_sets name, a single scheme's name, or an inline list; anything else is refused at load. An annotator in no group falls back to the global annotation_schemes -- but only for questions. Under assignment_strategy: batch they are served no items at all, and get the no-work page telling them every item has its full complement of annotators. That is the wrong reason, and the log says the same: on an unlimited num_annotators_per_item it reports every item as already having as many annotators as it needs. If you use batch_assignment with a fixed roster, anybody who arrives outside it is turned away with a misleading explanation, so list every annotator you expect.

The keys accept several spellings. annotators or users; instances, items or instance_ids; instances_file, items_file, instance_ids_file, data_file, input_data_file or input_file to read the ids from a file instead; max_annotators, annotator_count or slots for a per-group cap.

Choosing the number of annotators¶

Situation	Number
Near-mechanical labels, or prototyping, nobody reporting agreement	1
Any judgement involved and an agreement number will be reported	3
Genuinely subjective constructs, or per-annotator reliability is the object of study	5+

Three is the smallest number where a majority can break a tie. More annotators on fewer items beats fewer annotators on more items whenever you do not yet know the labels are learnable — find that out on 100 items before spending the budget on 10,000.

This is the researcher's decision, not yours. Propose a number with the reason and say what it costs.

Order¶

assignment_strategy takes: random, fixed_order, active_learning, llm_confidence, max_diversity, least_annotated, category_based, diversity_clustering, batch, priority, model_review, psychometric.

random with a random_seed is right unless there is a reason:

fixed_order when items are a narrative and order carries meaning
least_annotated when finishing every item matters more than annotator experience — the one to reach for when the quota trap above is biting and you cannot raise the cap
active_learning when a model is in the loop and the point is to spend annotator time where it changes something. It needs assignment_strategy: active_learning. With active_learning.enabled: true on its own the classifier is still fitted and its ordering serves nobody; the reorder log line and ordering_is_served in /admin/active-learning/stats both name which case you are in. llm_confidence sits beside it in the list, is not implemented, and assigns at random

The response format decides the metric¶

Potato classifies each scheme and computes what fits. You do not choose the metric separately, which makes this a measurement decision rather than a cosmetic one.

Kind	Schemes	What it computes
nominal	`radio`, `select`, `triage`, `conjoint`, `multiselect` capped at 1	pairwise Cohen's κ, Fleiss' κ, nominal α
ordinal	`likert`, `confidence`, `vas`, `range_slider`, `semantic_differential`	linear and quadratic weighted κ, Spearman ρ, ordinal α
continuous	`slider`, `number`	Pearson r, MAE, RMSE, interval α, ICC(2,k)
multilabel	`multiselect`, `hierarchical_multiselect`, `card_sort`	mean Jaccard, MASI α
matrix	`multirate`, `constant_sum`, `soft_label`, `bws`	the ordinal or continuous measures per row, plus a pooled block over every (item, row) pair
ranking	`ranking`	Kendall's τ, Spearman footrule
span	`span`, `error_span`, `coreference`, `span_link`, `event_annotation`, `extractive_qa`, `tree_annotation`	token-level κ, exact and partial span F1, Krippendorff's αU, γ
geometry	boxes, polygons, points	matched IoU, detection F1, chance-corrected σ, ks, detection α
		(the only way to score a geometry task against a known answer — `gold_label` cannot hold a box)
free text	`text`, `text_edit`	nothing

pairwise does not follow its type name. It renders three widgets and stores three shapes, so mode: scale is scored continuous, mode: multi_dimension as a matrix, and the binary default as nominal. Read the kind out of the report.

mean_matched_iou averages over matched pairs, not over items. A region one annotator drew and the other did not is not a zero in it; it is absent from it. Six matched pairs on a study where twelve of twenty regions never matched still reported 0.674, the mean of those six. That number answers "when both of them drew the same thing, how well did the boxes line up" and nothing else. What the disagreement about whether a thing is there costs you is in detection_f1 and mean_object_count_diff; quote all three or none.

Region-caption agreement anchors on one annotator, so it is asymmetric. The first annotator by sorted id defines the units, and everyone else is matched onto them at match_iou. Regions only the second annotator drew are never units and their captions are never counted, so n_captions is the anchor's captions plus one per match, not the number of captions in the data. Twelve anchor regions with six matches gave n_captions: 18 over twenty stored captions. Swapping who annotates first changes the units, and the module says as much rather than hiding it. agreement_distance defaults to token, which scores a correct paraphrase as complete disagreement; embedding is the one that does not, and it falls back to token with a note in the report if sentence-transformers is missing.

Consequences worth stating to a researcher up front:

A 1–5 rating stored as a radio is scored as unordered, so "1 vs 2" counts the same as "1 vs 5" and the agreement number is badly understated. Store ordered judgements as likert.
Span agreement tolerates disagreement about boundaries. Annotators asked to mark sentences will not agree on where a sentence starts; token-level and partial-overlap measures handle that, whereas splitting the text into sentence rows and computing per-sentence κ requires the units to match exactly.
Free text has no metric. If a judgement is going to be counted, make it a scale. If it is genuinely prose, plan to read it and say so.

Enable the admin-side computation with:

num_annotators_per_item: 3
agreement_metrics:
  enabled: true

agreement_metrics recognizes enabled and nothing else — metrics, measures, types, min_annotators and by_scheme are all rejected by --strict. The measures follow from the schema kinds; there is nothing to select.

Reading the agreement report¶

Nothing is computed at boot, and nothing is linked from the annotation page.

Agreement appears once an item reaches its full annotator cap, not once it has two annotators. An item sitting at 2 of num_annotators_per_item: 3 is not scored. Measured on a six-item study: two annotators finished and every coefficient was null, the third finished and the whole report populated.

The report tells you which of the two it is. Alongside n_overlap_items it returns n_items_below_cap, and an items block giving each item its cap, the annotators it has, and how many answered each scheme:

{"n_overlap_items": 6, "n_items_below_cap": 0,
 "items": {"p01": {"annotators": ["r1@x.com", "r2@x.com", "r3@x.com"], "cap": 3,
                   "schemas": {"decision": {"n_annotators": 3},
                               "topics": {"n_annotators": 2}}}}}

Read those two before the coefficients. A study waiting on its third annotator and a study nobody has touched both produce nulls, and n_items_below_cap is what separates them. The per-item schemas counts also show a scheme an annotator skipped, which is the usual reason one scheme is null while the rest of the report is fine.

cat admin_api_key.txt        # written into task_dir on the first admin request
curl -H "X-API-Key: $(cat admin_api_key.txt)" localhost:8000/admin/iaa

The file does not exist at boot. It appears the first time anything asks for an admin route, so the honest sequence is to make one unauthenticated request, let it be refused, and then read the key the refusal caused to be written.

/admin/iaa returns, per scheme, the kind it inferred and the metrics that follow from it, plus n_overlap_items. Confirming the kind matters more than the number: it is where you find out a rating you meant as ordinal is being scored as nominal.

Driven over two studies, eight items each, the kinds came back as:

sentiment   radio            nominal     alpha_nominal, fleiss_kappa, pairwise_cohen_kappa
severity    likert           ordinal     alpha_ordinal, spearman_rho,
                                         weighted_kappa_linear, weighted_kappa_quadratic
confidence  confidence       ordinal     the same four
issues      multiselect      multilabel  alpha_masi, mean_jaccard
rating      slider           continuous  alpha_interval, icc_2_k, mae, pearson_r, rmse
reasons     constant_sum     matrix      one block per row, plus `pooled` and `n_rows`
design      soft_label       matrix      the same
evidence    span             span        token_level_kappa, span_f1_exact,
                                         span_f1_partial, krippendorff_alpha_u, gamma_mathet
boxes       image_annotation geometry    detection_f1, mean_agreement, mean_matched_iou,
                                         mean_object_count_diff, detection_alpha

A matrix schema is scored twice over: once per sub-question, and once pooled over every (item, row) pair. Read the per-row block. A rating grid usually has one row everybody agrees on and one that carries the disagreement, and pooling averages the two together. On a three-row constant_sum over six items and three annotators, per-row interval α ran 0.86 to 1.0 against a pooled 0.94.

Two things to read off that. The kind follows the declared annotation_type, not the widget that rendered. A likert given an explicit labels: list renders as a radio group — the boot log says "Complex labels detected, using radio layout" — and is still scored ordinal, which is what you wanted. And a free-text scheme is absent from the report entirely rather than present with empty metrics, so do not read a missing scheme as a failure.

n_items is per scheme and counts only items with more than one annotator, so it is normally below n_overlap_items. A scheme that reports n_annotators: 0 and all-null metrics has no item two people both answered — for spans that is easy to hit by accident, because one annotator forgetting to select the label leaves nothing to compare.

Without the header, every admin JSON route answers 403 {"error":"Admin access required"} — which reads like a broken config if you do not know the key exists. /admin itself serves HTML with no key.

An empty report on a fresh task means "no overlapping annotations yet".

Norming a codebook in a live room¶

rooms is a live multiplayer session at /rooms. It belongs in this file rather than in a feature list because its output is annotations and its headline number is Krippendorff's alpha, computed twice.

rooms:
  enabled: true
  schema: sarcasm          # the radio or likert scheme members vote on
  persist_votes: true
  who_can_create: any

Members join, vote blind on the current item, the host reveals, everyone argues, anyone may change their vote, the host advances. Three kinds of room:

`room_type`	Items come from	Who votes
`norming`	the `item_ids` the creator passes, or a random sample	every active member
`huddle`	items the team already disagrees on, found live	every active member
`shadow`	as for norming	the host; everyone else joins as an observer

Only the host reveals, advances and closes — a member who asks gets Only the host can do that — and observers cannot vote. The blind phase is genuinely blind on the room page: before the reveal, a member's poll of the room state and of the event stream carries who has voted and not what they voted.

Set rooms.schema rather than leaving it to the autopick. With the key absent, the vote target is the first radio or likert scheme in the config, which moves when you reorder the file. potato validate covers both ways of getting it wrong, and --strict fails on either: nothing to vote on at all warns that "rooms will be disabled and /rooms will return 404", and a rooms.schema naming a scheme that does not exist warns that "rooms will refuse to open" and lists the names it did find. That second one is the dangerous shape, because a misspelling boots looking healthy and only fails when somebody tries to open a room.

What a room writes into each member's annotations¶

With persist_votes: true (the default), each member's vote is written into their own annotations for the room's schema, at the moment the host advances past a revealed item, or closes the room on one. An item nobody revealed is marked skipped and writes nothing, so a room abandoned mid-discussion leaves that item's votes only in the log.

Two properties to say out loud to a researcher before they run one:

What lands is the vote after the discussion, not the blind one. A member who voted Sincere, watched the room go the other way and changed to Sarcastic has Sarcastic in her annotations. Her blind vote survives only in <output_annotation_dir>/rooms/room-<id>.jsonl. A study that needs independent judgements cannot source them from a room.
The write replaces that member's existing answer for the room's schema, and touches no other schema. Measured on a config carrying sarcasm and a text scheme called notes: the room overwrote both members' sarcasm answer for the item and left both notes entries as they were.

The two alphas¶

The room state carries a metrics block that computes nominal Krippendorff's alpha over revealed items twice, on the blind votes and on the current ones. The gap between them is the room's norming lift. Three annotators, two revealed items, one of whom changed her vote after both reveals:

blind_alpha       -0.111
final_alpha        1.000
alpha_lift         1.111
total_changes          2
toward_majority        2

toward_majority counts a post-reveal change that landed on the majority of everyone else: the changer's own vote is excluded from the count, so a two-person room can record conformity instead of sitting at a permanent tie.

Both alphas are null until two items have been revealed. A room one item in reports n_revealed: 1 and three nulls, which is undefined rather than broken.

In a huddle those two numbers do not mean what they mean in a norming room. A huddle seeds itself from the team's live disagreements and shows every member's original annotation, by name, as context, before anyone in the room has voted. Measured: a member's state during the voting phase, no votes cast, already carried

seed_votes  {"frank@example.com": "Sincere", "gina@example.com": "Sarcastic"}

That is what a huddle is for, and it is not blind, so blind_alpha there is agreement among people who are looking at each other's answers. Quote the lift out of a norming room. Out of a huddle, quote final_alpha and the conformity counts and say where they came from.

A room raises the study's own agreement score, and the report cannot see why. A room-written vote registers its annotator against the item like any other annotation, so a room can carry an item to its cap and put it into the agreement report above. Measured: two annotators who voted Sarcastic and Sincere blind, argued, and converged left /admin/iaa reporting percent_agreement: 1.0 for that item, with nothing anywhere in the report to say they had been in a room together. Kappa and alpha came back null, with a note that every annotator gave the same label, because that one item was all the report had. Norm on items you are willing to spend, and keep them out of the set you report agreement on.

quality-control.md — the checks that consume the quota
designing-a-task.md — the seven decisions, including how many annotators
running-a-task.md — reading instance_id_ordering to see what was assigned