# Attention Checks and Gold Standards The quality-control ladder, and what each rung costs. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Four blocks, three of which need a side file whose format is documented nowhere and whose error messages mostly do not name the missing field. All formats below were recovered by booting the server against candidate files and reading the log. **The failure mode to design against:** every one of these can be enabled, validate clean under `--strict`, log a warning nobody reads, and run the entire study with the feature off. Two of them do not even log an error. After any change here, boot the server and check the count. The second failure mode is subtler and worse: the feature runs, scores everyone, and reports a number that is not what you think it is. Read "Where the results live" below before you act on an accuracy figure. ```bash grep -E "Loaded [0-9]+ (training instances|attention check items|gold standard items)" server.log ``` | Feature | Healthy line | If it says 0 | |---|---|---| | training | `Loaded 3 training instances` | Boot **fails** with an exception. Loud. | | attention checks | `Loaded 2 attention check items` | `WARNING … missing required fields` then silence. Study runs unchecked. | | gold standards | `Loaded 2 gold standard items` | `ERROR: Gold standards are enabled but NOT RUNNING`. Names the missing field. | ## Training A qualification round of worked examples, marked as the annotator answers, before they touch real items. It is the only quality-control feature that improves the annotations rather than only measuring them. ```yaml training: enabled: true data_file: data/training.json allow_retry: true feedback: show_explanations: true passing_criteria: min_correct: 3 ``` Recognized sub-keys: `enabled`, `data_file`, `allow_retry`, `feedback`, `passing_criteria`, `failure_action`, `annotation_schemes`. That is all seven. At this level `min_accuracy`, `max_attempts`, `num_items`, `show_feedback`, `threshold` and every other plausible name is rejected by `--strict`, which is how you find the real ones. Inside the two nested blocks an unknown key is a warning, not an error. `passing_criteria` takes `min_correct`, `require_all_correct`, `max_mistakes`, `max_mistakes_per_question` and `max_attempts`; `feedback` takes `show_explanations` and `allow_retry`. Anything else passes validation and logs `Unknown key training.feedback.enabled is ignored (known: allow_retry, show_explanations)`, so `min_accuracy: 0.6` under `passing_criteria` validates and does nothing. The pass mark is `min_correct`, a count of correct answers rather than a share; a value that is not a positive integer is a fatal error. `failure_action` is `move_to_done` (default) or `repeat_training`, and any other value is a fatal error. `annotation_schemes` lets the practice questions use different schemes from the real task; omit it and training uses the task's own. `feedback` and `passing_criteria` must be dictionaries; a bare `true` or `0.6` is a fatal error. Also add the phase, or the block does nothing: ```yaml phases: order: [consent, training, annotation] consent: type: consent title: Consent to take part file: surveyflow/consent.jsonl training: type: training title: Practice annotation: type: annotation ``` Define every phase you name in `order`. Naming one you do not define is a `--strict` failure, and the phase is dropped from the order rather than silently half-present. See `phases-and-pages.md`. ### The training data file An **object**, not an array, with a `training_instances` key: ```json { "training_instances": [ { "id": "train01", "text": "The energy regulator fined the utility £2 million on Monday …", "correct_answers": { "tone": "Neutral", "responsibility": "A company or developer", "misleading": "No" }, "explanation": "Wire copy. Every sentence is attributed reporting, so …" } ] } ``` **Required per instance: `id`, `text`, `correct_answers`.** Nothing else. Keys in `correct_answers` are scheme `name`s; values are the label strings exactly as they appear in `labels:` (the stored value, not the humanized display form). Three ways this goes wrong, with the exact message: | Message | Cause | |---|---| | `Training data must be a JSON object` | You wrote a JSON array | | `Training data must contain 'training_instances' field` | Right shape, wrong key | | `Training instance missing required fields: {…}` | Dumps the whole instance and **never names the field**. It is one of `id`, `text`, `correct_answers`. | Notes: - `explanation` is shown with the mark. Write it — a wrong answer with no explanation teaches nothing, and this is the one place in the task where you can correct a misreading before it contaminates 500 items. - The training page renders the question text only. Any context an annotator needs (a headline, a speaker, an outlet) must be folded into that string; the display fields from `instance_display` are not used here. From Potato 2.8.2 a training instance keeps all its fields and the question comes from the project's `text_key`, so `{"text": ...}` and `{"caption": ...}` both work where `text_key: caption`. Earlier builds kept six keys and dropped the rest. - **A practice round still cannot show media.** With no `instance_display` there is no `` for an `image_annotation` or other geometry scheme to take its bitmap from, and `source_field` does not save it: the instance fields the client would read are not serialized onto a phase page. The trainee gets an empty canvas, the console says `[ImageAnnotation] No image URL found! Check that the instance data contains an image URL`, and the config validates clean. `check_ui.py` flags it as "a geometry scheme on a page with no image on it". Potato ships no example combining an image display with a training phase, which is why nothing catches it. Either drop `training` on a media task, or use it only for the classification schemes and say in the `explanation` that the drawing tool is practised on the first real item. (Before 2.8.2 this failed more loudly and more confusingly: the canvas read "Failed to load image. Check the URL or CORS settings." and the console showed a 404 for a URL made out of the instance's own prose.) - Grading is per scheme against `correct_answers`. Schemes you omit are not graded. Span schemes are not graded. There is no way to express a gold span here. - Progress shows as `Question 1 / 3` with a running `Correct: n · Mistakes: n`, and a correct answer advances with `Correct! Moving to next question.` - State lands in `annotation_output//user_state.json` under `training_state`: per-question `correct`/`attempts`, plus `passed`. ## Attention checks Items with an obvious right answer, mixed into the stream, to catch someone who stopped reading halfway through. ```yaml attention_checks: enabled: true frequency: 3 # every third item; use probability OR frequency, never both items_file: data/attention.json min_response_time: 8 # a faster answer fails the check failure_handling: warn_threshold: 2 # warn from the second failure warn_message: "Please read each item before answering." block_threshold: 5 # block at the fifth ``` `failure_handling` reads four keys and no others: `warn_threshold`, `warn_message`, `block_threshold`, `block_message`. There is no `action` key. A sub-key it does not know is a warning rather than an error, so a config naming one validates without `--strict` and does nothing. The thresholds then fall back to 2 and 5 (`quality_control.py:375`). Setting both `frequency` and `probability` is a fatal error. `min_response_time` fails a check answered faster than the threshold, whatever the content was. Set it well under a plausible reading time — the point is to catch a straight-through click, not to punish someone who recognised the item — and remember it is one signal among several: `/admin/api/suspicious_activity` and the behavioral analytics see patterns a per-item threshold cannot. ### The attention item file A plain JSON array. **Required per item: `id` and `expected_answer`.** ```json [ { "id": "check01", "text": "This item is a check that you are still reading carefully. It is not a real article. Highlight this sentence and mark it Factual reporting. For the tone question, select Neutral …", "expected_answer": {"tone": "Neutral", "responsibility": "No one", "misleading": "No"} } ] ``` Give the items whatever display fields your `instance_display` reads (`headline`, `body`, …) as well. They render as a normal item and look broken without them. Two traps worth designing around: - **Instruct every required scheme. Span schemes included.** A check item that says "select Neutral" but says nothing about the highlight leaves the annotator stuck: `Next` returns `400 Required annotation(s) not completed: ` and the only feedback is a small toast naming the internal scheme name. - **Attention checks consume item slots.** They are inserted into the annotator's ordering, so with a per-annotator cap equal to the item count they displace real items — annotators finish having never seen the last two. See `assignment-and-agreement.md`. ## Gold standards Items with known labels used to score annotators continuously, rather than once in training. Use them when annotators are paid strangers. ```yaml gold_standards: enabled: true items_file: data/gold.json mode: mixed # training | mixed | separate frequency: 5 accuracy: min_threshold: 0.7 evaluation_count: 10 auto_promote: enabled: false ``` **`mode` takes three values and `mixed` is the one that is implemented.** `separate` behaves identically to it and says so at boot — `gold_standards.mode 'separate' behaves as 'mixed': 3 gold items are injected into the annotation stream every 2 items` — and `training` serves nothing at all, because no code path delivers gold during the training phase. That last one fails `--strict` and warns twice at boot, naming the count that will never be served. Grade a training phase with its own practice questions instead. **`accuracy` and `auto_promote` must both be dictionaries**, even though the key documentation types them `number|object` and `boolean|object`. `accuracy: 0.7` and `auto_promote: false` both fail with `must be a dictionary`. Their documented sub-keys are `min_threshold` / `evaluation_count` and `min_annotators` / `agreement_threshold` — and nothing checks those, because validation stops one level in. A wrong sub-key here validates clean and does nothing. **The accuracy threshold reports to you, not to the annotator.** Crossing it logs ` is below the gold standard accuracy threshold: 0.50 over 2 evaluated items, threshold 0.90` and puts the annotator in `gold_standards.below_accuracy_threshold` on the admin route. It does not block them and there is no setting that does — attention checks block, gold standards never do. If you need underperformers stopped rather than counted, that is `attention_checks.failure_handling` or your own procedure. ### How many gold items an annotator sees `frequency` sets the spacing, but **the file governs the count**: every annotator is served every item in `items_file`, once. Two items in the file with `frequency: 4` over eight real items gives two insertions per annotator, not `8 / 4`. Size the file to how many checks you want each person to get. They are injected on top of the per-annotator quota rather than inside it, so there is nothing to pad (see `assignment-and-agreement.md`). ### The gold item file A JSON array. **Required per item: `gold_label`** (plus `id` and the display fields). Both shapes load: ```json [ {"id": "g1", "text": "…", "gold_label": {"scheme_name": "label"}}, {"id": "g2", "text": "…", "gold_label": "label"} ] ``` Use the dict form on any task with more than one scheme. Two things about gold scoring worth telling a researcher before they read the number: - **A drawn answer can be a gold answer.** Boxes, polygons, masks and points are graded by overlap at 0.5 IoU, not by equality, so "did they find the thing I planted" is a question gold standards can now answer. Write the shape in the form the client stores — normalized fractional coordinates nested under `coordinates`, not pixels: ```json {"id": "gold_dog", "image_url": "…", "gold_label": {"region": [{"type": "bbox", "label": "subject", "coordinates": {"x": 0.18, "y": 0.20, "width": 0.43, "height": 0.40}}]}} ``` On an `image_annotation` task with that item, an annotator drawing the same box scored 1.0 and one drawing a box in the opposite corner scored 0.0. Every gold shape must be matched by a user shape of the same label, with no extras — boxing the whole image does not count as finding the object. Spans are stored elsewhere and I have not tested them; assume a gold span does not work. - **Both scores are keyed by (user, item)**, so re-saving an answer replaces the earlier result rather than adding one. The denominator is the number of gold items the annotator was served, and an annotator who changes their mind three times still counts once. This is the one file in the pack with a good error message: `Gold standard item skipped, missing gold_label: {…}`, followed by `Gold standards are enabled but NOT RUNNING: no usable items were loaded`. Copy that standard when judging the others: if the log does not say the feature is running, assume it is not. ### Where the results live Neither feature tells the annotator anything by default and neither writes a number into the output. Gold standards are silent unless you ask otherwise: ```yaml gold_standards: feedback: show_correct_answer: true show_explanation: true ``` Both default to false, and both off means the annotator never learns how they scored. That is the right choice when the gold items are also the measurement, and the wrong one when they are meant to teach. I have not driven a study with the block on, so I am claiming only that the key is recognised and read, not what the annotator sees. The scores live on the admin API: ```bash curl -s -o /dev/null localhost:8000/admin # makes the key file exist K=$(cat admin_api_key.txt) curl -H "X-API-Key: $K" localhost:8000/admin/api/quality_control ``` which reports pass/fail counts and per-user and per-item breakdowns for both. Results are written to disk as they are recorded and read back at boot, so they survive a restart along with the block list. Pull that route at handover the way you would pull the export — it is the only place the pass rates are assembled, and it is not in the export. The raw answers are also in `annotation_output//user_state.json` under the check and gold item ids, so the scores can be recomputed by hand against the items files if you ever need to show your working. ## Seeding answers with `pre_annotation` Not quality control in the sense the rest of this file is — it is the same manager, and it is the block a researcher means by "we already have model labels, have people correct them". ```yaml pre_annotation: enabled: true field: predictions # the field on each item holding the labels allow_modification: true show_confidence: true highlight_low_confidence: 0.7 ``` The named field is a dict on the item, keyed by scheme name: ```json {"id": "t01", "body": "The export button spins forever ...", "predictions": {"category": "Bug"}} ``` A field that is not a dict is skipped with a warning naming the item; a scheme name that is not in the config is skipped silently. Seeds are applied only when the user has **no** annotation for that item yet, so a returning annotator sees their own answers rather than the model's. I ran this with plain labels. `show_confidence` and `highlight_low_confidence` need per-prediction confidences in the same field, and I did not supply any, so take those two keys as documented rather than driven. **A seed the annotator never looks at is saved as their answer.** Measured: an item seeded `category: Bug`, with the annotator answering only the other required scheme and pressing Next, stored `[{"schema": "category", "name": "Bug"}, "Bug"]` in their `user_state.json`. The export cannot distinguish it from a label they chose, because nothing records that it came from the model. Two consequences worth stating out loud to the researcher: - Agreement between two annotators over seeded items measures the seeds, not the people. Seed one arm of the study or neither. - If the point is to measure whether people accept model labels, the thing you need — did they change it — is in the behavioural log rather than the annotations, or in `export_include_annotation_changes`. `display_logic` reads seeded answers at load: a scheme gated on `category equals Bug` is visible on first paint for an item seeded that way, and the behavioural log records `display_logic_change ... "reason": "Conditions met"`. ## Adjudication A queue where a named person resolves disagreements into one final label. Use it when you need a single label per item rather than a distribution. ```yaml adjudication: enabled: true adjudicator_users: [alice] agreement_threshold: 0.8 ``` `adjudicator_users` must be a non-empty list of usernames — the validator says so outright. `agreement_threshold` is the other recognized key; `mode`, `strategy`, `queue`, `auto_resolve` and `adjudicators` are all rejected. A named adjudicator works at **`/adjudicate`**, which is not linked from the annotation page: a queue of items that fell below the threshold, each showing its agreement percentage and every annotator's answer per scheme, plus a decision form with confidence, an error taxonomy and a guideline-update flag. Items enter the queue on disagreement rather than on reaching the annotator cap, so an item two of three annotators disagree about is adjudicable before the third arrives. Tell the adjudicator the URL; `/annotate` shows them the completion page instead. Decisions are written to `annotation_output/adjudication/decisions.json` as they are made: ```json {"instance_id": "p01", "adjudicator_id": "adj@x.com", "label_decisions": {"decision": "Include", "quality": 3}, "source": {"decision": "adjudicator"}, "confidence": "medium", "notes": "...", "error_taxonomy": [], "time_spent_ms": 39553} ``` **They are not in `csv`, `jsonl` or `parquet`** — those carry the annotators' answers, so a study run for the resolved labels would export the disagreements and leave the resolution behind. Ask for the `adjudication` format, which writes `adjudicated.csv` (one row per instance and scheme, with `source`, `adjudicator_id`, `confidence` and a timestamp) beside `adjudication_log.jsonl` (the full decision record, notes and error taxonomy included). Request it explicitly: it is not in the default set, and `export_annotation_format` has to name it. Before you design a workflow around adjudication, **open `/adjudicate` on a real item and check that the schemes you care about have a control there.** The panel shows every scheme's annotator responses above a decision form; a composite scheme is decided by adopting one annotator's whole answer rather than by re-entering it, and `label_decisions` records only what had an input. ## `require_fully_annotated` ```yaml require_fully_annotated: true ``` Free, and nearly always right. Combined with `label_requirement: {required: true}` per scheme it stops an annotator advancing past a half-answered item. Know what it does at the boundary: the client checks radios, checkboxes and text boxes and shows `Please answer the required questions: ` inline. A **required span scheme** is not checked client-side — the server refuses the navigation with `400 Required annotation(s) not completed: ` and the annotator gets a small corner toast naming the internal scheme name, not the question. Requiring a span is fine; just make sure every item, attention checks included, is one where drawing a span makes sense. ## `text_as_image` ```yaml text_as_image: true ``` Offer it when a researcher is worried that annotators will paste items into a chatbot. The server renders each item's text to a PNG and blanks the words everywhere the page could leak them. On a classification task in 2.10.0 the item's sentence was absent from the `/annotate` HTML, and `/api/current_instance` returned the item with only its `id`. It makes copying more work, not impossible: an annotator can retype the text or run OCR on a screenshot. Say that when you offer it. It costs accessibility. `potato validate` warns that a screen reader gets nothing from the picture, and a study that has to stay accessible should leave it off. It refuses anything that still shows or selects the text. With a span scheme, `potato validate` and `potato start` both exit 1. `pocket.enabled: true` and `rooms.enabled: true` each fail `potato validate` the same way, with a message naming the conflict: ```text text_as_image is incompatible with these annotation schemes: span. Each one anchors its annotations to character offsets in the instance text, and text_as_image removes that text from the page, so an annotator would have nothing to select. Turn off text_as_image, or use a scheme that does not select text. ``` Two more are refused by `potato validate`. An `audio_annotation` scheme whose `source_field` is not the `text_key` field shows the item text in a box beside the player, which the picture does not replace. A scheme whose own field attribute names the text field reads a field the feature blanks: `text_edit` with `source_field: text` is refused, with a message naming the scheme and the attribute, and the same scheme pointed at an `mt_output` field validates. Potato's `docs/advanced/text_as_image.md` has the full list, including video and tiered schemes. Ask about anti-copying before you choose span annotation, because the two cannot be combined. ## Choosing among them Add in this order, and stop when the risk is covered. Each one costs annotator time. 1. `require_fully_annotated: true`. Free. 2. Agreement. Automatic once items overlap; nothing to configure. 3. `training`, when the label scheme needs explaining. The only one that improves the data. 4. `attention_checks`. Cheap to author, and catches clicking-through. 5. `gold_standards`, when annotators are paid strangers and you need a running accuracy per person. Costs someone writing correct answers. 6. `adjudication`, when you need one final label per item.