Attention Checks and Gold Standards¶

The quality-control ladder, and what each rung costs. It ships with the potato-tasks skill:

/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-tasks@potato

Four blocks, three of which need a side file whose format is documented nowhere and whose error messages mostly do not name the missing field. All formats below were recovered by booting the server against candidate files and reading the log.

The failure mode to design against: every one of these can be enabled, validate clean under --strict, log a warning nobody reads, and run the entire study with the feature off. Two of them do not even log an error. After any change here, boot the server and check the count.

The second failure mode is subtler and worse: the feature runs, scores everyone, and reports a number that is not what you think it is. Read "Where the results live" below before you act on an accuracy figure.

grep -E "Loaded [0-9]+ (training instances|attention check items|gold standard items)" server.log

Feature	Healthy line	If it says 0
training	`Loaded 3 training instances`	Boot fails with an exception. Loud.
attention checks	`Loaded 2 attention check items`	`WARNING … missing required fields` then silence. Study runs unchecked.
gold standards	`Loaded 2 gold standard items`	`ERROR: Gold standards are enabled but NOT RUNNING`. Names the missing field.

Training¶

A qualification round of worked examples, marked as the annotator answers, before they touch real items. It is the only quality-control feature that improves the annotations rather than only measuring them.

training:
  enabled: true
  data_file: data/training.json
  allow_retry: true
  feedback:
    enabled: true
  passing_criteria:
    min_accuracy: 0.6

Recognized sub-keys: enabled, data_file, allow_retry, feedback, passing_criteria. That is all of them. min_accuracy, max_attempts, num_items, show_feedback, threshold and every other plausible name is rejected by --strict, which is how you find the real ones. feedback and passing_criteria must be dictionaries; a bare true or 0.6 is a fatal error.

Also add the phase, or the block does nothing:

phases:
  order: [consent, training, annotation]
  consent:
    type: consent
    title: Consent to take part
    file: surveyflow/consent.jsonl
  training:
    type: training
    title: Practice
  annotation:
    type: annotation

Define every phase you name in order. Naming one you do not define is not a no-op — if the annotator reaches it before annotation, every page of the study becomes a 500 behind a clean --strict and a boot log that says it skipped the phase. See phases-and-pages.md.

The training data file¶

An object, not an array, with a training_instances key:

{
  "training_instances": [
    {
      "id": "train01",
      "text": "The energy regulator fined the utility £2 million on Monday …",
      "correct_answers": {
        "tone": "Neutral",
        "responsibility": "A company or developer",
        "misleading": "No"
      },
      "explanation": "Wire copy. Every sentence is attributed reporting, so …"
    }
  ]
}

Required per instance: id, text, correct_answers. Nothing else. Keys in correct_answers are scheme names; values are the label strings exactly as they appear in labels: (the stored value, not the humanized display form).

Three ways this goes wrong, with the exact message:

Message	Cause
`Training data must be a JSON object`	You wrote a JSON array
`Training data must contain 'training_instances' field`	Right shape, wrong key
`Training instance missing required fields: {…}`	Dumps the whole instance and never names the field. It is one of `id`, `text`, `correct_answers`.

Notes:

explanation is shown with the mark. Write it — a wrong answer with no explanation teaches nothing, and this is the one place in the task where you can correct a misreading before it contaminates 500 items.
The training page renders the question text only. Any context an annotator needs (a headline, a speaker, an outlet) must be folded into that string; the display fields from instance_display are not used here. From Potato 2.8.2 a training instance keeps all its fields and the question comes from the project's text_key, so {"text": ...} and {"caption": ...} both work where text_key: caption. Earlier builds kept six keys and dropped the rest.
A practice round still cannot show media. With no instance_display there is no <img> for an image_annotation or other geometry scheme to take its bitmap from, and source_field does not save it: the instance fields the client would read are not serialized onto a phase page. The trainee gets an empty canvas, the console says [ImageAnnotation] No image URL found! Check that the instance data contains an image URL, and the config validates clean. check_ui.py flags it as "a geometry scheme on a page with no image on it". Potato ships no example combining an image display with a training phase, which is why nothing catches it. Either drop training on a media task, or use it only for the classification schemes and say in the explanation that the drawing tool is practised on the first real item. (Before 2.8.2 this failed more loudly and more confusingly: the canvas read "Failed to load image. Check the URL or CORS settings." and the console showed a 404 for a URL made out of the instance's own prose.)
Grading is per scheme against correct_answers. Schemes you omit are not graded. Span schemes are not graded. There is no way to express a gold span here.
Progress shows as Question 1 / 3 with a running Correct: n · Mistakes: n, and a correct answer advances with Correct! Moving to next question.
State lands in annotation_output/<user>/user_state.json under training_state: per-question correct/attempts, plus passed.

Attention checks¶

Items with an obvious right answer, mixed into the stream, to catch someone who stopped reading halfway through.

attention_checks:
  enabled: true
  frequency: 3            # every third item; use probability OR frequency, never both
  items_file: data/attention.json
  min_response_time: 8    # accepted, and currently does nothing — see below
  failure_handling:
    action: warn

failure_handling must be a dictionary despite the key reference typing it string|object. Setting both frequency and probability is a fatal error.

min_response_time is inert. The check needs a client timestamp on the save request and annotation.js does not send one, so the branch never runs — driven with min_response_time: 8 and answers submitted in under two seconds, the log has no fast-response line at all. Even reached, it only logs: pass and fail are decided by content alone. Do not count on it to catch clicking-through; that is what /admin/api/suspicious_activity and the behavioral analytics are for.

The attention item file¶

A plain JSON array. Required per item: id and expected_answer.

[
  {
    "id": "check01",
    "text": "This item is a check that you are still reading carefully. It is not a real article. Highlight this sentence and mark it Factual reporting. For the tone question, select Neutral …",
    "expected_answer": {"tone": "Neutral", "responsibility": "No one", "misleading": "No"}
  }
]

Give the items whatever display fields your instance_display reads (headline, body, …) as well. They render as a normal item and look broken without them.

Two traps worth designing around:

Instruct every required scheme. Span schemes included. A check item that says "select Neutral" but says nothing about the highlight leaves the annotator stuck: Next returns 400 Required annotation(s) not completed: <scheme> and the only feedback is a small toast naming the internal scheme name.
Attention checks consume item slots. They are inserted into the annotator's ordering, so with a per-annotator cap equal to the item count they displace real items — annotators finish having never seen the last two. See assignment-and-agreement.md.

Gold standards¶

Items with known labels used to score annotators continuously, rather than once in training. Use them when annotators are paid strangers.

gold_standards:
  enabled: true
  items_file: data/gold.json
  mode: mixed             # training | mixed | separate
  frequency: 5
  accuracy:
    min_threshold: 0.7
    evaluation_count: 10
  auto_promote:
    enabled: false

mode must be one of training, mixed, separate; the error names them.

accuracy and auto_promote must both be dictionaries, even though the key documentation types them number|object and boolean|object. accuracy: 0.7 and auto_promote: false both fail with must be a dictionary. Their documented sub-keys are min_threshold / evaluation_count and min_annotators / agreement_threshold — and nothing checks those, because validation stops one level in. A wrong sub-key here validates clean and does nothing.

How many gold items an annotator sees¶

frequency sets the spacing, but the file governs the count: every annotator is served every item in items_file, once. Two items in the file with frequency: 4 over eight real items gives two insertions per annotator, not 8 / 4. Size the file to how many checks you want each person to get. They are injected on top of the per-annotator quota rather than inside it, so there is nothing to pad (see assignment-and-agreement.md).

The gold item file¶

A JSON array. Required per item: gold_label (plus id and the display fields). Both shapes load:

[
  {"id": "g1", "text": "…", "gold_label": {"scheme_name": "label"}},
  {"id": "g2", "text": "…", "gold_label": "label"}
]

Use the dict form on any task with more than one scheme.

Two things about gold scoring worth telling a researcher before they read the number:

A drawn answer can be a gold answer. Boxes, polygons, masks and points are graded by overlap at 0.5 IoU, not by equality, so "did they find the thing I planted" is a question gold standards can now answer. Write the shape in the form the client stores — normalized fractional coordinates nested under coordinates, not pixels:

{"id": "gold_dog", "image_url": "…",
 "gold_label": {"region": [{"type": "bbox", "label": "subject",
   "coordinates": {"x": 0.18, "y": 0.20, "width": 0.43, "height": 0.40}}]}}

On an image_annotation task with that item, an annotator drawing the same box scored 1.0 and one drawing a box in the opposite corner scored 0.0. Every gold shape must be matched by a user shape of the same label, with no extras — boxing the whole image does not count as finding the object. Spans are stored elsewhere and I have not tested them; assume a gold span does not work. - The accuracy denominator counts saves, not gold items. Every save against a gold item appends another result, so the number measures how much the annotator fiddled. One gold item, four annotators: three quick clicks scored total: 1, a box drag scored 2, four changes of mind scored 7. Someone who revises a correct answer twice reads as 1/3. Attention checks do not have this problem — they are keyed by (user, item) and a re-save replaces the earlier result — so read the attention numbers at face value and treat the gold fraction as a floor until this is fixed.

This is the one file in the pack with a good error message: Gold standard item skipped, missing gold_label: {…}, followed by Gold standards are enabled but NOT RUNNING: no usable items were loaded. Copy that standard when judging the others: if the log does not say the feature is running, assume it is not.

Where the results live¶

Neither feature tells the annotator anything by default and neither writes a number into the output. Gold standards are deliberately silent. The loader reads gold_standards.feedback.show_correct_answer and .show_explanation (quality_control.py:186), but the validator rejects gold_standards.feedback as an unrecognized key, so that switch cannot pass --strict — I have not run a study with it on. Plan on the annotator never learning how they scored. The scores live on the admin API:

curl -s -o /dev/null localhost:8000/admin          # makes the key file exist
K=$(cat admin_api_key.txt)
curl -H "X-API-Key: $K" localhost:8000/admin/api/quality_control

which reports pass/fail counts and per-user and per-item breakdowns for both.

Those results are held in memory and a restart erases them. One annotator who had failed an attention check and missed a gold item read pass_rate: 0.5 and accuracy: 0.5 before a restart, and after it — with all sixteen of her annotations still on disk, gold and check items included — the same route reported zero checks, zero evaluations and an empty by_user, with enabled: true. That is indistinguishable from a study where nobody has failed anything. Blocked annotators are unblocked by the same restart.

So on any study that will outlive a single server process: pull /admin/api/quality_control and keep it before you restart anything, and treat a zero on that route as "no record" rather than "no failures" unless you know the server has been up the whole time. The raw answers are always in annotation_output/<user>/user_state.json under the check and gold item ids, so the scores can be recomputed by hand against the items files.

Adjudication¶

A queue where a named person resolves disagreements into one final label. Use it when you need a single label per item rather than a distribution.

adjudication:
  enabled: true
  adjudicator_users: [alice]
  agreement_threshold: 0.8

adjudicator_users must be a non-empty list of usernames — the validator says so outright. agreement_threshold is the other recognized key; mode, strategy, queue, auto_resolve and adjudicators are all rejected.

`require_fully_annotated`¶

require_fully_annotated: true

Free, and nearly always right. Combined with label_requirement: {required: true} per scheme it stops an annotator advancing past a half-answered item.

Know what it does at the boundary: the client checks radios, checkboxes and text boxes and shows Please answer the required questions: <question text> inline. A required span scheme is not checked client-side — the server refuses the navigation with 400 Required annotation(s) not completed: <scheme name> and the annotator gets a small corner toast naming the internal scheme name, not the question. Requiring a span is fine; just make sure every item, attention checks included, is one where drawing a span makes sense.

Choosing among them¶

Add in this order, and stop when the risk is covered. Each one costs annotator time.

require_fully_annotated: true. Free.
Agreement. Automatic once items overlap; nothing to configure.
training, when the label scheme needs explaining. The only one that improves the data.
attention_checks. Cheap to author, and catches clicking-through.
gold_standards, when annotators are paid strangers and you need a running accuracy per person. Costs someone writing correct answers.
adjudication, when you need one final label per item.