Attention Checks and Gold Standards¶

The quality-control ladder, and what each rung costs. It ships with the potato-skill skill:

/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato

Four blocks, three of which need a side file whose format is documented nowhere and whose error messages mostly do not name the missing field. All formats below were recovered by booting the server against candidate files and reading the log.

The failure mode to design against: every one of these can be enabled, validate clean under --strict, log a warning nobody reads, and run the entire study with the feature off. Two of them do not even log an error. After any change here, boot the server and check the count.

The second failure mode is subtler and worse: the feature runs, scores everyone, and reports a number that is not what you think it is. Read "Where the results live" below before you act on an accuracy figure.

grep -E "Loaded [0-9]+ (training instances|attention check items|gold standard items)" server.log

Feature	Healthy line	If it says 0
training	`Loaded 3 training instances`	Boot fails with an exception. Loud.
attention checks	`Loaded 2 attention check items`	`WARNING … missing required fields` then silence. Study runs unchecked.
gold standards	`Loaded 2 gold standard items`	`ERROR: Gold standards are enabled but NOT RUNNING`. Names the missing field.

Training¶

A qualification round of worked examples, marked as the annotator answers, before they touch real items. It is the only quality-control feature that improves the annotations rather than only measuring them.

training:
  enabled: true
  data_file: data/training.json
  allow_retry: true
  feedback:
    show_explanations: true
  passing_criteria:
    min_correct: 3

Recognized sub-keys: enabled, data_file, allow_retry, feedback, passing_criteria, failure_action, annotation_schemes. That is all seven. At this level min_accuracy, max_attempts, num_items, show_feedback, threshold and every other plausible name is rejected by --strict, which is how you find the real ones.

Inside the two nested blocks an unknown key is a warning, not an error. passing_criteria takes min_correct, require_all_correct, max_mistakes, max_mistakes_per_question and max_attempts; feedback takes show_explanations and allow_retry. Anything else passes validation and logs Unknown key training.feedback.enabled is ignored (known: allow_retry, show_explanations), so min_accuracy: 0.6 under passing_criteria validates and does nothing. The pass mark is min_correct, a count of correct answers rather than a share; a value that is not a positive integer is a fatal error.

failure_action is move_to_done (default) or repeat_training, and any other value is a fatal error. annotation_schemes lets the practice questions use different schemes from the real task; omit it and training uses the task's own.

feedback and passing_criteria must be dictionaries; a bare true or 0.6 is a fatal error.

Also add the phase, or the block does nothing:

phases:
  order: [consent, training, annotation]
  consent:
    type: consent
    title: Consent to take part
    file: surveyflow/consent.jsonl
  training:
    type: training
    title: Practice
  annotation:
    type: annotation

Define every phase you name in order. Naming one you do not define is a --strict failure, and the phase is dropped from the order rather than silently half-present. See phases-and-pages.md.

The training data file¶

An object, not an array, with a training_instances key:

{
  "training_instances": [
    {
      "id": "train01",
      "text": "The energy regulator fined the utility £2 million on Monday …",
      "correct_answers": {
        "tone": "Neutral",
        "responsibility": "A company or developer",
        "misleading": "No"
      },
      "explanation": "Wire copy. Every sentence is attributed reporting, so …"
    }
  ]
}

Required per instance: id, text, correct_answers. Nothing else. Keys in correct_answers are scheme names; values are the label strings exactly as they appear in labels: (the stored value, not the humanized display form).

Three ways this goes wrong, with the exact message:

Message	Cause
`Training data must be a JSON object`	You wrote a JSON array
`Training data must contain 'training_instances' field`	Right shape, wrong key
`Training instance missing required fields: {…}`	Dumps the whole instance and never names the field. It is one of `id`, `text`, `correct_answers`.

Notes:

explanation is shown with the mark. Write it — a wrong answer with no explanation teaches nothing, and this is the one place in the task where you can correct a misreading before it contaminates 500 items.
The training page renders the question text only. Any context an annotator needs (a headline, a speaker, an outlet) must be folded into that string; the display fields from instance_display are not used here. From Potato 2.8.2 a training instance keeps all its fields and the question comes from the project's text_key, so {"text": ...} and {"caption": ...} both work where text_key: caption. Earlier builds kept six keys and dropped the rest.
A practice round still cannot show media. With no instance_display there is no <img> for an image_annotation or other geometry scheme to take its bitmap from, and source_field does not save it: the instance fields the client would read are not serialized onto a phase page. The trainee gets an empty canvas, the console says [ImageAnnotation] No image URL found! Check that the instance data contains an image URL, and the config validates clean. check_ui.py flags it as "a geometry scheme on a page with no image on it". Potato ships no example combining an image display with a training phase, which is why nothing catches it. Either drop training on a media task, or use it only for the classification schemes and say in the explanation that the drawing tool is practised on the first real item. (Before 2.8.2 this failed more loudly and more confusingly: the canvas read "Failed to load image. Check the URL or CORS settings." and the console showed a 404 for a URL made out of the instance's own prose.)
Grading is per scheme against correct_answers. Schemes you omit are not graded. Span schemes are not graded. There is no way to express a gold span here.
Progress shows as Question 1 / 3 with a running Correct: n · Mistakes: n, and a correct answer advances with Correct! Moving to next question.
State lands in annotation_output/<user>/user_state.json under training_state: per-question correct/attempts, plus passed.

Attention checks¶

Items with an obvious right answer, mixed into the stream, to catch someone who stopped reading halfway through.

attention_checks:
  enabled: true
  frequency: 3            # every third item; use probability OR frequency, never both
  items_file: data/attention.json
  min_response_time: 8    # a faster answer fails the check
  failure_handling:
    warn_threshold: 2       # warn from the second failure
    warn_message: "Please read each item before answering."
    block_threshold: 5      # block at the fifth

failure_handling reads four keys and no others: warn_threshold, warn_message, block_threshold, block_message. There is no action key. A sub-key it does not know is a warning rather than an error, so a config naming one validates without --strict and does nothing. The thresholds then fall back to 2 and 5 (quality_control.py:375).

Setting both frequency and probability is a fatal error.

min_response_time fails a check answered faster than the threshold, whatever the content was. Set it well under a plausible reading time — the point is to catch a straight-through click, not to punish someone who recognised the item — and remember it is one signal among several: /admin/api/suspicious_activity and the behavioral analytics see patterns a per-item threshold cannot.

The attention item file¶

A plain JSON array. Required per item: id and expected_answer.

[
  {
    "id": "check01",
    "text": "This item is a check that you are still reading carefully. It is not a real article. Highlight this sentence and mark it Factual reporting. For the tone question, select Neutral …",
    "expected_answer": {"tone": "Neutral", "responsibility": "No one", "misleading": "No"}
  }
]

Give the items whatever display fields your instance_display reads (headline, body, …) as well. They render as a normal item and look broken without them.

Two traps worth designing around:

Instruct every required scheme. Span schemes included. A check item that says "select Neutral" but says nothing about the highlight leaves the annotator stuck: Next returns 400 Required annotation(s) not completed: <scheme> and the only feedback is a small toast naming the internal scheme name.
Attention checks consume item slots. They are inserted into the annotator's ordering, so with a per-annotator cap equal to the item count they displace real items — annotators finish having never seen the last two. See assignment-and-agreement.md.

Gold standards¶

Items with known labels used to score annotators continuously, rather than once in training. Use them when annotators are paid strangers.

gold_standards:
  enabled: true
  items_file: data/gold.json
  mode: mixed             # training | mixed | separate
  frequency: 5
  accuracy:
    min_threshold: 0.7
    evaluation_count: 10
  auto_promote:
    enabled: false

mode takes three values and mixed is the one that is implemented. separate behaves identically to it and says so at boot — gold_standards.mode 'separate' behaves as 'mixed': 3 gold items are injected into the annotation stream every 2 items — and training serves nothing at all, because no code path delivers gold during the training phase. That last one fails --strict and warns twice at boot, naming the count that will never be served. Grade a training phase with its own practice questions instead.

accuracy and auto_promote must both be dictionaries, even though the key documentation types them number|object and boolean|object. accuracy: 0.7 and auto_promote: false both fail with must be a dictionary. Their documented sub-keys are min_threshold / evaluation_count and min_annotators / agreement_threshold — and nothing checks those, because validation stops one level in. A wrong sub-key here validates clean and does nothing.

The accuracy threshold reports to you, not to the annotator. Crossing it logs <user> is below the gold standard accuracy threshold: 0.50 over 2 evaluated items, threshold 0.90 and puts the annotator in gold_standards.below_accuracy_threshold on the admin route. It does not block them and there is no setting that does — attention checks block, gold standards never do. If you need underperformers stopped rather than counted, that is attention_checks.failure_handling or your own procedure.

How many gold items an annotator sees¶

frequency sets the spacing, but the file governs the count: every annotator is served every item in items_file, once. Two items in the file with frequency: 4 over eight real items gives two insertions per annotator, not 8 / 4. Size the file to how many checks you want each person to get. They are injected on top of the per-annotator quota rather than inside it, so there is nothing to pad (see assignment-and-agreement.md).

The gold item file¶

A JSON array. Required per item: gold_label (plus id and the display fields). Both shapes load:

[
  {"id": "g1", "text": "…", "gold_label": {"scheme_name": "label"}},
  {"id": "g2", "text": "…", "gold_label": "label"}
]

Use the dict form on any task with more than one scheme.

Two things about gold scoring worth telling a researcher before they read the number:

A drawn answer can be a gold answer. Boxes, polygons, masks and points are graded by overlap at 0.5 IoU, not by equality, so "did they find the thing I planted" is a question gold standards can now answer. Write the shape in the form the client stores — normalized fractional coordinates nested under coordinates, not pixels:

{"id": "gold_dog", "image_url": "…",
 "gold_label": {"region": [{"type": "bbox", "label": "subject",
   "coordinates": {"x": 0.18, "y": 0.20, "width": 0.43, "height": 0.40}}]}}

On an image_annotation task with that item, an annotator drawing the same box scored 1.0 and one drawing a box in the opposite corner scored 0.0. Every gold shape must be matched by a user shape of the same label, with no extras — boxing the whole image does not count as finding the object. Spans are stored elsewhere and I have not tested them; assume a gold span does not work. - Both scores are keyed by (user, item), so re-saving an answer replaces the earlier result rather than adding one. The denominator is the number of gold items the annotator was served, and an annotator who changes their mind three times still counts once.

This is the one file in the pack with a good error message: Gold standard item skipped, missing gold_label: {…}, followed by Gold standards are enabled but NOT RUNNING: no usable items were loaded. Copy that standard when judging the others: if the log does not say the feature is running, assume it is not.

Where the results live¶

Neither feature tells the annotator anything by default and neither writes a number into the output. Gold standards are silent unless you ask otherwise:

gold_standards:
  feedback:
    show_correct_answer: true
    show_explanation: true

Both default to false, and both off means the annotator never learns how they scored. That is the right choice when the gold items are also the measurement, and the wrong one when they are meant to teach. I have not driven a study with the block on, so I am claiming only that the key is recognised and read, not what the annotator sees. The scores live on the admin API:

curl -s -o /dev/null localhost:8000/admin          # makes the key file exist
K=$(cat admin_api_key.txt)
curl -H "X-API-Key: $K" localhost:8000/admin/api/quality_control

which reports pass/fail counts and per-user and per-item breakdowns for both.

Results are written to disk as they are recorded and read back at boot, so they survive a restart along with the block list. Pull that route at handover the way you would pull the export — it is the only place the pass rates are assembled, and it is not in the export. The raw answers are also in annotation_output/<user>/user_state.json under the check and gold item ids, so the scores can be recomputed by hand against the items files if you ever need to show your working.

Seeding answers with `pre_annotation`¶

Not quality control in the sense the rest of this file is — it is the same manager, and it is the block a researcher means by "we already have model labels, have people correct them".

pre_annotation:
  enabled: true
  field: predictions          # the field on each item holding the labels
  allow_modification: true
  show_confidence: true
  highlight_low_confidence: 0.7

The named field is a dict on the item, keyed by scheme name:

{"id": "t01", "body": "The export button spins forever ...",
 "predictions": {"category": "Bug"}}

A field that is not a dict is skipped with a warning naming the item; a scheme name that is not in the config is skipped silently. Seeds are applied only when the user has no annotation for that item yet, so a returning annotator sees their own answers rather than the model's.

I ran this with plain labels. show_confidence and highlight_low_confidence need per-prediction confidences in the same field, and I did not supply any, so take those two keys as documented rather than driven.

A seed the annotator never looks at is saved as their answer. Measured: an item seeded category: Bug, with the annotator answering only the other required scheme and pressing Next, stored [{"schema": "category", "name": "Bug"}, "Bug"] in their user_state.json. The export cannot distinguish it from a label they chose, because nothing records that it came from the model. Two consequences worth stating out loud to the researcher:

Agreement between two annotators over seeded items measures the seeds, not the people. Seed one arm of the study or neither.
If the point is to measure whether people accept model labels, the thing you need — did they change it — is in the behavioural log rather than the annotations, or in export_include_annotation_changes.

display_logic reads seeded answers at load: a scheme gated on category equals Bug is visible on first paint for an item seeded that way, and the behavioural log records display_logic_change ... "reason": "Conditions met".

Adjudication¶

A queue where a named person resolves disagreements into one final label. Use it when you need a single label per item rather than a distribution.

adjudication:
  enabled: true
  adjudicator_users: [alice]
  agreement_threshold: 0.8

adjudicator_users must be a non-empty list of usernames — the validator says so outright. agreement_threshold is the other recognized key; mode, strategy, queue, auto_resolve and adjudicators are all rejected.

A named adjudicator works at /adjudicate, which is not linked from the annotation page: a queue of items that fell below the threshold, each showing its agreement percentage and every annotator's answer per scheme, plus a decision form with confidence, an error taxonomy and a guideline-update flag. Items enter the queue on disagreement rather than on reaching the annotator cap, so an item two of three annotators disagree about is adjudicable before the third arrives. Tell the adjudicator the URL; /annotate shows them the completion page instead.

Decisions are written to annotation_output/adjudication/decisions.json as they are made:

{"instance_id": "p01", "adjudicator_id": "adj@x.com",
 "label_decisions": {"decision": "Include", "quality": 3},
 "source": {"decision": "adjudicator"}, "confidence": "medium",
 "notes": "...", "error_taxonomy": [], "time_spent_ms": 39553}

They are not in csv, jsonl or parquet — those carry the annotators' answers, so a study run for the resolved labels would export the disagreements and leave the resolution behind. Ask for the adjudication format, which writes adjudicated.csv (one row per instance and scheme, with source, adjudicator_id, confidence and a timestamp) beside adjudication_log.jsonl (the full decision record, notes and error taxonomy included). Request it explicitly: it is not in the default set, and export_annotation_format has to name it.

Before you design a workflow around adjudication, open /adjudicate on a real item and check that the schemes you care about have a control there. The panel shows every scheme's annotator responses above a decision form; a composite scheme is decided by adopting one annotator's whole answer rather than by re-entering it, and label_decisions records only what had an input.

`require_fully_annotated`¶

require_fully_annotated: true

Free, and nearly always right. Combined with label_requirement: {required: true} per scheme it stops an annotator advancing past a half-answered item.

Know what it does at the boundary: the client checks radios, checkboxes and text boxes and shows Please answer the required questions: <question text> inline. A required span scheme is not checked client-side — the server refuses the navigation with 400 Required annotation(s) not completed: <scheme name> and the annotator gets a small corner toast naming the internal scheme name, not the question. Requiring a span is fine; just make sure every item, attention checks included, is one where drawing a span makes sense.

`text_as_image`¶

text_as_image: true

Offer it when a researcher is worried that annotators will paste items into a chatbot. The server renders each item's text to a PNG and blanks the words everywhere the page could leak them. On a classification task in 2.10.0 the item's sentence was absent from the /annotate HTML, and /api/current_instance returned the item with only its id. It makes copying more work, not impossible: an annotator can retype the text or run OCR on a screenshot. Say that when you offer it.

It costs accessibility. potato validate warns that a screen reader gets nothing from the picture, and a study that has to stay accessible should leave it off.

It refuses anything that still shows or selects the text. With a span scheme, potato validate and potato start both exit 1. pocket.enabled: true and rooms.enabled: true each fail potato validate the same way, with a message naming the conflict:

text_as_image is incompatible with these annotation schemes: span. Each one
anchors its annotations to character offsets in the instance text, and
text_as_image removes that text from the page, so an annotator would have
nothing to select. Turn off text_as_image, or use a scheme that does not select
text.

Two more are refused by potato validate. An audio_annotation scheme whose source_field is not the text_key field shows the item text in a box beside the player, which the picture does not replace. A scheme whose own field attribute names the text field reads a field the feature blanks: text_edit with source_field: text is refused, with a message naming the scheme and the attribute, and the same scheme pointed at an mt_output field validates. Potato's docs/advanced/text_as_image.md has the full list, including video and tiered schemes. Ask about anti-copying before you choose span annotation, because the two cannot be combined.

Choosing among them¶

Add in this order, and stop when the risk is covered. Each one costs annotator time.

require_fully_annotated: true. Free.
Agreement. Automatic once items overlap; nothing to configure.
training, when the label scheme needs explaining. The only one that improves the data.
attention_checks. Cheap to author, and catches clicking-through.
gold_standards, when annotators are paid strangers and you need a running accuracy per person. Costs someone writing correct answers.
adjudication, when you need one final label per item.