Reminder. One line per rule, not the full definitions. Those live on the instructions page.
annotation_schemes: - annotation_type: span name: evidence description: Highlight the part that decides your answer. target_field: body # names the span_target field above labels: [Evidence, Counter-evidence] humanize_labels: false sequential_key_binding: true show_span_labels: true - annotation_type: radio name: category description: Which category is this? labels: [A, B, C, Unclear] # always have a "none" or "unclear" option humanize_labels: false sequential_key_binding: true label_requirement: {required: true} - annotation_type: likert # ordered judgement -> ordinal metric name: severity description: How severe is it? size: 3 min_label: Low max_label: High humanize_labels: false labels: [Low, Medium, High] # label every point, not just the ends sequential_key_binding: true label_requirement: {required: true} - annotation_type: text # follow-up, hidden until it applies name: why_unclear description: What made it unclear? multiline: true # without this a `text` scheme is a one-line rows: 3 min_chars: 10 display_logic: show_when: - {schema: category, operator: equals, value: Unclear} num_annotators_per_item: 3 assignment_strategy: random random_seed: 1 max_annotations_per_user: 12 # every annotator sees all 12 items require_fully_annotated: true training: enabled: true data_file: data/training.json allow_retry: true feedback: {show_explanations: true} attention_checks: enabled: true frequency: 4 items_file: data/attention.json min_response_time: 5 # These four keys are the whole of `failure_handling`. It used to read # `{action: warn}` here, which is not one of them -- so it was dropped and # the thresholds silently fell back to these same defaults. failure_handling: warn_threshold: 2 warn_message: "Please read each item carefully before answering." block_threshold: 5 block_message: "Too many attention checks were missed." agreement_metrics: {enabled: true} phases: order: [consent, instructions, training, annotation, poststudy] consent: {type: consent, title: Consent, file: pages/consent.jsonl} instructions: {type: instructions, title: Instructions, file: pages/instructions.jsonl} training: {type: training, title: Practice} annotation: {type: annotation} poststudy: {type: poststudy, title: Last questions, file: pages/poststudy.jsonl} login: {type: password} user_config: {allow_all_users: true} ``` ## data/items.json Any of JSON array, JSONL, CSV or TSV. Fields beyond `id_key` and `text_key` are available to `instance_display` by name. ```json [ {"id": "c001", "source": "forum", "title": "Thread 1", "body": "Body text of item 1. It has two sentences so a span has somewhere to go."}, {"id": "c002", "source": "forum", "title": "Thread 2", "body": "Body text of item 2. It has two sentences so a span has somewhere to go."}, {"id": "c003", "source": "forum", "title": "Thread 3", "body": "Body text of item 3. It has two sentences so a span has somewhere to go."}, {"id": "c004", "source": "forum", "title": "Thread 4", "body": "Body text of item 4. It has two sentences so a span has somewhere to go."}, {"id": "c005", "source": "forum", "title": "Thread 5", "body": "Body text of item 5. It has two sentences so a span has somewhere to go."}, {"id": "c006", "source": "forum", "title": "Thread 6", "body": "Body text of item 6. It has two sentences so a span has somewhere to go."}, {"id": "c007", "source": "forum", "title": "Thread 7", "body": "Body text of item 7. It has two sentences so a span has somewhere to go."}, {"id": "c008", "source": "forum", "title": "Thread 8", "body": "Body text of item 8. It has two sentences so a span has somewhere to go."}, {"id": "c009", "source": "forum", "title": "Thread 9", "body": "Body text of item 9. It has two sentences so a span has somewhere to go."}, {"id": "c010", "source": "forum", "title": "Thread 10", "body": "Body text of item 10. It has two sentences so a span has somewhere to go."}, {"id": "c011", "source": "forum", "title": "Thread 11", "body": "Body text of item 11. It has two sentences so a span has somewhere to go."}, {"id": "c012", "source": "forum", "title": "Thread 12", "body": "Body text of item 12. It has two sentences so a span has somewhere to go."} ] ``` ## data/training.json An **object**, not an array. Each instance needs `id`, `text`, `correct_answers`. ```json { "training_instances": [ {"id": "t1", "text": "Worked example one. The model answer is A.", "correct_answers": {"category": "A", "severity": "Low"}, "explanation": "Why A is right, in a sentence the annotator can act on."}, {"id": "t2", "text": "Worked example two. Nothing here decides the category, so it is Unclear.", "correct_answers": {"category": "Unclear", "severity": "Low"}, "explanation": "Unclear is a real answer, not a way of skipping the item."} ] } ``` `correct_answers` keys are scheme names; values are the label strings exactly as written in `labels:`. Span schemes cannot be graded here. The training page renders `text` only, so fold any context the annotator needs into that string. ## data/attention.json An array. Each item needs `id` and `expected_answer`, plus whatever fields your `instance_display` renders. ```json [ {"id": "chk1", "title": "Attention check", "body": "This is a check, not a real item. Highlight this sentence and mark it Evidence, select A for category, and Low for severity.", "text": "This is a check, not a real item. Highlight this sentence and mark it Evidence, select A for category, and Low for severity.", "expected_answer": {"category": "A", "severity": "Low"}} ] ``` Instruct **every required scheme**, span schemes included. A check item that leaves a required span unanswered dead-ends the annotator on a 400 with only a corner toast for feedback. ## pages/consent.jsonl One annotation scheme per line, same syntax as `annotation_schemes`. Prose goes in a `pure_display` scheme with `allow_html: true`. ```jsonl {"annotation_type":"pure_display","name":"consent_text","allow_html":true,"description":"Draft. Replace with the approved wording.
"} {"annotation_type":"radio","name":"consent_agree","description":"Do you agree to take part?","labels":["I agree","I do not agree"],"humanize_labels":false,"label_requirement":{"required":true}} ``` ## pages/instructions.jsonl The full definitions live here, not in `annotation_instructions` — that banner sits on top of every item and pushes the questions below the fold. ```jsonl {"annotation_type":"pure_display","name":"instructions_text","allow_html":true,"description":"Read the item, highlight the part that decides your answer, then pick a category and a severity.
A. Replace with the real definition.
B. Replace with the real definition.
C. Replace with the real definition.
Unclear. Use it when the item genuinely does not decide. It is an answer, not a skip.
"} ``` ## pages/poststudy.jsonl A phase page can ask real questions, not just display prose. These land in `annotation_output/Two questions, then you are done.
"} {"annotation_type":"likert","name":"task_clarity","description":"How clear were the instructions?","size":5,"min_label":"Very unclear","max_label":"Very clear","label_requirement":{"required":true}} {"annotation_type":"text","name":"comments","description":"Anything that was hard to judge?","multiline":true,"rows":3} ``` ## static/study.css Phase-page prose lands in a `