Phases and Pages¶

Consent, instructions, training, surveys, and the order they run in. It ships with the potato-skill skill:

/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato

Everything a researcher describes as "they should see X first" and "at the end I want to ask them Y" lives here. None of it is in the generated key reference: phases and surveyflow are documented as object with no sub-keys, and potato validate --strict does not check inside either of them. Every fact below was established by running the server and reading the startup log.

Which of these you actually need¶

Start from none of them. 7 of Potato's 218 bundled examples declare a phases block at all; the other 207 open on the first item. A phase is a page an annotator has to get through before doing any work, and every one you add costs attrition and setup time, so add them because someone asked or because the protocol requires it — not because a study "should" have them.

Phase	Add it when	Skip it when
`consent`	Real participants, real data, and an IRB or ethics protocol that requires recorded consent — or you are paying strangers on a crowd platform	Prototyping, pilots with colleagues, your own data, any run whose output is a screenshot. A consent form on a prototype is theatre, and it is the wrong wording anyway
`instructions`	The rules do not fit in `annotation_instructions`, or annotators need worked examples before the first item	The task is one question with obvious labels. `annotation_instructions` renders on every page and is read more than a page nobody returns to
`training`	Labels need calibrating, the budget is large enough that a bad annotator is expensive, or you want a pass/fail gate	Fewer than a few hundred items, or the annotators are the researchers. Training needs a graded item file per scheme, which is real work
`prestudy`	You are screening — language, expertise, a qualification question that decides who continues	You already know who is annotating
`poststudy`	Someone named a question they want answered at the end	Always, otherwise. Do not add a post-study questionnaire nobody asked for. It is the phase most often added on reflex, it is where the identifiable answers accumulate, and it makes the export two files instead of one (`getting-the-data-out.md`)

Two things follow that are easy to get backwards.

Consent is a protocol decision, not a config decision. If the study needs consent, the wording comes from the approved protocol and you should be asking for it rather than drafting it — asking-the-experimenter.md treats it as one of the questions worth interrupting for. A placeholder consent page that ships is worse than no consent page, because it looks like the study has ethics coverage.

A phase you add for the pilot tends to stay. Adding poststudy later is a config edit and a page file; removing one after annotators have answered it means you now hold survey responses you did not need. Default to adding phases late.

The six phase types¶

consent | instructions | training | annotation | poststudy | prestudy

That is the complete list. Anything else fails at boot with Failed to load phase 'x': Unknown phase: x — and potato validate --strict passes it. A config naming fourteen invented phase types validates clean and then loses fourteen phases at startup.

There is no survey phase, no demographics, no debrief, no end, no done. A post-study questionnaire is a poststudy phase whose page happens to contain survey questions. Screening before the task is prestudy.

order is honoured as written, including where you put prestudy. Walked in a browser, this order

phases:
  order: [consent, prestudy, instructions, training, annotation, poststudy]

serves consent, then the screening page, then the instructions page, then the training questions, in exactly that sequence. A phase with required questions gates the next one — the annotator cannot advance past it unanswered — which is what "only people who answer the screening questions get through" means in practice. Phase pages record; they do not route. There is no mechanism for screening someone out on the basis of what they answered, and no documented way to feed a survey answer into assignment. The nearest thing is category_assignment, which routes by item category and can gate on a qualification earned in training, not on a survey response. If a researcher says "only screen-reader users get through", tell them the phase collects the answer and someone has to act on it — do not imply the phase enforces it.

Note also that the top-level prestudy: key is dead code — the key documentation says so outright ("the only code that reads it is an unreferenced method"). Screening is a prestudy phase, not the prestudy block.

Two ways to write it¶

The names in order are yours, not the type names, so a type can appear more than once. Two instructions phases called intro_a and intro_b serve in that order with a Previous button between them, which is the way to split long instructions across pages rather than one wall of text.

The exception is annotation: a second one is never served. The annotation flow owns the item queue and ends the study when it is empty, so a study declaring first_pass and second_pass runs the first, reaches the last item and goes straight to the completion page — I drove three items and watched progress go 0/3, 1/3, 2/3, done, with the second phase's title never appearing. Since 2.8.2 validate warns and names both phases, and --strict refuses; the warning fires on the bare name annotation in order as well as on a named block, since ["annotation", "second_pass"] is the same mistake spelled differently. For a second pass over the same items, look at adjudication or a second annotator instead.

Mapping form — an order list plus one entry per phase, keyed by name:

phases:
  order: [consent, instructions, training, annotation, poststudy]
  consent:
    type: consent
    title: Consent to take part
    file: surveyflow/consent.jsonl
  instructions:
    type: instructions
    title: About this study
    file: surveyflow/instructions.jsonl
  training:
    type: training
    title: Practice
  annotation:
    type: annotation
  poststudy:
    type: poststudy
    title: Your own news habits
    file: surveyflow/news_habits.jsonl

List form — the same thing with name inside each entry, no order:

phases:
  - {name: consent, type: consent, title: Consent, file: surveyflow/consent.jsonl}
  - {name: annotation, type: annotation}
  - {name: poststudy, type: poststudy, title: Wrap up, file: surveyflow/end.jsonl}

Both work. Mapping form reads better once any phase has more than three keys.

Naming a phase in order without defining it fails --strict, and is worth letting fail rather than downgrading:

OTHER WARNINGS:
  - Phase 'consent' is listed in phases.order but no 'phases.consent' block
    defines it. It will be dropped from the phase order.
FAILED — 1 warning(s); --strict treats these as errors.

The phase is then genuinely dropped — the boot log says dropping it from the phase order and the study runs without it. So the failure mode is quiet rather than loud: a consent step you meant to have, and a study that consents nobody. --strict is the check that catches it, which is the argument for running --strict rather than plain validate.

Define every phase you name. If you want a phase gone, take it out of order.

The annotator never sees it and the config still validates. If a phase you configured is not appearing, this is the first line to grep for.

Every non-annotation phase needs a page¶

ERROR: Phase consent requires 'instrument', 'instruments', or 'file' to specify its annotation schemes

file: — a path to a .jsonl page (below). Use it for anything you wrote yourself.
instrument: / instruments: — take a string (or list of strings) naming a built-in instrument from the library below. Handing instrument a list of scheme dicts fails with unhashable type: 'list', which is the error you get if you assume it is an inline scheme list.

The two combine: instrument questions are laid out first, then the file's. Note that instrument and instruments do not — the singular wins and the list is ignored, silently. Use instruments: for one or many and forget the singular exists.

The instrument library¶

Potato ships 55 validated questionnaires, so a post-study survey that asks for a standard measure should almost never be hand-written into a page file. Naming one writes the whole questionnaire, wording and response options included, into the phase.

phases:
  order: [annotation, poststudy]
  poststudy:
    type: poststudy
    instruments: [tipi, who-5]

That boots to a page of fifteen questions — ten from tipi, five from who-5 — with no page file anywhere. Eight categories:

Category	Examples
`personality`	`bfi-2` (60 items), `tipi` (10), `nfc`, `gse`
`mental_health`	`phq-9`, `gad-7`, `k6`, `ces-d`, `pss-10`, `who-5`
`affect`	`panas`, `iri`
`social`	`rse`, `ucla-loneliness`, `mos-ss`, `macarthur-ladder`
`attitudes`	`mfq`, `sdo-7`, `rwa`, `trust-ess`, `political-efficacy`
`response_style`	`mc-sds` — social desirability, for detecting it in the rest
`short_forms`	`bfi-10`, `mini-ipip`, `phq-2`, `gad-2`, `pss-4`
`demographics`	batteries lifted from ANES, GSS, ESS, WVS, MIDUS, IPUMS and the ACS

The registry is the authority, and it carries the item count and a source URL for each one:

from potato.survey_instruments import list_instruments, get_categories
get_categories()                      # category -> instrument ids
list_instruments("mental_health")     # id, name, items, description, url

Two things to tell a researcher before they pick one. Long instruments are long: bfi-2 is 60 questions and ztpi is 56, a bigger ask than the annotation task in most studies. The short_forms category exists for that reason. And a fifteen-question page runs about two and a half screens, so the submit button ends up below the fold. check_ui.py --config config.yaml --phase poststudy reports the exact position — use --phase rather than the ordinary walk, which starts at the landing page and never gets past annotation on a corpus of any size.

psychometrics is a different thing entirely despite the neighbouring name: it fits an item-response model over the annotation labels to estimate annotator ability and item difficulty. It has nothing to do with questionnaires.

training and annotation need no page: training reads training.data_file and annotation reads annotation_schemes.

The page file format¶

A phase page is JSONL, one annotation scheme per line, using exactly the scheme syntax from annotation-types.md:

{"annotation_type":"pure_display","name":"consent_text","allow_html":true,"description":"<h3>Consent</h3><p>…</p>"}
{"annotation_type":"radio","name":"consent_agree","description":"Do you agree to take part?","labels":["I agree","I do not agree"],"humanize_labels":false,"label_requirement":{"required":true}}

Notes that cost time to find:

Prose goes in a pure_display scheme, in its description, with allow_html: true. There is no text: key on a phase that renders anything — title renders, text is ignored.
pure_display logs WARNING: No labels provided for pure display content on every render. It is noise; the content renders.
Phase-page prose lands inside a <legend>, which the theme sets to font-weight: 600. A page of it reads as entirely bold. Fix with base_css:

form.pure-display legend { font-weight: 400; }
form.pure-display legend b, form.pure-display legend h3 { font-weight: 650; }

base_css is inlined into a <style> tag rather than served as a file, so the path only has to be readable at boot. - Anything with label_requirement: {required: true} is enforced on the phase page the same way it is on the annotation page. - Keys inside these lines are not validated at all. A misspelled annotation_type here fails at boot; a misspelled option key is ignored.

Phase-level schemes replace the top-level list, and you cannot mix¶

phases:
  poststudy:
    type: poststudy
    annotation_schemes: [...]     # with top-level annotation_schemes present

fails validation:

Configuration has both top-level 'annotation_schemes' and phase-level
'annotation_schemes' in: poststudy. Use only one location to avoid confusion.

This is all-or-nothing across the config. Defining a survey inline on one phase forces every phase, including annotation, to carry its own scheme list. For a normal study — one question set for the items, different questions on the survey pages — use file: for the phase pages and keep the top-level annotation_schemes for annotation. Reach for phase-level schemes only when you genuinely want different questions per phase and are willing to write them all out.

`annotation_instructions` is not the instructions phase¶

annotation_instructions is the collapsible banner at the top of every page, including consent, training and post-study. It is a standing reminder, not a page.

Two consequences:

Long label definitions in it push the item and the questions below the fold on every single item. Put the definitions on an instructions phase and keep the banner to a few lines.
It appears above the consent text, which looks like a mistake to an annotator reading it for the first time. Word it so it survives that context.

Rendering a phase page¶

potato preview --phase X --screenshot out.png renders the page for phase X. Two things about it:

It accepts only five names. --phase prestudy is refused:

Error: phase must be one of ('consent', 'instructions', 'training', 'annotation', 'poststudy'), got 'nonsense'

prestudy is a valid phase for the server and not for the preview CLI, so a screening page cannot be rendered this way. Walk it in a browser instead.

It exits 1 on a healthy phase page. Every phase page fires Potato's own /api/current_instance 404 and two [SpanManager] console errors, and --screenshot counts those as things to report:

$ potato preview config.yaml --phase consent --screenshot consent.png ; echo $?
console.error: [SpanManager] Failed to get server instance ID during initialization
HTTP 404: http://127.0.0.1:9080/api/current_instance
1

$ potato preview config.yaml --screenshot annotation.png ; echo $?
Rendered cleanly — no browser errors.
0

Both configs are fine. Do not gate on the exit code for a phase page, and do not go looking for the bug — look at the PNG and check the error list contains nothing beyond those three lines.

Only pass a phase the task actually has, or every route redirects toward a page that does not exist and the render fails with a redirect loop.

`surveyflow` versus `phases`¶

surveyflow is the older way to attach pages before and after annotation:

surveyflow:
  "on": true
  order: [pre_annotation, post_annotation]
  pre_annotation: [surveyflow/consent.jsonl, surveyflow/instructions.jsonl]
  post_annotation: [surveyflow/news_habits.jsonl]
  sample_status: []

Quote the "on" key. Bare on: is a YAML boolean, and the loader warns:

`on:` at surveyflow was read as the boolean True … quote the key ("on":) to be explicit.

Prefer phases. It names the phase types explicitly, it controls order in one place, and it is what the training phase and --debug-phase key off. A directory called surveyflow/ holding the page files is a convention worth keeping even when the surveyflow: block is absent — that is where the pages in the example above live.

Checking the workflow actually assembled¶

Validation cannot see any of this. Boot the server and read the log:

grep -E "phase|Loaded [0-9]+ (training|attention|gold)" server.log

A healthy boot is silent about phases. Any of these means a phase is missing from the annotator's walk:

Line	Meaning
`Phase 'x' in order but not defined`	You listed it and never described it
`Failed to load phase 'x': Unknown phase: y`	`type:` is not one of the six
`Phase x requires 'instrument', 'instruments', or 'file'`	No page attached
`Failed to load phase 'x': unhashable type: 'list'`	`instrument:` got a list of schemes

Then walk it in a browser. A phase can load cleanly and still be unreachable because of what precedes it. See running-a-task.md.