What to Confirm With the Researcher¶
Which design choices to make yourself and which to hand back. It ships with the potato-tasks skill:
/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-tasks@potato
Most task descriptions specify what to annotate and leave the rest open. Some of what is left open has a defensible default and some does not, and the difference is what this file is about.
Before the design questions¶
The design questions below assume a brief and a data file. Half the time the researcher is holding more than that and has not thought to mention it, because from where they sit it is not part of the request. Four things change what you build, and all four are cheap to ask and expensive to discover late:
| Ask | Why it changes the build |
|---|---|
| "What format is the data in, and where does the media live?" | Decides data_files versus data_directory versus data_sources, and whether images need media_directory. A folder of transcripts or a ConvoKit corpus has its own converter |
| "Do you already have annotations for any of this?" | potato import reads fourteen formats and produces a runnable project. Rebuilding one by hand is the most avoidable way to lose an afternoon |
| "What do you need to come out at the end?" | Twenty-nine export formats. A study aimed at training a detector should be checked against coco before anyone annotates, not after |
| "Who is running this once I hand it over, and for how long?" | Decides hosting, whether logins must survive a restart, and whether it needs an allowlist |
importing-existing-work.md and getting-the-data-out.md cover the first three.
The fourth is in deploying.md and after-annotators-start.md.
The rule¶
Decide it yourself if being wrong is cheap to fix. Ask if being wrong is expensive, irreversible, or not yours to decide.
Changing a keyboard shortcut costs nothing. Re-running 5,000 items because the unit of annotation was wrong costs the whole budget. Publishing consent wording nobody approved is not a cost you get to weigh at all.
Ask about these¶
The unit, when the data does not settle it. A file of paragraphs where the question is about sentences is a real fork, not a detail. Whichever you choose you may be re-running the study.
Annotators per item, when agreement is going to be reported. This is the budget question, and it is theirs. Propose a number with the reason.
Consent wording, and anything else with an ethics review behind it. Their institution has approved text. Draft from what they told you, hand it over as a draft, and say plainly that it needs replacing with the approved version.
Whether annotators are paid strangers or colleagues. It decides whether gold standards, attention checks and a completion code are worth the annotator time. You usually cannot infer it.
The label definitions, whenever they have any. Ask for the codebook from a
previous round, the paper the scheme comes from, or an annotation manual, before
writing definitions yourself. They carry the construct the study is about, so
getting them wrong invalidates the result rather than costing a re-render. If
they have none, draft them and hand them back marked as drafts —
writing-guidelines.md has the four parts each one needs.
Any label set that looks incomplete. Three labels with no "none" and no "unclear" is nearly always an oversight, and it is the kind that quietly poisons the data — annotators put genuinely unclear items somewhere arbitrary and nothing downstream can see it happened. Say which items you think have nowhere to go.
Anything touching identifiable people. If the data has usernames, faces or locations in it, ask before showing them to annotators.
Whether the label set is final, if annotation is starting soon. Say plainly
that this is the last cheap moment. Once people have answered, adding a question
collects nothing from anyone who already finished, and renaming one silently
strips the old answers out of every report — after-annotators-start.md has
what actually happens. A researcher who knows the deadline will usually take the
extra day.
Defaults to take without asking¶
| Decision | Default to take |
|---|---|
| Keyboard shortcuts | Add them on any scheme with ≤9 labels |
| Question order | Gating question first, follow-ups behind display_logic |
| Required vs optional | All required, plus require_fully_annotated: true |
| Item order | random, and set random_seed |
| Scale points | 5, every point labelled, unless they named a number |
| Instructions page | Write a draft from their description, marked as a draft |
| Login | Whatever the example you started from uses |
| Output format | JSON |
| Progress indicator, navigation | Leave the defaults on |
State these in a short "what I assumed" list when you hand the task over. An assumption they can see is one they can correct in ten seconds.
How to ask¶
Batch it. One message with every question, not a conversation. They are answering between other work.
Propose an answer to each. "How many annotators?" makes them do the design. "I've set 3 annotators per item, since you mentioned reporting agreement — say if the budget is tighter" gives them something to accept or push back on, and they can do it in one word.
Say what it costs to change later. They cannot rank your questions by consequence, and you can. Lead with the ones that are expensive to get wrong.
Cap it at five or six. Anything past that is a decision you should have made yourself.
Never block on it. Build the task with your assumptions, verify it, and ask alongside something working. A running task with three open questions is worth more than three questions and nothing to look at.
Shape of the message¶
Here's the task —
config.yaml, screenshot attached. It validates and renders. Four things I'd like you to confirm:
- Unit of annotation. Your CSV has one row per thread, and I've made a thread the thing an annotator judges. If the question is really about individual posts, that changes the data file and everything downstream, so it is worth getting right now.
- Three annotators per item, since you mentioned agreement. That is 3× the cost of one — say if the budget is tighter and I'll drop to 1 and overlap a subset instead.
- Consent page. I wrote a draft from what you described. It needs replacing with whatever your IRB approved.
- "Unclear" option. The label set is yes/no. Some of these comments are genuinely ambiguous (item c003, for instance), and without a third option annotators will split them arbitrarily. Add one?
Assumed, easily changed:
y/nshortcuts on the yes/no question, all questions required, random item order with a fixed seed, 5-point confidence scale with every point labelled.
When they cannot be reached¶
Build it, make every assumption explicit in a comment in the config and in your handover, and pick the conservative option: more context per item rather than less, an extra "unclear" label rather than a forced choice, one annotator rather than a budget they did not approve. Then say which choices you would revisit first.