# What to Confirm With the Researcher

Which design choices to make yourself and which to hand back. It ships with the `potato-skill` skill:

```
/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato
```

---

Most task descriptions specify what to annotate and leave the rest open. Some
of what is left open has a defensible default and some does not, and the
difference is what this file is about.

## Before the design questions

The design questions below assume a brief and a data file. Half the time the
researcher is holding more than that and has not thought to mention it, because
from where they sit it is not part of the request. Four things change what you
build, and all four are cheap to ask and expensive to discover late:

| Ask | Why it changes the build |
|---|---|
| "What format is the data in, and where does the media live?" | Decides `data_files` versus `data_directory` versus `data_sources`, and whether images need `media_directory`. A folder of transcripts or a ConvoKit corpus has its own converter |
| "Do you already have annotations for any of this?" | `potato import` reads 20 formats and produces a runnable project. Rebuilding one by hand is the most avoidable way to lose an afternoon |
| "What do you need to come out at the end?" | 32 export formats. A study aimed at training a detector should be checked against `coco` **before** anyone annotates, not after |
| "Who is running this once I hand it over, and for how long?" | Decides hosting, whether logins must survive a restart, and whether it needs an allowlist |

`importing-existing-work.md` and `getting-the-data-out.md` cover the first three.
The fourth is in `deploying.md` and `after-annotators-start.md`.

## The rule

**Decide it yourself if being wrong is cheap to fix. Ask if being wrong is
expensive, irreversible, or not yours to decide.**

Changing a keyboard shortcut costs nothing. Re-running 5,000 items because the
unit of annotation was wrong costs the whole budget. Publishing consent wording
nobody approved is not a cost you get to weigh at all.

## Ask about these

**The unit, when the data does not settle it.** A file of paragraphs where the
question is about sentences is a real fork, not a detail. Whichever you choose
you may be re-running the study.

**Annotators per item, when agreement is going to be reported.** This is the
budget question, and it is theirs. Propose a number with the reason.

**Consent wording, and anything else with an ethics review behind it.** Their
institution has approved text. Draft from what they told you, hand it over as a
draft, and say plainly that it needs replacing with the approved version.

**Whether annotators are paid strangers or colleagues.** It decides whether
gold standards, attention checks and a completion code are worth the annotator
time. You usually cannot infer it.

**The label definitions, whenever they have any.** Ask for the codebook from a
previous round, the paper the scheme comes from, or an annotation manual, before
writing definitions yourself. They carry the construct the study is about, so
getting them wrong invalidates the result rather than costing a re-render. If
they have none, draft them and hand them back marked as drafts —
`writing-guidelines.md` has the four parts each one needs.

**Any label set that looks incomplete.** Three labels with no "none" and no
"unclear" is nearly always an oversight, and it is the kind that quietly
poisons the data — annotators put genuinely unclear items somewhere arbitrary
and nothing downstream can see it happened. Say which items you think have
nowhere to go.

**Anything touching identifiable people.** If the data has usernames, faces or
locations in it, ask before showing them to annotators.

**Whether the label set is final, if annotation is starting soon.** Say plainly
that this is the last cheap moment. Once people have answered, adding a question
collects nothing from anyone who already finished, and renaming one silently
strips the old answers out of every report — `after-annotators-start.md` has
what actually happens. A researcher who knows the deadline will usually take the
extra day.

## Defaults to take without asking

| Decision | Default to take |
|---|---|
| Keyboard shortcuts | Add them on any scheme with ≤9 labels |
| Question order | Gating question first, follow-ups behind `display_logic` |
| Required vs optional | All required, plus `require_fully_annotated: true` |
| Item order | `random`, and set `random_seed` |
| Scale points | 5, every point labelled, unless they named a number |
| Instructions page | Write a draft from their description, marked as a draft |
| Login | Whatever the example you started from uses |
| Output format | JSON |
| Progress indicator, navigation | Leave the defaults on |

State these in a short "what I assumed" list when you hand the task over. An
assumption they can see is one they can correct in ten seconds.

## How to ask

**Batch it.** One message with every question, not a conversation. They are
answering between other work.

**Propose an answer to each.** "How many annotators?" makes them do the design.
"I've set 3 annotators per item, since you mentioned reporting agreement — say
if the budget is tighter" gives them something to accept or push back on, and
they can do it in one word.

**Say what it costs to change later.** They cannot rank your questions by
consequence, and you can. Lead with the ones that are expensive to get wrong.

**Cap it at five or six.** Anything past that is a decision you should have
made yourself.

**Never block on it.** Build the task with your assumptions, verify it, and ask
alongside something working. A running task with three open questions is worth
more than three questions and nothing to look at.

### Shape of the message

> Here's the task — `config.yaml`, screenshot attached. It validates and
> renders. Four things I'd like you to confirm:
>
> 1. **Unit of annotation.** Your CSV has one row per thread, and I've made a
>    thread the thing an annotator judges. If the question is really about
>    individual posts, that changes the data file and everything downstream, so
>    it is worth getting right now.
> 2. **Three annotators per item**, since you mentioned agreement. That is 3×
>    the cost of one — say if the budget is tighter and I'll drop to 1 and
>    overlap a subset instead.
> 3. **Consent page.** I wrote a draft from what you described. It needs
>    replacing with whatever your IRB approved.
> 4. **"Unclear" option.** The label set is yes/no. Some of these comments are
>    genuinely ambiguous (item c003, for instance), and without a third option
>    annotators will split them arbitrarily. Add one?
>
> Assumed, easily changed: `y`/`n` shortcuts on the yes/no question, all
> questions required, random item order with a fixed seed, 5-point confidence
> scale with every point labelled.

## When they cannot be reached

Build it, make every assumption explicit in a comment in the config *and* in
your handover, and pick the conservative option: more context per item rather
than less, an extra "unclear" label rather than a forced choice, one annotator
rather than a budget they did not approve. Then say which choices you would
revisit first.