Data Files and Admin Access¶

Input formats, side files, and reading results back out. It ships with the potato-skill skill:

/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato

Where items come from, and who is allowed to annotate them. Nothing else in this pack covers either.

Items¶

data_files:
  - data/articles.json
item_properties:
  id_key: id
  text_key: body

data_files accepts JSON, JSONL, CSV and TSV. JSON may be an array of objects or one object per line. Every object needs the fields named by id_key and text_key.

Several files merge in the order listed, and they do not have to carry the same fields — I loaded a three-item file and a two-item file whose objects had different extra keys and got five instances, the first file's items first. But an id repeated across files stops the server dead, and validate does not see it coming:

$ potato validate config.yaml --strict   -> OK — no issues found.
$ potato start config.yaml               -> ValueError: Duplicate instance ID
                                            'SHARED' found at item 2

Failing rather than silently keeping one of the two is the right call, and the message names the id. Worth knowing before you split a corpus across files, because it is the one data problem that passes validation and then refuses to boot.

Key	Effect
`item_properties.id_key`	Field holding the unique identifier
`item_properties.text_key`	Field shown to the annotator when there is no `instance_display`
`item_properties.category_key`	Field holding a category, used by `category_based` assignment
`item_properties.kwargs`	Extra per-item fields carried through to the display layer

text_key still matters when you define instance_display. It is what plain-text rendering falls back to, what span schemes anchor to by default, and what the training and search surfaces read. Point it at the field with the actual content, not at a title.

Every other field in the row is available to the display layer by name. A row with outlet, headline and body can render all three as separate instance_display.fields without any of them being text_key. Fields you do not display are still stored, which is how you keep a condition label out of the annotator's view without dropping it from the data — but only parquet exports them. csv, tsv and jsonl carry instance_id and the answers and no item fields at all, so plan to hand the data file over with the export, or to use parquet. getting-the-data-out.md has the comparison.

Every path in the config has to resolve inside the project directory. A data_files entry that climbs out with ../ is refused at boot with ConfigSecurityError: Path '...' resolves to '...' which is outside the project directory, and potato validate reports the same thing without booting. Copy shared data into the project, or move the config up to a directory that contains both.

A directory instead of a list¶

data_directory: ./data/incoming
data_directory_encoding: utf-8
watch_data_directory: true      # rescan while running and pick up new files
watch_poll_interval: 5.0

It reads .json, .jsonl, .csv and .tsv, and skips everything else without comment. A directory of .md or .txt files loads nothing, and the only sign is Loaded 0 instances from data_directory: ./incoming at INFO in the boot log — the config validates, the server starts, and the first annotator gets an empty study. Count the instances in the log before handing out the URL. This is also not the way to get documents into a document field; see building-the-ui.md.

Remote and live sources¶

# type: url, s3, huggingface, google_sheets, database, google_drive,
#       dropbox or file
data_sources:
  - type: url
    url: https://example.org/items.json
data_cache:
  enabled: true
  ttl_seconds: 3600
  max_size_mb: 200

Sources can be polled for new rows, which is how a task keeps ingesting while annotators work. Pair with automation (rules over incoming items) and triage (rank what arrives). Both are in modes-and-subsystems.md.

Media¶

media_directory: media      # default; served at /media/

Put images, audio and video under it, and write the path including the directory in the data field: media/photo.png, not photo.png. The displays emit the field value as the URL; only depth_map prepends the directory itself, so a bare filename resolves against the site root and 404s. Measured across one page on 2.8.2-26: image, gallery, video, audio, pdf, audio_dialogue and web_agent_trace all need the prefix in the data, and all of them except pdf fail without a word on the page.

Without an instance_display field of the right type, that path renders as text — the classic "the item shows a file path instead of an image". See building-the-ui.md for the display types.

Big corpora¶

partial_loading: {...}          # load lazily rather than reading everything at boot
item_store:
  backend: paged                # memory (default) or paged
  cache_size: 1000
  path: .item_cache.sqlite      # under the output directory when unset

An unknown item_store.backend warns and silently falls back to memory rather than refusing to start, so check the boot log if paging was the point.

Output¶

output_annotation_dir: annotation_output/
export_annotation_format: [csv]        # periodic auto-export; empty means none
auto_export_interval: 60               # seconds
export_include_phase_data: false       # consent and survey answers; usually where the PII is
export_include_annotation_changes: false

There is one storage format and it is not configurable. Annotations live at annotation_output/<user>/user_state.json, one file per annotator, whatever else the config says. Any other shape is an export, and there are two ways to ask for one: export_annotation_format, which writes it on a timer, and the admin export route, which writes it on demand. output_annotation_format is deprecated — the loader reads it as export_annotation_format and warns. Write the live key in anything you author.

So what a researcher gets at the end of a study is a directory of per-annotator JSON, not a table. If they want one row per item, produce it: either set export_annotation_format: [csv], or read the user_state.json files and write the table yourself. Decide which before handover and say so, because "where is my CSV" is the first question after the annotators finish.

annotation_output/ is rewritten wholesale by the server. Never hand-edit anything in it — potato repair-annotations exists for when something already did.

Per-annotator state lives at annotation_output/<user>/user_state.json and is the fastest way to check what the server actually stored. Keys worth knowing: instance_id_ordering, instance_id_to_label_to_value, instance_id_to_span_to_value, phase_to_page_to_label_to_value, training_state, max_assignments.

Who may log in¶

login:
  type: password          # password | url_direct | none
user_config:
  allow_all_users: true   # anyone may register

Key	Effect
`login.type`	`password` (register and sign in), `url_direct` (identity in a query parameter), `none`
`login.url_argument`	The query parameter carrying the user id under `url_direct`
`user_config.allow_all_users`	Default `true`. Anyone can register
`user_config.users`	Allowlist, used when `allow_all_users: false`
`require_password` / `require_no_password`	Force the choice regardless of `login.type`
`authentication`	SSO and database-backed accounts
`rbac`	Role assignments and SSO role mapping

url_direct is what crowdsourcing platforms want: the platform appends the worker id and the annotator never sees a login screen.

Running on a crowd platform¶

login: {type: url_direct, url_argument: PROLIFIC_PID}
crowdsourcing: {provider: prolific}
completion_code: CODE-ABC123
auto_redirect_on_completion: true
auto_redirect_delay: 3000        # milliseconds on the completion page first

crowdsourcing.provider takes prolific, mturk, connect, sona, microworkers, clickworker, expert, generic or url_direct, and the boot log confirms it — Initialized crowdsourcing provider: prolific. Set login.url_argument to whatever the platform appends: Prolific sends PROLIFIC_PID, so ?PROLIFIC_PID=worker777 seats that worker with no login screen and their id in the navbar.

Naming a provider changes the completion page, which is the part worth seeing before you launch. With the redirect off it reads

Thank You! You have completed the annotation task and your responses are saved.
CODE-ABC123
Click to copy your completion code
Return to Prolific

— the code, a copy control, and a return link named after the provider. With auto_redirect_on_completion: true the worker is sent to the platform's completion URL instead, after auto_redirect_delay. I drove both: the redirect really does leave for the platform, so test with it off first unless you enjoy losing your browser session mid-check.

The login page has two tabs, login and register, in one document — if you are driving it in a browser, switch with switchTab('register') before filling the register form. Both forms post action (login or register), email and pass; the fields are #login-email/#login-pass and #register-email/#register-pass.

The default authenticator keeps accounts in memory. The boot log names the one you have: Initialized UserAuthenticator with method: in_memory. That method has no store behind it, so a username is only as durable as the process. authentication is the block that gives database-backed or SSO accounts; secret_key and persist_sessions cover the Flask session, which is a different thing. Decide this before you recruit rather than after the first restart.

Serving it¶

Key	Effect	Default
`port`	Port; `-p` overrides	8000
`host`	Interface to bind; `0.0.0.0` exposes beyond localhost	localhost
`secret_key`	Flask session signing key, and the key that decides whether a login survives a restart: unset, every boot signs with a fresh random key and everyone is logged out	—
`session_lifetime_days`	How long a session lasts. The login response carries it as an explicit cookie `Expires`	2
`persist_sessions`	Asks for sessions to be kept across a restart; refuses to start without `secret_key`	false
`admin_api_key`	Admin API key, sent as `X-API-Key`. Generated into `{task_dir}/admin_api_key.txt` when unset	—

debug: true is for a server on your own machine, never for one an annotator can reach. The same goes for --debug on the command line. On its own it still shows the login page. With --debug-phase annotation it signs every visitor in as one shared debug_user and drops them into the task, which is what makes it useful for checking the interface. It also opens the admin pages without a key to requests from the server's own machine, so an admin-auth check you run locally under debug proves nothing. On 2.11.1, with --debug on the default 0.0.0.0 bind, /admin/api/overview answered 200 on localhost and 403 on the machine's network address. 2.11.0 refused that bind outright ("Refusing to start: debug is enabled and the server is bound to 0.0.0.0 rather than loopback"), so on 2.11.0 pass --host 127.0.0.1.

Binding 0.0.0.0 is how a researcher on the same network opens the task. Say so explicitly in a handover, along with the port, rather than leaving them to guess that localhost is not the only address.