Skip to content

Agent Instructions

Codex and Cursor read AGENTS.md from the project root, so it needs no install step: copy it into the project you are working in. Claude Code reads CLAUDE.md instead, and a CLAUDE.md holding the line @AGENTS.md imports it.

It is the short version. The references and helper scripts it points to come with the skill, installed as below. The text after the rule is the file itself.

Installing the skill

/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato

For Codex and Cursor, the skills CLI copies the whole skill -- SKILL.md, the references and the helpers -- into .agents/skills/potato-skill/:

npx skills add davidjurgens/potato-skill --agent codex cursor

The skill is a plain directory, so it also travels:

python scripts/package_skill.py

builds dist/potato-skill/ -- SKILL.md, the references, and the 9 executable helpers -- and dist/potato-skill.zip. One build, three destinations:

Where How
Claude Code, every project on the machine cp -r dist/potato-skill ~/.claude/skills/, or --install-personal
Claude Agent SDK Point the SDK's skill directory at dist/; it loads the skill the same way Claude Code does
Claude API Upload dist/potato-skill.zip as a skill, then name it on the request. The endpoint is in beta -- check Anthropic's Agent Skills documentation for the current call

The scripts in the bundle drive the potato command and import Potato's registries, so wherever the skill runs, Potato has to be installed in the environment running the commands. SKILL.md opens by checking exactly that.

Potato is a web annotation platform. A task is one YAML config plus a data file. This file tells you how to build one correctly.

The loop

Do not write a config and hand it over. Run it through these steps first.

potato validate config.yaml --strict              # structural check
potato preview config.yaml                        # what form did I declare?
potato preview config.yaml --screenshot out.png   # what does it look like, did it break?

All three return promptly. potato start config.yaml -p 8000 runs the task for a human to click through, but it does not return -- it holds the server in the foreground until killed, and will hang a non-interactive session indefinitely. The screenshot step already boots a real server and shuts it down, so nothing in this loop needs it. Background it if you genuinely want one running.

potato validate runs the same checks potato start runs at boot, so passing it means the server will accept the config. Its errors name the alternatives: an unknown annotation_type comes back with the list of valid ones. --strict turns unrecognized keys from a warning into a failure, which is how you catch a typo that silently disabled a feature.

--screenshot is the step that catches what validation cannot. It boots the task, opens it in a headless browser, and reports every uncaught exception, console.error and failed request. A config can validate cleanly and still render a broken interface, because most of the annotation UI is built by JavaScript after the HTML arrives. Exit code is 0 only if the page rendered with nothing to report.

Deciding what to build

A config is the last step, and the decisions before it are the ones that are expensive to reverse: what one annotator sees at a time, what they are asked, how they answer, how many people answer each item, and how you find out whether the answers are any good.

When a description leaves those open, decide the cheap ones yourself and confirm the expensive ones -- the unit of annotation, the annotator budget, consent wording, and anything involving identifiable people. Build with stated assumptions rather than waiting on an answer.

Two that are easy to get wrong and expensive to discover late:

  • The response format decides the agreement metric. Potato picks the metric from the schema kind, so a 1--5 rating stored as a radio is scored as unordered and "1 vs 2" counts the same as "1 vs 5".
  • num_annotators_per_item sets the count. Unset, there is no limit. min_annotators_per_instance and max_annotations_per_item are older spellings of the same setting, and two of them with different numbers fail validation.

The potato-skill skill has the full versions in its references/ directory: designing-a-task.md, asking-the-experimenter.md and building-the-ui.md. Without the skill, read them at https://davidjurgens.github.io/potato-skill/designing-a-task.md and the pages beside it.

Start from a working example

There are 218 example projects. Copying one beats assembling a config from field lists, and every one of them is checked in CI.

# Search the catalog
python -c "from potato.server_utils.examples_manifest import search_examples; \
           print([e['dir'] for e in search_examples(annotation_type='span')])"

The catalog is also published as JSON at potato/schemas/potato-examples.manifest.json. Each entry records the annotation types and display types a config uses, the features it switches on, and the command that runs it.

The minimum config

Four keys are always required, plus a data source:

# yaml-language-server: $schema=https://potatoannotator.readthedocs.io/en/latest/schemas/potato-config.schema.json
annotation_task_name: Sentiment Annotation
task_dir: .
output_annotation_dir: annotation_output/
data_files:
  - data/items.json
item_properties:
  id_key: id
  text_key: text
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: How does this text feel?
    labels: [positive, neutral, negative]

Keep the first line. It switches on live validation in VS Code, JetBrains, Zed and Helix, and it tells the next agent where the contract is.

The data file is JSON, JSONL, CSV or TSV. JSON may be an array of objects or one object per line. Every object needs the fields named by id_key and text_key.

Rules that will bite you

Paths resolve against task_dir, and anything resolving outside it is rejected. This is a security check, not a convenience. If you see ConfigSecurityError: Path ... is outside the project directory, you are running from the wrong directory or pointing outside the project.

annotation_type must come from the registry. There are 61 of them. Do not guess a plausible-sounding name. sentiment and classification are not types.

python -c "from potato.server_utils.schemas.registry import schema_registry; \
           print(schema_registry.get_supported_types())"

Unknown config keys only produce a warning. A typo'd key is silently ignored and the feature you thought you enabled is off. Run potato validate --strict, which turns those warnings into a failure.

Phase-level annotation_schemes replace the top-level list. They do not add to it. Having both is an error.

Never hand-edit anything in output_annotation_dir. The server rewrites those files wholesale.

Reference

Every key, with its type, default and description, is in docs/configuration/config_reference.md, and the same information machine-readable is in potato/schemas/potato-config.schema.json.

From Python:

from potato.server_utils.config_key_docs import get_key_doc
get_key_doc("attention_checks.frequency")

from potato.server_utils.schema_examples import example_scheme_for
example_scheme_for("bws")   # a scheme from a config that really runs

MCP

potato mcp is two servers, both listed under other commands in potato --help.

potato mcp serve --root . answers authoring questions out of the same registries as everything above, through tools such as list_annotation_types, describe_annotation_type, list_examples, validate_config, preview_config and render_task_screenshot. potato mcp tools prints all of them. Same answers as the shell, and render_task_screenshot hands you the rendered page as an image.

potato mcp connect --url … --token … bridges a running task, if its config carries an mcp block listing tools in mcp.tools and an agent holds a token from potato mcp issue-token. The bridge adds 17 live tools named live_get_status, live_get_progress, live_list_items, live_get_item, live_list_annotators, live_get_config, live_get_agreement, live_assign_items, live_export_data, live_submit_annotation, live_add_items and live_delete_annotations, plus five that drive the labeling engine: live_engine_status, live_engine_requests, live_engine_run, live_engine_certify and live_engine_certificate. live_delete_annotations destroys an annotator's work on an item, so it also has to be listed in mcp.destructive and called with confirm: true; Potato refuses it otherwise. Each publishes a real parameter schema, so tools/list tells you what to pass.

potato mcp config --root .    # prints a client config block
potato mcp tools              # the tool list, without starting anything

Reporting a bug or asking for a feature

Sort it before reporting it. potato validate config.yaml --strict failing, or a boot log line reading Loaded 0 <something>, means a config mistake -- a key misspelled inside a block Potato treats as opaque is accepted, ignored, and never mentioned again. That is the usual cause of "the config is right and nothing happens", and it is not a defect.

If Potato does something other than what it documents, that is a bug. If it does exactly what it documents and that is not enough, that is a feature request -- and before filing one, name the closest existing annotation type or config key and say why it does not fit, because it exists more often than people expect.

Issues go to https://github.com/davidjurgens/potato/issues. A useful report carries the version (ask potato --version; releases up to 2.7 have no such flag, and neither importlib.metadata.version("potato-annotation") nor potato.__version__ answers the same way from every directory, so say which you used and name the commit if you run from a clone), the --strict result, the shortest config that shows it, the shape of one data record rather than the records, and what you measured against what you expected. Strip secret_key, ai_support.ai_config.api_key, crowd credentials and annotator email addresses first. The potato-skill skill has a helper that assembles all of that, scripts/report_issue.py in the skill's directory. npx skills add davidjurgens/potato-skill --agent codex cursor puts that directory at .agents/skills/potato-skill/.

Filing is public and cannot be withdrawn. Show the report to whoever owns the study and get an answer before you file it.

Where to read more