Letting Models Label and People Certify¶

The 2.11 labeling engine: blind samples, contracts, and what to ask first. It ships with the potato-skill skill:

/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato

Potato 2.11 adds a labeling engine: models label every item, people label a blind random sample, and each contract in the config tests the machine labels against that sample. The machine labels a contract accepts come with a stated error bound, and the rest go to people. The researcher gets the dataset and a certificate saying what was tested, on how many human labels.

It is configured in one labeling_engine block, and annotators are served its requests only under assignment_strategy: engine. Potato's own guide is docs/labeling-engine/ in the checkout, five pages with every key; this page is what to decide with the researcher and what I saw when I ran it.

Corpus size¶

The engine saves human labels when the corpus is much larger than the human budget. On a small corpus it can sample most of the corpus and accept nothing, as it did on the 120-item run below.

I ran it on 120 product reviews with Gemma 12B on vLLM as the only source and a selective_risk contract at 10% error and 0.9 confidence. Gemma was wrong on 8 of the 120 (6.7%), under the target, but the corpus was too small to test the contract and it never passed. The planner opened 107 calibration requests and 22 resolve requests, and every one of the 120 delivered labels came from a person.

So ask how many items there are and how many human labels the researcher can pay for before proposing it. If every item will get a human label anyway, use an ordinary assignment strategy.

What to ask the researcher¶

Question	Where the answer goes
"How often may an accepted machine label be wrong, and how sure do you need to be?"	`error` and `confidence` on a `selective_risk` contract
"Do you need every label to be usable, or a rate, such as the share of toxic posts?"	`selective_risk` for labels; `ppi` for a prevalence with an interval
"Should the model match your annotators rather than a fixed accuracy?"	`no_worse_than_human`, which needs `humans.overlap_k` of at least 3
"How many human labels can you pay for?"	`budgets.human_labels`
"Which questions should the model answer?"	`labeling_engine.schemas`; free-text questions cannot be among them

The error target and the confidence are the researcher's call. They are what the certificate will claim, so do not pick them yourself.

A minimal block¶

assignment_strategy: engine
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: "Overall sentiment?"
    labels: [positive, negative, neutral]
labeling_engine:
  enabled: true
  sources:
    - id: gemma
      kind: llm
      endpoint_type: vllm
      model: google/gemma-4-12B-it-qat-w4a16-ct
      base_url: http://localhost:8001/v1
      confidence: {method: logprob}
  contracts:
    - {id: accuracy, kind: selective_risk, schema: sentiment, error: 0.1, confidence: 0.9}
    - {id: prevalence, kind: ppi, schema: sentiment, confidence: 0.9}
  budgets: {human_labels: 200}

logprob confidence reads the probabilities of the model's one-letter answer. It needs a backend that returns logprobs, such as vLLM, and a scheme with at most 20 labels. native on an llm source is refused: confidence.method native comes from decision models and detectors; an llm source uses verbalized, k_sample or logprob.

Contract keys are checked. A misspelt error names the key and the ones the kind reads: labeling_engine.contracts[0] (selective_risk) has unknown keys: eror (known: confidence, error, id, kind, per_class, schema, tolerance).

Free-text questions¶

The engine labels only schemes it can score a machine answer on, and free text is not one of them. A contract on a free-text scheme is a validation error: these contracts are on schemes the engine cannot score machine answers for: accuracy (note: text).

A free-text scheme can still sit beside the engine's schemes, but under assignment_strategy: engine people see it only on the items the engine sends them, and potato validate says so:

assignment_strategy: engine serves only the engine's requests, and the engine
does not label note, so people will answer them only on the items the engine
sends them for the other schemes, not on every item. If every item needs them,
use another assignment_strategy.

That is a warning, so --strict fails on it. If the researcher needs a comment on every item, the study cannot use assignment_strategy: engine. Tell them before building it.

An engine with no contracts¶

A machine label is accepted only when a contract has tested it. An engine with contracts: [] accepts nothing. I enabled one with a static source holding the right answer for all 120 items. It validated clean under --strict, opened 62 calibration requests, and accepted no machine label.

The budget check¶

potato validate works out whether budgets.human_labels could certify each selective_risk contract at all:

budgets.human_labels 20 cannot certify accuracy (error 0.1, confidence 0.9)
even with no machine errors: it needs at least 54 human labels (27 held out,
the rest for choosing the threshold). Raise the budget or loosen the target.

That is the floor for a model that is never wrong. In the 120-item run, 124 answered requests were not enough for a model wrong on 6.7% of the items.

assignment_strategy: engine with labeling_engine.enabled: false fails validation: assignment_strategy: engine serves the labeling engine's requests, but labeling_engine.enabled is false.

Running it¶

potato label run config.yaml        # label every item, draw the blind sample
potato start config.yaml -p 8000    # people answer the sample
potato label status config.yaml     # contracts, requests, what is still missing

label run made 120 model calls on 120 items, opened 62 calibration requests and accepted nothing: with no human labels yet, the contract read insufficient.

Only one process drives a study. While the server runs, its planner holds the study's lock, and the command line refuses with exit 2:

server:<host>:<pid>:1 is driving this study. Certify through it instead
(POST /api/engine/v1/certify, or the admin page's Certify button), or stop it
first.

status, certificate, export and calls are read-only and work while the server runs. The planner certifies on its own as labels arrive, so a running study rarely needs an explicit certify. POST /api/engine/v1/certify with the admin key in X-API-Key worked, and the same route without the key returned 403. The admin page is /admin/labeling-engine.

What annotators see¶

Every calibration item showed "Label this one yourself. Suggestions are off for this item." above the form. Calibration requests are blind: the person does not see a machine answer, so their label is an independent check of it.

Reading the result¶

potato label certificate config.yaml --format md writes the certificate. A contract that has not passed says so in its statement:

Target, not established: at most 10.0% of accepted labels wrong, with
probability at least 0.90 (overall). No machine label on this scheme is
accepted until it is met (43 held-out labels, look 1).

When reporting to the researcher, quote the statement rather than the config: the config holds the target, and only a met statement reports a result.

Getting the labels out¶

potato label export config.yaml --out DIR, or the labeling_engine export format, writes labels.jsonl, labels.csv, the certificate as JSON, Markdown and HTML, a dataset card in README.md, and model_calls.jsonl. labels.csv has one row per item and scheme:

id,schema,label,label_origin,status,source,model,source_version,confidence,contract_version,request_ids
r0001,sentiment,neutral,human,accepted,,,,,,[9]

The ordinary csv export still holds the annotators' answers. With the engine on it adds engine_purpose and engine_request columns, and the second one holds the request's split, visibility and inclusion_prob. Use them to tell the blind random sample from the other answers before anyone computes an accuracy from the human labels.

Configs with the older keys¶

pre_annotation, icl_labeling and judge_calibration still work in 2.11, and each prints a note pointing at the engine:

NOTE: pre_annotation still works, but the labeling engine now does this: use
labeling_engine (a static source). `potato label migrate-config` writes the
equivalent block. pre_annotation is planned for removal in 3.0.

potato label migrate-config config.yaml --out new.yaml writes a copy with a labeling_engine block. For pre_annotation it wrote a static source on the same field, enabled: false, and contracts: []. Enabling that block as it stands accepts nothing (see An engine with no contracts), so add the contract with the researcher before turning it on.

What I have and have not verified¶

Run on 2.11.0 (eb284819), from a checkout, against Gemma 12B on a vLLM server:

the 120-item study end to end: label run, the server's planner, three annotators labeling over HTTP from the items' true labels, status, certificate, export, and the labeling_engine and csv exports;
the writer-lock refusals of label certify and label run, POST /api/engine/v1/certify with and without the admin key, and the admin page;
every validation message quoted above, by potato validate --strict on a config built to trigger it;
migrate-config on a pre_annotation config, and that block enabled.

Not verified here: decision-model sources (Laya, hosted Jev, OpenJev), the detector, judge, jury and distilled sources, the no_worse_than_human, budget_first and distributional contracts, verify requests, crowd recruiting on Prolific, Pocket Mode with the engine, and the Python SDK.