# Letting Models Label and People Certify The 2.11 labeling engine: blind samples, contracts, and what to ask first. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- Potato 2.11 adds a labeling engine: models label every item, people label a blind random sample, and each contract in the config tests the machine labels against that sample. The machine labels a contract accepts come with a stated error bound, and the rest go to people. The researcher gets the dataset and a certificate saying what was tested, on how many human labels. It is configured in one `labeling_engine` block, and annotators are served its requests only under `assignment_strategy: engine`. Potato's own guide is `docs/labeling-engine/` in the checkout, five pages with every key; this page is what to decide with the researcher and what I saw when I ran it. ## Corpus size The engine saves human labels when the corpus is much larger than the human budget. On a small corpus it can sample most of the corpus and accept nothing, as it did on the 120-item run below. I ran it on 120 product reviews with Gemma 12B on vLLM as the only source and a `selective_risk` contract at 10% error and 0.9 confidence. Gemma was wrong on 8 of the 120 (6.7%), under the target, but the corpus was too small to test the contract and it never passed. The planner opened 107 calibration requests and 22 resolve requests, and every one of the 120 delivered labels came from a person. So ask how many items there are and how many human labels the researcher can pay for before proposing it. If every item will get a human label anyway, use an ordinary assignment strategy. ## What to ask the researcher | Question | Where the answer goes | |---|---| | "How often may an accepted machine label be wrong, and how sure do you need to be?" | `error` and `confidence` on a `selective_risk` contract | | "Do you need every label to be usable, or a rate, such as the share of toxic posts?" | `selective_risk` for labels; `ppi` for a prevalence with an interval | | "Should the model match your annotators rather than a fixed accuracy?" | `no_worse_than_human`, which needs `humans.overlap_k` of at least 3 | | "How many human labels can you pay for?" | `budgets.human_labels` | | "Which questions should the model answer?" | `labeling_engine.schemas`; free-text questions cannot be among them | The error target and the confidence are the researcher's call. They are what the certificate will claim, so do not pick them yourself. ## A minimal block ```yaml assignment_strategy: engine annotation_schemes: - annotation_type: radio name: sentiment description: "Overall sentiment?" labels: [positive, negative, neutral] labeling_engine: enabled: true sources: - id: gemma kind: llm endpoint_type: vllm model: google/gemma-4-12B-it-qat-w4a16-ct base_url: http://localhost:8001/v1 confidence: {method: logprob} contracts: - {id: accuracy, kind: selective_risk, schema: sentiment, error: 0.1, confidence: 0.9} - {id: prevalence, kind: ppi, schema: sentiment, confidence: 0.9} budgets: {human_labels: 200} ``` `logprob` confidence reads the probabilities of the model's one-letter answer. It needs a backend that returns logprobs, such as vLLM, and a scheme with at most 20 labels. `native` on an `llm` source is refused: `confidence.method native comes from decision models and detectors; an llm source uses verbalized, k_sample or logprob`. Contract keys are checked. A misspelt `error` names the key and the ones the kind reads: `labeling_engine.contracts[0] (selective_risk) has unknown keys: eror (known: confidence, error, id, kind, per_class, schema, tolerance)`. ## Free-text questions The engine labels only schemes it can score a machine answer on, and free text is not one of them. A contract on a free-text scheme is a validation error: `these contracts are on schemes the engine cannot score machine answers for: accuracy (note: text)`. A free-text scheme can still sit beside the engine's schemes, but under `assignment_strategy: engine` people see it only on the items the engine sends them, and `potato validate` says so: ``` assignment_strategy: engine serves only the engine's requests, and the engine does not label note, so people will answer them only on the items the engine sends them for the other schemes, not on every item. If every item needs them, use another assignment_strategy. ``` That is a warning, so `--strict` fails on it. If the researcher needs a comment on every item, the study cannot use `assignment_strategy: engine`. Tell them before building it. ## An engine with no contracts A machine label is accepted only when a contract has tested it. An engine with `contracts: []` accepts nothing. I enabled one with a `static` source holding the right answer for all 120 items. It validated clean under `--strict`, opened 62 calibration requests, and accepted no machine label. ## The budget check `potato validate` works out whether `budgets.human_labels` could certify each `selective_risk` contract at all: ``` budgets.human_labels 20 cannot certify accuracy (error 0.1, confidence 0.9) even with no machine errors: it needs at least 54 human labels (27 held out, the rest for choosing the threshold). Raise the budget or loosen the target. ``` That is the floor for a model that is never wrong. In the 120-item run, 124 answered requests were not enough for a model wrong on 6.7% of the items. `assignment_strategy: engine` with `labeling_engine.enabled: false` fails validation: `assignment_strategy: engine serves the labeling engine's requests, but labeling_engine.enabled is false.` ## Running it ```bash potato label run config.yaml # label every item, draw the blind sample potato start config.yaml -p 8000 # people answer the sample potato label status config.yaml # contracts, requests, what is still missing ``` `label run` made 120 model calls on 120 items, opened 62 calibration requests and accepted nothing: with no human labels yet, the contract read `insufficient`. Only one process drives a study. While the server runs, its planner holds the study's lock, and the command line refuses with exit 2: ``` server:::1 is driving this study. Certify through it instead (POST /api/engine/v1/certify, or the admin page's Certify button), or stop it first. ``` `status`, `certificate`, `export` and `calls` are read-only and work while the server runs. The planner certifies on its own as labels arrive, so a running study rarely needs an explicit certify. `POST /api/engine/v1/certify` with the admin key in `X-API-Key` worked, and the same route without the key returned 403. The admin page is `/admin/labeling-engine`. ## What annotators see Every calibration item showed "Label this one yourself. Suggestions are off for this item." above the form. Calibration requests are blind: the person does not see a machine answer, so their label is an independent check of it. ## Reading the result `potato label certificate config.yaml --format md` writes the certificate. A contract that has not passed says so in its statement: ``` Target, not established: at most 10.0% of accepted labels wrong, with probability at least 0.90 (overall). No machine label on this scheme is accepted until it is met (43 held-out labels, look 1). ``` When reporting to the researcher, quote the statement rather than the config: the config holds the target, and only a `met` statement reports a result. ## Getting the labels out `potato label export config.yaml --out DIR`, or the `labeling_engine` export format, writes `labels.jsonl`, `labels.csv`, the certificate as JSON, Markdown and HTML, a dataset card in `README.md`, and `model_calls.jsonl`. `labels.csv` has one row per item and scheme: ``` id,schema,label,label_origin,status,source,model,source_version,confidence,contract_version,request_ids r0001,sentiment,neutral,human,accepted,,,,,,[9] ``` The ordinary `csv` export still holds the annotators' answers. With the engine on it adds `engine_purpose` and `engine_request` columns, and the second one holds the request's `split`, `visibility` and `inclusion_prob`. Use them to tell the blind random sample from the other answers before anyone computes an accuracy from the human labels. ## Configs with the older keys `pre_annotation`, `icl_labeling` and `judge_calibration` still work in 2.11, and each prints a note pointing at the engine: ``` NOTE: pre_annotation still works, but the labeling engine now does this: use labeling_engine (a static source). `potato label migrate-config` writes the equivalent block. pre_annotation is planned for removal in 3.0. ``` `potato label migrate-config config.yaml --out new.yaml` writes a copy with a `labeling_engine` block. For `pre_annotation` it wrote a `static` source on the same field, `enabled: false`, and `contracts: []`. Enabling that block as it stands accepts nothing (see **An engine with no contracts**), so add the contract with the researcher before turning it on. ## What I have and have not verified Run on 2.11.0 (`eb284819`), from a checkout, against Gemma 12B on a vLLM server: - the 120-item study end to end: `label run`, the server's planner, three annotators labeling over HTTP from the items' true labels, `status`, `certificate`, `export`, and the `labeling_engine` and `csv` exports; - the writer-lock refusals of `label certify` and `label run`, `POST /api/engine/v1/certify` with and without the admin key, and the admin page; - every validation message quoted above, by `potato validate --strict` on a config built to trigger it; - `migrate-config` on a `pre_annotation` config, and that block enabled. Not verified here: decision-model sources (Laya, hosted Jev, OpenJev), the detector, judge, jury and distilled sources, the `no_worse_than_human`, `budget_first` and `distributional` contracts, verify requests, crowd recruiting on Prolific, Pocket Mode with the engine, and the Python SDK.