Model Assistance From a Self-Hosted Endpoint¶

Pointing ai_support at your own vLLM or other OpenAI-compatible server. It ships with the potato-skill skill:

/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-skill@potato

ai_support puts a row of assistant buttons beside each annotation scheme — Hint, Keyword, Rationale — and fills them from an LLM. This file covers pointing that at a server you run yourself: vLLM, SGLang, LM Studio, llama.cpp, anything speaking the OpenAI chat API. Commercial providers use the same keys and are not covered here, because I have not run one.

The three endpoint types I drove against a live server all work. What is left to get wrong is the config: the keys sit one level deeper than they read, and the switch that decides whether any button appears is off by default and warns about nothing.

Where the keys go¶

Three levels, and only the first is guessable:

ai_support:
  enabled: true
  endpoint_type: openai_vision
  ai_config:
    base_url: "http://your-server:8001/v1"
    model: "google/gemma-4-12B-it-qat-w4a16-ct"
    temperature: 0.1
    include:
      all: true

endpoint_type sits beside ai_config. Everything else — base_url, model, api_key, max_tokens, temperature — goes inside it. Written one level up, where it reads naturally, the keys are dropped and the endpoint falls back to its own default host. validate --strict catches that one:

Unrecognized config key 'ai_support.base_url'. This key will be ignored.

include.all is off by default and nothing warns you. Without it the config validates clean — OK — no issues found — the endpoint connects, the boot log says the endpoint is ready, and the page renders <div class="ai-help none"> with no buttons in it. This is the single most likely reason a correctly configured ai_support shows nothing. ai_config.include.special_include is the per-page, per-scheme alternative, keyed page number then annotation id.

config-keys-nested.md documents these, include.all included. Setting base_url makes api_key optional, on the validator as well as in the endpoint, so a local server needs no placeholder key.

One ai_support block serves every scheme, and each gets its own buttons and its own answer. Three schemes on a page — a radio, a multiselect and a free text — gave three separate Hint/Keyword/Rationale rows, and each hint was about that scheme's question: the radio's discussed weighing the complaint against the resolution, the multiselect's walked its own five labels and said which phrases mapped to which, the text one described the summary it wanted. The suggestions landed on the right scheme too, highlighted with a sparkle — Mixed on the radio, and Shipping, Build Quality and Support but not Battery or Price on the multiselect. There is nothing per-scheme to configure.

The model's output is escaped before it reaches the page, which matters because a hint is generated from the item: markup in your corpus comes back through the model. I put <b> and <img> in an item's text and the hint panel rendered neither — 607 characters of hint, no elements the renderer did not create. The same holds for the rationale panel. Markdown in the output is rendered rather than shown as asterisks; I have driven the escaping half against a live model and taken the markdown half from the fix, because the model did not happen to emit any on the runs I made.

One rough edge left: on a narrow scheme the three buttons lose their labels and render as bare icons, so the same row reads differently depending on how wide its scheme box is.

Keeping the endpoint out of the config¶

ai_support.ai_config_file names a second YAML file, merged in at load. It is how the bundled examples keep API keys and lab-internal hostnames out of the committed config, and it is worth using for the same reason.

The file is flat. Its keys go straight into ai_config, and endpoint_type is lifted out to sit beside it:

endpoint_type: openai_vision
base_url: "http://your-server:8001/v1"
model: "google/gemma-4-12B-it-qat-w4a16-ct"

Give that file its own ai_config: block and you get ai_config.ai_config, every real setting is invisible, and the boot log complains about a missing API key, which points at the wrong key entirely.

If the named file does not exist, Potato warns and sets enabled: false. validate --strict fails on that warning, so the check catches a study whose whole point is model assistance running with the model switched off. Ship an ai-config.yaml.example beside the config for whoever clones it, as examples/image/image-vllm-rationale/ does.

Which endpoint_type to use¶

endpoint_type: vllm for a vLLM server, openai for anything else OpenAI-compatible, openai_vision if items include images. Driven against a live vLLM 0.24.0 serving a 12B text+vision model, clicking every assistant in a browser and reading the response body:

`endpoint_type`	Text	Images	Constrains the output with
`vllm`	yes	no	`response_format: json_schema`
`openai`	yes	no	`response_format: json_schema`
`openai_vision`	yes	yes	`json_schema`, falling back to `json_object` then nothing if the server refuses

All three returned correctly shaped JSON for all three assistants, and suggestive_choice highlighted the label in the form.

openai_vision is named for its capability, not its backend: its docstring is explicit that base_url is there for vLLM, SGLang, LM Studio, llama.cpp and LiteLLM, and it appends /v1 if you leave it off. Pick it whenever any item is an image, because it is the only one of the three that sends one.

On 2.11.1 the validator accepts openai, openai_vision, anthropic, anthropic_vision, gemini, huggingface, ollama, ollama_vision, openrouter (also spelt open_router; 2.11.0 refused both), vllm, and yolo for detection. Anything else fails with ai_support.endpoint_type must be one of: and that list. sam and sam3 have endpoint classes in Potato's code but are refused here. I have run three of them.

max_tokens¶

Defaults to 800, which covers the multi-label formats. Rationale and Keyword both produce one entry per label, so a three-label scheme runs 300-500 tokens; the old default of 100 cut them off mid-object. Lower it, or give a scheme many labels, and from Potato 2.9.0 the log says so:

The model hit max_tokens (100) before finishing this response, so the reply is
cut off and may parse into the wrong shape. Raise ai_config.max_tokens.

Worth knowing because truncation and a genuinely broken endpoint share a symptom: the salvage step returns whatever key-value pairs completed, so the renderer gets a plausible dict of the wrong type and the tooltip reads "No rationales available" either way.

From 2.9.0, all ten text and vision endpoints warn for the assistants, on text and image items alike. Their chat methods warn too, with this chat reply where an assistant's warning says this response: on 2.9.0 only anthropic, ollama, openai and vllm, and on all ten from 2.9.1. Before 2.9.0 only openai, openai_vision and vllm warned, and never from chat. On anthropic, anthropic_vision, gemini, huggingface, ollama, ollama_vision and openrouter a cut-off reply reached the renderer with nothing in the log and produced the empty tooltip above, which is also what a broken endpoint produces. A quiet log is evidence the reply was complete only on 2.9.0 or later, and for chat on 2.9.0 only on anthropic, ollama, openai or vllm.

Which field the model gets¶

Two keys decide, and only one of them is the obvious one.

item_properties.text_key names the field that goes to the model on the text path. It is the same key that decides plain-text rendering, span anchoring, training and search (data-and-access.md), so it is not free to point somewhere else just to change what the assistants read.

ai_support.image_key names the field holding the picture, for an item that carries both. item_properties.image_key is the same key one level up, and ai_support.image_key wins:

item_properties:
  id_key: id
  text_key: body                  # the ticket text
ai_support:
  enabled: true
  endpoint_type: openai_vision
  image_key: screenshot           # the field holding the image URL

With both keys set the model receives both: the picture, and the text field named by text_key, in the same request. Without image_key a vision endpoint on a text-keyed item never receives an image, and nothing says so — the endpoint connects, the buttons render and the model answers about the half it was given.

Read the response body to find out what it was given. The tooltip cannot tell you, and a model answering about the wrong half of the item is fluent about it — the hint reads like a hint, and the label it suggests is highlighted in the form either way. One /api/get_ai_suggestion call on one multimodal item settles which field reached the model; nothing in the config does.

Which images the model can reach¶

The AI path loads the image itself, server-side. From 2.8.2-29 a bare filename, or a media/... or /media/... path, under media_directory is read off disk and sent as base64. Anything else has to be an http(s) URL, and one resolving to a private, loopback or link-local address is refused. That refusal is SSRF protection on a URL that comes from your data file, and it means a full URL to your own Potato instance (http://localhost:8000/media/cat.png) never reaches the model, even though the same file written as /media/cat.png does. A data: URI is refused too: the page displays one (modalities.md), and the model never gets it. Annotators see the image in every one of these cases, so checking the page will not catch it. Write media paths in the data as /media/<path>, the form the page serves, and the model gets the same file.

When the image cannot be loaded, the request goes to the model as text only, and the answer comes back in the same shape as an answer about the picture. From 2.9.0 the log says so the model is being sent TEXT ONLY, and by the source the /api/get_ai_suggestion response carries image_attached: false. Before 2.9.0 nothing said so.

A separate detail: whether an item counts as an image at all is decided by sniffing the text for an extension (.jpg, .png, .gif, .webp, .bmp), a known image host, or the word "image" in an http URL. An extensionless URL from your own object store is read as text and the vision path never runs.

The three text assistants¶

Three, for text schemes, each with its own expected output shape:

Button	Produces	Where it lands
Hint	Guidance that avoids naming the answer	A tooltip, plus `suggestive_choice` highlighting that label in the form
Keyword	Per-label keywords	Overlays on the item text, labelled — not the tooltip, which stays empty
Rationale	One argument per label, for and against	A tooltip

Which appear is filtered per item by what the endpoint declares it can do, so an image item correctly drops Keyword while a text item keeps it, on the vision endpoints too. The prompts are per annotation type and live in potato/ai/prompt/*.json, one file per type — radio, likert, multiselect, select, slider, number, span, text, plus the image and video ones. A scheme whose type has no file there gets no assistants and the tooltip says the annotation type does not exist in the prompts.

ai_support.ai_config.model_module and annotation_path override the schemas and the prompt files respectively, if the defaults do not fit the study.

Letting them ask the model instead¶

chat_support is a separate block with the same endpoint shape, and it puts a sidebar beside the form rather than buttons beside each scheme. Reach for it when the researcher says "let them ask a model while annotating" — the assistants answer one question about one scheme, the chat answers whatever the annotator types.

chat_support:
  enabled: true
  endpoint_type: openai
  ai_config:
    base_url: "http://your-server:8001/v1"
    model: "google/gemma-4-12B-it-qat-w4a16-ct"
    temperature: 0.3
    max_tokens: 2000
  ui:
    title: Ask the model
    sidebar_width: 420
    max_history_per_instance: 50

endpoint_type and ai_config sit at the same level as they do under ai_support, and include.all is not involved: the sidebar renders whenever the block is enabled. The boot log's line is Chat endpoint initialized: <endpoint_type>.

The sample sets max_tokens well above the 800 default, and should. An assistant returns one entry per label, so the author knows roughly how long the reply will be. Chat length is set by whatever the annotator types. Asked a question covering three labels, this endpoint at 400 stopped on the bare heading ### 3. Unclear with nothing under it; at 800 it reached the last section but not its closing sentence; at 2000 it finished. Those runs were on a build before 2.9.0, whose chat path never checked the finish reason, and nothing appeared in the log. From 2.9.0 this endpoint logs before finishing this chat reply, but the sidebar still shows the reply with nothing to mark it as cut off, because only its text reaches the browser, so the log is the only place a cut-off shows. A cap costs nothing unless the model reaches it.

It is given far more than the assistants are. The default prompt carries the task name, the task description, every scheme's name and its labels, and the item's text_key field. Asked what it had been given, the model listed all eight scheme names in the study. It is not given the image, on any endpoint type. On a task whose item is a picture, the chat only ever sees the text field beside it.

The default prompt refuses to name a label. Its closing line is an instruction not to tell the annotator which label to choose, only to help them reason. That is the right default for a study measuring human judgement and the wrong one for a study measuring how people use an assistant. Replace it with a system_prompt block holding a template string, which is str.formated with task_name, task_description, annotation_labels, instance_text and instance_id. An unknown placeholder falls back to the default prompt with a warning rather than failing.

The blocks past `ai_support`¶

ai_support puts a button beside a scheme. Three other blocks put a model in the loop at other points. On 2.11.1 Potato documents their sub-keys (25 under active_learning, 23 under icl_labeling, 2 under arena), and config-keys-nested.md lists them. What follows is what I set and what the server then said it had.

active_learning reorders the queue by what a classifier finds hard, and needs assignment_strategy: active_learning to have any effect on assignment. classifier.name and vectorizer.name take a short name or an import path:

assignment_strategy: active_learning
active_learning:
  enabled: true
  schema_names: ["stance"]
  min_instances_for_training: 5      # refuses to fit below this
  update_frequency: 2                # refit every N new annotations
  query_strategy: uncertainty
  classifier: {name: sklearn.linear_model.LogisticRegression}
  vectorizer: {name: sklearn.feature_extraction.text.TfidfVectorizer}
  model_persistence: {enabled: true} # a dict, not a boolean

{name: logistic} and {name: tfidf} validate and train the same models as the paths above. The short classifier names are logistic, random_forest and svm, with a few spellings of each; the vectorizer ones are tfidf, count (or bow) and sentence-transformers (or embeddings), and from 2.11.1 image (or clip) for items that are image files. On 46 coloured images under media_directory, referenced as /media/..., {name: image} trained to 0.978 accuracy and warned about the two remote URLs it could not read. A name Potato cannot build fails validation and stops the boot, with the full list in the message: active_learning.classifier.name: 'logreg' is not a classifier Potato can build. Use a short name (logistic, logistic_regression, ...).

Two log lines tell you it is working, and they say different things. The first is Trained classifier for schema <name> with N instances, accuracy: ..., which proves a classifier was fitted. The second names the strategy, because a fitted classifier and a served ordering are separate questions:

Reordered 2 instances (2 ranked, 0 exploration); assignment_strategy:
active_learning, so this is the order annotators get

Reordered 2 instances (2 ranked, 0 exploration); assignment_strategy is
'fixed_order', NOT active_learning, so this ordering is computed and never
served

/admin/active-learning/stats carries the same fact as assignment_strategy and ordering_is_served. Read one of those two rather than the training line: active_learning.enabled: true with the wrong strategy trains a classifier whose ordering reaches nobody, and until this pair existed the two cases were indistinguishable in the log, in the admin report and in --strict.

icl_labeling builds few-shot prompts out of annotations you already have. Its sub-keys are three levels deep. On 2.11.1 --strict refuses a name it does not know at any level, and a real sub-key at the wrong level too: min_agreement_threshold written directly under icl_labeling fails with Unrecognized config key 'icl_labeling.min_agreement_threshold'. The layout:

icl_labeling:
  enabled: true
  example_selection:
    min_agreement_threshold: 0.5
    min_annotators_per_instance: 2   # the default, and the usual reason for zero
    max_examples_per_schema: 6
    refresh_interval_seconds: 60
  llm_labeling:
    trigger_threshold: 3
    batch_size: 5
    batch_interval_seconds: 90
  verification: {enabled: true, sample_rate: 0.2}
  persistence: {predictions_file: icl_predictions.json}

It needs two annotators who agree on an item before that item can become an example. On a one-annotator study it runs on schedule and reports Refreshed high-confidence examples: 0 examples across 0 schemas forever. Prove the keys landed by reading example_refresh= in the startup line back: it echoes refresh_interval_seconds, so if it still says 300 you are in the defaults.

arena fans one prompt out to several models and keeps a leaderboard. Each entry is endpoint_type plus model, with optional label, base_url and temperature; the page is /admin/arena and it answers with per-model latency, an error field, and Elo, Bradley-Terry and win-rate-interval columns once somebody picks a winner. The same server can appear twice under different labels, which is the cheap way to compare decoding settings rather than models.

Verifying one¶

validate --strict will pass a config whose assistants never appear. Boot it and drive it instead:

Read the boot log for the endpoint's own line. What you want names the class that was actually constructed, and agrees with what you configured:

OpenAI Vision client initialized with model: <model> at http://<host>/v1
AI endpoint ready (OpenAIVisionEndpoint), vision: True

Confirm the buttons exist. No buttons means include.all, nine times out of ten. The container renders as <div class="ai-help none"> when there are none.
Click one and read the network response, not the tooltip. The tooltip says "No hint available" for a truncated answer, a wrongly-shaped answer and a real failure alike. /api/get_ai_suggestion shows you which:

{"hint":"...","suggestive_choice":"negative"}   rendered
{"response":"### Annotation Guidance..."}       unconstrained; nothing renders
{"label":...,"reasoning":...}                   truncated: raise max_tokens

For Keyword, look at the item text rather than the tooltip: it draws .ai-keyword-overlay spans titled <label>: "<keyword>".

Check the cost surface if the study is long. ai_budget.cap_usd refuses a run projected to cross it, and /admin/api/ai-cost reports spend. Both are aimed at metered providers; against your own GPU they measure tokens rather than money.

The cap only binds a model Potato has a price for. Prices come from a table of 59 rows compiled into potato/ai/cost.py, matched by the longest substring of the model name. That gives five states, and they are not equally loud:

State	Cap	Log
Its own row	binds	silent
A dated snapshot of a row (`<model>-20260901`)	binds	silent
Matched to a shorter row, so priced as a relative	binds, on that rate	warns, and names `ai_budget.prices`
No row contains it	does not bind, run proceeds	warns
Local endpoint	binds, at zero	silent

Measured on 2026-09-07 against Potato at 1f8dab21, 500 items of about 100 words: an unpriced model projected 107,500 tokens and no dollar figure, and ran through a cap_usd: 0.10 untouched --

ai_budget.cap_usd is set but 'gemini-3.0-pro' has no price on record, so
this run cannot be checked against it. Projected 107,500 tokens.

The third row of that table is the one that used to be silent, and it is the more dangerous of the two: an absent price gives you no number, while a price taken from a shorter row gives you a confident one. It runs in both directions -- an unlisted new generation picks up an older, cheaper rate and the cap lets spend through; an unlisted cheap variant picks up its expensive parent's rate and the cap refuses runs you could afford.

Fix it in the config rather than waiting for a release. ai_budget.prices is merged over the built-in table by the same matching rule, so a family name or a dated snapshot both work:

ai_budget:
  cap_usd: 0.10
  prices:
    gemini-3.0-pro: [1.0, 8.0]   # USD per million tokens, input then output

Measured on the same run: without it the batch above was unpriced and ran; with it the batch was projected at $0.46 and refused. The warning stops too, because it was only ever saying the cap had nothing to bind.

No rate is written down here, and none should be: the next repricing makes it wrong without making it look wrong. Ask at the moment you need it:

python <skill-dir>/scripts/model_prices.py config.yaml

It finds every model the config names, including one that lives only in ai_config_file, fetches a catalogue when it runs, and puts the live rate next to what Potato would charge. It names the row the charge came from, whether that row was the model's own or a relative's, and whether the number came from your ai_budget.prices rather than the table. Run it before choosing a cap_usd, not after. Self-hosted endpoints are skipped rather than reported as unpriced, because zero is already the right answer for them. With no network it says so and falls back, which is a different sentence from "that model is not listed"; do not read one as the other.

A disagreement it reports is a question, not a verdict. The catalogue is a router's published rate, which is not always the vendor's list price. Measured on 2026-09-07: OpenAI lists gpt-5.6-sol at $4/$20 per million and the catalogue publishes $2/$10, while gpt-5.6-terra and gpt-5.6-luna agree exactly across both. Potato was right about sol. Check the vendor's own page before you change a number on the strength of this.

Watch /admin/api/ai-cost as well. The cap is checked against a projection, and the projection is the only thing the running total is made of.

cache_config.disk_cache is worth turning on for anything with more than a few dozen items: without it, the same item re-queries the model on every page view. cache_config.prefetch.warm_up_page_count generates ahead so the first annotator is not waiting on the first token.

What I have and have not verified¶

Run against a Potato checkout at v2.8.2 plus the endpoint fixes that followed it (bullets naming 2.9.0 or 2.9.1 were run on that release, except where a bullet says "from the source"), and a vLLM 0.24.0 server on a 12B text+vision model, in headless Chromium, reading the response body and the rendered DOM:

the ai_config nesting, and what --strict says when you get it wrong
include.all deciding whether any button exists
ai_config_file merging flat, and a missing one failing --strict
all three text assistants on vllm, openai and openai_vision, including the keyword overlays
Hint on an image_annotation task through openai_vision, including the model reading the photograph correctly and suggesting a label
api_key being optional, in the validator and the endpoint, when base_url is set
text_key deciding what the model receives, both ways round, and the capability filter dropping Keyword for an image item and keeping it for a text one
ai_support.image_key resolving the picture on an item that carries text and an image, and the answer changing when it is set
chat_support end to end against the same server: the sidebar, a reply, the ui keys, and a custom system_prompt.template reaching the model
on a build before 2.9.0, the same chat question at max_tokens 400, 800 and 2000, reading each reply back from /api/chat/send and each server log: cut at a heading, cut before its closing sentence, complete — no truncation warning in any of the three
on 2.9.0, the vllm and openai chat methods against the same server at max_tokens 40 and 2000, called directly rather than through /api/chat/send: the this chat reply warning at 40 on both, nothing at 2000
on 2.9.1, the openai_vision, vllm and openai chat methods called the same way: the this chat reply warning at 40 on all three, nothing at 2000
which endpoint methods call the truncation check, from the source: on 2.9.0, query on all ten text and vision endpoints, query_with_image on the three vision ones, and chat_query on anthropic, ollama, openai and vllm; on 2.9.1, a chat_query of its own on all ten, each logging this chat reply
on 2.9.0, from the source, chat_manager sending the browser only the reply's text
on 2.9.0, the vision loader against a media_directory holding one PNG: cat.png, media/cat.png and /media/cat.png read off disk, http://localhost:8000/media/cat.png and a data: URI of the same PNG refused, and http://127.0.0.1:8000/media/cat.png sent through resolve_vision_image logging the TEXT ONLY warning; from the source, get_ai_help returning the image_attached stamp to the route unchanged; from Potato's history, the disk read arriving in v2.8.2-29-g39173a45, and the warning and the stamp later, in 2b24a056
judge_alignment.inline rendering a judge's suggestion beside the item, and /admin/judge-alignment reporting agreement against the human labels
active_learning training and reordering a twenty-item queue, and the dotted sklearn paths above being the names it accepts
active_learning retraining as annotations arrive (six fits across 24 items at update_frequency: 2) and the reordered pool actually reaching the annotator — and the same log and the same /admin/active-learning/stats appearing on a fixed_order arm where it reaches nobody
llm_confidence assigning at random, on 24 items, with a fitted classifier available to it
icl_labeling collecting examples once a second annotator agreed — 0 across 0 on one annotator, 6 across 2 schemas on two
arena running two models against one prompt and recording a preference
the price table against a live catalogue, row by row, on 2026-09-07: 52 of the 59 rows name a model the catalogue still lists, and 51 of those agree with it to the cent. The one that does not is gpt-5.6-sol, where the vendor and the router publish different rates and the vendor's page backs the table. The 7 unresolvable rows are models the router no longer carries (gemini-1.5-*, gpt-3.5-turbo-1106, both claude-haiku-3-5 spellings), never carried (claude-mythos-5), or carried only as a preview (gemini-3.1-pro). The table is accurate where it is confirmable; what it is not is complete, and an absent model is the case ai_budget.prices exists for
cost.record_spend having one caller outside the tests (server_utils/judge_alignment.py), which passes the projection with estimated=True. The estimated column anticipates measured figures and nothing writes one, so the running total is projections summing projections and cannot be checked against an invoice

On 2.8.2 itself none of the three endpoint types worked against a local server: vllm sent a parameter vLLM ignores, openai's correct answer was delivered to the browser labelled as an error, openai_vision never sent the schema, and the 100-token default truncated two of the three assistants. All four fail the same way — the model answers, the tokens are spent, and the annotator reads "No hint available" — so if you are on 2.8.2 or earlier, read the response body before you conclude anything about your config.

Not run, and not claimed: every commercial provider; ollama and the rest of the endpoint list; icl_labeling, judge_calibration, arena and solo_mode, all of which take the same endpoint config and so inherit whatever is true of it above.