# Model Assistance From a Self-Hosted Endpoint Pointing `ai_support` at your own vLLM or other OpenAI-compatible server. It ships with the `potato-skill` skill: ``` /plugin marketplace add davidjurgens/potato-skill /plugin install potato-skill@potato ``` --- `ai_support` puts a row of assistant buttons beside each annotation scheme — Hint, Keyword, Rationale — and fills them from an LLM. This file covers pointing that at a server you run yourself: vLLM, SGLang, LM Studio, llama.cpp, anything speaking the OpenAI chat API. Commercial providers use the same keys and are not covered here, because I have not run one. The three endpoint types I drove against a live server all work. What is left to get wrong is the config: the keys sit one level deeper than they read, and the switch that decides whether any button appears is off by default and warns about nothing. ## Where the keys go Three levels, and only the first is guessable: ```yaml ai_support: enabled: true endpoint_type: openai_vision ai_config: base_url: "http://your-server:8001/v1" model: "google/gemma-4-12B-it-qat-w4a16-ct" temperature: 0.1 include: all: true ``` `endpoint_type` sits beside `ai_config`. Everything else — `base_url`, `model`, `api_key`, `max_tokens`, `temperature` — goes *inside* it. Written one level up, where it reads naturally, the keys are dropped and the endpoint falls back to its own default host. `validate --strict` catches that one: ``` Unrecognized config key 'ai_support.base_url'. This key will be ignored. ``` **`include.all` is off by default and nothing warns you.** Without it the config validates clean — `OK — no issues found` — the endpoint connects, the boot log says the endpoint is ready, and the page renders `
` with no buttons in it. This is the single most likely reason a correctly configured `ai_support` shows nothing. `ai_config.include.special_include` is the per-page, per-scheme alternative, keyed page number then annotation id. `config-keys-nested.md` documents these, `include.all` included. Setting `base_url` makes `api_key` optional, on the validator as well as in the endpoint, so a local server needs no placeholder key. **One `ai_support` block serves every scheme, and each gets its own buttons and its own answer.** Three schemes on a page — a radio, a multiselect and a free text — gave three separate Hint/Keyword/Rationale rows, and each hint was about that scheme's question: the radio's discussed weighing the complaint against the resolution, the multiselect's walked its own five labels and said which phrases mapped to which, the text one described the summary it wanted. The suggestions landed on the right scheme too, highlighted with a sparkle — `Mixed` on the radio, and `Shipping`, `Build Quality` and `Support` but not `Battery` or `Price` on the multiselect. There is nothing per-scheme to configure. **The model's output is escaped before it reaches the page**, which matters because a hint is generated *from* the item: markup in your corpus comes back through the model. I put `` and `` in an item's text and the hint panel rendered neither — 607 characters of hint, no elements the renderer did not create. The same holds for the rationale panel. Markdown in the output is rendered rather than shown as asterisks; I have driven the escaping half against a live model and taken the markdown half from the fix, because the model did not happen to emit any on the runs I made. One rough edge left: on a narrow scheme the three buttons lose their labels and render as bare icons, so the same row reads differently depending on how wide its scheme box is. ### Keeping the endpoint out of the config `ai_support.ai_config_file` names a second YAML file, merged in at load. It is how the bundled examples keep API keys and lab-internal hostnames out of the committed config, and it is worth using for the same reason. The file is **flat**. Its keys go straight into `ai_config`, and `endpoint_type` is lifted out to sit beside it: ``` endpoint_type: openai_vision base_url: "http://your-server:8001/v1" model: "google/gemma-4-12B-it-qat-w4a16-ct" ``` Give that file its own `ai_config:` block and you get `ai_config.ai_config`, every real setting is invisible, and the boot log complains about a missing API key, which points at the wrong key entirely. If the named file does not exist, Potato warns and sets `enabled: false`. `validate --strict` fails on that warning, so the check catches a study whose whole point is model assistance running with the model switched off. Ship an `ai-config.yaml.example` beside the config for whoever clones it, as `examples/image/image-vllm-rationale/` does. ## Which endpoint_type to use `endpoint_type: vllm` for a vLLM server, `openai` for anything else OpenAI-compatible, `openai_vision` if items include images. Driven against a live vLLM 0.24.0 serving a 12B text+vision model, clicking every assistant in a browser and reading the response body: | `endpoint_type` | Text | Images | Constrains the output with | |---|---|---|---| | `vllm` | yes | no | `response_format: json_schema` | | `openai` | yes | no | `response_format: json_schema` | | `openai_vision` | yes | yes | `json_schema`, falling back to `json_object` then nothing if the server refuses | All three returned correctly shaped JSON for all three assistants, and `suggestive_choice` highlighted the label in the form. `openai_vision` is named for its capability, not its backend: its docstring is explicit that `base_url` is there for vLLM, SGLang, LM Studio, llama.cpp and LiteLLM, and it appends `/v1` if you leave it off. Pick it whenever any item is an image, because it is the only one of the three that sends one. On 2.11.1 the validator accepts `openai`, `openai_vision`, `anthropic`, `anthropic_vision`, `gemini`, `huggingface`, `ollama`, `ollama_vision`, `openrouter` (also spelt `open_router`; 2.11.0 refused both), `vllm`, and `yolo` for detection. Anything else fails with `ai_support.endpoint_type must be one of:` and that list. `sam` and `sam3` have endpoint classes in Potato's code but are refused here. I have run three of them. ## max_tokens Defaults to 800, which covers the multi-label formats. Rationale and Keyword both produce one entry per label, so a three-label scheme runs 300-500 tokens; the old default of 100 cut them off mid-object. Lower it, or give a scheme many labels, and from Potato 2.9.0 the log says so: ``` The model hit max_tokens (100) before finishing this response, so the reply is cut off and may parse into the wrong shape. Raise ai_config.max_tokens. ``` Worth knowing because truncation and a genuinely broken endpoint share a symptom: the salvage step returns whatever key-value pairs completed, so the renderer gets a plausible dict of the wrong type and the tooltip reads "No rationales available" either way. **From 2.9.0, all ten text and vision endpoints warn for the assistants**, on text and image items alike. Their chat methods warn too, with `this chat reply` where an assistant's warning says `this response`: on 2.9.0 only `anthropic`, `ollama`, `openai` and `vllm`, and on all ten from 2.9.1. Before 2.9.0 only `openai`, `openai_vision` and `vllm` warned, and never from chat. On `anthropic`, `anthropic_vision`, `gemini`, `huggingface`, `ollama`, `ollama_vision` and `openrouter` a cut-off reply reached the renderer with nothing in the log and produced the empty tooltip above, which is also what a broken endpoint produces. A quiet log is evidence the reply was complete only on 2.9.0 or later, and for chat on 2.9.0 only on `anthropic`, `ollama`, `openai` or `vllm`. ## Which field the model gets Two keys decide, and only one of them is the obvious one. `item_properties.text_key` names the field that goes to the model on the text path. It is the same key that decides plain-text rendering, span anchoring, training and search (`data-and-access.md`), so it is not free to point somewhere else just to change what the assistants read. `ai_support.image_key` names the field holding the picture, for an item that carries both. `item_properties.image_key` is the same key one level up, and `ai_support.image_key` wins: ```yaml item_properties: id_key: id text_key: body # the ticket text ai_support: enabled: true endpoint_type: openai_vision image_key: screenshot # the field holding the image URL ``` With both keys set the model receives both: the picture, and the text field named by `text_key`, in the same request. Without `image_key` a vision endpoint on a text-keyed item never receives an image, and nothing says so — the endpoint connects, the buttons render and the model answers about the half it was given. **Read the response body to find out what it was given.** The tooltip cannot tell you, and a model answering about the wrong half of the item is fluent about it — the hint reads like a hint, and the label it suggests is highlighted in the form either way. One `/api/get_ai_suggestion` call on one multimodal item settles which field reached the model; nothing in the config does. ## Which images the model can reach The AI path loads the image itself, server-side. From 2.8.2-29 a bare filename, or a `media/...` or `/media/...` path, under `media_directory` is read off disk and sent as base64. Anything else has to be an http(s) URL, and one resolving to a private, loopback or link-local address is refused. That refusal is SSRF protection on a URL that comes from your data file, and it means a full URL to your own Potato instance (`http://localhost:8000/media/cat.png`) never reaches the model, even though the same file written as `/media/cat.png` does. A `data:` URI is refused too: the page displays one (`modalities.md`), and the model never gets it. Annotators see the image in every one of these cases, so checking the page will not catch it. Write media paths in the data as `/media/`, the form the page serves, and the model gets the same file. When the image cannot be loaded, the request goes to the model as text only, and the answer comes back in the same shape as an answer about the picture. From 2.9.0 the log says `so the model is being sent TEXT ONLY`, and by the source the `/api/get_ai_suggestion` response carries `image_attached: false`. Before 2.9.0 nothing said so. A separate detail: whether an item counts as an image at all is decided by sniffing the text for an extension (`.jpg`, `.png`, `.gif`, `.webp`, `.bmp`), a known image host, or the word "image" in an http URL. An extensionless URL from your own object store is read as text and the vision path never runs. ## The three text assistants Three, for text schemes, each with its own expected output shape: | Button | Produces | Where it lands | |---|---|---| | Hint | Guidance that avoids naming the answer | A tooltip, plus `suggestive_choice` highlighting that label in the form | | Keyword | Per-label keywords | Overlays on the item text, labelled — not the tooltip, which stays empty | | Rationale | One argument per label, for and against | A tooltip | Which appear is filtered per item by what the endpoint declares it can do, so an image item correctly drops Keyword while a text item keeps it, on the vision endpoints too. The prompts are per annotation type and live in `potato/ai/prompt/*.json`, one file per type — `radio`, `likert`, `multiselect`, `select`, `slider`, `number`, `span`, `text`, plus the image and video ones. A scheme whose type has no file there gets no assistants and the tooltip says the annotation type does not exist in the prompts. `ai_support.ai_config.model_module` and `annotation_path` override the schemas and the prompt files respectively, if the defaults do not fit the study. ## Letting them ask the model instead `chat_support` is a separate block with the same endpoint shape, and it puts a sidebar beside the form rather than buttons beside each scheme. Reach for it when the researcher says "let them ask a model while annotating" — the assistants answer one question about one scheme, the chat answers whatever the annotator types. ```yaml chat_support: enabled: true endpoint_type: openai ai_config: base_url: "http://your-server:8001/v1" model: "google/gemma-4-12B-it-qat-w4a16-ct" temperature: 0.3 max_tokens: 2000 ui: title: Ask the model sidebar_width: 420 max_history_per_instance: 50 ``` `endpoint_type` and `ai_config` sit at the same level as they do under `ai_support`, and `include.all` is not involved: the sidebar renders whenever the block is enabled. The boot log's line is `Chat endpoint initialized: `. **The sample sets `max_tokens` well above the 800 default, and should.** An assistant returns one entry per label, so the author knows roughly how long the reply will be. Chat length is set by whatever the annotator types. Asked a question covering three labels, this endpoint at 400 stopped on the bare heading `### 3. Unclear` with nothing under it; at 800 it reached the last section but not its closing sentence; at 2000 it finished. Those runs were on a build before 2.9.0, whose chat path never checked the finish reason, and nothing appeared in the log. From 2.9.0 this endpoint logs `before finishing this chat reply`, but the sidebar still shows the reply with nothing to mark it as cut off, because only its text reaches the browser, so the log is the only place a cut-off shows. A cap costs nothing unless the model reaches it. **It is given far more than the assistants are.** The default prompt carries the task name, the task description, every scheme's name and its labels, and the item's `text_key` field. Asked what it had been given, the model listed all eight scheme names in the study. It is *not* given the image, on any endpoint type. On a task whose item is a picture, the chat only ever sees the text field beside it. **The default prompt refuses to name a label.** Its closing line is an instruction not to tell the annotator which label to choose, only to help them reason. That is the right default for a study measuring human judgement and the wrong one for a study measuring how people use an assistant. Replace it with a `system_prompt` block holding a `template` string, which is `str.format`ed with `task_name`, `task_description`, `annotation_labels`, `instance_text` and `instance_id`. An unknown placeholder falls back to the default prompt with a warning rather than failing. ## The blocks past `ai_support` `ai_support` puts a button beside a scheme. Three other blocks put a model in the loop at other points. On 2.11.1 Potato documents their sub-keys (25 under `active_learning`, 23 under `icl_labeling`, 2 under `arena`), and `config-keys-nested.md` lists them. What follows is what I set and what the server then said it had. **`active_learning`** reorders the queue by what a classifier finds hard, and needs `assignment_strategy: active_learning` to have any effect on assignment. `classifier.name` and `vectorizer.name` take a short name or an import path: ```yaml assignment_strategy: active_learning active_learning: enabled: true schema_names: ["stance"] min_instances_for_training: 5 # refuses to fit below this update_frequency: 2 # refit every N new annotations query_strategy: uncertainty classifier: {name: sklearn.linear_model.LogisticRegression} vectorizer: {name: sklearn.feature_extraction.text.TfidfVectorizer} model_persistence: {enabled: true} # a dict, not a boolean ``` `{name: logistic}` and `{name: tfidf}` validate and train the same models as the paths above. The short classifier names are `logistic`, `random_forest` and `svm`, with a few spellings of each; the vectorizer ones are `tfidf`, `count` (or `bow`) and `sentence-transformers` (or `embeddings`), and from 2.11.1 `image` (or `clip`) for items that are image files. On 46 coloured images under `media_directory`, referenced as `/media/...`, `{name: image}` trained to 0.978 accuracy and warned about the two remote URLs it could not read. A name Potato cannot build fails validation and stops the boot, with the full list in the message: `active_learning.classifier.name: 'logreg' is not a classifier Potato can build. Use a short name (logistic, logistic_regression, ...)`. Two log lines tell you it is working, and they say different things. The first is `Trained classifier for schema with N instances, accuracy: ...`, which proves a classifier was fitted. The second names the strategy, because a fitted classifier and a served ordering are separate questions: ``` Reordered 2 instances (2 ranked, 0 exploration); assignment_strategy: active_learning, so this is the order annotators get Reordered 2 instances (2 ranked, 0 exploration); assignment_strategy is 'fixed_order', NOT active_learning, so this ordering is computed and never served ``` `/admin/active-learning/stats` carries the same fact as `assignment_strategy` and `ordering_is_served`. Read one of those two rather than the training line: `active_learning.enabled: true` with the wrong strategy trains a classifier whose ordering reaches nobody, and until this pair existed the two cases were indistinguishable in the log, in the admin report and in `--strict`. **`icl_labeling`** builds few-shot prompts out of annotations you already have. Its sub-keys are **three levels deep**. On 2.11.1 `--strict` refuses a name it does not know at any level, and a real sub-key at the wrong level too: `min_agreement_threshold` written directly under `icl_labeling` fails with `Unrecognized config key 'icl_labeling.min_agreement_threshold'`. The layout: ```yaml icl_labeling: enabled: true example_selection: min_agreement_threshold: 0.5 min_annotators_per_instance: 2 # the default, and the usual reason for zero max_examples_per_schema: 6 refresh_interval_seconds: 60 llm_labeling: trigger_threshold: 3 batch_size: 5 batch_interval_seconds: 90 verification: {enabled: true, sample_rate: 0.2} persistence: {predictions_file: icl_predictions.json} ``` It needs **two annotators who agree** on an item before that item can become an example. On a one-annotator study it runs on schedule and reports `Refreshed high-confidence examples: 0 examples across 0 schemas` forever. Prove the keys landed by reading `example_refresh=` in the startup line back: it echoes `refresh_interval_seconds`, so if it still says 300 you are in the defaults. **`arena`** fans one prompt out to several models and keeps a leaderboard. Each entry is `endpoint_type` plus `model`, with optional `label`, `base_url` and `temperature`; the page is `/admin/arena` and it answers with per-model latency, an error field, and Elo, Bradley-Terry and win-rate-interval columns once somebody picks a winner. The same server can appear twice under different labels, which is the cheap way to compare decoding settings rather than models. ## Verifying one `validate --strict` will pass a config whose assistants never appear. Boot it and drive it instead: 1. **Read the boot log for the endpoint's own line.** What you want names the class that was actually constructed, and agrees with what you configured: ``` OpenAI Vision client initialized with model: at http:///v1 AI endpoint ready (OpenAIVisionEndpoint), vision: True ``` 2. **Confirm the buttons exist.** No buttons means `include.all`, nine times out of ten. The container renders as `
` when there are none. 3. **Click one and read the network response**, not the tooltip. The tooltip says "No hint available" for a truncated answer, a wrongly-shaped answer and a real failure alike. `/api/get_ai_suggestion` shows you which: ``` {"hint":"...","suggestive_choice":"negative"} rendered {"response":"### Annotation Guidance..."} unconstrained; nothing renders {"label":...,"reasoning":...} truncated: raise max_tokens ``` For Keyword, look at the item text rather than the tooltip: it draws `.ai-keyword-overlay` spans titled `