Model Assistance From a Self-Hosted Endpoint¶

Pointing ai_support at your own vLLM or other OpenAI-compatible server. It ships with the potato-tasks skill:

/plugin marketplace add davidjurgens/potato-skill
/plugin install potato-tasks@potato

ai_support puts a row of assistant buttons beside each annotation scheme — Hint, Keyword, Rationale — and fills them from an LLM. This file covers pointing that at a server you run yourself: vLLM, SGLang, LM Studio, llama.cpp, anything speaking the OpenAI chat API. Commercial providers use the same keys and are not covered here, because I have not run one.

The three endpoint types I drove against a live server all work. What is left to get wrong is the config: the keys sit one level deeper than they read, and the switch that decides whether any button appears is off by default and warns about nothing.

Where the keys go¶

Three levels, and only the first is guessable:

ai_support:
  enabled: true
  endpoint_type: openai_vision
  ai_config:
    base_url: "http://your-server:8001/v1"
    model: "google/gemma-4-12B-it-qat-w4a16-ct"
    temperature: 0.1
    include:
      all: true

endpoint_type sits beside ai_config. Everything else — base_url, model, api_key, max_tokens, temperature — goes inside it. Written one level up, where it reads naturally, the keys are dropped and the endpoint falls back to its own default host. validate --strict catches that one:

Unrecognized config key 'ai_support.base_url'. This key will be ignored.

include.all is off by default and nothing warns you. Without it the config validates clean — OK — no issues found — the endpoint connects, the boot log says the endpoint is ready, and the page renders <div class="ai-help none"> with no buttons in it. This is the single most likely reason a correctly configured ai_support shows nothing. ai_config.include.special_include is the per-page, per-scheme alternative, keyed page number then annotation id.

config-keys-nested.md documents these, include.all included. Setting base_url makes api_key optional, on the validator as well as in the endpoint, so a local server needs no placeholder key.

Keeping the endpoint out of the config¶

ai_support.ai_config_file names a second YAML file, merged in at load. It is how the bundled examples keep API keys and lab-internal hostnames out of the committed config, and it is worth using for the same reason.

The file is flat. Its keys go straight into ai_config, and endpoint_type is lifted out to sit beside it:

endpoint_type: openai_vision
base_url: "http://your-server:8001/v1"
model: "google/gemma-4-12B-it-qat-w4a16-ct"

Give that file its own ai_config: block and you get ai_config.ai_config, every real setting is invisible, and the boot log complains about a missing API key, which points at the wrong key entirely.

If the named file does not exist, Potato warns and sets enabled: false. validate --strict fails on that warning, so the check catches a study whose whole point is model assistance running with the model switched off. Ship an ai-config.yaml.example beside the config for whoever clones it, as examples/image/image-vllm-rationale/ does.

Which endpoint_type to use¶

endpoint_type: vllm for a vLLM server, openai for anything else OpenAI-compatible, openai_vision if items include images. Driven against a live vLLM 0.24.0 serving a 12B text+vision model, clicking every assistant in a browser and reading the response body:

`endpoint_type`	Text	Images	Constrains the output with
`vllm`	yes	no	`response_format: json_schema`
`openai`	yes	no	`response_format: json_schema`
`openai_vision`	yes	yes	`json_schema`, falling back to `json_object` then nothing if the server refuses

All three returned correctly shaped JSON for all three assistants, and suggestive_choice highlighted the label in the form.

openai_vision is named for its capability, not its backend: its docstring is explicit that base_url is there for vLLM, SGLang, LM Studio, llama.cpp and LiteLLM, and it appends /v1 if you leave it off. Pick it whenever any item is an image, because it is the only one of the three that sends one.

The full list is openai, openai_vision, anthropic, anthropic_vision, gemini, huggingface, ollama, ollama_vision, openrouter, vllm, plus yolo, sam and sam3 for detection and segmentation. I have run three of them.

max_tokens¶

Defaults to 800, which covers the multi-label formats. Rationale and Keyword both produce one entry per label, so a three-label scheme runs 300-500 tokens; the old default of 100 cut them off mid-object. If you lower it, or a scheme has many labels, the log says so:

The model hit max_tokens (100) before finishing this response, so the reply is
cut off and may parse into the wrong shape. Raise ai_config.max_tokens.

Worth knowing because truncation and a genuinely broken endpoint share a symptom: the salvage step returns whatever key-value pairs completed, so the renderer gets a plausible dict of the wrong type and the tooltip reads "No rationales available" either way.

Images have to be publicly reachable¶

The AI path fetches the image itself, server-side, and refuses any URL resolving to a private, loopback or link-local address. That is deliberate — it is SSRF protection on a URL that comes from your data file — but it means images served by your own Potato instance, or sitting behind media_directory, cannot reach the model. When the fetch is refused the code falls back to a plain text query on the URL string, so you get an answer about a filename.

Annotators see local images fine; only the model cannot. If a study needs both, the images have to be somewhere public.

A separate detail: whether an item counts as an image at all is decided by sniffing the text for an extension (.jpg, .png, .gif, .webp, .bmp), a known image host, or the word "image" in an http URL. An extensionless URL from your own object store is read as text and the vision path never runs.

The three text assistants¶

Three, for text schemes, each with its own expected output shape:

Button	Produces	Where it lands
Hint	Guidance that avoids naming the answer	A tooltip, plus `suggestive_choice` highlighting that label in the form
Keyword	Per-label keywords	Overlays on the item text, labelled — not the tooltip, which stays empty
Rationale	One argument per label, for and against	A tooltip

Which appear is filtered per item by what the endpoint declares it can do, so an image item correctly drops Keyword while a text item keeps it, on the vision endpoints too. The prompts are per annotation type and live in potato/ai/prompt/*.json, one file per type — radio, likert, multiselect, select, slider, number, span, text, plus the image and video ones. A scheme whose type has no file there gets no assistants and the tooltip says the annotation type does not exist in the prompts.

ai_support.ai_config.model_module and annotation_path override the schemas and the prompt files respectively, if the defaults do not fit the study.

Verifying one¶

validate --strict will pass a config whose assistants never appear. Boot it and drive it instead:

Read the boot log for the endpoint's own line. What you want names the class that was actually constructed, and agrees with what you configured:

OpenAI Vision client initialized with model: <model> at http://<host>/v1
AI endpoint ready (OpenAIVisionEndpoint), vision: True

Confirm the buttons exist. No buttons means include.all, nine times out of ten. The container renders as <div class="ai-help none"> when there are none.
Click one and read the network response, not the tooltip. The tooltip says "No hint available" for a truncated answer, a wrongly-shaped answer and a real failure alike. /api/get_ai_suggestion shows you which:

{"hint":"...","suggestive_choice":"negative"}   rendered
{"response":"### Annotation Guidance..."}       unconstrained; nothing renders
{"label":...,"reasoning":...}                   truncated: raise max_tokens

For Keyword, look at the item text rather than the tooltip: it draws .ai-keyword-overlay spans titled <label>: "<keyword>".

Check the cost surface if the study is long. ai_budget.cap_usd refuses a run projected to cross it, and /admin/api/ai-cost reports spend. Both are aimed at metered providers; against your own GPU they measure tokens rather than money.

cache_config.disk_cache is worth turning on for anything with more than a few dozen items: without it, the same item re-queries the model on every page view. cache_config.prefetch.warm_up_page_count generates ahead so the first annotator is not waiting on the first token.

What I have and have not verified¶

Run against a Potato checkout at v2.8.2 plus the endpoint fixes that followed it, and a vLLM 0.24.0 server on a 12B text+vision model, in headless Chromium, reading the response body and the rendered DOM:

the ai_config nesting, and what --strict says when you get it wrong
include.all deciding whether any button exists
ai_config_file merging flat, and a missing one failing --strict
all three text assistants on vllm, openai and openai_vision, including the keyword overlays
Hint on an image_annotation task through openai_vision, including the model reading the photograph correctly and suggesting a label
api_key being optional, in the validator and the endpoint, when base_url is set

On 2.8.2 itself none of the three endpoint types worked against a local server: vllm sent a parameter vLLM ignores, openai's correct answer was delivered to the browser labelled as an error, openai_vision never sent the schema, and the 100-token default truncated two of the three assistants. All four fail the same way — the model answers, the tokens are spent, and the annotator reads "No hint available" — so if you are on 2.8.2 or earlier, read the response body before you conclude anything about your config.

Not run, and not claimed: every commercial provider; ollama and the rest of the endpoint list; chat_support, llm_labeling, icl_labeling, active_learning, judge_alignment, judge_calibration, arena and solo_mode, all of which take the same endpoint config and so inherit whatever is true of it above.