Machine-Readable Specs¶

Potato publishes three generated, machine-checkable contracts alongside the prose documentation. All are produced from the running code rather than maintained by hand, so they describe what Potato actually does today.

If you are pointing a coding agent (Claude Code, Codex, Cursor) at Potato, these files plus llms.txt are the highest-value things to give it.

Artifact	What it describes	URL
Config JSON Schema	Every valid key, annotation type, and display type in a task `config.yaml`	`schemas/potato-config.schema.json`
OpenAPI 3.1 spec	Every HTTP endpoint the server registers	`api-reference/openapi.json`
Examples catalog	Every example project, and what each one demonstrates	`schemas/potato-examples.manifest.json`
`llms.txt`	Curated index of the documentation	`llms.txt`
`llms-full.txt`	The entire documentation set in one file	`llms-full.txt`

Config JSON Schema¶

Validates a task config.yaml before you ever start the server. It is generated from the same registries the server validates against, so a newly registered annotation type appears in the schema immediately.

It currently covers 165 top-level config keys, 61 annotation types, and 24 display types.

Editor validation¶

Every config in examples/ carries a modeline that switches on live validation in VS Code (with the YAML extension), JetBrains IDEs, Zed, and Helix:

# yaml-language-server: $schema=https://potatoannotator.readthedocs.io/en/latest/schemas/potato-config.schema.json
annotation_task_name: My Task
task_dir: .

Add that line to your own configs to get autocomplete and inline errors.

Validating in CI¶

pip install check-jsonschema
check-jsonschema --schemafile \
  https://potatoannotator.readthedocs.io/en/latest/schemas/potato-config.schema.json \
  path/to/config.yaml

The schema also ships inside the installed package, so it resolves offline:

import json, importlib.resources as res
schema = json.loads(
    res.files("potato").joinpath("schemas/potato-config.schema.json").read_text()
)

What it catches¶

Typo'd or hallucinated annotation types (radioo, sentiment), missing per-type required fields (a constant_sum scheme without labels), unknown display types, invalid assignment_strategy values, a missing item_properties.text_key, and configs with no data source at all.

Deliberately permissive about extra keys

additionalProperties is true throughout. Potato only warns about unrecognized config keys, so a schema that rejected them would be stricter than the server and would flag working configs as invalid.

OpenAPI 3.1 spec¶

An index of Potato's HTTP surface: 449 paths / 486 operations, generated by walking the live Flask url_map.

Two extensions carry information a plain OpenAPI document cannot:

x-potato-auth — how the operation is guarded. Decorator names (admin_required, api_login_required, same_origin_required, login_required, require_permission) where a decorator is used, and inline: markers where the check is written into the view body instead: inline:admin_api_key, inline:admin_access, inline:adjudicator, inline:session_username, inline:debug_only. Markers propagate one level through helper calls, so a view that delegates its check to _context() or to admin_dashboard.<method>() is recorded as guarded.
x-potato-requires-config — set when the endpoint only exists if a config flag is on. /admin/arena, for example, is registered only under arena.enabled: true. About a quarter of all operations are config-gated.

Example¶

# Every admin-only endpoint
jq -r '.paths | to_entries[] | .key as $p | .value | to_entries[]
       | select(.value["x-potato-auth"] // [] | index("admin_required"))
       | "\(.key|ascii_upcase) \($p)"' openapi.json

# Everything that needs datasets.enabled
jq -r '.paths | to_entries[] | .key as $p | .value | to_entries[]
       | select(.value["x-potato-requires-config"] // "" | test("datasets"))
       | "\(.key|ascii_upcase) \($p)"' openapi.json

Scope¶

This spec is an authoritative index of what exists, not a payload contract — request and response bodies are not modelled. For narrative descriptions of the main endpoints and their payloads, see the API Reference.

Most endpoints are session-authenticated: POST /auth with username and password as form data, then reuse the session cookie.

Examples catalog¶

A derived index of the 215 example projects in the repository, across 15 categories. Each entry records the annotation types and display types the config uses, the subsystems it switches on, whether it ships data and a README, and the command that runs it.

examples/README.md groups examples by category for browsing. This one is searchable by what a config actually contains, which is what you need when picking one to copy.

# Every example using best-worst scaling
jq -r '.examples[] | select(.annotation_types | index("bws")) | .dir' \
  potato-examples.manifest.json

# Examples that demonstrate gold-standard quality control
jq -r '.examples[] | select(.config_keys | index("gold_standards")) | .run' \
  potato-examples.manifest.json

From Python, without a checkout:

from potato.server_utils.examples_manifest import search_examples
search_examples(annotation_type="span", display_type="image")

examples/ is not part of the wheel, so the packaged copy at potato/schemas/potato-examples.manifest.json is the only record of the example tree an installed Potato has.

Regenerating¶

All five artifacts are generated, and CI fails if a checked-in copy drifts from the code:

python scripts/generate_config_schema.py     # config JSON Schema
python scripts/generate_openapi.py           # OpenAPI spec
python scripts/generate_examples_manifest.py # examples catalog
python scripts/generate_llms_full.py         # llms-full.txt

# Verify without writing (what CI runs)
python scripts/generate_config_schema.py --check
python scripts/generate_openapi.py --check
python scripts/generate_examples_manifest.py --check
python scripts/generate_llms_full.py --check

Run them after adding an annotation type, a display type, a config key, or a route. See Testing for the drift tests that enforce this.