Skip to content

Potato Documentation

Potato is a free, open-source, self-hosted annotation and agent-evaluation platform for NLP, agentic, and GenAI research. You configure tasks entirely in YAML — no coding — to annotate text, audio, video, images, documents, and AI agent traces, and to run a full agent-evaluation loop (programmatic evaluators, versioned datasets/experiments, automation, CI gating, LLM-as-judge calibration, and a multi-model arena) as a free alternative to LangSmith, LabelBox, and Braintrust.

New here? Start with the FAQ, the Glossary, or the Quick Start.

Looking for guides and tutorials?

This Read the Docs site is the complete, version-matched technical reference — every config option, the full HTTP API, and internals. For guided walkthroughs, use-cases, and higher-level docs, visit potatoannotator.com/docs.

Guides

Role-based guides that walk you through Potato for your specific use case:

Getting Started

For Coding Agents

Generated, machine-checkable specs built from the running code — prefer these over prose when writing a config or calling the API.

Annotation Schemas

Workflow & Quality

  • Annotation Navigation - Navigation tools and status indicators
  • Task Assignment - Assignment strategies and configuration
  • Per-Cohort Schemas - Show different annotation schemes to different annotator cohorts
  • Heterogeneous Coverage - Single-annotator default with a multi-annotator overlap sample, adaptive boost, per-annotator quotas, and full IAA reporting
  • Diversity Ordering - Embedding-based clustering for diverse item presentation
  • Multi-Document Event Annotation - Cross-document events with a 2D corpus map, cluster browser, KNN, and evidence-cited template slots
  • Training Phase - Annotator training and qualification
  • Quality Control - Attention checks and gold standards
  • Boundary Lab - Counterfactual boundary probing: collect contrast sets during ordinary annotation, capture boundary rationales, and get paraphrase-invariance quality control
  • Truth Serum - Surprisingly-popular scoring: gold-free item verdicts that beat majority vote, plus annotator calibration
  • Paper Mode - One command generates a cut-paste LaTeX dataset report: description, distributions, annotator table, IAA, limitations
  • Think-Aloud Mode - Voice rationales with fully-local STT and rule-based spoken-label commitment; verbatim reasoning streams, no LLM
  • Pocket Mode - Mobile-first card-stack annotation PWA: thumb-zone labels, swipe navigation, offline queue with auto-sync
  • Psychometrics - Labels with error bars: live IRT (ability + difficulty, no gold, no LLM), information-gain adaptive routing, codebook-bug detection, and pre-study power analysis
  • Multiplayer Rooms - Live group annotation: blind-vote norming sessions with a real-time agreement meter and conformity logging, adjudication huddles, and expert shadowing
  • Adjudication - Multi-annotator disagreement resolution
  • MACE - Multi-Annotator Competence Estimation via variational inference
  • Iterative BWS - Adaptive Best-Worst Scaling for fine-grained ordinal rankings
  • Category Assignment - Category-based item assignment
  • Surveyflow - Pre/post annotation surveys
  • Annotation Filtering - Filter data based on prior annotations
  • Survey Instruments - 55 pre-built validated psychological instruments
  • QDA Mode - Qualitative data analysis workspace: composes codebook + memos + cases + search with single-coder defaults
  • Memos - Universal annotator notes (instance/span-anchored, private/shared)
  • Search - Universal FTS5 search; admin search + guarded annotator search-and-claim
  • Codebook - Universal mutable code set (nested, opt-in per scheme, on-the-fly add)
  • Cases - Group instances into units of analysis; QDA auto-detect; crosstab integration

Agent Evaluation

  • Coding Agent Annotation - Evaluate agentic coding systems (Claude Code, SWE-Agent, Aider) with diff rendering, PRM annotation, and code review
  • CoT Process Reward (LLM pre-label + verify) - Segment a long chain-of-thought into steps, have an LLM pre-label each step's reward, and have a human verify — fast PRM data collection
  • VLM Grounding & Pointing - Bind referring expressions to regions or points, with an explicit not-present answer, and localize what a caption says that is not there
  • World-Model & Rollout Evaluation - Frame-locked video panels; mark the frame where a generated rollout stops making sense, and measure whether annotators agree on when
  • Agent Traces - Evaluate AI agent traces and trajectories
  • Turn-Level Annotation - Bind any rating/tagging/comment schema per-turn with declarative filters (by speaker, agent, step type, tool)
  • Multi-Agent Discussion - Annotate agent-to-agent discussions/debates with agent identity, addressees, reply threading, and consensus tracking
  • Agent Task Recipes - Ready-to-run configs: debate judging, plan review, negotiation, safety escalation, context-use annotation
  • Session-Level Scoring - Group traces by session_id/thread_id and score whole sessions on a dedicated queue page
  • Sub-Agent Run Tree - Interactive run-hierarchy sidebar for orchestrator traces; bind per-turn schemes to specific sub-agent runs
  • Reviewer Routing + Kanban - Route instances to reviewers with first-match rules; track review states on a kanban board with adjudication handoff
  • Three-Pane Trace Eval - Reasoning | function calls | final answer side-by-side, for continuous evaluation
  • Trajectory Correction - Edit traces into SFT/DPO training data
  • Datasets & Experiments - Versioned eval datasets + experiment runs that score outputs over time
  • Programmatic Evaluators - Trajectory match, tool-use, LLM-judge & heuristic evaluators (Flask-free library)
  • Automation Rules - filter→sample→action rules that route incoming items to queues/datasets/evaluators (production→eval loop)
  • CI Evaluation - pytest plugin to run evals in your suite and gate the build on score-threshold regressions
  • Semantic Curation - embedding search + dynamic slices to find traces by similarity and curate them into datasets
  • LLM-Judge ↔ Human Alignment - Measure & calibrate an LLM judge against human gold (Cohen's κ)
  • Signal-Based Triage Queue - Prioritize the queue by a quality signal (errors / low score first)
  • Hotkey Review Mode - Keyboard-driven review queue with auto-advance on completion
  • Live Agent Interaction - Observe and interact with a live AI agent in real time
  • Model Arena - Compare N models side by side on one prompt; pick the best, build a win-rate leaderboard (provider-agnostic)
  • Web Agent Annotation - Review and create web agent browsing traces

Solo Mode

AI & Intelligence

Authentication & User Management

Crowdsourcing

Administration

Data & Output

UI & Customization

Integrations

Tools & Utilities

Productivity Features

Release Notes

  • v2.7.1 - Transcripts In, Without the Reformatting (21 input formats, sidecar files, potato transcripts CLI)
  • v2.7.0 - Seven New Ways to Annotate (Psychometrics, Multiplayer Rooms, Boundary Lab, Truth Serum, Think-Aloud, Paper Mode, Pocket Mode)
  • v2.6.1 - Agentic Evaluation Suite (evaluators, datasets/experiments, automation, CI gating, tracing SDK, curation, arena)
  • v2.6.0 - QDA Mode, LLM-as-Judge Calibration & Trajectory Editing
  • v2.4.4 - Span Annotation Fixes & UX Improvements
  • v2.4.3 - Coding Agent Annotation, Localization & Stability
  • v2.4.1 - Bug Fixes
  • v2.4.0 - Agent Evaluation, AI-Assisted Annotation & Enterprise Integration
  • v2.3.0 - Solo Mode, Agent Workflows & Security Hardening
  • v2.2.0 - Comprehensive Annotation & Export Platform
  • v2.1.0 - Adjudication & Multi-Modal Annotation
  • v2.0.0 - Backend Refactor

Contributing

Task Documentation
Set up a basic annotation task Quick Start
Choose an annotation type Schema Gallery
Display images/video with radio buttons Instance Display
Show/hide questions based on answers Conditional Logic
Annotate PDFs, Word docs, or code Format Support
Set up SSO/OAuth login SSO Authentication
Reset a user's password Password Management
Use a database for user storage Password Management
Configure for MTurk MTurk Integration
Configure for Prolific Crowdsourcing Guide
Monitor annotation progress Admin Dashboard
Add AI suggestions AI Support
Set up quality control Quality Control
Present items diversely Diversity Ordering
Debug configuration issues Debugging Guide
Create custom visual layouts Layout Customization
Rapidly filter/triage data Triage
Link entities to knowledge bases Entity Linking
Annotate coreference chains Coreference Annotation
Annotate conversation trees Conversation Tree Annotation
Navigate efficiently through items Annotation Navigation
Evaluate AI agent traces Agent Traces
See reasoning, tool calls & answer side-by-side Three-Pane Trace Eval
Edit agent traces into SFT/DPO training data Trajectory Correction
Build versioned eval sets & track scores over time Datasets & Experiments
Score agent outputs programmatically Programmatic Evaluators
Capture agent runs from your own code Tracing SDK
Auto-route incoming traces to queues/datasets/evals Automation Rules
Gate CI on eval-score regressions CI Evaluation
Find traces by similarity / curate slices Semantic Curation
Compare models side by side on a prompt Model Arena
Align/calibrate an LLM judge to human labels Judge Alignment
Auto-label with LLM judges + calibrate blind Judge Calibration
Use Solo Mode for collaborative annotation Solo Mode
Export annotations to Parquet Export Formats
Export to COCO/YOLO/CoNLL Export Formats
Import COCO annotations (RLE masks included) Image Annotation Formats
Push annotations to HuggingFace Hub HuggingFace Export
Deploy on HuggingFace Spaces HuggingFace Spaces
Run behind a /app1/ reverse proxy Reverse Proxy / URL Prefix
Set up webhook notifications Webhooks
Use LLM chat assistant for annotators Chat Support
Evaluate coding agents Coding Agent Annotation
Review web agent traces Web Agent Annotation
Load data from S3/GDrive/Dropbox Remote Data Sources
Use pre-built survey instruments Survey Instruments
Set up multilingual interface Multilingual
Resolve annotator disagreements Adjudication
Browse the REST API API Reference

Example Projects

Ready-to-use example configurations are available in the examples/ directory:

# Run a simple radio button example
python potato/flask_server.py start examples/classification/single-choice/config.yaml -p 8000

# Run a custom layout example (content moderation, dialogue QA, medical review)
python potato/flask_server.py start examples/custom-layouts/content-moderation/config.yaml -p 8000

See the examples directory for more examples, including:

  • classification/ - Classification annotation examples (radio, checkbox, likert, etc.)
  • span/ - Span annotation examples (NER, linking, coreference, etc.)
  • image/ - Image annotation examples
  • video/ - Video annotation examples
  • audio/ - Audio annotation examples
  • advanced/ - Advanced features (conditional logic, quality control, etc.)
  • agent-traces/ - Agent trace evaluation examples (RAG, GUI agents, comparisons)
  • custom-layouts/ - Custom layout examples

Citation

If you use Potato in your research, please cite the Potato 2.0 paper (ACL 2026 System Demonstrations):

@inproceedings{jurgens-etal-2026-potato,
    title = "Potato 2.0: A Comprehensive Annotation Platform with {AI}-in-the-Loop Support",
    author = "Jurgens, David  and Chen, Michael  and Iyer, Lina",
    booktitle = "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-demo.37/",
    pages = "374--386",
}

The original Potato release is described in the Potato 1.0 paper (EMNLP 2022 System Demonstrations, Pei et al., 2022). See the README for both BibTeX entries.