Skip to content

Frequently Asked Questions

Answers to common questions about Potato, annotation, and agent (agentic) evaluation. See the linked guides for detail.

What is Potato?

Potato is a free, open-source, self-hosted annotation and agent-evaluation platform for NLP, agentic, and GenAI research. You configure tasks entirely in YAML — no coding — to annotate text, audio, video, images, documents, and AI agent traces. See Quick Start.

What is agentic annotation?

Agentic annotation is the practice of evaluating AI agent runs — their reasoning steps, tool calls, and final outputs — rather than just labeling static text. Potato renders agent trajectories and lets humans (and LLM judges) rate correctness step by step, mark error spans, edit trajectories, and compare agents. See the Agent Evaluation Guide.

Is Potato a free alternative to LangSmith, LabelBox, or Braintrust?

Yes. Potato is free and self-hosted, and covers the agent-evaluation loop those tools provide — programmatic evaluators, versioned datasets and experiments, automation rules, CI gating, LLM-as-judge calibration, and a multi-model arena — without per-seat or per-trace fees and without sending your data to a SaaS. See the comparison.

How do I evaluate AI agent trajectories?

Import a trace (OpenAI, Anthropic, LangChain, LangGraph, CrewAI, OpenTelemetry, and more), display it with the agent_trace or three-pane eval_trace view, and score it with human schemes or programmatic evaluators (deterministic trajectory match, tool-use correctness, LLM-judge). See Agent Traces and Trajectory Evaluation.

How do I annotate agent traces and tool calls?

Convert traces with python -m potato.trace_converter, then annotate tool calls, reasoning, and observations per step. Tool calls render natively in the coding_trace and eval_trace displays. See Agent Traces.

How do I do LLM-as-judge evaluation and calibration?

Configure an LLM judge, run it over human-labeled items, and measure agreement (Cohen's κ, ECE/Brier). Potato can auto-calibrate the judge by turning human corrections into few-shot examples, and judges categorical, span, and free-text outputs. See Judge Alignment and Judge Calibration.

How do I collect RLHF / DPO / preference data?

Use pairwise or ranking schemes (or the model arena for side-by-side model comparison), then export to SFT/DPO via the trajectory-correction and dataset exporters.

Can I gate CI on evaluation quality?

Yes. The pytest plugin runs your evaluations in CI and fails the build when a metric drops below a threshold (--potato-threshold correct=0.8), so prompt/model regressions are caught on every PR.

How do I capture agent runs from my own code?

Instrument your agent with the potato_trace SDK: decorate functions with @traceable and runs are captured and sent to Potato. An OpenTelemetry exporter is also available.

Does Potato support crowdsourcing (MTurk, Prolific)?

Yes — native MTurk and Prolific integration with platform-specific authentication, plus quality control, training phases, and inter-annotator agreement.

What annotation types does Potato support?

61 schemes: radio/checkbox/Likert, span/NER, spans linking, coreference, ranking, best-worst scaling, pairwise/conjoint comparison, sliders, soft labels, rubric grids, and agent-specific schemes (trajectory eval, process-reward, code review). See the Schema Gallery.

How large a dataset can Potato handle?

Potato holds items in an ID-keyed in-memory index, so lookups are O(1), not file scans — the "un-indexed files" concern does not apply. Memory scales with the number of items (the ML stack is not loaded unless you enable a feature that needs it): the reference 50k-item boot runs in ~5.7 s at ~365 MB RSS, and a benchmark (tests/performance/test_large_dataset_boot.py) guards against regressions. For very large projects, shard work across cohorts and cap workload with per_annotator_quota so annotators only ever load their own queue. See Scaling & Large Datasets for details, including bulk export behavior.

Can different annotators see different questions or have different permissions?

Yes to both. Assign different annotation schemes to different cohorts with Per-Cohort Schemas, and control who can do what with role-based access control (admin, adjudicator, annotator, or custom roles — assigned by user or SSO claim). Dynamic assignment of items to unknown annotators is also built in via batch assignment (cohorts, auto-assignment, restart-persistent pins).

Is my data private?

Yes. Potato is self-hosted — your data stays on your infrastructure. There is no required external service.

How do I install and run Potato?

pip install potato-annotation
python potato/flask_server.py start <config.yaml> -p 8000

See Quick Start and Usage.