Skip to content

Adjudication Mode

Overview

Adjudication is a quality assurance workflow in which designated reviewers (adjudicators) resolve disagreements between annotators and produce gold-standard final labels. In any multi-annotator project some items will be labeled differently by different people, whether because the text is ambiguous, the guidelines have a gap, or an annotator was wrong. A subject-matter expert reviews those disagreements, reads each annotator's response alongside timing data and agreement scores, and makes the final call.

Adjudication in Potato is implemented as a parallel workflow, not a phase. Adjudicators access a dedicated /adjudicate route with its own interface, separate from the standard annotation flow. That keeps the phase progression (consent, instructions, training, annotation, post-study) untouched and gives adjudicators an interface built for reviewing conflicts.

When to Use Adjudication

  • You have multiple annotators labeling the same items and need to produce a single gold-standard label per item.
  • You want a human-in-the-loop process for resolving inter-annotator disagreements rather than relying solely on majority vote.
  • You need provenance tracking that records whether each final label came from unanimous agreement or an adjudicator's decision.
  • You want to categorize sources of disagreement (ambiguous text, guideline gaps, annotator error) to improve your annotation guidelines over time.

Configuration

Adjudication is configured through the adjudication section of your project's YAML config file. Below is a complete example with all available options and their descriptions.

# Adjudication configuration
adjudication:
  # Enable or disable adjudication mode.
  # Type: bool
  # Default: false
  enabled: true

  # List of usernames authorized to access the adjudication interface.
  # These users can log in and navigate to /adjudicate. Regular annotators
  # cannot access this route.
  # Type: list of strings
  adjudicator_users:
    - "adjudicator"
    - "senior_annotator"

  # Minimum number of annotations an item must have before it becomes
  # eligible for adjudication. Items with fewer annotations are excluded
  # from the queue.
  # Type: int
  # Default: 2
  min_annotations: 2

  # Agreement threshold (0.0 to 1.0). Items with overall agreement at or
  # above this threshold are considered resolved and excluded from the
  # adjudication queue (unless show_all_items is true). Lower values mean
  # fewer items need adjudication; higher values are more conservative.
  # Type: float
  # Default: 0.75
  agreement_threshold: 0.75

  # When true, all items that meet the min_annotations criterion appear
  # in the queue, including items where annotators unanimously agree.
  # Useful for auditing or when you want adjudicators to verify even
  # unanimous decisions.
  # Type: bool
  # Default: false
  show_all_items: false

  # Whether to display annotator usernames in the adjudication interface.
  # When false, annotations are shown as "Annotator" without identifying
  # information, which can reduce bias.
  # Type: bool
  # Default: true
  show_annotator_names: true

  # Whether to display per-annotator timing data (how long each annotator
  # spent on the item). Helps adjudicators identify rushed annotations.
  # Type: bool
  # Default: true
  show_timing_data: true

  # Whether to display per-schema agreement scores in the interface.
  # Type: bool
  # Default: true
  show_agreement_scores: true

  # Time threshold in milliseconds. If an annotator spent less than this
  # amount on an item, their annotation is flagged with a warning icon
  # to alert the adjudicator that the decision may have been rushed.
  # Set to 0 to disable the warning.
  # Type: int
  # Default: 2000
  fast_decision_warning_ms: 2000

  # Whether the adjudicator is required to select a confidence level
  # (high, medium, low) for each decision.
  # Type: bool
  # Default: true
  require_confidence: true

  # Whether the adjudicator must provide notes when overriding all
  # annotators' choices (i.e., when the final decision does not match
  # any annotator's response).
  # Type: bool
  # Default: false
  require_notes_on_override: false

  # List of error category labels that adjudicators can assign to
  # classify the source of disagreement. Customize these to match your
  # project's needs.
  # Type: list of strings
  # Default: [ambiguous_text, guideline_gap, annotator_error,
  #           edge_case, subjective_disagreement, other]
  error_taxonomy:
    - "ambiguous_text"
    - "guideline_gap"
    - "annotator_error"
    - "edge_case"
    - "subjective_disagreement"
    - "other"

  # Similarity-based item grouping (not yet available -- reserved for
  # a future release). When enabled, similar items would be shown
  # alongside the current item to help adjudicators maintain consistency.
  # Type: object
  similarity:
    enabled: false
    # model: "all-MiniLM-L6-v2"
    # top_k: 5
    # precompute_on_start: true

  # Subdirectory name within the annotation output directory where
  # adjudication decisions are stored.
  # Type: string
  # Default: "adjudication"
  output_subdir: "adjudication"

Required Supporting Configuration

Adjudication requires that multiple annotators can be assigned to the same item. Make sure your main config includes:

# Allow multiple annotators per item (must be >= min_annotations)
max_annotations_per_item: 3

The three stages

The adjudication workflow consists of three stages: annotation, adjudication, and export.

Stage 1: Annotation

Regular annotators complete their work through the standard /annotate interface. As annotations accumulate, the adjudication system monitors each item's annotation count and inter-annotator agreement.

Stage 2: Queue Building and Adjudication

When an adjudicator logs in and navigates to /adjudicate, the system builds the adjudication queue by scanning all items. An item enters the queue when:

  1. It has at least min_annotations completed annotations (excluding adjudicator users).
  2. Its overall agreement score falls below agreement_threshold (unless show_all_items is true), or it has no score at all.

An item has no score when none of its schemas can be compared: a span-only item, or one where every schema was answered by a single annotator. Such items are queued with a no score badge, ahead of the scored ones: without a score, nothing says their annotators agreed.

Agreement is computed as pairwise percentage agreement: for each annotation schema, the system counts how many annotator pairs chose the same label and divides by the total number of pairs. The overall agreement is the mean across all schemas.

The queue is sorted with pending items first (lowest agreement first), followed by in-progress items, and finally completed items. Adjudicators therefore reach the most contentious items first.

Stage 3: Export

After adjudication is complete (or at any point during the process), you can generate a final dataset using the export CLI. The export merges three categories of items:

  • Unanimous items: All annotators agreed. The shared label is used directly.
  • Adjudicated items: An adjudicator reviewed the disagreement and submitted a decision. The adjudicator's decision is used.
  • Unresolved items: Annotators disagreed but no adjudication decision has been submitted yet. These are excluded by default but can be included with a flag.

Each item in the exported dataset includes a source field indicating its provenance ("unanimous", "adjudicated", or "unresolved").

Data Flow Diagram

Annotators                    Adjudicator                   Export
    |                              |                           |
    |-- annotate items -->         |                           |
    |   (via /annotate)            |                           |
    |                              |                           |
    |   [items reach min_annotations and low agreement]        |
    |                              |                           |
    |                    view queue at /adjudicate              |
    |                    review annotations + timing            |
    |                    submit decisions                       |
    |                              |                           |
    |                              |-- decisions saved -->     |
    |                              |   (annotation_output/     |
    |                              |    adjudication/           |
    |                              |    decisions.json)         |
    |                              |                           |
    |                              |           python -m potato.adjudication_export
    |                              |                    merges unanimous + adjudicated
    |                              |                    outputs final dataset

Setting Up

Follow these steps to add adjudication to an existing annotation project.

Step 1: Configure Multiple Annotators Per Item

Ensure your config allows multiple annotators to label the same items:

max_annotations_per_item: 3

Step 2: Add the Adjudication Section

Add the adjudication block to your config file. At minimum:

adjudication:
  enabled: true
  adjudicator_users:
    - "adjudicator"
  min_annotations: 2
  agreement_threshold: 0.75

Step 3: Create Adjudicator Accounts

Adjudicator usernames listed in adjudicator_users are standard user accounts. They log in through the same login page as regular annotators. The system checks the username against the adjudicator_users list to grant access to the /adjudicate route.

If your project uses password authentication, adjudicators need valid credentials. If it uses standard (passwordless) login, they simply enter their username.

Step 4: Run Annotations

Start the server and have annotators complete their work:

python potato/flask_server.py start config.yaml -p 8000

Annotators use the standard /annotate interface. Adjudicators should wait until a sufficient number of items have been annotated before beginning adjudication.

Step 5: Begin Adjudication

Once annotations are underway, an adjudicator logs in and navigates to:

http://localhost:8000/adjudicate

The adjudication queue automatically populates with items that meet the configured criteria.

Step 6: Export the Final Dataset

After adjudication is complete, generate the merged dataset:

python -m potato.adjudication_export --config config.yaml --output final_dataset.jsonl

Using the Adjudication Interface

The adjudication interface is a dedicated page with three main areas: the sidebar queue, the main content area, and the bottom navigation bar.

The left sidebar displays the adjudication queue as a scrollable list. Each item shows:

  • The instance ID.
  • The number of annotators who labeled it.
  • A color-coded agreement badge (red for low, amber for medium, green for high).

Items are sorted by status (pending first) and then by agreement score (lowest first). Three filter buttons at the top of the sidebar allow switching between views:

  • Pending -- Items that have not yet been adjudicated (default view).
  • All -- All items in the queue regardless of status.
  • Done -- Only items that have been adjudicated.

A progress bar at the top of the sidebar shows overall completion percentage.

Main Content Area

When you select an item from the queue, the main area displays:

  1. Item Header -- The instance ID, agreement score badge, and annotator count.

  2. The Item -- Rendered through the project's instance_display when one is configured, so the adjudicator sees exactly what the annotators saw, images included. Projects without instance_display fall back to listing the item's data fields with their keys as labels; fields Potato writes for itself, such as displayed_text, are not shown.

  3. Annotator Responses -- For each annotation schema, a grouped panel shows what each annotator selected. Each annotator's response appears in a card with:

  4. The annotator's username (or "Annotator" if show_annotator_names is false).
  5. Their selected label(s) or text response.
  6. Time spent on the item (if show_timing_data is true), with a warning icon if the time was below fast_decision_warning_ms.
  7. A per-schema agreement badge (if show_agreement_scores is true).

  8. Quality flags, where any apply. Each names its own scope: "many edits on this item" is about this item, "low agreement across all items" is about the annotator's record across the whole task. The second one sits beside a per-schema agreement percentage and is not about it. An annotator can carry it on a scheme everyone agreed on. Hover a flag for the number behind it.

  9. Your Decision -- For each schema, a compact decision form is rendered based on the annotation type:

  10. Radio/single-choice: Options listed vertically with radio buttons. Colored annotator chips appear next to each option showing which annotators chose it. Click an option or an annotator chip to select it.
  11. Multiselect/checkbox: Similar to radio but with checkboxes for multiple selection.
  12. Likert/slider: A track showing dots for each annotator's rating, plus a slider for the adjudicator's decision. Click an annotator dot to adopt their value.
  13. Text/number: Each annotator's text response is shown as a clickable card. Click a card to adopt its text into your response textarea, or type your own.
  14. Everything else: A list of the answers annotators actually gave, one radio each. Pick one and it becomes the decision, in the same shape and value space as the answers it resolves. This covers the composite widgets, whose answers cannot be re-elicited from a generic input: constant_sum, soft_label, hierarchical_multiselect, ranking and card_sort.

A likert that declares a labels: list renders as a radio group here, not a slider, because that is how the annotators answered it. A slider would store 3 against their "Good".

  1. Decision Metadata -- Below the schema forms:
  2. Confidence (if require_confidence is true): A dropdown to select high, medium, or low confidence.
  3. Error Type: Clickable tags from the error_taxonomy list. Select one or more to classify the source of disagreement.
  4. Notes: A free-text area for optional notes about the decision.
  5. Flag for Guideline Update: A checkbox to flag items that reveal a gap or ambiguity in the annotation guidelines, with an optional text field to describe the issue.

Bottom Navigation Bar

The navigation bar has three actions:

  • Previous -- Go to the previous item in the queue.
  • Skip -- Mark the current item as skipped and move to the next.
  • Submit & Next -- Submit the decision and advance to the next item.

Keyboard Shortcuts

The following keyboard shortcuts are available when focus is not in an input field, textarea, or dropdown:

Key Action
n or Right Arrow Navigate to the next item in the queue
p or Left Arrow Navigate to the previous item in the queue
Ctrl+Enter Submit the current decision and advance to the next item

These shortcuts are disabled when you are typing in a textarea (such as notes or text-based annotation decisions) or interacting with a select dropdown.

Getting the decisions out

potato export --format adjudication writes the decisions themselves: adjudicated.csv, one row per (item, schema) with the value, its source and the adjudicator's confidence, plus adjudication_log.jsonl with the notes, error taxonomy and timings. It runs from the same place as every other format, including the admin export API. See Export formats.

The CLI below merges those decisions with the items annotators already agreed on, to make one final dataset. Reach for it if you want that merge. The exporter gives you the decisions on their own.

The csv, jsonl and parquet formats carry neither. They read per-annotator state, so an export made after adjudication ships the disagreements and not the resolution.

Export CLI

The export CLI generates a final dataset by merging unanimous agreements and adjudication decisions. It reads the project's annotation output directory, loads all user annotations and adjudication decisions, and produces a single output file.

Basic Usage

# Export as JSONL (default format, one JSON object per line)
python -m potato.adjudication_export --config config.yaml --output final_dataset.jsonl

# Export as CSV
python -m potato.adjudication_export --config config.yaml --output final_dataset.csv --format csv

# Export as JSON (single JSON array)
python -m potato.adjudication_export --config config.yaml --output final_dataset.json --format json

Including Unresolved Items

By default, items where annotators disagreed but no adjudication decision has been submitted are excluded from the output. To include them:

python -m potato.adjudication_export --config config.yaml --output final_dataset.jsonl --include-unresolved

Verbose Output

For detailed logging during the export process:

python -m potato.adjudication_export --config config.yaml --output final_dataset.jsonl --verbose

Command Reference

python -m potato.adjudication_export [OPTIONS]

Required:
  --config CONFIG       Path to the Potato config YAML file
  --output OUTPUT       Output file path

Optional:
  --format {jsonl,json,csv}
                        Output format (default: jsonl)
  --include-unresolved  Include items without adjudication or consensus
  --verbose, -v         Verbose output with debug logging

Output Format

Each item in the exported dataset includes the following fields:

Field Description
instance_id The unique identifier for the item
item_data The original item data (JSONL/JSON only)
labels The final label decisions (keyed by schema name)
source Provenance: "unanimous", "adjudicated", or "unresolved"
adjudicator Username of the adjudicator (adjudicated items only)
confidence Adjudicator's confidence level (adjudicated items only)
provenance Per-schema source mapping (adjudicated items only)
num_annotators Number of annotators (unanimous/unresolved items)

Export Summary

After export, a summary is printed to the console:

Export complete: final_dataset.jsonl
  Total items: 150
  Unanimous:   98
  Adjudicated: 47
  Format:      jsonl

If --include-unresolved is used, unresolved items are included in the count:

Export complete: final_dataset.jsonl
  Total items: 155
  Unanimous:   98
  Adjudicated: 47
  Unresolved:  10
  Format:      jsonl

Decisions Storage

Adjudication decisions are persisted to disk automatically after each submission. They are stored in:

<output_annotation_dir>/<output_subdir>/decisions.json

With the default settings, this is:

annotation_output/adjudication/decisions.json

The file contains a JSON object with a decisions array and a last_updated timestamp. Each decision record includes:

  • instance_id -- The item that was adjudicated.
  • adjudicator_id -- Who made the decision.
  • timestamp -- ISO format timestamp of the decision.
  • label_decisions -- The final labels (keyed by schema name).
  • span_decisions -- Any span annotation decisions.
  • source -- Per-schema provenance (which annotator's response was adopted, or "adjudicator" for a novel decision).
  • source_origins -- Recorded only in a study with declared machine annotators. Per schema, what produced the final answer, as it stood when the decision was made: the adopted annotator ("human", or e.g. "pipeline_a 2.1.0 (tool)"), or, when the adjudicator chose the value themselves, everyone who had given that value, joined with "; ". A value no annotator gave has no entry.
  • confidence -- The adjudicator's confidence rating.
  • notes -- Free-text notes.
  • error_taxonomy -- Selected error classification tags.
  • guideline_update_flag -- Whether the item was flagged for guideline review.
  • guideline_update_notes -- Description of the guideline issue (if flagged).
  • time_spent_ms -- Time the adjudicator spent reviewing the item.

Previously saved decisions are automatically loaded when the server restarts, so adjudication progress is not lost.

Example Project

A complete, self-contained example project is available at:

examples/advanced/adjudication/

This example includes a sentiment analysis task with two annotation schemas (single-choice sentiment and multi-select topics), sample data with 8 review texts, and pre-created annotation output from three simulated annotators.

Running the Example

From the repository root:

python potato/flask_server.py start examples/advanced/adjudication/config.yaml -p 8000

Then:

  1. Open http://localhost:8000 in your browser.
  2. Log in as user_1, user_2, or user_3 to view or add annotations.
  3. Log in as adjudicator to access the adjudication interface at http://localhost:8000/adjudicate.

The example config has show_all_items: true so all items with sufficient annotations appear in the queue, including those with unanimous agreement.

Example Data

The sample data (data/adjudication-example.json) contains items designed to produce a mix of agreement and disagreement:

[
  {
    "id": "item_001",
    "text": "The restaurant had amazing food but terrible service. I waited 45 minutes for my appetizer."
  },
  {
    "id": "item_002",
    "text": "This product exceeded my expectations in every way. Highly recommend!"
  }
]

Items like item_001 with mixed sentiment ("amazing food but terrible service") are likely to produce disagreement between annotators, so they are good candidates for adjudication.

Supported Annotation Types

The adjudication interface renders custom decision forms for the following annotation schema types:

Annotation Type Adjudication Form
radio / select Radio buttons with annotator chips showing who chose each option
multiselect Checkboxes with annotator chips
likert Visual track with annotator dots plus a slider
slider Same as likert (visual track with slider)
text / number Clickable response cards with a textarea for the final answer

Display-only types (pure_display, video) are skipped in the adjudication interface since they do not produce annotation data.

Troubleshooting

"Not authorized as adjudicator" error

The logged-in username must exactly match one of the entries in adjudicator_users. Usernames are case-sensitive. Verify the username in your config:

adjudication:
  adjudicator_users:
    - "adjudicator"    # Must match exactly

Empty adjudication queue

If the queue shows no items, check:

  1. That annotators have completed annotations on items. Each item needs at least min_annotations completed annotations.
  2. That agreement_threshold is not set too low. A threshold of 0.0 would exclude all items unless show_all_items is true.
  3. That annotator usernames are not listed in adjudicator_users. Annotations from adjudicator users are excluded from agreement calculations.

Adjudication is not enabled error (export CLI)

The export CLI requires adjudication.enabled: true in the config file. If you see this error, verify your config.

Decisions not persisting after restart

Decisions are saved to <output_annotation_dir>/<output_subdir>/decisions.json. Ensure the output directory exists and is writable. Check the server logs for any "Failed to save adjudication decisions" error messages.