Quality Control Features¶

Five quality control features, all optional and independent of each other:

Attention Checks - Verify annotator engagement with known-answer items
Gold Standards - Track accuracy against expert-labeled items
Pre-annotation Support - Pre-fill forms with model predictions
Agreement Metrics - Calculate inter-annotator agreement in real-time
Step-Level QC - Per-step quality control for agent trace evaluation

Attention Checks¶

Attention checks are items with known correct answers that are periodically injected into the annotation flow to verify that annotators are paying attention and not randomly clicking.

Configuration¶

attention_checks:
  enabled: true

  # Path to JSON file containing attention check items
  items_file: "attention_checks.json"

  # How often to inject attention checks (choose one):
  frequency: 10              # Insert one every 10 items
  # OR
  probability: 0.1           # 10% chance per item

  # Optional: fail responses that arrive too fast to have been read
  min_response_time: 3.0     # Fail the check if answered in under 3 seconds

  # Failure handling
  failure_handling:
    warn_threshold: 2        # Show warning after 2 failures
    warn_message: "Please read items carefully before answering."
    block_threshold: 5       # Block user after 5 failures
    block_message: "You have been blocked due to too many incorrect responses."

Attention Check Items File Format¶

Create a JSON file with your attention check items:

[
  {
    "id": "attn_001",
    "text": "Please select 'Positive' for this item to verify you are reading carefully.",
    "expected_answer": {
      "sentiment": "positive"
    }
  },
  {
    "id": "attn_002",
    "text": "This is a test item. The correct answer is 'Negative'. Please select it now.",
    "expected_answer": {
      "sentiment": "negative"
    }
  }
]

Fields: - id (required): Unique identifier for the attention check - text (required): The text to display to annotators - expected_answer (required): Dictionary mapping schema names to expected values. For a multiselect, a single label means "this box and no other"; a list means exactly those boxes, in any order. An annotator who ticks extra boxes fails.

When a check is injected and what happens next¶

Attention check items are loaded at server startup
Based on frequency or probability, checks are injected into the annotation flow
When an annotator submits a response, it's compared to the expected answer
A response faster than min_response_time fails whatever it says. The clock runs from when the server sent the item to when it received the save, and it is measured on the server. The annotator's own client also reports a duration; that number is not what the check reads, and a large disagreement between the two is logged
Failures are tracked per-user, at most one result per annotator per check item, so changing an answer does not add a second result. Re-saves merge rather than replace. The answer is taken from the latest save, the recorded time is the fastest one seen, and a check that failed on speed stays failed — the annotation page saves twice for one answer, on selection and again on Next, and that second save carries the seconds spent looking at an answer already given
The comparison accepts either shape the annotation endpoint takes for a selection: the label in the value, or the label in the key with the form's own on as the value

If the server restarts between sending an item and receiving its answer, it cannot time that answer. The check is skipped for that one response and a line naming the annotator and the item goes to the log, rather than the answer passing quietly. 7. Warnings and blocks are triggered at configured thresholds 8. Once a warning or block has been shown for a failed check, that failure stands: answering the same check again does not change it. An answer changed before any verdict was shown (a misclick corrected on the same page) still counts 9. Moving past a check without answering it fails it 10. A blocked annotator gets no more items. Reloading the annotation page ends their task, and further saves are refused with HTTP 403. On a crowd platform the end page carries the platform's failure code; elsewhere it shows the block message. A block cannot be lifted by re-answering

Consent, instruction and survey pages are never graded as attention checks or gold items, and their answers are not considered for auto-promotion.

Attention-check and gold results are written to <output_annotation_dir>/quality_control_results.json as they are recorded, and read back at startup, so a restart for a deploy or a crash does not clear an annotator's failure history or unblock someone who had been blocked.

Admin Dashboard¶

View attention check statistics in the admin dashboard at /admin: - Overall pass/fail rates - Per-annotator statistics - Individual failure history

Gold Standards¶

Gold standards are expert-labeled items used to measure annotator accuracy. By default, gold standards are silent - results are recorded for admin review in the dashboard, but annotators don't see feedback. You therefore track quality without influencing annotator behavior.

Configuration¶

gold_standards:
  enabled: true

  # Path to JSON file containing gold standard items
  items_file: "gold_standards.json"

  # How to use gold standards
  mode: "mixed"              # Options: mixed, separate, training
  # - mixed: inject into the annotation stream on the frequency below
  # - separate: accepted, and behaves exactly as mixed
  # - training: serves nothing at all. See the note below.

  # For mixed mode, how often to inject
  frequency: 20              # Insert one every 20 items

  # Accuracy requirements (tracked in admin dashboard)
  accuracy:
    min_threshold: 0.7       # Minimum required accuracy (70%)
    evaluation_count: 10     # Evaluate after this many gold items
  # Evaluated whether or not feedback is on. Annotators under the threshold
  # appear in `gold_standards.below_accuracy_threshold` in
  # /admin/api/quality_control, and are named in the server log the first time
  # they cross it. Only the message TO the annotator depends on `feedback`.

  # Feedback settings (disabled by default for silent tracking)
  # Enable for training scenarios where you want to give annotators feedback
  feedback:
    show_correct_answer: false  # Show correct answer after submission
    show_explanation: false     # Show explanation if provided

  # Auto-promotion: items become gold standards when annotators agree
  auto_promote:
    enabled: true
    min_annotators: 3          # Minimum annotators before checking
    agreement_threshold: 1.0   # 1.0 = unanimous, 0.8 = 80% agree

What each mode does¶

mixed is the implemented mode: gold items are injected into the annotation stream every frequency items, and graded silently unless feedback is on.

separate is accepted and takes the same code path as mixed. There is no dedicated evaluation block; the server says so at boot.

training serves no gold items. Nothing delivers them during the training phase, so a study configured this way loads its gold file, logs the same lines as a working config, and measures nothing. Both the server and potato validate --strict now refuse to let that pass quietly. To grade during training, put practice questions in the training phase's own config — they use the same comparator.

Grading and the payload shape¶

Answers reach the server as {"schema:label": value} or {"schema:::label": value}; both are accepted and both are graded the same way. When the value is a selection marker — on, true, yes, 1, checked or selected — the answer is the label, which is what a browser form, the simulator and most API callers send. When the value is anything else it is the answer, which is what a free-text or numeric schema stores.

Auto-promotion¶

An item is promoted when min_annotators have answered it and they agree. Agreement is measured per schema on the resolved label, so two annotators giving opposite answers do not promote, and a schema only one of them answered is not treated as unanimous. A tie never promotes, whatever agreement_threshold is: at 0.5, a one-to-one split has no majority label. The promotion record's agreement is the share of annotators behind the promoted label, taken from the least-agreed schema.

Raters declared under machine_annotators are not counted toward promotion. Tools that share a reference database make the same mistakes, so counting them would let a few tools write their shared error into the answer key before any person had looked at the item. People who agree with the tools still promote it.

A promoted item is graded when an annotator meets it in the ordinary stream; it is never injected. The annotators whose answers promoted it are not graded on it, since its gold label is their own consensus. The headline gold accuracy therefore depends on when an annotator arrived: items promoted behind them count, items promoted ahead of them do not. /admin/api/quality_control reports configured_correct / configured_total per annotator alongside the totals, because the configured pool is the same for everyone and is the comparable number. total_items counts everything currently gradeable, with configured_items and promoted_items broken out.

Gold Standard Items File Format¶

[
  {
    "id": "gold_001",
    "text": "The service was absolutely terrible and I will never return.",
    "gold_label": {
      "sentiment": "negative"
    },
    "explanation": "Strong negative language ('absolutely terrible', 'never return') clearly indicates negative sentiment.",
    "difficulty": "easy"
  },
  {
    "id": "gold_002",
    "text": "The food was okay but nothing special.",
    "gold_label": {
      "sentiment": "neutral"
    },
    "explanation": "Mixed signals balance to neutral sentiment.",
    "difficulty": "medium"
  }
]

Fields: - id (required): Unique identifier - text (required): The text to display - gold_label (required): Dictionary with correct annotations - explanation (optional): Explanation shown to annotators - difficulty (optional): Metadata for analysis

Gold Standards for Drawn Answers¶

Image annotation gold standards work, and are graded by overlap rather than equality — two annotators never produce byte-identical geometry, so a string comparison would fail every drawn answer no matter how well it was drawn.

The gold_label value is the same JSON the annotation client stores: a list of objects with normalized (0–1) coordinates.

[
  {
    "id": "gold_img_001",
    "image": "images/street_042.jpg",
    "gold_label": {
      "objects": "[{\"type\": \"bbox\", \"label\": \"car\", \"coordinates\": {\"x\": 0.12, \"y\": 0.40, \"width\": 0.22, \"height\": 0.18}}]"
    },
    "explanation": "The parked car on the left; the reflection in the window is not a second car."
  }
]

An answer passes when every gold shape is matched by a drawn shape of the same label at or above the IoU tolerance, with no extra shapes. Right place, wrong label is a failure — that is what a gold standard exists to catch.

quality_control:
  geometry_iou_tolerance: 0.5   # default; also accepted inside gold_standards

Tolerance	Effect
`0.5` (default)	The Pascal VOC / COCO detection convention. The annotator clearly found and outlined the right object.
`0.75`–`0.9`	Strict. Appropriate for segmentation work where boundary precision is the skill being tested.
`0.25`–`0.4`	Loose. Appropriate for "did you spot it at all" screening, or for small objects where a few pixels dominate the IoU.

IoU punishes small objects

A fixed pixel error costs far more IoU on a small object than a large one. If your gold items are dominated by small objects, a 0.5 tolerance is stricter in practice than it sounds — consider lowering it or choosing larger gold objects.

The same comparator grades training-phase practice questions, so a project's gold standards and its training answers are held to the same standard.

Feedback Display¶

After submitting a gold standard item, annotators see: - Whether their answer was correct or incorrect - The correct answer (if show_correct_answer: true) - An explanation (if show_explanation: true and explanation provided) - Accuracy warning if below threshold

Admin Dashboard¶

View gold standard metrics in the admin dashboard: - Overall accuracy across all annotators - Per-annotator accuracy tracking - Per-item difficulty analysis (which items are most often missed) - Users below accuracy threshold

Auto-Promotion to Gold Standard¶

You can configure Potato to automatically promote items to the gold standard pool when multiple annotators agree on the label. This is useful for: - Growing your gold standard pool organically - Identifying "easy" items where everyone agrees - Reducing the burden of manually creating gold standards

gold_standards:
  enabled: true
  items_file: "initial_gold_standards.json"  # Seed items (optional)

  auto_promote:
    enabled: true
    min_annotators: 3          # Wait for at least 3 annotators
    agreement_threshold: 1.0   # 1.0 = all must agree (unanimous)
                               # 0.8 = 80% must agree

How it works: 1. As items are annotated, the system tracks all responses 2. When min_annotators have annotated an item, agreement is checked 3. If agreement meets agreement_threshold, the item is promoted 4. Promoted items are added to the gold standard pool and used for future quality checks

Admin visibility: - View promoted items in /admin/api/quality_control - See "promotion candidates" (items close to threshold) - Track which items were auto-promoted vs. manually defined

Pre-annotation Support¶

Pre-annotation pre-fills the form with model predictions, which is useful for: - Active learning workflows - Correcting model outputs - Bootstrapping from existing annotations

Configuration¶

pre_annotation:
  enabled: true

  # Field in data items containing predictions
  field: "predictions"

  # Can annotators change pre-filled values?
  allow_modification: true

  # Show confidence scores if available
  show_confidence: true

  # Highlight items below this confidence threshold
  highlight_low_confidence: 0.7

Data Format¶

Include predictions in your data items:

{
  "id": "item_001",
  "text": "I love this product!",
  "predictions": {
    "sentiment": "positive",
    "confidence": 0.92
  }
}

For span annotations:

{
  "id": "item_002",
  "text": "Apple announced new iPhone in California.",
  "predictions": {
    "entities": [
      {"start": 0, "end": 5, "label": "ORG", "confidence": 0.85},
      {"start": 27, "end": 37, "label": "LOC", "confidence": 0.91}
    ]
  }
}

When predictions are applied¶

When an item is loaded, the predictions field is extracted
If the annotator hasn't already annotated this item, predictions are used to pre-fill the form
Annotators can modify the pre-filled values (if allow_modification: true)
Low-confidence items can be visually highlighted

Best Practices¶

Use pre-annotation for correction workflows where you have model predictions
Set allow_modification: true to let annotators fix errors
Use confidence thresholds to flag items needing more attention
Track modification rates to assess model quality

Agreement Metrics¶

Real-time inter-annotator agreement metrics are available in the admin dashboard, using Krippendorff's alpha.

Configuration¶

agreement_metrics:
  enabled: true

  # Minimum annotators per item for calculation
  min_overlap: 2

  # Auto-refresh settings
  auto_refresh: true
  refresh_interval: 60       # Seconds between updates

Interpreting Krippendorff's Alpha¶

Alpha Value	Interpretation
α ≥ 0.8	Good agreement - reliable for most purposes
0.67 ≤ α < 0.8	Tentative agreement - draw tentative conclusions
0.33 ≤ α < 0.67	Low agreement - review guidelines
α < 0.33	Poor agreement - significant issues

Admin Dashboard¶

The Agreement tab in the admin dashboard shows: - Overall average alpha across all schemas - Per-schema agreement metrics - Number of items evaluated - Metric type (nominal vs interval) - Human-readable interpretation

When to Use Different Metrics¶

The system automatically selects the appropriate metric: - Nominal metric: For categorical annotations (radio, multiselect) - Interval metric: For numeric annotations (likert, slider, number)

An interval metric subtracts the two ratings, so a likert, slider or number scheme whose answers are not numbers is measured nominally instead. The report says so, in metric_note on the schema and in the top-level warnings list.

Nothing is reported for a schema until min_overlap annotators have answered the same item, so a study with one annotator shows "No items with 2+ annotators" rather than a number.

API Endpoints¶

Quality Control Metrics¶

GET /admin/api/quality_control

Returns:

{
  "enabled": true,
  "attention_checks": {
    "enabled": true,
    "total_checks": 50,
    "total_passed": 45,
    "total_failed": 5,
    "by_user": {
      "user1": {"passed": 10, "failed": 0, "pass_rate": 1.0},
      "user2": {"passed": 8, "failed": 2, "pass_rate": 0.8}
    }
  },
  "gold_standards": {
    "enabled": true,
    "total_evaluations": 30,
    "total_correct": 25,
    "by_user": {...},
    "by_item": {...}
  }
}

Agreement Metrics¶

GET /admin/api/agreement

Returns:

{
  "enabled": true,
  "overall": {
    "average_krippendorff_alpha": 0.75,
    "interpretation": "Tentative agreement"
  },
  "by_schema": {
    "sentiment": {
      "krippendorff_alpha": 0.82,
      "items_evaluated": 100,
      "interpretation": "Good agreement"
    }
  }
}

Example Configuration¶

Here's a complete example with all quality control features enabled:

annotation_task_name: "Sentiment Analysis with Quality Control"

# Main annotation scheme
annotation_schemes:
  - name: sentiment
    annotation_type: radio
    labels: [positive, negative, neutral]
    description: "Select the sentiment of the text"

# Quality Control Configuration
attention_checks:
  enabled: true
  items_file: "data/attention_checks.json"
  frequency: 15
  failure_handling:
    warn_threshold: 2
    block_threshold: 5

# Optional: discard all completed work from QC-blocked users.
# If omitted, completed annotations are preserved, while the failed
# attention-check response itself is not kept.
instance_reclaim:
  quality_control:
    preserve_completed_annotations: false

gold_standards:
  enabled: true
  items_file: "data/gold_standards.json"
  mode: mixed
  frequency: 25
  accuracy:
    min_threshold: 0.7
    evaluation_count: 5
  feedback:
    show_correct_answer: true
    show_explanation: true

pre_annotation:
  enabled: true
  field: "model_prediction"
  allow_modification: true

agreement_metrics:
  enabled: true
  min_overlap: 2
  refresh_interval: 60

Step-Level Quality Control¶

For agent trace evaluation and trajectory annotation tasks, Potato supports step-level quality control — checking annotator performance at the granularity of individual agent steps rather than whole instances.

Configuration¶

quality_control:
  step_level:
    enabled: true

    # Gold standards with known-correct step labels
    gold_standards_file: "data/gold_steps.json"

    # Attention checks at the step level
    attention_checks:
      enabled: true
      frequency: 5  # Insert attention check every N steps

Step-level QC is particularly useful for: - Trajectory evaluation tasks where each step has independent correctness - Process reward model annotation where per-step labels are critical - Code review tasks with multiple files to review

Step-level agreement metrics (Cohen's kappa) are computed per annotator pair for step-level schemas and available in the admin dashboard.

Writing-Process Signals (free-text quality)¶

Attention checks and gold standards work on categorical answers. They cannot tell you whether a free-text rationale was written by the annotator or pasted in from an LLM — the finished text usually looks fine either way.

Keystroke logging adds a complementary signal for free-text fields by recording how the response was produced:

keystroke_logging:
  enabled: true
  detection:
    enabled: true
    on_external_insert: flag   # allow | warn | block | flag

This produces per-response flags — paste_dominant, silent_insertion, transcription_rhythm, offscreen_composition, implausible_speed, synthetic_input — each carrying the evidence that fired it. They surface in the Writing Process panel of the admin dashboard and in the typing_dynamics.csv export.

These flags are not attention checks

A failed attention check is a discrete, defensible fact: the annotator gave a known-wrong answer. A writing-process flag is a statistical signal about a behaviour that has innocent explanations — fast typists, mobile keyboards, dictation, assistive technology. Do not wire them to automatic rejection.

See false positives and the ethics guide before using them in any decision that affects an annotator.

Troubleshooting¶

Attention checks not appearing¶

Verify items_file path is correct (relative to task directory)
Check that items have required fields (id, expected_answer)
Ensure frequency or probability is set

Gold standard feedback not showing¶

Check feedback.show_correct_answer is true
Verify items have gold_label field
Check browser console for JavaScript errors

Agreement metrics showing "No items with N+ annotators"¶

Ensure items have been annotated by multiple users
Reduce min_overlap if needed
Check that annotations are being saved correctly

Pre-annotations not appearing¶

Verify field matches the field name in your data
Check that predictions format matches expected schema
Ensure user hasn't already annotated the item (pre-annotations only appear for un-annotated items)