Potato API Reference

This document describes the HTTP API endpoints provided by Potato, enabling integration with custom frontends, automation scripts, or external systems.

Overview

Base URL

http://localhost:8000

Authentication

Most endpoints require an active session. Sessions are established via the login endpoints and maintained via cookies.

For programmatic access, you can: 1. Use session cookies from a browser 2. Use the debug mode with automatic user creation 3. Implement the login flow programmatically

Response Format

All API endpoints return JSON responses. Error responses include an error field with a description.

Session Management

Login

POST /auth
Content-Type: application/x-www-form-urlencoded

Request Body:

username=your_username&password=your_password

Response: Redirects to annotation page on success, or returns error.

Check Session

GET /api/current_instance

Response (authenticated):

{
  "instance_id": "item_001",
  "current_index": 0,
  "total_instances": 100
}

Response (not authenticated):

{
  "error": "No active session"
}

Status: 401

Logout

POST /logout

Response: Redirects to home page.

Annotation Endpoints

Get Current Instance

GET /api/current_instance

Returns information about the user's current annotation instance.

Response:

{
  "instance_id": "item_001",
  "current_index": 5,
  "total_instances": 100
}

Get Instance Content

GET /api/spans/{instance_id}

Returns the text content and existing span annotations for an instance.

Response:

{
  "instance_id": "item_001",
  "text": "The quick brown fox jumps over the lazy dog.",
  "spans": [
    {
      "id": "span_123",
      "schema": "entities",
      "label": "ANIMAL",
      "title": "ANIMAL",
      "start": 16,
      "end": 19,
      "text": "fox",
      "color": "#ff6b6b"
    }
  ]
}

Get Annotations

GET /get_annotations?instance_id={instance_id}

Returns all annotations (labels and spans) for a specific instance.

Response:

{
  "label_annotations": {
    "sentiment": {
      "positive": "1"
    }
  },
  "span_annotations": {
    "entities:ANIMAL:16:19": "true"
  }
}

Submit Annotation

POST /updateinstance
Content-Type: application/json

Submit or update annotations for an instance.

Request Body (Frontend Format):

{
  "instance_id": "item_001",
  "annotations": {
    "sentiment:positive": "1",
    "sentiment:negative": "0"
  },
  "span_annotations": [
    {
      "schema": "entities",
      "name": "PERSON",
      "title": "PERSON",
      "start": 0,
      "end": 5,
      "value": "true"
    }
  ],
  "client_timestamp": "2024-01-15T10:30:00Z"
}

Request Body (Backend Format):

{
  "instance_id": "item_001",
  "schema": "sentiment",
  "type": "label",
  "state": [
    {"name": "positive", "value": "1"},
    {"name": "negative", "value": "0"}
  ]
}

Response (Success):

{
  "status": "success",
  "processing_time_ms": 15,
  "performance_metrics": {
    "total_annotations": 50,
    "session_duration": 1800
  }
}

Response (With Quality Control Feedback):

{
  "status": "success",
  "processing_time_ms": 12,
  "qc_result": {
    "type": "gold_standard",
    "correct": false,
    "gold_label": {"sentiment": "positive"},
    "explanation": "Strong positive language indicates positive sentiment."
  }
}

Response (Attention Check Warning):

{
  "status": "success",
  "warning": true,
  "warning_message": "Please read items carefully before answering.",
  "qc_result": {
    "type": "attention_check",
    "passed": false,
    "warning": true
  }
}

Response (Blocked):

{
  "status": "blocked",
  "message": "You have been blocked due to too many incorrect responses."
}

Clear Span Annotations

POST /api/spans/{instance_id}/clear

Clears all span annotations for an instance.

Response:

{
  "status": "success",
  "cleared_count": 3
}

POST /go_to
Content-Type: application/json

Navigate to a specific instance by ID or index.

Request Body:

{
  "go_to": "item_005"
}

Or by action:

{
  "action": "next_instance"
}

Schema Information

Get Annotation Schemas

GET /api/schemas

Returns information about configured annotation schemas (primarily for span annotations).

Response:

{
  "entities": {
    "name": "entities",
    "description": "Named entity recognition",
    "labels": ["PERSON", "ORGANIZATION", "LOCATION"],
    "type": "span"
  },
  "sentiment_spans": {
    "name": "sentiment_spans",
    "description": "Sentiment-bearing phrases",
    "labels": ["positive", "negative"],
    "type": "span"
  }
}

Get Label Colors

GET /api/colors

Returns the color mapping for annotation labels.

Response:

{
  "entities": {
    "PERSON": "#ff6b6b",
    "ORGANIZATION": "#4ecdc4",
    "LOCATION": "#45b7d1"
  }
}

Get Keyword Highlights

GET /api/keyword_highlights/{instance_id}

Returns admin-defined keyword matches found in the instance text. Keywords are configured via keyword_highlights_file in the config.

Parameters: - instance_id (path): The ID of the instance to get keyword highlights for

Response:

{
  "instance_id": "item_1",
  "keywords": [
    {
      "label": "positive",
      "start": 10,
      "end": 19,
      "text": "excellent",
      "reasoning": "Keyword: excel* → positive",
      "schema": "sentiment",
      "color": "rgba(34, 197, 94, 0.8)"
    },
    {
      "label": "negative",
      "start": 45,
      "end": 53,
      "text": "terrible",
      "reasoning": "Keyword: terrible → negative",
      "schema": "sentiment",
      "color": "rgba(239, 68, 68, 0.8)"
    }
  ]
}

Response Fields: | Field | Description | |-------|-------------| | instance_id | The instance ID | | keywords | Array of keyword matches found in the text | | keywords[].label | The label associated with this keyword | | keywords[].start | Character offset where the match starts | | keywords[].end | Character offset where the match ends | | keywords[].text | The actual matched text from the instance | | keywords[].reasoning | Description of the keyword pattern that matched | | keywords[].schema | The annotation schema this keyword belongs to | | keywords[].color | RGBA color string for rendering the highlight |

Notes: - Returns empty keywords array if no keyword_highlights_file is configured - Keywords are sorted by start position - Duplicate matches at the same position are deduplicated - Colors are taken from ui.spans.span_colors config or auto-assigned

Admin API Endpoints

Admin endpoints require admin privileges or API key authentication.

Health Check

GET /admin/health

Response:

{
  "status": "healthy",
  "timestamp": "2024-01-15T10:30:00Z",
  "version": "2.0.0"
}

Dashboard Overview

GET /admin/api/overview

Response:

{
  "total_items": 1000,
  "total_annotations": 5000,
  "total_users": 25,
  "active_users": 10,
  "completion_rate": 0.45,
  "items_by_status": {
    "completed": 450,
    "in_progress": 150,
    "unassigned": 400
  }
}

Get Annotators

GET /admin/api/annotators

Response:

{
  "annotators": [
    {
      "user_id": "user1",
      "annotations_count": 200,
      "current_phase": "annotation",
      "last_active": "2024-01-15T10:30:00Z",
      "assigned_items": 250,
      "completed_items": 200
    }
  ],
  "total_count": 25
}

Get Instances

GET /admin/api/instances

Query Parameters: - status - Filter by status (completed, in_progress, unassigned) - limit - Maximum number of results (default: 100) - offset - Pagination offset

Response:

{
  "instances": [
    {
      "id": "item_001",
      "text_preview": "The quick brown fox...",
      "annotations_count": 3,
      "status": "completed"
    }
  ],
  "total_count": 1000
}

Get Configuration

GET /admin/api/config

Response:

{
  "annotation_task_name": "Sentiment Analysis",
  "annotation_schemes": [...],
  "total_items": 1000,
  "output_format": "json"
}

Update Configuration

POST /admin/api/config
Content-Type: application/json

Request Body:

{
  "setting_name": "value"
}

Get Annotation History

GET /admin/api/annotation_history

Query Parameters: - user_id - Filter by user - instance_id - Filter by instance - minutes - Time window in minutes

Response:

{
  "history": [
    {
      "user_id": "user1",
      "instance_id": "item_001",
      "action_type": "add_label",
      "schema_name": "sentiment",
      "label_name": "positive",
      "timestamp": "2024-01-15T10:30:00Z"
    }
  ]
}

Get Suspicious Activity

GET /admin/api/suspicious_activity

Response:

{
  "suspicious_users": [
    {
      "user_id": "user5",
      "flags": ["rapid_submissions", "low_time_per_item"],
      "avg_time_per_item": 2.5,
      "annotations_count": 500
    }
  ]
}

Get Crowdsourcing Data

GET /admin/api/crowdsourcing

Response:

{
  "summary": {
    "total_workers": 50,
    "prolific_workers": 30,
    "mturk_workers": 15,
    "other_workers": 5
  },
  "prolific": {
    "workers": [...],
    "completion_codes_issued": 28
  },
  "mturk": {
    "workers": [...],
    "hits_completed": 15
  }
}

Quality Control API

Get Quality Control Metrics

GET /admin/api/quality_control

Response:

{
  "enabled": true,
  "attention_checks": {
    "enabled": true,
    "total_items": 5,
    "total_checks": 150,
    "total_passed": 140,
    "total_failed": 10,
    "by_user": {
      "user1": {
        "passed": 10,
        "failed": 0,
        "pass_rate": 1.0
      },
      "user2": {
        "passed": 8,
        "failed": 2,
        "pass_rate": 0.8
      }
    }
  },
  "gold_standards": {
    "enabled": true,
    "total_items": 10,
    "total_evaluations": 200,
    "total_correct": 170,
    "total_incorrect": 30,
    "by_user": {
      "user1": {
        "correct": 9,
        "total": 10,
        "accuracy": 0.9
      }
    },
    "by_item": {
      "gold_001": {
        "correct": 18,
        "total": 20,
        "accuracy": 0.9
      }
    }
  },
  "auto_promotion": {
    "enabled": true,
    "min_annotators": 3,
    "agreement_threshold": 1.0,
    "promoted_count": 5,
    "promoted_items": [
      {
        "item_id": "item_042",
        "consensus_label": {"sentiment": "positive"},
        "annotator_count": 3,
        "promoted_at": "2024-01-15T10:30:00Z"
      }
    ],
    "candidates": [
      {
        "item_id": "item_055",
        "annotator_count": 2,
        "needed_annotators": 3,
        "schema_agreement": {
          "sentiment": {
            "value": "negative",
            "count": 2,
            "total": 2,
            "agreement": 1.0
          }
        }
      }
    ]
  },
  "pre_annotation": {
    "enabled": true,
    "items_with_predictions": 1000
  }
}

Get Agreement Metrics

GET /admin/api/agreement

Response:

{
  "enabled": true,
  "overall": {
    "average_krippendorff_alpha": 0.78,
    "schemas_evaluated": 2,
    "interpretation": "Tentative agreement"
  },
  "by_schema": {
    "sentiment": {
      "krippendorff_alpha": 0.82,
      "metric_type": "nominal",
      "items_evaluated": 100,
      "total_annotations": 300,
      "interpretation": "Good agreement"
    },
    "intensity": {
      "krippendorff_alpha": 0.74,
      "metric_type": "interval",
      "items_evaluated": 100,
      "total_annotations": 300,
      "interpretation": "Tentative agreement"
    }
  },
  "warnings": []
}

AI Assistant API

Get AI Suggestion

GET /get_ai_suggestion?annotationId={annotation_index}&aiAssistant={ai_assistant_type}

Response:

{
  "suggestion": "Based on the text, this appears to be positive sentiment.",
  "confidence": 0.85,
  "labels": {
    "sentiment": "positive"
  }
}

Get AI Assistant Help

GET /api/ai_assistant?instance_id={instance_id}&schema={schema_name}

Response:

{
  "html": "<div class='ai-hint'>This text expresses positive sentiment...</div>",
  "suggestions": [
    {
      "schema": "sentiment",
      "label": "positive",
      "confidence": 0.9
    }
  ]
}

User State Endpoints (Admin)

Get User State

GET /admin/user_state/{user_id}

Response:

{
  "user_id": "user1",
  "current_phase": "annotation",
  "current_instance_index": 45,
  "instance_id_ordering": ["item_001", "item_002", ...],
  "assigned_count": 100,
  "completed_count": 45,
  "annotation_history": [...]
}

Get All Users

GET /admin/system_state

Response:

{
  "users": {
    "user1": {
      "phase": "annotation",
      "annotations": 200,
      "last_active": "2024-01-15T10:30:00Z"
    }
  },
  "items": {
    "total": 1000,
    "assigned": 800,
    "completed": 500
  }
}

Item State Endpoints (Admin)

Get All Items

GET /admin/all_instances

Response:

{
  "instances": [
    {
      "id": "item_001",
      "text": "The quick brown fox...",
      "assigned_to": ["user1", "user2"],
      "annotations_count": 2
    }
  ]
}

Get Item State

GET /admin/item_state/{item_id}

Response:

{
  "id": "item_001",
  "text": "The quick brown fox jumps over the lazy dog.",
  "data": {
    "id": "item_001",
    "text": "The quick brown fox jumps over the lazy dog.",
    "metadata": {...}
  },
  "assigned_to": ["user1", "user2"],
  "annotations": {
    "user1": {
      "sentiment": {"positive": "1"}
    },
    "user2": {
      "sentiment": {"positive": "1"}
    }
  }
}

Media Endpoints

Get Audio Waveform

GET /api/waveform/{cache_key}

Returns cached waveform data for audio annotation.

Generate Audio Waveform

POST /api/waveform/generate
Content-Type: application/json

Request Body:

{
  "audio_url": "https://example.com/audio.mp3"
}

Response:

{
  "cache_key": "abc123",
  "waveform_url": "/api/waveform/abc123"
}

Proxy Audio

GET /api/audio/proxy?url={encoded_audio_url}

Proxies external audio files to handle CORS issues.

Get Video Metadata

POST /api/video/metadata
Content-Type: application/json

Request Body:

{
  "video_url": "https://example.com/video.mp4"
}

Response:

{
  "duration": 120.5,
  "width": 1920,
  "height": 1080,
  "fps": 30
}

Behavioral Tracking API

Track Interactions

POST /api/track_interactions
Content-Type: application/json

Records batched interaction events from the frontend (mouse clicks, focus time, scroll depth).

Track AI Usage

POST /api/track_ai_usage
Content-Type: application/json

Records AI assistance requests, responses, and user decisions for analytics.

Track Annotation Change

POST /api/track_annotation_change
Content-Type: application/json

Records annotation changes from the frontend (select, deselect, update, clear actions).

Get Behavioral Data

GET /api/behavioral_data/{instance_id}

Returns behavioral data for a specific instance (interaction events, timing, AI usage).

Option Highlighting API

Get Option Highlighting Config

GET /api/option_highlights/config

Returns the option highlighting configuration for AI-suggested labels.

Trigger Option Highlight Prefetch

POST /api/option_highlights/prefetch

Triggers prefetching of option highlights for upcoming items.

Get Option Highlights

GET /api/option_highlights/{annotation_id}

Returns AI-suggested option highlights for a specific annotation scheme index.

Entity Linking API

Search Knowledge Base

GET /api/entity_linking/search?query={query}&kb={kb_name}

Searches a configured knowledge base for entities matching the query.

Get Entity Details

GET /api/entity_linking/entity/{kb_name}/{entity_id}

Returns detailed information about a specific entity from a knowledge base.

Get Configured Knowledge Bases

GET /api/entity_linking/configured_kbs

Returns the list of configured knowledge bases.

POST /api/entity_linking/update_span
Content-Type: application/json

Updates a span annotation with entity linking information.

GET /api/links/{instance_id}

Returns link annotations (span relationships) for a specific instance.

DELETE /api/links/{instance_id}/{link_id}

Deletes a specific link annotation.

Get Events

GET /api/events/{instance_id}

Returns event annotations for a specific instance.

Delete Event

DELETE /api/events/{instance_id}/{event_id}

Deletes a specific event annotation.

Span API

Get Span Data

GET /api/spans/{instance_id}

Returns text content and existing span annotations for an instance.

Clear Spans

DELETE /api/spans/{instance_id}/clear

Clears all span annotations for an instance.

Get Span Colors

GET /api/colors

Returns the color mapping for span labels.

Chat Support API

Send Chat Message

POST /api/chat/send
Content-Type: application/json

Sends a message to the LLM chat assistant and returns the response.

Get Chat History

GET /api/chat/history?instance_id={instance_id}

Returns chat history for a specific instance.

Get Chat Config

GET /api/chat/config

Returns chat UI configuration.

Agent Chat API

Send Agent Message

POST /agent_chat/send
Content-Type: application/json

Sends a message to the agent and returns a response.

Finish Agent Chat

POST /agent_chat/finish

Finishes the chat and writes conversation data to the item.

Get Agent Chat Status

GET /agent_chat/status

Returns the current agent chat session status (for page refresh recovery).

ICL Labeling API (Admin)

Get ICL Status

GET /admin/api/icl/status

Returns ICL labeler status and statistics.

Get ICL Examples

GET /admin/api/icl/examples?schema={schema_name}

Returns current high-confidence examples, optionally filtered by schema.

Get ICL Predictions

GET /admin/api/icl/predictions?schema={schema_name}&status={status}&limit={limit}

Returns LLM predictions with filtering options.

Get ICL Accuracy

GET /admin/api/icl/accuracy

Returns accuracy metrics for ICL labeling.

Trigger ICL Operation

POST /admin/api/icl/trigger
Content-Type: application/json

Request Body:

{
  "action": "refresh_examples | batch_label | save_state"
}

Record ICL Verification

POST /api/icl/record_verification
Content-Type: application/json

Records human verification of an LLM prediction.

Adjudication API

Get Adjudication Queue

GET /adjudicate/api/queue

Returns the adjudication queue (items needing adjudication).

Get Adjudication Item

GET /adjudicate/api/item/{instance_id}

Returns full item detail for adjudication review.

Submit Adjudication Decision

POST /adjudicate/api/submit
Content-Type: application/json

Submits an adjudication decision.

Get Adjudication Stats

GET /adjudicate/api/stats

Returns adjudication progress statistics.

Skip Adjudication Item

POST /adjudicate/api/skip/{instance_id}

Skips an adjudication item.

Get Next Adjudication Item

GET /adjudicate/api/next

Returns the next adjudication item in the queue.

Get Similar Items

GET /adjudicate/api/similar/{instance_id}

Returns similar items for a specific instance (lazy-loading).

Admin Adjudication Overview

GET /admin/api/adjudication

Returns admin dashboard overview of adjudication status.

BWS / IBWS Scoring API (Admin)

Get BWS Scoring Status

GET /admin/api/bws_scoring

Returns current BWS (Best-Worst Scaling) scoring status and cached scores.

Generate BWS Scores

POST /admin/api/bws_scoring/generate

Generates BWS scores from current annotations.

Get IBWS Status

GET /admin/api/ibws_status

Returns current IBWS (Iterative Best-Worst Scaling) round status and progress.

Get IBWS Ranking

GET /admin/api/ibws_ranking

Returns current IBWS ordinal ranking.

MACE API (Admin)

Get MACE Overview

GET /admin/api/mace/overview

Returns MACE (Multiple Annotator Competence Estimation) results overview.

Get MACE Predictions

GET /admin/api/mace/predictions?schema={schema_name}&instance_id={instance_id}

Returns MACE predicted labels, optionally filtered by schema and instance.

Trigger MACE Computation

POST /admin/api/mace/trigger

Manually triggers a MACE recomputation.

Embedding Visualization API (Admin)

Get Visualization Data

GET /admin/api/embedding_viz/data

Returns visualization data for the embedding scatter plot.

Reorder Queue from Embeddings

POST /admin/api/embedding_viz/reorder
Content-Type: application/json

Reorders the annotation queue based on selected instances in the visualization.

Refresh Embeddings

POST /admin/api/embedding_viz/refresh

Forces re-computation of embeddings and UMAP projection.

Get Embedding Stats

GET /admin/api/embedding_viz/stats

Returns embedding visualization statistics.

Export API (Admin)

List Export Formats

GET /admin/api/export/formats

Returns available export formats with metadata.

Run Export

POST /admin/api/export
Content-Type: application/json

Runs an export in the requested format and returns the result.

Admin Analytics API

Get Behavioral Analytics

GET /admin/api/behavioral_analytics

Returns behavioral analytics data for all annotators (AI usage, interaction types, suspicion scores).

Get Stale Assignments

GET /admin/api/stale_assignments

Returns a list of stale instance assignments (assigned but not annotated past timeout).

Reclaim Instance

POST /admin/api/reclaim_instance
Content-Type: application/json

Manually reclaims a specific instance assignment from a user.

Set User Instances

POST /admin/api/user/{username}/set_instances
Content-Type: application/json

Sets the maximum number of instances for a specific user.

Get Question Analytics

GET /admin/api/questions

Returns aggregate analysis data for all annotation schemas/questions.

Schema API

Get Annotation Schemas

GET /api/schemas

Returns annotation schema information for the current configuration.

Error Responses

All endpoints may return error responses in the following format:

{
  "error": "Description of the error"
}

Common HTTP status codes: - 400 - Bad Request (invalid parameters) - 401 - Unauthorized (no session or invalid credentials) - 403 - Forbidden (insufficient permissions) - 404 - Not Found (resource doesn't exist) - 500 - Internal Server Error

Example: Complete Annotation Flow

Here's an example of a complete annotation flow using the API:

import requests

BASE_URL = "http://localhost:8000"
session = requests.Session()

# 1. Login
session.post(f"{BASE_URL}/auth", data={
    "username": "annotator1",
    "password": "password123"
})

# 2. Get current instance
response = session.get(f"{BASE_URL}/api/current_instance")
instance_info = response.json()
instance_id = instance_info["instance_id"]

# 3. Get instance content
response = session.get(f"{BASE_URL}/api/spans/{instance_id}")
content = response.json()
print(f"Text: {content['text']}")

# 4. Submit annotation
response = session.post(f"{BASE_URL}/updateinstance", json={
    "instance_id": instance_id,
    "annotations": {
        "sentiment:positive": "1"
    }
})
result = response.json()
print(f"Status: {result['status']}")

# 5. Navigate to next instance
session.post(f"{BASE_URL}/go_to", json={
    "action": "next_instance"
})

# 6. Repeat steps 2-5 until done

Example: Admin Monitoring

import requests

BASE_URL = "http://localhost:8000"

# Get overview
overview = requests.get(f"{BASE_URL}/admin/api/overview").json()
print(f"Progress: {overview['completion_rate']*100:.1f}%")

# Get annotator stats
annotators = requests.get(f"{BASE_URL}/admin/api/annotators").json()
for a in annotators["annotators"]:
    print(f"{a['user_id']}: {a['completed_items']}/{a['assigned_items']}")

# Get agreement metrics
agreement = requests.get(f"{BASE_URL}/admin/api/agreement").json()
print(f"Agreement: α = {agreement['overall']['average_krippendorff_alpha']:.3f}")

# Get quality control metrics
qc = requests.get(f"{BASE_URL}/admin/api/quality_control").json()
if qc["attention_checks"]["enabled"]:
    pass_rate = qc["attention_checks"]["total_passed"] / qc["attention_checks"]["total_checks"]
    print(f"Attention check pass rate: {pass_rate*100:.1f}%")

WebSocket Events (Future)

Currently, Potato uses polling for real-time updates. WebSocket support may be added in future versions for: - Real-time annotation updates - Live progress notifications - Collaborative annotation features