Skip to content

Behavioral Tracking and Analytics

Overview

Potato records what annotators do during a session: clicks and focus events, how long each item took, when AI assistance was asked for and whether the suggestion was taken, and every change to an answer. The record is what an audit of annotation quality is built from, and it is the only way to tell a careful annotator from a fast one after the fact.

What Gets Tracked

1. Interaction Events

Every user interaction with the annotation interface is captured:

Event Type Description Example Target
click Mouse clicks on elements label:positive, nav:next
focus_in Element receives focus textbox:explanation
focus_out Element loses focus label:negative
keypress Keyboard shortcuts key:1, nav:ArrowRight
navigation Instance navigation next, prev, instance_load
save Annotation save events instance:123
annotation_change Label modifications schema:sentiment

2. AI Assistance Usage

Complete lifecycle tracking for AI-assisted annotation:

{
  "request_timestamp": 1706500010.0,
  "response_timestamp": 1706500012.5,
  "schema_name": "sentiment",
  "suggestions_shown": ["positive", "neutral"],
  "suggestion_accepted": "positive",
  "time_to_decision_ms": 3500
}

Tracked metrics: - When AI assistance was requested - Response latency - Which suggestions were shown - Whether the suggestion was accepted or rejected - Time from seeing suggestion to making a decision

3. Annotation Changes

Detailed change history for all annotations:

{
  "timestamp": 1706500002.5,
  "schema_name": "sentiment",
  "label_name": "positive",
  "action": "select",
  "old_value": null,
  "new_value": true,
  "source": "user"
}

Source types: - user - Direct user interaction - ai_accept - User accepted AI suggestion - keyboard - Keyboard shortcut used - prefill - Pre-filled from configuration

4. Timing and Focus Data

  • Focus time by element: Time spent focused on each UI element
  • Scroll depth: Maximum scroll percentage reached
  • Session timing: Start/end timestamps and total duration
  • Navigation history: Complete path through instances

Data Format

Behavioral Data Structure

Each annotation instance includes a behavioral_data object:

{
  "id": "instance_123",
  "annotations": {
    "sentiment": {"positive": true}
  },
  "behavioral_data": {
    "instance_id": "instance_123",
    "session_start": 1706500000.0,
    "session_end": 1706500045.0,
    "total_time_ms": 45000,
    "interactions": [
      {
        "event_type": "click",
        "timestamp": 1706500002.5,
        "target": "label:positive",
        "instance_id": "instance_123",
        "client_timestamp": 1706500002500,
        "metadata": {"x": 150, "y": 320}
      }
    ],
    "ai_usage": [
      {
        "request_timestamp": 1706500010.0,
        "response_timestamp": 1706500012.5,
        "schema_name": "sentiment",
        "suggestions_shown": ["positive"],
        "suggestion_accepted": "positive",
        "time_to_decision_ms": 3500
      }
    ],
    "annotation_changes": [
      {
        "timestamp": 1706500002.5,
        "schema_name": "sentiment",
        "label_name": "positive",
        "old_label": null,
        "action": "select",
        "old_value": null,
        "new_value": true,
        "source": "user",
        "phase": "annotation",
        "page": null
      }
    ],
    "navigation_history": [
      {
        "action": "next",
        "from_instance": "instance_122",
        "to_instance": "instance_123",
        "timestamp": 1706500000.0
      }
    ],
    "focus_time_by_element": {
      "label:positive": 2500,
      "label:negative": 1200,
      "textbox:explanation": 8000
    },
    "scroll_depth_max": 75.5,
    "keyword_highlights_shown": [
      {"text": "excellent", "label": "positive", "type": "keyword"}
    ],
    "chat_history": [
      {
        "role": "user",
        "content": "Is sarcasm counted as negative here?",
        "timestamp": 1706500020.0,
        "instance_id": "instance_123",
        "response_time_ms": null
      }
    ],
    "typing_summaries": {
      "explanation:::explanation": {
        "keystrokes": 172,
        "final_chars": 154,
        "iki_median_ms": 133.0,
        "pause_counts": {"2000": 2},
        "pasted_fraction": 0.0,
        "verdict": {"level": "ok", "flag_names": []}
      }
    },
    "annotation_telemetry": {
      "objects": {
        "shapes_added": 7,
        "shape_interval_median_ms": 4200.0,
        "revision_ratio": 0.22,
        "max_zoom": 3.5,
        "ai_suggested": 9,
        "ai_accepted": 6,
        "ai_accept_latency_median_ms": 3100.0,
        "verdict": {"flags": [], "scores": {}, "notes": {}}
      }
    }
  }
}

chat_history is populated only when the LLM chat sidebar is enabled. typing_summaries is populated only when keystroke logging is enabled; the raw keystroke streams live in the project's SQLite database rather than here.

annotation_telemetry is populated only when annotation telemetry is enabled, and follows the same split: the summary here, the raw event streams in SQLite. It is keyed by schema name alone rather than schema:::label, because a drawn annotation is stored under a single _data key and there is no label to key on.

Storage Location

Behavioral data is stored in: 1. User state files: annotation_output/<user_id>/user_state.json 2. Exported annotations: Included with annotation data when exported

Configuration

Enabling Behavioral Tracking

Behavioral tracking is enabled by default. No additional configuration is required.

Keystroke logging is a separate, opt-in feature

Fine-grained typing dynamics on free-text fields — pauses, bursts, revisions, paste detection — are not part of this default tracking. They are configured under keystroke_logging, default off, and documented in Keystroke Logging.

Annotation telemetry is also separate and opt-in

Drawing dynamics on geometry schemas — time per shape, zoom behaviour, revision counts, and AI-suggestion accept latency — are configured under annotation_telemetry, default off, and documented in Annotation Telemetry.

Frontend Debug Mode

To enable debug logging for the interaction tracker:

// In browser console
window.interactionTracker.setDebugMode(true);

Flush Interval

The default flush interval is 5 seconds. Events are also flushed on page navigation.

API Endpoints

Track Interactions

POST /api/track_interactions
Content-Type: application/json

{
  "instance_id": "instance_123",
  "events": [...],
  "focus_time": {"element": ms},
  "scroll_depth": 75.5
}

Track AI Usage

POST /api/track_ai_usage
Content-Type: application/json

{
  "instance_id": "instance_123",
  "schema_name": "sentiment",
  "event_type": "request|response|accept|reject",
  "suggestions": [...],
  "accepted_value": "positive"
}

Track Annotation Change

POST /api/track_annotation_change
Content-Type: application/json

{
  "instance_id": "instance_123",
  "schema_name": "sentiment",
  "label_name": "positive",
  "action": "select|deselect|update|clear",
  "old_label": null,
  "old_value": null,
  "new_value": true,
  "source": "user|ai_accept|keyboard|prefill"
}

The workflow phase and page are stamped server-side rather than sent by the client: behavioral data is bucketed by instance id, and every non-annotation page shares the same __phase_page__ sentinel, so without them the trail cannot say which survey a change belongs to.

Get Behavioral Data

GET /api/behavioral_data/<instance_id>

Returns the complete behavioral data for an instance.

Analysis Examples

Loading Behavioral Data

import json
from pathlib import Path

def load_behavioral_data(annotation_dir: str) -> dict:
    """Load all behavioral data from annotation output directory."""
    data = {}

    for user_dir in Path(annotation_dir).iterdir():
        if not user_dir.is_dir():
            continue

        state_file = user_dir / 'user_state.json'
        if state_file.exists():
            with open(state_file) as f:
                user_state = json.load(f)

            user_id = user_state.get('user_id')
            behavioral = user_state.get('instance_id_to_behavioral_data', {})
            data[user_id] = behavioral

    return data

Analyzing Annotation Time

def analyze_annotation_time(behavioral_data: dict) -> dict:
    """Calculate annotation time statistics per user."""
    stats = {}

    for user_id, instances in behavioral_data.items():
        times = []
        for instance_id, bd in instances.items():
            if 'total_time_ms' in bd:
                times.append(bd['total_time_ms'] / 1000)  # Convert to seconds

        if times:
            stats[user_id] = {
                'mean_time': sum(times) / len(times),
                'min_time': min(times),
                'max_time': max(times),
                'total_instances': len(times)
            }

    return stats

AI Assistance Analysis

def analyze_ai_usage(behavioral_data: dict) -> dict:
    """Analyze AI assistance usage patterns."""
    ai_stats = {
        'total_requests': 0,
        'total_accepts': 0,
        'total_rejects': 0,
        'avg_decision_time_ms': 0,
        'by_schema': {}
    }

    decision_times = []

    for user_id, instances in behavioral_data.items():
        for instance_id, bd in instances.items():
            for ai_event in bd.get('ai_usage', []):
                ai_stats['total_requests'] += 1

                schema = ai_event.get('schema_name', 'unknown')
                if schema not in ai_stats['by_schema']:
                    ai_stats['by_schema'][schema] = {'accepts': 0, 'rejects': 0}

                if ai_event.get('suggestion_accepted'):
                    ai_stats['total_accepts'] += 1
                    ai_stats['by_schema'][schema]['accepts'] += 1
                else:
                    ai_stats['total_rejects'] += 1
                    ai_stats['by_schema'][schema]['rejects'] += 1

                if ai_event.get('time_to_decision_ms'):
                    decision_times.append(ai_event['time_to_decision_ms'])

    if decision_times:
        ai_stats['avg_decision_time_ms'] = sum(decision_times) / len(decision_times)

    return ai_stats

Detecting Suspicious Behavior

def detect_suspicious_annotators(behavioral_data: dict,
                                  min_time_threshold: float = 2.0,
                                  min_interactions_threshold: int = 2) -> list:
    """
    Identify annotators with potentially low-quality behavior.

    Flags:
    - Very fast annotation times (< min_time_threshold seconds)
    - Very few interactions per instance
    - No annotation changes (just clicking through)
    """
    suspicious = []

    for user_id, instances in behavioral_data.items():
        fast_count = 0
        low_interaction_count = 0
        no_change_count = 0

        for instance_id, bd in instances.items():
            time_sec = bd.get('total_time_ms', 0) / 1000
            interactions = len(bd.get('interactions', []))
            changes = len(bd.get('annotation_changes', []))

            if time_sec < min_time_threshold:
                fast_count += 1
            if interactions < min_interactions_threshold:
                low_interaction_count += 1
            if changes == 0:
                no_change_count += 1

        total = len(instances)
        if total > 0:
            fast_rate = fast_count / total
            low_rate = low_interaction_count / total
            no_change_rate = no_change_count / total

            if fast_rate > 0.5 or low_rate > 0.5 or no_change_rate > 0.8:
                suspicious.append({
                    'user_id': user_id,
                    'fast_rate': fast_rate,
                    'low_interaction_rate': low_rate,
                    'no_change_rate': no_change_rate,
                    'total_instances': total
                })

    return suspicious

Integration with Admin Dashboard

The Admin Dashboard includes a Behavioral Analytics tab that provides:

  1. User Interaction Heatmap: Visual representation of interaction patterns
  2. AI Assistance Metrics: Accept/reject rates, decision times
  3. Timing Distribution: Histogram of annotation times
  4. Suspicious Activity Alerts: Flagged annotators requiring review

See Admin Dashboard Documentation for details.

Best Practices

For Researchers

  1. Baseline Establishment: Collect behavioral data from known-good annotators to establish baselines
  2. Quality Metrics: Use behavioral data alongside annotation agreement for quality assessment
  3. Training Evaluation: Compare pre- and post-training behavioral patterns
  4. AI Impact Analysis: Measure how AI assistance affects annotation quality and speed

For Annotation Projects

  1. Monitor in Real-Time: Use the admin dashboard to spot issues early
  2. Set Thresholds: Define acceptable ranges for timing and interaction metrics
  3. Provide Feedback: Use behavioral insights to provide targeted annotator feedback
  4. Iterate on Guidelines: Identify confusing instances through interaction patterns

Jupyter Notebook Tutorial

For a hands-on tutorial on analyzing behavioral data, see the example notebook:

examples/behavioral_analysis/analyze_behavioral_data.ipynb

This notebook demonstrates: - Loading and parsing behavioral data - Visualizing interaction patterns - Analyzing AI assistance effectiveness - Detecting quality issues - Generating reports

Troubleshooting

No Behavioral Data Being Collected

  1. Verify interaction_tracker.js is loaded (check browser Network tab)
  2. Check browser console for JavaScript errors
  3. Verify API endpoints are accessible (/api/track_interactions)

Data Not Persisting

  1. Check that user state is being saved (look for user_state.json)
  2. Verify serialization is working (check server logs)
  3. Ensure the annotation output directory is writable

High Network Traffic

  1. Increase the flush interval in interaction_tracker.js
  2. Filter out unnecessary events
  3. Use sampling for high-volume deployments

See Also