Crowdsourcing Guide

This guide walks you through running annotation tasks on crowdsourcing platforms like Prolific and Amazon MTurk.

Platform Setup

• Crowdsourcing Guide - General setup for Prolific and MTurk integration
• MTurk Integration - Detailed Amazon MTurk HIT setup, payment, and management

Authentication for Crowd Workers

For crowdsourcing, you typically want low-friction authentication:

• Passwordless Login - Workers access tasks without passwords (recommended for most crowd tasks)
• SSO Authentication - Institutional SSO for studies requiring verified identity

Platform-specific authentication (Prolific IDs, MTurk Worker IDs) is handled automatically when using the crowdsourcing integration.

Quality Assurance for Crowd Tasks

Quality control is especially important with crowd workers:

• Quality Control - Attention checks and gold standard items to verify engagement
• Training Phase - Qualification training before the real task (filter unqualified workers)
• Adjudication - Resolve disagreements between multiple annotators
• MACE - Statistical estimation of annotator competence and label recovery

Task Design Best Practices

• Choosing Annotation Types - Select appropriate schemas for non-expert annotators
• Form Layout - Design clear, easy-to-use form layouts
• Conditional Logic - Adaptive forms that simplify complex tasks
• Survey Instruments - 55 pre-built validated instruments for demographic and psychological surveys

Multi-Phase Workflows

Set up consent, instructions, training, annotation, and post-study surveys:

• Multi-Phase Workflows - Configure the full workflow pipeline

Deployment

• Installation & Usage - Server setup and configuration
• HuggingFace Spaces - Deploy on HuggingFace Spaces (free hosting)

Monitoring and Export

• Admin Dashboard - Monitor progress, track completions, detect suspicious activity
• Behavioral Tracking - Timing analysis and engagement metrics
• Export Formats - Export results in multiple formats