Skip to content

Machine Annotators

A machine annotator is a rater that is a program rather than a person: an annotation pipeline, a detector, a language model. Declaring one lets Potato measure it as a rater — computing agreement between tools, routing their conflicts to a human, estimating each one's reliability — without counting it as a person in statistics that describe people.

This matters when several automated systems label the same data and no ground truth exists to say which is right. Benchmarks of genome annotation pipelines, OCR engines, ASR systems and LLM extractors all have that shape: the tools disagree, the disagreement is measurable, and the question of who was right on any given item goes unanswered.

Why a declaration is needed

Without a declaration, machine output reaches Potato by one of two paths, and neither measures it as a rater.

The importers refuse to write machine output into a user state at all, because a fabricated user "would make machine output indistinguishable from human work and fabricate agreement between annotators who never touched the item." Imported predictions live in item data as pre-annotations, where no agreement, adjudication or psychometric code can reach them.

The simulator does the opposite. It registers over HTTP, authenticates and saves through the ordinary route, so an LLM-driven annotator gets a real user state — and is counted as a person by /admin/iaa, MACE and the IRT engine, because nothing on the state says it is not one.

A declaration fixes both. The importers' guard is against machine output posing as a person, and a declared rater is a machine rater on the record.

Configuration

machine_annotators:
  enabled: true
  annotators:
    - id: prokka                  # the user id this rater annotates under
      kind: tool                  # tool | llm
      tool: prokka
      version: "1.14.6"
      database_version: "2023-05"
      run_date: "2024-11-03"

    - id: gpt4o_judge
      kind: llm
      endpoint_type: openai
      model: gpt-4o
      version: "2024-08-06"
Field Meaning
id The user id the rater annotates under. This is the join key between the declaration and the state on disk.
kind tool or llm. Anything else is accepted, logged, and treated as a machine — not being human is what the metrics depend on.
tool, model, endpoint_type Which program produced the answers.
version, database_version, run_date Which version of it, and when.

Fill in the version fields. A benchmark of four genome annotation tools could not analyse how annotations drifted between releases because the prior annotation's date was never recorded: a re-run that finds 139 more features cannot be told apart from a reference database that gained them.

Absent means human

A participant with no declaration is a person. Every user state written before this feature existed loads as human, so adding the block changes nothing for raters you do not name, and a study with no machine_annotators block produces the same output it produced before.

Disabling the block stops new stamping. It does not unmake origins already written to disk, because a rater that was a tool last week was still a tool.

Effects on the measurement surfaces

Agreement

/admin/iaa reports three partitions instead of one:

Partition What it measures
human_human Inter-human reliability. This is the headline.
machine_machine Whether the tools agree with each other.
pooled Everyone in one matrix.

The headline figures cover humans only. The top-level number is read as inter-human reliability, so if it pooled everyone, a run of the LLM simulator against a live study would change it with nothing on the page to say why.

The pooled figure is still reported, under a name that says what it mixes. A coefficient over a mixed human/machine reliability matrix is not an estimate of either group's reliability — two groups that are each perfectly self-consistent can pool to worse than chance, because the number is then describing the distance between them.

The admin page says which raters were excluded and which versions produced them, so an admin can see why the annotator count dropped.

/admin/api/agreement excludes machine raters for the same reason and lists them under machine_annotators_excluded.

MACE

MACE estimates per-annotator competence, and its output is used to decide whose labels to trust. Declared machines are excluded with no opt-in: a competence score for an annotation pipeline is a different claim from a competence score for a person, and mixing them puts a tool's number beside a colleague's.

Psychometrics (IRT)

The IRT engine is the exception, because modelling a tool's ability is the point:

psychometrics:
  enabled: true
  schema: sentiment
  include_machine_annotators: true

Off by default, so an existing study's estimates do not move when a machine annotator is declared. On, the model fits an ability per rater and a difficulty per item from the disagreement pattern alone, with no gold labels — which is how you get a reliability estimate per tool on a corpus that has no ground truth. Annotator rows carry an origin label for machine raters.

Correlated error

GLAD assumes raters err independently given the true label. Tools that share a reference database do not: they make the same mistake together, which inflates the apparent ability of whichever cluster is largest. An ability estimate over machine raters needs someone who knows which tools share evidence before it is published. The psychometrics dashboard says so above the ability chart whenever a machine rater is in it, and marks each machine's row.

Gold auto-promotion

Machine raters never count toward gold_standards.auto_promote, and no setting lets them. Gold is what later annotators are graded against, and agreement between tools that share evidence is correlated error. Enough people agreeing still promotes an item, whatever the tools answered.

Adjudication

adjudication:
  enabled: true
  adjudicator_users: [curator]
  include_machine_annotators: true
  min_human_annotations: 0

Machines stay out of the queue unless include_machine_annotators is set, which extends the exclusion the queue already applies to adjudicators. When they are in, each of their answers carries a badge (TOOL or LLM) on its response card and on its chip in the decision form; hovering it shows the tool or model and its version. Under show_annotator_names: false the badge still appears beside the alias, because knowing an answer came from a tool is part of judging it.

min_human_annotations is a floor on people, counted separately from the participant total. Zero is allowed: several tools and no human annotator is a valid queue, because the adjudicator is the person in that design.

Queue items carry annotator_origins, so a curator choosing between two labels can see that one came from a tool and which version produced it. Each decision records source_origins: per schema, which raters had given the answer the adjudicator settled on, e.g. pipeline_a 2.1.0 db refdb-2024-03 (tool); pipeline_b 1.8.2 db refdb-2024-03 (tool). A label only two tools that share a database ever gave is a different result from one the curators gave. It is recorded when the decision is made, so a later config edit cannot change what a past decision appears to have meant.

The final dataset distinguishes who agreed. A study with no machine rater still writes source: "unanimous"; one with machine raters writes unanimous_machine or unanimous_mixed where they apply, because several tools agreeing is not the same as several people agreeing.

Export

Once any machine rater has annotated, the per-annotator export formats mark every row with who produced it. csv, tsv, parquet, HuggingFace and the publish bundle get an annotator_origin column (human, tool or llm) and an annotator_origin_detail column holding the declaration as JSON; jsonl gets the whole declaration under annotator_origin. The columns sit next to user_id:

instance_id,user_id,annotator_origin,annotator_origin_detail,function_category.information_processing,...
TOY_00030,curator_1,human,,,...
TOY_00030,pipeline_a,tool,"{""database_version"": ""refdb-2024-03"", ""declared_in"": ""config"", ""id"": ""pipeline_a"", ...}",,...

A study with no machine rater exports without them. An export run before the server has loaded a declared rater's state (right after an import, for example) still labels it, from the config.

The adjudication outputs record what was adopted: adjudicated.csv adds a source_origin column beside source, and the potato.adjudication_export dataset carries source_origins on each adjudicated item. See Export Formats.

Getting machine output in

From a run you already have

Import the predictions and declare the seeded rater:

python -m potato.importers \
  --input-format coco --input predictions.json \
  --output-dir myproject \
  --seed-user prokka --as-machine-annotator tool

The seeded state then carries an origin. Declare the same id under machine_annotators in the project config so the server keeps the declaration when it reloads that state.

Without --as-machine-annotator, --seed-user still fabricates an undeclared annotator and still warns that its rows must be kept out of agreement analysis.

From a model running against the server

The simulator's llm and agent strategies drive a real endpoint. Declare the user ids it will use:

machine_annotators:
  enabled: true
  annotators:
    - {id: sim_user_0000, kind: llm, model: my-model}
    - {id: sim_user_0001, kind: llm, model: my-model}

The simulator warns at startup when a model-backed strategy is selected, naming the ids that should be declared. It cannot fix the problem itself — the roster lives in the target server's config, not the simulator's.

Requiring every participant to be declared

By default an undeclared participant is treated as a person, which is what makes the feature safe to add to a running study. A study that wants the stronger guarantee can refuse annotations from anyone it does not recognise:

machine_annotators:
  enabled: true
  require_declaration: true
  annotators:
    - {id: prokka, kind: tool, version: "1.14.6"}

user_config:
  allow_all_users: false
  users: [alice, bob]

A write is refused when the participant is neither declared under machine_annotators.annotators nor on the human roster, and the annotator is told to ask the administrator rather than being left with a save that silently did nothing.

Two deliberate limits:

It is checked where annotations are written, not at login, and never from a request header. A header asserting "I am a machine" is supplied by the caller, so honouring one would let anybody opt out of the human count — the opposite of the guarantee. Only the study's config can declare an origin.

It requires a roster to compare against, and says so at load. Under open registration every username that registers becomes a valid account, so the setting would have nothing to refuse, so the config is rejected at startup unless user_config.allow_all_users is false and a roster is named — inline under user_config.users, or as a file at authentication.user_config_path.

Consent, instruction and survey pages are exempt. Those are not dataset annotations, and a rater that never reaches an item has nothing to contaminate.

Prioritising what the tools disagree about

Cross-tool disagreement is a per-item number, so it drives the existing triage queue with no new machinery. Compute it when the data is built, put it on the item, and point triage at the field:

triage:
  enabled: true
  signal_field: tool_disagreement

assignment_strategy: priority

signal_field is read as items are added, before any annotation exists, so it works for a batch import: the contested items are served first and a human never opens the ones every tool already agreed on.

Can the machine replace an annotator?

Agreement between a machine and your annotators does not answer this on its own: a model can agree with the majority 80% of the time while each person agrees with the rest even more often. The Alternative Annotator Test (Calderon, Reichart and Dror, ACL 2025) asks the question directly. For each human annotator in turn, it compares how well the machine matches the other annotators with how well that annotator does, and tests whether the machine is at least as good, within a margin ε. The machine passes when it does as well as at least half the annotators after a Benjamini–Yekutieli correction across them.

GET /admin/api/alt-test?annotator=<machine user id>&schema=<scheme name>&epsilon=0.2

The request needs the X-API-Key header with admin_api_key. Called without annotator or schema, it returns HTTP 400 with the declared machine annotators and the scheme names to choose from.

Parameter Meaning
annotator The machine's user id. It has to be a declared machine annotator; a person is refused
schema The scheme to test on: radio and select are scored by exact match, likert, slider and number by negative RMSE
epsilon How much better than the machine an annotator has to be before the machine fails against them. The paper suggests 0.2 for experts, 0.15 for skilled annotators and 0.1 for crowd workers. Default 0.2

The response gives passed, the winning rate winning_rate (the share of annotators the machine matched), the average advantage probability advantage_probability, and a row per annotator with their item count, p-value and adjusted p-value. It reads labels the same way /admin/iaa does.

The test needs enough data to mean anything. By default each human annotator needs at least 30 items that the machine also labelled, and at least three annotators have to qualify. Below that, passed is null and insufficient says what is missing; annotators with too few items are listed under skipped_annotators. A Solo Mode study has one human annotator and cannot run the test. Its blind agreement is the measure there.

Example project

examples/advanced/machine-annotators/ is a synthetic gene-function curation task: three annotation pipelines and one LLM have labelled twelve predicted genes, and two curators have labelled eight of them. It shows agreement by partition, triage on tool disagreement, adjudication with machine raters, IRT over tools, and the export columns. It leaves require_declaration and MACE off.

python examples/advanced/machine-annotators/setup_demo.py
python potato/flask_server.py start examples/advanced/machine-annotators/config.yaml -p 8000

Log in with any new username to curate (the contested loci come first), as lead_curator to open /adjudicate, and open /admin/iaa?format=html with the admin key demo-admin-key for the agreement report.