Machine Annotators¶
A machine annotator is a rater that is a program rather than a person: an annotation pipeline, a detector, a language model. Declaring one lets Potato measure it as a rater — computing agreement between tools, routing their conflicts to a human, estimating each one's reliability — without counting it as a person in statistics that describe people.
This matters when several automated systems label the same data and no ground truth exists to say which is right. Benchmarks of genome annotation pipelines, OCR engines, ASR systems and LLM extractors all have that shape: the tools disagree, the disagreement is measurable, and the question of who was right on any given item goes unanswered.
Why a declaration is needed¶
Without a declaration, machine output reaches Potato by one of two paths, and neither measures it as a rater.
The importers refuse to write machine output into a user state at all, because a fabricated user "would make machine output indistinguishable from human work and fabricate agreement between annotators who never touched the item." Imported predictions live in item data as pre-annotations, where no agreement, adjudication or psychometric code can reach them.
The simulator does the opposite. It registers over HTTP, authenticates and
saves through the ordinary route, so an LLM-driven annotator gets a real user
state — and is counted as a person by /admin/iaa, MACE and the IRT engine,
because nothing on the state says it is not one.
A declaration fixes both. The importers' guard is against machine output posing as a person, and a declared rater is a machine rater on the record.
Configuration¶
machine_annotators:
enabled: true
annotators:
- id: prokka # the user id this rater annotates under
kind: tool # tool | llm
tool: prokka
version: "1.14.6"
database_version: "2023-05"
run_date: "2024-11-03"
- id: gpt4o_judge
kind: llm
endpoint_type: openai
model: gpt-4o
version: "2024-08-06"
| Field | Meaning |
|---|---|
id |
The user id the rater annotates under. This is the join key between the declaration and the state on disk. |
kind |
tool or llm. Anything else is accepted, logged, and treated as a machine — not being human is what the metrics depend on. |
tool, model, endpoint_type |
Which program produced the answers. |
version, database_version, run_date |
Which version of it, and when. |
Fill in the version fields. A benchmark of four genome annotation tools could not analyse how annotations drifted between releases because the prior annotation's date was never recorded: a re-run that finds 139 more features cannot be told apart from a reference database that gained them.
Absent means human¶
A participant with no declaration is a person. Every user state written before
this feature existed loads as human, so adding the block changes nothing for
raters you do not name, and a study with no machine_annotators block produces
the same output it produced before.
Disabling the block stops new stamping. It does not unmake origins already written to disk, because a rater that was a tool last week was still a tool.
Effects on the measurement surfaces¶
Agreement¶
/admin/iaa reports three partitions instead of one:
| Partition | What it measures |
|---|---|
human_human |
Inter-human reliability. This is the headline. |
machine_machine |
Whether the tools agree with each other. |
pooled |
Everyone in one matrix. |
The headline figures cover humans only. The top-level number is read as inter-human reliability, so if it pooled everyone, a run of the LLM simulator against a live study would change it with nothing on the page to say why.
The pooled figure is still reported, under a name that says what it mixes. A coefficient over a mixed human/machine reliability matrix is not an estimate of either group's reliability — two groups that are each perfectly self-consistent can pool to worse than chance, because the number is then describing the distance between them.
The admin page says which raters were excluded and which versions produced them, so an admin can see why the annotator count dropped.
/admin/api/agreement excludes machine raters for the same reason and lists
them under machine_annotators_excluded.
MACE¶
MACE estimates per-annotator competence, and its output is used to decide whose labels to trust. Declared machines are excluded with no opt-in: a competence score for an annotation pipeline is a different claim from a competence score for a person, and mixing them puts a tool's number beside a colleague's.
Psychometrics (IRT)¶
The IRT engine is the exception, because modelling a tool's ability is the point:
psychometrics:
enabled: true
schema: sentiment
include_machine_annotators: true
Off by default, so an existing study's estimates do not move when a machine
annotator is declared. On, the model fits an ability per rater and a difficulty
per item from the disagreement pattern alone, with no gold labels — which is
how you get a reliability estimate per tool on a corpus that has no ground
truth. Annotator rows carry an origin label for machine raters.
Correlated error
GLAD assumes raters err independently given the true label. Tools that share a reference database do not: they make the same mistake together, which inflates the apparent ability of whichever cluster is largest. An ability estimate over machine raters needs someone who knows which tools share evidence before it is published. The psychometrics dashboard says so above the ability chart whenever a machine rater is in it, and marks each machine's row.
Gold auto-promotion¶
Machine raters never count toward gold_standards.auto_promote, and no setting
lets them. Gold is what later annotators are graded against, and agreement
between tools that share evidence is correlated error. Enough people agreeing
still promotes an item, whatever the tools answered.
Adjudication¶
adjudication:
enabled: true
adjudicator_users: [curator]
include_machine_annotators: true
min_human_annotations: 0
Machines stay out of the queue unless include_machine_annotators is set,
which extends the exclusion the queue already applies to adjudicators. When
they are in, each of their answers carries a badge (TOOL or LLM) on its
response card and on its chip in the decision form; hovering it shows the tool
or model and its version. Under show_annotator_names: false the badge still
appears beside the alias, because knowing an answer came from a tool is part
of judging it.
min_human_annotations is a floor on people, counted separately from the
participant total. Zero is allowed: several tools and no human
annotator is a valid queue, because the adjudicator is the person in that
design.
Queue items carry annotator_origins, so a curator choosing between two labels
can see that one came from a tool and which version produced it. Each decision
records source_origins: per schema, which raters had given the answer the
adjudicator settled on, e.g. pipeline_a 2.1.0 db refdb-2024-03 (tool);
pipeline_b 1.8.2 db refdb-2024-03 (tool). A label only two tools that share a
database ever gave is a different result from one the curators gave. It is
recorded when the decision is made, so a later config edit cannot change what
a past decision appears to have meant.
The final dataset distinguishes who agreed. A study with no machine rater still
writes source: "unanimous"; one with machine raters writes
unanimous_machine or unanimous_mixed where they apply, because several
tools agreeing is not the same as several people agreeing.
Export¶
Once any machine rater has annotated, the per-annotator export formats mark
every row with who produced it. csv, tsv, parquet, HuggingFace and the publish
bundle get an annotator_origin column (human, tool or llm) and an
annotator_origin_detail column holding the declaration as JSON; jsonl gets
the whole declaration under annotator_origin. The columns sit next to
user_id:
instance_id,user_id,annotator_origin,annotator_origin_detail,function_category.information_processing,...
TOY_00030,curator_1,human,,,...
TOY_00030,pipeline_a,tool,"{""database_version"": ""refdb-2024-03"", ""declared_in"": ""config"", ""id"": ""pipeline_a"", ...}",,...
A study with no machine rater exports without them. An export run before the server has loaded a declared rater's state (right after an import, for example) still labels it, from the config.
The adjudication outputs record what was adopted: adjudicated.csv adds a
source_origin column beside source, and the potato.adjudication_export
dataset carries source_origins on each adjudicated item. See
Export Formats.
Getting machine output in¶
From a run you already have¶
Import the predictions and declare the seeded rater:
python -m potato.importers \
--input-format coco --input predictions.json \
--output-dir myproject \
--seed-user prokka --as-machine-annotator tool
The seeded state then carries an origin. Declare the same id under
machine_annotators in the project config so the server keeps the declaration
when it reloads that state.
Without --as-machine-annotator, --seed-user still fabricates an
undeclared annotator and still warns that its rows must be kept out of
agreement analysis.
From a model running against the server¶
The simulator's llm and agent strategies drive a real endpoint. Declare the
user ids it will use:
machine_annotators:
enabled: true
annotators:
- {id: sim_user_0000, kind: llm, model: my-model}
- {id: sim_user_0001, kind: llm, model: my-model}
The simulator warns at startup when a model-backed strategy is selected, naming the ids that should be declared. It cannot fix the problem itself — the roster lives in the target server's config, not the simulator's.
Requiring every participant to be declared¶
By default an undeclared participant is treated as a person, which is what makes the feature safe to add to a running study. A study that wants the stronger guarantee can refuse annotations from anyone it does not recognise:
machine_annotators:
enabled: true
require_declaration: true
annotators:
- {id: prokka, kind: tool, version: "1.14.6"}
user_config:
allow_all_users: false
users: [alice, bob]
A write is refused when the participant is neither declared under
machine_annotators.annotators nor on the human roster, and the annotator is
told to ask the administrator rather than being left with a save that silently
did nothing.
Two deliberate limits:
It is checked where annotations are written, not at login, and never from a request header. A header asserting "I am a machine" is supplied by the caller, so honouring one would let anybody opt out of the human count — the opposite of the guarantee. Only the study's config can declare an origin.
It requires a roster to compare against, and says so at load. Under open
registration every username that registers becomes a valid account, so the
setting would have nothing to refuse, so the config is rejected at startup unless
user_config.allow_all_users is false and a roster is named — inline under
user_config.users, or as a file at authentication.user_config_path.
Consent, instruction and survey pages are exempt. Those are not dataset annotations, and a rater that never reaches an item has nothing to contaminate.
Prioritising what the tools disagree about¶
Cross-tool disagreement is a per-item number, so it drives the existing triage
queue with no new machinery. Compute it when the data is built, put it on the
item, and point triage at the field:
triage:
enabled: true
signal_field: tool_disagreement
assignment_strategy: priority
signal_field is read as items are added, before any annotation exists, so it
works for a batch import: the contested items are served first and a human never opens the ones every tool already agreed on.
Can the machine replace an annotator?¶
Agreement between a machine and your annotators does not answer this on its own: a model can agree with the majority 80% of the time while each person agrees with the rest even more often. The Alternative Annotator Test (Calderon, Reichart and Dror, ACL 2025) asks the question directly. For each human annotator in turn, it compares how well the machine matches the other annotators with how well that annotator does, and tests whether the machine is at least as good, within a margin ε. The machine passes when it does as well as at least half the annotators after a Benjamini–Yekutieli correction across them.
GET /admin/api/alt-test?annotator=<machine user id>&schema=<scheme name>&epsilon=0.2
The request needs the X-API-Key header with admin_api_key. Called without annotator or schema, it returns HTTP 400 with the declared machine annotators and the scheme names to choose from.
| Parameter | Meaning |
|---|---|
annotator |
The machine's user id. It has to be a declared machine annotator; a person is refused |
schema |
The scheme to test on: radio and select are scored by exact match, likert, slider and number by negative RMSE |
epsilon |
How much better than the machine an annotator has to be before the machine fails against them. The paper suggests 0.2 for experts, 0.15 for skilled annotators and 0.1 for crowd workers. Default 0.2 |
The response gives passed, the winning rate winning_rate (the share of annotators the machine matched), the average advantage probability advantage_probability, and a row per annotator with their item count, p-value and adjusted p-value. It reads labels the same way /admin/iaa does.
The test needs enough data to mean anything. By default each human annotator needs at least 30 items that the machine also labelled, and at least three annotators have to qualify. Below that, passed is null and insufficient says what is missing; annotators with too few items are listed under skipped_annotators. A Solo Mode study has one human annotator and cannot run the test. Its blind agreement is the measure there.
Example project¶
examples/advanced/machine-annotators/ is a synthetic gene-function curation
task: three annotation pipelines and one LLM have labelled twelve predicted
genes, and two curators have labelled eight of them. It shows agreement by
partition, triage on tool disagreement, adjudication with machine raters, IRT
over tools, and the export columns. It leaves require_declaration and MACE
off.
python examples/advanced/machine-annotators/setup_demo.py
python potato/flask_server.py start examples/advanced/machine-annotators/config.yaml -p 8000
Log in with any new username to curate (the contested loci come first), as
lead_curator to open /adjudicate, and open /admin/iaa?format=html with
the admin key demo-admin-key for the agreement report.
Related¶
- Heterogeneous Coverage — overlap samples, adaptive boost, and the agreement report
- Psychometrics — the IRT model and adaptive routing
- MACE — competence estimation over human annotators
- Task Assignment — assignment strategies including
priority