Skip to content

Writing-Process Detection

Turns the typing dynamics captured by Keystroke Logging into named, auditable flags: was this response composed, transcribed, or pasted?

Read this before you use any of it

Flags are evidence for human review. They are not proof of misconduct, and they must never be wired to automatic rejection, payment withholding, or participant bans.

A fast, fluent typist producing a clean first draft genuinely resembles transcription. An annotator on a phone genuinely resembles someone pasting. The rules below are built to minimise those collisions, and they still happen. Read the false positives section — it is part of the feature, not a disclaimer.

Three tiers

Tier What it is Needs labeled data? Default
1. Rules Six named flags with explicit thresholds and visible evidence No On
2. Calibration Thresholds refitted to your project's own annotators No Off
3. Supervised A classifier you train on your own labels Yes Off

No pre-trained model ships with Potato

There is no labeled corpus in this repository, so shipping fitted coefficients would be fabricating validation. Tier 1 is transparent rules you can read and argue with. Tier 3 exists for researchers who have real labels — that is the path that reproduces the accuracies in the literature.

Tier 1: the rules

Every flag returns the feature values that fired it, so you can defend or discard any individual one.

paste_dominant · severity: suspect

Fires when pasted_fraction ≥ 0.5 — half or more of the final text arrived by paste.

Suppressed when every paste was classified as self (re-arranging your own draft) or instance_text (quoting the passage under annotation).

silent_insertion · severity: suspect

Fires when external_insert_ratio ≥ 0.3 — nearly a third of inserted characters appeared with no corresponding keystroke.

This is the highest-value single signal. Paste, autofill, dictation and programmatic injection all show up here, even when the paste event itself is suppressed by the page. It is a direct operationalization of Asher et al.'s "keystroke count anomalously low relative to response length".

Suppressed on soft keyboards (virtual_keyboard) and during IME composition, where keydown is not reliably emitted and every insert would look silent. Uses the source-aware ratio, so legitimate quoting does not count.

transcription_rhythm · severity: review

Fires when all three hold:

  • iki_log_cv ≤ 0.06 — metronomic typing rhythm
  • revision_ratio ≤ 0.02 — essentially no deletion
  • fewer than 0.5 pauses ≥2 s per 100 characters

Conjunctive by design. Any one alone has an innocent reading; together they are the copy-typing signature Crossley et al. describe — linear, burst-oriented, low variance.

Skipped entirely for responses under 80 characters or 40 keystrokes, where there is no rhythm to speak of.

offscreen_composition · severity: suspect

Fires when a large externally-sourced insertion (≥80 chars) immediately follows ≥10 s away from the page. The "switched to ChatGPT, came back, pasted" pattern.

Suppressed when the insertion's source was the passage or the annotator's own text — stepping away and then quoting the passage is ordinary behaviour.

implausible_speed · severity: review

Fires above 900 characters/minute (~180 wpm) sustained across a whole response.

synthetic_input · severity: suspect

Fires on any event with isTrusted === false — the browser reporting that input was generated by script rather than a person. Browser automation, injected scripts, some accessibility tooling.

Verdict levels

Level Meaning
ok No flags fired
review At least one review-severity flag
suspect At least one suspect-severity flag

Overriding thresholds

keystroke_logging:
  detection:
    thresholds:
      paste_dominant.pasted_fraction: 0.4
      silent_insertion.ratio: 0.25
      transcription_rhythm.iki_log_cv: 0.05
      transcription_rhythm.revision_ratio: 0.02
      transcription_rhythm.pause_2s_per_100_chars: 0.5
      offscreen_composition.blur_ms: 15000
      offscreen_composition.insert_chars: 100
      implausible_speed.chars_per_min: 1000

Thresholds are evaluated server-side and never sent to the browser — publishing them would tell an annotator exactly how slowly to paste in order to stay under the flag.

Tier 2: project calibration

Keystroke features vary substantially by writing task (Conijn et al. 2019), and fixed thresholds on a mixture-distributed quantity are biased (Roeser et al. 2021). A cutoff that works for a one-sentence rationale is wrong for a five-paragraph essay.

Calibration sets each threshold at a tail percentile of your project's sessions.

# Inspect the fit without saving it
python -m potato.typing_detect calibrate config.yaml --dry-run

# Save it
python -m potato.typing_detect calibrate config.yaml

keystroke_logging:
  detection:
    calibrate: true    # use the saved fit

Requires at least 30 usable sessions (≥80 characters, non-mobile). Below that it returns insufficient_data and the defaults stay in force.

What a calibrated flag does and does not mean

A percentile cutoff is a relative outlier definition. By construction it flags roughly tail_fraction (default 5%) of sessions even in a population where nobody is doing anything wrong. It tells you where to look first. It is not evidence.

Calibrated thresholds are additionally clamped to within 3× the built-in defaults, so a homogeneous population cannot drag a cutoff onto its own median and start flagging honest annotators.

An explicit thresholds: override always beats a calibrated value, which always beats the built-in default.

Tier 3: supervised classifier

If you have labeled sessions, train a real model:

from potato import typing_store
from potato.typing_detect import fit_supervised

rows = typing_store.feature_matrix(task_dir, project)
labels = [...]   # 1 = non-composed, 0 = composed

result = fit_supervised(rows, labels, model="random_forest")
print(result["cv_accuracy_mean"], result["feature_importances"])

Requires scikit-learn (lazily imported; not a Potato dependency).

Getting labels without an external study

Potato's training phase can generate them for you. Ask annotators to copy a supplied passage as a warm-up task: their copying sessions are genuine transcription exemplars, and their normal answers are composed exemplars — a labeled set from within your own project, on your own task, with your own annotators. This mirrors how Crossley et al. built their corpus.

False positives

The failure mode of this feature is accusing an honest annotator. These are the cases that legitimately resemble the patterns above.

Situation Which rule it resembles How Potato handles it
Quoting the passage under annotation paste_dominant, silent_insertion, offscreen_composition Suppressed via paste_source: instance_text
Moving your own draft around Same Suppressed via paste_source: self
Typing on a phone or tablet silent_insertion Suppressed via virtual_keyboard
Non-Latin input via an IME silent_insertion Suppressed via composition_events
Fast, fluent typist, clean draft transcription_rhythm, implausible_speed Conjunctive rules; review not suspect; calibration
Very short answers transcription_rhythm Skipped below 80 chars / 40 keystrokes
Dictation / speech-to-text silent_insertion Not handled. Will flag. Exclude those annotators or raise the threshold.
Screen readers and some AT synthetic_input Not fully handled. Some tooling produces untrusted events.
Autocorrect on mobile silent_insertion Partly — insertReplacementText counts as external.
Browser extensions (grammar tools) silent_insertion, synthetic_input Not handled. Will flag.

The last four are real limitations, not oversights we forgot to mention. If your participant pool includes dictation users, assistive-technology users, or people who use writing extensions, either exclude those rules or treat every flag as a prompt to ask the annotator rather than to act.

Accessibility

synthetic_input keys off isTrusted, which some assistive technologies also trip. Flagging a disabled annotator for using the tools they need to work is an unacceptable outcome. If your study is open to AT users, disable that rule:

keystroke_logging:
  detection:
    thresholds:
      # There is no explicit off switch; make it unreachable.
      implausible_speed.chars_per_min: 1000000

or set detection.enabled: false and analyse the exported features yourself.

Real-time intervention

keystroke_logging:
  detection:
    on_external_insert: flag   # allow | warn | block | flag
Value Behaviour
allow Do nothing beyond recording
warn Non-blocking notice on external paste (not on self/passage quotes)
block Prevent paste and drop into instrumented fields
flag Default. Record silently; you decide later

block is a blunt instrument: it also blocks legitimate quoting, and a determined participant can retype instead. flag plus review is usually better.

Admin dashboard

The Writing Process panel under the Behavioral tab shows per-annotator medians, paste rates, silent-insertion rates, and every flagged session with its evidence.

It reports writing_process_risk — the share of a user's sessions that fired each flag, weighted toward the signals with the least innocent explanation. It is a ranking aid, deliberately kept separate from the existing suspicion_score so that neither number silently changes the other's meaning.

Research grounding

Every citation verified against the Crossref or DataCite registry.

Detection. Crossley, Tian, Choi, Holmes & Morris (2024) collected 500 argumentative essays, had other workers transcribe them, and separated authentic from transcribed at 99% accuracy with a random forest (96–98% for other models). Their feature families — pause times before sentences and words, insertion/deletion counts, product-to-process ratios, bursts, revision, process variance — are what Potato captures. Their finding is the design target: authentic writing shows longer pauses, more insertions and deletions, and greater variance; transcription is linear and burst-oriented.

Deane, Zhang, Hao & Li and Zhang, Feng, He, Li & Zhu independently confirm the copy-typing/natural-writing separation, the latter with a deep-learning model. Asher, Gold, Chen & Carvalho is the crowdsourcing-specific case: a keystroke tool on Prolific flagging participants who paste into response fields or whose keystroke count is anomalously low for their response length.

Process measures. Leijten & Van Waes (Inputlog) for the standard log measures; Chenoweth & Hayes for the burst construct; Conijn, Roeser & van Zaanen for task-dependence of keystroke features; Roeser, De Maeyer, Leijten & Van Waes for why fixed pause thresholds are biased.

References

  1. Crossley, S., Tian, Y., Choi, J. S., Holmes, L., & Morris, W. (2024). Plagiarism Detection Using Keystroke Logs. EDM 2024 (Short Papers). doi:10.5281/zenodo.12729864
  2. Deane, P., Zhang, M., Hao, J., & Li, C. Using Keystroke Dynamics to Detect Nonoriginal Text. Journal of Educational Measurement, 63(1). doi:10.1111/jedm.12431
  3. Asher, M. W., Gold, G., Chen, E., & Carvalho, P. F. (2026). Chatbots Are Undermining Crowdsourced Research in the Behavioral Sciences: Detecting Artificial Intelligence–Assisted Cheating With a Keystroke-Based Tool. Advances in Methods and Practices in Psychological Science, 9(1). doi:10.1177/25152459261424723
  4. Zhang, M., Feng, L., He, X., Li, C., & Zhu, M. (2026). Disentangling copy typing and natural writing behaviors using keystroke logs and deep learning model. Assessing Writing. doi:10.1016/j.asw.2026.101070
  5. Leijten, M., & Van Waes, L. (2013). Keystroke Logging in Writing Research: Using Inputlog to Analyze and Visualize Writing Processes. Written Communication, 30(3), 358–392. doi:10.1177/0741088313491692
  6. Chenoweth, N. A., & Hayes, J. R. (2001). Fluency in Writing: Generating Text in L1 and L2. Written Communication, 18(1), 80–98. doi:10.1177/0741088301018001004
  7. Conijn, R., Roeser, J., & van Zaanen, M. (2019). Understanding the keystroke log: the effect of writing task on keystroke features. Reading and Writing, 32(9), 2353–2374. doi:10.1007/s11145-019-09953-8
  8. Roeser, J., De Maeyer, S., Leijten, M., & Van Waes, L. (2021). Modelling typing disfluencies as finite mixture process. Reading and Writing. doi:10.1007/s11145-021-10203-z
  9. Lee, M., Liang, P., & Yang, Q. (2022). CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. CHI 2022. doi:10.1145/3491102.3502030