How Kerata decides, and how often it's wrong

What the score means

The AI likelihood is a calibrated probability that the code was written by an AI model rather than a person, assuming a 50/50 prior. It is computed for the whole file and for overlapping ~40-line windows. Files with fewer than about 15 substantive lines get “too short to judge” instead of a score. On the displayed scale, “likely AI” starts at 96% and “mixed / uncertain” at 44%; below that the code reads as human-written.

The “likely AI” cutoff is chosen on held-out human code from several populations (GitHub projects old and new, student homework, competitive programming) so that about 1 in 100 human files crosses it in each population. That protects people from false accusations at the cost of recall: a fair share of AI-written files land in “mixed / uncertain”.

Kerata only scores source code. Prose (an essay, a cover letter) is refused with “This doesn't look like source code” rather than given a score the model was never trained to produce.

Measured accuracy

Share of files called “likely AI” at the served cutoff, on data the model never trained on:

Test sets are group-disjoint from training: the same repository or task never appears in both.

Class checks: cutoffs for student homework

When Kerata knows it is looking at homework (a class check), it holds the same guarantee against human student code (classroom submissions, competitive programming, beginner exercises) instead of against the hardest population of all, which is professional repository code. In class checks “likely AI” starts at 61% and “mixed” at 24%. Pasted code and repository scans keep the general cutoffs above.

Ranking quality (AUROC)

How well the score ranks AI-written files above human-written ones, independent of any cutoff (1.0 = perfect):

Robustness

• “Contains AI-like section” fires on none of the 385 pure human files in the held-out test. It needs a run of at least 6 lines that each look strongly AI-written, inside a file that itself scores at least “mixed”.
• Line highlighting: median within-file AUROC 0.85 on real files partly edited by AI. Line shading is relative within a file: it shows which lines look most AI-like, not a probability per line.
• Finding a pasted-in AI block (as an “AI-like section”): 23% of blocks under 12 lines, 59% of 12–24 lines, 64% of 25+ lines. Short pastes and light AI edits of a person's own code are often missed; when a section is reported, at least 85% of its lines were AI-written in every held-out edit set.

Robustness tests against stripped comments and against AI models held out from training were run on an earlier version of the model; they are being repeated for the current one and aren't quoted here until then.

The models

• File score: a fine-tuned ModernBERT code encoder combined with a stylometric model, then calibrated so the percentage means what it says.
• Line highlighting: a token-classification tagger that rates each line relative to the rest of the file. Lines are only shaded when the file's verdict is “likely AI”, “mixed” or contains an AI-like section; blank and import-only lines are never shaded.
• AI-like sections: a run of 6+ strongly AI-like lines found by the line tagger, reported only when the file itself scores at least “mixed”.
• Most similar to: a vendor head (OpenAI / Anthropic / open-weights), shown only when the verdict is “likely AI”, with a strong / moderate / weak tier instead of a raw number. It's a lead, not an identification.
• Repositories: per-file scores weighted by length (files too short to judge don't count toward the total), plus README and scaffold signals, shown separately.

Training data

Human code from GitHub (including a 2021–22 slice so “old code = human” isn't a shortcut, and repositories that use code formatters, all last changed before ChatGPT), student homework and competitive programming. AI code from 20+ OpenAI models, Claude, Gemini and open-weights models (DeepSeek, Qwen, Kimi, Mistral, gpt-oss): written from tasks, re-implemented from specs of real human files, written or edited inside real repositories the way coding agents work, and as AI “fixes” of real student submissions. Plus files from public repositories whose every commit was made with an AI coding agent (Claude Code, Codex, Devin, Copilot, Cursor, Jules), and the public DroidCollection and CoDET-M4 datasets.

Before any model sees the code, indentation is normalised (tabs, 2 or 4 spaces all become one form), and training includes Prettier / Black re-formatted copies, so a formatter's style can't decide the verdict.

Known limitations

• Short files are noisy; that's why they get “too short to judge”.
• Heavily human-edited AI code drifts toward human: the detector measures style, not provenance.
• Idiomatic, dense code (for example research-style Python) can look human even when AI-assisted, and very textbook-like human code can look AI-like.
• Re-implementations from a spec are the hardest case: the AI copies a person's structure.
• Languages outside the evaluation (for example C# and Dart) are accepted but not measured.
• Brand-new models may score lower until the detector is retrained.

How to use a result fairly

1. Read the highlighted code yourself. Does it fit what you know about the author and the task?
2. Look for corroborating evidence: edit history, similarity to other submissions, a conversation about the code.
3. Give the author a chance to explain. A score alone should never decide a grade, a hire or a sanction.