What the Rating Actually Measures

FramingThe brief, and the trap in it

A company runs performance reviews across roughly two thousand people. Managers rate their reports on a five-point scale. The evidence about what each person actually did is scattered across five systems, the ticket tracker, the code host, the KPI warehouse, the weekly self-updates, and the HR roster, and nobody has the time to read it, so the rating is made mostly from impression and whatever is easiest to recall.

The request, verbatim in spirit, was: "we should use AI to make performance reviews more objective and consistent."

So what I built is not a rating engine. It is a pre-conversation brief and a review queue: for each person it gathers the evidence, scores it against a published rubric, and surfaces where the manager's rating and the evidence disagree, with the specific gap named. It has no write path into anyone's rating and never proposes one. The most useful thing it produces on a quarter of the roster is the sentence "I cannot score this person fairly, and here is exactly why."

This started life as a take-home design-and-build exercise. I did not join the company and signed nothing, so it is published here with the client unnamed and with only the synthetic dataset that came with the exercise. Every person, email, rating and KPI below is fabricated test data; the employee IDs (E0047, E0073, and so on) are the dataset's own placeholders, not real people. The repository is private for the same reason; see how to read the code at the end.

Part 1What the rating actually measures

Before designing anything I ran read-only analyses over the raw data. The headline is uncomfortable and it is the reason the rest of the design looks the way it does.

Rank correlation (Spearman rho), across the people whose KPI row could be attributed to them unambiguously:

Read the first two rows against the third and fourth. The two measures that best predict the rating a person receives are the two with almost nothing to do with their business result. The rating is not random, at +0.26 against the result it carries real signal, and any replacement has to beat that rather than dismiss it. But roughly twice as much of it is explained by writing volume as by outcomes.

It gets worse when you read the text. The weekly-update corpus is 444 updates built from 47 distinct sentences, and eight filler templates account for 76% of every sentence written:

"I proactively drove clarity on priorities and helped
 everyone move in the same direction."
"I championed best practices and helped raise the bar
 on quality across the board."
"I spent meaningful time unblocking dependencies…"

Against a handful of concrete ones, "Onboarded 3 merchants; hit quota." · "Cut report runtime from 40s to 6s." Longer updates are longer because they repeat the filler, often verbatim within one update. So length is not merely uninformative, it is inversely related to substance while being the single strongest predictor of the rating. Any design that rewards writing volume, even indirectly, would automate this bias and hand it a number.

Part 2Four traps in the data

Each of these produces a wrong answer silently, an attributed number rather than an error, which is the dangerous kind. None was visible in the data dictionary; each came out of measurement.

1. Email is not a unique key. The roster has 222 people and 211 email addresses. Eleven addresses belong to two different people each, distinct IDs, different departments, tenure and rating. Because the ticket handle derives from the email, it collides too, and one shared handle carries KPI attainment of both 45.0 and 95.0. Guessing which is which moves a person between the bottom and top decile of their cohort. Everything here keys on the immutable employee ID, never on email.
2. 61% of the task tracker is not delivered work. Of the tracker's rows, only the delivery type is real output; the rest are self-filed status items and stretch goals. Every self-update row and every stretch-goal row was filed by the person it is assigned to. A naive "tasks closed" count therefore scores people mostly on how diligently they file their own paperwork, which is why delivery is the only task type this system scores.
3. Alias handles carry real work. Twelve alias handles in the tracker carry delivery work only. Ignoring them silently deletes real delivered work for twelve people. Each resolves to a person with a matching name stem via a single dominant task creator, so they are recovered rather than dropped.
4. Confidential HR matters sit in the self-updates. Twelve weekly updates, all written by Employee Relations staff, describe live HR and legal matters, performance plans, harassment allegations, grievances, conduct investigations, and name 22 real people from the roster. Fed into a scoring pipeline unfiltered, that text would copy legally sensitive third-party information into a system a different set of managers can read. It is withheld in full before anything else reads it (Part 4).

Part 3Identity is the fairness core

The single most consequential decision in the system is not the rubric. It is what happens when the data cannot say, with confidence, whose work this is. The instinct is to pick the most likely match and move on. On a performance record, that instinct is the bug.

A handle resolves to a candidate set, not a person, narrowed only by role-consistent evidence, project codes and KPI names that are specific to a department and learned from the unambiguous people at runtime rather than hardcoded.

• Collisions that split across departments resolve cleanly, because an analytics project code cannot belong to the engineer who shares the email.
• Collisions that are two people in the same department with the same KPI have nothing to separate them. Those abstain.
• Derivable-but-missing handles are recovered by an explicit, testable rule; opaque handles with no relation to the name are declared unrecoverable rather than guessed.

Abstention has its own failure mode, and the design states it rather than hiding it: 25% abstention means silence exactly where the evidence is thinnest, which may correlate with roles that are already under-observed. That is a reason to drive identity coverage up before rollout, not a reason to paper over it with a guess.

Part 4Redaction runs before anything reads

The privacy control is a gate on input, not a filter on output. It runs before any stage that reads text, so no later stage, model or rule, can see an unscreened string even by accident. This is the opposite of the common pattern, where text flows freely and names are stripped from the final report; by then the sensitive string has already been read, joined and possibly copied into someone else's evidence.

Sensitive updates are withheld in full, not name-masked, because the surrounding sentence usually identifies the case anyway, "the grievance filed against" a masked name still points at one person on a small team. Any roster colleague named in somebody else's ordinary update is replaced with a neutral placeholder, so no evidence is ever created about a person from text a different person wrote. The affected ER staff still get credit for the case work through their tracked case tasks; only the text is withheld.

Part 5What is scored, and what is deliberately not

Four dimensions, each ranked within department because delivery volume differs by an order of magnitude across them, and weights redistributed when a dimension has no evidence behind it rather than scoring a zero it did not earn.

KPI attainment must be ranked within its own KPI, never across the company: the KPIs have different spreads, so a raw cross-role comparison would rank the KPI, not the person. For departments whose KPI is itself an output count, the score carries a confidence penalty, not a score penalty, because there the outcome and the delivery evidence are the same signal wearing two hats, and the system should rest less weight on it and widen its flag threshold accordingly.

Part 6Where AI earns its place, and where it is banned

Most of the pipeline is deterministic arithmetic and set membership, and a model has no business there. There is exactly one job where a model earns its place, and it earns it on a measurement rather than an assertion.

The self-updates are free text. To use them for corroboration, did what a person said they did show up in the systems of record, you first have to turn prose into checkable claims. The templated corpus here is easy, a regex handles it, and that proves nothing, because real weekly updates are not templated. So I hand-labelled eighteen realistic updates, written the way people actually write, with hedging, negation, multi-clause sentences and claims that span sentence boundaries, and measured two extractors against them:

Recall is the whole story. The pattern extractor is precise about what it finds and misses more than half of it, it reads "I had hoped to ship X but it is still in review" as a delivery, and it cannot follow a claim across a sentence boundary. Doubling recall is the difference between a corroboration dimension that works on real prose and one that only works on templates. The model earns its place, and it does so on the cheapest tier, which is the other half of the point.

LLM proposes, deterministic code disposes. The boundary is enforced, not asked for politely. The screening gate raises rather than degrading if handed unscreened text; invented citations are stripped by a checker a unit test keeps honest; and the default configuration runs with no model at all, fully deterministic, so the repository runs anywhere and nobody's evidence depends on a network call.

Part 7The evaluation, built without a ground truth

There is no ground truth for performance here, the manager rating is explicitly not one, so every check is built to be falsifiable without one. Checks that assert can fail the run; checks that measure print their number either way, because a harness that can only pass measures nothing.

E1 is non-circular by construction. The Outcome dimension is 40% of the production score, so measuring that score against KPI attainment would be measuring the KPI against itself. E1 rebuilds the score with the outcome data excluded entirely, delivery, consistency and corroboration only, and then tests it against the result. It beats the manager rating by 2.8× on a target neither was allowed to see.

E4 is reported honestly. Split-half +0.457 is moderate, and it is a limit on the whole exercise rather than a property of the rubric: this dataset covers four weeks, which is not a performance period. Nothing here should reach a calibration meeting without being re-measured over a real quarter, and the eval says so in the number rather than in a footnote.

Part 8Running it for two thousand people

The whole roster scores in about a second on one core with no dependencies; two thousand is roughly ten seconds. Compute is not the constraint; identity is. Drive the 25% abstention rate down before anything else, because that is where the system is silent.

What stays human

The feedback loop matters more than the rubric. Every decision is captured as (person, evidence score, rating, accepted or overridden, reason). Override reasons are clustered each cycle, and a recurring cluster is a rubric bug, not a manager problem, "runs the on-call rota" appearing twenty times means on-call is real work the instrumentation cannot see. Clusters drive changes in priority order: instrument the missing work first, widen abstention as the safe fallback, re-weight the rubric last and only with a published diff. The eval suite gates every rubric change, a weight change that improves predictive validity but degrades the bias audit does not ship.

Part 9What a governance reviewer would ask

The metric, and the guardrails

Three things are watched every cycle. Working: flag precision (below 60% the queue is noise), flag volume per manager, and predictive validity as a time series. Fair: the score's correlation with every excluded input, thresholded at 0.20 and currently 0.044, plus abstention rate by department, because a concentration in one team is an instrumentation gap being mistaken for a performance gap. Safe: the privacy assertion must be zero leaks every run, or the run does not publish.

Part 10Limits, and what I'd do with another quarter

The limits are stated up front because they bound how much any of the above should be trusted. The period is four weeks, not a performance period, which is why stability is only moderate. Nearly every manager in the sample has exactly one report, so per-manager leniency, the single most valuable calibration signal in a real deployment, cannot be measured here at all. And the data is synthetic: the method transfers, the specific numbers do not.

1. Drive identity coverage up. 25% abstention is the one thing that most limits the system's reach, and it is a data-plumbing problem with named owners, not a modelling problem.
2. Re-measure over a real quarter. Split-half reliability of +0.457 on four weeks is not enough to put a rating conversation on. Everything predictive should be re-run on a genuine performance period before it informs a calibration meeting.
3. Add per-manager calibration. Once the roster has managers with several reports, manager leniency becomes the largest correctable bias and the first thing worth adding.
4. Broaden the negative set for the claim reader. The realistic prose set bounds recall; it does not yet stress how often the model invents a claim across the full range of real updates.
5. Exclude structurally invisible roles until they have instrumented outcomes. Some functions do work that will never appear in a tracker, and it is precisely the work that must not be described in text this system reads. They should be out of automated scoring, not scored on a proxy.

CodeReading the code

Everything described above is a working repository, not a diagram: the identity resolver and its claiming and disambiguation rules, the redaction gate that runs first, the four-dimension rubric with its abstention and confidence logic, the flags that compare rating against evidence, the optional model adapters that are off by default, the E1–E7 evaluation, 29 unit tests, an analysis write-up in which every number is recomputed by one command, and an operations blueprint for running it across two thousand people.