Where AI Was Considered and Rejected

FramingThe brief, and the trap in it

A B2B SaaS company moved to a new pricing & packaging model in which the contract price depends on one number: the customer's audience count. Reps take that number from whatever is nearest to hand. It is often stale, and it often disagrees with the warehouse. Nobody finds out until the quote is already in front of the customer, and the fix is a repricing conversation.

The request, verbatim in spirit, was: "we should have AI pull this from Databricks and keep it current by account."

It is a trust problem. The number is not missing; reps have numbers. The number was wrong at the moment a quote went out. So what I built is not a sync. It is a check that fires at that moment, reads from two systems, writes to neither, and is allowed to say "I cannot stand behind any number here."

This started life as a take-home design-and-build exercise. I did not join the company and signed nothing, so it is published here with the client unnamed and with only the synthetic twelve-account dataset that came with the exercise. The account names below (Acme, Globex, Hooli, Wayne) are the fictional placeholders from that dataset, not customers. The repository is private for the same reason; see how to read the code at the end.

Part 1Five questions before any code

Each of these changes what gets built. None of them can be answered by engineering, which is exactly why they go first.

1. What does "audience count" mean when an account has multiple business units? One account in the set has three Databricks rows (NA, EMEA, APAC) summing to 535,400 against a 500,000 quote. Its note says the audience has "historically" been quoted as a single combined figure. Historically describes the old model. Nobody has said whether the new one prices the sum, each unit separately, or a deduplicated figure. Those are three different prices for the same customer, and no amount of engineering resolves it. Largest revenue consequence in the dataset; a pricing policy decision, not a data question.
2. Is customer_key the contractual join key, or a convenience field? The dataset's own documentation calls it "the field Sales Ops believes is the join key", which is not the same as it being one. The data agrees with the scepticism: one key is held by two different Salesforce accounts after a corporate split, one key carries a -OLD suffix on the warehouse side, and one exists in Salesforce and nowhere in Databricks. If this field is contractually load-bearing it needs an owner and a uniqueness constraint. If it is a convenience field, the whole reconciliation is built on sand.
3. What is the refresh SLA, and what happens when the pipeline silently stops? One account last refreshed 52 days before the export. Every other account refreshed the same morning. That is not a slow pipeline, that is one account that fell out of it, with nothing raising an alarm. A validation service that reads silently-stale data and reports it as current is worse than no service, because it converts an unnoticed data problem into a confident wrong answer.
4. What happens today when a quote goes out with a wrong count? Who notices, how long it takes, what the correction costs, whether it is absorbed or renegotiated. This is the baseline the pilot gets measured against, and it also locates the pain. Caught reliably in a Deal Desk review two days later? Cycle-time problem. Reaching the customer? Revenue and credibility problem, and the design should be more conservative.
5. Who owns the escalation when the validated count disagrees with the quote? Four accounts in this dataset come back BLOCKED. Each needs a named team and an expected turnaround, or the tool just relocates the problem into a queue nobody owns. Tooling that blocks without a working escalation path gets routed around.

The assumptions made instead

None of the five can be answered from inside the exercise, so the prototype proceeds on stated assumptions that a reader can correct:

• Multi-BU accounts are summed, and the sum is flagged, never settled. Summing is the most likely reading, but the engine does not get to set pricing policy, so that account returns REVIEW with the per-unit breakdown attached.
• customer_key is the intended join key but is neither reliably unique nor stable. Key quality therefore becomes a first-class output: exact, normalised, ambiguous, or missing, each with its own verdict.
• The audience pipeline is meant to run daily. Fourteen days without a refresh is a broken pipeline, not slow growth.
• A wrong count is materially more expensive than a blocked quote. This one assumption is behind every asymmetry in the design.
• Audience count is the only field in scope. One account has a pending tier upgrade noted against it. Real problem; not this service's problem.

Part 2The slice, and three verdicts

A rep is about to send a quote. Before they do, they trigger a check on that one account and get back a verdict, with reasons and provenance attached.

Why this slice and not the obvious one

The obvious build is a nightly batch that writes a validated count onto every Salesforce account. I rejected it, and the reasoning matters more than the conclusion.

The moment of the quote is the only point where an intervention changes the commercial outcome, and the only point where a human is already present, already paying attention, and already accountable for what they send. A check that fires there costs one person ten seconds and prevents a repricing conversation.

It also has the smallest blast radius: it reads from two systems and writes to neither. No state to reconcile, no partial-failure mode that leaves Salesforce half-updated, and no window in which a stale synced value sits on an account looking authoritative.

A dashboard for Sales Ops was the third option and is one more step removed from the moment of harm. It also tends to produce a backlog that is never fully cleared, which quietly becomes the evidence that the tool is noisy.

Part 3Where AI was considered and rejected

This is the argument the whole design rests on, so it is stated as an argument rather than as a preference.

Consider what this service actually has to answer. Which warehouse rows belong to this Salesforce account. What do they sum to. How old is the oldest of them. How far is that sum from the number on the last quote. Is that week-over-week move consistent with this account's own history. Every one of those questions has exactly one correct answer, computable from the data, with no judgement to exercise. There is nothing for a model to contribute.

What a model would contribute is variance, and variance is disqualifying here, because this number is an input to a contract price. The property that makes the system defensible is that the same account yields the same verdict every time. That property is what makes the eval suite meaningful, what lets Data Governance challenge a threshold instead of a vibe, and what lets you tell a VP that the tool cannot surprise them. Putting a model on the count path trades all of that away for nothing.

The test I actually applied

No agent runs anywhere, and where one would be right

An agent is the right shape when the sequence of steps cannot be known in advance. Here it is fully known: resolve the key, fetch the rows, sum, compare against history and against the quote, apply thresholds. Writing that as an agent replaces a function with a negotiation. It would sometimes produce a different number for the same account, and sometimes is disqualifying for an input to a contract price.

That does not make agents wrong here forever. Consider the analyst question that follows a block: "EMEA audience dropped across six accounts last month, what do they have in common?" That has no fixed query. Answering it means forming a hypothesis, querying, reading the result, revising, and querying again. The steps cannot be enumerated in advance, a human interprets the output, and nothing is priced against it. That is the shape an agent fits.

The distinction is not "simple versus complex". It is whether the sequence of steps is knowable in advance, and whether the output feeds a contract. Pre-quote validation is knowable and does feed a contract. Root-cause exploration is neither.

Natural-language SQL: available for analysis, not for pricing

The warehouse offers a natural-language-to-SQL layer, and it was on the table for the count itself. It translates a business question into SQL at runtime, which is genuinely valuable for exploration, nobody wrote the query for "why did EMEA move last month?" in advance.

It is the wrong tool for a number about to be written onto a contract, for one reason: generated SQL is not stable across runs. The same question can come back with a different grouping, a different filter, or a silently different definition of the metric. Pricing needs the same input to produce the same number every time, and it needs the query reviewable by Data Governance before it runs rather than reconstructed from logs afterwards.

There is a second, less obvious cost. If the count comes from generated SQL, then when a rep says "this doesn't match the warehouse", answering requires reconstructing what SQL ran that time. With a fixed, parameterised, version-controlled query, the answer is a file in the repo with a git history.

Part 4The one model call, and why it earns its place

Exactly one language-model call exists in the system. On accounts the rules engine has already flagged, an advisory reader looks at the freeform account_notes field and returns a routing hypothesis: suspected_cause, suggested_owner, confidence. It cannot change a count, a flag, or a verdict.

The justification is specific. account_notes is the only unstructured field in either system, and it carries information that exists nowhere else in the data: that an account was renamed after a rebrand, that a corporate split is pending on the data team's backlog, that onboarding may not have completed. That information does not change the count, it explains why the count is flagged and points the rep at the right team.

At twelve accounts you would read the notes yourself. At the scale a real sales org runs, you cannot, and the sentences are arbitrary enough that regex over them is a losing game. Three of the four blocked accounts in this dataset have a note that names the exact cause; leaving that on the floor to preserve architectural purity is a worse trade than making one bounded, advisory, non-authoritative call.

The boundary is enforced in code, not by convention

• notes_hypothesis is attached after the assessment has returned its verdict, and nothing reads it back.
• The deterministic eval suite never invokes the model at all, which is why it is reproducible.
• The whole service runs with no API key set, and no verdict changes when one is present.
• The output schema is validated on the way in: an unrecognised owner becomes null rather than being passed through, and a negative result is stripped of any cause the model attached to it.
• The model call has a 20-second ceiling and fails open into silence. The data calls fail closed into an error, because there is no safe default answer to "what is this customer's audience".

Model selection, and why the usual cost argument does not apply

The cost argument is different here because of where the call sits. Volume scales with the number of flagged accounts carrying a populated note, not the number of accounts. The call happens downstream of a verdict that already exists, so it is spend on the exception path, not on the book. That structure is what makes per-call price a weak consideration.

What this dataset cannot tell you is how large that exception path is in production. Seven of its twelve accounts flag, which is a property of a dataset built to be adversarial, not a forecast, so no production rate is assumed anywhere in the design.

What decides it is the other side of the trade: a misread note routes a rep to the wrong team and costs a day. Accuracy wins. The task is also harder than it looks, it is not extraction, it is judging whether an arbitrary sentence is relevant to a specific flag. "Tier upgrade pending approval" is a real issue that has nothing to do with a stale refresh timestamp, and the correct answer is "this does not explain the flag". That judgement, and the discipline to return a negative when a note is merely adjacent, is exactly where a smaller model's errors show up. A cheaper, faster model is the documented fallback and becomes the right call the moment this moves to a synchronous per-quote path where latency is user-visible, which it is not today.

Part 5Rules, thresholds, and deliberate asymmetries

Every threshold lives in one config file with its rationale written beside it, because every number in that file is a business rule wearing a programmer's clothes. "8% quote drift" is a statement about how much organic growth Sales considers unremarkable, and the person qualified to challenge it works in Sales Ops, not engineering. Putting it in one file means that conversation can happen without anybody reading the rules module.

The asymmetry I wrote down instead of hiding

The implausible-jump rule has two conditions and they are not equally load-bearing at this data depth. Three weekly snapshots give two growth observations, so the "prior trend" baseline is a single number. Five times a 0.8% baseline is 4%, which would flag any account that had a good week. The 25% absolute floor is what actually does the work today; the trend multiple is what will tighten the rule as history deepens and the baseline becomes a real distribution.

That is written into the config file rather than buried behind a tuned constant, because a reviewer who works it out for themselves will reasonably wonder what else was quietly papered over. It is a limitation of three weeks of history, not of the rule, and it resolves itself as data accumulates.

Learned thresholds were rejected. Ten accounts and three weeks: any threshold fitted to that is fitted to noise, on data that contains a known pipeline fault. Worse, a learned threshold cannot be explained to the person it blocks. "The model considers this anomalous" is not an answer a rep can act on; "this moved 4,815% against its own 4.3% trend" is.

Key matching: normalisation proposes, a human ratifies

Account mapping is the integration risk that actually bites, so it is a first-class output rather than an error. Four distinct outcomes, each with its own verdict:

Matching is deterministic string normalisation, case, whitespace, and a list of five lifecycle suffixes Sales Ops appends when an account is renamed or superseded. Not a model, and not fuzzy distance. Fuzzy matching would cheerfully join CK-1008 to CK-1009: one character apart, two entirely unrelated customers. A model would do better and still be non-deterministic on the one operation that must never vary.

The interesting half is the verdict. A normalised match is a heuristic join feeding a contract price. It is almost certainly right, and "almost certainly" is the wrong confidence level to have silently, so the count is returned, the verdict is REVIEW, and the message asks a human to confirm the mapping once. That is the general shape: a heuristic is allowed to propose a join and is not allowed to ratify one. The confirmation happens once per account, not once per quote.

Why BLOCKED returns no number at all

The asymmetry with REVIEW is deliberate rather than inconsistent. REVIEW means "this number is probably right, check the thing we named", the rep needs the number in order to do the checking. BLOCKED means "we cannot stand behind any number here", and in that state handing over a number is handing over the thing we just said we could not vouch for. Falling back to the last quoted count would be worse still: that is precisely the number under suspicion, returned with a verification stamp on it.

Part 6Twelve accounts, and what the engine did

Expectations were written by reading the raw data, not by recording what the code does. Every account is covered, including the ones expected to be boring, a rules engine that flags nothing is useless and one that flags everything is worse, so the negative cases carry as much weight as the positive ones.

Five trusted, three to review, four blocked. Drift is blank on the two Wayne accounts because the service declines to compute a comparison against a number it has already refused to attribute.

The trap that looks like the clever answer

Wayne Industries and Wayne Analytics both hold CK-1009 after a corporate split the data team has not yet processed, and the warehouse reports one combined 155,000. The obvious fix is to split it in proportion to each account's last quoted number.

Their last quotes are 80,000 and 75,000. They sum to exactly 155,000, precisely the combined warehouse figure. So a proportional split returns 80,000 and 75,000: the two numbers it was asked to validate, handed back with a verification stamp on them. The output is derived from the input it is supposed to check. It would agree with the quote every time, including on the day someone put a wrong number in the quote.

Returning the full 155,000 to each account with a warning is worse in a quieter way: it double-counts. Two accounts each showing 155,000 means 310,000 across the pair, and if both are quoted in the same quarter the company has priced the same audience twice. Blocking both, rather than blocking whichever one is encountered first, matters too, otherwise the outcome depends on row order.

The negative cases carry equal weight

Hooli Consumer Tech is the most important negative case in the set. Its absolute drift is +53,450, larger than the entire audience of several other accounts here. Any rule expressed in absolute counts flags this account, and is wrong to. As a percentage it is 4.6% of a large base, matching a steady ~0.6% weekly trend.

Globex Media Group refreshed 52 days before the export, against a pipeline every other account refreshed on the morning of the export. That is not a slow pipeline; that is one account that fell out of it silently. Quote drift is +4.0%, under the threshold, so staleness is correctly the only flag, and the count is still returned, with a message saying to treat it as a floor rather than a current figure.

Stark Retail Solutions comes back TRUSTED even though its note mentions a pending tier upgrade. That is a genuine issue and not this service's problem, tier is a separate field. It is in the suite specifically to confirm the engine does not flag on note content it has no rule for.

Part 7The defect that hid behind another rule

This is the part of the build I would bring to a review, because what broke and how it was caught is part of the design argument rather than separate from it.

The rule. IMPLAUSIBLE_JUMP compares the latest weekly change against the median of the changes that preceded it, and fires only when the move exceeds both 5× that baseline and 25% in absolute terms.

The defect. The first version computed that baseline over all growth observations, including the one being tested.

The arithmetic, on Initech Digital. Weekly snapshots run 8,050 → 8,400 → 412,900, giving two growth observations of +4.35% and +4,815.48%:

The outlier raises its own threshold. A single enormous jump drags the median past its own 5× limit and walks straight through the rule.

What it actually cost, which is not what it first appears

I reproduced it by reintroducing the defect in a scratch copy and re-running the suite. On Initech as it appears in this dataset, the account still blocks. The IMPLAUSIBLE_JUMP flag is lost, but QUOTE_DRIFT_SEVERE fires independently at +4,997.5% against the 8,100 quote and blocks the account anyway. The defect is masked by a second, unrelated rule.

That masking is the genuinely instructive part, so it is worth being precise rather than claiming a scarier outcome than the data supports. The severity is not visible on Initech; it is visible on the account Initech could have been. Take the same audience series and move only the last quoted figure to 400,000, a plausible stale quote near the post-jump number:

  quote=400,000  current=412,900  drift=+3.23%
  growth=[4.35, 4815.48]

  FIXED tree:      VERDICT=BLOCKED   usable_count=None    flags=['IMPLAUSIBLE_JUMP']
  DEFECTIVE tree:  VERDICT=TRUSTED   usable_count=412900  flags=(none)

Quote drift is +3.23%, silent. The implausible-jump rule is the only thing standing between a pipeline fault and a contract, and with the defect it says nothing. The service returns TRUSTED and a count of 412,900 for an account whose real audience is about 8,400.

The guard is a unit test, and note precisely what catches the regression. The test's verdict assertion passes even with the defect present, for the masking reason above. It is the flag assertion that actually fails. The test docstring says so, so that nobody later "simplifies" the flag assertion away on the grounds that the verdict assertion covers it.

Two smaller defects came out of the same pass. The eval runner miscounted its own pass rate, reporting individual assertion mismatches rather than failed cases, so one case failing on verdict, count and flags would have been reported as three failures out of twelve. And two prompt-eval expectations cited account notes that do not exist in the dataset, both fields are empty, so those cases would have failed on every run for a reason that has nothing to do with the prompt. The suite had never been executed, so nobody had found out.

Part 8Measuring a non-deterministic component

Two eval suites, kept separate on purpose. The deterministic suite covers the rules engine, needs no API key, and gates a merge: 12/12 cases and 16 unit tests pass. The prompt suite calls a real model, costs money, and can disagree with itself between runs. Mixing them would make the deterministic guarantee untrue.

The prompt suite runs every case three times and marks any case that disagreed with itself as unstable. Across two prompt versions and eleven cases, no case was unstable.

$ python -m evals.run_prompt_evals --version v2 --repeat 3
prompt version: v2 · 3 run(s) per case

account                      expected   got        owner        result
------------------------------------------------------------------------------
Fabrikam Consumer Brands     True       True       Data Team    PASS
Umbrella CPG                 True       True       Sales Ops    PASS
Wayne Industries             True       True       Data Team    PASS
Wayne Analytics              True       True       Data Team    PASS
Northwind Global Holdings    True       True       Deal Desk    PASS
Globex Media Group           skipped    skipped    None         PASS
Initech Digital              skipped    skipped    None         PASS
------------------------------------------------------------------------------
7/7 cases passed

true positives 7/7  ·  adversarial negatives 4/4

Why the two scores are reported separately and never summed

A score on true positives alone cannot characterise this component. Consider two degenerate readers that do no work at all, both run through the real harness with a stubbed model, so these are measured figures, not assertions:

Each degenerate reader scores full marks on one set and zero on the other while doing nothing. Either number read alone endorses a useless component. Only the pair constrains behaviour: there is no way to score well on both without actually reading the note and judging it against the specific flag. A combined "11/13" would be satisfied by a reader that is right for the wrong reasons, and would hide which half is weak. They are two different questions, does it recognise a relevant note, and does it decline an irrelevant one.

The comparison that did not validate what I expected

Across eleven cases and thirty-three model calls per version, the two prompt versions differ in exactly one cell: the suggested owner on Umbrella CPG. v1 routes it to the Data Team and fails; v2 routes it to Sales Ops and passes. Note that both versions correctly see that the note is relevant, v1 is not blind to it, it just sends the rep to the wrong team.

The changelog had credited v2's improvement to negative pressure: making "this note does not explain the flag" an explicitly correct answer. That change turned out to be unmeasured, and then measured as no difference, v1 also declines all four adversarial cases. What the run actually validates is the other change in v2: defining the four owner values instead of listing them as bare labels. Umbrella's note records a rename in Salesforce after a rebrand, which is record hygiene and belongs to Sales Ops, but the flag it explains is about a customer key, and a key looks like pipeline territory. With bare labels the model has nothing to distinguish "the Salesforce record was renamed" from "the key mapping is wrong", and it reaches for the Data Team.

Pre-registration, and a commitment made before the run

The adversarial set is four hand-written notes, each deliberately topically adjacent to a flag rather than obviously unrelated, a note that is plainly off topic tests nothing. The hardest one mentions audience growth against a flag about audience growth, where accepting the link would tell a rep that a pipeline fault was expected. Both versions decline it across three runs.

The circularity is stated rather than hidden: the same author wrote the notes, the expected answers, and the prompt. That cannot be removed, only bounded, expectations were recorded before the first run and not revised afterwards, and the fixture is readable in full so a reviewer can judge whether it is fair.

What this still does not prove

Of the twelve accounts, seven flag. Five of those carry a note, and all five genuinely explain their flag. So the provided dataset contains no account that is flagged and carries a note that fails to explain the flag, and that is exactly the case that matters most, because a fluent wrong hypothesis routes a rep to the wrong team with more confidence than silence would.

On the provided data alone, this suite can measure the reader's true-positive rate and not its false-positive rate. The adversarial set narrows that gap without closing it: four sentences composed by the same author as the prompt are not production text. Neither version falls for those four specific traps; how often the reader invents an explanation across the range of real notes is still unknown.

Part 9What a governance reviewer would ask

Provenance is not decoration

Every response carries what the number is made of, whether or not anything was flagged: the resolved key and how it was resolved, the contributing record IDs, the per-business-unit breakdown behind any sum, the oldest refresh timestamp across contributing rows and its age in days, and the date the check was run against.

This is what makes the "the AI's count doesn't match the warehouse" conversation resolvable in minutes rather than days, the rep can see exactly which rows produced the figure without opening Databricks. And because there is no AI in the count, that conversation is never about a model behaving unpredictably; it is only ever about which records were used. Usually the rep is looking at one business unit rather than the combined figure. Occasionally the tool is wrong, and we want to know urgently. A disagreement is information about the system, not a rep being wrong.

Note one deliberate detail: staleness is computed from the oldest contributing row, not the newest, so one fresh business unit cannot mask two stale ones.

The open governance item

Audience counts are customer data, and the notes field is free text a human wrote, it may contain more than the schema implies. Whether account names in notes are acceptable to send off-platform is a governance call, not an engineering one, and would need a retention and no-training commitment on the endpoint. Before any pilot, the notes-reader payload should be reviewed for what it actually carries in production, since the sample here is twelve curated sentences and real notes are messier.

The metric, and the guardrail

The primary metric is quote rework caused by an incorrect audience count, measured against the pre-launch baseline for the same sales pod. If that does not fall, the tool has not earned a rollout. Coverage matters too, a tool with a perfect hit rate that nobody triggers has changed nothing.

Deliberately not success metrics: number of accounts validated, or share returning TRUSTED. Both go up if the thresholds are loosened, which is the opposite of the goal.

One more honest limit, stated in the leadership brief rather than buried: TRUSTED does not mean "final". It means the count reconciles with the warehouse at a stated timestamp, not that it is contractually correct, that depends on how the pricing definition applies. This tool is very good at catching a number that is definitely wrong. It cannot certify that one is right. It tells you when to stop, not when to stop thinking.

Part 10What I'd do with another day

1. Build negative cases for the notes reader, sourced from someone who did not write the prompt. This is the only measurement gap that can hurt a user, and four self-authored sentences bound the component without putting a rate on it.
2. Add a regression test for the masked-defect case. The scenario where the implausible-jump rule is the only guard is reproduced in the results document but is not yet a test. It should be.
3. Replace normalisation with a persisted alias table. CK-1008 ↔ CK-1008-OLD is re-derived by string manipulation on every run. Once a human confirms the mapping it should be recorded, and REVIEW should become TRUSTED on subsequent runs, that is the whole point of asking for the confirmation.
4. Deepen the history window. Three weekly snapshots make the trend term nearly inert. Twelve weeks would let it carry its intended weight and would let the absolute floor come down.
5. Get the multi-BU pricing definition answered. The single largest revenue question in the dataset, and no amount of engineering resolves it.
6. Then, and only then, the write-back path. A nightly sync that puts a validated count into Salesforce needs a field-level audit trail, a defined behaviour when it disagrees with a human entry, and a partial-failure story. It is a different product with a different trust model, not a bigger version of this one.

CodeReading the code

Everything described above is a working repository, not a diagram: the rules engine and its thresholds, the fixed parameterised query, the single model call and the schema that fences it in, 16 unit tests, the twelve-account deterministic eval suite, the prompt-eval runner with its adversarial negatives, an ADR log recording what was rejected at each step, and a one-page brief written for people who do not read code.