FramingWorking assumptions
This design describes an assist layer inside an existing Search experience. I treat the brief as a starting point and state my assumptions wherever it leaves room.
- Catalog size. Tens of thousands of titles, not millions. The platform has several million subscribers, but a single-market licensed catalog is small. This one fact drives most of the retrieval and storage choices below.
- No mandated stack. No onboarded technology was specified, so I choose by problem fit and justify each call.
- Assist layer, not a standalone product. The assistant augments search and must never block or degrade the core playback and search flow.
- Availability is dynamic. Licensing changes mean that a title's presence in the catalog and its current availability are not the same thing.
Part 1What the hard problem actually is
The easy reading is "add a chatbot to search." That is not the hard problem.
Reliable, grounded intent-to-catalog mapping in Czech, produced as strict structured output, cheaply, at scale, with state held across turns, without ever recommending something the user cannot actually watch.
Building a demo of this is easy. Building it so it holds up is not. The real difficulty lives in the tensions:
Czech quality vs. cost
Czech is the explicit quality gate and the hardest axis: less training data, rich inflection, and vague colloquial intent — "něco oddechového na večer", "ten špión, ten starší chlápek z devadesátek". The models that handle Czech best are usually the larger, more expensive ones, yet the cost ceiling at 1.2M conversations/day pushes hard toward small models. These two forces pull in opposite directions, and most of the design is about resolving that tension.
Fluency vs. grounding
The reply must feel natural and helpful, but the model is strictly forbidden from inventing titles. Free-text fluency and zero-hallucination grounding are in direct conflict and must be separated architecturally — not by prompt wording.
Latency budget vs. work inside it
5–10 s end-to-end is generous for one model call, but it must also absorb a safety pass, retrieval, generation and validation — and hold at p99 during the 150k/hour peak, not just on average.
Statefulness vs. cost
Sticky constraints require memory across turns, but stuffing conversation history into the prompt on every turn is both expensive at this volume and a source of drift.
Freshness
"Only available titles" is a correctness contract. A title indexed yesterday whose licence lapsed today must not be recommended — which makes the index alone an unsafe source of truth.
Part 2A bounded pipeline, not an agent
I deliberately design this as a bounded, deterministic pipeline rather than an autonomous agent. The task is well-scoped — understand → retrieve → respond — with hard correctness and latency constraints. A general agent framework would add latency, cost and failure modes I do not need and cannot easily reason about.
The LLM is a powerful component inside a controlled flow, not the controller of the flow.
An input guard runs first as a cheap blocking filter. A cache lookup follows, keyed on the normalized utterance plus the active constraints carried in the session, so it works before any LLM parsing. On a miss, the LLM extracts the user's intent and updates the sticky constraint object. Hybrid retrieval then returns a candidate set of real catalog IDs filtered by availability and active constraints. The LLM ranks that set and emits the structured response, choosing only from the provided IDs. A deterministic validation step confirms the schema, and that every recommended ID exists and is currently available; on any failure it falls back deterministically, never with a second LLM call. Logging and eval sampling happen asynchronously, off the critical path.
GroundingThe catalog is the source of truth
- Data & enrichment. Existing per-title tags and descriptions, enriched offline by an LLM pass into richer semantic descriptions and normalized tags, then embedded. This runs once, and again only when a title is added or changed. Content is near-static, so it is cheap.
- Indexing & retrieval. Titles, structured metadata and embeddings live together in Postgres with pgvector. Retrieval is a single hybrid query: hard SQL filters for availability and constraints, vector similarity for vague intent over the enriched text.
- Freshness. Availability is a column in the same database and is the single source of truth. There is no separate vector index to keep in sync, which removes a whole class of staleness bugs. Embeddings are recomputed only on content change.
StateSticky constraints, not chat history
State is an explicit structured constraint object — active filters such as genre, local-originals, era and maturity — not conversation history replayed into the prompt. Each turn the model reads the current object plus the new utterance and returns an updated object. It is persisted per session in Redis, and constraints flow into retrieval as hard filters.
This is cheap, debuggable and unit-testable. It is the difference between a constraint that reliably sticks and one that silently drifts — and it keeps token cost flat regardless of conversation length.
SafetyStructured output is the defence
The response is produced with native structured outputs / function calling bound to a strict {reply, picks, chips} schema. picks is constrained to the candidate IDs from retrieval and validated against the catalog afterwards.
Even if a user manipulates the model, it cannot return anything outside the schema and cannot recommend a title outside the available catalog. Grounding is structural, not a matter of prompt phrasing. The input guard is only a cheap first filter on top of this.
When validation does fail — a malformed structure, which is rare with structured outputs, or an invalid or unavailable ID, which is the more likely case — the fallback is deterministic and never a second model call. The pipeline drops the offending picks and substitutes the top results already returned by hybrid retrieval in step 4, together with a static Czech catch-all message. A self-correction prompt would add a full round-trip to the latency budget, so it is deliberately avoided.
Where the LLM fits
Language work
Czech intent understanding, constraint extraction, ranking and selection from the candidate set, and the fluent Czech reply plus tap-chips.
Anything load-bearing
Store state, decide availability, generate catalog IDs, perform retrieval, or act as the sole safety boundary. Those are owned by deterministic components I can test and reason about.
BudgetLatency and the number of model calls
A cache miss can involve two LLM steps (3 and 5), and two sequential calls are the main threat to the latency budget at peak — made worse by Czech tokenizing less efficiently. I keep this under control in three ways.
- Different model sizes per step. State extraction (step 3) is a narrow, structured task suited to a small, fast model, optionally fine-tuned. Only ranking and Czech generation (step 5) need the more capable model.
- Most turns need only one call. When the user simply continues without changing a constraint — taps a chip, says "spíš něco kratšího" — the state does not change in a way that alters retrieval, so step 3 is skipped and a single call handles the turn. The two-call path is reserved for turns that introduce a new constraint that must reshape retrieval before generation.
- Per-stage timeouts protect the tail. Folding both steps into one function-calling prompt is possible, but it gives up the ability to filter the catalog between them — so I use that only as the single-call path above, never as the default.
Part 3Technology choices
| Component | Choice | Reason |
|---|---|---|
| LLM model | Eval-gated. Small fast model (mini / Flash class) for state extraction and most turns, escalating to a stronger multilingual model for harder ranking and generation. | Czech quality must decide the models via a golden-dataset eval, and using the small model for the narrow steps is what keeps 1.2M conversations/day affordable. |
| Retrieval layer | Hybrid: hard metadata filters + vector similarity, with the LLM ranking only from the returned candidates. | Constraints and availability are exact filters while vague intent is semantic, so I need both in one step. |
| Vector store | pgvector inside the catalog Postgres database, not a dedicated vector DB. | The catalog is small and structured, hybrid filtering wants vectors and metadata in one query, and freshness comes for free because availability lives in the same store. |
| Serving framework | FastAPI async service orchestrating an explicit pipeline, not an agent framework. | A bounded, latency-sensitive flow does not need agent overhead and is safer when each stage is explicit. |
| Cache | Redis, keyed on the normalized utterance + active constraints — no LLM needed to build the key. | Verbatim Czech caches poorly, so normalizing the text and adding the session constraints gives a safe, high-hit key that is available before any model call. |
| Session state | Redis session store holding the constraint object. | Low-latency per-session reads and writes for the sticky-constraint state on every turn. |
| Embeddings | Eval-gated multilingual embedding model. | Retrieval quality on Czech depends directly on embedding quality, so it is chosen the same way as the LLM. |
Part 4MVP scope cut
The goal of the internal beta is to prove the two riskiest axes — Czech quality, and grounded no-hallucination retrieval — with a credible end-to-end loop. Everything deferred below is additive, not foundational.
Ships
- Czech as a hard quality gate, with an evaluation harness and a Czech golden dataset from day one.
- The core loop: guard → intent → hybrid retrieval → structured output → deterministic validation.
- Sticky constraints over a small, well-defined set (genre, local-originals, era, maturity).
- Hard catalog and availability validation on every recommendation.
- Basic input guard, plus the schema-and-filter grounding that doubles as jailbreak defence.
Waits
- Personalization and watch-history awareness: start session-only, no user profile.
- Semantic (vector) caching: start with exact caching on normalized text + constraints.
- Advanced multi-constraint reasoning and negotiation across many simultaneous filters.
- Full jailbreak hardening beyond schema + catalog filtering.
- Sophisticated model tiering: begin with one small model plus a single escalation rule, tune later.
- Voice input and any non-Czech language support.
Part 5Top risks
Czech quality
The model misreads vague or inflected Czech, or generates plausible-but-wrong Czech. This is the explicit quality gate and the main differentiator: get it wrong and the assistant feels worse than plain search.
Mitigation. A Czech golden dataset with LLM-as-judge evaluation gating both model selection and every release; a bias toward constrained generation over free text wherever possible; regression runs on each change. This is the axis I would invest the most in, and it is where I have direct production experience.
Cost blow-up at scale
Running a frontier model on every turn at 1.2M conversations/day breaks the economics.
Mitigation. Model tiering, caching on intent and constraints, and using the LLM only where it genuinely adds value — retrieval and validation are non-LLM. A measured cost-per-conversation budget with alerting, treated as a release gate alongside quality.
Catalog freshness / availability drift
Recommending a title whose licence expired directly violates the "only available titles" contract and erodes trust.
Mitigation. Availability as a single source of truth in the catalog DB, and a final validation against live availability — never against a stale index — before any recommendation reaches the user.
Latency tail at peak
Average latency can look healthy while p99 blows past the budget during the 150k/hour peak, hurting the search flow it is embedded in.
Mitigation. Per-stage timeouts, graceful degradation to normal search results, async handling of non-critical work, caching to shed load, and keeping most turns to a single model call with the two-call path used only when a constraint changes.
Sticky-constraint correctness
A constraint is silently dropped, or gets stuck on — subtle, hard to notice, and quietly frustrating.
Mitigation. The explicit state object with targeted unit tests, plus surfacing the active constraints back in the chips so they are visible and correctable by the user.
Assist-layer containment
Because this lives inside Search, an assistant failure must never disrupt playback or the core search experience.
Mitigation. Fail open to the existing search results on any error or timeout, strict component boundaries, and a circuit breaker that disables the assist layer cleanly under sustained failure.
Keep the LLM inside a bounded pipeline, make grounding structural rather than prompted, and let a Czech golden dataset decide every model in the stack.
Fluency is a model problem. Correctness is an architecture problem — and it is the one that decides whether this survives contact with production.