AI & LLM Systems

LLM features now show up in design interviews as their own question. The good news is that most of it is ordinary systems work with unfamiliar names.

Frame the task before the model. The most common failure is talking about model choice before defining what the system must do, what "good" means, and what data exists. Spend the first minutes on the task, the success metric, and the failure cost — the same scoping discipline any design question needs.

RAG is a retrieval problem wearing an AI hat. Retrieval-augmented generation embeds your documents, finds the ones similar to the query, and passes them to the model as context. Quality is dominated by retrieval, not generation: if the right chunk is not retrieved, no model will answer correctly.

Chunking is the lever people underrate. Chunks that are too small lose the context that makes them meaningful; too large and the relevant sentence is diluted among irrelevant text while you burn context window. Hybrid search — dense vectors plus keyword — matters because embeddings are bad at exact identifiers like SKUs and error codes. Re-ranking runs a slower, more accurate model over the top-k candidates to fix the ordering that fast approximate search got roughly right.

Vector search is approximate. Indexes like HNSW trade recall for latency, so "did we retrieve the right document" is a tunable, measurable property rather than a guarantee.

Serving has its own shape. Generation runs in two phases: prefill processes the whole prompt in parallel and is compute-bound, then decode emits one token at a time and is bound by memory bandwidth. The KV cache stores attention state for tokens already processed so each new token does not recompute the whole prompt — it is why long prompts cost memory, not just compute. Continuous batching raises GPU throughput by packing concurrent requests, at some cost to individual latency. Streaming tokens does not make generation faster; it makes time to first token the number the user feels.

Evaluate like a system, not a demo. Hold out a reference set and measure retrieval recall separately from answer quality, so you know which half is failing. Automated judging is useful and biased — sample and read real outputs. Guardrails on input and output, plus a path to refuse when retrieval is weak, beat hoping the model behaves.