LLM Fundamentals

Working with language models is mostly about managing context and cost, and knowing which lever to reach for.

Tokens are the unit of everything. Models read and write tokens, not words — roughly a token per three-quarters of an English word. You are billed per token, latency scales with tokens, and the context window caps how many can be in play at once. A long conversation does not "remember" anything; the whole history is re-sent every turn, which is why costs grow quadratically in a naive chat loop.

Three ways to make a model do your task, in ascending cost:

Prompting — instructions and examples in the context. Instant to iterate, no training, and limited by context size.

RAG — retrieve relevant documents at request time and put them in context. The right answer when knowledge changes often, because updating the knowledge base is an indexing job, not a training run. It grounds answers in sources you can cite.

Fine-tuning — adjust the model's weights on your examples. It teaches form, style, and task shape far better than it teaches facts, and facts you fine-tune in go stale and cannot be cited. Reach for it when you need consistent structure or a specialised behaviour, not to load a knowledge base.

The common mistake is fine-tuning to inject knowledge that RAG should supply.

Temperature controls randomness in sampling. Zero is near-deterministic, which is what you want for extraction and classification; higher values suit generation where variety helps. Deterministic is not the same as correct.

Structured output. When you need machine-readable results, constrain the output with a schema rather than parsing prose and hoping. It removes a whole class of brittle string handling.