LLMs and RAG in plain English
A short tour of how large language models work and the pattern most teams reach for first when they want one to use their own data: RAG.
What an LLM is
A large language model is a probabilistic model of text. Given the words so far, it predicts which word (strictly, which token) is most likely to come next, and repeats that until it has an answer. Everything else it appears to do, such as summarising, writing code or translating, comes from doing that one thing at enormous scale.
The acronym GPT says how it is built:
- Generative: it produces new content instead of only labelling existing content.
- Pre-trained: it first learns general language patterns from a very large body of text.
- Transformer: the neural network design that turns one sequence of tokens into another, using attention to decide which earlier words matter most.
"Large" refers to two things: the size of the training data and the number of parameters, the adjustable weights inside the network. "General purpose" means one model is good enough for many common tasks without being built for any one of them.
How the learning works
Supervised and unsupervised learning
Supervised learning trains on labelled examples, such as emails marked spam or not spam, and is typically used for classification. Unsupervised learning gets no labels and finds structure by itself, for example by clustering similar items. LLM pre-training is usually called self-supervised: the "label" is simply the next word in the text, so no human has to annotate anything.
Neural networks and deep learning
An artificial neural network is a stack of layers of simple units loosely inspired by the brain: an input layer, one or more hidden layers, and an output layer. When there are many hidden layers it is called deep learning. LLMs are deep learning models, which is why they are described as a subset of it.
Pre-training, fine-tuning and prompting
There are three ways to get a general model to do a specific job, and they differ a lot in cost.
| Approach | What changes | When to use it |
|---|---|---|
| Prompting | Only the input text | Always start here. Cheap and instant. |
| RAG | The input, plus retrieved facts | The model needs knowledge it was never trained on, or that changes often. |
| Fine-tuning | The model's weights | You need a consistent style, format or specialised behaviour. |
Prompting itself has a few levels. Zero-shot means asking for the task with no examples. Few-shot means including a handful of worked examples in the prompt so the model copies the pattern. Good prompt engineering is mostly being specific about the task, the audience, the format of the answer and the constraints.
RAG: answering from your own data
A model only knows what was in its training data, and that data has a cut-off date. It also cannot know anything private to you. Asked about either, it may answer confidently and be wrong, which is what people call a hallucination.
Retrieval-augmented generation avoids this. Before the model answers, you search your own documents for the passages most relevant to the question and paste them into the prompt. The model then writes its answer from that material instead of from memory.
The model does not learn your documents. It reads the relevant ones fresh, every time. That is why RAG answers can be updated just by updating the documents.
The main advantages over fine-tuning are that answers can cite their sources, knowledge is always as fresh as the index, access controls can be applied at retrieval time, and there is no training run to pay for.
The RAG pipeline
1. Ingest and index
Documents are split into chunks small enough to fit in a prompt, usually a few hundred words. An embedding model turns each chunk into a vector, a list of numbers that places it in a space where similar meanings sit close together. The vectors are stored in a vector database alongside the original text.
2. Retrieve
At question time the question is embedded the same way, and the database returns the chunks whose vectors are nearest to it.
3. Prompt and generate
The retrieved chunks are placed in the prompt with an instruction such as "answer only from the context below, and say so if it isn't there." The model generates the final answer.
Three common shapes
- Classic RAG: one retrieval step, then one generation step.
- Agent-assisted RAG: an agent decides what to search for, can search several times or in several sources, and checks whether it has enough before answering.
- RAG with guardrails: checks run on the question and on the answer, for example to block unsafe requests, remove personal data, or reject answers that are not supported by the retrieved text.
Frameworks such as LlamaIndex and LangChain provide ready-made building blocks for ingestion, retrieval and prompting, so you rarely write this plumbing from scratch.
Using it responsibly
- Treat outputs as drafts. Verify facts that matter.
- Keep sensitive data out of prompts unless your provider's terms cover it.
- Watch for bias in both the training data and your own documents.
- Be transparent with users when they are talking to a model.
- Log and review real conversations so you can find failure cases.