AI Glossary

Every technical term used in the courses, explained in plain words.

activation: The technical name for the squish function a neuron applies to its result.
agent skill: A folder of instructions (and sometimes templates or scripts) that an agent loads only when a task needs it.
AI: Artificial Intelligence: computer systems that do things we normally think need human intelligence, like recognising faces, translating or answering questions. Today most AI learns from examples instead of following hand-written rules.
AI agent: An AI system that works towards a goal in steps: it decides what to do, uses tools to act, looks at the results and repeats.
Amdahl’s law: The rule that the part of a job which must run step by step limits the total speed-up: if 5% can’t be shared out, no number of workers makes it more than 20× faster.
API: Application Programming Interface: an agreed way for one program to ask another for something, such as an app sending a request to an AI model.
API key: A secret code that identifies who is calling an API, so the provider can allow the request and bill for it.
approximate nearest neighbour: Search that finds very close vectors quickly by skipping most of the collection, occasionally missing the very closest one.
arithmetic intensity: How many calculations you do for each byte you load from memory. High intensity keeps the math units busy; low intensity leaves them waiting.
attention: The step where each word looks at the other words and decides which ones matter for understanding it, then mixes in their information.
backpropagation: The method that works out, for every weight at once, how much it contributed to a mistake. It starts at the output and passes the blame backwards, layer by layer.
backward pass: The extra step during training: going back through the network to work out how much each weight was to blame for the mistake, so it can be nudged. Experts call it "backpropagation".
base model: A model after pre-training only: excellent at continuing text, but not yet trained to act as a helpful assistant.
batch: A group of inputs processed together in one go. Bigger batches reuse the same weights for more work.
benchmark: A standard test, with fixed questions and known answers, used to compare models on the same task.
bias: A number each neuron adds at the end, no matter what the input is. Think of it as a starting amount, like a base price.
block: A group of threads that runs together on one SM and can share its fast memory. A kernel is split into many blocks.
cache: A small, fast memory close to the chip that keeps copies of recently used data, so it does not have to be fetched from far away again.
chain of thought: Having a model write out intermediate reasoning steps before its final answer, which often makes the answer better.
chat template: The fixed format, with special tokens, that wraps a conversation before the model reads it: who is speaking, and where the assistant’s turn begins.
chunking: Splitting documents into smaller pieces before embedding them, so each piece can be found and retrieved on its own.
context engineering: Choosing what information goes into the model’s context window: relevant instructions, facts, memory and examples, and nothing more.
context window: The maximum number of tokens a model can look at at once: your prompt plus its answer so far.
continuous batching: Rebuilding the batch at every step, so a finished request’s place goes straight to the next waiting request.
core: One worker inside a chip that can do calculations. A CPU has a handful of strong cores; a GPU has thousands of simple ones.
cosine similarity: A measure of how closely two vectors point the same way: 1 = same direction (similar), 0 = unrelated, −1 = opposite.
CPU: Central Processing Unit: the main chip of a computer. It has a few very smart, very fast cores that are great at doing different things one after another.
data leakage: When information from the test data sneaks into training, for example duplicate examples on both sides of the split. It makes a model look better than it really is.
decode: The second phase of answering: the reply is written one token at a time, each needing a full pass that rereads the weights.
deep learning: Machine learning with neural networks that have many layers ("deep" means many layers). It powers face recognition, speech recognition, image generators and LLMs.
deep neural network: A neural network with many hidden layers. Each layer builds on the one before it, from simple features to complex ideas.
distillation: Training a smaller “student” model to imitate a bigger “teacher” model, keeping much of its skill at a fraction of the cost.
embedding: A list of numbers that represents a token's meaning. Tokens with similar meanings get similar lists, so they sit close together on a "map of meaning".
embedding model: A model that turns a piece of text into a single vector capturing its meaning, so similar texts get similar vectors.
epoch: The technical name for one practice round over all the training examples.
feed-forward: The part of each transformer block that processes every token on its own, through two big layers of neurons. It holds most of the model's weights.
few-shot prompting: Including a few worked examples of a task in the prompt, so the model copies their pattern and format.
fine-tuning: Extra training on a small, carefully chosen dataset to teach a pre-trained model a specific behaviour, like answering questions.
floating point: The way computers store decimal numbers: a sign, an exponent (how big) and a mantissa (the digits), like scientific notation (3.14 × 10²).
FLOP: A FLoating-point OPeration: one multiplication or addition of decimal numbers. "FLOP/s" counts how many a chip can do per second.
forward pass: Running an input through the network, layer by layer, to get its answer. Every time you use a model, it does a forward pass.
GPU: Graphics Processing Unit. A chip with thousands of simple workers that all do math at the same time. Built for video games, now the engine of AI.
gradient: The technical name for the "which way and how much" signal that tells each weight how to change.
gradient descent: How almost every neural network learns: measure the mistakes, work out which way each weight should move to reduce them (the gradient), take a small step that way, and repeat.
hallucination: A fluent, confident statement from an AI model that is false or made up.
HBM: High Bandwidth Memory: the GPU's main memory, stacked right next to the chip. Big (tens of GB) and fast compared to normal memory, but slow compared to the chip itself.
hidden layer: Neurons between the input and the output. They are "hidden" because you never see their answers directly; they work out helpful in-between ideas.
hybrid search: Combining keyword search and meaning search and merging their results, so each covers the other’s blind spots.
inference: Using a trained model to get answers. The opposite of training.
input: The number(s) you give the network: the question.
JSON: A simple text format for structured data: named fields inside curly brackets, easy for programs to read and write.
kernel: A program that runs on the GPU. Thousands of threads all run the same kernel, each on its own piece of the data.
knowledge cutoff: The date after which a model’s training data stops. It knows nothing about later events.
KV cache: Memory where the model keeps the keys and values it already computed for earlier tokens, so it does not redo that work for every new token.
label: The right answer attached to a training example, such as "spam" for an email or the sale price for a house.
latency: How long one user waits: for the first token, between tokens, or for the whole answer.
layer: A group of neurons that all work on the same inputs at the same time. A network passes numbers from one layer to the next.
learning rate: The technical name for the step size.
learning step: One round of practice: the network checks its answers on all examples, then nudges every weight and bias a little to make fewer mistakes. Experts call one round an "epoch".
LLM: Large Language Model: a huge neural network trained on enormous amounts of text to predict the next token. ChatGPT, Claude, Gemini and Llama are LLMs.
long-term memory: Facts an agent saves outside the model, such as user preferences, and retrieves in later conversations when they are relevant.
LoRA: Low-rank adaptation: fine-tuning by freezing the original weights and training two small matrices whose product is added to them.
loss: The technical name for the mistake score. Lower is better; 0 means every answer is perfect.
machine learning: A way to build AI where the computer finds the rules itself, by spotting patterns in lots of examples, instead of a programmer writing them.
matrix: A grid of numbers with rows and columns. A layer of a network keeps its weights in a matrix.
matrix multiplication: Multiplying two grids of numbers: each output is a row of one grid times a column of the other, summed up. Almost all of the work in a neural network is this.
MCP: Model Context Protocol: an open standard for connecting AI apps to tools and data, so one server works with many apps.
memory bandwidth: How many bytes per second can travel between memory and the chip. Often the real speed limit for AI, not the math.
metadata: Extra information stored with a document, such as its source, date, department or who may see it.
mistake score: How wrong the network is on average across all training examples. Each mistake is squared first, so big mistakes count extra. Experts call this the "loss".
mixture of experts: A model design with many expert sub-networks per layer, where a router sends each token to only a few of them.
model: What learning produces: a program whose behaviour is set by the numbers (parameters) it learned from data. You give it an input and it gives back a prediction.
multi-agent system: Several AI agents, each with its own context and job, working together on a larger task.
multi-head attention: Running several attention calculations side by side, each with its own learned weights, so a layer can track several kinds of relationship at once.
neural network: A program made of many small calculators (neurons) connected together. Instead of being told the rules, it learns them from examples.
neuron: One small calculator. It takes numbers in, multiplies each by a weight, adds a bias, and passes the result on.
nudge: The network works out, for every weight, "if I made this a bit bigger, would the mistakes get smaller or bigger?" and moves it in the helpful direction. Experts call this direction the "gradient".
OCR: Optical character recognition: reading the text out of an image, such as a scanned page.
open-weight model: A model whose trained weights are published for anyone to download, run and fine-tune, under the terms of its licence.
output: The number the network gives back: its answer.
overfitting: When a model memorises its training examples, noise included, instead of learning the general pattern. It looks great on those examples and poor on new ones.
PagedAttention: Storing the KV cache in small pages handed out on demand anywhere in memory, instead of reserving one big block per request.
parallel: Doing many things at the same time instead of one after another.
parameter: Any number the model learns: every weight and every bias. "A 70B model" means a model with about 70 billion of them.
pattern: A regularity in data, like "bigger houses usually cost more". Machine learning finds patterns in examples and uses them on new cases.
pipeline parallelism: Splitting a model’s stack of layers into blocks on different GPUs, with each token passed from one GPU to the next.
positional encoding: Extra information added to each token’s vector to say where it sits in the text, so the model can tell word order.
pre-training: The first and most expensive training stage: learning to predict the next token on trillions of words of text.
precision: How many bits a number uses. More bits = more exact, but more memory and slower math.
prediction: A model's answer for an input it has not seen before, based on the patterns it learned. It can be a number, a label ("spam") or the next word.
prefill: The first phase of answering: the whole prompt is read in one parallel pass. It keeps the GPU’s maths units busy.
probability: How likely something is, from 0% (never) to 100% (certain). An LLM gives every possible next token a probability.
prompt: The text you give a language model: your question or instruction, plus anything else in the conversation.
prompt engineering: The craft of writing prompts that give a model everything it needs: purpose, audience, format, facts and constraints.
prompt injection: An attack where text the AI reads, such as an email or web page, contains instructions meant to hijack what it does.
quantization: Storing a model's numbers with fewer bits (for example 8 or 4 instead of 16) to save memory and gain speed, at a small cost in accuracy.
RAG: Retrieval-augmented generation: finding relevant passages in your own documents and adding them to the prompt so the model answers from them.
rate limit: The most requests or tokens an account may send per minute. Going over it gets a “429 Too Many Requests” error.
ReAct: Short for “reasoning and acting”: an agent pattern that alternates written reasoning with tool calls and their observations.
reasoning model: An LLM trained, usually with reinforcement learning, to think at length in writing before answering.
register: The tiniest, fastest memory: a few slots inside each worker that hold the numbers it is calculating with right now.
reinforcement learning: Learning by trial and error: the model acts, gets a reward or a penalty, and gradually prefers the actions that paid off.
ReLU: Short for "rectified linear unit": a squish function that turns negative numbers into 0 and lets positive ones through unchanged.
reranker: A model that reads the question together with each candidate passage and reorders a shortlist by how well each one answers it.
reward: In reinforcement learning, the score the model receives after an action: positive for good outcomes, negative for bad ones.
reward model: A helper model trained on human ratings that predicts how much people would like an answer.
RLHF: Reinforcement Learning from Human Feedback: people rate answers, and the model is tuned to produce the kind of answers people prefer.
RNN: Recurrent neural network: an older kind of language model that reads one word at a time and carries a single running memory vector.
scaling law: A predictable relationship between a model’s size, its training data, the compute used and how good it gets.
self-supervised learning: Learning from data that supplies its own answers, for example hiding the next word of a sentence and asking the model to guess it.
semantic search: Searching by meaning rather than exact words, by comparing embedding vectors of the question and the documents.
serving engine: Software that turns a model into a fast shared service by scheduling requests, managing memory and running the model efficiently, such as vLLM.
shared memory: A small fast memory shared by one group of GPU workers (one SM). Programmers fill it on purpose with data the group will reuse.
sigmoid: A squish function that turns any number into a value between 0 and 1, often read as "how sure" the neuron is.
SM: Streaming Multiprocessor: one "workshop" on an NVIDIA GPU with its own workers, registers and shared memory. An H100 has 132 of them.
softmax: A formula that turns any list of scores into probabilities that add up to 100%. Higher score = bigger share.
special token: A token that isn’t ordinary text but a marker, such as “start of the user’s message” or “end of the answer”.
speculative decoding: Letting a small model guess several tokens ahead and having the big model check them all in one pass, keeping only what it agrees with.
squish function: An extra step some neurons do at the end to reshape their result. For example, sigmoid squishes any number into the range 0 to 1, useful for yes/no answers. Experts call this an "activation function".
step size: How big each nudge is. Too small and learning is slow. Too big and it overshoots and goes wild. Experts call this the "learning rate".
structured output: Making a model return data in an exact format, such as JSON that matches a schema, often enforced while it generates.
supervised learning: Learning from examples that come with the right answer attached, like photos labelled "cat" or "dog".
system prompt: Hidden instructions from the app, sent first with every request, that set the assistant’s role, tone and rules.
temperature: A setting that controls how adventurous the model is when picking the next token. Low = safe and repetitive, high = creative but riskier.
tensor core: A special unit in modern NVIDIA GPUs that multiplies small matrices in one go, much faster than ordinary cores, especially with low-precision numbers.
tensor parallelism: Splitting every layer of a model across several GPUs, which work on each token together and swap results after every layer.
test set: Data locked away until the very end and used once, to measure how well the finished model handles examples it has never seen.
thread: One small sequence of instructions run by one GPU worker, usually responsible for one tiny piece of the result.
throughput: How much work a system does in total, such as tokens per second summed over all users.
tile: A small square piece of a big matrix, sized so it fits in fast memory.
time to first token: How long you wait before the first word of an answer appears: queueing, reading the prompt and network time. Often shortened to TTFT.
token: A small piece of text (a word or part of a word) that a language model reads or writes one at a time.
tokenizer: The program that cuts text into tokens and turns them into ID numbers (and back again).
tool calling: How a model uses tools: it writes a structured request for a function, the app runs it, and the result is sent back to the model.
top-p: A setting that throws away the unlikely tokens, keeping only the most likely ones that together cover p (for example 90%) of the probability.
traditional software: A program that follows exact step-by-step rules written by a programmer. It does exactly what it was told, and nothing it was not told.
training: Repeating the learning step many times so the mistake score keeps going down.
training data: The examples the network learns from: questions together with their correct answers.
training set: The part of the data the model learns from.
transformer: The neural network design behind all modern LLMs (2017). It is a tall stack of identical blocks, each with an attention step and a feed-forward step.
underfitting: When a model is too simple to capture even the real pattern, so it does poorly on old and new examples alike.
unsupervised learning: Learning from data with no answers at all. The model finds structure on its own, such as groups of things that look alike.
validation set: Data held back from training and used to make choices, such as which model or settings to keep.
vector: A list of numbers, which you can picture as an arrow or a point in space. Embeddings are vectors.
vector database: A database that stores vectors with their text and metadata, and quickly finds the vectors closest to a question’s vector.
vocabulary: The fixed list of all tokens a model knows, typically 30,000 to 200,000 of them. Each has an ID number.
VRAM: The memory on a graphics card (on AI GPUs this is HBM). The model must fit in it to run fast.
warp: A team of 32 threads on an NVIDIA GPU that always execute the same instruction at the same time.
weight: A number on a connection that says how much a signal counts. Big = it matters a lot. Negative = it pushes the answer the other way. Learning = adjusting weights.

AI Glossary

Every technical term used in the courses, explained in plain words.

activation: The technical name for the squish function a neuron applies to its result.
agent skill: A folder of instructions (and sometimes templates or scripts) that an agent loads only when a task needs it.
AI: Artificial Intelligence: computer systems that do things we normally think need human intelligence, like recognising faces, translating or answering questions. Today most AI learns from examples instead of following hand-written rules.
AI agent: An AI system that works towards a goal in steps: it decides what to do, uses tools to act, looks at the results and repeats.
Amdahl’s law: The rule that the part of a job which must run step by step limits the total speed-up: if 5% can’t be shared out, no number of workers makes it more than 20× faster.
API: Application Programming Interface: an agreed way for one program to ask another for something, such as an app sending a request to an AI model.
API key: A secret code that identifies who is calling an API, so the provider can allow the request and bill for it.
approximate nearest neighbour: Search that finds very close vectors quickly by skipping most of the collection, occasionally missing the very closest one.
arithmetic intensity: How many calculations you do for each byte you load from memory. High intensity keeps the math units busy; low intensity leaves them waiting.
attention: The step where each word looks at the other words and decides which ones matter for understanding it, then mixes in their information.
backpropagation: The method that works out, for every weight at once, how much it contributed to a mistake. It starts at the output and passes the blame backwards, layer by layer.
backward pass: The extra step during training: going back through the network to work out how much each weight was to blame for the mistake, so it can be nudged. Experts call it "backpropagation".
base model: A model after pre-training only: excellent at continuing text, but not yet trained to act as a helpful assistant.
batch: A group of inputs processed together in one go. Bigger batches reuse the same weights for more work.
benchmark: A standard test, with fixed questions and known answers, used to compare models on the same task.
bias: A number each neuron adds at the end, no matter what the input is. Think of it as a starting amount, like a base price.
block: A group of threads that runs together on one SM and can share its fast memory. A kernel is split into many blocks.
cache: A small, fast memory close to the chip that keeps copies of recently used data, so it does not have to be fetched from far away again.
chain of thought: Having a model write out intermediate reasoning steps before its final answer, which often makes the answer better.
chat template: The fixed format, with special tokens, that wraps a conversation before the model reads it: who is speaking, and where the assistant’s turn begins.
chunking: Splitting documents into smaller pieces before embedding them, so each piece can be found and retrieved on its own.
context engineering: Choosing what information goes into the model’s context window: relevant instructions, facts, memory and examples, and nothing more.
context window: The maximum number of tokens a model can look at at once: your prompt plus its answer so far.
continuous batching: Rebuilding the batch at every step, so a finished request’s place goes straight to the next waiting request.
core: One worker inside a chip that can do calculations. A CPU has a handful of strong cores; a GPU has thousands of simple ones.
cosine similarity: A measure of how closely two vectors point the same way: 1 = same direction (similar), 0 = unrelated, −1 = opposite.
CPU: Central Processing Unit: the main chip of a computer. It has a few very smart, very fast cores that are great at doing different things one after another.
data leakage: When information from the test data sneaks into training, for example duplicate examples on both sides of the split. It makes a model look better than it really is.
decode: The second phase of answering: the reply is written one token at a time, each needing a full pass that rereads the weights.
deep learning: Machine learning with neural networks that have many layers ("deep" means many layers). It powers face recognition, speech recognition, image generators and LLMs.
deep neural network: A neural network with many hidden layers. Each layer builds on the one before it, from simple features to complex ideas.
distillation: Training a smaller “student” model to imitate a bigger “teacher” model, keeping much of its skill at a fraction of the cost.
embedding: A list of numbers that represents a token's meaning. Tokens with similar meanings get similar lists, so they sit close together on a "map of meaning".
embedding model: A model that turns a piece of text into a single vector capturing its meaning, so similar texts get similar vectors.
epoch: The technical name for one practice round over all the training examples.
feed-forward: The part of each transformer block that processes every token on its own, through two big layers of neurons. It holds most of the model's weights.
few-shot prompting: Including a few worked examples of a task in the prompt, so the model copies their pattern and format.
fine-tuning: Extra training on a small, carefully chosen dataset to teach a pre-trained model a specific behaviour, like answering questions.
floating point: The way computers store decimal numbers: a sign, an exponent (how big) and a mantissa (the digits), like scientific notation (3.14 × 10²).
FLOP: A FLoating-point OPeration: one multiplication or addition of decimal numbers. "FLOP/s" counts how many a chip can do per second.
forward pass: Running an input through the network, layer by layer, to get its answer. Every time you use a model, it does a forward pass.
GPU: Graphics Processing Unit. A chip with thousands of simple workers that all do math at the same time. Built for video games, now the engine of AI.
gradient: The technical name for the "which way and how much" signal that tells each weight how to change.
gradient descent: How almost every neural network learns: measure the mistakes, work out which way each weight should move to reduce them (the gradient), take a small step that way, and repeat.
hallucination: A fluent, confident statement from an AI model that is false or made up.
HBM: High Bandwidth Memory: the GPU's main memory, stacked right next to the chip. Big (tens of GB) and fast compared to normal memory, but slow compared to the chip itself.
hidden layer: Neurons between the input and the output. They are "hidden" because you never see their answers directly; they work out helpful in-between ideas.
hybrid search: Combining keyword search and meaning search and merging their results, so each covers the other’s blind spots.
inference: Using a trained model to get answers. The opposite of training.
input: The number(s) you give the network: the question.
JSON: A simple text format for structured data: named fields inside curly brackets, easy for programs to read and write.
kernel: A program that runs on the GPU. Thousands of threads all run the same kernel, each on its own piece of the data.
knowledge cutoff: The date after which a model’s training data stops. It knows nothing about later events.
KV cache: Memory where the model keeps the keys and values it already computed for earlier tokens, so it does not redo that work for every new token.
label: The right answer attached to a training example, such as "spam" for an email or the sale price for a house.
latency: How long one user waits: for the first token, between tokens, or for the whole answer.
layer: A group of neurons that all work on the same inputs at the same time. A network passes numbers from one layer to the next.
learning rate: The technical name for the step size.
learning step: One round of practice: the network checks its answers on all examples, then nudges every weight and bias a little to make fewer mistakes. Experts call one round an "epoch".
LLM: Large Language Model: a huge neural network trained on enormous amounts of text to predict the next token. ChatGPT, Claude, Gemini and Llama are LLMs.
long-term memory: Facts an agent saves outside the model, such as user preferences, and retrieves in later conversations when they are relevant.
LoRA: Low-rank adaptation: fine-tuning by freezing the original weights and training two small matrices whose product is added to them.
loss: The technical name for the mistake score. Lower is better; 0 means every answer is perfect.
machine learning: A way to build AI where the computer finds the rules itself, by spotting patterns in lots of examples, instead of a programmer writing them.
matrix: A grid of numbers with rows and columns. A layer of a network keeps its weights in a matrix.
matrix multiplication: Multiplying two grids of numbers: each output is a row of one grid times a column of the other, summed up. Almost all of the work in a neural network is this.
MCP: Model Context Protocol: an open standard for connecting AI apps to tools and data, so one server works with many apps.
memory bandwidth: How many bytes per second can travel between memory and the chip. Often the real speed limit for AI, not the math.
metadata: Extra information stored with a document, such as its source, date, department or who may see it.
mistake score: How wrong the network is on average across all training examples. Each mistake is squared first, so big mistakes count extra. Experts call this the "loss".
mixture of experts: A model design with many expert sub-networks per layer, where a router sends each token to only a few of them.
model: What learning produces: a program whose behaviour is set by the numbers (parameters) it learned from data. You give it an input and it gives back a prediction.
multi-agent system: Several AI agents, each with its own context and job, working together on a larger task.
multi-head attention: Running several attention calculations side by side, each with its own learned weights, so a layer can track several kinds of relationship at once.
neural network: A program made of many small calculators (neurons) connected together. Instead of being told the rules, it learns them from examples.
neuron: One small calculator. It takes numbers in, multiplies each by a weight, adds a bias, and passes the result on.
nudge: The network works out, for every weight, "if I made this a bit bigger, would the mistakes get smaller or bigger?" and moves it in the helpful direction. Experts call this direction the "gradient".
OCR: Optical character recognition: reading the text out of an image, such as a scanned page.
open-weight model: A model whose trained weights are published for anyone to download, run and fine-tune, under the terms of its licence.
output: The number the network gives back: its answer.
overfitting: When a model memorises its training examples, noise included, instead of learning the general pattern. It looks great on those examples and poor on new ones.
PagedAttention: Storing the KV cache in small pages handed out on demand anywhere in memory, instead of reserving one big block per request.
parallel: Doing many things at the same time instead of one after another.
parameter: Any number the model learns: every weight and every bias. "A 70B model" means a model with about 70 billion of them.
pattern: A regularity in data, like "bigger houses usually cost more". Machine learning finds patterns in examples and uses them on new cases.
pipeline parallelism: Splitting a model’s stack of layers into blocks on different GPUs, with each token passed from one GPU to the next.
positional encoding: Extra information added to each token’s vector to say where it sits in the text, so the model can tell word order.
pre-training: The first and most expensive training stage: learning to predict the next token on trillions of words of text.
precision: How many bits a number uses. More bits = more exact, but more memory and slower math.
prediction: A model's answer for an input it has not seen before, based on the patterns it learned. It can be a number, a label ("spam") or the next word.
prefill: The first phase of answering: the whole prompt is read in one parallel pass. It keeps the GPU’s maths units busy.
probability: How likely something is, from 0% (never) to 100% (certain). An LLM gives every possible next token a probability.
prompt: The text you give a language model: your question or instruction, plus anything else in the conversation.
prompt engineering: The craft of writing prompts that give a model everything it needs: purpose, audience, format, facts and constraints.
prompt injection: An attack where text the AI reads, such as an email or web page, contains instructions meant to hijack what it does.
quantization: Storing a model's numbers with fewer bits (for example 8 or 4 instead of 16) to save memory and gain speed, at a small cost in accuracy.
RAG: Retrieval-augmented generation: finding relevant passages in your own documents and adding them to the prompt so the model answers from them.
rate limit: The most requests or tokens an account may send per minute. Going over it gets a “429 Too Many Requests” error.
ReAct: Short for “reasoning and acting”: an agent pattern that alternates written reasoning with tool calls and their observations.
reasoning model: An LLM trained, usually with reinforcement learning, to think at length in writing before answering.
register: The tiniest, fastest memory: a few slots inside each worker that hold the numbers it is calculating with right now.
reinforcement learning: Learning by trial and error: the model acts, gets a reward or a penalty, and gradually prefers the actions that paid off.
ReLU: Short for "rectified linear unit": a squish function that turns negative numbers into 0 and lets positive ones through unchanged.
reranker: A model that reads the question together with each candidate passage and reorders a shortlist by how well each one answers it.
reward: In reinforcement learning, the score the model receives after an action: positive for good outcomes, negative for bad ones.
reward model: A helper model trained on human ratings that predicts how much people would like an answer.
RLHF: Reinforcement Learning from Human Feedback: people rate answers, and the model is tuned to produce the kind of answers people prefer.
RNN: Recurrent neural network: an older kind of language model that reads one word at a time and carries a single running memory vector.
scaling law: A predictable relationship between a model’s size, its training data, the compute used and how good it gets.
self-supervised learning: Learning from data that supplies its own answers, for example hiding the next word of a sentence and asking the model to guess it.
semantic search: Searching by meaning rather than exact words, by comparing embedding vectors of the question and the documents.
serving engine: Software that turns a model into a fast shared service by scheduling requests, managing memory and running the model efficiently, such as vLLM.
shared memory: A small fast memory shared by one group of GPU workers (one SM). Programmers fill it on purpose with data the group will reuse.
sigmoid: A squish function that turns any number into a value between 0 and 1, often read as "how sure" the neuron is.
SM: Streaming Multiprocessor: one "workshop" on an NVIDIA GPU with its own workers, registers and shared memory. An H100 has 132 of them.
softmax: A formula that turns any list of scores into probabilities that add up to 100%. Higher score = bigger share.
special token: A token that isn’t ordinary text but a marker, such as “start of the user’s message” or “end of the answer”.
speculative decoding: Letting a small model guess several tokens ahead and having the big model check them all in one pass, keeping only what it agrees with.
squish function: An extra step some neurons do at the end to reshape their result. For example, sigmoid squishes any number into the range 0 to 1, useful for yes/no answers. Experts call this an "activation function".
step size: How big each nudge is. Too small and learning is slow. Too big and it overshoots and goes wild. Experts call this the "learning rate".
structured output: Making a model return data in an exact format, such as JSON that matches a schema, often enforced while it generates.
supervised learning: Learning from examples that come with the right answer attached, like photos labelled "cat" or "dog".
system prompt: Hidden instructions from the app, sent first with every request, that set the assistant’s role, tone and rules.
temperature: A setting that controls how adventurous the model is when picking the next token. Low = safe and repetitive, high = creative but riskier.
tensor core: A special unit in modern NVIDIA GPUs that multiplies small matrices in one go, much faster than ordinary cores, especially with low-precision numbers.
tensor parallelism: Splitting every layer of a model across several GPUs, which work on each token together and swap results after every layer.
test set: Data locked away until the very end and used once, to measure how well the finished model handles examples it has never seen.
thread: One small sequence of instructions run by one GPU worker, usually responsible for one tiny piece of the result.
throughput: How much work a system does in total, such as tokens per second summed over all users.
tile: A small square piece of a big matrix, sized so it fits in fast memory.
time to first token: How long you wait before the first word of an answer appears: queueing, reading the prompt and network time. Often shortened to TTFT.
token: A small piece of text (a word or part of a word) that a language model reads or writes one at a time.
tokenizer: The program that cuts text into tokens and turns them into ID numbers (and back again).
tool calling: How a model uses tools: it writes a structured request for a function, the app runs it, and the result is sent back to the model.
top-p: A setting that throws away the unlikely tokens, keeping only the most likely ones that together cover p (for example 90%) of the probability.
traditional software: A program that follows exact step-by-step rules written by a programmer. It does exactly what it was told, and nothing it was not told.
training: Repeating the learning step many times so the mistake score keeps going down.
training data: The examples the network learns from: questions together with their correct answers.
training set: The part of the data the model learns from.
transformer: The neural network design behind all modern LLMs (2017). It is a tall stack of identical blocks, each with an attention step and a feed-forward step.
underfitting: When a model is too simple to capture even the real pattern, so it does poorly on old and new examples alike.
unsupervised learning: Learning from data with no answers at all. The model finds structure on its own, such as groups of things that look alike.
validation set: Data held back from training and used to make choices, such as which model or settings to keep.
vector: A list of numbers, which you can picture as an arrow or a point in space. Embeddings are vectors.
vector database: A database that stores vectors with their text and metadata, and quickly finds the vectors closest to a question’s vector.
vocabulary: The fixed list of all tokens a model knows, typically 30,000 to 200,000 of them. Each has an ID number.
VRAM: The memory on a graphics card (on AI GPUs this is HBM). The model must fit in it to run fast.
warp: A team of 32 threads on an NVIDIA GPU that always execute the same instruction at the same time.
weight: A number on a connection that says how much a signal counts. Big = it matters a lot. Negative = it pushes the answer the other way. Learning = adjusting weights.