AI Glossary

Also: tokens

The unit a language model reads and bills by — roughly four characters of English, so a token is often part of a word rather than a whole one. Prices are quoted per million tokens.

The component that splits text into tokens before the model sees it. Each model family uses its own vocabulary, so the same sentence can be a different number of tokens on GPT, Claude or Gemini.

Also: context length, context size

The maximum number of tokens a model can hold at once — your prompt plus its reply. Everything beyond it is truncated or has to be dropped.

The full text you send the model: instructions, examples, documents and the question. The model only knows what is in the prompt plus what it learned in training.

Also: system message

A separate instruction that sets the model's role and rules for a whole conversation, sent ahead of the user messages and usually resent on every turn.

Also: output, generation

The text the model generates in response to your prompt. Output tokens are billed separately from input, almost always at a higher rate.

A setting from 0 to about 2 that controls randomness. Low values make the model pick the most likely next token (focused, repeatable); high values spread the choice out (varied, less predictable).

Also: nucleus sampling

An alternative to temperature that limits the model to the smallest set of next tokens whose probabilities add up to p. Lower p keeps output on the safe, likely path.

When a model states something false or invented as if it were fact. It happens because the model predicts plausible text, not verified truth — grounding it with retrieved sources reduces but never removes it.

Also: large language model

A large language model — a neural network trained on huge amounts of text to predict the next token. GPT, Claude and Gemini are all LLMs.

Also: weights

The learned numbers inside a model, counted in billions. More parameters can mean more capability but also higher cost and latency; a bigger model is not always the right one.

Running a trained model to get an answer, as opposed to training it. Every API call you make is an inference call, billed by the tokens in and out.

Continuing to train a base model on your own examples so it adopts a style or task. It has a one-time training cost and the resulting model is billed per token at inference like any other.

Also: vector, embeddings

A list of numbers that captures the meaning of a piece of text so similar texts sit close together. Embeddings power semantic search and retrieval.

Also: vector store

A database built to store embeddings and find the nearest ones to a query fast. It is the memory a retrieval system searches at query time.

Also: retrieval augmented generation

Retrieval-augmented generation — fetching relevant documents and pasting them into the prompt so the model answers from your data instead of memory alone.

A provider feature that stores a repeated prompt prefix (like a long system prompt) so resending it costs a fraction of the normal input rate.

Also: requests per minute

Requests per minute — a rate limit on how many API calls you can make each minute, separate from the token limit.

Also: tokens per minute

Tokens per minute — a rate limit on the total tokens (input plus output) you can push through each minute. Large requests hit this before the request limit.

The time between sending a request and getting the response. It grows with output length and model size; streaming hides it by showing tokens as they arrive.

Returning the completion token by token as it is generated, so the user sees text appear immediately instead of waiting for the whole reply.

Also: tool use, tools

Letting a model return a structured request to call one of your functions — check the weather, query a database — so it can act on the world instead of only writing text.

A system that lets a model plan and take multiple steps toward a goal, calling tools and reacting to results in a loop rather than answering in one shot.

Also: CoT, reasoning

Prompting a model to work through a problem step by step before answering, which improves accuracy on maths and logic at the cost of more output tokens.

Also: in-context learning

Putting a handful of worked examples in the prompt so the model copies the pattern. Zero-shot means giving none and relying on the instruction alone.

Asking a model to do a task with only an instruction and no examples. It works when the task is common; harder tasks usually benefit from a few examples.

The neural-network architecture behind modern language models. Its attention mechanism lets every token weigh every other token, which is what made large-scale language models work.

The mechanism a transformer uses to decide which other tokens matter when processing each token. It is why context length is expensive — cost grows quickly as the window gets longer.

Storing a model's weights at lower numeric precision (say 8-bit instead of 16-bit) to shrink memory and speed inference, usually with a small quality trade-off.

Training a smaller "student" model to imitate a larger "teacher" so you get most of the quality at a fraction of the cost and latency.

The first, most expensive training phase, where a model learns language from a huge general corpus. Fine-tuning and alignment come afterward on top of it.

Also: reinforcement learning from human feedback

Reinforcement learning from human feedback — using human preference ratings to steer a model toward helpful, safe answers after pre-training.

The work of making a model behave the way people intend — helpful, honest and safe — rather than just fluent. RLHF is one common alignment technique.

Also: vision, vision model

A model that handles more than text — images, audio or video as well. Vision models turn an image into tokens and bill for them alongside the text prompt.

An attack where hidden instructions in retrieved or user content trick a model into ignoring its real rules. A leading risk for agents that read untrusted text.

Another name for the context window — the token budget a model can process in a single call, shared between your input and its output.

Also: max output

A cap you set on how many tokens the model may generate in its reply. It controls cost and stops runaway output, but set too low it truncates the answer.

A cheaper, slower processing mode where you submit many requests at once and collect results later — often around half price for work that is not time-sensitive.

Checks around a model that filter inputs or outputs — blocking unsafe content, enforcing a format, or catching prompt injection — so the raw model is not trusted blindly.

Tying a model's answer to specific supplied sources so it can cite them and is less likely to invent facts. Retrieval is the usual way to ground a response.