Software and versions used
This explanation refers to:
- Concepts: Prompts, tokenization, system prompts, temperature
- Context: Tool-independent prompt fundamentals
Two questions with identical meaning can produce completely different answers. At first glance, this seems contradictory—a model that supposedly “understands” should respond the same way to the same intent. That apparent contradiction disappears once it becomes clear what a prompt actually influences.
A prompt is the starting point of the computation, not a question asked of another party
How a Language Model Works describes inference as a token-by-token computation: every token is predicted based on all previous tokens. A prompt is nothing more than the first part of this context—even before the model generates a single token of its own, the foundation for every subsequent computation has already been established.
Why wording changes the answer even though the meaning stays the same
The model has no concept of “meaning” separate from the exact wording. It calculates probabilities based on the tokens that are actually present—not on what the human intended. “Explain Docker” and “What is Docker?” may mean the same thing to a person; for the model, they are different token sequences that were associated during training with slightly different continuation patterns. “Explain X” appears more often in training data before detailed explanations, while “What is X?” more often precedes concise definitions. The model reacts to the wording, not to the intent behind it.
This also explains why two people can evaluate the same prompt differently: they bring different expectations to the same wording. The model itself has no knowledge of those expectations—it responds exclusively to what is actually present in the context.
Tokenization breaks text into pieces because a model cannot maintain an unlimited vocabulary
Before a prompt is processed, a tokenizer breaks the text into fragments called tokens. A model could not maintain a separate entry for every conceivable word in a language; the number of possible word forms, compounds, and technical terms is effectively unlimited. Instead, the tokenizer learns a limited set of common fragments—frequently occurring words often remain intact, while rare or compound words are divided into smaller pieces.
German compound words are therefore often split more aggressively than English words. “Containerisierung” might be divided into “Container” and “isierung” because the model encountered the complete combination less frequently during training than the individual parts. This is not a weakness of the German language, but a consequence of training data being predominantly English. The tokenizer learns the fragmentation that was most efficient for the training data.
More fragments per word mean more tokens per prompt—and therefore more of the context window consumed by the same amount of text. A prompt containing many technical terms or compound words can therefore use noticeably more context budget than a linguistically simpler prompt with the same word count.
System prompts establish a framework, but they do not enforce it
A system prompt is an instruction that is sent to the model before the actual user input. The model processes the system prompt and the user input within the same context. Many chat models additionally distinguish the different roles using special control tokens or chat templates. This distinction influences the learned behavior, but it does not enforce the instruction like an access rule implemented outside the model.
Why a model sometimes ignores a system prompt anyway
A system prompt is not a command in the sense of a rule the model must obey—it merely shifts which continuations become more likely. A system prompt such as “Always answer in German” significantly increases the probability of German tokens, but it does not completely exclude English tokens. If the following context becomes long and is strongly shaped by English—for example through English technical terms or quoted source code—that context can outweigh the original instruction. Prompt injection follows a related principle: an opposing instruction placed later in the context competes with the system prompt for the same probability distribution. How strong this competition becomes depends on the model’s training, the role markers, the surrounding context, and the specific wording—there is no fixed hierarchy in which later text always wins.
System prompts do not modify the model’s parameters—they affect only the current inference context. Every new conversation begins without previous system prompts unless they are supplied again.
Prompt engineering takes advantage of how wording shifts probabilities
Because the model responds to the actual wording, a more precise formulation also changes the answer—not because the model “understands better,” but because a more precise formulation was associated during training with more suitable continuations.
Why examples in a prompt help
An example in a prompt—for example, a question preceded by a matching example answer—does not define new knowledge. Instead, it establishes a pattern within the immediate context. The model does not have to “learn” this pattern in the sense of training; it simply applies its existing ability to use the preceding context for the next prediction—the same ability that keeps a multi-turn conversation coherent. An example in a prompt acts like a very concrete writing pattern, not an instruction.
Why longer prompts do not automatically produce better answers
Self-attention relates every token to every other token in the context, but not as a fixed, evenly distributed budget. A long prompt containing irrelevant information provides the model with additional, potentially competing patterns. The relevant instruction remains in the context, but it may become less reliable when surrounded by unimportant details, repetitions, or contradictory hints. More text therefore does not automatically mean more relevant information. A short, precise prompt reduces this competition and makes the actual instruction more consistently effective.
Chain-of-Thought prompting—the explicit request to proceed step by step—uses a similar effect. The request shifts the probability toward continuations that themselves contain intermediate reasoning steps. Those intermediate steps then become part of the context for the following tokens, so a more structured beginning encourages a more structured continuation.
Temperature changes not the computation itself, but the selection from its result
For every token, the model calculates a probability distribution over the possible continuations, regardless of the temperature setting. Temperature comes into play only afterward: when selecting which token is actually chosen from that distribution.
Why the same distribution can lead to different answers
At a low temperature, the most probable token is chosen almost every time, making the output deterministic and nearly identical across repeated requests. At a higher temperature, less probable tokens also have a realistic chance of being selected. The underlying probability distribution itself does not change—only the rule used to choose from it. That is precisely why the same prompt can produce a different, yet still plausible, answer each time when using a higher temperature.
Very low temperature values produce repetitive, less natural text because the same most probable tokens are selected every time. Very high values increasingly allow unlikely tokens and can eventually lead to incoherent output. Besides temperature, additional techniques exist to constrain the selection from the probability distribution—for a basic understanding, however, the concept of temperature is sufficient, while the details depend on the specific tool and use case.
Context length limits how much of a prompt can fit
Local Models explains the context window as the fixed upper limit of processable tokens. A long prompt consumes part of this budget before the model even begins generating a response—a 4K context window with a 3K-token prompt leaves only 1K tokens available for the answer.
Multi-turn conversations accumulate over time. Every additional message extends the context until the window limit is reached. Once that limit is exceeded, the oldest messages are removed. The model loses that part of the conversation completely, without being aware that anything has been lost.
Uniformly structured training data can reduce later flexibility
A model that sees almost exclusively prompts in a fixed format during fine-tuning—for example, consistently using “Question: … Answer: …"—may begin treating that format as part of the expected response structure. As a result, it may reproduce those formatting markers even when the current prompt does not require them.
The cause does not lie in the current prompt, but in the patterns reinforced during fine-tuning. Highly uniform training data can therefore reduce the flexibility with which a base model originally responded to different prompt formats.
Summary of the relationship
A prompt is not a command, but context that influences the probability distribution of every subsequent token—just as previously generated tokens do. Tokenization determines the units in which that context exists in the first place. System prompts, examples, and precise wording all work through the same mechanism: they shift which continuations become more likely without changing the model itself. Temperature comes into play only after this computation, during the selection from the already established probability distribution.
The next article in the series covers fine-tuning. Prompting changes the context. Fine-tuning changes the model itself.
