How LLMs Actually Work: Tokenization, Vectorization, Generation

A language model doesn't understand language. At least not the way we understand it. It gives the impression that it understands. It behaves as if it understands. But underneath, there's a massive counting machine. Let's look under the hood — along the way, you'll also get better at prompting.

Tokenization: turning language into numbers

This part of the process is actually pretty simple to explain.

[30720, 3577, 164251, 12637, 49755, 24762, 88, 621, 14962, 2067, 2259, 135779, 13, 220]

For you, language is made of letters, words, sentences, and paragraphs sprinkled with punctuation. For a language model, everything is made of tokens. The Polish sentence “Ten element procesu jest bardzo prosty do wyjaśnienia” (“This part of the process is really simple to explain”) turns into the string of numbers shown above. A token isn't the same as a word — that sentence has eight words, fifty-five characters, and gets represented as fourteen tokens. Thirteen, if I drop the space after the period. If you want to play around with turning language into tokens yourself, OpenAI has a tool called Tokenizer.

What can you learn from playing with the Tokenizer?

• One token is roughly four letters. Though — as you can see in the example above — Polish is actually very token-efficient. Only the word “wyjaśnienia” (“explain”) breaks into more than one token.
• The same words get different tokens depending on context. Think of “bank” as in a river bank versus “bank” as in where you keep your money — different meaning, different token.
• Punctuation marks, spaces, and special characters — proper quotation marks (“ ”) or an ellipsis (…) — all have their own tokens. The model doesn't see sentences or paragraphs. It only ever sees strings of numbers.

Tokenization means the model no longer has to deal with the raw elements of language (words, sentences, paragraphs, letters, commas). Everything becomes a number. Think of a token as a numeric ID assigned to every unit of language.

Vectorization: giving tokens meaning

This next part is a genuinely interesting exercise. Imagine we take a dictionary and tokenize it. That means: we assign a numeric ID to every word we find in it. No cleverness involved — just alphabetically, one after another. We end up with a long list.

Now imagine that long list is actually a long line stretched out in space. A mathematician would call that a “vector.” This vector has one dimension — length. Give it the right number, and you land at a specific point on that vector and get the word that corresponds to that number on the list. Because we've moved from the world of abstraction into the world of geometry, we can start talking about relationships between words. Some words sit close together, others sit far apart.

It quickly becomes clear that the alphabetical, arbitrary ordering we started with doesn't make much sense. On our vector, the word “lampshade” sits closer to “amnesia” (because they start with the same letter) than to “lamp” (which it's logically related to). So maybe we should replace the alphabetical vector with something more meaningful? But with what?

First idea: let's build a vector that captures how “tangible” something is. On one side we'd put words describing things you can touch, kick, or knock off a table — “lampshade” and “lamp,” for instance. On the other end of that axis we'd put things with no physical properties at all — like “war” — and further still, fully abstract things, like “hope.” On that scale, “lamp” and “lampshade” would sit much closer together than either does to “amnesia.” Makes sense, right?

Now consider that we don't have to limit ourselves to a single dimension. If you put a “tangibility” vector on one axis and add a “positivity” vector on another (is the word used in a positive or negative context?), you get a two-dimensional space where words cluster into “blobs.” This is where tokens start to matter — because when I try to figure out where to place the word “turkey” on that axis, its position depends on whether I'm thinking of the bird (neutral) or using it as an insult for a person (negative).

Watch this:

• turkey (the bird) [1; 0] — it gets a one on the “tangibility” scale and zero (the midpoint) on the “positivity” vector;
• turkey (the insult) [1; -1] — used as an insult for a person, it gets a one on the tangibility scale and minus one on the positivity scale;
• war [-0.7; -1] — war isn't tangible, but it has a beginning and an end, so let's give it minus 0.7 on tangibility and minus one on positivity.

In this scheme, the words “lamp” and “lampshade” would occupy nearly the same space: both are tangible and both are neutral. And there can be many more dimensions than these two. For example:

• whether the word refers to something living;
• whether the word is contemporary;
• whether the word is slang;
• which language the word belongs to.

The more parameters you add, the more “dimensions of understanding” the model assigns to a single word, and the more precise the space that word ends up occupying. Modern models like ChatGPT use more than 12,000 (!) parameters to describe the meaning of a single token. That's something our brains genuinely can't wrap themselves around. It's as if every time you learned a new word, you asked twelve thousand questions to help you understand it.

Just as tokenization turned words into numbers, vectorization arranges those numbers into spaces. That's why GPUs are so much better suited to AI work — they were built to do calculations in space. An LLM doesn't see a table of numbers; it sees a multidimensional vector space.

Transformers and attention: what to actually look at for things to make sense

Vectorization on its own doesn't solve anything. It makes understanding easier, but meaning only shows up in the third step. The model looks at a token and asks itself: which other token should this one be tied to with an especially strong bond?

This is a lot like the sentence diagramming your grammar teacher tortured you with in school. Look at this example: “The boy, whose grandmother bought him a Harley-Davidson motorcycle the day before yesterday, is riding up the hill to the school located there.”

Before the paper Attention Is All You Need, language models worked by looking at neighboring words or keywords. But if we take the word “riding” and start asking who is riding, then — going purely by proximity — our strongest candidate is “motorcycle,” with “grandmother” right before it, and “boy” all the way back at the start of the sentence.

Transformers — massively simplified — rework that sentence using strong connections between some parts and weak connections between others. Here's how a transformer sees this sentence:

• The boy is riding to school — the first cluster of strong connections, anchored on “boy”
• Grandmother bought a motorcycle — the second cluster of strong connections, anchored on “grandmother”
• The motorcycle is a Harley-Davidson — anchored on “motorcycle”

And so on. Not every word matters equally, and not every connection is equally strong. A transformer reads a sentence the way an editor would (paying attention to meaning), not the way a stenographer would (reading everything in strict sequence). And the attention mechanism lets it ignore linguistic noise, tie pronouns to the right nouns, and spot relationships across long distances.

Probability distributions: ready to predict

We've got tokens, we've got vectors, we've got connections between them. At this point the model is ready to predict what comes next. One important note here: the model doesn't pick the next token. The model calculates the probability distribution of which token is likely to come next.

So, for example, given the sentence: “A good ad campaign is…” it might predict the following endings:

• …one that sells (28%)
• …the key to success (15%)
• …an investment, not a cost (12%)
• …the foundation of every business (9%)

All of those endings make sense, so if I were grading the model's performance, I'd say it “understands” language. None of the predicted results are nonsense. But if I actually want the model to generate text, I need to tell it which of these options I prefer. Because I won't always want the single most probable answer.

Two key parameters come in here:

• Temperature controls the model's “boldness.” Low temperature means the text is correct but as dull as a store's terms and conditions. High temperature means more creativity — the model reaches for less obvious options more often. The downside? A higher risk of generating nonsense. Important: temperature doesn't add creativity to the model. It adds randomness to the output — and we're the ones who interpret that randomness as creativity.
• Top-p, or nucleus sampling. This parameter controls how deep even that “bolder” model is allowed to dig. In plain terms: only pull tokens from the part of the distribution that adds up to 64% (that's the sum of the percentages in my example above). Cut off the rest. The higher you set top-p, the more often the model will surprise you mid-conversation with some exotic pick.

Important: in the models available to the average user, you can't directly fiddle with the temperature or top-p sliders. Instead, ChatGPT lets you choose a “personality.” Or you get models tuned for specific tasks. For example, the Gemini you talk to inside NotebookLM runs at a lower temperature than standard Gemini. Why? Because NotebookLM is meant to pull knowledge out of the material you give it without making things up, while standard Gemini is often used for creative tasks.

What does this mean for prompting?

Once you understand that an LLM doesn't “answer” — it continues a piece of text — prompting stops feeling like a conversation and starts looking more like setting a scene in a movie. If you drop a model into a vacuum and ask “write a brand strategy,” it'll do exactly what it does in a vacuum — reach for the most generic, safest version of “brand strategy” it has seen thousands of times. But if you instead start with “You are a brand strategist working with a B2B company that just lost 20% of its leads after a positioning change,” the probability distribution suddenly shifts into completely different territory. The model doesn't get smarter — it just gets a better starting point.

The same applies to style. The model doesn't read between the lines and can't sense the author's mood. If you don't tell it to write calmly, analytically, without coaching-speak and without “a rapidly changing market,” there's a very good chance that's exactly where it'll land. That's not spite — it's statistics. There was simply a lot of that kind of text in the training data. But add a single sentence like: “tone: matter-of-fact, dry irony in the background, zero marketing clichés,” and the model shifts into a completely different part of the language space. Not because it understood the irony, but because it has seen thousands of texts where that kind of instruction led to a different form.

The trickiest moment comes when the model doesn't know something. An LLM has no built-in mechanism for raising its hand and saying “I don't have that data.” If you don't set boundaries for it, it will generate an answer that sounds convincing. It might cite studies that don't exist, attribute a quote to the wrong person, or invent a definition that happens to fit the rest of the paragraph. That's not lying — it's pure continuation of language. That's why requests like “if you don't have the data, say so directly” or “cite your sources, and flag it if you don't know them” are some of the most effective prompting tools. They narrow the space of guessing.

In practice, a good interaction with an LLM looks less like asking questions and more like writing a brief for a very literal, very hard-working intern. You have to tell it who it is, what situation it's in, what register to speak in, and where its competence ends. Do that, and the model can be surprisingly useful. Skip it, and you'll get correct-sounding text that's indistinguishable from hundreds of other correct-sounding texts.

This isn't the end of the story — just another layer

Looking back at the road artificial intelligence has traveled, it's easy to spot something that gets lost in the daily noise of news and product launches. This isn't a story about machines suddenly waking up. It's a story about patiently taming complexity. About repeated attempts to write down fragments of the world in a form that can be counted, tested, and corrected.

From simple neurons acting as switches, through IF–THEN rules, brute-force Deep Blue, statistical algorithms, backpropagation, ImageNet, all the way to transformers and multimodal models — every stage promised something and fell short of something else. And every stage left behind tools that turned out to be invaluable later, once the technology finally caught up to them. In AI, very little ever really gets thrown away. More often it just lands on the “too early” shelf.

Today's language and multimodal models aren't the culmination of this story. They're more like the first moment when a lot of old ideas started working at the same time. Data scale met architecture, hardware caught up with algorithms, and statistics started to resemble competence. That's why today's AI looks so convincing — not because it understands the world, but because it has seen it, in the data, in an unimaginable number of variations.

This distinction matters, especially for marketers. AI isn't a new member of the team. It has no intent, no ambition, no brand vision. What it is, is an extremely sensitive instrument for picking up patterns — styles, narratives, argument structures, linguistic clichés, cultural shorthand. It can speed up work, widen the range of options, and lift the burden of mechanical writing off people. But it doesn't replace decisions. It doesn't know why we're doing something.

That's why the most meaningful shift AI brings isn't about replacing people — it's about moving the boundary between thinking and executing. Where you once had to manually generate dozens of versions, today it's enough to describe the problem well. Where you once burned time on form, today you can focus on substance. AI doesn't take away responsibility — it makes it more visible.

If anything about this history should ease your anxiety, it's its continuity. AI didn't appear out of nowhere, and it doesn't operate outside any known rules. It's just the next generation of algorithms doing what algorithms do best: taming complexity through simplification. Sometimes a little too well. Which is exactly why we still need people who know what actually matters, and what just sounds good.

At the end of this road, there's no artificial mind waiting. What's waiting is something far more mundane, but also far more useful: a statistical echo of the world we ourselves built, in data, in text, in images, and in decisions. An echo that can answer — as long as we ask it a sensible question, and know how to listen to the answer.