When I first encountered programming, someone told me it wasn’t for my “artistic brain.” Years passed, I studied law, I freelanced, but mostly spent countless hours diving into the world wide web.

Then 2023 happened, and – MIND. BLOWN 

How can a computer understand what I’m saying and respond so… naturaly? So of course, I had to know.

It starts with Tokenization

How does the program understand?

It’s through tokenization, and that means breaking down language into tiny, digestible pieces that a computer can understand. And it’s way more than turning each word into a number.

Note to self: How much computer power is needed to break every letter in token and would it mean more or overburden.

Let me show you what I mean.

When you type “unlikeliest”, the AI doesn’t see it as one word. Instead, it breaks it down into pieces like “un” + “like” + “li” + “est”. Each of these pieces is called a token, and each token gets converted into a number.

Why do it this way? Well, think about how many words exist in English. Now add all the misspellings, slang, and new words people create. It would be impossible to have a number for every possible word!

Ok, not yet.

But what will happen when we turn the impossible to possible, and attach number to every letter? Are all the possible combinatons, how much numbers there are? I have no idea what I’m talking about, like the image above, all context is fluid.

Oops.

Ok, Let’s go back to the here.

Instead, by breaking words into smaller pieces, AI can understand even words it’s never seen before.

This is actually pretty similar to how we learn new words ourselves. When you see a word like “ungooglable” (yup, that’s a real word!), your brain naturally breaks it down into pieces you know: “un” + “google” + “able”.

Token breakdown up closely

Let’s look at how some words get split up :

“butterflies” → “butter” + “flies”
(Makes sense, right?)

“homework” → “home” + “work”
(Pretty logical so far…)

“cryptocurrency” → “crypto” + “curr” + “ency”
(Not quite what you’d expect!)

Let’s look at some modern language:

“blockchain” → “block” + “chain”
(Simple enough)

“#ThrowbackThursday” → “#” + “Throw” + “back” + “Thursday”

The same word might get split differently depending on what’s around it. Take the word “ing”:

“jumping” → “jump” + “ing”
“walking” → “walk” + “ing”
“ring” → “ring” (not “r” + “ing”!)

In next phase, each one gets transformed into a vector with hundreds of dimensions, let’s get into it.

From Tokens to Vectors

We downplayed when we said that each token gets turned into a number? More correctly, each token actually gets turned into a whole LIST of numbers – typically hundreds of them! That’s vector and it’s where things get really trippy.

So when you think of the word “king”, your brain doesn’t just think “male monarch.” You might think:

  • Male ✓
  • Royal ✓
  • Powerful ✓
  • Human ✓
  • Leader ✓
  • Racist ✓

Each of these attributes could be represented by a number. So instead of one number, “king” becomes something like: [0.8, 0.9, 0.95, 1.0, 0.9, … ] (but hundreds more numbers)

In this mathematical space:

  • “king” – “man” + “woman” = “queen”
  • “paris” – “france” + “italy” = “rome”
  • “walking” – “walk” + “swim” = “swimming”

It’s doing math with meaning!

Even more fascinating it is, that the AI figured out these relationships BY ITSELF just by reading tons of text. It noticed that “king” appears in similar contexts as “queen” except when gender is involved, and it captured that in these number patterns.

If you think this is magical, wait till you think about how LLMs use all these number lists to actually understand and respond to us. Attention. And that’s a whole another chapter. Stay tuned.

What is the definition of Vector

In AI, a vector is simply a list of numbers that represent something’s characteristics or features. Like a very detailed ID card for a word, or fingerprint, even better analogy.

Vector Math

Let’s break this down with the “king – man + woman = queen” example. I’ll use smaller vectors (just 5 numbers instead of hundreds) to make it perceivable:

Let’s say our vectors look like this:

king = [0.8, 0.7, 0.9, 0.3, 0.6]
man = [0.7, 0.3, 0.4, 0.3, 0.2]
woman = [0.7, 0.3, 0.4, -0.3, 0.2]

Where these numbers might represent things like:

  • How royal something is (0-1)
  • How powerful (0-1)
  • How respected (0-1)
  • Gender (-1 for female, +1 for male)
  • How human (0-1)

When we do “king – man + woman”, we’re literally:

  1. Subtracting man’s numbers from king’s numbers
  2. Adding woman’s numbers
  3. Looking for what word has a vector closest to our result

Like this:

king = [ 0.8, 0.7, 0.9, 0.3, 0.6]
man = [ 0.7, 0.3, 0.4, 0.3, 0.2]
king-man= [ 0.1, 0.4, 0.5, 0.0, 0.4]
woman = [ 0.7, 0.3, 0.4, -0.3, 0.2]
result = [ 0.8, 0.7, 0.9, -0.3, 0.6]

This result vector ends up being super close to the vector for “queen”!

The mind-blowing part? The AI learned these numbers BY ITSELF, by reading lots of text and noticing how these words are used.

Context windows and token limits – when ChatGPT cracks

When GPT start having amnesia, it’s hitting its token limit.

Like trying to remember a conversation where you can only keep the last 50 sentences in your head. No matter how interesting the discussion, those first sentences have to go to make room for new ones. That’s basically what’s happening with these models.

But here’s where it gets interesting: counting tokens isn’t as simple as counting words. Remember our token breakdowns from earlier? Let’s see how this plays out in real life:

“I love programming” = 3 words but 3 tokens
“I love tokenization” = 3 words but 4 tokens (token + ization)

Different models have different memory limits. That’s about memory and processing power – the longer the context window, the more relationships between tokens the model needs to track. It’s like trying to keep track of how every sentence in a book relates to every other sentence – boom.

Special tokens or how AI punctuates

You know how in texting, we have all these unwritten rules? Like how “ok.” feels passive-aggressive but “ok!” feels friendly? AI models have their own special characters that tell them important stuff, but these are more structured.

i.e:

<|endoftext|> – This is like the AI’s version of “THE END”. It’s how the model knows one piece of text is finished and another is starting. Think of it as the AI’s version of turning the page.

[INST] and [/INST] – Some models use these to know when they’re getting instructions versus when they’re supposed to generate content. It’s like “I have a question” instead of spilling the question on the first touch point.

[SYS] – This is like setting the mood. When models see this, they know they’re getting their character background. Like psychographics for systems.

Multilingual Tokenization

Why GPT uses up your tokens faster when you’re writing in a different language?

This is why:

“I love programming” = 3 tokens
“Me encanta programar” = 3 tokens
“私はプログラミングが好きです” = 8-9 tokens!

This difference it’s because of how writing systems work.

Let’s show it with an example and break down “programming” in different languages:

programming = program + ming
プログラミング = プ + ログ + ラ + ミ + ング

Chinse don’t use spaces between words:

“我喜欢编程” (I love programming)

How does the model know where one word ends and another begins? It figures it out character by character.

This is why you might run out of tokens faster when writing in Asian languages, the model is juggling different tokenization systems at the same time.

Model’s vocabulary

Think about how many words exist in English. Spanish. Japanese. Add every programming language, emoji, hashtag, and that weird way your friend types “hellooooooo” with too many o’s.

This is exactly the problem AI models face. They can’t possibly have a token for every possible combination of characters (not yet), so they have to make some tough choices about their vocabulary size. Let’s see this:

Common words: Get their own tokens
“the” = 1 token
“and” = 1 token
“programming” = 1 token

Less common words: Get split
“tokenization” = token + ization
“uncommon” = un + common

Rare/New words: Get broken down into tiny pieces
“cryptocurrency” = cry + pt + o + curr + ency
“NFT” = N + F + T

Most models have a vocabulary of around 50,000 tokens. That might sound like a lot, but:

  • English alone has ~170,000 words
  • Add technical terms
  • Add internet slang
  • Add other languages
  • Add all possible misspellings…

And here’s where it gets interesting – when the model encounters a word it doesn’t have in its vocabulary, it breaks it down into the smallest pieces it knows. This is why sometimes when you ask it about very new terms (like latest tech or slang), it might:

  1. Break them down weirdly
  2. Understand them differently
  3. Use more tokens than you’d expect

Practical stuff: making friends with tokens

Let me show you how this vocabulary thing plays out in real life and why it matters when you’re chatting with AI:

When models get creative (not always in a good way)

Take something like “blockchain” – when this was a new term, models would break it down as:

blockchain = block + chain

Makes sense! But what about:

dogecoin = dog + e + coin
bitcoin = bit + coin
web3 = web + 3

This breaking-down process affects how the model understands these terms. Sometimes it gets lucky – like with “blockchain”, the meaning is pretty clear from its parts. But other times… well, let’s just say there’s a reason why AI sometimes gets confused about crypto terms!

Why this matters for you

Here’s how knowing this helps you work better with AI:

  1. Writing Prompts:

Bad: “Plz explane blockchane tecnology”
Better: “Explain blockchain technology”

The first one wastes tokens on misspellings AND might confuse the model!

  1. Technical Discussions:

Less efficient: “Can you tell me about cryptocurrency blockchain distributed ledger technology?”
More efficient: “Explain blockchain tech”

Same question, fewer tokens!

  1. When Using Code:

This whole block might use more tokens than you think

def analyze_cryptocurrency_blockchain_distributed_ledger():
pass

vs

More token-efficient

def analyze_blockchain():
pass

And here’s my favorite tip: when you’re hitting token limits, try rephrasing with simpler, more common words. The model might actually understand you better, AND you’ll use fewer tokens!

Putting it all together: from letters to understanding

Ok, so let’s connect all these pieces we’ve explored – it’s like following the journey of your message from the moment you hit “send” to getting an AI response:

  1. Tokenization (Breaking it Down)
  • Your text gets chopped into tokens
  • Each word might become one or more pieces
  • Special tokens get added (like markers for instructions)
  • Different languages get processed differently
  1. Vector Transformation (Making it Mathematical)
  • Each token becomes a vector (that list of hundreds of numbers)
  • These numbers capture meaning and relationships
  • Similar concepts get similar vectors
  • We can do “word math” (like king – man + woman = queen)
  1. Vocabulary Limitations (Working with What We’ve Got)
  • Model has a fixed vocabulary (around 50K tokens)
  • Common words get their own tokens
  • New or rare words get split up
  • This affects both understanding and token usage

And here’s the cool part – this is just the FIRST step in how AI understands language! All these tokens and vectors then feed into attention mechanisms, which figure out how all these pieces relate to each other.

Think of it like this:

  1. Tokenization breaks your message into pieces (like individual notes)
  2. Vectors give each piece meaning (like the pitch and duration of notes)
  3. Attention mechanisms figure out how they work together (like turning notes into music)

Why should you care about all this?

You don’t.

Care only if you want.