If you have used ChatGPT, Claude or Gemini to debug a college assignment or prepare for a placement interview, you have already used a transformer. The core idea is simpler than it sounds. This guide walks you through what a transformer is, why it replaced older models, and how the main pieces fit together, so that you can talk about it confidently in interviews and start building with it for free.
The problem before transformers
Before 2017, most language models were built on recurrent neural networks (RNNs) and their improved cousins, LSTMs. An RNN reads a sentence the way you might read with your finger on the page: one word at a time, left to right, carrying a small "memory" forward at each step.
This approach had two big weaknesses:
- It was slow to train. Because word five depends on the memory from word four, you cannot process all words at once. GPUs, which excel at parallel work, sat underused.
- It forgot things over long distances. By the time an RNN reached the end of a long paragraph, information from the beginning had faded.
In 2017, a group of researchers at Google published a paper titled "Attention Is All You Need". It proposed a new architecture, the transformer, that dropped recurrence entirely. Instead of reading word by word, a transformer looks at all the words in a sentence at the same time and lets each word decide which other words matter to it.
Key takeaway: RNNs read in sequence and forget. Transformers read everything at once and use attention to connect related words, no matter how far apart they are.
Step one: tokens and embeddings
A computer cannot read text directly, so the first job is to convert words into numbers.
Tokens
The text is split into small pieces called tokens. A token can be a whole word ("college"), part of a word ("intern" and "ship"), or punctuation. Sub word pieces let the model handle rare words and typos with a manageable vocabulary.
Each token is mapped to an ID from a fixed vocabulary. You can see this yourself with the Hugging Face transformers library:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
text = "Transformers helped me crack my internship interview"
print(tokenizer.tokenize(text))
print(tokenizer(text)["input_ids"])
Embeddings
Each token ID is then turned into an embedding: a long list of numbers (a vector) that represents its meaning. Words with similar meanings end up with similar vectors. "Placement" sits close to "job" and far from "mango".
At this stage, though, each embedding knows nothing about its neighbours. The word "bank" has the same vector whether you are talking about a river bank or a State Bank account. Fixing that is the job of self attention.
Self attention, the heart of the transformer
An everyday analogy
Imagine a group discussion round during campus placements. Ten students sit around a table. When one student speaks, everyone else quietly asks themselves: "Is what this person is saying relevant to my point?" Each student pays more attention to the people whose ideas connect with their own and less to the rest.
Self attention works the same way. Every token in a sentence looks at every other token, decides how relevant each one is, and then updates its own meaning by blending in information from the relevant ones.
Queries, keys and values
To make this precise, each token creates three vectors from its embedding:
- Query (Q): what am I looking for?
- Key (K): what do I contain that others might look for?
- Value (V): what information will I actually share if someone attends to me?
Think of a library: the query is your search, keys are catalogue labels, values are the books' contents.
A tiny worked example
Take the sentence "Riya submitted her resume". Let us focus on the token "her" and ask which word it should attend to. Suppose, using made up small numbers, the query for "her" is compared with each word's key using a dot product (multiply matching numbers and add them up):
- Riya: score 4.0
- submitted: score 1.0
- her: score 2.0
- resume: score 1.0
These raw scores are passed through a function called softmax, which turns them into weights that are all positive and add up to 1. Roughly, the weights come out as:
- Riya: 0.81
- submitted: 0.04
- her: 0.11
- resume: 0.04
Now the new representation of "her" is a weighted mix of every word's value vector: 81 percent from "Riya", a little from itself, and very little from the rest. In effect, the model has learned that "her" refers to Riya. Real transformers also divide scores by the square root of the vector size before softmax for stability.
Key takeaway: self attention lets each word ask "which other words help me understand my meaning here?" and then mix in their information, weighted by relevance.
Multi head attention and positional information
Many heads, many perspectives
A single attention calculation captures one kind of relationship. But language has many: who did what, which adjective describes which noun, what tense the sentence is in. Multi head attention runs several attention calculations in parallel, each with its own set of query, key and value weights. One head might learn to link pronouns to names, another might track verbs and their objects. The outputs of all heads are combined into one richer representation.
A full transformer stacks many such layers. Each layer pairs multi head attention with a small feed forward network.
Where is the word order?
Since a transformer looks at all tokens at once, it has no built in sense of order. "Dog bites man" and "Man bites dog" would look identical. To fix this, the model adds positional information to each embedding before attention starts. The original paper used fixed sine and cosine patterns; later models often learn positions or encode relative distance. Either way, each token knows what it is and where it sits.
Encoder, decoder, and encoder decoder models
The original transformer had two halves, and modern models use one or both of them. This split comes up often in interviews.
Encoder only (BERT style)
An encoder reads the full input and lets every token attend to every other token, both left and right. This makes it excellent at understanding text. BERT and its smaller variants like DistilBERT are encoder models. They are used for:
- Sentiment analysis and text classification
- Named entity recognition (find company names in a job description)
- Semantic search (match a resume to a job posting)
Decoder only (GPT style)
A decoder generates text one token at a time. Each token is only allowed to attend to the tokens before it, never the future ones, because during generation the future does not exist yet. This is called masked or causal attention. GPT style models are decoders, and they are trained on one simple task at enormous scale: predict the next token. They power chat, writing and code generation.
Encoder decoder (T5 style)
An encoder decoder model uses an encoder to understand an input and a decoder to produce a new output, with the decoder also attending to the encoder's results. This setup is natural for tasks that transform one text into another: translation, summarisation, and rewriting. T5, FLAN T5 and BART follow this pattern. For example, google/flan-t5-small can answer instructions, and facebook/bart-large-cnn is widely used for summarising news style articles.
Key takeaway: encoders understand, decoders generate, and encoder decoders convert one text into another.
Why transformers power ChatGPT, Claude and Gemini
The large chat assistants you use daily, including ChatGPT, Claude and Gemini, are built on the transformer idea, mostly in the decoder style, at a very large scale. A few properties made this possible:
- Parallel training. Attention processes all tokens together, so training spreads across many GPUs.
- Scaling works. Bigger transformers trained on more data kept getting better.
- Long context. Distant words connect directly, so models handle long documents and conversations.
Companies then add training with human feedback and safety work so the assistant follows instructions helpfully, and reasoning models work through problems step by step before answering. Underneath, attention still does the core work.
Transformer knowledge is now baseline for machine learning and data science roles. A structured career roadmap helps you decide how deep to go.
Where to try a transformer for free
You do not need a paid GPU or an expensive laptop to start. Two free resources will cover almost everything a beginner needs:
- Hugging Face Hub: a public library of thousands of open source models and datasets. Most of the models mentioned above, such as
distilbert-base-uncased-finetuned-sst-2-englishandgoogle/flan-t5-small, are hosted there and can be downloaded with one line of code. - Google Colab: a free, browser based Python notebook with optional GPU access on its free tier. Free tier limits change over time, so check the current details on the Colab page.
Here is a complete first experiment you can paste into a Colab cell. It downloads a small encoder model and classifies sentiment:
!pip install -q transformers
from transformers import pipeline
classifier = pipeline(
"sentiment-analysis",
model="distilbert-base-uncased-finetuned-sst-2-english",
)
feedback = [
"The mock interview really boosted my confidence.",
"The internship tasks were confusing and poorly explained.",
]
print(classifier(feedback))
Small models like this run comfortably even on a laptop CPU. Later posts in this series cover Hugging Face and Colab step by step, plus fine tuning and deploying a portfolio demo. You can find them on our blog.
What to do next
Here is a simple plan to turn this explanation into real skill:
- Explain it back. Write a one page note on tokens, embeddings, self attention and the three model types in your own words.
- Run the code above in Colab and change the input sentences. Then see how the tokenizer splits your own name.
- Try all three model types. Use an encoder for classification,
google/flan-t5-smallfor an encoder decoder task, and a small decoder model for text generation. - Build one small project, such as a resume and job description matcher, and add it to your resume. Get it checked with our AI resume review before placement season.
- Go deeper with structure. If you want guided practice, explore our courses on Python and machine learning.
Transformers are not magic. They are a clever way of letting words pay attention to each other. Once that clicks, the rest of modern AI becomes far easier to learn.
