I Built an LLM From Scratch

What happens when you type a prompt into an LLM like ChatGPT or Claude or Gemini and press enter ? The answer, as it turns out, involves over 80 years of research, billions of dollars, and some of the most elegant [music] math humans have ever invented . Now, to answer this question, I built an LLM from scratch, and I'm going to walk you through that full journey from keystroke to stream response with working code at every step . Now, our story actually starts back in 1950 . Claude Shannon, the father of information theory, sat down with his wife Betty to play a game . And he showed her a passage of text with the next letter hidden, and she had to guess what came next, letter by letter . And he measured how often she was right . But what he was really measuring is how predictable is written language . And his finding was, given enough context, the next letter is often nearly certain . Now, this means that language has deep statistical structure, and over 70 years later, that's pretty much what an LLM is doing . It's predicting what is coming next . And LLMs are statistical models of language . They extract , and they build a mathematical representation of how words relate to each other . And people often describe them as next token predictors , and while that's true, there is a massive amount of machinery involved before you can get to the point of what comes next . And that's really what I wanted to understand . Right? We are increasingly interacting with these AI services on a daily basis , and whether it's at home or at work , and a lot of the details are hidden from us . And most people kind of just hand-wave over , "Oh, it predicts the next token." And And I wanted to dive deeper . I wanted to have a better understanding of all this stuff . And so, that's exactly what I'm going to break down in this video . You're going to walk away with a deeper understanding of how LLMs work . If that sounds good, let's dive in . My name is CJ. Welcome to Syntax . Now, before LLMs existed, we had chatbots of forms over the past 60-plus years . And this really all started back in 1950 when Alan Turing published his paper computing machinery and intelligence