# The spelled-out intro to neural networks and backpropagation: building micrograd

- Channel: [Andrej Karpathy](https://www.youtube.com/channel/UCXUPKJO5MZQN11PqgIvyuvQ)
- Published: 2022-08-16
- Source video: [Watch on YouTube](https://youtube.com/watch?v=VMj-3S1tku0)
- Read on ReadTube: https://www.read.tube/p/videos/VMj-3S1tku0
- Language: English

## Summary

**Building Micrograd: Neural Network Training Under the Hood**

This video provides a deep, intuitive look at how neural networks are trained from scratch by building 'micrograd', a small scalar-valued autograd engine. The lecture demonstrates how backpropagation and the chain rule can be used to efficiently calculate gradients and tune network parameters, illustrating that modern deep learning libraries like PyTorch are built on these same fundamental concepts.

- Micrograd is a minimal autograd engine that implements backpropagation to efficiently calculate gradients of loss functions for neural network training.
- Neural networks are essentially complex mathematical expressions where forward passes compute outputs and backpropagation recursively applies the chain rule to determine the influence of internal parameters on the final result.
- Training involves initializing parameters, performing a forward pass to calculate loss, and running backpropagation to find the gradients.
- Gradient descent is then used to iteratively adjust weights and biases in the direction that minimizes the loss, allowing the network to learn patterns in the data.
- While libraries like PyTorch optimize these processes using multidimensional tensors and hardware parallelism, the underlying mathematical principles of backpropagation and gradient-based optimization remain the same as those implemented in micrograd.

## Article

### Building micrograd from scratch

Hello, my name is Andre, and I have been training deep neural networks for a bit more than a decade. In this lecture, I would like to show you what neural network training looks like under the hood. In particular, we are going to start with a blank Jupyter notebook, and by the end of this lecture we will define and train a neural net, and you will get to see everything that goes on under the hood and exactly how that works on an intuitive level.

What I would like to do specifically is take you through building **micrograd**. Micrograd is a library that I released on GitHub about two years ago, but at the time I only uploaded the source code, and you would have to go in by yourself and really figure out how it works. In this lecture, I will take you through it step by step and comment on all the pieces of it.

#### What micrograd is

What is micrograd, and why is it interesting? Micrograd is basically an **autograd engine**. Autograd is short for **automatic gradient**, and really what it does is implement **backpropagation**. Backpropagation is the algorithm that allows you to efficiently evaluate the gradient of some kind of a loss function with respect to the weights of a neural network. That allows us to iteratively tune the weights of that neural network to minimize the loss function and therefore improve the accuracy of the network. Backpropagation would be at the mathematical core of any modern deep neural network library like, say, **PyTorch** or **JAX**.

#### An example expression graph

The functionality of micrograd is, I think, best illustrated by an example. If we scroll down here, you will see that micrograd basically allows you to build out mathematical expressions. Here we are building an expression where you have two inputs, `a` and `b`, and you will see that `a` and `b` are `-4` and `2`, but we are wrapping those values into this `Value` object that we are going to build out as part of micrograd.

This `Value` object will wrap the numbers themselves, and then we are going to build out a mathematical expression here where `a` and `b` are transformed into `c`, `d`, and eventually `e`, `f`, and `g`. I am showing some of the functionality of micrograd and the operations that it supports, so you can add two `Value` objects, you can multiply them, you can raise them to a constant power, you can offset by one, negate, squash at zero, square, divide by constant, divide by it, etc. We are building out an expression graph with these two inputs `a` and `b`, and we are creating an output value of `g`, and micrograd will in the background build out this entire mathematical expression, so it will, for example, know

### Scalar values and expression graphs

that `c` is also a value. `c` was a result of an addition operation, and the child nodes of `c` are `a` and `b` because the `add` will maintain pointers to `a` and `b` value objects, so we'll basically know exactly how all of this is laid out. And then not only can we do what we call the forward pass, where we actually look at the value of `g`—of course, that's pretty straightforward, we will access that using the `.data` attribute—and so the output of the forward pass, the value of `g`, is `24.7`, it turns out—but the big deal is that we can also take this `g` value object and we can call that `backward`, and this will basically initialize backpropagation at the node `g`.

And what backpropagation is going to do is it's going to start at `g` and it's going to go backwards through that expression graph, and it's going to recursively apply the chain rule from calculus. And what that allows us to do then is we're going to evaluate basically the derivative of `g` with respect to all the internal nodes like `e`, `d`, and `c`, but also with respect to the inputs `a` and `b`. And then we can actually query this derivative of `g` with respect to `a`, for example; that's `a.grad` in this case, it happens to be `138`, and the derivative of `g` with respect to `b`, which also happens to be here `645`. And this derivative, we'll see soon, is very important information because it's telling us how `a` and `b` are affecting `g` through this mathematical expression. So in particular, `a.grad` is `138`, so if we slightly nudge `a` and make it slightly larger, `138` is telling us that `g` will grow, and the slope of that growth is going to be `138`, and the slope of growth of `b` is going to be `645`. So that's going to tell us about how `g` will respond if `a` and `b` get tweaked a tiny amount in a positive direction.

Now you might be confused about what this expression is that we built out here, and this expression, by the way, is completely meaningless. I just made it up. I'm just flexing about the kinds of operations that are supported by micrograd. What we actually really care about are neural networks, but it turns out that neural networks are just mathematical expressions, just like this one, but actually slightly less crazy. Even neural networks are just a mathematical expression. They take the input data as an input and they take the weights of a neural network as an input, and it's a mathematical expression, and the output are your predictions of your neural net or the loss function. We'll see this in a bit. Basically, neural networks just happen to be a certain class of mathematical expressions, but backpropagation is actually significantly more general. It doesn't actually care about neural networks at all. It only tells us about arbitrary mathematical expressions, and then we happen to use that machinery for training of neural networks.

Now one more note I would like to make at this stage is that, as you see here, micrograd is a scalar-valued autograd engine, so it's working on the level of individual scalars like negative four and two, and we're taking neural nets and we're breaking them down all the way to these atoms of individual scalars and all the little pluses and times, and it's just excessive. And so obviously you would never be doing any of this in production. It's really just put down for pedagogical reasons because it allows us to not have to deal with these n-dimensional tensors that you would use in a modern deep neural network library. So this is really done so that you understand and refactor out backpropagation and chain rule and understanding of neural network training, and then if you actually want to train bigger networks, you have to be using these tensors, but none of the math changes. This is done purely for efficiency. We are basically taking scalar values—all the scalar values—we're packaging them up into tensors, which are just arrays of these scalars, and then because we have these large arrays, we're making operations on those large arrays. That allows us to take advantage of the parallelism in a computer, and all those operations can be done in parallel, and then the whole thing runs faster. But really, none of the math changes, and that's done purely for efficiency. So I don't think that it's pedagogically useful to be dealing with tensors from scratch, and I think—and that's why I fundamentally wrote micrograd—because you can understand how things work at the fundamental level and then you can speed it up later.

Okay, so here's the fun part. My claim is that micrograd is what you need to train your networks, and everything else is just efficiency. So you'd think that micrograd would be a very complex piece of code, and that turns out to not be the case. So if we just go to micrograd, and you'll see that there's only two files here in micrograd. This is the actual engine. It doesn't know anything about neural nets, and this is the entire neural nets library on top of micrograd: `engine` and `nn.py`. So the actual backpropagation autograd engine that gives you the power of neural networks is literally 100 lines of code of very simple Python, which we'll understand by the end of this lecture, and then `nn.py`, this neural network library built on top of the autograd engine, is like a joke. It's like we have to define what a neuron is, and then we have to define what a layer of neurons is, and then we define what a multi-layer perceptron is, which is just a sequence of layers of neurons, and so it's just a total joke. So basically, there's a lot of power that comes from only 150 lines of code, and that's all you need to understand to understand neural network training, and everything else is just efficiency. And of course there's a lot to efficiency, but fundamentally that's all that's happening.

Okay, so now let's dive right in and implement micrograd step by step. The first thing I'd like to do is I'd like to make sure that you have a very good understanding intuitively of what a derivative is and exactly what information it gives you. So let's start with some basic imports that I copy paste in every Jupyter notebook always, and let's define a function, a scalar-valued function `f` of `x`, as follows. So I just make this up randomly. I just want a scalar-valued function that takes a single scalar `x` and returns a single scalar `y`, and we can call this function, of course. So we can pass in, say, `3.0` and get `20` back. Now we can also plot this function to get a sense of its shape. You can tell from the mathematical expression that this is probably a parabola, it's a quadratic, and so if we just create a set of scalar values that we can feed in using, for example, a range from negative five to five in steps of `0.25`—so this axis is just from negative 5 to 5, not including 5, in steps of `0.25`.

### Measuring derivatives numerically

and we can actually call this function on this NumPy array as well, so we get a set of `y` values. If we call `f` on `x`, then these `y` values are basically also applying a function on every one of these elements independently. And we can plot this using Matplotlib, so `plt.plot(x, y)`, and we get a nice parabola.

Previously here we fed in `3.0` somewhere here and we received `20` back, which is here the `y` coordinate. So now I'd like to think through what is the derivative of this function at any single input point `x`, right? What is the derivative at different points `x` of this function?

If you remember back to your calculus class, you've probably derived derivatives. So we take this mathematical expression `3x^2 - 4x + 5`, and you would write out on a piece of paper and apply the product rule and all the other rules and derive the mathematical expression of the derivative of the original function. Then you could plug in different `x` values and see what the derivative is.

We're not going to actually do that, because no one in neural networks actually writes out the expression for the neural net. It would be a massive expression — thousands, tens of thousands of terms. No one actually derives the derivative, of course. So we're not going to take this symbolic approach.

Instead, what I'd like to do is look at the definition of derivative and make sure that we really understand what derivative is measuring, what it's telling you about the function. If we just look up derivative, we see that this is not a very good definition of derivative; this is a definition of what it means to be differentiable. But if you remember from calculus, it is the limit as `h` goes to zero of `f(x + h) - f(x) / h`.

Basically, what it's saying is: if you slightly bump up some point `x` that you're interested in, or `a`, and if you slightly increase it by a small number `h`, how does the function respond? With what sensitivity does it respond? What is the slope at that point? Does the function go up or does it go down, and by how much? That's the slope of that function, the slope of that response at that point.

So we can basically evaluate the derivative here numerically by taking a very small `h`. Of course the definition would ask us to take `h` to zero; we're just going to pick a very small `h`, `0.001`. And let's say we're interested in point `3.0`, so we can look at `f(x)`, of course, as `20`, and now `f(x + h)`.

If we slightly nudge `x` in a positive direction, how is the function going to respond? And just looking at this, do you expect `f(x + h)` to be slightly greater than `20`, or do you expect it to be slightly lower than `20`? Since this `3` is here and this is `20`, if we slightly go positively, the function will respond positively. So you'd expect this to be slightly greater than `20`.

And now by how much? It's telling you the strength of that slope, the size of the slope. So `f(x + h) - f(x)`: this is how much the function responded in the positive direction, and we have to normalize by the run. So we have the rise over run to get the slope. This is, of course, just a numerical approximation of the slope, because we have to make `h` very, very small to converge to the exact amount.

Now if I'm doing too many zeros, at some point I'm going to get an incorrect answer because we're using floating point arithmetic and the representations of all these numbers in computer memory is finite, and at some point we get into trouble. So we can converge toward the right answer with this approach, but basically at `3` the slope is `14`.

And you can see that by taking `3x^2 - 4x + 5` and differentiating it in our head, so `3x^2` would be `6x - 4`, and then we plug in `x = 3`, so that's `18 - 4`, which is `14`. So this is correct. That's at `3`.

Now how about the slope at, say, `-3`? Would you expect — would you expect for the slope now, telling the exact value is really hard, but what is the sign of that slope? At `-3`, if we slightly go in the positive direction at `x`, the function would actually go down, and so that tells you that the slope would be negative. So we'll get a slight number below `20`, and so if we take the slope, we expect something negative: `-22`.

And at some point here, of course, the slope would be zero. Now for this specific function, I looked it up previously, and it's at point `2/3`, so at roughly `2/3` — that's somewhere here — this derivative would be zero. So basically at that precise point, if we nudge in a positive direction, the function doesn't respond; this stays the same almost. And so that's why the slope is zero.

#### A slightly more complex case

Now let's look at a bit more complex case, so we're going to start complexifying a bit. Now we have a function here with output variable `d` that is a function of three scalar inputs `a`, `b`, and `c`. So `a`, `b`, and `c` are some specific values, three inputs into our expression graph, and a single output `d`.

If we just print `d`, we get `4`. And now what I have to do is I'd like to again look at the derivatives of `d` with respect to `a`, `b`, and `c`, and think through again just the intuition of what this derivative is telling us.

In order to evaluate this derivative, we're going to get a bit hacky here. We're going to again have a very small value of `h`, and then we're going to fix the inputs at some values that we're interested in. So this is the point `abc` at which we're going to be evaluating the derivative of `d` with respect to all `a`, `b`, and `c` at that point. So there are the inputs.

And now we have `d1` as that expression, and then we're going to, for example, look at the derivative of `d` with respect to `a`. So we'll take `a` and we'll bump it by `h`, and then we'll get `d2` to be the exact same function. And now we're going to print `d1`, `d2`, and print slope. So the derivative, or slope, here will be, of course, `d2 - d1 / h`.

`d2 - d1` is how much the function increased when we bumped the specific input that we're interested in by a tiny amount, and this is then normalized by `h` to get the slope.

### Derivative intuition with finite differences

Yeah, so if I just run this, we are going to print `d1`, which we know is `4`. Now `d2` will be bumped, `a` will be bumped by `h`, so let’s think through a little bit what `d2` will be printed out here. In particular, `d1` will be `4`, but will `d2` be a number slightly greater than `4` or slightly lower than `4`? That is going to tell us the sign of the derivative.

We are bumping `a` by `h`. `b` is `-3`, `c` is `10`, so you can just intuitively think through this derivative and what it is doing. `a` will be slightly more positive, but `b` is a negative number, so if `a` is slightly more positive, because `b` is `-3`, we are actually going to be adding less to `d`. So you would actually expect that the value of the function will go down. Let’s just see this.

And so we went from `4` to `3.9996`, and that tells you that the slope will be negative, and then the exact number of the slope will be the exact amount of slope is `-3`. You can also convince yourself that `-3` is the right answer mathematically and analytically because if you have `a times b plus c` and you have calculus, then differentiating `a times b plus c` with respect to `a` gives you just `b`, and indeed the value of `b` is `-3`, which is the derivative that we have. So you can tell that that is correct.

Now if we do this with `b`, if we bump `b` by a little bit in a positive direction, we would get different slopes. So what is the influence of `b` on the output `d`? If we bump `b` by a tiny amount in a positive direction, then because `a` is positive, we will be adding more to `d`, right? And now what is the sensitivity, what is the slope of that addition? It might not surprise you that this should be `2`, and why is `2`? Because `d` of `d` by `db`, differentiating with respect to `b`, would give us `a`, and the value of `a` is `2`, so that is also working well.

Then if `c` gets bumped a tiny amount in `h` by `h`, then of course `a times b` is unaffected, and now `c` becomes slightly bit higher. What does that do to the function? It makes it slightly bit higher because we are simply adding `c`, and it makes it slightly bit higher by the exact same amount that we added to `c`. And so that tells you that the slope is `1`. That will be the rate at which `d` will increase as we scale `c`.

Okay, so we now have some intuitive sense of what this derivative is telling you.

### The Value object for computations

about the function, and we would like to move to neural networks now. As I mentioned, neural networks will be pretty massive expressions, mathematical expressions, so we need some data structures that maintain these expressions, and that is what we are going to start to build out now.

We are going to build out this `Value` object that I showed you in the README page of micrograd. Let me copy-paste a skeleton of the first, very simple `Value` object:

```python
class Value:
    def __init__(self, data):
        self.data = data
```

`Value` takes a single scalar value that it wraps and keeps track of, and that is it. So we can, for example, do `Value(2.0)`, and then we can look at its content, and Python will internally use the wrapper function to return this string, oops, like that. So this is a `Value` object with `data = 2` that we are creating here.

What we would like to do is not just have single values, but have a *bluffy*, right? We would like to add them. Currently, you would get an error because Python does not know how to add two `Value` objects, so we have to tell it. Here is addition: you have to basically use these special double underscore methods in Python to define these operators for these objects.

If we use this plus operator, Python will internally call `a.__add__(b)`. That is what will happen internally. `b` will be the other and `self` will be `a`. And so what we see is that what we are going to return is a new `Value` object, and it is just going to be wrapping the plus of their `data`. But remember, now because `data` is the actual numbered Python number, this operator here is just the typical floating-point plus addition. It is not an addition of `Value` objects, and it will return a new `Value`.

So `a + b` should work, and it should print `Value(-1)` because that is `2 + -3`. There we go.

#### Multiplication and printing

Let us now implement multiply just so we can recreate this expression here. So multiply, I think it will not surprise you, will be fairly similar. Instead of `add`, we are going to be using `mul`, and then here, of course, we want to do times. Now we can create a `c` `Value` object, which will be `10.0`, and now we should be able to do `a * b`. Well, let us just do `a * b` first.

That is `Value(-6)` now. And by the way, I skipped over this a little bit: suppose that I did not have the wrapper function here, then you would get some kind of an ugly expression. So what `wrapper` is doing is providing us a way to print out a nicer-looking expression in Python, so we do not just have something cryptic. Actually, you know, it is `Value(-6)`.

So this gives us a times, and then this we should now be able to add `c` to it because we have defined and told Python how to do `mul` and `add`. And so this will call, this will basically be equivalent to `a.dot.mul(b)`, and then this new `Value` object will be `.add(c)`. And so let us see if that worked. Yep, that worked well. That gave us `4`, which is what we expect from before. And I believe we can just call them manually as well. There we go.

#### Keeping expression graphs

So now what we are missing is the connective tissue of this expression. As I mentioned, we want to keep these expression graphs, so we need to know and keep pointers about what values produce what other values. Here, for example, we are going to introduce a new variable, which we will call `children`, and by default it will be an empty tuple. And then we are actually going to keep a slightly different variable in the class, which we will call `_prev`, which will be the set of children. This is how I did it in the original micrograd, looking at my code here. I cannot remember exactly the reason; I believe it was efficiency. But this `_children` will be a tuple for convenience, and then when we actually maintain it in the class, it will be just this set. Yeah, I believe for efficiency.

So now when we are creating a `Value` like this with a constructor, `children` will be empty and `prev` will be the empty set. But when we are creating a `Value` through addition or multiplication, we are going to feed in the children of this `Value`, which in this case is `self` and `other`. So those are the children here.

Now we can do `d.prev`, and we will see that the children of the value, we now know, are this `Value(-6)` and `Value(10)`. And this, of course, is the value resulting from `a * b`, and the `c` value, which is `10`.

The last piece of information we do not know: we know the children of every single value, but we do not know what operation created this value. So we need one more element here. Let us call it `_op`, and by default this is the empty set for leaves, and then we will just maintain it here. Now the operation will be just a simple string, and in the case of addition it is `+`, in the case of multiplication it is `*`.

So now we not just have `d.prev`, we also have `d.op`, and we know that `d` was produced by an addition of those two values. And so now we have the full mathematical expression, and we are building out this data structure, and we know exactly how each value came to be, by what expression and from what other values.

### Visualizing computation graphs

Values now, because these expressions are about to get quite a bit larger, we would like a way to nicely visualize the expressions we are building out. For that, I am going to copy and paste a bunch of slightly scary code that is going to visualize these expression graphs for us.

Here is the code, and I will explain it in a bit, but first let me show you what it does. Basically, it creates a new function `drawDot` that we can call on some root node, and then it is going to visualize it. So if we call `drawDot` on `d`, which is this final value here, that is `a * b + c`, it creates something like this. So this is `d`, and you can see that this is `a * b`, creating an intermediate value, plus `c`, gives us this output node `d`. So that is `drawDot` of `d`.

I am not going to go through this in complete detail. You can take a look at Graphviz and its API. Graphviz is an open source graph visualization software, and what we are doing here is building out this graph with the Graphviz API. You can basically see that `trace` is this helper function that enumerates all of the nodes and edges in the graph, so that just builds a set of all the nodes and edges. Then we iterate over all the nodes and create special node objects for them using `dot.node`, and then we also create edges using `dot.edge`.

The only thing that is slightly tricky here is that I basically add these fake nodes, which are these operation nodes. So for example, this node here is just a plus node, and I create these special op nodes here and connect them accordingly. These nodes, of course, are not actual nodes in the original graph. They are not actually a `Value` object. The only value objects here are the things in squares; those are actual `Value` objects, or representations thereof. And these op nodes are just created in this `drawDot` routine so that it looks nice.

Let us also add labels to these graphs, just so we know what variables are where. So let us create a special underscore label, or let us just do `label =` empty by default, and save it in each node. Then here we are going to do `label` as a label is the label `a`, `c`, and then let us create a special `e = a * b`, and `e.label` will be `e`. It is kind of naughty, and `e` will be `e + c`, and `d.label` will be `d`. Okay? So nothing really changes. I just added this new `e` function, a new `e` variable. Then here, when we are printing this, I am going to print the label here. So this will be a `%s` bar, and this will be `end.label`. And so now we have the label on the left here, so it says `a b` creating `e`, and then `e + c` creates `d`, just like we have it here.

Finally, let us make this expression just one layer deeper. So `d` will not be the final output node. Instead, after `d`, we are going to create a new `Value` object called `f` — we are going to start running out of variables soon. `f` will be `-2.0`, and its label will, of course, just be `f`. And then `L`, capital `L`, will be the output of our graph, and `L` will be `p * f`. Okay? So `L` will be negative eight as the output. So now we do not just draw a `d`, we draw `L`. And somehow the label of `L` was undefined. Oops. All that label has to be explicitly sort of given to it. There we go.

So `L` is the output. Let us quickly recap what we have done so far. We are able to build out mathematical expressions using only plus and times so far. They are scalar valued along the way, and we can do this forward pass and build out a mathematical expression. So we have multiple inputs here, `a`, `b`, `c`, and `f`, going into a mathematical expression that produces a single output `L`. And this here is visualizing the forward pass. So the output of the forward pass is negative eight. That is the value now. What we would like to do next is we would...

### Manual backpropagation on a toy graph

#### Backpropagation in reverse

We like to run backpropagation, and in backpropagation we are going to start here at the end and reverse direction, calculating the gradient along all these intermediate values. What we are really computing for every single value here is the derivative of that node with respect to `l`. The derivative of `l` with respect to `l` is just `1`, and then we are going to derive what is the derivative of `l` with respect to `f`, with respect to `d`, with respect to `c`, with respect to `e`, with respect to `b`, and with respect to `a`.

In the neural network setting, you would be very interested in the derivative of this loss function `l` with respect to the weights of a neural network. Here, of course, we just have these variables `a`, `b`, `c`, and `f`, but some of these will eventually represent the weights of a neural net. So we need to know how those weights are impacting the loss function. We will therefore be interested in the derivative of the output with respect to some of its leaf nodes, and those leaf nodes will be the weights of the neural net. The other leaf nodes, of course, will be the data itself, but usually we will not want or use the derivative of the loss function with respect to data, because the data is fixed. The weights, however, will be iterated on using the gradient information.

#### Adding `grad` to each value

Next we are going to create a variable inside the `Value` class that maintains the derivative of `l` with respect to that value, and we will call this variable `grad`. So there is `data`, and there is `self.grad`, and initially it will be zero. Remember that zero basically means no effect. At initialization, we are assuming that every value does not impact the output, because if the gradient is zero, that means that changing this variable is not changing the loss function. So by default we assume that the gradient is zero.

Now that we have `grad` and it is `0.0`, we are going to be able to visualize it here after `data`. So here `grad` is `0.4 f`, and this will be in that graph, and now we are going to be showing both the `data` and the `grad` initialized at zero. We are just about getting ready to calculate the backpropagation. And of course this `grad`, again, as I mentioned, is representing the derivative of the output in this case `l` with respect to this value. So this is the derivative of `l` with respect to `f`, with respect to `d`, and so on.

#### Filling in the gradients by hand

So let's now fill in those gradients and actually do backpropagation manually. Let's start filling in these gradients and start all the way at the end. As I mentioned, here first we are interested to fill in this gradient here. So what is the derivative of `l` with respect to `l`? In other words, if I change `l` by a tiny amount of `h`, how much does `l` change? It changes by `h`, so it is proportional, and therefore the derivative will be `1`.

We can of course measure these or estimate these numerical gradients numerically, just like we have seen before. If I take this expression and I create a `def lol` function here and put this here, now the reason I am creating a gating function `lol` here is because I do not want to pollute or mess up the global scope here. This is just kind of like a little staging area, and as you know in Python all of these will be local variables to this function, so I am not changing any of the global scope here. So here `l1` will be `l`, and then copy-pasting this expression, we are going to add a small amount `h` in, for example `a`, and this would be measuring the derivative of `l` with respect to `a`. So here this will be `l2`, and then we want to print this derivative, so `print(l2 - l1)`, which is how much `l` changed, and then normalize it by `h`. This is the rise over run, and we have to be careful because `l` is a `Value` node, so we actually want its `data`, so that these are floats dividing by `h`.

This should print the derivative of `l` with respect to `a`, because `a` is the one that we bumped a little bit by `h`. So what is the derivative of `l` with respect to `a`? It is `6`. And obviously if we change `l` by `h`, then that would be here, effectively. This looks really awkward, but changing `l` by `h`, you see the derivative here is `1`. That is kind of like the base case of what we are doing here. So basically we can come up here and manually set `l.grad` to `1`. This is our manual backpropagation: `l.grad` is `1`.

Let's redraw and we will see that we filled in `grad` as `1` for `l`. We are now going to continue the backpropagation, so let's look at the derivatives of `l` with respect to `d` and `f`. Let's do `d` first. What we are interested in, if I create a markdown node here, is that we would like to know basically we have that `l` is `d` times `f`, and we would like to know what is `dl/dd`. What is that? If you know your calculus, `l` is `d` times `f`, so `dl/dd` would be `f`.

And if you do not believe me, we can also just derive it, because the proof would be fairly straightforward. We go to the definition of the derivative, which is `f(x + h) - f(x)` divided by `h`, as a limit of `h` goes to zero of this kind of expression. So when we have `l` is `d` times `f`, then increasing `d` by `h` would give us the output of `d + h` times `f` — that is basically `f(x + h)`, right? — minus `d` times `f`, and then divide by `h`. Symbolically expanding out here, we would have basically `d` times `f` plus `h` times `f` minus `d` times `f`, divided by `h`, and then you see how the `df` minus `df` cancels, so you are left with `h` times `f` divided by `h`, which is `f`. So in the limit as `h` goes to zero of the derivative definition, we just get `f` in the case of `d` times `f`.

Symmetrically, `dl/df` will just be `d`. So what we have is that `f.grad` now is just the value of `d`, which is `4`. And we see that `d.grad` is just the value of `f`, and so the value of `f` is `-2`. So we will set those manually. Let me erase this markdown node and then let's redraw what we have.

Okay, and let's just make sure that these were correct. So we seem to think that `dl/dd` is `-2`. So let's double check. Let me erase this `+ h` from before, and now we want the derivative with respect to `f`, so let's just come here, create `f`, and let's do a `+ h` here, and this should print the derivative of `l` with

### Checking gradients with numerical tests

respect to `f`, so we expect to see four — yeah, and this is four up to floating-point funkiness. And then `dl/dd` should be `f`, which is negative two, `grad` is negative two. So if we again come here and we change `dd.data += h` right here, then we expect — so we have added a little `h`, and then we see how `l` changed, and we expect to print negative two. There we go. So we have numerically verified what we are doing here is kind of like an inline gradient check.

A gradient check is when we are deriving this backpropagation and getting the derivative with respect to all the intermediate results, and then numerical gradient is just estimating it using a small step size. Now we are getting to the crux of backpropagation. So this will be the most important node to understand, because if you understand the gradient for this node, you understand all of backpropagation and all of training of neural nets basically.

So we need to derive `dl/dc`. In other words, the derivative of `l` with respect to `c`, because we have computed all these other gradients already. Now we are coming here and we are continuing the backpropagation manually. So we want `dl/dc`, and then we will also derive `dl/de`.

#### The local derivative

Now here is the problem: how do we derive `dl/dc`? We actually know the derivative of `l` with respect to `d`, so we know how `l` is sensitive to `d`. But how is `l` sensitive to `c`? If we wiggle `c`, how does that impact `l` through `d`? So we know `dl/dc`, and we also here know how `c` impacts `d`, and so just very intuitively, if you know the impact that `c` is having on `d` and the impact that `d` is having on `l`, then you should be able to somehow put that information together to figure out how `c` impacts `l`.

And indeed, this is what we can actually do. So in particular, we know, just concentrating on `d` first, let us look at the derivative basically of `d` with respect to `c`. So in other words, what is `dd/dc`? Here we know that `d` is `c * c + e`, that is what we know. And now we are interested in `dd/dc`.

If you just know your calculus again and you remember that differentiating `c + e` with respect to `c`, that gives you `1.0`. And we can also go back to the basics and derive this, because again we can go to our `f(x + h) - f(x)` divide by `h` — that is the definition of a derivative as `h` goes to zero. So here, focusing on `c` and its effect on `d`, we can basically do the `f(x + h)` will be `c` incremented by `h` plus `e` — that is the first evaluation of our function — minus `c + e`, and then divide by `h`.

So what is this? Just expanding this out, this will be `c + h + e - c - e` divided by `h`. And then you see here how `c - c` cancels, `e - e` cancels, and we are left with `h/h`, which is `1.0`. And so by symmetry, also `dd/de` will be `1.0` as well. So basically the derivative of a sum expression is very simple.

And this is the local derivative. I call this the local derivative because we have the final output value all the way at the end of this graph, and we are now like a small node here, and this is a little plus node. And the little plus node does not know anything about the rest of the graph that it is embedded in. All it knows is that it did a plus: it took a `c` and an `e`, added them, and created `d`.

This plus node also knows the local influence of `c` on `d`, or rather the derivative of `d` with respect to `c`. And it also knows the derivative of `d` with respect to `e`, but that is not what we want. That is just a local derivative. What we actually want is `dl/dc`, and `l` could — `l` is here just one step away, but in a general case this little plus node could be embedded in a massive graph.

### Applying the chain rule

so again we know how `l` impacts `d`, and now we know how `c` and `e` impact `d`. How do we put that information together to write `dl/dc`? And the answer, of course, is the **chain rule** in calculus.

I pulled up the chain rule here from Kapedia, and I am going to go through this very briefly. Chain rule Wikipedia sometimes can be very confusing, and calculus can be very confusing. This is the way I learned chain rule, and it was very confusing: what is happening? It is just complicated. So I like this expression much better. If a variable `z` depends on a variable `y`, which itself depends on the variable `x`, then `z` depends on `x` as well, obviously through the intermediate variable `y`. In this case, the chain rule is expressed as: if you want `dz/dx`, then you take `dz/dy` and you multiply it by `dy/dx`.

So the chain rule fundamentally is telling you how we chain these derivatives together correctly. To differentiate through a function composition, we have to apply a multiplication of those derivatives. That is really what chain rule is telling us. And there is a nice little intuitive explanation here, which I also think is kind of cute. The chain rule says that knowing the instantaneous rate of change of `z` with respect to `y`, and `y` relative to `x`, allows one to calculate the instantaneous rate of change of `z` relative to `x` as a product of those two rates of change — simply the product of those two.

Here is a good one: if a car travels twice as fast as bicycle, and the bicycle is four times as fast as walking man, then the car travels two times four, eight times as fast as demand. And so this makes it very clear that the correct thing to do sort of is to multiply. So cars are twice as fast as bicycle, and bicycle is four times as fast as man, so the car will be eight times as fast as the man. We can take these intermediate rates of change, if you will, and multiply them together, and that justifies the chain rule intuitively.

So have a look at chain rule. What it means for us is there is a very simple recipe for deriving what we want, which is `dl/dc`. What we have so far is we know `w`, and we know what is the impact of `d` on `l`, so we know `dl/dd`, the derivative of `l` with respect to `d`. We know that that is `-2`. And now, because of this local reasoning that we have done here, we know `dd/dc`. So how does `c` impact `d`? And in particular this is a `+` node, so the local derivative is simply `1.0`. It is very simple.

And so the chain rule tells us that `dl/dc`, going through this intermediate variable, will just be simply `dl/dd` multiplied by `dd/dc`. That is chain rule. This is identical to what is happening here, except `z` is our `l`, `y` is our `d`, and `x` is our `c`. So we literally just have to multiply these, and because these local derivatives like `dd/dc` are just one, we basically just copy over `dl/dd`, because this is just times one.

So what does it do? Because `dl/dd` is `-2`, what is `dl/dc`? Well, it is the local gradient `1.0` times `dl/dd`, which is `-2`. So literally what a `+` node does, you can look at it that way, is it literally just routes the gradient, because the `+` node's local derivatives are just one. And so in the chain rule, one times `dl/dd` is just `dl/dd`, and so that derivative just gets routed to both `c` and to `e` in this case. So basically we have that grad, or let's start with `c` since that is the one we looked at, is `-2` times `1`, `-2`. And in the same way, by symmetry, `e` that grad will be `-2`. That is the claim.

So we can set those, we can redraw, and you see how we just assign negative `2` to negative `2`. So this backpropagating signal, which is carrying the information of what is the derivative of `l` with respect to all the intermediate nodes, we can imagine it almost like flowing backwards through the graph. And a `+` node will simply distribute the derivative to all the leaf nodes — sorry, to all the children nodes of it. So this is the claim, and now let us verify it.

Let me remove the `+ h` here from before, and now instead what we are going to do is we are going to increment `c`, so `c.data` will be credited by `h`. When I run this, we expect to see `-2 -2`. And then of course for `e`, so `e.data += h`, and we expect to see `-2`. Simple.

So those are the derivatives of these internal nodes, and now we are going to recurse our way backwards again, and we are again going to apply the chain rule. So here we go, our second application of chain rule, and we will apply it all the way through the graph. We just happen to only have one more node remaining. We have that `dl/de`, as we have just calculated, is `-2`. So we know that. We know the derivative of `l` with respect to `e`, and now we want `dl/da`, right? And the chain rule is telling us that that is just `dl/de`, `-2`, times the local gradient. So what is the local gradient? Basically `de/da`. We have to look at that.

So I am a `*` node inside a massive graph, and I only know that I did `a * b` and I produced an `e`. So now what is `de/da` and `de/db`? That is the only thing that I sort of know about. That is my local gradient. So because we have that `e` is `a * b`, we are asking what is `de/da`, and of course we just did that here. We had `a *`, so I am not going to rederive it, but if you want to differentiate this with respect to `a`, you will just get `b`, right? The value of `b`, which in this case is `-3.0`.

So basically we have that `dl/da`, well, let me just do it right here. We have that `a.grad`, and we are applying chain rule here, is `dl/de`, which we see here is `-2`, times what is `de/da`? It is the value of `b`, which is `-3`. That is it. And then we have `b.grad` is again `dl/de`, which is `-2`, just the same way, times what is `de/db` is the value of `a`, which is `2.0`, as the value of `a`.

So these are our claimed derivatives. Let us redraw, and we see here that `a.grad` turns out to be `6` because

### Propagating gradients to inputs

that is negative 2 times negative 3, and `b.dot.grad` is negative 4 times—sorry—negative 2 times 2, which is negative 4. So those are our claims. Let's delete this and let's verify them. We have here `a.dot.data += h`, so the claim is that `a.dot.grad` is six. Let's verify six. And we have `b.data += h`, so nudging `b` by `h` and looking at what happens, we claim it's negative four, and indeed it's negative four, plus minus again, float oddness, and that's it.

That was the manual backpropagation all the way from here to all the leaf nodes, and we've done it piece by piece. Really, all we've done is, as you saw, we iterated through all the nodes one by one and locally applied the chain rule. We always know what is the derivative of `L` with respect to this little output, and then we look at how this output was produced. This output was produced through some operation, and we have the pointers to the children nodes of this operation. In this little operation, we know what the local derivatives are, and we just multiply them onto the derivative always. So we just go through and recursively multiply on the local derivatives, and that's what backpropagation is: just a recursive application of chain rule backwards through the computation graph.

Let's see this power in action just very briefly. What we're going to do is we're going to nudge our inputs to try to make `L` go up. In particular, what we're doing is we want `a.data`—we're going to change it—and if we want `L` to go up, that means we just have to go in the direction of the gradient. So `a` should increase in the direction of gradient by some small step amount; this is the step size. And we don't just want this for `b`, but also for `c`, also for `f`. Those are leaf nodes, which we usually have control over, and if we nudge in direction of the gradient, we expect a positive influence on `L`, so we expect `L` to go up positively. So it should become less negative; it should go up to, say, negative six or something like that. It's hard to tell exactly, and we'd have to rewrite the forward pass, so let me just do that here. This would be the forward pass, `f` would be unchanged. This is effectively the forward pass.

And now if we print `L.data`, we expect, because we nudged all the values, all the inputs in the rational gradient, that we expected a less negative `L`. We expect it to go up, so maybe it's negative six or so. Let's see what happens. Okay, negative seven. And this is basically one step of an optimization that we'll end up running, and really does gradient give us

### Backpropagating through a neuron

some power because we know how to influence the final outcome, and this will be extremely useful for training, as well as, you'll see. So now I would like to do one more example of manual backpropagation using a bit more complex and useful example: we are going to backpropagate through a neuron.

We want to eventually build up neural networks, and in the simplest case these are multilayer perceptrons, as they're called. So this is a two-layer neural net, and it's got these hidden layers made up of neurons, and these neurons are fully connected to each other.

Biologically, neurons are very complicated devices, but we have very simple mathematical models of them, and so this is a very simple mathematical model of a neuron. You have some input axes, and then you have these synapses that have weights on them. So the `w`'s are weights, and then the synapse interacts with the input to this neuron multiplicatively. What flows to the cell body of this neuron is `w × x`, but there's multiple inputs, so there's many `w × x`'s flowing into the cell body. The cell body then has also some bias, so this is kind of like the innate sort of trigger happiness of this neuron. This bias can make it a bit more trigger happy or a bit less trigger happy regardless of the input. But basically we're taking all the `w × x` of all the inputs, adding the bias, and then we take it through an activation function.

This activation function is usually some kind of a squashing function, like a sigmoid or `tanh` or something like that. As an example, we're going to use the `tanh` in this example. NumPy has a `np.tanh`, so we can call it on a range and we can plot it. This is the `tanh` function, and you see that the inputs as they come in get squashed on the `y` coordinate here. Right at zero, we're going to get exactly zero, and then as you go more positive in the input, you'll see that the function will only go up to one and then plateau out. So if you pass in very positive inputs, we're going to cap it smoothly at one, and on the negative side we're going to cap it smoothly at negative one. So that's `tanh`, and that's the squashing function, or an activation function.

What comes out of this neuron is just the activation function applied to the dot product of the weights and the inputs. So let's write one out. I'm going to copy-paste because I don't want to type too much, but okay. So here we have the inputs `x1`, `x2`, so this is a two-dimensional neuron, so two inputs are going to come in. These are thought of as the weights of this neuron, `w1`, `w2`, and these weights again are the synaptic strengths for each input. And this is the bias of the neuron, `b`.

Now what we want to do is, according to this model, we need to multiply `x1` times `w1` and `x2` times `w2`, and then we need to add bias on top of it. It gets a little messy here, but all we are trying to do is `x1w1 + x2w2 + b`, and these are multiply here, except I'm doing it in small steps so that we actually have pointers to all these intermediate nodes. So we have `x1w1` variable, `x2w2` variable, and I'm also labeling them so `n` is now the cell body raw activation without the activation function for now. And this should be enough to basically plot it, so `draw_dot(n)` gives us `x1` times `w1`, `x2` times `w2` being added, then the bias gets added on top of this, and this `n` is this sum.

So we're now going to take it through an activation function, and let's say we use the `tanh` so that we produce the output. So what we'd like to do here is we'd like to do the output, and I'll call it `o`, is `n.tanh()`. Okay, but we haven't yet written the `tanh`. The reason that we need to implement another `tanh` function here is that `tanh` is a hyperbolic function, and we've only so far implemented a plus and a times and

### Implementing tanh and its gradient

you can’t make a `tanh` out of just pluses and times; you also need exponentiation. So `tanh` is this kind of formula here: you can use either one of these, and you see that there is exponentiation involved, which we have not implemented yet for our low `Value` node. So we are not going to be able to produce `tanh` yet, and we have to go back up and implement something like it.

One option here is that we could actually implement exponentiation, right? And we could return the `x` of a `Value` instead of a `tanh` of a `Value`, because if we had `x` then we have everything else that we need. Because we know how to add, and we know how to multiply, so we would be able to create `tanh` if we knew how to `x`. But for the purposes of this example, I specifically wanted to show you that we do not necessarily need to have the most atomic pieces in this `Value` object. We can actually create functions at arbitrary points of abstraction. They can be complicated functions, but they can also be very, very simple functions, like a plus. And it is totally up to us.

The only thing that matters is that we know how to differentiate through any one function. We take some inputs and we make an output. The only thing that matters is that it can be an arbitrarily complex function, as long as you know how to create the local derivative. If you know the local derivative of how the inputs impact the output, then that is all you need. So we are going to cluster up all of this expression, and we are not going to break it down to its atomic pieces. We are just going to directly implement `tanh`.

So let’s do that. `__tanh__` will return a `Value` of this expression here, so let me actually copy-paste. Let’s grab `n`, which is `self.data`, and then this, I believe, is the `tanh` math: `exp` of two `n` minus one over two `n` plus one. Maybe I can call this `x` just so that it matches exactly. Okay, and now this will be `t`, and the children of this node, there is just one child, and I am wrapping it in a tuple. So this is a tuple of one object, just `self`, and here the name of this operation will be `tanh`, and we are going to return that. Okay, so now `Value` should be implementing `tanh`.

And now we can scroll all the way down here, and we can actually do `n.tanh`, and that is going to return the `tanh`-ed output of `n`. And now we should be able to draw it out of `o`, not of `n`. So let’s see how that worked. There we go: `n` went through `tanh` to produce this output. So now `tanh` is sort of our little micrograd-supported node here as an operation, and as long as we know the derivative of `tanh`, then we will be able to backpropagate through it.

Now let’s see this `tanh` in action. Currently it is not squashing too much because the input to it is pretty low. So if the bias was increased to, say, `8`, then we will see that what is flowing into the `tanh` now is `2`, and `tanh` is squashing it to `0.96`. So we are already hitting the tail of this `tanh`, and it will sort of smoothly go up to `1` and then plateau out over there.

Okay, so now I am going to do something slightly strange. I am going to change this bias from `8` to this number `6.88...`, and I am going to do this for specific reasons, because we are about to start backpropagation, and I want to make sure that our numbers come out nice. They are not very crazy numbers; they are nice numbers that we can sort of understand in our head. Let me also add a pose label: `o` is short for output here, so that is `0`. So `0.88` flows into `tanh`, comes out `0.7`, and so on.

So now we are going to do backpropagation, and we are going to fill in all the gradients. What is the derivative of `o` with respect to all the inputs here? And of course, in the typical neural network setting, what we really care about the most is the derivative of these neurons on the weights, specifically the `w2` and `w1`, because those are the weights that we are going to be changing as part of the optimization. And the other thing that we have to remember is that here we have only a single neuron. But in the neural network, we typically have many neurons and they are connected, so this is only like one small neuron, a piece of a much bigger puzzle. And eventually there is a loss function that sort of measures the accuracy of the neural net, and we are backpropagating with respect to that accuracy and trying to increase it.

So let’s start off by backpropagation here. In the end, what is the derivative of `o` with respect to `o`? The base case, sort of, we know always is that the gradient is just `1.0`. So let me fill it in, and then let me split out the drawing function here, and then here `cell clear` this output here. Okay, so now when we draw `o`, we will see that `o.grad` is `1`.

So now we are going to backpropagate through the `tanh`. To backpropagate through `tanh`, we need to know the local derivative of `tanh`. So if we have that `o` is `tanh` of `n`, then what is `do/dn`? Now what you could do is you could come here and you could take this expression and you could do your calculus derivative taking, and that would work. But we can also just scroll down Wikipedia here into a section that hopefully tells us that the derivative `d/dx` of `tanh` of `x` is any of these. I like this one: `1 - tanh^2(x)`. So this is `1 - tanh(x)^2`.

So basically what this is saying is that `do/dn` is `1 - tanh(n)^2`, and we already have `tanh(n)`, that is just `o`. So it is `1 - o^2`. So `o` is the output here, so the output is this number, `data` is this number, and then what this is saying is that `do/dn` is `1 - this squared`. So `1` minus that `data` squared is `0.5`, conveniently. So the local derivative of this `tanh` operation here is `0.5`, and so that would be `do/dn`. So we can fill that in: that `grad` is `0.5`. We will just fill in. So this is exactly `0.5`, one-half.

So now we are going to continue the backpropagation. This is `0.5`, and this is a plus node. So what is backprop going to do here? If you remember our previous example, a plus is just a distributor of gradient. So this gradient will simply flow to both of these equally. And that is because the local derivative of this operation is `1` for every one of its nodes. So `1` times `0.5` is `0.5`.

### Working backward through a neuron

So therefore we know that this node here, which we called its grad, is just `0.5`, and we know that `b` dot grad is also `0.5`, so let's set those and let's draw. So `0.5` continuing. We have another plus `0.5`; again, we'll just distribute it, so `0.5` will flow to both of these. So we can set theirs `x2w2` as well, that grad is `0.5`, and let's redraw.

Pluses are my favorite operations to backpropagate through because it's very simple. So now it's flowing into these expressions is `0.5`, and so really, again, keep in mind what the derivative is telling us at every point in time along here: this is saying that if we want the output of this neuron to increase, then the influence on these expressions is positive. On the output, both of them are positive contribution to the output.

Now backpropagating to `x2` and `w2` first, this is a times node, so we know that the local derivative is, you know, the other term. So if we want to calculate `x2.grad`, then can you think through what it's going to be? So `x2.grad` will be `w2.data` times this `x2w2.grad`, right? And `w2.grad` will be `x2.data` times `x2w2.grad`, right? So that's the local piece of chain rule. Let's set them and let's redraw.

So here we see that the gradient on our weight 2 is `0` because `x2.data` was `0`, right? But `x2` will have the gradient `0.5` because data here was `1`. And so what's interesting here, right, is because the input `x2` was `0`, then because of the way the times works, of course this gradient will be zero. And think about intuitively why that is: derivative always tells us the influence of this on the final output. If I wiggle `w2`, how is the output changing? It's not changing because we're multiplying by zero. So because it's not changing, there's no derivative, and zero is the correct answer because we're squashing it at zero.

And let's do it here. `0.5` should come here and flow through this times, and so we'll have that `x1.grad` is, can you think through a little bit what this should be? The local derivative of times with respect to `x1` is going to be `w1`, so `w1` is `data` times `x1w1.dot.grad`, and `w1.grad` will be `x1.data` times `x1w1.grad` with graph. Let's see what those came out to be.

So this is `0.5`, so this would be `-1.5`, and this would be `1`. And we've backpropagated through this expression. These are the actual final derivatives. So if we want this neuron's output to increase, we know that what's necessary is that `w2` — we have no gradient, `w2` doesn't actually matter to this neuron right now — but this neuron, this weight, should go up. So if this weight goes up, then this neuron's output would have gone up and...

### Automating the backward pass

proportionally because the gradient is one. So doing the backpropagation manually is obviously ridiculous, and we are now going to put an end to this suffering. We are going to see how we can implement the backward pass a bit more automatically. We are not going to be doing all of it manually out here anymore.

It is now pretty obvious from the examples how these pluses and times are backpropagation ingredients, so we go up to the `Value` object and start codifying what we have seen below. We do this by storing a special `._backward` cell, and this will be a function that does that little piece of chain rule at each little node that computed, took inputs, and produced output. We are going to store how we are going to chain the output gradient into the input gradients.

By default, this will be a function that does nothing. You can also see that here in the `Value` in micrograd: with this backward function by default doing nothing, this is an empty function, and that would be the case, for example, for a leaf node. For a leaf node, there is nothing to do. But now, when we are creating these `out` values, these `out` values are an addition of `self` and `other`, and so we will want to set `out._backward` to be the function that propagates the gradient.

#### Addition backward function

Let us define what should happen, and we are going to store it in a closure. Let us define what should happen when we call `out.grad` for addition. Our job is to take `out.grad` and propagate it into `self.grad` and `other.grad`. Basically, we want to set `self.grad` to something, and we want to set `other.grad` to something.

The way we saw below how chain rule works, we want to take the local derivative times the global derivative, which is the derivative of the final output of the expression with respect to `out`, with respect to `out` data. The local derivative of `self` in an addition is `1.0`, so it is just `1.0 * out.grad`. That is the chain rule. And `other.grad` will be `1.0 * out.grad`. What you are seeing here is that `out.grad` will simply be copied onto `self.grad` and `other.grad`, as we saw happens for an addition operation. We are going to later call this function to propagate the gradient.

#### Multiplication backward function

Having done addition, let us now do multiplication. We are going to also define that backward, and we are going to set its backward to be backward. We want to chain `out.grad` into `self.grad` and `other.grad`, and this will be a little piece of chain rule for multiplication.

What should this be? Can you think it through? The local derivative here was `other.data`, and then `other.data * out.grad`, that is chain rule. And here we have `self.data * out.grad`, that is what we have been doing.

#### Tanh backward function

Finally, for `tanh`, let us also define `backward`, and then we want to set `out._backward` to be just backward. Here we need to backpropagate. We have `out.grad`, and we want to chain it into `self.grad`, and `self.grad` will be the local derivative of this operation that we have done here, which is `tanh`. We saw that the local gradient is `1 - tanh(x)^2`, which here is `t`; that is the local derivative, because `t` is the output of this `tanh`. So `1 - t^2` is the local derivative, and then the gradient has to be multiplied because of the chain rule, so `out.grad` is chained through the local gradient into `self.grad`, and that should be basically it.

We are going to redefine our `Value` node. We are going to swing all the way down here and redefine our expression. Make sure that all the grads are zero. But now we do not have to do this manually anymore. We are going to basically be calling the `.backward()` in the right order. So first we want to call `o.backward()`, so `o` was the outcome of `tanh`, right? So calling `o.backward()` will be this function. This is what it will do.

Now we have to be careful because there is a times `out.grad`, and `out.grad`, remember, is initialized to zero. So here we see `grad = 0`. As a base case, we need to set `o.grad` to `1.0` to initialize this with `1`, and then once this is `1`, we can call `o.backward()`, and what that should do is it should propagate this grad through `tanh`, so the local derivative times the global derivative, which is initialized at `1`.

This should work. I thought about redoing it, but I figured I should just leave the error in here because it is pretty funny: why is `anti-object` not callable? It is because I screwed up. We are trying to save these functions, so this is correct. This here we do not want to call the function because that returns `None`; these functions return `None`. We just want to store the function. So let me redefine the `Value` object, and then we are going to come back and redefine the expression, draw a dot, everything is great, `o.grad` is `1`, `o.grad` is `1`, and now this should work, of course. Okay, so `o.backward()` should... this grad should now be `0.5` if we redraw, and if everything went correctly, `0.5`, yay.

So now we need to call `n.backward()`. Sorry, `n.backward()`. So that seems to have worked. `tanh.backward()` routed the gradient to both of these, so this is looking great. Now we could, of course, call `b.backward()` too. What is going to happen? Well, `b` does not have a backward; `b` is a leaf node. `b`'s backward is, by initialization, the empty function, so nothing would happen. But we can call it on it. When we call this one, its backward, then we expect this `0.5` to get further routed. Right? So there we go: `0.5`, `0.5`. And then finally we want to call it here on `x2`, `w2`, and on `x1`, `w1`. Do both of those, and there we go. So we get `0`, `0.5`, `-1.5`, and `1`, exactly as we did before, but now we have done it through calling that backward, sort of manually.

We have the lamp. One last piece to get rid of, which is us calling `._backward` manually. So let us think through what we are actually doing. We have laid out a mathematical expression, and now we are trying to go backwards through that expression. Going backwards through the expression just means that we never want to call a `._backward` for any node before we have done everything after it. We have to do everything after it before we are ever going to call that backward on any one node. We have to get all of its full dependencies, everything that it depends on, to propagate to it before we can continue backpropagation, so this ordering of

### Fixing topological backpropagation

graphs can be achieved using something called **topological sort**. Topological sort is basically a laying out of a graph such that all the edges go only from left to right. Here we have a graph, a directed acyclic graph, a DAG, and this is two different topological orders of it, I believe, where basically you will see that it is laying out the nodes such that all the edges go only one way from left to right. Implementing topological sort, you can look in Wikipedia and so on; I am not going to go through it in detail, but basically this is what builds a topological graph.

We maintain a set of visited nodes, and then we are going through, starting at some root node, which for us is `o`, that is where we want to start the topological sort. Starting at `o`, we go through all of its children, and we need to lay them out from left to right. Basically this starts at `o`; if it is not visited, then it marks it as visited, and then it iterates through all of its children and calls `build_topological` on them. Then, after it has gone through all the children, it adds itself. So this node that we are going to call it on, say `o`, is only going to add itself to the topo list after all of the children have been processed, and that is how this function is guaranteeing that you are only going to be in the list once all your children are in the list. That is the invariant that is being maintained.

So if we built upon `o` and then inspect this list, we are going to see that it ordered our `Value` objects, and the last one is the value of `0.707`, which is the output. So this is `o`, and then this is `n`, and then all the other nodes get laid out before it. So that builds the topological graph, and really what we are doing now is we are just calling `. _backward` on all of the nodes in a topological order. If we just reset the gradients, they are all zero. What did we do? We started by setting `o.grad` to `b1`; that is the base case. Then we built the topological order, and then we went `for node in reversed(topo)`, now in reverse order because this list goes from, you know, we need to go through it in reversed order, so starting at `o`. Note that `backward` and this should be it. There we go, those are the correct derivatives.

Finally, we are going to hide this functionality. I am going to copy this, and we are going to hide it inside the `Value` class because we do not want to have all that code lying around. So instead of an `_backward`, we are now going to define an actual `backward`, so that is `backward` without the underscore, and that is going to do all the stuff that we just arrived. Let me just clean this up a little bit. We are first going to build a topological graph starting at `self`, so `build_topo(self)` will populate the topological order into the `topo` list, which is a local variable. Then we set `self.grad` to be one, and then for each node in the reversed list, so starting at us and going to all the children, `_backward`. And that should be it. Save, come down here, redefine... okay, all the grads are zero, and now what we can do is `o.backward()` without the underscore. And there we go, and that is backpropagation in place for one neuron.

Now we should not be too happy with ourselves, actually, because we have a bad bug, and we have not surfaced the bug because of some specific conditions that we have to think about right now. So here is the simplest case that shows the bug. Say I create a single node `a`, and then I create a `b` that is `a + a`, and then I call `backward`. What is going to happen is `a` is 3, and then `b` is `a + a`, so there are two arrows on top of each other here. Then we can see that `b` is of course, the forward pass works, `b` is just `a + a`, which is 6, but the gradient here is not actually correct that we calculate it automatically. That is because, of course, just doing calculus in your head, the derivative of `b` with respect to `a` should be two, one plus one; it is not one.

Intuitively what is happening here? `b` is the result of addition, and then when we called backward on it, `self.grad` was set to one and then `other.grad` was set to one, but because we are doing `a + a`, `self` and `other` are actually the exact same object, so we are overriding the gradient. We are setting it to one and then we are setting it again to one, and that is why it stays at one. So that is a problem.

There is another way to see this in a little bit more complicated expression. So here we have `a` and `b`, and then `d` will be the multiplication of the two and `e` will be the addition of the two, and then we multiply `e` times `d` to get `f`, and then we call `f.da.backward`, and these gradients, if you check, will be incorrect. So fundamentally what is happening here again is basically we are going to see an issue anytime we use a variable more than once. Until now in these expressions above, every variable is used exactly once, so we did not see the issue. But here, if a variable is used more than once, what is going to happen during backward pass? We are backpropagating from `f` to `e` to `d`, so far so good, but now `e` gets its backward and it deposits its gradients to `a` and `b`, but then we come back to `d` and call backward and it overwrites those gradients at `a` and `b`. So that is obviously a problem.

The solution here, if you look at the multivariate case of the chain rule and its generalization there, the solution there is basically that we have to accumulate these gradients. These gradients add, and so instead of setting those gradients, we can simply do `+=`. We need to accumulate those gradients, `+=`, `+=`, `+=`, `+=`, and this will be okay. Remember because we are initializing them at zero, so they start at zero and then any contribution that flows backwards will simply add.

So now if we redefine this one because of the `+=`, this now works because `a.grad` started at zero and we called `beta.backward`, we deposit one and then we deposit one again, and now this is two, which is correct. And here this will also work and we will get correct gradients because when we call `eta.backward`, we will deposit the gradients from this branch and then we get to back into `detail.backward`, it will deposit its own gradients and then those gradients simply add on top of each other, and so we just accumulate those gradients, and that fixes the issue.

### Refactoring the Value API

#### Cleaning up the intermediate work

Move on. Let me actually do a bit of cleanup here and delete some of this intermediate work so we are not going to need any of this now that we have derived all of it. We are going to keep this because I want to come back to it. Delete the `10h`, delete our morning example, delete the step, delete this, keep the code that draws, and then delete this example and leave behind only the definition of `Value`.

#### Returning to tanh

Now let us come back to this non-linearity here that we implemented, the `tanh`. I told you that we could have broken down `tanh` into its explicit atoms in terms of other expressions if we had the `x` function. If you remember, `tanh` is defined like this, and we chose to develop `tanh` as a single function. We can do that because we know its derivative and we can backpropagate through it. But we can also break down `tanh` into and express it as a function of `x`, and I would like to do that now because I want to prove to you that you get all the same results and all those ingredients, but also because it forces us to implement a few more expressions. It forces us to do exponentiation, addition, subtraction, division, and things like that, and I think it is a good exercise to go through a few more of these.

#### Wrapping numeric literals

Okay, so let us scroll up to the definition of `Value`. Here one thing that we currently cannot do is we can do, like, a `Value` of say `2.0`, but we cannot do, you know, here for example we want to add constant one and we cannot do something like this. And we cannot do it because it says object has no attribute `data`. That is because `a + 1` comes right here to `add`, and then `other` is the integer `1`, and then here Python is trying to access `1.data`, and that is not a thing. That is because basically `1` is not a `Value` object, and we only have addition for `Value` objects.

So as a matter of convenience, so that we can create expressions like this and make them make sense, we can simply do something like this: basically we let `other` alone if `other` is an instance of `Value`, but if it is not an instance of `Value`, we are going to assume that it is a number, like an integer or float, and we are going to simply wrap it in `Value`. Then `other` will just become `Value(other)`, and then `other` will have a `data` attribute, and this should work. So if I just say this predefined `Value`, then this should work. There we go.

Okay, now let us do the exact same thing for multiply, because we cannot do something like this again for the exact same reason. So we just have to go to `mul`, and if `other` is not a `Value`, then let us wrap it in `Value`. Let us redefine `Value`, and now this works.

#### Right-hand multiplication

Now here is a kind of unfortunate and not obvious part: `a * 2` works, we saw that, but `2 * a` — is that going to work? You would expect it to, right? But actually it will not. The reason it will not is because Python does not know, like, when you do `a * 2`, basically Python will go and it will basically do something like `a.__mul__(2)`. That is basically what it will call. But to it, `2 * a` is the same as `2.__mul__(a)`, and `2` cannot multiply `Value`, so it is really confused about that.

Instead, what happens is, in Python, the way this works is you are free to define something called `__rmul__`, and `__rmul__` is kind of like a fallback. So if Python cannot do `2 * a`, it will check if, by any chance, `a` knows how to multiply `2`, and that will be called into `__rmul__`. So because Python cannot do `2 * a`, it will check whether there is an `__rmul__` in `Value`, and because there is, it will now call that. What we will do here is we will swap the order of the operands. So basically `2 * a` will redirect to `__rmul__`, and `__rmul__` will basically call `a * 2`, and that is how that will work. So redefining now with `__rmul__`, `2 * a`...

### Implementing power and division

Now, looking at the other elements that we still need, we need to know how to exponentiate and how to divide. So first, the explanation to the exponentiation part: we are going to introduce a single function `x` here, and `x` is going to mirror `tanh` in the sense that it is a simple single function that transforms a single scalar value and outputs a single scalar value. So we pop out the Python number, we use `math.x` to exponentiate it, create a new `Value` object. Everything that we have seen before. The tricky part, of course, is how do you propagate through `e^x`? Here you can potentially pause the video and think about what should go here. Basically, we need to know what is the local derivative of `e^x`. `d/dx of e^x` is famously just `e^x`, and we have already just calculated `e^x` and it is inside `out.data`, so we can do `out.data * out.grad`, and that is the chain rule. So we are just chaining on to the current running grad, and this is what the expression looks like. It looks a little confusing, but this is what it is, and that is the exponentiation. So redefining, we should now be able to call `a.x`, and hopefully the backward pass works as well.

And the last thing we would like to do, of course, is we would like to be able to divide. Now I actually will implement something slightly more powerful than division, because division is just a special case of something a bit more powerful. In particular, just by rearranging, if we have some kind of `a / b` and we have `b = value of 4.0` here, we would like to basically be able to do `a / b`, and we would like this to be able to give us `0.5`.

Division actually can be reshuffled as follows: if we have `a / b`, that is actually the same as `a` multiplying `1 / b`, and that is the same as `a` multiplying `b` to the power of negative one. And so what I would like to do instead is I would basically like to implement the operation of `x` to the `k` for some constant `k`, so it is an integer or a float, and we would like to be able to differentiate this. Then as a special case, negative one will be division, and so I am doing that just because it is more general, and yeah, you might as well do it that way.

So basically what I am saying is we can redefine division, which we will put here somewhere. Yeah, we can put it here somewhere. What I am saying is that we can redefine division so `self.divide(other)` can actually be rewritten as `self * other` to the power of negative one, and now a value raised to the power of negative one, we have now defined that.

### The power rule behind tanh

#### Implementing exponentiation

So here is the need to implement the `pow` function. Where am I going to put the power function? Maybe here somewhere. This is the skeleton for it. This function will be called when we try to raise a value to some power, and `other` will be that power.

I would like to make sure that `other` is only an `int` or a `float`. Usually `other` is some kind of different value object, but here `other` will be forced to be an `int` or a `float`, otherwise the math will not work for what we are trying to achieve. In this specific case, that would be a different derivative expression if we wanted `other` to be a value.

So here we create the output value, which is just this data raised to the power of `other`. And `other` here could be, for example, negative one. That is what we are hoping to achieve. Then this is the backward stub, and this is the fun part: what is the chain rule expression here for backpropagating through the power function where the power is to the power of some kind of constant? This is the exercise, and maybe pause the video here and see if you can figure it out yourself as to what we should put here.

You can actually go here and look at derivative rules as an example, and we see lots of derivatives that you can hopefully know from calculus. In particular, what we are looking for is the power rule, because that is telling us that if we are trying to take `d/dx` of `x^n`, which is what we are doing here, then that is just `n * x^(n - 1)`, right?

That is telling us about the local derivative of this power operation. So all we want here basically, `n` is now `other`, and `self.data` is `x`, and so this now becomes `other * self.data ** (other - 1)`. `self.data` is now a Python `int` or `float`; it is not a value object, we are accessing the `data` attribute raised to the power of `other - 1`, or `n - 1`. I can put brackets around this, but this does not matter because power takes precedence over multiply in Python, so that would have been okay.

And that is the local derivative only, but now we have to chain it, and we chain just simply by multiplying by `out.grad`. That is chain rule. This should technically work, and we are going to find out soon. But now if we do this, this should now work, and we get `0.5`. So the forward pass works, but does the backward pass work?

I realize that we actually also have to know how to subtract. Right now `a - b` will not work. To make it work, we need one more piece of code here, and basically this is subtraction. The way we are going to implement subtraction is we are going to implement it by addition of a negation, and then to implement negation we are going to multiply by negative one. So again, using the stuff we have already built and just expressing it in terms of what we have, `a - b` is now working.

#### Expanding tanh

So now let's scroll again to this expression here for this neuron, and let's just compute the backward pass here once we have defined `o`. And let's draw it. So here are the gradients for all these leaf nodes for this two-dimensional neuron that has a `tanh` that we have seen before.

What I would like to do now is to break up this `tanh` into this expression here. Let me copy-paste this here, and now instead of, we will preserve the label, and we will change how we define `o`. In particular, we are going to implement this formula here. So we need `e^(2x) - 1 / e^x + 1`. So `e^(2x)`, we need to take `2 * n`, and we need to exponentiate it. That is `e` to the two `x`. Then because we are using it twice, let's create an intermediate variable `e`, and then define `o` as `e + 1 / e - 1 / e + 1`, `e - 1 / e + 1`, and that should be it. And then we should be able to draw that of `o`.

Now before I run this, what do we expect to see? Number one, we are expecting to see a much longer graph here because we have broken up `tanh` into a bunch of other operations, but those operations are mathematically equivalent. And so what we are expecting to see is, number one, the same result here, so the forward pass works, and number two, because of that mathematical equivalence, we expect to see the same backward pass and the same gradients on these leaf nodes. So these gradients should be identical.

So let's run this. Number one, let's verify that instead of a single `tanh` node we have now `x`, and we have `+`, we have `times negative one`, this is the division, and we end up with the same forward pass here. And then the gradients, we have to be careful because they are in slightly different order potentially. The gradients for `w2`, `x2` should be `0` and `0.5`; `w2` and `x2` are `0` and `0.5`; and `w1`, `x1` are `1` and `-1.5`, `1` and `-1.5`.

So that means that both our forward passes and backward passes were correct, because this turned out to be equivalent to `tanh` before. And so the reason I wanted to go through this exercise is, number one, we got to practice a few more operations and writing more backwards passes, and number two, I wanted to illustrate the point that the level at which you implement your operations is totally up to you. You can implement backward passes for tiny ...

### Comparing micrograd with PyTorch

The same thing can be done with a modern deep neural network library like **PyTorch**, which is roughly what micrograd is modeled after. PyTorch is what you would use in production, and the point here is to show that you can do the exact same thing using the PyTorch API.

#### Scalars versus tensors

The code is essentially a copy-paste version of the earlier micrograd example, just translated into PyTorch. You start by importing PyTorch and then defining the same kinds of value objects. The big difference is that micrograd is a **scalar-valued engine**, so it only works with scalar values like `2.0`, whereas PyTorch is based around **tensors**. As mentioned earlier, tensors are just *n*-dimensional arrays of scalars, so the library is a little more complicated in that respect.

Here, the goal is to get a scalar value tensor, meaning a tensor with just a single element. By default, though, when you work with PyTorch, you usually use more complicated tensors. For example, you can create tensors like this, and one such tensor might be a **2 by 3 array of scalars** in one compact representation. You can check its shape and see that it is a `2 x 3` array, and so on. That is the normal way you would use PyTorch in actual libraries.

#### Creating scalar tensors

In this example, a tensor is created that has only a single element, `2.0`, and then it is cast to `double`. The reason is that Python by default uses double precision for floating-point numbers, and the goal is to keep everything identical. By default, PyTorch tensors use `float32`, which is a single-precision float, so it is cast to `double` so that it becomes `float64`, matching Python.

At that point, you get something similar to the value `2`.

The next thing that has to be done is to tell PyTorch that these are leaf nodes that require gradients. By default, PyTorch assumes that leaf nodes do **not** require gradients, so you need to explicitly set that they do. This constructs scalar-valued one-element tensors and makes sure that PyTorch knows they require gradients.

By the way, the default is `false` for efficiency reasons, because usually you do not want gradients for leaf nodes like the inputs to the network. PyTorch is trying to be efficient in the most common cases.

#### Forward and backward pass in PyTorch

Once all the values are defined in Python, you can perform arithmetic exactly like you can in micrograd. That just works. There is also `torch.tanh`, and when it comes back, it is a tensor again. Just like in micrograd, it has a `.data` attribute and a `.grad` attribute.

The only difference is that you need to call `.item()`, because `.item()` on a PyTorch tensor with one element strips out the tensor and returns the underlying value.

Running the example should print the forward pass, which is `0.707`, and then the gradients, which are expected to be `0.5`, `0`, `-1.5`, and `1`.

When it is run, the forward pass does agree: `0.7...`, and the gradients come out as `0.5`, `0`, `-1.5`, and `1`.

### Single-element tensors in PyTorch

So PyTorch agrees with us, and just to show you here, basically, here is a tensor with a single element, and it is a double, and we can call `.item()` on it to just get the single number out. That is what `item` does. And `o` is a tensor object, like I mentioned, and it has a backward function just like we have implemented. Then all of these also have a dot graph, so like `x2` for example in the grad, and it is a tensor, and we can pop out the individual number with that action.

Basically, PyTorch can do what we did in micrograd as a special case when your tensors are all single-element tensors. But the big deal with PyTorch is that everything is significantly more efficient because we are working with these tensor objects, and we can do lots of operations in parallel on all of these tensors. Otherwise, what we have built very much agrees with the API of PyTorch.

Okay, so now that we have some machinery to build out pretty complicated mathematical expressions, we can also start building out neural nets. And as I mentioned, neural nets are just a specific class of mathematical expressions, so we are going to start building out a neural net piece by piece, and eventually we will build out a two-layer multilayer perceptron, as it is called, and I will show you exactly what that means.

Let us start with a single individual neuron. We have implemented one here, but here I am going to implement one that also subscribes to the PyTorch API in how it designs its neural network modules. So just like we saw that we can match the API of PyTorch on the autograd side, we are going to try to do that on the neural network modules side.

Here is `class Neuron`, and just for the sake of efficiency I am going to copy-paste some sections that are relatively straightforward. The constructor will take the number of inputs to this neuron, which is how many inputs come to a neuron. So this one, for example, has three inputs. Then it is going to create a weight that is some random number between negative one and one for every one of those inputs, and a bias that controls the overall trigger happiness of this neuron. Then we are going to implement a `def __call__(self, x)`, some input `x`, and really what we do here is `w times x plus b`, where `w times x` here is a dot product specifically.

Now, if you have not seen `__call__`, let me just return `0.0` here for now. The way this works now is we can have an `x` which is say like `2.0, 3.0`, then we can initialize a neuron that is two-dimensional because these are two numbers, and then we can feed those two numbers into that neuron to get an output. And so when you use this notation `n(x)`, Python will use `__call__`. So currently `__call__` just returns `0.0`.

Now we would like to actually do the forward pass of this neuron instead. So the first thing we are going to do here is we need to basically multiply all of the elements of `w` with all of the elements of `x` pairwise. We need to multiply them. So the first thing we are going to do is we are going to `zip` up `self.w` and `x`, and in Python `zip` takes two iterators and it creates a new iterator that iterates over the tuples of the corresponding entries. So for example, just to show you, we can print this list and still return `0.0` here, sorry, so we see that these `w`s are paired up with the `x`s, `w` with `x`.

Now what we want to do is, for `wi, xi` in, we want to multiply `wi times xi`, and then we want to sum all of that together to come up with an activation, and add also `self.b` on top, so that is the raw activation. And then of course we need to pass that through a non-linearity, so what we are going to be returning is `act.tanh()`, and here is `out`. So now we see that we are getting some outputs, and we get a different output from a neuron each time because we are initializing different weights and biases.

Then, to be a bit more efficient here actually, `sum` by the way takes a second...

### Building neurons, layers, and losses

optional parameter, which is the start, and by default the start is zero, so these elements of this sum will be added on top of zero to begin with, but actually we can just start with `cell.b`, and then we just have an expression like this. Then the generator expression here must be parenthesized in Python. There we go. Yep. So now we can forward a single neuron.

#### Defining a layer of neurons

Next up, we are going to define a layer of neurons. Here we have a schematic for an MLP. We see that these MLPs, each layer — this one layer — has actually a number of neurons, and they're not connected to each other, but all of them are fully connected to the input.

So what is a layer of neurons? It's just a set of neurons evaluated independently. So, in the interest of time, I'm going to do something fairly straightforward here. Literally, a layer is just a list of neurons. And then how many neurons do we have? We take that as an input argument here: how many neurons do you want in your layer, number of outputs in this layer. So we just initialize completely independent neurons with this given dimensionality, and when we call on it, we just independently evaluate them.

So now, instead of a neuron, we can make a layer of neurons. They are two-dimensional neurons, and let's have three of them. And now we see that we have three independent evaluations of three different neurons. Right.

#### Building the MLP

Finally, let's complete this picture and define an entire multi-layer perceptron, or MLP. As we can see here in an MLP, these layers just feed into each other sequentially. So let's come here, and I'm just going to copy the code here in the interest of time. An MLP is very similar: we're taking the number of inputs as before, but now instead of taking a single `nout`, which is number of neurons in a single layer, we're going to take a list of `nouts`, and this list defines the sizes of all the layers that we want in our MLP. So here we just put them all together, and then iterate over consecutive pairs of these sizes and create layer objects for them. And then in the `call` function, we are just calling them sequentially. So that's an MLP, really.

And let's actually re-implement this picture. So we want three input neurons, and then two layers of four, and an output unit. So we want a three-dimensional input. Say this is an example input. We want three inputs into two layers of four and one output. And this, of course, is an MLP. And there we go: that's a forward pass of an MLP.

To make this a little bit nicer, you see how we have just a single element but it's wrapped in a list because `Layer` always returns lists. For convenience, return `out[0]` if `len(out)` is exactly a single element, else return `out` and this will allow us to just get a single value out at the last layer that only has a single neuron. And finally, we should be able to draw `dot(n(x))`.

As you might imagine, these expressions are now getting relatively involved. So this is an entire MLP that we're defining now, all the way until a single output. And so obviously you would never differentiate on pen and paper these expressions, but with micrograd we will be able to backpropagate all the way through this and backpropagate into these weights of all these neurons. So let's see how that works.

#### Example data set and predictions

Okay, so let's create ourselves a very simple example data set here. This data set has four examples, and so we have four possible inputs into the neural net, and we have four desired targets. So we'd like the neural net to assign, or output, `1.0` when it's fed this example, `-1` when it's fed these examples, and `1` when it's fed this example. So it's a very simple binary classifier neural net, basically, that we would like here.

Now let's think what the neural net currently thinks about these four examples. We can just get their predictions — basically we can just call `n(x) for x in xs` — and then we can print. So these are the outputs of the neural net on those four examples. So the first one is `0.91`, but we'd like it to be one, so we should push this one higher. This one we want to be higher. This one says `0.88`, and we want this to be negative one. This is `0.8`, we want it to be negative one. And this one is `0.8`, we want it to be one.

So how do we make the neural net, and how do we tune the weights to better predict the desired targets? And the trick used in deep learning to achieve this is to calculate a single number that somehow measures the total performance of your neural net, and we call this single number the loss. The loss first is a single number that we're going to define that basically measures how well the neural net is performing. Right now we have the intuitive sense that it's not performing very well because we're not very much close to this, so the loss will be high, and we'll want to minimize the loss.

In particular, in this case, what we're going to do is we're going to implement the mean squared error loss. So this is doing, we're going to basically iterate for `y_ground_truth` and `y_output` in `zip` of `ys` and `y_pred`. So we're going to pair up the ground truths with the predictions, and this `zip` iterates over tuples of them. And for each `y_ground_truth` and `y_output`, we're going to subtract them and square them.

So let's first see what these losses are. These are individual loss components, and so basically for each one of the four we are taking the prediction and the ground truth, we are subtracting them and squaring them. So because this one is so close to its target, `0.91` is almost one, subtracting them gives a very small number. So here we would get something like negative point one, and then squaring it just makes sure that regardless of whether we are more negative or more positive, we always get a positive number.

Instead of squaring, we could also take, for example, the absolute value. We need to discard the sign. And so you see that the expression is arranged so that you only get zero exactly when `y_out` is equal to `y_ground_truth`. When those two are equal, so your prediction is exactly the target, you are going to get zero. And if your prediction is not the target, you are going to get some other number.

So here, for example, we are way off, and so that's why the loss is quite high. And the more off we are, the greater the loss will be. So we don't want high loss, we want low loss. And so the final loss here will be just the sum of all of these numbers. So you see that this should be zero roughly plus zero roughly, but plus seven, so loss should be about seven here. And now we want to minimize the loss. We

### Visualizing loss backpropagation

We want the loss to be low, because if the loss is low then every one of the predictions is equal to its target. So the lowest the loss can be is zero, and the greater it is, the worse off the neural net is predicting. So now, of course, if we do `loss.backward()`, something magical happened when I hit Enter. And the magical thing, of course, that happened is that we can look at `n.layers.neuron`, and that layer, at say the first layer, the neuron at zero, because remember that `mlp` has the `layers`, which is a list, and each layer has `neurons`, which is a list, and that gives us an individual neuron. Then it's got some weights, and so we can, for example, look at the weights at zero. Oops, it's not called `weights`, it's called `w`, and that's a `Value`. But now this `Value` also has a `grad` because of the backward pass, and so we see that because this gradient here on this particular weight of this particular neuron of this particular layer is negative, we see that its influence on the loss is also negative. So slightly increasing this particular weight of this neuron of this layer would make the loss go down. And we actually have this information for every single one of our neurons and all their parameters, actually.

It's worth looking at also the `draw_dot(loss)`, by the way. So previously we looked at the `draw_dot` of a single neural neuron forward pass, and that was already a large expression. But what is this expression? We actually forwarded every one of those four examples, and then we have the loss on top of them with the mean squared error. And so this is a really massive graph because this graph that we've built up now, oh my gosh, this graph that we've built up now, which is kind of excessive, it's excessive because it has four forward passes of a neural net for every one of the examples, and then it has the loss on top, and it ends with the value of the loss, which was `7.12`. And this loss will now backpropagate through all the four forward passes, all the way through just every single intermediate value of the neural net, all the way back to, of course, the parameters of the weights, which are the input.

So these weight parameters here are inputs to this neural net, and these numbers here, these scalars, are inputs to the neural net. So if we went around here, we'll probably find some of these examples, this `1.0`, potentially maybe this `1.0`, or, you know, some of the others, and you'll see that they all have gradients as well. The thing is, these gradients on the input data are not that useful to us, and that's because the input data seems to be not changeable. It's a given to the problem, and so it's a fixed input.

### Collecting neural net parameters

We are not going to be changing it or messing with it, even though we do have gradients for it. But some of these gradients here will be for the neural network parameters, the `ws` and the `bs`, and those we of course want to change.

So now we are going to want some convenience code to gather up all of the parameters of the neural net so that we can operate on all of them simultaneously, and every one of them we will nudge a tiny amount based on the gradient information. So let us collect the parameters of the neural net all in one array.

#### Collecting parameters in a list

Let us create a `parameters` of `self` that just returns `celta w`, which is a list concatenated with a list of `self.b`. So this will just return a list; list plus list just gives you a list. So that is `parameters` of neuron, and I am calling it this way because also PyTorch has a `parameters` on every single `nn.Module`, and it does exactly what we are doing here: it just returns the parameter tensors for us as the parameter scalars.

Now `Layer` is also a module, so it will have `parameters` itself. And basically what we want to do here is something like this: `params` is here, and then for neuron in `self.out neurons` we want to get `neuron.parameters`, and we want to `params.extend`, right? So these are the parameters of this neuron, and then we want to put them on top of `params`, so `params.extend` of these, and then we want to return `params`.

So this is way too much code. Actually, there is a way to simplify this, which is `return p for neuron in self.neurons for p in neuron.parameters`. So it is a single list comprehension in Python. You can sort of nest them like this, and then create the desired array. So these are identical. We can take this out, and then let us do the same here: `def parameters self` and return a parameter for `layer` in `self.layers` for `p` in `layer.parameters`, and that should be good.

Now let me pop out this so we do not re-initialize our network, because we need to re-initialize our... so unfortunately we will have to probably re-initialize the network because we just add functionality because this class, of course, we want to get all the and that parameters, but that is not going to work because this is the old class. So unfortunately we do have to re-initialize the network, which will change some of the numbers, but let me do that so that we pick up the new API.

We can now do in the parameters, and these are all the weights and biases inside the entire neural net. So in total this MLP has 41 parameters, and now we will be able to change them if we recalculate the loss here we see.

### Training with gradient descent

that unfortunately we have slightly different predictions and slightly different laws, but that's okay.

We see that this neuron's gradient is slightly negative. We can also look at its data right now, which is `0.85`, so this is the current value of this neuron, and this is its gradient on the loss. What we want to do now is iterate for every `p` in `n.parameters()`, so for all the 41 parameters in this neural net, we actually want to change `p.data` slightly according to the gradient information. `...` to do here, but this will be basically a tiny update in this gradient descent scheme.

In gradient descent, we are thinking of the gradient as a vector pointing in the direction of increased loss. So in gradient descent, we are modifying `p.data` by a small step size in the direction of the gradient. The step size, as an example, could be a very small number like `0.01` times `p.grad`, right? But we have to think through some of the signs here. In particular, working with this specific example here, we see that if we just left it like this, then this neuron's value would be currently increased by a tiny amount of the gradient. The gradient is negative, so this value of this neuron would go slightly down. It would become like `0.84` or something like that. But if this neuron's value goes lower, that would actually increase the loss. That's because the derivative of this neuron is negative, so increasing this makes the loss go down. So increasing it is what we want to do instead of decreasing it.

Basically, what we're missing here is a negative sign. Again, there's this other interpretation, and that's because we want to minimize the loss. We don't want to maximize the loss. We want to decrease it. And the other interpretation, as I mentioned, is you can think of the gradient vector, so basically just the vector of all the gradients, as pointing in the direction of increasing the loss, but then we want to decrease it, so we actually want to go in the opposite direction. You can convince yourself that this sort of plug does the right thing here with the negative because we want to minimize the loss. So if we nudge all the parameters by a tiny amount, then we'll see that this data will have changed a little bit. So now this neuron is a tiny amount greater value, so `0.854` went to `0.857`, and that's a good thing because slightly increasing this neuron's data makes the loss go down according to the gradient, and so the correct thing has happened sign-wise.

Now what we would expect, of course, is that because we've changed all these parameters, we expect that the loss should have gone down a bit. So we want to re-evaluate the loss. This is just a data definition that hasn't changed, but the forward pass here of the network we can recalculate. Actually, let me do it outside here so that we can compare the two loss values. If I recalculate the loss, we'd expect the new loss now to be slightly lower than this number. So hopefully what we're getting now is a tiny bit lower than `4.84`, `4.36`.

Remember, the way we've arranged this is that low loss means that our predictions are matching the targets. So our predictions now are probably slightly closer to the targets. And now all we have to do is iterate this process. Again, we've done the forward pass and this is the loss. Now we can call `loss.backward()`. Let me take these out, and we can do a step size, and now we should have a slightly lower loss: `4.36` goes to `3.9`. So we've done the forward pass, here's the backward pass, nudge, and now the loss is `3.66`, `3.47`, and you get the idea. We just continue doing this, and this is gradient descent. We're just iteratively doing forward pass, backward pass, update, forward pass, backward pass, update, and the neural net is improving its predictions.

Here, if we look at `ypred`, we see that this value should be getting closer to one, so this value should be getting more positive, these should be getting more negative, and this one should be also getting more positive. If we just iterate this a few more times, actually we may be able to afford to go a bit faster. Let's try a slightly higher learning rate. Oops, okay, there we go. So now we're at `0.31`.

If you go too fast, by the way, if you try to make it too big of a step, you may actually overstep. It's overconfidence because, again, remember we don't actually know exactly about the loss function. The loss function has all kinds of structure, and we only know about the very local dependence of all these parameters on the loss. But if we step too far, we may step into a part of the loss that is completely different, and that can destabilize training and make your loss actually blow up even.

So the loss is now `0.04`, so actually the predictions should be really quite close. Let's take a look. You see how this is almost one, almost negative one, almost one. We can continue going. So yep, backward, update. Oops, there we go. So we went way too fast, and we actually overstepped, so we got too eager.

Where are we now? Oops, okay, `7e-9`. So this is very, very low loss, and the predictions are basically perfect. So somehow we basically were doing way too big updates, and we briefly exploded, but then somehow we ended up getting into a really good spot. Usually this learning rate and the tuning of it is a subtle art. You want to set your learning rate so that if it's too low, you're going to take way too long to converge. But if it's too high, the whole thing gets unstable, and you might actually even explode the loss depending on your loss function. So finding the step size to be just right is a pretty subtle art sometimes when you're using sort of vanilla gradient descent. But we happened to get into a good spot. We can look at `n.parameters()`. So this is the setting of weights and biases that makes our network predict the desired targets very, very close, and basically we've successfully trained a neural net.

Okay, let's make this a tiny bit more respectable and implement an actual training loop and what that looks like. So this is the data definition that stays, this is the forward pass. For `k in range`, you know, we're going to take a bunch of steps. First you do the forward pass, we validate the loss. Let's re-initialize the neural net from scratch, and here's the data, and we first do the forward pass, then we do the backward pass, and then we do an update. That's gradient descent, and then we should be able to iterate.

### Training pitfalls and final recap

this and we should be able to print the current step, the current loss. Let’s just print the sort of number of the loss, and that should be it. Then the learning rate `0.01` is a little too small; `0.1` we saw is a little bit dangerously too high. Let’s go somewhere in between, and we’ll optimize this for not 10 steps but let’s go for, say, 20 steps. Let me erase all of this junk, and let’s run the optimization.

And you see how we’ve actually converged slower in a more controlled manner and got to a loss that is very low, so I expect white bread to be quite good. There we go. And that’s it.

#### A subtle bug

Okay, so this is kind of embarrassing, but we actually have a really terrible bug in here, and it’s a subtle bug, and it’s a very common bug, and I can’t believe I’ve done it for the 20th time in my life, especially on camera. And I could have reshot the whole thing, but I think it’s pretty funny, and you get to appreciate a bit what working with neural nets maybe is like sometimes. We are guilty of some bug.

I actually tweeted the most common neural net mistakes a long time ago now, and I’m not really going to explain any of these except for we are guilty of number three: **you forgot to zero grad before that backward**.

What is that basically happening? It’s a subtle bug, and I’m not sure if you saw it, is that all of these weights here have a `.data` and a `.grad`, and that grad starts at zero. Then we do backward and we fill in the gradients, and then we do an update on the data, but we don’t flush the grad. It stays there. So when we do the second forward pass and we do backward again, remember that all the backward operations do a `+=` on the grad, and so these gradients just add up, and they never get reset to zero. So basically we didn’t zero grad.

Here’s how we zero grad before backward: we need to iterate over all the parameters and we need to make sure that `p.grad` is set to zero. We need to reset it to zero, just like it is in the constructor. So remember, all the way here, for all these value nodes, `grad` is reset to zero, and then all these backward passes do a `+=` from that grad. But we need to make sure that we reset these graphs to zero so that when we do backward, all of them start at zero and the actual backward pass accumulates the loss derivatives into the grads. So this is zero grad in PyTorch.

And we will slightly get a different optimization. Let’s reset the neural net; the data is the same; this is now, I think, correct. And we get a much slower descent. We still end up with pretty good results, and we can continue this a bit more to get down lower and lower and lower. Yeah.

So the only reason that the previous thing worked, it’s extremely buggy, the only reason that worked is that this is a very, very simple problem and it’s very easy for this neural net to fit this data. And so the grads ended up accumulating, and it effectively gave us a massive step size, and it made us converge extremely fast. But basically now we have to do more steps to get to very low values of loss and get white bread to be really good. We can try to step a bit greater. Yeah, we’re going to get closer and closer to `1`, `-1`, and `1`.

#### Neural nets in one picture

Working with neural nets is sometimes tricky because you may have lots of bugs in the code, and your network might actually work just like ours worked. But chances are, if we had a more complex problem, then actually this bug would have made us not optimize the loss very well, and we were only able to get away with it because the problem is very simple.

So let’s now bring everything together and summarize what we learned. What are neural nets? Neural nets are these mathematical expressions, fairly simple mathematical expressions in the case of a multi-layer perceptron, that take input as the data and they take input the weights and the parameters of the neural net. It is a mathematical expression for the forward pass, followed by a loss function, and the loss function tries to measure the accuracy of the predictions. Usually the loss will be low when your predictions are matching your targets or when the network is basically behaving well.

So we manipulate the loss function so that when the loss is low, the network is doing what you want it to do on your problem. And then we backward the loss, use backpropagation to get the gradient, and then we know how to tune all the parameters to decrease the loss locally. But then we have to iterate that process many times in what’s called gradient descent. So we simply follow the gradient information, and that minimizes the loss, and the loss is arranged so that when the loss is minimized, the network is doing what you want it to do.

And yeah, so we just have a blob of neural stuff, and we can make it do arbitrary things, and that’s what gives neural nets their power. It’s, you know, this is a very tiny network with 41 parameters, but you can build significantly more complicated neural nets with billions — at this point almost trillions — of parameters, and it’s a massive blob of neural tissue, simulated neural tissue roughly speaking, and you can make it do extremely complex problems. And these neurons then have all kinds of very fascinating emergent properties when you try to make them do significantly hard problems.

As in the case of GPT, for example, we have massive amounts of text from the internet, and we’re trying to get a neural net to predict — to take a few words and try to predict the next word in a sequence. That’s the learning problem. And it turns out that when you train this on all of the internet, the neural net actually has really remarkable emergent properties. But that neural net would have hundreds of billions of parameters, and it works on fundamentally the exact same principles.

The neural net, of course, will be a bit more complex, but otherwise the value in the gradient is there and would be identical, and the gradient descent would be there and would be basically identical. But people usually use slightly different updates. This is a very simple stochastic gradient descent update, and the loss function would not be mean squared error; they would be using something called the cross-entropy loss for predicting the next token. So there are a few more details, but fundamentally the neural network setup and neural network training is identical and pervasive, and now you understand intuitively how that works under the hood.

At the beginning of this video I told you that by the end of it you would understand everything in micrograd, and then we’d slowly build it up. Let me briefly prove that to you.

### Walking through the micrograd code

So I am going to step through all the code that is in **micrograd** as of today. Actually, potentially some of the code will change by the time you watch this video, because I intend to continue developing micrograd, but let us look at what we have so far at least.

`engine.py` is empty when you go to `engine.py`; that has the value. Everything here you should mostly recognize. We have the `data`, `grad` attributes. We have the `backward` function. We have the previous set of children and the operation that produced this value. We have addition, multiplication, and raising to a scalar power. We have the `ReLU` non-linearity, which is a slightly different type of non-linearity than `tanh` that we used in this video. Both of them are non-linearities, and notably `tanh` is not actually present in micrograd as of right now, but I intend to add it later with the backward, which is identical.

Then all of these other operations are built up on top of the operations here, so values should be very recognizable except for the non-linearity used in this video. There is no massive difference between `ReLU` and `tanh` and `sigmoid` and these other non-linearities. They are all roughly equivalent and can be used in MLPs. I use `tanh` because it is a bit smoother and because it is a little bit more complicated than `ReLU` and therefore it stresses the local gradients and working with those derivatives a little bit more, which I thought would be useful.

Then `nn.py` is the neural networks library, as I mentioned, so you should recognize the identical implementation of `Neuron`, `Layer`, and `MLP`. Notably, or not so much, we have a class `Module` here. There is a parent class of all these modules. I did that because there is an `nn.Module` class in PyTorch, and so this exactly matches that API. And `nn.Module` in PyTorch has also a `zero_grad`, which I have refactored out here.

So that is micrograd, really. Then there is a test, which you will see basically creates two chunks of code, one in micrograd and one in PyTorch, and we will make sure that the forward and the backward pass agree identically for a slightly less complicated expression, a slightly more complicated expression. Everything agrees. So we agree with PyTorch on all of these operations.

Finally, there is a `demo.ipynb` here, and it is a bit more complicated binary classification demo than the one I covered in this lecture. We only had a tiny data set of four examples. Here we have a bit more complicated example with lots of blue points and lots of red points, and we are trying to again build a binary classifier to distinguish two-dimensional points as red or blue. It is a bit more complicated MLP here, with a bigger MLP. The loss is a bit more complicated because it supports batches, so because our dataset was so tiny we always did a forward pass on the entire

### Mini-batches and PyTorch internals

In the data set of four examples, you can do everything directly. But when your data set is like a million examples, what we usually do in practice is batch it: we basically pick out some random subset, call that a **batch**, and only process the batch forward, backward, and update, so we do not have to forward the entire training set. This supports batching because there is a lot more examples here.

Here we do a forward pass, and the loss is slightly different. This is a **max margin loss** that I implement here. The one that we used was the **mean squared error loss** because it is the simplest one. There is also the **binary cross entropy loss**. All of them can be used for binary classification and do not make too much of a difference in the simple examples that we looked at so far.

There is something called **L2 regularization** used here. This has to do with generalization of the neural net and controls overfitting in a machine learning setting, but I did not cover these concepts in this video, potentially later.

The training loop you should recognize: forward, backward, zero grad, update, and so on. You will notice that in the update here, the learning rate is scaled as a function of number of iterations, and it shrinks. This is something called **learning rate decay**. In the beginning you have a high learning rate, and as the network sort of stabilizes near the end, you bring down the learning rate to get some of the fine details in the end.

In the end we see the decision surface of the neural net, and we see that it learns to separate out the red and the blue area based on the data points. So that is the slightly more complicated example, and then we will demo that hyper ymb that you are free to go over.

But yeah, as of today, that is micrograd. I also wanted to show you a little bit of real stuff so that you get to see how this is actually implemented in a production grade library like PyTorch. In particular, I wanted to find and show you the backward pass for `tanh` in PyTorch.

Here in micrograd, we see that the backward pass of `tanh` is `1 - t^2`, where `t` is the output of the `tanh` of `x`, times the gradient, which is the chain rule. So we are looking for something that looks like this.

Now I went to PyTorch, which has an open source GitHub codebase, and I looked through a lot of its code. Honestly, I spent about 15 minutes and I could not find `tanh`. That is because these libraries, unfortunately, they grow in size and entropy, and if you just search for `tanh`, you get apparently 2,800 results and 406 files. So I do not know what these files are doing, honestly, and why there are so many mentions of `tanh`, but unfortunately these libraries are quite complex. They are meant to be used, not really inspected.

Eventually I did stumble on someone who tries to change the `tanh` backward code for some reason, and someone here pointed to the CPU kernel and the CUDA kernel for `tanh` backward. So basically this depends on if you are using PyTorch on a CPU device or on a GPU, which are different devices, and I have not covered this. But this is the `tanh` backward kernel for CPU, and the reason it is so large is that number one, this is like if you are using a complex type, which we have not even talked about, and if you are using a specific data type of bfloat16, which we have not talked about, and then if you are not, then this is the kernel.

And deep here we see something that resembles our backward pass. They have a `* (1 - b^2)`, so this `b` here must be the output of the `tanh`, and this is the `health.grad`. So here we found it, deep inside PyTorch from this location for some reason inside `binaryops_kernel` when `tanh` is not actually a binary op, and then this is the GPU kernel. We are not complex, we are here, and here we go with one line of code.

So we did find it, but basically unfortunately these code pieces are very large and micrograd is very, very simple. But if you actually want to use real stuff, ...

### Custom autograd functions in PyTorch

Finding the code for it, you will actually find that difficult. I also wanted to show you a little example here where PyTorch is showing you how you can register a new type of function that you want to add to PyTorch as a LEGO building block.

So here, if you want to, for example, add a general polynomial 3, here is how you could do it. You will register it as a class that subclasses `torch.autograd.Function`, and then you have to tell PyTorch how to forward your new function and how to backward through it. So as long as you can do the forward pass of this little function piece that you want to add, and as long as you know the local derivative, the local gradients, which are implemented in the backward, PyTorch will be able to backpropagate through your function. Then you can use this as a LEGO block in a larger LEGO castle of all the different LEGO blocks that PyTorch already has. That is the only thing you have to tell PyTorch, and everything would just work. You can register new types of functions in this way, following this example.

And that is everything that I wanted to cover in this lecture. So I hope you enjoyed building out micrograd with me. I hope you find it interesting, insightful, and yeah, I will post a lot of the links that are related to this video in the video description below. I will also probably post a link to a discussion forum or discussion group where you can ask questions related to this video, and then I can answer or someone else can answer your questions. I may also do a follow-up video that answers some of the most common questions. But for now, that is it. I hope you enjoyed it. If you did, then please like and subscribe so that YouTube knows to feature this video to more people. And that is it for now. I will see you later.

Now here is the problem. We know `dL` by... wait, what is the problem? And that is everything I wanted to cover in this lecture, so I hope you enjoyed us building up microcraft, micro crab. Okay, now let's do the exact same thing for multiply, because we can't do something like `a * 2`. Oops, I know what happened there.