How Neural Networks Work: Neurons, Learning, AI

A neuron made of arithmetic

In your brain a neuron collects signals from others and fires if the total is strong enough. The human brain has roughly 86 billion of them. An artificial neuron is a loose cartoon of that, not a copy. It takes numbers in, multiplies each by a weight that says how important it is, adds them together with a bias, and passes the sum through an activation function that squashes it into a range such as 0 to 1. The idea goes back to McCulloch and Pitts in 1943 and to Frank Rosenblatt's perceptron in 1958.

One neuron, worked out

Give a neuron inputs 0.5 and 0.8, weights 0.9 and 0.3, and a bias of −0.2. The sum is 0.5 × 0.9 + 0.8 × 0.3 − 0.2 = 0.45 + 0.24 − 0.2 = 0.49. The sigmoid activation, 1 / (1 + e to the power −x), turns 0.49 into about 0.62. A positive weight pushes the neuron on, a negative weight holds it back. That is all: multiplication and addition. Everything a network knows is stored in its weights. In the picture, thick lines are strong weights and thin ones are weak.

Layers

A single neuron can only draw a straight line between two groups. Stack neurons in layers and the network can draw curves. The input layer holds the data, hidden layers combine it, and the output layer gives the answer. For a 28 × 28 image of a digit you can use 784 inputs, a hidden layer of 128 neurons and 10 outputs. The number of weights and biases is 784 × 128 + 128 + 128 × 10 + 10 = 101,770 knobs. The signal flows forward layer by layer. The largest models today have billions or trillions of knobs.

Learning is nudging

At the start the weights are random and the answers are nonsense. Show the network an example with a known answer and measure the error, a single number called the loss. Then ask each weight: if you grew a little, would the loss go up or down? The gradient says which way is uphill, so you step downhill: new weight = weight − η × gradient, where η is a small learning rate. With w = 0.80, gradient +0.5 and η = 0.1, the new weight is 0.80 − 0.05 = 0.75. Repeat for thousands of examples. This is gradient descent.

Blame, passed backwards

A real network has thousands or millions of weights. How can we find every gradient? Backpropagation applies the chain rule of calculus backwards from the output to the input, sharing the blame layer by layer. Seppo Linnainmaa and Paul Werbos described the idea in 1970 and 1974, and the 1986 Nature paper of David Rumelhart, Geoffrey Hinton and Ronald Williams made it famous. In 2024 Hinton and John Hopfield shared the Nobel Prize in Physics for foundations of machine learning with artificial neural networks. Press train in the sandbox and watch the dividing line move.

Reading a digit

MNIST is a collection of 70,000 handwritten digits, each 28 × 28 grey pixels. 60,000 are for training and 10,000 are kept back for testing. The brightness of every pixel, from 0 to 1, enters one input neuron. The output layer has ten neurons, one per digit, and the strongest one wins, for example 0.93 for the digit 7 and tiny values for the rest. Small networks get about 98 percent of unseen test digits right, and convolutional networks, which look for edges and shapes, pass 99 percent. They match patterns; they do not understand digits as you do.

Predicting the next word

A large language model, the engine inside most chatbots, works with tokens, which are pieces of words. Given the tokens so far, the network outputs a probability for every possible next token. One is chosen and added to the text, and the process repeats. The model is trained on a huge amount of text and has billions of weights. The architecture used today, the transformer, was introduced in the 2017 paper Attention Is All You Need. A model does not look an answer up; it generates one, word after word.

Fluent is not the same as true

Because a language model predicts what text is likely, it can write a smooth, confident sentence that is false: an invented book title, a wrong date, a wrong sum. This is called a hallucination. The model has no built-in way to check facts, unless it is connected to tools or sources. Treat its answer as a helpful draft, not an oracle, and check anything important in reliable sources such as those listed under this chapter. Finally, train your own tiny network in the sandbox and see how little it takes to learn.

Neural networks and AI