Find the best-fit line through noisy data by minimising prediction error — gradient descent in action, step by step.
Linear regression fits a straight line (or plane) to data by finding parameters that minimise the total squared error. Gradient descent does it automatically — nudging the line a little in the downhill direction of the error surface on every step, until it settles at the best fit.
Yellow dots are training samples. The blue line starts at a random position and iterates toward the best fit. Residuals show the prediction error for each point.
Three ideas drive every linear regression: the model, the cost function, and the optimiser.
The model is just a line: slope m times the input, plus a bias b. With multiple inputs (features) it becomes a plane or hyperplane — same idea, more dimensions.
We measure fit by squaring the distance from each data point to the line and averaging them. Squaring penalises large errors more, and gives a smooth surface to optimise.
Compute the slope of the error surface (the gradient), take a small step in the opposite direction. Repeat until the error stops falling. The step size is the learning rate.
The vertical distance from each data point to the fitted line. Residuals should look random — if they have a pattern, the model is missing something the data contains.