Reverse-Mode Automatic Differentiation

Reverse-mode automatic differentiation is the mechanism underlying the backpropagation algorithm used in training neural networks.

Automatic differentiation (AD) numerically evaluates the derivative function specified by a computer programme. It leverages the chain rule of calculus to compute derivatives systematically.

There are two modes of AD. Forward mode computes derivatives alongside the function evaluation. It is suitable for function with fewer inputs than outputs. Reverse mode AD computes derivatives by performing first the forward pass to evaluate the function. Later, a backward pass is performed to propagate derivatives from outputs to inputs. It suits for functions with fewer outputs them inputs.

It is particularly more effective in neural networks as they have a large number of parameters (inputs) and fewer loss values (outputs).

Backpropagation is an application of reverse mode AD. It works by performing a forward pass to compute the output. Loss is calculated. Then a backward pass is performed to compute the gradients of the loss with respect to each parameter in the network, using the chain rule.

Backpropagation, in a nutshell, is a specialized use of reverse mode AD customized for computing gradients in neural networks.