Fast LLMs

There is heavy math behind LLMs so that they generate text shockingly fast.

These run on GPUs or TPUs built to do massive parallel operations. It boils down to linear algebra, mainly rallel matrix operations. It boils down to linear algebra, mainly matrix multiplications and additions. They perform thousands of these operations at once. The model does it in parallel, and not step-by-step as in CPU.

ently. The transformer architecture is designed for speed. Instead of processing language word by word (like RNNs), they use self-attention, allowing them to look at the entire context at once. The design generates output (inference) faster and more efficiently.

LLMs generate text one token at a time but the computation for each token is highly optimized. There are frameworks such as PyTorch, TensorFlow, JAY, CUDA backends that make the model run extremely efficiently.

While generating multiple tokens in a sequence (such as a full sentence), models cache the internal activations, (such as key and value vectors in attention layers) from previous steps. It is called KV caching and massively reduces computation time in generation.

Some models are optimized further by quantization and pruning, thus reducing memory and increasing speed with minimal loss of accuracy.

A modern LLM such GPT4, runs on a GPU such as A100, which can generate hundreds of tokens per second (though it involves billions of parameters and trillions of multiplications). It is thus a triumph of smart architecture, parallel computing and engineering optimization.