Microsoft has introduced 1-bit LLM, a new language model that quite differs from the traditional models. Research on BitNet has contributed to the development of this project.

What is new is the representation of parameters in the model. Parameters refer to the weights which use 1.58 bits. Traditional LLMs use 16 bit floating point values (FTP 16) for weights. However, BitNet’s each weight is restricted to -1, 0 or 1 values. It is a substantial reduction in bit usage.

Bitnet (b 1.58) despite the reduction performs as well as the traditional models. It is a cost effective model. Suitable hardware can be customized for BitNet model. It supports long sequences. It reduces the GPU usage as there will be only additions of weights as multiplication of inputs and weights will be bypassed since weights are -1, 0 and 1 (In forward pass).

1-Bit LLM uses quantization process. It reduces energy consumption and computational resources.