At present, for natural language processing (NLP), there are two models in use — BERT and GPT. Both are large language models (LLMs). Both these models are based on transformer architecture. However, there are some key differences between these models.

BERT stands for Bidirectional Encoder Representations from Transformers. This model is bidirectional, i.e. it can process text in both the directions. It is thus appropriate for question answering and sentiment analysis since the model has to comprehend the full context. BERT has been trained on a dataset of text and code. It has 340 million parameters. BERT is open source. It is easier to fine tune BERT for specific tasks. BERT works on encoding mechanisms to generate language. BERT is a transformer encoder. BERT is bi-directional. BERT’s input and output positions of each token are the same.

GPT stands for Generative Pre-trained Transformer. It is an autoregressive model, i.e. it can process text in one direction only. It is appropriate for text summarization and translation, where the model has to generate new text based on a given prompt. GPT has been trained on a dataset of text only, and it has 1.5 billion parameters. GPT is proprietary model. GPT relies on decoder part of the Transformer to generate text. GPT is a Transformer decoder. GPT is unidirectional. GPT is meant for autoregressive inference. These models generate an output of one token at a time. The probability distributions over the next token depends on the previous tokens. In short, these models generate output by predicting one token at a time based on previous tokens.