An RRN is Recurrent Neural Network. It is a type of neural network that can handle sequential data. It is commonly used in tasks like NLP, speech recognition and time series prediction.
Feedforward networks process each input independently. In RNN networks, the order of input matters. They capture inputs information about previous inputs. the information is retained over a period of time. The decisions are based on both current and past inputs.
RNNs maintain a hidden state of memory of past inputs. This is updated recursively (it means it considers both current and past inputs). Thus, there is a feedback loop. They thus capture temporal dependencies in the data.
They handle inputs and outputs of variable lengths. They are thus suitable for text generation, machine translation, speech recognition.
RNNs face vanishing gradient problem. The gradients become extremely small during training. That makes it difficult for the model to learn long-term dependencies. That has led to more advanced RNNs such as LSTM: Long Short-Term Memory and GRU: Gated Recurrent Unit. These versions are better at capturing long-term dependencies.
In short, RNNs are good at processing sequential data since they retain memory and capture temporal dependencies. They have limitations and these are overcome by using advanced versions such as LSTM and GRU.
LSTM’s heart consists of memory cells. They store information over longer periods of time. These memory cells have an internal state which can be updated over time based on the input data and previous states.
There are three types of gates in LSTMs. They control the flow of information in and out of memory cells. The Forget Gate decides which information to discard from the cell state. It takes as input the previous hidden state and the current input. The outputs are a number between 0 and 1 for each element of the cell state. 0 means forget completely. 1 means remember completely. The Input Gate decides which new information to store in the cell state. It also takes previous hidden state and the current input as input. The outputs are numbers between 0 and 1. It indicates how much of the new information should be added to the cell state. The Output Gate decides which information to output from the current cell state. It takes previous hidden state and current input as input. The outputs are a number between 0 and 1. It indicates how much of each element should be output.
The cell state is updated based on information from forget gate, input gate and the current input.
The hidden state of the LSTM is computed based on the updated cell state and the input data.