Neural network is basically a mathematical model implemented by a software. It runs on computer hardware. The physical manifestations of neural networks are seen as neuromorphic chips. The architecture itself is a computational model represented in a code.

Even transformers and its encoder-decoder are conceptual components, rather than physical objects. They consist of layers of neural network computations. They are implemented through software libraries such PyTorch or TensorFlow. The architecture can be visualized in diagrams. One can inspect their code implementation.

Nodes in such architecture are computational units that perform specific operations on the input data. In transformer architecture, they represent neurons. These neurons perform operations such as (matrix) multiplications, activation functions (ReLU, SoftMax) and layer normalization.

Each node receives input from the previous layer. It processes the input as per defined function. It passes the result to the next layer. The collective behaviour of these nodes enables the model to learn complex patterns and make predictions on the new data.