Sora, a video making software of OpenAI, is powered by DiT. AI innovation is stimulated by transformers and diffusion models. These are the architectures. Transformers are used in neural networks. They have rejuvenated language and text-based models. Diffusion models are used for image generation.
Diffusion basically means the process of spread. The particles spread from dense space to less dense area. Diffusion transformer: DiT is a diffusion model based on transformer architecture.
DiT has been developed William Peebles (formerly with UC Berkeley, now with OpenAI) and Saining XE (NY University) in 2023. It uses U-Net — iterative image de-noising. Making an image is akin to solving a jigsaw puzzle. The puzzle pieces can be arranged in a number of ways. Every time it is not the best solution. DiT enables us to solve big puzzles.
Sora uses diffusion to predict videos. It then uses the strength of the transformers for scaling. It has to focus on the attention pooling stage — important parts of the video. It then uses the strength of the transformers for scaling. It has to focus on the attention pooling stage — important parts of the video. There is random noise added on the prompt. It is to be seen how the noise affects the video. The model predicts what should come next by trying various combinations. It learns from its mistakes and improves. The finished video at the end is smooth and clear and is bereft of all extra noise. DiT, thus, helps Sora to understand text prompts and produce cool videos. It draws on the images it has stored during its training.
DiT deploys transformers where noise is slowly transformed into a target image. It reverses the diffusion process by being guided by the transformer. It is like editing a blurry image. DiT can handle larger input data without compromising quality.