Storage for the AI Data

Organizations adopting AI have to deal with their stored data — the system of storage has to be redesigned with built-in intelligence, guardrails and GPU-level performance. Independent organizations such as NetApp manage the external storage. It provides a new system of architecture called AFX — the outcome is AI-ready data. There is a combination of extreme-performance storage with GPUs. It enables processing in place, rather than data being copied repeatedly across applications. It makes a shift towards insanely faster system, and eliminates six or more redundant copies created during AI workload steps such as annotation, tagging, governance and training.

AFX works alongside NetAPP — it is called AI Data Engine. It consists of metadata engine, security, guardrails, and data transformation layer. All this is done without having to create secondary copies. Both training and inference become faster. There are no copies of petabytes and exabytes of enterprise data.

NetApp employs hundreds of engineers to make this possible. Previously, AI workloads were smaller and predictable. However, with LLMs around, the access to the data should be really fast. The number of times the GPUs hit the storage has increased exponentially.

It is a disintegrated architecture. The compute and storage are split. They can scale independently. There is rewriting of deep layers of NetApp’s storage stack. It creates a metadata engine capable of handling vast indexes. It builds system of vector embeddings. The data could be fed into model-training pipelines. The pipeline is cut short. One box moves through which data moves very fast, classified and creating metadata. This performance talks to directly to GPUs.

In generative AI, metadata engine, composable architecture, near-compute design and zero latency inferencing play a vital role. India delivers on all these four factors.

NetApp like organizations get ready for AI. India has re-architected the entire stack so that compute and storage can scale independently by adding metadata engines, classification tools, vector embeddings directly at the storage level. It shortens AI pipelines and lets enterprise train at high speed without creating endless data copies.