Three takeaways from NVIDIA GTC 2024

NVIDIA’s GPU Technology Conference (GTC) is one of the most anticipated events in the AI industry, showcasing the latest innovations and trends in GPU computing, deep learning, computer vision, and more. This year, GTC 2024 was held from March 18 to 21, and featured over 900 sessions, keynotes, demos, and workshops from experts and leaders in the field. As a proud partner of NVIDIA, Micron was there at the event, highlighting how our memory and storage solutions are enabling the next generation of AI platforms and applications. Here are three key observations from GTC 2024 that demonstrate the importance of memory and storage in the AI ecosystem.
 

1. Memory and storage are key enablers of AI platforms
 

As data is critical to AI, a key theme of GTC 2024 was the increasing demand for memory and storage performance and capacity in AI workloads. As AI models grow in size and complexity, they require more data to be processed and stored at faster speeds and lower latencies. This poses a challenge for traditional memory and storage architectures, which can become bottlenecks for AI performance and efficiency. To address this challenge, Micron showcased its portfolio of memory and storage solutions that are designed to optimize the data flow and availability for AI platforms. These include:

• The industry-leading performance and power efficiency of Micron’s HBM3E 8-high 24GB and 12-high 36GB cubes. Micron HBM3E 8-high 24GB is now in volume production and will be part of NVIDIA’s H200 GPU shipping in Q2 of calendar year 2024
• Micron CZ120 CXL™ Memory Module which provides the required capacity, bandwidth, and flexibility to accelerate AI and in-memory workloads
• Micron 9000 and 7000 series SSDs that support up to 17 GPUs in the 3D Medical Imaging benchmark Unet3D
• Micron 6000 series SSDs that boost AI data lake ingest by up to 48% when compared to our competitor’s capacity-focused SSDs¹
• Real-world lab results demonstrating how Micron’s technologies improve AI training and inference including LLMs, computer imaging, GNN, and more

By leveraging these memory and storage solutions throughout the entire data stack (near memory, main memory, expansion memory, SSD data cache and networked data lakes), Micron is helping accelerate the AI revolution, enabling AI platforms to handle more data, faster, and more efficiently.
 

2. AI is the killer use case for PCIe^® Gen5
 

Another highlight of GTC 2024 was the introduction of the new NVIDIA B100 accelerators, based on the PCIe Gen5 interface. The PCI Express standard is the most widely used interface for connecting high-performance CPUs, GPUs, SSDs, and network cards. PCIe Gen5 offers double the bandwidth of Gen4, enabling up to 32GT/s of data transfer per lane. This is a game-changer for AI workloads, which can benefit from the increased data throughput.

However, to fully exploit the potential of PCIe Gen5 in the data center, the devices that are connected to the interface must also be able to support the higher speed and lower latency. This is where Micron's PCIe Gen5 SSDs come in.

NVIDIA, Dell and Micron recently collaborated to showcase the benefits of Micron’s PCIe Gen5 SSD, Big Accelerator Memory² (BaM), NVIDIA H100 accelerators, and PowerEdge servers. The demonstration showed a 50% reduction in graph neural network training time when offloading the GNN training model to a PCIe Gen5 high-performance SSD vs. Gen4.
 

This test also showed a 5x performance improvement of NVIDIA H100 (Gen5) over A100 (Gen4). With 5x generation over generation improvements in GPU performance, storage devices will need to advance quickly to keep up. A typical deep learning recommendation (DLRM) workload will result in queue depths of 10 to 100, reading 128K to 512K chunks. On a Gen5 SSD this will typically hit the maximum drive throughput of ~14GB/s. With AI model offload, small block performance becomes critically important. The GNN demo detailed above reads 4K blocks at queue depths of over 1,000, easily reaching the maximum random read throughput of the fastest PCIe Gen5 SSD.
 

In Micron’s PCIe Gen5 technology demo, we've shown not only 14GB/s in sequential throughput, but also random reads of 3,300,000 IOPS. In a 4K workload that translates into a throughput of 13.2GB/s, between 22% and 32% faster than competitive offerings in the market today.
 

By providing such high performance and efficiency, Micron's PCIe Gen5 SSDs can enable AI platforms to leverage the full power of the new NVIDIA accelerators, resulting in faster outcomes and a better return on investment on your AI hardware purchases.
 

3. Networked data lakes are increasingly being deployed on high capacity SSDs
 

A third observation from GTC 2024 was the growing trend of deploying networked data lakes on SSDs, rather than HDDs, to store and access the massive amounts of data that are generated and consumed by AI applications. Networked data lakes are large and distributed repositories of data that are connected to the AI platform via a network, such as InfiniBand or Ethernet. While networked data lakes provide scalable and flexible storage capacity for AI data, enabling data sharing and collaboration across different platforms and users, they also introduce challenges in terms of data transfer speeds and density, which greatly affect the total cost of ownership (TCO) calculation.

To overcome these challenges, many AI users and developers are opting for high capacity SSDs such as the Micron 6500 ION, rather than HDDs, to build and operate their networked data lakes. This class of SSDs offers several advantages over HDDs for networked data lakes, such as:

• Faster data ingestion and processing rates, which can reduce the time and cost of training and inference for AI models. PCIe Gen4 SSDs can deliver up 6.8GB/s of sequential read performance and over 5.7GB/s of sequential write performance¹, which is much faster than HDDs, which can only deliver less than 300 MB/s of sequential read and write performance³. This means that these high capacity SSDs can process data over 22 times faster than HDDs, which can significantly accelerate the AI workflows and outcomes.
• Higher data density and lower power consumption, which can reduce TCO and improve the efficiency of networked data lakes. SSDs can store 30.72TB of data per drive as dense as 4.8TB per cubic inch, which is nearly five times more than the densest 24TB nearline HDDs today⁴. This means that SSDs can store more data in less space, which can reduce the hardware and infrastructure costs of networked data lakes.

In conclusion, GTC 2024 was a remarkable event that showcased the latest innovations and trends in the AI industry, and how Micron's memory and storage solutions are powering the AI revolution. We are proud to be a partner of NVIDIA, and we look forward to continuing our collaboration and contribution to the AI ecosystem.