Hardware Design for LLMs — NVIDIA Chief Scientist Bill Dally

NVIDIA Chief Scientist Bill Dally presented hardware innovations driving LLM performance.

Memory Bandwidth

Memory bandwidth is the primary bottleneck for LLM inference. HBM technology has scaled from 1TB/s (A100) to 4TB/s (B200). HBM4 targets 8TB/s by 2027.

Sparsity

2:4 structured sparsity doubles throughput on Ampere+ architectures by skipping zero-weight computations. Fine-grained sparsity requires specialized hardware.

Precision

FP16 -> FP8 reduces memory and compute by 2x with minimal accuracy loss. FP4 inference is emerging for latency-sensitive applications.

Future Trends

On-chip SRAM for KV-cache reduces HBM traffic. Optical interconnects enable higher GPU-to-GPU bandwidth. Domain-specific accelerators for attention and softmax.