AI Infrastructure and GPU Computing

AI, ML and deep learning

The terms nest. Artificial intelligence is the broad goal of machines doing tasks that need intelligence. Machine learning is the subset that learns patterns from data. Deep learning is the subset of ML using many-layered neural networks. Generative AI uses deep models, mostly transformers, to produce text, images or code.

Two phases drive infrastructure needs. Training adjusts billions of parameters over huge datasets and needs many GPUs working in lockstep. Inference serves the trained model and cares about latency, throughput and cost per request. Growth in data, model size and available compute is what pushed purpose-built AI data centers into existence.

Layers of an AI data center

It helps to think in layers, from metal up to applications.

1. Physical: servers with several GPUs each, power, cooling and racks. Efficiency is often tracked with PUE (total facility energy divided by IT equipment energy; closer to 1.0 is better).
2. Data movement and I/O: NVLink inside a node, InfiniBand or Ethernet between nodes, and technologies such as RDMA that bypass the CPU.
3. OS, drivers and virtualization: a Linux OS, the GPU driver, and partitioning features.
4. Core libraries: CUDA, cuDNN and NCCL.
5. Monitoring and management: telemetry, health checks and cluster tooling.
6. Applications and frameworks: PyTorch, TensorFlow, inference servers, MLOps and job scheduling.

Compute, network and storage all scale together. A fast GPU starved by slow storage or a congested network wastes expensive hardware.

GPUs and why they fit AI

A CPU has a few powerful cores tuned for sequential work. A GPU has thousands of simpler cores that run the same operation on many data elements at once, which suits the matrix multiplications at the heart of neural networks. Modern data-center GPUs also include specialised tensor cores for low-precision matrix math, and high-bandwidth memory (HBM) so the cores are fed quickly.

Servers are commonly built with eight GPUs linked by a fast fabric, and multiple servers are joined into larger clusters. Each generation increases memory capacity, bandwidth and supported precisions, so capacity planning focuses on GPU memory (will the model fit?) and bandwidth (will it run fast?), not just core counts.

A DPU (data processing unit) is a smart network card that offloads networking, storage and security work from the host CPU, leaving more of it for the workload.

Networking and interconnects

Technology	Role
PCIe	General host-to-device link; often the bottleneck for GPU-to-GPU traffic
NVLink / NVSwitch	High-bandwidth GPU-to-GPU links within a node
InfiniBand	Low-latency, lossless fabric between nodes
Ethernet (RoCE)	Ethernet carrying RDMA, tuned to be lossless

RDMA lets one machine read or write another machine's memory without involving the remote CPU or extra memory copies, cutting latency. GPUDirect RDMA extends this so a network adapter reads GPU memory directly, and GPUDirect Storage lets storage DMA straight into GPU memory, skipping a bounce through system RAM.

InfiniBand is purpose-built for HPC and AI and gives predictable latency; Ethernet is ubiquitous and familiar, and with congestion control and lossless configuration it is increasingly used for AI clusters. Training traffic is bursty and synchronised, so oversubscribed networks hurt.

CUDA and the software stack

CUDA is NVIDIA's parallel computing platform: a programming model, compiler and runtime for writing code that runs on the GPU. The core idea is a kernel, a function launched across a grid of thread blocks, each made up of many threads. Most practitioners use CUDA indirectly through libraries (cuBLAS, cuDNN) and frameworks that call them.

Check your setup with:

nvidia-smi          # driver version, GPUs, memory, utilisation
nvcc --version      # CUDA toolkit compiler version

Keep the driver, CUDA toolkit and framework builds compatible. A driver generally supports toolkits up to a given version, so mismatches are the most common cause of "CUDA not available" errors. In containers, use the NVIDIA Container Toolkit so the host driver is exposed to the container.

NCCL collectives

The NVIDIA Collective Communications Library (NCCL) implements multi-GPU and multi-node communication, picking efficient algorithms such as ring or tree over NVLink and the network. Distributed training frameworks call it for you. The key operations:

Collective	What happens
Broadcast	One rank sends the same data to all ranks
Scatter	One rank splits data and sends a piece to each rank
Gather	All ranks send data to one rank
AllGather	Every rank ends up with everyone's data concatenated
Reduce / AllReduce	Combine values (sum, max) and deliver the result to one rank or to all
ReduceScatter	Combine values, then each rank keeps one slice of the result
AlltoAll	Every rank sends distinct data to every other rank

AllReduce is the workhorse of data-parallel training: each GPU computes gradients on its own batch, and AllReduce averages them so all replicas stay identical. AllGather and ReduceScatter underpin sharded approaches, and AlltoAll appears in mixture-of-experts models.

GPU sharing and virtualization

A whole GPU is often too large for small jobs. Two approaches share it:

• vGPU: software-based virtualization that time-slices a GPU between virtual machines. It is flexible but provides weaker isolation of performance.
• MIG (Multi-Instance GPU): on supported GPUs, hardware partitioning into isolated instances, each with dedicated compute and memory slices. Profiles are named by slices and memory, for example 1g.5gb.

sudo nvidia-smi -i 0 -mig 1        # enable MIG mode on GPU 0
nvidia-smi mig -lgip               # list available instance profiles
sudo nvidia-smi mig -cgi 1g.5gb -C # create an instance (profile names vary by GPU)

Use MIG for predictable multi-tenant inference and vGPU where VM-level sharing is needed.

Operations and monitoring

• nvidia-smi is the first stop: utilisation, memory use, temperature, power and running processes.
• DCGM (Data Center GPU Manager) provides fleet-grade telemetry, health checks and diagnostics, and exports metrics for Prometheus-style monitoring.
• Cluster managers and schedulers such as Slurm or Kubernetes (with a GPU device plugin) allocate GPUs to jobs. Slurm suits batch training queues; Kubernetes suits long-running services and containerised pipelines.

What to watch

• GPU utilisation and memory use (low utilisation usually means a data-loading or network bottleneck).
• Temperature, power draw and throttling.
• ECC and Xid errors, which point to failing hardware.
• Network throughput and retransmits during collectives.