Designing Infrastructure for AI Workloads

AI infrastructure design has diverged sharply from traditional enterprise compute. Training runs push interconnect and memory bandwidth to their limits; inference at scale pushes latency, batching, and power efficiency. Treating "AI" as one workload is the fastest way to buy the wrong system.

Start by profiling the workload

Before any hardware conversation, separate the workload into three classes:

• Training — long-running, all-reduce-heavy, sensitive to interconnect bandwidth and checkpoint throughput.
• Fine-tuning / post-training — smaller than training but repeats often; benefits from elastic GPU pools.
• Inference — latency-bound, throughput-measured, driven by request patterns and model size. KV-cache and batching strategy dominate cost economics.

The public MLPerf Training and Inference benchmarks remain the most rigorous reference for comparing systems under like-for-like conditions^[1].

Compute: beyond "how many GPUs"

For dense transformer training at scale, NVIDIA Hopper (H100 / H200) and Blackwell (B200/GB200) remain the reference designs; AMD Instinct MI300X is a credible alternative on memory-bound workloads thanks to 192 GB HBM3 per accelerator^[2][3]. For inference, the calculus changes: aggregate throughput per watt and per rack matters more than peak FLOPs, and lower-precision formats (FP8, INT4) often halve the footprint without meaningful quality loss.

Interconnect is the hidden bottleneck

Training throughput above 8 GPUs is a networking problem, not a compute problem. Two interconnect domains matter:

• Intra-node — NVLink / NVSwitch on NVIDIA platforms provides multi-hundred-GB/s all-to-all between GPUs in a server or pod^[4].
• Inter-node — InfiniBand NDR (400 Gb/s) or high-speed RoCEv2 Ethernet fabrics for scale-out. Topology (rail-optimised fat-tree or dragonfly), adaptive routing, and congestion control determine whether advertised FLOPs translate into useful training time.

Storage and data pipelines

Checkpointing a frontier-scale model can move terabytes in minutes; a slow filesystem will idle an expensive GPU fleet. Parallel filesystems such as Lustre, IBM Storage Scale (GPFS), and WekaFS are the standard choices, paired with an object tier for datasets^[5]. Two specific design decisions matter:

• Checkpoint throughput sized to write a full optimizer state in a fraction of a training step — not the job runtime.
• Dataloader locality — either prefetch to node-local NVMe or engineer the fabric to serve random reads at line rate.

Orchestration: Kubernetes, Slurm, or both

Kubernetes (typically with Kubeflow, KubeRay, or the NVIDIA GPU Operator) is now the default for inference and for elastic fine-tuning. Slurm remains dominant for long, gang-scheduled training jobs because of its job-array semantics, fair-share accounting, and tight MPI integration. Mature organisations run both, with shared identity and storage.

Power, cooling, and facility

A single 8-GPU H100 node draws 10 kW; a densely packed rack can push 30–70 kW. Conventional perimeter-cooled data halls top out well below this envelope. Direct-to-chip liquid cooling is becoming the default for new AI halls, and rear-door heat exchangers are a practical retrofit path — a shift now reflected in Open Compute Project reference designs^[6]. Facility readiness (power density, liquid loops, redundancy) is frequently the longest-lead-time item in an AI build — worth starting earlier than feels necessary.

An inference-first economic lens

Most enterprises will spend materially more on inference than on training over a model's lifetime. Controlling inference cost means:

• Right-sizing the model — a distilled or quantised model often matches business quality at a fraction of the cost.
• Batching and continuous batching for transformer serving (vLLM, TensorRT-LLM, TGI) can yield 3–10× throughput on the same hardware.
• Tiering — route easy requests to a cheap model, hard ones to a larger model, with explicit quality SLAs on each tier.

Reference architectures, not reinvention

Unless you are building at hyperscaler scale, start from a validated reference: NVIDIA DGX SuperPOD, OEM-specific AI factory blueprints from Dell, HPE, Lenovo, and Supermicro, or the Open Compute Project designs^[6][7]. These compress months of risk into a known-good starting point; customise only where your workload genuinely demands it.

Bottom line

Enterprise AI infrastructure is an end-to-end problem: compute, interconnect, storage, orchestration, power, and cooling have to be co-designed for the actual workload. Get any one of them wrong and the others cannot compensate. The organisations that get this right treat infrastructure as a platform — versioned, observable, and optimised against measurable business outcomes.