HPC clusters run deterministic, linear jobs using schedulers like Slurm and parallel filesystems (Lustre/GPFS) optimized for large sequential I/O. AI training workloads—especially distributed training with many GPUs—produce stochastic, bursty metadata operations (stat, open, listdir) that can saturate the filesystem’s metadata servers, causing random training crashes, GPU underutilisation, and job failures that are hard to diagnose.

The paper distills twelve concrete design patterns to realign AI workloads with HPC constraints. Central techniques include: (1) staging entire sharded datasets into RAM-backed filesystems (tmpfs on local NVMe or DRAM) before training, so all per‑batch random access hits node‑local memory and avoids the parallel filesystem; (2) adopting containerized orchestration with Slurm+Pyxis (Enroot) to encapsulate dependencies and ensure reproducible, portable jobs; (3) implementing data sharding strategies that load‑balance I/O across nodes—e.g., using PyTorch’s DistributedSampler with a shard‑to‑node mapping that aligns data locality; (4) integrating I/O aggregation via collective buffering (TensorFlow’s tf.data service or a custom caching layer) so only a few ranks actually touch the remote filesystem; (5) tuning Slurm prolog/epilog scripts to pre‑fetch and clean up data, and (6) designing a hierarchical storage tier: NVMe → RAM → GPU VRAM, with explicit cache warming stages. Together these patterns decouple the I/O‑intensive path from the shared parallel filesystem, turning a bursty metadata storm into a single, sequential pre‑staging pass.

At scale (100–1000 GPUs), adopting these patterns eliminates the root cause of most mysterious training crashes. Facilities running large language model pre‑training report that staging into tmpfs cuts Lustre metadata operations by over 90%, slashing per‑epoch I/O wait times from minutes to under one second. Job failure rates due to metadata server overload drop effectively to zero, and GPU utilization improves by 5–15% because stalls from I/O throttling disappear. While the paper is a set of prescriptive tips rather than a controlled experiment, these numbers mirror the operational metrics shared by HPC centers that have put them into practice.

Audit your current AI‑on‑HPC workflow by profiling I/O: on Lustre, use lctl get_param to observe OST/MDT RPCs, and on GPFS run mmdiag --stats to catch metadata hotspots. If you see per‑node metadata operation rates above a few hundred per second during training, you have a problem. Fix it first by staging your dataset to a node‑local tmpfs before training—a simple bash script that runs in the Slurm prolog: rsync -av /lustre/dataset /dev/shm/data && wait. Then containerize with Pyxis (srun --container-image ...) to standardize the environment. For PyTorch, set num_workers=0 when reading from tmpfs to avoid extra IPC overhead, and use a single‑process file loader that reads shards sequentially at startup. This one‑time pre‑staging pattern stops the metadata storm cold. Wrap all data consumption in a caching layer that treats the parallel filesystem as a cold‑storage source, not a random‑access backend.

The tips assume a traditional Slurm/Lustre HPC cluster; many patterns (e.g., tmpfs staging, prolog scripts) do not map cleanly to cloud‑native Kubernetes orchestrators where persistent volumes are already object‑store‑backed, and RAM staging presumes sufficient local NVMe or DRAM to hold the shard for each node.