Protocol

NCCL

NVIDIA Collective Communications Library

Lab ready Full lab

NVIDIA Collective Communications Library implementing allreduce, broadcast, and related GPU collectives over NVLink, PCIe, and RDMA. Topology detection and ring/tree algorithms dominate training performance at scale. It is the default collective layer for many PyTorch/TensorFlow multi-GPU jobs.

Domains

Tags

collectivesgpuallreducetraining

Sources

  • NVIDIA NCCL documentation

Related protocols

RCCL

Lab ready

AMD ROCm Collective Communications Library — the ROCm counterpart to NCCL for GPU collectives. It uses Infinity Fabric, PCIe, and RDMA pa…

ai

RDMA Verbs

Lab ready

Kernel-bypass networking model: queue pairs, memory regions, and work requests move data between registered buffers with minimal CPU. Com…

aistorage

NVLink

Lab ready

NVIDIA high-bandwidth GPU interconnect for peer-to-peer and coherent-ish memory traffic inside a node or NVSwitch domain. Generations dif…

ai

MPI

Lab ready

Message Passing Interface — portable parallel messaging with point-to-point and collective operations across ranks. Implementations map t…

ai