Protocol

NCCL

NVIDIA Collective Communications Library

Lab ready Full lab

NVIDIA Collective Communications Library implementing allreduce, broadcast, and related GPU collectives over NVLink, PCIe, and RDMA. Topology detection and ring/tree algorithms dominate training performance at scale. It is the default collective layer for many PyTorch/TensorFlow multi-GPU jobs.

Domains

ai

Layers

application

Tags

collectivesgpuallreducetraining

Related protocols

RCCL

AMD ROCm Collective Communications Library — the ROCm counterpart to NCCL for GPU collectives. It uses Infinity Fabric, PCIe, and RDMA pa…

RDMA Verbs

Kernel-bypass networking model: queue pairs, memory regions, and work requests move data between registered buffers with minimal CPU. Com…

NVLink

NVIDIA high-bandwidth GPU interconnect for peer-to-peer and coherent-ish memory traffic inside a node or NVSwitch domain. Generations dif…

MPI

Message Passing Interface — portable parallel messaging with point-to-point and collective operations across ranks. Implementations map t…