Multi-GPU Distributed Training

Set up PyTorch DDP or DeepSpeed on a multi-GPU instance for distributed model training.

Train large models faster by distributing work across multiple GPUs on a single instance.

1. Deploy a multi-GPU instance

runcrate instances create --name multi-gpu --gpu A100 --gpu-count 4 --template ubuntu-train
runcrate instances status multi-gpu

2. Verify GPU topology

runcrate ssh multi-gpu -- "nvidia-smi"
runcrate ssh multi-gpu -- "python -c 'import torch; print(f\"GPUs: {torch.cuda.device_count()}\")'"

3. PyTorch DDP training

Upload and run your training script with torchrun:

runcrate cp ./train_ddp.py multi-gpu:/workspace/

runcrate ssh multi-gpu -- "cd /workspace && torchrun \
  --nproc_per_node=4 \
  --master_port=29500 \
  train_ddp.py \
  --batch-size 32 \
  --epochs 10 \
  --lr 1e-4"

4. DeepSpeed ZeRO (for larger models)

runcrate ssh multi-gpu -- "pip install deepspeed"

runcrate cp ./ds_config.json multi-gpu:/workspace/
runcrate cp ./train_deepspeed.py multi-gpu:/workspace/

runcrate ssh multi-gpu -- "cd /workspace && deepspeed \
  --num_gpus=4 \
  train_deepspeed.py \
  --deepspeed_config ds_config.json"

5. Monitor training

runcrate ssh multi-gpu -- nvidia-smi
runcrate ssh multi-gpu -- "nvidia-smi topo -m"
runcrate ssh multi-gpu -- "tail -30 /workspace/output/training.log"

6. Download results

runcrate cp multi-gpu:/workspace/output/ ./training-output/

Tips

• DDP scales linearly with GPU count for data-parallel workloads. 4 GPUs = ~3.8x throughput.
• DeepSpeed ZeRO-3 shards model parameters, gradients, and optimizer states across GPUs — use it when the model does not fit on a single GPU.
• Use --gradient_accumulation_steps to simulate larger batch sizes without increasing per-GPU memory.

Cleanup

runcrate instances delete multi-gpu