docs: add NCCL troubleshooting notes for multi-GPU training

[binzily]

Mirrors the documentation changes from #5195 onto develop.

Description

Adds NCCL troubleshooting notes to the multi-GPU docs and links them from the
general troubleshooting page.

Refs #4011
Refs #2756

Type of change

• Documentation update

[greptile-apps]

Greptile Summary

This PR adds an NCCL troubleshooting subsection to docs/source/features/multi_gpu.rst (with a Sphinx anchor label multi-gpu-nccl-troubleshooting) and a short forwarding entry in docs/source/refs/troubleshooting.rst that cross-references it. The documentation-only change is well-structured and the RST formatting is consistent with the rest of both files.

Confidence Score: 5/5

Documentation-only PR with no code changes; safe to merge.

All changes are RST documentation. The Sphinx anchor label and cross-reference are correctly paired, heading underlines meet RST length requirements, the NCCL environment variables are valid, and the new sections are stylistically consistent with the rest of both files. No P0/P1 findings.

No files require special attention.

Vulnerabilities

No security concerns identified.

Important Files Changed

Filename	Overview
docs/source/features/multi_gpu.rst	Adds a properly anchored "Troubleshooting NCCL Errors" subsection (^-level) with three env-var workarounds and a performance-impact note; RST syntax and label placement are correct.
docs/source/refs/troubleshooting.rst	Adds a top-level (-level) section that briefly describes the NCCL error and defers to the multi_gpu.rst anchor via :ref:multi-gpu-nccl-troubleshooting; heading underline length matches exactly (48 chars each), cross-reference resolves correctly.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[User hits NCCL error\nduring multi-GPU training] --> B{Error type}
    B -->|CUDA illegal memory access\nfrom ProcessGroupNCCL| C[Set NCCL_SHM_DISABLE=1]
    C --> D{Resolved?}
    D -->|No| E[Set NCCL_IB_DISABLE=1\nand NCCL_ALGO=Ring]
    E --> F[Relaunch distributed training]
    D -->|Yes| F
    B -->|Other| G[See general troubleshooting.rst]

Loading

_{Reviews (1): Last reviewed commit: "docs: add NCCL troubleshooting notes for..." | Re-trigger Greptile}

[Copilot]

Pull request overview

Adds documentation guidance for diagnosing and mitigating NCCL-related failures seen during Linux multi-GPU distributed training, and links this guidance from the general troubleshooting page.

Changes:

• Add a new troubleshooting entry in the general troubleshooting guide that points users to NCCL-specific guidance.
• Add an NCCL troubleshooting section to the multi-GPU training docs documenting environment-variable workarounds (NCCL_SHM_DISABLE, NCCL_IB_DISABLE, NCCL_ALGO).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
docs/source/refs/troubleshooting.rst	Adds a troubleshooting section and links to the multi-GPU NCCL troubleshooting anchor.
docs/source/features/multi_gpu.rst	Introduces a new NCCL troubleshooting subsection with suggested environment-variable mitigations.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[isaaclab-review-bot]

Docs-only PR — LGTM. The NCCL env vars are correct, the escalation order (shared memory → IB → ring algorithm) is sensible, RST formatting is clean, and the cross-reference from troubleshooting.rst resolves properly.

A couple of optional suggestions for future improvement (not blocking):

1. NCCL_P2P_DISABLE=1 — worth mentioning as another fallback. Peer-to-peer transport failures can produce the same illegal memory access symptom, especially on multi-GPU desktops where P2P isn't well supported across PCIe bridges.

2. NCCL_DEBUG=INFO — a brief mention that users can set this to identify which transport is failing would help them narrow down which NCCL_*_DISABLE variable they actually need, rather than blindly stacking all of them.

3. The .. note:: block could be slightly more specific about the trade-off — e.g., "may reduce inter-GPU communication throughput by falling back to slower transports" rather than the current generic "may change communication behavior or performance".

None of these are blockers. Good addition — the referenced issues (#4011, #2756) show this trips up real users.

[AntoineRichard]

Thanks @binzily !