feat: Use CDI for GPU injection instead of nvidia-container-cli

[klueska]

Problem Statement

GPU access currently relies on the legacy nvidia-container-runtime +
nvidia-container-cli stack at two layers: once when Docker injects GPUs
into the k3s cluster container, and again when the nvidia-device-plugin +
nvidia-container-runtime inject them into individual sandbox pods.

Proposed Design

Both layers should be migrated to CDI instead. The general idea:

1. Generate a CDI spec on the host before starting the cluster:
   nvidia-ctk cdi generate
2. Use Docker's native CDI support (available since Docker 25) to pass GPUs
   into the k3s container: --device nvidia.com/gpu=all
3. Mount /etc/cdi into the k3s container, enable enable_cdi_devices = true
   in the containerd config, and configure the nvidia-device-plugin to use CDI
   device IDs so containerd handles injection natively

CDI is the canonical way NVIDIA supports GPU access in containerized
environments going forward. Some platforms require CDI and are incompatible
with the legacy runtime stack, so this would also broaden the set of platforms
OpenShell can run on. It also makes what gets injected explicit and
auditable via the CDI spec rather than delegating to a CLI with broad host
access.

/cc @elezar @jgehrcke

Alternatives Considered

None

Agent Investigation

No response

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request

[tyeth]

Thanks for this, looks like it might solve my related issue in #404 and the PR in #411

[elezar]

Thanks @klueska. This is definitely the direction we would want to move this so as to not depend on a specific runtime / k8s implementation. Using CDI as the mechanism for defining what

• As of the NVIDIA Container Toolkit v1.18.0 the CDI specs for available devices SHOULD be generated automatically when installing the NVIDIA Container Toolkit. This should remove the need for explicit CDI spec generation. With that said, we may want to consider a situation where a sandbox is only given access to a subset of the GPUs so as to allow at least minimal isolation between multiple agents. Here the important thing is to separate CDI spec generation from consumption to allow for flexibility when defining new use cases. For example, if we are starting a REMOTE sandbox.

• Although CDI support is enabled in docker as of Docker 25, it was an opt-in feature up to 28.2.0 -- where it was enabled by default. The logic for injection was not changed significantly between these versions.

• The containerd config option that needs to be enabled is the enable_cdi plugin option. The exact config path depends on the containerd version / CRI plugin used. Furthermore, it is not available in v1.6 of container, is disabled by default in v1.7, and enabled by default as of v2.0. The option is also marked as deprecated in the current HEAD.

• The default paths for CDI specs are /etc/cdi and /var/run/cdi, with specifications in /var/run/cditaking precedence. As such we would also have to mount/var/run/cdi` to get access to the specs there. Also note that the device plugin (when CDI is activated) generates CDI specs itself -- assuming a containerized node these would have to be scoped to the container and not the host.

[klueska]

Thanks @elezar for the clarifications.

Regarding your first bullet -- This sounds like a host-level dependency that OpenShell would need to clarify in its README. From my understanding of how OpenShell works, I think the k3s container would still want access to all GPUs, and then each of the agent sandboxes it orchestrates would (potentially) want limited access to a subset of the GPUs (which would be made possible via the device plugin).

Regarding your fourth bullet -- I guess this means we don't actually need to inject these folders at all. Putting the device plugin into CDI mode would generate the specs on the fly so that these folders (and any generated specs) would be created on the fly.

[klueska]

Depending on the version of k8s in use, it might also make sense to use the DRA driver for GPUs instead of the device plugin so that controlled GPU sharing could be introduced more easily in the future. Also, it uses CDI natively, so no need to "configure" it in CDI mode.

[elezar]

Regarding your fourth bullet -- I guess this means we don't actually need to inject these folders at all. Putting the device plugin into CDI mode would generate the specs on the fly so that these folders (and any generated specs) would be created on the fly.

Yes, I think we should separate giving the containerized node access to the driver and GPUs from ensuring that individual containers running on this containerized node (which I assume is each as sandbox).

Depending on the version of k8s in use, it might also make sense to use the DRA driver for GPUs instead of the device plugin so that controlled GPU sharing could be introduced more easily in the future. Also, it uses CDI natively, so no need to "configure" it in CDI mode.

I was also thinking of DRA in this context -- especially considering that the DRA driver doesn't have a dependency on the NVIDIA Container Toolkit. This would mean that it SHOULD be possible to simplify the node image and have the DRA driver generate CDI specs and provide the CDI hook as it does today.