Skip to content

[BUG]: Intermittent CUDA_ERROR_OUT_OF_MEMORY cascade in cuda_core test suite #2381

Description

@rluo8

Is this a duplicate?

Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

When running the full cuda_core v1.1.0 test suite (python -m pytest .) repeatedly across several Python environments on a single GPU, some of the runs intermittently fails with a large, identical cascade of failures.
Most of the failures are about CUDA_ERROR_OUT_OF_MEMORY.

This is similar to internal bug 5815123.

Observations:
The failures are random when tests with Python 3.10/3.11/3.12/3.13/3.14/3.14t.
When testing a single failed case, it could pass. If I added a sleep 2 or 3 seconds hook between the two test cases, mostly it could pass.

Attached the logs for several runs without sleeping.

running_tests_logs.zip

How to Reproduce

pip install cuda-core[cu13]
pip install cuda-bindings==13.3.1

Checkout the source codes for tag cuda-core-v1.1.0
cd cuda_core
pytest -v tests (Run for every python version from 3.10 to 3.14t)

Expected behavior

The tests could all pass.

Operating System

Windows 2025 server

nvidia-smi output

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions