Skip to content

[BUG] Iterative CAGRA graph build fails while benchmarking with override_kv=dataset_memory_type:"mmap" #1637

@irina-resh-nvda

Description

@irina-resh-nvda

Iterative CAGRA graph build fails with an exception
CUDA call='cudaFreeAsync(ptr, stream)' at file=/workspace/repos/cuvs/cpp/src/neighbors/detail/cagra/compute_distance.hpp line=248 failed with misaligned address

override_kv=dataset_memory_type:"device" and override_kv=dataset_memory_type:"host" work

** To reproduce **
Run

CUVS_CAGRA_ANN_BENCH --build --force --data_prefix=/workspace/datasets/ --benchmark_out_format=csv --benchmark_out=res_build.csv --benchmark_counters_tabular=true --override_kv=dataset_memory_type:"mmap" /workspace/repos/vdb_bench/laion_1M_cagra_iterative.json

** laion_1M_cagra_iterative.json **

{
  "dataset": {
    "name": "laion_1M",
    "base_file": "laion_1M/base.1M.fbin",
    "subset_size": 1000000,
    "query_file": "laion_1M/queries.fbin",
    "groundtruth_neighbors_file": "laion_1M/groundtruth.1M.neighbors.ibin",
    "distance": "euclidean"
  },
  "search_basic_param": {
    "batch_size": 10000,
    "k": 10
  },
  "index": [
  
    {
      "name": "cuvs_cagra_iterative",
      "algo": "cuvs_cagra",
      "build_param": { 
        "graph_degree": 64,
        "intermediate_graph_degree": 128,
        "search_width": 1
      },
      "file": "laion_1M/cagra/q_coarse.ibin",
      "search_params": [
        {"itopk": 64, "search_width": 2, "max_iterations": 8},
        {"itopk": 128, "search_width": 1, "max_iterations": 12},
        {"itopk": 128, "search_width": 1, "max_iterations": 16},
        {"itopk": 256, "search_width": 1, "max_iterations": 16},
        {"itopk": 512, "search_width": 2, "max_iterations": 10},
        {"itopk": 256, "search_width": 2, "max_iterations": 12},
        {"itopk": 32, "search_width": 1, "max_iterations": 32},
        {"itopk": 32, "search_width": 1, "max_iterations": 64},
        {"itopk": 192, "search_width": 4, "max_iterations": 12},
        {"itopk": 256, "search_width": 4, "max_iterations": 12},
        {"itopk": 256, "search_width": 16, "max_iterations": 10},
        {"itopk": 512, "search_width": 16, "max_iterations": 32}
      ]
    }
  ]
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    Status

    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions