Milvus 2.6.6 with Custom Knowhere Build

[kumaramit01]

Closes #

💸 TL;DR

This PR introduces build against custom knowhere that has performance optimizations for sparse inverted index search, focusing on SIMD acceleration and filter-aware algorithms. The changes deliver speedups for sparse vector search operations, particularly when filters are applied.

• Automatic optimization: No API changes required - works transparently

The PR builds against https://github.com/reddit/knowhere/commits/sparse-simd-optimization/

📜 Details

Seek Operation Enhancements

• Aligned allocator: Memory alignment for better SIMD performance
• Galloping search: Fast seek operations for sparse data structures, instead of using a while loop with a next to seek.

Filter-Aware Block-Max WAND

• Block-level maximum tracking: Pre-computed max scores per 128-document block
• Tighter upper bounds: Position-aware score bounds for better pruning
• Memory efficient: Block max scores rebuilt at load time, not serialized

SIMD Optimizations for Sparse Search [Only relevant for Term at a Time algorithms]

• Multi-architecture SIMD support: AVX2, AVX-512, NEON, and reference implementations
• Critical path acceleration: Optimized max_score_from_block() and compute_all_distances() functions
• BM25 support: SIMD-accelerated BM25 scoring with proper average document length handling
• Runtime CPU detection: Automatic dispatch to best available SIMD implementation

Jira

🧪 Testing Steps / Validation

✅ Checks

• CI tests (if present) are passing
• Adheres to code style for repo
• Contributor License Agreement (CLA) completed if not a Reddit employee