VSIM support filter with post processing#1530
Closed
hailangx wants to merge 345 commits intovectorApiPoC-storeV2from
Closed
VSIM support filter with post processing#1530hailangx wants to merge 345 commits intovectorApiPoC-storeV2from
hailangx wants to merge 345 commits intovectorApiPoC-storeV2from
Conversation
…ostics" This reverts commit 0aa68d1.
… have a batch in the first place
…ope it was in there
…rrent directory is / which does not play nice with this path style; instead base on location of assemblies if initial lookup fails
…SpanBytes" This reverts commit 6d144ac.
This reverts commit 333b4e1.
This reverts commit 9104b92.
…y slowly, but still)
…Allure part of the website is wiped out until next nightly is ran. This fix keeps Allure site untouched when BDN is ran (#1517)
* replace net9 with net10 in project files * Update net9.0 to net10.0 in ci and nightly * Updated BDN to run net10 instead of net90 the website is not done yet. * Updating BDNs to not use .net9.0. Unfortunately, .net10 is not supported in current BDNs. * Updated External Release to release 8.0 and 10.0 only (no 9.0) * fix * Enabled Win10 for BDN since recent fix * Updated a couple BDNs expected values that were a bit out of range * Updated .net9.0 for CodeQL --------- Co-authored-by: Badrish Chandramouli <badrishc@microsoft.com>
* Bugfix in SortedSetObject * Fixed in HashObject as well
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request introduces a comprehensive post-filtering capability for vector search results based on flexible filter expressions evaluated against JSON attributes. The implementation adds a complete expression engine that can tokenize, parse, and evaluate complex filter conditions.
Changes:
- Adds a tokenizer, parser, and evaluator for filter expressions supporting arithmetic, comparison, logical, and containment operators
- Integrates post-filtering into the vector search pipeline in
VectorManager.cs - Adds comprehensive test coverage for both basic and advanced filtering scenarios
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/Garnet.test/RespVectorSetTests.cs | Adds two comprehensive test methods covering basic attribute filtering and advanced filtering with complex expressions |
| libs/server/Resp/Vector/VectorManager.cs | Integrates post-filtering into vector search methods and implements filter evaluation logic |
| libs/server/Resp/Vector/VectorFilterTokenizer.cs | Implements tokenization of filter strings into tokens for parsing |
| libs/server/Resp/Vector/VectorFilterParser.cs | Implements recursive descent parser to build expression trees from tokens |
| libs/server/Resp/Vector/VectorFilterExpression.cs | Defines expression tree node types for representing filter expressions |
| libs/server/Resp/Vector/VectorFilterEvaluator.cs | Implements expression evaluation against JSON attribute data |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (left == null || right == null) return false; | ||
|
|
||
| if (left is double || right is double) | ||
| return Math.Abs(ToNumber(left) - ToNumber(right)) < 0.0001; |
There was a problem hiding this comment.
The hard-coded epsilon value of 0.0001 for floating-point comparison should be extracted as a named constant (e.g., private const double ComparisonEpsilon = 0.0001;) to improve maintainability and make the tolerance value explicit and adjustable.
Member
Author
There was a problem hiding this comment.
@copilot open a new pull request to apply changes based on this feedback
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Contributor
hailangx
requested review from
TalZaccai,
TedHartMS and
badrishc
as code owners
hailangx
changed the base branch from
vectorApiPoC-storeV2
to
vectorApi-storev1
hailangx
changed the base branch from
vectorApi-storev1
to
vectorApiPoC-storeV2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adds support for post-filtering of vector search results using flexible filter expressions evaluated against JSON attributes. The main changes introduce a tokenizer, parser, and evaluator for filter expressions, and integrate post-filtering into the vector search pipeline.
Vector filter expression engine:
VectorFilterTokenizer.cs) that converts filter strings into tokens, supporting numbers, strings, booleans, identifiers, operators, keywords, and delimiters.VectorFilterParser.cs) that builds an expression tree from tokens, supporting arithmetic, comparison, logical, containment (in), and grouping operators.VectorFilterEvaluator.cs) that evaluates parsed expression trees against JSON attribute data, handling type conversions, logical operations, and containment checks.VectorFilterExpression.cs) for literals, member access, unary, and binary operations.Integration into vector search pipeline:
VectorManager.cs, applying the filter expression to each result's attributes and updating the result set accordingly. [1] [2] [3]These changes enable expressive filtering of vector search results based on complex conditions over JSON attributes.