Skip to content

two queue filtered search with max effort#929

Draft
hailangx wants to merge 7 commits intomainfrom
haixu/two-queue-filtered-search
Draft

two queue filtered search with max effort#929
hailangx wants to merge 7 commits intomainfrom
haixu/two-queue-filtered-search

Conversation

@hailangx
Copy link
Copy Markdown
Member

@hailangx hailangx commented Apr 8, 2026

Summary

  • Introduces a new two-queue search algorithm for graph-based vector search that decouples exploration from
    filter evaluation, improving recall for low-selectivity filtered queries
  • Adds a callback-based filter (GarnetFilterProvider) to diskann-garnet alongside the existing bitmap filter, enabling
    per-candidate FFI filter evaluation from Garnet/C#

Motivation

The existing beta-filtered search works well when filters are moderately selective, but struggles with low-selectivity
filters where most candidates are rejected. In those cases, the search converges prematurely because pruning is based
on distance to filtered results that haven't been found yet. The two-queue approach keeps exploration broad until
enough filtered results are accumulated.

Design

Two-Queue Search (two_queue_search.rs)

  • Maintains two separate queues: a min-heap (candidates) for exploration ordered by distance, and a max-heap
    (filtered_results) for filter-passing neighbors
  • All graph neighbors are explored regardless of filter status; only filter-passing nodes are added to filtered_results
  • Convergence: terminates when filtered_results has enough results and the closest unexplored candidate is farther than
    the worst filtered result
  • Supports a max_candidates hop limit as a safety cap and result_size_factor (default 10) to control result queue
    capacity (k * result_size_factor)
  • Reports termination reason via TwoQueueTermination enum (Exhausted, MaxCandidates, Converged, FilterTerminated)

Benchmark

  • Todo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant