Vector search¶

v3 — see SRS overview → Scope. Full-text search (SearchIndex, SQLite + FTS5) and NotesSearchIndex already give PriorisMCP literal/lexical search across fetched documents and, since NotesBackend shipped, cross-document notes search. Both are showing the limits inherent to keyword matching: a query has to share vocabulary with the target text to find it, which is exactly the gap Architecture → Anchoring already named as the reason a future VectorSearchBackend was left as future work — "the better long-term answer to 'find notes relevant to this passage' is semantic, not positional." This chapter is that design.

EmbeddingBackend is a pluggable interface, injected into VectorSearchBackend¶

Rather than committing to local-only or API-only embedding generation, embedding computation is its own interface — mirroring ParserBackend's pattern of one interface with swappable implementations selected via configuration. It is injected as a dependency into VectorSearchBackend, not implemented inside it: "which model turns text into vectors" and "where vectors are stored/searched" are independent axes, and coupling them would force a change to one every time the other changes (e.g. upgrading the embedding model would otherwise mean touching the storage engine's own code, and vice versa). Concretely, VectorSearchBackend.index_entries takes raw text (mirroring SearchIndex.index_entries's existing {"key", "start", "length", "text"} entry shape) and calls an injected EmbeddingBackend.embed(text) internally before writing vectors.

embed's signature is settled as async def embed(text: str) -> list[float] — a single text in, one embedding vector out, no batching at the interface level (index_entries calls it once per chunk in its own loop). Async regardless of whether a given implementation actually needs it: fastembed's local ONNX inference doesn't strictly require it, but an API-backed implementation (OpenAI, Cohere, Voyage AI) makes a real network call with real latency, and the interface shouldn't differ per implementation. This is deliberately not paired with ctx/report_progress, the FastMCP mechanism for a single call a caller is actively blocked on and waiting for. embed/index_entries are not that — "Embedding computation is asynchronous, including per-document," below, already settled that embedding runs as a detached background task the triggering tool call doesn't wait on at all, so there's no live, connected caller for ctx.report_progress to notify partway through. The not_built/building/ready index_status polling already settled below is this design's whole answer to "let the caller know something is happening" — coarser-grained than per-chunk progress, but matched to a model where the caller was never waiting synchronously in the first place.

Corpus topology: two corpus-wide instances, not one unified index and not per-document¶

The concrete use case the architecture doc already names — "similarity search between a note's text and the chunk currently being read finds relevant notes" — is a directional query (a chunk's embedding used to search the notes corpus), not a merged ranked list spanning both document chunks and notes. That removes the main argument for a single unified, type-tagged table.

Instead: one VectorSearchBackend instance covering all document chunks across every fetched document, and a second covering all notes across every document — mirroring today's search.sqlite3 and notes-search.sqlite3, which are each already single, corpus-wide files rather than one per document. See ADR-00020: Corpus topology — two corpus-wide VectorSearchBackend instances for why, including the rejected unified-table and per-document alternatives. Cross-type queries (chunk → relevant notes) are handled by querying the notes-corpus instance with a chunk-derived embedding, optionally filtered by provider/identifier/format — filtering, not separate storage, is how a caller narrows to "notes on this specific document."

"Corpus-wide" here implicitly means within one provider grouping (ResearchPublicationProvider, the only one that exists) — see Architecture → Provider groupings for the constructor-level seam a second grouping would attach through.

Engine: sqlite-vec (asg017)¶

sqlite-vec (asg017), chosen over DuckDB+vss, LanceDB, ChromaDB, Qdrant Edge, and sqlite-vector — see ADR-00021: Vector search engine: sqlite-vec (asg017) for the full comparison. It extends the same sqlite3 stdlib connection pattern StorageBackend, SqliteFts5SearchIndex, and SqliteFts5NotesSearchIndex already use (conn.enable_load_extension() + sqlite_vec.load(conn), same to_thread.run_sync wrapper idiom). Its vec0 virtual table does brute-force, exact KNN rather than true ANN (no HNSW/IVF-PQ).

Storage layout: one file per corpus, separate from every other backend's files¶

VectorSearchBackend instances persist to their own files — e.g. vectors.sqlite3 (documents) and notes-vectors.sqlite3 (notes) — separate from catalogue.sqlite/manifest.sqlite (StorageBackend) and from search.sqlite3/notes-search.sqlite3 (SearchIndex/NotesSearchIndex), even though all of them are, today, "SQLite." This mirrors the existing precedent that SearchIndex already sits apart from StorageBackend specifically so an S3-backed StorageBackend wouldn't need to touch search infrastructure at all — "same engine" has never implied "same file or same swappability boundary" in this codebase. The same reasoning extends to VectorSearchBackend: a future hosted vector-DB swap should not require touching catalogue.sqlite, manifest.sqlite, search.sqlite3, or vice versa.

Interfaces stay separate even if a future engine could serve more than one¶

SearchIndex (FTS5) and VectorSearchBackend remain two distinct Python interfaces regardless of which concrete engine(s) back them — see ADR-00022 for why.

Chunking granularity: heading-bounded, split only when oversized¶

VectorSearchBackend reuses SearchIndex's existing heading-bounded chunks as its unit of record, rather than a separate fixed-length-window chunking strategy computed independently. Heading-bounded chunks already respect the document's own semantic structure — a section is "about" one thing — which a fixed window (e.g. 400 tokens with overlap) doesn't: it cuts through sentences and section boundaries wherever the counter happens to land, regardless of whether that bisects a claim from its evidence. Where a heading-bounded chunk exceeds the embedding model's usable input length, VectorSearchBackend sub-splits it internally into multiple embedded windows, but tags every sub-split with the parent chunk's chunk_id — so externally, search results still resolve to the same chunk identity SearchIndex.search uses, and mode="hybrid" fusion doesn't have to reconcile two independent identity systems for the same document. The split is an overflow valve, not the primary chunking strategy.

A corollary: a chunked vector index returning several hits that all cover one semantic idea, because that idea happened to land across more than one chunk (or sub-split), is expected of chunked retrieval generally — not a defect of this splitting approach specifically. Because every sub-split carries its parent chunk's chunk_id, VectorSearchBackend.search can collapse multiple hits from the same parent into one result (e.g. keep the max-scoring sub-split) before returning, which a fixed-window scheme without parent linkage couldn't do as cleanly.

Chunk identity: a UUID minted per parse pass, not derived from heading text or position¶

Each detected chunk gets chunk_id = str(uuid.uuid4()), generated fresh during chunk detection. See ADR-00026: Chunk identity is a UUID minted per parse pass, not derived from heading text or position for why, and why chunk_id never needs to stay stable across parse passes — only within one.

SearchIndex.index_entries's existing implementation already deletes every prior entry for a document by document identity (DELETE FROM search WHERE provider = ? AND identifier = ? AND format = ?) before inserting the new set — never by matching individual entry keys across passes. VectorSearchBackend's document-replace operations must follow the identical delete-by-document-identity-then-insert pattern.

Known rough edge, not actioned now: chunk detection trusts the parser's ATX headings at face value. Low-quality parser output — most plausibly a PDF running header/footer (e.g. "Journal of Foo, Vol. 3") repeated on every page and misdetected as its own heading on each occurrence — can produce a pile of spurious, near-identical-looking chunks, each with its own chunk_id. This is a parser/chunk-detection signal-quality problem, not an identity collision (UUIDs make collision moot regardless), and isn't specific to vector search — it would affect SearchIndex's existing FTS5 chunking today too. Recorded here so it isn't lost; no fix planned as part of this chapter.

Default EmbeddingBackend: fastembed, model configurable via environment variable¶

The default EmbeddingBackend implementation is fastembed (Qdrant's library, MIT-licensed, ONNX-based, no PyTorch dependency), not sentence-transformers. The default model is BAAI/bge-small-en-v1.5 (384 dimensions, English, fastembed's own default model). See ADR-00023: Default EmbeddingBackend is fastembed, not sentence-transformers for why, including why 1536 dimensions was rejected as a design target.

The model name is exposed as PRIORIS_MCP_EMBEDDING_MODEL, left as an open string rather than OneOf-validated against a maintained allowlist — fastembed already errors on an unrecognized model name, so there's no need for this project to duplicate that validation. A user needing multilingual support sets it to something like intfloat/multilingual-e5-large (1024 dimensions) without a new EmbeddingBackend implementation: which model computes embeddings and which class implements the interface are different axes, the same distinction already drawn for chunking-vs-storage.

Index status is per-document/note, derived by comparing recorded vs. configured model¶

Each document's/note's vector record stores which model produced it (embedded_model). Status is derived per document/note, not per corpus, by comparing that recorded value against the currently configured PRIORIS_MCP_EMBEDDING_MODEL:

• not_built — no vector record exists yet for this document/note, including one invalidated by a model change (see below).
• stale — reserved in the IndexStatus type but not reachable through the vector mechanism in practice, for the same reason fts never reaches it either (see the last paragraph below) — a record naming a different model than the one configured is deleted, not merely flagged, the moment anything connects under the new model (see A model-mismatch drop invalidates status rows too, not just the vector table below).
• ready — a record exists and its embedded_model matches the configured one.

This resolves a question that looked separate at first: corpus-wide reindexing isn't a structurally different operation from per-document embedding, just the same per-document trigger invoked in bulk over every currently not_built document after a model change. There is no partial reindex for a model change specifically: vec0 fixes one dimension per table, so the moment the configured model changes, every existing row in that table is the wrong shape — "partial" can only mean per-corpus (documents vs. notes, since they're separate instances) or an implementation detail of how the bulk rebuild executes (batched/resumable), not a subset of rows that remains valid. Document-scoped partial reindexing already exists for a different trigger: it's what already happens whenever a single document is re-parsed or a note is updated.

building is deliberately never persisted. It's derived transiently from whether a live in-process task exists for that key. A process restart mid-embedding leaves the record honestly at not_built — indistinguishable, on the next check, from "not started yet." This is self-healing by construction and needs no separate crash-recovery logic.

index_status values are typed as the IndexStatus Literal["not_built", "stale", "ready", "building"] (src/prioris_mcp/vector/backend.py), not a free-form string — both SearchFetchedResult (documents) and NotesSearchResult (notes) declare index_status as dict[str, IndexStatus], so ty type-check catches an assignment outside that four-value set. Neither fts nor (per the above) vector actually reports stale in practice today, for two unrelated reasons — fts has no "model" to go stale against in the first place, while vector has one but never lets a mismatched record survive long enough to be read as stale — but this remains a documented, code-level invariant, not a schema-level one: both mechanisms share the same IndexStatus type, so nothing stops a future code path (or a future engine that keeps a mismatched record around, e.g. the generation-token design considered and rejected below) from assigning stale and having it type-check cleanly.

A model-mismatch drop invalidates status rows too, not just the vector table¶

A follow-up branch review found that _connect()'s model-mismatch drop only ever touched the vec0 table itself, leaving the corresponding *_vectors_status table's rows untouched. That silently reintroduced staleness as a correctness bug, not merely a cosmetic one: a rollback (model-a → model-b → model-a, e.g. an operator reverting PRIORIS_MCP_EMBEDDING_MODEL after testing an alternative) left model-a's original status rows sitting on disk, undisturbed through the entire model-b generation. Reconciliation's indexed_under("model-a") then genuinely believed those items were already ready under model-a — despite the model-a-configured vec0 table having been dropped twice by that point and holding zero rows for them — so it never rescheduled them, leaving vector search silently and permanently empty for exactly those items until something else happened to re-index them.

The fix: _connect() now deletes every row of the corresponding *_vectors_status table in the same step it drops the mismatched vec0 table, for both documents and notes. A status row can therefore never outlive the generation of the table it was written against, which closes the rollback gap directly — there is no longer a status row left over from a prior generation for a rollback to misread as current.

Trade-off, deliberately accepted: this makes stale unreachable in practice for vector too (see Index status is per-document/note above) — a caller sees not_built immediately after any model change, dimension-changing or same-dimension rename alike, rather than a stale value in the window before reconciliation catches up. A generation-token design (tagging each status row with the vec0 generation it was written against, so status()/indexed_under() could tell a genuinely-stale-but-real record apart from an orphaned one without deleting it) was considered and rejected for now: it would preserve stale as an observable signal, but at the cost of a schema migration on both meta/status tables, transactional-write-ordering guarantees across the migration, and new generation-aware branches in both backends' read paths — engineering cost judged disproportionate to what stale actually buys a caller over not_built today, since both values mean the same actionable thing ("this needs (re-)embedding") and nothing in this codebase branches on the distinction. It remains available as a later upgrade if that judgement changes.

Reconciliation runs automatically at server startup¶

A model change (PRIORIS_MCP_EMBEDDING_MODEL) making existing records stale is only half the story above — something still has to actually re-embed them. That "same per-document trigger invoked in bulk" is PriorisMCP.reconcile_vector_index(): it enumerates every persisted document/note not yet ready under the currently configured model and schedules each through the same EmbeddingScheduler a fresh parse or note update already uses. It runs automatically, once, at server startup — wired into FastMCP's lifespan — not as a separate tool a caller has to remember to invoke. See ADR-00030: Corpus-wide vector-index reconciliation runs automatically at server startup, not as a tool for the full design, including why only the destructive model-mismatch detection/swap (_force_vector_reconnect()) blocks server readiness while corpus enumeration and re-embed scheduling run as a background task.

Reconciliation's own live progress — how many documents/notes a given run started with and how many remain — is exposed via the research://vector-index/rebuild-status resource (see Tools → Vector index reconciliation), process-local and in-memory like building status itself, not a durable job log.

Embedding computation is asynchronous, including per-document¶

Both triggers — per-document embedding (after parse_full_text/note create-or-update) and bulk reindexing — run asynchronously, not only the bulk case. Even local ONNX inference (the fastembed default) is materially slower than FTS5 indexing, and blocking parse_full_text/note-update on it was rejected as the wrong tradeoff for a call whose job is ingesting a document, not embedding it. There is consequently no synchronous embedding path in this design at all: not_built/building is a normal, expected transient state for a document that hasn't caught up yet, not an error condition — the index-status mechanism above is what makes that an acceptable default rather than something needing to be hidden from the caller.

index_status travels with every search response, not a separate status-check call¶

index_status is folded into the same response search/research_notes_search already returns, not gated behind a separate status-check call — a caller polling readiness re-invokes the same search call rather than orchestrating a check-then-search sequence. See ADR-00025: index_status travels with every search response, not a separate status tool for why, including why fts — despite being synchronous and normally immediate — is included on the documents side.

The two corpora populate it under genuinely different rules, though, not one unconditional shape:

• Documents (research_search_fetched): index_status is dict[str, IndexStatus], one entry per registered mechanism (e.g. {"fts": "ready", "vector": "stale"}) — but only when provider, identifier, and format are all given, identifying exactly one document; otherwise it's {}. This has no meaningful unscoped aggregation today (mixed ready/stale/building states across an entire corpus don't collapse to one value per mechanism), so an unscoped call — e.g. corpus-wide mode="fts" with no identifier/provider/format — cannot poll readiness this way at all.
• Notes (research_notes_search): index_status is dict[str, IndexStatus] | None, None unless the vector mechanism actually ran this call (mode in ("vector", "hybrid") with a keyword given), and when present contains only a "vector" key — there is no per-call "fts" existence check for notes to report. Its value aggregates over the caller's full structurally-matched note-id scope (not merely the notes that landed in vector.matches this page — see Composition below for why that distinction matters), worst-case-wins in the order building > not_built > stale > ready, with an empty scope reporting not_built.

fts's status is computed differently from vector's, and is narrower as a result. vector status is a comparison (recorded-vs-configured model), which is how stale arises; FTS5 has no "model" to go stale against, only an existence check (does an entry for this key exist in search.sqlite3) — so fts only ever takes two values in practice, not_built/ready, and never stale. This is a documented, code-level invariant, not a schema-level one: fts and vector share the same four-value IndexStatus Literal["not_built", "stale", "ready", "building"] (see Index status is per-document/note above) rather than fts getting its own narrower type — so ty type-check catches a wrongly-typed value outside that four-value set, but nothing at the type level stops a future code path from assigning fts="stale"; only inspection of the code paths that actually populate fts's status confirms none of them do.

vec0 filtering: metadata columns, not auxiliary columns or a join¶

sqlite-vec's vec0 virtual table supports three distinct kinds of non-vector columns, verified directly against the maintainer's documentation of the feature rather than assumed: metadata columns (normal columns, stored/indexed alongside the vectors, usable in a KNN query's WHERE clause via =, !=, >, >=, <, <=, BETWEEN, IN), partition keys (a metadata column that additionally pre-shards the vector index by that key, a performance optimisation for large corpora where one filter dimension dominates), and auxiliary columns (+-prefixed, stored in a separate side table and joined at SELECT time, explicitly barred from appearing in a KNN WHERE clause).

provider, identifier, format (documents) and note_id (notes) are declared as metadata columns, not auxiliary columns. This gets WHERE identifier = ? AND provider = ?-style filtering, or note_id IN (...) for a note-id-set scope, evaluated inside vec0 itself as part of the same KNN query — no auxiliary-table join, and no over-fetch-then-filter in application code. Partition keys aren't adopted now: they're a scaling optimisation, and this project's stated corpus scale (thousands to tens of thousands of vectors, per the sqlite-vec engine choice above) doesn't yet justify one. They remain available later if one filter dimension turns out to dominate query patterns at larger scale — the same "swap when the scaling trigger is actually hit" reasoning already applied to choosing sqlite-vec over a true-ANN engine.

Composition: a mode parameter, not a new opaque "smart" search¶

research_search_fetched (and its notes equivalent) gains a mode parameter — fts (today's behaviour, always available), vector, and hybrid. mode="hybrid" fans out to every mechanism that exists as a capability of the running server and returns each one's results as its own separately-labeled set — {"fts": {...}, "vector": {...}, "index_status": {...}} — rather than merging them into one server-fused ranked list. See ADR-00024: Composition — a mode parameter, no server-side result fusion for why, including why an earlier draft's server-side RRF fusion was reversed.

This matters operationally, not just philosophically: FTS indexing is synchronous and cheap (available the moment a parse pass completes), while vector indexing needs an embedding pass (possibly a network round-trip). A document can therefore be FTS-ready long before it's vector-ready.

Which keys appear in the results shape is a capability question, not a per-document readiness question — the same distinction the ResearchPublicationProvider capability table already draws for providers: a mechanism absent from the running server's configuration is absent from both the results shape and index_status entirely, rather than present with an empty/not_built placeholder. A non-hybrid mode (e.g. mode="fts") only returns that one mechanism's results key. index_status itself, on the documents side, is populated independently of mode — it's gated by whether the call identifies exactly one document (provider+identifier+format all given), not by which mechanism mode requested — so a fully-scoped call with mode="fts" can still see index_status.vector == "not_built" and decide whether to also try mode="vector"; an unscoped mode="fts" call cannot, since index_status is {} for it (see index_status travels with every search response above). Requesting a mode that isn't a capability of the running server at all is a distinct case — an error, not a gracefully-empty response, since the caller asked for something structurally absent rather than merely not yet ready.

An agent that wants a specific mechanism deliberately can still request it explicitly via mode.

Result shape: mirror FTS's fields per corpus, uniform excerpts, no score-comparability requirement¶

Documents corpus. VectorSearchBackend.search mirrors SearchIndex.search's shape as closely as possible: {"provider", "identifier", "format", "chunk_id", "offset", "snippet", "score"} — same fields, same meaning for provider/identifier/format/offset/chunk_id (see chunk identity, above; FTS gains chunk_id too, tracked in #31), score a cosine distance instead of BM25 — not a similarity: 0 = identical, larger = less similar, so lower is better, the opposite of a typical relevance score (see Interface specification → research_search_fetched for the same polarity note on the response shape). snippet keeps its name for shape symmetry with FTS (useful to mode="hybrid" callers treating both uniformly) even though its meaning shifts slightly: FTS5's snippet() centers a highlighted window on the matched keyword, which has no semantic-search equivalent, so VectorSearchBackend's snippet is a plain truncated excerpt of the chunk text (first N characters + ellipsis), not highlighted. Every result gets one, uniformly — not only the top-K — since a truncated excerpt is already cheap regardless of rank, and giving every result the same self-describing shape avoids the caller having to cross-reference a separate top-K-only field by index or chunk_id. The excerpt earns its place the same way FTS's snippet/offset pair already does per the interface spec ("enough context... to locate the match"): it lets a caller triage which of several ranked hits are worth a follow-up parse_full_text call before paying for one, which matters at least as much for vector search as FTS — a bare cosine-similarity number is close to meaningless to judge on its own, unlike a BM25 score next to a visible keyword match.

Notes corpus. {"note_id", "score", "text_preview"} — richer than today's shipped NotesSearchIndex.search, which returns only a bare rank-ordered list[str] of note_ids (no score, no preview). This is a deliberate asymmetry, not an oversight to fix here: a vector score is the only relevance signal available for a semantic match (no keyword to eyeball), so it's needed more here than it was for keyword-matched notes, where the existing minimal shape was presumably judged sufficient. This does leave notes-FTS as another candidate for a follow-up-issue-shaped gap, parallel to chunk_id — worth a future issue, not resolved as part of this chapter.

Notes' vector: null-without-keyword is intentional, not an asymmetry bug. research_notes_search's keyword is optional (a keyword-less call is a valid structural-filter-only listing), unlike research_search_fetched's query, which is required — so mode="hybrid" with no keyword still populates fts (a keyword-less listing) while vector stays null, a null/populated split that can't arise on the documents side. This follows directly from the null/populated convention this project already settled on (null means the mechanism wasn't invoked this call; a populated object with an empty matches list means it was invoked but found no matches — see Interface specification → research_notes_search): with no keyword, there is nothing to embed, so the vector mechanism is correctly never invoked at all, the same way it would be for any other mode not selected.

No raw-score comparability requirement. mode="hybrid" no longer fuses results into one ranked list (see Composition, above), so score never needs to be comparable across mechanisms — each mechanism's score only needs to make sense within its own ranked list. This removes what would otherwise have been a much harder constraint on this section.

Deletion cascade: a plain scoped delete¶

research_delete_fetched and research_notes_delete must call a VectorSearchBackend remove operation — remove_document(provider, identifier, format) for documents, remove_note(note_id) for notes — mirroring SearchIndex.remove_document/NotesSearchIndex.remove_note exactly, both in signature and in being scoped by document/note identity rather than individual chunk_id. This is safe to settle now: a vector row is self-contained, leaf-level data (one chunk's embedding, tied to exactly one document) with no structural references pointing into it from elsewhere, so deleting every row for a document identity is a complete, uncomplicated cleanup — the same reasoning that already lets SearchIndex.remove_document be a plain scoped DELETE.

Rate limiting is implementation-owned, not an EmbeddingBackend-interface concern¶

Whether an EmbeddingBackend implementation needs rate limiting isn't a property of "is it API-backed" — it's a property of whether it's talking to a genuinely rate-limited third party. A local/self-hosted inference server (llama.cpp, Ollama, vLLM, text-embeddings-inference) is technically "API-backed" in the sense of being an HTTP call, but has no terms-of-use rate limit to respect — the only constraint is the user's own hardware throughput, which isn't PriorisMCP's business to arbitrarily throttle. Imposing ResearchPublicationProvider-style fixed request spacing on a local server would be a pure regression: needlessly capping throughput the hardware could otherwise sustain, for a compliance reason that doesn't exist in that case.

This mirrors how rate limiting already works for ResearchPublicationProvider: Architecture → Caching and rate limiting frames it explicitly as "a provider concern" — the interface itself doesn't mandate ProviderRequestQueue; individual providers (arXiv, Europe PMC) opt into it because each is talking to a real, rate-limited third-party API. EmbeddingBackend follows the identical pattern: the interface stays agnostic, and a concrete implementation that does talk to a genuinely rate-limited hosted API (OpenAI, Cohere, Voyage AI) owns its own ProviderRequestQueue instance, the same way each ResearchPublicationProvider does — a decision made per-implementation, not imposed by this chapter or by the interface itself.

This resolves what looked like an open design question rather than deferring it: there was never one uniform "does EmbeddingBackend need rate limiting" answer to give, because the premise — one answer for every "API-backed" implementation — doesn't hold once local-inference-server implementations are in view.