Document storage¶

StorageBackend and SearchIndex are separate interfaces (see Architecture → SearchIndex for why), each with exactly one v1 implementation: FilesystemStorageBackend and SqliteFts5SearchIndex respectively. Both write under the same local-disk root, $XDG_DATA_HOME/prioris-mcp/downloads/, but to different files for different reasons — see Separate SQLite files, not one shared database below.

catalogue.sqlite is the one top-level, cross-document index: a single row per (provider, canonical identifier, format, artefact) entry, backing list/exists/delete. Everything else content-bearing lives one level down, under a per-document documents/<document-hash>/ directory keyed by sha256(provider, canonical identifier) — never a raw identifier, for the path-safety reasons in Storage keys are hashed below. Each such directory holds one manifest.sqlite (structural leaf/chunk/summary rows, shared across every format fetched for that document) alongside one subdirectory per format actually fetched (pdf/, html/, ...), each holding that format's own document/markdown artefacts and metadata.jsonl.

search.sqlite3 sits beside catalogue.sqlite at the top level but is architecturally distinct from it: a disposable FTS5 cache synced from manifest.sqlite chunk/leaf rows whenever a document is parsed or deleted, never itself a source of truth, and safe to delete and rebuild at any time — unlike catalogue.sqlite and manifest.sqlite, which are durable.

Purpose¶

fetch_full_text and parse_full_text are the two capabilities in ResearchPublicationProvider that produce content requiring persistence. search, list_top_n, and fetch_metadata results are small and already covered by the server's response-caching middleware. Full text (a PDF, an HTML document, ...) and its parsed Markdown are different: potentially large (full text) or expensive to produce (parsing is CPU-heavy), so a repeat request for either shouldn't need to redo the work if it doesn't have to. The StorageBackend abstraction is where both are persisted.

StorageBackend¶

Interface¶

Full-text search is deliberately not a StorageBackend operation — it lives on a separate SearchIndex abstraction instead, see ADR-00004: SearchIndex as a separate interface from StorageBackend for why. This document covers only v1's one SearchIndex implementation and the physical index file it manages — see Full-text search: the search.sqlite3 index below.

fetch_full_text checks exists before performing a network fetch, and can return the already-persisted copy instead of downloading again. This is the mechanism that avoids redundant downloads for full text — it lives in the storage abstraction itself, not in the server's generic response-caching middleware, which is a poor fit for potentially large binary documents.

list and delete back the grouping-level research_list_fetched/research_delete_fetched MCP tools (see Architecture → list_fetched/delete_fetched) — a caller-driven way to enumerate or remove specific entries (e.g. correcting a mistakenly-fetched wrong identifier), distinct from the deferred, disk-pressure-driven retention/eviction policy below. Both operate on whichever catalogue entries already exist, for any provider, without touching content that isn't already persisted.

parse_full_text follows the same pattern one level up: it checks exists for the parsed Markdown first and returns it if present; only if that's missing does it check exists for the source full text, parse it (CPU-heavy), write the Markdown, and return it. It never triggers a fetch_full_text itself (see Architecture) — if the source full text isn't there either, it fails with a "not found" error.

Raw full text and its parsed Markdown are now two artefacts of the same (provider, canonical identifier, format) entry — document and markdown respectively — rather than two independent format values (see Directory layout below). This reuses the existing exists/write/read contract, parameterised by an additional artefact argument, rather than introducing a second, parallel concept for derived content. Because canonical identifiers are immutable once resolved (see Identifier canonicalisation), a persisted parse result is exactly as permanently valid as the source content it was derived from — no separate invalidation logic is needed for it.

Identity and location¶

Persisted content is identified by provider + canonical item identifier + format + artefact (e.g. arXiv item 2601.05525v2, format pdf, artefact markdown), not by a filename the caller chooses. This keeps exists/write/read consistent regardless of which backend is in use, and avoids collisions between providers that might otherwise reuse the same identifier scheme. "Canonical" is doing real work here — see Identifier canonicalisation below.

Storage keys are hashed, not built from the raw identifier¶

The top-level storage key — the document-hash — is derived by hashing (provider, canonical identifier) — a SHA-256 hex digest — rather than encoding the identifier into the path. format is deliberately not part of this hash: it's a plain, literal path segment (pdf, html, ...) nested under the document-hash directory, so every format fetched for the same document lands under one shared parent rather than at unrelated, independently-hashed locations (see Directory layout). See ADR-00008: Storage keys are hashed, not built from the raw identifier for why.

Human-readability is preserved separately: each format directory has a small metadata.jsonl (provider, original identifier, canonical identifier, format, fetch/parse timestamps, ...) rather than a descriptive filename, and the top-level catalogue.sqlite (see The catalogue) indexes every entry across the whole store. This also gives future eviction/inspection tooling a natural place to look.

Directory layout¶

$XDG_DATA_HOME/prioris-mcp/downloads/
  documents/
    <document-hash>/                 sha256([provider, canonical identifier])
      manifest.sqlite                 per-document structure, shared across formats — see below
      pdf/
        document                      raw fetched bytes
        markdown                      parsed Markdown (artefact "markdown")
        images/<image-id>             extracted raster images — see Future below; only present when enabled
        metadata.jsonl                 per-artefact fetch/parse timestamps, sizes, ...
      html/
        document
        markdown
        metadata.jsonl
  catalogue.sqlite
  search.sqlite3                       disposable FTS5 index — see below

Every format a caller fetches for the same document (e.g. arXiv exposing both pdf and html full text for the same item) gets its own subdirectory under the same <document-hash>, since document/markdown content genuinely differs per format — a PDF's page structure has no HTML equivalent, and the two parses can legitimately disagree on wording (OCR artefacts, layout reconstruction) even for the same underlying publication. The per-document manifest.sqlite holds rows for every format side by side, disambiguated by its own format column, rather than needing a separate file per format the way the old structure.jsonl design did.

catalogue.sqlite, manifest.sqlite, and search.sqlite3 are all SQLite, but deliberately kept as three separate files rather than one shared database — see Separate SQLite files, not one shared database below — and embedded SQLite itself comes with a concurrency scope that bounds where this backend can be deployed at all — see Embedded SQLite vs. a client-server database below.

Per-document structure: manifest.sqlite (replaces structure.jsonl)¶

One SQLite file per document-hash directory, shared across every format fetched under that document, replacing the earlier plain structure.jsonl sidecar. Like structure.jsonl before it, every row's spans are pointers into a markdown artefact — {start, length} character ranges — never a copy of the text itself; duplicating parsed Markdown inside its own sidecar would be exactly the kind of drift-prone redundancy the rest of this design avoids. Its content is always deterministic, produced synchronously as part of parse_full_text, with two exceptions noted explicitly below (an LLM-derived chunk fallback, and the future summary kind) that are not produced by anything in v1.

Schema — one table, one row per entry:

Entry order is derived, not stored: span-internal order comes from JSON array order (inherently ordered, unlike a normalized child table needing its own sequence column); a row's position among its siblings comes from sorting by its first span's start (or, for leaf, its numeric key/page number directly) — there is no ordinal column. Nesting (e.g. a subsection inside its enclosing section) is likewise derived, not stored: a chunk row's span is a sub-range of its parent's span, so containment among chunk rows sharing the same (format, scheme) reconstructs the hierarchy on demand; there is no parent_id. This is deliberately scoped to one scheme at a time — two different chunking passes' boundaries aren't guaranteed to align, so containment across schemes wouldn't mean anything.

Entry kinds:

1. leaf — the page-span level: parser-determined, one span each, ordered, and exhaustively partitioning the blob (or a single trivial span covering the whole blob, for whole-document formats with no native page concept, like JATS/HTML). Comes directly from the parser: ParserBackend.to_markdown() returns {"markdown": str, "leaf_spans": list[{"start", "length"}]} rather than a bare string, so leaf boundaries are recorded as the Markdown blob is assembled, not recovered from it afterwards. For PDF, this means building the blob directly from each page's own rendered Markdown and tracking its offset range while joining pages together with an ordinary block-separating joiner — not by inserting, and later trying to parse back out, a page-separator marker, which would be fragile if a page's own content ever happened to contain one.
2. chunk — a structural unit of the source document (e.g. a heading-bounded section), not a pattern matched against already-flattened text. Optional: the parser emits chunk rows only when it can recover source-document structure; when it can't, no chunk rows are produced for that leaf/document, and callers needing sub-leaf granularity fall back to leaf-level only. Detected by a single format-agnostic mechanism, run over the assembled markdown blob once leaf_spans come back: a walk over ATX-style Markdown headings (^#{1,6}\s+...), skipping any that fall inside a fenced code block. Each heading's span runs from its own start offset to the start of the next heading at the same or shallower level (or the end of the blob), which naturally produces one row per nesting level rather than one row per section — a subsection and its enclosing section both get their own overlapping row. This works uniformly across PDF, JATS, and HTML because all three converge on genuine ATX headings in their rendered Markdown (PDF via the parser's own font/layout heuristics; JATS via its stylesheet's depth-mapped headings; HTML via its DOM headings), with no per-format special case needed. provenance is parser for this walk; a chunk row with provenance = llm is reserved for a future fallback when structure can't be recovered at all (e.g. an untagged, OCR'd PDF), not produced by anything in v1.
3. summary — reserved for a future document-level summarisation capability (see Future: document-level generated content); not a span at all, but its own generated-text artefact referencing child chunk/summary row IDs. When it exists, it should reference chunk rows (not leaf rows) as its base layer — including for JATS/HTML, which have only one (trivial) leaf each but can still have many parser-derived chunk rows to build on. What a document with zero chunk rows at all falls back to (e.g. a PDF where structure recovery failed and no LLM-fallback chunking ran) is not yet resolved, and is deferred along with summary's design generally.

Indexing: a B-tree index on (format, kind, key) covers exact lookups, including the page parameter on parse_full_text, which resolves to format='pdf' AND kind='leaf' AND key=<page>. A generated column, span_start INTEGER GENERATED ALWAYS AS (json_extract(spans, '$[0].start')) VIRTUAL, indexed separately, covers page_range (which leaf/leaves an arbitrary offset/limit window spans — see Non-functional requirements → Response size) via sorted lookup rather than an interval query, exploiting that leaf rows are contiguous and exhaustive by construction — no gaps or overlaps to reason about. chunk rows can legitimately overlap (per the every-nesting-level rule above), but manifest files hold at most a few hundred rows per document, so overlap/containment queries over chunk rows are handled by loading all of a document's rows and filtering in application code, rather than with interval/R-tree indexing — not worth the complexity at this store's target scale.

Extracted PDF images (see Future: extracted PDF images below) get their own kind = "image" row, carrying page, bbox, and duplicate_of (the parser's own byte-level image dedup) instead of spans.

The catalogue: catalogue.sqlite¶

A single, top-level table — one row per (provider, canonical identifier, format, artefact) entry — rather than requiring list/exists/find_canonical_identifier to read every format directory's own metadata.jsonl individually. It is the sole source those three operations read from; write/delete are the only two operations that mutate it, mirroring how they're already the only two operations that touch metadata.jsonl today.

Forward lookup never actually needs the catalogue: the document-hash is deterministically sha256(provider, canonical identifier), computable on the spot from a request. What the catalogue buys is cheap reverse enumeration instead — hash-named directories don't carry the original identifier back out, so list/research_list_fetched would otherwise mean opening every document's own metadata.jsonl (an open per document) instead of one indexed table (one lookup covering every entry) — and, for the local filesystem source specifically, the catalogue is the only place its caller-facing-ID ↔ content-hash mapping exists at all, since that ID has no deterministic hash formula of its own.

A SQLite table, keyed by a unique constraint on (provider, canonical_identifier, format, artefact) with INSERT ... ON CONFLICT — see ADR-00009: Catalogue as SQLite, not append-only JSONL-plus-replay for why. See Embedded SQLite vs. a client-server database below for the concurrency model this depends on, and its limits.

Full-text search: the search.sqlite3 index¶

SearchIndex (see Architecture → SearchIndex) doesn't require SQLite — it's its own pluggable abstraction, and a different implementation (a hosted search service, a different embedded engine) could back it just as validly. SQLite + FTS5 is v1's one concrete choice, described here: one global index (search.sqlite3 above), not one index per document, and not through StorageBackend (see Interface above for why search doesn't belong there).

The indexed unit is a manifest.sqlite chunk row (or a leaf row, for a document with no chunks — the same leaf-fallback principle chunk-based callers apply generally), not a whole markdown artefact: a match is a specific section or page, not merely "this document contains the term somewhere." It is a plain FTS5 virtual table, not a SQLite content='' external-content table — see ADR-00010: FTS5 as a plain virtual table, not an external-content table for why. The table's own text column holds only the matched chunk/leaf's span (never a whole document's Markdown), tokenized directly, alongside a reference back to that span's position (provider, identifier, format, and the source row's key/span_start) as UNINDEXED columns, so a query can be scoped to a single document (MATCH ... AND identifier = ?) or left unscoped to search everything through the same table — no separate per-document index files. span_start is denormalised as its own UNINDEXED column specifically so a search result's reported offset means the matched entry's position in the document's own coordinate space, not an FTS5-internal one: FTS5's own offsets()/snippet() are relative to the indexed text column itself, which isn't what a caller re-fetching context via parse_full_text's offset/limit parameters needs.

One global index, not one per document, and not an in-memory scan either — see ADR-00011: One global search index, not one per document for why.

Synced incrementally: a whole-document replace of a document's indexed rows whenever it's freshly parsed (there's nothing to incrementally diff against, since a parse pass produces the complete manifest fresh each time), and a removal of all of a document's indexed rows at the same point its markdown artefact is deleted — no triggers, no periodic full rebuild needed. Treated as a pure, disposable cache, never a source of truth: catalogue.sqlite, manifest.sqlite, and the markdown files remain durable; search.sqlite3 can be deleted and rebuilt from them at any time, and doesn't need to live on the same (possibly cloud-mounted) volume as the durable files — it can live on local/ephemeral disk where available, sidestepping cloud-storage corruption risk entirely rather than tolerating it. See Separate SQLite files, not one shared database below.

Overlapping matches are an accepted characteristic of chunk-based indexing, not a defect: a query can legitimately match both a broad section chunk and one of its nested subsection chunks for the same underlying text, since every heading-nesting level gets its own manifest row. bm25 naturally favours the more topically-dense (usually narrower) chunk, but a caller should expect overlapping hits in the result set, not treat them as an indexing bug.

Embedded SQLite vs. a client-server database¶

SQLite genuinely solves the atomicity problem catalogue.sqlite, manifest.sqlite, and search.sqlite3 all need, for any number of writer processes sharing one local, properly lock-capable filesystem — including multiple PriorisMCP instances on the same machine (e.g. two parallel agent sessions each launching their own MCP server against the same storage root). This is not a single-process limitation; it's exactly the scenario SQLite's file-locking protocol is designed for, and a strict improvement over an append-only-JSONL design's implicit single-writer assumption (real transactions instead of hoping O_APPEND behaves).

It does not, however, work across machines or replicas, nor over a network-mounted filesystem (NFS/SMB) or a consumer file-sync client (Dropbox, Google Drive, OneDrive, and similar): SQLite has no distributed-replication story of its own, and file-locking semantics are unreliable to nonexistent over those substrates. Any deployment where writer processes cannot share a single local, properly lock-capable filesystem — i.e. genuinely different machines or replicas — MUST use a real client-server database (e.g. Postgres, or a distributed SQLite-compatible option) for the catalogue/manifest/search roles instead of embedded SQLite. This is stated on writer-process topology, not on where content blobs live: the two axes usually coincide (an object-store backend tends to get adopted specifically to enable multiple replicas) but aren't logically identical, and multi-replica deployment is out of v1's scope regardless — see Future: S3 below.

The local-filesystem storage root (PRIORIS_MCP_STORAGE_DIR) MUST NOT reside inside a directory managed by a consumer cloud-sync client or a network-mounted filesystem. These sync/mount layers don't understand SQLite's locking/journaling protocol and can propagate a torn write or produce unmergeable "conflicted copies" — SQLite's own documentation warns against exactly this. This restriction applies to the whole local-filesystem backend, not just its use of SQLite.

No alternative embedded engine is adopted in place of SQLite. Any alternative would need its own explicit multi-process concurrency audit before being treated as interchangeable — SQLite's decades-proven locking guarantees are specific to SQLite, not assumed to generalise to nominally "SQLite-compatible" alternatives.

Separate SQLite files, not one shared database¶

catalogue.sqlite, each document's manifest.sqlite, and search.sqlite3 are separate SQLite files (the same engine, not the same file), even though they could technically share one — see ADR-00012: Separate SQLite files, not one shared database for why, including the measured per-document overhead this costs.

Journal mode (WAL vs. rollback) and backup/restore for the embedded-database local backend are both deferred to implementation/configuration time — neither changes the schema or file layout decided here. A naive file copy of a live SQLite file mid-write isn't a safe backup mechanism regardless of journal mode; a real backup needs the engine's own consistent-snapshot mechanism (its backup API, or VACUUM INTO).

Losing human grep/cat-ability for the catalogue and manifest (plain JSONL in earlier drafts of this document) is a minor cost, not a real capability loss: the sqlite3 CLI, including -json output pipeable into jq, is as close to a single command as jq was for JSONL.

Deletion is per-artefact, not per-format¶

delete(provider, identifier, format, artefact) takes an artefact of document, markdown, or all — preserving the independent-deletability guarantee the current design already has (today expressed as two separately-deletable formats, pdf and pdf-markdown; here as two separately-deletable artefacts within one format directory). A caller can drop the bulky raw document while keeping the cheap markdown, or vice versa, exactly as before. artefact="all" removes the whole format directory (document, markdown, any extracted images/, metadata.jsonl) and that format's rows from the shared manifest.sqlite (DELETE FROM manifest WHERE format = ?, not dropping the whole file, since sibling formats' rows must survive); if that was the last format directory under a <document-hash>, the document-hash directory itself — including its now-empty manifest.sqlite — is removed too, in v1's scope — see Future for how this rule needs to change once document-level generated content exists. Deleting the markdown artefact also removes any extracted image artefacts and their kind="image" manifest rows that reference it, since an image placeholder anchored to a markdown blob that no longer exists has nothing left to anchor to.

Identifier canonicalisation¶

Some providers' identifiers don't have a fixed meaning over time. arXiv is the v1 example: an unversioned identifier (2601.05525) means "whatever is currently the latest version," while a versioned identifier (2601.05525v2) is permanently pinned, because arXiv versions are immutable once published.

A pinned identifier is a safe, permanent storage key on its own — its content can never change, so it's always safe to reuse a persisted copy. An unversioned identifier is not: the content behind it can legitimately change (a new version gets published) without the identifier string itself changing, so keying storage on the bare unversioned identifier risks silently serving stale content once a newer version exists. Hashing the unversioned string doesn't fix this — it just produces a very stable-looking hash of an answer that isn't stable.

resolve_identifier (see Architecture) is responsible for resolving an unversioned identifier to its current concrete version before it is used anywhere as a storage key. Storage itself never sees a bare unversioned identifier — only the canonical, version-pinned one that resolve_identifier produced for it, this call, and a pinned identifier passed in by the caller needs no resolution at all. Consequences:

• A canonical (version-pinned) identifier's storage entry can be kept and reused indefinitely.
• An unversioned request may resolve to a different canonical identifier — and therefore land on a different document-hash — once a new version is published, which is correct behaviour, not a bug to work around.
• Resolving "what's current" is a light, metadata-level check, done on every unversioned request regardless of whether anything changed; that per-call cost is the accepted price of never silently serving stale full text, and it's far cheaper than the full-text download it protects against re-serving incorrectly.

Content hashing of the downloaded bytes is a separate concern — useful as an optional integrity check (e.g. detecting a truncated download) — but it does not substitute for version resolution, since it can only detect a change after paying for the download it was meant to avoid.

Content-hash canonicalisation for the local filesystem source¶

The local filesystem source has no equivalent of resolve_identifier, because it has no external authority asserting what "the current version" of caller-sent content is — unlike arXiv, where an unversioned identifier's mutability is a known, bounded fact (it always means "whatever arXiv currently says is latest"), caller-sent content's mutability is unbounded and unannounced: the caller can send edited or replaced bytes on any subsequent call, with nothing to notify PriorisMCP.

Content hashing, dismissed above as insufficient on its own for network sources (it can only detect staleness after paying for the download), is exactly sufficient here, because reading a local file to hash it is not the expensive operation being protected against — copying it into storage is. Every fetch_full_text call for this source reads the file's current bytes and computes their SHA-256 hash unconditionally, then uses (provider="localfile", identifier=content_hash, format="pdf") to locate its entry: the document-hash directory is sha256([provider, content_hash]) — a hash of a hash, distinct from content_hash itself, which plays the role of identifier here exactly as an arXiv ID or DOI would for other providers. This is the canonicalisation step, taking the place resolve_identifier fills for arXiv, just performed inline by the local filesystem source's fetch_full_text rather than exposed as a separate capability. If the hash already exists in storage, write is skipped (a no-op re-fetch); if the file's content has changed since any previous fetch, this produces a new hash and therefore a new, independent storage entry, leaving whatever entry an earlier fetch produced untouched and still validly readable — the same guarantee a pinned arXiv version already provides, arrived at by hashing actual content instead of trusting an external version number.

Caller-facing identifiers for sources without one¶

Storage keys are hashed specifically so they're safe to use as a path segment (see Storage keys are hashed above) — but a content hash, while segment-safe, is not something a caller can usefully reuse in conversation, and the local filesystem source has no caller-supplied identifier at all to fall back on: the only caller-supplied value is the base64-encoded content itself (plus an optional, non-identifying filename hint that isn't even segment-safe, since it can contain /).

The local filesystem source therefore assigns a third, distinct value — a caller-facing identifier — at fetch_full_text time: a minute-resolution timestamp plus a short random suffix (e.g. 20260729-1430-a3f2), segment-safe and legible enough for a caller to recognise in a conversation transcript, without needing either the original content or the content hash to refer back to it. This is what fetch_full_text returns to the caller, what appears in the resource URI, and what parse_full_text subsequently takes as its input — not the content hash (which is an internal storage-key implementation detail, never surfaced).

This gives three distinct values, each with one job, for the local filesystem source specifically:

A catalogue entry (see The catalogue above) maps each caller-facing ID to its content hash and format, so parse_full_text can resolve an ID back to the right storage entry without needing the original content again. Re-fetching unchanged content (same hash) reuses its existing caller-facing ID rather than minting a new one, so a caller who already has an ID for that content keeps using the same one; changed content (new hash) gets a new ID, consistent with Content-hash canonicalisation above never repointing an existing identifier at different content.

Collision handling for the caller-facing ID needs no shared, persistent counter: on generating an ID, the catalogue is checked for that exact value, and a new random suffix is drawn and rechecked in the rare case of a collision — the minute-resolution timestamp already scopes the collision space to whatever falls within the same minute, and a 4-character base-36 suffix (~1.68M values) keeps that risk low even under a burst of concurrent fetches within one minute.

Migration¶

Stores created before this layout landed use the flat <hash>/<hash>.json scheme (one file pair per (provider, identifier, format) triple, format included in the hash). A one-time migration walks that flat layout and regroups each entry under documents/<document-hash>/<format>/, splitting what was a single pdf/pdf-markdown pair of format values into document/markdown artefacts of one pdf format directory, builds catalogue.sqlite from the resulting metadata.jsonl records, and creates each document's manifest.sqlite — initially populated with leaf rows only where page boundaries can be recovered from the pre-migration data, since the flat layout predates chunk/summary rows entirely; a document needing chunk rows gets them the same way any other document does, via a subsequent re-parse. This must be idempotent and safe to run against a store that's a mix of old and new layout (e.g. gated behind a version marker in the storage directory), since there's no way to guarantee every deployment migrates in one atomic step.

v1: local filesystem backend¶

The default — and, for v1, only implemented — backend persists to a directory on local disk, under XDG_DATA_HOME, not XDG_CONFIG_HOME: downloaded content is data, not configuration. If XDG_DATA_HOME is unset, the XDG Base Directory Specification default of ~/.local/share applies.

• Default location: $XDG_DATA_HOME/prioris-mcp/downloads (i.e. ~/.local/share/prioris-mcp/downloads when XDG_DATA_HOME is unset), laid out as shown in Directory layout above.
• Configurable through an environment variable declared on EnvVars (see src/prioris_mcp/__init__.py), consistent with how the rest of the server is configured — not hardcoded, and not read from os.environ elsewhere.
• Must not resolve, directly or via a configured override, to a directory managed by a consumer cloud-sync client or a network-mounted filesystem — see Embedded SQLite vs. a client-server database above.

Future: S3 (or other remote/object) backend¶

Explicitly out of scope for v1 (see the out-of-scope list in the SRS overview), but the exists/write/read contract above is designed so an S3-backed implementation is a second implementation of the same interface, selected by configuration — not a different code path through the providers or tools.

This is scoped more narrowly than earlier drafts of this section claimed. Real object stores have no append operation — a single growing object needs a full GET-modify-PUT, which is a read-modify-write race under concurrent writers — so catalogue.sqlite and each document's manifest.sqlite are not simply portable as single growing objects the way an earlier draft of this section suggested; the same is true of SQLite generally, which has no story for treating an S3 object as a live, lock-capable file at all (see Embedded SQLite vs. a client-server database above). A genuinely S3-safe design for the catalogue/manifest roles needs either an event-per-write log merged at read time and periodically compacted (e.g. catalogue/events/<ts>-<uuid>.jsonl), or a real client-server database in front of them — not a drop-in swap of the local StorageBackend alone.

A hosted, single-process deployment — one dedicated server process, content blobs on S3, catalogue.sqlite/manifest.sqlite/search.sqlite3 on that process's own local disk — is the scenario this contract already supports without further design. Multiple replicas sharing one S3 bucket is not: each replica's local SQLite files would only reflect that replica's own writes, silently returning incomplete results from research_list_fetched/research_search_fetched under concurrent replicas. That scenario needs the client-server database path described in Embedded SQLite vs. a client-server database above, is out of scope for v1, and should not be assumed to fall out of an S3-backed StorageBackend alone.

Future: document-level generated content¶

A reserved slot, not a v1 capability: manifest.sqlite's summary kind (see Per-document structure above) is reserved for a future, format-independent, generated summarisation capability, produced only by an explicit, non-cascading generate_full_text_summaries tool (never triggered automatically by parse_full_text, matching the existing precedent that parse_full_text never auto-triggers fetch_full_text) — out of scope for the storage redesign covered by this document; tracked separately once that tool's own design, including which summarisation strategy it uses, is settled.

This is deferred deliberately, not merely left unscheduled: whatever the eventual strategy turns out to be, its main practical payoff — efficient semantic drill-down through a large document (e.g. navigating a long document down to one relevant section and its cross-references) without dumping every leaf into an LLM's context window — depends on embedding-based retrieval that doesn't exist yet in this design (vector search is itself future work — see Architecture). A plain FTS5 index already captures much of the practical benefit for literal/referential text (a direct keyword search for a specific heading or term already surfaces cross-references well, at a corpus size FTS5 handles efficiently) without needing a generated summary hierarchy at all. Building summary generation now would mean paying real costs — LLM calls, non-determinism, the open questions below — for a capability that can't deliver its main benefit until vector search exists.

Two consequences are already accounted for in the manifest design above so this doesn't require a second migration later:

• summary rows should reference chunk rows (not leaf rows) as their base layer once this is designed — including for JATS/HTML, which have only one (trivial) leaf each but can still have many parser-derived chunks to build on. What a document with zero chunk rows falls back to is not yet resolved.
• search.sqlite3 indexes at the manifest-row level (see Full-text search above), so adding summary rows to the index later is an additive population of new rows, not a schema migration.

Not yet designed: whether summary ends up with a real key (some kind of label), and whether its provenance is unconditionally llm or could ever be something else (e.g. a future non-LLM extractive method) — both are left open pending summary's own design.

Once implemented, the deletion rule above — removing the last format directory also removes the document-hash directory — will need a guard (e.g. a purge_summaries flag, off by default) so that deleting the last format doesn't silently discard summary rows that are still potentially reusable by a later fetch of any format for the same document.

Future: extracted PDF images¶

Off by default in v1, gated by a new PRIORIS_MCP_PDF_EXTRACT_IMAGES env var (EnvVars, default False) — named to match the existing PRIORIS_MCP_PDF_OCR_* family (see Security → OCR language data) rather than a generic extraction toggle, since this is specifically about the PDF parser backend's embedded raster images. JATS/HTML images are link references only (<graphic>/<img src>), with no embedded bytes available at parse time in either pipeline — there is no equivalent "extraction" concept for those formats.

When disabled (the default), an inline Markdown reference to each image still appears in the markdown artefact, but no image bytes are pulled, no kind="image" manifest rows are written, and no image resources are registered — behaviourally identical to today.

When enabled: each extracted image's raw bytes are persisted via the existing StorageBackend.write() — another artefact under the document's pdf/ directory (images/<image-id>), keyed the same way any other artefact is, with no interface change to StorageBackend needed. Its manifest row (kind="image") carries page, bbox, and duplicate_of (the parser's own byte-level dedup for images, a separate concern from StorageBackend's own content-addressing) instead of spans; the image's id is the same identifier already embedded in the Markdown placeholder, so a row ties directly back to its position in the blob. Each image is also exposed as its own MCP resource (a binary resource with its own mime_type) — the URI scheme is not yet decided, but must not leak filesystem paths, consistent with Security. research_delete_fetched needs to cascade-delete a document's image artefacts and rows when its markdown artefact is deleted (see Deletion is per-artefact above).

Search indexing over images is explicitly out of scope: no OCR happens here, so extracted images aren't text-searchable content, and stay linked artefacts rather than SearchIndex entries.

Future: retention and redistribution-aware persistence¶

Two further concerns are explicitly deferred beyond v1 (see SRS overview → Out of scope for v1), and kept distinct from each other despite both bearing on "how long persisted content sticks around":

• Retention/eviction. v1's local filesystem backend never evicts anything — persisted full text and Markdown accumulate indefinitely. Because storage keys are derived from immutable canonical identifiers (see Identifier canonicalisation), nothing ever becomes incorrect by staying persisted — the only concern is unbounded disk growth, not staleness. The intended future direction is a size-based cap with LRU eviction (evicting least-recently-read entries once a configured disk quota is reached), using an extended catalogue entry tracking last-read time alongside the existing fetch/parse timestamps. A simpler time-based TTL could be offered as a secondary option, but LRU-by-size is the primary direction since a TTL alone doesn't bound disk usage under heavy recent use, and can evict content that's still being actively reused simply for being old.
• Redistribution-policy-aware persistence. Whether persisted content may ever be shared beyond the MCP client that originally fetched it is a licensing question, not a disk-management one, and v1 does not have full visibility into per-article licences to make that decision safely — Europe PMC's metadata already carries a license field (see Interface specification), but arXiv's Export API exposes none; only the already-deferred OAI-PMH arXiv format does. This is future work gated on that visibility, not on the eviction mechanism above.