How Processing Works
From an imported passage to a stored record, a retrieved result, and reviewable model context.
Papyrus separates document handling into stages so a researcher can see what kind of mechanism is used and what record it produces. Technical names identify the intended architecture; runtime behavior and tests still need further confirmation.
Three kinds of processing
| Approach | Suitable work | What it does not establish |
|---|---|---|
| Conventional code | Parsing, segmentation, identifiers, database writes, pattern matching, formatting, and cache checks. | Deterministic code can still misparse, omit, or connect information incorrectly. |
| Small task-specific models | Embeddings, reranking, classification, transcription, and narrow predictions. | A narrow scope does not make a prediction correct or unbiased. |
| Generative models | Selected synthesis, comparison, or relationship judgments. | Fluent output is not evidence and can introduce unsupported interpretation. |
What happens when a source is added
- Register and parseRecord the source, extract available text or a transcript, and divide it into blocks, sections, and addressable passages.
- Extract selected recordsConfigured stages look for names, dates, citations, tables, or candidate claims. These are classifications and associations, not verified facts.
- Prepare retrievalText can enter a full-text index and an embeddings-based vector index for lexical and meaning-based search.
- Reuse eligible workA cache can compare content, configuration, and model identity. Reuse avoids repetition; it does not prove correctness.
| Stage | Example record | Reason to review it |
|---|---|---|
| Parsed passage | A paragraph about a proposed water rule receives a source identifier and location. | Page order or extraction may be wrong. |
| Extracted record | “Department of Natural Resources” is marked as an organization. | Entity boundaries and aliases can be misclassified. |
| Retrieved result | A query about agency authority ranks that passage highly. | Rank does not establish relevance or legal significance. |
| Assisted output | A comparison cites the passage and exposes assembled context. | The model may overstate support or miss conflicting material. |
Where it all gets stored
The intended design uses SQLite for structured project records and a local vector index for embeddings. A record carries an artifact identifier, type, source provenance, and payload. Provenance records origin; it does not certify truth.
Finding the right passage
- PlanIdentify query type, names, dates, filters, or structured constraints.
- RetrieveRun lexical, embedding, entity-aware, or structured lookup as appropriate.
- Fuse and rerankReciprocal rank fusion combines ranked lists; an optional reranker reconsiders a smaller candidate set.
- Assemble contextPack selected records and exact passages within a configured budget, retaining source identifiers.
Fusion and reranking can still promote an irrelevant passage, but increases the quality of retrieved results significantly. These steps help determine the final context package the LLM receives when answering a query.
Which model does what
- Embeddings model
- Turns passages and queries into vectors for similarity search; similarity is not a factual judgment.
- Reranker
- Scores a small query–passage set after retrieval; it changes order rather than supplying evidence.
- Classifier
- Classifies sentences and words by type, like recognizing "John Smith" as a person.
- Generative model
- Produces selected synthesis from assembled context. Runs using ollama or llama cpp backend.
Current limitations
- Deterministic stages can be repeatable and wrong; repeatability depends on inputs, configuration, and versions.
- Cache reuse and determinism are separate properties; a cache may preserve an error.
- Source edits can change offsets, passage boundaries, content hashes, and identifiers.