This document describes the v1.0.0 architecture baseline.
victus-processing is a local, file-oriented batch-processing pipeline exposed
through a Python CLI.
The system shape is:
CLI command
-> domain stage
-> local artifacts under data/
-> optional external API call
-> next stage input
The architecture favors explicit stage boundaries and durable local artifacts.
Each stage can be inspected independently because handoffs are represented by
files, not hidden in memory or remote services.
- Path:
src/cli.py
- Responsibility: public command surface and command routing.
- Inputs: user arguments.
- Outputs: invoked stage behavior and CLI status output.
- Dependencies: domain modules.
- Boundary: covers command grouping, not stage internals.
- Path:
src/workspace/
- Responsibility: repository paths, config loading,
.env loading, and
canonical data layout helpers.
- Inputs:
config/*.yaml, optional config.yaml, environment variables.
- Outputs: resolved paths and runtime constants.
- Dependencies: local filesystem.
- Boundary: covers resolution, not business processing.
- Path:
src/application/metadata_extraction/
- Responsibility: discover, fetch, classify, and store paper metadata
candidates.
- Inputs: seed DOI queues, DOI arguments, Semantic Scholar responses.
- Outputs:
data/lake/paper_metadata.jsonl plus compatibility candidate
state for legacy exploration paths.
- Dependencies: Semantic Scholar API, internal LLM client port.
- Boundary: covers pre-PDF candidate state.
- Path:
src/application/bibliography_export.py
- Responsibility: generate DOI-only BibTeX outputs for external PDF
retrieval workflows such as Zotero.
- Inputs: metadata records.
- Outputs: BibTeX artifacts.
- Dependencies: local filesystem.
- Boundary: utility export only; it is not a full pipeline stage and does
not own PDF retrieval or PDF artifacts.
- Path:
src/application/pdf_intake.py
- Responsibility: link manually obtained PDFs to metadata and promote them
to canonical PDF artifacts.
- Inputs: explicit
metadata_id values and manually staged PDFs.
- Outputs:
data/artifacts/pdfs/{paper_id}.pdf and
data/lake/paper_pdf_links.jsonl.
- Dependencies: local filesystem.
- Boundary: covers manual PDF intake, not external PDF retrieval.
- Path:
src/application/pdf_processing/
- Responsibility: convert PDF artifacts into Markdown, split Markdown into
batches, call the internal LLM client, validate batch outputs, and merge structured paper JSON.
- Inputs: PDF artifacts, prompt files, PDF-processing config.
- Outputs: Markdown, raw batch JSON, merged structured paper JSON, status.
- Dependencies: Docling, internal LLM client port.
- Boundary: covers post-PDF structured paper artifacts.
- Path:
src/application/evidence_extraction/
- Responsibility: classify processed papers, trim structured blocks, map
explicit experimental scopes, build packets, and extract canonical evidence.
- Inputs: metadata, blocks, experiment-scope prompt outputs, canonical
evidence prompt outputs.
- Outputs: trimmed paper JSON, experiment map JSON, canonical evidence JSON.
- Dependencies: internal LLM client port.
- Boundary: covers paper-level evidence normalization, not downstream
retrieval, synthesis, recommendations, or analytics.
- Path:
src/application/testing_pipeline/
- Responsibility: prepare per-paper review folders and testing artifacts.
- Inputs: PDF artifacts, optional reused Markdown, PDF-processing outputs,
evidence outputs.
- Outputs:
data/testing/{paper_id}/ review workspaces.
- Dependencies: PDF-processing and evidence-extraction stages.
- Boundary: covers local review artifact assembly, not production runtime
artifact storage.
- Path:
src/prompts/
- Responsibility: store model instructions used by LLM-mediated stages.
- Inputs: repository-authored prompt text.
- Outputs: prompt content consumed by metadata, PDF-processing, and
evidence stages.
- Dependencies: none.
- Boundary: prompts shape behavior but do not execute stages directly.
Internal boundaries:
- CLI routing is separated from stage execution.
- Configuration/path resolution is separated from processing logic.
- Metadata is pre-PDF state.
- PDF-processing is post-PDF structured extraction.
- Canonical evidence is the downstream derived output produced by this repository.
- Canonical evidence is the active downstream extraction terminology.
- Prompts are separate artifacts consumed by model-mediated stages.
External boundaries:
- Semantic Scholar provides metadata and citation graph data.
- LiteLLM provides provider routing, model selection, retries, fallbacks, and
provider credentials for all LLM requests.
- Langfuse observes only LLM request boundaries.
- Analytics, RAG indexing, vector stores, and production deployment are outside
this repository.
High-level runtime sequence:
seed DOI or DOI argument
-> metadata stage
-> metadata lake records
-> bibliography export for Zotero/manual retrieval
-> manual PDF intake
-> PDF artifact
-> PDF processing
-> structured paper JSON
-> trimming
-> experiment scope mapping
-> canonical evidence extraction
-> canonical evidence JSON
The CLI is the orchestrating boundary. It does not run every stage
automatically; operators and agents select the stage to execute.
Main artifact movement:
data/inputs/
-> data/lake/paper_metadata.jsonl
-> data/artifacts/pdfs/
-> data/artifacts/markdown/
-> PostgreSQL scientific output tables
-> PostgreSQL paper_pipeline_state
-> PostgreSQL paper_processing_state
Artifact roles:
data/inputs/: seed queues, rules, and imports.data/lake/paper_metadata.jsonl: canonical pre-PDF metadata state.data/artifacts/pdfs/: canonical PDFs ready for processing.data/artifacts/markdown/: Markdown artifacts ready for structuring.- PostgreSQL scientific tables: StructuredBlocks, classifications, experiment
maps, and canonical evidence.
paper_pipeline_state: per-paper, per-stage attempt state written at pipeline
lifecycle boundaries.paper_processing_state: operational dashboard state derived from inputs and
PostgreSQL outputs and pipeline state.
Detailed path, handoff, configuration, CLI, and schema contracts live in
Contracts.
- Inspectability: stage state is visible as local files.
- Reproducibility: commands consume stable inputs and write stable outputs.
- Recoverability: stages can resume from existing artifacts and status.
- Operational transparency: expensive model-mediated stages are explicit.
- Composability: stages can be run independently or chained by operators.
- Local-first execution: the core workflow does not require deployment
infrastructure.
- Agent readability: docs, prompts, config, and artifacts are discoverable
without requiring hidden service context.
- Semantic Scholar API: metadata and citation exploration.
- LiteLLM: provider abstraction for all LLM requests.
- Langfuse: LLM request boundary tracing.
- Docling: local PDF-to-Markdown conversion.
- Local filesystem: primary durable state store.