PDF processing converts canonical PDF or Markdown artifacts into structured
blocks persisted in PostgreSQL.
Runtime sequence:
PDF/Markdown artifact -> Markdown batches -> LLM JSON -> merged block JSON
-> processed-paper contract enforcement -> structured_blocks
Run all available PDF artifacts:
uv run victus-processing pdf-processing run
Run one PDF:
uv run victus-processing pdf-processing run --pdf path/to/paper.pdf
Limit work:
uv run victus-processing pdf-processing run --limit 10
uv run victus-processing pdf-processing run --pdf path/to/paper.pdf --max-batches 3
Run Markdown-only conversion:
uv run victus-processing pdf-processing markdown --skip-existing
Process an existing Markdown artifact into structured JSON:
uv run victus-processing pdf-processing json-from-markdown --markdown data/artifacts/markdown/{paper_id}.md
Process a random subset of existing Markdown artifacts:
uv run victus-processing pdf-processing json-from-markdown --input-dir data/artifacts/markdown --shuffle --limit 5
Next stages:
Operational inputs:
data/artifacts/pdfs/;data/artifacts/markdown/;src/prompts/;config/pdf_processing.yaml;Batching behavior:
config/pdf_processing.yaml are:target: 10000 characters
soft limit: 14000 characters
hard limit: 20000 characters
max output tokens: provider/model default (`max_tokens: null`)
pdf_processing.max_tokens controls the output budget for local Markdownnull to avoid sending an explicit LiteLLMmax_tokens limit and rely on the provider/model limit. Truncated modelraw_batches/batch_XXXX.failed.json.Prompt and batch-continuity behavior:
src/prompts/pdf_processing/markdown_first_batch.md andsrc/prompts/pdf_processing/markdown_continuation_batch.md.section_registry andbatch_end state from prior batches.original_title, canonical_title,section_type, and parent-path fields.title and type are still accepted.section_registry, updated_section_registry, and batch_end are internalFinal block behavior:
paper_processing_state should show last_successful_stage=pdf.processnext_stage=classification.classify after successful Markdown-to-JSONOperational outputs:
structured_blocks PostgreSQL rows;paper_processing_state PostgreSQL row update after refresh;data/reports/exports/paper_processing_state.csv.LLM behavior:
Processing state refresh:
uv run victus-processing processing-state refresh --csv data/reports/exports/paper_processing_state.csv
Related: Operations,
Data Layout Contract,
Evidence extraction,
Testing pipeline.