This document defines stable system guarantees for victus-rag.
These contracts govern the interfaces and artifacts that retrieval, indexing, evaluation,
telemetry, and generated datasets depend on. Future changes should preserve these
guarantees or explicitly document a compatibility break.
Covered contracts:
Out of scope:
The repository exposes the victus-rag CLI entrypoint.
Stable top-level command groups:
indexqueryevalexportStable retrieval modes:
query sparsequery densequery hybridStable indexing modes:
index sparseindex qdrantStable evaluation modes:
eval sparseeval denseeval hybrideval generate-beireval sparse-beireval dense-beireval hybrid-beirStable export mode:
export claims-csvThe CLI may add flags or commands over time. Existing command names should not be renamed
without an explicit compatibility decision.
Default application configuration lives at:
config/default.yaml
Synthetic BEIR generation configuration lives at:
config/beir_generation.yaml
Stable top-level configuration sections:
pathsqdrantdenseparquet_contractquery_providertelemetryStable BEIR generation keys:
output_dirtotal_queriessingle_specificity_countmulti_relevance_countseedsplitopenai_modelllm_temperaturellm_claim_batch_sizesingle_prompt_fileConfiguration values may be overridden by CLI flags where supported. YAML sections must be
objects/mappings when present.
Embedded claim Parquet files must contain the configured id, text, and embedding columns.
Default claim columns:
claim_id: string or large stringclaim_text: string or large stringembedding: list of float32 or float64 valuesOptional metadata behavior:
metadata_column is configured, that column must be a struct.metadata_column is empty, all non-id, non-text, and non-embedding columns are copiedRecord-level invariants:
Dense indexing writes one Qdrant point per embedded claim record.
Point ID guarantee:
Payload guarantees:
external_id identifies the claim.claim_text stores the claim text.source_path is present and defaults to source_file or an empty string.source_locator is present and defaults to the external claim ID.paper_id is present and is derived from existing payload metadata, the claim ID, or aDense retrieval must be able to normalize legacy payloads that contain enough information
to recover an external_id.
Retrieval payloads must include:
querytop_kresult_countresultsEach result must include:
rankscoreexternal_idunit_typepaper_idsource_pathsource_locatorsnippetclaim_textmetadataSparse and hybrid results include grounding data when available:
support_sectionevidence_spancontext_matchesSparse results include matched-term explanations.
Dense results include the backend collection and query-provider context in the parent
payload.
Hybrid results include component scores and ranks for sparse and dense sources.
Invariants:
unit_type for retrieval results is claim.rank is one-based.result_count reflects the number of returned results.external_id is the primary identifier used by evaluation and telemetry.Local evaluation input is JSON loaded from the configured evaluation path.
Accepted query fields:
query_id or idquery or query_textrelevant_doc_ids or relevant_idsEvaluation examples may include:
categorydifficultymetadataRelevant IDs are canonicalized when they use supported claim identifier forms.
Local evaluation output includes:
top_kInvariant:
Synthetic BEIR generation produces a BEIR-compatible dataset directory.
Required files:
corpus.jsonlqueries.jsonlqrels/<split>.tsvmanifest.jsongeneration_status.jsonCorpus rows must contain:
_idtitletextmetadataQuery rows must contain:
_idtextQrels must be tab-separated and include the header:
query-id corpus-id score
Generation invariants:
single_specificity_count + multi_relevance_count must equal total_queries.Current configured defaults:
total_queries: 187single_specificity_count: 180multi_relevance_count: 7split: testopenai_model: gpt-5-mini-2025-08-07llm_claim_batch_size: 30BEIR evaluation writes metrics and optional run artifacts under the configured artifacts
directory.
Stable artifact families:
artifacts/metrics/artifacts/runs/Metrics payloads include:
Optional run artifacts:
Failure-analysis rows include:
Retrieval telemetry is optional.
When enabled, retrieval commands emit an OpenTelemetry span named:
rag.query
Required span attributes:
app.pipeline_stagedataset.nameuser.query.length_wordsinput.valueopeninference.span.kindretrieval.backendretrieval.top_kretrieval.result_countOptional backend attributes:
retrieval.collectionretrieval.query_providerretrieval.dense_model_nameRetrieved document attributes use indexed document keys:
retrieval.documents.<index>.document.rank
retrieval.documents.<index>.document.score
retrieval.documents.<index>.document.id
retrieval.documents.<index>.document.content
retrieval.documents.<index>.document.source_locator
retrieval.documents.<index>.document.metadata
retrieval.documents.<index>.document.grounding
Telemetry invariants:
telemetry.max_documents.document.content records claim text for auditability.Validation failures should fail fast with explicit messages.
Expected failure categories:
Partial BEIR generation writes generation_status.json with failure context when generation
fails after the writer has been initialized.
Future changes should preserve:
external_id usage across componentsexternal_id and claim_textAny change that breaks these expectations should be treated as a compatibility break and
recorded in docs/decisions/.
docs/000-SYSTEM-CONTEXT.md for repository purpose, scope, and vocabulary.docs/100-ARCHITECTURE.md for system shape and component interaction.docs/200-OPERATIONS.md for operational workflows and validation.docs/decisions/ for compatibility-breaking or architecture-level decisions.