This runbook describes production deployment through GitHub Actions, Infisical,
Tailscale, SSH, and Docker Compose.
Production deploys use the full sequential workflow:
.github/workflows/deploy-all.yml
The workflow deploys stacks in order:
observability -> core -> llm
\-> wiki
The observability stack is deployed first so monitoring is available before
the core stack is rolled out. The llm and wiki stacks are deployed after
core so the shared Docker network and edge routing are already present.
Internal job order:
validate -> deploy -> notify
Inside the deploy job, Ansible runs stack playbooks in dependency order:
preflight.yml -> deploy-observability.yml -> deploy.yml -> deploy-llm.yml -> deploy-wiki.yml -> review.yml
Manual:
GitHub Actions -> Deploy All Stacks -> Run workflow
Manual input:
git_ref branch, tag, or commit to deploy
Production secrets come from Infisical through GitHub OIDC. Secrets are not
stored in git or GitHub repository secrets.
The deploy workflow reads scoped Infisical paths:
/Hetzner-Server/global host access and shared deploy secrets
/Hetzner-Server/core core stack runtime secrets
/Hetzner-Server/llm LiteLLM and Langfuse runtime secrets
/Hetzner-Server/wiki Wiki.js runtime secrets
/Hetzner-Server/api-keys provider API keys named KEY_*
Required secrets are listed in security.md.
The deployment workflow pulls secrets from Infisical, validates host readiness
with Ansible, materializes stack runtime files in the runner, and passes those
files to repository-owned Ansible playbooks. Ansible is the deployment source of
truth for copying Compose files, staging runtime secrets, managing firewall
rules, rendering NGINX config, obtaining TLS certificates, and running Docker
Compose in dependency order.
The full sequential workflow validates:
Workflows materialize temporary runtime files and copy them into the server
runtime layout over SSH.
Expected production locations:
/srv/apps/
/srv/data/
/srv/logs/
/srv/secrets/runtime/
/srv/backups/
The workflow verifies expected containers are running, including:
nginx-private
nginx-public
seaweedfs
loki
prometheus
llm-postgres
litellm
langfuse
wiki
wiki-database
LLM service endpoints:
LiteLLM http://litellm.victus.io
Langfuse http://langfuse.victus.io
Postgres internal Docker network only
The llm deploy does not publish LiteLLM or Langfuse service ports directly.
Private NGINX binds to TAILSCALE_IPV4 and proxies to the services over
infra_shared_backend.
Wiki.js is published through nginx-public and proxies to wiki:3000 over
infra_shared_backend. The default production hostname is:
https://wiki.victus.fit
If nginx-private also binds port 80, sync-core-dns.sh derives
NGINX_PUBLIC_BIND_IP from the VPS public route so public and private NGINX do
not compete for the same host socket. Override NGINX_PUBLIC_BIND_IP in
CORE_RUNTIME_ENV only when the host has multiple public IPv4 addresses.
To preserve an existing Wiki.js database from the previous media stack, set
this in WIKI_RUNTIME_ENV before deploy:
WIKIJS_DB_DATA_LOCATION=/srv/data/media/wiki/postgres
After the first Langfuse login, create a Langfuse project, generate API keys,
update LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY in Infisical, and
rerun the deployment.
LiteLLM deployments are defined in Infisical through LITELLM_DEPLOYMENTS_JSON.
Each entry references a provider key by env var name:
[
{
"model_name": "gemini-flash-lite",
"model": "gemini/gemini-3.1-flash-lite",
"api_key_env": "KEY_GEMINI_FLASH_LITE_01",
"rpm": 15,
"tpm": 100000
}
]
To add a Gemini key:
KEY_GEMINI_FLASH_LITE_NN in Infisical under /Hetzner-Server/api-keys.To change limits, edit rpm or tpm in LITELLM_DEPLOYMENTS_JSON and rerun
the deploy workflow. Ansible regenerates:
/srv/apps/llm/litellm/config.yaml
LiteLLM uses simple-shuffle routing, so multiple entries with the same
model_name balance/fail over across deployments.
Validate through the private endpoint:
curl http://litellm.victus.io/v1/models \
-H "Authorization: Bearer <LITELLM_VIRTUAL_KEY>"
For request traces, inspect the Langfuse project linked by
LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY.
Manual inspection:
ssh carlos@<PROD_HOST> "docker ps --all"
gh run list --workflow=deploy-all.yml
gh run view <run-id> --log
Prefer redeploying a known-good git ref through the deployment workflow.
Manual rollback on the host should be treated as an emergency operation and
followed by a repository-backed deployment to restore source-of-truth
alignment.