Bilko Operational Runbook — Azure Container Apps

Bilko — Operational Runbook

Azure is canonical. Bilko runs on Azure Container Apps (swedencentral, resource group rg-bilko-demo, subscription 5b0b4d9b). The previous non-Azure environment this runbook described is dead — legacy billing was exhausted 2026-06-14 (MC #103633). Platform-specific legacy references have been removed below.

Deep architecture reference (source of truth for topology/diagrams): DEPLOY-MAP.md (root) Environment model: docs/infrastructure/ENVIRONMENTS.md Live CI/CD config: azure-pipelines.yml (root)

Version: 4.0 (Azure Container Apps era — B4 doc cleanup, MC #101088) Last updated: 2026-07-28 Supersedes: v3.1 (2026-04-30, pre-Azure era)

Table of Contents

  1. System Overview
  2. Pre-Deployment Checklist
  3. Deploying the API
  4. Deploying the Web Frontend
  5. Database Operations
  6. Rollback Procedures
  7. Database Server Operations (Azure PostgreSQL)
  8. Troubleshooting Guide
  9. Monitoring and Alerts
  10. Emergency Contacts and Escalation

1. System Overview

Architecture

Bilko is a Turborepo monorepo with two deployed apps, both running as Azure Container Apps:

Full topology diagram (Cloudflare edge → ACA → Azure PG) lives in DEPLOY-MAP.md — do not duplicate it here; that file is kept current on every infra change.

Apps in This Monorepo

App Path Port (dev) Deployed to (ACA app name)
Web apps/web/ 3000 bilko-web-demo / bilko-web-stage
API apps/api/ 8080 bilko-api-demo / bilko-api-stage
E2E apps/e2e/ — CI only (Playwright, stage FQDNs)

Key Packages

Package Path Purpose
Database apps/api/src/main/resources/db/migration/ Flyway SQL migrations (canonical schema source)
UI packages/ui/ Shared UI components

CI/CD Pipeline

Single canonical pipeline: azure-pipelines.yml (root), Azure DevOps Bilko-CI-CD (definitionId=1), org alai-holding, project Bilko, git remote azdo. Triggers on push to main and tag v*. GitHub Actions is dead/abandoned — do not rely on .github/workflows/* for deploys.

Stages: CI_Gates (8 jobs) → Build → Flyway_Migrate → Deploy_Stage → E2E_UAT → Promote_Demo (manual CEO approval). Full stage-by-stage detail: DEPLOY-MAP.md § CI/CD Pipeline.

2. Pre-Deployment Checklist

Run this before EVERY production (Promote_Demo) deployment:

3. Deploying the API

API source: apps/api/ (Kotlin + Ktor + Exposed) Deploy target: Azure Container App bilko-api-demo (+ bilko-api-stage) — resource group rg-bilko-demo, environment bilko-demo-env Build pipeline: azure-pipelines.yml Build stage — docker buildx linux/amd64 → Azure Container Registry Docker image: bilkodemo.azurecr.io/bilko-api:demo-{SHA} (production) / bilkodemo.azurecr.io/bilko-api:stage-{SHA} (stage)

3a. Automatic Deploy (Stage — on every main push)

Deploy_Stage runs automatically on the bilko-selfhosted pool (FORGE agent bilko-forge-1) after Build + Flyway_Migrate succeed. It updates bilko-api-stage and bilko-web-stage to the digest-pinned image from Build.

Monitor: az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko --pipeline-ids 1 --top 5 or the Azure DevOps pipeline UI.

3b. Promote to Production (CEO-gated manual approval)

Promote_Demo requires manual CEO approval in the ADO environment bilko-demo, then rebuilds demo-{SHA} images from the same commit and shifts bilko-api-demo ingress traffic 100% to the new revision (multiple-revisions mode, MC #105368).

# Trigger a manual pipeline run when Promote_Demo did not auto-open (BatchedCI can skip it)
az pipelines run --org https://dev.azure.com/alai-holding --project Bilko --branch main --id 1

# Approve via Azure DevOps UI (Pipelines > Environments > bilko-demo) once queued,
# or via API per CEO chat-authorization (see project memory for the API-approve pattern).

3c. Verify API Deployment

# Azure Container Apps status + FQDN
az containerapp list -g rg-bilko-demo \
  --query "[].{name:name, state:properties.provisioningState, fqdn:properties.configuration.ingress.fqdn}" \
  -o table

# Latest revision
az containerapp revision list -n bilko-api-demo -g rg-bilko-demo --query "[0].name" -o tsv

# Health check (brand domain, via CF Worker)
curl -s https://api.bilko.cloud/api/v1/health
curl -s https://api.bilko.io/api/v1/health
curl -s https://api.bilko.company/api/v1/health

# Direct ACA FQDN health check
curl -s https://bilko-api-demo.purplebeach-f004d490.swedencentral.azurecontainerapps.io/api/v1/health

3d. Required Secrets / Environment Variables (API)

Managed as Azure Container Apps secrets + env vars, sourced from Key Vault kv-bilko-demo2 (variable group bilko-kv-demo in ADO). ACA write-only secrets return NULL on read — always cross-check against Key Vault, never assume a value from a stale doc.

Secret / Var Description
db-password bilko_admin Azure PostgreSQL password
jwt-secret Access token signing secret
jwt-refresh-secret Refresh token signing secret
groq-api-key LLM provider key (API)
resend-api-key Transactional email provider key
field-encryption-key PII field-level encryption key
field-hmac-key PII field-level HMAC key
CORS_ORIGINS Allowed origins — see DEPLOY-MAP.md OCD-12 for the stage custom-domain gotcha

4. Deploying the Web Frontend

Frontend source: apps/web/ (Next.js 15) Deploy target: Azure Container App bilko-web-demo (+ bilko-web-stage) Build pipeline: Same azure-pipelines.yml Build stage — builds apps/web/ alongside the API in one pipeline run Production domains: app.bilko.io, app.bilko.cloud, app.bilko.company (routed through Cloudflare Worker bilko-edge-proxy, which rewrites Host/SNI to the ACA FQDN)

4a. Automatic Deploy (Stage) / 4b. Promote to Production

Same pipeline and same manual-approval gate as the API (§3a/§3b) — web and API are built and promoted together from one Azure DevOps run.

4c. Required Environment Variables (Web)

Managed via ACA env vars on bilko-web-demo / bilko-web-stage.

Variable Description
NEXT_PUBLIC_API_URL API base URL — currently baked at build time (see DEPLOY-MAP.md OCD-9 for the runtime-URL follow-up)
NEXT_PUBLIC_APP_ENV production or staging

4d. Verify Web Deployment

az containerapp revision list -n bilko-web-demo -g rg-bilko-demo --query "[0].name" -o tsv

curl -sI https://app.bilko.cloud/ | head -1
curl -sI https://app.bilko.io/ | head -1
curl -sI https://app.bilko.company/ | head -1

5. Database Operations

Migration tool: Flyway 11 (NOT Prisma — Prisma was tied to the deleted Express backend). Migration files live in apps/api/src/main/resources/db/migration/V*.sql.

5a. Running Migrations

Migrations run automatically in the Flyway_Migrate pipeline stage against bilko-demo-pg on every main push, before Deploy_Stage.

Manual run (rare — CI is the default path):

docker run --rm \
  -v "$(pwd)/apps/api/src/main/resources/db/migration:/flyway/sql" \
  flyway/flyway:11 \
  -url="jdbc:postgresql://bilko-demo-pg.postgres.database.azure.com:5432/bilko?sslmode=require" \
  -user=bilko_admin \
  -password="$DB_PASSWORD" \
  migrate

RULE: Never edit an existing V*.sql migration. Always create a new V{N+1}__description.sql. Reserve the version number carefully — two independent branches taking the same V number is a recurring collision (see technical_bilko_promote_manual_and_flyway_collisions_2026-07-20.md); check git ls-tree azdo/main -- apps/api/src/main/resources/db/migration/ before picking a number.

5b. Checking Migration Status

psql "host=bilko-demo-pg.postgres.database.azure.com port=5432 dbname=bilko user=bilko_admin sslmode=require" \
  -c "SELECT version, description, success FROM flyway_schema_history ORDER BY installed_rank DESC LIMIT 20;"

All rows should show success = t.

5c. Rolling Back a Migration

Flyway (community edition, as used here) does not support automatic migration rollback.

Option 1 (preferred): Write a new compensating migration that undoes the change (new V{N+1} file with inverse DDL/DML).

Option 2 (last resort): Restore from an Azure PostgreSQL backup (§7f). Data written after the migration will be lost.

5d. Database Backup and Restore

Azure PostgreSQL flexible server provides automatic backups (default retention — verify current setting via az postgres flexible-server show). For an on-demand logical backup:

pg_dump "host=bilko-demo-pg.postgres.database.azure.com port=5432 dbname=bilko user=bilko_admin sslmode=require" \
  --format=custom --no-acl --no-owner \
  --file=bilko-backup-$(date +%Y%m%d-%H%M%S).dump

Restore:

pg_restore --dbname="host=... user=bilko_admin sslmode=require" \
  --no-acl --no-owner \
  bilko-backup-YYYYMMDD-HHMMSS.dump

6. Rollback Procedures

6a. API Rollback (Azure Container Apps)

# List revisions, newest first
az containerapp revision list --name bilko-api-demo --resource-group rg-bilko-demo \
  --query "[].{name:name,created:properties.createdTime,active:properties.active}" -o table

# Shift 100% traffic to the last known-good revision
az containerapp ingress traffic set --name bilko-api-demo --resource-group rg-bilko-demo \
  --revision-weight <previous-revision-name>=100

6b. Web Rollback (Azure Container Apps)

az containerapp revision list --name bilko-web-demo --resource-group rg-bilko-demo \
  --query "[].{name:name,created:properties.createdTime,active:properties.active}" -o table

az containerapp ingress traffic set --name bilko-web-demo --resource-group rg-bilko-demo \
  --revision-weight <previous-revision-name>=100

6c. Edge Rollback (Cloudflare)

# Worker (bilko-edge-proxy) rollback
cd apps/edge-proxy && wrangler rollback

# CF Pages (marketing landings) — via dashboard:
# Pages > project (bilko-io / bilko-cloud / bilko-company) > Deployments > "Rollback to this deployment"

6d. Database Rollback

IMPORTANT: Database rollbacks are destructive if data was written after the migration ran. Always confirm with the team before rolling back the database.

Steps:

  1. Take a current-state backup first (§5d)
  2. Write a compensating migration (preferred over restore — see §5c Option 1)
  3. Only restore from backup if data loss is acceptable and a pre-migration backup exists

7. Database Server Operations (Azure PostgreSQL)

Database: Azure PostgreSQL flexible server bilko-demo-pg, B1ms, PostgreSQL 16, region swedencentral, resource group rg-bilko-demo Schema source of truth: Flyway migrations, apps/api/src/main/resources/db/migration/ App user: bilko_admin (NOT bilko) — password in ACA secret db-password / Key Vault kv-bilko-demo2 Shared instance: stage and demo currently share this one server (see DEPLOY-MAP.md OCD-3 for the isolation tradeoff)

7a. Connecting (Local / Admin)

psql "host=bilko-demo-pg.postgres.database.azure.com port=5432 dbname=bilko user=bilko_admin sslmode=require"

Firewall: FORGE's public IP is allow-listed (rule forge-ci-runner); Microsoft-hosted CI agents use the allow-azure-services rule. Connecting from an unlisted IP will time out — add a temporary firewall rule via az postgres flexible-server firewall-rule create rather than widening allow-azure-services.

7b. Running the RLS Audit

# Via test suite (preferred)
./gradlew :apps:api:test --tests "*RlsAudit*"

Tables that MUST have RLS enabled include (non-exhaustive — see the audit test for the authoritative list): organizations, users, invoices, invoice_items, expenses, transactions, bank_accounts, bank_transactions, contacts, logged_actions.

7c. Verifying RLS Policies in PostgreSQL

-- List all RLS policies
SELECT schemaname, tablename, policyname, cmd, qual
FROM pg_policies
ORDER BY tablename, policyname;

-- Check which tables have RLS enabled
SELECT relname AS table_name, relrowsecurity AS rls_enabled
FROM pg_class
WHERE relnamespace = 'public'::regnamespace AND relkind = 'r'
ORDER BY relname;

7d. Enabling RLS on a Table

ALTER TABLE public.table_name ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.table_name FORCE ROW LEVEL SECURITY;

-- Example org-scoped policy
CREATE POLICY org_isolation ON public.table_name
  USING (organization_id = current_setting('app.current_org_id')::uuid);

7e. Backup and Restore (Azure PostgreSQL server-level)

# Server status and current config
az postgres flexible-server show -g rg-bilko-demo -n bilko-demo-pg \
  --query "{name:name, state:state, version:version, location:location}" -o table

# Point-in-time restore to a new server (does NOT overwrite the source — verify before promoting)
az postgres flexible-server restore \
  --resource-group rg-bilko-demo \
  --name bilko-demo-pg-restored \
  --source-server bilko-demo-pg \
  --restore-time "2026-07-28T00:00:00Z"

Also see the logical pg_dump/pg_restore path in §5d for a single-database, portable backup.

8. Troubleshooting Guide

API Down

Symptoms: GET /api/v1/health returns non-200 or times out.

Steps:

  1. Check Azure status for the region: az containerapp show -n bilko-api-demo -g rg-bilko-demo --query "properties.provisioningState"
  2. Check container logs: az containerapp logs show -n bilko-api-demo -g rg-bilko-demo --tail 100
  3. Check recent revisions for a bad deploy: az containerapp revision list -n bilko-api-demo -g rg-bilko-demo -o table
  4. If a bad revision is live, roll back (§6a)
  5. Check the Cloudflare Worker (bilko-edge-proxy) isn't misrouting — curl the direct ACA FQDN to isolate edge vs. origin
  6. If not resolved in 10 minutes: rollback (§6) and open a P1 per §10

High Error Rate

Symptoms: More than 5% of API requests returning 5xx errors.

Steps:

  1. az containerapp logs show -n bilko-api-demo -g rg-bilko-demo --tail 200 — look for error patterns
  2. Check Application Insights / Azure Monitor (if wired for this service) for new error groups
  3. Health check: curl https://api.bilko.cloud/api/v1/health
  4. Check if a Flyway migration ran recently — look for schema mismatches in the logs
  5. Check for too many connections errors (see §8 DB Connection Pool Exhausted below)
  6. Restart the revision if connection-pool errors persist: az containerapp revision restart -n bilko-api-demo -g rg-bilko-demo --revision <name>

Database Unreachable

Symptoms: Health endpoint shows "database": "error", connection errors in logs.

Steps:

  1. az postgres flexible-server show -g rg-bilko-demo -n bilko-demo-pg --query "state"
  2. Verify the db-password secret / connection string is current: az containerapp secret list -n bilko-api-demo -g rg-bilko-demo (values are NULL by design — cross-check against Key Vault kv-bilko-demo2, not against this doc)
  3. Test connection directly: psql "host=bilko-demo-pg.postgres.database.azure.com ... sslmode=require" -c "SELECT 1;"
  4. Check firewall rules haven't been tightened/removed: az postgres flexible-server firewall-rule list -g rg-bilko-demo -n bilko-demo-pg -o table
  5. Check connection pool exhaustion (below)

DB Connection Pool Exhausted

Symptoms: too many connections errors, requests queuing indefinitely.

Steps:

  1. Check current connections: 
    SELECT count(*), state FROM pg_stat_activity GROUP BY state;
  2. Find long-running queries: 
    SELECT pid, now() - query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active' AND now() - query_start > interval '30 seconds';
  3. Kill a stuck query if safe: 
    SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid = target_pid;
  4. Restart the API revision to release the connection pool (az containerapp revision restart)
  5. Consider PgBouncer / connection-pooling middleware if this recurs under load

Slow Responses

Symptoms: p95 response time above 2 seconds.

Steps:

  1. Check for slow queries: 
    SELECT query, calls, mean_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC LIMIT 20;
  2. Check for missing indexes on frequently filtered columns
  3. Look for N+1 query patterns in Exposed usage
  4. Use EXPLAIN ANALYZE on suspect queries

SSL Certificate Issues

echo | openssl s_client -connect api.bilko.cloud:443 2>/dev/null | openssl x509 -noout -dates

Deployment Fails in CI

Common causes:

Failure Fix
CI_Gates lint/TSC fail Fix and push — see which of the 8 jobs failed in the ADO run
Trivy HIGH/CRITICAL Update the dependency or image base, re-run
Flyway_Migrate fails Bad SQL or a version-number collision (see §5a) — never edit an applied migration
Deploy_Stage health check fails New revision not passing /api/v1/health or / within the 20s wait — check logs
E2E_UAT skips Usually a stale BILKO_E2E_TOKEN_SECRET in bilko-kv-demo — see DEPLOY-MAP.md OCD-1

9. Monitoring and Alerts

See monitoring/ directory:

Quick health commands:

# Production API (brand domains)
curl -s https://api.bilko.cloud/api/v1/health
curl -s https://api.bilko.io/api/v1/health
curl -s https://api.bilko.company/api/v1/health

# Production Web
curl -s -o /dev/null -w "HTTP %{http_code} in %{time_total}s\n" https://app.bilko.cloud

# Stage
curl -s https://api-stage.bilko.cloud/api/v1/health
curl -sI https://web-stage.bilko.cloud/ | head -1

Log locations:

Component Log location
API (ACA) az containerapp logs show -n bilko-api-demo -g rg-bilko-demo
Web (ACA) az containerapp logs show -n bilko-web-demo -g rg-bilko-demo
DB (Azure PG) Azure Portal > bilko-demo-pg > Monitoring > Logs, or pg_stat_activity/pg_stat_statements
CI/CD Azure DevOps > Bilko-CI-CD pipeline > run > stage logs
Edge (CF Worker) wrangler tail against bilko-edge-proxy, or Cloudflare dashboard

10. Emergency Contacts and Escalation

Role Contact Method
On-call / CEO Alem Basic (alem@alai.no) Direct / Slack
Azure support portal.azure.com support Web
Azure DevOps dev.azure.com/alai-holding Web
Cloudflare support dash.cloudflare.com support Web

Escalation path:

Last updated: 2026-07-28 — B4 doc cleanup pass (MC #101088): rewritten for Azure Container Apps (all legacy platform and ORM references removed; content grounded against DEPLOY-MAP.md and apps/api/src/main/resources/db/migration/).

Revision #1
Created 2026-07-28 21:33:35 UTC by John
Updated 2026-07-28 21:33:35 UTC by John