# Bilko Operational Runbook — Azure Container Apps

# Bilko — Operational Runbook

> **Azure is canonical.** Bilko runs on **Azure Container Apps** (swedencentral, resource group
> `rg-bilko-demo`, subscription `5b0b4d9b`). The previous non-Azure environment this
> runbook described is **dead** — legacy billing was exhausted 2026-06-14 (MC #103633).
> Platform-specific legacy references have been removed below.
>
> **Deep architecture reference (source of truth for topology/diagrams):** `DEPLOY-MAP.md` (root)
> **Environment model:** `docs/infrastructure/ENVIRONMENTS.md`
> **Live CI/CD config:** `azure-pipelines.yml` (root)

**Version:** 4.0 (Azure Container Apps era — B4 doc cleanup, MC #101088)
**Last updated:** 2026-07-28
**Supersedes:** v3.1 (2026-04-30, pre-Azure era)

---

## Table of Contents

1. [System Overview](#1-system-overview)
2. [Pre-Deployment Checklist](#2-pre-deployment-checklist)
3. [Deploying the API](#3-deploying-the-api)
4. [Deploying the Web Frontend](#4-deploying-the-web-frontend)
5. [Database Operations](#5-database-operations)
6. [Rollback Procedures](#6-rollback-procedures)
7. [Database Server Operations (Azure PostgreSQL)](#7-database-server-operations-azure-postgresql)
8. [Troubleshooting Guide](#8-troubleshooting-guide)
9. [Monitoring and Alerts](#9-monitoring-and-alerts)
10. [Emergency Contacts and Escalation](#10-emergency-contacts-and-escalation)

---

## 1. System Overview

### Architecture

Bilko is a Turborepo monorepo with two deployed apps, both running as **Azure Container Apps**:

- **Web** (`apps/web/`) — Next.js 15 frontend, ACA app `bilko-web-demo` (+ `bilko-web-stage`)
- **API** (`apps/api/`) — Kotlin/Ktor + Exposed backend, ACA app `bilko-api-demo` (+ `bilko-api-stage`). This is the sole canonical backend (ADR-020/ADR-021) — the old Express `apps/api-express/` was deleted 2026-05-02 (MC #10493).
- **Database** — Azure PostgreSQL flexible server `bilko-demo-pg` (PostgreSQL 16, swedencentral)

Full topology diagram (Cloudflare edge → ACA → Azure PG) lives in `DEPLOY-MAP.md` — do not duplicate it here; that file is kept current on every infra change.

### Apps in This Monorepo

| App | Path        | Port (dev) | Deployed to (ACA app name)           |
| --- | ----------- | ---------- | ------------------------------------ |
| Web | `apps/web/` | 3000       | `bilko-web-demo` / `bilko-web-stage` |
| API | `apps/api/` | 8080       | `bilko-api-demo` / `bilko-api-stage` |
| E2E | `apps/e2e/` | —          | CI only (Playwright, stage FQDNs)    |

### Key Packages

| Package  | Path                                        | Purpose                                         |
| -------- | ------------------------------------------- | ----------------------------------------------- |
| Database | `apps/api/src/main/resources/db/migration/` | Flyway SQL migrations (canonical schema source) |
| UI       | `packages/ui/`                              | Shared UI components                            |

### CI/CD Pipeline

**Single canonical pipeline:** `azure-pipelines.yml` (root), Azure DevOps `Bilko-CI-CD` (definitionId=1), org `alai-holding`, project `Bilko`, git remote `azdo`. Triggers on push to `main` and tag `v*`. GitHub Actions is dead/abandoned — do not rely on `.github/workflows/*` for deploys.

Stages: `CI_Gates` (8 jobs) → `Build` → `Flyway_Migrate` → `Deploy_Stage` → `E2E_UAT` → `Promote_Demo` (manual CEO approval). Full stage-by-stage detail: `DEPLOY-MAP.md` § CI/CD Pipeline.

---

## 2. Pre-Deployment Checklist

Run this before EVERY production (`Promote_Demo`) deployment:

- [ ] `CI_Gates` stage is green on the branch (Prettier, TSC, Vitest, Kotest, Trivy FS, Gitleaks, Semgrep)
- [ ] `Build` stage produced digest-pinned images in `bilkodemo.azurecr.io`
- [ ] `Flyway_Migrate` succeeded against `bilko-demo-pg` — no failed migrations in `flyway_schema_history`
- [ ] `Deploy_Stage` + `E2E_UAT` passed against `bilko-*-stage`
- [ ] If schema changed: new `V{N+1}__description.sql` exists in `apps/api/src/main/resources/db/migration/` — NEVER edit an existing migration
- [ ] No open Trivy/Semgrep HIGH/CRITICAL findings introduced by this change
- [ ] Environment variables/secrets verified in Azure Key Vault `kv-bilko-demo2` (via ADO variable group `bilko-kv-demo`)
- [ ] ZAKON PI2 six hard checks ready to run post-deploy (see `~/system/rules/zakon-pi2-deploy-verification.md`)

---

## 3. Deploying the API

**API source:** `apps/api/` (Kotlin + Ktor + Exposed)
**Deploy target:** Azure Container App `bilko-api-demo` (+ `bilko-api-stage`) — resource group `rg-bilko-demo`, environment `bilko-demo-env`
**Build pipeline:** `azure-pipelines.yml` `Build` stage — `docker buildx` linux/amd64 → Azure Container Registry
**Docker image:** `bilkodemo.azurecr.io/bilko-api:demo-{SHA}` (production) / `bilkodemo.azurecr.io/bilko-api:stage-{SHA}` (stage)

### 3a. Automatic Deploy (Stage — on every `main` push)

`Deploy_Stage` runs automatically on the `bilko-selfhosted` pool (FORGE agent `bilko-forge-1`) after `Build` + `Flyway_Migrate` succeed. It updates `bilko-api-stage` and `bilko-web-stage` to the digest-pinned image from `Build`.

Monitor: `az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko --pipeline-ids 1 --top 5` or the Azure DevOps pipeline UI.

### 3b. Promote to Production (CEO-gated manual approval)

`Promote_Demo` requires manual CEO approval in the ADO environment `bilko-demo`, then rebuilds `demo-{SHA}` images from the same commit and shifts `bilko-api-demo` ingress traffic 100% to the new revision (multiple-revisions mode, MC #105368).

```bash
# Trigger a manual pipeline run when Promote_Demo did not auto-open (BatchedCI can skip it)
az pipelines run --org https://dev.azure.com/alai-holding --project Bilko --branch main --id 1

# Approve via Azure DevOps UI (Pipelines > Environments > bilko-demo) once queued,
# or via API per CEO chat-authorization (see project memory for the API-approve pattern).
```

### 3c. Verify API Deployment

```bash
# Azure Container Apps status + FQDN
az containerapp list -g rg-bilko-demo \
  --query "[].{name:name, state:properties.provisioningState, fqdn:properties.configuration.ingress.fqdn}" \
  -o table

# Latest revision
az containerapp revision list -n bilko-api-demo -g rg-bilko-demo --query "[0].name" -o tsv

# Health check (brand domain, via CF Worker)
curl -s https://api.bilko.cloud/api/v1/health
curl -s https://api.bilko.io/api/v1/health
curl -s https://api.bilko.company/api/v1/health

# Direct ACA FQDN health check
curl -s https://bilko-api-demo.purplebeach-f004d490.swedencentral.azurecontainerapps.io/api/v1/health
```

### 3d. Required Secrets / Environment Variables (API)

Managed as Azure Container Apps secrets + env vars, sourced from Key Vault `kv-bilko-demo2` (variable group `bilko-kv-demo` in ADO). ACA write-only secrets return NULL on read — always cross-check against Key Vault, never assume a value from a stale doc.

| Secret / Var           | Description                                                                     |
| ---------------------- | ------------------------------------------------------------------------------- |
| `db-password`          | `bilko_admin` Azure PostgreSQL password                                         |
| `jwt-secret`           | Access token signing secret                                                     |
| `jwt-refresh-secret`   | Refresh token signing secret                                                    |
| `groq-api-key`         | LLM provider key (API)                                                          |
| `resend-api-key`       | Transactional email provider key                                                |
| `field-encryption-key` | PII field-level encryption key                                                  |
| `field-hmac-key`       | PII field-level HMAC key                                                        |
| `CORS_ORIGINS`         | Allowed origins — see `DEPLOY-MAP.md` OCD-12 for the stage custom-domain gotcha |

---

## 4. Deploying the Web Frontend

**Frontend source:** `apps/web/` (Next.js 15)
**Deploy target:** Azure Container App `bilko-web-demo` (+ `bilko-web-stage`)
**Build pipeline:** Same `azure-pipelines.yml` `Build` stage — builds `apps/web/` alongside the API in one pipeline run
**Production domains:** `app.bilko.io`, `app.bilko.cloud`, `app.bilko.company` (routed through Cloudflare Worker `bilko-edge-proxy`, which rewrites Host/SNI to the ACA FQDN)

### 4a. Automatic Deploy (Stage) / 4b. Promote to Production

Same pipeline and same manual-approval gate as the API (§3a/§3b) — web and API are built and promoted together from one Azure DevOps run.

### 4c. Required Environment Variables (Web)

Managed via ACA env vars on `bilko-web-demo` / `bilko-web-stage`.

| Variable              | Description                                                                                          |
| --------------------- | ---------------------------------------------------------------------------------------------------- |
| `NEXT_PUBLIC_API_URL` | API base URL — currently baked at build time (see DEPLOY-MAP.md OCD-9 for the runtime-URL follow-up) |
| `NEXT_PUBLIC_APP_ENV` | `production` or `staging`                                                                            |

### 4d. Verify Web Deployment

```bash
az containerapp revision list -n bilko-web-demo -g rg-bilko-demo --query "[0].name" -o tsv

curl -sI https://app.bilko.cloud/ | head -1
curl -sI https://app.bilko.io/ | head -1
curl -sI https://app.bilko.company/ | head -1
```

---

## 5. Database Operations

**Migration tool:** Flyway 11 (NOT Prisma — Prisma was tied to the deleted Express backend). Migration files live in `apps/api/src/main/resources/db/migration/V*.sql`.

### 5a. Running Migrations

Migrations run automatically in the `Flyway_Migrate` pipeline stage against `bilko-demo-pg` on every `main` push, before `Deploy_Stage`.

Manual run (rare — CI is the default path):

```bash
docker run --rm \
  -v "$(pwd)/apps/api/src/main/resources/db/migration:/flyway/sql" \
  flyway/flyway:11 \
  -url="jdbc:postgresql://bilko-demo-pg.postgres.database.azure.com:5432/bilko?sslmode=require" \
  -user=bilko_admin \
  -password="$DB_PASSWORD" \
  migrate
```

**RULE:** Never edit an existing `V*.sql` migration. Always create a new `V{N+1}__description.sql`. Reserve the version number carefully — two independent branches taking the same `V` number is a recurring collision (see `technical_bilko_promote_manual_and_flyway_collisions_2026-07-20.md`); check `git ls-tree azdo/main -- apps/api/src/main/resources/db/migration/` before picking a number.

### 5b. Checking Migration Status

```bash
psql "host=bilko-demo-pg.postgres.database.azure.com port=5432 dbname=bilko user=bilko_admin sslmode=require" \
  -c "SELECT version, description, success FROM flyway_schema_history ORDER BY installed_rank DESC LIMIT 20;"
```

All rows should show `success = t`.

### 5c. Rolling Back a Migration

Flyway (community edition, as used here) does not support automatic migration rollback.

**Option 1 (preferred):** Write a new compensating migration that undoes the change (new `V{N+1}` file with inverse DDL/DML).

**Option 2 (last resort):** Restore from an Azure PostgreSQL backup (§7f). Data written after the migration will be lost.

### 5d. Database Backup and Restore

Azure PostgreSQL flexible server provides automatic backups (default retention — verify current setting via `az postgres flexible-server show`). For an on-demand logical backup:

```bash
pg_dump "host=bilko-demo-pg.postgres.database.azure.com port=5432 dbname=bilko user=bilko_admin sslmode=require" \
  --format=custom --no-acl --no-owner \
  --file=bilko-backup-$(date +%Y%m%d-%H%M%S).dump
```

Restore:

```bash
pg_restore --dbname="host=... user=bilko_admin sslmode=require" \
  --no-acl --no-owner \
  bilko-backup-YYYYMMDD-HHMMSS.dump
```

---

## 6. Rollback Procedures

### 6a. API Rollback (Azure Container Apps)

```bash
# List revisions, newest first
az containerapp revision list --name bilko-api-demo --resource-group rg-bilko-demo \
  --query "[].{name:name,created:properties.createdTime,active:properties.active}" -o table

# Shift 100% traffic to the last known-good revision
az containerapp ingress traffic set --name bilko-api-demo --resource-group rg-bilko-demo \
  --revision-weight <previous-revision-name>=100
```

### 6b. Web Rollback (Azure Container Apps)

```bash
az containerapp revision list --name bilko-web-demo --resource-group rg-bilko-demo \
  --query "[].{name:name,created:properties.createdTime,active:properties.active}" -o table

az containerapp ingress traffic set --name bilko-web-demo --resource-group rg-bilko-demo \
  --revision-weight <previous-revision-name>=100
```

### 6c. Edge Rollback (Cloudflare)

```bash
# Worker (bilko-edge-proxy) rollback
cd apps/edge-proxy && wrangler rollback

# CF Pages (marketing landings) — via dashboard:
# Pages > project (bilko-io / bilko-cloud / bilko-company) > Deployments > "Rollback to this deployment"
```

### 6d. Database Rollback

**IMPORTANT:** Database rollbacks are destructive if data was written after the migration ran. Always confirm with the team before rolling back the database.

Steps:

1. Take a current-state backup first (§5d)
2. Write a compensating migration (preferred over restore — see §5c Option 1)
3. Only restore from backup if data loss is acceptable and a pre-migration backup exists

---

## 7. Database Server Operations (Azure PostgreSQL)

**Database:** Azure PostgreSQL flexible server `bilko-demo-pg`, B1ms, PostgreSQL 16, region swedencentral, resource group `rg-bilko-demo`
**Schema source of truth:** Flyway migrations, `apps/api/src/main/resources/db/migration/`
**App user:** `bilko_admin` (NOT `bilko`) — password in ACA secret `db-password` / Key Vault `kv-bilko-demo2`
**Shared instance:** stage and demo currently share this one server (see `DEPLOY-MAP.md` OCD-3 for the isolation tradeoff)

### 7a. Connecting (Local / Admin)

```bash
psql "host=bilko-demo-pg.postgres.database.azure.com port=5432 dbname=bilko user=bilko_admin sslmode=require"
```

Firewall: FORGE's public IP is allow-listed (rule `forge-ci-runner`); Microsoft-hosted CI agents use the `allow-azure-services` rule. Connecting from an unlisted IP will time out — add a temporary firewall rule via `az postgres flexible-server firewall-rule create` rather than widening `allow-azure-services`.

### 7b. Running the RLS Audit

```bash
# Via test suite (preferred)
./gradlew :apps:api:test --tests "*RlsAudit*"
```

Tables that MUST have RLS enabled include (non-exhaustive — see the audit test for the authoritative list): `organizations`, `users`, `invoices`, `invoice_items`, `expenses`, `transactions`, `bank_accounts`, `bank_transactions`, `contacts`, `logged_actions`.

### 7c. Verifying RLS Policies in PostgreSQL

```sql
-- List all RLS policies
SELECT schemaname, tablename, policyname, cmd, qual
FROM pg_policies
ORDER BY tablename, policyname;

-- Check which tables have RLS enabled
SELECT relname AS table_name, relrowsecurity AS rls_enabled
FROM pg_class
WHERE relnamespace = 'public'::regnamespace AND relkind = 'r'
ORDER BY relname;
```

### 7d. Enabling RLS on a Table

```sql
ALTER TABLE public.table_name ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.table_name FORCE ROW LEVEL SECURITY;

-- Example org-scoped policy
CREATE POLICY org_isolation ON public.table_name
  USING (organization_id = current_setting('app.current_org_id')::uuid);
```

### 7e. Backup and Restore (Azure PostgreSQL server-level)

```bash
# Server status and current config
az postgres flexible-server show -g rg-bilko-demo -n bilko-demo-pg \
  --query "{name:name, state:state, version:version, location:location}" -o table

# Point-in-time restore to a new server (does NOT overwrite the source — verify before promoting)
az postgres flexible-server restore \
  --resource-group rg-bilko-demo \
  --name bilko-demo-pg-restored \
  --source-server bilko-demo-pg \
  --restore-time "2026-07-28T00:00:00Z"
```

Also see the logical `pg_dump`/`pg_restore` path in §5d for a single-database, portable backup.

---

## 8. Troubleshooting Guide

### API Down

**Symptoms:** `GET /api/v1/health` returns non-200 or times out.

**Steps:**

1. Check Azure status for the region: `az containerapp show -n bilko-api-demo -g rg-bilko-demo --query "properties.provisioningState"`
2. Check container logs: `az containerapp logs show -n bilko-api-demo -g rg-bilko-demo --tail 100`
3. Check recent revisions for a bad deploy: `az containerapp revision list -n bilko-api-demo -g rg-bilko-demo -o table`
4. If a bad revision is live, roll back (§6a)
5. Check the Cloudflare Worker (`bilko-edge-proxy`) isn't misrouting — `curl` the direct ACA FQDN to isolate edge vs. origin
6. If not resolved in 10 minutes: rollback (§6) and open a P1 per §10

### High Error Rate

**Symptoms:** More than 5% of API requests returning 5xx errors.

**Steps:**

1. `az containerapp logs show -n bilko-api-demo -g rg-bilko-demo --tail 200` — look for error patterns
2. Check Application Insights / Azure Monitor (if wired for this service) for new error groups
3. Health check: `curl https://api.bilko.cloud/api/v1/health`
4. Check if a Flyway migration ran recently — look for schema mismatches in the logs
5. Check for `too many connections` errors (see §8 DB Connection Pool Exhausted below)
6. Restart the revision if connection-pool errors persist: `az containerapp revision restart -n bilko-api-demo -g rg-bilko-demo --revision <name>`

### Database Unreachable

**Symptoms:** Health endpoint shows `"database": "error"`, connection errors in logs.

**Steps:**

1. `az postgres flexible-server show -g rg-bilko-demo -n bilko-demo-pg --query "state"`
2. Verify the `db-password` secret / connection string is current: `az containerapp secret list -n bilko-api-demo -g rg-bilko-demo` (values are NULL by design — cross-check against Key Vault `kv-bilko-demo2`, not against this doc)
3. Test connection directly: `psql "host=bilko-demo-pg.postgres.database.azure.com ... sslmode=require" -c "SELECT 1;"`
4. Check firewall rules haven't been tightened/removed: `az postgres flexible-server firewall-rule list -g rg-bilko-demo -n bilko-demo-pg -o table`
5. Check connection pool exhaustion (below)

### DB Connection Pool Exhausted

**Symptoms:** `too many connections` errors, requests queuing indefinitely.

**Steps:**

1. Check current connections:
   ```sql
   SELECT count(*), state FROM pg_stat_activity GROUP BY state;
   ```
2. Find long-running queries:
   ```sql
   SELECT pid, now() - query_start AS duration, query
   FROM pg_stat_activity
   WHERE state = 'active' AND now() - query_start > interval '30 seconds';
   ```
3. Kill a stuck query if safe:
   ```sql
   SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid = target_pid;
   ```
4. Restart the API revision to release the connection pool (`az containerapp revision restart`)
5. Consider PgBouncer / connection-pooling middleware if this recurs under load

### Slow Responses

**Symptoms:** p95 response time above 2 seconds.

**Steps:**

1. Check for slow queries:
   ```sql
   SELECT query, calls, mean_exec_time
   FROM pg_stat_statements
   ORDER BY mean_exec_time DESC LIMIT 20;
   ```
2. Check for missing indexes on frequently filtered columns
3. Look for N+1 query patterns in Exposed usage
4. Use `EXPLAIN ANALYZE` on suspect queries

### SSL Certificate Issues

```bash
echo | openssl s_client -connect api.bilko.cloud:443 2>/dev/null | openssl x509 -noout -dates
```

- **Cloudflare-fronted domains** (`app.*`, `api.*`, `bilko.io/.cloud/.company`): CF-managed, auto-renews — check Cloudflare dashboard if issues.
- **Stage custom domains** (`web-stage.bilko.cloud`, `api-stage.bilko.cloud`): ACA-managed certs (`SniEnabled`), direct DNS-only CNAME (not proxied) — check `az containerapp hostname list -n bilko-api-stage -g rg-bilko-demo` for cert binding state.

### Deployment Fails in CI

Common causes:

| Failure                           | Fix                                                                                   |
| --------------------------------- | ------------------------------------------------------------------------------------- |
| `CI_Gates` lint/TSC fail          | Fix and push — see which of the 8 jobs failed in the ADO run                          |
| `Trivy` HIGH/CRITICAL             | Update the dependency or image base, re-run                                           |
| `Flyway_Migrate` fails            | Bad SQL or a version-number collision (see §5a) — never edit an applied migration     |
| `Deploy_Stage` health check fails | New revision not passing `/api/v1/health` or `/` within the 20s wait — check logs     |
| `E2E_UAT` skips                   | Usually a stale `BILKO_E2E_TOKEN_SECRET` in `bilko-kv-demo` — see DEPLOY-MAP.md OCD-1 |

---

## 9. Monitoring and Alerts

See `monitoring/` directory:

- `monitoring/uptime.json` — uptime check configuration
- `monitoring/alert-rules.json` — alert thresholds and routing
- `monitoring/metrics.md` — key metrics with targets

**Quick health commands:**

```bash
# Production API (brand domains)
curl -s https://api.bilko.cloud/api/v1/health
curl -s https://api.bilko.io/api/v1/health
curl -s https://api.bilko.company/api/v1/health

# Production Web
curl -s -o /dev/null -w "HTTP %{http_code} in %{time_total}s\n" https://app.bilko.cloud

# Stage
curl -s https://api-stage.bilko.cloud/api/v1/health
curl -sI https://web-stage.bilko.cloud/ | head -1
```

**Log locations:**

| Component        | Log location                                                                                   |
| ---------------- | ---------------------------------------------------------------------------------------------- |
| API (ACA)        | `az containerapp logs show -n bilko-api-demo -g rg-bilko-demo`                                 |
| Web (ACA)        | `az containerapp logs show -n bilko-web-demo -g rg-bilko-demo`                                 |
| DB (Azure PG)    | Azure Portal > `bilko-demo-pg` > Monitoring > Logs, or `pg_stat_activity`/`pg_stat_statements` |
| CI/CD            | Azure DevOps > `Bilko-CI-CD` pipeline > run > stage logs                                       |
| Edge (CF Worker) | `wrangler tail` against `bilko-edge-proxy`, or Cloudflare dashboard                            |

---

## 10. Emergency Contacts and Escalation

| Role               | Contact                     | Method         |
| ------------------ | --------------------------- | -------------- |
| On-call / CEO      | Alem Basic (alem@alai.no)   | Direct / Slack |
| Azure support      | portal.azure.com support    | Web            |
| Azure DevOps       | dev.azure.com/alai-holding  | Web            |
| Cloudflare support | dash.cloudflare.com support | Web            |

**Escalation path:**

- **P1 (Production down):** Fix immediately per §8 Troubleshooting; notify Alem directly if not resolved within 15 min.
- **P2 (Degraded performance):** Investigate per §8; notify Alem if unresolved within 1 hour.
- **P3 (Non-urgent):** Log as an MC task (`mc.js add`), handle in the next work cycle.

---

_Last updated: 2026-07-28 — B4 doc cleanup pass (MC #101088): rewritten for Azure Container Apps (all legacy platform and ORM references removed; content grounded against `DEPLOY-MAP.md` and `apps/api/src/main/resources/db/migration/`)._