# Deployment Runbooks

Operational runbooks for staging, demo, and production deployments across ALAI products

# Bilko Stage→Demo→Prod Cutover Runbook

# Bilko Stage→Demo→Prod Cutover Runbook

**Document ID:** BILKO-RUNBOOK-001  
**Last Updated:** 2026-05-10  
**Owner:** Skillforge (ALAI Holding AS)  
**Status:** Active

## Purpose

This runbook documents the promotion path from Bilko `stage` environment to `demo` and eventually `prod`. It incorporates all changes delivered in Rounds 11-27, including massive test coverage expansion, 7+ production bug fixes, RBAC enforcement, permanent CORS configuration, and database migrations V14-V20.

**Context:** As of 2026-05-10, Bilko stage is on revision `00185-fay` (image `stage-3a4a9e9`). Demo environment has only a subset of these changes. Prod does not exist as a separate environment yet and currently uses demo as customer-facing.

## 1. Pre-Cutover Checklist

Complete ALL items before proceeding with cutover:

<table id="bkmrk-item-verification-co"><thead><tr><th>Item</th><th>Verification Command</th><th>Success Criteria</th><th>Status</th></tr></thead><tbody><tr><td>**High/Blocker MCs Resolved**</td><td>`node ~/system/tools/mc.js list --project bilko --priority H,BLOCKER --status open`</td><td>0 open tasks, or all explicitly waived by CEO</td><td>☐</td></tr><tr><td>**Cloud Build Health**</td><td>`gcloud builds list --project=tribal-sign-487920-k0 --limit=5 --filter="source.repoSource.branchName=main"`</td><td>Last 5 builds on main = SUCCESS</td><td>☐</td></tr><tr><td>**Coverage Gate**</td><td>`cd apps/api && ./gradlew koverVerify`</td><td>Exit code 0 (current threshold ≥60%)</td><td>☐</td></tr><tr><td>**E2E Regression Tests**</td><td>`cd apps/web && npm run test:e2e`</td><td>Phase A pass rate ≥50% (Round 24 regression specs)</td><td>☐</td></tr><tr><td>**Database Backup**</td><td>`gcloud sql backups create --instance=bilko-demo-db --project=tribal-sign-487920-k0`</td><td>Backup ID returned + status=SUCCESSFUL</td><td>☐</td></tr><tr><td>**CEO Sign-Off**</td><td>MC task comments or Slack approval</td><td>Explicit approval captured with timestamp</td><td>☐</td></tr></tbody></table>

## 2. Database Migration Plan (V14-V20)

Flyway migrations are applied automatically when Cloud Run starts. This section documents migration order and critical dependencies.

### Migration Sequence

<table id="bkmrk-version-description-"><thead><tr><th>Version</th><th>Description</th><th>MC Reference</th><th>Critical Notes</th></tr></thead><tbody><tr><td>**V14**</td><td>Demo + CI test seed</td><td>Round 12.7</td><td>Creates demo@bilko.rs user + test tenant for CI</td></tr><tr><td>**V15**</td><td>Trial fields (plan\_tier, trial\_start, trial\_end)</td><td>MC #100326</td><td>Adds subscription trial mechanic to organizations table</td></tr><tr><td>**V16**</td><td>RLS column widening (country constraint)</td><td>MC #100406</td><td>Expands country VARCHAR to support TaxJurisdiction enum values</td></tr><tr><td>**V17**</td><td>RLS PERMISSIVE policies</td><td>MC #100387</td><td>**REQUIRES:** bilko\_app role exists. V17 includes idempotent `CREATE ROLE IF NOT EXISTS bilko_app` block (Round 25 fix)</td></tr><tr><td>**V18**</td><td>InvoiceStatus enum</td><td>MC #100406 (drugi agent)</td><td>Creates PostgreSQL ENUM for invoice states (DRAFT, SENT, PAID, etc.)</td></tr><tr><td>**V19**</td><td>ExpenseStatus enum</td><td>MC #100406 (drugi agent)</td><td>Creates PostgreSQL ENUM for expense states (DRAFT, APPROVED, PAID, etc.)</td></tr><tr><td>**V20**</td><td>Demo org name fix</td><td>Round 25</td><td>Updates demo organization name to "Bilko Demo Firma" (Croatian/Serbian localization)</td></tr></tbody></table>

### Migration Execution

Migrations are applied automatically by Flyway during Cloud Run startup via `FlywayPlugin` in `apps/api/src/main/kotlin/no/alai/bilko/Application.kt`.

**Manual Verification After Deployment:**

```bash
# Connect to bilko-demo-db
gcloud sql connect bilko-demo-db --user=bilko_admin --project=tribal-sign-487920-k0

# Check Flyway schema history
SELECT version, description, success, installed_on
FROM flyway_schema_history
ORDER BY installed_rank DESC
LIMIT 10;

# Verify V20 demo org name
SELECT id, name FROM organizations WHERE name = 'Bilko Demo Firma';

```

## 3. Cloud Run Image Promotion Sequence

### Stage → Demo Promotion

**Current State (2026-05-10):**

- **Stage API:** `bilko-api-stage` — image `europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/api:stage-3a4a9e9`
- **Demo API:** `bilko-api-demo` — image TBD (promote from stage)
- **Stage Web:** `bilko-web-stage` — image `europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web:stage-<SHA>`
- **Demo Web:** `bilko-web-demo` — image TBD (promote from stage)

### Promotion Commands

#### API Service

```bash
# Get current stage image SHA
STAGE_SHA=$(gcloud run services describe bilko-api-stage \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --format='value(spec.template.spec.containers[0].image)' | \
  awk -F':' '{print $2}')

echo "Promoting API image: stage-${STAGE_SHA}"

# Promote to demo (use gcloud-write.sh for safety)
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --image=europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/api:stage-${STAGE_SHA}

```

#### Web Service

```bash
# Get current stage web image SHA
WEB_SHA=$(gcloud run services describe bilko-web-stage \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --format='value(spec.template.spec.containers[0].image)' | \
  awk -F':' '{print $2}')

echo "Promoting Web image: stage-${WEB_SHA}"

# Promote to demo
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update bilko-web-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --image=europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web:stage-${WEB_SHA}

```

### Traffic Shift (Gradual Rollout)

If gradual rollout is needed:

```bash
# Split traffic 10% new revision, 90% old
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update-traffic bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --to-revisions=bilko-api-demo-00186-xyz=10,bilko-api-demo-00185-abc=90

# After soak period (30 min), shift to 100%
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update-traffic bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --to-latest

```

## 4. Cloudflare DNS + CORS Verification

### DNS Configuration

**Demo Environment URLs:**

- **API:** `https://bilko-demo-api.alai.no` → Cloud Run `bilko-api-demo`
- **Web:** `https://bilko-demo.alai.no` → Cloud Run `bilko-web-demo`

DNS records managed in Cloudflare dashboard under `alai.no` zone.

### CORS Configuration

**Permanent CORS fix delivered in Round 12.8 (PR #110, MC #100297):**

CORS\_ORIGINS env var is permanently set in `cloudbuild-stage.yaml` and `cloudbuild-demo-api.yaml`:

```yaml
- name: 'gcr.io/cloud-builders/gcloud'
  args:
    - 'run'
    - 'services'
    - 'update'
    - 'bilko-api-demo'
    - '--region=europe-north1'
    - '--update-env-vars=CORS_ORIGINS=https://bilko-demo.alai.no,http://localhost:3000'

```

### CORS Preflight Verification

```bash
# Test CORS preflight from demo domain
curl -X OPTIONS https://bilko-demo-api.alai.no/auth/login \
  -H "Origin: https://bilko-demo.alai.no" \
  -H "Access-Control-Request-Method: POST" \
  -H "Access-Control-Request-Headers: Content-Type" \
  -v

# Expected headers in response:
# Access-Control-Allow-Origin: https://bilko-demo.alai.no
# Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
# Access-Control-Allow-Headers: Content-Type, Authorization
# Access-Control-Allow-Credentials: true

```

## 5. Rollback Procedure

### Cloud Run Rollback

Cloud Run keeps all previous revisions for 60 days. Rollback shifts traffic to a previous stable revision.

```bash
# List recent revisions
gcloud run revisions list \
  --service=bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --limit=10

# Identify previous stable revision (e.g., bilko-api-demo-00185-fay)
PREV_REV="bilko-api-demo-00185-fay"

# Rollback to previous revision
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update-traffic bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --to-revisions=${PREV_REV}=100

```

### Database Rollback

**CRITICAL:** Flyway migrations are forward-only. There is NO automatic rollback for schema changes.

**If V17+ RLS Policies Cause Issues:**

```sql
-- Disable RLS policies manually (Cloud SQL console or psql)
DROP POLICY IF EXISTS org_isolation ON invoices;
DROP POLICY IF EXISTS org_isolation ON expenses;
DROP POLICY IF EXISTS org_isolation ON contacts;
DROP POLICY IF EXISTS org_isolation ON transactions;
DROP POLICY IF EXISTS org_isolation ON bank_accounts;
DROP POLICY IF EXISTS org_isolation ON recurring_invoices;

-- Or disable RLS entirely (NOT recommended for prod)
ALTER TABLE invoices DISABLE ROW LEVEL SECURITY;

```

**Database Point-in-Time Recovery (PITR):**

```bash
# Clone database to a specific point in time (last known good state)
gcloud sql instances clone bilko-demo-db bilko-demo-db-rollback \
  --point-in-time="2026-05-10T10:00:00.000Z" \
  --project=tribal-sign-487920-k0

# Swap Cloud Run to rollback instance (update DATABASE_URL secret)
# This is a LAST RESORT — requires DNS cutover + application restart

```

## 6. Post-Deploy Smoke Tests

Run immediately after cutover to verify core functionality.

### Manual Smoke Tests

<table id="bkmrk-test-command%2Faction-"><thead><tr><th>Test</th><th>Command/Action</th><th>Expected Result</th><th>Status</th></tr></thead><tbody><tr><td>**1. Login**</td><td>`curl -X POST https://bilko-demo-api.alai.no/auth/login -H "Content-Type: application/json" -d '{"email":"demo@bilko.rs","password":"Demo2026!"}'`</td><td>HTTP 200 + JWT token in `accessToken` field</td><td>☐</td></tr><tr><td>**2. User Info**</td><td>`curl https://bilko-demo-api.alai.no/auth/me -H "Authorization: Bearer <JWT>"`</td><td>HTTP 200 + user object with org name "Bilko Demo Firma"</td><td>☐</td></tr><tr><td>**3. Invoice PDF**</td><td>`curl https://bilko-demo-api.alai.no/invoices/<demo-invoice-id>/pdf -H "Authorization: Bearer <JWT>" > /tmp/invoice.pdf`</td><td>HTTP 200 + PDF bytes + PDF contains "Bilko Demo Firma" (V20 regression gate)</td><td>☐</td></tr><tr><td>**4. Health Check**</td><td>`curl https://bilko-demo-api.alai.no/api/v1/health`</td><td>HTTP 200 + `{"status":"healthy"}`</td><td>☐</td></tr><tr><td>**5. RBAC Enforcement**</td><td>Login as viewer role, attempt DELETE /invoices/:id</td><td>HTTP 403 Forbidden (Round 19 RBAC enforcement)</td><td>☐</td></tr></tbody></table>

### Automated E2E Tests (Playwright)

```bash
# Run Playwright tests against demo environment
cd apps/web
PLAYWRIGHT_BASE_URL=https://bilko-demo.alai.no npm run test:e2e

# Expected: 13/15 PASS
# Known failures (Round 12):
# - session persistence (3 specs)
# - navigation edge cases
# - logout redirect

```

### Coverage Verification

```bash
# Verify backend coverage threshold is maintained
cd apps/api
./gradlew koverVerify

# Expected: EXIT 0 (threshold ≥60% per Round 20)

```

## 7. Rounds 11-27 Deliverables Manifest

This section provides traceability for all changes delivered in Rounds 11-27.

### Coverage Progression

<table id="bkmrk-round-mc-id-coverage"><thead><tr><th>Round</th><th>MC ID</th><th>Coverage Delta</th><th>Final Threshold</th><th>Test Count</th></tr></thead><tbody><tr><td>Round 16</td><td>\#100336</td><td>Baseline established</td><td>7%</td><td>~50 HTTP integration tests</td></tr><tr><td>Round 17</td><td>\#100336</td><td>+4%</td><td>11%</td><td>+110 service tests (Invoice, Expense, Contact, Report)</td></tr><tr><td>Round 18</td><td>\#100336</td><td>+39%</td><td>50%</td><td>+90 tests (4 streams: Banking, Auth, Stripe, Routes)</td></tr><tr><td>Round 19</td><td>\#100338</td><td>+7%</td><td>57%</td><td>+20 RBAC edge case tests (403 assertions)</td></tr><tr><td>Round 20</td><td>\#100353</td><td>+3%</td><td>60%</td><td>+81 tests (3 streams: RecurringInvoice, Compliance, Archive/Settings/Account)</td></tr><tr><td>Round 21</td><td>\#100359</td><td>Threshold held</td><td>60%</td><td>+75 edge case tests across 17 HttpIntegrationTest files</td></tr><tr><td>Round 22</td><td>\#100382</td><td>Threshold held</td><td>60%</td><td>+33 billing/webhook integration tests</td></tr><tr><td>Round 23</td><td>—</td><td>Threshold held</td><td>60%</td><td>Auth package coverage push (~30 tests, 4 streams)</td></tr><tr><td>Round 24</td><td>\#100396</td><td>Regression suite</td><td>60%</td><td>Failing-by-design specs for CEO Round 12 bugs</td></tr><tr><td>Round 27</td><td>\#100420</td><td>Deep edge coverage</td><td>60%</td><td>+113 deep edge tests across 8 HttpIntegrationTest files</td></tr></tbody></table>

### Production Bug Fixes (Rounds 12-26)

<table id="bkmrk-fix-mc-id-commit-sha"><thead><tr><th>Fix</th><th>MC ID</th><th>Commit SHA</th><th>Description</th></tr></thead><tbody><tr><td>**CORS Permanent**</td><td>\#100297</td><td>3943837</td><td>Permanent CORS\_ORIGINS + EMAIL\_FROM in cloudbuild-stage.yaml (PR #110)</td></tr><tr><td>**Double Login**</td><td>\#100367</td><td>ef1ed07</td><td>Eliminate double /auth/login call on registration (PR #111)</td></tr><tr><td>**RLS Column Widening**</td><td>\#100406</td><td>660795d</td><td>V16 column widening + V17 line 50 ALTER DATABASE removal</td></tr><tr><td>**bilko\_app Role**</td><td>Round 25</td><td>30d2de4</td><td>V17 idempotent CREATE ROLE IF NOT EXISTS bilko\_app before RLS policies (PR #113)</td></tr><tr><td>**Demo Org Name**</td><td>Round 25</td><td>dbbd80d</td><td>V20 migration ensures demo org name is "Bilko Demo Firma"</td></tr><tr><td>**Wizard Save Draft**</td><td>Round 25</td><td>9319d02</td><td>Step 5 "Save kao nacrt" shows localized toast + redirect</td></tr><tr><td>**Refresh Token**</td><td>Round 26</td><td>b55c8dd</td><td>Eliminate refresh token consumption on login</td></tr><tr><td>**HR L10n Critical**</td><td>\#100355</td><td>e3ede8e</td><td>OIB 11 digits, VAT 25/13/5%, locale routing, register language dynamic (Lexicon audit)</td></tr><tr><td>**Demo API Rebuild**</td><td>\#100400</td><td>9ffe3c4</td><td>Add cloudbuild-demo-api.yaml to rebuild bilko-api-demo (PR #116)</td></tr></tbody></table>

### Feature Deliveries

<table id="bkmrk-feature-mc-id-commit"><thead><tr><th>Feature</th><th>MC ID</th><th>Commit SHA</th><th>Description</th></tr></thead><tbody><tr><td>**RBAC HTTP Layer**</td><td>\#100338</td><td>2a933d8</td><td>Enforce role-check at HTTP layer on all CRUD route handlers (Round 19)</td></tr><tr><td>**RLS Permissive**</td><td>\#100387</td><td>6712fcd</td><td>Phase 2A PERMISSIVE RLS migrations + OrgScope middleware</td></tr><tr><td>**Trial Mechanic**</td><td>\#100326</td><td>d20cb13, e0d200b</td><td>D1+D2+D4 backend + D3 frontend trial countdown banner</td></tr><tr><td>**TaxJurisdiction Expansion**</td><td>\#100386</td><td>e346e9b</td><td>Expand TaxJurisdiction {BA\_FED, BA\_RS} + CountryPlugin interface (ADR-015)</td></tr><tr><td>**ADRs 015-019**</td><td>\#100362</td><td>d17f0ce</td><td>Write ADR-015/016/017/019 per Plan v3 Phase 0</td></tr></tbody></table>

### Infrastructure Changes

<table id="bkmrk-change-mc-id-commit-"><thead><tr><th>Change</th><th>MC ID</th><th>Commit SHA</th><th>Description</th></tr></thead><tbody><tr><td>**GCP Deploy Dormant Fix**</td><td>\#100110</td><td>b1ddecf</td><td>gcp-deploy.yml service name targets bilko-{web,api}-demo (ADR-023 gap closure)</td></tr><tr><td>**V14-V20 Migrations**</td><td>Various</td><td>—</td><td>7 Flyway migrations (demo seed, trial, RLS, enums, org name fix)</td></tr></tbody></table>

## 8. Escalation Contacts

<table id="bkmrk-issue-type-contact-c"><thead><tr><th>Issue Type</th><th>Contact</th><th>Channel</th></tr></thead><tbody><tr><td>**Cloud Run Outage**</td><td>Kelsey Hightower (FlowForge)</td><td>MC task assignment via John</td></tr><tr><td>**Database Corruption**</td><td>Bruce Momjian (CodeCraft)</td><td>MC task assignment via John</td></tr><tr><td>**RBAC/Auth Issues**</td><td>Parisa Tabriz (Securion)</td><td>MC task assignment via John</td></tr><tr><td>**Frontend Errors**</td><td>Lee Robinson (CodeCraft)</td><td>MC task assignment via John</td></tr><tr><td>**CEO Approval**</td><td>Alem Basic</td><td>Email: alem@alai.no, Slack: @alem</td></tr></tbody></table>

## 9. Appendix: Key File Locations

<table id="bkmrk-file-path-stage-clou"><thead><tr><th>File</th><th>Path</th></tr></thead><tbody><tr><td>**Stage Cloud Build**</td><td>`cloudbuild-stage.yaml`</td></tr><tr><td>**Demo API Cloud Build**</td><td>`cloudbuild-demo-api.yaml`</td></tr><tr><td>**Flyway Migrations**</td><td>`apps/api/src/main/resources/db/migration/`</td></tr><tr><td>**Application Config**</td><td>`apps/api/src/main/kotlin/no/alai/bilko/Application.kt`</td></tr><tr><td>**CORS Config**</td><td>`apps/api/src/main/kotlin/no/alai/bilko/plugins/CorsPlugin.kt`</td></tr><tr><td>**Playwright Tests**</td><td>`apps/web/tests/`</td></tr><tr><td>**Kover Config**</td><td>`apps/api/build.gradle.kts`</td></tr></tbody></table>

---

**Document Revision History:**

- **v1.0 (2026-05-10):** Initial runbook creation covering Rounds 11-27 deliverables (MC #100424)