Skip to main content

Bilko Stage→Demo→Prod Cutover Runbook

Bilko Stage→Demo→Prod Cutover Runbook

Document ID: BILKO-RUNBOOK-001
Last Updated: 2026-05-10
Owner: Skillforge (ALAI Holding AS)
Status: Active

Purpose

This runbook documents the promotion path from Bilko stage environment to demo and eventually prod. It incorporates all changes delivered in Rounds 11-27, including massive test coverage expansion, 7+ production bug fixes, RBAC enforcement, permanent CORS configuration, and database migrations V14-V20.

Context: As of 2026-05-10, Bilko stage is on revision 00185-fay (image stage-3a4a9e9). Demo environment has only a subset of these changes. Prod does not exist as a separate environment yet and currently uses demo as customer-facing.

1. Pre-Cutover Checklist

Complete ALL items before proceeding with cutover:

Item Verification Command Success Criteria Status
High/Blocker MCs Resolved node ~/system/tools/mc.js list --project bilko --priority H,BLOCKER --status open 0 open tasks, or all explicitly waived by CEO ☐
Cloud Build Health gcloud builds list --project=tribal-sign-487920-k0 --limit=5 --filter="source.repoSource.branchName=main" Last 5 builds on main = SUCCESS ☐
Coverage Gate cd apps/api && ./gradlew koverVerify Exit code 0 (current threshold ≥60%) ☐
E2E Regression Tests cd apps/web && npm run test:e2e Phase A pass rate ≥50% (Round 24 regression specs) ☐
Database Backup gcloud sql backups create --instance=bilko-demo-db --project=tribal-sign-487920-k0 Backup ID returned + status=SUCCESSFUL ☐
CEO Sign-Off MC task comments or Slack approval Explicit approval captured with timestamp ☐

2. Database Migration Plan (V14-V20)

Flyway migrations are applied automatically when Cloud Run starts. This section documents migration order and critical dependencies.

Migration Sequence

Version Description MC Reference Critical Notes
V14 Demo + CI test seed Round 12.7 Creates [email protected] user + test tenant for CI
V15 Trial fields (plan_tier, trial_start, trial_end) MC #100326 Adds subscription trial mechanic to organizations table
V16 RLS column widening (country constraint) MC #100406 Expands country VARCHAR to support TaxJurisdiction enum values
V17 RLS PERMISSIVE policies MC #100387 REQUIRES: bilko_app role exists. V17 includes idempotent CREATE ROLE IF NOT EXISTS bilko_app block (Round 25 fix)
V18 InvoiceStatus enum MC #100406 (drugi agent) Creates PostgreSQL ENUM for invoice states (DRAFT, SENT, PAID, etc.)
V19 ExpenseStatus enum MC #100406 (drugi agent) Creates PostgreSQL ENUM for expense states (DRAFT, APPROVED, PAID, etc.)
V20 Demo org name fix Round 25 Updates demo organization name to "Bilko Demo Firma" (Croatian/Serbian localization)

Migration Execution

Migrations are applied automatically by Flyway during Cloud Run startup via FlywayPlugin in apps/api/src/main/kotlin/no/alai/bilko/Application.kt.

Manual Verification After Deployment:

# Connect to bilko-demo-db
gcloud sql connect bilko-demo-db --user=bilko_admin --project=tribal-sign-487920-k0

# Check Flyway schema history
SELECT version, description, success, installed_on
FROM flyway_schema_history
ORDER BY installed_rank DESC
LIMIT 10;

# Verify V20 demo org name
SELECT id, name FROM organizations WHERE name = 'Bilko Demo Firma';

3. Cloud Run Image Promotion Sequence

Stage → Demo Promotion

Current State (2026-05-10):

  • Stage API: bilko-api-stage — image europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/api:stage-3a4a9e9
  • Demo API: bilko-api-demo — image TBD (promote from stage)
  • Stage Web: bilko-web-stage — image europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web:stage-<SHA>
  • Demo Web: bilko-web-demo — image TBD (promote from stage)

Promotion Commands

API Service

# Get current stage image SHA
STAGE_SHA=$(gcloud run services describe bilko-api-stage \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --format='value(spec.template.spec.containers[0].image)' | \
  awk -F':' '{print $2}')

echo "Promoting API image: stage-${STAGE_SHA}"

# Promote to demo (use gcloud-write.sh for safety)
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --image=europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/api:stage-${STAGE_SHA}

Web Service

# Get current stage web image SHA
WEB_SHA=$(gcloud run services describe bilko-web-stage \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --format='value(spec.template.spec.containers[0].image)' | \
  awk -F':' '{print $2}')

echo "Promoting Web image: stage-${WEB_SHA}"

# Promote to demo
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update bilko-web-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --image=europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web:stage-${WEB_SHA}

Traffic Shift (Gradual Rollout)

If gradual rollout is needed:

# Split traffic 10% new revision, 90% old
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update-traffic bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --to-revisions=bilko-api-demo-00186-xyz=10,bilko-api-demo-00185-abc=90

# After soak period (30 min), shift to 100%
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update-traffic bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --to-latest

4. Cloudflare DNS + CORS Verification

DNS Configuration

Demo Environment URLs:

  • API: https://bilko-demo-api.alai.no → Cloud Run bilko-api-demo
  • Web: https://bilko-demo.alai.no → Cloud Run bilko-web-demo

DNS records managed in Cloudflare dashboard under alai.no zone.

CORS Configuration

Permanent CORS fix delivered in Round 12.8 (PR #110, MC #100297):

CORS_ORIGINS env var is permanently set in cloudbuild-stage.yaml and cloudbuild-demo-api.yaml:

- name: 'gcr.io/cloud-builders/gcloud'
  args:
    - 'run'
    - 'services'
    - 'update'
    - 'bilko-api-demo'
    - '--region=europe-north1'
    - '--update-env-vars=CORS_ORIGINS=https://bilko-demo.alai.no,http://localhost:3000'

CORS Preflight Verification

# Test CORS preflight from demo domain
curl -X OPTIONS https://bilko-demo-api.alai.no/auth/login \
  -H "Origin: https://bilko-demo.alai.no" \
  -H "Access-Control-Request-Method: POST" \
  -H "Access-Control-Request-Headers: Content-Type" \
  -v

# Expected headers in response:
# Access-Control-Allow-Origin: https://bilko-demo.alai.no
# Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
# Access-Control-Allow-Headers: Content-Type, Authorization
# Access-Control-Allow-Credentials: true

5. Rollback Procedure

Cloud Run Rollback

Cloud Run keeps all previous revisions for 60 days. Rollback shifts traffic to a previous stable revision.

# List recent revisions
gcloud run revisions list \
  --service=bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --limit=10

# Identify previous stable revision (e.g., bilko-api-demo-00185-fay)
PREV_REV="bilko-api-demo-00185-fay"

# Rollback to previous revision
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update-traffic bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --to-revisions=${PREV_REV}=100

Database Rollback

CRITICAL: Flyway migrations are forward-only. There is NO automatic rollback for schema changes.

If V17+ RLS Policies Cause Issues:

-- Disable RLS policies manually (Cloud SQL console or psql)
DROP POLICY IF EXISTS org_isolation ON invoices;
DROP POLICY IF EXISTS org_isolation ON expenses;
DROP POLICY IF EXISTS org_isolation ON contacts;
DROP POLICY IF EXISTS org_isolation ON transactions;
DROP POLICY IF EXISTS org_isolation ON bank_accounts;
DROP POLICY IF EXISTS org_isolation ON recurring_invoices;

-- Or disable RLS entirely (NOT recommended for prod)
ALTER TABLE invoices DISABLE ROW LEVEL SECURITY;

Database Point-in-Time Recovery (PITR):

# Clone database to a specific point in time (last known good state)
gcloud sql instances clone bilko-demo-db bilko-demo-db-rollback \
  --point-in-time="2026-05-10T10:00:00.000Z" \
  --project=tribal-sign-487920-k0

# Swap Cloud Run to rollback instance (update DATABASE_URL secret)
# This is a LAST RESORT — requires DNS cutover + application restart

6. Post-Deploy Smoke Tests

Run immediately after cutover to verify core functionality.

Manual Smoke Tests

Test Command/Action Expected Result Status
1. Login curl -X POST https://bilko-demo-api.alai.no/auth/login -H "Content-Type: application/json" -d '{"email":"[email protected]","password":"Demo2026!"}' HTTP 200 + JWT token in accessToken field ☐
2. User Info curl https://bilko-demo-api.alai.no/auth/me -H "Authorization: Bearer <JWT>" HTTP 200 + user object with org name "Bilko Demo Firma" ☐
3. Invoice PDF curl https://bilko-demo-api.alai.no/invoices/<demo-invoice-id>/pdf -H "Authorization: Bearer <JWT>" > /tmp/invoice.pdf HTTP 200 + PDF bytes + PDF contains "Bilko Demo Firma" (V20 regression gate) ☐
4. Health Check curl https://bilko-demo-api.alai.no/api/v1/health HTTP 200 + {"status":"healthy"} ☐
5. RBAC Enforcement Login as viewer role, attempt DELETE /invoices/:id HTTP 403 Forbidden (Round 19 RBAC enforcement) ☐

Automated E2E Tests (Playwright)

# Run Playwright tests against demo environment
cd apps/web
PLAYWRIGHT_BASE_URL=https://bilko-demo.alai.no npm run test:e2e

# Expected: 13/15 PASS
# Known failures (Round 12):
# - session persistence (3 specs)
# - navigation edge cases
# - logout redirect

Coverage Verification

# Verify backend coverage threshold is maintained
cd apps/api
./gradlew koverVerify

# Expected: EXIT 0 (threshold ≥60% per Round 20)

7. Rounds 11-27 Deliverables Manifest

This section provides traceability for all changes delivered in Rounds 11-27.

Coverage Progression

Round MC ID Coverage Delta Final Threshold Test Count
Round 16 #100336 Baseline established 7% ~50 HTTP integration tests
Round 17 #100336 +4% 11% +110 service tests (Invoice, Expense, Contact, Report)
Round 18 #100336 +39% 50% +90 tests (4 streams: Banking, Auth, Stripe, Routes)
Round 19 #100338 +7% 57% +20 RBAC edge case tests (403 assertions)
Round 20 #100353 +3% 60% +81 tests (3 streams: RecurringInvoice, Compliance, Archive/Settings/Account)
Round 21 #100359 Threshold held 60% +75 edge case tests across 17 HttpIntegrationTest files
Round 22 #100382 Threshold held 60% +33 billing/webhook integration tests
Round 23 — Threshold held 60% Auth package coverage push (~30 tests, 4 streams)
Round 24 #100396 Regression suite 60% Failing-by-design specs for CEO Round 12 bugs
Round 27 #100420 Deep edge coverage 60% +113 deep edge tests across 8 HttpIntegrationTest files

Production Bug Fixes (Rounds 12-26)

Fix MC ID Commit SHA Description
CORS Permanent #100297 3943837 Permanent CORS_ORIGINS + EMAIL_FROM in cloudbuild-stage.yaml (PR #110)
Double Login #100367 ef1ed07 Eliminate double /auth/login call on registration (PR #111)
RLS Column Widening #100406 660795d V16 column widening + V17 line 50 ALTER DATABASE removal
bilko_app Role Round 25 30d2de4 V17 idempotent CREATE ROLE IF NOT EXISTS bilko_app before RLS policies (PR #113)
Demo Org Name Round 25 dbbd80d V20 migration ensures demo org name is "Bilko Demo Firma"
Wizard Save Draft Round 25 9319d02 Step 5 "Save kao nacrt" shows localized toast + redirect
Refresh Token Round 26 b55c8dd Eliminate refresh token consumption on login
HR L10n Critical #100355 e3ede8e OIB 11 digits, VAT 25/13/5%, locale routing, register language dynamic (Lexicon audit)
Demo API Rebuild #100400 9ffe3c4 Add cloudbuild-demo-api.yaml to rebuild bilko-api-demo (PR #116)

Feature Deliveries

Feature MC ID Commit SHA Description
RBAC HTTP Layer #100338 2a933d8 Enforce role-check at HTTP layer on all CRUD route handlers (Round 19)
RLS Permissive #100387 6712fcd Phase 2A PERMISSIVE RLS migrations + OrgScope middleware
Trial Mechanic #100326 d20cb13, e0d200b D1+D2+D4 backend + D3 frontend trial countdown banner
TaxJurisdiction Expansion #100386 e346e9b Expand TaxJurisdiction {BA_FED, BA_RS} + CountryPlugin interface (ADR-015)
ADRs 015-019 #100362 d17f0ce Write ADR-015/016/017/019 per Plan v3 Phase 0

Infrastructure Changes

Change MC ID Commit SHA Description
GCP Deploy Dormant Fix #100110 b1ddecf gcp-deploy.yml service name targets bilko-{web,api}-demo (ADR-023 gap closure)
V14-V20 Migrations Various — 7 Flyway migrations (demo seed, trial, RLS, enums, org name fix)

8. Escalation Contacts

Issue Type Contact Channel
Cloud Run Outage Kelsey Hightower (FlowForge) MC task assignment via John
Database Corruption Bruce Momjian (CodeCraft) MC task assignment via John
RBAC/Auth Issues Parisa Tabriz (Securion) MC task assignment via John
Frontend Errors Lee Robinson (CodeCraft) MC task assignment via John
CEO Approval Alem Basic Email: [email protected], Slack: @alem

9. Appendix: Key File Locations

File Path
Stage Cloud Build cloudbuild-stage.yaml
Demo API Cloud Build cloudbuild-demo-api.yaml
Flyway Migrations apps/api/src/main/resources/db/migration/
Application Config apps/api/src/main/kotlin/no/alai/bilko/Application.kt
CORS Config apps/api/src/main/kotlin/no/alai/bilko/plugins/CorsPlugin.kt
Playwright Tests apps/web/tests/
Kover Config apps/api/build.gradle.kts

Document Revision History:

  • v1.0 (2026-05-10): Initial runbook creation covering Rounds 11-27 deliverables (MC #100424)

No comments to display

Back to top