Bilko Stage→Demo→Prod Cutover Runbook Bilko Stage→Demo→Prod Cutover Runbook Document ID: BILKO-RUNBOOK-001 Last Updated: 2026-05-10 Owner: Skillforge (ALAI Holding AS) Status: Active Purpose This runbook documents the promotion path from Bilko stage environment to demo and eventually prod . It incorporates all changes delivered in Rounds 11-27, including massive test coverage expansion, 7+ production bug fixes, RBAC enforcement, permanent CORS configuration, and database migrations V14-V20. Context: As of 2026-05-10, Bilko stage is on revision 00185-fay (image stage-3a4a9e9 ). Demo environment has only a subset of these changes. Prod does not exist as a separate environment yet and currently uses demo as customer-facing. 1. Pre-Cutover Checklist Complete ALL items before proceeding with cutover: Item Verification Command Success Criteria Status High/Blocker MCs Resolved node ~/system/tools/mc.js list --project bilko --priority H,BLOCKER --status open 0 open tasks, or all explicitly waived by CEO ☐ Cloud Build Health gcloud builds list --project=tribal-sign-487920-k0 --limit=5 --filter="source.repoSource.branchName=main" Last 5 builds on main = SUCCESS ☐ Coverage Gate cd apps/api && ./gradlew koverVerify Exit code 0 (current threshold ≥60%) ☐ E2E Regression Tests cd apps/web && npm run test:e2e Phase A pass rate ≥50% (Round 24 regression specs) ☐ Database Backup gcloud sql backups create --instance=bilko-demo-db --project=tribal-sign-487920-k0 Backup ID returned + status=SUCCESSFUL ☐ CEO Sign-Off MC task comments or Slack approval Explicit approval captured with timestamp ☐ 2. Database Migration Plan (V14-V20) Flyway migrations are applied automatically when Cloud Run starts. This section documents migration order and critical dependencies. Migration Sequence Version Description MC Reference Critical Notes V14 Demo + CI test seed Round 12.7 Creates demo@bilko.rs user + test tenant for CI V15 Trial fields (plan_tier, trial_start, trial_end) MC #100326 Adds subscription trial mechanic to organizations table V16 RLS column widening (country constraint) MC #100406 Expands country VARCHAR to support TaxJurisdiction enum values V17 RLS PERMISSIVE policies MC #100387 REQUIRES: bilko_app role exists. V17 includes idempotent CREATE ROLE IF NOT EXISTS bilko_app block (Round 25 fix) V18 InvoiceStatus enum MC #100406 (drugi agent) Creates PostgreSQL ENUM for invoice states (DRAFT, SENT, PAID, etc.) V19 ExpenseStatus enum MC #100406 (drugi agent) Creates PostgreSQL ENUM for expense states (DRAFT, APPROVED, PAID, etc.) V20 Demo org name fix Round 25 Updates demo organization name to "Bilko Demo Firma" (Croatian/Serbian localization) Migration Execution Migrations are applied automatically by Flyway during Cloud Run startup via FlywayPlugin in apps/api/src/main/kotlin/no/alai/bilko/Application.kt . Manual Verification After Deployment: # Connect to bilko-demo-db gcloud sql connect bilko-demo-db --user=bilko_admin --project=tribal-sign-487920-k0 # Check Flyway schema history SELECT version, description, success, installed_on FROM flyway_schema_history ORDER BY installed_rank DESC LIMIT 10; # Verify V20 demo org name SELECT id, name FROM organizations WHERE name = 'Bilko Demo Firma'; 3. Cloud Run Image Promotion Sequence Stage → Demo Promotion Current State (2026-05-10): Stage API: bilko-api-stage — image europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/api:stage-3a4a9e9 Demo API: bilko-api-demo — image TBD (promote from stage) Stage Web: bilko-web-stage — image europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web:stage- Demo Web: bilko-web-demo — image TBD (promote from stage) Promotion Commands API Service # Get current stage image SHA STAGE_SHA=$(gcloud run services describe bilko-api-stage \ --region=europe-north1 \ --project=tribal-sign-487920-k0 \ --format='value(spec.template.spec.containers[0].image)' | \ awk -F':' '{print $2}') echo "Promoting API image: stage-${STAGE_SHA}" # Promote to demo (use gcloud-write.sh for safety) bash ~/system/tools/gcloud-write.sh --mc 100424 run services update bilko-api-demo \ --region=europe-north1 \ --project=tribal-sign-487920-k0 \ --image=europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/api:stage-${STAGE_SHA} Web Service # Get current stage web image SHA WEB_SHA=$(gcloud run services describe bilko-web-stage \ --region=europe-north1 \ --project=tribal-sign-487920-k0 \ --format='value(spec.template.spec.containers[0].image)' | \ awk -F':' '{print $2}') echo "Promoting Web image: stage-${WEB_SHA}" # Promote to demo bash ~/system/tools/gcloud-write.sh --mc 100424 run services update bilko-web-demo \ --region=europe-north1 \ --project=tribal-sign-487920-k0 \ --image=europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web:stage-${WEB_SHA} Traffic Shift (Gradual Rollout) If gradual rollout is needed: # Split traffic 10% new revision, 90% old bash ~/system/tools/gcloud-write.sh --mc 100424 run services update-traffic bilko-api-demo \ --region=europe-north1 \ --project=tribal-sign-487920-k0 \ --to-revisions=bilko-api-demo-00186-xyz=10,bilko-api-demo-00185-abc=90 # After soak period (30 min), shift to 100% bash ~/system/tools/gcloud-write.sh --mc 100424 run services update-traffic bilko-api-demo \ --region=europe-north1 \ --project=tribal-sign-487920-k0 \ --to-latest 4. Cloudflare DNS + CORS Verification DNS Configuration Demo Environment URLs: API: https://bilko-demo-api.alai.no → Cloud Run bilko-api-demo Web: https://bilko-demo.alai.no → Cloud Run bilko-web-demo DNS records managed in Cloudflare dashboard under alai.no zone. CORS Configuration Permanent CORS fix delivered in Round 12.8 (PR #110, MC #100297): CORS_ORIGINS env var is permanently set in cloudbuild-stage.yaml and cloudbuild-demo-api.yaml : - name: 'gcr.io/cloud-builders/gcloud' args: - 'run' - 'services' - 'update' - 'bilko-api-demo' - '--region=europe-north1' - '--update-env-vars=CORS_ORIGINS=https://bilko-demo.alai.no,http://localhost:3000' CORS Preflight Verification # Test CORS preflight from demo domain curl -X OPTIONS https://bilko-demo-api.alai.no/auth/login \ -H "Origin: https://bilko-demo.alai.no" \ -H "Access-Control-Request-Method: POST" \ -H "Access-Control-Request-Headers: Content-Type" \ -v # Expected headers in response: # Access-Control-Allow-Origin: https://bilko-demo.alai.no # Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS # Access-Control-Allow-Headers: Content-Type, Authorization # Access-Control-Allow-Credentials: true 5. Rollback Procedure Cloud Run Rollback Cloud Run keeps all previous revisions for 60 days. Rollback shifts traffic to a previous stable revision. # List recent revisions gcloud run revisions list \ --service=bilko-api-demo \ --region=europe-north1 \ --project=tribal-sign-487920-k0 \ --limit=10 # Identify previous stable revision (e.g., bilko-api-demo-00185-fay) PREV_REV="bilko-api-demo-00185-fay" # Rollback to previous revision bash ~/system/tools/gcloud-write.sh --mc 100424 run services update-traffic bilko-api-demo \ --region=europe-north1 \ --project=tribal-sign-487920-k0 \ --to-revisions=${PREV_REV}=100 Database Rollback CRITICAL: Flyway migrations are forward-only. There is NO automatic rollback for schema changes. If V17+ RLS Policies Cause Issues: -- Disable RLS policies manually (Cloud SQL console or psql) DROP POLICY IF EXISTS org_isolation ON invoices; DROP POLICY IF EXISTS org_isolation ON expenses; DROP POLICY IF EXISTS org_isolation ON contacts; DROP POLICY IF EXISTS org_isolation ON transactions; DROP POLICY IF EXISTS org_isolation ON bank_accounts; DROP POLICY IF EXISTS org_isolation ON recurring_invoices; -- Or disable RLS entirely (NOT recommended for prod) ALTER TABLE invoices DISABLE ROW LEVEL SECURITY; Database Point-in-Time Recovery (PITR): # Clone database to a specific point in time (last known good state) gcloud sql instances clone bilko-demo-db bilko-demo-db-rollback \ --point-in-time="2026-05-10T10:00:00.000Z" \ --project=tribal-sign-487920-k0 # Swap Cloud Run to rollback instance (update DATABASE_URL secret) # This is a LAST RESORT — requires DNS cutover + application restart 6. Post-Deploy Smoke Tests Run immediately after cutover to verify core functionality. Manual Smoke Tests Test Command/Action Expected Result Status 1. Login curl -X POST https://bilko-demo-api.alai.no/auth/login -H "Content-Type: application/json" -d '{"email":"demo@bilko.rs","password":"Demo2026!"}' HTTP 200 + JWT token in accessToken field ☐ 2. User Info curl https://bilko-demo-api.alai.no/auth/me -H "Authorization: Bearer " HTTP 200 + user object with org name "Bilko Demo Firma" ☐ 3. Invoice PDF curl https://bilko-demo-api.alai.no/invoices//pdf -H "Authorization: Bearer " > /tmp/invoice.pdf HTTP 200 + PDF bytes + PDF contains "Bilko Demo Firma" (V20 regression gate) ☐ 4. Health Check curl https://bilko-demo-api.alai.no/api/v1/health HTTP 200 + {"status":"healthy"} ☐ 5. RBAC Enforcement Login as viewer role, attempt DELETE /invoices/:id HTTP 403 Forbidden (Round 19 RBAC enforcement) ☐ Automated E2E Tests (Playwright) # Run Playwright tests against demo environment cd apps/web PLAYWRIGHT_BASE_URL=https://bilko-demo.alai.no npm run test:e2e # Expected: 13/15 PASS # Known failures (Round 12): # - session persistence (3 specs) # - navigation edge cases # - logout redirect Coverage Verification # Verify backend coverage threshold is maintained cd apps/api ./gradlew koverVerify # Expected: EXIT 0 (threshold ≥60% per Round 20) 7. Rounds 11-27 Deliverables Manifest This section provides traceability for all changes delivered in Rounds 11-27. Coverage Progression Round MC ID Coverage Delta Final Threshold Test Count Round 16 #100336 Baseline established 7% ~50 HTTP integration tests Round 17 #100336 +4% 11% +110 service tests (Invoice, Expense, Contact, Report) Round 18 #100336 +39% 50% +90 tests (4 streams: Banking, Auth, Stripe, Routes) Round 19 #100338 +7% 57% +20 RBAC edge case tests (403 assertions) Round 20 #100353 +3% 60% +81 tests (3 streams: RecurringInvoice, Compliance, Archive/Settings/Account) Round 21 #100359 Threshold held 60% +75 edge case tests across 17 HttpIntegrationTest files Round 22 #100382 Threshold held 60% +33 billing/webhook integration tests Round 23 — Threshold held 60% Auth package coverage push (~30 tests, 4 streams) Round 24 #100396 Regression suite 60% Failing-by-design specs for CEO Round 12 bugs Round 27 #100420 Deep edge coverage 60% +113 deep edge tests across 8 HttpIntegrationTest files Production Bug Fixes (Rounds 12-26) Fix MC ID Commit SHA Description CORS Permanent #100297 3943837 Permanent CORS_ORIGINS + EMAIL_FROM in cloudbuild-stage.yaml (PR #110) Double Login #100367 ef1ed07 Eliminate double /auth/login call on registration (PR #111) RLS Column Widening #100406 660795d V16 column widening + V17 line 50 ALTER DATABASE removal bilko_app Role Round 25 30d2de4 V17 idempotent CREATE ROLE IF NOT EXISTS bilko_app before RLS policies (PR #113) Demo Org Name Round 25 dbbd80d V20 migration ensures demo org name is "Bilko Demo Firma" Wizard Save Draft Round 25 9319d02 Step 5 "Save kao nacrt" shows localized toast + redirect Refresh Token Round 26 b55c8dd Eliminate refresh token consumption on login HR L10n Critical #100355 e3ede8e OIB 11 digits, VAT 25/13/5%, locale routing, register language dynamic (Lexicon audit) Demo API Rebuild #100400 9ffe3c4 Add cloudbuild-demo-api.yaml to rebuild bilko-api-demo (PR #116) Feature Deliveries Feature MC ID Commit SHA Description RBAC HTTP Layer #100338 2a933d8 Enforce role-check at HTTP layer on all CRUD route handlers (Round 19) RLS Permissive #100387 6712fcd Phase 2A PERMISSIVE RLS migrations + OrgScope middleware Trial Mechanic #100326 d20cb13, e0d200b D1+D2+D4 backend + D3 frontend trial countdown banner TaxJurisdiction Expansion #100386 e346e9b Expand TaxJurisdiction {BA_FED, BA_RS} + CountryPlugin interface (ADR-015) ADRs 015-019 #100362 d17f0ce Write ADR-015/016/017/019 per Plan v3 Phase 0 Infrastructure Changes Change MC ID Commit SHA Description GCP Deploy Dormant Fix #100110 b1ddecf gcp-deploy.yml service name targets bilko-{web,api}-demo (ADR-023 gap closure) V14-V20 Migrations Various — 7 Flyway migrations (demo seed, trial, RLS, enums, org name fix) 8. Escalation Contacts Issue Type Contact Channel Cloud Run Outage Kelsey Hightower (FlowForge) MC task assignment via John Database Corruption Bruce Momjian (CodeCraft) MC task assignment via John RBAC/Auth Issues Parisa Tabriz (Securion) MC task assignment via John Frontend Errors Lee Robinson (CodeCraft) MC task assignment via John CEO Approval Alem Basic Email: alem@alai.no, Slack: @alem 9. Appendix: Key File Locations File Path Stage Cloud Build cloudbuild-stage.yaml Demo API Cloud Build cloudbuild-demo-api.yaml Flyway Migrations apps/api/src/main/resources/db/migration/ Application Config apps/api/src/main/kotlin/no/alai/bilko/Application.kt CORS Config apps/api/src/main/kotlin/no/alai/bilko/plugins/CorsPlugin.kt Playwright Tests apps/web/tests/ Kover Config apps/api/build.gradle.kts Document Revision History: v1.0 (2026-05-10): Initial runbook creation covering Rounds 11-27 deliverables (MC #100424)