Bilko Stage→Demo→Prod Cutover Runbook
Bilko Stage→Demo→Prod Cutover Runbook
Document ID: BILKO-RUNBOOK-001
Last Updated: 2026-08-15
Owner: Skillforge (ALAI Holding AS)
Status: Active — Azure canonical
Purpose
This runbook documents the promotion path from Bilko stage environment to demo and eventually prod. It incorporates all changes delivered in Rounds 11-27, including massive test coverage expansion, 7+ production bug fixes, RBAC enforcement, permanent CORS configuration, and database migrations V14-V20.
Context: Bilko now runs on Azure Container Apps in rg-bilko-demo. The apps named bilko-web-demo and bilko-api-demo are customer-facing production; app.bilko.cloud and api.bilko.cloud route to them. Stage uses bilko-web-stage and bilko-api-stage. Both environments currently share the production Azure PostgreSQL server bilko-demo-pg, so every database action in this runbook is production-sensitive. Sources: live read-only Azure query 2026-08-15; BookStack pages 3305 and 3121; DEPLOY-MAP.md at azdo/main 3db716d7.
1. Pre-Cutover Checklist
Complete ALL items before proceeding with cutover:
| Item | Verification Command | Success Criteria | Status |
|---|---|---|---|
| High/Blocker MCs Resolved | node ~/system/tools/mc.js list --project bilko --priority H,BLOCKER --status open |
0 open tasks, or all explicitly waived by CEO | ☐ |
| Azure DevOps Pipeline Health | az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko --pipeline-ids 1 --branch main --top 5 --query "[].{id:id,status:status,result:result,sourceBranch:sourceBranch,sourceVersion:sourceVersion,finishTime:finishTime}" -o tableSource: verified read-only az query on 2026-08-15; pipeline definition azure-pipelines.yml at azdo/main 3db716d7. |
Last 5 completed runs on main = SUCCESS, or every failure is explicitly adjudicated before promotion | ☐ |
| Coverage Gate | cd apps/api && ./gradlew koverVerify |
Exit code 0 (current threshold ≥60%) | ☐ |
| E2E Regression Tests | Open the exact Azure DevOps run for the source commit and inspect E2E_UAT plus its published Playwright results.az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko --pipeline-ids 1 --branch main --top 5 -o tableSource: live azure-pipelines.yml stage E2E_UAT at azdo/main 3db716d7. |
E2E_UAT succeeded and the published Playwright results contain no failures for the exact promoted commit |
☐ |
| Database Backup | pg_dump "host=bilko-demo-pg.postgres.database.azure.com port=5432 dbname=bilko user=bilko_admin sslmode=require" --format=custom --no-acl --no-owner --file="bilko-backup-$(date +%Y%m%d-%H%M%S).dump"Source: BookStack page 3305, §5d “Database Backup and Restore”. Supply the password through the approved secure mechanism; never paste it into this page or shell history. |
Dump exits 0 and pg_restore --list <dump-file> exits 0; backup path and checksum recorded in the cutover evidence |
☐ |
| CEO Sign-Off | MC task comments or Slack approval | Explicit approval captured with timestamp | ☐ |
2. Database Migration Plan (V14-V20)
Flyway migrations are applied by the Flyway_Migrate stage in the canonical Azure DevOps pipeline, after Build and before Deploy_Stage. This section preserves the migration order and critical dependencies. Source: azure-pipelines.yml at azdo/main 3db716d7, stage Flyway_Migrate.
Migration Sequence
| Version | Description | MC Reference | Critical Notes |
|---|---|---|---|
| V14 | Demo + CI test seed | Round 12.7 | Creates demo@bilko.rs user + test tenant for CI |
| V15 | Trial fields (plan_tier, trial_start, trial_end) | MC #100326 | Adds subscription trial mechanic to organizations table |
| V16 | RLS column widening (country constraint) | MC #100406 | Expands country VARCHAR to support TaxJurisdiction enum values |
| V17 | RLS PERMISSIVE policies | MC #100387 | REQUIRES: bilko_app role exists. V17 includes idempotent CREATE ROLE IF NOT EXISTS bilko_app block (Round 25 fix) |
| V18 | InvoiceStatus enum | MC #100406 (drugi agent) | Creates PostgreSQL ENUM for invoice states (DRAFT, SENT, PAID, etc.) |
| V19 | ExpenseStatus enum | MC #100406 (drugi agent) | Creates PostgreSQL ENUM for expense states (DRAFT, APPROVED, PAID, etc.) |
| V20 | Demo org name fix | Round 25 | Updates demo organization name to "Bilko Demo Firma" (Croatian/Serbian localization) |
Migration Execution
Migrations are applied by Flyway 11 in the pipeline against bilko-demo-pg.postgres.database.azure.com:5432/bilko as bilko_admin. The pipeline fails closed if the database contains an applied migration newer than the checked-out source tree. Source: azure-pipelines.yml at azdo/main 3db716d7, stage Flyway_Migrate.
Manual Verification After Deployment:
Production warning: bilko-demo-pg serves customer-facing production. Use a read-only transaction for verification and do not run ad-hoc DDL/DML.
# Connect to Azure PostgreSQL Flexible Server
psql "host=bilko-demo-pg.postgres.database.azure.com port=5432 dbname=bilko user=bilko_admin sslmode=require"
# In psql, enter a read-only transaction before checks
BEGIN READ ONLY;
SELECT version, description, success, installed_on
FROM flyway_schema_history
ORDER BY installed_rank DESC
LIMIT 10;
SELECT id, name FROM organizations WHERE name = 'Bilko Demo Firma';
ROLLBACK;
Source: BookStack page 3305, §5b and §7a; database identity re-verified by read-only Azure query on 2026-08-15.
3. Azure Container Apps Image Promotion Sequence
Stage → Demo (Production) Promotion
Environment classification: the resources with -demo in their names are production. The canonical pipeline promotes API and Web together from one commit; do not copy a stage tag into production or call az containerapp update manually from this runbook.
- Stage API:
bilko-api-stage—bilkodemo.azurecr.io/bilko-api:stage-{SHA}@sha256:{digest} - Production API:
bilko-api-demo—bilkodemo.azurecr.io/bilko-api:demo-{SHA}@sha256:{digest} - Stage Web:
bilko-web-stage—bilkodemo.azurecr.io/bilko-web:stage-{SHA}@sha256:{digest} - Production Web:
bilko-web-demo—bilkodemo.azurecr.io/bilko-web:demo-{SHA}@sha256:{digest}
Sources: live read-only ACA/ACR query 2026-08-15; DEPLOY-MAP.md and azure-pipelines.yml at azdo/main 3db716d7.
Promotion Commands
API Service
# Queue the canonical pipeline from main. It builds, scans, migrates, deploys stage,
# runs E2E UAT, then waits for the bilko-demo environment approval.
az pipelines run --org https://dev.azure.com/alai-holding --project Bilko --branch main --id 1
# Monitor recent main runs; record the run ID and sourceVersion in cutover evidence.
az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko --pipeline-ids 1 --branch main --top 5 -o table
Do not approve Promote_Demo until CI_Gates → Build → Flyway_Migrate → Deploy_Stage → E2E_UAT are green and the pre-cutover checklist is complete. Approval is performed in Azure DevOps under Pipelines → Environments → bilko-demo. The approved stage rebuilds demo-{SHA} images from the same commit, scans them, deploys digest-pinned images, and shifts bilko-api-demo to the new revision.
Source: BookStack page 3305 §3b and live azure-pipelines.yml Promote_Demo at azdo/main 3db716d7.
Web Service
There is no separate Web promotion command. The same approved Bilko-CI-CD run builds and deploys bilko-api-demo and bilko-web-demo from the same source commit. A split API/Web manual promotion would bypass the canonical release contract.
Source: live azure-pipelines.yml Promote_Demo deployment job at azdo/main 3db716d7.
Traffic Shift (Gradual Rollout)
TODO/UNKNOWN — intentional safety stop: no reviewed gradual 10/90 production rollout command exists in the allowed current sources. The canonical pipeline shifts bilko-api-demo 100% to its latest revision; live state shows API in Multiple revision mode and Web in Single mode. If gradual rollout is required, stop here and obtain a FlowForge-authored, revision-specific procedure. Do not translate the retired traffic-weight commands by analogy.
Source for current behavior: live azure-pipelines.yml Promote_Demo and read-only ACA query 2026-08-15.
4. Cloudflare Routing + CORS Verification
DNS / Routing Configuration
Customer-facing production URLs:
- API:
https://api.bilko.cloud→ Cloudflare Workerbilko-edge-proxy→ ACAbilko-api-demo - Web:
https://app.bilko.cloud→ Cloudflare Workerbilko-edge-proxy→ ACAbilko-web-demo - Stage API:
https://api-stage.bilko.cloud→ direct ACA custom-domain binding onbilko-api-stage - Stage Web:
https://web-stage.bilko.cloud→ direct ACA custom-domain binding onbilko-web-stage
The retired *.alai.no demo hosts are not release targets and must not be used in health checks or browser tests.
Sources: BookStack page 3121; DEPLOY-MAP.md at azdo/main 3db716d7; live read-only ACA query 2026-08-15.
CORS Configuration
CORS is set by the canonical pipeline, not by a standalone operator update:
- Stage API:
https://bilko-web-stage.purplebeach-f004d490.swedencentral.azurecontainerapps.io,https://web-stage.bilko.cloud, and localhost for development. - Production API:
https://app.bilko.cloud,https://app.bilko.io,https://app.bilko.company, and localhost for development.
Source: live azure-pipelines.yml Deploy_Stage and Promote_Demo environment updates at azdo/main 3db716d7.
CORS / Route Verification
# Public production health (current brand domains)
curl -s https://api.bilko.cloud/api/v1/health
curl -sI https://app.bilko.cloud/ | head -1
# Pipeline evidence: E2E_UAT performs browser-level CSP/CORS-sensitive flows on stage.
az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko --pipeline-ids 1 --branch main --top 5 -o table
Success requires HTTP 200 on both public routes and a green E2E_UAT stage for the exact commit being promoted. The retired password-login preflight is invalid because Bilko is Entra/demo-session based; do not recreate it.
Sources: BookStack page 3305 §3c/§4d and live azure-pipelines.yml E2E/PI2 checks at azdo/main 3db716d7.
5. Rollback Procedure
Azure Container Apps Rollback
Live state verified 2026-08-15: bilko-api-demo uses Multiple revision mode; bilko-web-demo uses Single. Record the current revision and image before any rollback.
# API: list revisions and their state
az containerapp revision list --name bilko-api-demo --resource-group rg-bilko-demo --query "[].{name:name,created:properties.createdTime,active:properties.active}" -o table
# API: after explicit rollback authorization, shift 100% to the reviewed known-good revision
az containerapp ingress traffic set --name bilko-api-demo --resource-group rg-bilko-demo --revision-weight <previous-revision-name>=100
Source: BookStack page 3305 §6a and DEPLOY-MAP.md “Rollback (Azure)”; revision mode re-verified by read-only ACA query 2026-08-15.
TODO/UNKNOWN — production Web rollback: the live Web app is in Single revision mode, while one related wiki page still shows a traffic-weight command that is only valid for multiple-revision handling. No approved production Web rollback command consistent with live state was found in the allowed sources. Stop and escalate to FlowForge; do not reuse the API command or the stage-only rollback pipeline.
Database Rollback
CRITICAL: Flyway migrations are forward-only. Prefer a new compensating migration. Do not run ad-hoc policy drops or destructive SQL on bilko-demo-pg; it is production and shared with stage.
Point-in-time restore is a last resort: Azure PostgreSQL restore creates a new server; it does not overwrite the source. A separate, explicitly authorized application/edge cutover is required afterward.
az postgres flexible-server restore --resource-group rg-bilko-demo --name bilko-demo-pg-restored --source-server bilko-demo-pg --restore-time "<verified-UTC-timestamp>"
Source: BookStack page 3305 §5c and §7e. Live server bilko-demo-pg, PostgreSQL 16, state Ready, backup retention 7 days verified read-only on 2026-08-15.
6. Post-Deploy Smoke Tests
Run immediately after the approved production promotion. Record the pipeline run ID, commit SHA, deployed image digests, and every result.
Manual Smoke Tests
| Test | Command/Action | Expected Result | Status |
|---|---|---|---|
| 1. Demo Auth Bootstrap | curl -s -o /dev/null -w "%{http_code}\n" "https://bilko-api-demo.purplebeach-f004d490.swedencentral.azurecontainerapps.io/api/v1/auth/demo?country=HR" |
HTTP 200 | ☐ |
| 2. Real-Browser Login | TODO/UNKNOWN: no approved standalone authenticated user-info command was found without reintroducing retired password/JWT handling. Confirm the Real-browser MSAL login smoke (DEMO artifact) step in the exact Promote_Demo run instead. |
Pipeline login smoke succeeds for app.bilko.cloud | ☐ |
| 3. Invoice PDF | TODO/UNKNOWN: the current allowed sources do not identify a safe production fixture ID and authenticated PDF probe. Do not invent a customer ID or token. Add a sourced probe before treating this row as automated. | Explicitly waived by CEO or a sourced, tenant-safe probe passes | ☐ |
| 4. Health Check | curl -s https://api.bilko.cloud/api/v1/health |
HTTP 200 with the healthy Bilko API response | ☐ |
| 5. RBAC Enforcement | Open the exact Azure DevOps run and confirm E2E_UAT plus its published Playwright results succeeded before Promote_Demo. |
No failed E2E tests; approval evidence ties to the same source commit | ☐ |
Sources: live azure-pipelines.yml PI2 and real-browser login gates at azdo/main 3db716d7; BookStack page 3305 §3c/§4d.
Automated E2E Tests (Playwright)
The canonical suite runs in E2E_UAT against api-stage.bilko.cloud and web-stage.bilko.cloud. After promotion, the pipeline reruns tests/login-dashboard.spec.ts against https://app.bilko.cloud. Do not run the retired apps/web command against a dead demo hostname.
az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko --pipeline-ids 1 --branch main --top 5 -o table
Source: live azure-pipelines.yml stages E2E_UAT and Promote_Demo at azdo/main 3db716d7.
Coverage Verification
# Verify backend coverage threshold is maintained
cd apps/api
./gradlew koverVerify
Expected: exit 0. Keep this local quality gate separate from infrastructure promotion.
7. Rounds 11-27 Deliverables Manifest
This section provides traceability for all changes delivered in Rounds 11-27.
Coverage Progression
| Round | MC ID | Coverage Delta | Final Threshold | Test Count |
|---|---|---|---|---|
| Round 16 | #100336 | Baseline established | 7% | ~50 HTTP integration tests |
| Round 17 | #100336 | +4% | 11% | +110 service tests (Invoice, Expense, Contact, Report) |
| Round 18 | #100336 | +39% | 50% | +90 tests (4 streams: Banking, Auth, Stripe, Routes) |
| Round 19 | #100338 | +7% | 57% | +20 RBAC edge case tests (403 assertions) |
| Round 20 | #100353 | +3% | 60% | +81 tests (3 streams: RecurringInvoice, Compliance, Archive/Settings/Account) |
| Round 21 | #100359 | Threshold held | 60% | +75 edge case tests across 17 HttpIntegrationTest files |
| Round 22 | #100382 | Threshold held | 60% | +33 billing/webhook integration tests |
| Round 23 | — | Threshold held | 60% | Auth package coverage push (~30 tests, 4 streams) |
| Round 24 | #100396 | Regression suite | 60% | Failing-by-design specs for CEO Round 12 bugs |
| Round 27 | #100420 | Deep edge coverage | 60% | +113 deep edge tests across 8 HttpIntegrationTest files |
Production Bug Fixes (Rounds 12-26)
| Fix | MC ID | Commit SHA | Description |
|---|---|---|---|
| CORS Permanent | #100297 | 3943837 | Historical pre-Azure CORS fix (PR #110); current release configuration is in azure-pipelines.yml (see §4) |
| Double Login | #100367 | ef1ed07 | Eliminate double /auth/login call on registration (PR #111) |
| RLS Column Widening | #100406 | 660795d | V16 column widening + V17 line 50 ALTER DATABASE removal |
| bilko_app Role | Round 25 | 30d2de4 | V17 idempotent CREATE ROLE IF NOT EXISTS bilko_app before RLS policies (PR #113) |
| Demo Org Name | Round 25 | dbbd80d | V20 migration ensures demo org name is "Bilko Demo Firma" |
| Wizard Save Draft | Round 25 | 9319d02 | Step 5 "Save kao nacrt" shows localized toast + redirect |
| Refresh Token | Round 26 | b55c8dd | Eliminate refresh token consumption on login |
| HR L10n Critical | #100355 | e3ede8e | OIB 11 digits, VAT 25/13/5%, locale routing, register language dynamic (Lexicon audit) |
| Demo API Rebuild | #100400 | 9ffe3c4 | Historical pre-Azure demo build path (PR #116); superseded by Promote_Demo in azure-pipelines.yml |
Feature Deliveries
| Feature | MC ID | Commit SHA | Description |
|---|---|---|---|
| RBAC HTTP Layer | #100338 | 2a933d8 | Enforce role-check at HTTP layer on all CRUD route handlers (Round 19) |
| RLS Permissive | #100387 | 6712fcd | Phase 2A PERMISSIVE RLS migrations + OrgScope middleware |
| Trial Mechanic | #100326 | d20cb13, e0d200b | D1+D2+D4 backend + D3 frontend trial countdown banner |
| TaxJurisdiction Expansion | #100386 | e346e9b | Expand TaxJurisdiction {BA_FED, BA_RS} + CountryPlugin interface (ADR-015) |
| ADRs 015-019 | #100362 | d17f0ce | Write ADR-015/016/017/019 per Plan v3 Phase 0 |
Infrastructure Changes
| Change | MC ID | Commit SHA | Description |
|---|---|---|---|
| GCP Deploy Dormant Fix | #100110 | b1ddecf | gcp-deploy.yml service name targets bilko-{web,api}-demo (ADR-023 gap closure) |
| V14-V20 Migrations | Various | — | 7 Flyway migrations (demo seed, trial, RLS, enums, org name fix) |
8. Escalation Contacts
| Issue Type | Contact | Channel |
|---|---|---|
| Azure Container Apps Outage | Kelsey Hightower (FlowForge) | MC task assignment via John |
| Database Corruption | Bruce Momjian (CodeCraft) | MC task assignment via John |
| RBAC/Auth Issues | Parisa Tabriz (Securion) | MC task assignment via John |
| Frontend Errors | Lee Robinson (CodeCraft) | MC task assignment via John |
| CEO Approval | Alem Basic | Email: alem@alai.no, Slack: @alem |
9. Appendix: Key File Locations
| File | Path |
|---|---|
| Canonical CI/CD + promotion | azure-pipelines.yml |
| Live topology and operator commands | DEPLOY-MAP.md |
| Stage rollback rehearsal only | azure-pipelines-stage-rollback.yml |
| Flyway Migrations | apps/api/src/main/resources/db/migration/ |
| Application Config | apps/api/src/main/kotlin/no/alai/bilko/Application.kt |
| CORS Config | azure-pipelines.yml (Deploy_Stage and Promote_Demo) |
| Playwright Tests | apps/e2e/ |
| Kover Config | apps/api/build.gradle.kts |
Source: BUILD-BLUEPRINT.md, DEPLOY-MAP.md, and azure-pipelines.yml at azdo/main 3db716d7.
Document Revision History:
- v1.1 (2026-08-15): MC #107170 — replaced retired deployment, database, registry, hostname, smoke-test, and rollback commands with sourced Azure/ADO procedures; retained explicit safety stops where no current command could be established.
- v1.0 (2026-05-10): Initial runbook creation covering Rounds 11-27 deliverables (MC #100424)