Bilko Stage→Demo→Prod Cutover Runbook

Bilko Stage→Demo→Prod Cutover Runbook

Document ID: BILKO-RUNBOOK-001
Last Updated: 2026-05-1008-15
Owner: Skillforge (ALAI Holding AS)
Status: Active — Azure canonical

Purpose

This runbook documents the promotion path from Bilko stage environment to demo and eventually prod. It incorporates all changes delivered in Rounds 11-27, including massive test coverage expansion, 7+ production bug fixes, RBAC enforcement, permanent CORS configuration, and database migrations V14-V20.

Context: As of 2026-05-10, Bilko now runs on Azure Container Apps in rg-bilko-demo. The apps named bilko-web-demo and bilko-api-demo are customer-facing production; app.bilko.cloud and api.bilko.cloud route to them. Stage uses bilko-web-stage and bilko-api-stage. Both environments currently share the production Azure PostgreSQL server bilko-demo-pg, so every database action in this runbook is onproduction-sensitive. revisionSources: 00185-faylive (image stage-3a4a9e9). Demo environment has read-only aAzure subsetquery of2026-08-15; theseBookStack changes.pages Prod does not exist as a separate environment yet3305 and currently3121; usesDEPLOY-MAP.md demoat asazdo/main customer-facing.3db716d7.

1. Pre-Cutover Checklist

Complete ALL items before proceeding with cutover:

2. Database Migration Plan (V14-V20)

Flyway migrations are applied automaticallyby whenthe CloudFlyway_Migrate Runstage starts.in the canonical Azure DevOps pipeline, after Build and before Deploy_Stage. This section documentspreserves the migration order and critical dependencies. Source: azure-pipelines.yml at azdo/main 3db716d7, stage Flyway_Migrate.

Migration Sequence

Migration Execution

Migrations are applied automatically by Flyway during Cloud Run startup via FlywayPlugin11 in the pipeline against apps/api/src/main/kotlin/no/alai/bilko/Application.ktbilko-demo-pg.postgres.database.azure.com:5432/bilko as bilko_admin. The pipeline fails closed if the database contains an applied migration newer than the checked-out source tree. Source: azure-pipelines.yml at azdo/main 3db716d7, stage Flyway_Migrate.

Manual Verification After Deployment:

Production warning: bilko-demo-pg serves customer-facing production. Use a read-only transaction for verification and do not run ad-hoc DDL/DML.

# Connect to Azure PostgreSQL Flexible Server
psql "host=bilko-demo-dbpg.postgres.database.azure.com gcloudport=5432 sqldbname=bilko connect bilko-demo-db --user=bilko_admin --project=tribal-sign-487920-k0sslmode=require"

# CheckIn Flywaypsql, schemaenter historya read-only transaction before checks
BEGIN READ ONLY;

SELECT version, description, success, installed_on
FROM flyway_schema_history
ORDER BY installed_rank DESC
LIMIT 10;

# Verify V20 demo org name
SELECT id, name FROM organizations WHERE name = 'Bilko Demo Firma';
ROLLBACK;

Source: BookStack page 3305, §5b and §7a; database identity re-verified by read-only Azure query on 2026-08-15.

3. CloudAzure RunContainer Apps Image Promotion Sequence

Stage → Demo (Production) Promotion

CurrentEnvironment Stateclassification: (2026-05-10):the resources with -demo in their names are production. The canonical pipeline promotes API and Web together from one commit; do not copy a stage tag into production or call az containerapp update manually from this runbook.

• Stage API: bilko-api-stage — image europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/bilkodemo.azurecr.io/bilko-api:stage-3a4a9e9{SHA}@sha256:{digest}
• DemoProduction API: bilko-api-demo — image TBD (promote from stage)bilkodemo.azurecr.io/bilko-api:demo-{SHA}@sha256:{digest}
• Stage Web: bilko-web-stage — image europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/bilkodemo.azurecr.io/bilko-web:stage-<SHA>{SHA}@sha256:{digest}
• DemoProduction Web: bilko-web-demo — image TBD (promote from stage)bilkodemo.azurecr.io/bilko-web:demo-{SHA}@sha256:{digest}

Sources: live read-only ACA/ACR query 2026-08-15; DEPLOY-MAP.md and azure-pipelines.yml at azdo/main 3db716d7.

Promotion Commands

API Service

# GetQueue currentthe canonical pipeline from main. It builds, scans, migrates, deploys stage,
# runs E2E UAT, then waits for the bilko-demo environment approval.
az pipelines run --org https://dev.azure.com/alai-holding --project Bilko --branch main --id 1

# Monitor recent main runs; record the run ID and sourceVersion in cutover evidence.
az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko   --pipeline-ids 1 --branch main --top 5 -o table

Do not approve Promote_Demo until CI_Gates → Build → Flyway_Migrate → Deploy_Stage → E2E_UAT are green and the pre-cutover checklist is complete. Approval is performed in Azure DevOps under Pipelines → Environments → bilko-demo. The approved stage imagerebuilds SHAdemo-{SHA} STAGE_SHA=$(gcloudimages runfrom servicesthe describesame commit, scans them, deploys digest-pinned images, and shifts bilko-api-stage \ --region=europe-north1 \ --project=tribal-sign-487920-k0 \ --format='value(spec.template.spec.containers[0].image)' | \ awk -F':' '{print $2}') echo "Promoting API image: stage-${STAGE_SHA}" # Promotedemo to demothe (usenew gcloud-write.shrevision.

Source: safety)BookStack bashpage ~/system/tools/gcloud-write.sh3305 --mc§3b 100424and runlive servicesazure-pipelines.yml updatePromote_Demo bilko-api-demoat \azdo/main --region=europe-north1 \ --project=tribal-sign-487920-k0 \ --image=europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/api:stage-${STAGE_SHA} 3db716d7.

Web Service

There is no separate Web promotion command. The same approved # Get current stage web image SHA
WEB_SHA=$(gcloudBilko-CI-CD run servicesbuilds describeand deploys bilko-web-stageapi-demo \and --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --format='value(spec.template.spec.containers[0].image)' | \
  awk -F':' '{print $2}')

echo "Promoting Web image: stage-${WEB_SHA}"

# Promote to demo
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update bilko-web-demo \from --region=europe-north1the \same --project=tribal-sign-487920-k0source \commit. --image=europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web:stage-${WEB_SHA}A split API/Web manual promotion would bypass the canonical release contract.

Source: live azure-pipelines.yml

Traffic Shift (Gradual Rollout)

TODO/UNKNOWN — intentional safety stop: no reviewed gradual 10/90 production rollout command exists in the allowed current sources. The canonical pipeline shifts bilko-api-demo 100% to its latest revision; live state shows API in Multiple revision mode and Web in Single mode. If gradual rollout is needed:required, stop here and obtain a FlowForge-authored, revision-specific procedure. Do not translate the retired traffic-weight commands by analogy.

Source for current behavior: live #azure-pipelines.yml SplitPromote_Demo trafficand 10%read-only newACA revision,query 90% old
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update-traffic bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --to-revisions=bilko-api-demo-00186-xyz=10,bilko-api-demo-00185-abc=90

# After soak period (30 min), shift to 100%
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update-traffic bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --to-latest

4. Cloudflare DNSRouting + CORS Verification

DNS / Routing Configuration

DemoCustomer-facing Environmentproduction URLs:

• API: https://bilko-demo-api.alai.nobilko.cloud → CloudCloudflare RunWorker bilko-edge-proxy → ACA bilko-api-demo
• Web: https://bilko-demo.alai.noapp.bilko.cloud → CloudCloudflare RunWorker bilko-edge-proxy → ACA bilko-web-demo
• Stage API: https://api-stage.bilko.cloud → direct ACA custom-domain binding on bilko-api-stage
• Stage Web: https://web-stage.bilko.cloud → direct ACA custom-domain binding on bilko-web-stage

DNSThe records managed in Cloudflare dashboard underretired *.alai.no zone.demo hosts are not release targets and must not be used in health checks or browser tests.

Sources: BookStack page 3121; DEPLOY-MAP.md at azdo/main 3db716d7; live read-only ACA query 2026-08-15.

CORS Configuration

Permanent CORS fix delivered in Round 12.8 (PR #110, MC #100297):

CORS_ORIGINS env var is permanently set inby the canonical pipeline, not by a standalone operator update:

• Stage API: cloudbuild-https://bilko-web-stage.yamlpurplebeach-f004d490.swedencentral.azurecontainerapps.io, https://web-stage.bilko.cloud, and localhost for development.
• Production API: https://app.bilko.cloud, https://app.bilko.io, https://app.bilko.company, and localhost for development.

Source: live azure-pipelines.yml Deploy_Stage and cloudbuild-demo-api.yamlPromote_Demo: environment updates at azdo/main 3db716d7.

- name: 'gcr.io/cloud-builders/gcloud'
  args:
    - 'run'
    - 'services'
    - 'update'
    - 'bilko-api-demo'
    - '--region=europe-north1'
    - '--update-env-vars=CORS_ORIGINS=https://bilko-demo.alai.no,http://localhost:3000'

CORS Preflight/ Route Verification

# TestPublic CORSproduction preflighthealth from(current demobrand domaindomains)
curl -X OPTIONSs https://bilko-demo-api.alai.no/auth/loginbilko.cloud/api/v1/health
\curl -H "Origin:sI https://bilko-demo.alai.no"app.bilko.cloud/ \| head -H "Access-Control-Request-Method: POST" \
  -H "Access-Control-Request-Headers: Content-Type" \
  -v1

# ExpectedPipeline headersevidence: inE2E_UAT response:performs #browser-level Access-Control-Allow-Origin:CSP/CORS-sensitive flows on stage.
az pipelines runs list --org https://bilko-demo.alai.nodev.azure.com/alai-holding #--project Access-Control-Allow-Methods:Bilko   GET,--pipeline-ids POST,1 PUT,--branch DELETE,main OPTIONS--top #5 Access-Control-Allow-Headers:-o Content-Type, Authorization
# Access-Control-Allow-Credentials: truetable

Success requires HTTP 200 on both public routes and a green E2E_UAT stage for the exact commit being promoted. The retired password-login preflight is invalid because Bilko is Entra/demo-session based; do not recreate it.

Sources: BookStack page 3305 §3c/§4d and live azure-pipelines.yml E2E/PI2 checks at azdo/main 3db716d7.

5. Rollback Procedure

CloudAzure RunContainer Apps Rollback

CloudLive Runstate keepsverified all2026-08-15: previousbilko-api-demo revisionsuses forMultiple 60revision days.mode; Rollbackbilko-web-demo shiftsuses trafficSingle. toRecord athe previouscurrent stablerevision revision.and image before any rollback.

# ListAPI: recentlist revisions gcloudand runtheir revisionsstate
az containerapp revision list \
  --service=bilko-api-demo \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0 \
  --limit=10

# Identify previous stable revision (e.g., bilko-api-demo-00185-fay)
PREV_REV="bilko-api-demo-00185-fay"

# Rollback to previous revision
bash ~/system/tools/gcloud-write.sh --mc 100424 run services update-trafficname bilko-api-demo \--resource-group rg-bilko-demo   --region=europe-north1query \"[].{name:name,created:properties.createdTime,active:properties.active}" -o table

# API: after explicit rollback authorization, shift 100% to the reviewed known-good revision
az containerapp ingress traffic set --project=tribal-sign-487920-k0name \bilko-api-demo --to-revisions=${PREV_REV}resource-group rg-bilko-demo   --revision-weight <previous-revision-name>=100

Source: BookStack page 3305 §6a and DEPLOY-MAP.md “Rollback (Azure)”; revision mode re-verified by read-only ACA query 2026-08-15.

TODO/UNKNOWN — production Web rollback: the live Web app is in Single revision mode, while one related wiki page still shows a traffic-weight command that is only valid for multiple-revision handling. No approved production Web rollback command consistent with live state was found in the allowed sources. Stop and escalate to FlowForge; do not reuse the API command or the stage-only rollback pipeline.

Database Rollback

CRITICAL: Flyway migrations are forward-only. TherePrefer a new compensating migration. Do not run ad-hoc policy drops or destructive SQL on bilko-demo-pg; it is NOproduction automaticand rollbackshared forwith schema changes.stage.

IfPoint-in-time V17+restore RLSis Policiesa Causelast Issues:resort: Azure PostgreSQL restore creates a new server; it does not overwrite the source. A separate, explicitly authorized application/edge cutover is required afterward.

--az Disablepostgres RLSflexible-server policies manually (Cloud SQL console or psql)
DROP POLICY IF EXISTS org_isolation ON invoices;
DROP POLICY IF EXISTS org_isolation ON expenses;
DROP POLICY IF EXISTS org_isolation ON contacts;
DROP POLICY IF EXISTS org_isolation ON transactions;
DROP POLICY IF EXISTS org_isolation ON bank_accounts;
DROP POLICY IF EXISTS org_isolation ON recurring_invoices;restore   --resource-group Orrg-bilko-demo   disable--name RLSbilko-demo-pg-restored   entirely--source-server (NOTbilko-demo-pg   recommended--restore-time for prod)
ALTER TABLE invoices DISABLE ROW LEVEL SECURITY;"<verified-UTC-timestamp>"

DatabaseSource: Point-in-TimeBookStack Recoverypage (PITR):

§5c and §7e. Live server # Clone database to a specific point in time (last known good state)
gcloud sql instances clone bilko-demo-dbpg, bilko-demo-db-rollbackPostgreSQL \16, --point-in-time="state Ready, backup retention 7 days verified read-only on 2026-05-10T10:00:00.000Z" \
  --project=tribal-sign-487920-k0

# Swap Cloud Run to rollback instance (update DATABASE_URL secret)
# This is a LAST RESORT — requires DNS cutover + application restart

6. Post-Deploy Smoke Tests

Run immediately after cutoverthe toapproved verifyproduction corepromotion. functionality.Record the pipeline run ID, commit SHA, deployed image digests, and every result.

Manual Smoke Tests

Sources: live azure-pipelines.yml PI2 and real-browser login gates at azdo/main 3db716d7; BookStack page 3305 §3c/§4d.

Automated E2E Tests (Playwright)

The canonical suite runs in # Run Playwright testsE2E_UAT against demoapi-stage.bilko.cloud environmentand cdweb-stage.bilko.cloud. apps/webAfter PLAYWRIGHT_BASE_URL=promotion, the pipeline reruns tests/login-dashboard.spec.ts against https://bilko-demo.alai.noapp.bilko.cloud. npmDo not run test:e2ethe #retired Expected:apps/web 13/15command PASSagainst #a Knowndead failuresdemo (Roundhostname.
12):
az #pipelines runs list --org session persistence (3 specs)
#https://dev.azure.com/alai-holding --project navigation edge cases
#Bilko   --pipeline-ids logout1 redirect--branch main --top 5 -o table

Source: live azure-pipelines.yml stages E2E_UAT and Promote_Demo at azdo/main 3db716d7.

Coverage Verification
# Verify backend coverage threshold is maintained
cd apps/api
./gradlew koverVerify
#
Expected: EXITexit 00. (thresholdKeep ≥60%this perlocal Roundquality 20)gate 

7. Rounds 11-27 Deliverables Manifest

This section provides traceability for all changes delivered in Rounds 11-27.

Coverage Progression

Production Bug Fixes (Rounds 12-26)

Feature Deliveries

Infrastructure Changes

8. Escalation Contacts

9. Appendix: Key File Locations

Source: BUILD-BLUEPRINT.md, DEPLOY-MAP.md, and azure-pipelines.yml at azdo/main 3db716d7.

Document Revision History:

• v1.1 (2026-08-15): MC #107170 — replaced retired deployment, database, registry, hostname, smoke-test, and rollback commands with sourced Azure/ADO procedures; retained explicit safety stops where no current command could be established.
• v1.0 (2026-05-10): Initial runbook creation covering Rounds 11-27 deliverables (MC #100424)