Deployment Runbooks

Operational runbooks for staging, demo, and production deployments across ALAI products

Bilko Stage→Demo→Prod Cutover Runbook

Bilko Stage→Demo→Prod Cutover Runbook

Document ID: BILKO-RUNBOOK-001
Last Updated: 2026-08-15
Owner: Skillforge (ALAI Holding AS)
Status: Active — Azure canonical

Purpose

This runbook documents the promotion path from Bilko stage environment to demo and eventually prod. It incorporates all changes delivered in Rounds 11-27, including massive test coverage expansion, 7+ production bug fixes, RBAC enforcement, permanent CORS configuration, and database migrations V14-V20.

Context: Bilko now runs on Azure Container Apps in rg-bilko-demo. The apps named bilko-web-demo and bilko-api-demo are customer-facing production; app.bilko.cloud and api.bilko.cloud route to them. Stage uses bilko-web-stage and bilko-api-stage. Both environments currently share the production Azure PostgreSQL server bilko-demo-pg, so every database action in this runbook is production-sensitive. Sources: live read-only Azure query 2026-08-15; BookStack pages 3305 and 3121; DEPLOY-MAP.md at azdo/main 3db716d7.

1. Pre-Cutover Checklist

Complete ALL items before proceeding with cutover:

Item Verification Command Success Criteria Status
High/Blocker MCs Resolved node ~/system/tools/mc.js list --project bilko --priority H,BLOCKER --status open 0 open tasks, or all explicitly waived by CEO ☐
Azure DevOps Pipeline Health az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko --pipeline-ids 1 --branch main --top 5 --query "[].{id:id,status:status,result:result,sourceBranch:sourceBranch,sourceVersion:sourceVersion,finishTime:finishTime}" -o table
Source: verified read-only az query on 2026-08-15; pipeline definition azure-pipelines.yml at azdo/main 3db716d7.		 Last 5 completed runs on main = SUCCESS, or every failure is explicitly adjudicated before promotion ☐
Coverage Gate cd apps/api && ./gradlew koverVerify Exit code 0 (current threshold ≥60%) ☐
E2E Regression Tests Open the exact Azure DevOps run for the source commit and inspect E2E_UAT plus its published Playwright results.
az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko --pipeline-ids 1 --branch main --top 5 -o table
Source: live azure-pipelines.yml stage E2E_UAT at azdo/main 3db716d7.		 E2E_UAT succeeded and the published Playwright results contain no failures for the exact promoted commit ☐
Database Backup pg_dump "host=bilko-demo-pg.postgres.database.azure.com port=5432 dbname=bilko user=bilko_admin sslmode=require" --format=custom --no-acl --no-owner --file="bilko-backup-$(date +%Y%m%d-%H%M%S).dump"
Source: BookStack page 3305, §5d “Database Backup and Restore”. Supply the password through the approved secure mechanism; never paste it into this page or shell history.		 Dump exits 0 and pg_restore --list <dump-file> exits 0; backup path and checksum recorded in the cutover evidence ☐
CEO Sign-Off MC task comments or Slack approval Explicit approval captured with timestamp ☐

2. Database Migration Plan (V14-V20)

Flyway migrations are applied by the Flyway_Migrate stage in the canonical Azure DevOps pipeline, after Build and before Deploy_Stage. This section preserves the migration order and critical dependencies. Source: azure-pipelines.yml at azdo/main 3db716d7, stage Flyway_Migrate.

Migration Sequence

Version Description MC Reference Critical Notes
V14 Demo + CI test seed Round 12.7 Creates demo@bilko.rs user + test tenant for CI
V15 Trial fields (plan_tier, trial_start, trial_end) MC #100326 Adds subscription trial mechanic to organizations table
V16 RLS column widening (country constraint) MC #100406 Expands country VARCHAR to support TaxJurisdiction enum values
V17 RLS PERMISSIVE policies MC #100387 REQUIRES: bilko_app role exists. V17 includes idempotent CREATE ROLE IF NOT EXISTS bilko_app block (Round 25 fix)
V18 InvoiceStatus enum MC #100406 (drugi agent) Creates PostgreSQL ENUM for invoice states (DRAFT, SENT, PAID, etc.)
V19 ExpenseStatus enum MC #100406 (drugi agent) Creates PostgreSQL ENUM for expense states (DRAFT, APPROVED, PAID, etc.)
V20 Demo org name fix Round 25 Updates demo organization name to "Bilko Demo Firma" (Croatian/Serbian localization)

Migration Execution

Migrations are applied by Flyway 11 in the pipeline against bilko-demo-pg.postgres.database.azure.com:5432/bilko as bilko_admin. The pipeline fails closed if the database contains an applied migration newer than the checked-out source tree. Source: azure-pipelines.yml at azdo/main 3db716d7, stage Flyway_Migrate.

Manual Verification After Deployment:

Production warning: bilko-demo-pg serves customer-facing production. Use a read-only transaction for verification and do not run ad-hoc DDL/DML.

# Connect to Azure PostgreSQL Flexible Server
psql "host=bilko-demo-pg.postgres.database.azure.com port=5432 dbname=bilko user=bilko_admin sslmode=require"

# In psql, enter a read-only transaction before checks
BEGIN READ ONLY;

SELECT version, description, success, installed_on
FROM flyway_schema_history
ORDER BY installed_rank DESC
LIMIT 10;

SELECT id, name FROM organizations WHERE name = 'Bilko Demo Firma';
ROLLBACK;

Source: BookStack page 3305, §5b and §7a; database identity re-verified by read-only Azure query on 2026-08-15.

3. Azure Container Apps Image Promotion Sequence

Stage → Demo (Production) Promotion

Environment classification: the resources with -demo in their names are production. The canonical pipeline promotes API and Web together from one commit; do not copy a stage tag into production or call az containerapp update manually from this runbook.

Sources: live read-only ACA/ACR query 2026-08-15; DEPLOY-MAP.md and azure-pipelines.yml at azdo/main 3db716d7.

Promotion Commands

API Service

# Queue the canonical pipeline from main. It builds, scans, migrates, deploys stage,
# runs E2E UAT, then waits for the bilko-demo environment approval.
az pipelines run --org https://dev.azure.com/alai-holding --project Bilko --branch main --id 1

# Monitor recent main runs; record the run ID and sourceVersion in cutover evidence.
az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko   --pipeline-ids 1 --branch main --top 5 -o table

Do not approve Promote_Demo until CI_Gates → Build → Flyway_Migrate → Deploy_Stage → E2E_UAT are green and the pre-cutover checklist is complete. Approval is performed in Azure DevOps under Pipelines → Environments → bilko-demo. The approved stage rebuilds demo-{SHA} images from the same commit, scans them, deploys digest-pinned images, and shifts bilko-api-demo to the new revision.

Source: BookStack page 3305 §3b and live azure-pipelines.yml Promote_Demo at azdo/main 3db716d7.

Web Service

There is no separate Web promotion command. The same approved Bilko-CI-CD run builds and deploys bilko-api-demo and bilko-web-demo from the same source commit. A split API/Web manual promotion would bypass the canonical release contract.

Source: live azure-pipelines.yml Promote_Demo deployment job at azdo/main 3db716d7.

Traffic Shift (Gradual Rollout)

TODO/UNKNOWN — intentional safety stop: no reviewed gradual 10/90 production rollout command exists in the allowed current sources. The canonical pipeline shifts bilko-api-demo 100% to its latest revision; live state shows API in Multiple revision mode and Web in Single mode. If gradual rollout is required, stop here and obtain a FlowForge-authored, revision-specific procedure. Do not translate the retired traffic-weight commands by analogy.

Source for current behavior: live azure-pipelines.yml Promote_Demo and read-only ACA query 2026-08-15.

4. Cloudflare Routing + CORS Verification

DNS / Routing Configuration

Customer-facing production URLs:

The retired *.alai.no demo hosts are not release targets and must not be used in health checks or browser tests.

Sources: BookStack page 3121; DEPLOY-MAP.md at azdo/main 3db716d7; live read-only ACA query 2026-08-15.

CORS Configuration

CORS is set by the canonical pipeline, not by a standalone operator update:

Source: live azure-pipelines.yml Deploy_Stage and Promote_Demo environment updates at azdo/main 3db716d7.

CORS / Route Verification

# Public production health (current brand domains)
curl -s https://api.bilko.cloud/api/v1/health
curl -sI https://app.bilko.cloud/ | head -1

# Pipeline evidence: E2E_UAT performs browser-level CSP/CORS-sensitive flows on stage.
az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko   --pipeline-ids 1 --branch main --top 5 -o table

Success requires HTTP 200 on both public routes and a green E2E_UAT stage for the exact commit being promoted. The retired password-login preflight is invalid because Bilko is Entra/demo-session based; do not recreate it.

Sources: BookStack page 3305 §3c/§4d and live azure-pipelines.yml E2E/PI2 checks at azdo/main 3db716d7.

5. Rollback Procedure

Azure Container Apps Rollback

Live state verified 2026-08-15: bilko-api-demo uses Multiple revision mode; bilko-web-demo uses Single. Record the current revision and image before any rollback.

# API: list revisions and their state
az containerapp revision list --name bilko-api-demo --resource-group rg-bilko-demo   --query "[].{name:name,created:properties.createdTime,active:properties.active}" -o table

# API: after explicit rollback authorization, shift 100% to the reviewed known-good revision
az containerapp ingress traffic set --name bilko-api-demo --resource-group rg-bilko-demo   --revision-weight <previous-revision-name>=100

Source: BookStack page 3305 §6a and DEPLOY-MAP.md “Rollback (Azure)”; revision mode re-verified by read-only ACA query 2026-08-15.

TODO/UNKNOWN — production Web rollback: the live Web app is in Single revision mode, while one related wiki page still shows a traffic-weight command that is only valid for multiple-revision handling. No approved production Web rollback command consistent with live state was found in the allowed sources. Stop and escalate to FlowForge; do not reuse the API command or the stage-only rollback pipeline.

Database Rollback

CRITICAL: Flyway migrations are forward-only. Prefer a new compensating migration. Do not run ad-hoc policy drops or destructive SQL on bilko-demo-pg; it is production and shared with stage.

Point-in-time restore is a last resort: Azure PostgreSQL restore creates a new server; it does not overwrite the source. A separate, explicitly authorized application/edge cutover is required afterward.

az postgres flexible-server restore   --resource-group rg-bilko-demo   --name bilko-demo-pg-restored   --source-server bilko-demo-pg   --restore-time "<verified-UTC-timestamp>"

Source: BookStack page 3305 §5c and §7e. Live server bilko-demo-pg, PostgreSQL 16, state Ready, backup retention 7 days verified read-only on 2026-08-15.

6. Post-Deploy Smoke Tests

Run immediately after the approved production promotion. Record the pipeline run ID, commit SHA, deployed image digests, and every result.

Manual Smoke Tests

TestCommand/ActionExpected ResultStatus
1. Demo Auth Bootstrap curl -s -o /dev/null -w "%{http_code}\n" "https://bilko-api-demo.purplebeach-f004d490.swedencentral.azurecontainerapps.io/api/v1/auth/demo?country=HR" HTTP 200☐
2. Real-Browser Login TODO/UNKNOWN: no approved standalone authenticated user-info command was found without reintroducing retired password/JWT handling. Confirm the Real-browser MSAL login smoke (DEMO artifact) step in the exact Promote_Demo run instead. Pipeline login smoke succeeds for app.bilko.cloud☐
3. Invoice PDF TODO/UNKNOWN: the current allowed sources do not identify a safe production fixture ID and authenticated PDF probe. Do not invent a customer ID or token. Add a sourced probe before treating this row as automated. Explicitly waived by CEO or a sourced, tenant-safe probe passes☐
4. Health Check curl -s https://api.bilko.cloud/api/v1/health HTTP 200 with the healthy Bilko API response☐
5. RBAC Enforcement Open the exact Azure DevOps run and confirm E2E_UAT plus its published Playwright results succeeded before Promote_Demo. No failed E2E tests; approval evidence ties to the same source commit☐

Sources: live azure-pipelines.yml PI2 and real-browser login gates at azdo/main 3db716d7; BookStack page 3305 §3c/§4d.

Automated E2E Tests (Playwright)

The canonical suite runs in E2E_UAT against api-stage.bilko.cloud and web-stage.bilko.cloud. After promotion, the pipeline reruns tests/login-dashboard.spec.ts against https://app.bilko.cloud. Do not run the retired apps/web command against a dead demo hostname.

az pipelines runs list --org https://dev.azure.com/alai-holding --project Bilko   --pipeline-ids 1 --branch main --top 5 -o table

Source: live azure-pipelines.yml stages E2E_UAT and Promote_Demo at azdo/main 3db716d7.

Coverage Verification

# Verify backend coverage threshold is maintained
cd apps/api
./gradlew koverVerify

Expected: exit 0. Keep this local quality gate separate from infrastructure promotion.

7. Rounds 11-27 Deliverables Manifest

This section provides traceability for all changes delivered in Rounds 11-27.

Coverage Progression

Round MC ID Coverage Delta Final Threshold Test Count
Round 16 #100336 Baseline established 7% ~50 HTTP integration tests
Round 17 #100336 +4% 11% +110 service tests (Invoice, Expense, Contact, Report)
Round 18 #100336 +39% 50% +90 tests (4 streams: Banking, Auth, Stripe, Routes)
Round 19 #100338 +7% 57% +20 RBAC edge case tests (403 assertions)
Round 20 #100353 +3% 60% +81 tests (3 streams: RecurringInvoice, Compliance, Archive/Settings/Account)
Round 21 #100359 Threshold held 60% +75 edge case tests across 17 HttpIntegrationTest files
Round 22 #100382 Threshold held 60% +33 billing/webhook integration tests
Round 23 — Threshold held 60% Auth package coverage push (~30 tests, 4 streams)
Round 24 #100396 Regression suite 60% Failing-by-design specs for CEO Round 12 bugs
Round 27 #100420 Deep edge coverage 60% +113 deep edge tests across 8 HttpIntegrationTest files

Production Bug Fixes (Rounds 12-26)

Fix MC ID Commit SHA Description
CORS Permanent #100297 3943837 Historical pre-Azure CORS fix (PR #110); current release configuration is in azure-pipelines.yml (see §4)
Double Login #100367 ef1ed07 Eliminate double /auth/login call on registration (PR #111)
RLS Column Widening #100406 660795d V16 column widening + V17 line 50 ALTER DATABASE removal
bilko_app Role Round 25 30d2de4 V17 idempotent CREATE ROLE IF NOT EXISTS bilko_app before RLS policies (PR #113)
Demo Org Name Round 25 dbbd80d V20 migration ensures demo org name is "Bilko Demo Firma"
Wizard Save Draft Round 25 9319d02 Step 5 "Save kao nacrt" shows localized toast + redirect
Refresh Token Round 26 b55c8dd Eliminate refresh token consumption on login
HR L10n Critical #100355 e3ede8e OIB 11 digits, VAT 25/13/5%, locale routing, register language dynamic (Lexicon audit)
Demo API Rebuild #100400 9ffe3c4 Historical pre-Azure demo build path (PR #116); superseded by Promote_Demo in azure-pipelines.yml

Feature Deliveries

Feature MC ID Commit SHA Description
RBAC HTTP Layer #100338 2a933d8 Enforce role-check at HTTP layer on all CRUD route handlers (Round 19)
RLS Permissive #100387 6712fcd Phase 2A PERMISSIVE RLS migrations + OrgScope middleware
Trial Mechanic #100326 d20cb13, e0d200b D1+D2+D4 backend + D3 frontend trial countdown banner
TaxJurisdiction Expansion #100386 e346e9b Expand TaxJurisdiction {BA_FED, BA_RS} + CountryPlugin interface (ADR-015)
ADRs 015-019 #100362 d17f0ce Write ADR-015/016/017/019 per Plan v3 Phase 0

Infrastructure Changes

Change MC ID Commit SHA Description
GCP Deploy Dormant Fix #100110 b1ddecf gcp-deploy.yml service name targets bilko-{web,api}-demo (ADR-023 gap closure)
V14-V20 Migrations Various — 7 Flyway migrations (demo seed, trial, RLS, enums, org name fix)

8. Escalation Contacts

Issue Type Contact Channel
Azure Container Apps Outage Kelsey Hightower (FlowForge) MC task assignment via John
Database Corruption Bruce Momjian (CodeCraft) MC task assignment via John
RBAC/Auth Issues Parisa Tabriz (Securion) MC task assignment via John
Frontend Errors Lee Robinson (CodeCraft) MC task assignment via John
CEO Approval Alem Basic Email: alem@alai.no, Slack: @alem

9. Appendix: Key File Locations

FilePath
Canonical CI/CD + promotionazure-pipelines.yml
Live topology and operator commandsDEPLOY-MAP.md
Stage rollback rehearsal onlyazure-pipelines-stage-rollback.yml
Flyway Migrationsapps/api/src/main/resources/db/migration/
Application Configapps/api/src/main/kotlin/no/alai/bilko/Application.kt
CORS Configazure-pipelines.yml (Deploy_Stage and Promote_Demo)
Playwright Testsapps/e2e/
Kover Configapps/api/build.gradle.kts

Source: BUILD-BLUEPRINT.md, DEPLOY-MAP.md, and azure-pipelines.yml at azdo/main 3db716d7.

Document Revision History:

Promote_Demo kapija — zamjena Write_UAT uslova (MC #900135, 2026-08-22)

Promote_Demo kapija — zamjena Write_UAT uslova (MC #900135, 2026-08-22)

Odluka CEO-a: DA (2026-08-22). Ovo je zapis odluke, ne prijedlog.

Problem

azure-pipelines.yml:2090-2101, stage Promote_Demo:

condition: and( succeeded('E2E_UAT'), succeeded('Write_UAT'), or(Manual, refs/tags/v) )
dependsOn: [ Build, Write_UAT ]

Write_UAT je ljuska (MC #900104, mjereno 2026-08-21):

Dakle kapija ne štiti ništa — a blokira sve.

Šta je konkretno blokirala

Ispravan pravni subjekt (SMART FORGE d.o.o.) je na mainu u tri fajla apps/web, ali nije živ na app.bilko.cloud — tamo i dalje piše ALAI Holding AS. To je business_profile.url uslov za Stripe prijavu (MC #900044/#900048), dakle kapija je stajala na putu prihodu.

Ručno guranje na demo nije opcija: Bilko Stage→Demo→Prod Cutover Runbook izričito zabranjuje kopiranje stage oznake u produkciju i ručnu izmjenu container appa iz runbooka. Jedini put je kroz liniju.

Izmjena

condition: and( succeeded('E2E_UAT'), succeeded('Build'), or(eq(Build.Reason,'Manual'), startsWith(Build.SourceBranch,'refs/tags/v')) )
dependsOn: [ Build, E2E_UAT ]

Write_UAT ostaje u liniji i dalje se izvršava — samo prestaje biti uslov za promote.

Šta ostaje zaštićeno (namjerno se NE dira)

  1. Ljudsko odobrenje — deployment: approve_and_deploy_demo, environment: 'bilko-demo'. Nijedan promote ne ide bez klika.
  2. Digest-pinned deploy iz Build outputa (API_DIGEST / WEB_DIGEST).
  3. succeeded('E2E_UAT'), ne not(failed(...)) — fail-open klasa iz MC #107171/#107242 se NE uvodi nazad.
  4. Samo Manual ili v* tag.

Dokazni minimum prije klika na odobrenje

Nije u liniji nego je Johnov korak, po roadmap pravilu "promote samo s dokazom":

Šta ovo NE rješava

Gradnja prave write-UAT kontrolne ravni ostaje MC #900104 — po ugovoru docs/testing/WRITE-UAT-CONTRACT.md to je 10 blokirajućih preflight kapija, dokazi izolacije i uloga, paket dokaza, quiesce/cleanup/rotacija. To je pojačanje, ne blokator prihoda, i radi se odvojeno.