# Bilko CI/CD — Stage→Prod Pipeline (MC #99477)

## Overview

**Stage pipeline:** push-main → bilko-stage-auto-deploy → cloudbuild-stage.yaml → bilko-{web,api}-stage

**Prod pipeline:** tag v\* → bilko-main-deploy → cloudbuild.yaml → bilko-{web,api}

Stage pipeline is optimized for **FAST FEEDBACK** — no quality gates. Prod pipeline has 8 production gates including SHA verification, Trivy scanning, Flyway migrations, and Cloud Build native approval.

## Stage Pipeline

<table id="bkmrk-step-purpose-image-t"><thead><tr><th>Step</th><th>Purpose</th><th>Image Tag</th><th>Duration (avg)</th></tr></thead><tbody><tr><td>sanity-check</td><td>Verify Docker socket + Artifact Registry reachability (environment health, NOT a quality gate)</td><td>—</td><td>~2.3s</td></tr><tr><td>build-web</td><td>Build Next.js app with docker buildx (apps/web/Dockerfile)</td><td>:stage-${SHORT\_SHA}  
:stage-latest</td><td>~3m</td></tr><tr><td>push-web</td><td>Push image to Artifact Registry (europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web)</td><td>—</td><td>~7s</td></tr><tr><td>migrate-db</td><td>Run Flyway migrations against Cloud SQL bilko-staging-db (POSTGRES\_16) via Cloud SQL proxy</td><td>—</td><td>~22s</td></tr><tr><td>deploy-web-stage</td><td>Deploy bilko-web-stage Cloud Run service with :stage-${SHORT\_SHA} image, --no-traffic</td><td>—</td><td>~39s</td></tr><tr><td>promote-web-stage</td><td>Route 100% traffic to new revision (no canary for stage)</td><td>—</td><td>~10s</td></tr><tr><td>deploy-api-stage</td><td>Deploy bilko-api-stage (redeploys EXISTING image only — no API build step, see OCD-1)</td><td>—</td><td>~19s</td></tr><tr><td>smoke-test</td><td>curl -sf https://bilko-api-stage-dh4m46blja-lz.a.run.app/api/v1/health — exit 1 if non-200</td><td>—</td><td>~2.5s</td></tr></tbody></table>

**Total duration:** ~5 minutes (build 6f2236f6, validated 2026-05-06)

## Prod Pipeline

Existing prod pipeline (cloudbuild.yaml) has 8 gates and MUST NOT be rewritten. References:

- SHA verification (Git commit SHA in image metadata)
- Trivy vulnerability scanning
- Flyway migration validation
- Cloud Build native approval (approval\_required=true in modules/build/main.tf)
- Smoke tests (health endpoint + web homepage)
- Gradual traffic rollout (0% → 100%)
- Rollback on smoke test failure

**Prod pipeline is BLOCKED** on OCD-5 (bilko-db Cloud SQL instance does not exist — requires CEO approval for provisioning).

## Triggers

<table id="bkmrk-trigger-name-filenam"><thead><tr><th>Trigger Name</th><th>Filename</th><th>Branch/Tag</th><th>Approval</th><th>Service Account</th></tr></thead><tbody><tr><td>bilko-stage-auto-deploy</td><td>infrastructure/gcp/cloudbuild-stage.yaml</td><td>^main$</td><td>No (auto-deploy)</td><td>762788903040@cloudbuild.gserviceaccount.com</td></tr><tr><td>bilko-main-deploy</td><td>infrastructure/gcp/cloudbuild.yaml</td><td>v\* (semver tag)</td><td>Yes (Cloud Build UI)</td><td>762788903040@cloudbuild.gserviceaccount.com</td></tr></tbody></table>

GCP project: `tribal-sign-487920-k0`, region: `europe-north1`

## Open Risks — 5 CEO Decisions Required

These items require CEO judgment and are NOT resolved in this implementation:

### OCD-1: bilko-api Build Pipeline Gap

**Status:** OPEN — BLOCKER for API continuous delivery

**Current state:** bilko-api-stage is live and serving traffic at [https://bilko-api-stage-dh4m46blja-lz.a.run.app/api/v1](https://bilko-api-stage-dh4m46blja-lz.a.run.app/api/v1) with image `api:stage-b7e8a59`. No Cloud Build pipeline exists for the Kotlin/Ktor API. Dockerfile path unconfirmed.

**Impact:** Stage cloudbuild-stage.yaml `deploy-api-stage` step redeploys the EXISTING API image only — cannot build new API images. API deployments must be manual via `gcloud run deploy` until resolved.

**CEO decisions needed:**

1. What is the canonical Dockerfile path for apps/api?
2. Should API have its own Cloud Build step in cloudbuild-stage.yaml or a separate trigger?
3. Is bilko-api currently deployed manually via `gcloud run deploy`?

### OCD-2: Stage Hostname — bilko-stage.alai.no vs Raw .run.app URL

**Status:** OPEN — affects CORS configuration

**Current state:** ENV-MATRIX.md CORS\_ORIGINS for staging references `staging.bilko.io` (STALE). terraform.tfvars stage\_api\_url points to raw `.a.run.app` URL. Stage pipeline uses raw .run.app URL as default.

**Impact:** Frontend CORS errors if staging.bilko.io DNS is ever pointed at stage services.

**CEO decision needed:** Should `bilko-stage.alai.no` be the canonical stage hostname? If yes: Cloudflare DNS entry (manual — not in Bilko TF stack) + CORS\_ORIGINS update required via separate MC.

### OCD-3: Postgres Version Mismatch — Stage POSTGRES\_16 vs Prod POSTGRES\_15

**Status:** OPEN — CRITICAL for financial data integrity

**Current state:** bilko-staging-db runs POSTGRES\_16 (confirmed live). envs/prod/main.tf line 94 specifies POSTGRES\_15 for prod (bilko-db does not exist yet — see OCD-5). Stage validates migrations and queries against PG16; prod would run PG15.

**Impact:** For a financial accounting SaaS, stage validation on PG16 while prod runs PG15 invalidates the "stage-as-test-environment" premise. Schema compatibility unverified. SQL dialect differences (PG15→PG16) may surface as prod-only bugs.

**CEO decision needed:** Upgrade prod to POSTGRES\_16 (requires maintenance window, pg\_upgrade or dump/restore) OR downgrade stage to POSTGRES\_15? ALAI standard tech stack (ALAI/CLAUDE.md) mandates POSTGRES\_16 for all products, suggesting prod config is non-compliant.

### OCD-4: Stage → Prod SHA Promotion Strategy

**Status:** OPEN — architectural decision

**Current state:** Prod trigger fires on semver tag push, rebuilds from source. Stage-validated image digest is NOT carried to prod build. Stage tests one SHA and prod deploys a different build. If a hot dependency updates between stage build and prod build (e.g., npm registry serves new patch version), stage and prod can diverge on identical Git SHAs.

**CEO decision needed:**

1. **Option A:** Accept rebuild-on-tag (simpler, current model) with acknowledgment of hot-dependency risk.
2. **Option B:** Implement digest promotion where prod trigger accepts an image digest input parameter and skips rebuild. Requires Cloud Build trigger API call from a promotion script or Google Cloud Deploy.

### OCD-5: Prod Cloud SQL bilko-db Existence

**Status:** OPEN — BLOCKER for prod terraform apply

**Current state:** `gcloud sql instances list --project=tribal-sign-487920-k0` shows ONLY bilko-staging-db. No bilko-db (prod) exists. envs/prod/main.tf explicitly notes "bilko-db (prod) — TBD — audit required" (lines 4-6 and import.sh).

**Impact:** Any `terraform apply` on envs/prod would attempt to create a REGIONAL HA POSTGRES\_15 `db-custom-2-7680` instance (~$100+/month). Without CEO sign-off, prod infra is BLOCKED.

**CEO decision needed:** Approve prod DB provisioning (cost + data migration strategy if migrating from elsewhere) before ANY envs/prod TF apply is ever run. If bilko-db exists elsewhere (on-prem? Railway?), import.sh must be run first.

## Validation

**Evidence file:** `/tmp/99477-proveo-evidence.md`

**Build ID:** 6f2236f6-86ec-444c-96b7-7c22f63cf5a2

**Build log:** [View in GCP Console](https://console.cloud.google.com/cloud-build/builds;region=europe-north1/6f2236f6-86ec-444c-96b7-7c22f63cf5a2?project=762788903040)

**Validation date:** 2026-05-06T20:28Z

**Validator:** Angie Jones (Proveo)

**Verdict:** PASS — 7/7 Acceptance Criteria met

- AC1: Build SUCCESS (all 8 steps SUCCESS)
- AC2: bilko-web-stage HTTP/2 200
- AC3: bilko-api-stage health endpoint 200 {"status":"ok"}
- AC4: New web revision deployed within 5min window
- AC5: Flyway migrate-db ran without error (21.5s)
- AC6: No gate-\* steps executed (0 quality gates)
- AC7: Image pushed with :stage-${SHORT\_SHA} tag (stage-277dd5a confirmed in Artifact Registry)

## Related MCs

- **\#99395** — VAT enum-cast genesis (billing\_country ENUM cast to TEXT in Flyway migration)
- **\#99422** — Sibling task (stage Cloud Run services health check)
- **\#99477** — This task (Stage CI/CD pipeline implementation)

## ZAKON PI2 Compliance Status

**Stage pipeline:** ✅ COMPLIANT

- DEPLOY-MAP.md exists at repo root ✅
- Pre-flight checks executed (4 probes: triggers, GCS bucket, Cloud Run services, SQL instances) ✅
- Post-deploy validation (curl 200 + Cloud Run revision evidence) ✅
- Evidence files delivered (/tmp/99477-preflight.txt, /tmp/99477-proveo-evidence.md) ✅

**Prod pipeline:** ⏸ BLOCKED (awaiting OCD-5 bilko-db provisioning approval)

## Last Updated

2026-05-06, owner: FlowForge (Kelsey Hightower)

---

## 2026-07-07 — Pipeline migriran na bilko-selfhosted (MC #104933)

Root cause: org bez hosted minuta; svi jobovi migrirani na bilko-selfhosted (merge a1bec755, PR #65; gitleaks fetchDepth:0 5febf0ad). Run 298: CI\_Gates/Build/Flyway/DeployStage GREEN, E2E 109/142 FAILED (prvi E2E signal, triage MC #104956). FORGE docker prune 37GB. Evidence: ~/system/evidence/104933/.