Infrastructure & DevOps

Deployment Guide

Bilko Deployment Guide

Last Updated: 2026-04-16
Current State: Stable Cloud Run deployment with custom domain provisioning

GCP Project Configuration

Secret Manager

Secret NameVersionPurpose
bilko-cors-originsv2Comma-separated list of allowed CORS origins
bilko-database-urllatestCloud SQL connection string (password reset 2026-04-16)
bilko-jwt-refresh-secretlatestJWT refresh token secret

CORS Parsing: Secret bilko-cors-origins is parsed by comma in apps/api/src/app.ts:61

Environment Variables

bilko-web

bilko-api

Custom Domain Setup

Current Domain

Domain Verification Constraint

Critical: Only alai.no is verified in GCP Search Console (via dev@alai.no).
basicconsulting.no is NOT verified. All custom domains MUST use *.alai.no subdomains until basicconsulting.no is verified.

Custom Domain Runbook

Prerequisites

  1. Domain must be verified in Google Search Console by the GCP account owner
  2. DNS provider access (one.com for alai.no, Vercel for basicconsulting.no)
  3. gcloud CLI authenticated: gcloud auth login

Step-by-Step

1. Create Domain Mapping

gcloud beta run domain-mappings create \
  --service=bilko-web \
  --domain=bilko-demo.alai.no \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0

2. Configure DNS

Add CNAME record at DNS provider:

Type: CNAME
Host: bilko-demo
Value: ghs.googlehosted.com.
TTL: 3600

3. Wait for Certificate Provisioning

gcloud beta run domain-mappings describe bilko-demo.alai.no \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0

Look for status.conditions → CertificateProvisioned: True

4. Update CORS Allowed Origins

# Get current value
gcloud secrets versions access latest --secret=bilko-cors-origins

# Add new domain
echo "https://bilko-demo.alai.no,https://bilko-web-dh4m46blja-lz.a.run.app" | \
  gcloud secrets versions add bilko-cors-origins --data-file=-

5. Deploy New Revision

gcloud run services update-traffic bilko-api \
  --to-latest \
  --region=europe-north1 \
  --project=tribal-sign-487920-k0

6. Verify CORS Preflight

curl -sSI -X OPTIONS \
  https://bilko-api-dh4m46blja-lz.a.run.app/api/v1/auth/login \
  -H "Origin: https://bilko-demo.alai.no" \
  -H "Access-Control-Request-Method: POST"

Expected: HTTP 204 with Access-Control-Allow-Origin: https://bilko-demo.alai.no

GitHub Actions CI/CD

Deployment Steps

  1. Authenticate via WIF
  2. Build Docker images (api + web)
  3. Push to Google Container Registry
  4. Deploy to Cloud Run (europe-north1)
  5. Run smoke tests (Playwright E2E)

Testing

Backend Tests

End-to-End Tests

Recent Fixes (2026-04-16)

Commits

Key Changes

  1. Service Naming: Production services now named bilko-api and bilko-web (no -staging suffix)
  2. API Enhancements: Invoice list now includes currency, new profile settings endpoint
  3. Frontend Fixes: Accessibility improvements (ARIA), avatar initials when no image, visual polish

Rollback Procedure

Rollback to Previous Revision

# List revisions
gcloud run revisions list --service=bilko-api --region=europe-north1

# Rollback
gcloud run services update-traffic bilko-api \
  --to-revisions=bilko-api-00036=100 \
  --region=europe-north1

Rollback via GitHub Actions

git revert HEAD
git push origin main  # Triggers deploy workflow

Troubleshooting

Issue: CORS errors in browser

Cause: Custom domain not in bilko-cors-origins secret
Fix: Update secret (see step 4 in Custom Domain Runbook), deploy new revision

Issue: 502 Bad Gateway

Cause: Service unhealthy or startup timeout
Fix: Check Cloud Run logs: gcloud run services logs read bilko-api --region=europe-north1 --limit=50

Issue: Database connection timeout

Cause: Cloud SQL Proxy misconfiguration or secret outdated
Fix: Verify bilko-database-url secret, check Cloud SQL instance status

Issue: Custom domain SSL pending

Cause: DNS not propagated or domain not verified in Search Console
Fix: Wait 15-30 min after DNS change, verify domain ownership in Search Console

Architecture Diagram

┌─────────────────┐
│  one.com DNS    │
│  bilko-demo.    │
│  alai.no        │
└────────┬────────┘
         │ CNAME
         ▼
┌─────────────────────────────┐
│  ghs.googlehosted.com       │
│  (Google Cloud Load Balancer)│
└────────┬────────────────────┘
         │
         ▼
┌─────────────────────────────┐       ┌──────────────────┐
│  bilko-web (Cloud Run)      │──────▶│  bilko-api       │
│  Next.js 15 + React 19      │  HTTP │  Express + TS    │
│  europe-north1              │       │  europe-north1   │
└─────────────────────────────┘       └────────┬─────────┘
                                               │
                                               ▼
                                      ┌─────────────────┐
                                      │  Cloud SQL      │
                                      │  PostgreSQL 16  │
                                      │  europe-north1  │
                                      └─────────────────┘

CI/CD Pipeline

Bilko — CI/CD Pipeline

Status: PLANNED (GitHub Actions workflows not yet configured)

This document describes the target continuous integration and deployment pipeline for Bilko.

Overview

Bilko uses GitHub Actions for CI/CD automation:

Why GitHub Actions?

Pipeline Overview

flowchart TD
    PUSH(["git push / PR opened"])
    TRIGGER{{"Branch?"}}

    subgraph PARALLEL["Stage 1 — Parallel Quality Checks"]
        LINT["Lint\nESLint + Prettier\n<2 min"]
        TC["Type Check\nTypeScript strict\n<2 min"]
        UT["Unit Tests\nVitest + coverage\n<3 min"]
        IT["Integration Tests\nSupertest + real PG\n<5 min"]
    end

    BUILD["Build (Turborepo)\napps/web → .next\napps/api → dist\n<4 min"]

    subgraph E2E_BLOCK["Stage 3 — E2E Tests"]
        VP["Wait for Vercel\nPreview URL"]
        E2E["Playwright E2E\nChromium + Firefox + WebKit\n<8 min"]
    end

    subgraph DEPLOY["Stage 4 — Deploy (main only)"]
        DF["Deploy Frontend\nVercel Production\nbilko.io"]
        DB["Deploy Backend\nRailway Production\napi.bilko.io"]
        MIGRATE["DB Migrations\nnpx prisma migrate deploy"]
    end

    NOTIFY["Slack Notification\n#bilko-deploys"]

    PUSH --> TRIGGER
    TRIGGER -->|"PR"| PARALLEL
    TRIGGER -->|"main"| PARALLEL
    PARALLEL --> BUILD
    BUILD --> E2E_BLOCK
    VP --> E2E
    E2E_BLOCK --> DEPLOY
    DEPLOY --> NOTIFY

    LINT & TC & UT & IT -->|"All pass"| BUILD

Pipeline Stages

1. Code Quality (Parallel)

ESLint + Prettier

- name: Lint
  run: npm run lint

Checks:

Fail Conditions:

TypeScript Type Check

- name: Type Check
  run: npm run type-check

Checks:

Fail Conditions:

2. Unit Tests (Vitest)

- name: Unit Tests
  run: npm run test:unit

Coverage Requirements:

Test Types:

Fail Conditions:

3. Integration Tests (Supertest)

- name: Integration Tests
  run: npm run test:integration
  env:
    DATABASE_URL: ${{ secrets.TEST_DATABASE_URL }}

Setup:

Test Types:

Fail Conditions:

Job Dependency Graph

graph LR
    LINT["lint"]
    TC["type-check"]
    UT["unit-tests"]
    IT["integration-tests"]
    BUILD["build\nneeds: lint, type-check,\nunit-tests, integration-tests"]
    E2E["e2e-tests\nneeds: build"]
    DF["deploy-frontend\nneeds: build, e2e-tests\nif: main branch"]
    DB_JOB["deploy-backend\nneeds: build, e2e-tests\nif: main branch"]

    LINT --> BUILD
    TC --> BUILD
    UT --> BUILD
    IT --> BUILD
    BUILD --> E2E
    E2E --> DF
    E2E --> DB_JOB

4. Build (Turborepo)

- name: Build
  run: npm run build

Build Targets:

Fail Conditions:

Artifacts:

5. E2E Tests (Playwright)

- name: E2E Tests
  run: npm run test:e2e
  env:
    PLAYWRIGHT_BASE_URL: ${{ env.PREVIEW_URL }}

Setup:

Test Scenarios:

Browsers:

Fail Conditions:

6. Deploy

Frontend (Vercel)

- name: Deploy Frontend
  uses: vercel/action@v1
  with:
    vercel-token: ${{ secrets.VERCEL_TOKEN }}
    vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID }}
    vercel-org-id: ${{ secrets.VERCEL_ORG_ID }}
    production: ${{ github.ref == 'refs/heads/main' }}

Deployment Strategy:

Rollback:

Backend (Railway)

- name: Deploy Backend
  uses: railway-app/action@v1
  with:
    railway-token: ${{ secrets.RAILWAY_TOKEN }}
    service: api
    environment: ${{ github.ref == 'refs/heads/main' && 'production' || 'staging' }}

Pre-Deploy:

  1. Run database migrations: npx prisma migrate deploy
  2. Health check on current deployment

Deployment Strategy:

Rollback:

Workflow Files

Main Workflow (.github/workflows/main.yml)

name: CI/CD Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 18
          cache: npm
      - run: npm ci
      - run: npm run lint

  type-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 18
          cache: npm
      - run: npm ci
      - run: npm run type-check

  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 18
          cache: npm
      - run: npm ci
      - run: npm run test:unit -- --coverage
      - uses: codecov/codecov-action@v3
        with:
          files: ./coverage/coverage-final.json

  integration-tests:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_USER: bilko_test
          POSTGRES_PASSWORD: bilko_test
          POSTGRES_DB: bilko_test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 18
          cache: npm
      - run: npm ci
      - run: npx prisma migrate deploy
        env:
          DATABASE_URL: postgresql://bilko_test:bilko_test@localhost:5432/bilko_test
      - run: npm run test:integration
        env:
          DATABASE_URL: postgresql://bilko_test:bilko_test@localhost:5432/bilko_test

  build:
    needs: [lint, type-check, unit-tests, integration-tests]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 18
          cache: npm
      - run: npm ci
      - run: npm run build
      - uses: actions/upload-artifact@v3
        with:
          name: build-artifacts
          path: |
            apps/web/.next
            apps/api/dist

  e2e-tests:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 18
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps
      - name: Wait for Vercel Preview
        uses: patrickedqvist/wait-for-vercel-preview@v1.3.1
        id: vercel-preview
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          max_timeout: 300
      - run: npm run test:e2e
        env:
          PLAYWRIGHT_BASE_URL: ${{ steps.vercel-preview.outputs.url }}
      - uses: actions/upload-artifact@v3
        if: failure()
        with:
          name: playwright-screenshots
          path: test-results/

  deploy-frontend:
    needs: [build, e2e-tests]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: vercel/action@v1
        with:
          vercel-token: ${{ secrets.VERCEL_TOKEN }}
          vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID }}
          vercel-org-id: ${{ secrets.VERCEL_ORG_ID }}
          production: true

  deploy-backend:
    needs: [build, e2e-tests]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: railway-app/action@v1
        with:
          railway-token: ${{ secrets.RAILWAY_TOKEN }}
          service: api
          environment: production

Hotfix Workflow (.github/workflows/hotfix.yml)

Fast-track workflow for urgent production fixes (bypasses full pipeline):

name: Hotfix Deploy

on:
  workflow_dispatch:
    inputs:
      reason:
        description: 'Reason for hotfix'
        required: true

jobs:
  hotfix:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 18
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm run type-check
      - run: npm run build
      - uses: vercel/action@v1
        with:
          vercel-token: ${{ secrets.VERCEL_TOKEN }}
          vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID }}
          vercel-org-id: ${{ secrets.VERCEL_ORG_ID }}
          production: true
      - uses: railway-app/action@v1
        with:
          railway-token: ${{ secrets.RAILWAY_TOKEN }}
          service: api
          environment: production
      - name: Notify Team
        uses: slackapi/slack-github-action@v1.24.0
        with:
          webhook-url: ${{ secrets.SLACK_WEBHOOK }}
          payload: |
            {
              "text": "🚨 Hotfix deployed: ${{ github.event.inputs.reason }}"
            }

Secrets Configuration

GitHub repository secrets (Settings → Secrets and variables → Actions):

Secret Name Description How to Generate
VERCEL_TOKEN Vercel deployment token Vercel Dashboard → Settings → Tokens
VERCEL_PROJECT_ID Vercel project ID vercel link output
VERCEL_ORG_ID Vercel organization ID vercel link output
RAILWAY_TOKEN Railway deployment token Railway Dashboard → Settings → Tokens
TEST_DATABASE_URL PostgreSQL test DB URL Use GitHub Actions service
SLACK_WEBHOOK Slack notification webhook Slack → Apps → Incoming Webhooks

Branch Protection Rules

Configure on GitHub (Settings → Branches → Branch protection rules for main):

Performance Targets

Pipeline Duration

Optimization Strategies

Failure Handling

Failure & Rollback Flow

flowchart TD
    FAIL(["Pipeline Failure"])
    WHERE{{"Failed\nStage?"}}

    BLOCK_PR["PR Blocked\nLogs in GitHub Actions UI\nArtifacts uploaded"]
    FIX_CODE["Fix code\nPush new commit"]

    HEALTH{{"Health check\npasses?"}}
    AUTO_ROLL["Automatic Rollback\nPrevious deployment promoted"]
    SLACK_ALERT["Slack Alert\n#bilko-deploys"]
    MANUAL["Manual Investigation\nRailway / Vercel logs"]

    FLAKY{{"Flaky\nE2E test?"}}
    RETRY["Playwright retry\n(retries: 1)"]
    CRITICAL["Mark critical\nCreate GitHub issue"]

    FAIL --> WHERE
    WHERE -->|"Quality / Tests"| BLOCK_PR --> FIX_CODE
    WHERE -->|"Deploy"| HEALTH
    WHERE -->|"E2E"| FLAKY

    HEALTH -->|No| AUTO_ROLL --> SLACK_ALERT
    HEALTH -->|Yes| MANUAL

    FLAKY -->|Yes| RETRY
    RETRY -->|Still fails| CRITICAL
    FLAKY -->|No| BLOCK_PR

Test Failures

  1. Pipeline stops immediately (fail-fast)
  2. Logs available in GitHub Actions UI
  3. Artifacts uploaded (screenshots, coverage reports)
  4. PR blocked until fixed

Deployment Failures

  1. Automatic rollback to previous version
  2. Slack notification to team
  3. Health check endpoint monitored
  4. Manual intervention if health check fails

Flaky Tests

Monitoring & Notifications

Slack Notifications

Notify on:

Email Notifications

GitHub Actions built-in:

Local Testing

Developers can run the full pipeline locally before pushing:

# Lint
npm run lint

# Type check
npm run type-check

# Unit tests with coverage
npm run test:unit -- --coverage

# Integration tests (requires local PostgreSQL)
npm run test:integration

# Build
npm run build

# E2E tests (requires build)
npm run test:e2e

Pre-commit Hook (Recommended): Install Husky to run lint + type-check before every commit:

npx husky install
npx husky add .husky/pre-commit "npm run lint && npm run type-check"

Future Enhancements

Security Scanning

Performance Testing

Database Migration Testing

Last Updated: 2026-02-20 Status: PLANNED — No GitHub Actions workflows configured yet Next Steps: Create .github/workflows/main.yml, configure secrets, test on staging branch

Environment Configuration

Bilko — Development Environment Setup

This guide walks through setting up a local development environment for Bilko.

Environment Configuration Overview

graph TD
    subgraph DEV["Development Environment"]
        D_ENV["apps/api/.env\napps/web/.env.local"]
        D_PG["PostgreSQL 15\nlocalhost:5432\nbilko_dev"]
        D_WEB["Next.js\nlocalhost:3000"]
        D_API["Express\nlocalhost:4000"]
        D_PRISMA["Prisma Studio\nlocalhost:5555"]
    end

    subgraph STAGING["Staging / Preview"]
        S_SECRETS["Railway Dashboard Env Vars\n(staging environment)"]
        S_VERCEL["Vercel Preview\nbilko-pr-{n}.vercel.app"]
        S_RAIL["Railway Staging\nbilko-api-staging"]
        S_PG["Railway PostgreSQL\nbilko_staging"]
    end

    subgraph PROD["Production"]
        P_SECRETS["Railway + Vercel\nDashboard Secrets"]
        P_WEB["Vercel Production\nbilko.io"]
        P_API["Railway Production\napi.bilko.io"]
        P_PG["Railway PostgreSQL\nbilko_prod"]
        P_R2["Cloudflare R2\nbilko-receipts"]
    end

    D_ENV --> D_API --> D_PG
    D_ENV --> D_WEB
    D_API --> D_PRISMA

    S_SECRETS --> S_RAIL --> S_PG
    S_SECRETS --> S_VERCEL

    P_SECRETS --> P_API --> P_PG
    P_SECRETS --> P_WEB
    P_API --> P_R2

Prerequisites

Required Software

Software Version Check Command Install
Node.js 18+ node --version https://nodejs.org
npm 9+ npm --version Included with Node.js
PostgreSQL 15+ psql --version https://postgresql.org/download
Git Latest git --version https://git-scm.com

Optional Tools

Tool Purpose Install
Prisma Studio Database GUI npx prisma studio
Postman API testing https://postman.com
VS Code Recommended IDE https://code.visualstudio.com

Installation Steps

1. Clone Repository

git clone https://github.com/your-org/bilko.git
cd bilko

2. Install Dependencies

# Install all workspace dependencies
npm install

This installs dependencies for:

Local Setup Flow

flowchart TD
    CLONE["git clone bilko"]
    INSTALL["npm install\n(Turborepo workspace)"]
    PG{{"PostgreSQL\navailable?"}}
    LOCAL_PG["Create local DB\npsql -U postgres\nCREATE DATABASE bilko_dev"]
    DOCKER_PG["Docker PostgreSQL\npostgres:15\nport 5432"]
    ENV["Configure .env files\napps/api/.env\napps/web/.env.local"]
    MIGRATE["npx prisma migrate dev\n(applies 15 table schema)"]
    GENERATE["npx prisma generate\n(Prisma Client)"]
    SEED["npx prisma db seed\n(demo org + user) — optional"]
    DEV["npm run dev\nlocalhost:3000 + 4000"]

    CLONE --> INSTALL --> PG
    PG -->|"Local install"| LOCAL_PG --> ENV
    PG -->|"Docker"| DOCKER_PG --> ENV
    ENV --> MIGRATE --> GENERATE --> SEED --> DEV

3. Set Up PostgreSQL Database

Option A: Local PostgreSQL Installation

Create database and user:

psql -U postgres

CREATE DATABASE bilko_dev;
CREATE USER bilko WITH PASSWORD 'bilko';
GRANT ALL PRIVILEGES ON DATABASE bilko_dev TO bilko;
\q

Option B: Docker PostgreSQL

docker run --name bilko-postgres \
  -e POSTGRES_USER=bilko \
  -e POSTGRES_PASSWORD=bilko \
  -e POSTGRES_DB=bilko_dev \
  -p 5432:5432 \
  -d postgres:15

4. Configure Environment Variables

apps/api/.env

Create .env file in apps/api/ directory:

# Database
DATABASE_URL=postgresql://bilko:bilko@localhost:5432/bilko_dev

# JWT Secrets (use `openssl rand -base64 32` to generate)
JWT_SECRET=your-secret-here-change-in-production
JWT_REFRESH_SECRET=your-refresh-secret-here-change-in-production

# Email (optional for local dev, required for staging/production)
SENDGRID_API_KEY=

# File Storage (optional for local dev)
R2_ACCESS_KEY_ID=
R2_SECRET_ACCESS_KEY=
R2_BUCKET_NAME=bilko-receipts-dev
R2_ENDPOINT=

# App Config
PORT=4000
NODE_ENV=development
ALLOWED_ORIGINS=http://localhost:3000

apps/web/.env.local

Create .env.local file in apps/web/ directory:

# API URL (backend)
NEXT_PUBLIC_API_URL=http://localhost:4000

# App Environment
NEXT_PUBLIC_APP_ENV=development

5. Run Database Migrations

cd packages/database
npx prisma migrate dev
npx prisma generate

This will:

  1. Apply all migrations to bilko_dev database
  2. Create 15 tables from schema.prisma
  3. Generate Prisma Client

6. Seed Database (Optional)

Create seed script: packages/database/prisma/seed.ts

import { PrismaClient } from '@prisma/client';

const prisma = new PrismaClient();

async function main() {
  // Create demo organization
  const org = await prisma.organization.create({
    data: {
      name: 'Demo Company d.o.o.',
      registrationNumber: '12345678',
      vatNumber: 'RS123456789',
      baseCurrency: 'RSD',
      country: 'RS',
      language: 'sr',
    },
  });

  // Create demo user
  await prisma.user.create({
    data: {
      organizationId: org.id,
      email: 'demo@bilko.io',
      passwordHash: '$2b$12$...', // bcrypt hash of "demo123"
      fullName: 'Demo User',
      role: 'owner',
    },
  });

  console.log('✅ Seed data created');
}

main()
  .catch((e) => {
    console.error(e);
    process.exit(1);
  })
  .finally(async () => {
    await prisma.$disconnect();
  });

Run seed:

npx prisma db seed

Running the Application

Start All Services (Recommended)

From root directory:

npm run dev

Turborepo starts:

Start Individual Services

Frontend Only

cd apps/web
npm run dev

Backend Only

cd apps/api
npm run dev

Development Tools

Prisma Studio (Database GUI)

cd packages/database
npx prisma studio

Opens at http://localhost:5555

Features:

Hot Reload

Both frontend and backend support hot reload:

Tech Stack Overview

Frontend (apps/web/)

Technology Version Purpose
Next.js 15.0.0 React framework with SSR
React 19.0.0 UI library
TypeScript 5.3.0 Type safety
Tailwind CSS 4.0.0 Styling
shadcn/ui Latest Component library (Radix UI + Tailwind)
Zustand 4.5.0 State management
Recharts 2.15.0 Charts (revenue, expenses)
Lucide React Latest Icons

Backend (apps/api/)

Technology Version Purpose
Express TBD Web framework
TypeScript 5.3.0 Type safety
Prisma Latest ORM + migrations
PostgreSQL 15+ Database
Passport.js TBD Authentication
Zod TBD Validation
Helmet TBD Security headers
bcrypt TBD Password hashing
jsonwebtoken TBD JWT tokens

Database (packages/database/)

Feature Implementation
ORM Prisma
Database PostgreSQL 15
Models 15 (see schema.prisma)
Migrations Prisma Migrate
Seeding prisma/seed.ts

Common Tasks

Create Database Migration

After modifying schema.prisma:

cd packages/database
npx prisma migrate dev --name describe_your_changes

Reset Database (DEV ONLY)

WARNING: Deletes all data.

cd packages/database
npx prisma migrate reset

Generate Prisma Client

After pulling new migrations:

cd packages/database
npx prisma generate

Run Linter

npm run lint

Runs ESLint + Prettier on all workspaces.

Run Type Check

npm run type-check

Runs TypeScript compiler in --noEmit mode (checks types without building).

Build for Production

npm run build

Builds:

Troubleshooting

Database Connection Errors

Error: Can't reach database server at localhost:5432

Solutions:

  1. Check PostgreSQL is running: pg_isready
  2. Verify credentials in .env
  3. Check port 5432 is not blocked

Port Already in Use

Error: Port 3000 is already in use

Solutions:

  1. Kill process using port: lsof -ti:3000 | xargs kill
  2. Change port: PORT=3001 npm run dev

Prisma Client Not Generated

Error: @prisma/client not found

Solution:

cd packages/database
npx prisma generate

TypeScript Errors After Pulling Changes

Solution:

npm install
npx prisma generate
npm run type-check

Hot Reload Not Working

Solution:

  1. Restart dev server
  2. Clear Next.js cache: rm -rf apps/web/.next
  3. Check file watcher limits (Linux): sysctl fs.inotify.max_user_watches

VS Code Configuration

Create .vscode/extensions.json:

{
  "recommendations": [
    "dbaeumer.vscode-eslint",
    "esbenp.prettier-vscode",
    "bradlc.vscode-tailwindcss",
    "prisma.prisma",
    "ms-vscode.vscode-typescript-next"
  ]
}

Settings

Create .vscode/settings.json:

{
  "editor.formatOnSave": true,
  "editor.defaultFormatter": "esbenp.prettier-vscode",
  "editor.codeActionsOnSave": {
    "source.fixAll.eslint": true
  },
  "typescript.tsdk": "node_modules/typescript/lib",
  "tailwindCSS.experimental.classRegex": [
    ["cva\\(([^)]*)\\)", "[\"'`]([^\"'`]*).*?[\"'`]"]
  ]
}

Secrets Management

flowchart LR
    DEV_SECRET["Developer\n.env file\n(gitignored)"]
    GH_SECRET["GitHub Secrets\nActions → Settings\nVERCEL_TOKEN\nRAILWAY_TOKEN\nTEST_DATABASE_URL\nSLACK_WEBHOOK"]
    VERCEL_ENV["Vercel Dashboard\nEnvironment Variables\nNEXT_PUBLIC_API_URL\nNEXT_PUBLIC_APP_ENV"]
    RAILWAY_ENV["Railway Dashboard\nEnvironment Variables\nDATABASE_URL (auto)\nJWT_SECRET\nJWT_REFRESH_SECRET\nSENDGRID_API_KEY\nR2_ACCESS_KEY_ID\nR2_SECRET_ACCESS_KEY"]

    subgraph NEVERCOMMIT["NEVER commit to git"]
        SECRET_FILE[".env files\nAPI keys\nJWT secrets\nDB passwords"]
    end

    DEV_SECRET -->|"local only"| NEVERCOMMIT
    GH_SECRET -->|"CI/CD pipeline"| VERCEL_ENV
    GH_SECRET -->|"CI/CD pipeline"| RAILWAY_ENV

Environment Variables Reference

apps/api/.env

Variable Required Default Description
DATABASE_URL Yes — PostgreSQL connection string
JWT_SECRET Yes — Access token secret (32+ chars)
JWT_REFRESH_SECRET Yes — Refresh token secret (32+ chars)
SENDGRID_API_KEY No — SendGrid API key (emails)
R2_ACCESS_KEY_ID No — Cloudflare R2 access key
R2_SECRET_ACCESS_KEY No — Cloudflare R2 secret key
R2_BUCKET_NAME No — R2 bucket name
R2_ENDPOINT No — R2 endpoint URL
PORT No 4000 API server port
NODE_ENV No development Environment (development/production)
ALLOWED_ORIGINS No * CORS allowed origins (comma-separated)

apps/web/.env.local

Variable Required Default Description
NEXT_PUBLIC_API_URL Yes — Backend API URL
NEXT_PUBLIC_APP_ENV No development Environment name

Testing Locally

Unit Tests

npm run test:unit

Integration Tests

Requires test database:

# Create test database
createdb bilko_test

# Run tests
npm run test:integration

E2E Tests

Requires both frontend and backend running:

# Terminal 1: Start dev servers
npm run dev

# Terminal 2: Run E2E tests
npm run test:e2e

Next Steps

After setting up your environment:

  1. Read the docs:

  2. Create your first feature:

    • Pick a task from the backlog
    • Create feature branch: git checkout -b feature/your-feature
    • Make changes, test locally
    • Submit PR

  3. Join the team:

    • Slack: #bilko-dev
    • Weekly sync: Fridays 10:00 CET
    • Documentation: Bilko Wiki

Last Updated: 2026-02-20 Status: CURRENT — Reflects actual setup as of this date Maintainer: John (AI Director)

Bilko Stage Environment — Cloud SQL & IAM (Phase 1)

Summary

MC #10177 Phase 1 (FlowForge, 2026-04-29): bilko-staging-db Cloud SQL instance brought under Flyway management. Pre-existing instance (2026-04-15, Prisma-managed). V1+V2+V4+V5 baselined, V3 actually executed. IAM SA created. Phase 2 (Cloud Run) pending.

Instance Details

FieldValue
Instance namebilko-staging-db
Connection nametribal-sign-487920-k0:europe-north1:bilko-staging-db
IP35.228.33.112
Tierdb-g1-small
VersionPOSTGRES_16
StateRUNNABLE (pre-existing since 2026-04-15; reused)
Databasebilko
App userbilko
Migration adminmigration_admin
Secretbilko-staging-db-password (Secret Manager, 2026-04-15)
IAM SAbilko-api-stage-sa@tribal-sign-487920-k0.iam.gserviceaccount.com
IAM SA rolesroles/cloudsql.client + roles/secretmanager.secretAccessor
Total tables24 (public schema)

Flyway State (2026-04-29)

VersionScriptStatus
V1V1__initial_schema.sqlBaselined (DDL existed via Prisma)
V2V2__add_missing_prisma_columns.sqlBaselined (DDL existed via Prisma)
V3V3__add_jmbg_oib_encryption.sqlEXECUTED LIVE — jmbg/jmbg_hash/oib/oib_hash + 2 indexes added to contacts (ADR-014)
V4V4__add_supplementary_tables.sqlBaselined (DDL existed via Prisma)
V5V5__add_logo_url_to_organizations.sqlBaselined (DDL existed via Prisma)

Open Risks

Phase Status

References

Bilko Stage Environment — Cloud Run Services (Phase 2)

Overview

MC: #10177 Phase 2  |  Deployed: 2026-04-30  |  Git SHA: 1f48fdc  |  Status: LIVE, healthy

GCP Project: tribal-sign-487920-k0  |  Region: europe-north1

WARNING — TD-3 PROD CUTOVER BLOCKER (MC #10241): bilko-staging-db uses public IP (0.0.0.0/0 authorized network, requireSsl=false). Acceptable for stage only. MUST NOT be replicated to production. Production deploy is blocked until Cloud SQL private IP + VPC connector is configured.

Live Services

ServiceURLImageMin/MaxMemoryStatus
bilko-api-stagebilko-api-stagebilko/api:stage-1f48fdc0/2512Mi, CPU 1LIVE
bilko-web-stagebilko-web-stagebilko/web:stage-1f48fdc0/2512Mi, CPU 1LIVE

Full Artifact Registry prefix: europe-north1-docker.pkg.dev/tribal-sign-487920-k0/

bilko-api-stage Detail

FieldValue
DockerfileDockerfile.api-kotlin (Kotlin/Ktor, port 4001)
JAVA_OPTSHikariCP connection pool tuned
Cloud SQLtribal-sign-487920-k0:europe-north1:bilko-staging-db via direct TCP 35.228.33.112:5432 (TD-2 + TD-3)
Secretsbilko-staging-db-password, bilko-jwt-secret, bilko-jwt-refresh-secret, bilko-staging-field-encryption-key (NEW, ADR-014), bilko-staging-field-hmac-key (NEW, ADR-014)
SAbilko-api-stage-sa@tribal-sign-487920-k0.iam.gserviceaccount.com
SA rolescloudsql.client, secretmanager.secretAccessor
SmokeGET /api/v1/health → 200 {"status":"ok","service":"bilko-api","version":"1.0.0"}
Revisionbilko-api-stage-00001-5x8 (100% traffic)

bilko-web-stage Detail

FieldValue
Dockerfileapps/web/Dockerfile (Next.js 15)
NEXT_PUBLIC_API_URLhttps://bilko-api-stage-dh4m46blja-lz.a.run.app/api/v1
NEXT_PUBLIC_APP_ENVstage
SmokeGET / → 200 (HTML, lang=sr-Latn)
Revisionbilko-web-stage-00001-c45 (100% traffic)
Build noteFresh npm install (no lockfile) — workaround TD-1 MC #10239

Smoke Test Commands

# API health (expected: {"status":"ok","service":"bilko-api","version":"1.0.0"})
curl -s https://bilko-api-stage-dh4m46blja-lz.a.run.app/api/v1/health

# Web root (expected: HTTP 200)
curl -s -o /dev/null -w "HTTP %{http_code}" https://bilko-web-stage-dh4m46blja-lz.a.run.app

Stage Rollback

# List revisions
gcloud run revisions list --service bilko-api-stage --project=tribal-sign-487920-k0 --region=europe-north1

# Route to prior revision
gcloud run services update-traffic bilko-api-stage --project=tribal-sign-487920-k0 --region=europe-north1 --to-revisions=REVISION_NAME=100

Stage Redeploy (image update only)

gcloud run services update bilko-api-stage --project=tribal-sign-487920-k0 --region=europe-north1 --image=europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/api:NEW_TAG
gcloud run services update bilko-web-stage --project=tribal-sign-487920-k0 --region=europe-north1 --image=europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web:NEW_TAG

Phase 2 Tech Debt Tracker

IDMCDescriptionSeverityBlocks
TD-1#10239package-lock.json macOS arm64 missing linux-x64 native bins — fresh npm install workaroundMediumClean stage re-deploys
TD-2#10240postgres-socket-factory not in build.gradle.kts — Kotlin API uses direct TCP public IPMediumSecure DB connectivity
TD-3#10241bilko-staging-db: 0.0.0.0/0 + requireSsl=false — STAGE ONLY, NEVER replicate to prodBLOCKERPROD CUTOVER Phase 5

Key Learnings

  1. Lockfile drift macOS/linux: fresh npm install required per build until TD-1 fixed
  2. Kotlin Cloud SQL TCP via public IP works for stage, NOT prod (TD-2 + TD-3)
  3. --no-traffic flag invalid on new service creation — route 100% on first deploy
  4. Field encryption/HMAC keys are random per env (stage isolated from prod — ADR-014)
  5. HikariCP socketPath URL param silently ignored — always use explicit host:port for direct TCP

References

Bilko demo — receipt upload/download fix (GCS shared storage) — MC #103095 (2026-06-07)

1. Symptom

CEO reported that receipt upload (PNG, PDF, JPG) was not working — a central part of the app. Investigation showed upload itself succeeded (HTTP 201) but viewing/downloading the receipt returned intermittent HTTP 404 approximately 60% of the time. From the user seat, intermittent 404 on download reads as a broken upload. The UI button "Priloženi dokumenti" -> "Preuzmi" triggered the failing request.

2. Root Cause

The demo API had no shared object storage configured. It used BILKO_LOCAL_UPLOAD_DIR=/tmp/bilko-uploads — per-instance ephemeral local disk, routed through ReceiptService.kt persistLocalIfEnabled, storing files as local:// URLs.

Cloud Run (bilko-api-demo) runs up to 5 instances with concurrency=1 (set during the earlier MC #103057 hang mitigation). An upload landing on instance A wrote the file to that instance's /tmp; a subsequent download request routed to instance B, which had no copy of the file, returning 404. Files were also permanently lost on any instance restart or recycle.

A secondary symptom was occasional 15-second frontend timeouts on the expense detail page: the several parallel API calls the page makes on load were serialised by concurrency=1.

Contributing config drift: the active deploy step in infrastructure/gcp/cloudbuild.yaml (deploy-api-demo) used --set-env-vars which replaces the entire env set, making a separate cloudbuild-demo-api.yaml with BILKO_LOCAL_UPLOAD_DIR ineffective.

3. Fix

Applied by FlowForge (Kelsey Hightower). No application code changes were required — ReceiptService.kt is unchanged and uses a transparent filesystem abstraction.

ChangeDetail
GCS bucket provisionedgs://bilko-receipts-demo, region europe-north1, uniform bucket-level access, IAM: bilko-api-stage-sa = roles/storage.objectAdmin
Cloud Run exec environmentUpgraded to gen2 (required for gcsfuse volume mounts)
Volume mount added--add-volume=name=receipts,type=cloud-storage,bucket=bilko-receipts-demo
Mount path--add-volume-mount=volume=receipts,mount-path=/mnt/bilko-uploads
Env var updatedBILKO_LOCAL_UPLOAD_DIR: /tmp/bilko-uploads -> /mnt/bilko-uploads
Config persistedAll changes committed to infrastructure/gcp/cloudbuild.yaml (deploy-api-demo step)

Deployed: tag v0.2.30, commit 642bbc0cefdc63777d8c12d61aa61a8257716290, revision bilko-api-demo-00135-rmv, 100% traffic on new revision. Cloud Build ID: 793f929b-f41a-49e9-afa9-65b54d3972ff.

Note: Cloud Build reported FAILURE due to a pre-existing flaky test timeout in coverage artifact upload (expenses-ux-102887). This is unrelated to the GCS change. All 8 deploy gates (lint, typecheck, unit, coverage, trivy, gitleaks, semgrep, npm-audit) PASSED; build, push, trivy, migrate, deploy, promote, smoke-test, and verify-sha steps all succeeded.

4. Validation

Validated by Proveo (Angie Jones) — GLOBAL VERDICT: PASS.

TestResult
PDF upload + 10x download10/10 HTTP 200 (was intermittent 404)
PNG upload + 10x download10/10 HTTP 200
JPEG upload + 10x download10/10 HTTP 200
GCS persistence (15 total calls)15/15 HTTP 200 — confirmed shared across instances
UI: Priloženi dokumenti sectionVisible; download icon -> /content HTTP 200
Health checkhttps://bilko-demo-api.alai.no/api/v1/health -> 200 {"status":"ok"}

Company Mesh: mesh-thr-03f166bb-f001-4293-b9ec-db245e5790b3 — PASS.

Open item (non-blocker): 1 pre-fix orphan document (uploaded 07:58 before GCS deploy at 08:30) returns 404 as expected — the file lived in ephemeral /tmp on a recycled instance. Not a regression.

5. Known Follow-Up Tasks

MCDescription
#103102Graceful 404 handling for missing documents in UI; fix misleading BILKO-INV-001 error code returned on expense document content misses
#103103Flaky coverage test (expenses-ux-102887 dialog upload test) blocking clean Cloud Build artifact upload — unrelated to this fix
#103104Invoice-receipt download gap: POST /invoices/{id}/receipts returns 201 but no download endpoint exists and receiptUrl stays null in invoice record

6. Operational Notes — Demo Deploy Pipeline

Bilko Azure Observability + MS for Startups Credit Setup (2026-06-15)

Purpose

Cross $100/month in foundational Azure spend to automatically unlock the Microsoft for Startups $25K credit tier. The model is usage-triggered, not referral-gated: once cumulative Azure spend reaches the threshold, the Founders Hub dashboard upgrades the credit allocation automatically. This work establishes the baseline infrastructure telemetry and security services that generate billable spend from day one.

What Was Done (MC #103599, 2026-06-15)

Application Insights Wiring

Container App Revisions (state at time of work)

ServiceActive RevisionStatusTrafficHTTP Check
bilko-api-demo bilko-api-demo--0000003 Running / Healthy 100% HTTP 404 on root (Ktor baseline — app live, no root handler)
bilko-web-demo bilko-web-demo--0000002 Running / Healthy 100% HTTP 200 (Next.js)

Note (Proveo-corrected): bilko-web-demo--0000001 carries 0% traffic; a second deploy superseded it. Do not confuse with the active revision when diagnosing issues.

Microsoft Defender for Containers

Spend Mechanics

Verification

Independently verified by Proveo (Angie Jones). Verdict: PARTIAL — only a revision-name reporting discrepancy found (--0000001 vs --0000002 for bilko-web-demo), no functional defect.

Telemetry Status

Wired and operational. First metrics pending ingestion delay of approximately 10–15 minutes from fresh deploy (normal behaviour for App Insights cold start).

Open Items (flagged, out of scope for MC #103599)

How to Verify

# Confirm App Insights resource is healthy
az monitor app-insights component show --app appi-bilko -g rg-bilko-demo

# Check Defender pricing tier
az security pricing show --name Containers

# Check Container App active revision
az containerapp revision list -n bilko-api-demo -g rg-bilko-demo --query "[].{name:name,traffic:properties.trafficWeight,state:properties.runningState}"

# Monitor spend trajectory toward $100/month threshold
# Azure Portal: Cost Management > bilko subscription > cost analysis

Watch the Microsoft Founders Hub dashboard for automatic $25K credit tier upgrade once monthly spend crosses $100.

References

Bilko ACA Telemetry & Observability Wiring (Azure)

Context

GCP Cloud Monitoring dashboard (070613fa) was decommissioned 2026-06-23 after migration to Azure (MC #104228 closed). This page documents the ACA→Log Analytics + App Insights telemetry wiring done as follow-on MC #104266.

Resources

Root cause that was fixed

ACA env appLogsConfiguration.destination was not effectively set → ContainerApp logs not reaching the workspace. Fixed via:

az containerapp env update -n bilko-demo-env -g rg-bilko-demo --logs-destination log-analytics --logs-workspace-id 71443731-... --logs-workspace-key <key>

Result: ContainerAppSystemLogs_CL now flows (tool-verified 54 rows/15m, sustained).

App Insights instrumentation

Each ACA app needs env var APPLICATIONINSIGHTS_CONNECTION_STRING (from az monitor app-insights component show -g rg-bilko-demo -n appi-bilko --query connectionString -o tsv), set via az containerapp update -n <app> -g rg-bilko-demo --set-env-vars APPLICATIONINSIGHTS_CONNECTION_STRING=<value>. All 4 apps confirmed set.

KNOWN REMAINING GAP (important for runbook)

Setting the env var ALONE does not produce App Insights request/dependency telemetry — requests table = 0. The app code must initialize the App Insights SDK (Node: applicationinsights package; Spring Boot: azure-monitor starter). Until then, App-Insights-request-based workbook panels stay empty. The KQL log panel (ContainerAppSystemLogs_CL) and ACA platform-metric panels work regardless. Track app-code SDK init as a separate CodeCraft task if request tracing is needed.

Troubleshooting note

az monitor log-analytics query returns a FLAT array [{col:val}] — do NOT filter with --query "tables[0].rows" (returns empty falsely). az monitor app-insights query uses {tables:[{rows}]} shape.

Verification queries (runbook)

MC #104332 — Bilko URA LocalDate ISO deploy evidence

MC #104332 / URA3 LocalDate + UI polish deploy evidence (2026-06-25)

What changed

Validation evidence

Demo deployment

Name Image Latest Ready Running Traffic

bilko-api-demo bilkodemo.azurecr.io/bilko-api:demo-104332ura3 bilko-api-demo--ura3-api bilko-api-demo--ura3-api Running 100 bilko-web-demo bilkodemo.azurecr.io/bilko-web:demo-104332ura3 bilko-web-demo--ura3-web bilko-web-demo--ura3-web Running 100

Health probes:

Live UAT evidence

Azure DevOps merge evidence

Status

Local evidence directory: /Users/makinja/business/ALAI-Holding-AS/products/Bilko/docs/evidence/104332