Infrastructure & DevOps

Deployment, CI/CD, monitoring, DevOps/SRE stack, WAF

Deployment & Environment

Deployment & Environment

Environment Setup

Drop Environment Configuration

Last updated: 2026-02-13 Source: src/drop-app/package.json, next.config.ts, Dockerfile, docker-compose.yml, fly.toml

Technology Stack

Layer Technology Version Source
Runtime Node.js 22 (Alpine) Dockerfile:2
Framework Next.js 16.1.6 package.json:14
UI React 19.2.3 package.json:15-16
Database (all environments) PostgreSQL 16 via Drizzle ORM drizzle-orm src/shared/db/schema.ts
Auth JWT via jose ^6.1.3 package.json:8
Password hashing bcryptjs ^3.0.3 package.json:5
Styling Tailwind CSS ^4 package.json:33
UI Components Radix UI ^1.4.3 package.json:13
Icons Lucide React ^0.563.0 package.json:9
Theme next-themes ^0.4.6 package.json:10
Toasts Sonner ^2.0.7 package.json:17

Dev Dependencies

Tool Version Purpose Source
Vitest ^4.0.18 Unit/integration testing package.json:36
Playwright ^1.58.2 E2E testing package.json:21
TypeScript ^5 Type checking package.json:35
ESLint ^9 Linting package.json:29
shadcn ^3.8.4 UI component generation package.json:32

NPM Scripts

Source: src/drop-app/package.json:5-12

Script Command Description
dev next dev Start development server (port 3000)
build next build Build for production (standalone output)
start next start Start production server
lint eslint Run ESLint
test vitest run Run unit/integration tests (single run)
test:watch vitest Run tests in watch mode

Next.js Configuration

Source: src/drop-app/next.config.ts:1-49

Setting Value Purpose
output "standalone" Self-contained server for Docker (next.config.ts:4)
devIndicators false Disable dev indicators (next.config.ts:5)

Security Headers

All responses include these headers (configured in next.config.ts:6-58):

Header Value (Production) Value (Development) Purpose
Content-Security-Policy default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; font-src 'self'; img-src 'self' data: blob:; connect-src 'self'; frame-ancestors 'none' default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; font-src 'self'; img-src 'self' data: blob:; connect-src 'self'; frame-ancestors 'none' XSS and injection protection
X-Frame-Options DENY DENY Clickjacking prevention
X-Content-Type-Options nosniff nosniff MIME sniffing prevention
Referrer-Policy strict-origin-when-cross-origin strict-origin-when-cross-origin Referrer leakage prevention
Permissions-Policy camera=(self), microphone=(), geolocation=(self) camera=(self), microphone=(), geolocation=(self) Feature restriction
Strict-Transport-Security max-age=63072000; includeSubDomains; preload max-age=63072000; includeSubDomains; preload Force HTTPS

Note: CSP is stricter in production (no unsafe-eval for scripts). Development mode allows unsafe-inline and unsafe-eval for HMR (Hot Module Replacement) to work.

Environment Modes

Development

Production

Test

Port Mapping

Service Internal Port External Port Protocol
Drop App 3000 3000 HTTP
PostgreSQL (local dev) 5432 5433 TCP
PostgreSQL (production RDS) 5432 5432 TCP

Docker Image Details

Base: node:22-alpine User: nextjs (UID 1001) Working dir: /app Exposed port: 3000 Entrypoint: node server.js Build context: src/drop-app/

Image contents (runner stage):

Deployment & Environment

Secrets Management

Secrets Management

Last updated: 2026-02-17 Source: src/drop-app/src/lib/secrets.ts

Overview

Drop uses an abstracted secrets management system with pluggable providers. The system is backward compatible -- if no secrets provider is configured, it reads directly from environment variables (existing behavior).

Provider Selection

The provider is selected automatically based on which environment variables are set:

Priority Condition Provider Description
1 DOPPLER_TOKEN set Doppler Cloud secrets manager via Doppler API
2 AWS_SECRET_ARN set AWS AWS Secrets Manager (requires AWS SDK)
3 (default) env Reads from process.env

Initialization (call once at app startup):

import { initSecrets } from '@/lib/secrets';

// Auto-detect provider based on env vars
initSecrets();

// Optional: custom cache TTL (default 5 minutes)
initSecrets({ ttlMs: 10 * 60 * 1000 }); // 10 minutes

Usage:

import { getSecret } from '@/lib/secrets';

const jwtSecret = await getSecret('JWT_SECRET');
const dbUrl = await getSecret('DATABASE_URL');

Caching

All secret values are cached in memory with a configurable TTL (default: 5 minutes). This reduces API calls to external providers while ensuring secrets are refreshed periodically.

Rotation Procedures

JWT_SECRET

Impact: All active user sessions will be invalidated.

  1. Generate new secret: openssl rand -base64 48
  2. Update in secrets provider (Doppler/AWS/env)
  3. Call rotateSecret('JWT_SECRET', newValue) or restart the app
  4. Users will need to log in again

DATABASE_URL (PostgreSQL credentials)

Impact: Application loses DB connectivity until updated.

  1. Create new PostgreSQL credentials
  2. Update PostgreSQL user: ALTER USER drop WITH PASSWORD 'new_value';
  3. Update DATABASE_URL in secrets provider with new credentials
  4. Restart the application (or call rotateSecret)

SENTRY_DSN

Status: REMOVED (MC #1271 — Sentry deinstalled)

SLACK_WEBHOOK_URL

Impact: Alerts stop sending to Slack until updated.

  1. Create new incoming webhook in Slack workspace
  2. Update SLACK_WEBHOOK_URL in secrets provider
  3. Restart the application

Open Banking API Keys

Impact: Bank connectivity (AISP/PISP) stops working.

  1. Regenerate keys in the Open Banking provider dashboard
  2. Update the relevant env vars in secrets provider
  3. Restart the application
  4. Verify bank account connectivity via /api/health

Environment Setup per Provider

Environment Variables (Default)

No setup required. Set secrets as environment variables:

# .env.local (development)
JWT_SECRET=dev-secret-do-not-use-in-production

# Production (Fly.io)
fly secrets set JWT_SECRET="$(openssl rand -base64 48)"
fly secrets set DATABASE_URL="postgresql://..."

# Production (Docker)
# Pass via -e flags or docker-compose environment section

Doppler

  1. Create account at doppler.com
  2. Create project "drop" with environments (dev, staging, production)
  3. Add all secrets in the Doppler dashboard
  4. Generate a service token for each environment
  5. Set DOPPLER_TOKEN in your deployment:
# Fly.io
fly secrets set DOPPLER_TOKEN="dp.st.production.xxxxx"

# Docker (pass as environment variable)

AWS Secrets Manager

  1. Create a secret in AWS Secrets Manager (JSON format): 
    {
  "JWT_SECRET": "your-jwt-secret",
  "DATABASE_URL": "postgresql://...",
  "SLACK_WEBHOOK_URL": "https://..."
}
  2. Note the secret ARN
  3. Ensure the application has IAM permissions for secretsmanager:GetSecretValue
  4. Install the AWS SDK: npm install @aws-sdk/client-secrets-manager
  5. Set AWS_SECRET_ARN in your deployment

Audit Trail

All secret rotation events are logged to the audit_log table:

Field Value
action secret_rotated
resource_type secret
resource_id Secret key name (e.g., JWT_SECRET)
details JSON with provider name and rotation timestamp

Query rotation history:

SELECT * FROM audit_log
WHERE action = 'secret_rotated'
ORDER BY timestamp DESC;
Deployment & Environment

Deployment Checklist

Deployment Checklist: [PROJECT NAME]

Release: v[X.Y.Z] Date: YYYY-MM-DD Deploy Lead: DevOps Approved by: Tech Lead + John Environment: Staging → Production

Pre-Deployment (T-1 Day)

Verification

Preparation

Configuration

Deployment (T-0)

Window

Execution

Post-Deployment (T+0)

Smoke Tests

Monitoring (First 30 Minutes)

Communication

Rollback Plan

Rollback Triggers

Rollback Procedure

  1. Announce rollback in channel
  2. Revert to previous version
  3. Restore database backup (if schema changed)
  4. Verify rollback successful
  5. Announce rollback complete
  6. Create incident report

Rollback Time Targets

Sign-off

Role Name Pre-Deploy Post-Deploy
DevOps ☐ ☐
Tech Lead ☐ ☐
John ☐ ☐
Deployment & Environment

DR Runbook

Drop — Disaster Recovery Runbook

Infrastructure Overview

Production Environment

Database

Staging Environment

Domain

Backup Strategy

RDS PostgreSQL (Production)

Staging PostgreSQL (RDS)

Recovery Procedures

Scenario 1: App Runner Service Down

Symptoms

Investigation Steps

# 1. Check service status
aws apprunner describe-service \
  --service-arn arn:aws:apprunner:eu-west-1:324480209768:service/drop-web/8e45b0d335304487a1880f4e32d6aeec \
  --region eu-west-1

# 2. View recent logs (last 10 minutes)
aws logs tail /aws/apprunner/drop-web/8e45b0d335304487a1880f4e32d6aeec/application \
  --follow \
  --since 10m \
  --region eu-west-1

# 3. Check deployment history
aws apprunner list-operations \
  --service-arn arn:aws:apprunner:eu-west-1:324480209768:service/drop-web/8e45b0d335304487a1880f4e32d6aeec \
  --region eu-west-1

Recovery Actions

Option A: Restart Service

# Trigger new deployment (no code change)
aws apprunner start-deployment \
  --service-arn arn:aws:apprunner:eu-west-1:324480209768:service/drop-web/8e45b0d335304487a1880f4e32d6aeec \
  --region eu-west-1

# Monitor deployment status
aws apprunner describe-service \
  --service-arn arn:aws:apprunner:eu-west-1:324480209768:service/drop-web/8e45b0d335304487a1880f4e32d6aeec \
  --query 'Service.Status' \
  --region eu-west-1

Option B: Rollback to Previous Image

# 1. List recent ECR images
aws ecr describe-images \
  --repository-name drop-web \
  --region eu-west-1 \
  --query 'sort_by(imageDetails,& imagePushedAt)[-5:]'

# 2. Update service to use previous image tag
# (Manual step: Update .github/workflows/deploy-aws.yml with previous tag and push)

# 3. Or update directly via App Runner console (rollback to previous deployment)

RTO: 5-10 minutes (restart) / 15-20 minutes (rollback)

Scenario 2: RDS Database Failure

Symptoms

Investigation Steps

# 1. Check RDS instance status
aws rds describe-db-instances \
  --db-instance-identifier drop-db \
  --region eu-west-1 \
  --query 'DBInstances[0].DBInstanceStatus'

# 2. Check for automated snapshots
aws rds describe-db-snapshots \
  --db-instance-identifier drop-db \
  --region eu-west-1 \
  --query 'DBSnapshots[?SnapshotType==`automated`] | sort_by(@, &SnapshotCreateTime)[-5:]'

# 3. Review recent events
aws rds describe-events \
  --source-identifier drop-db \
  --source-type db-instance \
  --region eu-west-1 \
  --duration 60

Recovery Actions

Option A: Restore from Latest Automated Snapshot

# 1. Identify latest snapshot
LATEST_SNAPSHOT=$(aws rds describe-db-snapshots \
  --db-instance-identifier drop-db \
  --region eu-west-1 \
  --query 'DBSnapshots[?SnapshotType==`automated`] | sort_by(@, &SnapshotCreateTime)[-1].DBSnapshotIdentifier' \
  --output text)

echo "Latest snapshot: $LATEST_SNAPSHOT"

# 2. Restore to new instance
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier drop-db-restored \
  --db-snapshot-identifier $LATEST_SNAPSHOT \
  --db-instance-class db.t4g.micro \
  --vpc-security-group-ids sg-XXXXX \
  --db-subnet-group-name default \
  --region eu-west-1

# 3. Wait for restore to complete (10-20 minutes)
aws rds wait db-instance-available \
  --db-instance-identifier drop-db-restored \
  --region eu-west-1

# 4. Update DATABASE_URL in App Runner
# (Manual step: Update environment variable via AWS Console or CLI)

# 5. Verify connection
NEW_ENDPOINT=$(aws rds describe-db-instances \
  --db-instance-identifier drop-db-restored \
  --query 'DBInstances[0].Endpoint.Address' \
  --output text \
  --region eu-west-1)

echo "New endpoint: $NEW_ENDPOINT"

Option B: Point-in-Time Recovery

# Restore to specific timestamp (e.g., 1 hour ago)
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier drop-db \
  --target-db-instance-identifier drop-db-pitr \
  --restore-time $(date -u -d '1 hour ago' '+%Y-%m-%dT%H:%M:%SZ') \
  --db-instance-class db.t4g.micro \
  --region eu-west-1

# Wait for restore
aws rds wait db-instance-available \
  --db-instance-identifier drop-db-pitr \
  --region eu-west-1

RPO: 24 hours (snapshot) / 5 minutes (PITR) RTO: 30 minutes (snapshot) / 30 minutes (PITR)

Scenario 3: Data Corruption

Symptoms

Investigation Steps

# 1. Connect to RDS and inspect data
psql -h drop-db.czu2qe4quy4v.eu-west-1.rds.amazonaws.com \
     -U dropuser \
     -d dropapp \
     -c "SELECT COUNT(*) FROM users WHERE deleted_at IS NOT NULL;"

# 2. Check audit_log table for suspicious activity
psql -h drop-db.czu2qe4quy4v.eu-west-1.rds.amazonaws.com \
     -U dropuser \
     -d dropapp \
     -c "SELECT * FROM audit_log WHERE action IN ('DELETE', 'UPDATE') ORDER BY timestamp DESC LIMIT 50;"

# 3. Identify time of corruption
# Review application logs and database query logs

Recovery Actions

Option A: Selective Data Restore (if corruption is isolated)

# 1. Create temporary snapshot of current state
aws rds create-db-snapshot \
  --db-instance-identifier drop-db \
  --db-snapshot-identifier drop-db-before-restore-$(date +%Y%m%d-%H%M) \
  --region eu-west-1

# 2. Restore clean snapshot to temporary instance
CLEAN_SNAPSHOT=<snapshot-before-corruption>

aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier drop-db-temp \
  --db-snapshot-identifier $CLEAN_SNAPSHOT \
  --db-instance-class db.t4g.micro \
  --region eu-west-1

# 3. Export affected tables from clean instance
pg_dump -h <temp-endpoint> \
        -U dropuser \
        -d dropapp \
        -t users \
        -t transactions \
        --data-only \
        > clean_data.sql

# 4. Selectively import into production (after verification)
psql -h drop-db.czu2qe4quy4v.eu-west-1.rds.amazonaws.com \
     -U dropuser \
     -d dropapp \
     < clean_data.sql

# 5. Terminate temporary instance
aws rds delete-db-instance \
  --db-instance-identifier drop-db-temp \
  --skip-final-snapshot \
  --region eu-west-1

Option B: Full Database Restore (see Scenario 2)

RTO: 1-2 hours (selective) / 30 minutes (full restore) RPO: Depends on snapshot age

Scenario 4: Full Region Outage (eu-west-1)

Current State

Investigation Steps

# 1. Check AWS Service Health Dashboard
# https://health.aws.amazon.com/health/status

# 2. Verify RDS snapshots are accessible
aws rds describe-db-snapshots \
  --db-instance-identifier drop-db \
  --region eu-west-1

# 3. Check ECR images (may need to copy to secondary region)
aws ecr describe-images \
  --repository-name drop-web \
  --region eu-west-1

Recovery Actions (Manual Failover to eu-north-1)

# 1. Copy latest RDS snapshot to eu-north-1
LATEST_SNAPSHOT=$(aws rds describe-db-snapshots \
  --db-instance-identifier drop-db \
  --region eu-west-1 \
  --query 'DBSnapshots[?SnapshotType==`automated`] | sort_by(@, &SnapshotCreateTime)[-1].DBSnapshotIdentifier' \
  --output text)

aws rds copy-db-snapshot \
  --source-db-snapshot-identifier arn:aws:rds:eu-west-1:324480209768:snapshot:$LATEST_SNAPSHOT \
  --target-db-snapshot-identifier drop-db-failover-$(date +%Y%m%d) \
  --region eu-north-1

# 2. Restore RDS in eu-north-1
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier drop-db-failover \
  --db-snapshot-identifier drop-db-failover-$(date +%Y%m%d) \
  --db-instance-class db.t4g.micro \
  --region eu-north-1

# 3. Copy ECR image to eu-north-1
# (Manual: create ECR repo in eu-north-1, retag and push latest image)

# 4. Deploy App Runner in eu-north-1
# (Manual: create new App Runner service via console with failover database endpoint)

# 5. Update DNS (when getdrop.no is active)
# Point getdrop.no to new App Runner URL

RTO: 2-4 hours (manual process) RPO: Last snapshot before outage (24 hours worst case, 5 minutes with PITR if available)

Scenario 5: Security Incident

Symptoms

Investigation Steps

# 1. Check audit logs for suspicious activity
psql -h drop-db.czu2qe4quy4v.eu-west-1.rds.amazonaws.com \
     -U dropuser \
     -d dropapp \
     -c "SELECT * FROM audit_log WHERE timestamp > NOW() - INTERVAL '24 hours' ORDER BY timestamp DESC;"

# 2. Review AML alerts
psql -h drop-db.czu2qe4quy4v.eu-west-1.rds.amazonaws.com \
     -U dropuser \
     -d dropapp \
     -c "SELECT * FROM aml_alerts WHERE status = 'open' OR created_at > NOW() - INTERVAL '24 hours';"

# 3. Check AWS CloudTrail for API activity
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=drop-db \
  --region eu-west-1 \
  --max-results 50

# 4. Review App Runner access logs
aws logs filter-log-events \
  --log-group-name /aws/apprunner/drop-web/8e45b0d335304487a1880f4e32d6aeec/application \
  --start-time $(date -u -d '24 hours ago' +%s)000 \
  --region eu-west-1

Containment Actions

# 1. Revoke compromised sessions
psql -h drop-db.czu2qe4quy4v.eu-west-1.rds.amazonaws.com \
     -U dropuser \
     -d dropapp \
     -c "UPDATE sessions SET revoked = 1 WHERE user_id IN (SELECT user_id FROM aml_alerts WHERE status = 'open');"

# 2. Temporarily disable affected users
psql -h drop-db.czu2qe4quy4v.eu-west-1.rds.amazonaws.com \
     -U dropuser \
     -d dropapp \
     -c "UPDATE users SET kyc_status = 'rejected' WHERE id IN (SELECT user_id FROM aml_alerts WHERE severity = 'critical');"

# 3. Rotate database credentials
aws rds modify-db-instance \
  --db-instance-identifier drop-db \
  --master-user-password <new-password> \
  --apply-immediately \
  --region eu-west-1

# Update DATABASE_URL in App Runner with new password

# 4. Enable enhanced monitoring
aws rds modify-db-instance \
  --db-instance-identifier drop-db \
  --monitoring-interval 1 \
  --monitoring-role-arn arn:aws:iam::324480209768:role/rds-monitoring-role \
  --region eu-west-1

# 5. Take forensic snapshot
aws rds create-db-snapshot \
  --db-instance-identifier drop-db \
  --db-snapshot-identifier drop-db-incident-$(date +%Y%m%d-%H%M) \
  --region eu-west-1

Investigation & Remediation

  1. Analyze audit logs — identify scope of breach
  2. File STR reports — if financial crime suspected (via str_reports table)
  3. Notify Finanstilsynet — if user data compromised (GDPR requirement)
  4. Update security policies — patch vulnerabilities
  5. User communication — notify affected users if required by GDPR

RTO: Immediate containment (revoke sessions) / 24-48 hours full investigation

RTO/RPO Targets

Scenario RTO RPO
App Runner restart 5-10 minutes 0 (no data loss)
App Runner rollback 15-20 minutes 0 (no data loss)
RDS snapshot restore 30 minutes 24 hours (last snapshot)
RDS PITR restore 30 minutes 5 minutes (PITR granularity)
Full region failover 2-4 hours 24 hours (manual process)
Security incident containment Immediate 0 (logs preserved)

Contacts

Primary

AI Operations

External Support

Runbook Maintenance

Review Schedule

Test Schedule

Change Log

Date Change Author
2026-02-18 Initial version created Builder 3 (AI)

Appendix: Useful Commands

Quick Health Check

# Check App Runner status
aws apprunner describe-service \
  --service-arn arn:aws:apprunner:eu-west-1:324480209768:service/drop-web/8e45b0d335304487a1880f4e32d6aeec \
  --query 'Service.Status' \
  --output text \
  --region eu-west-1

# Check RDS status
aws rds describe-db-instances \
  --db-instance-identifier drop-db \
  --query 'DBInstances[0].DBInstanceStatus' \
  --output text \
  --region eu-west-1

# Check latest snapshot age
aws rds describe-db-snapshots \
  --db-instance-identifier drop-db \
  --region eu-west-1 \
  --query 'DBSnapshots[?SnapshotType==`automated`] | sort_by(@, &SnapshotCreateTime)[-1].SnapshotCreateTime' \
  --output text

Database Connection Test

# Test connection from local machine
psql -h drop-db.czu2qe4quy4v.eu-west-1.rds.amazonaws.com \
     -U dropuser \
     -d dropapp \
     -c "SELECT 1;"

Log Streaming

# Stream App Runner application logs
aws logs tail /aws/apprunner/drop-web/8e45b0d335304487a1880f4e32d6aeec/application \
  --follow \
  --region eu-west-1

# Stream RDS error logs
aws rds download-db-log-file-portion \
  --db-instance-identifier drop-db \
  --log-file-name error/postgresql.log \
  --region eu-west-1
Deployment & Environment

Deployment Guide

Drop Deployment Guide

Last updated: 2026-03-03 Source: src/drop-app/Dockerfile, docker-compose.yml, DOCKER.md

NOTE (2026-03-03): This document was updated for ADR-014 (PostgreSQL-only). The SQLite single-container deployment and better-sqlite3 native dependency have been removed. Current deployment: Docker + PostgreSQL 16 (dev), AWS App Runner + RDS (production).

Architecture Overview

Drop uses a multi-stage Docker build producing a minimal Node.js 22 Alpine production image. The application is a Next.js 16 standalone server.

Build stages (from Dockerfile:1-41):

Stage Base Purpose
deps node:22-alpine Install node_modules via npm ci.
builder node:22-alpine Copy deps + source, run npm run build (Next.js standalone output).
runner node:22-alpine Minimal production image. Copies only public/, .next/standalone/, .next/static/.

Security features in the runner stage (Dockerfile:25-26):

Deployment Configurations

1. Local Development -- docker-compose.yml

PostgreSQL 16 + Drop app (ADR-014).

File: src/drop-app/docker-compose.yml:1-22

services:
  drop-app:
    build: .
    ports:
      - "3000:3000"
    environment:
      - JWT_SECRET=${JWT_SECRET:?JWT_SECRET is required}
      - NODE_ENV=production
      - NEXT_PUBLIC_SERVICE_MODE=mock
    volumes:
      - drop_data:/app/data
    healthcheck:
      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:3000/api/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
    restart: unless-stopped

Quick start:

export JWT_SECRET="your-secure-random-string-min-32-chars"
docker compose up -d

Data persistence: PostgreSQL data stored in Docker volume drop_pgdata.

2. Production (PostgreSQL) -- docker-compose.production.yml

Multi-container setup with separate PostgreSQL 16 database.

File: src/drop-app/docker-compose.production.yml:1-38

services:
  drop-app:
    build: .
    ports:
      - "3000:3000"
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    environment:
      - POSTGRES_DB=drop
      - POSTGRES_USER=drop
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-drop_local_dev}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U drop"]
      interval: 10s
      timeout: 5s
      retries: 5

Quick start:

export JWT_SECRET="your-secure-random-string-min-32-chars"
export POSTGRES_PASSWORD="secure-postgres-password"
docker compose -f docker-compose.production.yml up -d

3. Fly.io Staging -- fly.toml

File: src/drop-app/fly.toml:1-28

Setting Value
App name drop-staging
Region arn (Stockholm -- closest to Norway)
Internal port 3000
Force HTTPS true
Auto-stop machines stop (scales to zero)
Auto-start machines true
Min machines 0
Persistent storage Volume drop_data mounted at /app/data

Health check: GET /api/health every 30s, 5s timeout, 10s grace period.

Environment Variables

Variable Required Default Description
JWT_SECRET Yes (production) Dev: process.cwd() hash JWT signing secret. Minimum 32 characters. Fatal error if missing in production.
NODE_ENV No development Set to production in containers. Controls seed data gating.
NEXT_PUBLIC_SERVICE_MODE No - Set to mock for MVP mode (no external API calls).
DATABASE_URL Yes - PostgreSQL 16 connection string. Required in all environments. Local dev: postgresql://drop:dev_only_not_a_secret@localhost:5433/drop_dev
POSTGRES_PASSWORD Production only drop_local_dev PostgreSQL password (production compose).
PORT No 3000 HTTP server port.
HOSTNAME No 0.0.0.0 Server bind address.

Database: PostgreSQL 16 is required in all environments. There is no SQLite fallback (ADR-014).

Health Check

Endpoint: GET /api/health Source: src/drop-app/src/app/api/health/route.ts:1-35

The health check performs a real database query (SELECT 1 as ok) and reports latency.

Success response (200):

{
  "status": "ok",
  "version": "0.1.0",
  "uptime": 123,
  "db": "connected",
  "dbLatencyMs": 5,
  "timestamp": "2026-02-13T12:00:00.000Z"
}

Failure response (503):

{
  "status": "error",
  "db": "disconnected",
  "timestamp": "..."
}

Building from Source

# Build Docker image
docker build -t drop-app .

# Run standalone container
docker run -d \
  -p 3000:3000 \
  -e JWT_SECRET="your-secret-min-32-chars" \
  -v drop_data:/app/data \
  --name drop-app \
  drop-app

Data Backup and Restore

Production Backups (AWS RDS)

Production database is PostgreSQL 16 on AWS RDS. Backups are managed by AWS:

Create a manual RDS snapshot before deployments:

aws rds create-db-snapshot \
  --db-instance-identifier drop-production \
  --db-snapshot-identifier drop-pre-deploy-$(date +%Y%m%d-%H%M%S)

Restore from snapshot: Via AWS Console → RDS → Snapshots → Restore.

Local Dev Backups (Docker)

Local development data in the drop_pgdata Docker volume is disposable. Recreate with:

docker compose down -v   # Remove volume (deletes local data)
docker compose up -d
make db-push && npm run db:seed

Backup Verification

Verify production database connectivity and integrity:

# Check health endpoint
curl https://your-app-runner-url/api/health

# Connect to RDS (requires VPN or bastion)
psql $DATABASE_URL -c "SELECT COUNT(*) FROM users;"

Demo User

In non-production mode (NODE_ENV !== 'production'), a demo user is seeded:

Field Value
Email amir@example.com
Password demo1234
Role merchant

Source: Drizzle seed script in src/shared/db/seed.ts. Gated behind NODE_ENV !== 'production'.

Troubleshooting

Container won't start:

docker compose logs
docker compose exec drop-app env | grep JWT_SECRET

Database connection issues:

# Check PostgreSQL container is running
docker compose ps

# Test connection
docker compose exec db psql -U drop -d drop_dev -c "SELECT COUNT(*) FROM users;"

# Check app DATABASE_URL is set correctly
docker compose exec drop-app env | grep DATABASE_URL

Permission denied:

docker compose down -v   # Remove volumes
docker compose up -d     # Recreate with correct permissions

Cleanup:

docker compose down      # Stop containers
docker compose down -v   # Stop + remove volumes (WARNING: deletes data)
docker rmi drop-app      # Remove image

CI/CD & Monitoring

CI/CD & Monitoring

CI/CD Pipeline

Drop CI/CD Pipeline

Last updated: 2026-02-13 Source: src/drop-app/package.json, Dockerfile, fly.toml, vitest.config.ts, playwright.config.ts

Current State

Drop is in MVP/pre-production stage. Core CI/CD infrastructure exists including a GitHub Actions workflow.

What exists:

What does not exist yet:

Build Pipeline

Step 1: Install Dependencies

npm ci

Installs exact versions from package-lock.json.

Step 2: Lint

npm run lint     # eslint

Step 3: Type Check

npx tsc --noEmit

Step 4: Unit + Integration Tests

npm test         # vitest run

Runs all tests in tests/**/*.test.ts (from vitest.config.ts:7). Test setup: tests/setup.ts sets NODE_ENV=test.

Step 5: Build

npm run build    # next build

Produces standalone output for Docker deployment.

Step 6: Docker Build

docker build -t drop-app .

Multi-stage build: deps -> builder -> runner.

Step 7: E2E Tests (requires running server)

npx playwright test

Requires dev server on http://localhost:3000. Playwright auto-starts it via webServer config.

Test Framework Configuration

Vitest (Unit + Integration)

Config: src/drop-app/vitest.config.ts:1-15

Setting Value
Environment node
Include tests/**/*.test.ts
Setup tests/setup.ts
Path alias @ -> ./src

Playwright (E2E)

Config: src/drop-app/playwright.config.ts:1-39

Setting Value
Test dir ./tests/e2e
Parallel false (serial -- rate limiter is shared)
Workers 1
Retries (CI) 2
Timeout 30,000ms
Base URL http://localhost:3000
Reporter HTML
Trace on-first-retry

Test projects:

  1. user-flows -- Basic user journey tests (user-flows.spec.ts)
  2. full-flows -- Complete feature journeys (full-flows.spec.ts)
  3. input-chaos -- Malicious/edge-case input testing (input-chaos.spec.ts). Depends on user-flows.

Web server config: Auto-starts npm run dev for E2E tests. Reuses existing server if running. 30s timeout.

Deployment Targets

Fly.io (Staging)

Config: fly.toml:1-28

# Deploy to Fly.io staging
fly deploy

# Set secrets
fly secrets set JWT_SECRET="your-secret"
fly secrets set NEXT_PUBLIC_SERVICE_MODE="mock"

Region: arn (Stockholm) Auto-scaling: Scales to 0 when idle, auto-starts on request.

Docker (Self-hosted)

# Local dev (PostgreSQL 16 via Docker)
docker compose up -d

# Apply schema
make db-push

Existing GitHub Actions CI Workflow

File: .github/workflows/ci.yml

Triggers on push/PR to main or master:

Jobs:
  1. lint-and-typecheck — npm ci, npm run lint, tsc --noEmit
  2. test — npm ci, npm test --if-present (depends on lint-and-typecheck)
  3. build — npm ci, npm run build with JWT_SECRET placeholder (depends on lint-and-typecheck)
  4. e2e — npm ci, npx playwright install chromium, npm run build, npm run start (production mode), npx playwright test user-flows + full-flows, generate QA report, upload artifacts (depends on build)
  5. docker-build — docker build -t drop-app:ci (depends on test + build + e2e)

Artifacts uploaded:

Not yet implemented:

Status: Full CI pipeline including E2E tests in place. CD deployment tracked in security hardening checklist (security/hardening-checklist.md:120-126).

CI/CD & Monitoring

Monitoring & Alerting

Drop Monitoring

Last updated: 2026-02-17 Source: src/drop-app/src/app/api/health/route.ts, docker-compose.yml, fly.toml, src/lib/alerts.ts

Health Check Endpoint

Route: GET /api/health Source: src/drop-app/src/app/api/health/route.ts:1-35

What It Checks

  1. Database connectivity -- Executes SELECT 1 as ok against the database
  2. Database latency -- Measures query execution time in milliseconds
  3. Database driver -- Reports pg (PostgreSQL 16 via Drizzle ORM)
  4. Service mode -- Reports NEXT_PUBLIC_SERVICE_MODE (mock or live)
  5. Application uptime -- Tracks seconds since server start
  6. Application version -- Reads from npm_package_version env var, defaults to 0.1.0

Status Values

Response Format

Healthy (200 OK):

{
  "data": {
    "status": "ok",
    "version": "0.1.0",
    "uptime": 3600,
    "checks": {
      "db": { "status": "pass", "latencyMs": 2, "driver": "pg" },
      "services": { "mode": "live" }
    },
    "timestamp": "2026-02-17T12:00:00.000Z"
  }
}

Down (503 Service Unavailable):

{
  "data": {
    "status": "down",
    "version": "0.1.0",
    "uptime": 3600,
    "checks": {
      "db": { "status": "fail" },
      "services": { "mode": "live" }
    },
    "timestamp": "2026-02-17T12:00:00.000Z"
  }
}

Container Health Checks

Docker Compose (MVP)

Source: docker-compose.yml:12-17

healthcheck:
  test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:3000/api/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 10s

Docker Compose (Production)

Source: docker-compose.production.yml:9-14

Same health check configuration as MVP. Additionally, PostgreSQL has its own health check:

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U drop"]
  interval: 10s
  timeout: 5s
  retries: 5

The drop-app service depends on PostgreSQL being healthy before starting (depends_on.postgres.condition: service_healthy).

Fly.io

Source: fly.toml:19-23

[[http_service.checks]]
  grace_period = "10s"
  interval = "30s"
  method = "GET"
  path = "/api/health"
  timeout = "5s"

Fly.io uses this health check to determine machine readiness and to route traffic.

Current Monitoring State

What Exists

What Does Not Exist Yet

Sentry Error Tracking

Status: REMOVED (MC #1271 — Sentry deinstalled)

Slack Alerting

Status: Implemented (MC #1183) Source: src/lib/alerts.ts, instrumentation.ts

Features

Setup Instructions

  1. Create incoming webhook in Slack workspace:
    • Go to Slack App Directory → Incoming Webhooks
    • Choose channel (e.g., #ops or #alerts)
    • Copy webhook URL
  2. Set environment variable: 
    # .env.local (server-side secret)
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXX

Required Environment Variable

Variable Required Description
SLACK_WEBHOOK_URL Yes (production) Slack incoming webhook URL

Note: When SLACK_WEBHOOK_URL is not set, alerts are logged to console but not sent to Slack.

Alert Types and Severities

Severity Emoji Use Case
info ℹ️ Application startup, normal operations
warning ⚠️ Degraded performance, non-critical issues
critical 🚨 Service outages, data loss, security incidents

Cooldown Behavior

Example: If "Database connection failed" is sent at 10:00, the next attempt before 10:10 will be skipped. But "High latency detected" can still be sent at 10:05.

Usage in Code

import { sendAlert } from '@/lib/alerts';

// Basic alert
await sendAlert({
  severity: 'critical',
  title: 'Database connection failed',
  message: 'PostgreSQL unreachable after 3 retries',
});

// Alert with details
await sendAlert({
  severity: 'warning',
  title: 'High error rate detected',
  message: '15 errors in last 5 minutes',
});

Current Integrations

Error Spike Detection

The alerting system automatically detects error spikes using a rolling window approach:

How it works:

  1. Every server error (HTTP 5xx) is tracked via trackError()
  2. Maintains rolling 1-minute window of error timestamps
  3. When count exceeds threshold (5 errors in 60 seconds), sends critical alert
  4. Integrates with middleware error handling

Threshold: 5 errors within 60 seconds Alert severity: Critical (🚨) Implementation: src/lib/alerts.ts:trackError(), wired into src/lib/middleware.ts:jsonError()

Note: Error counter is in-memory and resets on app restart. For production workloads requiring persistent tracking, consider Redis-backed counters.

BetterStack Uptime Monitoring

Status: Ready to configure (setup guide available) Documentation: BETTERSTACK-SETUP.md

Overview

BetterStack provides external uptime monitoring independent of Drop's infrastructure. Unlike internal health checks (Docker, Fly.io) that only work when containers are running, BetterStack detects total infrastructure failures.

Free tier includes:

Monitor URL Purpose Expected Response
Health Endpoint https://drop.alai.no/api/health API + DB connectivity 200, body contains "status":"ok"
Landing Page https://drop.alai.no Public website 200, body contains Send penger
Multi-Region Check https://drop.alai.no/api/health Geographic availability 200, body contains "status":"ok"

Alert Escalation

BetterStack sends alerts through multiple channels:

Minute 0:   Alert fires → Slack #drop-ops (immediate)
Minute 5:   Still down → Email to alem@alai.no
Minute 15:  Still down → SMS (requires paid plan)

Status Page

Public status page shows real-time service status:

Setup Instructions

Complete setup guide with step-by-step instructions: BETTERSTACK-SETUP.md

Setup includes:

  1. Account creation (free tier)
  2. Configure 3 monitors (health, landing, multi-region)
  3. Slack integration (#drop-ops channel)
  4. On-call schedule and escalation policy
  5. Public status page creation
  6. Testing and verification

Key Features

Proactive monitoring:

Incident management:

Reporting:

Integration with Drop Alerting

BetterStack complements Drop's internal alerting (src/lib/alerts.ts):

Feature Drop Internal Alerts BetterStack External
Detects Application errors, error spikes Infrastructure outages
When App is running App is unreachable
Source Application logs External HTTP checks
Delivery Slack webhook (direct) Escalation policy
Use case Code bugs, DB issues Container crashes, network failures

Example: Database connection fails:

  1. Drop internal alert: "Database connection failed" → Slack #drop-ops (immediate)
  2. BetterStack: Health check returns 503 → Slack #drop-ops + Email after 5 min

Maintenance Windows

When performing planned maintenance (deployments, upgrades):

  1. Create maintenance window in BetterStack
  2. Select affected monitors
  3. Set duration (e.g., 1 hour)
  4. Effect: Alerts suppressed, status page shows "Scheduled Maintenance"

Prevents: False downtime alerts during intentional service interruptions.

Best Practices

Do's:

Don'ts:

External Uptime Monitoring (Alternative: UptimeRobot)

Status: Alternative to BetterStack (not recommended)

BetterStack is recommended over UptimeRobot for Drop because:

UptimeRobot Setup (if BetterStack unavailable)

Cost: Free tier (50 monitors, 5-minute interval)

  1. Create account at uptimerobot.com
  2. Add HTTP(S) monitor:
    • Friendly Name: Drop Production
    • URL: https://drop.alai.no/api/health
    • Monitoring Interval: 5 minutes (free tier) or 1 minute (paid)
  3. Configure alert contacts:
    • Slack webhook (via Alert Contacts)
    • Email (alem@alai.no)
  4. Set Keyword Monitoring: Response contains "status":"ok"

Limitations:

Monitoring Stack Summary

Implemented (MC #1184)

Recommended (Manual Setup)

Future Enhancements (TODO)

Environment Variables Reference

Required for Production

# Slack alerting
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXX

Dev Mode (All Optional)

All monitoring features gracefully degrade when env vars are not set:

This allows development to work without external services configured.

CI/CD & Monitoring

Production Deployment

Drop AWS Amplify Deployment Guide

Rebrand note (2026-02-14): Originally titled "FontelePay". Product rebranded to Drop. Some env var references (Swan, Stripe) are FUTURE integrations — Drop uses a PSD2 pass-through model. See Drop CLAUDE.md.

This guide covers deploying Drop to AWS Amplify in the Frankfurt (eu-central-1) region.

Prerequisites

Step 1: Create Amplify App

  1. Go to AWS Amplify Console
  2. Ensure you're in eu-central-1 (Frankfurt) region
  3. Click Create new app
  4. Select Host web app

Step 2: Connect Repository

  1. Choose GitHub as your Git provider
  2. Authorize AWS Amplify to access your GitHub account
  3. Select the Drop repository
  4. Choose the branch to deploy (e.g., main or production)

Step 3: Configure Build Settings

Amplify will auto-detect Next.js. Verify the settings match amplify.yml:

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - npm ci
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
      - .next/cache/**/*

Step 4: Configure Environment Variables

In Amplify Console, go to App settings > Environment variables and add:

Required Variables

Variable Description Example
NODE_ENV Environment production
NEXT_PUBLIC_APP_URL Your app URL https://drop.amplifyapp.com

Swan BaaS

Variable Description
SWAN_API_URL https://api.swan.io (production)
SWAN_CLIENT_ID OAuth2 Client ID
SWAN_CLIENT_SECRET OAuth2 Client Secret
SWAN_PROJECT_ID Project ID
SWAN_WEBHOOK_SECRET Webhook validation secret

Stripe

Variable Description
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY Publishable key (pk_live_...)
STRIPE_SECRET_KEY Secret key (sk_live_...)
STRIPE_WEBHOOK_SECRET Webhook secret (whsec_...)

Sumsub KYC

Variable Description
SUMSUB_APP_TOKEN App token
SUMSUB_SECRET_KEY Secret key
SUMSUB_WEBHOOK_SECRET Webhook secret
SUMSUB_LEVEL_NAME KYC flow level

Database

Variable Description
DATABASE_URL PostgreSQL connection string
REDIS_URL Redis connection string

Authentication

Variable Description
JWT_SECRET Min 32 characters
SESSION_SECRET Min 32 characters

Step 5: Configure Next.js for Standalone Output

Update next.config.ts to enable standalone output for optimal Amplify deployment:

import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  output: 'standalone',
};

export default nextConfig;

Step 6: Deploy

  1. Click Save and deploy
  2. Monitor the build in the Amplify Console
  3. Once complete, your app will be available at https://<branch>.<app-id>.amplifyapp.com

Step 7: Configure Custom Domain (Optional)

  1. Go to App settings > Domain management
  2. Click Add domain
  3. Enter your domain (e.g., app.getdrop.no)
  4. Follow DNS configuration instructions
  5. SSL certificate is automatically provisioned

Step 8: Set Up Branch Deployments

For staging/production workflows:

  1. Go to App settings > General
  2. Click Edit
  3. Enable Branch auto-detection
  4. Configure branch patterns:
    • main -> Production
    • staging -> Staging
    • feature/* -> Preview environments

Monitoring & Health Checks

Health Endpoint

The app exposes /api/health for load balancer health checks:

curl https://your-app.amplifyapp.com/api/health

Response:

{
  "status": "healthy",
  "timestamp": "2026-02-05T12:00:00.000Z",
  "version": "0.1.0",
  "uptime": 3600,
  "checks": {}
}

CloudWatch Logs

  1. Go to App settings > Monitoring
  2. View build logs and access logs
  3. Set up CloudWatch alarms for errors

Troubleshooting

Build Fails

  1. Check build logs in Amplify Console
  2. Verify package.json scripts are correct
  3. Ensure all dependencies are in package.json

Environment Variables Not Working

  1. Verify variables are set in Amplify Console
  2. Remember: NEXT_PUBLIC_ prefix required for client-side access
  3. Redeploy after changing environment variables

502/503 Errors

  1. Check /api/health endpoint
  2. Review CloudWatch logs
  3. Verify database connections are correct
  4. Check memory limits (adjust if needed)

Cold Starts

For serverless functions, cold starts may occur. Mitigate by:

  1. Using connection pooling for databases
  2. Keeping functions warm with scheduled pings
  3. Optimizing bundle size

Security Checklist

Cost Optimization

Rollback

To rollback to a previous deployment:

  1. Go to Deployments in Amplify Console
  2. Find the previous successful deployment
  3. Click Redeploy this version

Support

CI/CD & Monitoring

BetterStack Setup

BetterStack Uptime Monitoring Setup Guide

Last updated: 2026-02-20 Related: MONITORING.md, health-check.sh Purpose: External uptime monitoring for Drop production environment

Why BetterStack?

BetterStack provides external uptime monitoring independent of Drop's infrastructure:

Key difference from internal health checks: Internal checks (Docker, Fly.io) only work when the container is running. BetterStack catches total outages.

Free Tier Limits

Plan: Free tier (no credit card required) Limits:

Upgrade required for:

Account Setup

Step 1: Create Account

  1. Go to https://betterstack.com/uptime
  2. Click "Start free trial" (becomes free tier after trial)
  3. Sign up with Alem's email: alem@alai.no
  4. Verify email address
  5. Create workspace name: "ALAI Products" (shared across Drop, BasicFakta)

Step 2: Configure Team

  1. Navigate to Settings > Team
  2. Add team members:
    • alem@alai.no (Owner)
    • john@basicconsulting.no (Admin)
  3. Set Default timezone: Europe/Oslo (UTC+1)

Monitor Configuration

Monitor 1: Health Endpoint (Primary)

Purpose: Verify API health and database connectivity

  1. Go to Monitors > Create Monitor

  2. Configure:

    • Monitor name: Drop Health Check
    • Monitor type: HTTP
    • URL: https://drop.alai.no/api/health
    • Check interval: 3 minutes (free tier)
    • Request timeout: 5 seconds
    • Method: GET
    • Confirmation period: 30 seconds (1 retry before alerting)

  3. Expected Response:

    • Status code: 200
    • Keyword check: Enable
      • Response body contains: "status":"ok"
      • Why: Ensures health endpoint returns valid JSON, not just HTTP 200

  4. Advanced settings:

    • Follow redirects: Enabled (default)
    • Verify SSL certificate: Enabled
    • SSL expiry warning: 14 days before expiration

  5. Click Create Monitor

Monitor 2: Landing Page

Purpose: Verify public website availability

  1. Go to Monitors > Create Monitor

  2. Configure:

    • Monitor name: Drop Landing Page
    • Monitor type: HTTP
    • URL: https://drop.alai.no
    • Check interval: 3 minutes
    • Request timeout: 10 seconds (landing page has more assets)
    • Method: GET
    • Confirmation period: 30 seconds

  3. Expected Response:

    • Status code: 200
    • Keyword check: Enable
      • Response body contains: Send penger (tagline verification)

  4. Click Create Monitor

Monitor 3: Multi-Region Health Check

Purpose: Detect regional networking issues

  1. Go to Monitors > Create Monitor

  2. Configure:

    • Monitor name: Drop Health (US East)
    • Monitor type: HTTP
    • URL: https://drop.alai.no/api/health
    • Check interval: 3 minutes
    • Request timeout: 5 seconds
    • Method: GET
    • Confirmation period: 30 seconds

  3. Expected Response:

    • Status code: 200
    • Keyword check: Response body contains "status":"ok"

  4. Advanced settings:

    • Region: US East (different from default EU region)
    • Why: Detects if Drop is unreachable from specific geographies

  5. Click Create Monitor

Slack Integration

Step 1: Create Slack Incoming Webhook

  1. Go to your Slack workspace: alai-talk.slack.com
  2. Navigate to Slack App Directory > Incoming Webhooks
  3. Click Add to Slack
  4. Select channel: #drop-ops (create if doesn't exist)
  5. Click Add Incoming Webhooks Integration
  6. Copy webhook URL (format: https://hooks.slack.com/services/T.../B.../XXX)
  7. Save this URL securely (needed for BetterStack)

Step 2: Add Slack Integration in BetterStack

  1. In BetterStack, go to Integrations
  2. Click Add Integration > Slack
  3. Paste webhook URL from Step 1
  4. Configure:
    • Integration name: Drop Ops Slack
    • Notification channel: #drop-ops
  5. Test integration: Click Send test message
    • Verify message appears in #drop-ops channel
  6. Click Save Integration

On-Call Team Setup

Step 1: Create On-Call Schedule

  1. Go to On-Call > Create Schedule
  2. Configure:
    • Schedule name: Drop Primary On-Call
    • Timezone: Europe/Oslo
  3. Add rotation:
    • Team member: alem@alai.no
    • Schedule type: 24/7 (always on-call for now)
  4. Click Create Schedule

Step 2: Configure Escalation Policy

  1. Go to Escalation Policies > Create Policy

  2. Configure:

    • Policy name: Drop Production Incidents

  3. Add escalation steps:

    Step 1 (Immediate):

    • Who: Drop Ops Slack integration
    • Delay: 0 minutes

    Step 2 (If still down after 5 minutes):

    • Who: alem@alai.no (Email)
    • Delay: 5 minutes

    Step 3 (If still down after 15 minutes):

    • Who: alem@alai.no (SMS) — Requires phone number
    • Delay: 15 minutes
    • Note: SMS requires paid plan or verified phone number

  4. Click Create Policy

Step 3: Assign Policy to Monitors

  1. Go to Monitors
  2. For each monitor (Drop Health Check, Drop Landing Page, Drop Health (US East)):
    • Click monitor name
    • Go to Settings > Escalation Policy
    • Select: Drop Production Incidents
    • Click Save

Status Page Setup

Purpose

Public status page allows clients and stakeholders to check Drop availability without contacting support.

Step 1: Create Status Page

  1. Go to Status Pages > Create Status Page

  2. Configure:

    • Page name: Drop Status
    • Subdomain: drop-status (URL: https://drop-status.betteruptime.com)
    • Custom domain (optional): status.drop.alai.no (requires DNS setup)

  3. Design settings:

    • Logo: Upload Drop logo (green rounded rectangle)
    • Brand color: #0B6E35 (Drop primary green)
    • Header text: Drop Status
    • Tagline: Real-time service status and incident updates

  4. Visibility:

    • Public: Yes (anyone can view)
    • Search engine indexing: No (prevent Google indexing)

  5. Click Create Status Page

Step 2: Add Components

  1. In the status page settings, go to Components

  2. Click Add Component

  3. Add three components:

    Component 1:

    • Name: API & Health Endpoint
    • Linked monitor: Drop Health Check
    • Description: Core API functionality and database connectivity

    Component 2:

    • Name: Landing Page
    • Linked monitor: Drop Landing Page
    • Description: Public website and marketing content

    Component 3:

    • Name: Global Network
    • Linked monitor: Drop Health (US East)
    • Description: International access and routing

  4. Click Save Components

Step 3: Configure Incident Communication

  1. Go to Status Pages > Settings > Incident Updates
  2. Enable:
    • Auto-create incidents: Yes (when monitor goes down)
    • Auto-resolve incidents: Yes (when monitor recovers)
  3. Notification subscribers:
    • Email subscriptions: Enabled (users can subscribe to updates)
    • Webhook notifications: Disabled (optional for future)

Step 4: Share Status Page

Once created, share the status page URL:

Status Page URL: https://drop-status.betteruptime.com

Verification Checklist

After completing setup, verify:

Testing the Setup

Test 1: Manual Down Alert

  1. Go to Monitors > Drop Health Check
  2. Click Pause Monitor (simulates downtime)
  3. Expected behavior:
    • Slack alert in #drop-ops within 30 seconds
    • Email to alem@alai.no after 5 minutes (if still paused)
  4. Click Resume Monitor to clear alert

Test 2: Actual Downtime

  1. SSH into production server (or use AWS App Runner console)
  2. Stop the Drop application container temporarily
  3. Wait for BetterStack to detect downtime (max 3 minutes + 30s confirmation)
  4. Expected behavior:
    • Monitor shows red status
    • Slack alert in #drop-ops
    • Status page component shows "Down"
  5. Restart application and verify recovery alert

Test 3: SSL Expiry Warning

  1. Go to Monitors > Drop Health Check
  2. Verify SSL expiry warning is enabled (14 days)
  3. Expected behavior:
    • Alert sent 14 days before SSL certificate expiration
    • Action required: Renew certificate before expiry

Alert Examples

Downtime Alert (Slack)

🚨 Drop Health Check is DOWN

Monitor: Drop Health Check
Status: DOWN
Response: Connection timeout
Region: EU West
Time: 2026-02-20 10:30 UTC

View incident: https://betterstack.com/incidents/...

Recovery Alert (Slack)

✅ Drop Health Check is UP

Monitor: Drop Health Check
Status: UP
Response: 200 OK (2ms)
Downtime duration: 3 minutes
Time: 2026-02-20 10:33 UTC

Incident closed: https://betterstack.com/incidents/...

SSL Expiry Warning (Email)

Subject: [BetterStack] SSL certificate expiring in 14 days

Monitor: Drop Health Check
Domain: drop.alai.no
Certificate expiry: 2026-03-06 23:59 UTC

Action required: Renew SSL certificate before expiration.

Maintenance Mode

When performing planned maintenance (deployments, infrastructure upgrades):

  1. Go to Maintenance Windows > Create Window
  2. Configure:
    • Name: Drop Deployment
    • Start time: 2026-02-20 22:00 UTC
    • Duration: 1 hour
    • Affected monitors: Select all Drop monitors
  3. Notification:
    • Status page update: Yes (shows maintenance banner)
    • Alert suppression: Yes (no downtime alerts during window)
  4. Click Create Maintenance Window

Effect: During maintenance, downtime alerts are suppressed and status page shows "Scheduled Maintenance" instead of "Down".

Best Practices

Do's

Don'ts

Troubleshooting

Monitor shows false positives (frequent up/down)

Cause: Network instability or slow response times Fix:

  1. Increase Request timeout from 5s to 10s
  2. Increase Confirmation period from 30s to 60s
  3. Check Drop API latency in logs

Slack alerts not received

Cause: Webhook URL incorrect or channel archived Fix:

  1. Go to Integrations > Drop Ops Slack
  2. Click Send test message
  3. If fails, regenerate webhook in Slack and update BetterStack

Email alerts delayed

Cause: Email provider spam filtering Fix:

  1. Whitelist notifications@betterstack.com in email settings
  2. Check spam/junk folder
  3. Verify email address in BetterStack team settings

Status page not updating

Cause: Monitor not linked to status page component Fix:

  1. Go to Status Pages > Drop Status > Components
  2. Ensure each component has a Linked monitor assigned
  3. Save changes and trigger test alert

Support

BetterStack Support:

Internal Contact:

CI/CD & Monitoring

Sentry Setup

Drop Sentry Setup

Last updated: 2026-02-20 Source: src/drop-app/src/lib/sentry.ts, src/drop-app/src/lib/sentry-server.ts, src/drop-api/src/lib/sentry.ts, src/drop-app/.env.example

Overview

Drop uses Sentry for error tracking and performance monitoring across three components:

  1. drop-app (client-side) - Browser errors via @sentry/browser
  2. drop-app (server-side) - Next.js middleware/API errors via custom envelope API
  3. drop-api - Backend API errors via @sentry/node

All three components share the same DSN and gracefully degrade to console-only logging when Sentry is not configured.

Sentry Account Setup

1. Create Free Sentry Account

  1. Visit sentry.io and sign up (free tier: 5,000 errors/month)
  2. Confirm email and log in

2. Create Projects

Create two separate projects (one for app, one for API):

Project 1: drop-app

  1. Click Projects → Create Project
  2. Platform: Next.js
  3. Project name: drop-app
  4. Team: Default team (or create drop-team)
  5. Alert frequency: On every new issue
  6. Click Create Project
  7. Copy the DSN (format: https://examplePublicKey@o0.ingest.sentry.io/0)

Project 2: drop-api

  1. Repeat steps above with platform Node.js
  2. Project name: drop-api
  3. Copy the DSN (different from drop-app)

IMPORTANT: Use separate projects to keep frontend and backend errors isolated.

Environment Variables Configuration

drop-app (.env.local)

Add these variables to src/drop-app/.env.local:

# --- Sentry (Error Tracking) ---
# Client-side error tracking (browser)
NEXT_PUBLIC_SENTRY_DSN=https://YOUR_PUBLIC_KEY@o0.ingest.sentry.io/YOUR_PROJECT_ID

# Server-side error tracking (middleware/API routes)
# NOTE: drop-app server uses custom envelope API (no @sentry/nextjs due to Turbopack incompatibility)
# Both client and server use the SAME DSN (NEXT_PUBLIC_SENTRY_DSN)

# Optional: Performance monitoring sample rate (0.0 to 1.0, default: 0.1 = 10%)
NEXT_PUBLIC_SENTRY_TRACES_SAMPLE_RATE=0.1

# Optional: For source map uploads (requires auth token from Sentry → Settings → Auth Tokens)
SENTRY_ORG=your-org-slug
SENTRY_PROJECT=drop-app
SENTRY_AUTH_TOKEN=your-auth-token

drop-api (.env)

Add these variables to src/drop-api/.env:

# --- Sentry (Error Tracking) ---
SENTRY_DSN=https://YOUR_PUBLIC_KEY@o0.ingest.sentry.io/YOUR_API_PROJECT_ID

# Optional: Performance monitoring sample rate (0.0 to 1.0, default: 0.1 = 10%)
SENTRY_TRACES_SAMPLE_RATE=0.1

# Optional: For source map uploads
SENTRY_ORG=your-org-slug
SENTRY_PROJECT=drop-api
SENTRY_AUTH_TOKEN=your-auth-token

Where to find these values:

Verification

Test Client-Side Error Capture (drop-app)

  1. Start the app: npm run dev (in src/drop-app/)
  2. Open browser console: http://localhost:3000
  3. Trigger test error via console: 
    throw new Error("Sentry test error - client-side");
  4. Check Sentry dashboard: Projects → drop-app → Issues
  5. You should see the test error appear within 10 seconds

Expected behavior:

Test Server-Side Error Capture (drop-app)

  1. Create test API route: src/drop-app/src/app/api/sentry-test/route.ts 
    import { NextResponse } from 'next/server';
import { captureServerError } from '@/lib/sentry-server';

export async function GET() {
  try {
    throw new Error('Sentry test error - server-side');
  } catch (error) {
    captureServerError(error as Error, { tags: { test: 'true' } });
    return NextResponse.json({ error: 'Test error sent to Sentry' }, { status: 500 });
  }
}
  2. Visit: http://localhost:3000/api/sentry-test
  3. Check server console: [Sentry Server] Error captured: Error: Sentry test error - server-side
  4. Check Sentry dashboard: Projects → drop-app → Issues

Test API Error Capture (drop-api)

  1. Start the API: npm run dev (in src/drop-api/)
  2. Trigger test error via curl: 
    curl http://localhost:4000/api/sentry-test
  3. OR create test endpoint in src/drop-api/src/routes/test.ts: 
    import { Router } from 'express';
import { captureError } from '../lib/sentry.js';

const router = Router();

router.get('/sentry-test', (req, res) => {
  try {
    throw new Error('Sentry test error - API');
  } catch (error) {
    captureError(error as Error, { tags: { test: 'true' } });
    res.status(500).json({ error: 'Test error sent to Sentry' });
  }
});

export default router;
  4. Check Sentry dashboard: Projects → drop-api → Issues

Source Map Upload Setup

Source maps allow Sentry to show readable stack traces instead of minified code.

1. Install Sentry CLI

# macOS (Homebrew)
brew install getsentry/tools/sentry-cli

# Or via npm (global)
npm install -g @sentry/cli

2. Configure Sentry CLI

Create .sentryclirc in project root:

[defaults]
url=https://sentry.io/
org=your-org-slug
project=drop-app

[auth]
token=your-auth-token

IMPORTANT: Add .sentryclirc to .gitignore (contains auth token).

3. Add Build Script (drop-app)

Update src/drop-app/package.json:

{
  "scripts": {
    "build": "next build",
    "build:sentry": "next build && sentry-cli sourcemaps upload --validate .next/static"
  }
}

4. Test Source Map Upload

cd src/drop-app
npm run build:sentry

Expected output:

> Analyzing source maps for sentry
> Uploading source maps to Sentry
✓ Successfully uploaded source maps

5. CI/CD Integration

For automated uploads in CI/CD, add these secrets to your deployment platform:

Vercel/Railway/Fly.io:

Then update build command:

npm run build && sentry-cli sourcemaps upload --validate .next/static

Alert Rules Configuration

1. New Issue Alert (drop-app)

  1. Go to Projects → drop-app → Settings → Alerts
  2. Click Create Alert Rule
  3. Configure:
    • Conditions: When a new issue is created
    • Filters: Environment = production
    • Actions:
      • Send notification to: Slack channel #drop-alerts
      • Send email to: alem@alai.no
  4. Save rule

2. High Error Rate Alert (drop-app)

  1. Create new alert rule
  2. Configure:
    • Conditions: Number of events in an issue is more than 100 in 1 hour
    • Filters: Environment = production, Level = error
    • Actions:
      • Send notification to: Slack channel #drop-alerts
      • Send email to: alem@alai.no
  3. Save rule

3. Critical Error Alert (drop-api)

  1. Go to Projects → drop-api → Settings → Alerts
  2. Create alert rule:
    • Conditions: When a new issue is created AND Level = fatal
    • Filters: Environment = production
    • Actions:
      • Send notification to: Slack channel #drop-critical
      • Send email to: alem@alai.no
  3. Save rule

4. Performance Degradation Alert (drop-app)

  1. Create alert rule:
    • Conditions: Average transaction duration is above 2000ms for 5 minutes
    • Filters: Environment = production, Transaction = /api/transactions/*
    • Actions:
      • Send notification to: Slack channel #drop-performance
  2. Save rule

Slack Integration (Optional)

  1. Go to Settings → Integrations → Slack
  2. Click Add Workspace
  3. Authorize Sentry to access your Slack workspace
  4. Select channels: #drop-alerts, #drop-critical, #drop-performance
  5. Test integration by triggering a test error

PII Scrubbing

All three Sentry integrations automatically scrub sensitive data before sending events:

Scrubbed fields:

Implementation:

Verification:

  1. Trigger error with sensitive data: 
    try {
  throw new Error('Login failed for user with password=secret123');
} catch (error) {
  captureError(error, { extra: { cardNumber: '1234567890123456' } });
}
  2. Check Sentry event:
    • Message should show: Login failed for user with password=[REDACTED]
    • Extra context should show: cardNumber: [REDACTED]

Environment-Specific Configuration

Development

# .env.local (development)
NEXT_PUBLIC_SENTRY_DSN=  # Leave empty to disable Sentry in dev

Staging

# .env.staging
NEXT_PUBLIC_SENTRY_DSN=https://YOUR_KEY@sentry.io/YOUR_PROJECT_ID
NEXT_PUBLIC_SENTRY_TRACES_SAMPLE_RATE=0.5
SENTRY_AUTH_TOKEN=your-auth-token

Production

# .env.production
NEXT_PUBLIC_SENTRY_DSN=https://YOUR_KEY@sentry.io/YOUR_PROJECT_ID
NEXT_PUBLIC_SENTRY_TRACES_SAMPLE_RATE=0.1
SENTRY_AUTH_TOKEN=your-auth-token

Troubleshooting

No errors appearing in Sentry dashboard

Check 1: DSN configured?

# drop-app
echo $NEXT_PUBLIC_SENTRY_DSN

# drop-api
echo $SENTRY_DSN

Check 2: Console output?

Check 3: Network errors?

Check 4: Environment mismatch?

Source maps not working (minified stack traces)

Check 1: Source maps uploaded?

cd src/drop-app
sentry-cli releases list

Check 2: Release version matches?

Check 3: Upload command ran?

# Manually test upload
sentry-cli sourcemaps upload --validate .next/static

PII still appearing in events

Check 1: Verify beforeSend hook

Check 2: Add custom scrubbing

Cost Management

Sentry Free Tier:

Staying within free tier:

  1. Lower sample rate: Set SENTRY_TRACES_SAMPLE_RATE=0.1 (10%)
  2. Filter noisy errors: Use beforeSend to ignore expected errors (e.g., 404s)
  3. Set up quotas: Sentry → Settings → Quotas → Set monthly limits

Example: Ignore 404 errors

beforeSend(event, hint) {
  // Ignore 404 errors
  if (event.request?.url?.includes('/api/') && hint?.originalException?.message?.includes('404')) {
    return null; // Don't send to Sentry
  }
  return event;
}

Security Considerations

  1. Auth token storage:

    • NEVER commit .sentryclirc to git
    • Store SENTRY_AUTH_TOKEN in CI/CD secrets, not .env files

  2. DSN exposure:

    • NEXT_PUBLIC_SENTRY_DSN is exposed to client-side code (safe - it's public)
    • Sentry rate-limits abuse via DSN quotas

  3. PII scrubbing:

    • Always verify PII scrubbing works before deploying to production
    • Test with real-world data patterns (Norwegian fødselsnummer, BankID tokens)

  4. Access control:

    • Limit Sentry dashboard access to authorized team members only
    • Use Sentry Teams to restrict project access

References

Next Steps

  1. Create Sentry account and projects (drop-app, drop-api)
  2. Add DSN to .env.local (development) and .env.production (production)
  3. Test error capture in all three components
  4. Configure alert rules (new issues, high error rate, critical errors)
  5. Set up source map uploads for production builds
  6. Integrate Slack notifications (optional)
  7. Monitor error dashboard daily during initial deployment
CI/CD & Monitoring

CloudWatch Logs Setup

CloudWatch Logs Setup — Drop Production

Date: 2026-02-22 Priority: P0 (Production Blocker) Effort: 2 hours Cost: ~$5/month (30 GB ingestion)

Overview

AWS App Runner automatically streams application logs (stdout/stderr) to CloudWatch Logs. This setup guide configures retention policies, log insights queries, and alarms for production monitoring.

Prerequisites

Configuration

1. Set Log Retention Policy

Default: CloudWatch Logs retain forever (expensive) Recommendation: 30 days (production), 7 days (staging)

# Production: 30 days retention
aws logs put-retention-policy \
  --log-group-name /aws/apprunner/drop-production \
  --retention-in-days 30 \
  --region eu-west-1

# Staging: 7 days retention
aws logs put-retention-policy \
  --log-group-name /aws/apprunner/drop-staging \
  --retention-in-days 7 \
  --region eu-west-1

Verify retention:

aws logs describe-log-groups \
  --log-group-name-prefix /aws/apprunner/drop \
  --region eu-west-1 \
  | jq '.logGroups[] | {name: .logGroupName, retention: .retentionInDays}'

# Expected:
# {
#   "name": "/aws/apprunner/drop-production",
#   "retention": 30
# }

2. Create Log Insights Queries

Purpose: Pre-built queries for common investigations.

Query 1: All Errors (Last Hour)

fields @timestamp, level, message, metadata.error, metadata.userId, requestId
| filter level = "error"
| sort @timestamp desc
| limit 100

Save as: drop-errors-last-hour

Query 2: User Activity Trace

fields @timestamp, level, message, metadata.userId, metadata.action, requestId
| filter metadata.userId = "usr_123"
| sort @timestamp desc
| limit 500

Save as: drop-user-activity-trace

Query 3: Request Trace by ID

fields @timestamp, level, message, metadata
| filter requestId = "req_abc123"
| sort @timestamp asc

Save as: drop-request-trace

Query 4: API Endpoint Performance

fields @timestamp, message, metadata.endpoint, metadata.latencyMs
| filter metadata.latencyMs > 1000
| stats avg(metadata.latencyMs) as avg_latency, max(metadata.latencyMs) as max_latency, count() as slow_requests by metadata.endpoint
| sort slow_requests desc

Save as: drop-slow-endpoints

Query 5: Authentication Events

fields @timestamp, level, message, metadata.action, metadata.userId, metadata.ip
| filter metadata.action in ["login_success", "login_failure", "logout"]
| sort @timestamp desc
| limit 100

Save as: drop-auth-events

Query 6: Payment Failures

fields @timestamp, level, message, metadata.errorCode, metadata.transactionId, metadata.userId
| filter metadata.errorCode in ["INSUFFICIENT_FUNDS", "PAYMENT_REJECTED", "TIMEOUT"]
| sort @timestamp desc
| limit 50

Save as: drop-payment-failures

3. Create CloudWatch Alarms

Alarm 1: High Error Rate

Metric: Error log entries per minute Threshold: >10 errors/minute for 2 consecutive periods Action: Send SNS notification → Slack webhook

# Create metric filter
aws logs put-metric-filter \
  --log-group-name /aws/apprunner/drop-production \
  --filter-name drop-error-count \
  --filter-pattern '{ $.level = "error" }' \
  --metric-transformations \
    metricName=ErrorCount,metricNamespace=Drop/Logs,metricValue=1,unit=Count \
  --region eu-west-1

# Create alarm
aws cloudwatch put-metric-alarm \
  --alarm-name drop-high-error-rate \
  --alarm-description "Alert when error rate exceeds threshold" \
  --metric-name ErrorCount \
  --namespace Drop/Logs \
  --statistic Sum \
  --period 60 \
  --evaluation-periods 2 \
  --threshold 10 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions <SNS-TOPIC-ARN> \
  --region eu-west-1

Alarm 2: No Logs Received (Service Down)

Metric: Log ingestion stopped Threshold: No logs for 5 minutes Action: Send SNS notification

aws cloudwatch put-metric-alarm \
  --alarm-name drop-no-logs-received \
  --alarm-description "Alert when no logs received (service may be down)" \
  --metric-name IncomingLogEvents \
  --namespace AWS/Logs \
  --dimensions Name=LogGroupName,Value=/aws/apprunner/drop-production \
  --statistic Sum \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 1 \
  --comparison-operator LessThanThreshold \
  --treat-missing-data breaching \
  --alarm-actions <SNS-TOPIC-ARN> \
  --region eu-west-1

Alarm 3: Database Errors

Metric: Database connection errors Threshold: >5 DB errors in 5 minutes

aws logs put-metric-filter \
  --log-group-name /aws/apprunner/drop-production \
  --filter-name drop-db-errors \
  --filter-pattern '{ $.message = "*database*" && $.level = "error" }' \
  --metric-transformations \
    metricName=DatabaseErrors,metricNamespace=Drop/Logs,metricValue=1,unit=Count \
  --region eu-west-1

aws cloudwatch put-metric-alarm \
  --alarm-name drop-database-errors \
  --metric-name DatabaseErrors \
  --namespace Drop/Logs \
  --statistic Sum \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 5 \
  --comparison-operator GreaterThanThreshold \
  --alarm-actions <SNS-TOPIC-ARN> \
  --region eu-west-1

4. SNS Topic for Alerts

Create SNS topic (if not exists):

aws sns create-topic \
  --name drop-cloudwatch-alerts \
  --region eu-west-1

# Output:
# {
#   "TopicArn": "arn:aws:sns:eu-west-1:324480209768:drop-cloudwatch-alerts"
# }

Subscribe Slack webhook:

# Option 1: Email subscription (immediate)
aws sns subscribe \
  --topic-arn arn:aws:sns:eu-west-1:324480209768:drop-cloudwatch-alerts \
  --protocol email \
  --notification-endpoint alem@alai.no \
  --region eu-west-1

# Confirm subscription via email link

# Option 2: Lambda → Slack (requires Lambda function)
# See: infrastructure/cloudwatch-to-slack-lambda.md (future enhancement)

5. Export Logs to S3 (Compliance/Archival)

Purpose: Long-term storage (>30 days) for compliance, cheaper than CloudWatch.

Create S3 bucket:

aws s3 mb s3://drop-logs-archive --region eu-west-1

# Set lifecycle policy (move to Glacier after 90 days)
cat > lifecycle.json <<EOF
{
  "Rules": [
    {
      "Id": "archive-old-logs",
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ],
      "Expiration": {
        "Days": 1825
      }
    }
  ]
}
EOF

aws s3api put-bucket-lifecycle-configuration \
  --bucket drop-logs-archive \
  --lifecycle-configuration file://lifecycle.json

Create export task (manual, run monthly):

# Export last 30 days to S3 (run on day 1 of each month)
START_TIME=$(date -u -d '60 days ago' +%s)000
END_TIME=$(date -u -d '30 days ago' +%s)000

aws logs create-export-task \
  --log-group-name /aws/apprunner/drop-production \
  --from $START_TIME \
  --to $END_TIME \
  --destination drop-logs-archive \
  --destination-prefix logs/$(date +%Y-%m) \
  --region eu-west-1

# Check export status
aws logs describe-export-tasks --region eu-west-1

Automate with Lambda (future):

Log Format

Current Format (Structured JSON)

Example log entry:

{
  "timestamp": "2026-02-22T10:30:45.123Z",
  "level": "info",
  "message": "User logged in",
  "requestId": "req_abc123",
  "metadata": {
    "userId": "usr_456",
    "email": "user@example.com",
    "ip": "1.2.3.4",
    "action": "login_success"
  }
}

CloudWatch Logs Insights automatically parses JSON fields, enabling queries like:

| filter metadata.userId = "usr_456"

Cost Estimate

CloudWatch Logs Pricing (EU-West-1)

Expected Usage (Production)

Total: ~$17/month

Cost Optimization

  1. Reduce log verbosity (filter debug logs in production):

    // src/lib/logger.ts
const minLevel = process.env.NODE_ENV === 'production' ? 'info' : 'debug';

  2. Use sampling for high-volume events:

    if (Math.random() < 0.1) { // Log 10% of requests
  logger.debug('Request details', { ... });
}

  3. Export to S3 for long-term storage ($0.023/GB/month, 23% cheaper)

Querying Logs

Via AWS Console

  1. Open CloudWatch Console: https://console.aws.amazon.com/cloudwatch/
  2. Navigate to: Logs → Log groups → /aws/apprunner/drop-production
  3. Click "Search log group" or "Insights queries"
  4. Select saved query or write custom query

Via AWS CLI

# Run saved query
aws logs start-query \
  --log-group-name /aws/apprunner/drop-production \
  --start-time $(date -u -d '1 hour ago' +%s) \
  --end-time $(date -u +%s) \
  --query-string 'fields @timestamp, level, message | filter level = "error" | sort @timestamp desc' \
  --region eu-west-1

# Get query results (use queryId from previous command)
aws logs get-query-results --query-id <query-id> --region eu-west-1

Via Log Streaming (Real-Time)

# Stream logs in real-time (like tail -f)
aws logs tail /aws/apprunner/drop-production \
  --follow \
  --format short \
  --region eu-west-1

# Filter by error level
aws logs tail /aws/apprunner/drop-production \
  --follow \
  --filter-pattern '{ $.level = "error" }' \
  --region eu-west-1

Troubleshooting

Issue: No logs appearing in CloudWatch

Diagnosis:

# Check if log group exists
aws logs describe-log-groups \
  --log-group-name-prefix /aws/apprunner/drop \
  --region eu-west-1

# Check App Runner service logs integration
aws apprunner describe-service \
  --service-arn <ARN> \
  --region eu-west-1 \
  | jq '.Service.ObservabilityConfiguration'

Solution:

Issue: Logs not in JSON format

Diagnosis:

# Check log entries
aws logs tail /aws/apprunner/drop-production --format short --region eu-west-1 | head -10

Solution:

Checklist

Next Steps

  1. Deploy retention policies (run commands above)
  2. Test alarms (trigger error spike, verify alert received)
  3. Save Log Insights queries (via AWS Console)
  4. Schedule monthly S3 export (manual for now, automate later)
  5. Monitor costs (set billing alert at $20/month)

Last Updated: 2026-02-22 Owner: John (AI Director)

DevOps Stack

DevOps Stack

DevOps/SRE Stack

DevOps/SRE Stack for Drop (originally FontelePay)

Rebrand note (2026-02-14): FontelePay was renamed to Drop. Some references to FontelePay remain in this document (metric names, Sentry projects, API URLs). These should be updated when implementing the actual DevOps stack. Drop uses a PSD2 pass-through model — no wallet, no balance held by Drop.

Table of Contents

  1. Executive Summary
  2. CI/CD Pipeline
  3. Testing Strategy
  4. Monitoring & Observability
  5. Error Tracking
  6. Alerting & Incident Management
  7. Documentation
  8. Security Operations
  9. Cost Summary
  10. Implementation Priority
  11. Integration Diagram

1. Executive Summary

Stack Philosophy

Drop requires a DevOps/SRE stack that balances:

Area MVP Tool Scale Tool Reason
CI/CD GitHub Actions GitHub Actions + ArgoCD Native GitHub, EU runners available
E2E Testing Playwright Playwright Open-source, excellent mobile web
Load Testing k6 k6 + Grafana Cloud Grafana ecosystem, scriptable
APM Grafana Cloud Grafana Cloud EU-hosted, cost-effective
Logs Grafana Loki Grafana Loki Part of Grafana stack
Errors Sentry Sentry Best-in-class, EU hosting
Alerts Slack + PagerDuty PagerDuty Start simple, scale
Secrets AWS Secrets Manager AWS Secrets Manager Native AWS, compliant
Security Scan Snyk Snyk + DAST Developer-friendly

Total MVP Monthly Cost: EUR 800-1,200/month

Total Scale Monthly Cost: EUR 2,500-4,000/month

2. CI/CD Pipeline

2.1 Recommendation: GitHub Actions

Why GitHub Actions over alternatives:

Criteria GitHub Actions GitLab CI CircleCI
Native Integration Best (GitHub) Requires migration Good
EU Runners Yes (Azure EU) Yes Limited
Free Tier 2,000 min/month 400 min/month 6,000 min/month
Secrets Management Native Native Native
Self-hosted Runners Yes Yes Limited
Marketplace Largest Growing Medium
Learning Curve Low Medium Medium
OIDC for AWS Native Requires setup Requires setup

Decision: GitHub Actions

2.2 Pipeline Architecture

# .github/workflows/main.yml structure

Triggers:
  - push to main/develop
  - pull request
  - manual dispatch

Jobs:
  1. lint-and-format
     - ESLint, Prettier
     - Parallel for speed

  2. security-scan
     - Snyk dependency check
     - Secret scanning
     - SAST (CodeQL)

  3. test-unit
     - Jest (backend/frontend)
     - Coverage threshold: 80%

  4. test-integration
     - Database tests
     - API contract tests

  5. build
     - Docker image build
     - Multi-arch (amd64/arm64)

  6. test-e2e (staging only)
     - Playwright
     - Against staging environment

  7. deploy-staging
     - Automatic on develop merge

  8. deploy-production
     - Manual approval required
     - Canary deployment

2.3 Deployment Strategies

MVP Phase: Rolling Deployment

Scale Phase: Canary Deployment

Production Traffic:
  ├── 95% → Current Version
  └── 5%  → New Version (canary)

Promotion: Manual after metrics validation
Rollback: Automatic on error rate spike

Implementation: ArgoCD + Argo Rollouts

2.4 Branch Strategy

main (production)
  ↑
  └── develop (staging)
        ↑
        └── feature/* (development)
        └── hotfix/* (emergency fixes)

Rules:

2.5 GitHub Actions Cost Estimate

Phase Minutes/Month Cost
MVP (5 devs) ~3,000 Free (2,000) + EUR 20
Scale (15 devs) ~15,000 EUR 120/month

3. Testing Strategy

3.1 Testing Pyramid

          ┌─────────┐
          │   E2E   │  ~10% of tests
          │ (Slow)  │  Critical user journeys
          └────┬────┘
               │
        ┌──────┴──────┐
        │ Integration │  ~20% of tests
        │  (Medium)   │  API contracts, DB
        └──────┬──────┘
               │
     ┌─────────┴─────────┐
     │       Unit        │  ~70% of tests
     │      (Fast)       │  Business logic
     └───────────────────┘

3.2 Unit Testing

Current Stack: Jest (already configured)

Coverage Requirements:

Component Minimum Target
Business Logic 90% 95%
API Controllers 80% 90%
Utilities 70% 80%
UI Components 60% 70%

Best Practices:

3.3 Integration Testing

Tools:

What to Test:

3.4 E2E Testing

Recommendation: Playwright

Criteria Playwright Cypress
Browser Support All major + mobile Chrome, Firefox, Edge
Speed Faster (parallel) Slower
Auto-wait Built-in Built-in
Mobile Testing Better (device emulation) Limited
CI Integration Excellent Good
Cost Free Free (cloud paid)
Learning Curve Medium Lower

Decision: Playwright

Critical User Journeys to Test:

  1. User registration + KYC start
  2. Login flow (email + biometric)
  3. View balance and transactions
  4. Send P2P transfer
  5. Card top-up flow
  6. Card freeze/unfreeze
  7. SEPA transfer initiation

Playwright Configuration:

// playwright.config.ts
{
  projects: [
    { name: 'Desktop Chrome', use: { ...devices['Desktop Chrome'] } },
    { name: 'Mobile Safari', use: { ...devices['iPhone 14'] } },
    { name: 'Mobile Chrome', use: { ...devices['Pixel 7'] } },
  ],
  retries: 2,
  reporter: [['html'], ['junit', { outputFile: 'results.xml' }]],
}

3.5 Load Testing

Recommendation: k6

Why k6:

Load Test Scenarios:

Scenario Virtual Users Duration Success Criteria
Baseline 50 5 min p95 < 500ms
Peak 200 10 min p95 < 1000ms
Stress 500 5 min No crashes
Soak 100 1 hour No memory leaks

Critical Endpoints:

3.6 Security Testing

SAST (Static Analysis):

DAST (Dynamic Analysis):

Dependency Scanning:

Schedule:

Test Type Frequency Blocker?
SAST Every PR Yes (high severity)
Dependency Scan Daily Yes (critical)
DAST Weekly No (review)
Pen Test Quarterly N/A (manual)

4. Monitoring & Observability

4.1 Strategy: Unified Grafana Stack

Why Grafana Cloud over alternatives:

Criteria Grafana Cloud Datadog New Relic
EU Hosting Yes (Frankfurt) Yes Yes
Pricing Model Usage-based Per-host Per-user
MVP Cost EUR 0-200 EUR 400+ EUR 300+
Scale Cost EUR 500-1,000 EUR 2,000+ EUR 1,500+
Open Standards Full (Prometheus, OTel) Partial Partial
Vendor Lock-in Low High High
Self-host Option Yes (fallback) No No

Decision: Grafana Cloud

4.2 Metrics (Prometheus + Grafana)

Infrastructure Metrics:

Application Metrics:

Business Metrics (Custom):

fontelepay_transactions_total{type="p2p|sepa|card"}
fontelepay_transaction_value_eur{type="p2p|sepa|card"}
fontelepay_users_registered_total
fontelepay_users_kyc_passed_total
fontelepay_cards_issued_total{type="virtual|physical"}
fontelepay_api_latency_seconds{endpoint="/api/..."}

4.3 Log Aggregation (Loki)

Why Loki:

Log Structure (JSON):

{
  "timestamp": "2026-02-05T10:30:00Z",
  "level": "info",
  "service": "payment-service",
  "trace_id": "abc123",
  "user_id": "usr_xxx",  // pseudonymized
  "message": "Transfer initiated",
  "amount_eur": 100,
  "transfer_type": "sepa"
}

Retention Policy:

Log Type Retention Reason
Application 30 days Debugging
Security/Audit 7 years Compliance
Access Logs 90 days Security review

GDPR Considerations:

4.4 Distributed Tracing (Tempo)

Implementation: OpenTelemetry

Why OpenTelemetry:

Trace Critical Paths:

  1. User login (app -> API -> auth -> DB)
  2. Payment initiation (app -> API -> payment -> BaaS -> ledger)
  3. Card transaction (webhook -> processor -> notification)

Sampling Strategy:

4.5 Real User Monitoring (RUM)

For Web (Next.js):

For Mobile (Flutter):

Key Metrics:

Metric Target Threshold
LCP (Largest Contentful Paint) <2.5s <4s
FID (First Input Delay) <100ms <300ms
CLS (Cumulative Layout Shift) <0.1 <0.25
App Cold Start <2s <3s
API Response (p95) <500ms <1s

4.6 Grafana Cloud Cost Estimate

Component MVP Usage MVP Cost Scale Usage Scale Cost
Metrics 10K series Free 50K series EUR 150
Logs 50 GB/mo Free 200 GB/mo EUR 200
Traces 10 GB/mo Free 50 GB/mo EUR 100
Total - EUR 0-50 - EUR 450

5. Error Tracking

5.1 Recommendation: Sentry

Comparison:

Criteria Sentry Bugsnag Rollbar
EU Hosting Yes Yes No
Flutter SDK Excellent Good Limited
Source Maps Automatic Automatic Manual
Performance Included Separate Included
Pricing (MVP) Free EUR 100 EUR 100
Pricing (Scale) EUR 300 EUR 400 EUR 350
Slack Integration Native Native Native
Issue Grouping Best Good Good

Decision: Sentry

5.2 Sentry Configuration

Projects:

Settings:

// sentry.config.js
{
  dsn: "https://xxx@sentry.io/xxx",
  environment: process.env.NODE_ENV,
  release: process.env.GIT_SHA,
  tracesSampleRate: 0.1,  // 10% of transactions

  // Filter sensitive data
  beforeSend(event) {
    // Remove PII
    if (event.user) {
      delete event.user.email;
      delete event.user.ip_address;
    }
    return event;
  }
}

Alert Rules:

Condition Action Priority
New issue (high severity) Slack + PagerDuty P1
Issue spike (>10x baseline) Slack + PagerDuty P1
New issue (medium) Slack only P2
Regression (resolved reopened) Slack P2

5.3 Source Maps

Web (Next.js):

Mobile (Flutter):

5.4 Sentry Cost Estimate

Phase Events/Month Cost
MVP <5,000 Free
Growth ~50,000 EUR 26/month
Scale ~500,000 EUR 300/month

6. Alerting & Incident Management

6.1 Phased Approach

MVP (Team <5): Slack + Grafana Alerts

Growth (Team 5-15): Add PagerDuty

Scale (Team 15+): Full Incident Management

6.2 Alert Levels

Level Response Time Examples Notification
P1 - Critical 15 min Payment processing down, data breach PagerDuty + Slack + SMS
P2 - High 1 hour High error rate, degraded performance PagerDuty + Slack
P3 - Medium 4 hours Non-critical service degraded Slack only
P4 - Low Next business day Warning thresholds Slack (daily digest)

6.3 Critical Alerts (P1)

Alert Condition Action
API Down 0 successful requests for 2 min Page on-call
Payment Failures >5% failure rate for 5 min Page on-call
Database Unreachable Connection failures >10/min Page on-call
Security Event Suspicious activity detected Page on-call + security
Error Spike 10x baseline errors Page on-call

6.4 On-Call Rotation

MVP Setup:

Week 1: Dev A (primary)
Week 2: Dev B (primary)
Week 3: Dev A (primary)
...

Escalation:
  0-15 min: Primary on-call
  15-30 min: Secondary on-call
  30+ min: Engineering lead

PagerDuty Cost:

Plan Cost Features
Free EUR 0 5 users, basic
Professional EUR 21/user/mo Full features

MVP: Free tier (5 users) Scale: Professional for core team

6.5 Incident Response Runbook Template

## Incident: [Title]

### Detection
- Alert source: [Grafana/Sentry/PagerDuty]
- Time detected: [timestamp]
- Severity: [P1/P2/P3]

### Impact
- Users affected: [estimate]
- Services affected: [list]
- Financial impact: [if applicable]

### Timeline
- HH:MM - [Event]
- HH:MM - [Event]

### Root Cause
[Description]

### Resolution
[Steps taken]

### Action Items
- [ ] [Preventive measure]
- [ ] [Process improvement]

### Participants
- Incident Commander: [name]
- Responders: [names]

7. Documentation

7.1 API Documentation

Recommendation: OpenAPI 3.1 + Swagger UI

Why:

Implementation:

# openapi.yaml (partial)
openapi: 3.1.0
info:
  title: Drop API
  version: 1.0.0
  description: Mobile banking API

servers:
  - url: https://api.fontelepay.com/v1
    description: Production
  - url: https://api.staging.fontelepay.com/v1
    description: Staging

security:
  - bearerAuth: []

paths:
  /accounts/{id}/balance:
    get:
      summary: Get account balance
      tags: [Accounts]
      ...

Hosting:

7.2 Runbooks

Location: /docs/runbooks/ in repository

Required Runbooks:

Runbook Purpose
deploy-production.md Production deployment steps
rollback.md How to rollback a bad deploy
database-migration.md Safe DB migration process
incident-response.md General incident handling
scaling.md How to scale services
secrets-rotation.md Rotating API keys, certs
disaster-recovery.md Full recovery procedures

Runbook Template:

# Runbook: [Title]

## Overview
[What this runbook covers]

## Prerequisites
- [ ] Access to [system]
- [ ] Permissions: [list]

## Steps
1. [Step with command examples]
2. [Step with verification]

## Verification
[How to confirm success]

## Rollback
[If something goes wrong]

## Contacts
- Primary: [name/slack]
- Escalation: [name/slack]

7.3 Architecture Decision Records (ADRs)

Location: /docs/adr/ in repository

Format:

# ADR-001: Use PostgreSQL as Primary Database

## Status
Accepted

## Context
We need a reliable, ACID-compliant database for financial transactions.

## Decision
Use PostgreSQL 16 as our primary database.

## Consequences
### Positive
- Strong ACID compliance
- Excellent JSON support
- Proven in fintech

### Negative
- Requires more ops than managed NoSQL
- Horizontal scaling more complex

## Alternatives Considered
- MySQL: Less JSON support
- MongoDB: Not ACID by default
- CockroachDB: Higher cost, complexity

Key ADRs to Create:

7.4 Documentation Tooling

Type Tool Cost
API Docs Swagger/OpenAPI Free
Internal Docs Notion or Confluence Free-EUR 50/mo
Runbooks Git repository Free
Diagrams Mermaid (in Markdown) Free
Postmortems Notion template Free

8. Security Operations

8.1 Dependency Scanning

Recommendation: Snyk

Why Snyk:

Integration:

# .github/workflows/security.yml
- name: Snyk Security Scan
  uses: snyk/actions/node@master
  with:
    args: --severity-threshold=high

Policy:

Severity Action SLA
Critical Block PR, fix immediately 24 hours
High Block PR, fix before merge 72 hours
Medium Warning, fix in sprint 2 weeks
Low Track, fix when convenient 1 month

Snyk Cost:

Plan Cost Limits
Free EUR 0 200 tests/month
Team EUR 52/dev/mo Unlimited

MVP: Free tier Scale: Team plan

8.2 Secret Management

Recommendation: AWS Secrets Manager

Why AWS Secrets Manager:

Alternative: HashiCorp Vault

Secrets to Manage:

Secret Rotation Access
Database credentials 90 days Backend services
API keys (Swan, Stripe) 180 days Backend services
JWT signing keys 365 days Auth service
Encryption keys Never (versioned) All services

Implementation:

// secrets.ts
import { SecretsManager } from '@aws-sdk/client-secrets-manager';

const client = new SecretsManager({ region: 'eu-central-1' });

export async function getSecret(name: string): Promise<string> {
  const response = await client.getSecretValue({ SecretId: name });
  return response.SecretString!;
}

AWS Secrets Manager Cost:

Secrets Cost
10 secrets EUR 4/month
50 secrets EUR 20/month
100 secrets EUR 40/month

8.3 Penetration Testing

Schedule:

Test Type Frequency Provider
Automated DAST Weekly OWASP ZAP
Web App Pen Test Quarterly External firm
Mobile App Pen Test Quarterly External firm
Infrastructure Pen Test Annually External firm

Budget:

Test Cost
Web + API Pen Test EUR 5,000-10,000
Mobile Pen Test EUR 5,000-8,000
Infrastructure EUR 8,000-15,000
Annual Total EUR 25,000-45,000

EU-Based Pen Testing Firms:

8.4 Security Monitoring

SIEM Considerations:

Security Alerts:

Event Action
Failed login spike Alert + temp block
New device login User notification
Large transfer Manual review queue
Admin action Audit log + alert
API key usage anomaly Alert + investigate

8.5 Compliance Automation

Tools:

Automated Checks:

9. Cost Summary

9.1 MVP Phase (Monthly)

Category Tool Cost (EUR)
CI/CD GitHub Actions 20-50
Monitoring Grafana Cloud (free tier) 0-50
Error Tracking Sentry (free tier) 0
Alerting Slack + PagerDuty Free 0
Security Snyk (free tier) 0
Secrets AWS Secrets Manager 10
Testing Playwright, k6 (OSS) 0
Total EUR 30-110

9.2 Growth Phase (Monthly)

Category Tool Cost (EUR)
CI/CD GitHub Actions 100-150
Monitoring Grafana Cloud 200-400
Error Tracking Sentry Team 100-300
Alerting PagerDuty Professional 100-200
Security Snyk Team 200-400
Secrets AWS Secrets Manager 20-40
Testing k6 Cloud (load testing) 100-200
Total EUR 820-1,690

9.3 Scale Phase (Monthly)

Category Tool Cost (EUR)
CI/CD GitHub Actions + ArgoCD 200-300
Monitoring Grafana Cloud 500-1,000
Error Tracking Sentry Business 300-500
Alerting PagerDuty + Statuspage 300-500
Security Snyk + DAST 500-800
Secrets AWS Secrets Manager 40-60
Testing k6 Cloud 200-400
Documentation Confluence 50-100
Total EUR 2,090-3,660

9.4 Annual Security Costs

Item Cost (EUR)
Penetration Testing (4x/year) 25,000-45,000
Compliance Audit (annual) 10,000-20,000
Security Training 2,000-5,000
Total EUR 37,000-70,000

10. Implementation Priority

10.1 Phase 1: Foundation (Week 1-2)

Must Have:

Outcome: Can deploy safely with visibility into errors

10.2 Phase 2: Observability (Week 3-4)

Must Have:

Outcome: Can monitor application health

10.3 Phase 3: Testing (Week 5-6)

Must Have:

Outcome: Confidence in deployments

10.4 Phase 4: Security (Week 7-8)

Must Have:

Outcome: Security baseline established

10.5 Phase 5: Operations (Week 9-12)

Should Have:

Outcome: Production-ready operations

10.6 Checklist Summary

Week 1-2:  CI/CD + Errors + Secrets
Week 3-4:  Monitoring + Logs + Alerts
Week 5-6:  Tests + E2E + Load
Week 7-8:  Security + Audit + Pen Test
Week 9-12: On-call + Docs + DR

11. Integration Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                              DEVELOPER WORKFLOW                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   ┌─────────┐    ┌─────────┐    ┌─────────────────────────────────────────┐ │
│   │  Code   │───>│  PR     │───>│            GitHub Actions                │ │
│   │ (IDE)   │    │ (GitHub)│    │  ┌─────┐ ┌────┐ ┌────┐ ┌─────┐ ┌─────┐ │ │
│   └─────────┘    └─────────┘    │  │Lint │ │Test│ │SAST│ │Build│ │Snyk │ │ │
│                                 │  └──┬──┘ └──┬─┘ └──┬─┘ └──┬──┘ └──┬──┘ │ │
│                                 └────┼───────┼──────┼──────┼───────┼─────┘ │
│                                      └───────┴──────┴──────┴───────┘       │
│                                                    │                        │
└────────────────────────────────────────────────────┼────────────────────────┘
                                                     │
                                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                              DEPLOYMENT (ArgoCD)                             │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   ┌───────────────┐         ┌───────────────┐         ┌───────────────┐     │
│   │    Staging    │────────>│    Canary     │────────>│   Production  │     │
│   │  (automatic)  │         │  (5% traffic) │         │  (95% -> 100%)│     │
│   └───────────────┘         └───────────────┘         └───────────────┘     │
│          │                         │                         │              │
│          └─────────────────────────┴─────────────────────────┘              │
│                                    │                                        │
└────────────────────────────────────┼────────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         KUBERNETES CLUSTER (AWS EKS)                         │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐       │
│   │  API Gateway│  │   Auth      │  │  Payment    │  │    Card     │       │
│   │   (Kong)    │  │  Service    │  │  Service    │  │   Service   │       │
│   └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘       │
│          │                │                │                │              │
│          └────────────────┴────────────────┴────────────────┘              │
│                                    │                                        │
│          ┌─────────────────────────┼─────────────────────────┐             │
│          │                         │                         │              │
│          ▼                         ▼                         ▼              │
│   ┌─────────────┐           ┌─────────────┐           ┌─────────────┐      │
│   │ PostgreSQL  │           │    Redis    │           │    Kafka    │      │
│   │   (RDS)     │           │(ElastiCache)│           │   (MSK)     │      │
│   └─────────────┘           └─────────────┘           └─────────────┘      │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
                                     │ Telemetry
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           OBSERVABILITY STACK                                │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   ┌─────────────────────────────────────────────────────────────────────┐   │
│   │                        GRAFANA CLOUD (EU)                            │   │
│   │                                                                      │   │
│   │   ┌────────────┐    ┌────────────┐    ┌────────────┐               │   │
│   │   │ Prometheus │    │    Loki    │    │   Tempo    │               │   │
│   │   │  (Metrics) │    │   (Logs)   │    │  (Traces)  │               │   │
│   │   └─────┬──────┘    └─────┬──────┘    └─────┬──────┘               │   │
│   │         └─────────────────┴─────────────────┘                       │   │
│   │                           │                                         │   │
│   │                    ┌──────┴──────┐                                  │   │
│   │                    │  Dashboards │                                  │   │
│   │                    │   & Alerts  │                                  │   │
│   │                    └─────────────┘                                  │   │
│   └─────────────────────────────────────────────────────────────────────┘   │
│                                                                              │
│   ┌────────────────┐                              ┌────────────────┐        │
│   │     Sentry     │                              │   PagerDuty    │        │
│   │ (Error Track)  │                              │   (Alerting)   │        │
│   └───────┬────────┘                              └───────┬────────┘        │
│           │                                               │                 │
│           └───────────────────┬───────────────────────────┘                 │
│                               │                                             │
│                               ▼                                             │
│                        ┌─────────────┐                                      │
│                        │    Slack    │                                      │
│                        │ (Notif Hub) │                                      │
│                        └─────────────┘                                      │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────┐
│                            SECURITY LAYER                                    │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐       │
│   │    Snyk     │  │   CodeQL    │  │  OWASP ZAP  │  │ AWS Secrets │       │
│   │  (Deps)     │  │   (SAST)    │  │   (DAST)    │  │  Manager    │       │
│   └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘       │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Appendix A: Tool Links

Tool URL Purpose
GitHub Actions github.com/features/actions CI/CD
ArgoCD argoproj.github.io/cd GitOps deployment
Grafana Cloud grafana.com/cloud Monitoring
Sentry sentry.io Error tracking
PagerDuty pagerduty.com Incident management
Snyk snyk.io Security scanning
Playwright playwright.dev E2E testing
k6 k6.io Load testing
OpenTelemetry opentelemetry.io Observability

Appendix B: Decision Matrix

Decision Options Considered Winner Key Factor
CI/CD GitHub Actions, GitLab, CircleCI GitHub Actions Native GitHub, EU runners
Monitoring Datadog, New Relic, Grafana Grafana Cloud Cost, EU hosting, open standards
E2E Testing Playwright, Cypress Playwright Mobile web support, speed
Error Tracking Sentry, Bugsnag, Rollbar Sentry Flutter SDK, EU hosting
Alerting PagerDuty, Opsgenie, Slack PagerDuty Industry standard, free tier
Secrets AWS SM, Vault, GCP SM AWS Secrets Manager Already on AWS, simple
Security Snyk, Dependabot, Sonar Snyk Best JS/TS coverage

Appendix C: Compliance Mapping

Requirement Solution Evidence
PCI DSS 10.x (Logging) Grafana Loki, 7yr retention CloudTrail + Loki
GDPR (Data Residency) Grafana EU, Sentry EU Region configs
GDPR (Right to Erasure) Pseudonymized logs No PII in logs
SOC 2 (Change Mgmt) GitHub PRs, ArgoCD Audit trail
ISO 27001 (Incident) PagerDuty, Runbooks Incident records

Document created: 2026-02-05 Last updated: 2026-02-05 Author: DevOps Research

DevOps Stack

WAF Rules

WAF Rules — Drop Payment App

MC #1229 — Web Application Firewall configuration for Drop fintech.

Overview

Drop runs on Fly.io which does not provide a built-in WAF. Protection is layered:

  1. Middleware-level (Next.js Edge Middleware) — first line of defense
  2. Fly.io Proxy — TLS termination, DDoS mitigation at network edge
  3. Application-level — input validation, parameterized SQL, CSRF checks

Middleware WAF Rules (Implemented in src/drop-app/src/middleware.ts)

1. CSRF Origin Validation

2. Rate Limiting

3. Content-Security-Policy

Recommended Reverse Proxy Rules (Fly.io / Cloudflare)

If a CDN or reverse proxy is added in front of Fly.io, configure these rules:

SQL Injection (SQLi)

Cross-Site Scripting (XSS)

Path Traversal

Request Size Limits

Geo-blocking (Optional)

Bot Protection

Implementation Priority

Priority Rule Status
P0 CSRF Origin check Implemented (middleware.ts)
P0 CSP headers Implemented (middleware.ts + next.config.ts)
P0 Rate limiting Implemented (per-endpoint)
P1 Trivy container scan Implemented (CI/CD)
P1 npm audit Implemented (CI/CD)
P2 SQLi WAF rules Pending — requires CDN/proxy
P2 XSS WAF rules Pending — requires CDN/proxy
P2 Path traversal rules Pending — requires CDN/proxy
P3 Geo-blocking Pending — requires CDN/proxy
P3 Bot protection (CAPTCHA) Pending — requires frontend integration

Testing WAF Rules

When WAF rules are deployed via CDN:

# Test SQLi blocking
curl -X POST "https://getdrop.no/api/test" -d "id=1 OR 1=1"
# Expected: 403 Forbidden

# Test XSS blocking
curl -X POST "https://getdrop.no/api/test" -d "name=<script>alert(1)</script>"
# Expected: 403 Forbidden

# Test path traversal blocking
curl "https://getdrop.no/../../etc/passwd"
# Expected: 403 Forbidden

Monitoring

DevOps Stack

Cloud Deployment Options

Cloud Deployment Options for Drop

Rebrand note (2026-02-14): Originally titled "FontelePay". Product rebranded to Drop. See Drop CLAUDE.md.

Date: 2026-02-05 Purpose: Evaluate cloud deployment options for European mobile banking MVP

Requirements Summary

Requirement Priority
Next.js support (static + SSR/API routes) Must-have
EU data residency (GDPR) Must-have
Financial compliance ready (PCI-DSS, SOC2) Must-have
Cost-effective for MVP High
Easy CI/CD integration High
Scalability for production Medium

Provider Comparison

Overview Table

Feature Vercel AWS (Amplify/Lambda) Google Cloud Run
Next.js Support Native (created by Vercel) Full SSR support (v15) Via container deployment
EU Regions Edge caching only Frankfurt, Ireland, Paris, Stockholm + ESC Frankfurt, Belgium, Netherlands, Zurich
Data Residency US-based storage* Full EU residency available Full EU residency available
PCI-DSS v4.0 (SAQ-D AOC) v4.0.1 certified v4.0.1 certified
SOC 2 Type 2 certified Type 2 certified Type 2 certified
ISO 27001 Certified Certified Certified
GDPR EU-US DPF certified Compliant Compliant
Ease of Use Excellent Moderate Moderate
Vendor Lock-in Medium Low Low

*Vercel: Static assets and function responses cached in EU, but primary storage remains US-based.

Detailed Analysis

1. Vercel

Strengths:

Weaknesses:

Pricing:

Tier Cost Includes
Hobby Free 100GB bandwidth, limited features
Pro $20/user/month 1TB bandwidth, $20 credits, viewer seats free
Enterprise Custom SAML SSO, SLAs, dedicated support

GDPR Concern: Vercel is certified under EU-US Data Privacy Framework, but for banking applications requiring strict EU data residency, this may not be sufficient. Functions can run in EU regions, but metadata and logs may still traverse US infrastructure.

2. AWS (Amplify + Lambda)

Strengths:

Weaknesses:

Pricing (AWS Amplify):

Resource Free Tier Paid
Build minutes 1,000/month $0.01/min
Data served 15 GB/month $0.15/GB
Data stored 5 GB/month $0.023/GB
SSR requests Varies ~$0.20/1M

Estimated MVP Cost: $5-25/month for low-moderate traffic

European Sovereign Cloud (ESC): Launched January 2026, provides EU-resident personnel and hardware-enforced access restrictions. Ideal for regulated financial services.

3. Google Cloud Run

Strengths:

Weaknesses:

Pricing (Tier 1 - EU regions):

Resource Free Tier Paid
CPU 180,000 vCPU-seconds/month $0.000024/vCPU-second
Memory 360,000 GiB-seconds/month $0.0000025/GiB-second
Requests 2 million/month $0.40/million

Estimated MVP Cost: $0-15/month for low-moderate traffic (often within free tier)

Compliance Matrix for Fintech

Certification Vercel AWS GCP Required for Drop
PCI-DSS v4.0+ Yes Yes Yes Yes (payment processing)
SOC 2 Type 2 Yes Yes Yes Yes (enterprise clients)
ISO 27001 Yes Yes Yes Recommended
GDPR DPF Full Full Yes (EU operations)
EU Data Residency Partial Full Full Critical

Recommendation

MVP Phase (0-6 months)

Primary: AWS Amplify (Frankfurt region)

Rationale:

  1. True EU data residency - critical for banking MVP regulatory approval
  2. Full Next.js support - SSR, API routes, ISR all work
  3. Cost-effective - likely $10-30/month for MVP traffic
  4. Compliance-ready - PCI-DSS, SOC 2, ISO 27001 from day one
  5. No per-seat pricing - scales with team growth
  6. Path to production - same platform, just scale up

Setup recommendation:

Production Phase (6+ months)

Stay with AWS but consider:

Why Not Vercel?

Despite excellent DX, Vercel's partial EU data residency is a significant concern for a banking application. While Vercel is PCI-DSS compliant, regulators may question data flows through US infrastructure. For an MVP seeking banking licenses or partnerships, demonstrating full EU data residency is simpler with AWS or GCP.

Why Not GCP Cloud Run?

GCP is technically excellent but:

Cost Projection (12 months)

Scenario Vercel Pro AWS Amplify GCP Cloud Run
MVP (2 devs, 10k users) $480/year $120-300/year $0-180/year
Growth (5 devs, 50k users) $1,200/year $300-600/year $200-400/year
Scale (10 devs, 200k users) $2,400/year $600-1,500/year $500-1,200/year

AWS and GCP costs vary based on usage patterns; Vercel costs fixed per-seat

Action Items

  1. Set up AWS account with Frankfurt region default
  2. Configure Amplify for Next.js deployment
  3. Implement GitHub Actions CI/CD pipeline
  4. Document compliance controls for future audits
  5. Evaluate AWS ESC when banking license process begins

Sources

DevOps Stack

Infrastructure Overview

Infrastructure Resources

Infrastructure resources for Drop project: deployment, monitoring, CI/CD.

Cloud Migration Strategy — GCP → Azure

Cloud Migration Strategy — Drop

Dato: 2026-02-18 Status: Planlegging Beslutning: Azure som produksjonsplattform, GCP for dev/staging

SpareBank 1 — Teknisk Stack (Research)

Lag Teknologi
Cloud Azure (primær) — Eunomia-plattformen for 13 banker
Sekundær AWS (mindre workloads)
Backend Kotlin/Java (Spring Boot)
Frontend React + TypeScript
Orkestrering Kubernetes / OpenShift
Meldingskø Apache Kafka
Autentisering BankID (norsk eID)
API Gateway Axway
Partnerskap Microsoft (strategisk partner)

Drops Nåværende Stack

Lag Teknologi
Frontend Next.js 16 + React 19 + Tailwind v4
Backend Next.js API Routes (Node.js)
Database SQLite (better-sqlite3) → PostgreSQL (prod)
Auth JWT (jose) i httpOnly cookies + BankID
Hosting Fly.io (staging), Vercel (planned prod)

Migreringsstrategi

Fase 1: GCP Dev/Staging (NÅ)

Fase 2: Azure Produksjon (Når credits kommer)

Fase 3: Multi-Cloud Beredskap

GCP Deploy Plan (Fase 1)

Steg 1: Containerisering

# Dockerfile for Drop Next.js
FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --production
COPY . .
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]

Steg 2: GCP Oppsett

  1. Opprett GCP-prosjekt (allerede: project-72cd303f)
  2. Aktiver Cloud Run API
  3. Opprett Cloud SQL PostgreSQL-instans (db-f1-micro for dev)
  4. Konfigurer secrets i Secret Manager
  5. Sett opp Cloud Build for CI/CD

Steg 3: Deploy Pipeline

# Build og push container
gcloud builds submit --tag gcr.io/PROJECT_ID/drop-web

# Deploy til Cloud Run
gcloud run deploy drop-web \
  --image gcr.io/PROJECT_ID/drop-web \
  --platform managed \
  --region europe-north1 \
  --allow-unauthenticated \
  --set-env-vars DATABASE_URL=postgresql://...

Steg 4: DNS + SSL

Regulatoriske Krav (Finanstilsynet)

Kotlin/Java Vurdering

Beslutning: Nei — beholder Next.js/TypeScript

Hvorfor:

  1. Drop er ~7,600 linjer TypeScript — full rewrite til Kotlin = 2-3 mnd
  2. SpareBank 1 sin API er REST/GraphQL — språket på vår side er irrelevant
  3. TypeScript fullstack = ett språk, ett team, raskere iterasjon
  4. Next.js API Routes er tilstrekkelig for vår skala (fintech MVP)
  5. Når vi trenger mikrotjenester → da vurderer vi Kotlin for spesifikke tjenester

Estimerte Kostnader

Tjeneste GCP (dev) Azure (prod)
Compute Cloud Run: ~$0 (free tier) App Service B1: ~$13/mnd
Database Cloud SQL micro: ~$7/mnd PostgreSQL Basic: ~$25/mnd
Storage 5GB: ~$0.10/mnd 5GB: ~$0.10/mnd
Totalt ~$7/mnd (dekkes av credits) ~$38/mnd (dekkes av credits)

Relaterte Oppgaver

Load Test Results — 2026-02-18

Load Test Results — Drop Staging

Dato: 2026-02-18 Verktøy: k6 v1.6.1 Mål: https://drop-staging.fly.dev Server: Fly.io shared-cpu-1x (256MB RAM, 1 delt CPU, Stockholm)

Testoppsett

To scenarier kjørt samtidig:

Scenario 1: Public Stress (helse + valutakurser)

Scenario 2: Autentisert brukerflyt

Resultater

Samtidige brukere Median latens p95 latens Feilrate Status
1-10 74ms ~90ms 0% Fungerer utmerket
25-50 ~500ms ~3s ~5% Degradering starter
75-100 ~2-3s ~6s ~30% Alvorlige problemer
150-200 3s+ 27s+ 47% Praktisk talt nede

Detaljerte tall (k6 output)

Public endpoints:

Autentiserte endpoints:

Totalt:

Breaking Point

~25-30 samtidige brukere

Etter dette eksploderer responstidene og endepunkter begynner å feile.

Flaskehalser identifisert

1. Maskinressurser (KRITISK)

2. SQLite single-writer (HØY)

3. Null caching (MEDIUM)

4. bcrypt 12 rounds (MEDIUM)

5. Enkeltinstans (MEDIUM)

Oppgraderingsplan

Oppgradering Effekt Kostnad
Fly.io performance-1x (2GB RAM, 1 dedikert CPU) ~3x kapasitet (~75 brukere) ~$30/mnd
+ PostgreSQL i stedet for SQLite Samtidige skrivinger, connection pooling ~$15/mnd
+ Redis cache (kurser, sesjoner) 10x raskere på lese-endepunkter ~$10/mnd
+ 2 instanser (auto-scale) ~150+ brukere ~$60/mnd totalt
Full produksjonsoppsett ~500+ brukere ~$100/mnd

Konklusjon

For MVP/demo med SpareBank 1: Nåværende oppsett holder 10-15 samtidige brukere — tilstrekkelig for demo. For pilot med ekte brukere trengs minimum PostgreSQL + større maskin.

Se også: Cloud Migration Strategy — GCP → Azure for migreringsplan.

GCP Architecture — Cloud Run + Cloud SQL

GCP Architecture for Drop

Dato: 2026-02-18 Region: europe-north1 (Finland — nærmest Norge) Kontekst: Migrering fra Fly.io shared-cpu-1x som takler ~25 samtidige brukere

Nåværende Fly.io vs GCP — Sammenligning

Tier 1: Minimum (Dev/Demo) — ~25 brukere

Komponent Fly.io (nå) GCP ekvivalent GCP kostnad
Compute shared-cpu-1x (256MB) Cloud Run: 1 vCPU, 512MB ~$0 (free tier)
Database SQLite på Fly Volume Cloud SQL db-f1-micro (0.6GB, 10GB) ~$9/mnd
Cache Ingen Ingen $0
Totalt ~$5/mnd ~$9/mnd
Kapasitet ~25 samtidige ~25 samtidige

Tier 2: Pilot (SpareBank 1 demo) — ~100 brukere

Komponent GCP tjeneste Spesifikasjon Kostnad
Compute Cloud Run 2 vCPU, 1GB RAM, min 1 instans ~$15/mnd
Database Cloud SQL db-g1-small (1.7GB, 20GB SSD) ~$30/mnd
Cache Memorystore Redis Basic 1GB ~$35/mnd
CDN Cloud CDN 10GB egress ~$1/mnd
Secrets Secret Manager 10 secrets ~$0 (free tier)
Monitoring Cloud Monitoring Basic ~$0 (free tier)
Totalt ~$81/mnd
Kapasitet ~100-150 samtidige

Tier 3: Produksjon (Ekte brukere) — ~500+ brukere

Komponent GCP tjeneste Spesifikasjon Kostnad
Compute Cloud Run 2 vCPU, 2GB RAM, min 2 instanser, auto-scale til 10 ~$50/mnd
Database Cloud SQL Enterprise: 2 vCPU, 8GB RAM, 50GB SSD, HA ~$150/mnd
Cache Memorystore Redis Standard 1GB (HA) ~$70/mnd
CDN Cloud CDN + Load Balancer Global ~$25/mnd
Secrets Secret Manager ~$0
Monitoring Cloud Monitoring + Logging ~$10/mnd
Backup Automated DB backup ~$5/mnd
Totalt ~$310/mnd
Kapasitet ~500-1000 samtidige

Cloud Run Pricing (Tier 1 region)

Ressurs Pris Gratis tier
CPU $0.000024/vCPU-sekund 180,000 vCPU-sek/mnd
Minne $0.0000025/GiB-sekund 360,000 GiB-sek/mnd
Forespørsler $0.40/million 2 millioner/mnd
Egress $0.12/GB (etter 1GB gratis) 1 GB/mnd

Gratis tier dekker: ~50 timer med 1 vCPU + 256MB — nok for dev/staging med lav trafikk.

Viktig: Gratis tier gjelder kun us-central1/us-east1/us-west1. I europe-north1 faktureres ALT fra første bruk — men dekkes av $300 free trial credits.

Cloud SQL PostgreSQL Pricing

Instanstype vCPU RAM Pris (ca.)
db-f1-micro Delt 0.6 GB ~$9/mnd
db-g1-small Delt 1.7 GB ~$27/mnd
db-custom-2-8192 2 8 GB ~$130/mnd

Lagring: ~$0.17/GB/mnd (SSD) Backup: ~$0.08/GB/mnd

Deploy-arkitektur på GCP

┌─────────────────────────────────────────────┐
│                 Cloud CDN                    │
│          (statiske filer + caching)          │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────▼──────────────────────────┐
│            Cloud Run Service                 │
│     drop-web (Next.js standalone)            │
│     Region: europe-north1                    │
│     Auto-scale: 0-10 instanser               │
│     CPU: 1-2 vCPU, RAM: 512MB-2GB            │
└──────┬──────────────────┬───────────────────┘
       │                  │
┌──────▼──────┐    ┌──────▼──────┐
│  Cloud SQL  │    │ Memorystore │
│ PostgreSQL  │    │    Redis    │
│ europe-n1   │    │  (cache)    │
│ Private IP  │    │  Basic 1GB  │
└─────────────┘    └─────────────┘

Alt: Serverless VPC Connector for private nettverk

Migreringssteg

Steg 1: Containerisering (dag 1)

Steg 2: GCP-oppsett (dag 1-2)

# Opprett prosjekt (allerede: project-72cd303f)
gcloud config set project project-72cd303f-66e5-46ee-a4c

# Aktiver APIer
gcloud services enable run.googleapis.com
gcloud services enable sqladmin.googleapis.com
gcloud services enable secretmanager.googleapis.com
gcloud services enable cloudbuild.googleapis.com

# Cloud SQL instans
gcloud sql instances create drop-db \
  --database-version=POSTGRES_15 \
  --tier=db-f1-micro \
  --region=europe-north1 \
  --storage-size=10GB \
  --storage-type=SSD

# Opprett database
gcloud sql databases create drop --instance=drop-db

# Opprett bruker
gcloud sql users create drop-user \
  --instance=drop-db \
  --password=<generert>

Steg 3: Deploy til Cloud Run (dag 2)

# Build og push container
gcloud builds submit --tag gcr.io/PROJECT_ID/drop-web

# Deploy
gcloud run deploy drop-web \
  --image gcr.io/PROJECT_ID/drop-web \
  --platform managed \
  --region europe-north1 \
  --allow-unauthenticated \
  --memory 512Mi \
  --cpu 1 \
  --min-instances 0 \
  --max-instances 5 \
  --set-env-vars NODE_ENV=production \
  --set-cloudsql-instances PROJECT_ID:europe-north1:drop-db \
  --set-secrets DATABASE_URL=drop-db-url:latest

Steg 4: DNS + SSL (dag 2)

# Custom domain
gcloud run domain-mappings create \
  --service drop-web \
  --domain drop-dev.alai.no \
  --region europe-north1

Steg 5: CI/CD (dag 3)

Kostnadsdekning

Kilde Beløp Dekker
GCP Free Trial $300 (kr 2,884) Tier 1+2 i ~3 mnd
GCP Startups Program (søkt) Inntil $100,000 Alt i 1-2 år
Microsoft Founders Hub (søkt) Inntil $150,000 Azure Azure-migrering senere

Med free trial alene: $300 / ~$9 per mnd (Tier 1) = 33 måneder for dev. Med Tier 2 ($81/mnd) = ~3.7 måneder.

Alternativt: Billigere cache enn Memorystore

Memorystore Redis (1GB basic = ~$35/mnd) er dyrt for MVP. Alternativer:

Alternativ Pris Fordel
Upstash Redis (serverless) Gratis opptil 10K kommandoer/dag Null kostnad for dev
In-memory cache i Cloud Run $0 Forsvinner ved restart
Cloud Run + node-cache $0 Enkel, per-instans cache

Anbefaling: Start uten Redis. Legg til Upstash eller in-memory cache først. Memorystore kun hvis vi trenger delt cache mellom instanser.

Kapasitetsestimat per tier

Tier Samtidige brukere Responstid (p95) Kostnad
Tier 1 (db-f1-micro, 1 vCPU) ~25-30 <200ms ~$9/mnd
Tier 2 (db-g1-small, 2 instanser) ~100-150 <500ms ~$80/mnd
Tier 3 (2 vCPU DB, auto-scale) ~500-1000 <300ms ~$310/mnd

PostgreSQL alene gir ~2-3x bedre concurrent performance enn SQLite pga. connection pooling og parallelle skrivinger.

AWS Deploy — App Runner + RDS (Live)

AWS Deploy — Drop Staging

Dato: 2026-02-18 Status: LIVE Region: eu-west-1 (Ireland)

Infrastruktur

Komponent Tjeneste Detaljer
Compute App Runner 1 vCPU, 2GB RAM, auto-scale
Container ECR 324480209768.dkr.ecr.eu-west-1.amazonaws.com/drop-web
Database RDS PostgreSQL 16.6 db.t3.micro (Free Tier), 20GB gp3
Region eu-west-1 Ireland

URLer

Tjeneste URL
App https://9ef3szvvsb.eu-west-1.awsapprunner.com
RDS drop-db.czu2qe4quy4v.eu-west-1.rds.amazonaws.com:5432

Credentials

Nøkkel Verdi
AWS Account 324480209768
IAM User john-deploy (AdministratorAccess)
RDS User dropuser
RDS Database dropapp
JWT Secret drop-aws-jwt-secret-2026-xK9mP2vL

NB: Passord i Vaultwarden, ikke i BookStack.

Load Test — Sammenligning

Metrikk Fly.io (256MB) AWS (2GB) Forbedring
Throughput 74 req/s 186 req/s 2.5x
Health p95 6,216ms 614ms 10x raskere
Kapasitet ~25 brukere ~75-100 brukere 3-4x

Neste steg

  1. Koble App Runner til RDS PostgreSQL (DATABASE_URL) — trenger VPC Connector
  2. Sett opp custom domene (drop-staging.alai.no)
  3. CI/CD via GitHub Actions → ECR → App Runner
  4. Load test med PostgreSQL (forventet ytterligere forbedring)

Kostnad

Tjeneste Estimert
App Runner (1 vCPU, 2GB) ~$7/mnd (idle)
RDS db.t3.micro $0 (Free Tier 12 mnd)
ECR ~$1/mnd
Totalt ~$8/mnd

Dekkes av AWS Activate credits ($1,000 søkt).

Cloud Audit

Cloud infrastructure audit and multi-cloud design

Cloud Audit

Cloud Audit: Resource Inventory

Drop — AWS Resource Inventory

Date: 2026-02-19 Region: eu-west-1 (Ireland) Account: Drop production Auditor: infra-lead (CloudForge cloud-audit team) MC Task: #1443

Executive Summary

Drop runs a minimal AWS footprint: one App Runner service fronting a PostgreSQL RDS instance, with container images stored in ECR. Total estimated cost is $48-60/month.

Three CRITICAL security findings require immediate action:

  1. RDS database is publicly accessible with security group open to the entire internet (0.0.0.0/0 on port 5432)
  2. Database storage is unencrypted
  3. Plaintext secrets (DATABASE_URL with password, JWT_SECRET) in App Runner environment variables

No WAF, no CloudFront, no CloudWatch monitoring, no Route53 DNS management, and Secrets Manager is provisioned but empty.

Resource Table

Resource Type ID / Name Region Status Key Config
App Runner Service drop-web eu-west-1 RUNNING 1 vCPU, 2 GB RAM, port 3000
RDS PostgreSQL 16.6 drop-db eu-west-1a Available db.t3.micro, 20 GB gp3, single-AZ
ECR Repository drop-web eu-west-1 Active ScanOnPush: TRUE, Encryption: AES256
Security Group SG drop-db-sg eu-west-1 In use Inbound: 0.0.0.0/0 : 5432
VPC Default — eu-west-1 Active 172.31.0.0/16
IAM User User john-deploy Global Active Programmatic access
IAM Role Role AppRunnerECRAccessRole Global Active ECR pull permissions
Secrets Manager — (empty) eu-west-1 Provisioned 0 secrets stored
CloudWatch — — — NOT CONFIGURED No alarms, no dashboards
CloudFront — — — NOT PROVISIONED No CDN
WAF — — — NOT PROVISIONED No web application firewall
Route53 — — — NOT PROVISIONED DNS managed externally
S3 — — — NOT PROVISIONED No buckets

Architecture Diagram

                         INTERNET
                            |
                            | HTTPS (public ingress)
                            v
                   +------------------+
                   |   App Runner     |
                   |   drop-web       |
                   |                  |
                   |  1 vCPU / 2 GB   |
                   |  Port 3000       |
                   |  ECR source      |
                   |                  |
                   |  ENV (plaintext):|
                   |  - DATABASE_URL  |
                   |  - JWT_SECRET    |
                   +--------+---------+
                            |
                            | VPC Connector (egress)
                            |
              +-------------+-------------+
              |       Default VPC         |
              |     172.31.0.0/16         |
              |                           |
              |   +-------------------+   |
              |   |  drop-db-sg       |   |
              |   |  0.0.0.0/0:5432   |   |
              |   +--------+----------+   |
              |            |              |
              |   +--------v----------+   |
              |   |   RDS             |   |
              |   |   drop-db         |   |
              |   |                   |   |
              |   |   PostgreSQL 16.6 |   |
              |   |   db.t3.micro     |   |
              |   |   20 GB gp3       |   |
              |   |   single-AZ (a)   |   |
              |   |                   |   |
              |   |   Public: YES     |   |
              |   |   Encrypted: NO   |   |
              |   |   Backup: 7 days  |   |
              |   |   DeletionProt: ON|   |
              |   |   Monitoring: OFF |   |
              |   +-------------------+   |
              +---------------------------+

        +----------+          +---------------------+
        |   ECR    |          |   Secrets Manager   |
        | drop-web |          |   (EMPTY)           |
        | ScanPush |          +---------------------+
        +----------+

        +----------+          +---------------------+
        |   IAM    |          |   MISSING           |
        | john-    |          |   CloudWatch        |
        |  deploy  |          |   CloudFront        |
        | ECR Role |          |   WAF / Route53 / S3|
        +----------+          +---------------------+

Security Findings

CRITICAL

# Finding Resource Risk Remediation
C1 Database publicly accessible RDS drop-db Direct internet access to PostgreSQL. Any attacker can attempt connections. Set PubliclyAccessible=false. App Runner already uses VPC Connector for egress — RDS only needs private subnet access.
C2 Security group allows 0.0.0.0/0 on port 5432 drop-db-sg Combined with C1, the database is wide open to brute-force and exploitation from any IP on Earth. Restrict inbound rule to App Runner VPC Connector security group only. Remove 0.0.0.0/0 CIDR.
C3 Plaintext secrets in App Runner env vars App Runner drop-web DATABASE_URL contains full connection string with password. JWT_SECRET in plaintext. Anyone with console/API access sees credentials. Visible in CloudTrail, config exports, and deployment logs. Migrate secrets to AWS Secrets Manager (already provisioned, currently empty). Reference via App Runner secret ARN configuration. Rotate both DATABASE_URL password and JWT_SECRET after migration.
C4 Database storage unencrypted RDS drop-db Data at rest is not encrypted. Violates baseline security posture and most compliance frameworks (SOC2, GDPR, PCI). Enable storage encryption. Note: cannot enable on existing instance — requires snapshot, restore to encrypted instance, DNS/connection swap. Plan downtime window.

HIGH

# Finding Resource Risk Remediation
H1 Single-AZ deployment RDS drop-db AZ failure = full database outage. No automatic failover. Enable Multi-AZ for production. Cost increase ~$14/mo for db.t3.micro.
H2 No monitoring or alerting CloudWatch (missing) No CPU, memory, connection, or storage alarms. No visibility into failures, performance degradation, or security events. Silent failures. Configure CloudWatch alarms: CPU > 80%, FreeStorageSpace < 2 GB, DatabaseConnections > 80%, FreeableMemory < 200 MB. Enable Enhanced Monitoring on RDS.
H3 No WAF WAF (missing) No protection against OWASP Top 10 attacks (SQLi, XSS, SSRF, etc.) at the edge. App Runner public endpoint is directly exposed. Deploy AWS WAF with managed rule groups (AWSManagedRulesCommonRuleSet, AWSManagedRulesSQLiRuleSet). Attach to CloudFront distribution (see H4).

MEDIUM

# Finding Resource Risk Remediation
M1 No CDN / CloudFront CloudFront (missing) All traffic hits App Runner origin directly. No edge caching, no DDoS protection (Shield Standard), higher latency for distant users. Deploy CloudFront distribution in front of App Runner. Enables WAF attachment, caching, and Shield Standard.
M2 Default VPC VPC 172.31.0.0/16 Default VPC has broad routing, public subnets by default, and no network segmentation. Not suitable for production workloads. Create custom VPC with private subnets for RDS, public subnets for NAT Gateway / ALB if needed. Migrate RDS to private subnet.
M3 No DNS management Route53 (missing) DNS managed outside AWS. No health checks, no failover routing, no alias records for AWS resources. Consider Route53 for DNS if domain is Drop-owned. Enables health-check-based routing and simpler AWS integration.
M4 TCP health check only App Runner drop-web TCP checks confirm port is open but not that the application is healthy. A process could accept connections while returning 500s. Configure HTTP health check on a dedicated /health endpoint that verifies database connectivity.

LOW

# Finding Resource Risk Remediation
L1 No S3 buckets S3 (missing) If the app needs file storage in future, ensure encryption-at-rest (SSE-S3 or SSE-KMS), versioning, and public access block from day one. Provision with secure defaults when needed.
L2 IAM user john-deploy IAM Long-lived access keys. No indication of key rotation policy or MFA. Audit key age. Enable MFA. Consider OIDC federation for CI/CD instead of IAM user. Rotate keys on a 90-day schedule.

Cost Breakdown

Service Specification Estimated Monthly Cost
App Runner 1 vCPU, 2 GB, always running $29 - $36
RDS db.t3.micro, 20 GB gp3, single-AZ $15 - $18
ECR Image storage (~1-5 GB) $0.50 - $1.00
Data Transfer Minimal (< 10 GB/mo estimate) $1 - $2
Secrets Manager 0 secrets (currently unused) $0
Total $46 - $57/mo

Cost Notes

Gaps Analysis

Category Current State Target State Priority
Secrets management Plaintext env vars Secrets Manager with rotation CRITICAL
Network security Public RDS + open SG Private subnet + restricted SG CRITICAL
Encryption at rest Disabled AES-256 (KMS or default) CRITICAL
Monitoring None CloudWatch alarms + dashboards HIGH
High availability Single-AZ Multi-AZ RDS HIGH
Edge security No WAF / CDN CloudFront + WAF HIGH
Network architecture Default VPC Custom VPC with segmentation MEDIUM
Health checks TCP only HTTP application-level MEDIUM
IAM hygiene Long-lived keys OIDC + key rotation + MFA MEDIUM
DNS External Route53 (optional) LOW
Backup/DR 7-day automated only Cross-region snapshot copy LOW

Recommendations (Priority Order)

Phase 1 — Immediate (Week 1) — CRITICAL Security

  1. Lock down RDS network access

    • Set PubliclyAccessible=false on drop-db
    • Update drop-db-sg: remove 0.0.0.0/0, allow only App Runner VPC Connector SG
    • Verify App Runner can still connect via VPC Connector

  2. Migrate secrets to Secrets Manager

    • Create secrets: drop/database-url, drop/jwt-secret
    • Update App Runner service to reference secret ARNs
    • Remove plaintext env vars from App Runner config
    • Rotate database password and JWT secret post-migration

  3. Enable RDS encryption

    • Snapshot current instance
    • Restore snapshot with encryption enabled
    • Update connection string to new endpoint
    • Verify, then delete old unencrypted instance
    • Requires brief downtime — schedule maintenance window

Phase 2 — Short Term (Week 2-3) — HIGH Priority

  1. Configure CloudWatch monitoring

    • RDS alarms: CPU, storage, connections, memory
    • App Runner alarms: request count, error rate, latency
    • SNS topic for alert notifications
    • Enable RDS Enhanced Monitoring

  2. Enable Multi-AZ RDS

    • Modify instance to Multi-AZ
    • Near-zero downtime (AWS handles failover setup)

  3. Deploy CloudFront + WAF

    • CloudFront distribution pointing to App Runner
    • WAF with AWS managed rule sets (Common, SQLi, Known Bad Inputs)
    • Update DNS to point to CloudFront

Phase 3 — Medium Term (Month 2) — Hardening

  1. Custom VPC migration

    • Design VPC: 2 private subnets (RDS), 2 public subnets (NAT if needed)
    • Migrate RDS to private subnets
    • Update App Runner VPC Connector

  2. HTTP health checks

    • Implement /health endpoint in Drop application (DB connectivity check)
    • Configure App Runner HTTP health check path

  3. IAM improvements

    • Audit john-deploy key age
    • Enable MFA on IAM user
    • Consider GitHub Actions OIDC for CI/CD (eliminates long-lived keys)

Risk Matrix

Risk Likelihood Impact Severity Mitigation
Database breach via public access + open SG HIGH CRITICAL CRITICAL Phase 1: Lock down network (C1, C2)
Credential leak from plaintext env vars MEDIUM CRITICAL CRITICAL Phase 1: Secrets Manager (C3)
Data exposure from unencrypted storage LOW HIGH HIGH Phase 1: Enable encryption (C4)
Database outage (single-AZ failure) LOW HIGH HIGH Phase 2: Multi-AZ (H1)
Silent application failure (no monitoring) MEDIUM MEDIUM HIGH Phase 2: CloudWatch (H2)
Application-layer attack (no WAF) MEDIUM HIGH HIGH Phase 2: WAF (H3)
DDoS / performance degradation (no CDN) LOW MEDIUM MEDIUM Phase 2: CloudFront (M1)
Lateral movement via default VPC LOW MEDIUM MEDIUM Phase 3: Custom VPC (M2)
IAM key compromise LOW HIGH MEDIUM Phase 3: Key rotation + OIDC (L2)

Appendix: Raw Resource Details

App Runner — drop-web

Service:         drop-web
Status:          RUNNING
Region:          eu-west-1
Source:          ECR (container image)
CPU:             1 vCPU
Memory:          2 GB
Port:            3000
Ingress:         Public
Egress:          VPC Connector
Health Check:    TCP
Environment:     DATABASE_URL (plaintext, contains password)
                 JWT_SECRET (plaintext)

RDS — drop-db

Engine:          PostgreSQL 16.6
Instance Class:  db.t3.micro
Storage:         20 GB gp3
AZ:              eu-west-1a (single-AZ)
VPC:             Default (172.31.0.0/16)
Public Access:   TRUE
Encrypted:       FALSE
Deletion Prot:   TRUE
Backup:          7-day automated
Monitoring:      DISABLED

ECR — drop-web

Repository:      drop-web
Scan on Push:    TRUE
Encryption:      AES256 (default)

Security Groups — drop-db-sg

Inbound Rules:
  - Protocol: TCP
    Port: 5432
    Source: 0.0.0.0/0  (ALL TRAFFIC)

IAM

User:   john-deploy     (programmatic access, deployment)
Role:   AppRunnerECRAccessRole  (App Runner → ECR pull)

Secrets Manager

Secrets stored: 0 (service provisioned but unused)
Cloud Audit

Cloud Audit: Multi-Cloud Design

Drop — Multi-Cloud Architecture Design

Date: 2026-02-19 Auditor: solution-arch (CloudForge cloud-audit team) MC Task: #1443

Executive Summary

Drop is 85% cloud-portable thanks to Docker containerization and PostgreSQL. Main AWS lock-in: App Runner (easily replaceable). Recommendation: stay on AWS, optimize current setup, design Terraform with abstraction for future portability.

1. Provider Comparison Matrix

Service AWS (Current) Azure GCP
Compute App Runner ($25-35/mo) Container Apps ($20-30/mo) Cloud Run ($15-25/mo)
Database RDS PostgreSQL ($15-18/mo) Azure DB for PG ($15-20/mo) Cloud SQL ($12-18/mo)
Registry ECR ($1-2/mo) ACR ($5/mo) Artifact Registry ($1-2/mo)
Secrets Secrets Manager ($0.40/secret) Key Vault ($0.03/10k ops) Secret Manager ($0.06/10k ops)
CDN CloudFront ($0-5/mo) Front Door ($35+/mo) Cloud CDN ($0-5/mo)
WAF AWS WAF ($5+/mo) Azure WAF ($20+/mo) Cloud Armor ($5+/mo)
Monitoring CloudWatch ($3-10/mo) Azure Monitor ($5-15/mo) Cloud Monitoring ($0-8/mo)
Total estimate $50-75/mo $100-130/mo $35-60/mo

2. Portable Architecture

                    Cloudflare (DNS + CDN + WAF)  ← Cloud-agnostic edge
                              |
                              | HTTPS
                              v
                    ┌──────────────────┐
                    │  CaaS Platform   │  ← App Runner / Container Apps / Cloud Run
                    │  ┌──────────┐   │
                    │  │ Docker   │   │  ← Identical image everywhere
                    │  │ Next.js  │   │
                    │  │ :3000    │   │
                    │  └──────────┘   │
                    └────────┬────────┘
                             │ DATABASE_URL
                    ┌────────┴────────┐
                    │  Managed PG     │  ← RDS / Azure DB / Cloud SQL
                    └─────────────────┘

Abstraction Strategy

Layer Approach
Compute Docker image to any CaaS. No platform SDK
Database Standard PostgreSQL via DATABASE_URL
Secrets Terraform abstracts provider. App reads env vars
DNS/CDN/WAF Cloudflare (cloud-agnostic, free tier)
Monitoring Sentry (errors) + structured logs to any aggregator
CI/CD GitHub Actions (already cloud-agnostic)

3. Migration Paths

AWS to Azure (3-5 days)

  1. Push image to ACR
  2. Create Azure DB for PostgreSQL Flexible Server
  3. pg_dump/pg_restore data migration
  4. Deploy to Azure Container Apps
  5. Update Cloudflare DNS
  6. Write Azure Terraform modules

AWS to GCP (2-3 days)

  1. Push image to Artifact Registry
  2. Create Cloud SQL PostgreSQL
  3. pg_dump/pg_restore
  4. Deploy to Cloud Run (most similar to App Runner)
  5. Update Cloudflare DNS
  6. Write GCP Terraform modules

Lock-In Assessment

Component Lock-In Notes
App Runner LOW Standard Docker, replaceable
RDS PostgreSQL LOW Standard PG, any managed PG works
ECR LOW Standard OCI registry
VPC Connector MEDIUM AWS-specific networking
IAM Roles MEDIUM AWS-specific auth model
Secrets Manager LOW App reads env vars regardless

4. Recommendation: Stay AWS, Optimize

Rationale:

Immediate Actions

  1. Security fixes (encrypt RDS, restrict SG, use Secrets Manager)
  2. Add Cloudflare free tier (DNS, CDN, WAF — cloud-agnostic)
  3. Terraform all resources (reproducibility)
  4. Add CloudWatch basic alarms ($3-5/mo)

Future Migration Triggers

5. 12-Month Cost Projection

Scenario Monthly Annual
Current (no changes) $50-75 $600-900
Optimized AWS $55-80 $660-960
AWS + Cloudflare $55-80 $660-960
Azure equivalent $100-130 $1,200-1,560
GCP equivalent $35-60 $420-720
Cloud Audit

Cloud Audit: App Cloud Readiness

Drop Application Cloud-Readiness Audit

MC Task: #1443 Date: 2026-02-19 Auditor: software-arch (CloudForge team) Application: Drop Fintech Payment App (Next.js 15 + SQLite/PostgreSQL dual-driver)

NOTE (2026-03-03): This audit was performed on 2026-02-19. ADR-014 (2026-03-03) removed SQLite and the dual-driver architecture. Drop now uses PostgreSQL 16 exclusively in all environments. SQLite concerns noted in this audit are resolved. The better-sqlite3 dependency has been removed.

1. Twelve-Factor Compliance

I. Codebase — PASS

II. Dependencies — PASS

III. Config — PASS

IV. Backing Services — PASS

V. Build, Release, Run — PASS

VI. Processes — PARTIAL

VII. Port Binding — PASS

VIII. Concurrency — PARTIAL

IX. Disposability — PASS

X. Dev/Prod Parity — PASS

XI. Logs — PARTIAL

XII. Admin Processes — PASS

2. Containerization Quality

Multi-Stage Build — EXCELLENT

Image Size

Security

Layer Caching

Missing

3. Database Portability

Dual-Driver Architecture — STRONG

SQL Translation Layer

SQLite Idiom PostgreSQL Translation Location
? placeholders $1, $2, ... db.ts:47-50
INSERT OR IGNORE INTO INSERT INTO ... ON CONFLICT DO NOTHING db.ts:56, 104-118
INSERT OR REPLACE INTO INSERT INTO ... ON CONFLICT (col) DO UPDATE SET db.ts:58, 120-134
datetime('now') CURRENT_TIMESTAMP db.ts:60
INTEGER PRIMARY KEY AUTOINCREMENT SERIAL PRIMARY KEY db.ts:278 vs 530
hex(randomblob(32)) encode(gen_random_bytes(32), 'hex') db.ts:248 vs 504

Transaction Support

Migrations

Indexes

4. Config Externalization

Environment Variables

Category Variables Source
Core JWT_SECRET, JWT_EXPIRY, NODE_ENV .env.example:12-14
Database DATABASE_URL db.ts:9
Service Mode NEXT_PUBLIC_SERVICE_MODE, DROP_MODE .env.example:8
Auth (BankID) BANKID_CLIENT_ID/SECRET/URLS, BANKID_MOCK .env.example:19-29
Payments PISP_API_URL/KEY, AISP_API_URL/KEY .env.example:32-40
Cards STRIPE_SECRET_KEY, STRIPE_PUBLISHABLE_KEY .env.example:43-47
KYC SUMSUB_APP_TOKEN, SUMSUB_SECRET_KEY .env.example:50-52
Monitoring SENTRY_DSN, SENTRY_TRACES_SAMPLE_RATE .env.example:63-74
Feature Flags 8x NEXT_PUBLIC_FF_* .env.example:77-87
Exchange EXCHANGE_RATE_API_KEY/URL .env.example:55-59

Secrets Management

Feature Flags

5. CI/CD Quality

Pipeline Structure (ci.yml)

lint-test (parallel)          docker-scan (sequential, needs lint-test)
  -- npm ci                     -- docker build
  -- eslint                     -- Trivy scan (table, exit-code=1 on HIGH/CRITICAL)
  -- tsc --noEmit               -- Trivy SARIF -> GitHub Security
  -- vitest run
  -- npm audit (production)

Reproducibility

Security Scanning

Testing

Deployment

6. Overall Score and Top 5 Improvements

Overall Cloud-Readiness Score: 7.5 / 10

The application demonstrates strong cloud-native fundamentals:

Top 5 Improvements (Priority Order)

1. Eliminate Build Tools from Production Image (HIGH)

2. Add Structured Logging (HIGH)

3. Add CI Coverage Enforcement and E2E Tests (MEDIUM)

4. Automate Schema Parity Check (MEDIUM)

5. Add Deployment Pipeline and Environment Promotion (MEDIUM)

Honorable Mentions

Appendix: File Reference

File Purpose
src/drop-app/src/lib/db.ts Dual-driver database abstraction (SQLite + PostgreSQL)
src/drop-app/Dockerfile 3-stage multi-stage build
src/drop-app/.env.example Environment variable documentation (87 lines)
src/drop-app/fly.toml Fly.io deployment config (Stockholm region)
src/drop-app/docker-compose.production.yml Self-hosted production config
src/drop-app/package.json Dependencies and scripts
.github/workflows/ci.yml CI pipeline (lint, test, type-check, Trivy)
src/drop-app/migrations/0001_initial-schema.ts PostgreSQL migration (node-pg-migrate)
src/drop-app/next.config.ts Next.js config (standalone output, security headers)
src/drop-app/src/middleware.ts Edge middleware (CSRF, CSP nonce)
src/drop-app/src/lib/middleware.ts Server middleware (rate limiting, auth, validation, audit)
src/drop-app/src/app/api/health/route.ts Health endpoint (real DB check)
src/drop-app/src/lib/env.ts Environment validation at startup
Cloud Audit

Cloud Audit: Validation Report

Drop — Validation + Security + Cost Report

Date: 2026-02-19 Auditor: cloud-tester (CloudForge cloud-audit team) MC Task: #1443

Executive Summary

Drop's AWS infrastructure has 3 CRITICAL and 4 HIGH security findings requiring immediate remediation. Current spend is ~$50-75/mo, well-optimized for scale. The application is cloud-portable (7.5/10) and the recommended path is to stay on AWS with security hardening + Terraform IaC.

1. Security Posture Assessment

Current vs Improved

Area Current State After Remediation Risk Reduction
Secrets Plaintext in App Runner env vars AWS Secrets Manager CRITICAL → LOW
RDS Access Publicly accessible, SG open 0.0.0.0/0 Private, VPC-only access CRITICAL → LOW
Encryption RDS unencrypted at rest AES-256 encryption enabled CRITICAL → RESOLVED
Monitoring None (no CloudWatch) Basic alarms + Performance Insights HIGH → LOW
WAF None Cloudflare WAF (free tier) HIGH → LOW
CDN None (direct App Runner URL) Cloudflare CDN HIGH → LOW
SSL/TLS App Runner managed cert Cloudflare + App Runner MEDIUM → LOW
IAM Single user (john-deploy) Least-privilege roles MEDIUM → LOW

Security Findings Summary

# Severity Finding Remediation Effort
S1 CRITICAL RDS publicly accessible with SG allowing 0.0.0.0/0:5432 Set publicly_accessible=false, restrict SG to VPC CIDR 1 hour
S2 CRITICAL Database password in plaintext App Runner env var Migrate to Secrets Manager, update App Runner to read from SM 2 hours
S3 CRITICAL JWT_SECRET in plaintext App Runner env var Migrate to Secrets Manager 1 hour
S4 HIGH RDS storage not encrypted at rest Enable encryption (requires snapshot + restore for existing DB) 2-4 hours
S5 HIGH No monitoring or alerting configured Add CloudWatch alarms for CPU, memory, DB connections 1 hour
S6 HIGH No WAF protection Add Cloudflare WAF (free tier) 30 min
S7 HIGH No CDN (direct App Runner URL exposed) Add Cloudflare CDN 30 min
S8 MEDIUM Sentry DSN in plaintext (not secret, but cleanup) Move to Secrets Manager for consistency 30 min
S9 MEDIUM Docker image has build tools in runner (attack surface) Remove python3/make/g++ from runner stage 1 hour
S10 MEDIUM No structured logging (incident investigation gaps) Add pino/winston with JSON output 2 days
S11 LOW ECR image tag mutability (tag overwrite risk) Set image_tag_mutability = IMMUTABLE 5 min
S12 LOW No lifecycle policy for ECR images Add policy to clean old images 15 min

Compliance Checklist

Item Status Notes
GDPR data tables (consents, data_access_requests) PASS Schema includes consent tracking, DSAR, right to erasure
Audit logging PASS audit_log table with IP, user_agent, request_id
AML/KYC compliance PASS aml_alerts, str_reports, screening_results tables
Encryption at rest FAIL RDS storage unencrypted
Encryption in transit PARTIAL App Runner HTTPS, but RDS sslmode=no-verify
Secrets management FAIL Plaintext in env vars
Access control PARTIAL Single IAM user, no MFA enforcement
Backup & recovery PASS RDS 7-day automated backups
DeletionProtection PASS Enabled on RDS

2. Cost Comparison

Current AWS Spend

Resource Monthly Cost Notes
App Runner (1 vCPU, 2GB) $25-35 Always-on, no auto-stop
RDS db.t3.micro $15-18 Single-AZ, 20GB gp3
ECR $1-2 Image storage
VPC Connector $5 Flat fee
Data transfer $2-5 Low traffic
Total $48-65

Optimized AWS (after fixes)

Resource Monthly Cost Change
App Runner $25-35 No change
RDS (encrypted) $15-18 No cost increase
ECR $1-2 No change
Secrets Manager (3 secrets) $1.20 +$1.20
CloudWatch (basic alarms) $3-5 +$3-5
Cloudflare (free tier) $0 Free CDN/WAF/DNS
Total $52-70 +$4-7

Multi-Cloud Equivalent

Provider Monthly Annual vs Current
AWS (optimized) $52-70 $624-840 +$4-7/mo
Azure $100-130 $1,200-1,560 +$50-65/mo
GCP $35-60 $420-720 -$5-15/mo

Verdict: AWS is cost-effective. GCP saves ~$10/mo but migration effort not justified at current scale.

3. Risk Matrix

Risk Probability Impact Current Mitigation Recommended
Data breach via public RDS HIGH CRITICAL DeletionProtection only Restrict SG, disable public access
Secret exposure MEDIUM CRITICAL None (plaintext) Secrets Manager + rotation
Service downtime LOW HIGH App Runner auto-scaling Add health checks, CloudWatch alarms
Data loss LOW CRITICAL 7-day RDS backups Add cross-region backup copy
Cost overrun LOW MEDIUM None Add AWS Budgets alarm at $100
Vendor lock-in LOW MEDIUM Docker + PostgreSQL Terraform abstraction modules
DDoS attack MEDIUM HIGH None Cloudflare WAF + rate limiting
Compliance failure MEDIUM HIGH Tables exist, no encryption Enable encryption, structured logging

4. Implementation Roadmap

Phase 1: Security Fixes (Immediate — Day 1)

Phase 2: IaC Migration (Week 1)

Phase 3: Monitoring & Observability (Week 2)

Phase 4: Edge Security (Week 2-3)

Phase 5: RDS Encryption (Week 3)

Phase 6: Multi-Cloud Readiness (Month 2+)

5. Recommendations Summary

Priority Action Status
P0 (NOW) Fix RDS public access + SG Terraform module created
P0 (NOW) Move secrets to Secrets Manager Terraform module created
P1 (Week 1) Enable RDS encryption Requires snapshot/restore
P1 (Week 1) Deploy Terraform IaC Modules ready
P2 (Week 2) Add monitoring (CloudWatch + Performance Insights) In Terraform
P2 (Week 2) Add Cloudflare CDN/WAF Manual setup
P3 (Month 1) Add structured logging Application code change
P3 (Month 1) Add graceful shutdown handler Application code change
P4 (Month 2+) Multi-cloud Terraform modules As needed

Overall Assessment: Drop's infrastructure is functional but needs immediate security hardening. The Terraform IaC created by this audit provides a complete, reproducible foundation. Total investment: ~1 week of engineering time, ~$5/mo additional cost, significant risk reduction.

Bilko Deploy — Standard Operating Procedure

$(cat /tmp/bilko-deploy-sop.html | jq -Rs .)

Bilko Deploy — Standard Operating Procedure

Bilko Deploy — Standard Operating Procedure

Last updated: 2026-04-22
Owner: FlowForge (Kelsey Hightower)
Status: ACTIVE

Cloud Run Architecture

GCP Project: tribal-sign-487920-k0
Region: europe-north1
Services:

Deploy Map

Branch Service URL CI Workflow Last Verified
main bilko-web https://bilko-demo.alai.no gcp-deploy.yml (BROKEN) 2026-04-22
main bilko-api https://bilko-api-762788903040.europe-north1.run.app gcp-deploy.yml (BROKEN) 2026-04-18
feat/intesa-bih-demo bilko-intesa-demo https://bilko-intesa-demo-762788903040.europe-north1.run.app Manual gcloud only 2026-04-17

Pre-Flight Checks (ZAKON PI2 Check 2)

OBAVEZNO — Run these 4 commands and paste output into MC task BEFORE touching code:

# 1. Target URL alive?
curl -sI https://bilko-demo.alai.no | head -3

# 2. Branch state?
git log main --oneline -5

# 3. CI health?
gh run list --repo alai-holding/bilko --branch main --limit 3

# 4. Cloud Run service status?
gcloud run services describe bilko-web \
  --region europe-north1 \
  --project tribal-sign-487920-k0 \
  --format='value(status.latestReadyRevisionName,status.url,status.traffic)'

If any returns unexpected: STOP, escalate to John. Do not proceed.

CI Pipeline Status

Status: BROKEN (2026-04-15 onwards)
Root Causes:

  1. GitHub Actions minutes quota exhausted (monthly limit reached)
  2. --no-traffic flag on line 206 of gcp-deploy.yml prevents traffic promotion for existing services

Workaround: Use manual deploy path (see below) until CI fixed.

Manual Deploy Path (Emergency + CI Broken)

When CI is broken or for emergency fixes, follow this path:

Step 1: Build Docker Image

cd /Users/makinja/ALAI/products/Bilko

docker build \
  --platform linux/amd64 \
  -f apps/web/Dockerfile \
  --build-arg NEXT_PUBLIC_API_URL=https://bilko-api-762788903040.europe-north1.run.app/api/v1 \
  -t europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web:fix-<purpose>-<DDmon> \
  .

Image tag convention:

Context reduction (.dockerignore): As of 2026-04-22, .dockerignore reduces build context from 4.1GB → 50MB by excluding node_modules, .next, apps/e2e, docs, etc.

Step 2: Push to Artifact Registry

gcloud auth configure-docker europe-north1-docker.pkg.dev

docker push europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web:fix-<purpose>-<DDmon>

Step 3: Deploy to Cloud Run

CRITICAL: Do NOT use --no-traffic flag for existing services. It blocks traffic promotion.

gcloud run deploy bilko-web \
  --image europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web:fix-<purpose>-<DDmon> \
  --region europe-north1 \
  --platform managed \
  --allow-unauthenticated \
  --max-instances 10 \
  --min-instances 0 \
  --memory 512Mi \
  --cpu 1 \
  --concurrency 100 \
  --timeout 60s \
  --port 3000 \
  --set-env-vars NEXT_PUBLIC_API_URL=https://bilko-api-762788903040.europe-north1.run.app/api/v1,NEXT_TELEMETRY_DISABLED=1 \
  --project=tribal-sign-487920-k0

Step 4: Verify Deployment

# Check revisions
gcloud run revisions list \
  --service bilko-web \
  --region europe-north1 \
  --project=tribal-sign-487920-k0 \
  --limit=5

# Verify traffic routing (should show 100% on latest revision)
gcloud run services describe bilko-web \
  --region europe-north1 \
  --project=tribal-sign-487920-k0 \
  --format='value(status.traffic)'

Post-Deploy Evidence Gate (ZAKON PI2 Check 5)

MC task CANNOT move to done without ALL three:

  1. curl checks: Paste output showing HTTP 200 for expected routes 
    curl -sI https://bilko-demo.alai.no | head -3
curl -sI https://bilko-demo.alai.no/invoices/new | head -3
curl -sI https://bilko-demo.alai.no/settings | head -3
curl -sI https://bilko-demo.alai.no/intesa-bridge | head -3  # Should be 404
  2. Playwright screenshots: Stored in docs/evidence/<task-id>/*.png
    • Home page
    • Feature verified (e.g., invoice template save button)
    • Any isolation checks (e.g., 404 for client routes on main)
  3. verification.json: Machine-readable evidence file 
    {
  "task_id": 8730,
  "timestamp": "2026-04-22T21:41:10Z",
  "revision": "bilko-web-00019-7tl",
  "traffic_100_percent": true,
  "curl_checks": { "home": 200, "intesa-bridge": 404, ... },
  "playwright_pass": true,
  "screenshots": ["home.png", "invoices-new.png", ...]
}

Deploy Flow Diagram

flowchart LR
    A[Code Change] --> B{CI Healthy?}
    B -->|Yes| C[CI: Build + Push]
    B -->|No| D[Manual Build]
    C --> E[Artifact Registry]
    D --> E
    E --> F[Cloud Run Deploy]
    F --> G{Traffic Routing}
    G -->|100%| H[Live]
    G -->|0%| I[Blocked - Check --no-traffic flag]
    H --> J[Evidence Gate]
    J --> K{All 3 checks pass?}
    K -->|Yes| L[MC task done]
    K -->|No| M[Block - Add evidence]

Known Issues + Workarounds

Issue 1: CI broken since 2026-04-15

Symptom: All main branch pushes fail at deploy step
Root cause: GitHub Actions quota + --no-traffic flag
Workaround: Use manual deploy path above

Issue 2: Intesa content leaked to public URL (fixed 2026-04-22)

Symptom: /intesa-bridge route returned 200 on bilko-demo.alai.no
Root cause: Intesa feature branch merged to main
Fix: Deleted intesa routes from main (commit 66d2220) + added branch-purity.yml CI check

Issue 3: Manual paste-copy anti-pattern

Symptom: CEO had to manually paste docker build output and gcloud commands
Root cause: FlowForge task dispatched after image built locally
Fix: Always dispatch FlowForge BEFORE build step, let agent own full flow

Branch Purity Rules

Client-specific routes MUST NOT appear on main. Reserved prefixes:

CI Enforcement: .github/workflows/branch-purity.yml runs on every PR to main:

find apps/web/app -type d \( -name "intesa-*" -o -name "corpint-*" \) | grep . && exit 1 || exit 0

Registry: ~/system/rules/client-prefix-registry.md

Domain Mapping

Escalation

Owner: FlowForge
Escalate to: John → pi-orchestrator
MC category: devops + priority: H

Created by ALAI Skillforge, 2026-04-22

Bilko CI/CD — Stage→Prod Pipeline (MC #99477)

Overview

Stage pipeline: push-main → bilko-stage-auto-deploy → cloudbuild-stage.yaml → bilko-{web,api}-stage

Prod pipeline: tag v* → bilko-main-deploy → cloudbuild.yaml → bilko-{web,api}

Stage pipeline is optimized for FAST FEEDBACK — no quality gates. Prod pipeline has 8 production gates including SHA verification, Trivy scanning, Flyway migrations, and Cloud Build native approval.

Stage Pipeline

Step Purpose Image Tag Duration (avg)
sanity-check Verify Docker socket + Artifact Registry reachability (environment health, NOT a quality gate) — ~2.3s
build-web Build Next.js app with docker buildx (apps/web/Dockerfile) :stage-${SHORT_SHA}
:stage-latest		 ~3m
push-web Push image to Artifact Registry (europe-north1-docker.pkg.dev/tribal-sign-487920-k0/bilko/web) — ~7s
migrate-db Run Flyway migrations against Cloud SQL bilko-staging-db (POSTGRES_16) via Cloud SQL proxy — ~22s
deploy-web-stage Deploy bilko-web-stage Cloud Run service with :stage-${SHORT_SHA} image, --no-traffic — ~39s
promote-web-stage Route 100% traffic to new revision (no canary for stage) — ~10s
deploy-api-stage Deploy bilko-api-stage (redeploys EXISTING image only — no API build step, see OCD-1) — ~19s
smoke-test curl -sf https://bilko-api-stage-dh4m46blja-lz.a.run.app/api/v1/health — exit 1 if non-200 — ~2.5s

Total duration: ~5 minutes (build 6f2236f6, validated 2026-05-06)

Prod Pipeline

Existing prod pipeline (cloudbuild.yaml) has 8 gates and MUST NOT be rewritten. References:

Prod pipeline is BLOCKED on OCD-5 (bilko-db Cloud SQL instance does not exist — requires CEO approval for provisioning).

Triggers

Trigger Name Filename Branch/Tag Approval Service Account
bilko-stage-auto-deploy infrastructure/gcp/cloudbuild-stage.yaml ^main$ No (auto-deploy) 762788903040@cloudbuild.gserviceaccount.com
bilko-main-deploy infrastructure/gcp/cloudbuild.yaml v* (semver tag) Yes (Cloud Build UI) 762788903040@cloudbuild.gserviceaccount.com

GCP project: tribal-sign-487920-k0, region: europe-north1

Open Risks — 5 CEO Decisions Required

These items require CEO judgment and are NOT resolved in this implementation:

OCD-1: bilko-api Build Pipeline Gap

Status: OPEN — BLOCKER for API continuous delivery

Current state: bilko-api-stage is live and serving traffic at https://bilko-api-stage-dh4m46blja-lz.a.run.app/api/v1 with image api:stage-b7e8a59. No Cloud Build pipeline exists for the Kotlin/Ktor API. Dockerfile path unconfirmed.

Impact: Stage cloudbuild-stage.yaml deploy-api-stage step redeploys the EXISTING API image only — cannot build new API images. API deployments must be manual via gcloud run deploy until resolved.

CEO decisions needed:

  1. What is the canonical Dockerfile path for apps/api?
  2. Should API have its own Cloud Build step in cloudbuild-stage.yaml or a separate trigger?
  3. Is bilko-api currently deployed manually via gcloud run deploy?

OCD-2: Stage Hostname — bilko-stage.alai.no vs Raw .run.app URL

Status: OPEN — affects CORS configuration

Current state: ENV-MATRIX.md CORS_ORIGINS for staging references staging.bilko.io (STALE). terraform.tfvars stage_api_url points to raw .a.run.app URL. Stage pipeline uses raw .run.app URL as default.

Impact: Frontend CORS errors if staging.bilko.io DNS is ever pointed at stage services.

CEO decision needed: Should bilko-stage.alai.no be the canonical stage hostname? If yes: Cloudflare DNS entry (manual — not in Bilko TF stack) + CORS_ORIGINS update required via separate MC.

OCD-3: Postgres Version Mismatch — Stage POSTGRES_16 vs Prod POSTGRES_15

Status: OPEN — CRITICAL for financial data integrity

Current state: bilko-staging-db runs POSTGRES_16 (confirmed live). envs/prod/main.tf line 94 specifies POSTGRES_15 for prod (bilko-db does not exist yet — see OCD-5). Stage validates migrations and queries against PG16; prod would run PG15.

Impact: For a financial accounting SaaS, stage validation on PG16 while prod runs PG15 invalidates the "stage-as-test-environment" premise. Schema compatibility unverified. SQL dialect differences (PG15→PG16) may surface as prod-only bugs.

CEO decision needed: Upgrade prod to POSTGRES_16 (requires maintenance window, pg_upgrade or dump/restore) OR downgrade stage to POSTGRES_15? ALAI standard tech stack (ALAI/CLAUDE.md) mandates POSTGRES_16 for all products, suggesting prod config is non-compliant.

OCD-4: Stage → Prod SHA Promotion Strategy

Status: OPEN — architectural decision

Current state: Prod trigger fires on semver tag push, rebuilds from source. Stage-validated image digest is NOT carried to prod build. Stage tests one SHA and prod deploys a different build. If a hot dependency updates between stage build and prod build (e.g., npm registry serves new patch version), stage and prod can diverge on identical Git SHAs.

CEO decision needed:

  1. Option A: Accept rebuild-on-tag (simpler, current model) with acknowledgment of hot-dependency risk.
  2. Option B: Implement digest promotion where prod trigger accepts an image digest input parameter and skips rebuild. Requires Cloud Build trigger API call from a promotion script or Google Cloud Deploy.

OCD-5: Prod Cloud SQL bilko-db Existence

Status: OPEN — BLOCKER for prod terraform apply

Current state: gcloud sql instances list --project=tribal-sign-487920-k0 shows ONLY bilko-staging-db. No bilko-db (prod) exists. envs/prod/main.tf explicitly notes "bilko-db (prod) — TBD — audit required" (lines 4-6 and import.sh).

Impact: Any terraform apply on envs/prod would attempt to create a REGIONAL HA POSTGRES_15 db-custom-2-7680 instance (~$100+/month). Without CEO sign-off, prod infra is BLOCKED.

CEO decision needed: Approve prod DB provisioning (cost + data migration strategy if migrating from elsewhere) before ANY envs/prod TF apply is ever run. If bilko-db exists elsewhere (on-prem? Railway?), import.sh must be run first.

Validation

Evidence file: /tmp/99477-proveo-evidence.md

Build ID: 6f2236f6-86ec-444c-96b7-7c22f63cf5a2

Build log: View in GCP Console

Validation date: 2026-05-06T20:28Z

Validator: Angie Jones (Proveo)

Verdict: PASS — 7/7 Acceptance Criteria met

ZAKON PI2 Compliance Status

Stage pipeline: ✅ COMPLIANT

Prod pipeline: ⏸ BLOCKED (awaiting OCD-5 bilko-db provisioning approval)

Last Updated

2026-05-06, owner: FlowForge (Kelsey Hightower)

2026-07-07 — Pipeline migriran na bilko-selfhosted (MC #104933)

Root cause: org bez hosted minuta; svi jobovi migrirani na bilko-selfhosted (merge a1bec755, PR #65; gitleaks fetchDepth:0 5febf0ad). Run 298: CI_Gates/Build/Flyway/DeployStage GREEN, E2E 109/142 FAILED (prvi E2E signal, triage MC #104956). FORGE docker prune 37GB. Evidence: ~/system/evidence/104933/.

ALAI CI/CD Blueprint Standardization 2026-05-08

ALAI CI/CD Blueprint Standardization — 2026-05-08

Master MC: #99881 Owner: John (AI Director) + Petter Graff persona for canonical refresh Status: All 4 phases verified closed. Triple-layer enforcement live. Cost: ~$15-30 LLM tokens

Context

CEO directive 2026-05-08 in single-day push: "Discuss CI/CD pipelines and blueprints" → triple-layer mechanical enforcement live + 7/7 fleet compliance + free-first routing across persona blueprints.

4-phase arc summary

Faza MC Outcome
1 — Audit #99882 4 artifacts in ~/system/specs/cicd-audit-2026-05-08/ (gap matrix, deploy-map matrix, canonical self-audit, summary). 1 real bug caught: DropSrbija/BUILD-BLUEPRINT.md line 225 stale "Postgres 5434" comment (actual port 5436).
2 — Canonical refresh #99886 UNIVERSAL bumped to v3.0 (§13 6-mandatory files including DEPLOY-MAP, §15 forma-only variant, §16.3 CI gates, ZAKON PI2 invariant). DEPLOY bumped to v2.0 (multi-profile §1A GCP / §1B Azure VM / §1C Cloudflare Pages / §1D Vercel deprecated). blueprint-format.md disambiguation header (YAML agent layer vs MD product layer). alai-cicd-architecture.md staleness notice (sections §5.2 AWS, §9 Phase 3 superseded).
3 — Product migration #99896 7 in-scope products migrated to v2 §1A/§1B/§1C profiles. 6 new mandatory files created (web PIPELINE/RUNBOOK/CHANGELOG, Gotiva RUNBOOK/CHANGELOG, Drop PIPELINE). Drop §1B refactor reached FULL_COMPLIANCE 5/5 schema. Excluded: BasicFakta (MC #99893 Vercel→CF Pages migration), DropSrbija (MC #99883 scope decision), akershus-fylke (forma-only).
4 — Enforcement #99911 Triple-layer mechanical enforcement live.

Triple-layer enforcement (all live, all verified)

1. Linter — ~/system/tools/blueprint-check.js v2

Dual-mode (backward compat with mehanik-commit + pre-dispatch-gate Check 9):

JSON output reusable by hook + daemon.

2. PostToolUse hook — ~/.claude/hooks/blueprint-schema-validator.sh

Registered in settings.json under Write|Edit|MultiEdit matcher. Triggers on writes to product-root DEPLOY-MAP.md files under ~/business/ALAI-Holding-AS/{products,web,finance}/*/. Blocks with exit 2 + structured BLOCKED message + missing sections + template pointers when schema fails. Override marker: <!-- blueprint-schema-validator: skip -->.

Trace log: ~/system/state/blueprint-schema-validator-trace.log.

3. Nightly daemon — ~/system/daemons/blueprint-fleet-watchdog.js

LaunchAgent com.alai.blueprint-fleet-watchdog schedules daily 06:15. Scans 10 product roots, persists state to ~/system/state/blueprint-fleet-status.json, detects regressions (verdict drop, schema score drop, file removal) with differential alert. Exit 1 on regression.

Free-first routing (CEO directive "ukljuci free modele gdje god mozes")

~/system/config/tier-routing.json updated:

Persona blueprint sweep (MC #99923): 13 yaml files — 9 all-sonnet personas (AgentForge, Axiom, Finverge, FlowForge, Lexicon, Proveo, Resolver, Skybound, Vizu) + 4 CodeCraft yaml (api-backend, codecraft-api, nextjs-app, openapi-sdk-package). 46 phase declarations swept sonnet → local-first (qwen2.5-coder:32b@anvil for general phases, qwen3-coder:latest@forge for code-gen phases). 6 KEPT-sonnet phases with explicit rationale: 3 Lexicon legal phases (Norwegian law / GDPR / PSD2 regulatory precision), 3 Resolver cross-company phases (multi-domain reasoning).

Verifier pattern dokazan

bp-verifier background agent ran ~15 rounds, ~178 atomic claims, 2 stvarna buga uhvaćena:

  1. DropSrbija/BUILD-BLUEPRINT.md line 225 stale comment "Postgres 5434" (actual port 5436 per docker-compose.yml). Fixed in both audit artifact + product blueprint.
  2. Drop/DEPLOY-MAP.md schema 3/5 PARTIAL — no formal OPEN RISK / OCD register, no SA distinction. Fixed via §1B-appropriate equivalents (SSH key → Trigger SA equivalent, container USER → Service SA equivalent).

Pattern recommendation: For every multi-phase project, spawn named bp-verifier in BG (Agent({subagent_type: "verifier", name: "bp-verifier", run_in_background: true})), send each artifact via SendMessage for atomic claim validation, fix-loop on FAIL. Cost: $0.10 per round Claude ($0 if MLX primary per new tier-routing).

Fleet compliance final (verified by daemon 2026-05-08)

Product Verdict Files Schema Profile
Bilko FULL_COMPLIANCE 6/6 5/5 §1A GCP
Tok FULL_COMPLIANCE 6/6 5/5 §1A GCP
Drop FULL_COMPLIANCE 6/6 5/5 §1B Azure VM
Lobby FULL_COMPLIANCE 6/6 5/5 §1A GCP (stub)
Plock FULL_COMPLIANCE 6/6 5/5 §1A GCP (stub)
Gotiva FULL_COMPLIANCE 6/6 5/5 §1A GCP multi-service
web FULL_COMPLIANCE 6/6 5/5 §1C CF Pages
akershus-fylke FORMA_ONLY_OK 1/1 N/A non-deployable
BasicFakta MISSING_FILES 5/6 0/5 §1D Vercel deprecated (MC #99893 migration backlog)
DropSrbija MISSING_FILES 3/6 0/5 scope decision pending (MC #99883)

Open follow-ups (parked, not blocking arc closure)

Git audit trail

Lessons

  1. Verifier-in-bg uhvati realne bugove — propagated stale comments + schema gaps. USE THIS PATTERN for every multi-phase project.
  2. Mehanik enforcement >> ZAKON-only — hook + daemon catch what memo can't. UNIVERSAL §13 / DEPLOY §4 sad mehanički enforced.
  3. Local-first viable for builder/verifier — qwen2.5-coder + qwen3-coder + MLX qwen3-coder-30b dovoljno za schema validation, code gen, doc draft. Sonnet ostaje za high-stakes synthesis (legal, cross-company).
  4. Closure-loop discipline — build-verify-mark-done pattern, ne build-verify-stop. CEO uhvatio gap u mid-session closure ("jel sve dokumentovano, merged, zatvoreno po propisima") and triggered this BookStack publish + git commit + memory entry.

References

Slack bot token SSOT — slack.json (MC #102830) — 2026-06-03

Summary

MC #102830 makes ~/system/config/slack.json the single source of truth (SSOT) for the Slack bot's tokens, with environment-variable fallback, and removes the hardcoded tokens from the LaunchAgent plist. Previously the com.john.slack-bot.plist hardcoded both SLACK_BOT_TOKEN and SLACK_APP_TOKEN in EnvironmentVariables — so a token rotation that wasn't mirrored into the plist would strand the daemon with a stale token.

Change

Token rotation procedure (new)

  1. Edit ~/system/config/slack.json — update token (xoxb) and/or app_token (xapp).
  2. bash ~/system/tools/run-slack-bot-reload.sh

No plist edit. No risk of stranding the daemon on rotation.

Verification

Security note

This also improves posture: secrets moved out of a (potentially world-readable) LaunchAgent plist into the 0600 slack.json. Token values are never logged (masked).

Bilko CI — integration-test job (Testcontainers) MC #102843 — 2026-06-03

Summary

MC #102843 adds an integration-test job to Bilko's .github/workflows/ci.yml. Previously the backend-test job ran only ./gradlew test, and tasks.test does excludeTags("integration") (apps/api/build.gradle.kts:159) — so the integrationTest task (Testcontainers/Postgres, includeTags("integration")) never ran in CI. PRs that broke integration tests passed green (surfaced manually by Proveo during MC #102798).

Change (PR #245, base main, not merged)

Why non-blocking (important)

Running the suite revealed it is currently broken on main: ~78/1147 integration tests fail (FlywayMigrateException in SettingsServiceRlsTest, ExposedSQLException in VatReportStatutoryGroupingTest, and others). These had never run in CI. Making the job a required gate immediately would red-lock every PR. So the job is visible on every PR (failures now surface) but does not block merges yet.

Path to required gate

Tracked in MC #102874 (H): fix the 78 failing integration suites. Once green, promotion is a one-line CI change — remove continue-on-error: true and add integration-test to build needs: [lint, unit, backend-test, integration-test].

Verification

Incident (logged, low severity)

During implementation a build branch was accidentally pushed to origin/main (commit ecf5a97) and immediately reverted (036e2c6). It triggered bilko-stage-auto-deploy twice; both SUCCESS, change was ci.yml-only (no app artifact change), stage is non-customer-facing. origin/main verified clean afterwards. Lesson recorded: build agents must git push -u origin HEAD:<branch> and verify upstream ≠ origin/main (push to Bilko main auto-deploys stage).

Bilko integrationTest suite green — 79->0 failures (MC #102874) — 2026-06-03

Summary

MC #102874 took the Bilko backend integrationTest suite from 79 failing → 0 failing (1213 tests, 91 suites). These integration tests had never run in CI (tasks.test does excludeTags("integration")); the new CI job from MC #102843 exposed the rot. PR #246 (base main, not merged).

Root-cause clusters fixed

Assertion-strength (anti-gate-gaming)

Two tests had been widened to multi-code accept lists; re-pinned to single deterministic codes:

Verification

Process note

Multi-session: one session produced the WIP fix (~76 failures), a John review session affirmed it ("NOT gate-gaming") and flagged 3 items, this session ran the decisive green run, fixed the last 3 + the 2 flags, and closed. Earlier in the campaign a build agent accidentally pushed to main (reverted) — lesson recorded; this PR was pushed cleanly with explicit refspec, main untouched.

Downstream (NOT part of this fix)

  1. Merge PR #246 → main green.
  2. Then flip the CI gate (MC #102843): remove continue-on-error + add integration-test to build needs:. Cannot flip before merge (main is still red until then).

Prometheus Best Practices — USE vs RED

Prometheus Best Practices and Pitfalls

Source: YouTube Learning — Julius Volz (Prometheus co-founder), Swiss Cloud Native Day 2021
Indexed: 2026-06-15 (MC #103620)

USE vs RED: Decision Framework

USE Method (Resource-Oriented Systems)

For infrastructure components (CPU, memory, disk, network):

When to use: Cloud Run instances, Azure Container Apps, database connections, worker threads, storage volumes.

RED Method (Request-Oriented Systems)

For services handling requests:

When to use: REST APIs, BFF layers, RPC services, HTTP endpoints.

Custom Metrics in Application Code

Best Practices

  1. Counter for events that only go up (requests, errors, jobs completed)
  2. Gauge for values that go up/down (active connections, queue size, temperature)
  3. Histogram for bucketed observations (latency, request size) — auto-generates _sum, _count, _bucket
  4. Summary for client-side quantiles (use histogram + server-side quantiles in PromQL instead)

Common Pitfalls

PromQL Essentials

# Rate of HTTP errors over 5min
rate(http_requests_total{status=~"5.."}[5m])

# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# CPU utilization (USE)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Error rate (RED)
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

How This Applies to ALAI

Current Infrastructure

  1. Instrument Bilko/LumisCare services with Micrometer (auto-exposes Prometheus /actuator/prometheus)
  2. Add RED dashboards for all user-facing APIs (Grafana template: https://grafana.com/grafana/dashboards/4701)
  3. Add USE dashboards for Cloud Run / ACA resource health
  4. Alert on SLIs: Error rate >1%, p95 latency >2s, CPU >80%

ALAI-Specific Pitfall to Avoid

Do NOT add per-user or per-client labels to core metrics. Use organization_id buckets (max ~50) or aggregate at service level. High cardinality = Prometheus death.

References

Bilko trunk crven 2026-08-16 — FX fail-closed, dva fail-open uslova, i zelen build koji nije mjerio nista (MC #107282/#107171/#107242)

Bilko trunk crven 2026-08-16 — FX fail-closed, dva fail-open uslova, i zelen build koji nije mjerio ništa

MC: #107282 (E2E), #107171 (Promote_Demo), #107242 (Build/CI_Gates), #107283 (produkt-rupa, otvoren) PR-ovi: 344, 343, 345 — svi merge-ovani 2026-08-16/17 Main poslije svega: 68be6405

Šta se desilo

Main je bio crven tri builda zaredom — 1089 (13:55), 1091 (15:00), 1092 (18:00) — sva tri na e970edfb, sva tri na istom mjestu: Playwright E2E (chromium, stage), Bash exited with code '1'.

Poruka iz testa:

invoice create failed: {"error":"Exchange rate not found for EUR -> RSD on 2026-08-16.
Add the rate before issuing documents in EUR.","code":"BAD_REQUEST"}
expect(received).toBe(201)

Komit e970edfb (#106892) je namjerno uveo fail-closed na FX kurs. Gate radi tačno kako je projektovan. Problem je bio drugdje.

Korijen — i zašto je bio nevidljiv

storno-credit-note-106104.spec.ts je izdavao fakturu s hardkodiranim currencyCode: 'EUR' (linija 126). Fajl je o storno/credit-note ponašanju: EUR se pojavljivao tačno jednom, a tvrdnji o valuti, kursu ili baznom iznosu nije bilo nijedne. Valuta je bila slučajna — ušla je iz HR/EUR helpera koji koristi i RS/RSD scenario-3 tenant.

Test je prolazio samo dok je kurs slučajno postojao. Fail-closed ga je pretvorio u dnevni pad: svaki dan bez unesenog kursa obara E2E, a E2E je zadnja kapija prije promocije — dakle blokira verifikaciju svakog rada na mainu.

Popravka (PR 344)

Fixture čita valutu organizacije umjesto da je pretpostavlja:

const currencyCode = await getOrganizationBaseCurrency(ctx, token)

Helper zove autentikovani GET /organization, traži HTTP 200 i troslovni kod. Postojeći ugovor, ništa novo: SettingsRoutes.kt izlaže rutu, SettingsService vraća baseCurrency.

Po tenantu: HR/EUR i dalje šalje EUR (to mu jeste bazna valuta), RS/RSD sad šalje RSD. apps/api netaknut, nijedna tvrdnja uklonjena.

Veći nalaz koji ovo NIJE riješilo — #107283

Ne postoji nijedan način da se kurs unese. Provjereno na azdo/main e970edfb:

Dakle korisnik na RSD orgu ne može izdati EUR dokument. Nikad. To je produkt-odluka vlasnika #106892: ruta/admin ekran, scheduled importer (HNB/NBS), ili svjesno suziti fail-closed.

Dva fail-open uslova nađena usput

Ista porodica predikata, dvije različite izloženosti:

linija stage stari uslov izloženost
1229 Promote_Demo not(failed('E2E_UAT')) skip — pao uzvodni stage ⇒ E2E skipped ≠ failed ⇒ uslov se nikad ne razriješi ⇒ stage vječno pending, run vječno inProgress
762 Build not(failed('CI_Gates')) cancel — CI_Gates je prvi stage i nema condition pa se ne može preskočiti; ali otkazan CI_Gates ostavlja uslov istinitim

Prva je proizvela devet duh-runova između 03. i 06.08. koje niko nije mogao razlikovati od legitimnog čekanja na odobrenje: build 879 (Flyway pao → E2E preskočen → promocija nemoguća) i build 1079 (svih pet zeleno, stvarno čeka CEO) izgledali su identično — oba pending.

Druga znači da otkazan CI_Gates može pustiti Build da gurne slike u bilkodemo.azurecr.io (ACR login 783-790, API push 818-826, Web push 858-872) — a Trivy skenira tek poslije pusha.

Obje popravljene istim potezom, u odvojenim PR-ovima radi čistog reverta:

-        not(failed('E2E_UAT')),      →  +        succeeded('E2E_UAT'),
-        not(failed('CI_Gates')),     →  +        succeeded('CI_Gates'),

Time su svi stageovi usklađeni: Flyway_Migrate (924), Deploy_Stage (989), E2E_UAT (1139), Promote_Demo (1229), Build (762) — svi na succeeded().

Lekcija koja se ponavlja: zeleno koje ne mjeri ništa

PR 343 je imao zelen validation build na kodu na kojem je main padao. Ista stvar, suprotan ishod — jer PR-validation buildovi preskaču E2E. Izmjereno, ne pretpostavljeno:

build 1093 (PR 344, zelen):
  [Stage] E2E UAT (Playwright → Stage): result=skipped

Zato nijedan merge ovdje nije pravdan zelenim PR buildom. Dokaz je bio isključivo main build:

build 1092 (prije):  [Task] Playwright E2E (chromium, stage): result=failed
build 1094 (poslije): [Task] Playwright E2E (chromium, stage): result=succeeded

Zamka u alatu koja je skoro proizvela pogrešan izvještaj

azdo-build-logs.sh ispisuje samo taskove koji nose upozorenje ili grešku. Čist task je tamo nevidljiv. U buildu 1094 Playwright se nije pojavio — i u zelenom buildu 1087 se ne pojavljuje. Odsustvo u tom alatu ne razlikuje „prošlo tiho" od „nikad se nije pokrenulo".

Zato je napravljen ~/system/tools/azdo-build-timeline.sh — čita timeline API i daje result= po svakom stageu/jobu/tasku; ako traženog koraka nema, eksplicitno kaže da je odsutan iz timeline-a.

Uz njega ide ~/system/tools/run-azdo-build-wait.sh — blokira do kraja builda pa ispiše stvarni timeline. Postoji zato što je „provjeriću kad build završi" obećanje bez mehanizma: čeka da ga neko podsjeti. Ovako se orkestrator sam probudi.

Redoslijed isporuke (i zašto takav)

  1. PR 344 prvi — jedini koji otvara trunk
  2. main build 1094 → dokaz da E2E stvarno prolazi
  3. PR 343, uz rebase na novi main i requeue validacije (staro zeleno je bilo na drugom commitu)
  4. PR 345, isto rebase-ovan

Org ima jedan self-hosted CI slot, pa se buildovi ne smiju takmičiti; requeue je već ranije obarao žive buildove, zato se prije svakog requeue-a provjerava da ništa ne radi.

Kako je rađeno

Izvršilac: pi-orchestrator (openai-codex/gpt-5.6-sol) preko Company Mesh P2P kanala. Orkestracija i verifikacija: John (Claude). Svaki nalaz je nezavisno provjeren prije nego je prenesen — diff, linije u pinovanom blob-u, ruta i polje u API-ju, timeline builda.

Peer je jednom vratio BLOCKED umjesto da izmisli mehanizam za unos kursa; taj BLOCKED je i otkrio #107283. To je ispravno ponašanje izvršioca i vrijedi više od brzog PASS-a.

P2P isporuke — regwatch pausal izvor (#107169) i cutover runbook (#107170)

P2P isporuke — regwatch paušal izvor (#107169) i cutover runbook (#107170)

Datum: 2026-08-15/16 · Izvršilac: pi-orchestrator (openai-codex/gpt-5.6-sol) preko Company Mesh Orkestracija i nezavisna verifikacija: John (Claude) Dokazi: ~/system/evidence/107169/, ~/system/evidence/107170/

Obje stavke su isporučene i provjerene 15/16.08, ali su predane na review tek 17.08 — kašnjenje je moje, ne izvršiočevo. Zapisano ovdje jer je isporuka bez zatvaranja isto što i nezavršen posao.

#107169 — regwatch nije pratio paušalne razrede

Kvar: regwatch-hr prati mišljenja Porezne, Fiskalizaciju 2.0 i FINA certifikate, ali ne i stranicu s godišnjim paušalnim razredima. Kad se razredi promijene za 2027, Bilkov kalkulator tiho postaje pogrešan i ništa ne alarmira.

Popravka: dodan izvor porezna-obrtnici-pausalisti u SCRAPE_SOURCES (~/system/tools/regwatch-hr.js), po istom obrascu kao postojeća tri (id / label / url).

Šta sam nezavisno provjerio (John), ne preuzeo iz izvještaja:

tvrdnja provjera
URL živ, bez preusmjerenja curl → http=200 redirects=0, 234 KB
stranica stvarno nosi razrede 14 različitih iznosa; izvučen tekst pokazuje sedam razreda od 0–11.300,00 do 50.000,01–60.000,00 eura
upisan u state s hashom hr-state.json → hash c066d659…, lastCheckedAt 2026-08-15T19:37:54Z, url
izvor ne pada na short-body guard tijelo 20.952 znaka poslije stripovanja — prag se ne dira
bez commita/pusha promjena ostavljena u radnom stablu

Ograničenje koje ostaje: run je prošao granom „nema izmjena", koja sendMail uopšte ne zove. Da alarm stvarno stiže dokazuje tek prvi run koji detektuje promjenu. To nije tvrdnja da radi — to je zapisano ograničenje.

Uzgredno potvrđeno: isti run je prošao kroz izmijenjeni sendMail iz #107106 bez pada.

#107170 — cutover runbook je slao na ugašen GCP projekat

Kvar: BookStack stranica 2908 („Bilko Stage→Demo→Prod Cutover Runbook") propisivala je gcloud ... --project=tribal-sign-487920-k0. Bilko je na Azure Container Apps od maja, GCP billing je otkačen 02.08. Operater koji prati runbook dobija greške — ili gore, misli da je promocija prošla a ništa nije izvršeno. Logika i lista gate-ova su bile i dalje tačne; komande su bile mrtve.

Popravka: 20 zastarjelih poziva u 13 blokova zamijenjeno, jednim fail-closed API PUT-om (HTTP 200, revizija 4), pa ponovo dohvaćena stranica radi provjere.

Šta sam nezavisno provjerio (John):

tvrdnja provjera
tribal-sign uklonjen moj re-fetch stranice → 0 pojava
gcloud uklonjen 0
gcr.io uklonjen 0
komande nisu izmišljene az containerapp list -g rg-bilko-demo vraća bilko-api-demo, bilko-web-demo, bilko-api-stage, bilko-web-stage — imena koja stoje u tekstu
org/projekat u az pipelines alai-holding / Bilko poklapa se s git remote-om
rollback postoji backup prije pisanja, 0600, 21.887 B, SHA 3f6cb9b2…, sadrži 16 starih tribal-sign referenci

Najvrednije je ono što nije napisano. Četiri mjesta su označena TODO/UNKNOWN umjesto uvjerljive izmišljotine, i svako od njih je stvarna rupa u operativnoj spremnosti:

  1. nema odobrenog 10/90 gradual rollouta za produkciju
  2. Web app je u Single revision modu → nema rollback puta koji bi se mogao potvrditi
  3. nema sigurne standalone provjere prijavljenog korisnika poslije penzionisanog login toka
  4. nema tenant-safe fixture za invoice-PDF provjeru na produkciji

Stranica 3104 je odbijena kao izvor jer su joj komande iz pred-Azure ere.

Pravilo koje je ovdje potvrđeno

Izvršiocu je u oba zadatka izričito rečeno: ne izmišljaj komandu — ako se ne može utvrditi, piši UNKNOWN. Runbook s tri tačne komande i jednim poštenim UNKNOWN je upotrebljiv; onaj s četiri samouvjerene od kojih je jedna izmišljena je gori od zatečenog stanja.