This Region:document eu-central-1describes (Frankfurt)the Architecture:target EC2deployment (API)architecture +for S3/CloudFrontBilko. (Frontend)Infrastructure +has RDSnot PostgreSQLyet been provisioned.

Table of ContentsOverview

Bilko

1. Prerequisites
uses
2. Architecturea Overview
multi-service
3. First-Timearchitecture Setup
deployed
4. DNSto Recordsdifferent platforms optimized for Cloudflare
each
5. Terraform Deploy
6. EC2 Server Setup
7. SSL/TLS Setup
8. Application Deploy
9. Environment Variables
10. CI/CD Secrets
11. Monitoring
12. Rollback Procedures
13. Cost Estimate

1. Prerequisites

Tools Required (on your local machine)

AWS Account Setup

1. Create an AWS account or use existing ALAI account
2. Create an IAM user with AdministratorAccess for Terraform (only for initial provisioning — restrict after)
3. Configure AWS CLI:

   aws configure
   # Enter: Access Key ID, Secret, region: eu-central-1, output: json
   

4. Verify:

   aws sts get-caller-identity
   

Terraform State Backend (one-time)

Before running Terraform, create the S3 backend resources manually:

# Create S3 bucket for Terraform state
aws s3 mb s3://bilko-terraform-state --region eu-central-1

# Enable versioning on state bucket
aws s3api put-bucket-versioning \
  --bucket bilko-terraform-state \
  --versioning-configuration Status=Enabled

# Enable encryption on state bucket
aws s3api put-bucket-encryption \
  --bucket bilko-terraform-state \
  --server-side-encryption-configuration '{
    "Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]
  }'

# Block public access on state bucket
aws s3api put-public-access-block \
  --bucket bilko-terraform-state \
  --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

# Create DynamoDB table for state locking
aws dynamodb create-table \
  --table-name bilko-terraform-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --region eu-central-1

SSH Key Pair

Create the key pair in AWS Console BEFORE running Terraform:

1. Go to EC2 Console → Key Pairs → Create Key Pair
2. Name: bilko-prod-key
3. Type: ED25519
4. Download the .pem file → save to ~/.ssh/bilko-prod.pem
5. Set permissions: chmod 400 ~/.ssh/bilko-prod.pem

2. Architecture Overview

Internet
    │
    ├─── HTTPS bilko.io ──► CloudFront ──► S3 (Next.js static)
    │
    └─── HTTPS api.bilko.io ──► EC2 Nginx ──► PM2 (Express API) ──► RDS PostgreSQL

Components:component:

Target Domains:

• Primary: bilko.io (all regions)
• Redirect: bilko.rs (Serbia market, 301 redirect to bilko.io)

Deployment Architecture Overview

graph TD
    User["User Browser / Mobile PWA"]
    DNS["Cloudflare DNS\nbilko.io / bilko.rs"]
    CF_PROXY["Cloudflare Proxy\nDDoS + SSL Termination"]
    Vercel["Vercel Edge Network\nNext.js 15 Frontend\nbilko.io"]
    Railway["Railway EU West\nExpress API\napi.bilko.io"]
    PG["PostgreSQL 15\nRailway Managed DB"]
    R2["Cloudflare R2\nbilko-receipts bucket\n(Receipts + PDFs)"]
    SG["SendGrid\nTransactional Email\[email protected]"]

    User --> DNS
    DNS --> CF_PROXY
    CF_PROXY --> Vercel
    CF_PROXY --> Railway
    Railway --> PG
    Railway --> R2
    Railway --> SG
    Vercel -->|"HTTPS API calls"| Railway

Environment Topology

graph LR
    subgraph LOCAL["Local (Developer)"]
        L_WEB["localhost:3000\nNext.js dev"]
        L_API["localhost:4000\nExpress dev"]
        L_DB["localhost:5432\nbilko_dev DB"]
    end

    subgraph STAGING["Staging"]
        S_WEB["bilko-pr-{n}.vercel.app\nVercel Preview"]
        S_API["staging Railway env\nbilko-api-staging"]
        S_DB["Railway PostgreSQL\nbilko_staging DB"]
    end

    subgraph PROD["Production"]
        P_WEB["bilko.io\nVercel Production"]
        P_API["api.bilko.io\nRailway Production"]
        P_DB["Railway PostgreSQL\nbilko_prod DB"]
        P_R2["Cloudflare R2\nbilko-receipts"]
    end

    L_WEB --> L_API --> L_DB
    S_WEB --> S_API --> S_DB
    P_WEB --> P_API --> P_DB
    P_API --> P_R2

3.Frontend First-TimeDeployment Setup(Vercel)

StepPlatform: 1:Vercel

terraform.tfvars
• Why: Zero-config Next.js deployment, edge CDN, automatic HTTPS, preview deployments
• Pricing: Free tier for MVP (handles 10K+ users)
• Region: Global Edge Network (CDN)

Deployment Process

Initial Setup

1. Install Vercel CLI:

   npm install -g vercel
   

2. Login:

   vercel login
   

3. Link project:

   cd infrastructure/terraformapps/web
   cpvercel terraform.tfvars.example terraform.tfvarslink
   

4. EditDeploy terraform.tfvars:to production:

   aws_regionvercel = "eu-central-1"
   environment = "prod"
   
   domain_name   = "bilko.io"
   api_subdomain = "api"
   
   ec2_instance_type = "t3.medium"
   ec2_key_name      = "bilko-prod-key"
   ssh_allowed_cidrs = ["YOUR_OFFICE_IP/32", "YOUR_HOME_IP/32"]
   
   rds_instance_class    = "db.t3.micro"
   rds_allocated_storage = 20
   rds_db_name           = "bilko"
   rds_username          = "bilko_admin"
   rds_password          = "GENERATE_STRONG_PASSWORD_HERE"
   
   api_port = 4000
   node_env = "production"--prod
   

Environment Variables (Vercel Dashboard)

Required environment variables for production:

NEXT_PUBLIC_API_URL=https://api.bilko.io
NEXT_PUBLIC_APP_ENV=production

Custom Domain Setup

1. Go to Vercel Dashboard → Project Settings → Domains
2. Add custom domain: bilko.io
3. Configure DNS (see DNS section below)
4. Vercel auto-provisions SSL certificate

Preview Deployments

• Every PR automatically deploys to unique URL: bilko-pr-{number}.vercel.app
• Allows testing before merging to main
• Automatically deleted after PR merge/close

Backend Deployment (Railway)

Platform: Railway

• Important:Why: NeverPostgreSQL commitincluded, terraform.tfvarssimple deployment, EU region available (GDPR), affordable
• Pricing: ~€20/mo for MVP (starter tier: 2GB RAM, 2 vCPU, 10GB storage)
• Region: EU West (Frankfurt or Paris — itGDPR iscompliant)
in

Deployment Process

Initial Setup

1. Install Railway CLI:

   .gitignorenpm install -g @railway/cli
   .

   Generate
password:

2. Login:

   railway login
   

3. Create project:

   railway init
   

4. Provision PostgreSQL:

   railway add --database postgresql
   

5. Deploy API:

   cd apps/api
   railway up
   

Environment Variables (Railway Dashboard)

Required environment variables for production:

# Database (auto-provisioned by Railway)
DATABASE_URL=${{Postgres.DATABASE_URL}}

# JWT Secrets (generate with: openssl rand -base64 3232)
JWT_SECRET=<your-secret-here>
JWT_REFRESH_SECRET=<your-refresh-secret-here>

# SendGrid Email
SENDGRID_API_KEY=<your-sendgrid-key>

# Cloudflare R2 Storage
R2_ACCESS_KEY_ID=<your-r2-key>
R2_SECRET_ACCESS_KEY=<your-r2-secret>
R2_BUCKET_NAME=bilko-receipts
R2_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com

# App Config
PORT=4000
NODE_ENV=production
ALLOWED_ORIGINS=https://bilko.io,https://www.bilko.io

Database Migrations

Run Prisma migrations on Railway:

railway run npx prisma migrate deploy

Monitoring

Railway provides built-in monitoring:

• CPU, memory, network usage
• Logs: railway logs
• Metrics: Railway Dashboard → Metrics tab

4.File DNSStorage Records(Cloudflare R2)

Platform: Cloudflare R2

• Why: Zero egress fees (S3 charges for Cloudflaredownloads),

  S3-compatible API

• Context:Pricing: AWS Route53 is used~€1/mo for ACMMVP certificate(storage: DNS€0.015/GB, validationno only.egress Allfees)
public
• Use DNSCases: isReceipt managedphotos in(JPG, Cloudflare.
  PNG), invoice PDFs, expense attachments

AfterSetup Terraform ApplyProcess

1. Create R2 Bucket

1. Go to Cloudflare Dashboard → R2
2. Create bucket: bilko-receipts
3. Note Account ID from dashboard

2. Generate API Credentials

1. R2 → Manage R2 API Tokens
2. Create API token with permissions:
   • Object Read & Write
   • Bucket scope: bilko-receipts
3. Save Access Key ID and Secret Access Key

3. Configure CORS

GetAllow theuploads valuesfrom you need:frontend:

cd{
  infrastructure/terraform"AllowedOrigins": terraform["https://bilko.io", output"https://www.bilko.io"],
  cloudfront_domain_name"AllowedMethods": #["GET", e.g."PUT", d1abc123xyz.cloudfront.net"POST", terraform"DELETE"],
  output"AllowedHeaders": ec2_public_ip["*"],
  #"MaxAgeSeconds": e.g.3600
52.29.x.x}
terraform

Apply nameserversvia #Cloudflare Route53Dashboard NS→ recordsR2 → bilko-receipts → Settings → CORS

4. Backend Integration

Use AWS SDK for S3 (forR2 ACMis validationS3-compatible):

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

const s3Client = new S3Client({
  region: "auto",
  endpoint: process.env.R2_ENDPOINT,
  credentials: {
    accessKeyId: process.env.R2_ACCESS_KEY_ID!,
    secretAccessKey: process.env.R2_SECRET_ACCESS_KEY!,
  },
});

DNS Configuration (Cloudflare)

DNS + Traffic Routing Flow

flowchart TD
    REQ["Incoming Request"]
    CHK_RS{{"bilko.rs\nor www.bilko.rs?"}}
    REDIRECT["301 Permanent Redirect\nhttps://bilko.io$request.uri"]
    CHK_API{{"api.bilko.io?"}}
    VERCEL_CNAME["CNAME → cname.vercel-dns.com\n(Proxied via Cloudflare)"]
    RAIL_CNAME["CNAME → railway.app\n(DNS only, no proxy)"]
    SSL_VERCEL["Vercel SSL\n(Let's Encrypt auto-renew)"]
    SSL_RAIL["Railway SSL\n(Let's Encrypt auto-renew)"]

    REQ --> CHK_RS
    CHK_RS -->|Yes| REDIRECT
    CHK_RS -->|No| CHK_API
    CHK_API -->|Yes| RAIL_CNAME --> SSL_RAIL
    CHK_API -->|No| VERCEL_CNAME --> SSL_VERCEL

CloudflareDomains

• bilko.io — Primary domain (registered, DNS not configured)
• bilko.rs — Serbia redirect (not yet registered)

DNS Records to(Cloudflare CreateDashboard)

Log in to Cloudflare →

bilko.io → DNS:

Note on CloudFront + Cloudflare: CloudFront does NOT have a single IP — it's a CDN with many IPs. The correct approach is:

• Set the root domain CNAME

  bilko.rs (or use Cloudflare's CNAME flattening for root):

  • Type: CNAME, Name: @, Value: <cloudfront_domain>.cloudfront.net, Proxied: ON

Why api must be DNS-only (grey cloud):

• certbot's HTTP challenge (/.well-known/acme-challenge/) requires direct access to EC2
• Cloudflare proxy would intercept the challenge and break SSL provisioning
• After cert is obtained, you can optionally enable proxy for DDoS protection (but Nginx already handles TLS)

ACM Certificate DNS Validation

When Terraform creates the ACM certificate, it outputs validation CNAME records that must be added to Cloudflare (NOT Route53, since Cloudflare is authoritative for bilko.io):

# Get validation records
cd infrastructure/terraform
terraform output -json | jq '.cert_validation_records'

Add these CNAMEs to Cloudflare DNS. They look like:

Type: CNAME
Name: _abc123.bilko.io
Value: _xyz456.acm-validations.aws
Proxy: DNS only (grey)

Wait 5-10 minutes for ACM to validate. Check status:

aws acm describe-certificate \
  --certificate-arn $(terraform output -raw acm_certificate_arn) \
  --region us-east-1 \
  --query "Certificate.Status"

Note: The Terraform aws_route53_record.cert_validation resources create validation records in Route53. Since your authoritative DNS is Cloudflare (not Route53), you must add these CNAMEs in Cloudflare manually. Alternatively, delegate the bilko.io zone to Route53 by pointing Cloudflare nameservers to Route53 NS records — but that gives AWS full DNS control.

Recommended approach: Keep Cloudflare. Manually add the ACM validation CNAMEs there.

5. Terraform Deploy

cd infrastructure/terraform

# Initialize
terraform init

# Preview
terraform plan -var-file="terraform.tfvars"

# Apply (takes ~10 minutes)
terraform apply -var-file="terraform.tfvars"

What Gets Created

• VPC with public/private subnets in eu-central-1a and eu-central-1b
• EC2 t3.medium (Amazon Linux 2023) with Elastic IP
• RDS PostgreSQL 15 (db.t3.micro) in private subnet
• S3 bucket for static web (bilko-web-prod) — private, OAC
• S3 bucket for file storage (bilko-storage-prod) — private, encrypted, versioned
• CloudFront distribution with ACM certificate
• ACM certificate (us-east-1, for CloudFront)
• Route53 hosted zone (for ACM validation records)
• CloudWatch alarms for EC2 and RDS
• SNS topic for alarm notifications
• IAM role + instance profile for EC2 to access S3

Post-Apply: Subscribe to Alarms

# Subscribe your email to receive alarm notifications
aws sns subscribe \
  --topic-arn $(terraform output -raw sns_alarms_topic_arn) \
  --protocol email \
  --notification-endpoint [email protected]
# Check email and confirm subscription

6. EC2 Server Setup

After Terraform provisions the EC2 instance, run the setup script:

EC2_IP=$(cd infrastructure/terraform && terraform output -raw ec2_public_ip)

# Run setup script
ssh -i ~/.ssh/bilko-prod.pem ec2-user@$EC2_IP 'bash -s' < infrastructure/scripts/setup-ec2.sh

Then clone the repository

ssh -i ~/.ssh/bilko-prod.pem ec2-user@$EC2_IP

# On the EC2 instance:
git clone https://github.com/ALAI-Holding/bilko.git /opt/bilko
cd /opt/bilko

Copy Nginx config

# On EC2:
sudo cp /opt/bilko/infrastructure/nginx/bilko-api.conf /etc/nginx/conf.d/bilko-api.conf
sudo nginx -t
sudo systemctl restart nginx

7. SSL/TLS Setup

Frontend (bilko.io)

SSL is handled by ACM + CloudFront. No action needed — it's automated by Terraform.

API (api.bilko.io)

SSL is handled by Let's Encrypt on EC2 via certbot.

Prerequisites: Cloudflare DNS record for api.bilko.io must be pointing to the EC2 IP and DNS must have propagated.

# On EC2:
sudo certbot --nginx -d api.bilko.io \
  --non-interactive \
  --agree-tos \
  -m [email protected]

Certbot auto-renewal is handled by the systemd timer (installed by default on Amazon Linux 2023):

# Verify auto-renewal is active
sudo systemctl status certbot-renew.timer

# Test renewal (dry run)
sudo certbot renew --dry-run

8. Application Deploy

First Deploy (manual)

# On EC2 at /opt/bilko:

# Install dependencies
npm ci

# Create .env file
cp apps/api/.env.example apps/api/.env
# Edit with production values (see Environment Variables section)
nano apps/api/.env

# Generate Prisma client
npx prisma generate --schema=packages/database/prisma/schema.prisma

# Run migrations
npx prisma migrate deploy --schema=packages/database/prisma/schema.prisma

# Build API
npm run build --workspace=@bilko/api

# Start with PM2
pm2 start infrastructure/pm2/ecosystem.config.js
pm2 save

# Verify
pm2 status
curl http://localhost:4000/api/v1/health

Web (Frontend) First Deploy

# Locally:
export CLOUDFRONT_DISTRIBUTION_ID=$(cd infrastructure/terraform && terraform output -raw cloudfront_distribution_id)
export AWS_REGION=eu-central-1

./infrastructure/scripts/deploy-web.sh prod

Subsequent Deploys

API: Use CI/CD (push to main branch triggers deploy-prod.yml — manual approval required) Web: Same CI/CD pipeline

Manual API deploy:

EC2_HOST=<ec2-ip> SSH_KEY=~/.ssh/bilko-prod.pem ./infrastructure/scripts/deploy-api.sh prod

Manual migration:

EC2_HOST=<ec2-ip> SSH_KEY=~/.ssh/bilko-prod.pem ./infrastructure/scripts/db-migrate.sh prod

9. Environment Variables

apps/api/.env (on EC2)

# Database
DATABASE_URL="postgresql://bilko_admin:<PASSWORD>@<RDS_ENDPOINT>:5432/bilko"

# JWT (generate with: openssl rand -base64 64)
JWT_SECRET="<GENERATE_64_CHAR_RANDOM_SECRET>"
JWT_REFRESH_SECRET="<GENERATE_DIFFERENT_64_CHAR_SECRET>"
JWT_EXPIRES_IN="15m"
JWT_REFRESH_EXPIRES_IN="7d"

# App
PORT=4000
NODE_ENV=production

# CORS
ALLOWED_ORIGINS="https://bilko.io,https://www.bilko.io"

# AWS S3 (using EC2 instance profile — no access keys needed)
AWS_REGION=eu-central-1
S3_STORAGE_BUCKET=bilko-storage-prod

# Email (for sending invoices, notifications)
# TODO: Configure SES or SMTP provider
SMTP_HOST=
SMTP_PORT=
SMTP_USER=
SMTP_PASS=
SMTP_FROM="[email protected]"

Get RDS endpoint:

cd infrastructure/terraform && terraform output rds_endpoint

Important: Never commit .env to git. The file is in .gitignore.

GitHub Actions Secrets

Configure these in GitHub → Settings → Secrets and Variables → Actions:

10. CI/CD Secrets

Required GitHub Secrets

GitHubRedirect EnvironmentRules (Cloudflare)

Create redirect rule for bilko.rs → bilko.io:

• If: Hostname equals bilko.rs or www.bilko.rs
• Then: Dynamic redirect to https://bilko.io$request.uri (301 permanent)

Email Configuration (SendGrid)

Platform: SendGrid

• Why: Free tier (100 emails/day), email templates, delivery tracking
• Pricing: €0 for MVP (upgrade to €15/mo for 40K emails when needed)

Setup Process

1. Create SendGrid Account

1. GoSign toup GitHubat repositorysendgrid.com
2. Verify email address
3. Create API key with Mail Send permissions

2. Domain Authentication

1. SendGrid Dashboard → Settings → EnvironmentsSender Authentication
2. CreateAuthenticate domain: stagingbilko.io environment
3. Create production environment — add required reviewers for approval gate
4. Add secretsDNS records to eachCloudflare environment(CNAME for DKIM, TXT for SPF)

Creating

3. IAMEmail UserTemplates

Create GitHubtransactional Actionsemail templates:

• Welcome email (S3/CloudFrontuser only)registration)
• Invoice sent notification
• Payment received confirmation
• Password reset
• 2FA code

4. Backend Integration

Use SendGrid Node.js SDK:

#import Create deployment user (separatesgMail from EC2"@sendgrid/mail";
rolesgMail.setApiKey(process.env.SENDGRID_API_KEY!);

—await narrower permissions)
aws iam create-user --user-name bilko-github-deployer

aws iam put-user-policy \
  --user-name bilko-github-deployer \
  --policy-name bilko-deploy-policy \
  --policy-document 'sgMail.send({
  "Version":to: "2012-10-17"[email protected]",
  from: "Statement":[email protected]",
  [subject: "Invoice #12345",
  templateId: "d-abc123...",
  dynamicTemplateData: { "Effect":invoiceNumber: "Allow",
        "Action": ["s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetObject"],
        "Resource": ["arn:aws:s3:::bilko-web-prod", "arn:aws:s3:::bilko-web-prod/*",
                     "arn:aws:s3:::bilko-web-staging", "arn:aws:s3:::bilko-web-staging/*"]12345" },
{
        "Effect": "Allow",
        "Action": ["cloudfront:CreateInvalidation"],
        "Resource": "*"
      }
    ]
  }'

aws iam create-access-key --user-name bilko-github-deployer
# Copy AccessKeyId and SecretAccessKey → add to GitHub secrets);

11. Monitoring

CloudWatch Alarms

The following alarms are configured (SNS email notifications):

CloudWatch Dashboards

Access via AWS Console → CloudWatch → Dashboards.

Key metrics to watch:

• EC2: CPUUtilization, NetworkIn/Out, StatusCheckFailed
• RDS: CPUUtilization, FreeStorageSpace, DatabaseConnections, ReadLatency, WriteLatency
• CloudFront: Requests, BytesDownloaded, 4xxErrorRate, 5xxErrorRate

Application Logs

# PM2 logs (on EC2)
pm2 logs bilko-api
pm2 logs bilko-api --lines 100

# Log files
tail -f /var/log/bilko/api-out.log
tail -f /var/log/bilko/api-error.log

# Nginx logs
tail -f /var/log/nginx/bilko-api-access.log
tail -f /var/log/nginx/bilko-api-error.log

Health Check Endpoints

12. Rollback Procedures

API Rollback

Automatic: If health check fails during deploy, the deploy script automatically rolls back to the previous git commit.

Manual rollback:

ssh -i ~/.ssh/bilko-prod.pem ec2-user@<EC2_IP>

cd /opt/bilko

# Find last known good commit
git log --oneline -10

# Roll back
git checkout <COMMIT_HASH>
npm ci
npx prisma generate --schema=packages/database/prisma/schema.prisma
npm run build --workspace=@bilko/api
pm2 reload bilko-api --update-env

# Verify
curl http://localhost:4000/api/v1/health

Note on migrations: Database migrations cannot be automatically rolled back with Prisma (migrate deploy is forward-only). If a migration broke production:

1. Roll back the application code (above)
2. Write a new migration that undoes the schema changes
3. Deploy the new migration manually

Frontend Rollback

CloudFront serves S3 content. To roll back:

# Option 1: Re-deploy previous git tag
git checkout <PREVIOUS_TAG>
./infrastructure/scripts/deploy-web.sh prod

# Option 2: If previous build artifacts are available
aws s3 sync s3://bilko-web-backup s3://bilko-web-prod --delete
aws cloudfront create-invalidation \
  --distribution-id <DISTRIBUTION_ID> \
  --paths "/*"

Database Point-in-Time Recovery

RDS has automated backups with 7-day retention. To restore:

# List available restore points
aws rds describe-db-instances \
  --db-instance-identifier bilko-prod-db \
  --query "DBInstances[0].LatestRestorableTime"

# Restore to point in time (creates NEW instance)
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier bilko-prod-db \
  --target-db-instance-identifier bilko-prod-db-restored \
  --restore-time "2026-02-23T10:00:00Z"

13. Cost Estimate

Monthly

MVP AWS costsBudget (eu-central-1, approximate):

Monthly)

Scale-up Budget (1K-10K users)

Deployment Checklist

Development Environment

•  Local PostgreSQL 15+ installed
•  Node.js 18+ installed
•  Environment variables in .env files
•  Database migrated: npx prisma migrate dev
•  Frontend runs: npm run dev (port 3000)
•  Backend runs: npm run dev (port 4000)

Staging Environment

•  Railway project created
•  PostgreSQL provisioned on Railway
•  Environment variables configured
•  Database migrated: railway run npx prisma migrate deploy
•  Backend deployed: railway up
•  Vercel project created
•  Frontend deployed: vercel
•  Custom domain configured: staging.bilko.io
•  End-to-end smoke test passed

Production Environment

•  DNS configured (bilko.io → Vercel + Railway)
•  SSL certificates provisioned (automatic)
•  Environment variables configured (production secrets)
•  Database migrated: railway run npx prisma migrate deploy
•  Cloudflare R2 bucket created
•  CORS configured on R2
•  SendGrid domain authenticated
•  Email templates created
•  Backend deployed: railway up --production
•  Frontend deployed: vercel --prod
•  Monitoring configured (Railway metrics + Sentry)
•  Backup strategy tested
•  Load testing completed (1K concurrent users)
•  Incident response plan documented

Rollback Strategy

Rollback Decision Flow

flowchart TD
    INC["Incident Detected"]
    TYPE{{"Affected\nComponent?"}}

    FE_ROLL["Vercel Dashboard\n→ Deployments\n→ Promote to Production\n⏱ Instant (no rebuild)"]
    BE_ROLL["Railway Dashboard\n→ Deployments\n→ Redeploy previous\n⏱ ~2 minutes"]
    DB_ROLL{{"Data\nCorrupted?"}}
    DB_FORWARD["Forward Fix\n(preferred)\nDeploy corrective migration"]
    DB_RESTORE["Railway Backup Restore\n→ Select snapshot\n→ Test integrity\n→ Switch traffic"]

    INC --> TYPE
    TYPE -->|"Frontend only"| FE_ROLL
    TYPE -->|"Backend only"| BE_ROLL
    TYPE -->|"Database"| DB_ROLL
    DB_ROLL -->|No| DB_FORWARD
    DB_ROLL -->|Yes| DB_RESTORE

Frontend Rollback (Vercel)

Vercel keeps all previous deployments. To rollback:

1. Go to Vercel Dashboard → Deployments
2. Find previous working deployment
3. Click "Promote to Production"
4. Instant rollback (no rebuild needed)

Backend Rollback (Railway)

Railway keeps deployment history. To rollback:

1. Go to Railway Dashboard → Deployments
2. Select previous deployment
3. Click "Redeploy"
4. Rollback completes in ~2 minutes

Database Rollback

CostCRITICAL: optimizationDatabase tips:rollbacks are risky. Prefer forward-fixing.

If necessary:

1. Restore from Railway automated backup
2. Re-run migrations up to previous version
3. Test data integrity before switching traffic

Prevention:

• RDSAlways db.t3.microtest handlesmigrations upon tostaging ~80 connections — sufficient for early stagefirst
• CloudFrontUse PriceClass_100backward-compatible limits to US + Europe (covers Balkan region via Frankfurt PoP)changes
• EC2Never Reserveddrop Instancecolumns (1-year)without wouldmulti-step reducemigration cost by ~40% ($35 → ~$21)
• When traffic grows: scale EC2 vertically (t3.large) before adding ALB + Auto Scalingplan

Monitoring & Alerts

Railway Monitoring

StagingBuilt-in environmentmetrics:

• CPU usage
• Memory usage
• Network I/O
• Request rate
• Error rate

Vercel Analytics

Built-in analytics:

• Page views
• Core Web Vitals
• Geographic distribution
• Device breakdown

Uptime Monitoring (ifRecommended: needed):Uptime AddRobot)

Free fortier separatemonitors:

t3.small
• Frontend +(bilko.io)
RDS
• Backend db.t3.micro.(api.bilko.io/health)
• Check interval: 5 minutes
• Alert via email/Slack on downtime

Error Tracking (Recommended: Sentry)

Integrate Sentry for:

• JavaScript errors (frontend)
• API errors (backend)
• Performance monitoring
• Release tracking

Appendix:Security QuickConsiderations

HTTPS Commands

• Vercel: Automatic HTTPS
• Railway: Automatic HTTPS
• Cloudflare: Force HTTPS redirect

Secrets Management

• NEVER commit secrets to git
• Store in platform dashboards (Vercel, Railway)
• Rotate JWT secrets quarterly
• Rotate API keys annually

CORS Configuration

Restrict to production domains only:

#const SSHcorsOptions to= EC2{
  sshorigin: -iprocess.env.ALLOWED_ORIGINS.split(","),
  ~/.ssh/bilko-prod.pemcredentials: ec2-user@<EC2_IP>true,
# Check API health
curl https://api.bilko.io/api/v1/health

# PM2 status
pm2 status

# View recent logs
pm2 logs bilko-api --lines 50

# Restart API
pm2 reload bilko-api

# Nginx status
sudo systemctl status nginx

# Certbot renewal check
sudo certbot renew --dry-run

# Terraform outputs
cd infrastructure/terraform && terraform output

# Manual API deploy
EC2_HOST=<IP> SSH_KEY=~/.ssh/bilko-prod.pem ./infrastructure/scripts/deploy-api.sh prod

# Manual web deploy
CLOUDFRONT_DISTRIBUTION_ID=<ID> ./infrastructure/scripts/deploy-web.sh prod

# Manual DB migration
EC2_HOST=<IP> SSH_KEY=~/.ssh/bilko-prod.pem ./infrastructure/scripts/db-migrate.sh prod};

Rate Limiting

Protect API endpoints:

• General: 100 req/min per IP
• Auth: 5 req/min per IP
• Reports: 10 req/min per user

Backup & Disaster Recovery

Database Backups (Railway)

• Automated: Daily backups, 30-day retention
• Manual: Export via pg_dump before major migrations
• Restore: Railway Dashboard → Database → Backups → Restore

File Backups (Cloudflare R2)

• Strategy: Receipts/PDFs linked in database, R2 is source of truth
• Retention: Indefinite (regulatory requirement for financial records)
• Replication: Enable R2 multi-region replication (future)

Recovery Time Objective (RTO)

• Target: 1 hour (from incident to service restored)
• Components:
  • Database restore: 15 minutes
  • Backend redeploy: 5 minutes
  • Frontend redeploy: 2 minutes
  • DNS propagation: 5 minutes (cached)

Recovery Point Objective (RPO)

• Target: 24 hours (maximum data loss acceptable)
• Achieved via: Daily database backups

Related Documents

• CI/CD Pipeline: CI-CD.md
• Environment Setup: ENVIRONMENT.md
• Security Architecture: ../security/SECURITY-ARCHITECTURE.md
• Testing Guide: ../testing/TESTING-GUIDE.md

Last Updated: 2026-02-20 Status: PLANNED — No infrastructure deployed yet Next Steps: Provision Railway project + PostgreSQL, deploy staging environment