Deployment Guide

Bilko — Deployment Guide

Domain: bilko.io DNS Provider: Cloudflare (managed by CEO) AWS Region: eu-central-1 (Frankfurt) Architecture: EC2 (API) + S3/CloudFront (Frontend) + RDS PostgreSQL

Prerequisites
Architecture Overview
First-Time Setup
DNS Records for Cloudflare
Terraform Deploy
EC2 Server Setup
SSL/TLS Setup
Application Deploy
Environment Variables
CI/CD Secrets
Monitoring
Rollback Procedures
Cost Estimate

1. Prerequisites

Tools Required (on your local machine)

Tool	Version	Install
Terraform	>= 1.5.0	`brew install terraform`
AWS CLI	>= 2.x	`brew install awscli`
Node.js	20.x	`brew install node@20`
Docker	Latest	Docker Desktop
SSH key	—	Generated in AWS Console

AWS Account Setup

Create an AWS account or use existing ALAI account
Create an IAM user with AdministratorAccess for Terraform (only for initial provisioning — restrict after)

Configure AWS CLI:

aws configure
# Enter: Access Key ID, Secret, region: eu-central-1, output: json

Verify:
```
aws sts get-caller-identity
```

Terraform State Backend (one-time)

Before running Terraform, create the S3 backend resources manually:

# Create S3 bucket for Terraform state
aws s3 mb s3://bilko-terraform-state --region eu-central-1

# Enable versioning on state bucket
aws s3api put-bucket-versioning \
  --bucket bilko-terraform-state \
  --versioning-configuration Status=Enabled

# Enable encryption on state bucket
aws s3api put-bucket-encryption \
  --bucket bilko-terraform-state \
  --server-side-encryption-configuration '{
    "Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]
  }'

# Block public access on state bucket
aws s3api put-public-access-block \
  --bucket bilko-terraform-state \
  --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

# Create DynamoDB table for state locking
aws dynamodb create-table \
  --table-name bilko-terraform-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --region eu-central-1

SSH Key Pair

Create the key pair in AWS Console BEFORE running Terraform:

Go to EC2 Console → Key Pairs → Create Key Pair
Name: bilko-prod-key
Type: ED25519
Download the .pem file → save to ~/.ssh/bilko-prod.pem
Set permissions: chmod 400 ~/.ssh/bilko-prod.pem

2. Architecture Overview

Internet
    │
    ├─── HTTPS bilko.io ──► CloudFront ──► S3 (Next.js static)
    │
    └─── HTTPS api.bilko.io ──► EC2 Nginx ──► PM2 (Express API) ──► RDS PostgreSQL

Components:

Component	Service	Purpose
Frontend	CloudFront + S3	Next.js static export, global CDN
API server	EC2 t3.medium	Express API, PM2 cluster mode
Database	RDS PostgreSQL 15 (db.t3.micro)	Primary data store, private subnet
SSL (frontend)	ACM + CloudFront	Managed by AWS, auto-renewing
SSL (API)	Let's Encrypt via certbot	Managed by certbot on EC2
File storage	S3 (bilko-storage-prod)	Receipts, invoice PDFs — private
DNS	Route53 (AWS) + Cloudflare	Route53 for ACM validation; Cloudflare for public DNS
Monitoring	CloudWatch + SNS	CPU, memory, 5xx alarms

3. First-Time Setup

Step 1: Create terraform.tfvars

cd infrastructure/terraform
cp terraform.tfvars.example terraform.tfvars

Edit terraform.tfvars:

aws_region  = "eu-central-1"
environment = "prod"

domain_name   = "bilko.io"
api_subdomain = "api"

ec2_instance_type = "t3.medium"
ec2_key_name      = "bilko-prod-key"
ssh_allowed_cidrs = ["YOUR_OFFICE_IP/32", "YOUR_HOME_IP/32"]

rds_instance_class    = "db.t3.micro"
rds_allocated_storage = 20
rds_db_name           = "bilko"
rds_username          = "bilko_admin"
rds_password          = "GENERATE_STRONG_PASSWORD_HERE"

api_port = 4000
node_env = "production"

Important: Never commit terraform.tfvars — it is in .gitignore. Generate password: openssl rand -base64 32

4. DNS Records for Cloudflare

Context: AWS Route53 is used for ACM certificate DNS validation only. All public DNS is managed in Cloudflare.

After Terraform Apply

Get the values you need:

cd infrastructure/terraform
terraform output cloudfront_domain_name   # e.g. d1abc123xyz.cloudfront.net
terraform output ec2_public_ip            # e.g. 52.29.x.x
terraform output nameservers              # Route53 NS records (for ACM validation only)

Cloudflare DNS Records to Create

Type	Name	Value	Proxy	TTL	Notes
A	`@` (bilko.io)	CloudFront IP*	Proxied (orange)	Auto	Root domain → CloudFront
CNAME	`www`	`<cloudfront_domain>.cloudfront.net`	Proxied (orange)	Auto	www → CloudFront
A	`api`	`<ec2_public_ip>`	DNS only (grey)	300	API → EC2

Note on CloudFront + Cloudflare: CloudFront does NOT have a single IP — it's a CDN with many IPs. The correct approach is:

Set the root domain CNAME (or use Cloudflare's CNAME flattening for root):
- Type: CNAME, Name: @, Value: <cloudfront_domain>.cloudfront.net, Proxied: ON

Type	Name	Value	Proxy	Notes
CNAME	`@`	`<cloudfront_domain>.cloudfront.net`	Proxied ON	Cloudflare flattens CNAME at root
CNAME	`www`	`<cloudfront_domain>.cloudfront.net`	Proxied ON	www redirect
A	`api`	`<ec2_elastic_ip>`	DNS only (grey cloud)	Must be grey — certbot needs direct access

Why api must be DNS-only (grey cloud):

certbot's HTTP challenge (/.well-known/acme-challenge/) requires direct access to EC2
Cloudflare proxy would intercept the challenge and break SSL provisioning
After cert is obtained, you can optionally enable proxy for DDoS protection (but Nginx already handles TLS)

ACM Certificate DNS Validation

When Terraform creates the ACM certificate, it outputs validation CNAME records that must be added to Cloudflare (NOT Route53, since Cloudflare is authoritative for bilko.io):

# Get validation records
cd infrastructure/terraform
terraform output -json | jq '.cert_validation_records'

Add these CNAMEs to Cloudflare DNS. They look like:

Type: CNAME
Name: _abc123.bilko.io
Value: _xyz456.acm-validations.aws
Proxy: DNS only (grey)

Wait 5-10 minutes for ACM to validate. Check status:

aws acm describe-certificate \
  --certificate-arn $(terraform output -raw acm_certificate_arn) \
  --region us-east-1 \
  --query "Certificate.Status"

Note: The Terraform aws_route53_record.cert_validation resources create validation records in Route53. Since your authoritative DNS is Cloudflare (not Route53), you must add these CNAMEs in Cloudflare manually. Alternatively, delegate the bilko.io zone to Route53 by pointing Cloudflare nameservers to Route53 NS records — but that gives AWS full DNS control.

Recommended approach: Keep Cloudflare. Manually add the ACM validation CNAMEs there.

5. Terraform Deploy

cd infrastructure/terraform

# Initialize
terraform init

# Preview
terraform plan -var-file="terraform.tfvars"

# Apply (takes ~10 minutes)
terraform apply -var-file="terraform.tfvars"

What Gets Created

VPC with public/private subnets in eu-central-1a and eu-central-1b
EC2 t3.medium (Amazon Linux 2023) with Elastic IP
RDS PostgreSQL 15 (db.t3.micro) in private subnet
S3 bucket for static web (bilko-web-prod) — private, OAC
S3 bucket for file storage (bilko-storage-prod) — private, encrypted, versioned
CloudFront distribution with ACM certificate
ACM certificate (us-east-1, for CloudFront)
Route53 hosted zone (for ACM validation records)
CloudWatch alarms for EC2 and RDS
SNS topic for alarm notifications
IAM role + instance profile for EC2 to access S3

Post-Apply: Subscribe to Alarms

# Subscribe your email to receive alarm notifications
aws sns subscribe \
  --topic-arn $(terraform output -raw sns_alarms_topic_arn) \
  --protocol email \
  --notification-endpoint [email protected]
# Check email and confirm subscription

6. EC2 Server Setup

After Terraform provisions the EC2 instance, run the setup script:

EC2_IP=$(cd infrastructure/terraform && terraform output -raw ec2_public_ip)

# Run setup script
ssh -i ~/.ssh/bilko-prod.pem ec2-user@$EC2_IP 'bash -s' < infrastructure/scripts/setup-ec2.sh

Then clone the repository

ssh -i ~/.ssh/bilko-prod.pem ec2-user@$EC2_IP

# On the EC2 instance:
git clone https://github.com/ALAI-Holding/bilko.git /opt/bilko
cd /opt/bilko

Copy Nginx config

# On EC2:
sudo cp /opt/bilko/infrastructure/nginx/bilko-api.conf /etc/nginx/conf.d/bilko-api.conf
sudo nginx -t
sudo systemctl restart nginx

7. SSL/TLS Setup

Frontend (bilko.io)

SSL is handled by ACM + CloudFront. No action needed — it's automated by Terraform.

API (api.bilko.io)

SSL is handled by Let's Encrypt on EC2 via certbot.

Prerequisites: Cloudflare DNS record for api.bilko.io must be pointing to the EC2 IP and DNS must have propagated.

# On EC2:
sudo certbot --nginx -d api.bilko.io \
  --non-interactive \
  --agree-tos \
  -m [email protected]

Certbot auto-renewal is handled by the systemd timer (installed by default on Amazon Linux 2023):

# Verify auto-renewal is active
sudo systemctl status certbot-renew.timer

# Test renewal (dry run)
sudo certbot renew --dry-run

8. Application Deploy

First Deploy (manual)

# On EC2 at /opt/bilko:

# Install dependencies
npm ci

# Create .env file
cp apps/api/.env.example apps/api/.env
# Edit with production values (see Environment Variables section)
nano apps/api/.env

# Generate Prisma client
npx prisma generate --schema=packages/database/prisma/schema.prisma

# Run migrations
npx prisma migrate deploy --schema=packages/database/prisma/schema.prisma

# Build API
npm run build --workspace=@bilko/api

# Start with PM2
pm2 start infrastructure/pm2/ecosystem.config.js
pm2 save

# Verify
pm2 status
curl http://localhost:4000/api/v1/health

Web (Frontend) First Deploy

# Locally:
export CLOUDFRONT_DISTRIBUTION_ID=$(cd infrastructure/terraform && terraform output -raw cloudfront_distribution_id)
export AWS_REGION=eu-central-1

./infrastructure/scripts/deploy-web.sh prod

Subsequent Deploys

API: Use CI/CD (push to main branch triggers deploy-prod.yml — manual approval required) Web: Same CI/CD pipeline

Manual API deploy:

EC2_HOST=<ec2-ip> SSH_KEY=~/.ssh/bilko-prod.pem ./infrastructure/scripts/deploy-api.sh prod

Manual migration:

EC2_HOST=<ec2-ip> SSH_KEY=~/.ssh/bilko-prod.pem ./infrastructure/scripts/db-migrate.sh prod

9. Environment Variables

apps/api/.env (on EC2)

# Database
DATABASE_URL="postgresql://bilko_admin:<PASSWORD>@<RDS_ENDPOINT>:5432/bilko"

# JWT (generate with: openssl rand -base64 64)
JWT_SECRET="<GENERATE_64_CHAR_RANDOM_SECRET>"
JWT_REFRESH_SECRET="<GENERATE_DIFFERENT_64_CHAR_SECRET>"
JWT_EXPIRES_IN="15m"
JWT_REFRESH_EXPIRES_IN="7d"

# App
PORT=4000
NODE_ENV=production

# CORS
ALLOWED_ORIGINS="https://bilko.io,https://www.bilko.io"

# AWS S3 (using EC2 instance profile — no access keys needed)
AWS_REGION=eu-central-1
S3_STORAGE_BUCKET=bilko-storage-prod

# Email (for sending invoices, notifications)
# TODO: Configure SES or SMTP provider
SMTP_HOST=
SMTP_PORT=
SMTP_USER=
SMTP_PASS=
SMTP_FROM="[email protected]"

Get RDS endpoint:

cd infrastructure/terraform && terraform output rds_endpoint

Important: Never commit .env to git. The file is in .gitignore.

GitHub Actions Secrets

Configure these in GitHub → Settings → Secrets and Variables → Actions:

10. CI/CD Secrets

Required GitHub Secrets

Secret	Value	Environment
`EC2_HOST`	EC2 Elastic IP address	staging, production
`EC2_SSH_KEY`	Content of `~/.ssh/bilko-prod.pem`	staging, production
`AWS_ACCESS_KEY_ID`	IAM user access key for S3/CloudFront deploy	staging, production
`AWS_SECRET_ACCESS_KEY`	IAM user secret key	staging, production
`CLOUDFRONT_DISTRIBUTION_ID`	CloudFront distribution ID	staging, production

GitHub Environment Setup

Go to GitHub repository → Settings → Environments
Create staging environment
Create production environment — add required reviewers for approval gate
Add secrets to each environment

Creating IAM User for GitHub Actions (S3/CloudFront only)

# Create deployment user (separate from EC2 role — narrower permissions)
aws iam create-user --user-name bilko-github-deployer

aws iam put-user-policy \
  --user-name bilko-github-deployer \
  --policy-name bilko-deploy-policy \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": ["s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetObject"],
        "Resource": ["arn:aws:s3:::bilko-web-prod", "arn:aws:s3:::bilko-web-prod/*",
                     "arn:aws:s3:::bilko-web-staging", "arn:aws:s3:::bilko-web-staging/*"]
      },
      {
        "Effect": "Allow",
        "Action": ["cloudfront:CreateInvalidation"],
        "Resource": "*"
      }
    ]
  }'

aws iam create-access-key --user-name bilko-github-deployer
# Copy AccessKeyId and SecretAccessKey → add to GitHub secrets

11. Monitoring

CloudWatch Alarms

The following alarms are configured (SNS email notifications):

Alarm	Threshold	Action
EC2 CPU High	> 80% for 15 min	Email alert
EC2 Status Check Failed	Any failure	Email alert
RDS CPU High	> 80% for 10 min	Email alert
RDS Free Storage Low	< 2 GB	Email alert
RDS Connections High	> 50 connections	Email alert

CloudWatch Dashboards

Access via AWS Console → CloudWatch → Dashboards.

Key metrics to watch:

EC2: CPUUtilization, NetworkIn/Out, StatusCheckFailed
RDS: CPUUtilization, FreeStorageSpace, DatabaseConnections, ReadLatency, WriteLatency
CloudFront: Requests, BytesDownloaded, 4xxErrorRate, 5xxErrorRate

Application Logs

# PM2 logs (on EC2)
pm2 logs bilko-api
pm2 logs bilko-api --lines 100

# Log files
tail -f /var/log/bilko/api-out.log
tail -f /var/log/bilko/api-error.log

# Nginx logs
tail -f /var/log/nginx/bilko-api-access.log
tail -f /var/log/nginx/bilko-api-error.log

Health Check Endpoints

Endpoint	Expected	Notes
`https://api.bilko.io/api/v1/health`	HTTP 200	API liveness
`https://bilko.io`	HTTP 200	Frontend

12. Rollback Procedures

API Rollback

Automatic: If health check fails during deploy, the deploy script automatically rolls back to the previous git commit.

Manual rollback:

ssh -i ~/.ssh/bilko-prod.pem ec2-user@<EC2_IP>

cd /opt/bilko

# Find last known good commit
git log --oneline -10

# Roll back
git checkout <COMMIT_HASH>
npm ci
npx prisma generate --schema=packages/database/prisma/schema.prisma
npm run build --workspace=@bilko/api
pm2 reload bilko-api --update-env

# Verify
curl http://localhost:4000/api/v1/health

Note on migrations: Database migrations cannot be automatically rolled back with Prisma (migrate deploy is forward-only). If a migration broke production:

Roll back the application code (above)
Write a new migration that undoes the schema changes
Deploy the new migration manually

Frontend Rollback

CloudFront serves S3 content. To roll back:

# Option 1: Re-deploy previous git tag
git checkout <PREVIOUS_TAG>
./infrastructure/scripts/deploy-web.sh prod

# Option 2: If previous build artifacts are available
aws s3 sync s3://bilko-web-backup s3://bilko-web-prod --delete
aws cloudfront create-invalidation \
  --distribution-id <DISTRIBUTION_ID> \
  --paths "/*"

Database Point-in-Time Recovery

RDS has automated backups with 7-day retention. To restore:

# List available restore points
aws rds describe-db-instances \
  --db-instance-identifier bilko-prod-db \
  --query "DBInstances[0].LatestRestorableTime"

# Restore to point in time (creates NEW instance)
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier bilko-prod-db \
  --target-db-instance-identifier bilko-prod-db-restored \
  --restore-time "2026-02-23T10:00:00Z"

13. Cost Estimate

Monthly AWS costs (eu-central-1, approximate):

Service	Spec	Est. Monthly Cost
EC2	t3.medium (2 vCPU, 4GB)	~$35
RDS	db.t3.micro (1 vCPU, 1GB), 20GB gp3	~$20
S3 (web)	Static hosting, ~1GB	~$0.02
S3 (storage)	File uploads, 10GB	~$0.25
CloudFront	1TB transfer, EU+US	~$10
Route53	Hosted zone + queries	~$0.55
Data Transfer	EC2 outbound, ~10GB	~$1
CloudWatch	Alarms + logs	~$2
Elastic IP	1 IP (attached = free, detached = $0.005/hr)	$0
Total		~$69/month

Cost optimization tips:

RDS db.t3.micro handles up to ~80 connections — sufficient for early stage
CloudFront PriceClass_100 limits to US + Europe (covers Balkan region via Frankfurt PoP)
EC2 Reserved Instance (1-year) would reduce cost by ~40% ($35 → ~$21)
When traffic grows: scale EC2 vertically (t3.large) before adding ALB + Auto Scaling

Staging environment (if needed): Add ~$30/month for separate EC2 t3.small + RDS db.t3.micro.

Appendix: Quick Reference Commands

# SSH to EC2
ssh -i ~/.ssh/bilko-prod.pem ec2-user@<EC2_IP>

# Check API health
curl https://api.bilko.io/api/v1/health

# PM2 status
pm2 status

# View recent logs
pm2 logs bilko-api --lines 50

# Restart API
pm2 reload bilko-api

# Nginx status
sudo systemctl status nginx

# Certbot renewal check
sudo certbot renew --dry-run

# Terraform outputs
cd infrastructure/terraform && terraform output

# Manual API deploy
EC2_HOST=<IP> SSH_KEY=~/.ssh/bilko-prod.pem ./infrastructure/scripts/deploy-api.sh prod

# Manual web deploy
CLOUDFRONT_DISTRIBUTION_ID=<ID> ./infrastructure/scripts/deploy-web.sh prod

# Manual DB migration
EC2_HOST=<IP> SSH_KEY=~/.ssh/bilko-prod.pem ./infrastructure/scripts/db-migrate.sh prod