# Go-Live Runbook

# Go-Live Runbook

> **Project:** Bilko
> **Version:** 0.1 — Initial Production Launch
> **Date:** 2026-02-23
> **Author:** Ops Architect
> **Status:** Draft
> **Reviewers:** Tech Lead, Alem Bašić

## Document History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 0.1     | 2026-02-23 | Ops Architect | Initial draft |

---

## 1. Go-Live Overview

**What:** Bilko v1.0 — first production launch of the cloud accounting SaaS
**Target date:** TBD (set when MVP backend complete)
**Deployment window:** 2-hour window (prefer low-traffic: Tuesday–Thursday 10:00–12:00 CET)
**Go-Live Type:** New product launch — no existing users to migrate

**Incident Commander:** Alem Bašić (+47 40 47 42 51) — primary
**Technical Lead:** Alem Bašić (also, at MVP)
**War Room:** Slack #bilko-deploys (create dedicated #bilko-launch channel for day-of)
**Status Page:** status.bilko.io (PLANNED — configure BetterStack before launch)

---

## 2. Pre-Launch Checklist

### T-7 Days: Infrastructure Verification

- [ ] Railway project created with `api` service and PostgreSQL 15 database
- [ ] Railway EU West region confirmed (GDPR compliance)
- [ ] Vercel project created and linked to GitHub repo (`alai-holding/bilko`)
- [ ] Cloudflare R2 bucket `bilko-receipts` created with correct CORS policy
- [ ] All production environment variables set in Railway and Vercel dashboards
- [ ] Railway health check endpoint (`GET /health`) responding `{"status":"ok","db":"ok"}`
- [ ] Vercel build successful (`pnpm run build` on main branch)
- [ ] Database migrations applied: `railway run npx prisma migrate deploy`
- [ ] Cost estimate confirmed within budget (< €25/mo at MVP)

**Owner:** Alem Bašić | **Due:** T-7 days

---

### T-5 Days: DNS Configuration

- [ ] Cloudflare DNS records created for bilko.io:
  - `@` CNAME → `cname.vercel-dns.com` (Proxied: Yes)
  - `www` CNAME → `cname.vercel-dns.com` (Proxied: Yes)
  - `api` CNAME → `<railway-domain>.railway.app` (Proxied: No)
- [ ] bilko.io domain verified in Vercel project (Vercel → Settings → Domains)
- [ ] api.bilko.io custom domain set in Railway (Railway → Settings → Domains)
- [ ] DNS propagation verified: `dig bilko.io` and `dig api.bilko.io`
- [ ] Cloudflare "Always Use HTTPS" enabled for bilko.io
- [ ] bilko.rs redirect rule configured (if domain registered): bilko.rs → bilko.io

**Owner:** Alem Bašić | **Due:** T-5 days

---

### T-5 Days: SSL Certificates

- [ ] bilko.io TLS certificate provisioned (Vercel auto-provisions via Let's Encrypt)
- [ ] api.bilko.io TLS certificate provisioned (Railway auto-provisions)
- [ ] HTTPS verified: `curl -I https://bilko.io` → HTTP/2 200
- [ ] HTTPS verified: `curl https://api.bilko.io/health` → `{"status":"ok"}`
- [ ] HTTP → HTTPS redirect working: `curl -I http://bilko.io` → 301

**Owner:** Alem Bašić | **Due:** T-5 days

---

### T-3 Days: Third-Party Integrations

- [ ] SendGrid live API key in Railway production secrets
- [ ] SendGrid domain authentication for bilko.io completed (SPF, DKIM, DMARC)
- [ ] Test email sent from `noreply@bilko.io` and received successfully
- [ ] Cloudflare R2 live API credentials in Railway production secrets
- [ ] Test file upload to R2 `bilko-receipts` bucket via API

**Owner:** Alem Bašić | **Due:** T-3 days

---

### T-2 Days: Monitoring Setup

- [ ] Sentry project created for Bilko backend + frontend (if ready)
- [ ] SENTRY_DSN environment variable set in Railway and Vercel
- [ ] BetterStack uptime monitors created:
  - `https://bilko.io` — check every 1 min
  - `https://api.bilko.io/health` — check every 1 min
- [ ] BetterStack alert routing: Slack #bilko-alerts + email to alem@alai.no
- [ ] Alert test fired and received by Alem Bašić

**Owner:** Alem Bašić | **Due:** T-2 days

---

### T-1 Day: Legal / Compliance

- [ ] Privacy policy published at bilko.io/privacy
- [ ] Terms of service published at bilko.io/terms
- [ ] Cookie consent banner implemented (not required for SaaS with no tracking cookies, but review)
- [ ] GDPR data processing documentation completed
- [ ] Data retention policy documented (financial records: 10 years per Serbian law)
- [ ] Legal sign-off from Alem Bašić on compliance readiness

**Owner:** Alem Bašić | **Due:** T-1 day

---

### T-1 Day: Backup Verification

- [ ] Railway automated backup confirmed running (PostgreSQL → Backups tab)
- [ ] Manual backup taken and restore tested on staging:
  ```bash
  railway run pg_dump $DATABASE_URL -f test_backup.dump
  # Restore to staging DB
  railway run psql $STAGING_DATABASE_URL < test_backup.dump
  ```
- [ ] Backup verified: record counts match original

**Owner:** Alem Bašić | **Due:** T-1 day

---

### T-0: Final Checks (1 hour before launch)

- [ ] Staging smoke tests all green (last run: within 24h)
- [ ] Main branch is up to date with all intended changes
- [ ] Railway and Vercel dashboards open in browser
- [ ] BetterStack monitoring dashboard open
- [ ] Rollback procedure reviewed (< 2 min for frontend, < 5 min for backend)
- [ ] Slack #bilko-launch channel open with Alem

---

## 3. Launch Day Procedure

### H+0:00 — Deployment Start

| Time | Action | Owner | Status |
|------|--------|-------|--------|
| H+0:00 | Post in #bilko-launch: "Bilko launch starting" | Alem | |
| H+0:00 | Confirm Railway deployment pipeline ready | Alem | |
| H+0:05 | Trigger production deployment from main branch | Alem | |
| H+0:10 | Monitor Railway build logs | Alem | |

### H+0:10 → H+0:20 — Deploy Verification

| Time | Action | Owner | Status |
|------|--------|-------|--------|
| H+0:15 | Confirm Railway deployment successful | Alem | |
| H+0:15 | Verify health check: `curl https://api.bilko.io/health` | Alem | |
| H+0:20 | Confirm Vercel frontend deployment successful | Alem | |
| H+0:20 | Verify frontend: open https://bilko.io in browser | Alem | |

### H+0:20 → H+0:45 — Smoke Tests

| Time | Action | Owner | Status |
|------|--------|-------|--------|
| H+0:20 | Register new test account on bilko.io | Alem | |
| H+0:25 | Create test invoice (RSD, 20% VAT) | Alem | |
| H+0:30 | Verify invoice totals (subtotal + VAT = total) | Alem | |
| H+0:35 | Create test expense with receipt upload | Alem | |
| H+0:40 | Generate VAT report for current month | Alem | |
| H+0:45 | All smoke tests PASS → proceed | Alem | |

### H+0:45 — Go-Live Declaration

| Time | Action | Owner | Status |
|------|--------|-------|--------|
| H+0:45 | Post in #bilko-launch: "Bilko is LIVE! bilko.io" | Alem | |
| H+0:50 | Update status page: "All systems operational" | Alem | |
| H+1:00 | Send launch announcement (if planned) | Alem | |

---

## 4. Post-Launch Monitoring (First 48 Hours)

| Period | Check Frequency | What to Watch |
|--------|-----------------|---------------|
| H+0 to H+4 | Every 30 min | BetterStack dashboard, Railway metrics, Sentry errors |
| H+4 to H+24 | Every 60 min | Same as above |
| Day 2 | Every 4 hours | Same as above |
| Day 3+ | Standard monitoring | BetterStack alerts only |

**Healthy indicators:**
- BetterStack: all monitors green
- Railway CPU: < 50%, Memory: < 1GB
- Sentry: 0 new issues in first hour
- API health endpoint: `{"status":"ok","db":"ok"}`

---

## 5. Rollback Triggers & Procedure

**Rollback if:**
- Health check fails for > 3 consecutive minutes
- Error rate > 5% in any 5-minute window (Sentry)
- Financial calculation bug discovered (any VAT/total error)
- Authentication completely broken

**Rollback procedure:**
1. Post in #bilko-launch: "Rolling back — [reason]"
2. **Frontend:** Vercel Dashboard → Deployments → Promote previous → instant
3. **Backend:** Railway Dashboard → Deployments → Redeploy previous → ~2 min
4. Verify health: `curl https://api.bilko.io/health`
5. Post update when rollback complete

---

## 6. Communication Plan

### Launch Day Communications

| Audience | Channel | When | Message |
|----------|---------|------|---------|
| Internal | Slack #bilko-launch | H+0 | "Deployment started" |
| Internal | Slack #bilko-launch | H+0:45 | "Bilko is live! bilko.io" |
| Beta users (if any) | Email | H+1:00 | Launch announcement |
| Status page | status.bilko.io | H+0:45 | "All systems operational" |

---

## Related Documents

- [Deployment Checklist](../release/DEPLOYMENT-CHECKLIST.md)
- [Rollback Plan](../release/ROLLBACK-PLAN.md)
- [Operational Runbook](./OPERATIONAL-RUNBOOK.md)
- [Monitoring & Observability](../infrastructure/MONITORING.md)
- [Disaster Recovery Plan](../infrastructure/DISASTER-RECOVERY.md)

---

## Approval
| Role | Name | Date | Signature |
|------|------|------|-----------|
| Author | Ops Architect | 2026-02-23 | |
| Reviewer | Tech Lead | | |
| Approver | Alem Bašić | | |