Bilko demo — receipt upload/download fix (GCS shared storage) — MC #103095 (2026-06-07)
1. Symptom
CEO reported that receipt upload (PNG, PDF, JPG) was not working — a central part of the app. Investigation showed upload itself succeeded (HTTP 201) but viewing/downloading the receipt returned intermittent HTTP 404 approximately 60% of the time. From the user seat, intermittent 404 on download reads as a broken upload. The UI button "Priloženi dokumenti" -> "Preuzmi" triggered the failing request.
2. Root Cause
The demo API had no shared object storage configured. It used BILKO_LOCAL_UPLOAD_DIR=/tmp/bilko-uploads — per-instance ephemeral local disk, routed through ReceiptService.kt persistLocalIfEnabled, storing files as local:// URLs.
Cloud Run (bilko-api-demo) runs up to 5 instances with concurrency=1 (set during the earlier MC #103057 hang mitigation). An upload landing on instance A wrote the file to that instance's /tmp; a subsequent download request routed to instance B, which had no copy of the file, returning 404. Files were also permanently lost on any instance restart or recycle.
A secondary symptom was occasional 15-second frontend timeouts on the expense detail page: the several parallel API calls the page makes on load were serialised by concurrency=1.
Contributing config drift: the active deploy step in infrastructure/gcp/cloudbuild.yaml (deploy-api-demo) used --set-env-vars which replaces the entire env set, making a separate cloudbuild-demo-api.yaml with BILKO_LOCAL_UPLOAD_DIR ineffective.
3. Fix
Applied by FlowForge (Kelsey Hightower). No application code changes were required — ReceiptService.kt is unchanged and uses a transparent filesystem abstraction.
| Change | Detail |
|---|---|
| GCS bucket provisioned | gs://bilko-receipts-demo, region europe-north1, uniform bucket-level access, IAM: bilko-api-stage-sa = roles/storage.objectAdmin |
| Cloud Run exec environment | Upgraded to gen2 (required for gcsfuse volume mounts) |
| Volume mount added | --add-volume=name=receipts,type=cloud-storage,bucket=bilko-receipts-demo |
| Mount path | --add-volume-mount=volume=receipts,mount-path=/mnt/bilko-uploads |
| Env var updated | BILKO_LOCAL_UPLOAD_DIR: /tmp/bilko-uploads -> /mnt/bilko-uploads |
| Config persisted | All changes committed to infrastructure/gcp/cloudbuild.yaml (deploy-api-demo step) |
Deployed: tag v0.2.30, commit 642bbc0cefdc63777d8c12d61aa61a8257716290, revision bilko-api-demo-00135-rmv, 100% traffic on new revision. Cloud Build ID: 793f929b-f41a-49e9-afa9-65b54d3972ff.
Note: Cloud Build reported FAILURE due to a pre-existing flaky test timeout in coverage artifact upload (expenses-ux-102887). This is unrelated to the GCS change. All 8 deploy gates (lint, typecheck, unit, coverage, trivy, gitleaks, semgrep, npm-audit) PASSED; build, push, trivy, migrate, deploy, promote, smoke-test, and verify-sha steps all succeeded.
4. Validation
Validated by Proveo (Angie Jones) — GLOBAL VERDICT: PASS.
| Test | Result |
|---|---|
| PDF upload + 10x download | 10/10 HTTP 200 (was intermittent 404) |
| PNG upload + 10x download | 10/10 HTTP 200 |
| JPEG upload + 10x download | 10/10 HTTP 200 |
| GCS persistence (15 total calls) | 15/15 HTTP 200 — confirmed shared across instances |
| UI: Priloženi dokumenti section | Visible; download icon -> /content HTTP 200 |
| Health check | https://bilko-demo-api.alai.no/api/v1/health -> 200 {"status":"ok"} |
Company Mesh: mesh-thr-03f166bb-f001-4293-b9ec-db245e5790b3 — PASS.
Open item (non-blocker): 1 pre-fix orphan document (uploaded 07:58 before GCS deploy at 08:30) returns 404 as expected — the file lived in ephemeral /tmp on a recycled instance. Not a regression.
5. Known Follow-Up Tasks
| MC | Description |
|---|---|
| #103102 | Graceful 404 handling for missing documents in UI; fix misleading BILKO-INV-001 error code returned on expense document content misses |
| #103103 | Flaky coverage test (expenses-ux-102887 dialog upload test) blocking clean Cloud Build artifact upload — unrelated to this fix |
| #103104 | Invoice-receipt download gap: POST /invoices/{id}/receipts returns 201 but no download endpoint exists and receiptUrl stays null in invoice record |
6. Operational Notes — Demo Deploy Pipeline
- Demo deploys via semver tag (e.g. v0.2.30) pushed to the Bilko repo, which triggers the
bilko-main-deployCloud Build trigger, which runsinfrastructure/gcp/cloudbuild.yaml. This deploysbilko-api-demo+bilko-web-demoand migratesbilko-demo-db. - Do NOT push directly to
main— pushing to main auto-triggers the stage deploy (bilko-stage-auto-deploy), not the demo deploy. - Stage vs demo separation: Stage uses
bilko-api-stage(no GCS mount, different SA). Demo usesbilko-api-demowithgs://bilko-receipts-demovia gcsfuse. RLS bugs and storage configuration differ between the two environments — always verify fixes on demo, not only stage. - GCS FUSE driver:
gcsfuse.run.googleapis.com, requires Cloud Run gen2 execution environment.
No comments to display
No comments to display