Bilko demo — receipt upload/download fix (GCS shared storage) — MC #103095 (2026-06-07)

1. Symptom CEO reported that receipt upload (PNG, PDF, JPG) was not working — a central part of the app. Investigation showed upload itself succeeded (HTTP 201) but viewing/downloading the receipt returned intermittent HTTP 404 approximately 60% of the time . From the user seat, intermittent 404 on download reads as a broken upload. The UI button "Priloženi dokumenti" -> "Preuzmi" triggered the failing request. 2. Root Cause The demo API had no shared object storage configured. It used BILKO_LOCAL_UPLOAD_DIR=/tmp/bilko-uploads — per-instance ephemeral local disk, routed through ReceiptService.kt persistLocalIfEnabled , storing files as local:// URLs. Cloud Run (bilko-api-demo) runs up to 5 instances with concurrency=1 (set during the earlier MC #103057 hang mitigation). An upload landing on instance A wrote the file to that instance's /tmp ; a subsequent download request routed to instance B, which had no copy of the file, returning 404. Files were also permanently lost on any instance restart or recycle. A secondary symptom was occasional 15-second frontend timeouts on the expense detail page: the several parallel API calls the page makes on load were serialised by concurrency=1 . Contributing config drift: the active deploy step in infrastructure/gcp/cloudbuild.yaml (deploy-api-demo) used --set-env-vars which replaces the entire env set, making a separate cloudbuild-demo-api.yaml with BILKO_LOCAL_UPLOAD_DIR ineffective. 3. Fix Applied by FlowForge (Kelsey Hightower). No application code changes were required — ReceiptService.kt is unchanged and uses a transparent filesystem abstraction. Change Detail GCS bucket provisioned gs://bilko-receipts-demo , region europe-north1, uniform bucket-level access, IAM: bilko-api-stage-sa = roles/storage.objectAdmin Cloud Run exec environment Upgraded to gen2 (required for gcsfuse volume mounts) Volume mount added --add-volume=name=receipts,type=cloud-storage,bucket=bilko-receipts-demo Mount path --add-volume-mount=volume=receipts,mount-path=/mnt/bilko-uploads Env var updated BILKO_LOCAL_UPLOAD_DIR : /tmp/bilko-uploads -> /mnt/bilko-uploads Config persisted All changes committed to infrastructure/gcp/cloudbuild.yaml (deploy-api-demo step) Deployed: tag v0.2.30, commit 642bbc0cefdc63777d8c12d61aa61a8257716290, revision bilko-api-demo-00135-rmv, 100% traffic on new revision. Cloud Build ID: 793f929b-f41a-49e9-afa9-65b54d3972ff. Note: Cloud Build reported FAILURE due to a pre-existing flaky test timeout in coverage artifact upload (expenses-ux-102887). This is unrelated to the GCS change. All 8 deploy gates (lint, typecheck, unit, coverage, trivy, gitleaks, semgrep, npm-audit) PASSED; build, push, trivy, migrate, deploy, promote, smoke-test, and verify-sha steps all succeeded. 4. Validation Validated by Proveo (Angie Jones) — GLOBAL VERDICT: PASS. Test Result PDF upload + 10x download 10/10 HTTP 200 (was intermittent 404) PNG upload + 10x download 10/10 HTTP 200 JPEG upload + 10x download 10/10 HTTP 200 GCS persistence (15 total calls) 15/15 HTTP 200 — confirmed shared across instances UI: Priloženi dokumenti section Visible; download icon -> /content HTTP 200 Health check https://bilko-demo-api.alai.no/api/v1/health -> 200 {"status":"ok"} Company Mesh: mesh-thr-03f166bb-f001-4293-b9ec-db245e5790b3 — PASS. Open item (non-blocker): 1 pre-fix orphan document (uploaded 07:58 before GCS deploy at 08:30) returns 404 as expected — the file lived in ephemeral /tmp on a recycled instance. Not a regression. 5. Known Follow-Up Tasks MC Description #103102 Graceful 404 handling for missing documents in UI; fix misleading BILKO-INV-001 error code returned on expense document content misses #103103 Flaky coverage test (expenses-ux-102887 dialog upload test) blocking clean Cloud Build artifact upload — unrelated to this fix #103104 Invoice-receipt download gap: POST /invoices/{id}/receipts returns 201 but no download endpoint exists and receiptUrl stays null in invoice record 6. Operational Notes — Demo Deploy Pipeline Demo deploys via semver tag (e.g. v0.2.30) pushed to the Bilko repo, which triggers the bilko-main-deploy Cloud Build trigger, which runs infrastructure/gcp/cloudbuild.yaml . This deploys bilko-api-demo + bilko-web-demo and migrates bilko-demo-db . Do NOT push directly to main — pushing to main auto-triggers the stage deploy (bilko-stage-auto-deploy), not the demo deploy. Stage vs demo separation: Stage uses bilko-api-stage (no GCS mount, different SA). Demo uses bilko-api-demo with gs://bilko-receipts-demo via gcsfuse. RLS bugs and storage configuration differ between the two environments — always verify fixes on demo, not only stage. GCS FUSE driver: gcsfuse.run.googleapis.com , requires Cloud Run gen2 execution environment.