# Bilko demo — receipt upload/download fix (GCS shared storage) — MC #103095 (2026-06-07)

## 1. Symptom

CEO reported that receipt upload (PNG, PDF, JPG) was not working — a central part of the app. Investigation showed upload itself succeeded (HTTP 201) but **viewing/downloading the receipt returned intermittent HTTP 404 approximately 60% of the time**. From the user seat, intermittent 404 on download reads as a broken upload. The UI button "Priloženi dokumenti" -> "Preuzmi" triggered the failing request.

## 2. Root Cause

The demo API had no shared object storage configured. It used `BILKO_LOCAL_UPLOAD_DIR=/tmp/bilko-uploads` — per-instance ephemeral local disk, routed through `ReceiptService.kt persistLocalIfEnabled`, storing files as `local://` URLs.

Cloud Run (bilko-api-demo) runs up to 5 instances with `concurrency=1` (set during the earlier MC #103057 hang mitigation). An upload landing on instance A wrote the file to that instance's `/tmp`; a subsequent download request routed to instance B, which had no copy of the file, returning 404. Files were also permanently lost on any instance restart or recycle.

A secondary symptom was occasional 15-second frontend timeouts on the expense detail page: the several parallel API calls the page makes on load were serialised by `concurrency=1`.

Contributing config drift: the active deploy step in `infrastructure/gcp/cloudbuild.yaml` (deploy-api-demo) used `--set-env-vars` which replaces the entire env set, making a separate `cloudbuild-demo-api.yaml` with `BILKO_LOCAL_UPLOAD_DIR` ineffective.

## 3. Fix

Applied by FlowForge (Kelsey Hightower). No application code changes were required — `ReceiptService.kt` is unchanged and uses a transparent filesystem abstraction.

<table id="bkmrk-changedetailgcs-buck"><thead><tr><th>Change</th><th>Detail</th></tr></thead><tbody><tr><td>GCS bucket provisioned</td><td>`gs://bilko-receipts-demo`, region europe-north1, uniform bucket-level access, IAM: bilko-api-stage-sa = roles/storage.objectAdmin</td></tr><tr><td>Cloud Run exec environment</td><td>Upgraded to `gen2` (required for gcsfuse volume mounts)</td></tr><tr><td>Volume mount added</td><td>`--add-volume=name=receipts,type=cloud-storage,bucket=bilko-receipts-demo`</td></tr><tr><td>Mount path</td><td>`--add-volume-mount=volume=receipts,mount-path=/mnt/bilko-uploads`</td></tr><tr><td>Env var updated</td><td>`BILKO_LOCAL_UPLOAD_DIR`: `/tmp/bilko-uploads` -&gt; `/mnt/bilko-uploads`</td></tr><tr><td>Config persisted</td><td>All changes committed to `infrastructure/gcp/cloudbuild.yaml` (deploy-api-demo step)</td></tr></tbody></table>

**Deployed:** tag v0.2.30, commit 642bbc0cefdc63777d8c12d61aa61a8257716290, revision bilko-api-demo-00135-rmv, 100% traffic on new revision. Cloud Build ID: 793f929b-f41a-49e9-afa9-65b54d3972ff.

Note: Cloud Build reported FAILURE due to a pre-existing flaky test timeout in coverage artifact upload (expenses-ux-102887). This is unrelated to the GCS change. All 8 deploy gates (lint, typecheck, unit, coverage, trivy, gitleaks, semgrep, npm-audit) PASSED; build, push, trivy, migrate, deploy, promote, smoke-test, and verify-sha steps all succeeded.

## 4. Validation

**Validated by Proveo (Angie Jones) — GLOBAL VERDICT: PASS.**

<table id="bkmrk-testresultpdf-upload"><thead><tr><th>Test</th><th>Result</th></tr></thead><tbody><tr><td>PDF upload + 10x download</td><td>10/10 HTTP 200 (was intermittent 404)</td></tr><tr><td>PNG upload + 10x download</td><td>10/10 HTTP 200</td></tr><tr><td>JPEG upload + 10x download</td><td>10/10 HTTP 200</td></tr><tr><td>GCS persistence (15 total calls)</td><td>15/15 HTTP 200 — confirmed shared across instances</td></tr><tr><td>UI: Priloženi dokumenti section</td><td>Visible; download icon -&gt; /content HTTP 200</td></tr><tr><td>Health check</td><td>https://bilko-demo-api.alai.no/api/v1/health -&gt; 200 {"status":"ok"}</td></tr></tbody></table>

Company Mesh: `mesh-thr-03f166bb-f001-4293-b9ec-db245e5790b3` — PASS.

**Open item (non-blocker):** 1 pre-fix orphan document (uploaded 07:58 before GCS deploy at 08:30) returns 404 as expected — the file lived in ephemeral /tmp on a recycled instance. Not a regression.

## 5. Known Follow-Up Tasks

<table id="bkmrk-mcdescription%23103102"><thead><tr><th>MC</th><th>Description</th></tr></thead><tbody><tr><td>\#103102</td><td>Graceful 404 handling for missing documents in UI; fix misleading BILKO-INV-001 error code returned on expense document content misses</td></tr><tr><td>\#103103</td><td>Flaky coverage test (expenses-ux-102887 dialog upload test) blocking clean Cloud Build artifact upload — unrelated to this fix</td></tr><tr><td>\#103104</td><td>Invoice-receipt download gap: POST /invoices/{id}/receipts returns 201 but no download endpoint exists and receiptUrl stays null in invoice record</td></tr></tbody></table>

## 6. Operational Notes — Demo Deploy Pipeline

- **Demo deploys** via semver tag (e.g. v0.2.30) pushed to the Bilko repo, which triggers the `bilko-main-deploy` Cloud Build trigger, which runs `infrastructure/gcp/cloudbuild.yaml`. This deploys `bilko-api-demo` + `bilko-web-demo` and migrates `bilko-demo-db`.
- **Do NOT push directly to `main`** — pushing to main auto-triggers the stage deploy (bilko-stage-auto-deploy), not the demo deploy.
- **Stage vs demo separation:** Stage uses `bilko-api-stage` (no GCS mount, different SA). Demo uses `bilko-api-demo` with `gs://bilko-receipts-demo` via gcsfuse. RLS bugs and storage configuration differ between the two environments — always verify fixes on demo, not only stage.
- GCS FUSE driver: `gcsfuse.run.googleapis.com`, requires Cloud Run gen2 execution environment.