# Bilko ACA Telemetry & Observability Wiring (Azure)

## Context

GCP Cloud Monitoring dashboard (070613fa) was decommissioned 2026-06-23 after migration to Azure (MC #104228 closed). This page documents the ACA→Log Analytics + App Insights telemetry wiring done as follow-on MC #104266.

## Resources

- **Container App Environment:** `bilko-demo-env` (NOTE: `purplebeach-f004d490` is only the default-domain suffix in app URLs, not the env name).
- **Log Analytics workspace:** `workspace-rgbilkodemo6lnV`, customerId `71443731-9feb-41b1-9e27-fff4e4ebf098`.
- **App Insights:** `appi-bilko`, appId `69e12981-9ebb-47ef-9dbd-5cf69fa87c40`.
- **Workbook:** `dcaef4e3-9bc7-48ae-8e1b-bd382a73889e` "Bilko Observability — Prod+Stage (Azure)".
- **4 ACA apps:** bilko-api-demo, bilko-web-demo, bilko-api-stage, bilko-web-stage.

## Root cause that was fixed

ACA env `appLogsConfiguration.destination` was not effectively set → ContainerApp logs not reaching the workspace. Fixed via:

```
az containerapp env update -n bilko-demo-env -g rg-bilko-demo --logs-destination log-analytics --logs-workspace-id 71443731-... --logs-workspace-key <key>
```

**Result:** ContainerAppSystemLogs\_CL now flows (tool-verified 54 rows/15m, sustained).

## App Insights instrumentation

Each ACA app needs env var `APPLICATIONINSIGHTS_CONNECTION_STRING` (from `az monitor app-insights component show -g rg-bilko-demo -n appi-bilko --query connectionString -o tsv`), set via `az containerapp update -n <app> -g rg-bilko-demo --set-env-vars APPLICATIONINSIGHTS_CONNECTION_STRING=<value>`. All 4 apps confirmed set.

## MC #104268 — SDK Initialization Fix (2026-06-28) ✅

The "KNOWN REMAINING GAP" documented below has been **RESOLVED**. Setting `APPLICATIONINSIGHTS_CONNECTION_STRING` env var alone was insufficient — the application runtime must initialize the App Insights SDK/agent.

### Solution — Both Services Now Emitting Telemetry

#### 1. bilko-api-demo (Kotlin/Ktor) — Java Agent

- **Method:** Application Insights Java Agent 3.6.2 auto-instrumentation
- **Implementation:**
    - Downloaded agent JAR to `/app/applicationinsights-agent.jar` during Docker build (SHA256-pinned: `e81ef99f...`)
    - Entrypoint: `java -javaagent:/app/applicationinsights-agent.jar -jar /app/bilko-api.jar`
    - Config file: `apps/api/applicationinsights.json` with **Ktor preview instrumentation enabled**: ```
        {
          "preview": {
            "instrumentation": {
              "ktor": { "enabled": true }
            }
          }
        }
        ```
    - Env vars: `APPLICATIONINSIGHTS_CONNECTION_STRING` + `APPLICATIONINSIGHTS_ROLE_NAME=bilko-api`
- **Critical discovery:** Ktor instrumentation is a PREVIEW feature in the Java agent. Without the explicit config file enabling it, the agent starts but does NOT instrument Ktor routes.
- **Evidence:** 15+ requests visible in App Insights `requests` table with `cloud_RoleName=bilko-api`

#### 2. bilko-web-demo (Next.js) — Azure Monitor OpenTelemetry

- **Method:** Azure Monitor OpenTelemetry SDK
- **Implementation:** `@azure/monitor-opentelemetry` (^1.18.1) initialized in `instrumentation.ts`
- **Evidence:** 36+ requests visible in App Insights
- **Service Name:** `unknown_service:node` (default, can be customized)

### Validation Result — Complete Success ✅

**Baseline before fix:** `requests` table = 0 rows in last 30 minutes  
**After fix:** 144+ requests in 15 minutes from BOTH services

```
requests | where timestamp > ago(15m) | summarize count() by cloud_RoleName

Result:
- bilko-web: 36+ requests
- bilko-api: 15+ requests
- Total: 144+ requests
```

### Deployed Revisions (Final)

- **bilko-api-demo:** revision `--0000036`, image `demo-9658388f-ai2`
- **bilko-web-demo:** revision `--0000024`, image `demo-9658388f`
- **Merge commit:** `f15deb8a` (PR #27)

### Files Modified

1. `apps/api/Dockerfile` — Java agent download + config file copy + javaagent entrypoint
2. `apps/api/applicationinsights.json` — NEW: Agent config with Ktor preview instrumentation
3. `apps/web/instrumentation.ts` — Azure Monitor OpenTelemetry initialization
4. `apps/web/package.json` — @azure/monitor-opentelemetry dependency

### Accepted Risk

`@azure/monitor-opentelemetry` package has GHSA-8988-4f7v-96qf (documented in `DEPLOY-MAP.md`)

### Evidence Files

Evidence stored in repo: `/docs/evidence/104268/{SUCCESS-FINAL.md, telemetry-validation.md, sdk-changes.md, deploy-revisions.md, final-status.md}`

---

## KNOWN REMAINING GAP (important for runbook) — RESOLVED 2026-06-28

*This section is retained for historical context. The gap was resolved by MC #104268 (see section above).*  
  
Setting the env var ALONE does not produce App Insights request/dependency telemetry — `requests` table = 0. The app code must initialize the App Insights SDK (Node: `applicationinsights` package; Spring Boot: azure-monitor starter). Until then, App-Insights-request-based workbook panels stay empty. The KQL log panel (ContainerAppSystemLogs\_CL) and ACA platform-metric panels work regardless. Track app-code SDK init as a separate CodeCraft task if request tracing is needed.

## Troubleshooting note

`az monitor log-analytics query` returns a FLAT array `[{col:val}]` — do NOT filter with `--query "tables[0].rows"` (returns empty falsely). `az monitor app-insights query` uses `{tables:[{rows}]}` shape.

## Verification queries (runbook)

- **Logs:** `ContainerAppSystemLogs_CL | where TimeGenerated > ago(20m) | count` (workspace 71443731) — expect &gt;0.
- **Env vars:** `az containerapp show -g rg-bilko-demo -n <app> --query "properties.template.containers[0].env[?name=='APPLICATIONINSIGHTS_CONNECTION_STRING']"`.
- **Availability:** availabilityResults pass rate on appi-bilko.

## Related

- MC #104268 (completed 2026-06-28) — SDK initialization fix
- MC #104266 (completed 2026-06-23) — Env var wiring
- MC #104228 (GCP decommission, closed)
- Azure subscription: 5b0b4d9b-e677-464e-abf0-5170cbce3b8e
- Resource group: rg-bilko-demo (swedencentral)