Bilko ACA Telemetry & Observability Wiring (Azure)

Context

GCP Cloud Monitoring dashboard (070613fa) was decommissioned 2026-06-23 after migration to Azure (MC #104228 closed). This page documents the ACA→Log Analytics + App Insights telemetry wiring done as follow-on MC #104266.

Resources

• Container App Environment: bilko-demo-env (NOTE: purplebeach-f004d490 is only the default-domain suffix in app URLs, not the env name).
• Log Analytics workspace: workspace-rgbilkodemo6lnV, customerId 71443731-9feb-41b1-9e27-fff4e4ebf098.
• App Insights: appi-bilko, appId 69e12981-9ebb-47ef-9dbd-5cf69fa87c40.
• Workbook: dcaef4e3-9bc7-48ae-8e1b-bd382a73889e "Bilko Observability — Prod+Stage (Azure)".
• 4 ACA apps: bilko-api-demo, bilko-web-demo, bilko-api-stage, bilko-web-stage.

Root cause that was fixed

ACA env appLogsConfiguration.destination was not effectively set → ContainerApp logs not reaching the workspace. Fixed via:

az containerapp env update -n bilko-demo-env -g rg-bilko-demo --logs-destination log-analytics --logs-workspace-id 71443731-... --logs-workspace-key <key>

Result: ContainerAppSystemLogs_CL now flows (tool-verified 54 rows/15m, sustained).

App Insights instrumentation

Each ACA app needs env var APPLICATIONINSIGHTS_CONNECTION_STRING (from az monitor app-insights component show -g rg-bilko-demo -n appi-bilko --query connectionString -o tsv), set via az containerapp update -n <app> -g rg-bilko-demo --set-env-vars APPLICATIONINSIGHTS_CONNECTION_STRING=<value>. All 4 apps confirmed set.

MC #104268 — SDK Initialization Fix (2026-06-28) ✅

The "KNOWN REMAINING GAP" documented below has been RESOLVED. Setting APPLICATIONINSIGHTS_CONNECTION_STRING env var alone was insufficient — the application runtime must initialize the App Insights SDK/agent.

Solution — Both Services Now Emitting Telemetry

1. bilko-api-demo (Kotlin/Ktor) — Java Agent

• Method: Application Insights Java Agent 3.6.2 auto-instrumentation
• Implementation:
  • Downloaded agent JAR to /app/applicationinsights-agent.jar during Docker build (SHA256-pinned: e81ef99f...)
  • Entrypoint: java -javaagent:/app/applicationinsights-agent.jar -jar /app/bilko-api.jar
  • Config file: apps/api/applicationinsights.json with Ktor preview instrumentation enabled:

    {
      "preview": {
        "instrumentation": {
          "ktor": { "enabled": true }
        }
      }
    }

  • Env vars: APPLICATIONINSIGHTS_CONNECTION_STRING + APPLICATIONINSIGHTS_ROLE_NAME=bilko-api
• Critical discovery: Ktor instrumentation is a PREVIEW feature in the Java agent. Without the explicit config file enabling it, the agent starts but does NOT instrument Ktor routes.
• Evidence: 15+ requests visible in App Insights requests table with cloud_RoleName=bilko-api

2. bilko-web-demo (Next.js) — Azure Monitor OpenTelemetry

• Method: Azure Monitor OpenTelemetry SDK
• Implementation: @azure/monitor-opentelemetry (^1.18.1) initialized in instrumentation.ts
• Evidence: 36+ requests visible in App Insights
• Service Name: unknown_service:node (default, can be customized)

Validation Result — Complete Success ✅

Baseline before fix: requests table = 0 rows in last 30 minutes
After fix: 144+ requests in 15 minutes from BOTH services

requests | where timestamp > ago(15m) | summarize count() by cloud_RoleName

Result:
- bilko-web: 36+ requests
- bilko-api: 15+ requests
- Total: 144+ requests

Deployed Revisions (Final)

• bilko-api-demo: revision --0000036, image demo-9658388f-ai2
• bilko-web-demo: revision --0000024, image demo-9658388f
• Merge commit: f15deb8a (PR #27)

Files Modified

1. apps/api/Dockerfile — Java agent download + config file copy + javaagent entrypoint
2. apps/api/applicationinsights.json — NEW: Agent config with Ktor preview instrumentation
3. apps/web/instrumentation.ts — Azure Monitor OpenTelemetry initialization
4. apps/web/package.json — @azure/monitor-opentelemetry dependency

Accepted Risk

@azure/monitor-opentelemetry package has GHSA-8988-4f7v-96qf (documented in DEPLOY-MAP.md)

Evidence Files

Evidence stored in repo: /docs/evidence/104268/{SUCCESS-FINAL.md, telemetry-validation.md, sdk-changes.md, deploy-revisions.md, final-status.md}

KNOWN REMAINING GAP (important for runbook) — RESOLVED 2026-06-28

This section is retained for historical context. The gap was resolved by MC #104268 (see section above).

Setting the env var ALONE does not produce App Insights request/dependency telemetry — requests table = 0. The app code must initialize the App Insights SDK (Node: applicationinsights package; Spring Boot: azure-monitor starter). Until then, App-Insights-request-based workbook panels stay empty. The KQL log panel (ContainerAppSystemLogs_CL) and ACA platform-metric panels work regardless. Track app-code SDK init as a separate CodeCraft task if request tracing is needed.

Troubleshooting note

az monitor log-analytics query returns a FLAT array [{col:val}] — do NOT filter with --query "tables[0].rows" (returns empty falsely). az monitor app-insights query uses {tables:[{rows}]} shape.

Verification queries (runbook)

• Logs: ContainerAppSystemLogs_CL | where TimeGenerated > ago(20m) | count (workspace 71443731) — expect >0.
• Env vars: az containerapp show -g rg-bilko-demo -n <app> --query "properties.template.containers[0].env[?name=='APPLICATIONINSIGHTS_CONNECTION_STRING']".
• Availability: availabilityResults pass rate on appi-bilko.

Related

• MC #104268 (completed 2026-06-28) — SDK initialization fix
• MC #104266 (completed 2026-06-23) — Env var wiring
• MC #104228 (GCP decommission, closed)
• Azure subscription: 5b0b4d9b-e677-464e-abf0-5170cbce3b8e
• Resource group: rg-bilko-demo (swedencentral)