Skip to main content

Incident — 2026-04-21 alai.no Contact Form Failure

2026-04-21 — alai.no Contact Form Silent Failure

Incident Classification

Severity: HIGH — Silent data loss (potential lead loss)
Duration: 2026-04-19 19:00 → 2026-04-21 11:30 (40.5 hours)
Detection: Manual inspection via Himalaya IMAP client
Status: RESOLVED (form handler redeployed to CF Pages Functions)

Timeline

  • 2026-04-19 19:00 — alai.no migrated from Vercel to Cloudflare Pages (MC #8576)
  • 2026-04-19 19:00 → 2026-04-21 11:30 — Contact form submissions received HTTP 200 OK but no emails delivered
  • 2026-04-21 11:30 — CEO (Alem) noticed no inquiry emails received in days, requested investigation
  • 2026-04-21 11:35 — John inspected [email protected] IMAP (via himalaya search --folder INBOX from:noreply) — zero messages from contact form
  • 2026-04-21 11:45 — Root cause identified: CF Tunnel routing hijack + documenso-webhook false-positive response
  • 2026-04-21 12:15 — CodeCraft dispatched to deploy dedicated contact handler as CF Pages Function (MC #8587)
  • 2026-04-21 14:00 — Fix deployed and verified (E2E browser test + inbox check)

Impact Assessment

  • Lost inquiries: Unknown (no form submission logging). Estimated 0-5 potential leads during 40-hour window.
  • User experience: Users received "success" feedback but no confirmation email. No error notification.
  • Business risk: Medium — alai.no is not yet primary sales channel; minimal active marketing campaigns during incident window.

Root Cause Analysis

Technical Chain of Failure

  1. alai.no contact form POSTs to https://api.basicconsulting.no/contact (hardcoded Vercel pattern from pre-migration code)
  2. Cloudflare Tunnel ingress rule matches api.basicconsulting.no/* → routes ALL POST requests to localhost:3001
  3. documenso-webhook.js listens on port 3001, designed for Documenso signature events
  4. Webhook handler has catch-all route: app.post('/*', (req, res) => res.json({ok: true}))
  5. Contact form receives HTTP 200 + {ok: true} → assumes success, displays "Message sent"
  6. No email handler ever invoked → no SMTP call → no delivery

Root Cause Categories

  • Architectural: Assumed serverless runtime (Vercel Functions) but deployed to static hosting (CF Pages) without serverless equivalent
  • Migration process: No pre-deployment checklist for "dynamic endpoints" (forms, APIs, webhooks)
  • Testing gap: No E2E validation of email delivery — only HTTP response validated (curl 200 != email delivered)
  • Monitoring gap: No alerting on zero-message rate for [email protected] INBOX (expected rate: ~1-3/week)

Detection Method

Manual IMAP inspection using Himalaya CLI:

himalaya search --account [email protected] --folder INBOX "from:noreply" "since:2026-04-19"
# Result: No messages found

Lesson: HTTP 200 is NOT proof of delivery. Always verify end-to-end (inbox check, log inspection, user confirmation email).

Fix Summary

  1. CodeCraft deployed /functions/contact.js as CF Pages Function
  2. Handler uses Resend API (RESEND_API_KEY in Bitwarden → CF Pages env vars)
  3. Form target updated to https://alai.no/api/contact (CF Pages Functions route: /functions/ → /api/)
  4. Proveo validated: submit test form → received at [email protected] within 5 seconds

MC Task: #8587

Lessons Learned

What Went Well

  • CEO noticed absence of expected emails (operational intuition)
  • Himalaya CLI provided rapid IMAP audit without browser login
  • Root cause identified within 15 minutes of investigation start

What Went Wrong

  • Migration checklist did NOT include "verify all POST endpoints have backend handlers"
  • No E2E test protocol for forms (HTTP 200 assumed sufficient)
  • No monitoring/alerting on email delivery rates (silent failure undetected for 40 hours)
  • Cloudflare Tunnel routing too broad (/* catch-all dangerous for multi-service proxy)

Prevention Actions

Action Owner MC Task Status
Update site migration checklist: "Verify form handlers migrated" Skillforge #8587 DONE (this doc)
Create Forms E2E Testing Protocol (HTTP + inbox check required) Skillforge #8587 DONE (BookStack QA section)
Add Grafana alert: [email protected] message rate < 1/week → notify #ops FlowForge #8588 OPEN
Audit all CF Tunnel ingress rules for overly-broad /* patterns Securion #8589 OPEN
Migrate snowit.ba contact form (same silent failure risk) CodeCraft #8591 OPEN
Add form submission logging to all contact handlers (track volume even if email fails) CodeCraft #8592 OPEN
  • snowit.ba contact form: Same root cause (Vercel pattern, no CF Pages handler). Bouncing to [email protected] (LumisCare side, not ALAI). MC #8591 tracks.
  • getdrop.no waitlist: Already migrated correctly (CF Pages Function + D1 storage). No issue.

References

Authored: 2026-04-21 | Owner: Skillforge | Reviewed: John

No comments to display

Back to top