Integration Observability: Making Every Data Pipeline Traceable
Why Integrations Especially Need Observability
When a monolith breaks, the logs are in one place. When an integration breaks, data spans three to five systems, each with its own "view": the marketplace says the order was pushed, the ERP says nothing arrived, and nobody can say where it broke in between. An integration without observability is production entrusted to luck.
The Four-Layer Model
Layer 1: Task-level monitoring. Did the job run, and did it succeed? Metrics: trigger punctuality, duration, success/failure, record counts. This is the minimum alerting surface — if a job doesn't run, someone must know within minutes.
Layer 2: Record-level detail. What happened to this record? For every business record — an order, an outbound bill — log its journey through each node: input snapshot, output snapshot, status, duration, and the verbatim error. Given a business bill number, you should see exactly which step it died on and what the error said.
Layer 3: Lineage-level tracing. What is the full cross-system path of this transaction? Assign a trace ID that spans the whole pipeline (the source system's business number works well), linking the marketplace order, the ERP bill, and the platform callback into one lineage. Cross-system reconciliation becomes a lookup by that ID.
Layer 4: Semantics-level reconciliation. Do the books balance on both sides? A green technical pipeline does not mean correct data — wrong amount semantics or missed enum mappings still report "success". Run regular business-level reconciliation: daily comparisons of order counts, amount totals, and status distributions, with differences logged and assigned.
Alert Grading
| Level | Examples | Response |
|---|---|---|
| P1 Blocking | Scheduler down, auth expired, target unreachable | Page someone immediately, 15-min response |
| P2 Degraded | A record failing persistently, latency over threshold | Handle within 1 hour in business time |
| P3 Notice | Occasional retried successes, unmapped values logged | Daily digest; escalate on worsening trend |
Every alert must carry locating context: flow name, business bill number, failing node, and the original error. An alert that says only "the job failed" is barely better than none.
Replay
The endgame of observability is recoverability. Every failed record should support replay from the failing node — not a full pipeline rerun. Replay presupposes idempotency, which is why idempotency keys recur throughout these guides: without them, replay manufactures duplicates.
Implementation Checklist
Every pipeline has task-level metrics and alerts; any business number's full lineage is queryable within a minute; sanitized request/response payloads are retained for at least the reconciliation horizon; daily business reconciliation has a report and an owner; failed records support single or bulk replay, and replay actions are themselves audited.