Alert Deduplication and Correlation

When a database crashes, dozens of dependent services generate alerts simultaneously. Without deduplication and correlation, engineers receive fifty notifications about what is fundamentally one problem. The database crashes, and thirty-five services all raise their hands to tell you the same thing: "I can't reach the database." They're not wrong. They're just not helpful.

These techniques transform alert storms into coherent incident information that answers the only question that matters at 3 AM: what actually broke?

The Alert Storm Problem

Single failures trigger cascading alerts:

A database fails. Within seconds:

• 20 web servers: "Can't connect to database"
• 10 background workers: "Database query timeout"
• 5 API services: "Database unavailable"
• Monitoring system: "Health check failing"
• Load balancer: "All backends unhealthy"

That's 36 notifications about one problem. The most important information—the database failure—gets buried under a pile of symptoms.

A network partition occurs. Dozens of services on one side can't reach services on the other. Both sides generate alerts about unreachable dependencies. The flood obscures the actual issue: network connectivity.

A deployment breaks authentication. Every service requiring authentication fails. Each alerts separately despite sharing a root cause.

Deduplication: Same Alert, One Notification

Deduplication identifies identical or nearly-identical alerts and groups them:

Exact deduplication: Server CPU > 90% fires every minute for 10 minutes. Instead of 10 notifications, send one: "Server CPU > 90% — fired 10 times in the last 10 minutes."

Source-based deduplication: 20 web servers all alert "Disk space > 90%." Instead of 20 notifications: "Disk space > 90% on 20 web servers." One server with disk issues is a local problem. Twenty servers suggests something systemic.

Time-based deduplication: API response time degradation fires every 5 minutes for an hour. Instead of 12 notifications, send an initial alert, then periodic updates: "API response time still degraded — ongoing for 45 minutes."

Alert systems track fingerprints (unique identifiers) for each alert type and source. When the same fingerprint appears multiple times within a time window—typically 5-10 minutes—identical alerts merge.

Correlation: Finding the Root Cause

Correlation identifies relationships between different alerts:

Dependency-based correlation: Database fails. The alert system knows 35 services depend on this database. When those services alert, correlation identifies the database as probable root cause and aggregates the dependent alerts.

Notification becomes: "Database failure — HIGH severity" followed by "35 dependent services reporting failures, likely due to database issue."

Temporal correlation: Within a 2-minute window, authentication service reports high error rates, web servers report auth failures, and mobile API reports login problems. Temporal proximity suggests these are related, not three separate incidents.

Even without explicit dependency graphs, alerts firing together usually share a cause.

Pattern-based correlation: Multiple services in the same datacenter alert simultaneously. The pattern suggests a datacenter-level issue, not independent failures.

Patterns include:

• Same datacenter or availability zone
• Same network segment
• Same cloud provider region
• Same third-party dependency
• Same recent deployment

The First Alert Usually Wins

Correlation should identify likely root causes:

Timeline analysis: The first alert in a storm is usually the answer. Everything after is just the system agreeing with itself. If database alerts at 2:00:00 AM and web servers alert at 2:00:05 AM, the database is the root cause.

Dependency analysis: Services closer to the root of dependency graphs are more likely root causes. Database failures cause application failures, not the reverse.

Impact scope: Failures affecting many downstream services are likely root causes. Authentication failing across all services suggests authentication is the problem.

Infrastructure vs. application: Infrastructure failures cascade to application symptoms. Network, database, and compute problems appear first; applications complain second.

Present root cause hypotheses to engineers: "Database failure likely causing 23 related alerts" rather than just grouping alerts without context.

Notification Strategy

Primary alert: Send the likely root cause as high priority. "Database failure — HIGH severity."

Secondary summary: Send a grouped summary of related alerts with lower urgency. "23 services reporting failures likely due to database issue."

Progressive updates: As more alerts correlate, update the existing incident rather than sending new notifications. "Database incident: now affecting 35 services (was 23)."

Clear relationships: Explicitly state dependencies. "Payment API failure — dependency: database [reported failed 5 minutes ago]."

Resolution propagation: When root cause resolves, automatically resolve related alerts. Fixing the database clears all dependent service alerts.

Tuning the System

Time windows: Too short and related alerts don't correlate. Too long and unrelated issues incorrectly group. Start with 5 minutes and adjust.

Threshold counts: Don't deduplicate until at least 3-5 similar alerts fire. One or two might be coincidence.

Severity handling: Preserve highest severity when grouping. If one alert is critical and nineteen are warnings, the group is critical.

Where Correlation Goes Wrong

False correlation: Two unrelated incidents happen simultaneously and get treated as one. Both get delayed.

Root cause misidentification: Database slows because network fails. Correlation identifies database slowness (first observable symptom) as root cause when network is the real issue.

Over-suppression: Suppressing too many secondary alerts hides important information. Engineers need the full picture, just organized coherently.

Delayed notification: Waiting to correlate can delay initial response. Balance correlation benefits against notification speed.

Measuring Effectiveness

Reduction ratio: 100 alerts becoming 5 notifications shows effective correlation.

Accuracy rate: Post-incident analysis reveals whether the identified root cause was correct.

Time to root cause: Good correlation reduces how long engineers spend figuring out what actually broke.

Engineer feedback: Did grouped alerts help or hinder response? The people getting paged know best.