Anomaly Detection

You've done anomaly detection your whole life. When a friend's voice sounds slightly off. When traffic patterns feel strange even though you can't say why. When something in your environment has shifted and your brain registers unease before conscious thought catches up.

Anomaly detection teaches machines to develop that same intuition about system behavior.

The Limit of Rules

Traditional monitoring relies on static thresholds: alert if response time exceeds 500ms, if error rate surpasses 1%, if CPU usage goes above 80%. These are rules. They work when normal behavior fits predictable ranges.

But normal isn't static.

Consider a retail website where traffic follows daily rhythms—low overnight, moderate mornings, peak afternoons. A static traffic threshold either cries wolf during normal peaks or sleeps through problems at 3 AM. The rule doesn't know that 1000 requests per second is healthy at 3 PM but alarming at 3 AM.

A static threshold is a rule. Anomaly detection is intuition—the difference between "this number is too high" and "something feels wrong here."

Anomaly detection adapts to:

Temporal patterns where acceptable values change based on time of day, day of week, or season.

Gradual trends where systems grow or decline over time. Last month's normal is this month's below-normal.

Contextual dependencies where the normal value of one metric depends on others. High CPU during high traffic is expected. High CPU during low traffic is suspicious.

Shapes of Wrongness

Anomalies take different forms:

Point anomalies—individual data points that stand alone as unusual. A single request taking 30 seconds when typical requests complete in 100ms. The simplest anomalies to detect, not always the most important.

Contextual anomalies—values that are normal in one context, unusual in another. Ten requests per second is healthy mid-afternoon, alarming at 3 AM. Context determines whether the same number means health or illness.

Collective anomalies—sequences where each individual point looks fine, but the pattern is wrong. Every response time falls within acceptable ranges, but they're all clustering at the high end instead of following their typical distribution. No single request is anomalous. The collection is.

Trend anomalies—changes in the rate of change rather than absolute values. Error rate drifting from 0.1% to 0.5% over hours might never cross a threshold, but that trajectory points toward trouble.

Statistical Foundations

Statistical methods provide the first layer of machine intuition:

Standard deviation approaches flag values falling far from the mean. Points more than two or three standard deviations out are statistically unusual. Works well for normally distributed data, struggles with skewed distributions.

Interquartile range (IQR) identifies outliers based on the spread between the 25th and 75th percentiles. Values beyond 1.5 times the IQR from either quartile are suspicious. Handles skewed distributions better than standard deviation.

Moving averages detect trend changes by comparing recent values to smoothed history. A sudden jump from the smoothed trajectory signals something changed.

Z-scores standardize metrics to enable comparison across different scales. Converting everything to "number of standard deviations from mean" allows consistent detection regardless of whether you're measuring milliseconds or megabytes.

Time as a Dimension

Monitoring data is fundamentally time series—measurements at regular intervals, forming patterns across time. Specialized techniques reveal anomalies that simpler methods miss:

Seasonal decomposition separates data into trend, seasonal, and residual components. By removing known patterns—daily cycles, weekly rhythms—what remains reveals genuine anomalies. Tuesday mornings have a typical shape. Deviation from that shape becomes visible only after accounting for the pattern.

ARIMA modeling predicts expected values based on historical patterns, capturing complex autocorrelation. Significant gaps between prediction and reality indicate anomalies.

Change point detection identifies moments when statistical properties shift abruptly. A sudden change in mean or variance suggests something fundamental changed—often a deployment, configuration change, or emerging failure.

Spectral analysis examines frequency components. If a metric normally cycles every 24 hours but suddenly shows 12-hour cycles, something has changed in the system's rhythm.

Machine Learning Intuition

Machine learning handles complexity that statistical methods can't:

Clustering algorithms group similar data points. Points that don't fit any cluster—isolated, belonging nowhere—are potentially anomalous. K-means, DBSCAN, and hierarchical clustering all enable this unsupervised detection.

Isolation forests exploit the principle that anomalies are easier to isolate than normal points. By randomly partitioning data, points requiring few partitions for isolation reveal themselves as unusual. Scales well to high-dimensional data.

Autoencoders—neural networks trained to reconstruct their input—learn the shape of normal behavior. After training, they struggle to reconstruct anomalous patterns. High reconstruction error signals something the network hasn't seen before.

One-class SVM learns boundaries around normal data in high-dimensional space. Points outside the boundary are classified as anomalies. Requires only examples of normal behavior, not labeled anomalies.

LSTM networks excel at temporal dependencies. Trained on normal sequences, they predict expected next values. Large prediction errors indicate the sequence has deviated from learned patterns.

Metrics in Relationship

Real systems have many interrelated metrics. Examining each in isolation misses anomalies that exist only in their relationships:

Correlation-based detection identifies when normal correlations break down. CPU usage and request rate typically move together. When they diverge—CPU high despite low traffic, or low despite high traffic—something is wrong that neither metric reveals alone.

Principal Component Analysis (PCA) reduces high-dimensional metric spaces to key components. Anomalies appear as points far from the main cluster in reduced-dimension space—complex multivariate anomalies invisible in individual metrics.

Distance-based methods measure how far each data point sits from its neighbors in multidimensional space. Points with unusually distant neighbors are anomalies. k-nearest neighbors and local outlier factor algorithms implement this intuition.

Patterns Within Patterns

Many metrics exhibit patterns that complicate detection:

Multiple seasonality layers daily patterns (peak mid-afternoon) over weekly patterns (lower weekends) over yearly patterns (holiday surges). Effective detection accounts for all levels simultaneously.

Trend-adjusted detection prevents false positives from expected growth. If traffic grows 10% monthly, naive comparison to last month produces constant false alarms. Trend-aware detection adjusts expectations.

Holiday handling requires special baselines for known unusual days. Black Friday traffic looks anomalous compared to normal days—but it's entirely expected for that day.

Baseline evolution keeps detection relevant as systems change. A baseline from six months ago might describe a system that no longer exists.

Tuning the Intuition

Anomaly detection must balance catching problems against crying wolf:

Sensitivity measures how well the system detects real anomalies. High sensitivity catches subtle problems but generates false positives.

Specificity measures how well the system avoids flagging normal behavior. High specificity means fewer false alarms but risks missing real problems.

Adjustable thresholds let teams tune sensitivity to context. Lower thresholds during critical periods accept more false positives for the sake of catching problems. Higher thresholds during maintenance reduce noise from expected unusual behavior.

Confidence scores help prioritize. Rather than binary classification, systems that express how anomalous something appears enable better response decisions.

Making It Work

Effective anomaly detection requires thoughtful implementation:

Start with baselines. Anomaly detection builds on understanding normal behavior. Without that foundation, you're detecting deviation from nothing.

Combine methods for comprehensive coverage. Statistical methods catch obvious deviations. Machine learning handles complex patterns. Domain-specific rules encode known failure modes.

Include feedback loops where operators confirm or reject alerts. This feedback tunes algorithms and improves accuracy over time.

Provide context when alerting. Show the anomaly alongside historical patterns, related metrics, and potential causes. Context helps responders assess whether action is needed.

Roll out gradually. Start in informational mode—surface anomalies without alerting. After validating accuracy, enable alerting for high-confidence detections.

The Hard Problems

Anomaly detection faces real challenges:

Cold start—new services lack history. Detection needs examples of normal behavior to learn from. New services require time to establish baselines.

Concept drift—normal behavior changes. Models trained on old patterns might flag new normal as anomalous. Continuous retraining helps but adds complexity.

High cardinality—complex systems with thousands of unique metrics require sophisticated approaches and significant computational resources.

Explainability—when a deep learning model flags something, explaining why can be difficult. This complicates trust in the system.