Log Aggregation

When something breaks in a distributed system, the first question is always: what happened? But the answer is scattered across fifty servers, thirty microservices, and a hundred containers—some of which no longer exist.

Log aggregation solves this by collecting logs from everywhere into one searchable place. It's how distributed systems remember.

The Problem with Scattered Logs

Modern applications are distributed. A single user request might touch an API gateway, authentication service, payment processor, inventory system, and database—each running on different machines, each writing its own logs.

Without aggregation, investigating an issue means:

Manual archaeology: SSH into each server, grep through files, try to piece together what happened. For fifty servers, this takes hours.

Racing against deletion: Containers die and take their logs with them. Auto-scaled instances disappear. Old logs get rotated away. The evidence vanishes.

Impossible correlation: Even if you find relevant logs on multiple machines, aligning timestamps and connecting events across services is manual detective work.

Log aggregation centralizes everything. One search, all your infrastructure, instant results.

How It Works

Collection

Logs flow from sources to the aggregation system through several paths:

Log shippers like Filebeat or Fluentd run on servers, watch log files, and forward new entries to the central system.

Direct logging sends logs straight from applications via APIs—no files involved.

Container platforms capture stdout/stderr from containers automatically.

Cloud integration collects logs from managed services without any configuration.

Parsing

Raw log text gets transformed into structured data:

2024-01-15 10:30:45 ERROR User alice failed login from 192.168.1.100

Becomes:

{
  "timestamp": "2024-01-15T10:30:45Z",
  "level": "ERROR",
  "user": "alice",
  "action": "login",
  "result": "failed",
  "ip": "192.168.1.100"
}

Now you can search for all failed logins, or all events from alice, or all errors from that IP address.

Indexing and Storage

Billions of log entries need fast searching. Aggregation systems build indexes—organized by timestamp, by field values, by full text—so queries return in seconds, not hours.

Storage is tiered: recent logs on fast disks, older logs on cheaper storage, ancient logs in archives. Eventually, logs are deleted. Keeping everything forever costs too much.

Using Aggregated Logs

Debugging an Incident

Errors spike at 3 AM. With aggregated logs:

1. Search: level:ERROR AND timestamp:[now-5m TO now]
2. See all errors across all services in one view
3. Notice pattern: every error mentions "connection timeout to database"
4. Root cause found in minutes, not hours

Tracing a Request

A user reports a failed checkout. Their request ID is req-abc-123.

Search: request_id:req-abc-123

You see the request's journey: API gateway received it, auth service validated the token, payment service attempted the charge, and... "credit card validation failed." The whole story, one search.

This only works if your applications include correlation IDs in every log entry. Without them, you're back to guessing which logs belong to which request.

Alerting

Create alerts based on log patterns:

• Error rate exceeds threshold
• Specific error messages appear (database connection failures)
• Expected logs stop appearing (health check hasn't logged in 5 minutes)

Structured Logging

Log aggregation is only as good as the logs you send it.

Unstructured (hard to search):

logger.info(f"User {user_id} purchased product {product_id} for ${amount}")

Structured (easily searchable):

logger.info("purchase_completed", extra={
    "user_id": user_id,
    "product_id": product_id,
    "amount": amount,
    "currency": "USD"
})

With structured logs, you can search by any field: find all purchases over $100, all actions by a specific user, all events involving a specific product.

What to Include

Correlation IDs: Request IDs, session IDs, user IDs—anything that connects logs across services.

Consistent field names: If one service logs user_id and another logs userId, your searches won't find everything.

Appropriate levels: ERROR for failures, WARN for concerning situations, INFO for normal events, DEBUG for detailed diagnostics.

What to exclude: Passwords, credit card numbers, personal information. These create security and compliance nightmares.

Common Systems

ELK Stack (Elasticsearch, Logstash, Kibana): Open source, powerful, requires operational expertise.

Splunk: Commercial, advanced analytics, expensive.

Cloud-native (AWS CloudWatch, Google Cloud Logging, Azure Monitor): Integrated with cloud services, locked to that provider.

Grafana Loki: Cost-efficient, indexes metadata instead of full text.

The Challenges

Volume

Large systems generate terabytes of logs daily. Solutions:

• Sample verbose logs (keep 1% of debug logs)
• Shorter retention for high-volume, low-value logs
• Tiered storage to balance cost and access speed

Sensitive Data

Logs accidentally capture passwords, tokens, personal information. Solutions:

• Scrub sensitive data before logging
• Encrypt logs at rest and in transit
• Control who can access which logs

Cost

Commercial aggregation charges per GB. Costs grow fast. Manage by sampling, filtering noisy logs, and choosing appropriate retention periods.

Performance

Logging consumes CPU, disk, and network. Applications should log asynchronously—buffer entries and send them in batches rather than blocking on every log call.

Retention

Different logs need different retention:

7 days: Debug logs, high-volume verbose logs 30 days: Standard application logs 1 year+: Audit logs, security logs, compliance-required logs

Regulations like GDPR, HIPAA, and PCI-DSS mandate specific retention periods and access controls. Design policies accordingly.

Best Practices

Aggregate everything: Application logs, server logs, database logs, network devices. If it generates logs, centralize it.

Require correlation IDs: Every request should have an ID that appears in every log entry it generates.

Tag by environment: Distinguish production from staging from development.

Monitor the aggregation system itself: If log collection fails, you won't know until you need those missing logs.

Test your searches: Regularly verify you can find the information you'd need during an incident.

When a container dies, it takes its memories with it. Log aggregation is how distributed systems remember—turning scattered, ephemeral evidence into a searchable record of everything that happened.