Logging & Monitoring
The difference between finding out about an outage from a monitoring alert versus from an angry customer.
5.13.1Definition
Logging records discrete events — a request, an error, a state change — as a timestamped, searchable trail. Monitoring aggregates metrics over time (response latency, error rate, CPU usage) and alerts when they cross a threshold. Together they form a system's primary window into its own behavior once it's running in production, beyond what any individual developer can observe directly.
5.13.2Why It Exists
Production systems fail in ways that are rarely reproducible on a developer's machine — intermittent network issues, edge-case data, load-dependent race conditions. Logging and monitoring exist to make a system's internal behavior visible after the fact and to surface problems proactively, ideally before a user reports them, rather than relying entirely on user complaints as the detection mechanism.
5.13.3The Three Pillars
| Pillar | Answers | Example tool |
|---|---|---|
| Logs | What exactly happened, in detail, for one event | Datadog, CloudWatch Logs, self-hosted ELK |
| Metrics | How is the system performing in aggregate, over time | Prometheus, Grafana, Datadog |
| Traces | How did one request move through a multi-service system | Jaeger, OpenTelemetry, Datadog APM |
5.13.4Common Mistakes
- Logging so verbosely that signal is buried in noise, making genuine errors hard to find among routine debug output at production volume.
- Logging sensitive data in plaintext — passwords, tokens, full card numbers — turning log storage into a compliance and security liability.
- No alerting configured at all, treating logs and dashboards purely as a passive record checked only after a problem has already been reported by a user.
- Alert fatigue from poorly-tuned thresholds, causing real alerts to be ignored alongside a flood of false positives.
5.13.5Best Practices
- Use structured logging (JSON, not free-text) so logs are queryable rather than only readable line by line.
- Redact or omit sensitive fields from logs entirely, at the point of logging.
- Alert on symptoms users would notice (error rate, latency) rather than only on low-level internal metrics.