Monitor what matters. The four golden signals provide a powerful framework for understanding system health and performance, helping you detect and resolve issues before users notice.
As someone who's spent years in the trenches of systems reliability, I've learned that monitoring everything leads to notification fatigue and missed alerts. Four key metrics cut through the noise - latency, traffic, errors, and saturation. These golden signals form the foundation of effective Site Reliability Engineering practices.
Let's explore what makes these metrics so valuable and how to implement them in your monitoring strategy.
Table of Contents
What Are the SRE Golden Signals?
The golden signals concept comes from Google's Site Reliability Engineering (SRE) practices. These four metrics provide a comprehensive view of service health from a user perspective:
- Latency: The time it takes to service a request
- Traffic: The demand placed on your system
- Errors: The rate of failed requests
- Saturation: How "full" your service is
What makes these signals particularly valuable is their focus on user experience rather than system internals. They answer the question: "Is my service working well for users?" rather than "Are my servers running properly?"
I've found that teams often make the mistake of monitoring what's easy to measure rather than what actually matters. CPU usage might look normal while users experience terrible response times. Golden signals keep you focused on metrics that directly impact users.
Latency: Measuring Response Time
Latency measures how long it takes your system to respond to a request. It's the most direct indicator of user experience - when latency increases, users notice immediately.
Key aspects of latency monitoring:
- Measure successful and failed requests separately: Failed requests often have different latency characteristics
- Track percentiles, not just averages: A 99th percentile latency spike may affect only 1% of users, but their experience is still important
- Compare against baselines: Know what "normal" looks like for your service
For example, instead of just tracking average response time, consider this more nuanced approach:
| Metric | Description | Warning Threshold | Critical Threshold |
|---|---|---|---|
| p50 Latency | Median response time | 100ms | 200ms |
| p95 Latency | 95th percentile response time | 250ms | 500ms |
| p99 Latency | 99th percentile response time | 500ms | 1000ms |
I've seen cases where the average latency looked fine, but a small percentage of requests were timing out completely. This created a terrible experience for those users while our monitoring showed green across the board. After implementing percentile-based latency monitoring, we caught these issues immediately.
Traffic: Tracking System Demand
Traffic measures the load on your system - how many requests it's handling. This signal helps you understand utilization patterns and detect abnormal spikes or drops.
Effective traffic monitoring considerations:
- Define appropriate metrics: HTTP requests per second, database queries, message queue throughput, etc.
- Segment by important dimensions: Client type, region, endpoint, etc.
- Look for patterns: Daily, weekly, or seasonal variations
A traffic drop can indicate a serious problem upstream, while unexpected spikes might warn of a DDoS attack or viral content. Both scenarios require attention.
One startup I worked with couldn't figure out why their database was overloaded every Monday morning. By tracking traffic patterns, we discovered that a weekly batch job was running at the same time as the Monday traffic peak. Simply rescheduling the job to Sunday night solved the problem without any code changes or infrastructure upgrades.
Errors: Identifying Failures
The error signal tracks failed requests - explicit failures like HTTP 500s, implicit failures like serving the wrong content, or policy violations like exceeding time budgets.
Key aspects of error monitoring:
- Track error rates, not counts: A 1% error rate might be acceptable, but 1,000 errors per second probably isn't
- Categorize errors: Distinguish between server errors, client errors, and data errors
- Set appropriate thresholds: Some errors are expected, but sudden changes need attention
For web services, a simple approach is tracking HTTP status codes:
| Error Type | HTTP Codes | Example Alert Threshold |
|---|---|---|
| Client Errors | 400-499 | >5% of requests |
| Server Errors | 500-599 | >0.1% of requests |
But don't stop at HTTP codes. I once debugged a service that returned HTTP 200 responses containing error messages in the response body. From the monitoring perspective, everything looked fine, but users were seeing errors. We had to enhance our error detection to parse response bodies for error messages.
Saturation: Measuring System Capacity
Saturation measures how "full" your system is - how close it is to its capacity limit. As systems approach saturation, performance typically degrades before complete failure.
Key saturation metrics to monitor:
- Resource utilization: CPU, memory, disk I/O, network bandwidth
- Queueing: Request queues, thread pool utilization, database connection pools
- Constraints: Rate limits, quotas, circuit breakers
Saturation is often the most complex signal to measure correctly. A system might have multiple potential bottlenecks, and saturation in one component can cause cascading issues.
I remember debugging a mysteriously slow API that had plenty of CPU and memory available. The issue turned out to be database connection pool saturation - requests were queuing up waiting for database connections. Once we identified and expanded the connection pool, performance returned to normal.
This table shows some common saturation metrics and their warning signs:
| Resource | Metric | Warning Signs |
|---|---|---|
| CPU | Utilization percentage | Sustained periods >70% |
| Memory | Available memory, swap usage | <20% free, any swap usage |
| Disk | I/O utilization, queue length | Sustained I/O >80%, queue length >1 |
| Network | Bandwidth utilization | Sustained periods >70% of capacity |
| Threads | Thread pool utilization | >80% of maximum threads in use |
| Connections | Connection pool usage | >80% of available connections in use |
Implementing Golden Signals Monitoring
Implementing golden signals requires both technical setup and organizational buy-in. Here's a practical approach:
- Identify your services: Map your system architecture to understand service boundaries
- Define appropriate metrics: Choose the right metrics for each golden signal based on your technology stack
- Instrument your code: Add the necessary monitoring hooks
- Visualize the data: Create dashboards that make the signals understandable
- Set up alerts: Configure thresholds and notification channels
- Iterate and refine: Continuously improve your monitoring based on incidents and feedback
Start small and expand gradually. Pick a critical service, implement basic golden signals monitoring, and use that experience to refine your approach before scaling to other services.
Most modern monitoring tools support the metrics needed for golden signals. For example:
- Prometheus/Grafana: Great for metrics collection and visualization
- New Relic/Datadog: Provide application performance monitoring with built-in support for golden signals
- ELK Stack: Good for log-based metrics extraction
- OpenTelemetry: Open standard for instrumentation that works with many backends
Common Implementation Challenges
Implementing golden signals isn't always straightforward. Here are some challenges you might face:
Distributed systems complexity: In microservices architectures, a single user request might touch dozens of services. Distributed tracing becomes essential for accurate latency measurement.
Legacy systems: Older systems may lack instrumentation capabilities. You might need to rely on external monitoring or logs analysis.
Data volume: Collecting detailed metrics at scale generates significant data. Sampling strategies and data retention policies become important.
Alert fatigue: Setting thresholds too tight leads to constant alerts. Start with conservative thresholds and tighten them gradually.
Organizational silos: Different teams might own different parts of the system. Cross-team collaboration is essential for end-to-end monitoring.
One particularly tricky challenge is determining what "normal" looks like. I've seen teams set static thresholds that trigger constant alerts during peak hours and miss problems during quiet periods. Consider using adaptive thresholds that account for historical patterns.
Golden Signals vs. RED Method
The RED method is another popular monitoring framework, especially for microservices. RED stands for:
- Rate: The number of requests per second
- Errors: The number of failed requests per second
- Duration: The amount of time it takes to process a request
You might notice RED is essentially a subset of the golden signals, focusing on latency, traffic, and errors but omitting saturation. This makes RED somewhat simpler to implement but potentially less comprehensive.
If you're primarily concerned with service-level monitoring, RED might be sufficient. If you need to understand resource constraints and capacity planning, the golden signals provide a more complete picture.
I often recommend RED as a starting point for teams new to modern monitoring practices. Once you've mastered those three signals, adding saturation monitoring is a natural next step.
Golden Signals vs. USE Method
The USE method is a different approach focused on resources rather than services:
- Utilization: Percentage of time the resource is busy
- Saturation: Amount of work the resource can't service
- Errors: Count of error events
USE is particularly valuable for infrastructure monitoring, while golden signals excel at service monitoring. They're complementary rather than competitive approaches.
In practice, I've found that combining these methodologies works well:
- Use golden signals for service-level monitoring (user perspective)
- Apply USE for resource-level monitoring (system perspective)
This provides a comprehensive view from both user experience and infrastructure health angles.
Practical Example: Golden Signals in Action
Let's walk through a practical example of golden signals monitoring for a web API:
Latency monitoring:
histogram_quantile(0.50, sum(rate(http_request_ duration_seconds_bucket[5m])) by (le))
histogram_quantile(0.95, sum(rate(http_request_ duration_seconds_bucket[5m])) by (le))
histogram_quantile(0.99, sum(rate(http_request_ duration_seconds_bucket[5m])) by (le))
Traffic monitoring:
sum(rate(http_ requests_total[5m]))
Error monitoring:
sum(rate(http_r equests_total{status= ~"5.."}[5m])) / sum(rate(http_ requests_total[5m]))
Saturation monitoring:
avg by (instance) (rate (node_cpu_seconds_total {mode!="idle"}[5m]))
# Memory saturation
(node_memory_ MemTotal_bytes - node_memory_ MemAvailable_bytes) / node_memory_ MemTotal_bytes
# Connection pool saturation
max_over_time (connection_pool_active_ connections[5m]) / max_over_time (connection_pool_max_ connections[5m])
With these metrics in place, you could create a dashboard that looks something like this:
| Signal | Current | 24h Avg | Status |
|---|---|---|---|
| P95 Latency | 120ms | 95ms | ✅ |
| Requests/sec | 350 | 275 | ✅ |
| Error Rate | 0.02% | 0.01% | ✅ |
| CPU Saturation | 62% | 45% | ✅ |
| Memory Saturation | 75% | 70% | ⚠️ |
| Conn Pool Saturation | 83% | 60% | ⚠️ |
This gives you an at-a-glance view of system health across all four golden signals.
Advanced Techniques for Golden Signals
Once you've implemented basic golden signals monitoring, consider these advanced techniques:
Signal correlation: Analyze relationships between signals. For example, does latency increase when traffic spikes? This helps identify bottlenecks.
Synthetic transactions: Use automated tests to continuously verify service behavior, providing a consistent baseline for latency and error measurements.
Canary analysis: Compare golden signals between canary and production deployments to catch issues before full rollout.
Anomaly detection: Apply machine learning to detect unusual patterns in golden signals that might indicate problems.
Multi-dimensional analysis: Break down signals by region, user type, endpoint, etc., to identify specific problem areas.
One technique I've found particularly useful is "service level indicators" (SLIs) derived from golden signals. For example, an SLI might be "99% of requests complete in under 200ms." This translates raw metrics into business-relevant indicators.
Golden Signals and Business Metrics
While golden signals focus on technical performance, they directly impact business outcomes. Making these connections explicit helps communicate the importance of reliability to non-technical stakeholders.
Consider these relationships:
| Golden Signal | Business Impact |
|---|---|
| Latency | User satisfaction, conversion rate |
| Traffic | Customer engagement, growth |
| Errors | User frustration, support costs |
| Saturation | Scalability, cost efficiency |
For example, research shows that every 100ms of added latency can reduce conversion rates by up to 7%. Tracking both the technical metrics and business outcomes helps build a shared understanding of why reliability matters.
I worked with an e-commerce company that started correlating their golden signals with cart abandonment rates. They discovered that even slight increases in latency during checkout had a measurable impact on revenue. This made it much easier to justify investments in performance optimization.
Conclusion
The SRE golden signals provide a powerful framework for monitoring what matters most in your systems. By focusing on latency, traffic, errors, and saturation, you can build a monitoring strategy that directly connects to user experience and business outcomes.
Remember that implementing golden signals is a journey, not a destination. Start simple, iterate based on real incidents, and continuously refine your approach. The goal isn't perfect monitoring but rather effective monitoring that helps you deliver reliable services to your users.
If you're looking for an easy way to implement golden signals monitoring for your websites and APIs, Odown provides powerful tools to track these critical metrics. With Odown, you can monitor uptime, response times, and errors from multiple locations around the world. The platform also offers SSL certificate monitoring to prevent unexpected expirations and public status pages to keep your users informed during incidents. Start monitoring what matters today.
