What is Prometheus?
Prometheus, an open-source monitoring system, focuses on tracking metrics and performance analysis for applications and infrastructure. It boasts a straightforward yet potent data model and a query language for deep insights into your system’s behavior. Unlike trying to tackle various issues, Prometheus concentrates solely on metrics.
Developed mainly in Go and using the Apache 2.0 license, Prometheus employs a distinctive data model. Time series are identified not just by names but also by label sets, which are sets of key-value pairs. This enables flexible aggregation across labels, allowing analysis at various levels, like per process, datacenter, service, or other defined labels. These insights can be visualized in tools like Grafana.
The PromQL query language, used for graphing, also serves as the foundation for defining alerts. Anything graphable is alertable. Labels simplify alert maintenance, as one alert can cover all label values, as opposed to creating individual alerts for each application or machine. Additionally, service discovery automatically identifies what applications and machines to monitor from sources like Kubernetes, Consul, AWS EC2, Azure, GCE, and OpenStack.
Prometheus excels in efficiency and simplicity. It can handle a large number of samples per second, and its components can run in containers. Its design emphasizes compatibility with existing infrastructure and tools, rather than being a management platform itself. It’s all about seamless integration and built-in adaptability.
What Is Monitoring
The role of monitoring encompasses several essential aspects:
In summary, monitoring involves alerting for timely intervention, debugging to pinpoint and fix issues, trending for long-term insights, and even repurposing monitoring components for other tasks as needed.
Categories of Monitoring
In the realm of monitoring, the core focus remains consistent: events. These events can encompass a wide array of occurrences.
Every event carries its own context. For instance, consider an HTTP request; it brings along the source and destination IP addresses, the requested URL, cookies in use, and details about the requesting user. On the flip side, an HTTP response carries information like response duration, HTTP status code, and response body size. Events tied to functions come with their call stacks, along with the triggers like an initiating HTTP request.
While having complete event context would be ideal for comprehensive debugging and performance understanding – both technically and in business terms – dealing with such massive data volumes isn’t practical. Hence, there are roughly four strategies to handle this data volume and make it manageable: profiling, tracing, logging, and metrics.
Profiling
Profiling adopts a pragmatic approach, acknowledging that obtaining complete context for all events all the time isn’t feasible. Instead, it suggests gathering partial context during specific time intervals.
For example, tools like Tcpdump enable the recording of network traffic based on filters. Although vital for debugging, such tools can’t be run continuously due to space constraints. Debug builds of software, tracking profiling data, offer valuable insights, but the performance impact of collecting extensive information, such as timing of each function call, restricts their practical use in production environments.
The Linux kernel introduces enhanced Berkeley Packet Filters (eBPF), allowing detailed profiling of various kernel events like filesystem operations and network behaviors. This offers unprecedented insights and comes with benefits like portability and safety.
Profiling mainly serves tactical debugging. When used for more extended periods, data volume needs to be reduced to fit into other monitoring categories, or continuous profiling becomes necessary. Continuous profiling minimizes data volume and overhead by lowering profiling frequency. Emerging tools like the eBPF-based Parca Agent operate at a reduced frequency, aiming for statistically significant data over minutes rather than seconds. This approach provides the needed insights into CPU usage across infrastructure, enhancing application efficiency where required.
Tracing
Tracing, in contrast to profiling, doesn’t aim to capture all events. Instead, it selects a fraction of events, often around one in a hundred, that traverse specific functions of interest. This involves noting functions in stack traces at these points and often recording the execution times of these functions. This approach provides insights into program behavior and identifies latency-contributing code paths.
Some tracing systems go beyond snapshotting stack traces. They record timings for every function call beneath the function of interest. For instance, one in a hundred user HTTP requests might be sampled, revealing time spent on backend interactions like databases and caches. This allows analysis of timing discrepancies based on factors such as cache hits and misses.
Distributed tracing takes this further by extending tracing across processes. It attaches unique IDs to requests passed between processes in remote procedure calls (RPCs). This enables stitching together traces from different processes and machines using the request ID. Distributed tracing is crucial for debugging complex microservices architectures. Technologies like OpenZipkin and Jaeger serve in this capacity.
In tracing, the key lies in sampling, which maintains reasonable data volumes and minimizes performance impact from instrumentation.
Logging
Logging targets a specific set of events and captures some context for each of these events. For instance, it might focus on incoming HTTP requests or outgoing database calls. A practical guideline is to limit each log entry to about a hundred fields to prevent excessive resource consumption. Exceeding this limit could lead to bandwidth and storage concerns.
To illustrate, for a server handling 1,000 requests per second, a log entry with 100 fields of 10 bytes each translates to 1 megabyte per second. This size can strain a 100 Mbit network card and consume 84 GB of storage daily.
Logging offers the advantage of typically not requiring event sampling. This makes it possible to analyze the impact of slow requests on a specific user communicating with a particular API endpoint.
Logging, like monitoring, carries different meanings depending on who you ask, leading to confusion. It can be categorized into four somewhat overlapping groups:
Treating all log types uniformly can lead to challenges, combining the data volume of debug logs with the stringent reliability requirements of transaction logs. As systems grow, it’s advisable to separate debug logs for individual handling.
Examples of logging systems encompass the ELK stack, OpenSearch, Grafana Loki, and Graylog.
Metrics
Metrics-focused monitoring operates by aggregating over time, sidestepping detailed context tracking. To maintain reasonable resource usage, the number of distinct tracked values should be limited, often around 10,000 per process.
Metrics involve aggregating data like the count of received HTTP requests, request handling time, and current in-progress requests. By excluding context details, data volume and processing remain manageable.
However, some context can still be included. For example, you could have a metric for each URL path in an HTTP request. Yet, the 10,000 metric guideline applies, as each distinct path counts as a metric. Incorporating unbounded context, like a user’s email address, would be unwise due to high cardinality.
Metrics are great for tracking latency and data volumes within subsystems, aiding in diagnosing slowdowns. While logs can’t record as many fields, they’re useful once you identify the problematic subsystem, helping pinpoint specific user requests involved.
This highlights the trade-off between logs and metrics. Metrics provide data from various parts of the process, often with limited context fields. Logs excel at gathering information about a specific event type but with a higher context field limit.
Prometheus, as a metrics-based monitoring system, focuses on overall system health, behavior, and performance rather than individual events. It’s concerned with insights like 15 requests in the last minute taking 4 seconds to process, resulting in 40 database calls, 17 cache hits, and 2 customer purchases. Profiling and logging are more suited for detailed analysis of individual calls.
Prometheus Architecture
Prometheus employs service discovery to find targets for data collection. These targets can include your applications with instrumentation or third-party apps accessible through an exporter. Gathered data is stored, usable in dashboards via PromQL, and can trigger alerts. Alertmanager then transforms alerts into notifications like pages and emails.
Client Libraries
Client libraries play a pivotal role in generating metrics from applications. Adding instrumentation to code becomes effortless with just a few lines, a practice known as direct instrumentation.
Prometheus offers official client libraries for popular languages like Go, Python, Java/JVM, Ruby, and Rust. Additionally, third-party libraries exist for languages like C#/.Net, Node.js, Haskell, and Erlang.
These client libraries handle intricate aspects such as thread safety, bookkeeping, and producing Prometheus text or OpenMetrics exposition formats in response to HTTP requests. Unlike event-based monitoring, memory usage in client libraries isn’t influenced by the volume of events but rather by the number of metrics.
When a library dependency of your application includes Prometheus instrumentation, it’s automatically recognized. By instrumenting a crucial library like your RPC client, you extend instrumentation across all your applications.
Certain metrics, like CPU usage and garbage collection stats, are often readily available through client libraries, contingent on the library and runtime environment.
Exporters
Not all code you utilize can be directly instrumented due to limitations in control or access. For instance, expecting operating system kernels to produce Prometheus-formatted metrics over HTTP isn’t likely to happen soon.
In such cases, software often offers some form of metric access. This might involve an ad hoc format requiring custom handling, common for many Linux metrics, or a well-established standard like SNMP.
An exporter is a software component placed alongside the target application to fetch metrics. It receives requests from Prometheus, collects required data from the application, transforms it to the appropriate format, and responds to Prometheus. Imagine an exporter as a small proxy, translating data between the application’s metric interface and the Prometheus format.
Unlike direct instrumentation used for code under your control, exporters employ custom collectors or ConstMetrics for instrumentation.
Service Discovery
Once your applications are instrumented and exporters are up and running, it’s important for Prometheus to be aware of their presence. This ensures Prometheus monitors the right targets and detects unresponsive components. In dynamic environments, providing a static list of applications and exporters becomes outdated quickly. This is where **service discovery** comes into play.
You likely maintain a database detailing your machines, applications, and their functions. This information could reside in tools like Chef, Ansible inventory files, EC2 instance tags, Kubernetes labels, annotations, or even in your documentation wiki.
Prometheus seamlessly integrates with common service discovery mechanisms like Kubernetes, EC2, and Consul. There’s also a generic integration for setups that diverge from the norm, like using “File” and “HTTP” mechanisms.
However, a challenge remains. Having a list of machines and services isn’t enough to understand their role in your architecture. Different organizations might use distinct conventions, such as using the EC2 Name tag or an “app” tag to signify the application on a machine.
Acknowledging these variations, Prometheus offers relabeling to configure how metadata from service discovery is mapped to monitoring targets and their labels. This ensures flexibility in adapting to your organization’s unique setup.
Scraping
After identifying targets through service discovery and configuring them using relabeling, Prometheus proceeds to retrieve metrics. This is accomplished by sending an HTTP request, known as a scrape, to the target. The response received from the scrape is parsed and stored within the Prometheus storage. Importantly, this process introduces additional metrics, including the success status of the scrape and its duration.
Scrapes are conducted at regular intervals. Typically, they are configured to occur every 10 to 60 seconds for each target. This recurring fetch ensures that Prometheus continuously updates its metric data from the monitored applications and systems.
Storage
Prometheus stores data locally within a custom database. The choice not to implement clustering aims at ensuring reliability and simplifying operational management.
Over time, Prometheus has undergone several redesigns for its storage system, with the third iteration introduced in Prometheus 2.0. This storage system is capable of efficiently handling the ingestion of millions of samples per second. This capability enables monitoring of extensive systems – even thousands of machines – with a single Prometheus server. The storage system utilizes a compression algorithm that achieves efficient data representation at around 1.3 bytes per sample for real-world data.
Dashboards
Prometheus offers several HTTP APIs enabling data retrieval and PromQL query evaluation. These APIs facilitate graph and dashboard creation. While Prometheus provides the expression browser for ad hoc querying and data exploration, it isn’t a comprehensive dashboard system.
For dashboard needs, it’s recommended to use Grafana. Grafana boasts numerous features and officially supports Prometheus as a data source. Notably, Grafana can interact with multiple Prometheus servers, even within a single dashboard panel.
Recording Rules and Alerts
While PromQL and the storage engine are potent and efficient, real-time aggregation of metrics from numerous machines during graph rendering can occasionally lead to performance delays. Recording rules provide a solution by allowing regular evaluation of PromQL expressions, with their outcomes integrated into the storage engine.
Alerting rules, a type of recording rule, also conduct periodic PromQL expression evaluations. These evaluations produce alerts whenever the expressions yield results that satisfy predefined conditions. These alerts are then transmitted to the Alertmanager for further handling.
Alert Management
The Alertmanager serves as an intermediary between Prometheus servers and notifications. It transforms received alerts into various forms of notifications, which can include emails, communication platforms like Slack, and services like PagerDuty.
However, the Alertmanager’s role extends beyond simple alert-to-notification conversion. It has features like aggregating related alerts into a single notification, controlling alert frequency to avoid overwhelming notifications (known as pager storms), and configuring distinct routing and notification outputs for different teams. The Alertmanager also allows for alert silencing, handy for temporarily suppressing known issues during scheduled maintenance.
It’s crucial to note that the Alertmanager solely handles the dispatch of notifications. For managing human responses to incidents, it’s recommended to integrate services like PagerDuty and ticketing systems.
Long-Term Storage
Prometheus stores data locally, which means your storage capacity is constrained by the disk space on the hosting machine. Typically, you’re concerned with recent data, but for long-term planning, extending the retention period is valuable.
Prometheus doesn’t provide a clustered storage solution for distributing data across multiple machines. However, it offers remote read and write APIs, enabling external systems to step in and handle this task. With these APIs, PromQL queries can be executed seamlessly against both local and remote data sources. This approach helps bridge the gap when dealing with limited local storage for long-term data retention needs.
Visualizing with Grafana
When dealing with alerts and monitoring the real-time performance of your systems, dashboards become your primary resource. The expression browser, while useful for ad hoc graphing and PromQL debugging, isn’t meant to serve as a dashboard.
In this context, a dashboard refers to a collection of graphs, tables, and visualizations representing your system’s status. For instance, you might have a dashboard displaying global traffic distribution, service-specific traffic, latency, and other metrics. Each service could have its dashboard displaying latency, errors, request rate, CPU and memory usage, instance count, and more. Further drilling down, you can create dashboards for specific subsystems or services, even tailored dashboards like one for Java application garbage collection.
Grafana stands as a widely-used tool for constructing such dashboards, not only for monitoring systems like Prometheus but also other non-monitoring tools like Graphite, InfluxDB, Jaeger, Elasticsearch, and PostgreSQL. Grafana is highly recommended for designing dashboards when working with Prometheus, continually enhancing its Prometheus support for an improved experience.
Data Source
Grafana establishes a connection with Prometheus through data sources, which it uses to gather information for creating graphs. The tool offers support for various data source types, including InfluxDB, PostgreSQL, and, notably, Prometheus. You can have multiple data sources of the same type, often maintaining one per active Prometheus instance. Dashboards in Grafana can feature graphs from various data sources, even mixing sources within a time series panel.
In recent versions of Grafana, adding your first data source is straightforward. By selecting “Add your first data source,” provide a Name (e.g., Prometheus), choose Type as Prometheus, and input the URL (e.g., [http://localhost:9090](http://localhost:9090)). Keep the other settings at default values and click “Save & Test” at the bottom of the form. Be sure to check the buttons, as they might require scrolling on smaller screens. Successful configuration will result in a message confirming the data source’s functionality.
Dashboards and Panels
To begin, navigate to [http://localhost:3000/](http://localhost:3000/) in your browser and select “Create your first dashboard.” Then, click “Add a new panel” to choose the initial panel you wish to include. Panels are rectangular sections that house graphs, tables, or other visual content. You can keep adding more panels by using the “Add panel” button, recognizable by the orange plus sign on the top row. Panels adhere to a grid system and can be rearranged easily through drag-and-drop actions.
For further customization, access the dashboard’s settings through the gear icon at the top. Within the settings menu, options like changing the dashboard’s name are available. Moreover, you can duplicate dashboards using “Save As,” a useful feature when you’re experimenting with a dashboard’s layout or content.
Avoiding the Wall of Graphs
It’s common to end up with multiple dashboards for each service you manage. However, it’s important to avoid creating overly complex dashboards with numerous graphs, which can become difficult to interpret. To address this, it’s recommended to design dashboards that are specific to individual teams or purposes, rather than serving multiple functions.
The complexity of a dashboard should align with its high-level or in-depth purpose. A global overview dashboard should be concise and easily comprehensible at a glance. Dashboards used during on-call duties might have a few extra rows, while expert-focused performance tuning dashboards could span multiple screens.
Limiting the number of graphs on a dashboard is essential due to cognitive load. With each additional element, understanding the dashboard becomes more challenging, particularly in stressful situations like responding to alerts. When under pressure, trying to remember the nuances of each graph’s meaning can hinder your response time and appropriate actions.
For instance, a service dashboard with over 600 graphs, creates confusion and extended load times. Such dashboarding practices are referred to as the “Wall of Graphs” antipattern.
It’s crucial to understand that having numerous graphs doesn’t equate to effective monitoring. Ultimately, the value of monitoring lies in outcomes like quicker incident resolution and informed engineering decisions, not just visually appealing graphs.
Node Exporter – Enabling Insight into System Health
At the core of Prometheus, the Node Exporter is an emissary of system metrics, a bridge connecting Prometheus to the intricate details of the host machine’s operation. It is a utility that grants Prometheus the ability to peer into the vital signs of the operating system, relaying metrics such as CPU utilization, memory allocation, disk space utilization, network activity, and even the motherboard’s temperature. With its primary focus on Unix systems, it empowers administrators to gain insights into the crucial aspects of their infrastructure’s behavior.
For those familiar with the ‘df’ command, the analogy between the filesystem collector of the Node Exporter and ‘df’ is apt. This collector, unsurprisingly, collects metrics that echo the information gleaned from ‘df,’ revealing the status of mounted filesystems. However, what sets the Node Exporter apart is its capacity for customization. Administrators can selectively choose which filesystems to include or exclude, using flags like `collector.filesystem.mount-points-exclude` and `collector.filesystem.fs-types-exclude`. This allows for a tailored approach to monitoring, focusing on the filesystems of utmost significance.
The utility of the Node Exporter extends beyond merely collecting data; it embodies the Prometheus philosophy of simplicity and focus. Rather than aiming to monitor every facet of a machine, it concentrates on capturing metrics at the system level. This differs from traditional monitoring solutions that often employ a singular ‘uberagent’ tasked with tracking everything on the host. In the Prometheus ecosystem, individual services and processes expose their metrics through exporters, enabling a more dynamic and scalable approach to monitoring. This also avoids the pitfalls of a centralized agent potentially becoming an operational bottleneck.
The Node Exporter’s beauty lies in its adaptability and versatility. While Prometheus emphasizes direct instrumentation for metrics, the Node Exporter embraces a different terrain, dealing with metrics from sources not necessarily designed with Prometheus in mind. It navigates the complexity of operating systems, collecting an array of metrics that can range from well-documented CPU usage to more elusive memory consumption metrics that can vary between kernel versions. Its role as an intermediary, translating intricate kernel-level information into Prometheus-readable data, underscores its significance in the monitoring landscape.
Furthermore, the Node Exporter champions security and resource efficiency. Designed to be executed as a non-root user, it aligns with best practices and avoids potential security vulnerabilities. Moreover, its customizable metrics categories ensure that administrators can tailor their monitoring strategy to fit their specific needs, without overwhelming Prometheus with unnecessary data.
In conclusion, the Node Exporter emerges as an indispensable sentinel, guarding the doors to critical system metrics and insights. It stands as Prometheus’s emissary to the underlying operating system, transcending the barriers of machine intricacies to offer administrators a window into their infrastructure’s health and performance. In the intricate symphony of monitoring and maintenance, the Node Exporter’s distinct melody resounds, empowering administrators with a harmony of system awareness and informed decision-making.
PromQL: The Language of Metric Exploration and Analysis
At the heart of Prometheus lies PromQL, a querying language that acts as a gateway to unlocking the rich tapestry of insights hidden within the collected metrics.
PromQL, short for Prometheus Query Language, is more than just a querying tool; it is the lens through which administrators peer into the inner workings of their systems. It transcends the mere collection of metrics, elevating them to the realm of actionable insights. PromQL equips users with the power to dissect and analyze time-series data, providing answers to the most pressing questions about system health, performance bottlenecks, and anomalies.
The language’s syntax may appear deceptively simple, but its capabilities are nothing short of remarkable. With PromQL, users can formulate intricate queries that traverse time-series data, filtering, aggregating, and manipulating it to derive meaningful conclusions. From basic tasks such as retrieving the average CPU utilization over a specified time period to advanced operations like quantifying the rate of change of requests per second, PromQL is a versatile instrument for data exploration.
Central to PromQL’s prowess is its ability to manage the time dimension with finesse. In the context of monitoring, time-series data is not static; it evolves, capturing the dynamics of system behavior. PromQL understands this, enabling users to select and analyze time ranges, calculate rates of change, and identify trends. Its range vector selector allows for pinpointing specific time intervals, facilitating focused analysis of transient events.
Moreover, PromQL’s filtering capabilities empower users to dissect data based on labels, an essential feature in a multi-dimensional data ecosystem. The language’s operators enable logical combinations and comparisons, making it possible to extract insights from specific subsets of metrics. This versatility is crucial in situations where pinpointing the source of an issue requires precision.
While PromQL is a gateway to insight, it also upholds the Prometheus philosophy of simplicity. Its concise syntax is designed to be accessible, allowing both novice and experienced users to harness its power. Queries can be constructed without excessive complexity, encouraging quick ad hoc analysis as well as systematic, automated alerting.
The role of PromQL extends beyond ad hoc queries; it forms the bedrock of alerting rules and recording rules in Prometheus. These rules, formulated in PromQL, provide the foundation for timely notifications and long-term data transformation. PromQL’s ubiquity within the Prometheus ecosystem underlines its centrality in facilitating informed decisions and proactive responses.
In a world where data governs decision-making, PromQL emerges as Prometheus’s virtuoso, orchestrating a symphony of metrics into actionable insights. Its capability to dissect time-series data, its prowess in handling labels, and its elegant yet powerful syntax empower administrators to unearth hidden patterns, trends, and anomalies. PromQL stands as the bridge between raw metrics and informed decision-making, a testament to Prometheus’s commitment to equipping operations with the tools to navigate the complexities of modern IT landscapes.
Alertmanager in Prometheus: Orchestrating Proactive Operations
As an integral component of Prometheus’s monitoring ecosystem, Alertmanager transforms raw metrics into proactive actions, safeguarding the stability and reliability of systems in a dynamic environment.
At its core, Alertmanager is the guardian of alerts, translating raw data into actionable insights. It receives alerts from Prometheus servers and orchestrates their dissemination through a multitude of notification channels. Whether it’s an email, a Slack message, a PagerDuty notification, or a custom webhook, Alertmanager ensures that relevant parties are promptly informed about critical events, enabling swift responses.
Alertmanager’s role extends far beyond mere notification delivery. It acts as an intelligent filter, consolidating and aggregating related alerts. This aggregation minimizes the risk of alert fatigue, ensuring that operators receive meaningful and concise notifications rather than drowning in a deluge of individual alerts. This strategic approach enhances the effectiveness of incident response, allowing for a more focused and informed assessment of the situation.
The power of Alertmanager lies in its ability to tailor notifications to fit the unique needs of different teams and use cases. Teams often have distinct preferences for communication channels, escalation paths, and even timing for alerts. Alertmanager caters to these nuances, offering customization options that cater to diverse operational workflows. Its flexibility empowers administrators to define routing rules that ensure alerts reach the right people at the right time, optimizing incident response.
One of the defining features of Alertmanager is its capability to handle silencing and inhibition of alerts. In the fast-paced world of operations, silencing an alert temporarily or inhibiting subsequent alerts can be crucial during maintenance windows or known outages. Alertmanager seamlessly integrates this functionality, allowing operators to maintain a harmonious balance between incident response and scheduled activities.
For effective incident management, Alertmanager embraces collaboration and integration. It integrates seamlessly with external services such as PagerDuty and ticketing systems, ensuring that incident response workflows are streamlined and well-coordinated. This integration bridges the gap between alerting and resolution, facilitating a cohesive approach to incident management.
Furthermore, Alertmanager is not a monolithic entity; it collaborates harmoniously with Prometheus’s broader ecosystem. It complements Prometheus’s alerting and recording rules, acting as the final layer of defense in the incident response pipeline. By interfacing with Prometheus’s querying language, PromQL, Alertmanager ensures that only the most pertinent and validated alerts are propagated to the appropriate channels.
In the ever-evolving landscape of IT operations, Alertmanager stands as Prometheus’s sentinel, transforming raw metrics into actionable intelligence. Its capacity to filter, aggregate, and route alerts, coupled with its adaptability and integrative capabilities, places it at the forefront of proactive incident response. Alertmanager’s orchestration of notifications transcends simple alerting; it’s a symphony of action that empowers organizations to navigate complexities with confidence, safeguarding the integrity of their systems and services.
Copyright © Intrastellar. All Rights Reserved.
Company No: 14024326
1 Greatorex Street, London, E1 5NS
+447927272123