Skip to content

Prometheus Blackbox Exporter: Guide

  • 18 min read

Ever been in a situation where you need to keep tabs on a service you don’t control? Perhaps it’s an API you use, or a website you rely on. You can’t just install an agent on those systems. So how do you get the metrics you need? That’s where the Prometheus Blackbox Exporter comes in. It’s a tool that lets you monitor external endpoints, and it’s a key piece of many DevOps setups. This guide will show you how to use it effectively.

Understanding the Prometheus Blackbox Exporter

The Prometheus Blackbox Exporter is not your average metrics exporter. Instead of pulling metrics from a system, it actively probes endpoints. It then turns the results of these probes into metrics that Prometheus can collect. Think of it as a watchman, checking if things are up and running from the outside.

It works by sending requests to target services, just like a user would, and measuring the response. This could be a simple HTTP request to see if a website is up, a DNS query to check name resolution, or even a TCP connection to ensure a server is reachable. The Blackbox Exporter then takes these measurements and makes them available in a format that Prometheus understands.

This approach makes it perfect for monitoring services that you don’t have direct access to or when you want to test from the user’s perspective. You can see what the experience is like for someone outside your network, giving you a more complete picture of your service’s availability and performance.

Why Use the Blackbox Exporter?

Why not just monitor from inside? Good question. Internal monitoring is vital, but it doesn’t catch all the issues. External issues, like network problems or third-party service outages, can often be missed if your only view is from within your own infrastructure. Here are some benefits of using the Blackbox Exporter:

  • External Viewpoint: It lets you see how your services appear to the outside world. This helps catch problems that may be invisible from inside your network.
  • No Agent Needed: You don’t need to install an agent on the systems you are monitoring. This is helpful when you are working with services you don’t own.
  • Versatile Probing: It supports different types of checks such as HTTP, HTTPS, DNS, TCP, and ICMP. This makes it useful for monitoring all kinds of services.
  • Realistic Monitoring: The probes mimic the actual user requests, giving a more realistic view of the user experience.
  • Easy Integration: It works very well with Prometheus. So, if you’re already using it for monitoring, this exporter fits right in.

How the Blackbox Exporter Works

The Blackbox Exporter works in a fairly simple manner. Here’s a step by step guide:

  1. Configuration: You set up the exporter with the list of targets it should probe and how it should probe them. This is all done in a config file.
  2. Probing: When Prometheus scrapes the exporter, the exporter starts to send probe requests to the target. For instance, if you set up an HTTP check, the exporter will send an HTTP GET request to that target.
  3. Measurement: The exporter records the results of the probe. It will record things like time it took to connect, the status code of the HTTP response, and so forth.
  4. Metrics Export: The exporter converts these results into metrics. Then makes these metrics available for Prometheus to scrape.
  5. Scraping: Prometheus scrapes these metrics from the exporter at regular intervals. This data is then used to create graphs, alerts, and dashboards.

Setting Up the Blackbox Exporter

Setting up the Blackbox Exporter involves a few steps. Let’s walk through them:

  1. Download the Exporter: First you need to download the Blackbox Exporter binary from the Prometheus download page. Choose the correct version for your system.
  2. Extract the Binary: Once you download the binary, you’ll need to extract it. You can often do this with a simple tar -xvzf blackbox_exporter-<version>.tar.gz.
  3. Create Configuration File: The Blackbox Exporter needs a config file to know what to monitor. You’ll need to create a blackbox.yml file. It should include your setup.
  4. Run the Exporter: Now, you run the binary. You use the command, blackbox_exporter --config.file=blackbox.yml.
  5. Configure Prometheus: The last step is setting up Prometheus. It needs to know where to find and collect the exporter’s metrics.

Let’s examine each of these steps further.

Downloading the Exporter

To get the Blackbox Exporter, head over to the official Prometheus download page. Look for the “blackbox exporter” section and download the version that fits your system. For example, if you are using a Linux 64-bit system, you’d get blackbox_exporter-<version>.linux-amd64.tar.gz. It’s good to grab the latest stable version. After you download the archive, you can extract it with this command in your terminal:

tar -xvzf blackbox_exporter-<version>.linux-amd64.tar.gz

This gives you an executable called blackbox_exporter that you can run.

Creating the Configuration File

The heart of the Blackbox Exporter is its configuration file. This file tells the exporter what to probe, how to probe, and what to measure. Here’s a basic example of a configuration file, named blackbox.yml:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      method: GET
  dns_lookup:
    prober: dns
    timeout: 5s
    dns:
      query_type: A
  tcp_connect:
    prober: tcp
    timeout: 5s

Let’s break down this config.

  • modules: This section defines the different probing modules you can use. You can define many different modules.
  • http_2xx: This module probes using the HTTP protocol. It will only consider the target as up when a 200 status code is returned.
  • prober: http: This tells the exporter to use the HTTP prober.
  • timeout: 5s: This is the length of time the exporter waits before considering the probe as a failure. It will time out after 5 seconds.
  • http: This section is specific to the HTTP prober.
  • valid_status_codes: [200]: This lists the status codes considered to be a successful request. Here, we are only looking for 200.
  • method: GET: The request will use the HTTP GET method.
  • dns_lookup: This is a DNS probing module.
  • prober: dns: This tells the exporter to use the DNS prober.
  • dns: query_type: A: This tells the exporter to perform an A record query.
  • tcp_connect: This is a TCP probing module.
  • prober: tcp: This tells the exporter to use the TCP prober.

You can create more complex modules to meet your specific monitoring needs.

Running the Exporter

After you’ve created the config file, you can run the exporter. Use this command in your terminal:

./blackbox_exporter --config.file=blackbox.yml

This starts the Blackbox Exporter and loads the config. It also makes the metrics available at http://localhost:9115/metrics. You can change the port by using the --web.listen-address flag.

Configuring Prometheus

Now it’s time to set up Prometheus so it can start scraping metrics from the exporter. You need to add a new job to your prometheus.yml config file:

scrape_configs:
  - job_name: 'blackbox_exporter'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - localhost:9115
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target
      - source_labels: [__address__]
        target_label: instance
      - target_label: module
        replacement: http_2xx

Let’s break this down:

  • job_name: This gives a name to your Prometheus job, “blackbox_exporter”.
  • scrape_interval: How often Prometheus should scrape the exporter. Here, it’s set to every 15 seconds.
  • metrics_path: This shows the path on the exporter where the metrics can be found (/metrics).
  • static_configs: This is the list of the exporter’s addresses where Prometheus will get the metrics.
  • relabel_configs: These rules let you change the labels. They let Prometheus pass the target to the exporter.

With this config, Prometheus will look for metrics from the Blackbox Exporter, but it still needs to know what to probe and how. This is handled by relabel_configs.

Probing Targets

The relabel_configs section of your Prometheus config does a couple of very important things: It tells the exporter what to probe and which module to use.

How the Relabeling Works

  1. The source_labels: [__address__] takes the address that Prometheus is scraping, such as localhost:9115.
  2. The target_label: __param_target makes a copy of the address and names it __param_target. This new label now holds the exporter’s address.
  3. The source_labels: [__param_target] and target_label: target take that new address and assign it to a label named target.
  4. The source_labels: [__address__] and target_label: instance take the exporter’s address and set the value for label named instance.
  5. The target_label: module sets the module for the exporter, which is set to http_2xx in this case.

Now, you can add more targets to your Prometheus config. Then the exporter will probe these. You do this with the following:

scrape_configs:
  - job_name: 'blackbox_exporter'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - localhost:9115
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target
      - source_labels: [__address__]
        target_label: instance
      - target_label: module
        replacement: http_2xx
  - job_name: 'http_probes'
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://google.com
          - https://example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target

Let’s examine the additions:
* job_name: 'http_probes': This is a new job that is used to set up targets.
* metrics_path: /probe: This is the path that tells the exporter to probe, instead of getting its own metrics.
* params: module: [http_2xx]: This tells the exporter what modules to use to probe.
* static_configs: targets: These are the targets to probe. In this case, it’s https://google.com and https://example.com.
* relabel_configs: These are the rules that sets up the new labels as with the blackbox_exporter job.

With this, Prometheus will now probe the specified URLs using the http_2xx module. The target label will have the URL of the site being probed. This lets you know what site had a failure. And the instance label will have the Blackbox Exporter’s address.

You can also use other methods of setting targets. For instance, you can use DNS service discovery:

scrape_configs:
  - job_name: 'blackbox_exporter'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - localhost:9115
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target
      - source_labels: [__address__]
        target_label: instance
      - target_label: module
        replacement: http_2xx
  - job_name: 'http_probes'
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [http_2xx]
    dns_sd_configs:
      - names:
          - my-service.service.consul
        type: A
        port: 80
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target

In this config, Prometheus will query my-service.service.consul and use all the IPs found with an A record query and probes them.

Blackbox Exporter Metrics

The Blackbox Exporter exports a variety of metrics. Here are some of the main ones, with explanations:

  • probe_duration_seconds: The total duration of a probe, in seconds. This is useful to see how long the checks are taking.
  • probe_dns_lookup_time_seconds: The time taken for the DNS lookup, in seconds. This can help identify DNS resolution problems.
  • probe_http_status_code: The HTTP status code returned by the probe. For instance, a 200 for “OK,” a 404 for “Not Found,” and so on.
  • probe_success: Binary metric indicating if the probe was successful (1 for success, 0 for failure).
  • probe_tcp_duration_seconds: The time taken for a TCP connection, in seconds. This is useful for checking network reachability.
  • probe_ssl_earliest_cert_expiry: The expiration date of the earliest SSL certificate, in seconds.

These metrics allow you to create helpful alerts. They let you know if a service is down, or if there are network problems, or even if your SSL certificates are close to expiring.

Practical Use Cases

The Blackbox Exporter is useful in many scenarios. Here are a few practical use cases:

  • Website Monitoring: You can check if your website is online, and if the HTTPS certificate is still valid.
  • API Monitoring: You can check that your API endpoints are working and returning valid responses.
  • DNS Monitoring: You can ensure your domain resolves correctly and that DNS servers are functioning well.
  • Network Monitoring: You can check if you can connect to important services over TCP/IP.
  • Third Party Service Monitoring: You can check if the third-party services you rely on are available.

Let’s look at a few of these use cases in more detail.

Website Monitoring

With the HTTP module you can probe web pages with a GET request. Here is how you set it up in your blackbox.yml:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      method: GET

Then use the following in your prometheus.yml:

scrape_configs:
  - job_name: 'blackbox_exporter'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - localhost:9115
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target
      - source_labels: [__address__]
        target_label: instance
      - target_label: module
        replacement: http_2xx
  - job_name: 'http_probes'
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://mywebsite.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target

Now, Prometheus will start probing your website. If your site is down or the response code is not 200, the probe_success metric will be 0. You can create an alert based on that metric. For instance, use the following PromQL query:

probe_success{job="http_probes"} == 0

This query will trigger when the probe_success metric is zero. You can set up an alert to notify your on-call team when a site goes down.

API Monitoring

To monitor APIs, you can use a similar HTTP config. You’ll likely need to add extra headers to your probes if your API requires specific headers. Here is a config example with an authorization header:

modules:
  api_check:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: [200, 201, 204]
      method: GET
      headers:
        Authorization: "Bearer YOUR_API_TOKEN"

Then use this in your prometheus.yml:

scrape_configs:
  - job_name: 'blackbox_exporter'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - localhost:9115
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target
      - source_labels: [__address__]
        target_label: instance
      - target_label: module
        replacement: http_2xx
  - job_name: 'api_probes'
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [api_check]
    static_configs:
      - targets:
          - https://myapi.com/status
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target

This will tell the exporter to include an Authorization header with each request. The header needs a Bearer token, in this case, named YOUR_API_TOKEN.

DNS Monitoring

The Blackbox Exporter also supports DNS probes. This lets you check if your domain names can be resolved correctly. You can probe for various records, like A, AAAA, or CNAME. Here’s an example config:

modules:
  dns_lookup:
    prober: dns
    timeout: 5s
    dns:
      query_type: A

Use the following config for prometheus.yml:

scrape_configs:
  - job_name: 'blackbox_exporter'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - localhost:9115
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target
      - source_labels: [__address__]
        target_label: instance
      - target_label: module
        replacement: http_2xx
  - job_name: 'dns_probes'
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [dns_lookup]
    static_configs:
      - targets:
          - mydomain.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target

Prometheus will now try to resolve mydomain.com. If the lookup fails, the probe_success metric will be 0. You can then set up an alert to notify you of the problem.

TCP Monitoring

You can also use the exporter to check TCP connections. This is useful for making sure your servers or services are reachable over a network:

modules:
  tcp_connect:
    prober: tcp
    timeout: 5s

Set up Prometheus with the following:

scrape_configs:
  - job_name: 'blackbox_exporter'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - localhost:9115
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target
      - source_labels: [__address__]
        target_label: instance
      - target_label: module
        replacement: http_2xx
  - job_name: 'tcp_probes'
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
          - my-server.com:8080
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target

Prometheus will now try to connect to port 8080 on my-server.com. If the connection fails, probe_success will be 0. You can set up an alert if the probe fails.

Advanced Configuration

The Blackbox Exporter has features that go beyond the basics. Here are some advanced configurations:

  • TLS/SSL: You can check if the SSL certificates are valid and are not expired.
  • Custom Headers: You can include custom headers in your HTTP requests. This is useful for API checks that require authentication.
  • POST Requests: You can probe with POST requests for APIs that need a payload.
  • Different Status Codes: You can specify what status codes you consider to be successful for each probe.
  • Multiple Modules: You can define different modules to check the same target in different ways.
  • Authentication: You can set up basic authentication for probes to access secured resources.

Checking SSL Certificates

The Blackbox Exporter can check the SSL certificates of your websites. You can configure the exporter to send a GET request to the site and record the certificate’s expiration date:

modules:
  https_cert_check:
    prober: http
    timeout: 10s
    http:
      method: GET
      tls_config:
        insecure_skip_verify: false

Then, you can add this in your prometheus.yml:

scrape_configs:
  - job_name: 'blackbox_exporter'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - localhost:9115
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target
      - source_labels: [__address__]
        target_label: instance
      - target_label: module
        replacement: http_2xx
  - job_name: 'https_probes'
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [https_cert_check]
    static_configs:
      - targets:
          - https://mysite.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target

With this, the probe_ssl_earliest_cert_expiry metric will show you when the certificate expires, in Unix timestamp. This way, you can create alerts based on how close a certificate is to expiring.

Using POST requests

The Blackbox Exporter can also send HTTP POST requests, which is very helpful for checking API endpoints that need you to send data:

modules:
  api_post_check:
    prober: http
    timeout: 10s
    http:
      method: POST
      valid_status_codes: [200, 201, 204]
      body: '{"status": "up"}'
      headers:
        Content-Type: application/json

Here is how to set it up in Prometheus:

scrape_configs:
  - job_name: 'blackbox_exporter'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - localhost:9115
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target
      - source_labels: [__address__]
        target_label: instance
      - target_label: module
        replacement: http_2xx
  - job_name: 'api_post_probes'
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [api_post_check]
    static_configs:
      - targets:
          - https://myapi.com/check
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target

This will make the Blackbox Exporter send a POST request with a {"status": "up"} body.

Using Basic Authentication

If your target needs basic authentication, you can configure the Blackbox Exporter to include it:

modules:
  auth_http_check:
    prober: http
    timeout: 10s
    http:
      valid_status_codes: [200, 201, 204]
      method: GET
      basic_auth:
        username: myuser
        password: mypassword

Then, set up Prometheus to use the module:

scrape_configs:
  - job_name: 'blackbox_exporter'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - localhost:9115
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target
      - source_labels: [__address__]
        target_label: instance
      - target_label: module
        replacement: http_2xx
  - job_name: 'auth_http_probes'
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [auth_http_check]
    static_configs:
      - targets:
          - https://mysecuredsite.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: target

This configuration will include the user and the password when the Blackbox Exporter sends the requests.

Alerting with Prometheus

You can set up alerts based on the Blackbox Exporter’s metrics. This lets you know of problems before they impact your users. Here are a few example rules:

groups:
  - name: blackbox_alerts
    rules:
      - alert: HTTPProbeFailed
        expr: probe_success{job="http_probes"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "HTTP probe failed for {{ $labels.target }}"
          description: "The HTTP probe to {{ $labels.target }} failed for 5 minutes. Please investigate."
      - alert: DNSProbeFailed
        expr: probe_success{job="dns_probes"} == 0
        for: 5m
        labels:
            severity: critical
        annotations:
          summary: "DNS probe failed for {{ $labels.target }}"
          description: "The DNS probe to {{ $labels.target }} failed for 5 minutes. Please investigate."
      - alert: TCPProbeFailed
        expr: probe_success{job="tcp_probes"} == 0
        for: 5m
        labels:
            severity: critical
        annotations:
          summary: "TCP probe failed for {{ $labels.target }}"
          description: "The TCP probe to {{ $labels.target }} failed for 5 minutes. Please investigate."
      - alert: SSLCertificateExpiring
        expr: probe_ssl_earliest_cert_expiry{job="https_probes"} - time() < 86400 * 7
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "SSL certificate for {{ $labels.target }} is expiring soon"
          description: "The SSL certificate for {{ $labels.target }} will expire in less than 7 days."

Here’s what these rules do:

  • HTTPProbeFailed: This alerts you if an HTTP probe fails for more than 5 minutes.
  • DNSProbeFailed: This alerts you if a DNS probe fails for more than 5 minutes.
  • TCPProbeFailed: This alerts you if a TCP probe fails for more than 5 minutes.
  • SSLCertificateExpiring: This alerts you if an SSL certificate is expiring in less than 7 days.

These rules can help you catch problems early. You can tailor them to fit your specific needs.

Best Practices

Here are some best practices to keep in mind while using the Blackbox Exporter:

  • Keep it Simple: Start with simple configurations and add complexity only as needed. This makes it easy to understand the setup.
  • Use Descriptive Names: Use clear names for your modules and jobs in Prometheus. This helps to keep things organized.
  • Monitor the Exporter: Make sure to monitor the exporter itself. It’s good to know when it has problems so you don’t miss other issues.
  • Keep the Config Simple: Try to keep your configuration files concise and easy to read. This will help maintain them.
  • Test Configurations: Always test new configurations in a non-production environment first. This helps to prevent problems in production.
  • Avoid Overloading: Be careful not to overload the exporter by probing too many targets at the same time. You should avoid probing too many targets on a single exporter.

Wrapping it Up

The Prometheus Blackbox Exporter is a great tool for monitoring external endpoints. It helps catch problems before they get out of hand. Whether it’s a website, an API, or a network connection, the Blackbox Exporter helps you keep tabs on them. It’s a vital tool in the DevOps world, and by taking the time to learn how to use it well, you make your monitoring game much better. So, why not give it a try and see how it can help?

Related Posts

Grafana Tempo: Distributed Tracing

Splunk Search: Mastering It