Prometheus: Long-Term Storage

When Prometheus collects metrics, it stores them in its local time-series database, but this storage isn’t meant for long-term retention. If you’ve ever found yourself needing to access older Prometheus metrics for analysis or compliance, you know how crucial having a long-term storage solution is. Prometheus is a powerful tool, but for robust, long-lasting metric storage, you need to think beyond its default setup.

Why You Need Prometheus Long-Term Storage

Prometheus is great for real-time monitoring and alerting, but its local storage isn’t designed to keep data around forever. The default storage is local and not meant for backups or long-term analysis. It was made to be quick and to work with real-time data. As you scale up, or simply need historical data, you’ll soon discover that the data gets deleted or aggregated over time. This limits what you can do with it, like spotting trends over months or years. The following are common cases where long-term storage becomes a must:

• Trend Analysis: To understand how your systems behave over time, you need a way to access metrics from past weeks, months or even years. Knowing the normal behavior of your system over time can help you spot patterns and identify when something is going off-track.
• Capacity Planning: If you’re scaling up infrastructure, past data is crucial to see how resources were used, so you know how much more or less you need. With that insight, you can also see how you can make the best use of resources you have, or where you may need more of them, and when to have them.
• Compliance and Auditing: Some industries have rules that ask you to keep metric data for a long time. You’ll need to make sure you’re meeting such requirements. This way, if you need data for an audit, you can be sure that you have it.
• Root Cause Analysis: When issues come up, having more historical data can be useful to see what led up to that issue. It gives you the full picture and what happened before the problem. Having that data at hand is very useful to figure out what happened and how to fix it and to prevent that same problem in the future.

Understanding Prometheus Storage Limitations

Before diving into long-term storage options, let’s discuss the limitations of Prometheus’s default storage:

• Local and Ephemeral: Prometheus stores data locally on the server it runs on. This means if the server fails, you lose your data. It does not have backups, and has only one copy of it, locally.
• Limited Retention: By default, Prometheus keeps data only for a set time, which is normally just a few weeks. This is okay for real-time alerts and short-term issues, but it is not enough if you need to access metrics from months ago. After a specific time, old metrics get removed to free up space.
• No Clustering or Replication: Prometheus doesn’t natively support clustering or replication, which is crucial for high availability and redundancy. You can only run it in one place, and you can’t have a replica of it for backup.
• Scalability Challenges: As your metric volumes grow, Prometheus can struggle to manage it all on a single server. It might get slower, and you might get gaps in your data. The bigger your data set, the harder it is for Prometheus to work.
• Limited Query Capabilities: The query language, PromQL, is robust but not always the best for complex historical analysis. The query language has limits on how it can get and combine data over long time frames.

These limits mean that if you’re serious about metrics, you need a long-term solution. It gives you the reliability, backup, and the ability to store and get what you need from the data.

Exploring Long-Term Storage Options

Now that you know the limits of default Prometheus storage, let’s look at some popular long-term storage options:

Thanos

Thanos is a popular open-source project created by the same team that created Prometheus. It turns Prometheus into a system that can be scaled and have a long-term storage. Thanos works by taking Prometheus data and moving it to object storage (like AWS S3 or Google Cloud Storage). Here are its main features:

• Global Query View: Thanos lets you query data from all your Prometheus instances in one place. You don’t need to look at one instance to then look at another. You have a single query view of the data across all of your instances.
• Object Storage Integration: It uses object storage for long-term data storage, making it scalable and cost-effective. Object storage is good because it’s cheap and can hold a lot of data.
• Downsampling: To save on storage and improve performance, Thanos automatically reduces the resolution of older data. It keeps the important details but uses less space. This speeds up queries over long periods.
• High Availability: With Thanos, you can make your Prometheus setup highly available, with redundancy and failover. If a Prometheus instance fails, your data remains accessible.
• Sidecar Pattern: It uses a sidecar, a small process running along with Prometheus, to move data to object storage. This is simple to set up and doesn’t change Prometheus in any major way.

Thanos is a great choice if you like how Prometheus works and want to keep that but make it scale for the long run.

Cortex

Cortex is another open-source project, which also allows to create a scalable and robust long-term storage solution for Prometheus metrics. Like Thanos, Cortex is designed to work well with Prometheus. Here’s what it brings to the table:

• Scalable and Distributed: Cortex is built to handle large amounts of metric data by running as a distributed cluster, so you can scale your solution out as much as you need.
• Multi-Tenancy: It allows you to store metrics from many users or teams in a single cluster, while making sure each user’s data is separate. This is useful for large setups where many teams need to keep track of their metrics.
• High Availability: Cortex is designed to be resilient. Data is replicated and available even if parts of the system go down.
• PromQL Support: You can use the same query language (PromQL) as Prometheus, which is useful for anyone already familiar with Prometheus.
• Centralized Query Interface: Like Thanos, Cortex gives you a single place to query data across all your Prometheus instances.
• Write-Ahead Logging: To protect against data loss, Cortex uses a write-ahead log. It makes sure data gets saved before it gets used.
• Long-term Storage Backend: Cortex can use different storage backends, such as AWS DynamoDB, Google BigTable, or Cassandra.

Cortex is a solid pick if you have a complex environment and need multi-tenancy and scalability from day one.

M3DB

M3DB, a time-series database, is a project from Uber and designed for large-scale metric storage. Here’s what makes M3DB stand out:

• High Performance: It is designed for speed and can handle a lot of writes and queries, making it good for large metric workloads.
• Time-Series Optimized: M3DB was designed specifically for time-series data, so it works well for metrics from Prometheus.
• Scalable and Fault-Tolerant: It can scale out as much as you need, and handles hardware failures well. If a server goes down, the data is still available.
• Downsampling and Aggregation: Like Thanos, M3DB can downsample and combine data to make queries faster.
• Integrations: It works well with Prometheus and other systems. It has native support to be used with Prometheus.
• Customizable Storage: M3DB lets you choose where and how to store your data, which gives you extra control.

M3DB is a solid choice if performance and scalability are your top concerns, but it is also complex to set up and work with.

TimescaleDB

TimescaleDB is a time-series database built on top of PostgreSQL. It is good for those already using Postgres and want to keep their data there. Here are its key features:

• SQL Compatibility: Because it’s built on PostgreSQL, you can use standard SQL to get the data, making it easier for those already comfortable with SQL.
• Time-Series Optimizations: It has special features made just for time-series data, like fast data inserts and queries.
• Scalability: TimescaleDB can scale out and handle a good amount of data and queries.
• Easy to Use: For people who already know PostgreSQL, TimescaleDB is not too hard to start using.
• Mature Ecosystem: It is based on Postgres, so it has a strong community and a lot of tools.
• Long-Term Support: It can store data for a very long time and keep it safe.
• Flexibility: Can also be used as a general-purpose database, besides using it for metrics.

TimescaleDB is a solid option if you want to use a familiar SQL interface and don’t want to use a new storage engine, but it does not come with direct support for Prometheus out of the box.

InfluxDB

InfluxDB is a time-series database that focuses on handling large amounts of data and performing analytics. It has an integration with Prometheus, making it useful for storing Prometheus metrics. Here’s what you get with it:

• Time-Series Focus: It is built for time-series data and handles writes and queries well, so it works well with metrics.
• Prometheus Integration: It works with Prometheus metrics, making it easy to pull data from Prometheus instances.
• Scalability and Performance: It’s built to handle large datasets and gives good query performance.
• Downsampling: Like other options, it can automatically downsample data to save space and speed up queries.
• Query Language: It uses its own query language called Flux, which is built for time-series queries, but you can still use PromQL.
• Alerting Features: It has built-in features for setting up alerts based on metrics.
• Telegraf: Uses Telegraf to collect metrics. It is an agent that has inputs and outputs for different systems, allowing for flexibility in how and where to send metrics.

InfluxDB is a good choice if you want an easy way to store and analyze Prometheus metrics, but you will also need to understand its query language.

Choosing the Right Option

Choosing the right option for long-term storage relies on your specific requirements, such as:

• Scale: How big is your system? Do you need to store data from many services or just a few?
• Cost: How much are you willing to spend to store data? Object storage is usually cheap, and using a managed database can be more costly.
• Complexity: How much time do you want to spend setting it up? Some options are simpler to manage than others.
• Skills: What tools and technologies are you comfortable with? Do you prefer to use the same PromQL, or do you want to use SQL for complex queries?
• High Availability: How critical is your metric data? Do you need to ensure data access, even during system failures?

Take a close look at what you need, try some of these options out and see which one is the right fit for you. You should try them in a test environment before going to production.

Deep Dive into Thanos: A Practical Guide

Let’s focus on Thanos and see how to set it up. It’s one of the most used long-term storage solutions for Prometheus, and we’ll show you the process:

Thanos Architecture

Thanos follows a distributed architecture. Here’s what you need to know:

• Prometheus: You still use Prometheus to collect metrics, but now it’s not the only place data is stored.
• Thanos Sidecar: A small process that runs along with each Prometheus instance. It uploads metrics to object storage.
• Thanos Store: This part handles data from object storage, so you can query older metrics.
• Thanos Query: A central component that allows you to query data from multiple Prometheus instances and Thanos stores, making all data accessible in one place.
• Object Storage: Where all the historical data is stored (like AWS S3, Google Cloud Storage, Azure Blob Storage, or MinIO).

Step-by-Step Setup Guide

Here’s how to set up Thanos with Prometheus:

1. Set Up Object Storage

First, pick and configure your object storage. For this example, we will use AWS S3, but the steps would be similar with others:

• Create an S3 bucket: Make a new bucket in AWS S3 where Thanos can store your metrics. Be sure to set proper access rules for the bucket so only Thanos can access it.
• Get credentials: You’ll need the access key ID and secret access key of an IAM user that can read and write to that bucket. Keep these safe.

2. Deploy Thanos Sidecar

The sidecar is a small process you run alongside Prometheus to upload data to object storage. Here’s how to set it up:

• Download the sidecar: Get the latest Thanos release from the GitHub releases page.
• Configure the sidecar: Create a config file that tells the sidecar how to connect to Prometheus and object storage. It usually looks like this:
  yaml
type: s3
config:
bucket: <your-s3-bucket-name>
endpoint: <your-s3-endpoint>
access_key: <your-s3-access-key>
secret_key: <your-s3-secret-key>
insecure: false
  Adjust settings like your bucket name, S3 endpoint, access key and secret key.

• Run the sidecar: Start the sidecar using the command line, like this:
  bash
thanos sidecar \
--prometheus.url="http://localhost:9090" \
--objstore.config-file="path/to/your/thanos-sidecar.yaml" \
--http-address=:10902
  Replace http://localhost:9090 with your Prometheus address, and path/to/your/thanos-sidecar.yaml with the location of your config.

• Add Sidecar to your Prometheus Deployment: If you run Prometheus in Kubernetes, or some other containerization technology, you can run the sidecar alongside Prometheus as a container. Here’s how you may add it to your Kubernetes manifest:
  yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
spec:
template:
spec:
containers:
- name: prometheus
image: prom/prometheus:latest
ports:
- containerPort: 9090
- name: thanos-sidecar
image: quay.io/thanos/thanos:latest
args:
- sidecar
- --prometheus.url=http://localhost:9090
- --objstore.config-file=/etc/thanos/thanos-sidecar.yaml
ports:
- containerPort: 10902
volumeMounts:
- name: thanos-sidecar-config
mountPath: /etc/thanos
  In this example, a volume with the thanos-sidecar.yaml config is mounted to the /etc/thanos folder. You need to ensure you also mount your config to the correct folder as seen in the volumeMounts.

3. Deploy Thanos Store

The store component lets you query data from the object storage. Here’s how to set it up:

• Download Thanos Store: Get the binary from the GitHub releases page.
• Configure Thanos Store: Create a config file, like this:
  yaml
type: s3
config:
bucket: <your-s3-bucket-name>
endpoint: <your-s3-endpoint>
access_key: <your-s3-access-key>
secret_key: <your-s3-secret-key>
insecure: false
  Change it to fit your bucket, S3 endpoint and credentials.
• Run the Thanos Store: Start the store component from the command line:
  bash
thanos store \
--objstore.config-file="path/to/your/thanos-store.yaml" \
--http-address=:10901
  Replace path/to/your/thanos-store.yaml with the location of your config. If you run this in Kubernetes, you will need a manifest similar to the Prometheus deployment in the previous step.

4. Deploy Thanos Query

The query component gives you a single view of all Prometheus metrics, even from object storage. Here’s how to set up:

• Download Thanos Query: Get the binary from the GitHub releases page.
• Configure Thanos Query: Create a config file, like this:
  yaml
--query.replica-label="prometheus_replica" \
--store="<thanos-store-address>:10901" \
--store="<prometheus-sidecar-address>:10902"
  Replace <thanos-store-address>:10901 with the address where you run Thanos Store, and <prometheus-sidecar-address>:10902 with your Thanos Sidecar address. If you have more than one, list them separated by a comma.
• Run Thanos Query: Run Thanos Query from the command line, using this command:
  bash
thanos query \
--http-address=:10900 \
--query.replica-label="prometheus_replica" \
--store="<thanos-store-address>:10901" \
--store="<prometheus-sidecar-address>:10902"

5. Query Your Data

Once all parts are running, go to the Thanos Query address (http://localhost:10900 if you are running locally). You can now use PromQL to query the data. You should be able to see data from both your Prometheus instances and the data stored in object storage.

Important Considerations

• Downsampling: To save on storage, Thanos uses downsampling. Older data has less detail, but this also speeds up queries.
• Monitoring Thanos: Use Prometheus to keep an eye on Thanos. It’s a good way to know if everything is working well.
• Security: Always secure your object storage and use TLS for all network communication between components.
• Maintenance: Set up processes to back up data in object storage and keep your Thanos components up to date.

Setting up Thanos may seem a bit much at first, but once done, you’ll have a solid way to store and access your metrics over the long haul.

Cortex Deep Dive: Multi-Tenancy and Scalability

Now let’s look at Cortex. Cortex focuses on multi-tenancy and scalability. Here’s how to get going:

Cortex Architecture

Cortex has a microservices architecture. Here’s how each part fits together:

• Ingester: Receives metric data from Prometheus and writes it to long-term storage.
• Distributor: Takes metrics from Prometheus and sends them to the correct Ingester.
• Querier: Queries data from the storage for the users.
• Store: Interface for different storage backends (DynamoDB, BigTable, Cassandra).
• Ruler: Handles recording and alert rules.

Setting up Cortex

Here’s a brief guide on how to set up Cortex:

1. Select Storage Backend

First, pick your storage backend. For example, AWS DynamoDB:

• Create a DynamoDB table: Set up a DynamoDB table with enough write and read capacity to store the metric data.
• Set up authentication: Make sure your user for the API can access the DynamoDB table.

2. Deploy Cortex Components

Deploy the Cortex components as separate microservices. They can be run on Kubernetes or similar platforms. Here’s a sample configuration for Kubernetes:

• Distributor: Deploys the distributor. The code might look like this:
  yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: cortex-distributor
spec:
template:
spec:
containers:
- name: cortex-distributor
image: grafana/cortex:latest
args:
- -distributor.config.file=/etc/cortex/cortex-config.yaml
ports:
- containerPort: 8080
volumeMounts:
- name: cortex-config
mountPath: /etc/cortex
  You’ll need a volume with the configuration file mounted in /etc/cortex as seen in the example. The content of this config file will be explained next.
• Ingester: Create deployment for the Ingester, the code will look similar to the Distributor, but with different arguments to the container.
• Querier: Create deployment for the Querier, the code will look similar to the Distributor, but with different arguments to the container.
• Config: Create config files for each component with settings for connection to storage, and general settings such as how much memory or CPU to use, log settings and others. Here is a sample:
  yaml
distributor:
ring:
kvstore:
store: inmemory
ingester:
lifecycler:
ring:
kvstore:
store: inmemory
wal:
dir: /tmp/wal
query_range:
align_queries_with_step: true
max_query_parallelism: 16
storage:
backend: dynamodb
dynamodb:
region: <your-dynamodb-region>
aws_access_key_id: <your-aws-access-key>
aws_secret_access_key: <your-aws-secret-key>
dynamodb_table: <your-dynamodb-table>
  Change <your-dynamodb-region>, <your-aws-access-key>, <your-aws-secret-key>, and <your-dynamodb-table> to the information of your configuration.
• Run components: Deploy each component using your platform and make sure they all can access each other. You will need a service to expose the Distributor for Prometheus to send the metrics. You will also need to expose the Querier for the users to query the metrics.

3. Configure Prometheus to Write to Cortex

Tell Prometheus to send data to the Cortex Distributor. Add these settings to your prometheus.yaml configuration:

```yaml
remote_write:
  - url: http://<cortex-distributor-address>:8080/api/v1/push
```
 Replace `<cortex-distributor-address>` with the address to reach the Distributor.

4. Query Your Metrics

Query data using the Cortex Querier API. Cortex will return the data using PromQL. You should be able to query data from all your Prometheus instances in a single query, without the need to go from one instance to the other.

Key Features

• Multi-Tenancy: Cortex allows many teams or users to have their metrics on the same cluster, with a clear separation between each other.
• Scalability: The design of Cortex allows you to add more resources as needed. Each part can be scaled independently to fit the requirements.
• High Availability: If one part of the system fails, others take over, making sure you still have access to your metrics.

Cortex is a good choice if you want a scalable way to manage Prometheus data for several teams or users, while ensuring high availability.

M3DB: A Look into High-Performance Time-Series Storage

Let’s now discuss M3DB for performance and scalability. It is good when you need speed and have large data sets. Here’s what you should know:

M3DB Architecture

M3DB has a distributed architecture that is optimized for time-series data. Here are its components:

• Coordinator: Manages the cluster and distributes data and queries.
• Server Nodes: Store the time-series data across multiple nodes for scalability.
• Client: Integrates with Prometheus to send and query data.

Setting up M3DB

Here’s a quick guide on setting up M3DB:

1. Deploy M3DB

Deploy the M3DB cluster in Kubernetes or your preferred platform. You’ll need two deployments:

• Coordinator: You will need to create a Kubernetes manifest with the coordinator container. It’ll have the settings that allow it to communicate with the server nodes.
• Server Nodes: Create a deployment to run the server nodes. You’ll need several of them to store the data, each connected to the coordinator.

2. Configure M3DB

You’ll need to configure each component using a configuration file. Here’s a sample configuration file:
yaml
cluster:
placement:
num_shards: 1024
replication_factor: 3
This is a basic config file, you’ll need to add the addresses to connect with each other.

3. Integrate Prometheus with M3DB

To use Prometheus with M3DB, you’ll need to use the m3coordinator remote write endpoint. Configure it like this in prometheus.yaml:

```yaml
remote_write:
  - url: http://<m3coordinator-address>:7201/api/v1/prom/remote/write
```
  Replace `<m3coordinator-address>` with the address to access the coordinator API.

4. Query Your Data

Use the M3 query interface to query the metrics. The data is accessible via Prometheus API, so you can use your usual tools to access it.

Key Features

• High Performance: Designed to handle a lot of metrics with low latency.
• Time-Series Optimization: The system is built specifically for time-series data, giving the best performance for this use case.
• Scalability and Fault Tolerance: Can handle large amounts of data, and keeps it safe even when there are hardware issues.

M3DB is a solid choice for organizations that want the best performance for their Prometheus metrics, and can handle a complex set up.

TimescaleDB, InfluxDB: SQL and Time-Series Databases

Now, let’s look at TimescaleDB and InfluxDB, each offering a different way to manage your Prometheus metrics.

TimescaleDB: Time-Series on SQL

TimescaleDB uses a SQL interface to store and manage metrics. Here’s how you can use it:

1. Set Up TimescaleDB

Set up a PostgreSQL database with the TimescaleDB extension. You can use a managed database service or set up your own server:
* Create a database: Set up a database for your metrics.
* Install TimescaleDB: Follow the instructions to install the extension for PostgreSQL.

2. Configure Prometheus to Write to TimescaleDB

To send Prometheus data to TimescaleDB, you will need a remote write adapter. It can be set up with the following parameters:

```yaml
remote_write:
  - url: <your-remote-write-address>
```
  The remote write adapter should send the data to a database table.

3. Query with SQL

Query metrics using standard SQL. This makes it easy for those used to SQL databases, here’s a sample query:

```sql
SELECT time, value FROM metrics WHERE metric_name='cpu_usage' AND time > NOW() - INTERVAL '1 day';
```
This SQL query gets the metric `cpu_usage` over the last day.

InfluxDB: A Time-Series Focus

InfluxDB is designed from the ground up for time-series data. Here’s how you can use it for your Prometheus metrics:

1. Set Up InfluxDB

Set up InfluxDB by downloading the server and run it as a service. Create a database for your metrics, set up authentication and set any other configuration you may need.

2. Integrate with Prometheus

Use Telegraf or the Prometheus remote write endpoint to send data to InfluxDB. Here is the configuration for prometheus.yaml:

```yaml
remote_write:
  - url: http://<influxdb-write-address>:8086/api/v2/write?org=<your-organization>&bucket=<your-bucket>&precision=s
```
Replace the URL with your InfluxDB address, the name of your organization, and the name of the bucket to store the data. You should also configure InfluxDB to accept data from Prometheus using the remote write endpoint.

3. Query with Flux

Use InfluxDB’s Flux query language to analyze metrics. Here’s how you can get data from a bucket:

```flux
from(bucket: "<your-bucket>")
  |> range(start: -1d)
  |> filter(fn: (r) => r["_measurement"] == "cpu_usage")
```
This Flux query will get the `cpu_usage` metrics from the past day.

Key Differences

• SQL vs. Flux: TimescaleDB uses SQL, while InfluxDB uses Flux. Pick what’s more suitable for your team.
• Integrations: Both work with Prometheus, but their integration points are different.
• General-Purpose vs. Time-Series Focus: TimescaleDB is a more generic database optimized for time series, while InfluxDB is a dedicated time-series database.

Both TimescaleDB and InfluxDB are solid options, depending on how you want to query the data and your background with SQL or dedicated time-series tools.

Best Practices for Prometheus Long-Term Storage

Regardless of the storage option you choose, follow these best practices:

• Retention Policies: Create a good policy for how long to keep metrics, based on your requirements.
• Downsampling: Use downsampling to save on space, and make sure queries are fast, especially when querying old data.
• Backups: Regularly back up your metrics, regardless of the option you use, as a way to prevent data loss.
• Monitoring: Monitor the system used to store your metrics, and set up alerts for issues. Make sure your system is up and running correctly.
• Security: Secure your storage system and make sure only the needed components can access the metrics. Use encryption for data when in rest and when in transit.
• Testing: Test how the different solutions work with your metrics. Before pushing anything into production, test it in a non-production environment first.
• Stay Up to Date: Keep the storage tools and Prometheus components updated with the latest fixes. Keep your software updated to the latest stable versions.
• Optimize Queries: Make sure to only query the data you need, and don’t query more than needed to make sure the system runs fast.
• Capacity Planning: Plan how much storage, network, and CPU resources you’ll need, based on your system scale.
• Alerts: Set up alerts for your storage system to make sure it’s running correctly and you are aware of any issues.

By following these best practices, you will ensure that your long-term storage for Prometheus is reliable, scalable and secure.

How to Migrate Existing Prometheus Data

If you have data in Prometheus’s local storage, you may want to move it to the new storage option. This is not always a trivial task, but you should plan it. Here’s a plan you can use:

• Create a Backup: Before you move the data, make a copy of the local storage in Prometheus, in case anything goes wrong. You can back up the folder to disk or to another machine to keep it safe.
• Use a tool: You may be able to use a tool like prombackup to copy metrics to a remote storage in bulk. Be sure to follow all documentation and procedures to avoid data loss or inconsistencies.
• Plan Downtime: Plan a small downtime if you have to stop Prometheus to move data. In some cases, such as if you use a remote write system, it might not be needed, so plan it according to your specific use case.
• Set Up the New System: Be sure the new system is set up, running, and able to store new metrics. You can test with a small set of metrics before sending everything to it.
• Test Queries: Test the queries against the new storage. Make sure you have all metrics in the new system and they work as expected.
• Rollback: Have a plan to roll back to the old system. Make sure you can switch to the old system and keep it running until the new system is working as expected.
• Verify: Be sure to compare data in the new system to the old one to make sure the migration went well and all data is there.

Migrating data is a complex operation and should be done with a lot of care and caution. Take some time to understand your current environment, and the new one, to ensure a smooth transition.

Conclusion: Taking Control of Your Metrics

Choosing the right long-term storage for Prometheus is a big decision. It helps you make use of all the information you collect, and gives you a solid system for monitoring. With options such as Thanos, Cortex, M3DB, TimescaleDB and InfluxDB, you can now pick the solution that fits your needs the best. So, make sure you evaluate all of them, make a test, and decide which one works better for you. And always remember that proper planning, maintenance, and a focus on long-term vision will guarantee that your metrics are safe and will give you all the insights you need.