Maximize Productivity with PromQL Macros

I want to set a request SLO for my service foo. This is what my expression looks like.

Now:

• I want to set up a 7-day SLO as well.
• I want to encourage the same SLO to be used across different services. Additionally, I use many patterns for rather elaborate PromQL queries across many metrics and at many times. These queries can be difficult to reproduce, and new organizational users may struggle to understand them.

We have two options:

Recording Rules

Using a Prometheus recording rule doesn't quite make sense, as we don't necessarily know where or when we'll want to use the pattern, and applying the rule to every metric is too expensive.

Plain Text Copy and Paste

However, there are problems with copy-pasting queries everywhere:

• Repetition of code
• Reduced abstractions and readability
• Error-prone large text blobs
• Modifications are hard and tiresome.

In the above example, I realized that the denominator includes all status codes, even those requests that result in 404. These 404s are reducing my availability. There is an instant fix that requires patching across SLO computations, and it must look like this:

To implement this fix across all consumers, dashboards and alerts will take:

• Weeks of migrations and trial and error to roll out.
• A week of distraction away from feature development.

🌈 Introducing PromQL Macros

Step 1: Define an Availability function.

Step 2: Use it.

For a Service foo for 1 day

Or, for Service bar for 7 days

Or, to Package Complex Queries

Macros are one of the most powerful tools for increasing productivity when working with PromQL. They work in a way that is similar to how SQL developers use stored procedures. Macros take full advantage of the time-tested best practices of functions, abstractions, and reusability to replace cumbersome and error-prone methods.

You can create macros that are specific to each Levitate cluster. These macros can function as logic modules that are easily reusable across Grafana, Alert Manager, or the command line interface (CLI). This means that the macros you create will be accessible by all queries, making it possible to streamline your workflow and increase efficiency. With the ability to define your functions and use them across different queries and services, the possibilities for increased productivity are endless.

Macro Syntax

The Macros use ES6-like syntax where you define a function that takes any number of arguments. The function has to return a valid PromQL query as its return value.

It can define any intermediate variables using the keyword let to simplify the return value.

function percentile(p, metric){ 
  let clusterName = "test-ap-cluster"
  let filter =  {cluster=clusterName,status="200"}
  return histogram_quantile(p, sum(rate(metric{filter}[5m])) by (le))
}

Using Macros in PromQL

You can use the defined Macros in your PromQLs to visualize them or set alerts on them. Since macros are defined on a per-cluster basis, you can only use it for that particular cluster as a data source.

A query using the above configuration template example would look like percentile(90, http_request_duration_seconds_bucket).

Read more about how to use PromQL based Macros here:

🔌 PromQL Macros

Everything you need to know to start harnessing power of PromQL based Macros

Last9 Documentation