Exporting Metrics

Pyroscope Server offers a way to export values of a particular stack trace sample from your profiles as a Prometheus metric. This can be very useful if you already know a spot in your code that might be causing a problem, and you want to keep track of it.

Exported metrics are of counter type and are available at /exported-metrics endpoint. Samples of CPU time are counted in seconds based on the sample period, samples of other types are represented as-is:

Profile type	Units	Aggregation
`cpu`	seconds	sum
`inuse_objects`	objects	sum
`alloc_objects`	objects	sum
`inuse_space`	bytes	avg
`alloc_space`	bytes	avg

Metric export rules can be specified in the server configuration file:

metrics-export-rules: {}

metrics-export-rules is a map of rules for exporting metrics where the key is a unique valid prometheus metric name. A rule is defined as follows:

---# Expression in FlameQL syntax to be evaluated against samples.expr: ""
# Tree node filter expression. Should be either 'total' or a valid RE2 regexp.node: ""
# List of tags to be used for aggregation. The tags are exported as prometheus labels.group_by: []

For example, let's define a metric that shows how many sampled values of CPU time have been captured for main;slowFunction calls of app.name application in staging environment:

app_name_expensive_call:  expr: 'app.name.cpu{env="staging"}'  node: "main;slowFunction"  labels:   - "version"

note

When ingesting data, Pyroscope Agent adds a name suffix specifying the profile type: <application_name>.<profile_type>.

Node filter is a regular expression applied to sampled stack traces, frames are delimited by semicolon. Any call that includes main;slowFunction in its stack trace will match the rule and will be added to the metric value. If the sample has a version tag attached, it will be preserved as a label of the exported metric, and any other tag will be omitted – pretty much like GROUP BY statement in SQL.

Similarly, a metric for the whole application can be configured:

app_name_total:  expr: "app.name.cpu"  node: "total"

Example#

Let's configure some metrics to demonstrate how filtering and aggregation works:

a_simple_app_total counts application samples.
a_simple_app_slow_env_prod counts slowFunction samples with env=prod tag.
a_simple_app_slow_env counts slowFunction samples broken down by env tag.

You can find sample program in the Pyroscope repository. We will run two instances: one with env=prod tag and another one with env=staging.

metrics-export-rules:  a_simple_app_total:    expr: simple.golang.app.cpu
  a_simple_app_slow_env_prod:    expr: simple.golang.app.cpu{env="prod"}    node: slowFunction
  a_simple_app_slow_env:    expr: simple.golang.app.cpu    node: slowFunction    labels:      - "env"

After some time you should see exported metrics at /exported-metrics endpoint:

a_simple_app_slow_env_prod 1260a_simple_app_slow_env{env="staging"} 1701a_simple_app_slow_env{env="prod"} 1260a_simple_app_total 3768

You can see how exporting works using the example in the pyroscope repository. The example also demonstrates how to query the resulting metrics.