Exporting Metrics
Pyroscope Server offers a way to export values of a particular stack trace sample from your profiles as a Prometheus metric. This can be very useful if you already know a spot in your code that might be causing a problem, and you want to keep track of it.
Exported metrics are of counter type and are available
at /exported-metrics
endpoint. Samples of CPU time are counted in seconds based on the sample period, samples of other
types are represented as-is:
Profile type | Units | Aggregation |
---|---|---|
cpu | seconds | sum |
inuse_objects | objects | sum |
alloc_objects | objects | sum |
inuse_space | bytes | avg |
alloc_space | bytes | avg |
Metric export rules can be specified in the server configuration file:
metrics-export-rules: {}
metrics-export-rules
is a map of rules for exporting metrics where the key is a unique valid prometheus metric name.
A rule is defined as follows:
---# Expression in FlameQL syntax to be evaluated against samples.expr: ""
# Tree node filter expression. Should be either 'total' or a valid RE2 regexp.node: ""
# List of tags to be used for aggregation. The tags are exported as prometheus labels.group_by: []
For example, let's define a metric that shows how many sampled values of CPU time have been captured for
main;slowFunction
calls of app.name
application in staging
environment:
app_name_expensive_call: expr: 'app.name.cpu{env="staging"}' node: "main;slowFunction" labels: - "version"
note
When ingesting data, Pyroscope Agent adds a name suffix specifying the profile type: <application_name>.<profile_type>
.
Node filter is a regular expression applied to sampled stack traces, frames are delimited by semicolon. Any call that
includes main;slowFunction
in its stack trace will match the rule and will be added to the metric value. If the sample
has a version
tag attached, it will be preserved as a label of the exported metric, and any other tag will be
omitted โ pretty much like GROUP BY
statement in SQL.
Similarly, a metric for the whole application can be configured:
app_name_total: expr: "app.name.cpu" node: "total"
#
ExampleLet's configure some metrics to demonstrate how filtering and aggregation works:
a_simple_app_total
counts application samples.a_simple_app_slow_env_prod
countsslowFunction
samples withenv=prod
tag.a_simple_app_slow_env
countsslowFunction
samples broken down byenv
tag.
You can find sample program in the
Pyroscope repository. We will run two instances: one with env=prod
tag and another one with env=staging
.
metrics-export-rules: a_simple_app_total: expr: simple.golang.app.cpu
a_simple_app_slow_env_prod: expr: simple.golang.app.cpu{env="prod"} node: slowFunction
a_simple_app_slow_env: expr: simple.golang.app.cpu node: slowFunction labels: - "env"
After some time you should see exported metrics at /exported-metrics
endpoint:
a_simple_app_slow_env_prod 1260a_simple_app_slow_env{env="staging"} 1701a_simple_app_slow_env{env="prod"} 1260a_simple_app_total 3768
You can see how exporting works using the example in the pyroscope repository. The example also demonstrates how to query the resulting metrics.