Visualize profiles in Grafana with Pyroscope Flamegraph plugin

warning

Pyroscope is now part of Grafana Labs! As a result we have consolidated efforts around our Grafana plugins to make one recomended way of using Pyroscope. As a result this blogpost is now outdated. However, you can find updated docs on how to add profiling to Grafana with Pyroscope using the Grafana profiling documentation.

Grafana is an open-source observability and monitoring platform used by individuals and organizations to monitor their applications and infrastructures. Grafana leverages the three pillars of observability, metrics, logs, and traces, to deliver insights into how well your systems are doing. Nowadays, Observability involves a whole lot more than metrics, logs, and tracing; it also involves profiling.

In this article, I will:

1. Describe how to leverage continuous profiling in Grafana by installing the Pyroscope flamegraph panel and datasource plugin
2. Show how to configure the plugins properly
3. Explain how to setup your first dashboard that includes profiling
4. Give a sneak peak of an upcoming feature that will let you link profiles to logs, metrics, and traces

If you're new to flamegraphs and would like to learn more about what they are and how to use them, see this blog post.

Introduction

Grafana provides you with tools to visualize your metrics, view logs, and analyze traces, but it is incomplete without the added benefits of profiling. Continuous Profiling is super critical when you’re looking to debug an existing performance issue in your application. It enables you to monitor your application’s performance over time and provides insights into parts of your application that are consuming resources the most. Continuous profiling is used to locate and fix memory leaks, clean up unused code, and understand the call tree of your application. This results in a more efficient application.

Benefits of using Pyroscope in Grafana

Unified view for complete observability

Using Pyroscope in Grafana provides you with complete observability without leaving your Grafana dashboard. Grafana leverages the powerful features of Pyroscope to take complete control of your application’s end-to-end observability and makes things like debugging easy. You can now see your profiles alongside corresponding logs, metrics, and traces to tell the complete story of your application.

Zero migration cost

It costs nothing to migrate your application profile from Pyroscope’s UI dashboard into Grafana. Simply open Grafana and install both the Pyroscope panel and datasource plugin, and you’re all set!

Single authentication

Because your profiling data is visualized in Grafana, you can login using just your Grafana credentials. Single authentication reduces the stress of logging into both Grafana and Pyroscope applications to visualize your profiles.

Manage user permissions

Pyroscope leverages Grafana's multi user feature, which allows you as an admin to manage permissions for different users that have access to your dashboard. This comes in handy when you want to share access to multiple users within your team or organization.

Install Pyroscope’s Flamegraph’s plugin in Grafana

To install Pyroscope’s flamegraph plugin in Grafana, you will need the Pyroscope panel plugin and the Pyroscope datasource plugin.

The Pyroscope panel plugin serves as the visualization option in which we use to show our profiling data. This is represented as a flamegraph on your Grafana dashboard. The Pyroscope datasource plugin connects your Grafana instance to the host, Pyroscope server.

note

To run Pyroscope’s data source, you need to be running on Grafana 7.2 or later. To perform actions such as variable querying in Pyroscope’s flamegraph, you need to install version 1.1.0 or later of the pyroscope-datasource.

How to Install the Pyroscope Panel and Pyroscope Datasource plugins

To install the two plugins that you'll need, first navigate to your Grafana directory using your terminal. Then, run this command:

grafana-cli plugins install pyroscope-panel # Installs the Pyroscope panel plugingrafana-cli plugins install pyroscope-datasource # Installs the Pyroscope datasource plugin

These commands install the two plugin into your Grafana application.

Once you are done installing the two plugins, restart your Grafana instance so that your plugins can be loaded properly.

Configure the Pyroscope Data source

note

Make sure your Pyroscope server is up and running before attempting to configure your data source.

In order to use the Pyroscope datasource plugin, you need to configure your datasource appropriately. To configure the Pyroscope data source, you need to follow these steps:

1. Navigate to the datasources page

2. Click the button to add your datasource.
3. Search for Pyroscope datasource and click on it.
4. Specify Pyroscope host in instance field: http://localhost:4040
5. Click the button to save and test.

Adding API keys if using authentication

You can add API keys to your Grafana instance. Check out Pyroscope authentication page to learn more.

Setup Dashboard

Once you’ve successfully configured your datasource, follow these steps to create your dashboard:

1. Navigate to your dashboard
2. Click the “add a new panel” button to add an empty panel to your dashboard.
3. Navigate to the visualization list and select pyroscope-panel
4. Under panel view in Query tab, select pyroscope-datasource as your datasource
5. In the query input, enter the name to the specific profile you’re looking for. In this case, I’m interested in the Pyroscope server allocation space so I select pyroscope.server.alloc_space to view profile.
6. Click "Apply" to complete the setup and begin profiling!

You should see a flamegraph appear in the panel.

Running dockerized grafana example

If you’re interested in seeing the Pyroscope Grafana plugin in action, check out the example from the Pyroscope GitHub repository.

Sneak Peak at What's next for Pyroscope x Grafana integration

We believe that one of the biggest opportunities for Pyroscope to integrate into Grafana workflows is by linking directly with other observability data. Using our otel-pyroscope package we've shown how you can:

1. Auto-instrument your code with trace-specific profiles
2. Link directly from logs or traces to the profiles that are associated with them (i.e. via derived fields in the explore tab)

Here's a preview of what that ends up looking like in practice. Stay tuned for more info on our upcoming (official) release of this feature or if you want to try it out yourself this tracing example is available in the repo.