Skip to main content
info

In March 2023, Grafana Labs acquired Pyroscope, the company behind the eponymous open source continuous profiling project. In September 2023, Grafana Pyroscope v1.0.0 was released.

This documentation covers Pyroscope versions preceding v1.0.0. We're keeping it for the time being, but we recommend that you upgrade to Grafana Pyroscope v1.0.0 or higher and switch to the new Grafana Pyroscope Docs for most up-to-date documentation.

Troubleshooting/FAQ (Frequently Asked Questions)

What are the minimum system requirements for running pyroscope? (RAM, Disk)#

My pyroscope server runs out of memory#

It's difficult to accurately estimate memory consumption in advance, as it is highly dependent on the workload - mainly the size of profiles and the ingestion rate.

It's fair to estimate resource consumption per agent/target based on the actual memory consumption under the actual load. As a pessimistic estimation we suggest 32 MB per agent/target.

Also keep in mind pyroscope reserves some memory at startup (~400 MB), and also makes heavy use of various internal caches that grow over time and affect overall memory consumption. There are 2 parameters to manage this:

  • cache-evict-threshold: percentage of consumed memory at which cache evictions start. The parameter relies on the container limits.
  • cache-evict-volume: percentage of cache that is evicted per eviction run.

My pyroscope server is running out of space#

In these cases, you can set up a more aggressive retention policy. See the Data retention page for more instructions

My pyroscope server is crashing due to out of space#

Although pyroscope tries to leave enough space, it may be possible that it gets stuck in a crashing state, where it can't even start due to lack of space.

There are 2 options: increase the disk and free some data.

Free some data#

warning

We don't guarantee pyroscope will be able to deal with this correctly. Only use this as a last resort. Preferrably do a backup first.

It depends on where you are running, for kubernetes you can overwrite the Deployment's args to run a dummy command (like sleep infinity).

Then, you can the oldest .vlog files in the trees directory.