Skip to main content

Troubleshooting/FAQ (Frequently Asked Questions)

What are the minimum system requirements for running pyroscope? (RAM, Disk)#

My pyroscope server runs out of memory#

It's difficult to accurately estimate memory consumption in advance, as it is highly dependent on the workload - mainly the size of profiles and the ingestion rate.

It's fair to estimate resource consumption per agent/target based on the actual memory consumption under the actual load. As a pessimistic estimation we suggest 32 MB per agent/target.

Also keep in mind pyroscope reserves some memory at startup (~400 MB), and also makes heavy use of various internal caches that grow over time and affect overall memory consumption. There are 2 parameters to manage this:

  • cache-evict-threshold: percentage of consumed memory at which cache evictions start. The parameter relies on the container limits.
  • cache-evict-volume: percentage of cache that is evicted per eviction run.

My pyroscope server is running out of space#

In these cases, you can set up a more aggressive retention policy. See the Data retention page for more instructions

My pyroscope server is crashing due to out of space#

Although pyroscope tries to leave enough space, it may be possible that it gets stuck in a crashing state, where it can't even start due to lack of space.

There are 2 options: increase the disk and free some data.

Free some data#

warning

We don't guarantee pyroscope will be able to deal with this correctly. Only use this as a last resort. Preferrably do a backup first.

It depends on where you are running, for kubernetes you can overwrite the Deployment's args to run a dummy command (like sleep infinity).

Then, you can the oldest .vlog files in the trees directory.