Go Pull Mode
Modern observability systems generally fall into two categories: those where services push data, and those where the observability system pulls data from services. We find that both approaches are suitable: they both have advantages and disadvantages, and one of them can be more desirable than another under certain circumstances.
Pyroscope server can operate in both Pull and Push modes. The current implementation makes extensive use of Prometheus scrape and service discovery mechanisms. We express our deep gratitude to Prometheus team for their incredible work!
important
If you want to use mutex or block profiling you have to explicitly enable it in your application. See Mutex Profiling section for more information.
important
Before golang 1.19 version, goroutine profile will lead to a long time STW proportional to the number of goroutines, see here for more details.
#
Supported languages and platformsAny application that exposes data in pprof
format via HTTP can be set as a remote profiling target.
#
Scrape configurationPyroscope uses exactly the same mechanisms as Prometheus does in order to ensure smooth user experience. Therefore, it is configured in almost identical way, and Prometheus documentation can be used as a reference.
You will need to add the following content to your pyroscope/server.yml
Pyroscope config file. See the Server config documentation for more information on where this config is located by default on your system.
---# A list of scrape configurations.scrape-configs: # The job name assigned to scraped profiles by default. - job-name: pyroscope
# How frequently to scrape targets by default. scrape-interval: 10s
# The list of profiles to be scraped from the targets. enabled-profiles: [cpu, mem, goroutines, mutex, block]
# List of labeled statically configured targets for this job. static-configs: - application: my-application-name spy-name: gospy targets: - hostname:6060 labels: env: dev
note
application
is the name of the application being profiled. Mandatory field.targets
lists application instances. It is important to avoid mixing profiles of different applications.
#
Scrape intervalBy default, a target is scraped on 10 seconds interval which is specified with scrape-interval
parameter.
That is, with the default configuration a 10 seconds profile is collected every 10 seconds. Both interval and duration
parameters can be modified to allow sampling strategies, where only a subset of targets is being profiled at a given time
and only a subset of profiling data is being collected.
For example, configuring scrape-interval
to 60 seconds with the default 10 second profiling duration effectively
means that only 1/6 of the data is collected, because only 1/6 of the targets is being scraped, which in turn:
- Reduces the overall profiling overhead
- Decreases resource usage of the Pyroscope server
---scrape-configs: - job-name: pyroscope scrape-interval: 60s enabled-profiles: [cpu, mem, goroutines, mutex, block] static-configs: - application: my-application-name spy-name: gospy targets: - hostname:6060 labels: env: dev
There are two types of profiles in Go:
- Profiles that accumulate samples during a profiling session:
cpu
,mutex
, andblock
. - Instant profiles that represent the current state:
goroutines
andmem
โ Pyroscope stores the delta of two consecutive "snapshots".
For profiles of the first type, you can override the duration of the profiling session, which allows you to use even more flexible scenarios. However, setting the profiling duration shorter than the scrape interval degrades the accuracy of the resulting profiles and may significantly complicate their analysis.
For profiles of the second type, profiling duration is always equal to scrape-interval
.
The example configuration below instructs Pyroscope to collected cpu, block, and mutex profiles for 30 seconds every minute, goroutine and memory profiles are collected every minute as well but include all the stack trace samples emerged within this time window.
note
If you change profile duration, you also need to increase scrape-timeout
accordingly. scrape-timeout
defaults to 15 seconds.
---scrape-configs: - job-name: pyroscope scrape-interval: 60s scrape-timeout: 60s enabled-profiles: [cpu, mem, goroutines, mutex, block] use-delta-profiles: true profiles: cpu: params: seconds: [ "30" ] block: params: seconds: [ "30" ] mutex: params: seconds: [ "30" ] static-configs: - application: my-application-name spy-name: gospy targets: - hostname:6060 labels: env: dev
note
Because of the storage engine specifics, we advise you to set intervals (both profiling duration and scrape interval) to a multiple of 10 seconds. This allows you to reduce the use of resources by the pyroscope server.
#
Service DiscoveryPyroscope creates pull targets based on the discovered labels. At least __name__
and __address__
labels must be
present, where __name__
is the name of the application being profiled.
Optional labels:
__scheme__
: If the metrics endpoint is secured then you will need to set this tohttps
.__port__
: Scrape the target on the indicated port.__profile_{profile_name}_enabled
: Indicates whether a particular profile should be scraped.__profile_{profile_name}_path
: Specifies URL path exposing pprof profile.__profile_{profile_name}_param_{param_key}
: Overrides scrape URL parameters.
Where {profile_name}
must be a valid profile configuration name.
At this point, Pyroscope fully supports only Kubernetes Service Discovery.
#
Mutex Profilingnote
Mutex profiling requires pyroscope v0.20.0 or higher.
Mutex profiling is useful for finding sources of contention within your application. It helps you to find out which mutexes are being held by which goroutines.
To enable mutex profiling, you need to add the following code to your application:
runtime.SetMutexProfileFraction(rate)
rate
parameter controls the fraction of mutex contention events that are reported in the mutex profile. On average 1/rate events are reported.
#
Block Profilingnote
Block profiling requires pyroscope v0.20.0 or higher.
Block profiling lets you analyze how much time your program spends waiting on the blocking operations such as:
- select
- channel send/receive
- semacquire
- notifyListWait
To enable block profiling, you need to add the following code to your application:
runtime.SetBlockProfileRate(rate)
rate
parameter controls the fraction of goroutine blocking events that are reported in the blocking profile. The profiler aims to sample an average of one blocking event per rate nanoseconds spent blocked.
#
godeltaprofgodeltaprof is a memory profiler for cumulative profiles(heap, block, mutex). It is more efficient because it does the delta/merging before producing pprof data, avoiding extra decompression/parsing/allocations/compression.
To start using godeltaprof in pull mode in a Go application, you need to include godeltaprof module in your app:
go get github.com/pyroscope-io/godeltaprof@latest
Integration is very simillar to net/http/pprof
, you need to import a new package and it will expose new endpoints /debug/pprof/delta_heap
, /debug/pprof/delta_block
, /debug/pprof/delta_mutex
_ "github.com/pyroscope-io/godeltaprof/http/pprof"
In the scrape config you need to enable new delta endpoints with use-delta-endpoints: true
, for example:
scrape-configs: - job-name: pyroscope1 enabled-profiles: [cpu, mem, block, mutex] use-delta-profiles: true
#
Examples#
Static targetsYou can find an example of how Pyroscope scrapes static targets in the pyroscope repository.
#
Kubernetes service discoveryYou can see how Pyroscope discovers remote targets in Kubernetes using the example setup in the pyroscope repository.
Here is an example of how to configure Pyroscope to scrape targets from Kubernetes pods:
---pyroscopeConfigs: log-level: debug scrape-configs: - job-name: 'kubernetes-pods' enabled-profiles: [ cpu, mem ] kubernetes-sd-configs: - role: pod relabel-configs: - source-labels: [__meta_kubernetes_pod_annotation_pyroscope_io_scrape] action: keep regex: true - source-labels: [__meta_kubernetes_pod_annotation_pyroscope_io_application_name] action: replace target-label: __name__ - source-labels: [__meta_kubernetes_pod_annotation_pyroscope_io_scheme] action: replace regex: (https?) target-label: __scheme__ - source-labels: [__address__, __meta_kubernetes_pod_annotation_pyroscope_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target-label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source-labels: [__meta_kubernetes_namespace] action: replace target-label: kubernetes_namespace - source-labels: [__meta_kubernetes_pod_name] action: replace target-label: kubernetes_pod_name - source-labels: [__meta_kubernetes_pod_phase] regex: Pending|Succeeded|Failed|Completed action: drop - action: labelmap regex: __meta_kubernetes_pod_annotation_pyroscope_io_profile_(.+) replacement: __profile_$1
note
Kubernetes Service Discovery requires RBAC set up. Please refer to Pyroscope helm chart for details.
#
File service discoveryYou can configure Pyroscope server to scrape targets defined in files that can be updated dynamically without restarting the server. You can find an example configuration in the pyroscope repository.
#
Consul service discoveryConsul service discovery allow retrieving scrape targets from Consul's Catalog API. You can find an example configuration in the pyroscope repository.