Skip to main content

Go Pull Mode

Modern observability systems generally fall into two categories: those where services push data, and those where the observability system pulls data from services. We find that both approaches are suitable: they both have advantages and disadvantages, and one of them can be more desirable than another under certain circumstances.

Pyroscope server can operate in both Pull and Push modes. The current implementation makes extensive use of Prometheus scrape and service discovery mechanisms. We express our deep gratitude to Prometheus team for their incredible work!

important

If you want to use mutex or block profiling you have to explicitly enable it in your application. See Mutex Profiling section for more information.

important

Before golang 1.19 version, goroutine profile will lead to a long time STW proportional to the number of goroutines, see here for more details.

Supported languages and platforms#

Any application that exposes data in pprof format via HTTP can be set as a remote profiling target.

Scrape configuration#

Pyroscope uses exactly the same mechanisms as Prometheus does in order to ensure smooth user experience. Therefore, it is configured in almost identical way, and Prometheus documentation can be used as a reference.

You will need to add the following content to your pyroscope/server.yml Pyroscope config file. See the Server config documentation for more information on where this config is located by default on your system.

---# A list of scrape configurations.scrape-configs:  # The job name assigned to scraped profiles by default.  - job-name: pyroscope
    # The list of profiles to be scraped from the targets.    enabled-profiles: [cpu, mem, goroutines, mutex, block]
    # List of labeled statically configured targets for this job.    static-configs:      - application: my-application-name        spy-name: gospy        targets:          - hostname:6060        labels:          env: dev
note
  • application is the name of the application being profiled. Mandatory field.
  • targets lists application instances. It is important to avoid mixing profiles of different applications.

Service Discovery#

Pyroscope creates pull targets based on the discovered labels. At least __name__ and __address__ labels must be present, where __name__ is the name of the application being profiled.

Optional labels:

  • __scheme__: If the metrics endpoint is secured then you will need to set this to https.
  • __port__: Scrape the target on the indicated port.
  • __profile_{profile_name}_enabled: Indicates whether a particular profile should be scraped.
  • __profile_{profile_name}_path: Specifies URL path exposing pprof profile.
  • __profile_{profile_name}_param_{param_key}: Overrides scrape URL parameters.

Where {profile_name} must be a valid profile configuration name.

At this point, Pyroscope fully supports only Kubernetes Service Discovery.

Mutex Profiling#

note

Mutex profiling requires pyroscope v0.20.0 or higher.

Mutex profiling is useful for finding sources of contention within your application. It helps you to find out which mutexes are being held by which goroutines.

To enable mutex profiling, you need to add the following code to your application:

runtime.SetMutexProfileFraction(rate)

rate parameter controls the fraction of mutex contention events that are reported in the mutex profile. On average 1/rate events are reported.

Block Profiling#

note

Block profiling requires pyroscope v0.20.0 or higher.

Block profiling lets you analyze how much time your program spends waiting on the blocking operations such as:

  • select
  • channel send/receive
  • semacquire
  • notifyListWait

To enable block profiling, you need to add the following code to your application:

runtime.SetBlockProfileRate(rate)

rate parameter controls the fraction of goroutine blocking events that are reported in the blocking profile. The profiler aims to sample an average of one blocking event per rate nanoseconds spent blocked.

Examples#

Static targets#

You can find an example of how Pyroscope scrapes static targets in the pyroscope repository.

Kubernetes pods discovery#

You can see how Pyroscope discovers remote targets in Kubernetes using the example setup in the pyroscope repository.

note

Kubernetes Service Discovery requires RBAC set up. Please refer to Pyroscope helm chart for details.