Skip to main content

Go Pull Mode

Modern observability systems generally fall into two categories: those where services push data, and those where the observability system pulls data from services. We find that both approaches are suitable: they both have advantages and disadvantages, and one of them can be more desirable than another under certain circumstances.

Pyroscope server can operate in both Pull and Push modes. The current implementation makes extensive use of Prometheus scrape and service discovery mechanisms. We express our deep gratitude to Prometheus team for their incredible work!

Supported languages and platforms#

Any application that exposes data in pprof format via HTTP can be set as a remote profiling target.

Scrape configuration#

Pyroscope uses exactly the same mechanisms as Prometheus does in order to ensure smooth user experience. Therefore, it is configured in almost identical way, and Prometheus documentation can be used as a reference.

You will need to add the following content to your pyroscope/server.yml Pyroscope config file. See the Server config documentation for more information on where this config is located by default on your system.

---# A list of scrape configurations.scrape-configs:  # The job name assigned to scraped profiles by default.  - job-name: pyroscope
    # The list of profiles to be scraped from the targets.    enabled-profiles: [cpu, mem]
    # List of labeled statically configured targets for this job.    static-configs:      - application: my-application-name        spy-name: gospy        targets:          - hostname:6060        labels:          env: dev
note
  • application is the name of the application being profiled. Mandatory field.
  • targets lists application instances. It is important to avoid mixing profiles of different applications.

Service Discovery#

Pyroscope creates pull targets based on the discovered labels. At least __name__ and __address__ labels must be present, where __name__ is the name of the application being profiled.

Optional labels:

  • __scheme__: If the metrics endpoint is secured then you will need to set this to https.
  • __port__: Scrape the target on the indicated port.
  • __profile_{profile_name}_enabled: Indicates whether a particular profile should be scraped.
  • __profile_{profile_name}_path: Specifies URL path exposing pprof profile.
  • __profile_{profile_name}_param_{param_key}: Overrides scrape URL parameters.

Where {profile_name} must be a valid profile configuration name.

At this point, Pyroscope fully supports only Kubernetes Service Discovery.

Examples#

Static targets#

You can find an example of how Pyroscope scrapes static targets in the pyroscope repository.

Kubernetes pods discovery#

You can see how Pyroscope discovers remote targets in Kubernetes using the example setup in the pyroscope repository.

note

Kubernetes Service Discovery requires RBAC set up. Please refer to Pyroscope helm chart for details.