Go

Pyroscope uses the standard runtime/pprof package to collect profiling data. Refer to the official documentation for details.

Supported platforms#

Spy Name	Type	Linux	macOS	Windows	Docker
n/a	`standalone`	✅	✅	✅	✅

Profiling Go application#

To start profiling a Go application, you need to include our go module in your app:

# make sure you also upgrade pyroscope server to version 0.3.1 or highergo get github.com/pyroscope-io/client/pyroscope

Then add the following code to your application:

package main
import "github.com/pyroscope-io/client/pyroscope"
func main() {  // These 2 lines are only required if you're using mutex or block profiling  // Read the explanation below for how to set these rates:  runtime.SetMutexProfileFraction(5)  runtime.SetBlockProfileRate(5)
  pyroscope.Start(pyroscope.Config{    ApplicationName: "simple.golang.app",
    // replace this with the address of pyroscope server    ServerAddress:   "http://pyroscope-server:4040",
    // you can disable logging by setting this to nil    Logger:          pyroscope.StandardLogger,
    // optionally, if authentication is enabled, specify the API key:    // AuthToken:    os.Getenv("PYROSCOPE_AUTH_TOKEN"),        // you can provide static tags via a map:    Tags:            map[string]string{"hostname": os.Getenv("HOSTNAME")},
    ProfileTypes: []pyroscope.ProfileType{      // these profile types are enabled by default:      pyroscope.ProfileCPU,      pyroscope.ProfileAllocObjects,      pyroscope.ProfileAllocSpace,      pyroscope.ProfileInuseObjects,      pyroscope.ProfileInuseSpace,
      // these profile types are optional:      pyroscope.ProfileGoroutines,      pyroscope.ProfileMutexCount,      pyroscope.ProfileMutexDuration,      pyroscope.ProfileBlockCount,      pyroscope.ProfileBlockDuration,    },  })
  // your code goes here}

important

If you want to use mutex or block profiling you have to explicitly enable it in your application. See Mutex Profiling section for more information.

important

Before golang 1.19 version, goroutine profile will lead to a long time STW proportional to the number of goroutines, see here for more details.

Tags#

It is possible to add tags (labels) to the profiling data. These tags can be used to filter the data in the UI. We have a custom API that's in line with our other integrations (e.g Python or Ruby) as well as go-native pprof api:

// these two ways of adding tags are equivalent:pyroscope.TagWrapper(context.Background(), pyroscope.Labels("controller", "slow_controller"), func(c context.Context) {  slowCode()})
pprof.Do(context.Background(), pprof.Labels("controller", "slow_controller"), func(c context.Context) {  slowCode()})

Pull Mode#

Go integration supports pull mode, which means that you can profile applications without adding any extra code. For that to work you will need to make sure you have profiling routes (/debug/pprof) enabled in your http server. Generally, that means that you need to add net/http/pprof package:

import _ "net/http/pprof"

For information on how to use net/http/pprof refer to official net/http/pprof documentation.
For more information on how to set up pyroscope server to pull data from your applications, refer to pull mode documentation.

godeltaprof#

godeltaprof is a memory profiler for cumulative profiles(heap, block, mutex). It is more efficient because it does the delta/merging before producing pprof data, avoiding extra decompression/parsing/allocations/compression.

To start using godeltaprof in pull mode in a Go application, you need to include godeltaprof module in your app:

go get github.com/pyroscope-io/godeltaprof@latest

Integration is very simillar to net/http/pprof, you need to import a new package and it will expose new endpoints /debug/pprof/delta_heap, /debug/pprof/delta_block, /debug/pprof/delta_mutex

_ "github.com/pyroscope-io/godeltaprof/http/pprof"

In the scrape config you need to enable new delta endpoints with use-delta-endpoints: true, for example:

scrape-configs:  - job-name: pyroscope1    enabled-profiles: [cpu, mem, block, mutex]    use-delta-profiles: true

For push mode godeltaprof is used automatically since v0.6.0

Tracing Integration#

Pyroscope can integrate with distributed tracing systems supporting OpenTelemetry standard which allows you to link traces with the profiling data, and find specific lines of code related to a performance issue. For more information, refer to tracing integration documentation.

Mutex Profiling#

note

Mutex profiling requires pyroscope v0.20.0 or higher and client integration v.0.3.0 or higher.

Mutex profiling is useful for finding sources of contention within your application. It helps you to find out which mutexes are being held by which goroutines.

To enable mutex profiling, you need to add the following code to your application:

runtime.SetMutexProfileFraction(rate)

rate parameter controls the fraction of mutex contention events that are reported in the mutex profile. On average 1/rate events are reported.

Block Profiling#

note

Block profiling requires pyroscope v0.20.0 or higher and client integration v.0.3.0 or higher.

Block profiling lets you analyze how much time your program spends waiting on the blocking operations such as:

select
channel send/receive
semacquire
notifyListWait

To enable block profiling, you need to add the following code to your application:

runtime.SetBlockProfileRate(rate)

rate parameter controls the fraction of goroutine blocking events that are reported in the blocking profile. The profiler aims to sample an average of one blocking event per rate nanoseconds spent blocked.

Sending data to Phlare with Pyroscope Golang integration#

Starting with weekly-f8 you can ingest pyroscope profiles directly to phlare.

pyroscope.Start(pyroscope.Config{  ApplicationName:   "phlare.golang.app",  ServerAddress:     "<URL>",  // Optional HTTP Basic authentication  BasicAuthUser:     "<User>",  BasicAuthPassword: "<Password>",  // Optional Phlare tenant ID  TenantID:        "<TenantID>",  ProfileTypes: []pyroscope.ProfileType{    pyroscope.ProfileCPU,    pyroscope.ProfileInuseObjects,    pyroscope.ProfileAllocObjects,    pyroscope.ProfileInuseSpace,    pyroscope.ProfileAllocSpace,  },})

To configure the Golang integration to send data to Phlare, replace the <URL> placeholder with the appropriate server URL. This could be the grafana.com Phlare URL or your own custom Phlare server URL.

If you need to send data to grafana.com, you'll have to configure HTTP Basic authentication. Replace <User> with your grafana.com stack user and <Password> with your grafana.com API key.

If your Phlare server has multi-tenancy enabled, you'll need to configure a tenant ID. Replace <TenantID> with your Phlare tenant ID.

Golang profiling examples#

Check out the following resources to learn more about Golang profiling:

Golang examples
Golang Demo showing golang example with tags
Golang blog post

Frame width represents CPU time per function

Pyroscope