Go Tracing Integration
#
OpenTelemetry supportPyroscope can integrate with distributed tracing systems supporting OpenTelemetry standard which allows you to link traces with the profiling data, and find specific lines of code related to a performance issue.
note
- Only CPU profiling is supported at the moment.
- We recommend you using Go 1.18 to ensure accurate results.
- Because of how sampling profilers work, spans shorter than the sample interval may not be captured. Go CPU profiler probes stack traces 100 times per second, meaning that spans shorter than 10ms may not be captured.
Go code can be easily instrumented with otel-profiling-go package -
a TracerProvider
implementation, that annotates profiling data with span IDs which makes it possible to filter
out profile of a particular trace span in Pyroscope:
# Make sure you also upgrade pyroscope server to version 0.14.0 or higher.go get github.com/pyroscope-io/otel-profiling-go
Now we can create and configure the tracer provider:
import otelpyroscope "github.com/pyroscope-io/otel-profiling-go"
// Create an exporter that prints traces to stdout.exporter, _ := stdouttrace.New(stdouttrace.WithPrettyPrint())tp := trace.NewTracerProvider( trace.WithSampler(trace.AlwaysSample()), trace.WithBatcher(exporter),)defer func() { if err := tp.Shutdown(context.Background()); err != nil { log.Fatal(err) }}()
// We wrap the tracer provider to also annotate goroutines with Span ID so// that pprof would add corresponding labels to profiling samples.otel.SetTracerProvider(otelpyroscope.NewTracerProvider(tp, otelpyroscope.WithAppName("simple.golang.app"), otelpyroscope.WithPyroscopeURL("http://pyroscope-server:4040"), otelpyroscope.WithRootSpanOnly(true), otelpyroscope.WithAddSpanName(true), otelpyroscope.WithProfileURL(true), otelpyroscope.WithProfileBaselineURL(true),))
Please notice the WithRootSpanOnly
option: when enabled, the tracer will annotate only the first span created locally
(the root span), but the profile will include samples of all the nested spans. This may be helpful in case if the trace
consists of multiple spans shorter than 10ms and profiler can't collect and annotate samples properly.
Another option that you may find useful is WithAddSpanName
that controls whether the span name added to profile labels
automatically. You can find description of all the options in the package documentation.
important
If you enable WithAddSpanName
option, please make sure span names do not contain unique values, for example, a URL.
Otherwise, this can increase data cardinality and slow down queries.
Now that we set up the tracer, we can create a new trace from anywhere:
ctx, span := otel.Tracer("tracerName").Start(context.Background(), "ExampleSpan")defer span.End()
// Your code goes here.
Collected profiles can be viewed in Pyroscope UI using FlameQL:
simple.golang.app.cpu{profile_id="<spanID>"}
- Shows flamegraph for a particular span.simple.golang.app.cpu{span_name="ExampleSpan"}
- Shows aggregated profile for all spans named ExampleSpan.
For convenience, the tracer annotates profiled spans with extra attributes:
pyroscope.profile.id
- is set to span ID to indicate that profile was captured for a span.pyroscope.profile.url
- contains the URL to the flamegraph in Pyroscope UI.pyroscope.profile.baseline.url
- contains the URL to the baseline comparison view in Pyroscope UI.
#
Baseline profilesA baseline profile is an aggregate of all span profiles. For example, consider two exemplars (the number here replaces profiling data):
app.cpu{region="us=east",env="dev",span_name="FetchData",profile_id="abc"}
100app.cpu{region="us=east",env="dev",span_name="FetchData",profile_id="zyx"}
200
Then, the baseline profile for each of them is:
app.cpu{region="us=east",env="dev",span_name="FetchData"}
300
#
ExamplesCheck out the examples directory in our repository to find a complete example application that demonstrates tracing integration features (using a custom jaeger ui) and learn more!
#
Using tracing exemplars manuallyIf you're not using open telemetry integration you can still use exemplars storage to store profiles associated with some execution context (e.g individual HTTP / GRPC request). To create exemplars you need to tag specific parts of your code with a special profile_id
tag, for example, in golang you could do this:
pprof.Do(ctx, pprof.Labels("profile_id", "8474e98b95013e4f"), func(ctx context.Context) { slowRequest()})
"8474e98b95013e4f"
can be any ID that you use to identify execution contexts (individual HTTP / GRPC requests).