Skip to main content

eBPF

eBPF is an emerging Linux kernel technology that allows for user-supplied programs to run inside of the kernel. This enables a bunch of interesting use cases, particularly efficient CPU profiling of the whole Linux system.

Supported platforms#

Spy NameTypeLinuxmacOSWindowsDocker
ebpfspyembedded

Benefits of using eBPF for continuous profiling#

The pros of using the eBPF integration are:

  • Easy installation: You can add the agent to your node without making code changes as opposed to adding agent libraries to your code
  • Access to kernel stacktraces: While most agent libraries focus on user-space code performance, eBPF can also give insight into kernel-space code performance

Tradeoffs of using eBPF for continuous profiling#

Because eBPF is still a growing technology, there's a few tradeoffs that you have to make in order use eBPF for profiling your applications

  • Doesn't work as well with interpreted languages
  • Requires certain kernel versions

Getting Started with eBPF Profiling#

For the reasons mentioned above, we recommend an hybrid approach for better results: eBPF to profile the node and specific language instrumentation per application. The following steps will explain how to get started:

Prerequisites for profiling with eBPF#

For the eBPF integration to work you'll need:

Running eBPF profiler on Kubernetes#

Step 1. Install pyroscope agent#

First, add the helm repo

helm repo add pyroscope-io https://pyroscope-io.github.io/helm-chart

Then install the agent

helm install pyroscope-ebpf pyroscope-io/pyroscope-ebpf

It will install pyroscope eBPF agent on all of your nodes and start profiling applications across your cluster.

note

If you're using Pyroscope Cloud, you can specify an auth key by populating the PYROSCOPE_AUTH_TOKEN environment variable or passing an --auth-token flag:

- "--auth-token"- "PYROSCOPE_CLOUD_API_KEY"- "--server-address"- "https://ingest.pyroscope.cloud"

Running eBPF profiler from binary#

export PYROSCOPE_APPLICATION_NAME=my.ebpf.programexport PYROSCOPE_SERVER_ADDRESS=http://address-of-pyroscope-server:4040/export PYROSCOPE_SPY_NAME=ebpfspy# optionally, if authentication is enabled, specify the API key:# export PYROSCOPE_AUTH_TOKEN={YOUR_API_KEY}
# to wrap an existing program and profile itsudo -E pyroscope exec mongod
# to profile the whole systemsudo -E pyroscope ebpf

Dealing with [unknowns]#

eBPF relies on having debugging symbols available for each program installed in your system. If you don't have those you'll see a lot of stacktraces full of [unknown]s. On most systems you can get debugging symbols for most packages with debuginfo-install command:

sudo debuginfo-install -y <pkg>

Configuration#

All parameters below are also supported as CLI arguments, a full list can be accessed via pyroscope ebpf --help. For brevity only environment variables are listed.

  • PYROSCOPE_KUBERNETES_NODE Set to current k8s Node.nodeName for service discovery and labeling
  • PYROSCOPE_ONLY_SERVICES Ignore processes unknown to service discovery
  • PYROSCOPE_SYMBOL_CACHE_SIZE Max size of symbols cache (1 entry per process)
env vardefaultdescription
PYROSCOPE_KUBERNETES_NODE""Used by service discovery. It's automatically set in the Helm Chart.
PYROSCOPE_ONLY_SERVICESfalseIgnore processes unknown to service discovery. In a Kubernetes cluster it ignores processes like containerd, runc, kubelet etc
PYROSCOPE_SYMBOL_CACHE_SIZE256Max size of symbols cache (1 entry per process). Change this value if you’re experiencing memory pressure or have many individual services.

Example#

Check out this docker-compose example in our repository for an example of how you can use eBPF integration or see our demo.