This functionality is available starting from pyroscope version 0.19.0.
Make sure to upgrade before you use this.
#What is Remote Write?
Remote Write is a feature that allows writing data to a remote Pyroscope instance (ie. not the same one that is immediately ingesting the data).
#How does it work?
Its current implementation is simple: for all ingested data, the same data is sent to the remote targets'
/ingest endpoint, concurrently.
We currently do not provide any guarantees this data is going to be delivered (think for example, network failures). Although in future versions we will implement retries.
remote-write: enabled: true targets: bar: address: http://bar:4040 auth-token: MY_AUTH_TOKEN tags: foo: bar baz: address: http://baz:4040 auth-token: MY_AUTH_TOKEN tags: foo: baz headers: X-Extra-Header: "Value"
For every ingested profile, it will also write to 2 pyroscope instances:
Tags can (and should) be added, so that you can identify where the profiling data is coming from.
#What are the use cases?
There are mainly 2 use cases:
#Case 1: Aggregating Pyroscope data from multiple remote targets
Consider the following situation in an organization:
- Multiple Kubernetes clusters are running pull mode, ie. Pyroscope scrapes apps (ie. Pyroscope scrapes apps)
- A central pyroscope instance stores data in one place which aggregates profiling data from all of these apps
remote-write option enables collecting data on the centralized Pyroscope instance and only exposing this one instance while allowing the various internal Pyroscope instances remain private.
#Case 2: Migrating to Pyroscope Cloud
For users who are running Pyroscope's Open Source version and would like to migrate to Pyroscope's Cloud version, before
remote-write it required:
- Updating applications to point to Pyroscope Cloud
- Replacing API Keys
- Redeploying applications
With remote write, migrating to the cloud is simple as adding the Pyroscope Cloud instance as a target (and as you transition your open source data will be unaffected)! For example:
remote-write: enabled: true disable-local-writes: true targets: pyroscope-cloud: address: https://ingest.pyroscope.cloud auth-token: MY_AUTH_TOKEN tags: # optional tag to distinguish between different clusters you have cluster-name: my-cluster
#Disabling local writes
disable-local-writes configuration allows disabling writes to the local storage.
We recommend using this option if you only want to relay data to another pyroscope instance and don't need to write it locally.
Keep in mind, in some situations it may be useful to leave local writes enabled, so that if a remote target is having issues, it's still possible to access the ingested data locally.
remote-write: enabled: true disable-local-writes: true
We recommend using the following defaults, however, not every environment is the same. If you have trouble configuring these values please, create an issue, reach out to email@example.com or join our slack channel and we'd love to help.
That being said, Pyroscope has a few key configurable options:
remote-write: enabled: true targets: bar: address: http://bar:4040 auth-token: MY_AUTH_TOKEN tags: foo: bar # tweakable timeout: 1m queue-size: 100 queue-workers: 4
|Maximum amount of time each upload can take. It uses the standard Go time duration format.|
|Maximum amount of items that can be stored in the queue at a time.|
|How many queue workers are running.|
#Sending profiles to Phlare via remote-write
Starting with weekly-f8 you can ingest pyroscope profiles directly to phlare. You can do remote write from pyroscope to phlare as well.
remote-write: enabled: true targets: phlare: address: "<URL>" basic-auth-user: "<User>" basic-auth-password: "<Password>" tenant-id: "<TenantID>"
To configure .NET integration to send data to Phlare, replace the
<URL> placeholder with the appropriate server URL. This could be the grafana.com Phlare URL or your own custom Phlare server URL.
If you need to send data to grafana.com, you'll have to configure HTTP Basic authentication. Replace
<User> with your grafana.com stack user and
<Password> with your grafana.com API key.
If your Phlare server has multi-tenancy enabled, you'll need to configure a tenant ID. Replace
<TenantID> with your Phlare tenant ID.
Prometheus metrics are also exposed:
|counter||The total number of bytes of data (not metadata) sent to the remote target.|
|gauge||How many items the queue can hold.|
|counter||How many items were dropped (as in not accepted into the queue).|
|gauge||Total number of queue workers.|
Also notice that the key in the yaml is used as
targetName in the metric. For example, given the following configuration:
remote-write: enabled: true targets: my-cluster: address: http://bar:4040
It will create metrics with the following labels (
- Enabling remote write will increase bandwith usage, since every ingestion is replicated to the number of targets.