Self-Hosted eBPF Continuous Profiling with Parca and Pyroscope: A Practical Guide

Quick Summary (TL;DR)

What: Deploy self-hosted, eBPF-based continuous profiling to find CPU/memory hotspots in production — without code changes.

Tools: Parca (cloud-native, Prometheus-style querying, object-storage backend) vs. Pyroscope (simpler UX, built-in flame graphs, supports eBPF + language agents).

How: Run an eBPF agent on each node, ship profiles to a central server, visualize flame graphs in Grafana.

Overhead: Typically 1–5% CPU at ~19Hz sampling; eBPF programs are kernel-verified and safe for production.

Integrations: Both correlate with Prometheus metrics and render in Grafana.

Continuous profiling is the missing pillar of observability. Metrics tell you that CPU spiked; traces tell you which request was slow; profiles tell you exactly which function burned the cycles. This guide walks you through deploying both Parca and Pyroscope on Linux/Kubernetes using eBPF — no application instrumentation, no redeploys, no language-specific SDKs.

Introduction to Continuous Profiling with eBPF

Traditional profiling is a debugging activity: you attach a profiler locally, reproduce the problem, then remove it. Continuous profiling flips this — you sample stack traces from every process on every host, all the time, at low frequency. The result is a time-series database of “who consumed CPU/memory and where,” queryable retroactively. When an incident happens at 3 a.m., you already have the data.

eBPF (extended Berkeley Packet Filter) is the enabler. Instead of injecting instrumentation into your app, an eBPF program attaches to kernel-level events — CPU timer interrupts, memory allocations, syscalls — and captures user-space stack traces. The kernel’s verifier guarantees the program is safe before it ever runs.

flowchart LR
    A[Application<br/>any language] -->|runs on| B[Linux Kernel]
    B -->|perf/timer events| C[eBPF Agent<br/>Parca / Pyroscope]
    C -->|stack samples| D[Central Server]
    D -->|profile storage| E[(Object Store /<br/>Local Disk)]
    D -->|query API| F[Grafana]
    G[Prometheus] -->|metrics| F
    F --> H[Engineer<br/>flame graphs]

Why eBPF? Benefits Over Traditional Profiling

Dimension	Traditional profilers	eBPF continuous profiling
Code changes	Instrumentation/SDK required	None — kernel-level capture
Language support	Per-language tooling	Any runtime (Go, Java, Rust, C, Python…)
Overhead	Often 5–20%+ when active	1–5%, tunable via sampling rate
Coverage	One process, one session	All processes, all hosts, always-on
Production safety	Risky to attach	Kernel-verified, cannot crash the system
Historical analysis	None (ad-hoc)	Full time-series of profiles

The trade-off: eBPF captures native stack traces, so managed runtimes (JVM, Python) may show less expressive frame names unless you add symbolization or language agents. That’s a key reason Pyroscope offers both eBPF and language-specific agents.

Parca vs. Pyroscope: Key Differences and Comparison

Both are CNCF-adjacent, open-source, and Grafana-friendly. They diverge in storage philosophy and operational complexity.

Feature	Parca	Pyroscope
Primary capture	eBPF agent (Parca Agent)	eBPF agent + per-language agents
Query language	PromQL-like (`profile_type:cpu`)	Label-based UI queries
Storage backend	Object storage (S3/GCS/Azure) + local	Custom TSDB (blocks on disk)
Scalability model	Stateless server + object store (very scalable)	Monolithic or microservices mode
UI	Basic; Grafana-first	Rich built-in flame graph UI
Best for	Cloud-native, large fleets, cost-efficient retention	Fast onboarding, deep UI, mixed eBPF+SDK
Learning curve	Moderate (PromQL familiarity helps)	Low

Rule of thumb: choose Parca if you already think in PromQL and want cheap, infinitely-retained profiles in object storage. Choose Pyroscope if you want the fastest path to a beautiful flame graph UI and plan to mix eBPF with language agents.

Prerequisites and Environment Setup

Linux kernel ≥ 4.9 (5.x+ strongly recommended for BTF/CO-RE support).
Root or CAP_SYS_ADMIN + CAP_BPF on the agent (it attaches kernel programs).
BTF enabled (/sys/kernel/btf/vmlinux present) — check with ls /sys/kernel/btf/vmlinux.
Docker or Kubernetes (Helm 3 for K8s deployments).
Grafana ≥ 10 for visualization, Prometheus optional for correlation.

Verify your kernel supports eBPF:

uname -r                                   # expect 5.x+
ls /sys/kernel/btf/vmlinux                  # BTF present?
grep CONFIG_DEBUG_INFO_BTF /boot/config-$(uname -r)

Deploying Parca: Step-by-Step Configuration

Parca runs as two components: the agent (DaemonSet, one per node) and the server (Deployment, scrapes agents).

Step 1 — Install via Helm:

helm repo add parca https://parca-dev.github.io/helm-charts
helm repo update
helm install parca parca/parca \
  --namespace parca --create-namespace \
  --set server.persistentVolume.storageClassName=standard \
  --set server.persistentVolume.size=50Gi

Step 2 — Configure the agent (parca-agent values). The agent scrapes local processes and exposes them for the server:

# parca-agent-values.yaml
agent:
  scrapeInterval: "10s"
  profilingInterval: "19s"     # sampling frequency; lower = less overhead
  node: "$(NODE_NAME)"
  remoteStoreAddress: ""       # empty = server scrapes agent
  # Optional: filter to specific namespaces
  # profilingLabels: { namespace: "production" }

Step 3 — Point the server at object storage for scalable retention:

# parca-server-values.yaml
server:
  config:
    object_storage:
      bucket:
        type: "S3"
        config:
          bucket: "my-parca-profiles"
          region: "us-east-1"
          endpoint: "s3.amazonaws.com"
    scrape_configs:
      - job_name: "parca"
        static_configs:
          - targets: ["parca-agent.parca.svc:7071"]

Step 4 — Verify the server is ingesting:

kubectl port-forward -n parca svc/parca 7070:7070
# open http://localhost:7070, query: cpu

Parca’s query syntax mirrors PromQL: cpu returns all CPU profiles; cpu{namespace="production"} narrows by label; sum by (pod) (cpu) aggregates.

Deploying Pyroscope: Step-by-Step Configuration

Pyroscope is a single binary/server that accepts profiles from an eBPF agent or language SDKs.

Step 1 — Run the server with Docker:

docker run -d --name pyroscope \
  -p 4040:4040 \
  -v pyroscope-data:/var/lib/pyroscope \
  grafana/pyroscope:latest

Step 2 — Deploy the eBPF agent on each host:

docker run -d --name pyroscope-ebpf \
  --privileged \
  --pid=host \
  -v /sys/kernel/debug:/sys/kernel/debug:ro \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  grafana/pyroscope-ebpf:latest \
  --server-address http://pyroscope-server:4040

Step 3 — (Optional) Add a language agent for richer symbols. Example for a Go service:

import "github.com/grafana/pyroscope-go"

pyroscope.Start(pyroscope.Config{
    ApplicationName: "checkout-service",
    ServerAddress:   "http://pyroscope-server:4040",
    ProfileTypes: []pyroscope.ProfileType{
        pyroscope.ProfileCPU, pyroscope.ProfileAllocObjects,
        pyroscope.ProfileInuseSpace,
    },
})

Step 4 — Explore at http://localhost:4040. Select an application, pick CPU or memory, and a flame graph renders instantly.

Tip: Run eBPF for broad host coverage and language agents for deep runtime detail — they complement each other.

Analyzing Flame Graphs for CPU and Memory Hotspots

A flame graph stacks call frames bottom-up; width = resource consumed (CPU time or allocated bytes). Reading it correctly is the whole point:

flowchart TD
    A[Open flame graph] --> B{Wide plateau<br/>at top?}
    B -->|Yes| C[Hotspot: function<br/>consumes most time/mem]
    B -->|No| D[Look for many<br/>thin tall towers]
    D --> E[Deep recursion or<br/>per-call overhead]
    C --> F[Compare to baseline<br/>via diff view]
    E --> F
    F --> G[Correlate with<br/>Prometheus metrics]
    G --> H[Fix & re-profile]

Practical workflow:

Find the widest self-time frame — that’s your CPU sink (e.g., json.Marshal, regex compilation, lock contention).
Use the diff view — compare a slow hour against a baseline; newly-widened frames are regressions.
Search/highlight a suspected function to see its aggregate cost across all stacks.
For memory, switch to inuse_space / alloc_space profiles to spot leak-prone allocations vs. churn.

A frame like runtime.mallocgc being wide often points to allocation pressure — correlate with go_memstats_alloc_bytes or JVM heap metrics in Prometheus to confirm.

Integrating with Existing Observability Stacks

Grafana datasources: Install the Parca or Pyroscope datasource plugin, then use the Flame Graph panel. In Grafana 11+, Grafana Alloy can scrape both, unifying your pipeline.

Metrics correlation (the killer feature): Link a metric spike to its profile. In Grafana, add a data link on a Prometheus panel:

https://grafana.example.com/d/profiles?var-filters=pod_name%3D${pod}

Now clicking a CPU spike in your metrics dashboard jumps straight to the flame graph for that pod at that timestamp. This collapses the “metrics → logs → traces → profiles” hunt into one click.

Prometheus: Both servers expose /metrics for self-monitoring (ingestion rate, agent health). Scrape them so you can alert on profiling pipeline failures.

Best Practices and Production Considerations

Concern	Recommendation
Sampling rate	Start at 19Hz; raise to 99Hz only for targeted debugging.
Overhead budget	Monitor agent CPU; keep total profiling overhead <5%.
Retention	Parca → object storage lifecycle rules (e.g., 30d standard, 1y cold). Pyroscope → disk sizing per block retention.
Security	Agents need `CAP_SYS_ADMIN`/`CAP_BPF`; run in dedicated namespace, restrict RBAC.
Symbolization	Ensure debug symbols / BTF are available; stripped binaries yield `[unknown]` frames.
Cardinality	Limit label dimensions (avoid per-request labels) to prevent storage blowup.
Rollout	Canary on one node → validate overhead → expand to fleet.
Alerting	Alert on agent down and ingestion gaps — silent profilers are worse than none.

Production hardening checklist:

✅ Test in staging with production-like load.
✅ Pin agent versions; eBPF programs vary by kernel.
✅ Set resource requests/limits on servers.
✅ Back up object storage credentials securely (Parca needs write access).
✅ Document the “metrics → profile” jump link for on-call.

Conclusion

Continuous profiling with eBPF closes the last gap in observability: it tells you why a system is slow, not just that it is. Parca is the choice for cloud-native fleets that want PromQL-style querying and cheap object-storage retention; Pyroscope wins on time-to-value with a polished flame-graph UI and flexible eBPF+SDK capture. Start with a single node, a 19Hz sampling rate, and a Grafana flame-graph panel — then wire metric-to-profile jump links so your next incident is diagnosed in clicks, not hours. The overhead is negligible, the code changes are zero, and the payoff is a permanent, queryable record of every CPU cycle your fleet has ever spent.

FAQ

What is continuous profiling and why use eBPF? Continuous profiling collects performance data from production systems continuously with low overhead. eBPF lets you profile without code instrumentation by attaching to kernel events, giving deep CPU/memory insight with minimal performance impact.

Do I need to modify my application code to use Parca or Pyroscope? No. Both use eBPF to collect profiles at the system level, so no code changes or instrumentation are required. They can profile any application on the host, including languages without native profiling support.

What are the system requirements for running Parca or Pyroscope? Linux kernel 4.9+ (5.x recommended for better eBPF/BTF support), root or CAP_SYS_ADMIN/CAP_BPF for the agent, and sufficient CPU/memory. Parca is best on Kubernetes with object storage for production; Pyroscope runs happily on a single node or in Kubernetes.

How does Parca differ from Pyroscope? Parca focuses on eBPF profiling with a Prometheus-like query language and object-storage backend for scalability. Pyroscope supports eBPF and language agents, offers a simpler UI, and stores data in a custom format with built-in flame graphs. Parca is more cloud-native and scalable; Pyroscope is easier to start with.

Is it safe to run eBPF profilers in production? Yes. eBPF programs are verified by the kernel for safety and cannot crash the system. Test in staging first, monitor resource usage, ensure proper permissions, and tune the sampling rate — both tools are designed for production with rate limiting and sampling safeguards.