Self-Hosted Distributed Tracing with OpenTelemetry Collector, Tempo, and Tail Sampling

Quick Summary (TL;DR)

Build a vendor-neutral tracing pipeline: OpenTelemetry SDKs → OTel Collector (with tail sampling) → Grafana Tempo → Grafana.

Tail sampling keeps all error/slow traces while dropping routine ones—often cutting storage costs by 80–95%.

Use a load-balancing exporter in front of tail-sampling Collectors so every span of a trace lands on the same instance.

Store traces in object storage (S3/MinIO) via Tempo for cheap, durable retention.

Correlate traces ↔ logs by injecting trace_id/span_id into your log lines and configuring Grafana derived fields.

Distributed tracing is the difference between “the checkout endpoint is slow” and “the checkout endpoint is slow because the payment service’s gRPC call to the fraud API retries three times.” But commercial tracing vendors charge by the span, and at scale those bills get ugly fast. Self-hosting an open-source stack gives you full control over cost, data residency, and retention—if you architect it correctly.

This guide walks you through a production-ready pipeline: OpenTelemetry SDKs for instrumentation, the OpenTelemetry Collector with tail-based sampling for intelligent filtering, and Grafana Tempo for cost-effective storage. Let’s build it.

Why Self-Hosted Tracing? Benefits and Trade-offs

Before committing, understand what you’re signing up for.

Factor	Self-Hosted (OTel + Tempo)	Managed Vendor
Cost model	Infrastructure + object storage (predictable, low)	Per-span / per-GB ingestion (scales with volume)
Data ownership	Full—stays in your VPC	Vendor-controlled
Vendor lock-in	None (OTel is a standard)	Proprietary agents/protocols
Operational burden	You run & scale it	Vendor handles it
Feature depth	Solid; Grafana-native	Often richer UI/APM features
Sampling control	Complete (tail sampling)	Usually limited

The trade-off is clear: you trade operational effort for cost control and flexibility. For teams with high trace volumes and existing Kubernetes/Grafana expertise, self-hosting usually wins on economics within months.

Architecture Overview: OpenTelemetry, Collector, and Tempo

Key Components and Data Flow

flowchart LR
    subgraph Apps["Instrumented Services"]
        A1[Service A<br/>OTel SDK]
        A2[Service B<br/>OTel SDK]
        A3[Service C<br/>OTel SDK]
    end

    subgraph Collectors["OpenTelemetry Collectors"]
        LB["LB Collector<br/>(load-balancing exporter)"]
        TS1["Tail-Sampling<br/>Collector 1"]
        TS2["Tail-Sampling<br/>Collector 2"]
    end

    Tempo[(Grafana Tempo)]
    S3[(Object Storage<br/>S3 / MinIO)]
    Grafana[Grafana UI]

    A1 & A2 & A3 -->|OTLP/gRPC| LB
    LB -->|trace-ID hash| TS1
    LB -->|trace-ID hash| TS2
    TS1 & TS2 -->|OTLP| Tempo
    Tempo --> S3
    Grafana -->|TraceQL| Tempo
    Grafana -.->|derived fields| Logs[(Loki)]

The flow: services emit OTLP spans → a load-balancing Collector routes whole traces to one tail-sampling Collector → sampled traces are exported to Tempo → Tempo writes blocks to object storage → Grafana queries Tempo and links to Loki logs.

Instrumenting Services with OpenTelemetry SDKs

Auto-Instrumentation vs. Manual Instrumentation

Approach	Effort	Coverage	Best For
Auto-instrumentation	Minimal (agent/init container)	HTTP, DB, gRPC out of the box	Fast rollout, standard frameworks
Manual	Higher (code changes)	Custom spans & attributes	Business logic, internal libraries
Hybrid	Medium	Best of both	Most production systems

Start with auto-instrumentation, then add manual spans where business context matters (e.g., order.validate, payment.charge).

Example: Instrumenting a Node.js Service

// tracing.js — loaded before app code via `node --require ./tracing.js`
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { Resource } = require('@opentelemetry/resources');
const { SEMRESATTRS_SERVICE_NAME } = require('@opentelemetry/semantic-conventions');

const sdk = new NodeSDK({
  resource: new Resource({ [SEMRESATTRS_SERVICE_NAME]: 'checkout-service' }),
  traceExporter: new OTLPTraceExporter({ url: 'http://otel-collector:4317' }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();
process.on('SIGTERM', () => sdk.shutdown().then(() => process.exit(0)));

For custom spans, wrap business logic:

const { trace, SpanStatusCode } = require('@opentelemetry/api');
const tracer = trace.getTracer('checkout');

async function charge(order) {
  return tracer.startActiveSpan('payment.charge', async (span) => {
    try {
      span.setAttribute('order.id', order.id);
      span.setAttribute('payment.amount', order.total);
      const result = await gateway.charge(order);
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

The SDK automatically propagates W3C traceparent headers across HTTP/gRPC calls, so context flows end-to-end without manual plumbing.

Deploying the OpenTelemetry Collector

Collector Configuration: Receivers, Processors, Exporters

The Collector is the heart of the pipeline. A minimal config:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  batch:
    send_batch_size: 8192
    timeout: 5s

exporters:
  otlp:
    endpoint: tempo:4317
    tls: { insecure: true }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

Batching for Efficiency

The batch processor groups spans before export, dramatically reducing network round-trips and backend write pressure. 8192 spans / 5s timeout is a solid default. Always place memory_limiter first in the pipeline to shed load gracefully under pressure rather than OOM-ing.

Implementing Tail-Based Sampling to Control Cost

How Tail Sampling Works

Unlike head sampling (decided at trace start, blindly), tail sampling waits until all spans of a trace arrive, then decides based on the complete picture—errors, latency, attributes. This means you can keep 100% of problem traces while discarding the boring successful ones.

flowchart TD
    S[Spans arrive] --> W[Buffer trace by trace ID]
    W --> C{Trace complete<br/>or timeout?}
    C -->|No| W
    C -->|Yes| P{Evaluate policies}
    P -->|Error?| K[Keep]
    P -->|Latency > 2s?| K
    P -->|VIP customer?| K
    P -->|Routine success| D[Drop]
    K --> E[Export to Tempo]

Configuring Tail Sampling Policies

processors:
  tail_sampling:
    decision_wait: 10s          # wait for spans to arrive
    num_traces: 100000          # in-memory trace buffer
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-requests
        type: latency
        latency: { threshold_ms: 2000 }
      - name: vip-customers
        type: string_attribute
        string_attribute:
          key: customer.tier
          values: [platinum, gold]
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Policies are evaluated in order; the first match wins. The final probabilistic policy acts as a catch-all so you retain a representative sample of healthy traffic.

Scaling Tail Sampling with Load Balancing

Tail sampling requires all spans of a trace on the same Collector. If spans scatter across instances, decisions become unreliable. Solve this with the load-balancing exporter on a front tier, which hashes by trace ID:

# Front-tier LB collector
exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls: { insecure: true }
    resolver:
      static:
        hostnames:
          - tail-sampling-1:4317
          - tail-sampling-2:4317

Now scale tail-sampling Collectors horizontally—each sees a consistent subset of traces.

Storing Traces in Grafana Tempo

Tempo Deployment Options

Mode	Scale	Notes
Monolithic	Dev / small	Single binary, simplest
Microservices	Large	Separate ingester/querier/compactor
Single-binary + S3	Medium	Great starting point for production

Configuring Tempo for Scalability and Retention

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc: { endpoint: 0.0.0.0:4317 }

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: minio:9000
      access_key: ${MINIO_ACCESS_KEY}
      secret_key: ${MINIO_SECRET_KEY}
      insecure: true
    blocklist_poll: 5m
    wal:
      path: /var/tempo/wal

compactor:
  compaction:
    block_retention: 336h   # 14 days

Tempo writes immutable blocks to object storage and keeps only an index in memory/disk—this is why it’s dramatically cheaper than span-indexed stores. Tune block_retention to your compliance and cost needs.

Correlating Traces with Logs

Injecting Trace Context into Logs

The magic of correlation is having trace_id in your log lines. With OTel, the active span context is available:

const { trace } = require('@opentelemetry/api');
const logger = require('pino')();

function logWithTrace(level, msg, extra = {}) {
  const span = trace.getActiveSpan();
  const ctx = span?.spanContext();
  logger[level]({ ...extra, trace_id: ctx?.traceId, span_id: ctx?.spanId }, msg);
}

Ship these logs to Loki (or any log store) with trace_id as a structured field.

Configuring Grafana for Trace-to-Log Correlation

In Grafana’s Tempo datasource, add a Trace to logs setting:

# Grafana datasource provisioning (excerpt)
datasource:
  type: tempo
  jsonData:
    tracesToLogsV2:
      datasourceUid: loki-uid
      spanStartTimeShift: '-1m'
      spanEndTimeShift: '1m'
      filterByTraceID: true
      tags: [{ key: 'service.name', value: 'service_name' }]

Now clicking a span in Grafana jumps straight to matching Loki logs. The reverse (logs → trace) works via a derived field on trace_id.

Comparison: Tail Sampling vs. Head Sampling

Aspect	Head Sampling	Tail Sampling
Decision point	At trace start	After trace completes
Can keep all errors?	No (blind)	Yes
Resource cost	Low (drop early)	Higher (buffer full traces)
Scaling	Trivial	Needs trace-ID routing
Latency-aware?	No	Yes
Best for	Uniform, high-volume traffic	Cost-sensitive, error-focused SLOs

Recommendation: Use tail sampling at the Collector tier for intelligent retention, and optionally a light head-sampling rate at the SDK to reduce Collector ingress in extreme volumes.

Operational Best Practices

Monitoring the Pipeline

Enable the Collector’s internal telemetry and watch these signals:

otelcol_receiver_refused_spans — backpressure indicator
otelcol_exporter_send_failed_spans — downstream (Tempo) issues
otelcol_processor_tail_sampling_count_traces_sampled — policy effectiveness
Tempo tempo_ingester_traces_created_total and query latency

Alert on sustained export failures and on decision_wait timeouts (a sign your buffer is too small).

Security Considerations

TLS everywhere: enable mTLS between Collectors and Tempo; never run insecure: true in production.
Auth on OTLP: front the Collector with an auth proxy or use the bearertokenauth extension.
Network isolation: keep Tempo’s query port internal; expose only Grafana.
PII scrubbing: use the attributes/transform processors to drop sensitive attributes before export.

Conclusion

You now have a complete, vendor-neutral tracing pipeline: instrumented services emitting OTLP, a load-balanced Collector tier applying intelligent tail sampling, and Tempo storing traces cheaply in object storage—all visualized and correlated in Grafana. The economics are compelling: by keeping every error and slow trace while sampling routine traffic, you get better visibility at a fraction of managed-vendor cost. Start monolithic, monitor your pipeline metrics, and scale the tail-sampling tier horizontally as volume grows.

FAQ

What is tail-based sampling and why is it important for cost control? Tail-based sampling makes decisions after a trace is complete, letting you keep traces based on errors, latency, or attributes. You retain all interesting traces while sampling out routine successful ones—reducing storage and processing costs without losing visibility into issues.

Can I use the OpenTelemetry Collector with backends other than Tempo? Yes. The Collector is vendor-neutral and ships exporters for Jaeger, Zipkin, Prometheus, and commercial services. You can even export to multiple backends simultaneously for migration or redundancy.

How does Grafana Tempo compare to Jaeger for self-hosted tracing? Tempo emphasizes scalability and cost-efficiency, using object storage (S3/MinIO) for trace data and integrating tightly with Grafana and trace-to-log correlation. Jaeger offers richer built-in trace analysis but is more resource-intensive at scale. Tempo is often preferred for high-volume cloud-native environments.

What are the prerequisites for implementing this pipeline? A Kubernetes cluster (or Docker environment), basic OpenTelemetry knowledge, services instrumented with OTel SDKs, object storage (MinIO or S3) for Tempo, and Grafana for visualization.

How do I ensure trace context propagates across services? Use the SDKs’ built-in W3C Trace Context propagation and ensure the traceparent header flows through HTTP/gRPC calls (auto-instrumentation handles this). For messaging systems, manually inject and extract context at produce/consume boundaries.