Self-Hosted Durable Workflow Orchestration with Temporal

Quick Summary (TL;DR)

• Temporal gives you durable execution: workflow state survives crashes, restarts, and multi-day waits because it is rebuilt from an append-only event history.
• Model long-running business processes as sagas—each forward step paired with a compensating action that runs in reverse order on failure.
• Temporal guarantees at-least-once activity execution, not exactly-once. You must build idempotency into every external side effect.
• Derive stable idempotency keys from workflow identity (never randomness), and wrap external writes in an outbox or transactional boundary.
• Self-hosting requires a multi-node topology (frontend, history, matching, worker) plus a highly available database. Harden it before production.

Why Durable Execution Beats Hand-Rolled State Machines

Every team eventually writes a status column and a cron job to advance orders through PENDING → PAID → SHIPPED. It works until it doesn’t: a pod restarts mid-transition, a webhook arrives twice, and nobody knows whether the refund fired. Hand-rolled state machines encode what the state is but not how to safely resume.

Temporal flips the model. You write ordinary code—loops, conditionals, function calls—and the runtime persists every decision as an event. On restart, the workflow replays its history to reconstruct in-memory state, then continues from exactly where it stopped. You get retries, timeouts, and durability as platform primitives rather than bespoke glue.

The trade-off is discipline: workflow code must be deterministic, and every external call must be idempotent. This guide shows how to self-host the cluster and build those guarantees.

Self-Hosting Temporal: A Minimal Production Topology

flowchart TD
    W[Worker Fleet<br/>your workflow + activity code] -->|gRPC| F[Frontend<br/>x3, stateless]
    F --> H[History<br/>x3, sharded state]
    F --> M[Matching<br/>x3, task queues]
    H --> DB[(Persistence<br/>PostgreSQL / Cassandra)]
    M --> DB
    H --> V[(Visibility<br/>Elasticsearch)]
    UI[Web UI] --> F
    CLI[tctl] --> F

Four server roles matter: frontend (API gateway), history (owns workflow state and shards), matching (task queues), and worker (system background work). Your own application workers poll the frontend for tasks.

Docker Compose Baseline for Local Development

Start with the official dev server for iteration, then run the full stack via temporalio/docker-compose. A trimmed compose for local work:

services:
  temporal:
    image: temporalio/auto-setup:1.24
    environment:
      - DB=postgres12
      - DB_PORT=5432
      - POSTGRES_USER=temporal
      - POSTGRES_PWD=temporal
      - POSTGRES_SEEDS=postgresql
    ports: ["7233:7233"]
    depends_on: [postgresql]

  temporal-ui:
    image: temporalio/ui:2.27.0
    environment:
      - TEMPORAL_ADDRESS=temporal:7233
    ports: ["8080:8080"]

  postgresql:
    image: postgres:16
    environment:
      POSTGRES_USER: temporal
      POSTGRES_PASSWORD: temporal
    volumes: ["pgdata:/var/lib/postgresql/data"]

volumes: { pgdata: {} }

Hardening the Self-Hosted Cluster

• Separate persistence from the cluster. Use managed PostgreSQL with synchronous replication or a Cassandra ring; never co-locate the database on the same nodes.
• Run three instances of each server role across availability zones and front them with a load balancer.
• Enable mTLS between frontend and history/matching, and between workers and frontend.
• Size history shards deliberately (default 512); you cannot easily change this later.
• Back up the visibility store (Elasticsearch) and the primary database on independent schedules.

Workflow Design Fundamentals: Determinism and Versioning

Workflow code is replayed from history, so it must produce identical decisions given identical inputs. That rules out time.Now(), math/rand, direct network calls, and map iteration order. Use workflow.Now(), workflow.SideEffect, and activities for anything non-deterministic.

When you change workflow logic, version it so in-flight executions replay against their original history:

v := workflow.GetVersion(ctx, "add-fraud-check", workflow.DefaultVersion, 1)
if v == 1 {
    err = workflow.ExecuteActivity(ctx, CheckFraud, order).Get(ctx, nil)
}

Modeling Sagas: Compensating Transactions That Actually Roll Back

A saga is a sequence of local transactions, each with a compensating action. If step N fails, run compensations for steps N-1 … 1 in reverse. This trades atomicity for availability—the standard pattern for multi-service business processes.

A Payment and Fulfillment Saga in Go

func OrderSaga(ctx workflow.Context, order Order) error {
    ao := workflow.ActivityOptions{
        StartToCloseTimeout: 10 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{MaximumAttempts: 5},
    }
    ctx = workflow.WithActivityOptions(ctx, ao)

    var compensations []func(workflow.Context) error
    compensate := func() {
        for i := len(compensations) - 1; i >= 0; i-- {
            _ = compensations[i](ctx) // best-effort, idempotent
        }
    }

    if err := workflow.ExecuteActivity(ctx, ReserveInventory, order).Get(ctx, nil); err != nil {
        return err
    }
    compensations = append(compensations, func(c workflow.Context) error {
        return workflow.ExecuteActivity(c, ReleaseInventory, order).Get(c, nil)
    })

    if err := workflow.ExecuteActivity(ctx, ChargePayment, order).Get(ctx, nil); err != nil {
        compensate()
        return err
    }
    compensations = append(compensations, func(c workflow.Context) error {
        return workflow.ExecuteActivity(c, RefundPayment, order).Get(c, nil)
    })

    if err := workflow.ExecuteActivity(ctx, ShipOrder, order).Get(ctx, nil); err != nil {
        compensate()
        return err
    }
    return nil
}

Compensation Semantics and Failure Modes

Compensations must themselves be idempotent and retried with a bounded policy. If a compensation fails permanently, never swallow it—record it in workflow state, emit a metric, and route to a manual-intervention state. Some actions are irreversible (an email sent, a package shipped), so treat those as best-effort with explicit business acknowledgment.

Idempotency: The Contract Between Workflow and External System

Deriving Stable Idempotency Keys

Derive keys deterministically from workflow identity and business intent—never from randomness or timestamps:

func IdempotencyKey(ctx workflow.Context, step, entityID string) string {
    info := workflow.GetInfo(ctx)
    raw := info.WorkflowExecution.ID + ":" + step + ":" + entityID
    return fmt.Sprintf("%x", sha256.Sum256([]byte(raw)))
}

Because the key is stable across retries, the downstream service can deduplicate. Generating the key inside the activity with uuid.New() defeats the purpose—retries would produce new keys.

Idempotency at the Activity Boundary

Pass the key as an Idempotency-Key header. On the server side, store (key → response) in a unique-constrained table; a duplicate request returns the stored response instead of re-executing.

func ChargePayment(ctx context.Context, order Order) error {
    key := order.IdempotencyKey // computed in workflow, stored on the DTO
    return gateway.Charge(ctx, ChargeRequest{
        IdempotencyKey: key,
        Amount:         order.Total,
    })
}

Exactly-Once Side Effects: What Temporal Guarantees and What You Must Build

Temporal guarantees the workflow advances exactly once in terms of its history. Activities, however, are at-least-once: a timeout may trigger a retry even if the first attempt actually succeeded. Exactly-once external effects are your responsibility.

The Outbox Pattern Inside an Activity

Write the side effect and a record of it in the same local transaction, then publish asynchronously:

sequenceDiagram
    participant W as Workflow
    participant A as Activity
    participant DB as Local DB
    participant Q as Message Broker
    W->>A: ExecuteActivity(Charge)
    A->>DB: INSERT charge + outbox row (one tx)
    A-->>W: success
    DB->>Q: relay publishes outbox row
    Q->>Consumer: deliver (idempotent by key)

If the activity crashes before commit, nothing happened. If it crashes after commit, the retry hits the unique constraint and returns the prior result. Either way, the external effect happens once.

Reconciling Ambiguous Outcomes

When an activity times out, you don’t know if the downstream call succeeded. Build a reconciliation activity that queries the external system by idempotency key and resolves the true state before retrying. Schedule it as a separate step rather than relying on blind retries.

Comparison: Orchestration Approaches for Long-Running Processes

Approach	Durability	Retries / Timeouts	Saga Support	Operational Cost	Best For
Hand-rolled state machine + cron	Manual	Custom code	Manual	Low initially, high over time	Tiny, single-service flows
Queue + worker (SQS/RabbitMQ)	Message-level	Basic	None built-in	Medium	Short, fire-and-forget tasks
Temporal (self-hosted)	Full event history	First-class	Native via code	Medium–high (cluster ops)	Long-running, multi-step business flows
Temporal Cloud	Full	First-class	Native	Low ops, usage-based cost	Teams avoiding cluster ops
2PC / distributed tx	Strong atomicity	N/A	N/A	High, brittle	Short, homogeneous resources

Operational Checklist Before Going Live

• Workflow code passes determinism review (no time.Now, no random, no direct I/O).
• Every external call carries a stable idempotency key.
• Compensations are idempotent, reversed, and bounded-retry.
• Permanent compensation failures alert and route to manual review.
• Cluster runs ≥3 instances per role across AZs with mTLS.
• Database and visibility store backed up independently.
• Saga tests force failure at each step and assert compensation order.
• Dashboards track activity retries, saga rollbacks, and stuck workflows.

FAQ

Does Temporal give me exactly-once execution of activities? No. Temporal guarantees at-least-once activity execution: an activity may be retried after a timeout even if the first attempt succeeded. Exactly-once side effects come from making each activity idempotent or wrapping the external call in an outbox. The workflow itself advances exactly once in terms of its history, but external effects still need idempotency keys.

How do I choose an idempotency key for an external API call? Derive it deterministically from workflow ID, run ID, activity type, and a logical step name—for example sha256(workflowID + ':' + step + ':' + entityID). Pass it as an Idempotency-Key header. Because it is stable across retries, the downstream service deduplicates. Never generate the key inside the activity with randomness.

What is the difference between a saga and a distributed transaction? A distributed transaction (two-phase commit) provides atomicity but requires locking and coordinated participants—impractical across heterogeneous services and long durations. A saga is a sequence of local transactions with compensating actions; on failure, completed steps are undone by compensations. Sagas trade atomicity for availability.

Should compensations run in reverse order, and what if one fails? Yes—reverse order of forward steps. Compensations must be idempotent and retried with a bounded policy. If one fails permanently, record it in workflow state, alert, and route to manual intervention. Some compensations are irreversible (an email already sent), so design them as best-effort with explicit business acknowledgment.

How do I handle non-deterministic code in workflows? Avoid random numbers, wall-clock time, direct I/O, and map iteration order in workflow code. Use workflow.Now(), workflow.SideEffect, and activities for non-determinism. When changing logic, use versioning APIs (workflow.GetVersion in Go, patched in TypeScript) so in-flight executions replay correctly against their original history.