Temporal

Monitor your Temporal workflow orchestration platform with Last9. This integration covers metrics from both the Temporal Server (self-hosted) and Temporal SDKs, so you can see how your durable workflow executions behave.

Temporal is a durable execution platform that helps developers build scalable, reliable applications. Monitor Temporal to measure workflow performance, to find bottlenecks, and to confirm reliable task processing.

Last9 supports two methods for collecting Temporal telemetry:

• Prometheus Remote Write: Scrape metrics from Temporal Server and SDKs, then forward to Last9
• OpenTelemetry: Send traces and metrics directly to Last9 via OTLP

Prerequisites

Before setting up Temporal monitoring, ensure you have:

• Temporal Cluster: Running Temporal Server (self-hosted) or Temporal Cloud account
• Last9 Account: With integration credentials from Getting Started
• Prometheus (for metrics): Running instance with remote write capability
• Network Access: Collector can reach Temporal services and Last9 endpoints

Keep the following Last9 credentials handy:

• $last9_remote_write_url - Last9’s Remote write endpoint
• $last9_remote_write_username - Cluster ID
• $last9_remote_write_password - Write token created for the cluster
• $last9_otlp_endpoint - Last9’s OTLP endpoint (for OpenTelemetry)
• $last9_otlp_auth_header - OTLP authorization header

Temporal Server Metrics

1. Configure Temporal Server Metrics Endpoint

   Update your Temporal Server configuration to expose Prometheus metrics. Add the following to your Temporal Server configuration file:

   global:
     metrics:
       prometheus:
         framework: "opentelemetry"
         listenAddress: "0.0.0.0:9090"
         handlerPath: "/metrics"

   For Docker Compose deployments, ensure the metrics port is exposed:

   services:
     temporal:
       image: temporalio/auto-setup:latest
       ports:
         - "7233:7233" # gRPC frontend
         - "9090:9090" # Metrics endpoint
       environment:
         - PROMETHEUS_ENDPOINT=0.0.0.0:9090

2. Configure Prometheus to Scrape Temporal

   Add Temporal as a scrape target in your prometheus.yaml:

   global:
     scrape_interval: 15s
   
   scrape_configs:
     # Temporal Server metrics
     - job_name: "temporal-server"
       static_configs:
         - targets:
             - "temporal-frontend:9090"
             - "temporal-history:9090"
             - "temporal-matching:9090"
             - "temporal-worker:9090"
       metrics_path: /metrics
   
     # Temporal SDK Worker metrics (if exposed)
     - job_name: "temporal-workers"
       static_configs:
         - targets:
             - "your-worker-host:8077"
             - "your-worker-host:8078"

3. Configure Prometheus Remote Write to Last9

   Add the remote write configuration to forward metrics to Last9:

   remote_write:
     - url: "$last9_remote_write_url"
       basic_auth:
         username: "$last9_remote_write_username"
         password: "$last9_remote_write_password"
       remote_timeout: 60s

4. Restart Services

   Restart Prometheus to apply the configuration changes:

   # For systemd
   sudo systemctl restart prometheus
   
   # For Docker
   docker-compose restart prometheus

SDK Metrics Configuration

Temporal SDKs emit metrics for Workers, Workflows, and Activities. Configure your SDK to expose these metrics.

package main

import (
    "log"
    "time"

    "github.com/uber-go/tally/v4"
    "github.com/uber-go/tally/v4/prometheus"
    "go.temporal.io/sdk/client"
    "go.temporal.io/sdk/contrib/opentelemetry"
    sdktally "go.temporal.io/sdk/contrib/tally"
    "go.temporal.io/sdk/interceptor"
    "go.temporal.io/sdk/worker"
)

func main() {
    // Configure Prometheus metrics reporter — serve its handler yourself,
    // e.g. http.Handle("/metrics", reporter.HTTPHandler())
    reporter := prometheus.NewReporter(prometheus.Options{})

    scope, closer := tally.NewRootScope(tally.ScopeOptions{
        CachedReporter: reporter,
    }, time.Second)
    defer closer.Close()

    // Create OpenTelemetry tracing interceptor
    tracingInterceptor, err := opentelemetry.NewTracingInterceptor(
        opentelemetry.TracerOptions{},
    )
    if err != nil {
        log.Fatal(err)
    }

    // Create Temporal client with BOTH metrics and tracing
    c, err := client.Dial(client.Options{
        HostPort:       "temporal:7233",
        MetricsHandler: sdktally.NewMetricsHandler(scope),
        Interceptors:   []interceptor.ClientInterceptor{tracingInterceptor},
    })
    if err != nil {
        log.Fatal(err)
    }
    defer c.Close()

    // Create worker with metrics
    w := worker.New(c, "your-task-queue", worker.Options{})

    // Register workflows and activities
    w.RegisterWorkflow(YourWorkflow)
    w.RegisterActivity(YourActivity)

    // Start worker
    if err := w.Run(worker.InterruptCh()); err != nil {
        log.Fatal(err)
    }
}

import { NativeConnection, Worker } from "@temporalio/worker";
import { Runtime } from "@temporalio/worker";

// Configure metrics endpoint
Runtime.install({
  telemetryOptions: {
    metrics: {
      prometheus: {
        bindAddress: "0.0.0.0:8077",
      },
    },
  },
});

async function run() {
  const connection = await NativeConnection.connect({
    address: "temporal:7233",
  });

  const worker = await Worker.create({
    connection,
    namespace: "default",
    taskQueue: "your-task-queue",
    workflowsPath: require.resolve("./workflows"),
    activities: require("./activities"),
  });

  await worker.run();
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});

import {
  OpenTelemetryActivityInboundInterceptor,
  makeWorkflowExporter,
} from "@temporalio/interceptors-opentelemetry/lib/worker";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";
import { Resource } from "@opentelemetry/resources";

const resource = new Resource({ "service.name": "your-worker" });
const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter());

const worker = await Worker.create({
  connection,
  namespace: "default",
  taskQueue: "your-task-queue",
  workflowsPath: require.resolve("./workflows"),
  activities: require("./activities"),
  interceptors: {
    // `activityInbound` factories receive the Activity Context;
    // pass it through to the interceptor's constructor
    activityInbound: [
      (ctx) => new OpenTelemetryActivityInboundInterceptor(ctx),
    ],
    workflowModules: [
      require.resolve("@temporalio/interceptors-opentelemetry/lib/workflow"),
    ],
  },
  sinks: {
    exporter: makeWorkflowExporter(spanProcessor, resource),
  },
});

pip install "temporalio[opentelemetry]"

import asyncio
from temporalio.client import Client
from temporalio.worker import Worker
from temporalio.runtime import Runtime, TelemetryConfig, PrometheusConfig
from temporalio.contrib.opentelemetry import TracingInterceptor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

# Set up OpenTelemetry tracing (skip this if opentelemetry-instrument
# already configured a global TracerProvider for this process)
provider = TracerProvider()
trace.set_tracer_provider(provider)

# Configure runtime with Prometheus metrics
runtime = Runtime(
    telemetry=TelemetryConfig(
        metrics=PrometheusConfig(bind_address="0.0.0.0:8077")
    )
)

async def main():
    client = await Client.connect(
        "temporal:7233",
        runtime=runtime,
        interceptors=[TracingInterceptor()],
    )

    worker = Worker(
        client,
        task_queue="your-task-queue",
        workflows=[YourWorkflow],
        activities=[your_activity],
    )

    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())

OpenTelemetry Collector Setup

Use the OpenTelemetry Collector to receive metrics and traces from Temporal in one pipeline.

1. Install OpenTelemetry Collector

   • DEB Package
   • RPM Package
   • Docker

   wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.118.0/otelcol-contrib_0.118.0_linux_amd64.deb
   sudo dpkg -i otelcol-contrib_0.118.0_linux_amd64.deb

2. Configure the Collector

   Create /etc/otelcol-contrib/config.yaml:

   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: 0.0.0.0:4317
         http:
           endpoint: 0.0.0.0:4318
   
     prometheus:
       config:
         scrape_configs:
           - job_name: "temporal-server"
             scrape_interval: 15s
             static_configs:
               - targets:
                   - "temporal-frontend:9090"
                   - "temporal-history:9090"
                   - "temporal-matching:9090"
   
           - job_name: "temporal-workers"
             scrape_interval: 15s
             static_configs:
               - targets:
                   - "worker-host:8077"
   
   processors:
     batch:
       timeout: 15s
       send_batch_size: 10000
   
     resource:
       attributes:
         - key: service.name
           value: temporal
           action: upsert
         - key: deployment.environment
           value: production
           action: upsert
   
   exporters:
     otlp/last9:
       endpoint: "$last9_otlp_endpoint"
       headers:
         Authorization: "$last9_otlp_auth_header"
   
     prometheusremotewrite:
       endpoint: "$last9_remote_write_url"
       auth:
         authenticator: basicauth/last9
   
   extensions:
     basicauth/last9:
       client_auth:
         username: "$last9_remote_write_username"
         password: "$last9_remote_write_password"
   
   service:
     extensions: [basicauth/last9]
     pipelines:
       metrics:
         receivers: [prometheus, otlp]
         processors: [batch, resource]
         exporters: [prometheusremotewrite]
       traces:
         receivers: [otlp]
         processors: [batch, resource]
         exporters: [otlp/last9]

3. Start the Collector

   sudo systemctl daemon-reload
   sudo systemctl enable otelcol-contrib
   sudo systemctl start otelcol-contrib

Key Metrics to Monitor

Workflow Metrics

Activity Metrics

Worker Metrics

Server Metrics

Example Prometheus Queries

Query workflow completion rate:

sum(rate(temporal_workflow_completed_total[5m])) by (namespace, workflow_type)

Monitor Schedule-To-Start latency (P95):

histogram_quantile(0.95,
  sum(rate(temporal_workflow_task_schedule_to_start_latency_bucket[5m])) by (le, namespace)
)

Track activity failure rate:

sum(rate(temporal_activity_execution_failed_total[5m])) by (namespace, activity_type)
/
sum(rate(temporal_activity_execution_latency_count[5m])) by (namespace, activity_type)

Worker utilization:

temporal_worker_task_slots_used
/
(temporal_worker_task_slots_used + temporal_worker_task_slots_available)

Sticky cache hit ratio (alert if this drops below 80%):

sum(rate(temporal_sticky_cache_hit_total[5m])) by (namespace)
/
(sum(rate(temporal_sticky_cache_hit_total[5m])) by (namespace) + sum(rate(temporal_sticky_cache_miss_total[5m])) by (namespace))

Verification

1. Check Prometheus Targets

   Navigate to your Prometheus instance at http://localhost:9090/targets and verify that Temporal targets are being scraped successfully.

2. Query Temporal Metrics

   Run a test query in Prometheus:

   temporal_service_requests

3. Verify Last9 Ingestion

   Log into your Last9 account and open Metrics Explorer at https://app.last9.io/v2/organizations/<org_slug>/metrics (replace <org_slug> with your organization slug). Search for Temporal metrics:

   • temporal_workflow_completed_total (counters land with a _total suffix when forwarded through the Collector; see the note in Example Prometheus Queries)
   • temporal_activity_execution_latency
   • temporal_worker_task_slots_available

4. Generate Test Workflows

   Run some test workflows to generate metrics:

   # Using the Temporal CLI (tctl is deprecated; use `temporal`)
   temporal workflow start \
     --workflow-id your-workflow-id \
     --type YourWorkflow \
     --task-queue your-task-queue \
     --input '{"key": "value"}'

Next Steps

• Open Metrics Explorer (https://app.last9.io/v2/organizations/<org_slug>/metrics) to visualize Temporal metrics
• Set up Alerts for workflow failures and latency thresholds
• Explore Log Management for Temporal server logs

Troubleshooting

• High schedule-to-start latency

  If temporal_workflow_task_schedule_to_start_latency or temporal_activity_schedule_to_start_latency is high, check these causes:

  • Insufficient workers: Scale up the number of workers, or increase the task slots.
  • Worker overload: Check temporal_worker_task_slots_available for capacity limits.
  • Network latency: Deploy the workers close to the Temporal cluster.

• Missing metrics

  • Verify that the metrics endpoint is accessible: curl http://temporal-frontend:9090/metrics
  • Check the scrape targets in the Prometheus configuration.
  • Make sure that the firewall rules allow metrics traffic.

• SDK metrics do not appear

  • Verify that the SDK metrics bind address is accessible.
  • Check that the metrics handler is configured in your worker code.
  • Make sure that Prometheus can reach the metrics endpoint of the worker.

Please get in touch with us on Discord or Email if you have any questions.