# Temporal

> Monitor Temporal workflows and activities with Last9 using Prometheus and OpenTelemetry

Source: https://last9.io/docs/integrations/others/temporal/

Monitor your [Temporal](https://temporal.io/) workflow orchestration platform with Last9. This integration covers metrics from both the Temporal Server (self-hosted) and Temporal SDKs, so you can see how your durable workflow executions behave.

Temporal is a durable execution platform that helps developers build scalable, reliable applications. Monitor Temporal to measure workflow performance, to find bottlenecks, and to confirm reliable task processing.

Last9 supports two methods for collecting Temporal telemetry:

- **Prometheus Remote Write**: Scrape metrics from Temporal Server and SDKs, then forward to Last9
- **OpenTelemetry**: Send traces and metrics directly to Last9 via OTLP

## Prerequisites

Before setting up Temporal monitoring, ensure you have:

- **Temporal Cluster**: Running Temporal Server (self-hosted) or Temporal Cloud account
- **Last9 Account**: With integration credentials from [Getting Started](/docs/onboard/)
- **Prometheus** (for metrics): Running instance with remote write capability
- **Network Access**: Collector can reach Temporal services and Last9 endpoints

Keep the following Last9 credentials handy:

- `$last9_remote_write_url` - Last9's Remote write endpoint
- `$last9_remote_write_username` - Cluster ID
- `$last9_remote_write_password` - Write token created for the cluster
- `$last9_otlp_endpoint` - Last9's OTLP endpoint (for OpenTelemetry)
- `$last9_otlp_auth_header` - OTLP authorization header

## Temporal Server Metrics

1. **Configure Temporal Server Metrics Endpoint**

   Update your Temporal Server configuration to expose Prometheus metrics. Add the following to your Temporal Server configuration file:

   ```yaml
   global:
     metrics:
       prometheus:
         framework: "opentelemetry"
         listenAddress: "0.0.0.0:9090"
         handlerPath: "/metrics"
   ```

   For Docker Compose deployments, ensure the metrics port is exposed:

   ```yaml
   services:
     temporal:
       image: temporalio/auto-setup:latest
       ports:
         - "7233:7233" # gRPC frontend
         - "9090:9090" # Metrics endpoint
       environment:
         - PROMETHEUS_ENDPOINT=0.0.0.0:9090
   ```

2. **Configure Prometheus to Scrape Temporal**

   Add Temporal as a scrape target in your `prometheus.yaml`:

   ```yaml
   global:
     scrape_interval: 15s

   scrape_configs:
     # Temporal Server metrics
     - job_name: "temporal-server"
       static_configs:
         - targets:
             - "temporal-frontend:9090"
             - "temporal-history:9090"
             - "temporal-matching:9090"
             - "temporal-worker:9090"
       metrics_path: /metrics

     # Temporal SDK Worker metrics (if exposed)
     - job_name: "temporal-workers"
       static_configs:
         - targets:
             - "your-worker-host:8077"
             - "your-worker-host:8078"
   ```

   :::note
   When Temporal services are deployed independently, each service exposes its own metrics endpoint. Configure scrape targets for each service you want to monitor.
   :::

3. **Configure Prometheus Remote Write to Last9**

   Add the remote write configuration to forward metrics to Last9:

   ```yaml
   remote_write:
     - url: "$last9_remote_write_url"
       basic_auth:
         username: "$last9_remote_write_username"
         password: "$last9_remote_write_password"
       remote_timeout: 60s
   ```

4. **Restart Services**

   Restart Prometheus to apply the configuration changes:

   ```bash
   # For systemd
   sudo systemctl restart prometheus

   # For Docker
   docker-compose restart prometheus
   ```

## SDK Metrics Configuration

Temporal SDKs emit metrics for Workers, Workflows, and Activities. Configure your SDK to expose these metrics.

**Go**

`contrib/opentelemetry` and `contrib/tally` are separate Go modules from the main SDK — add them with `go get go.temporal.io/sdk/contrib/opentelemetry` and `go get go.temporal.io/sdk/contrib/tally`. Configure metrics and tracing together in one `client.Dial` call — if they're set up in separate calls, the second call overwrites the first client's config entirely:

```go
package main

import (
    "log"
    "time"

    "github.com/uber-go/tally/v4"
    "github.com/uber-go/tally/v4/prometheus"
    "go.temporal.io/sdk/client"
    "go.temporal.io/sdk/contrib/opentelemetry"
    sdktally "go.temporal.io/sdk/contrib/tally"
    "go.temporal.io/sdk/interceptor"
    "go.temporal.io/sdk/worker"
)

func main() {
    // Configure Prometheus metrics reporter — serve its handler yourself,
    // e.g. http.Handle("/metrics", reporter.HTTPHandler())
    reporter := prometheus.NewReporter(prometheus.Options{})

    scope, closer := tally.NewRootScope(tally.ScopeOptions{
        CachedReporter: reporter,
    }, time.Second)
    defer closer.Close()

    // Create OpenTelemetry tracing interceptor
    tracingInterceptor, err := opentelemetry.NewTracingInterceptor(
        opentelemetry.TracerOptions{},
    )
    if err != nil {
        log.Fatal(err)
    }

    // Create Temporal client with BOTH metrics and tracing
    c, err := client.Dial(client.Options{
        HostPort:       "temporal:7233",
        MetricsHandler: sdktally.NewMetricsHandler(scope),
        Interceptors:   []interceptor.ClientInterceptor{tracingInterceptor},
    })
    if err != nil {
        log.Fatal(err)
    }
    defer c.Close()

    // Create worker with metrics
    w := worker.New(c, "your-task-queue", worker.Options{})

    // Register workflows and activities
    w.RegisterWorkflow(YourWorkflow)
    w.RegisterActivity(YourActivity)

    // Start worker
    if err := w.Run(worker.InterruptCh()); err != nil {
        log.Fatal(err)
    }
}
```

**TypeScript**

```typescript
import { NativeConnection, Worker } from "@temporalio/worker";
import { Runtime } from "@temporalio/worker";

// Configure metrics endpoint
Runtime.install({
  telemetryOptions: {
    metrics: {
      prometheus: {
        bindAddress: "0.0.0.0:8077",
      },
    },
  },
});

async function run() {
  const connection = await NativeConnection.connect({
    address: "temporal:7233",
  });

  const worker = await Worker.create({
    connection,
    namespace: "default",
    taskQueue: "your-task-queue",
    workflowsPath: require.resolve("./workflows"),
    activities: require("./activities"),
  });

  await worker.run();
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

For OpenTelemetry tracing, use the `@temporalio/interceptors-opentelemetry` package. Pin your `@opentelemetry/*` packages to the major version this package bundles (v1.x as of `@temporalio/interceptors-opentelemetry@1.23.0`) — mixing major versions causes type and runtime incompatibilities:

```typescript
import {
  OpenTelemetryActivityInboundInterceptor,
  makeWorkflowExporter,
} from "@temporalio/interceptors-opentelemetry/lib/worker";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";
import { Resource } from "@opentelemetry/resources";

const resource = new Resource({ "service.name": "your-worker" });
const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter());

const worker = await Worker.create({
  connection,
  namespace: "default",
  taskQueue: "your-task-queue",
  workflowsPath: require.resolve("./workflows"),
  activities: require("./activities"),
  interceptors: {
    // `activityInbound` factories receive the Activity Context;
    // pass it through to the interceptor's constructor
    activityInbound: [
      (ctx) => new OpenTelemetryActivityInboundInterceptor(ctx),
    ],
    workflowModules: [
      require.resolve("@temporalio/interceptors-opentelemetry/lib/workflow"),
    ],
  },
  sinks: {
    exporter: makeWorkflowExporter(spanProcessor, resource),
  },
});
```

**Python**

Install the SDK with the OpenTelemetry extra:

```bash
pip install "temporalio[opentelemetry]"
```

Configure Prometheus metrics and OpenTelemetry tracing together on the same client. Pass both `runtime` (metrics) and `interceptors` (tracing) in the same `Client.connect()` call. If you use two separate calls, the second call drops the runtime from the first call:

```python
import asyncio
from temporalio.client import Client
from temporalio.worker import Worker
from temporalio.runtime import Runtime, TelemetryConfig, PrometheusConfig
from temporalio.contrib.opentelemetry import TracingInterceptor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

# Set up OpenTelemetry tracing (skip this if opentelemetry-instrument
# already configured a global TracerProvider for this process)
provider = TracerProvider()
trace.set_tracer_provider(provider)

# Configure runtime with Prometheus metrics
runtime = Runtime(
    telemetry=TelemetryConfig(
        metrics=PrometheusConfig(bind_address="0.0.0.0:8077")
    )
)

async def main():
    client = await Client.connect(
        "temporal:7233",
        runtime=runtime,
        interceptors=[TracingInterceptor()],
    )

    worker = Worker(
        client,
        task_queue="your-task-queue",
        workflows=[YourWorkflow],
        activities=[your_activity],
    )

    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())
```

:::note
By default, `TracingInterceptor` only creates workflow-level spans when the workflow was started from a call that already had an active parent span (for example, an HTTP request handled by auto-instrumentation). This avoids orphaned spans across workflow replays. Activity spans and client-call spans are always created. If a workflow starts from a source with no active span, such as a cron schedule or a CLI command, pass `TracingInterceptor(always_create_workflow_spans=True)`. This forces workflow spans, but it can produce orphaned spans after a replay.
:::

If your process already runs `opentelemetry-instrument` for HTTP auto-instrumentation, `TracingInterceptor()` uses the same global `TracerProvider` by default. Worker spans then join the same trace as the HTTP request that started the workflow. Add `TracingInterceptor` after auto-instrumentation sets up the provider.

## OpenTelemetry Collector Setup

Use the OpenTelemetry Collector to receive metrics and traces from Temporal in one pipeline.

1. **Install OpenTelemetry Collector**

**DEB Package**

   ```bash
   wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.118.0/otelcol-contrib_0.118.0_linux_amd64.deb
   sudo dpkg -i otelcol-contrib_0.118.0_linux_amd64.deb
   ```

**RPM Package**

   ```bash
   wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.118.0/otelcol-contrib_0.118.0_linux_amd64.rpm
   sudo rpm -ivh otelcol-contrib_0.118.0_linux_amd64.rpm
   ```

**Docker**

   ```yaml
   services:
     otel-collector:
       image: otel/opentelemetry-collector-contrib:0.118.0
       command: ["--config=/etc/otelcol/config.yaml"]
       volumes:
         - ./otel-config.yaml:/etc/otelcol/config.yaml
       ports:
         - "4317:4317" # OTLP gRPC
         - "4318:4318" # OTLP HTTP
         - "8889:8889" # Prometheus metrics
   ```

2. **Configure the Collector**

   Create `/etc/otelcol-contrib/config.yaml`:

   ```yaml
   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: 0.0.0.0:4317
         http:
           endpoint: 0.0.0.0:4318

     prometheus:
       config:
         scrape_configs:
           - job_name: "temporal-server"
             scrape_interval: 15s
             static_configs:
               - targets:
                   - "temporal-frontend:9090"
                   - "temporal-history:9090"
                   - "temporal-matching:9090"

           - job_name: "temporal-workers"
             scrape_interval: 15s
             static_configs:
               - targets:
                   - "worker-host:8077"

   processors:
     batch:
       timeout: 15s
       send_batch_size: 10000

     resource:
       attributes:
         - key: service.name
           value: temporal
           action: upsert
         - key: deployment.environment
           value: production
           action: upsert

   exporters:
     otlp/last9:
       endpoint: "$last9_otlp_endpoint"
       headers:
         Authorization: "$last9_otlp_auth_header"

     prometheusremotewrite:
       endpoint: "$last9_remote_write_url"
       auth:
         authenticator: basicauth/last9

   extensions:
     basicauth/last9:
       client_auth:
         username: "$last9_remote_write_username"
         password: "$last9_remote_write_password"

   service:
     extensions: [basicauth/last9]
     pipelines:
       metrics:
         receivers: [prometheus, otlp]
         processors: [batch, resource]
         exporters: [prometheusremotewrite]
       traces:
         receivers: [otlp]
         processors: [batch, resource]
         exporters: [otlp/last9]
   ```

3. **Start the Collector**

   ```bash
   sudo systemctl daemon-reload
   sudo systemctl enable otelcol-contrib
   sudo systemctl start otelcol-contrib
   ```

## Key Metrics to Monitor

### Workflow Metrics

| Metric                                             | Description                                                                                              | Alert Threshold   |
| -------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------- |
| `temporal_workflow_completed`                      | Count of completed workflows                                                                             | Track trends      |
| `temporal_workflow_failed`                         | Count of failed workflows                                                                                | > 0 (investigate) |
| `temporal_workflow_canceled`                       | Count of canceled workflows                                                                              | Track trends      |
| `temporal_workflow_continue_as_new`                | Workflows that continued as new                                                                          | Track trends      |
| `temporal_workflow_task_schedule_to_start_latency` | Time from the task being placed on the task queue to a worker's poller picking it up (the Started event) | P95 > 1s          |

### Activity Metrics

| Metric                                        | Description                                                                                              | Alert Threshold   |
| --------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------- |
| `temporal_activity_schedule_to_start_latency` | Time from the task being placed on the task queue to a worker's poller picking it up (the Started event) | P95 > 1s          |
| `temporal_activity_execution_latency`         | Time to complete activity execution                                                                      | Based on SLO      |
| `temporal_activity_execution_failed`          | Count of failed activity executions                                                                      | > 0 (investigate) |

### Worker Metrics

| Metric                                 | Description                           | Alert Threshold  |
| -------------------------------------- | ------------------------------------- | ---------------- |
| `temporal_worker_task_slots_available` | Available task slots on worker        | < 10% of max     |
| `temporal_sticky_cache_hit`            | Count of sticky workflow cache hits   | Track trends     |
| `temporal_sticky_cache_miss`           | Count of sticky workflow cache misses | Track trends     |
| `temporal_poller_start`                | Poller start events                   | Should be stable |
| `temporal_num_pollers`                 | Number of active pollers (gauge)      | Should be stable |

### Server Metrics

| Metric                         | Description                         | Alert Threshold |
| ------------------------------ | ----------------------------------- | --------------- |
| `temporal_service_requests`    | Total service requests by operation | Track trends    |
| `temporal_service_latency`     | Service request latency             | P99 > 500ms     |
| `temporal_persistence_latency` | Database operation latency          | P99 > 100ms     |

## Example Prometheus Queries

Query workflow completion rate:

```promql
sum(rate(temporal_workflow_completed_total[5m])) by (namespace, workflow_type)
```

:::note
The SDK's own `/metrics` endpoint exposes counters without a `_total` suffix (see the table above). If your pipeline forwards through an OpenTelemetry Collector's `prometheusremotewrite` exporter, as in the [Collector setup](#opentelemetry-collector-setup) above, the exporter appends `_total` to every monotonic counter on the way out — this is mandated by the [OpenTelemetry Prometheus compatibility spec](https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/), not something the Temporal SDK controls. The queries below use the `_total` form to match what actually lands in Last9. If you scrape the SDK's endpoint directly with Prometheus (no Collector in between), drop the `_total` suffix. Gauges (like `temporal_worker_task_slots_available`) are unaffected either way.
:::

Monitor Schedule-To-Start latency (P95):

```promql
histogram_quantile(0.95,
  sum(rate(temporal_workflow_task_schedule_to_start_latency_bucket[5m])) by (le, namespace)
)
```

Track activity failure rate:

```promql
sum(rate(temporal_activity_execution_failed_total[5m])) by (namespace, activity_type)
/
sum(rate(temporal_activity_execution_latency_count[5m])) by (namespace, activity_type)
```

Worker utilization:

```promql
temporal_worker_task_slots_used
/
(temporal_worker_task_slots_used + temporal_worker_task_slots_available)
```

Sticky cache hit ratio (alert if this drops below 80%):

```promql
sum(rate(temporal_sticky_cache_hit_total[5m])) by (namespace)
/
(sum(rate(temporal_sticky_cache_hit_total[5m])) by (namespace) + sum(rate(temporal_sticky_cache_miss_total[5m])) by (namespace))
```

## Verification

1. **Check Prometheus Targets**

   Navigate to your Prometheus instance at `http://localhost:9090/targets` and verify that Temporal targets are being scraped successfully.

2. **Query Temporal Metrics**

   Run a test query in Prometheus:

   ```promql
   temporal_service_requests
   ```

3. **Verify Last9 Ingestion**

   Log into your Last9 account and open Metrics Explorer at `https://app.last9.io/v2/organizations/<org_slug>/metrics` (replace `<org_slug>` with your organization slug). Search for Temporal metrics:

   - `temporal_workflow_completed_total` (counters land with a `_total` suffix when forwarded through the Collector; see the note in [Example Prometheus Queries](#example-prometheus-queries))
   - `temporal_activity_execution_latency`
   - `temporal_worker_task_slots_available`

4. **Generate Test Workflows**

   Run some test workflows to generate metrics:

   ```bash
   # Using the Temporal CLI (tctl is deprecated; use `temporal`)
   temporal workflow start \
     --workflow-id your-workflow-id \
     --type YourWorkflow \
     --task-queue your-task-queue \
     --input '{"key": "value"}'
   ```

## Next Steps

- Open Metrics Explorer (`https://app.last9.io/v2/organizations/<org_slug>/metrics`) to visualize Temporal metrics
- Set up [Alerts](/docs/alerting/) for workflow failures and latency thresholds
- Explore [Log Management](/docs/logs/) for Temporal server logs

---

## Troubleshooting

- **High schedule-to-start latency**

  If `temporal_workflow_task_schedule_to_start_latency` or `temporal_activity_schedule_to_start_latency` is high, check these causes:

  - **Insufficient workers**: Scale up the number of workers, or increase the task slots.
  - **Worker overload**: Check `temporal_worker_task_slots_available` for capacity limits.
  - **Network latency**: Deploy the workers close to the Temporal cluster.

- **Missing metrics**

  - Verify that the metrics endpoint is accessible: `curl http://temporal-frontend:9090/metrics`
  - Check the scrape targets in the Prometheus configuration.
  - Make sure that the firewall rules allow metrics traffic.

- **SDK metrics do not appear**

  - Verify that the SDK metrics bind address is accessible.
  - Check that the metrics handler is configured in your worker code.
  - Make sure that Prometheus can reach the metrics endpoint of the worker.

Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:support@last9.io) if you have any questions.