# Enable Kong Gateway Tracing in 5 Minutes

> Instrument Kong with OpenTelemetry for end-to-end API visibility, no code changes required.

Source: https://last9.io/blog/enable-kong-gateway-tracing-in-5-minutes/

Kong Gateway is a popular API gateway that sits at the edge of your infrastructure, routing and shaping traffic across microservices. It’s fast, pluggable, and battle-tested, but for many teams, it remains a black box.

You might have OpenTelemetry set up across your application stack. Traces flow from your app servers, databases, and third-party APIs.

But the moment a request enters through Kong, observability drops off. There’s no visibility into plugins, proxy logic, or where time is being spent before the request even hits your backend.

This blog walks through how to enable native OpenTelemetry tracing in Kong Gateway using its built-in plugin. The setup takes about 5 minutes, works with any OTLP-compatible backend, and doesn’t require writing any application code.

## Why Trace the API Gateway?

[Kong](https://konghq.com/) is often the first system that handles incoming requests, yet it's rarely instrumented for tracing. This creates a visibility gap at the very entry point of your system.

Without gateway tracing:

- Latency issues become ambiguous—was it DNS? A plugin? Upstream slowness?
- Errors like 504s offer no clues about where the timeout occurred
- Observability tools show partial traces starting after the request has passed through Kong

With gateway tracing:

- You see the full lifecycle of a request, starting at the ingress
- You can pinpoint latency introduced by specific plugins (like auth or rate-limiting)
- You can correlate upstream service errors with Kong retries, proxying delays, and failures

Here’s a quick example:

**Without tracing**:  
“API is slow. It could be the app, database, or Kong. Start checking everything.”

**With tracing**:  
“Auth plugin on Kong is adding 200ms to every request. Optimize JWT validation.”

Adding tracing at this layer helps close the loop, especially in production systems that rely on Kong for routing, auth, security, and more.

If you're exporting traces from Kong, it's worth understanding how [OpenTelemetry's semantic conventions](https://last9.io/blog/opentelemetry-semantic-conventions/) apply to things like HTTP routes, status codes, and service names.

## Prerequisites

Before you start, ensure the following:

- Kong Gateway v3.1 or later
- Access to the Kong Admin API (`http://localhost:8001` by default)
- An OpenTelemetry-compatible backend or collector  
  (e.g., Last9, Jaeger, Tempo)
- Tracing enabled via an environment variable

Kong does not emit traces by default. You need to explicitly enable tracing and tell Kong which phases to instrument.

Set the following environment variable before starting Kong:

```bash
export KONG_TRACING_INSTRUMENTATIONS="all"
```

This instructs Kong to generate spans for all instrumented phases (`router`, `balancer`, `plugins`, `proxy`, etc.). You can scope this more narrowly if needed.

Where to set this, depending on your deployment:

```yaml
environment:
  KONG_TRACING_INSTRUMENTATIONS: "all"
```

```yaml
env:
  - name: KONG_TRACING_INSTRUMENTATIONS
    value: "all"
```

```ini
Environment=KONG_TRACING_INSTRUMENTATIONS=all
```

> **Important:** Restart Kong after setting this variable. Kong will not emit any traces unless this is set at startup, even if the plugin is correctly configured.

## Set Up Tracing with the OpenTelemetry Plugin

Kong provides a native plugin that exports traces in OpenTelemetry format. This plugin can be applied globally or to individual routes or services.

### Configure the Plugin Globally

This setup sends traces to a collector running at `http://otel-collector:4318/v1/traces`.

```bash
curl -X POST http://localhost:8001/plugins \
  --data "name=opentelemetry" \
  --data "config.endpoint=http://otel-collector:4318/v1/traces"
```

To enable it on a specific service only:

```bash
curl -X POST http://localhost:8001/plugins \
  --data "name=opentelemetry" \
  --data "service.id=SERVICE_ID" \
  --data "config.endpoint=http://otel-collector:4318/v1/traces"
```

If your collector requires authentication, include headers like this:

```bash
--data "config.headers.Authorization=Bearer ${OTEL_TOKEN}"
```

**Note**: Never hardcode tokens in production. Use environment variables or Kong’s [Vault integration](https://docs.konghq.com/gateway/latest/kong-enterprise/secrets-management/).

## Verify Trace Export Is Working

Once the plugin is active and Kong is restarted, send a request through the gateway:

```bash
curl http://localhost:8000/your-api-endpoint
```

Then check your trace backend.

### In Last9:

- Go to Traces
- Search for the service name `kong`
- Look for spans like `kong:ingress`, `kong:plugins:<plugin-name>`, `kong:proxy`, or `kong:upstream:<service>`

### In Jaeger:

- Open `http://localhost:16686`
- Select the `kong` service
- View spans and breakdowns per phase

You should see a clear sequence of spans showing:

- How long Kong spent processing the request
- How much time each plugin took
- Upstream response time and status

You can check out this breakdown of [OpenTelemetry vs Jaeger](https://last9.io/blog/opentelemetry-vs-jaeger-which-should-you-pick/) if you're weighing your options for trace collection and analysis.

## Local Setup Using Docker Compose

To try gateway tracing locally, here's a complete Docker Compose setup that runs:

- Kong Gateway (with OpenTelemetry tracing enabled)
- OpenTelemetry Collector (to receive spans from Kong and forward them to Last9)

This gives you a working setup to see traces from Kong in your Last9 instance in just a few steps.

**Step 1: Create the Docker Compose File**

Create a new file called `docker-compose.yaml` in your working directory and add the following:

```yaml
version: "3.8"

services:
  kong:
    image: kong:3.2
    environment:
      KONG_DATABASE: "off"
      KONG_ADMIN_LISTEN: "0.0.0.0:8001"
      KONG_PROXY_LISTEN: "0.0.0.0:8000"
      KONG_TRACING_INSTRUMENTATIONS: "all"
    ports:
      - "8000:8000" # Kong proxy
      - "8001:8001" # Kong Admin API
    depends_on:
      - otel-collector

  otel-collector:
    image: otel/opentelemetry-collector:latest
    command: ["--config=/etc/otel/config.yaml"]
    volumes:
      - ./otel-config.yaml:/etc/otel/config.yaml
    ports:
      - "4318:4318" # OTLP HTTP receiver
```

**Step 2: Create the OpenTelemetry Collector Config**

In the same directory, create a file named `otel-config.yaml` with the following contents. Replace `YOUR_API_KEY` with your actual Last9 token.

```yaml
receivers:
  otlp:
    protocols:
      http:

exporters:
  otlphttp:
    endpoint: https://otel.last9.io/v1/traces
    headers:
      Authorization: "Bearer YOUR_API_KEY"

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp]
```

This config tells the OTEL Collector to accept spans from Kong over HTTP and forward them securely to Last9.

> **Security Note**: Do not hardcode tokens in production. Use Docker secrets, environment variables, or Kong's Vault integration.

**Step 3: Start the Stack**

Bring everything up with:

```bash
docker-compose up -d
```

This will start Kong and the OTEL Collector. Kong will be listening on ports `8000` (proxy) and `8001` (Admin API).

**Step 4: Enable the OpenTelemetry Plugin in Kong**

Once Kong is running, use the Admin API to configure the OpenTelemetry plugin:

```bash
curl -X POST http://localhost:8001/plugins \
  --data "name=opentelemetry" \
  --data "config.endpoint=http://otel-collector:4318/v1/traces"
```

This registers the plugin globally. You can also scope it to a specific service or route if needed.

**Step 5: Test the Setup**

Send a test request through Kong:

```bash
curl http://localhost:8000/your-api-endpoint
```

If you haven’t configured a service yet, a `404` is expected. That’s fine, it still generates spans for `ingress`, `router`, and `proxy` phases.

**Step 6: View Traces in Last9**

Head to your [Last9 Traces UI](https://last9.io/docs/traces/) :

- Search for the service name `kong`
- Use filters like `span.name starts with kong:` or check for spans such as:
  - `kong:ingress`
  - `kong:router`
  - `kong:proxy`
  - `kong:plugins:<plugin-name>`
  - `kong:upstream:<service-name>`

This gives you complete visibility into how Kong is handling your request: from routing decisions and plugin latency to upstream service response times.

If you're curious about how the OpenTelemetry Collector actually handles aggregation, exporting, and processing of spans behind the scenes, our guide, [The OpenTelemetry Collector deep dive](https://last9.io/guides/opentelemetry/the-opentelemetry-collector-deep-dive/), which breaks down its role in your tracing pipeline.

## What Happens if the Collector Fails?

If Kong cannot reach the collector endpoint, it silently drops spans. Request handling is not blocked or affected. Kong continues operating normally, which is by design to prevent observability issues from impacting request flow.

However, you won’t see trace data until the collector becomes reachable again.

For production:

- Use batch exporters with retries and backoff
- Monitor collector health
- Alert on dropped spans or disconnected exporters

## Performance and Overhead

The OpenTelemetry plugin in Kong Gateway is built to minimize impact while enabling detailed trace visibility. It operates asynchronously and is suitable for production traffic with moderate to high request volumes.

### Latency Impact

- Typical overhead: ~1–2ms per request under moderate load (up to 100–200 RPS)
- Spans are batched and exported in the background, so request handling remains non-blocking
- If the internal span queue fills (e.g., due to downstream collector slowness), new spans may be dropped silently to avoid blocking

### CPU & Resource Usage

- Negligible CPU impact under normal operating conditions
- Overhead becomes noticeable only when pushing thousands of RPS without proper sampling or throttling
- The plugin itself does not perform heavy computation; most processing happens in the collector or backend

### Reducing Overhead in Production

To optimize performance further:

- **Use trace sampling**:
  - Prefer collector-side sampling (e.g., tail-based sampling in OTEL Collector)
  - For high-throughput systems, apply upfront sampling at Kong using rate or probabilistic options once supported
- **Limit instrumentation scope** by setting `KONG_TRACING_INSTRUMENTATIONS` to only the necessary phases:

```bash
export KONG_TRACING_INSTRUMENTATIONS="proxy,plugins"
```

### Monitoring Collector Pressure

- To detect span drops due to queue limits, monitor collector metrics like:
  - `otelcol_processor_batch_dropped_spans`
  - `otelcol_exporter_send_failed_spans`
- Ensure the OTEL Collector has adequate buffer and batch size configuration to handle peak loads

## Backend Selection: What Should You Use?

Choosing the right observability backend depends on your environment and goals. Here's a quick comparison:

| Backend       | Best For                 | Notes                                                                                                    |
| ------------- | ------------------------ | -------------------------------------------------------------------------------------------------------- |
| **Last9**     | Production observability | Correlates traces with metrics and logs. Managed, scalable, and built for high-cardinality environments. |
| **Jaeger**    | Local testing            | Zero-config Docker setup. Great for validating traces during development.                                |
| **Tempo**     | Scalable storage         | Works well with Grafana dashboards. Good choice for self-hosted setups.                                  |
| **Honeycomb** | Sampling and debugging   | Strong UI for deep filtering and sampling strategies. Popular for debugging distributed systems.         |

## Final Thoughts

Enabling tracing for Kong Gateway closes a major observability gap. With a single plugin and minimal configuration, you can capture detailed request traces from the moment traffic enters your infrastructure.

- No code changes needed
- Supports any OTLP-compatible backend
- Works well in both local and production setups

And if you're looking to discuss specific use cases, [our Discord community is open](https://discord.com/invite/W8gMppQC4b), we’ve got a dedicated channel where you can connect with other developers and share what you’re building.