Enable Kong Gateway Tracing in 5 Minutes

Kong Gateway is a popular API gateway that sits at the edge of your infrastructure, routing and shaping traffic across microservices. It’s fast, pluggable, and battle-tested, but for many teams, it remains a black box.

You might have OpenTelemetry set up across your application stack. Traces flow from your app servers, databases, and third-party APIs.

But the moment a request enters through Kong, observability drops off. There’s no visibility into plugins, proxy logic, or where time is being spent before the request even hits your backend.

This blog walks through how to enable native OpenTelemetry tracing in Kong Gateway using its built-in plugin. The setup takes about 5 minutes, works with any OTLP-compatible backend, and doesn’t require writing any application code.

Why Trace the API Gateway?

Kong is often the first system that handles incoming requests, yet it's rarely instrumented for tracing. This creates a visibility gap at the very entry point of your system.

Without gateway tracing:

• Latency issues become ambiguous—was it DNS? A plugin? Upstream slowness?
• Errors like 504s offer no clues about where the timeout occurred
• Observability tools show partial traces starting after the request has passed through Kong

With gateway tracing:

• You see the full lifecycle of a request, starting at the ingress
• You can pinpoint latency introduced by specific plugins (like auth or rate-limiting)
• You can correlate upstream service errors with Kong retries, proxying delays, and failures

Here’s a quick example:

Without tracing:
“API is slow. It could be the app, database, or Kong. Start checking everything.”

With tracing:
“Auth plugin on Kong is adding 200ms to every request. Optimize JWT validation.”

Adding tracing at this layer helps close the loop, especially in production systems that rely on Kong for routing, auth, security, and more.

Prerequisites

Before you start, ensure the following:

• Kong Gateway v3.1 or later
• Access to the Kong Admin API (http://localhost:8001 by default)
• An OpenTelemetry-compatible backend or collector
  (e.g., Last9, Jaeger, Tempo)
• Tracing enabled via an environment variable

Kong does not emit traces by default. You need to explicitly enable tracing and tell Kong which phases to instrument.

Set the following environment variable before starting Kong:

export KONG_TRACING_INSTRUMENTATIONS="all"

This instructs Kong to generate spans for all instrumented phases (router, balancer, plugins, proxy, etc.). You can scope this more narrowly if needed.

Where to set this, depending on your deployment:

environment:
  KONG_TRACING_INSTRUMENTATIONS: "all"

env:
  - name: KONG_TRACING_INSTRUMENTATIONS
    value: "all"

Environment=KONG_TRACING_INSTRUMENTATIONS=all

Important: Restart Kong after setting this variable. Kong will not emit any traces unless this is set at startup, even if the plugin is correctly configured.

Set Up Tracing with the OpenTelemetry Plugin

Kong provides a native plugin that exports traces in OpenTelemetry format. This plugin can be applied globally or to individual routes or services.

Configure the Plugin Globally

This setup sends traces to a collector running at http://otel-collector:4318/v1/traces.

curl -X POST http://localhost:8001/plugins \
  --data "name=opentelemetry" \
  --data "config.endpoint=http://otel-collector:4318/v1/traces"

To enable it on a specific service only:

curl -X POST http://localhost:8001/plugins \
  --data "name=opentelemetry" \
  --data "service.id=SERVICE_ID" \
  --data "config.endpoint=http://otel-collector:4318/v1/traces"

If your collector requires authentication, include headers like this:

--data "config.headers.Authorization=Bearer ${OTEL_TOKEN}"

Note: Never hardcode tokens in production. Use environment variables or Kong’s Vault integration.

Verify Trace Export Is Working

Once the plugin is active and Kong is restarted, send a request through the gateway:

curl http://localhost:8000/your-api-endpoint

Then check your trace backend.

In Last9:

• Go to Traces
• Search for the service name kong
• Look for spans like kong:ingress, kong:plugins:<plugin-name>, kong:proxy, or kong:upstream:<service>

In Jaeger:

• Open http://localhost:16686
• Select the kong service
• View spans and breakdowns per phase

You should see a clear sequence of spans showing:

• How long Kong spent processing the request
• How much time each plugin took
• Upstream response time and status

Local Setup Using Docker Compose

To try gateway tracing locally, here's a complete Docker Compose setup that runs:

• Kong Gateway (with OpenTelemetry tracing enabled)
• OpenTelemetry Collector (to receive spans from Kong and forward them to Last9)

This gives you a working setup to see traces from Kong in your Last9 instance in just a few steps.

Step 1: Create the Docker Compose File

Create a new file called docker-compose.yaml in your working directory and add the following:

version: '3.8'

services:
  kong:
    image: kong:3.2
    environment:
      KONG_DATABASE: "off"
      KONG_ADMIN_LISTEN: "0.0.0.0:8001"
      KONG_PROXY_LISTEN: "0.0.0.0:8000"
      KONG_TRACING_INSTRUMENTATIONS: "all"
    ports:
      - "8000:8000"   # Kong proxy
      - "8001:8001"   # Kong Admin API
    depends_on:
      - otel-collector

  otel-collector:
    image: otel/opentelemetry-collector:latest
    command: ["--config=/etc/otel/config.yaml"]
    volumes:
      - ./otel-config.yaml:/etc/otel/config.yaml
    ports:
      - "4318:4318"   # OTLP HTTP receiver

Step 2: Create the OpenTelemetry Collector Config

In the same directory, create a file named otel-config.yaml with the following contents. Replace YOUR_API_KEY with your actual Last9 token.

receivers:
  otlp:
    protocols:
      http:

exporters:
  otlphttp:
    endpoint: https://otel.last9.io/v1/traces
    headers:
      Authorization: "Bearer YOUR_API_KEY"

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp]

This config tells the OTEL Collector to accept spans from Kong over HTTP and forward them securely to Last9.

Security Note: Do not hardcode tokens in production. Use Docker secrets, environment variables, or Kong's Vault integration.

Step 3: Start the Stack

Bring everything up with:

docker-compose up -d

This will start Kong and the OTEL Collector. Kong will be listening on ports 8000 (proxy) and 8001 (Admin API).

Step 4: Enable the OpenTelemetry Plugin in Kong

Once Kong is running, use the Admin API to configure the OpenTelemetry plugin:

curl -X POST http://localhost:8001/plugins \
  --data "name=opentelemetry" \
  --data "config.endpoint=http://otel-collector:4318/v1/traces"

This registers the plugin globally. You can also scope it to a specific service or route if needed.

Step 5: Test the Setup

Send a test request through Kong:

curl http://localhost:8000/your-api-endpoint

If you haven’t configured a service yet, a 404 is expected. That’s fine, it still generates spans for ingress, router, and proxy phases.

Step 6: View Traces in Last9

Head to your Last9 Traces UI :

• Search for the service name kong
• Use filters like span.name starts with kong: or check for spans such as:
  • kong:ingress
  • kong:router
  • kong:proxy
  • kong:plugins:<plugin-name>
  • kong:upstream:<service-name>

This gives you complete visibility into how Kong is handling your request: from routing decisions and plugin latency to upstream service response times.

What Happens if the Collector Fails?

If Kong cannot reach the collector endpoint, it silently drops spans. Request handling is not blocked or affected. Kong continues operating normally, which is by design to prevent observability issues from impacting request flow.

However, you won’t see trace data until the collector becomes reachable again.

For production:

• Use batch exporters with retries and backoff
• Monitor collector health
• Alert on dropped spans or disconnected exporters

Performance and Overhead

The OpenTelemetry plugin in Kong Gateway is built to minimize impact while enabling detailed trace visibility. It operates asynchronously and is suitable for production traffic with moderate to high request volumes.

Latency Impact

• Typical overhead: ~1–2ms per request under moderate load (up to 100–200 RPS)
• Spans are batched and exported in the background, so request handling remains non-blocking
• If the internal span queue fills (e.g., due to downstream collector slowness), new spans may be dropped silently to avoid blocking

CPU & Resource Usage

• Negligible CPU impact under normal operating conditions
• Overhead becomes noticeable only when pushing thousands of RPS without proper sampling or throttling
• The plugin itself does not perform heavy computation; most processing happens in the collector or backend

Reducing Overhead in Production

To optimize performance further:

• Use trace sampling:
  • Prefer collector-side sampling (e.g., tail-based sampling in OTEL Collector)
  • For high-throughput systems, apply upfront sampling at Kong using rate or probabilistic options once supported
• Limit instrumentation scope by setting KONG_TRACING_INSTRUMENTATIONS to only the necessary phases:

export KONG_TRACING_INSTRUMENTATIONS="proxy,plugins"

Monitoring Collector Pressure

• To detect span drops due to queue limits, monitor collector metrics like:
  • otelcol_processor_batch_dropped_spans
  • otelcol_exporter_send_failed_spans
• Ensure the OTEL Collector has adequate buffer and batch size configuration to handle peak loads

Backend Selection: What Should You Use?

Choosing the right observability backend depends on your environment and goals. Here's a quick comparison:

Final Thoughts

Enabling tracing for Kong Gateway closes a major observability gap. With a single plugin and minimal configuration, you can capture detailed request traces from the moment traffic enters your infrastructure.

• No code changes needed
• Supports any OTLP-compatible backend
• Works well in both local and production setups