OpenResty

Monitor your OpenResty reverse proxy with full-stack observability — distributed traces with per-upstream child spans, custom Lua metrics, and structured access logs — all forwarded to Last9 via OpenTelemetry Collector. No module compilation required.

Prerequisites

• OpenResty 1.21+ (includes LuaJIT 2.1)
• OpenResty Package Manager (opm) — bundled with OpenResty
• OpenTelemetry Collector

How it works

Each HTTP request proxied through OpenResty produces:

• A SERVER span covering the full request lifecycle with OTel HTTP semconv v1.23+ attributes
• One CLIENT child span per upstream attempt — retries and failovers appear as separate spans with reconstructed timings
• exception span events for 4xx and 5xx with typed error classification
• Access and error logs with OTel semconv attribute names
• Prometheus metrics: request counters, latency histograms, upstream response time histograms, active connection gauge

Trace context (traceparent + tracestate) is extracted from incoming requests and injected into upstream calls. Sampling is configurable via OTEL_TRACES_SAMPLER.

Installation

1. Install Lua dependencies

   • opm
   • Docker

   opm get knyar/nginx-lua-prometheus ledgetech/lua-resty-http

   lua-resty-http is required for the async OTLP export — it is not included in the OpenResty base image.

2. Configure nginx.conf

   Add to the http {} block:

   http {
       lua_shared_dict prometheus_metrics 10m;
       lua_package_path "/usr/local/openresty/site/lualib/?.lua;;";
   
       # Required for hostname resolution inside ngx.timer.at callbacks.
       # 127.0.0.11 is Docker's embedded DNS; replace with your resolver otherwise.
       resolver 127.0.0.11 valid=30s ipv6=off;
   
       init_worker_by_lua_block {
           require("metrics_init").init()
       }
   
       log_format json_combined escape=json '{'
           '"timestamp":"$time_iso8601",'
           '"remote_addr":"$remote_addr",'
           '"request_method":"$request_method",'
           '"request_uri":"$request_uri",'
           '"server_protocol":"$server_protocol",'
           '"status":$status,'
           '"body_bytes_sent":$body_bytes_sent,'
           '"request_length":$request_length,'
           '"request_time":$request_time,'
           '"upstream_response_time":"$upstream_response_time",'
           '"upstream_connect_time":"$upstream_connect_time",'
           '"upstream_addr":"$upstream_addr",'
           '"upstream_status":"$upstream_status",'
           '"upstream_cache_status":"$upstream_cache_status",'
           '"http_user_agent":"$http_user_agent",'
           '"http_referer":"$http_referer",'
           '"trace_id":"$otel_trace_id",'
           '"span_id":"$otel_span_id"'
       '}';
   
       access_log /var/log/nginx/access.log json_combined;
       include /etc/nginx/conf.d/*.conf;
   }

3. Add tracing hooks to your server block

   server {
       listen 80;
   
       # Required for log-trace correlation — populated by start_span()
       set $otel_trace_id "";
       set $otel_span_id  "";
   
       access_by_lua_block {
           require("otel_tracer").start_span()
       }
   
       log_by_lua_block {
           require("otel_tracer").finish_span()
           require("metrics_init").record()
       }
   
       location / {
           proxy_pass http://upstream-app;
           proxy_http_version 1.1;
           proxy_set_header Host             $host;
           proxy_set_header X-Real-IP        $remote_addr;
           proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
           # traceparent + tracestate injected automatically by start_span()
       }
   
       location /health {
           access_log off;
           return 200 '{"status":"ok"}';
           add_header Content-Type application/json;
       }
   }

4. Add a metrics and status server block

   Expose on a separate port not reachable from the internet:

   server {
       listen 9145;
   
       location /metrics {
           access_log  off;
           content_by_lua_block {
               require("metrics_init").collect()
           }
       }
   
       location /nginx_status {
           stub_status on;
           access_log  off;
           allow 127.0.0.1;
           allow 10.0.0.0/8;
           allow 172.0.0.0/8;
           allow 192.168.0.0/16;
           deny  all;
       }
   }

5. Configure the OTel Collector

   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: 0.0.0.0:4317
         http:
           endpoint: 0.0.0.0:4318
   
     prometheus/lua:
       config:
         scrape_configs:
           - job_name: openresty_lua
             scrape_interval: 30s
             static_configs:
               - targets: ["localhost:9145"]
             metrics_path: /metrics
   
     nginx:
       endpoint: "http://localhost:9145/nginx_status"
       collection_interval: 30s
   
     filelog/access:
       include:
         - /var/log/nginx/access.log
       operators:
         - type: json_parser
         - type: move
           from: attributes.request_method
           to: attributes["http.request.method"]
         - type: move
           from: attributes.request_uri
           to: attributes["url.path"]
         - type: move
           from: attributes.server_protocol
           to: attributes["network.protocol.version"]
         - type: move
           from: attributes.status
           to: attributes["http.response.status_code"]
         - type: move
           from: attributes.request_length
           to: attributes["http.request.body.size"]
         - type: move
           from: attributes.body_bytes_sent
           to: attributes["http.response.body.size"]
         - type: move
           from: attributes.request_time
           to: attributes["http.request.duration"]
         - type: move
           from: attributes.upstream_addr
           to: attributes["server.address"]
         - type: move
           from: attributes.upstream_cache_status
           to: attributes["http.cache_status"]
         - type: move
           from: attributes.remote_addr
           to: attributes["client.address"]
         - type: move
           from: attributes.trace_id
           to: attributes["trace_id"]
         - type: add
           field: attributes["service.name"]
           value: "openresty"
         - type: add
           field: attributes["log.source"]
           value: "nginx.access"
   
     filelog/error:
       include:
         - /var/log/nginx/error.log
       operators:
         - type: regex_parser
           regex: '^(?P<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?P<level>\w+)\] (?P<pid>\d+)#(?P<tid>\d+): (?P<message>.*)'
           timestamp:
             parse_from: attributes.timestamp
             layout: "%Y/%m/%d %H:%M:%S"
             layout_type: strptime
         - type: severity_parser
           parse_from: attributes.level
           mapping:
             debug: debug
             info: info
             notice: info2
             warn: warn
             error: error
             crit: fatal
             alert: fatal
             emerg: fatal
         - type: add
           field: attributes["service.name"]
           value: "openresty"
         - type: add
           field: attributes["log.source"]
           value: "nginx.error"
   
   processors:
     memory_limiter:
       check_interval: 5s
       limit_percentage: 85
       spike_limit_percentage: 15
     batch:
       timeout: 5s
       send_batch_size: 100000
     resource:
       attributes:
         - key: service.name
           value: openresty
           action: upsert
   
   exporters:
     otlp/last9:
       endpoint: ${env:LAST9_OTLP_ENDPOINT}
       headers:
         Authorization: ${env:LAST9_OTLP_AUTH}
   
   service:
     pipelines:
       traces:
         receivers:  [otlp]
         processors: [memory_limiter, batch]
         exporters:  [otlp/last9]
       metrics:
         receivers:  [prometheus/lua, nginx]
         processors: [memory_limiter, resource, batch]
         exporters:  [otlp/last9]
       logs:
         receivers:  [filelog/access, filelog/error]
         processors: [memory_limiter, resource, batch]
         exporters:  [otlp/last9]

6. Set environment variables

   export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
   export OTEL_SERVICE_NAME=openresty
   export OTEL_SERVICE_VERSION=1.0.0
   export DEPLOYMENT_ENVIRONMENT=production
   
   # Sampling strategy (default follows incoming traceparent; new roots always sample)
   export OTEL_TRACES_SAMPLER=parentbased_always_on
   # export OTEL_TRACES_SAMPLER_ARG=0.1   # for traceid_ratio / parentbased_traceid_ratio

7. Reload OpenResty

   nginx -t && nginx -s reload

Verification

# Send test requests including error scenarios
curl http://localhost/get
curl http://localhost/status/404
curl http://localhost/status/500

# Confirm Lua metrics endpoint
curl http://localhost:9145/metrics | grep openresty_http

# Confirm nginx stub_status
curl http://localhost:9145/nginx_status

# Collector health
curl http://localhost:13133

Traces appear in Last9 within seconds. Each request produces a SERVER span and one CLIENT child span per upstream attempt — retried requests show each attempt as a separate span.

Environment Variables

Sampler options:

Troubleshooting

No traces in Last9

• Confirm the collector is running: curl http://localhost:13133
• Check for Lua errors: tail -f /var/log/nginx/error.log | grep '\[otel\]'
• Verify OTEL_EXPORTER_OTLP_ENDPOINT points to the collector, not Last9 directly
• Ensure resolver directive is set in nginx.conf — without it, hostname resolution fails inside timer callbacks

no resolver defined error in logs

• Add resolver 127.0.0.11 valid=30s ipv6=off; to the http {} block in nginx.conf
• proxy_pass resolves DNS at config load time; resty.http inside ngx.timer.at requires a runtime resolver

No Lua metrics

• Verify lua_shared_dict prometheus_metrics 10m is in the http {} block
• Test: curl http://localhost:9145/metrics | grep openresty_http
• Check opm list shows knyar/nginx-lua-prometheus is installed

No upstream child spans

• CLIENT spans are only created when $upstream_response_time is populated — direct return statements and content_by_lua locations produce only the SERVER span

services:
  openresty:
    build: .
    ports:
      - "80:80"
      - "9145:9145"
    environment:
      - OTEL_SERVICE_NAME=openresty
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
      - DEPLOYMENT_ENVIRONMENT=production
      - OTEL_TRACES_SAMPLER=parentbased_always_on
    volumes:
      - nginx_logs:/var/log/nginx
    depends_on:
      - otel-collector

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.144.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    environment:
      - LAST9_OTLP_ENDPOINT=${LAST9_OTLP_ENDPOINT}
      - LAST9_OTLP_AUTH=${LAST9_OTLP_AUTH}
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro
      - nginx_logs:/var/log/nginx:ro
    ports:
      - "4318:4318"

volumes:
  nginx_logs: