# OpenResty > Instrument OpenResty (nginx + LuaJIT) with OpenTelemetry to send traces, metrics, and logs to Last9 Source: https://last9.io/docs/integrations/web-and-cdn/openresty/ Monitor your OpenResty reverse proxy with full-stack observability — distributed traces with per-upstream child spans, custom Lua metrics, and structured access logs — all forwarded to Last9 via OpenTelemetry Collector. No module compilation required. ## Prerequisites - OpenResty 1.21+ (includes LuaJIT 2.1) - OpenResty Package Manager (`opm`) — bundled with OpenResty - OpenTelemetry Collector ## How it works Each HTTP request proxied through OpenResty produces: - A **SERVER span** covering the full request lifecycle with OTel HTTP semconv v1.23+ attributes - One **CLIENT child span per upstream attempt** — retries and failovers appear as separate spans with reconstructed timings - `exception` span events for 4xx and 5xx with typed error classification - Access and error logs with OTel semconv attribute names - Prometheus metrics: request counters, latency histograms, upstream response time histograms, active connection gauge Trace context (`traceparent` + `tracestate`) is extracted from incoming requests and injected into upstream calls. Sampling is configurable via `OTEL_TRACES_SAMPLER`. ## Installation 1. **Install Lua dependencies** **opm** ```bash opm get knyar/nginx-lua-prometheus ledgetech/lua-resty-http ``` **Docker** ```dockerfile FROM openresty/openresty:1.25.3.1-bullseye-fat RUN opm get knyar/nginx-lua-prometheus ledgetech/lua-resty-http COPY nginx.conf /usr/local/openresty/nginx/conf/nginx.conf COPY conf.d/ /etc/nginx/conf.d/ COPY lua/ /usr/local/openresty/site/lualib/ EXPOSE 80 9145 CMD ["/usr/local/openresty/bin/openresty", "-g", "daemon off;"] ``` `lua-resty-http` is required for the async OTLP export — it is not included in the OpenResty base image. 2. **Configure `nginx.conf`** Add to the `http {}` block: ```nginx http { lua_shared_dict prometheus_metrics 10m; lua_package_path "/usr/local/openresty/site/lualib/?.lua;;"; # Required for hostname resolution inside ngx.timer.at callbacks. # 127.0.0.11 is Docker's embedded DNS; replace with your resolver otherwise. resolver 127.0.0.11 valid=30s ipv6=off; init_worker_by_lua_block { require("metrics_init").init() } log_format json_combined escape=json '{' '"timestamp":"$time_iso8601",' '"remote_addr":"$remote_addr",' '"request_method":"$request_method",' '"request_uri":"$request_uri",' '"server_protocol":"$server_protocol",' '"status":$status,' '"body_bytes_sent":$body_bytes_sent,' '"request_length":$request_length,' '"request_time":$request_time,' '"upstream_response_time":"$upstream_response_time",' '"upstream_connect_time":"$upstream_connect_time",' '"upstream_addr":"$upstream_addr",' '"upstream_status":"$upstream_status",' '"upstream_cache_status":"$upstream_cache_status",' '"http_user_agent":"$http_user_agent",' '"http_referer":"$http_referer",' '"trace_id":"$otel_trace_id",' '"span_id":"$otel_span_id"' '}'; access_log /var/log/nginx/access.log json_combined; include /etc/nginx/conf.d/*.conf; } ``` 3. **Add tracing hooks to your server block** ```nginx server { listen 80; # Required for log-trace correlation — populated by start_span() set $otel_trace_id ""; set $otel_span_id ""; access_by_lua_block { require("otel_tracer").start_span() } log_by_lua_block { require("otel_tracer").finish_span() require("metrics_init").record() } location / { proxy_pass http://upstream-app; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # traceparent + tracestate injected automatically by start_span() } location /health { access_log off; return 200 '{"status":"ok"}'; add_header Content-Type application/json; } } ``` 4. **Add a metrics and status server block** Expose on a separate port not reachable from the internet: ```nginx server { listen 9145; location /metrics { access_log off; content_by_lua_block { require("metrics_init").collect() } } location /nginx_status { stub_status on; access_log off; allow 127.0.0.1; allow 10.0.0.0/8; allow 172.0.0.0/8; allow 192.168.0.0/16; deny all; } } ``` 5. **Configure the OTel Collector** ```yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 prometheus/lua: config: scrape_configs: - job_name: openresty_lua scrape_interval: 30s static_configs: - targets: ["localhost:9145"] metrics_path: /metrics nginx: endpoint: "http://localhost:9145/nginx_status" collection_interval: 30s filelog/access: include: - /var/log/nginx/access.log operators: - type: json_parser - type: move from: attributes.request_method to: attributes["http.request.method"] - type: move from: attributes.request_uri to: attributes["url.path"] - type: move from: attributes.server_protocol to: attributes["network.protocol.version"] - type: move from: attributes.status to: attributes["http.response.status_code"] - type: move from: attributes.request_length to: attributes["http.request.body.size"] - type: move from: attributes.body_bytes_sent to: attributes["http.response.body.size"] - type: move from: attributes.request_time to: attributes["http.request.duration"] - type: move from: attributes.upstream_addr to: attributes["server.address"] - type: move from: attributes.upstream_cache_status to: attributes["http.cache_status"] - type: move from: attributes.remote_addr to: attributes["client.address"] - type: move from: attributes.trace_id to: attributes["trace_id"] - type: add field: attributes["service.name"] value: "openresty" - type: add field: attributes["log.source"] value: "nginx.access" filelog/error: include: - /var/log/nginx/error.log operators: - type: regex_parser regex: '^(?P\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?P\w+)\] (?P\d+)#(?P\d+): (?P.*)' timestamp: parse_from: attributes.timestamp layout: "%Y/%m/%d %H:%M:%S" layout_type: strptime - type: severity_parser parse_from: attributes.level mapping: debug: debug info: info notice: info2 warn: warn error: error crit: fatal alert: fatal emerg: fatal - type: add field: attributes["service.name"] value: "openresty" - type: add field: attributes["log.source"] value: "nginx.error" processors: memory_limiter: check_interval: 5s limit_percentage: 85 spike_limit_percentage: 15 batch: timeout: 5s send_batch_size: 100000 resource: attributes: - key: service.name value: openresty action: upsert exporters: otlp/last9: endpoint: ${env:LAST9_OTLP_ENDPOINT} headers: Authorization: ${env:LAST9_OTLP_AUTH} service: pipelines: traces: receivers: [otlp] processors: [memory_limiter, batch] exporters: [otlp/last9] metrics: receivers: [prometheus/lua, nginx] processors: [memory_limiter, resource, batch] exporters: [otlp/last9] logs: receivers: [filelog/access, filelog/error] processors: [memory_limiter, resource, batch] exporters: [otlp/last9] ``` 6. **Set environment variables** ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 export OTEL_SERVICE_NAME=openresty export OTEL_SERVICE_VERSION=1.0.0 export DEPLOYMENT_ENVIRONMENT=production # Sampling strategy (default follows incoming traceparent; new roots always sample) export OTEL_TRACES_SAMPLER=parentbased_always_on # export OTEL_TRACES_SAMPLER_ARG=0.1 # for traceid_ratio / parentbased_traceid_ratio ``` 7. **Reload OpenResty** ```bash nginx -t && nginx -s reload ``` ## Verification ```bash # Send test requests including error scenarios curl http://localhost/get curl http://localhost/status/404 curl http://localhost/status/500 # Confirm Lua metrics endpoint curl http://localhost:9145/metrics | grep openresty_http # Confirm nginx stub_status curl http://localhost:9145/nginx_status # Collector health curl http://localhost:13133 ``` Traces appear in Last9 within seconds. Each request produces a SERVER span and one CLIENT child span per upstream attempt — retried requests show each attempt as a separate span. ## Environment Variables | Variable | Default | Description | | ----------------------------------- | ----------------------- | ------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` | OTel Collector OTLP HTTP endpoint | | `OTEL_SERVICE_NAME` | `openresty` | `service.name` resource attribute | | `OTEL_SERVICE_VERSION` | `1.0.0` | `service.version` resource attribute | | `DEPLOYMENT_ENVIRONMENT` | `production` | `deployment.environment` resource attribute | | `OTEL_TRACES_SAMPLER` | `parentbased_always_on` | Sampling strategy — see table below | | `OTEL_TRACES_SAMPLER_ARG` | `1.0` | Ratio for `traceid_ratio` / `parentbased_traceid_ratio` | | `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT` | `256` | Max characters for string span attributes | **Sampler options:** | Value | Behaviour | | --------------------------- | ------------------------------------------------------------------ | | `parentbased_always_on` | Respect parent sampled flag; new roots always sample **(default)** | | `parentbased_traceid_ratio` | Respect parent sampled flag; new roots use ratio | | `always_on` | Sample every request | | `always_off` | Drop all traces | | `traceid_ratio` | Probabilistic sampling, ignores parent flag | ## Troubleshooting **No traces in Last9** - Confirm the collector is running: `curl http://localhost:13133` - Check for Lua errors: `tail -f /var/log/nginx/error.log | grep '\[otel\]'` - Verify `OTEL_EXPORTER_OTLP_ENDPOINT` points to the collector, not Last9 directly - Ensure `resolver` directive is set in `nginx.conf` — without it, hostname resolution fails inside timer callbacks **`no resolver defined` error in logs** - Add `resolver 127.0.0.11 valid=30s ipv6=off;` to the `http {}` block in `nginx.conf` - `proxy_pass` resolves DNS at config load time; `resty.http` inside `ngx.timer.at` requires a runtime resolver **No Lua metrics** - Verify `lua_shared_dict prometheus_metrics 10m` is in the `http {}` block - Test: `curl http://localhost:9145/metrics | grep openresty_http` - Check `opm list` shows `knyar/nginx-lua-prometheus` is installed **No upstream child spans** - CLIENT spans are only created when `$upstream_response_time` is populated — direct `return` statements and `content_by_lua` locations produce only the SERVER span
Docker Compose example ```yaml services: openresty: build: . ports: - "80:80" - "9145:9145" environment: - OTEL_SERVICE_NAME=openresty - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 - DEPLOYMENT_ENVIRONMENT=production - OTEL_TRACES_SAMPLER=parentbased_always_on volumes: - nginx_logs:/var/log/nginx depends_on: - otel-collector otel-collector: image: otel/opentelemetry-collector-contrib:0.144.0 command: ["--config=/etc/otel-collector-config.yaml"] environment: - LAST9_OTLP_ENDPOINT=${LAST9_OTLP_ENDPOINT} - LAST9_OTLP_AUTH=${LAST9_OTLP_AUTH} volumes: - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro - nginx_logs:/var/log/nginx:ro ports: - "4318:4318" volumes: nginx_logs: ```