Auto-Instrument Your Applications Using OTel Injector

As distributed systems scale, maintaining manual instrumentation across services quickly becomes unsustainable. The OTel Injector addresses this by automatically attaching OpenTelemetry instrumentation to applications, no code changes needed.

This blog covers how the OTel Injector works, how it integrates with Linux environments, and how to set it up for consistent telemetry across your stack.

What is the OTel Injector?

The OpenTelemetry Injector is a host-based tool that automatically adds OpenTelemetry instrumentation to your applications, with no need to touch source code or rebuild containers.

It runs on any Linux machine and is stable enough for production use. Originally developed by Splunk, it's now part of the OpenTelemetry project.

This comes in handy when you're working across services in different languages, Java here, Python there, maybe a bit of Node.js or .NET thrown in. Instead of figuring out how to instrument each one manually, the OTel Injector handles it for you at the host level.

Key Benefits:

• No code changes or redeployments
• Works across Java, Node.js, Python, and .NET
• Reliable enough for production workloads
• Lower effort than manual instrumentation, especially in polyglot setups

How the OTel Injector Works

The OTel Injector instruments applications by injecting environment variables during process startup. It detects the language runtime and applies the right configuration automatically, without modifying the app code or container image.

There are two supported methods:

1. LD_PRELOAD Hook

This approach relies on Linux's dynamic linking system. A shared library (libotelinject.so) is loaded using the LD_PRELOAD mechanism. Once active, the library:

• Intercepts calls to getenv() from the runtime
• Modifies or appends environment variables needed for OpenTelemetry
• Adds the correct agent configuration for the detected language runtime

This method works for most dynamically linked applications and requires minimal system-level setup.

2. Systemd Drop-ins

For applications managed by systemd, the OTel Injector uses drop-in files to configure environment variables per service. This avoids using LD_PRELOAD and integrates cleanly with how services are typically deployed on Linux.

Examples:

Java:

JAVA_TOOL_OPTIONS=-javaagent:/usr/lib/opentelemetry/otel-javaagent.jar

Node.js:

NODE_OPTIONS=-r /usr/lib/opentelemetry/otel-js/node_modules/@opentelemetry/auto-instrumentations-node/register

.NET: Sets CORECLR_ENABLE_PROFILING, CORECLR_PROFILER, DOTNET_ADDITIONAL_DEPS, and related variables

Both options achieve the same goal: automatic instrumentation at startup. The choice depends on your deployment model and control over service configurations.

Install and Configure the OTel Injector on Linux

The opentelemetry-injector package installs everything you need to get started with automatic instrumentation: the required OpenTelemetry agents, the libotelinject.so shared library, and sample configuration files.

Install the Injector Package

Download and install the appropriate package for your Linux distribution:

Debian/Ubuntu:

# Download the package and its checksum
wget https://github.com/open-telemetry/opentelemetry-injector/releases/latest/download/opentelemetry-injector_amd64.deb
wget https://github.com/open-telemetry/opentelemetry-injector/releases/latest/download/opentelemetry-injector_amd64.deb.sha256

# Verify package integrity (recommended)
sha256sum -c opentelemetry-injector_amd64.deb.sha256

# Install the package
sudo dpkg -i opentelemetry-injector_amd64.deb

RHEL/CentOS:

# Download the package and its checksum
wget https://github.com/open-telemetry/opentelemetry-injector/releases/latest/download/opentelemetry-injector.x86_64.rpm
wget https://github.com/open-telemetry/opentelemetry-injector/releases/latest/download/opentelemetry-injector.x86_64.rpm.sha256

# Verify package integrity (recommended)
sha256sum -c opentelemetry-injector.x86_64.rpm.sha256

# Install the package
sudo rpm -i opentelemetry-injector.x86_64.rpm

Method 1: Enabling Instrumentation with LD_PRELOAD

To instrument all supported processes on the host, add the OTel Injector's shared library to the system preload list:

echo /usr/lib/opentelemetry/libotelinject.so | sudo tee -a /etc/ld.so.preload

Note: This method affects all processes on the system. It's recommended to validate changes in a staging or non-production environment before rolling out system-wide.

Method 2: Configuring systemd Services

If your services are managed by systemd, you can enable instrumentation by applying a system-wide drop-in configuration:

# Create the systemd drop-in directory
sudo mkdir -p /usr/lib/systemd/system.conf.d/

# Copy the sample configuration
sudo cp /usr/lib/opentelemetry/examples/systemd/00-otelinject-instrumentation.conf \
        /usr/lib/systemd/system.conf.d/

# Reload systemd to apply changes
sudo systemctl daemon-reload

This method allows more control by limiting instrumentation to systemd-managed services, without touching global process startup behavior.

Runtime-Specific Configurations

The package also installs default config files for each supported language runtime under /etc/opentelemetry/otelinject/. These files define the environment variables needed to activate the corresponding instrumentation agents:

Java:

# /etc/opentelemetry/otelinject/java.conf
JAVA_TOOL_OPTIONS=-javaagent:/usr/lib/opentelemetry/otel-javaagent.jar

Node.js:

# /etc/opentelemetry/otelinject/node.conf
NODE_OPTIONS=-r /usr/lib/opentelemetry/otel-js/node_modules/@opentelemetry/auto-instrumentations-node/register

Python:

# /etc/opentelemetry/otelinject/python.conf
PYTHONPATH=/usr/lib/opentelemetry/otel-python
OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=

You can modify these files to customize agent behavior or override default options based on your environment.

Advanced Configuration Options

Once instrumentation is in place, the next step is telling the agents where to send telemetry. This is done using environment variables that control exporter settings, service metadata, and signal types.

Set Exporter and Metadata Variables

Each OpenTelemetry agent looks for a standard set of environment variables. These control the destination, protocol, and metadata for traces, metrics, and logs.

Here's a basic setup:

# Collector endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# Export protocol (grpc or http/protobuf)
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

# Service metadata
OTEL_SERVICE_NAME=my-application
OTEL_RESOURCE_ATTRIBUTES=service.version=1.0.0,deployment.environment=production

# Enable specific exporters
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp

These variables can go directly into runtime-specific files under /etc/opentelemetry/otelinject/, or be set globally depending on how you've enabled the OTel Injector.

Important: Configuration of the respective agents is supported by adding/updating the following environment variables in each of these files (any environment variable not in this list will be ignored): OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_LOGS_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_RESOURCE_ATTRIBUTES, and OTEL_SERVICE_NAME.

Manage Configuration in Dynamic Environments

For dynamic setups, like containers or CI/CD pipelines, you may want to avoid hardcoding values. Instead, you can template the config files and inject environment variables at runtime.

Example: Templated Java config using environment fallbacks

cat <<EOF > /etc/opentelemetry/otelinject/java.conf
JAVA_TOOL_OPTIONS=-javaagent:/usr/lib/opentelemetry/otel-javaagent.jar
OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_COLLECTOR_ENDPOINT:-http://localhost:4317}
OTEL_SERVICE_NAME=${SERVICE_NAME:-unknown-service}
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=${ENVIRONMENT:-development}
EOF

This makes it easier to reuse the same config across environments—just set the right environment variables before startup.

Language-Specific Instrumentation

The OTel Injector supports auto-instrumentation across major language runtimes. Here's what you need to know for each.

Java

Instrumentation is handled through the JAVA_TOOL_OPTIONS environment variable, which the JVM picks up automatically. This works reliably across:

• Spring Boot apps
• Micronaut and Quarkus services
• Traditional WAR-based deployments
• Containerized Java workloads

If your application already sets JAVA_TOOL_OPTIONS, double-check for conflicts before enabling the OTel Injector.

Node.js

Node.js uses the NODE_OPTIONS flag to preload the OpenTelemetry instrumentation module. Supported setups include:

• Express.js apps
• NestJS services
• PM2-managed processes
• Serverless functions (though some runtimes may limit environment overrides)

Performance note: In most cases, the overhead is low, typically between 2–5%.

Python

Python apps require the OpenTelemetry auto-instrumentation package to be installed. The OTel Injector sets up the necessary variables automatically for:

• Django
• Flask
• FastAPI
• Celery

If you're using virtual environments, make sure the agent path is reachable within that context.

.NET

For .NET, the OTel Injector configures multiple environment variables required for CLR profiling. This includes setting:

• CORECLR_ENABLE_PROFILING
• CORECLR_PROFILER
• DOTNET_STARTUP_HOOKS, among others

There's no need to wire these manually; the OTel Injector handles setup across both .NET Core and Framework applications.

Monitoring and Troubleshooting

You'll want to make sure Opentelemetry Injector is instrumenting processes and exporting telemetry correctly. Here's how to check that everything's wired up correctly and what to look into if something breaks.

How to Verify It's Working

Start by confirming that the OTel Injector's shared library is loaded into your application process:

ldd /proc/<PID>/exe | grep otelinject

Then check if the right OpenTelemetry environment variables are present:

cat /proc/<PID>/environ | tr '\0' '\n' | grep OTEL

And finally, check your service logs for any signs that telemetry is being initialized:

journalctl -u your-service | grep -i opentelemetry

If those checks pass, you should be seeing data in your backend.

Common Issues and How to Fix Them

No data showing up

First, make sure the OpenTelemetry Collector is actually running and listening on the endpoint you've configured. Network issues or mismatched protocols (like grpc vs http/protobuf) are often the culprit. Also, confirm that the service has access to the collector and isn't blocked by firewall rules or container networking.

App crashes or fails to start

This usually points to a conflict in environment variables. For example, if JAVA_TOOL_OPTIONS is already being used elsewhere, it can override the OTel Injector's settings. Version mismatches between the instrumentation agent and the runtime can also cause failures. Logs will usually show errors around agent loading or startup hooks.

High memory usage

Auto-instrumentation adds some overhead, especially if you're capturing a lot of spans. If memory spikes after rollout, start by reducing the sampling rate. For Java, keep an eye on GC behavior; some agents allocate more aggressively during startup. Also, check your resource attributes; long strings or too many custom tags can add up fast.

Monitor Performance Impact

You can track instrumentation overhead using basic system tools:

# CPU and memory usage
top -p $(pgrep -f your-application)

# Check if the app is connecting to the collector
netstat -an | grep :4317

# Collector export metrics (if exposed)
curl http://localhost:8888/metrics | grep otelcol_exporter

Integrate with Your Existing Observability Stack

The OTel Injector can be configured to forward telemetry to any backend that supports OpenTelemetry, including Prometheus for metrics and various APM platforms for end-to-end observability across metrics, traces, and logs.

Send Metrics to Prometheus

To expose metrics for Prometheus to scrape, use the built-in Prometheus exporter:

OTEL_METRICS_EXPORTER=prometheus
OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0
OTEL_EXPORTER_PROMETHEUS_PORT=9464

Prometheus can scrape metrics from http://<host>:9464/metrics. These settings can be added to your language-specific config files under /etc/opentelemetry/otelinject/ or passed through systemd for more control.

Exporting Telemetry to Last9

Last9 supports OTLP effortlessly. To send traces, metrics, and logs to Last9, point your OpenTelemetry agents to your project’s OTLP endpoint and set the appropriate authentication headers.

OTEL_EXPORTER_OTLP_ENDPOINT=https://api.last9.io/otlp/v1
OTEL_EXPORTER_OTLP_HEADERS="last9-api-key=your-api-key"

This setup works across all supported runtimes. Once configured, you’ll be able to view traces in real time, correlate them with metrics, and debug issues directly from the Last9 dashboard.

If you’re using multiple environments (e.g., staging and production), set OTEL_RESOURCE_ATTRIBUTES to tag telemetry with relevant metadata:

OTEL_RESOURCE_ATTRIBUTES=service.name=my-app,deployment.environment=production

When to Use the OTel Injector

The OTel Injector is built for cases where you need observability across a large, mixed set of services, fast. It works especially well when you're dealing with multiple languages, older applications, or teams that don't have the time (or access) to wire up manual instrumentation.

You'll get the most value from the OTel Injector when:

• You're managing a polyglot environment with Java, Node.js, Python, and .NET services
• The codebase is legacy or vendor-owned, and modifying it isn't an option
• You want consistent instrumentation across teams without enforcing how it's done in code
• You need to roll out telemetry quickly across dozens or hundreds of services

That said, the OTel Injector isn't always the right tool. You may want to consider manual instrumentation or SDK-based approaches when:

• You need to add custom spans, attributes, or business-specific context
• You're in environments where root access isn't available (e.g., certain containers or PaaS setups)
• Your app runs on a runtime that isn't supported by the OTel Injector
• You need precise control over what gets instrumented and how

The OTel Injector is great for bootstrapping observability across your stack, but you can always pair it with more targeted instrumentation later as your needs evolve.

Final Thoughts

The OTel Injector simplifies the rollout of observability across large, multi-language environments. It removes the need for manual instrumentation, works across standard runtimes, and is stable enough for production.

For many teams, it's the quickest way to get consistent telemetry in place, especially when dealing with legacy services or limited access to source code.

• Join the conversation in the #otel-injector channel on OpenTelemetry Slack
• Check out the opentelemetry-injector repository