How OpenTelemetry Auto-Instrumentation Works

Most developers use auto-instrumentation as it’s meant to be used — run the Java agent, add NODE_OPTIONS, and telemetry starts flowing.
When it stops, though, figuring out why can be tricky. Maybe the agent didn’t load, maybe there’s a framework version mismatch, or something else entirely.

Understanding how auto-instrumentation works makes it easier to spot and fix these issues. The OpenTelemetry project recently published a great breakdown of the five core techniques that power it.

In this post, we walk through what each one does and where you might run into problems.

What Auto-Instrumentation Does

Automatic instrumentation refers to collecting telemetry without changing your application’s source code. Depending on the language, this can use techniques like bytecode injection, monkey patching, or eBPF.

To make sense of it, it helps to think in three layers:

• Techniques layer:
  The low-level mechanisms, such as bytecode injection, monkey patching, or eBPF that make auto-instrumentation possible.
• Instrumentation libraries:
  These use the techniques above to target specific frameworks or libraries — for example, Spring Boot (Java), Express.js (Node.js), or Laravel (PHP).
• Complete solutions:
  Full-featured agents like the OpenTelemetry Java agent bundle multiple instrumentation libraries together, adding configuration for exporters, samplers, and other telemetry components.

When something breaks, knowing which layer to check saves time:

• Technique layer issue: The agent itself isn’t loading.
• Instrumentation library issue: The framework version isn’t supported.
• Configuration issue: The exporter endpoint or environment variables are incorrect.

The Five Auto-Instrumentation Techniques

Auto-instrumentation uses specific techniques that hook into your runtime, compiler, or kernel. Each approach works differently and has different characteristics.

Monkey Patching: Runtime Function Replacement

You’ve probably seen this in action if you’ve used OpenTelemetry with JavaScript or Python.

In dynamic languages, the agent simply replaces a function at runtime with an instrumented version — one that measures execution time, collects metadata, and then calls the original.

Here’s a simplified example in Node.js:

const originalFunction = exports.functionName;

function instrumentedFunction(...args) {
  const startTime = process.hrtime.bigint();
  const result = originalFunction.apply(this, args);
  const duration = process.hrtime.bigint() - startTime;
  // Send span data to collector
  return result;
}

exports.functionName = instrumentedFunction;

Libraries like require-in-the-middle make this possible by intercepting module loading. So when your app imports something like Express, the agent wraps methods such as app.get() before your code executes.

Here’s what to keep in mind:

• Timing matters. If your app loads a module before the OTel agent starts, that module won’t be instrumented.
• Overhead is small but real. Expect about a 2–5% latency increase, slightly higher for hot paths.
• Version mismatches happen. If Express or another framework changes internal logic, the instrumentation library might need an update.

You’ll see this pattern across OpenTelemetry’s JavaScript and Python instrumentation libraries — simple in concept, but powerful when timed right.

Bytecode Instrumentation: Modifying the Virtual Machine

If you’ve ever used the Java agent flag -javaagent, you’ve already seen bytecode instrumentation in action.
Here, the agent doesn’t modify your source code — it changes the compiled bytecode as it loads into the JVM. Think of it as inserting tracing instructions right before your code runs.

When the JVM starts, it calls the agent’s premain() method. From there, the agent registers a class transformer that updates classes on the fly:

public static void premain(String args, Instrumentation inst) {
  new AgentBuilder.Default()
    .type(ElementMatchers.nameStartsWith("com.example"))
    .transform((builder, type, classLoader, module, domain) ->
      builder.method(ElementMatchers.named("handleRequest"))
        .intercept(MethodDelegation.to(TimingInterceptor.class))
    ).installOn(inst);
}

The TimingInterceptor wraps the original method call to record latency or add tracing logic — without touching your code.

This approach gives you wide coverage:

• It works across JVM-based languages like Java, Kotlin, and Scala.
• It can instrument classes loaded dynamically or from external dependencies.

You’ll notice a small startup delay (200–500 ms), but once classes load, the runtime overhead is minimal. Watch out for two common issues, though:

• Custom classloaders that skip standard loading might prevent instrumentation.
• Multiple agents (like security tools or other APMs) can clash if they modify the same bytecode.

Most of this magic happens through ByteBuddy, the library that simplifies bytecode manipulation. OpenTelemetry’s Java agent uses it extensively, and similar techniques show up in its .NET instrumentation too.

Compile-Time Instrumentation: Baking Observability Into the Binary

If you’re working with a statically compiled language like Go, auto-instrumentation can’t rely on runtime hooks. Instead, it happens before your app is built — by injecting observability code directly into the source during compilation.

Here’s how it works: the compiler parses your code into an Abstract Syntax Tree (AST), modifies that tree to insert instrumentation logic, and then regenerates the source before building the binary. The result? Observability is baked right into the executable.

func instrumentFunction() {
  fset := token.NewFileSet()
  file, err := parser.ParseFile(fset, "app/target.go", nil, parser.ParseComments)
  
  // Find target function and inject timing
  ast.Inspect(file, func(n ast.Node) bool {
    if fn, ok := n.(*ast.FuncDecl); ok && fn.Name.Name == "handleRequest" {
      // Add defer statement for timing
      deferStmt := &ast.DeferStmt{...}
      fn.Body.List = append([]ast.Stmt{deferStmt}, fn.Body.List...)
    }
    return true
  })
  
  // Write modified file back
  printer.Fprint(f, fset, file)
}

This method has zero runtime overhead — once compiled, there’s no agent or wrapper adding latency.
But there are tradeoffs:

• You need access to the source code and build system, which limits use for third-party libraries or precompiled binaries.
• Working with ASTs is complex and can make building pipelines harder to maintain.

The OpenTelemetry Go Compile Instrumentation project is a great reference for seeing this approach in practice. It shows how observability can be built into the binary itself, without relying on runtime interception.

eBPF Instrumentation: Kernel-Level Observability

If you’ve ever needed observability without touching your code, eBPF is what makes that possible. Instead of modifying source or bytecode, eBPF operates directly in the Linux kernel — watching what your program does as it runs.

It works by attaching probes to function entry and exit points. These probes collect telemetry data like execution duration or system call activity without altering the application itself.

Here’s a simple example using bpftrace:

#!/usr/bin/env bpftrace
uprobe:/app/service:main.handleRequest
{
  @start[tid] = nsecs;
}

uretprobe:/app/service:main.handleRequest
{
  $delta = nsecs - @start[tid];
  printf("handleRequest() duration: %d ns\n", $delta);
  delete(@start[tid]);
}

This script attaches to the function handleRequest(). When it starts, the probe records a timestamp; when it ends, it calculates the duration.

Why this approach stands out:

• Works with any language running on Linux — no code changes or recompilation needed.
• Adds very little overhead (around 1–2%) since everything runs in the kernel.

That said, there are some practical considerations:

• You’ll need Linux and elevated privileges (root or CAP_BPF), which might not be available in containerized or restricted environments.
• You won’t get deep visibility into application logic — eBPF can observe function calls, but not easily capture internal state or arguments.

For production-grade use, frameworks like BCC (BPF Compiler Collection) or libbpf provide more flexibility and performance than bpftrace. The OpenTelemetry eBPF Instrumentation project demonstrates how these capabilities are being adapted for cross-language observability.

Language Runtime APIs: Native Instrumentation Support

Some languages have built-in APIs for instrumentation. PHP's Observer API, introduced in PHP 8.0, is an example.

The Observer API lets C extensions hook into the PHP engine's execution flow at the Zend engine level. This provides visibility into PHP application behavior without code modifications:

static void observer_begin(zend_execute_data *execute_data) {
  if (execute_data->func && execute_data->func->common.function_name) {
    const char *function_name = ZSTR_VAL(execute_data->func->common.function_name);
    if (strcmp(function_name, "handleRequest") == 0) {
      start_time = clock();
    }
  }
}

static void observer_end(zend_execute_data *execute_data, zval *retval) {
  if (execute_data->func && execute_data->func->common.function_name) {
    const char *function_name = ZSTR_VAL(execute_data->func->common.function_name);
    if (strcmp(function_name, "handleRequest") == 0) {
      clock_t end_time = clock();
      double duration = (double)(end_time - start_time) / CLOCKS_PER_SEC * 1000;
      php_printf("Function %s() took %.2f ms\n", function_name, duration);
    }
  }
}

The Observer API is efficient and well-integrated with the language. It requires writing C extensions, though, which adds complexity and requires familiarity with C and PHP's internal APIs.

OpenTelemetry's PHP instrumentation libraries show how this works in practice.

Language Runtime APIs: Native Instrumentation Support

Some languages make observability easier by exposing native hooks in their runtime. Instead of using external agents or bytecode manipulation, you can plug directly into the language’s execution engine.

A good example is PHP’s Observer API, introduced in PHP 8.0. It lets C extensions attach callbacks at key points in the Zend engine’s execution flow — giving you insight into function calls without touching application code.

Here’s a simple example of how it works:

static void observer_begin(zend_execute_data *execute_data) {
  if (execute_data->func && execute_data->func->common.function_name) {
    const char *function_name = ZSTR_VAL(execute_data->func->common.function_name);
    if (strcmp(function_name, "handleRequest") == 0) {
      start_time = clock();
    }
  }
}

static void observer_end(zend_execute_data *execute_data, zval *retval) {
  if (execute_data->func && execute_data->func->common.function_name) {
    const char *function_name = ZSTR_VAL(execute_data->func->common.function_name);
    if (strcmp(function_name, "handleRequest") == 0) {
      clock_t end_time = clock();
      double duration = (double)(end_time - start_time) / CLOCKS_PER_SEC * 1000;
      php_printf("Function %s() took %.2f ms\n", function_name, duration);
    }
  }
}

This code hooks into the PHP runtime to measure how long the handleRequest() function takes to execute — all without modifying the application itself.

The benefits are clear:

• It’s efficient, since it runs as part of the language runtime.
• It’s accurate, with direct access to function-level events.

But there’s also a tradeoff: you need to write C extensions and understand the PHP internals, which adds complexity and maintenance effort.

Choose an Approach for Your Stack

It really depends on the language you’re using and how much control you have over your build or runtime. Here’s a quick way to think about what fits your stack best:

• Python
  Same story as Node.js — it uses monkey patching too. Works with Django, Flask, and FastAPI.
  The only real gotcha? Make sure the OTel agent loads before your app imports these frameworks.
• Go
  Go gives you options. You can:
  • Inject instrumentation at compile time if you control the build.
  • Or use eBPF for zero code changes.
    eBPF won’t capture detailed function arguments or internal state, so many teams pair it with manual SDK instrumentation for extra context.
• PHP
  If you’re running PHP 8.0 or newer, you can use the Observer API — it hooks directly into the Zend engine. On older versions, you’ll need to fall back to manual SDK instrumentation.
• .NET
  Works a lot like Java. The OTel .NET agent instruments the CLR using profiling APIs. You’ll just need to enable it:

CORECLR_ENABLE_PROFILING=1

• Node.js
  Auto-instrumentation here relies on monkey patching. The OpenTelemetry Node.js agent wraps popular frameworks like Express, Fastify, and NestJS.
  Just make sure the agent loads before your app code:

NODE_OPTIONS=--require @opentelemetry/auto-instrumentations-node/register

• Java, Kotlin, Scala
  These languages live on the JVM, so the Java agent route is usually the easiest. It works out of the box with frameworks like Spring, Micronaut, and Quarkus.
  Just set:

JAVA_TOOL_OPTIONS=-javaagent:/path/to/opentelemetry-javaagent.jar

Once you do that, the agent automatically instruments supported libraries.

If you’re juggling multiple languages and want telemetry without touching code, eBPF is your friend. It runs in the kernel, works across runtimes, and tools like the OTel Injector can even deploy agents automatically — detecting what’s running and wiring things up for you.

Troubleshoot Common Issues

When telemetry stops flowing, the first step is verifying that the agent is loaded. For Linux systems:

# Check if the agent loaded
ldd /proc/<PID>/exe | grep otel

# Check environment variables
cat /proc/<PID>/environ | tr '\0' '\n' | grep OTEL

# Check service logs
journalctl -u your-service | grep -i opentelemetry

If the agent is loaded but the data isn't appearing, check the collector endpoint. Network issues or protocol mismatches (gRPC vs HTTP) are common. Verify the collector is running: netstat -an | grep :4317.

Missing spans for specific frameworks often means the instrumentation library doesn't support that framework version. The OTel repositories list supported versions for each instrumentation. If your version isn't listed, you'll need manual instrumentation for those code paths.

Application crashes on startup often point to environment variable conflicts. If you're already setting JAVA_TOOL_OPTIONS or NODE_OPTIONS elsewhere—common in CI/CD pipelines—the OTel agent's configuration might collide. Check container environment variables with docker inspect <container> | jq '.[0].Config.Env'.

Higher memory usage after enabling auto-instrumentation is expected to some degree. Excessive growth usually means too much data capture. Try reducing the sampling rate with OTEL_TRACES_SAMPLER=parentbased_traceratio and OTEL_TRACES_SAMPLER_ARG=0.1 for 10% sampling. Also check for high-cardinality labels like full user IDs or request IDs in span attributes.

Combine Auto and Manual Instrumentation

Auto-instrumentation handles infrastructure well: HTTP requests, database queries, cache calls, framework operations. It covers a lot of what's needed for debugging distributed systems.

Manual instrumentation adds business logic that's specific to your application—payment processing, order fulfillment, and domain logic that matters for debugging. Manual spans also let you add custom attributes like user ID, tenant ID, or feature flags that help filter traces.

Here's an example with Node.js:

// Auto-instrumentation handles the HTTP span
app.get('/api/orders', async (req, res) => {
  // Manual span for business logic
  const span = tracer.startSpan('process.order');
  span.setAttribute('user.id', req.user.id);
  span.setAttribute('order.total', order.total);
  span.setAttribute('payment.method', order.paymentMethod);
  
  try {
    const result = await processOrder(order);
    span.setStatus({ code: SpanStatusCode.OK });
    return result;
  } catch (error) {
    span.recordException(error);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw error;
  } finally {
    span.end();
  }
});

The HTTP request span comes from auto-instrumentation. The business logic span is manual. They nest correctly because context propagation works across both. The OTel SDK maintains a context stack that tracks active spans, so manually created spans automatically become children of auto-instrumented spans.

OTel Injector can help deploy auto-instrumentation at the host level. Manual spans, then layer on top in application code where needed.

A Note on Context Propagation

Context propagation—how trace IDs and span IDs flow across service boundaries—is worth mentioning briefly. Auto-instrumentation handles this in most cases. When your instrumented service makes an HTTP call, the agent injects traceparent headers. The receiving service's agent extracts them and continues the trace.

If you're mixing auto-instrumented services with manual instrumentation, you need to handle propagation explicitly in manual code. For more on how this works, the traceparent header explanation covers the details.

Integration Options

Most observability backends accept OTLP, so these instrumentation methods work with whatever you're using. For Last9, point your OTel agent to the endpoint:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.last9.io
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <your-token>"
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp

In mixed environments—Java with bytecode instrumentation, Node with monkey patching, Go with compile-time instrumentation—naming inconsistencies can appear across telemetry. Last9's Control Plane can normalize this at ingestion if needed, letting you rename metrics and standardize label keys.

For consistent instrumentation from the start, OTel Weaver generates type-safe constants from semantic conventions. This helps manual instrumentation match auto-instrumented services.

Try It Yourself

The OpenTelemetry blog post includes a lab repository with working examples for each technique:

• Monkey patching in Node.js
• Bytecode instrumentation in Java
• Compile-time instrumentation in Go
• eBPF with bpftrace
• PHP Observer API

Each example is self-contained and ready to run. Experimenting with the code and seeing what breaks helps understand these techniques better than reading about them.

The OpenTelemetry project continues to develop these techniques. Work includes better eBPF support for capturing application context, more language-specific runtime APIs similar to PHP's Observer, and improved compile-time tooling for statically compiled languages.

If you're interested in contributing, OpenTelemetry's Special Interest Groups cover language-specific instrumentation work.