Instrumenting fasthttp with OpenTelemetry: A Comprehensive Guide

As a developer who's worked extensively with Go and various web frameworks, I've come to appreciate the importance of observability in modern applications. 

Today, I'm excited to share my experience instrumenting a fasthttp server with OpenTelemetry. This tutorial will guide you through the process, highlighting the benefits of manual instrumentation and touching on related concepts like distributed tracing and context propagation.

Introduction

OpenTelemetry is an open-source observability framework that provides a unified way to instrument, generate, collect, and export telemetry data. While many developers are familiar with instrumenting net/HTTP-based servers using packages like otelhttp, fasthttp requires a different approach.

In this guide, we'll walk through the process of manually instrumenting a fasthttp server to emit RED (Rate, Error, Duration) metrics.

We'll configure our application to push these metrics to an OTLP (OpenTelemetry Protocol) endpoint, which can then be consumed by an OpenTelemetry Collector or other compatible systems for analysis and alerting.

Prerequisites

Before we begin, make sure you have the following installed:

• Go 1.16 or later
• Docker (optional, for running an OpenTelemetry Collector)

Step 1: Setting Up the Project

mkdir fasthttp-otel
cd fasthttp-otel
go mod init fasthttp-otel

Next, we'll install the necessary dependencies:

# Install fasthttp packages
go get github.com/valyala/fasthttp
go get github.com/fasthttp/router

# Install OpenTelemetry packages
go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp
go get go.opentelemetry.io/otel/sdk/metric

Note that we're using packages from the OpenTelemetry contrib repository, which provides additional instrumentation options beyond the core library.

Step 2: Create a Base Application

Let's create a basic fasthttp server in main.go:

package main

import (
    "flag"
    "fmt"
    "log"

    "github.com/fasthttp/router"
    "github.com/valyala/fasthttp"
)

var addr = flag.String("addr", ":8080", "TCP address to listen to")

func main() {
    flag.Parse()
    r := router.New()

    r.GET("/", func(ctx *fasthttp.RequestCtx) {
        ctx.Response.Header.Set("Content-Type", "application/json")
        ctx.Response.SetBodyString(`{"message": "Hello, World!"}`)
    })

    fmt.Printf("Server listening on %s\n", *addr)
    if err := fasthttp.ListenAndServe(*addr, r.Handler); err != nil {
        log.Fatalf("Error in ListenAndServe: %v", err)
    }
}

This sets up a basic HTTP server using fasthttp, which we'll now instrument with OpenTelemetry.

Step 3: Implementing OpenTelemetry Instrumentation

Now, let's create a instrumentation package to handle our OpenTelemetry setup. Create a new file instrumentation/instrumentation.go:

package instrumentation

import (
    "context"
    "log"
    "time"

    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
    api "go.opentelemetry.io/otel/metric"
    "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.25.0"
)

type Instrumentation struct {
    HTTPTotalRequestsCounter api.Int64Counter
    HTTPRequestHistogram     api.Float64Histogram
}

func New() *Instrumentation {
    // Initialize the OTLP exporter
    exporter, err := otlpmetrichttp.New(context.Background(), otlpmetrichttp.WithInsecure())
    if err != nil {
        log.Fatalf("Failed to create the collector exporter: %v", err)
    }

    // Create a meter provider with a periodic reader
    reader := metric.NewPeriodicReader(exporter, metric.WithInterval(15*time.Second))

    // Configure the resource with semantic conventions
    res := resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceName("fasthttp-otel-service"),
        attribute.String("environment", "production"),
    )

    // Initialize the meter provider
    provider := metric.NewMeterProvider(
        metric.WithReader(reader),
        metric.WithResource(res),
    )

    meter := provider.Meter("fasthttp-otel")

    // Create instruments for our metrics
    httpTotalRequestsCounter, _ := meter.Int64Counter(
        "http_total_requests",
        api.WithDescription("Total number of HTTP requests"),
    )

    httpRequestHistogram, _ := meter.Float64Histogram(
        "http_request_duration_seconds",
        api.WithDescription("Duration of HTTP requests"),
    )

    return &Instrumentation{
        HTTPTotalRequestsCounter: httpTotalRequestsCounter,
        HTTPRequestHistogram:     httpRequestHistogram,
    }
}

This setup initializes our OpenTelemetry instrumentation, creating a meter provider and defining our metrics. We're using semantic conventions to add metadata to our metrics, which will help with analysis later.

Step 4: Creating an Instrumented Router

Now, let's create a custom router that wraps the fasthttp router with our instrumentation. Create a new file router/router.go:

package router

import (
    "fasthttp-otel/instrumentation"
    "time"

    fr "github.com/fasthttp/router"
    "github.com/valyala/fasthttp"
    "go.opentelemetry.io/otel/attribute"
    api "go.opentelemetry.io/otel/metric"
)

type Router struct {
    r *fr.Router
    i *instrumentation.Instrumentation
}

func New() *Router {
    return &Router{
        r: fr.New(),
        i: instrumentation.New(),
    }
}

func (ir *Router) instrumentedHandler(handlerFunc fasthttp.RequestHandler) fasthttp.RequestHandler {
    return func(ctx *fasthttp.RequestCtx) {
        start := time.Now()
        handlerFunc(ctx)
        duration := time.Since(start).Seconds()

        labels := []attribute.KeyValue{
            attribute.String("method", string(ctx.Method())),
            attribute.Int("status", ctx.Response.StatusCode()),
            attribute.String("path", string(ctx.Path())),
        }

        ir.i.HTTPRequestHistogram.Record(ctx, duration, api.WithAttributes(labels...))
        ir.i.HTTPTotalRequestsCounter.Add(ctx, 1, api.WithAttributes(labels...))
    }
}

func (ir *Router) Handler(ctx *fasthttp.RequestCtx) {
    ir.r.Handler(ctx)
}

// Implement methods for each HTTP verb
func (ir *Router) GET(path string, handler fasthttp.RequestHandler) {
    ir.r.GET(path, ir.instrumentedHandler(handler))
}

// Implement POST, PUT, DELETE, etc. similarly

This router wraps each handler with our instrumentation, allowing us to measure the duration of each request and count the total number of requests.

Step 5: Updating the Main Application

Now, let's update our main.go to use our new instrumented router:

package main

import (
    "flag"
    "fmt"
    "log"

    "fasthttp-otel/router"
    "github.com/valyala/fasthttp"
)

var addr = flag.String("addr", ":8080", "TCP address to listen to")

func main() {
    flag.Parse()
    r := router.New()

    r.GET("/", func(ctx *fasthttp.RequestCtx) {
        ctx.Response.Header.Set("Content-Type", "application/json")
        ctx.Response.SetBodyString(`{"message": "Hello, World!"}`)
    })

    fmt.Printf("Server listening on %s\n", *addr)
    if err := fasthttp.ListenAndServe(*addr, r.Handler); err != nil {
        log.Fatalf("Error in ListenAndServe: %v", err)
    }
}

Running the Application

To run the application, you'll need to set the OTLP endpoint environment variable:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
go run .

This assumes you have an OpenTelemetry Collector running on localhost:4318. You can set up a collector using Docker for testing:

docker run -p 4318:4318 otel/opentelemetry-collector-contrib:latest

Conclusion

We've successfully instrumented a fasthttp server with OpenTelemetry, enabling the collection of important metrics like request count and duration. While this tutorial focused on server-side instrumentation, you could extend this to instrument HTTP clients as well. Additionally, you might consider implementing distributed tracing by propagating trace context between services.

OpenTelemetry's flexibility allows you to start small and gradually expand your observability setup as your needs grow. Whether you're working with Go, Java, Python, or other languages, the concepts we've covered here will serve you well.

Happy coding, and may your services always be observable!

We'd love to hear about your experiences with reliability, observability, or monitoring. Let’s share insights and chat about these topics in the SRE Discord community.

FAQs related to fasthttp with OTel

What is Instrumentation and Why Do We Need It?

Instrumentation is the process of adding code to measure an application's performance and behavior.

It's essential for:

• Identifying and diagnosing issues
• Understanding system behavior
• Making data-driven optimization decisions
• Ensuring service level objectives are met

What is OpenTelemetry?

OpenTelemetry (OTel) is an open-source observability framework that provides:

• APIs and SDKs for instrumenting applications
• A vendor-neutral way to collect metrics, logs, and traces
• Tools to export telemetry data to various backends

It aims to standardize observability data collection across different languages and platforms.

Why is the fasthttp API incompatible with net/http?

fasthttp's API differs from net/http for performance reasons:

• Uses a single context object for both request and response
• Implements custom memory management and connection pooling
• Simplifies some features to reduce overhead

These differences make fasthttp incompatible with standard net/http middleware and libraries.

How do I set up OTel instrumentation for a fasthttp application?

1. Install OpenTelemetry SDK and OTLP exporter
2. Initialize meter and tracer providers
3. Create custom middleware to wrap fasthttp handlers
4. Implement metric collection and span creation in the middleware
5. Configure the application to send telemetry data to a collector

This tutorial provides a detailed walkthrough for metrics. For tracing, additional steps for span creation and context propagation are needed.

What is OpenTelemetry auto-instrumentation?

Auto-instrumentation automatically adds telemetry data collection to an application without manual code changes. It typically uses agents or runtime manipulation. However, for fasthttp, manual instrumentation (as shown in this tutorial) is often necessary due to its unique API.