Custom Metrics in Go

Use the OpenTelemetry Go SDK to emit custom metrics — counters, histograms, and gauges — with attributes (labels) from any Go service.

Prerequisites

• Go 1.22 or higher
• Last9 account with OTLP credentials

Installation

1. Install dependencies

   go get go.opentelemetry.io/otel \
     go.opentelemetry.io/otel/metric \
     go.opentelemetry.io/otel/sdk/metric \
     go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp

2. Set environment variables

   export OTEL_SERVICE_NAME="your-service"
   export OTEL_EXPORTER_OTLP_ENDPOINT="$last9_otlp_endpoint"
   export OTEL_EXPORTER_OTLP_HEADERS="Authorization=$last9_otlp_auth_header"
   export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"

3. Set up MeterProvider

   package main
   
   import (
       "context"
       "log"
       "time"
   
       "go.opentelemetry.io/otel"
       "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
       "go.opentelemetry.io/otel/sdk/metric"
       "go.opentelemetry.io/otel/sdk/resource"
       semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
   )
   
   func initMeterProvider(ctx context.Context) (*metric.MeterProvider, error) {
       exporter, err := otlpmetrichttp.New(ctx)
       if err != nil {
           return nil, err
       }
   
       res, err := resource.New(ctx,
           resource.WithFromEnv(),
           resource.WithAttributes(
               semconv.ServiceNameKey.String("your-service"),
               semconv.DeploymentEnvironmentKey.String("production"),
           ),
       )
       if err != nil {
           return nil, err
       }
   
       mp := metric.NewMeterProvider(
           metric.WithResource(res),
           metric.WithReader(metric.NewPeriodicReader(exporter,
               metric.WithInterval(30*time.Second),
           )),
       )
       otel.SetMeterProvider(mp)
       return mp, nil
   }
   
   func main() {
       ctx := context.Background()
   
       mp, err := initMeterProvider(ctx)
       if err != nil {
           log.Fatalf("init meter provider: %v", err)
       }
       defer func() {
           mp.ForceFlush(ctx)
           mp.Shutdown(ctx)
       }()
   
       // your application code here
   }

4. Record metrics with attributes

   • Counter
   • Histogram
   • Gauge

   import (
       "go.opentelemetry.io/otel/attribute"
       otelmetric "go.opentelemetry.io/otel/metric"
   )
   
   meter := otel.Meter("your-service")
   
   counter, err := meter.Int64Counter(
       "subscription.upgrade.notification.resolution",
       otelmetric.WithDescription("Count of subscription upgrade notification resolutions"),
   )
   if err != nil {
       log.Fatalf("create counter: %v", err)
   }
   
   // Record with attributes
   counter.Add(ctx, 1, otelmetric.WithAttributes(
       attribute.String("status", status),
       attribute.String("reason", reason),
   ))

Metric naming

The OTel SDK converts . to _ in metric names when exporting. A counter named subscription.upgrade.notification.resolution appears in Last9 as subscription_upgrade_notification_resolution_total.

Querying in Last9

Once metrics are flowing, query them in Last9 using PromQL:

# Total rate of resolutions by status
sum by (status, reason) (
  rate(subscription_upgrade_notification_resolution_total[5m])
)

# Filter to a specific attribute value
subscription_upgrade_notification_resolution_total{status="success"}

Troubleshooting

• Attributes not appearing in Last9

  If a metric appears in Last9 but some attributes (labels) are missing, the most likely cause is that the attribute value is an empty string at record time.

  Last9 follows the Prometheus data model: labels with empty values are treated as if the label does not exist. The metric is stored and queryable, but that label dimension is absent.

  How to identify the problem — add a log line immediately before the Add() or Record() call:

  log.Printf("recording metric: status=%q reason=%q product_type=%q",
      status, reason, productType)
  
  counter.Add(ctx, 1, otelmetric.WithAttributes(
      attribute.String("status", status),
      attribute.String("reason", reason),
      attribute.String("product_type", productType),
  ))

  If any value prints as "", that label will not appear in Last9.

  Fix — use a sentinel value instead of empty string:

  if status == "" {
      status = "unknown"
  }
  if reason == "" {
      reason = "unknown"
  }
  if productType == "" {
      productType = "unknown"
  }
  
  counter.Add(ctx, 1, otelmetric.WithAttributes(
      attribute.String("status", status),
      attribute.String("reason", reason),
      attribute.String("product_type", productType),
  ))

  This behavior is silent — no error is returned and the metric value is recorded correctly. Only the label is dropped. This makes it easy to miss during development if attribute values happen to be non-empty in test environments but empty in production.

Please get in touch with us on Discord or Email if you have any questions.