# Custom Metrics in Go

> Emit custom metrics with attributes from Go services using the OpenTelemetry SDK and send them to Last9

Source: https://last9.io/docs/integrations/frameworks/go/custom-metrics/

Use the OpenTelemetry Go SDK to emit custom metrics — counters, histograms, and gauges — with attributes (labels) from any Go service.

## Prerequisites

- Go 1.22 or higher
- Last9 account with OTLP credentials

## Installation

1. **Install dependencies**

   ```bash
   go get go.opentelemetry.io/otel \
     go.opentelemetry.io/otel/metric \
     go.opentelemetry.io/otel/sdk/metric \
     go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp
   ```

2. **Set environment variables**

   ```bash
   export OTEL_SERVICE_NAME="your-service"
   export OTEL_EXPORTER_OTLP_ENDPOINT="$last9_otlp_endpoint"
   export OTEL_EXPORTER_OTLP_HEADERS="Authorization=$last9_otlp_auth_header"
   export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
   ```

3. **Set up MeterProvider**

   ```go
   package main

   import (
       "context"
       "log"
       "time"

       "go.opentelemetry.io/otel"
       "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
       "go.opentelemetry.io/otel/sdk/metric"
       "go.opentelemetry.io/otel/sdk/resource"
       semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
   )

   func initMeterProvider(ctx context.Context) (*metric.MeterProvider, error) {
       exporter, err := otlpmetrichttp.New(ctx)
       if err != nil {
           return nil, err
       }

       res, err := resource.New(ctx,
           resource.WithFromEnv(),
           resource.WithAttributes(
               semconv.ServiceNameKey.String("your-service"),
               semconv.DeploymentEnvironmentKey.String("production"),
           ),
       )
       if err != nil {
           return nil, err
       }

       mp := metric.NewMeterProvider(
           metric.WithResource(res),
           metric.WithReader(metric.NewPeriodicReader(exporter,
               metric.WithInterval(30*time.Second),
           )),
       )
       otel.SetMeterProvider(mp)
       return mp, nil
   }

   func main() {
       ctx := context.Background()

       mp, err := initMeterProvider(ctx)
       if err != nil {
           log.Fatalf("init meter provider: %v", err)
       }
       defer func() {
           mp.ForceFlush(ctx)
           mp.Shutdown(ctx)
       }()

       // your application code here
   }
   ```

4. **Record metrics with attributes**

**Counter**

   ```go
   import (
       "go.opentelemetry.io/otel/attribute"
       otelmetric "go.opentelemetry.io/otel/metric"
   )

   meter := otel.Meter("your-service")

   counter, err := meter.Int64Counter(
       "subscription.upgrade.notification.resolution",
       otelmetric.WithDescription("Count of subscription upgrade notification resolutions"),
   )
   if err != nil {
       log.Fatalf("create counter: %v", err)
   }

   // Record with attributes
   counter.Add(ctx, 1, otelmetric.WithAttributes(
       attribute.String("status", status),
       attribute.String("reason", reason),
   ))
   ```

**Histogram**

   ```go
   meter := otel.Meter("your-service")

   histogram, err := meter.Float64Histogram(
       "request.duration",
       otelmetric.WithDescription("Request duration in seconds"),
       otelmetric.WithUnit("s"),
   )
   if err != nil {
       log.Fatalf("create histogram: %v", err)
   }

   histogram.Record(ctx, duration.Seconds(), otelmetric.WithAttributes(
       attribute.String("method", "GET"),
       attribute.String("status_code", "200"),
   ))
   ```

**Gauge**

   ```go
   meter := otel.Meter("your-service")

   gauge, err := meter.Int64Gauge(
       "queue.depth",
       otelmetric.WithDescription("Current queue depth"),
   )
   if err != nil {
       log.Fatalf("create gauge: %v", err)
   }

   gauge.Record(ctx, int64(queueDepth), otelmetric.WithAttributes(
       attribute.String("queue_name", queueName),
   ))
   ```

## Metric naming

The OTel SDK converts `.` to `_` in metric names when exporting. A counter named `subscription.upgrade.notification.resolution` appears in Last9 as `subscription_upgrade_notification_resolution_total`.

## Querying in Last9

Once metrics are flowing, query them in Last9 using PromQL:

```promql
# Total rate of resolutions by status
sum by (status, reason) (
  rate(subscription_upgrade_notification_resolution_total[5m])
)

# Filter to a specific attribute value
subscription_upgrade_notification_resolution_total{status="success"}
```

---

## Troubleshooting

- **Attributes not appearing in Last9**

  If a metric appears in Last9 but some attributes (labels) are missing, the most likely cause is that the attribute value is an **empty string** at record time.

  Last9 follows the Prometheus data model: labels with empty values are treated as if the label does not exist. The metric is stored and queryable, but that label dimension is absent.

  How to identify the problem — add a log line immediately before the `Add()` or `Record()` call:

  ```go
  log.Printf("recording metric: status=%q reason=%q product_type=%q",
      status, reason, productType)

  counter.Add(ctx, 1, otelmetric.WithAttributes(
      attribute.String("status", status),
      attribute.String("reason", reason),
      attribute.String("product_type", productType),
  ))
  ```

  If any value prints as `""`, that label will not appear in Last9.

  Fix — use a sentinel value instead of empty string:

  ```go
  if status == "" {
      status = "unknown"
  }
  if reason == "" {
      reason = "unknown"
  }
  if productType == "" {
      productType = "unknown"
  }

  counter.Add(ctx, 1, otelmetric.WithAttributes(
      attribute.String("status", status),
      attribute.String("reason", reason),
      attribute.String("product_type", productType),
  ))
  ```

  This behavior is silent — no error is returned and the metric value is recorded correctly. Only the label is dropped. This makes it easy to miss during development if attribute values happen to be non-empty in test environments but empty in production.

Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:support@last9.io) if you have any questions.