Azure Event Hubs

Use OpenTelemetry to instrument your Azure Components and send logs and metrics to Last9 using Azure Event Hubs.

This guide outlines the steps to setup Azure Components to send logs and metrics to Last9 using OpenTelemetry Collector as a sidecar container.

Prerequisites

Before setting up Azure Event Hubs monitoring, ensure you have:

• Azure Container Apps
• Event Hubs
• Last9 account with integration credentials

1. Create Event Hubs Infrastructure

   Create Event Hubs namespace with name last9.

   Create Event hub in last9 namespace with name telemetry-hub.

2. Configure Azure Components

   Enable necessary logs, and metrics to be sent to Event hub created in Step 1 from various Azure Components. Read this guide for more details on how to enable streaming of logs and metrics to Event hub.

3. Setup Event Hub Access Policy

   Visit Event hub and setup a Shared Access Policy with following details:

   • Scope: Listen
   • Name: last9-telemetry

4. Get Connection String

   Copy the Primary connection string displayed after creating the shared access policy. It will look like following:

   Endpoint=sb://oteldemo.servicebus.windows.net/;SharedAccessKeyName=last9;SharedAccessKey=<access_key>;EntityPath=oteldemo

5. Run OpenTelemetry Collector

   Run the OpenTelemetry Collector to pull logs, and metrics from Event hub and send them to Last9. You can run this on a standalone VM or even locally as Docker Container.

   Create directory on the machine where you want to run the OpenTelemetry Collector:

   mkdir last9-otel-collector
   cd last9-otel-collector
   mkdir -p storage
   chmod 777 storage

   Create otel.yaml:

   receivers:
     azureeventhub:
       connection: <event_hub_primary_connection_string>
       storage: file_storage/otlp
       format: "azure"
       apply_semantic_conventions: true
   
   processors:
     batch:
       timeout: 5s
       send_batch_size: 20000
       send_batch_max_size: 20000
     transform/azure_logs:
       error_mode: ignore
       flatten_data: true
       log_statements:
         - context: log
           statements:
             - merge_maps(attributes, body, "insert")
             - flatten(attributes)
             # Set body to actual log message
             - set(body, attributes["properties.Log"])
             - delete_key(attributes, "properties.Log")
             # Set service name to container name
             - set(resource.attributes["service.name"], attributes["properties.ContainerAppName"])
             # Handle Severity mapping
             - set(severity_text, "TRACE") where Int(severity_text) >= 1 and Int(severity_text) <= 4
             - set(severity_text, "DEBUG") where Int(severity_text) >= 5 and Int(severity_text) <= 8
             - set(severity_text, "INFO") where Int(severity_text) >= 9 and Int(severity_text) <= 12
             - set(severity_text, "WARN") where Int(severity_text) >= 13 and Int(severity_text) <= 16
             - set(severity_text, "ERROR") where Int(severity_text) >= 17 and Int(severity_text) <= 20
             - set(severity_text, "FATAL") where Int(severity_text) >= 21 and Int(severity_text) <= 24
   
     transform/add_timestamp:
       error_mode: ignore
       flatten_data: true
       log_statements:
         - context: log
           conditions:
             - time_unix_nano == 0
           statements:
             - set(observed_time, Now())
             - set(time_unix_nano, observed_time_unix_nano)
   
   exporters:
     debug:
       verbosity: detailed
     otlp/last9:
       endpoint: "{{ .Logs.WriteURL }}"
       headers:
         "Authorization": "{{ .Logs.AuthValue }}"
   
   service:
     extensions: [file_storage/otlp]
     pipelines:
       logs:
         receivers: [azureeventhub]
         processors: [transform/azure_logs, transform/add_timestamp, batch]
         exporters: [otlp/last9]
       metrics:
         receivers: [azureeventhub]
         processors: [batch]
         exporters: [otlp/last9]
       traces:
         receivers: [azureeventhub]
         processors: [batch]
         exporters: [otlp/last9]
   
   extensions:
     file_storage/otlp:
       directory: /storage
       create_directory: true
       timeout: 10s
       compaction:
         directory: /storage
         on_rebound: true
         check_interval: 1s

6. Deploy OpenTelemetry Collector

   • Docker Compose
   • Docker
   • Standalone Binary

   Create docker-compose.yaml:

   services:
     otel-collector:
       image: otel/opentelemetry-collector-contrib:latest
       container_name: otel-collector
       volumes:
         - ./otel.yaml:/etc/otel-collector/config.yaml
         - ./storage:/storage
       command:
         [
           "--config",
           "/etc/otel-collector/config.yaml",
           "--feature-gates",
           "transform.flatten.logs",
         ]
       restart: unless-stopped

   Start the docker container:

   docker compose up

Verification

Once the Otel collector is running, the logs and metrics will be sent to Last9. You can verify this by checking the Logs Explorer and Grafana in Last9.

Need Help?

If you encounter any issues or have questions:

• Join our Discord community for real-time support
• Contact our support team at support@last9.io