# Azure Event Hubs

> Collect logs and metrics from Azure components using Azure Event Hubs with OpenTelemetry collector integration to Last9

Source: https://last9.io/docs/integrations/messaging/azure-event-hubs/

Use OpenTelemetry to instrument your Azure Components and send logs and metrics to Last9 using Azure Event Hubs.

This guide outlines the steps to setup Azure Components to send logs and metrics to Last9 using OpenTelemetry Collector as a sidecar container.

## Prerequisites

Before setting up Azure Event Hubs monitoring, ensure you have:

- Azure Container Apps
- Event Hubs
- Last9 account with integration credentials

1.  **Create Event Hubs Infrastructure**

    Create Event Hubs namespace with name `last9`.

    Create Event hub in `last9` namespace with name `telemetry-hub`.

2.  **Configure Azure Components**

    Enable necessary logs, and metrics to be sent to Event hub created in Step 1 from various Azure Components. Read this [guide](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/stream-monitoring-data-event-hubs#streaming-methods) for more details on how to enable streaming of logs and metrics to Event hub.

3.  **Setup Event Hub Access Policy**

    Visit Event hub and setup a Shared Access Policy with following details:

    - Scope: Listen
    - Name: last9-telemetry

4.  **Get Connection String**

    Copy the Primary connection string displayed after creating the shared access policy. It will look like following:

    ```json
    Endpoint=sb://oteldemo.servicebus.windows.net/;SharedAccessKeyName=last9;SharedAccessKey=<access_key>;EntityPath=oteldemo
    ```

5.  **Run OpenTelemetry Collector**

    Run the OpenTelemetry Collector to pull logs, and metrics from Event hub and send them to Last9. You can run this on a standalone VM or even locally as Docker Container.

    Create directory on the machine where you want to run the OpenTelemetry Collector:

    ```bash
    mkdir last9-otel-collector
    cd last9-otel-collector
    mkdir -p storage
    chmod 777 storage
    ```

    Create **otel.yaml**:

    ```yaml
    receivers:
      azureeventhub:
        connection: <event_hub_primary_connection_string>
        storage: file_storage/otlp
        format: "azure"
        apply_semantic_conventions: true

    processors:
      batch:
        timeout: 5s
        send_batch_size: 20000
        send_batch_max_size: 20000
      transform/azure_logs:
        error_mode: ignore
        flatten_data: true
        log_statements:
          - context: log
            statements:
              - merge_maps(attributes, body, "insert")
              - flatten(attributes)
              # Set body to actual log message
              - set(body, attributes["properties.Log"])
              - delete_key(attributes, "properties.Log")
              # Set service name to container name
              - set(resource.attributes["service.name"], attributes["properties.ContainerAppName"])
              # Handle Severity mapping
              - set(severity_text, "TRACE") where Int(severity_text) >= 1 and Int(severity_text) <= 4
              - set(severity_text, "DEBUG") where Int(severity_text) >= 5 and Int(severity_text) <= 8
              - set(severity_text, "INFO") where Int(severity_text) >= 9 and Int(severity_text) <= 12
              - set(severity_text, "WARN") where Int(severity_text) >= 13 and Int(severity_text) <= 16
              - set(severity_text, "ERROR") where Int(severity_text) >= 17 and Int(severity_text) <= 20
              - set(severity_text, "FATAL") where Int(severity_text) >= 21 and Int(severity_text) <= 24

      transform/add_timestamp:
        error_mode: ignore
        flatten_data: true
        log_statements:
          - context: log
            conditions:
              - time_unix_nano == 0
            statements:
              - set(observed_time, Now())
              - set(time_unix_nano, observed_time_unix_nano)

    exporters:
      debug:
        verbosity: detailed
      otlp/last9:
        endpoint: "{{ .Logs.WriteURL }}"
        headers:
          "Authorization": "{{ .Logs.AuthValue }}"

    service:
      extensions: [file_storage/otlp]
      pipelines:
        logs:
          receivers: [azureeventhub]
          processors: [transform/azure_logs, transform/add_timestamp, batch]
          exporters: [otlp/last9]
        metrics:
          receivers: [azureeventhub]
          processors: [batch]
          exporters: [otlp/last9]
        traces:
          receivers: [azureeventhub]
          processors: [batch]
          exporters: [otlp/last9]

    extensions:
      file_storage/otlp:
        directory: /storage
        create_directory: true
        timeout: 10s
        compaction:
          directory: /storage
          on_rebound: true
          check_interval: 1s
    ```

6.  **Deploy OpenTelemetry Collector**

**Docker Compose**

    Create **docker-compose.yaml**:

    ```yaml
    services:
      otel-collector:
        image: otel/opentelemetry-collector-contrib:latest
        container_name: otel-collector
        volumes:
          - ./otel.yaml:/etc/otel-collector/config.yaml
          - ./storage:/storage
        command:
          [
            "--config",
            "/etc/otel-collector/config.yaml",
            "--feature-gates",
            "transform.flatten.logs",
          ]
        restart: unless-stopped
    ```

    Start the docker container:

    ```bash
    docker compose up
    ```

**Docker**

    If you don't want to use docker compose, directly start the container:

    ```bash
    docker run -d \
      --name otel-collector \
      --restart unless-stopped \
      -v "$(pwd)/otel.yaml:/etc/otel-collector/config.yaml" \
      -v "$(pwd)/storage:/storage" \
      otel/opentelemetry-collector-contrib:latest \
      --config /etc/otel-collector/config.yaml \
      --feature-gates transform.flatten.logs
    ```

**Standalone Binary**

    Install the Collector:

    ```bash
    sudo apt-get update
    sudo apt-get -y install wget systemctl
    wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.118.0/otelcol-contrib_0.118.0_linux_amd64.deb
    sudo dpkg -i otelcol-contrib_0.118.0_linux_amd64.deb
    ```

    Start the collector:

    ```bash
    otelcol-contrib --config $(pwd)/otel.yaml --feature-gates transform.flatten.logs
    ```

## Verification

Once the Otel collector is running, the logs and metrics will be sent to Last9. You can verify this by checking the [Logs Explorer](https://app.last9.io/logs) and [Grafana](https://app.last9.io/grafana) in Last9.

---

## Troubleshooting

Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:support@last9.io) if you have any questions.