Karafka

Use OpenTelemetry to instrument your Karafka application and send telemetry data to Last9. You can either run OpenTelemetry Collector or send the telemetry directly from the application to Last9.

Note: This integration should be used if you are using Karafka outside of Ruby on Rails. If you are using it inside a Ruby on Rails application, then use the Ruby on Rails integration.

Install OpenTelemetry Packages

Install the following gems.

gem 'opentelemetry-sdk'
gem 'opentelemetry-exporter-otlp'
gem 'opentelemetry-instrumentation-all'

Set the Environment Variables

OTEL_SERVICE_NAME=<your-app-name>
OTEL_EXPORTER_OTLP_ENDPOINT={{ .Logs.WriteURL }}
OTEL_EXPORTER_OTLP_HEADERS="Authorization={{ .Logs.AuthValue }}"
OTEL_TRACES_EXPORTER=otlp
OTEL_RESOURCE_ATTRIBUTES="deployment.environment=local"
OTEL_LOG_LEVEL=error

Note: In case you are running an OpenTelemetry Collector, replace the OTEL_EXPORTER_OTLP_ENDPOINT with the collector URL.

Instrument the Karafka application

Create a file lib/otel_setup.rb, add the following code to instrument your application:

require 'opentelemetry/sdk'
require 'opentelemetry/exporter/otlp'
require 'opentelemetry/instrumentation/all'

class OtelSetup
  def initialize
    @otel_exporter = OpenTelemetry::Exporter::OTLP::Exporter.new
    @processor = OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(@otel_exporter)
  end

  def process
    OpenTelemetry::SDK.configure do |c|
      # Exporter and Processor configuration
      c.add_span_processor(@processor) # Created above this SDK.configure block

      # Resource configuration
      c.resource = OpenTelemetry::SDK::Resources::Resource.create({
                                                                    OpenTelemetry::SemanticConventions::Resource::DEPLOYMENT_ENVIRONMENT => "staging" # Change to deployment environment
                                                                  })

      c.use_all() # enables all instrumentation!
    end
  end
end

Include this file in your application setup as early as possible, immediately after bundler is setup or eager loading is done.

ENV['KARAFKA_ENV'] ||= 'development'
Bundler.require(:default, ENV['KARAFKA_ENV'])

# Zeitwerk custom loader for loading the app components before the whole
# Karafka framework configuration
APP_LOADER = Zeitwerk::Loader.new

%w[
  lib
  app/consumers
].each(&APP_LOADER.method(:push_dir))

APP_LOADER.setup
APP_LOADER.eager_load

# Setup OpenTelemetry instrumentation
OtelSetup.new.process

## Rest of the code

This code snippet configures the OpenTelemetry SDK to use the OTLP exporter and enables instrumentation for Karafka application.

Run the Application

bundle exec karafka s

The application will start sending traces to Last9.

Find sample code example of a Standalone Karafka application with OpenTelemetry here.