Enhancing Observability with Fluent Bit and OpenTelemetry

Building observability pipelines is like setting up an efficient postal system for your data: logs, metrics, and traces must travel from their sources to their destinations without getting lost or delayed.

Fluent Bit and OpenTelemetry, two powerful open-source tools, simplify this journey and help you process trace data, log data, and metadata efficiently.

Let's talk about why observability pipelines are essential, how OpenTelemetry (Otel) integrates with Fluent Bit, and some best practices for production.

Why Observability Pipelines Are Essential

Modern systems generate vast amounts of data every second. This data can help diagnose problems, optimize performance, and improve reliability—if it’s handled properly. Observability pipelines ensure:

• Data Collection: From various sources (containers, applications, nodes, etc.).
• Data Parsing: Converting raw data into a structured format like JSON.
• Data Transport: Securely deliver processed data to monitoring backends.

Full-Stack Observability for Better Application Performance | Last9

Achieve better application performance with full-stack observability, gaining real-time insights to troubleshoot, optimize, and enhance user experience.

Last9Last9

Fluent Bit

Developed by Calyptia and led by Eduardo Silva, Fluent Bit is a lightweight, high-performance logging and metrics tool. It excels at handling log data efficiently, even in environments with limited resources, such as edge devices or Kubernetes clusters.

Key Features

Input Plugins: Fluent Bit supports various plugins to gather logs and metrics, like Docker logs and standard output (stdout).

• Output Flexibility: You can export data to OpenTelemetry, Jaeger, Prometheus, or other tools.
• TLS Support: Fluent Bit ensures secure data transfer with TLS.

Here’s a typical configuration file for Fluent Bit, processing Docker logs:

[INPUT] 
    Name docker 
    Path /var/lib/docker/containers/*/*.log 

[OUTPUT] 
    Name stdout 
    Match * 
    Log_Response_Payload true

Getting the Most Out of Tracing Tools for Observability | Last9

Maximize your observability with tracing tools to track requests, identify bottlenecks, and optimize system performance across services.

Last9Last9

OpenTelemetry

OpenTelemetry, often used with tools like Jaeger, provides a framework for instrumentation across logs, metrics, and traces. It’s the go-to standard for creating observability pipelines that scale.

How It Integrates with Fluent Bit

When Fluent Bit collects and parses log data, it can route it directly to OpenTelemetry’s collector. This handoff ensures that trace data and metadata are consistently formatted and ready for further processing.

Setting Up a Fluent Bit and OpenTelemetry Pipeline

Step 1: Install Fluent Bit

Here’s how you can install Fluent Bit:

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
fluent-bit --version

Step 2: Configure Fluent Bit for OpenTelemetry

Create a Fluent Bit configuration file to parse JSON log data and send it to the OpenTelemetry collector:

[SERVICE] 
    Log_Level info 

[INPUT] 
    Name tail 
    Path /var/log/app/*.log 
    Tag app.logs 

[FILTER] 
    Name parser 
    Match * 
    Parser json 

[OUTPUT] 
    Name opentelemetry 
    Match * 
    Host otel-collector 
    Port 4318 
    TLS On 
    Log_Response_Payload true

Step 3: Set Up OpenTelemetry Collector

The OpenTelemetry collector handles incoming data and routes it to tools like Jaeger for trace visualization. A sample YAML configuration for the collector:

receivers: 
  otlp: 
    protocols: 
      grpc: 
      http: 
exporters: 
  jaeger: 
    endpoint: "http://jaeger-collector:14268/api/traces" 
service: 
  pipelines: 
    traces: 
      receivers: [otlp] 
      exporters: [jaeger]

Deploy the collector with:

kubectl apply -f otel-collector.yaml

Kubernetes Observability with OpenTelemetry Operator | Last9

Learn how the OpenTelemetry Operator makes monitoring Kubernetes easier, so you can focus on what matters—keeping your apps running smoothly!

Last9Last9

Advanced Parsing and Instrumentation

Parsing Logs with Fluent Bit

Complex applications like Python-based microservices often generate logs with multiple fields. Fluent Bit’s powerful parsing capabilities can handle this.

For instance:

[PARSER] 
    Name custom_json 
    Format json 
    Time_Key timestamp 
    Time_Format %Y-%m-%dT%H:%M:%S

Instrumentation with OpenTelemetry

Adding instrumentation to your Python application ensures trace data flows through the observability pipeline. Here’s an example:

from opentelemetry import trace
from opentelemetry.exporter.jaeger import JaegerExporter

trace.set_tracer_provider(TracerProvider())
jaeger_exporter = JaegerExporter(agent_host_name='jaeger', agent_port=6831)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(jaeger_exporter))

What are OpenTelemetry Metrics? A Comprehensive Guide | Last9

Learn about OpenTelemetry Metrics, types of instruments, and best practices for effective application performance monitoring and observability.

Last9Book a Demo

Best Practices for Production

• Monitor Resources: Use Fluent Bit’s built-in metrics to monitor performance.
• Secure Data Transfer: Always enable TLS for sensitive workloads.
• Validate Configurations: Test your config files to avoid runtime issues.

How to Send Logs to Last9

Sending logs to Last9 is straightforward with Fluent Bit and the OpenTelemetry output plugin. Below is a detailed guide to get you started.

Prerequisites

Before configuring Fluent Bit, ensure you have:

1. A Last9 cluster set up. Follow the Getting Started guide to create one.
2. The following details from your Last9 Integrations page:
   • $last9_otlp_endpoint: Your Last9 OTLP endpoint.
   • $last9_basic_auth_header: OpenTelemetry Basic authorization header.
   • $last9_username: Your OpenTelemetry username.
   • $last9_password: Your OpenTelemetry password.

Fluent Bit Configuration

To forward logs to Last9, add this configuration to your Fluent Bit config file:

[OUTPUT]
    Name                 opentelemetry
    Match                kube.*
    Host                 otlp.last9.io
    http_user            <last9_username>
    http_passwd          <last9_password>
    Port                 443
    Logs_uri             /v1/logs
    Tls                  On
    Tls.verify           On
    logs_body_key_attributes true

Key Details:

• Match: This field should correspond to the logs you are collecting in the Input or Filter sections.
• Tls and Tls.verify: Ensures secure log transmission.
• logs_body_key_attributes: Enables structured attributes in the log body.

This minimal setup will start sending logs to Last9.

Advanced Configuration

Lua Functions for Log Parsing

To enhance your log data, use Lua scripts for custom parsing. For example, you can flatten nested fields in JSON logs:

function set_fields(tag, timestamp, record)
    local new_record = {}

    local function flatten_table(prefix, t)
        for k, v in pairs(t) do
            if type(v) == "table" then
                flatten_table(prefix .. k .. ".", v)
            else
                new_record[prefix .. k] = v
            end
        end
    end
    flatten_table("", record)

    return 1, timestamp, new_record
end

Adding the Lua Script

To apply this function, include the Lua filter in your Fluent Bit configuration:

[FILTER]
    Name lua
    Match kube.*
    Script flatten.lua
    Call set_fields

Adding Unique Tags

To distinguish logs by environment, cluster, or application, you can append unique tags using the modify filter:

[FILTER]
    Name modify
    Match kube.*
    Add env production

This practice is particularly useful for managing logs across multiple clusters or environments.

Conclusion

Fluent Bit and OpenTelemetry together form the backbone of observability pipelines. They simplify data collection, parsing, and transport, ensuring clear visibility into complex systems.

FAQs

How do you version control Kubernetes?
Store Kubernetes YAML manifests in a Git repository. Use GitOps tools like ArgoCD or Flux to sync changes with your cluster for reproducibility and tracking.

How do you start cloud-native observability with telemetry pipelines?
Set up pipelines using tools like Fluent Bit and OpenTelemetry to collect, process, and export logs, metrics, and traces tailored to your application.

How does Fluent Bit enable observability pipelines?
It collects, processes, and exports logs and metrics from multiple sources to tools like OpenTelemetry, Prometheus, or Jaeger.

How to change the default path of the Prometheus exporter in OpenTelemetry Collector?
Update the endpoint in the exporter’s section of the Collector config file:

exporters:  
  prometheus:  
    endpoint: "localhost:8888/new-path"  

Do you need an OpenTelemetry Collector?
Not always, but it helps centralize and manage large-scale telemetry pipelines efficiently while supporting multiple observability tools.

Should you expose metrics directly or via OpenTelemetry Collector?
Routing metrics through OpenTelemetry Collector improves flexibility and scalability over direct exposure, especially in complex setups.

How to configure Fluent Bit to export logs and metrics to OpenTelemetry Collector?
Use the otlp output plugin:

[OUTPUT]  
    Name otlp  
    Host otel-collector  
    Port 4317  
    tls On  

How to integrate Fluent Bit with OpenTelemetry for logs and metrics?
Configure Fluent Bit's otlp and http plugins to send data to OpenTelemetry Collector, using Fluent Bit’s filtering and parsing features.

Can Fluent Bit work with OpenTelemetry for observability?
Yes, Fluent Bit preprocesses logs and metrics, while OpenTelemetry handles advanced routing, processing, and exporting.

How to improve log and metric collection with Fluent Bit and OpenTelemetry?
Use Fluent Bit for filtering and parsing data, then forward it to OpenTelemetry Collector for enhanced processing and analysis.