Google Cloud Managed Service for Apache Kafka

Use OpenTelemetry to collect Google Cloud Managed Kafka metrics and send telemetry data to Last9.

Prerequisites

Before setting up Google Cloud Managed Kafka monitoring, ensure you have:

• Google Cloud account with administrative privileges or Cloud Managed Kafka Admin privileges
• Access to a project in Google Cloud Platform (GCP)
• Google Cloud Managed Kafka APIs enabled - follow this guide to enable APIs
• Virtual Machine where you can run OpenTelemetry Collector to collect data from Google Cloud Managed Kafka and send to Last9

1. Install OpenTelemetry Collector

   You can install either rpm or deb package depending on flavor of operating system you are using.

   Note: systemd is required for automatic service configuration.

   • RPM Package
   • DEB Package

   Installing rpm package:

   wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.118.0/otelcol-contrib_0.118.0_linux_amd64.rpm
   sudo rpm -ivh otelcol-contrib_0.118.0_linux_amd64.rpm

   More installation options can be found here.

2. Configure Collector

   Use the following configuration for Otel Collector:

   receivers:
     googlecloudmonitoring:
       collection_interval: 60s
       project_id: gcp-project-id
       metrics_list:
         # Filter for specific metrics such cloud managedkafka
         - metric_descriptor_filter: 'metric.type = starts_with("managedkafka.googleapis.com")'
   
   processors:
     batch:
       timeout: 20s
       send_batch_size: 100000
       send_batch_max_size: 100000
     resourcedetection/gcp:
       detectors: ["gcp"]
   
   exporters:
     otlp/last9:
       endpoint: "{{ .Logs.WriteURL }}"
       headers:
         "Authorization": "{{ .Logs.AuthValue }}"
     debug:
       verbosity: detailed
   
   service:
     pipelines:
       metrics:
         receivers: [googlecloudmonitoring]
         processors: [batch, resourcedetection/gcp]
         exporters: [otlp/last9]

   Note: Replace gcp-project-id with your actual Google Cloud project ID.

   The complete list of supported metrics for GCP Kafka metrics can be found here.

   Read more about the Google Cloud monitoring receiver configuration options here.

3. Start Collector

   Run the otel collector using systemctl command:

   sudo systemctl start otelcol-contrib
   sudo systemctl status otelcol-contrib
   sudo systemctl restart otelcol-contrib

4. Check Collector Logs

   Monitor the collector logs to ensure it’s working correctly:

   sudo journalctl -u otelcol-contrib -f

Verification

To verify the setup is working:

1. Check that the collector service is running
2. Review the collector logs for any errors
3. Log into your Last9 account to confirm Managed Kafka metrics are being received

Need Help?

If you encounter any issues or have questions:

• Join our Discord community for real-time support
• Contact our support team at support@last9.io