AWS CloudWatch Logs — push via Firehose or Lambda

There are two ways to get CloudWatch logs into Last9:

Pull (Collector-based) — run an OpenTelemetry Collector with the awscloudwatch receiver. Covered in AWS CloudWatch Logs.
Push (Firehose or Lambda forwarder) — this page. CloudWatch streams logs out as they’re written. No long-running Collector. Lower latency. AWS-native data plane.

Use push when you do not want to manage a Collector, when latency matters, or when the pull-based approach has produced duplicate log entries for Lambda invocations (see Known issues).

Which push path?

Path	Use when
Kinesis Data Firehose	You already use Firehose, want managed buffering/retries, or expect sustained high log volume.
Lambda forwarder	You don’t use Firehose, want minimal infra, or need custom log transformation per event.

Both end at the Last9 OTLP/HTTP logs endpoint.

Prerequisites

AWS account with IAM permission to manage CloudWatch subscription filters
CloudWatch log group(s) you want to forward
Last9 OTLP/HTTP logs endpoint + write token from Integrations → OpenTelemetry

Option A: Kinesis Data Firehose

Create a Firehose delivery stream

In the AWS Console, open Kinesis → Data Firehose → Create delivery stream.
- Source: Direct PUT
- Destination: HTTP Endpoint
- HTTP endpoint URL: your Last9 logs ingestion URL (e.g. https://otlp-aps1.last9.io/v1/logs)
- Access key: your Last9 write token (sent in X-API-Key or Authorization header per the endpoint contract — see the integration page for the exact name)
- Content encoding: GZIP (Last9 accepts both compressed and uncompressed)
- Retry duration: keep AWS default (300 s)
- S3 backup: enable a backup S3 bucket for failed deliveries — strongly recommended
Create the IAM role for Firehose → HTTP

Firehose assumes a service role to deliver. Trust policy:
```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "firehose.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}
```
Inline policy must grant logs:PutLogEvents to the Firehose error-log group plus s3:PutObject to the failure-backup bucket.
Create the CloudWatch subscription filter

Subscription filter pipes log group events into Firehose:
```
aws logs put-subscription-filter \
  --log-group-name "/aws/lambda/my-fn" \
  --filter-name "to-last9" \
  --filter-pattern "" \
  --destination-arn "arn:aws:firehose:<region>:<account>:deliverystream/<stream-name>" \
  --role-arn "arn:aws:iam::<account>:role/CWLtoFirehoseRole"
```
The role-arn here is a separate role that allows CloudWatch Logs to write to Firehose (firehose:PutRecord, firehose:PutRecordBatch). Repeat the subscription filter per log group, or use aws logs describe-log-groups to script bulk creation.
Verify delivery

In the Firehose console, check Monitoring tab:
- DeliveryToHttpEndpoint.Success should be > 0 within a few minutes.
- DeliveryToHttpEndpoint.Records shows the per-batch record count.
In Last9, open Logs and filter for the matching aws.cloudwatch.log_group attribute.

Option B: Lambda forwarder

Author the forwarder Lambda

The function receives a awslogs.data payload (gzipped, base64-encoded), decodes it, and POSTs to Last9 OTLP/HTTP.

# lambda_function.py
import base64, gzip, json, os, urllib.request

LAST9_URL = os.environ["LAST9_LOGS_URL"]
LAST9_TOKEN = os.environ["LAST9_TOKEN"]

def lambda_handler(event, _ctx):
    payload = base64.b64decode(event["awslogs"]["data"])
    decoded = json.loads(gzip.decompress(payload))

    otlp_body = build_otlp_logs_request(decoded)  # convert to OTLP/JSON shape

    req = urllib.request.Request(
        LAST9_URL,
        data=json.dumps(otlp_body).encode("utf-8"),
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Basic {LAST9_TOKEN}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        if resp.status >= 300:
            raise RuntimeError(f"last9 returned {resp.status}: {resp.read()}")
    return {"forwarded": len(decoded.get("logEvents", []))}

build_otlp_logs_request maps each CloudWatch logEvents entry to OTLP LogRecord with body, timestamp, and resource attributes (aws.cloudwatch.log_group, aws.cloudwatch.log_stream).

IAM role for the Lambda

The Lambda needs the standard AWSLambdaBasicExecutionRole (for its own CloudWatch logs) — no extra perms needed since CloudWatch invokes the function with the log data inline.

Allow CloudWatch Logs to invoke the Lambda

aws lambda add-permission \
  --function-name last9-log-forwarder \
  --statement-id cw-logs-invoke \
  --principal logs.amazonaws.com \
  --action lambda:InvokeFunction \
  --source-arn "arn:aws:logs:<region>:<account>:log-group:*:*"

Subscription filter pointing to the Lambda

aws logs put-subscription-filter \
  --log-group-name "/aws/lambda/my-fn" \
  --filter-name "to-last9" \
  --filter-pattern "" \
  --destination-arn "arn:aws:lambda:<region>:<account>:function:last9-log-forwarder"

Verify delivery

Tail the forwarder’s own CloudWatch logs for delivery counts or errors. In Last9 Logs, filter by aws.cloudwatch.log_group to confirm arrival.

Permissions reference

Role	Service	Required actions
`CWLtoFirehoseRole` (Option A)	CloudWatch Logs assumes this	`firehose:PutRecord`, `firehose:PutRecordBatch` on the target stream
Firehose delivery role (Option A)	Firehose assumes this	`s3:PutObject` on backup bucket, `logs:PutLogEvents` on Firehose error log group
Forwarder Lambda execution role (Option B)	Lambda assumes this	`AWSLambdaBasicExecutionRole` (managed policy)
Lambda invoke permission (Option B)	CloudWatch Logs invokes Lambda	Resource policy on the Lambda, principal `logs.amazonaws.com`

Known issues

Duplicate logs via the pull-based awscloudwatch receiver. When using the OpenTelemetry Collector receiver path (the pull approach), some Lambda invocations produce duplicate log entries — the same log line appears twice in Last9 while CloudWatch itself shows only one occurrence. Root cause is under investigation. If you hit this, switch to the push approach on this page.
Subscription filter limits. Each log group supports up to two subscription filters. If you already pipe a log group elsewhere (e.g., another aggregator), the second filter to Last9 is the last slot.
Firehose backup bucket fills up. If Last9 ingestion is briefly unreachable, Firehose retries for up to 300 seconds, then writes failed records to the S3 backup bucket. Set a lifecycle policy on the backup bucket to expire old failures — otherwise it grows unbounded.

Choosing between push and pull

Concern	Push (this page)	Pull (Collector receiver)
Infra to manage	Firehose stream or one Lambda	Long-running Collector instance
Latency	Seconds	Seconds to minutes (poll interval)
Cost (AWS side)	Firehose: per-GB ingest + delivery	Collector: per-API-call (DescribeLogStreams, FilterLogEvents)
Duplicate-log bug	Not affected	Affected for some Lambda invocations
Retry semantics	Firehose: managed retry + S3 backup; Lambda: at-most-once + DLQ if configured	Collector retries via OTLP exporter retry policy
Best for	Pure AWS-native flow, no Collector available, high volume	Existing Collector deployment, log groups not covered by subscription filters

AWS CloudWatch Logs (Collector pull) — the existing receiver-based path
AWS Lambda OTel instrumentation — for trace + log correlation on Lambda

Troubleshooting

Logs not arriving in Last9. Check the delivery surface first: for Firehose, the Monitoring tab (DeliveryToHttpEndpoint.Success should be > 0); for the Lambda forwarder, the function’s own CloudWatch logs for errors. Confirm the subscription filter is attached to the right log group and the endpoint URL + write token are correct.
Duplicate log entries, subscription filter limits, and Firehose backup growth. See Known issues above.

Please get in touch with us on Discord or Email if you have any questions.