Change Events

Why Change Events Matter?

Software systems and their observability are not just about the telemetry data emitted from these systems. The software systems also get affected by external change events. These events can be from domains such as deployment, configuration, or external 3rd party systems. Last9 allows tracking such change events along with other metrics, seamlessly providing more context to the system observability.

Every deployment, configuration tweak, and external change ripples through your system. Last9 Change Events capture these moments and overlay them directly onto your performance metrics, giving you the context to:

• Spot performance dips exactly when deployments hit
• Correlate configuration changes with error spikes
• Validate whether a new release is performing as expected
• Share complete system context with your team in one glance

This document will show how to start tracking change events with Last9.

How It Works

Last9 offers an HTTP API that can be used to track any domain change event. Each event has two states, start and stop, and both can be tracked with Last9. Once Last9 receives the event, it converts it into a metric that can be used with other metrics for querying using PromQL in Grafana dashboards or alerting.

The data flow for change events is as follows.

Last9 Change Events API

Last9 offers a REST HTTP endpoint that can be used to send the change events. The API endpoint is as follows.

curl -XPUT https://app.last9.io/api/v4/organizations/{org_slug}/change_events \
--header 'Content-Type: application/json' \
--header 'X-LAST9-API-TOKEN: Bearer <LAST9_API_WRITE_TOKEN>' \
--data-raw '{
  "timestamp": "2024-01-15T17:57:22+05:30",
  "event_name": "new_deployment",
  "event_state": "start",
  "data_source_name": "{your_cluster_name}",
  "attributes": {
    "service_name": "frontend",
    "deployment_environment": "production",
    "version": "v2.1.4",
    "team": "platform",
    "change_type": "hotfix"
  }
}'

API Parameters

Last9 will convert the events into a metric named last9_change_events.

Visualize Change Events in Service Dashboards

After pushing a change event, navigate to your service dashboard. If service_name and deployment_environment match your APM data, the change event appears as a contextual overlay on all performance charts:

What you’ll see:

• Red vertical line marking your exact deployment time
• Rich context popup with all your event details (version, team, change type)
• Performance correlation across APDEX, response time, and throughput charts
• Error patterns before, during, and after changes

This lets you instantly answer questions like:

• Did response time spike after deployment?
• Is the new release performing as expected?
• Which specific configuration change caused the error spike?

Visualize Change Events in Grafana

The change events can also be visualized in Grafana just like any other metrics. Query change events using PromQL:

last9_change_events{event_name="new_deployment", deployment_environment="production"}

Change Events Storage

It is possible that you might be using multiple Last9 clusters. In such scenario, you can choose to store the change events in a Last9 cluster of your choice. The optional data_source_name attribute is used to specify the cluster where change event will be stored.

If this attribute is not passed, then Last9 will store the change event in a default cluster designated for change events. The default cluster for change events is set as follows.

You can override this by specifying the data_source_name in the request payload. Obtain the cluster name from the Data Sources section as follows.

Event Naming Best Practices

Use consistent, descriptive names for your events:

• deployment_start / deployment_complete — for application deployments
• config_update_redis — for configuration changes
• feature_flag_toggle — for feature flag changes
• db_migration_start / db_migration_complete — for database migrations

Add meaningful context through attributes:

{
  "service_name": "frontend",
  "deployment_environment": "production",
  "version": "v2.1.4",
  "team": "platform",
  "change_type": "hotfix"
}

Migrating to Last9 Change Events

If you’re already tracking deployments with another observability tool, the table below maps common concepts to their Last9 equivalents. This makes it straightforward to replace your existing deployment event calls with the Last9 API.

Concept Mapping

What’s Different in Last9

• Events become metrics. Unlike visual-only annotations or markers, Last9 converts every change event into a Prometheus metric (last9_change_events). This means you can query, alert, and build recording rules on deployment events using PromQL — not just view them on a chart.

• No entity pre-registration required. Some tools require the target service to already exist before you can record a deployment against it. Last9 accepts events for any service_name immediately.

• No timestamp restrictions. Some tools limit event timestamps to 18–24 hours in the past. Last9 accepts any valid ISO8601 timestamp.

• Automatic service dashboard correlation. When service_name and deployment_environment match your APM data, change events automatically appear as overlays on APDEX, response time, throughput, and error charts — no manual dashboard configuration needed.

• Flexible attributes instead of rigid schemas. Instead of fixed fields like deploymentType or entityGuid, Last9 uses open-ended attributes. Add any key-value pairs relevant to your workflow (team, change_type, rollback, ticket_id, etc.).

Example: Replacing an Existing Integration

If you’re currently sending deployment events via a POST to another provider’s API, the migration is typically a one-line change in your CI/CD pipeline:

# Replace your existing deployment event call with:
curl -XPUT https://app.last9.io/api/v4/organizations/{org_slug}/change_events \
--header 'Content-Type: application/json' \
--header 'X-LAST9-API-TOKEN: Bearer <LAST9_API_WRITE_TOKEN>' \
--data-raw '{
  "event_name": "deployment",
  "event_state": "start",
  "attributes": {
    "service_name": "'"$SERVICE_NAME"'",
    "deployment_environment": "'"$DEPLOY_ENV"'",
    "version": "'"$GIT_SHA"'",
    "team": "'"$TEAM"'"
  }
}'

Native Integrations for Change Events

• Prodvana
• LaunchDarkly

Troubleshooting

Please get in touch with us on Discord or Email if you have any questions.