This is the full developer documentation for Last9 # Introduction > Learn how to get started with Last9 [school](/docs/introduction/) ## [Getting Started](/docs/introduction/) [Understand what Last9 is and how to quickly start sending data](/docs/introduction/) [cell\_tower](/docs/control-plane/) ## [Control Plane](/docs/control-plane/) [Manage your telemetry data, its configurations, and its lifecycle](/docs/control-plane/) [wysiwyg](/docs/logs/) ## [Logs](/docs/logs/) [Explore your logs data, its details, and related telemetry](/docs/logs/) [waterfall\_chart](/docs/traces/) ## [Traces](/docs/traces/) [Explore your trace spans, its dependencies and timeline charts, and span details](/docs/traces/) [monitor\_heart](/docs/rum/) ## [Real User Monitoring](/docs/rum/) [Monitor your web application's performance from your users' perspective](/docs/rum/) [notifications\_active](/docs/alerting-overview/) ## [Alerting](/docs/alerting-overview/) [Set up alerts, pattern matching, receive notifications, an IaC tool for alerting](/docs/alerting-overview/) [extension](/docs/integrations/) ## [Instrumentation](/docs/integrations/) [Send data via OpenTelemetry, Prometheus, AWS Cloudwatch, and more](/docs/integrations/) [support](/docs/howto/) ## [Tutorials](/docs/howto/) [Common how-tos for Prometheus, Kubernetes, VictoriaMetrics, etc.](/docs/howto/) [help](/docs/faqs/) ## [FAQs](/docs/faqs/) [Frequently asked questions about Last9 — what, why, how](/docs/faqs/) ## Other Resources [assignment Changelog ](https://last9.io/changelog/)[public Blog ](https://last9.io/blog/)[group Community ](https://discord.com/invite/Q3p2EEucx9/)[comment X / Twitter ](https://x.com/last9io/)[smart\_display Youtube](https://youtube.com/@last9/) # Access Policies > Leverage Last9's access policies to perform traffic shaping of time series data in real-time. Last9 supports automatic data tiering of the metrics based on retention policies. These data tiers have different retention policies. E.g., Blaze Tier stores data for the last two hours, whereas Hot Tier stores data for the last six months. Depending on the use case, the tiers are designed to access their metrics from a fast or slow tier. It is extremely crucial to ensure that traffic for real-time alerting is always prioritized and served from the fastest Blaze tier. The Grafana queries can be served from the Hot tier without conflicting with alerting. Access Policies ensure that one can create these policy guardrails to ensure the metrics data is accessible from a specific tier based on its purpose. Note Access Policies are specific to a Last9 cluster. Different clusters can have different Access Policies. Each Access Policy is associated with a Token created for a Tier. Tokens allow ACL for time series data by providing a way to access data from specific tiers for either `read` `write` or both operations. ## Setting up Tokens To achieve this, create a read token first from Settings -> Tokens. [Creating a Read Token in Levitate](https://www.youtube.com/embed/qfdUYwAMZvw) ## Creating Access Policy Once the Token is created, one can create an Access Policies from Settings -> Policies. [Creating Access Policy in Levitate](https://www.youtube.com/embed/0j_N9CKyigY) That’s it, you don’t have to change anything more. Just use the token associated with the Alerting policy to configure alertmanager and the token associated with the visualization policy to [configure Grafana](/docs/grafana-config/). Last9 will take care of performing traffic shaping in real-time. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Alert Groups > Overview of Alert Groups An Alert Group is a container for Indicators (ie PromQL queries) and Alert Rules which evaluate these queries. Alerts that are generated by Alert Rules, send notification on the Channels that are configured within the Alert Group. ## Creating an Alert Group Note The following steps are to create an Alert Group using the UI. To use a gitops workflow, see [Declarative Alerting via IaC](/docs/alerting-via-iac/). Alert Groups created via IaC have the option to [disable any edits from the UI](/docs/alerting-via-iac/#entities) to avoid configuration conflicts. 1. Navigate to **Home** → **Alert Studio** → **Alert Groups** and click on **Add New**   2. Assign a descriptive name to the Alert Group and Select the data source from which you like to query metrics and click **Create.**  Ensure that you select the correct Data Source (Last9 Cluster) from which you like to query the metrics from or else the Alert Rules will not evaluate. Pro Tip - You can also use Last9’s Health Cluster as a data source to setup alerting to watch your Cluster’s health .. after all q*uis custodiet ipsos custodes?* 3. Click on the **Alert Group** to navigate to your newly created Alert Group  If this is your first Alert Group, next you would need to create the first Indicators followed by creating an Alert Rule. ## Deleting an Alert Group Deleting an Alert Group deletes all the Alert Rules, Indicators and all the generated Alerts. To delete an Alert Group: 1. Navigate to **Home** → **Alert Studio** → **Alert Groups** 2. Click the **…** button besides the Alert Group you wish to delete and select **Delete**   ## Features ### Labels Labels are are named pairs (key:value pairs) that add additional information and context to Alert Groups. To add labels to an Alert Group: 1. Click **Edit** in to update Alert Group meta fields  2. In the Labels card, click **Add Labels** to add a new label Labels must have a unique key (ie a name). Label value can contain alphanumeric text.  3. Click **Done** to exit edit mode To edit or delete labels from an Alert Group: 1. Click **Edit** in to update Alert Group meta fields 2. In the Labels card, hover on the label you wish to edit or delete. Click on the appropriate button to edit or delete the label  3. Click **Done** to exit edit mode ### Tags Tags help you categorize multiple Alert Groups To add tags to an Alert Group: 1. Click **Edit** in to update Alert Group meta fields 2. In the Details card, click on Assign Tags 3. Search from existing or add a new Tag to the Alert Group  4. Click **Done** to exit edit mode To remove tags from an Alert Group: 1. Click **Edit** in to update Alert Group meta fields 2. In the Details card, hover on the tags you wish to edit or delete. Click on the X button to remove tag 3. Click **Done** to exit edit mode ### Links Alert Group links allow you to add links to external resource used by your team. These can be very helpful for your team to quickly navigate to resources like CloudWatch, runbooks or repos, etc. Links are named URLs which can have any custom with several suggested links. To add links to Alert Groups: 1. Click **Edit** in to update Alert Group meta fields 2. In the Links card, add a link to a suggested field or add your own custom name for the link  3. Click **Done** to exit edit mode To edit or remove links from an Alert Group: 1. Click **Edit** in to update Alert Group meta fields 2. In the Links card, hover on the link you wish to edit or delete. Click on the appropriate button to edit or delete the link 3. Click **Done** to exit edit mode ## Alert Group Settings ### Channels Notifications from Last9 are sent on [Notifications Channels](/docs/notification-channels/). Ensure that you have at least one Notification Channel configured, before trying to add an Channels to an Alert Group To add a notification channel: 1. Navigate to `Home` → `Alert Studio` → `Alert Groups` → *Select an Alert Group* Press the ⚙️ icon on the top right to view Alert Group settings.  2. Under the Channels tab you can assign channels as per Alerts severity level, ie you can set different (or same) channels for Threat and Breach severity alerts  Slack integration also allows you to append additional *@mentions* to tag a person or group  3. The configured Alert Channel will now start receiving alerts  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Alert Rules > Alert Rules Overview ## Alert Rules Overview Alert Rules evaluate Indicators using algorithms and configured thresholds/sensitivity to generate Alerts. Alerts can be visualized using Health and sent to configured Notification Channels. ## Creating an Alert Rule Before you can configure an Alert Rule, you need at least one Indicator to be created in the Alert Group. To create an Alert Rule: 1. Navigate to the Alert Group in which you would like to create the Alert Rule: **Home** → **Alert Studio** → **Alert Groups** → *Select an Alert Group* → **Alert Rules** Tab  2. The following details are required for an Alert Rule: 1. **Rule Name**: Use a descriptive Alert Rule that can be easily identified by your team. This will be also sent as the part of notifications 2. **Indicator**: Select the indicator for which you would like to create the Alert Rule. If you have not created an Indicator, follow the steps as mentioned here 3. **Edit Label Filter**: … 4. **Algorithm**: Select from the available algorithms. For a detailed guide on how various algorithms, refer to this [guide](/docs/anomalous-pattern-detection-guide/) In this tutorial, we will step up an alert using Static Threshold 5. **Threshold / Sensitivity**: Specify the Threshold (or Sensitivity in case of Anomaly Detection algorithms) and the Operator (example: Alert when the Indicator value is *greater than or equal* to 10) 6. **Alert Sensitivity**: Using Alert Sensitivity you can define how reactive is the Alert Rule. Alert Sensitivity requires two inputs: * Total Minutes: This is the total duration of the rolling time window during which the Indicator is evaluated, with the maximum allowed duration being 60 minutes. (All Alert Rules are evaluated in one-minute intervals) * Bad Minutes: This value represents the number of minutes within the evaluation window that exhibit undesirable or unexpected behavior. These “bad” minutes need not be consecutive If the number of ‘Bad Minutes’ exceeds the predefined limit within the ‘Total Minutes’ rolling window, an alert is generated. A rolling time evaluation window offers continuous analysis by constantly updating the period under evaluation. It allows for immediate reaction to issues as they develop, rather than waiting for a static hourly evaluation to complete. This mechanism ensures that users are notified only when there is a significant deviation in expected metric performance, helping to avoid unnecessary alerts for minor or inconsequential fluctuations. 7. **Severity Level**: Helps you provide additional context to Alert Rule by categorizing alerts as either Threat or Breach. We indicate Threat alerts in amber and Breach as red colors 8. **Notification Group**: For Indicators with multiple timeseries, you can choose to receive individual alerts for every single timeseries or to group them into a single alert. We recommend that you group alerts as ungrouping them can lead to noise being generated 9. **Annotations** (Optional): Annotations are optional information labels in `key:value` format that can be sent with every Alert notification. You can use these specify additional description, Runbooks or trigger complex workflows in your incident management systems 10. When the threshold is configured, we generate a preview to help you visualize what values of the Indicators are considered anomalous. The number of timeseries that will be evaluated every minute by the alert rule are indicated Click **Save Rule** to enable alerting for this rule. To start receiving notification for this Alert Rule, ensure that at least one Notification Channel as been configured for this Alert Group.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Alerting Overview > Overview of Last9's Alerting Capabilities Last9 comes with complete monitoring support, including alerting and notification capabilities. Irrespective of your tool choice, a few problems plague today’s alerting journey — coverage, fatigue, and cleanup. Unfortunately, there are no easy answers to these complex problems. However, with advanced features like Pattern-based Alerting and a redesigned Alert Manager designed with High Cardinality in mind, Last9 helps you stay ahead. In addition to being fully PromQL compatible, it provides features like a real-time alert monitor and historical health view. You can also perform advanced tasks, such as correlating them with events while focusing on the desired outcome of keeping up with constantly evolving infrastructure and Services. In addition to being fully PromQL compatible, it provides features like a real-time alert monitor and historical health view. You can also perform advanced tasks, such as correlating them with events while focusing on the desired outcome of keeping up with constantly evolving infrastructure and Services. Alerting with Last9 starts by creating an **Alert Groups** which contain one or more **Alert Rules.** These Alert Rules evaluate the PromQL queries which are defined as **Indicators** in the Alert Group. Using **Alert Monitor** you can view a live updating stream of all your Alert Rules across all Alert Groups. In the following section we dive deeper into each of these components. ## Enabling Alerting Studio All new orgs needs to request access to Alert Studio, this is a one time action and takes about 30 minutes to get completed (usually done much faster). To enable Alert Studio: 1. Navigate to **Home** → **Alert Studio** and click on the **Request To Enable** button  2. Once the request has been sent, come back in some time to start using Alert Studio *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Declarative Alerting via IaC > Last9 supports configuring alerts and notifications automatically using a Python-based SDK tool which takes care of infrastructure changes Configurations for alerting and notifications for observability at scale are hard to start, maintain and fix manually, just like provisioning infrastructure at scale. With infrastructure changes, it’s important that the observability stack also catch up with it to avoid the chances of issues because of a lack of observability or black swarm events. Last9 has introduced.`l9iac` tool to solve the exact same problem. ## Installation Last9’s IaC (Infrastructure as Code) tool is available as a Docker image, providing a consistent and isolated environment for automating entity creation and alert configuration. 1. **Pull the Docker Image** ```bash docker pull last9system/iac:latest ``` The image is available on [DockerHub](https://hub.docker.com/repository/docker/last9system/iac/general). 2. **Prepare Your Working Directory** Create a directory containing: * Your IaC YAML files * `config.json` with your refresh tokens ([see file structure](#configuration-file-structure)) * Space for the state lock file 3. **Run the Docker Container** ```bash docker run --name l9iac -d -v : last9system/iac: ``` Example: ```bash docker run -d -v /home/user/iac-files:/app/rules last9system/iac:2.4.2 ``` > 💡 **Note**: If using Docker Desktop, ensure file sharing is enabled for the volume mount. 4. **Execute IaC Commands** ```bash docker exec -it l9iac -mf -c ``` Example: ```bash docker exec -it bcdea6660fd4 l9iac -mf /app/rules/alert-rules.yaml -c /app/rules/config.json plan ``` ## Configuration File Structure The IaC tool requires a `config.json` file with the following structure: ```json { "api_config": { "read": { "refresh_token": "", "api_base_url": "https://app.last9.io/api/v4", "org": "" }, "write": { "refresh_token": "", "api_base_url": "https://app.last9.io/api/v4", "org": "" }, "delete": { "refresh_token": "", "api_base_url": "https://app.last9.io/api/v4", "org": "" } }, "state_lock_file_path": "state.lock" // Should be in the same directory as model_file and config_file } ``` ### Important Notes * The `refresh_token` values can be obtained from the [API Access](https://app.last9.io/api-access) page in the Last9 dashboard ([know more](/docs/getting-started-with-api/)) * The `` can be obtained from the app’s URL: `app.last9.io/v2/organizations/` * For on-premise Last9 setups, contact to get the correct `api_base_url` * The `state_lock_file_path` should be accessible from the directory where you run the IaC commands ## Quick Start 1. Create a *YAML* as per your alert rule configuration **Example**: notification\_service\_am.yaml ```yaml # notification_service_am.yaml entities: - name: Notification Backend Alert Manager type: service_alert_manager data_source: prod-cluster entity_class: alert-manager external_ref: unqiue-slug-identifier indicators: - name: availability query: count(sum by (job, taskid)(up{job !~ "ome.*"}) > 0) / count(sum by (job, taskid) (up{job=~".*vmagent.*", job !~ "ome.*"})) * 100 - name: loss_of_signal query: 'absent(up{job !~ "ome.*"})' alert_rules: - name: Availability of notification service should not be less than 95% description: The error rate (5xx / total requests) is what defines the availability, lower value means more degradation indicator: availability less_than: 99.5 severity: breach bad_minutes: 3 total_minutes: 5 group_timeseries_notifications: false annotations: team: payments description: Error Rate described as number of 5xx/throughput runbook: https://notion.com/runbooks/payments/error_rates_fixing_strategies ``` 2. Prepare the configuration file for running the IaC tool The configuration file has the following structure. It is a JSON file. ```json { "api_config": { "read": { "refresh_token": "", "api_base_url": "https://app.last9.io/api/v4", "org": "" }, "write": { "refresh_token": "", "api_base_url": "https://app.last9.io/api/v4", "org": "" }, "delete": { "refresh_token": "", "api_base_url": "https://app.last9.io/api/v4", "org": "" } }, "state_lock_file_path": "state.lock" } ``` * The `refresh_token` can be obtained from the API Access page from the Last9 dashboard. You need to have `refresh_tokens` for all 3 operations - read, write and delete as the `l9iac` tool will perform all these 3 actions while applying the alert rules. * The `` is your organization’s unique slug in Last9. It can be obtained from the API access page of Last9 dashboard.i * The default `api_base_url` is `https://app.last9.io/api/v4`. If you are on an on-premise setup of Last9, contact to get the `api_base_url`. * The `state_lock_file_path` is name of the file where `l9iac` will store the state lock of current alerting state(on the same lines of terraform state.lock). 3. Run the following command to do a dry run for the changes ```shell l9iac -mf notification_service_am.yaml -c config.json plan ``` 4. Run the following command to apply the changes ```shell l9iac -mf notification_service_am.yaml -c config.json apply ``` Tip We will provision the GitOps flow that will run `apply` command once changes are merged to the master branch in the GitHub repo. Contact for more details. ## Schema Here is the complete schema for generating the above `.yaml` file: ### Entities | Field | Type | Unique | Required | Description | | --------------------------------------------------------- | --------------- | ------ | -------- | ---------------------------------------------------------------------------------------------- | | name | string | false | true | Name of the entity (alert manager) | | type | string | false | true | Type of the entity | | external\_ref | string | true | true | External reference for the entity, it’s a unique slug format identifier for each alert manager | | [adhoc\_filter](#common-rule-filters-adhoc-filters) | object | false | optional | List of common rule filters for the entity | | [alert\_rules](#alert-rules) | array | false | optional | List of alert rules for the entity | | data\_source | string | false | optional | Data source | | data\_source\_id | string | false | optional | The ID of the data source | | description | string | false | optional | Description of the entity | | entity\_class | string | false | optional | Denotes the class of the entity. Supported values: `alert-manager` | | [indicators](#indicators) | array | false | optional | List of indicators for the entity | | labels | object | false | optional | List of key value pairs of group label names and values | | [links](#links) | array | false | optional | List of links associated with the entity | | namespace | string | false | optional | The namespace of the entity | | [notification\_channels](#notification-channels) | string OR array | false | optional | List of notification channels applicable to the entity | | tags | array | false | optional | List of tags for the entity | | team | string | false | optional | The team that owns the entity | | tier | string | false | optional | Tier of the entity | | [ui\_readonly](/docs/alert-group#creating-an-alert-group) | boolean | false | optional | Disable any sort of edits to the alert group from the UI | | workspace | string | false | optional | Workspace of the entity | ### Common Rule Filters (Adhoc Filters) | Field | Type | Unique | Required | Description | | ------------ | ------ | ------ | -------- | ------------------------------------------------- | | labels | object | false | required | List of key value pairs of label names and values | | data\_source | string | false | required | Defaults to entity’s data source | ### Alert Rules | Field | Type | Unique | Required | Description | | -------------------------------- | ---------- | ------ | -------- | -------------------------------------------------------------------------------------------- | | name | string | true | required | Rule name that describes the alert | | indicator | string | false | required | Name of the indicator | | bad\_minutes | integer | false | required | Number of minutes the indicator must be in a bad state before alerting | | total\_minutes | integer | false | required | Total number of minutes the indicator is sampled over | | description | string | true | optional | Description for an alert rule that is included in the alert payload | | expression | string | false | optional | Alert rule expression, to be used only for pattern-based alerts | | greater\_than | number | false | optional | Alert triggers when the indicator value is greater than this | | greater\_than\_eq | number | false | optional | Alert triggers when the indicator value is greater than or equal to this | | less\_than | number | false | optional | Alert triggers when the indicator value is less than this | | less\_than\_eq | number | false | optional | Alert triggers when the indicator value is less than or equal to this | | equal\_to | number | false | optional | Alert triggers when the indicator value is equal to this | | not\_equal | number | false | optional | Alert triggers when the indicator value is not equal to this | | group\_timeseries\_notifications | boolean | false | optional | If multiple impacted time series in an alert need to be grouped as one notification or not | | is\_disabled | boolean | false | optional | Whether the alert is disabled or not | | label\_filter | map/object | false | optional | Mapping of the variables present in the indicator query and their pattern for the alert rule | | mute | boolean | false | optional | If alert notifications need to be muted or not | | [runbook](#runbook) | | false | optional | Runbook link to be included in the alert payload | | severity | string | false | optional | Can be a `threat` or `breach` | #### Runbook | Field | Type | Unique | Required | Description | | ----- | ------ | ------ | -------- | ------------------------------------------------ | | link | string | false | required | Runbook link to be included in the alert payload | ### Indicators | Field | Type | Unique | Required | Description | | ------------ | ------ | ----------------------------------------- | -------- | ------------------------------------ | | name | string | true, uniqueness enforced at entity level | required | Name of the indicator | | query | string | false | required | PromQL query for the indicator | | data\_source | string | false | optional | Data Source of the indicator (Last9) | | description | string | false | optional | Description of the indicator | | unit | string | false | optional | Unit of the indicator | ### Links | Field | Type | Unique | Required | Description | | ----- | ------ | ------ | -------- | ------------------------ | | name | string | false | required | Display name of the link | | url | string | false | required | URL of the link | ### Notification Channels | Field | Type | Unique | Required | Description | | -------- | ----------------------- | ------ | -------- | ------------------------------------------------------------------------------------------------- | | name | string | false | required | Name of the notification channel | | type | string | false | required | Type of notification channel. Allowed values: `slack`, `pagerduty`, `opsgenie`, `generic_webhook` | | mention | string OR list (string) | false | optional | Only applicable to Slack. The user(s) to tag in the alert message | | severity | string | false | optional | Severity of the alerts sent through this channel. Allowed values: `threat`, `breach` | Before a notification channel can be used in IaC, it needs to be configured. Please see [Notification Channels](/docs/notification-channels/) for more details. ## Supported Macros by IaC * `low_spike (tolerance, metric)` * `high_spike (tolerance, metric)` * `decreasing_changepoint (tolerance, metric)` * `increasing_changepoint (tolerance, metric)` * `increasing_trend (tolerance, metric)` * `decreasing_trend (tolerance, metric)` *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Anomalous Pattern Detection Guide > An overview of Pattern Detection algorithms supported by Last9 and guidelines on when to use them. ## Supported algorithms Last9’s Alert Studio supports following algorithms for anomalous pattern detection. ### High Spike The high spike algorithm is designed to detect sudden increases in signal values, particularly when the increase occur within a short time frame. It is especially useful for detecting sudden jumps in the number of 4xx, throughput, and edge hits, which are a good fit for the high spike. The high spike algorithm compares the current data point with the last 60 minutes worth of data points to check whether a given point has a considerably large amplitude or not. Tip The **high spike** algorithm is designed to detect sudden increases in signal values over a short period of time, while the **low spike** algorithm is useful for identifying sudden drops in signal values. #### Eligible Signals for High Spike Signals similar to following can be used for high spike pattern detection.   ### Low Spike The low spike algorithm is particularly helpful in identifying sudden drops in signal values, Signals such as CPU utilization, cache hit rate, and availability are good fits for the low spike algorithm. The algorithm compares the current data point with the previous 60 minutes of data to determine whether a given point represents a significant drop or not. Tip The **level change** algorithm is different from high/low spike algorithms in that it detects when a data pattern has changed rather than detecting a single or few large jumps or drops. #### Eligible Signals for Low Spike    ### Level Change The level change algorithm detects the point at which data begins to exhibit a new pattern that is different from the old. The data will have different patterns before and after the level change. To determine if an incoming point is a candidate level change, the algorithm checks if it is different (too high or too low) from the data over the last hour. #### How is this different from a high/low spike? If the data shows a single or a few large jumps or drops, this algorithm will not detect them. A single different value or even a few of them do not necessarily indicate that the pattern has changed or that there is a new pattern. #### Eligible Signals Level change Algorithm    ### Trend Deviation A trend algorithm is a useful tool for detecting deviations in a signal from its expected pattern compared to its behaviour over a certain number of previous days. * For each incoming data point, collect relevant data from the past (this is the reference period or seasonality). It is not necessary to collect all past data * To determine if an incoming data point is an anomaly, compare it with a reference period from the past Tip The **trend deviation** algorithm detects when unexpected data points or patterns have occurred compared to the previous 7 days of data. #### Illustrations In the below scenario, the trend algorithm will detect anomalies at 10 a.m. (red circled) because it is not expected when compared to its previous days (reference period- red-colored rectangular boxes)  In Figure 2, if we observe the signal pattern carefully, the trend algorithm will not detect any anomalies at 10 a.m. (red-circled) because the point or peak is expected when compared to its previous days.  In Figure 3, if we observe the signal pattern carefully, the trend algorithm will detect anomalies at 10 a.m. (red circle). Although it is a repetitive peak, the amplitude of this peak is much higher than the peaks of the previous days.   #### Eligible Signal for Trend (increasing / Decreasing)    ## How to select the right algorithm? Each algorithm matches a specific pattern and raises an alert when it is encountered. To use it effectively, the user should follow the below process when choosing an algorithm. 1. **Define normal behaviour**. It is important to know what the acceptable behaviour of the signal is. One simple way of doing this, is to look at the signal over the relevant span, and try and point out the timestamps where the signal deviates from the normal behaviour, and you would like to get alerted. Remember, an algorithm is not able to detect deviation from normal behaviour, if a trained human cannot 2. **Identify the anomalous pattern(s) in the signal**. Different signals exhibit different anomalous behaviour. Some might show spikes, some might show level change. Eg, for a signal like CPU usage, a sharp spike that returns to baseline may be perfectly normal behaviour, but for a business metric it may not. Knowledge of the underlying processes that generate the signal is essential to determine the correct pattern 3. **Check if a PromQL expression captures the intended deviation better**. PromQL is a very powerful language with many functions. For detecting deviations that can be defined in terms of relative values, percentages, or some rollup formulae on historical data, prefer defining the PromQLs accordingly For eg., if a signal has a normal range if ```text it stays in a range of minimum and maximum of the 15 minute medians over the last 2 days, with a tolerance of 20% ``` The PromQL to detect this would be ```text s < min_over_time(median_over_time(s)[15m])[2d]*0.8 || s > max_over_time(median_over_time(s)[15m])[2d]*0.8 ``` where `s` is the original signal metric. 4. **Check the Algorithm**. If the pattern that you want to match cannot be expressed easily like demonstrated above, check if any built-in algorithm can satisfactorily match the pattern. Remember that each algorithm has its own limitations, and it is important to understand them when working with signals Signals that don’t meet the requirements of any of the algorithms should be handled differently. By selecting the appropriate algorithm and adjusting the sensitivity to match your use case, you can improve the accuracy of these pattern detections. ## When not to choose a pattern matching algorithm? As a rule of thumb, a pattern matching algorithm should be chosen in situations where a human who is looking at the plot can define, with a high level of accuracy, where an alert should be generated and where it should not be generated. If, by looking at the plot, it is not possible for a human to determine the alert points, it is highly unlikely that any of the above algorithms can succeed. Below are a few signals which are not a good fit for any one of the above algorithms  The above signal is mostly zero-valued. Applying high spikes, low spikes, or increasing trend to these types of signals will cause each and every peak to be alerted. It is better to use a static threshold instead of pattern matching functions on these types of signals. Note Alert Studio supports static threshold based alerting as well. ***  This signal is a discrete-time signal. At any given point in time, it can have one of three possible values (1000, 1500, 2000) or no value at all. For this type of signal, a static threshold may be a better choice. ***     These signals should be handled differently as they do not follow a predictable pattern, making it difficult to detect patterns. ## Summary While deciding the pattern detection algorithm, it is important to understand the nature of the signal and the objective of the alert before choosing the algorithm. This guide describes a few guidelines which can be used while deciding the pattern algorithms with Last9 Alert Studio. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Cardinality Explorer > Identify metrics and labels impacted by cardinality. Cardinality Explorer helps you understand how the cardinality for metrics in a Cluster is trending. This powerful feature enables you to diagnose cardinality-related challenges with your metrics. ## Using Cardinality Explorer To view an individual metric’s cardinality contribution: 1. Navigate to **Control Plane** → **Cardinality Explore** & Select the Cluster you wish to explore  A report with all your metrics in the selected date is generated. When the current day is selected, the data shown in the table will continue to update throughout the day. The report also highlights metrics if they have crossed or are nearing their cardinality quota limits: * Metrics in Red have crossed their daily cardinality quota * Metrics in Amber have crossed 80% of their daily cardinality quota 2. To view how an individual metric is contributing towards the Cluster’s cardinality, click on a metric from the table:  You can use this detail view to diagnose issues with the selected metric using: * **Cardinality Trend:** Using this graph you can observe how the cardinality of the selected metric has trended over the last 7 days. A sudden spike or a dip may indicate unexpected changes to the cardinality of this metric * **Cardinality Details: V**iew all the metric’s label names and their top 5 occurring label values for a selected date. Using this, you can find which labels contribute to the metric’s cardinality growth. When the current day is selected, the reported cardinality and labels shown will continue to update throughout the day *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Change Events > With Last9, track change events such as configuration changes and deployment events with ease along with other metrics. ## Why Change Events Matter? Software systems and their observability are not just about the telemetry data emitted from these systems. The software systems also get affected by external change events. These events can be from domains such as deployment, configuration, or external 3rd party systems. Last9 allows tracking such change events along with other metrics, seamlessly providing more context to the system observability. This document will show how to start tracking change events with Last9. Last9 offers an HTTP API that can be used to track any domain change event. Each event has two states, start and stop, and both can be tracked with Last9. Once Last9 receives the event, it converts it into a metric that can be used with other metrics for querying using PromQL in Grafana dashboards or alerting. The data flow for change events is as follows.  ## Last9 Change Events API Last9 offers a REST HTTP endpoint that can be used to send the change events. The API endpoint is as follows. ```shell curl -XPUT https://app.last9.io/api/v4/organizations/{org_slug}/change_events \ --header 'Content-Type: application/json' \ --header 'X-LAST9-API-TOKEN: Bearer ' \ --data-raw ' { "timestamp": "2024-01-15T17:57:22+05:30", "event_name": "new_deployment", "event_state": "start", "data_source_name": {levitate_cluster_data_source_name}, "attributes": { "env": "production", "k8s_cluster": "prod-us-east-1", "app": "backend-api" } }' ``` Tip Refer to the [Getting Started with API](/docs/getting-started-with-api/) guide to obtain the token required for the change events API. * `timestamp` is the iso8601 formatted timestamp of the event. It is optional. If not passed, current timestamp is used * `event_name` can be any event name depending on the context. It will be added as a label to the resulting time series * `event_state` can be `start` or `stop` * `attributes` will be used as labels while converting the change event to a metric * `data_source_name` is the name of the Last9 cluster where the events will be stored. It is an optional field and can be obtained as described in the [Change Events Storage](#change-events-storage) section. Last9 will convert the events into a metric named `last9_change_events`. ## Change Events Storage It is possible that you might be using multiple Last9 clusters. In such scenario, you can choose to store the change events in a Last9 cluster of your choice. The optional `data_source_name` attribute is used to specify the cluster where change event will be stored. If this attribute is not passed, then Last9 will store the change event in a default cluster designated for change events. The default cluster for change events is set as follows.  You can override this by specifying the `data_source_name` in the request payload. Obtain the cluster name from the Data Sources section as follows.   Note Store events and related metrics in same Last9 cluster for automatic correlation between them. ## Visualize Change Events in Grafana The change events can be visualized in Grafana just like any other metrics.  ## Native Integrations for Change Events * [Prodvana](/docs/integrations-prodvana/) * [LaunchDarkly](/docs/integrations-launchdarkly/) *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Clusters > Overview of Clusters ## Cluster Overview To start using Last9 you need at least one Cluster, from which you read and write metric data. In this document, we dive deep into all things related to a cluster. To get up and running fast, see our [Quick Start Guide](/docs/onboard/). Think of a Cluster as a logically separated, Prometheus API-compatible data source for all your metric data. You can create as many Clusters as you want, the number of clusters has no impact on your billing. It is typically recommended that you create Clusters for each of your environments. Example: Production Cluster, Staging Cluster, etc. ## Creating a New Cluster To create a Cluster: 1. Navigate to **Home** → **Levitate**  2. Click the *Launch Cluster* button to launch the setup wizard  3. Select the AWS region you would like to deploy the cluster in. This should ideally be the same region as your application 4. Give the Cluster a descriptive name 5. Optionally, add a description which will be displayed on the Cluster Overview screen  6. Press the **Create** to create your new Cluster As the Cluster gets created, you will be presented with an access token that is automatically created. This token is required to start writing & reading data to the Cluster. Tokens are only shown once, so please copy or download credentials (or you can always create another token from Cluster settings).  Your new Cluster is now ready to receive metrics. 7. To start writing data to this new Cluster, please follow the Write Data steps start writing data from Kubernetes, Prometheus, AWS/CloudStream or quickly try out by running a local demo environment  Using the **Test Config** button you can verify if your Last9 cluster has started receiving data. Click the **Next** button to start reading data/querying metrics from your Cluster. See our guides on how you can send data from [Prometheus](#), [OpenTelemetry](#), [VMAgent](#), or other various s[Integrations](#) supported. 8. To start reading metrics from your new Cluster you can use Managed Grafana which comes included with every plan. Alternatively, you can use the provided **Read URL** to read data using any Prometheus HTTP API Compatible tool like AlertManager, your own Grafana, KEDA, etc. See the guide on [how to connect your own Grafana](/docs/grafana-config/) with a Last9 cluster.  *** ## Managing a Cluster ### Cluster Usage and Performance Last9 provides the following tools to observe the Cluster’s performance: * [Cluster Health Dashboard](#cluster-health-dashboard) - Performance & usage metrics report * [Query Logs](#query-logs) - Identify slow-running queries #### Cluster Usage  Usage for each cluster is reported in *Samples Ingested*. A **sample** refers to a single data point in a time series. Usage for each Cluster can be viewed from the Cluster’s details page. For more granular and historical usage, see the Cluster Health dashboard’s Sample Ingested panel. #### Cluster Quotas There are no *per-cluster* limits in Last9. You are billed for usage across all Clusters combined. The ingestion rate, read query rate, and data retention quotas are applied for all the data across all clusters. #### Default Cluster Quotas Last9’s default cluster quotas are fairly generous. In certain cases, keeping in mind performance and cost impacts, we may be able to increase a quota after a discussion with your team. #### Write Quotas | Type | Base Quota | Reset Period | Note | | -------------------------------------------- | ---------- | ------------ | ------------------------ | | Per Time Series Cardinality | 1M | Per Hour | Can be raised on request | | Per Time Series Cardinality | 20M | Per Day | Can be raised on request | | Streaming Aggregation Cardinality | 3M | Per Hour | Can be raised on request | | Ingestion Concurrency | 20K | Per Second | Can be raised on request | | Number of Metrics Aggregated in one Pipeline | 1 Metric | Per Query | Cannot be changed | #### Read Quotas | Type | Base Quota | Note | | ------------------------------------------ | ---------- | ------------------------ | | Time Series Scanned Per Query — Blaze Tier | 5M | Cannot be changed | | Time Series Scanned Per Query — Hot Tier | 10M | Cannot be changed | | Samples Scanned Per Query | 100M | Cannot be changed | | Query Time Range — Blaze Tier | 2 Hours | Can be raised on request | | Query Time Range — Hot Tier | 35 Days | Can be raised on request | If you wish to change your quotas, please raise a request by emailing us on: ### Cluster Health Dashboard Every Last9 Cluster comes with its own Health dashboard. To view the Health dashboard, navigate to the Cluster details page and click on the **View Health** link in the performance card.  The following Cluster Performance Metrics are available in the health dashboard:  * **Write Success** - Total successful write requests * **Write Error** - Total failed write requests * **Samples Ingested** - Total number of samples ingested * **Write Availability** - Percentage of write requests successful * **Write Latency** - Write request latency * **Lag** - Pending samples waiting to be indexed (in bytes) * **Read Success** - Total successful write requests * **Read Errors** - Total failed read requests * **Cardinality Limited** - Metrics whose cardinality has been limited * **Read Latency** - Query Latency * **Cardinality Limiter (Early Warning)** - Metrics whose cardinality is about to be limited * **Bytes Dropped** - Samples permanently failed to be indexed (in bytes) *** ## Query Logs Query Logs helps identify slow-running queries so that you can debug and optimize your PromQL. Query Logs displays slow queries in the last 24 hours, which were successfully executed but have taken more than 1000ms (ie one second) to execute.  When a slow query is identified the following details are displayed: * **Timestamp** - Time when the query was executed * **Query** - PromQL along with the query’s time range and query resolution step width * **Latency** - approximate time taken for the query to execute * **Token** Name - the name of the token used to query * **Tier** - storage tier that was used for this query *** ## Cluster Settings ### Tokens Tokens provide a mechanism for access management for your clients. We generate a default token when the Cluster is created for the first time #### Creating a New Token 1. Navigate to the Cluster that you wish to create a token for: **Control Plane** → **Tokens**  2. Click **New Token**  3. Provide a descriptive **Token Name** the access **Scope** (Write Only, Read Only, Read & Write) and click **Create** 4. Copy the generated token since it will be visible only once. This token can now be used along with the Read or Write URL (depending on the Scope selected)  #### Delete a Token To delete/revoke a token: 1. Navigate to the Cluster that you wish to revoke a token from: \*\*Control Plane → \*\*Tokens\*\* 2. Click the **…** button and select Delete  Note: * This action cannot be undone, once deleted tokens cannot be recovered * Tokens can only be deleted by your organization’s admin ### Write & Read Data Refer to the list of available [Integrations](/docs/integrations/) that can be used to start writing and reading data to a Last9 Cluster. ### Access Policies Last9 has built-in data-tiering capabilities based on retention policies. Access policies let you define policies to control which token or client can query a specified data tier. See our in-depth guide on how you can leverage this powerful feature - Guide on Access Policies #### To define a new access policy: 1. Navigate to **Control Plane** → **Access Policies**  Every cluster comes with a default access policy pre-configured. 2. To define a new policy click the Create button  Provide the following details: * Policy Name: Give a descriptive name for this access policy * Token: Select a specific Token for which this access policy is applied or select *Any* * Query Client: We can identify traffic from known clients or select Any for the policy to apply from any client * Tier: Select the Tier from which the queries will be served for this policy And click **Create** 3. Your new access policy will be applied instantly  #### To delete an Access Policy : 1. Select the **…** button beside the access policy you wish to delete  2. Select **Delete** from the menu Do Note: * Access policies can only be deleted by the admin user(s) of your org * Deleting an access policy may limit or lock access for a client or token, please be mindful before deleting ### Macros Macros lets you define PromQL queries as reusable functions and use them as abstracted metric names across Grafana, Alert Manager, or the CLI We cover how to define and use Macros in detail in [guide on PromQL Macros](/docs/promql-macros/) #### Enabling Macros: 1. Navigate to **Control Plane** → **Macros**  2. Write/Paste your Macro function and Click Save  We perform validation once you click Save  Once validated, we will save your Macro function. Do note that it will take upto 5 minutes for new Macros to be available for querying  #### Deleting Macros: 1. Navigate to **Control Plane** → **Macros**  2. Click the delete icon and click confirm Note: * Deleted Macros will impact any queries and dashboards where the macro functions were used * Deleted Macros may be available for queries up to 5 minutes after they have been deleted ### Streaming Aggregation Streaming aggregation is a powerful metric cardinality that is built-in with Last9. Refer to our [Guide on Streaming Aggregation](/docs/streaming-aggregations/) for an in-depth tutorial # Configuring an Alert > A step-by-step guide to configuring an alert rule in an Alert Group  ## Pre-requisites To be useful, each Alert Group needs Alert Rules to enable monitoring the health of the Alert Group. If you’ve created an Alert Group by importing from a Managed Grafana dashboard, indicators will already exist based on the PromQLs used in the dashboard. Else, you’ll need to create a new indicator. Indicators are required to be selected while creating an alert rule. ## Rule Configuration ### Rule Name Short and simple is good for quick identification when a notification is triggered. Keep in mind that Alert Rule names are shown along with the Alert Group names in the notification. If you want to add more context, use the Rule Description field in the [Annotations](#annotations-optional) section. ### Select Indicator Alert Rules are run against an Indicator. If you’ve imported a Grafana dashboard, Indicators are auto-generated based on the dashboard panel PromQLs, else you’ll have to first add the relevant Indicator. Indicators inherit the ALert Group’s datasource, but can also have their own as an override. ### Edit Label Filter (optional) Indicator queries support PromQL variables. If the query contains a variable, you’re able to specify a specific label filter for the Alert Rule to be triggered only for that. ### Select Alerting Algorithm ```plaintext By default, only Static Threshold is enabled. If you would like to use our [Anomaly Detection](/docs/anomalous-pattern-detection-guide/) algorithms, please write to us at [support@last9.io](mailto:support@last9.io). ``` ### Set Threshold This section is only visible if you’ve selected Static Threshold. You can select an operator and set the value of the threshold. The alert rule will only trigger when it matches the threshold’s criteria. ### Configure Alert Sensitivity Depending on the algorithm selected, the options may vary here. In case of Static Threshold, you can specify the no. of bad minutes the rule needs to be triggered out of no. of total minutes before it appears as firing in the Alert Monitor or send a notification. In case of the Anomaly Detection algorithms, you can specify a value ranging from 0 to 10, and decimal values are accepted. Lower the value, the more sensitive the algorithm will be. You can click on the backtest button in the preview panel to open the indicator and algorithm calculations in Grafana and see at what values will the algorithm trigger. Play around with the query in Grafana to find a balance that you’re comfortable with. ### Severity Level Select if this rule, when firing, should be treated as a threat or a breach. This is helpful as additional metadata for integrations like PagerDuty and OpsGenie to determine severity levels and route accordingly. ### Notification Grouping When the alert rule is firing for multiple labelsets, it may lead to noise. For such case, you may group notifications to a single instance. Such notifications do call out the no. of labelsets and values the alert rule is firing for. ### Annotations (optional) Annotations are used to include additional meta data to alert notifications for an alert rule. For example, to help your team members better understand the context of an alert notification, you may want to include a brief description outlining the behavior or circumstances when the rule should’ve triggered. Or, include a runbook link for quickly reaching the next to-do steps for your team member. #### Dynamic Annotations Annotations can be supercharged by inserting dynamic values using template variable. Currently, the following variables are supported: * **Labels**, where it is the value of the respective label of timeeries under alert, with the syntax `{{ $labels. }}` or `{{ .Labels. }}` * **Value**, where it is the worst value of timeseries under alert, with the syntax `{{ $value }}` or `{{ .Value }}` Template variables can be used alongside plain text as well. For example, `Service name is {{ $labels.service }}`. Usage of multiple variables in a field is also supported. Spaces in the template variable syntax are optional. Template variables can be used in any of the annotation fields — the rule description, runbook, or even custom annotations. Considerations: * Apart from the labels in the metric’s timeseries, the labels of the Alert Group can also be referenced in template variables. In case the labels match, preference to the metric’s timeseries is given * In case a label value is not present, the template variable is shown as is * In case the template variable syntax is incorrect, the UI will display an error. Please note the supported variables above and their respective syntaxes * Notifications with Dynamic Annotations display these dynamic values. In case of [grouped notifications](#notification-grouping), *Labels* are shown as a count of all label values and *Values* are shown as a P99 of all the worst values ##### **Sample usage of Dynamic Annotations with Splunk** A custom annotation named `splunk_debug_url` is added to an alert rule whose value is configured as `https://search.splunk.com/?service={{$labels.service}}&stack={{$labels.stack}}`. When alerts are generated for one or more timeseries, the values of the variable in this custom annotation will be interpolated using the labels in the timeseries. For example, `service=billing` and `stack=my-org` will lead to link `https://search.splunk.com/?service=billing&stack=my-org` and so on. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Send data using AWS PrivateLink > This guide walks you through how to configure AWS PrivateLink for use with Last9 will send read and write requests over the private network. The process consists of configuring an internal endpoint in your VPC, which can talk to the Last9 endpoint without going via a public network. ## Setup ### Get Last9 PrivateLink Service name Currently, Last9 supports three regions - `eu-west-1`, `us-east-1` and `ap-south-1`. You can choose the service name from the following list. Note This applies only to customers hosted on app.last9.io. If you use an on-premise hosted Last9 offering, the Last9 Customer Success team will provide the right endpoint details. Please contact us at . | Region | Type | Service Name | DNS | | ---------------- | ---------- | -------------------------------------------------------------- | ------------------------------------- | | `eu-west-1` | Write | `com.amazonaws.vpce.eu-west-1.vpce-svc-0381436dec59e8895` | `https://app-tsdb-euw1.last9.io` | | `eu-west-1` | Read | `com.amazonaws.vpce.eu-west-1.vpce-svc-0a101a671bd82e759` | `https://read-app-tsdb-euw1.last9.io` | | `us-east-1` | Write | `com.amazonaws.vpce.us-east-1.vpce-svc-0eda8f4ecd25af01f` | `https://app-tsdb-use1.last9.io` | | `us-east-1` | Read | `com.amazonaws.vpce.us-east-1.vpce-svc-0fb0502d1f58dc1bb` | `https://read-app-tsdb-use1.last9.io` | | `ap-south-1` | Write | `com.amazonaws.vpce.ap-south-1.vpce-svc-0240de6d26b096123` | `https://app-tsdb.last9.io` | | `ap-south-1` | Read | `com.amazonaws.vpce.ap-south-1.vpce-svc-01c9bdbb02e34fe2c` | `https://read-app-tsdb.last9.io` | | `us-east-1` | Read/Write | `com.amazonaws.vpce.us-east-1.vpce-svc-06931cb6d013ad0fc` | `https://otlp.last9.io` | | `ap-south-1` | Read/Write | `com.amazonaws.vpce.ap-south-1.vpce-svc-026c2a0d782b7ef32` | `https://otlp-aps1.last9.io` | | `ap-southeast-1` | Read/Write | `com.amazonaws.vpce.ap-southeast-1.vpce-svc-0f8965c5096ad1f65` | `https://otlp-apse1.last9.io` | ### Create an endpoint in your VPC Navigate to the VPC section and select `Endpoints` in the left sidebar.  Click on Create Endpoint in the top right corner. Enter an appropriate `Name tag`, and Select `Other endpoint services `.  Enter the Service name provided by Last9 in the earlier step and Click `Verify service`. ### Additional settings * Select the VPC for the endpoint, and in additional settings, check the box for Enable DNS Name.  * Select the subnets where you are running the workloads that will read or write to Last9.  * Attach a security group to the endpoint; the endpoint must allow traffic on port 443 from your origin VPC or the specific IP address where the requests originate. * Note: For any on-premise Last9 setups, the endpoint must allow traffic for ports 80 and 443.  ## Verification After the Endpoint status becomes available, validate that the DNS records change from any machine inside a subnet in your VPC for which the Endpoint is enabled. For example, `dig app-tsdb.last9.io` should not return the public IP address; instead, it should return the private IP address of the Endpoint we just created.  That’s all. Happy sending logs, metrics, and traces to Last9 using PrivateLink. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Control Plane > Manage your data, its configurations, and its lifecycle. ## Introduction  Last9’s Control Plane offers a first-class citizen experience for developers to manage their data, its settings, and its lifecycle. This document provides an overview of the main features and functionalities available in the Control Plane user interface. ## Tools and Configurations [input](/docs/control-plane-ingestion/) ## [Ingestion](/docs/control-plane-ingestion/) [Configurations for how your data is ingested into Last9](/docs/control-plane-ingestion/) [folder\_zip](/docs/control-plane-storage/) ## [Storage](/docs/control-plane-storage/) [Defaults and controls for storing and using your telemetry data](/docs/control-plane-storage/) [query\_stats](/docs/control-plane-query/) ## [Query](/docs/control-plane-query/) [Configure query reusability, reads, and pattern match alerts.](/docs/control-plane-query/) [monitoring](/docs/control-plane-analytics/) ## [Analytics](/docs/control-plane-analytics/) [Understand and debug system usage and performance](/docs/control-plane-analytics/) *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Analytics > Control Plane tools to understand and debug system usage and performance. ## Cardinality Explorer  While Last9 offers [superior defaults](/docs/managing-high-cardinality/) on per-metric per-day cardinality, you may need to identify the metrics and its labels that are impacted. Cardinality Explorer helps you understand how the cardinality for metrics and its labels is trending. This enables you to diagnose cardinality-related challenges with your metrics. [Read more](/docs/cardinality-explorer/) on how to use the Cardinality Explorer interface. ## Slow Query Logs  Quickly identify which queries are taking the longest to debug and optimize them. These queries could be originating from either Last9’s alerting, managed Grafana explore/dasboards, or from your own read workfloads. You can change the latency values on the filter to see slower queries, but the minimum is queries taking longer than 1 second. By default, logs are displayed for the last 1 hour, but the window can be customized to a maximum of last 24 hours. ## Health Dashboard  While Last9 provides an SLA of 99.9% writes and 99.5% reads, you can also view the health of Last9 by clicking on Health Dashboard. You are redirected to a system-generated Grafana dashboard with panels for availability, successes/errors, latencies, lags, bytes dropped, and more. ## Usage  View the ingestion trend and usage breakdown for your telemetry data by total and types (log, span, and metric events). By default, a summary of the last 30 days is displayed. You can select an area on the chart to zoom in or you can click on the icon in each date row of the breakdown table to view an hourly breakdown. You can also click on “Download CSV” to get a hourly breakdown for the last 30 days. ### What is an Event? Usage numbers are shown as Total Events. Each log line, trace span, and metric sample that is ingested by Last9 is considered an event. The number of events is calculated at the ingestion layer, before the data is used by any of the ingestion pipelines like [Streaming Aggregation](/docs/control-plane-ingestion/#streaming-aggregations), [Sensitive Data](/docs/control-plane-ingestion/#sensitive-data), [Forward](/docs/control-plane-ingestion/#forward), and [Drop](/docs/control-plane-ingestion/#drop). *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Cold Storage > Learn how to configure AWS S3 cold storage for log archival and cost optimization with Last9 Automatically archive logs older than 14 days to S3 for cost-effective storage and on-demand rehydration. Note The default log retention period in Last9 is 14 days. To modify this retention period for your specific needs, please reach out to our support team at .  ## Setup 1. **Create IAM Role** with permissions to the S3 bucket: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:ListBucket", "s3express:CreateSession" ], "Resource": [ "arn:aws:s3:::", "arn:aws:s3:::/*" ] } ] } ``` 2. **Add Trust Relationship**: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com", "AWS": "arn:aws:iam::" }, "Action": "sts:AssumeRole" } ] } ``` 3. Make sure that the role session expirry is set to **minimum 4 hours**. Note Contact Last9 support for LAST9\_STORAGE\_USER ARN. 4. **Enable Cold Storage** Configure your bucket name and role ARN in [Cold Storage](https://app.last9.io/control-plane/cold-storage). 5. Once the cold storage is enabled, you can rehydrate the logs on demand. Read the [Rehydration](/docs/control-plane-rehydration/) guide for more details. ## Service-Level Backup Configuration You can now configure which services you want to back up to your cold storage. When configuring, you have three options: 1. **Default:** With service-level backup not enabled, data is backed up with index-level granularity only, meaning you cannot rehydrate individual services. 2. **All Services:** All services are backed up and you can rehydrate individual services. 3. **Only Selected Services:** Specify which services you want to back up, giving you more granular control. Note While creating a Rehydrated Index, if service-level backup is available for your selected time range, you can choose which specific services to rehydrate. ### Benefits of Service-Level Backup * **Targeted cost optimization:** Save money where it makes sense without compromising on critical services * **Service-appropriate retention:** Match data lifecycle to each service’s actual needs * **Strategic resource allocation:** Invest observability resources based on service priority * **Simplified compliance:** Apply different retention rules only where legally necessary *** ## Troubleshooting Need help? Join our [Discord](https://discord.com/invite/Q3p2EEucx9) or email . # Access Cold Storage Logs via AWS Athena > Learn how to query and analyze your cold storage logs in S3 using AWS Athena's SQL interface Last9 automatically backs up your logs to a configured S3 bucket via [Cold Storage](/docs/control-plane-cold-storage). This doc will show you how to access and query these archived logs using AWS Athena, allowing you to perform powerful SQL-based analysis on your historical data. ## Create a database on Athena ```sql CREATE DATABASE last9; ``` ## Create a table in the database ```sql CREATE EXTERNAL TABLE last9.logs ( `timestamp` bigint, `traceid` string, `spanid` string, `traceflags` int, `severitytext` string, `severitynumber` int, `servicename` string, `body` string, `resourceschemaurl` string, `resourceattributes` array>, `scopeschemaurl` string, `scopename` string, `scopeversion` string, `scopeattributes` array, `logattributes` array> ) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 's3://customer_s3_bucket/snappy-files/' TBLPROPERTIES ('parquet.compression'='SNAPPY'); ``` ## Export AWS Profile Before running the script, ensure your AWS profile is properly configured with appropriate permissions to access both your source and destination S3 buckets, as well as Athena. ## Move logs to Athena from backup S3 bucket Save the following Python script as `insert_data_into_athena.py`. ```python import argparse import boto3 import os import pandas as pd import tempfile import lz4.frame from botocore.exceptions import ClientError class ParquetProcessor: def __init__(self): """Initialize the processor using AWS credentials from environment""" self.s3_client = boto3.client('s3') self.athena_client = boto3.client('athena') self.temp_dir = tempfile.mkdtemp() def download_from_s3(self, bucket_name, prefix): """Download all .parquet.lz4 files from the specified S3 path""" downloaded_files = [] try: print(f"Searching in bucket: {bucket_name}") print(f"Using prefix: {prefix}") paginator = self.s3_client.get_paginator('list_objects_v2') for page in paginator.paginate(Bucket=bucket_name, Prefix=prefix): if 'Contents' in page: print("\nObjects found:") for obj in page['Contents']: print(f"Key: {obj['Key']}") if obj['Key'].endswith('.parquet.lz4'): print(f"Found matching file: {obj['Key']}") local_file = os.path.join(self.temp_dir, os.path.basename(obj['Key'])) self.s3_client.download_file(bucket_name, obj['Key'], local_file) downloaded_files.append(local_file) else: print("No 'Contents' in this page") if not downloaded_files: print("No .parquet.lz4 files were found") return downloaded_files except ClientError as e: print(f"Error downloading files: {e}") return [] def decompress_lz4(self, file_path): """Decompress .parquet.lz4 file to .parquet""" try: output_file = file_path.replace('.lz4', '') print(f"Decompressing {file_path} to {output_file}") with open(file_path, 'rb') as compressed: compressed_data = compressed.read() decompressed_data = lz4.frame.decompress(compressed_data) with open(output_file, 'wb') as decompressed: decompressed.write(decompressed_data) os.remove(file_path) print(f"Successfully decompressed to {output_file}") return output_file except Exception as e: print(f"Error decompressing file {file_path}: {e}") return None def convert_to_snappy(self, file_path): """Convert decompressed parquet to Snappy compression""" try: df = pd.read_parquet(file_path) df.to_parquet(file_path, compression='snappy') return file_path except Exception as e: print(f"Error converting file {file_path}: {e}") return None def upload_to_s3(self, bucket, prefix, file_path): """Upload a file to S3""" try: file_name = os.path.basename(file_path) s3_key = os.path.join(prefix.rstrip('/'), file_name) print(f"Uploading {file_path} to s3://{bucket}/{s3_key}") self.s3_client.upload_file(file_path, bucket, s3_key) return True except Exception as e: print(f"Error uploading file: {e}") return False def cleanup_local_files(self, snappy_files): """Clean up temporary local files""" for file in snappy_files: try: os.remove(file) except Exception as e: print(f"Error removing file {file}: {e}") os.rmdir(self.temp_dir) def process_files(self, source_bucket, source_prefix, snappy_destination, athena_results_location=None): """Main process to handle the complete workflow""" # Download LZ4 files lz4_files = self.download_from_s3(source_bucket, source_prefix) if not lz4_files: print("No .parquet.lz4 files found") return # Decompress LZ4 files decompressed_files = [] for file in lz4_files: decompressed_file = self.decompress_lz4(file) if decompressed_file: decompressed_files.append(decompressed_file) if not decompressed_files: print("No files were successfully decompressed") return # Convert to Snappy snappy_files = [] for file in decompressed_files: snappy_file = self.convert_to_snappy(file) if snappy_file: snappy_files.append(snappy_file) if not snappy_files: print("No files were successfully converted to Snappy") return # Upload to snappy destination dest_bucket = snappy_destination.split('//')[1].split('/')[0] dest_prefix = '/'.join(snappy_destination.split('//')[1].split('/')[1:]) for file in snappy_files: if not self.upload_to_s3(dest_bucket, dest_prefix, file): print(f"Failed to upload {file}") continue # Cleanup local files self.cleanup_local_files(snappy_files) print("Processing completed successfully") if __name__ == "__main__": parser = argparse.ArgumentParser(description='Process .parquet.lz4 files and upload to S3') # S3 and Athena configuration parser.add_argument('--source-bucket', required=True, help='Source S3 bucket name where parquet.lz4 (last9 saves archives)') parser.add_argument('--source-prefix', required=True, help='Source S3 prefix path where parquet.lz4 files are stored') parser.add_argument('--snappy-destination', required=True, help='S3 path for converted snappy files') parser.add_argument('--athena-results', required=True, help='S3 path for Athena query results') args = parser.parse_args() processor = ParquetProcessor() processor.process_files( source_bucket=args.source_bucket, source_prefix=args.source_prefix, snappy_destination=args.snappy_destination, athena_results_location=args.athena_results ) ``` The script `insert_data_into_athena.py` is used to process `.parquet.lz4` files from the backup bucket and upload them to a separate S3 location for processing in Athena. ### Help Command Run the following command to see all available options and parameters: ```bash python insert_data_into_athena.py --help ``` ### Usage ```plaintext usage: insert_data_into_athena.py [-h] --source-bucket SOURCE_BUCKET --source-prefix SOURCE_PREFIX --snappy-destination SNAPPY_DESTINATION --athena-results ATHENA_RESULTS Process .parquet.lz4 files and upload to S3 options: -h, --help show this help message and exit --source-bucket SOURCE_BUCKET Source S3 bucket name where parquet.lz4 (where Last9 saves backup files) --source-prefix SOURCE_PREFIX Source S3 prefix path where parquet.lz4 files are stored --snappy-destination SNAPPY_DESTINATION S3 path for converted snappy files --athena-results ATHENA_RESULTS S3 path for Athena query results ``` ### Example Command Here’s a sample command that processes files from your backup bucket to prepare them for Athena queries: ```bash python insert_data_into_athena.py \ --source-bucket last9_backup_bucket \ --source-prefix "path/to/file/" \ --snappy-destination "s3://customer_s3_bucket/snappy-files" \ --athena-results "s3://customer_s3_bucket/athena-results/" ``` In this example: * `last9_backup_bucket` is your source bucket containing the archived logs * `path/to/file/` is the directory path where your .parquet.lz4 files are located * `s3://customer_s3_bucket/snappy-files` is where the converted files will be stored * `s3://customer_s3_bucket/athena-results/` is where Athena will store query results ## Check result on Athena After the data has been uploaded, you can query it using Athena with the following SQL: ```sql SELECT * FROM last9.logs; ``` This will retrieve all logs from the `last9.logs` table, allowing you to verify that your data has been successfully uploaded and is accessible through Athena. ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Drop > Drop unwanted telemetry at ingestion layer using Control Plane  Order of Last9's pipeline processing. [Drop](https://app.last9.io/control-plane/drop) lets you discard unwanted telemetry that you don’t want to store and query, at runtime — no code changes, no redeploys, no policy updates. For example, if you need debug logs during an incident, just remove the drop rule. It’s a faster, simpler alternative to code-level governance. ## Create new drop rule Head to the Control Plane and create a new Drop Rule.  You can configure rules with `==` and `!=` matching filters to drop incoming data. Do note, this data is not ingested and cannot be recovered as well. To verify the filters before saving the rule, you can click on “View in Dashboard”.  ## Supported Telemetry Types Supported telemetry types are logs, metrics and traces. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Forward > Forward telemetry data to object storage backends such as AWS S3  Order of Last9's pipeline processing. ## Forward telemetry data Last9 control plane allows forwarding telemetry data to object storage such as AWS S3 without storing it. Forwarding capability is useful for forwarding certain logs that are not useful for frequent querying but only need to be stored for compliance. Note The forwarded telemetry is stored in `.gz` format in the AWS S3 bucket. ## Configure AWS S3 backend Before creating the forward rule, configure an AWS S3 bucket where Last9 can forward telemetry data using [IAM AssumeRole](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html).  ## Create a new forward rule Head to the Control Plane and create a new Forward Rule.  To drop incoming data, you can configure rules with `==` and `!=` matching filters. Do note that this data is not ingested and cannot be recovered. To verify the filters before saving the rule, you can click on “View in Dashboard.”  ## Supported Telemetry data types Supported telemetry types are logs and traces. The supported storage backend is AWS S3. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Ingestion > Control Plane tools to configure for how your data is ingested into Last9. Ingestion is the second pillar of our telemetry data platform, Last9. Once you‘ve [instrumented](/docs/integrations/) your system, controls over how your telemetry data flows in to Last9 do a fair bit of heavy lifting. ## Ingestion Tokens  Ingestion Tokens authenticate your applications when sending telemetry data to Last9. These tokens control what data can be sent and from which origins, ensuring secure data collection. A system-generated ingestion token is created when you signup — this token is used in the setup wizard to help you configure sending data to Last9. System-generated tokens cannot be deleted. ## Access Policies  Setup how various clients access your data — depending on the client type and token used, you can control from which tier (blaze, hot, cold) your data is queried. We recommend alerting workloads to always use the Blaze Tier and reporting workfloads to use the Cold Tier. [Read more](/docs/access-policies/) on how to configure these policies. ## Streaming Aggregations  Streaming Aggregations allow you to transform data in real-time at the ingestion layer before it is stored in Last9. They enable you to generate scoped metrics on runtime without any instrumentation changes and improve performance of your read queries by controlling cardinality of the new metrics. [Read more](/docs/streaming-aggregations/) on how to configure these aggregations. *** Note The following configurations are applied in the same sequence as presented. Your telemetry data is first scanned for sensitive data, then matching data is forwarded, and then any matching data is dropped. These configurations allow you to not make any instrumentation level changes and give you a more run-time control. ## Sensitive Data  Redact sensitive data from your telemetry data at time of ingestion. Currently supported: * Telemetry Type: Logs * Actions: Redact (default), No Action Last9 provides built-in scan rules for PII like emails, phone numbers, and credit card numbers. While the default action is to redact, you can also choose to take no action. This is particularly helpful when you just want to attach additional labels to the telemetry. Configured rules are applied in a sequential order. Once saved, you can drag-and-drop to reorder the rules. ## Forward  While data after applicable retention periods can be moved to your configured S3 bucket for [Cold Storage](/docs/control-plane-storage/#cold-storage), you can also configure rules with `==` and `!=` matching filters to forward incoming data directly to your cold storage without being ingested and stored. To verify the filters before saving the rule, you can click on “View in Dashboard”. Do note, this data will not be available for querying when forwarded, but once [rehydrated](/docs/control-plane-storage/#rehydration), it can be queried. Supported telemetry types: Logs and Traces. ## Drop  You can configure rules with `==` and `!=` matching filters to drop incoming data. Do note, this data is not ingested and cannot be recovered as well. To verify the filters before saving the rule, you can click on “View in Dashboard”. Supported telemetry types: Logs, Metrics and Traces. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Query > Control Plane tools for query-level configurations. ## Macros  They work in a way that is similar to how SQL developers use stored procedures. Macros take full advantage of the time-tested best practices of functions, abstractions, and reusability to replace cumbersome and error-prone methods. Simplify your PromQLs that are reused often to avoid repition of code, and improve abstractions and readability. [Read more](/docs/promql-macros/) on how to configure these macros. ## Scrape Interval Set this to the typical scrape and evaluation interval configured in your agent’s config file. If you set this to a greater value than your agent’s config file interval, the embedded Grafana in Explore will evaluate the data according to this interval and you will see less data points. Notes: * Defaults to 1m. * This does not change your agent’s scrape interval. ## Read Data  If you’re looking to use your stored telemetry data outside of Last9’s [Alerting](https://app.last9.io/alert-studio) or [Managed Grafana](https://app.last9.io/explore), you can refer to the Read Data settings to configure your choice of visualization tool. For additional settings on how to configure your own Grafana to use Last9 as a datasource, [read this](/docs/grafana-config/). ## Scheduled Search  Create periodic searches on telemetry data and set alerts when patterns are found or missing. [Read more](/docs/scheduled-search/) on how configure these alerts. Supported telemetry types: Logs, and Traces coming soon. ## Query Tokens  Query Tokens provide read-only access to your telemetry data for external visualization tools like Grafana, alerting systems, and custom applications. A system-generated query token is created when you signup — this token is used to set up dashboard templates, including the Health Dashboard. System-generated tokens cannot be deleted. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Rehydration > Learn how to rehydrate logs from AWS S3 cold storage to Last9 Rehydrate logs from AWS S3 cold storage to Last9 on demand. Note 1. Make sure that the [cold storage](/docs/control-plane-cold-storage/) is enabled before rehydrating the logs. 2. While creating a Rehydrated Index, if service-level backup is available for your selected time range, you can choose which specific services to rehydrate. ## Steps to Rehydrate Logs 1. Go to [Rehydration](https://app.last9.io/control-plane/rehydration) and click on “New Rehydrated Index” 2. Select the time range for rehydration.  3. (Optional) Add email for completion notification.  4. Click **Rehydrate**. The job will appear as **pending** in the Rehydration tab.  5. Once complete, the status changes to **completed**.  6. Click **Query** to access the logs.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Remapping > Transform and standardize your logs data by extracting and mapping fields for better searchability and analysis.  Order of Last9’s pipeline processing. [Remapping](https://app.last9.io/control-plane/remapping) allows you to standardize your logs data by extracting fields from log lines and mapping them to consistent formats. This powerful feature helps you normalize data across different services and sources, making your logs more searchable and easier to analyze. Tip Remapping configurations are real-time and do not need any instrumentation or code changes, nor any re-deployment. Remapping consists of two primary functions: 1. **Extract**: Pull specific fields or patterns from your log lines 2. **Map**: Transform extracted fields into standardized formats This capability is valuable for scenarios like: * Normalizing different service names across your infrastructure * Standardizing severity levels from various sources (ERROR, err, Fatal, 500) * Creating consistent environment labels (prod, production, prd) * Extracting structured data from JSON or pattern-based logs * Maintaining consistent field naming conventions ## Working with Remapping ### Extract  1. Navigate to [Control Plane > Remapping](https://app.last9.io/control-plane/remapping) 2. Select the “Extract” tab 3. View existing extraction rules in the table showing: * Name: Descriptive name of the extraction rule * Method: JSON or Pattern Match extraction method * Scope: Which lines the extraction applies to * Fields/Pattern: Which fields or patterns to extract * Action: How the extracted data is handled (Upsert/Insert) * Active Since: When the rule was activated 4. Click ”+ NEW RULE” to create a new extraction rule #### Creating a New Extraction Rule 1. Select “Extraction Method”: * **JSON:** Extract fields from structured JSON logs * **Pattern Match:** Use regex patterns to extract fields from unstructured logs 2. Choose “Extraction Scope”: * **All Lines:** Apply extraction to every log line * **Lines that match:** Apply only to lines matching specific criteria 3. Field(s) to Extract: 1. For JSON method: * Select the field(s) to extract * Example fields: requestId, thread\_id, logger\_name, etc. 2. For Pattern Match method: * Enter the regex pattern in “Pattern to Extract” field * Example: `timeseries:\s*(?P\d+)` 4. Set “Action” to “Upsert” (update if exists, insert if not) or “Insert” 5. Choose “Extract Into” option: * **Log Attributes:** Adds fields to the log’s searchable attributes * **Resource Attributes:** Adds fields to the resource’s metadata 6. Optionally add a “Prefix” to extracted field names * Example: “ec2\_” would transform “id” to “ec2\_id” 7. Enter a descriptive “Rule Name” 8. Click “SAVE” to activate the rule ### Map  1. Navigate to [Control Plane > Remapping](https://app.last9.io/control-plane/remapping) 2. Select the “Map” tab 3. View “Remap Fields” section with existing mappings 4. Map common fields to standardized formats: * **Service:** Map various service names to consistent values * Example: `attributes["service_name"]` * **Severity:** Map different log levels to standard severity * Example: `attributes["level"]` and `attributes["levelname"]` * **Deployment Environment:** Map environment indicators * Select from available attributes 5. Preview the mapping results in the “Preview (Last 2 mins)” section below * SERVICE: How service names appear after mapping * SEVERITY: Standardized severity levels * DEPLOYMENT ENV: Normalized environment names * LOG ATTRIBUTES: Other log details * RESOURCE ATTR: Resource-related information 6. After configuring mappings, click “SAVE” ## Example Use Cases 1. **Standardizing Service Names**: Map various service identifiers to consistent names * Raw values: “auth-svc”, “auth\_service”, “authentication” * Mapped to: “authentication-service” 2. **Normalizing Severity Levels**: Create consistent severity levels across sources * Raw values: “ERROR”, “err”, “Fatal”, “500” * Mapped to: “ERROR” 3. **Extracting Thread Information**: Pull thread details from logs for better filtering * Extract fields: thread\_id, thread\_name, thread\_priority * Makes thread-based troubleshooting more efficient 4. **Environment Consistency**: Standardize environment naming * Raw values: “dev”, “development”, “preprod”, “staging” * Mapped to consistent environment names ## Tips for Effective Remapping * **Start Simple:** Begin with the most common fields you search by * **Use Consistent Naming:** Follow a naming convention for all mapped fields * **Check Preview Results:** Use the preview section to verify your mappings work as expected * **Consider Extraction Order:** Remember that attributes will be looked up in the sequence they are entered * **Use JSON When Possible:** JSON extraction is more reliable for structured logs * **Test Pattern Matches:** Validate regex patterns before implementing them *** ## Troubleshooting If your remapping rules aren’t working as expected: 1. Check the extraction pattern syntax for errors 2. Verify field names match exactly what appears in your logs 3. Ensure your extraction scope is appropriate 4. Look at the preview to confirm data is flowing as expected 5. Try simplifying complex regex patterns Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Sensitive Data > Redact sensitive data from telemetry at ingestion layer using Control Plane  Order of Last9's pipeline processing. [Sensitive Data](https://app.last9.io/control-plane/sensitive-data) automatically detects and redacts personally identifiable information (PII) and other sensitive data from your telemetry at ingestion time — no code changes, no redeploys, no policy updates. For example, if customer phone numbers start appearing in your logs, just create a redaction rule to automatically replace them with asterisks. It’s a faster, simpler alternative to code-level data sanitization. ## Create new sensitive data rule Head to the Control Plane and create a new Sensitive Data Rule.  You can configure rules to scan for different types of Personal Identifiable Information (PII) including email addresses, phone numbers, and credit card numbers. When sensitive data is detected, you can choose to redact it (replace with asterisks) or take no action. Additional labels can be attached to matching samples for filtering and alerts.  ## Configuration Options ### Telemetry Data Currently supported telemetry type is **logs only**. All samples for the selected telemetry data will be scanned using the configured rules. ### Scan Rules Choose which types of sensitive data to detect: * **Email** - Detects email addresses in your log data * **Phone Number** - Identifies phone numbers across various formats * **Credit Card Number** - Finds credit card numbers and payment card data ### Actions Available actions for detected sensitive data: * **Redact** - Replace matching sensitive data with asterisks (**\***) * **No Action** - Detect and label but don’t modify the data ### Additional Labels Add custom labels (key:value pairs) to samples containing sensitive data. These labels can be used for filtering, alerting, and downstream processing. Common examples: * `sensitive_data: true` * `redacted: true` * `pii_type: phone` ## Supported Telemetry Types Currently supported telemetry type is **logs**. Support for metrics and traces will be added in future releases. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Storage > Control Plane tools for defaults and configuring how your telemetry data is stored and re-used. ## Sampling, Tiering, and Retention 1. **Sampling:** Last9 applies no sampling on your data to ensure an accurate representation of your system’s health. 2. **Data Tiering:** Last9 offers automated data tiering by default for your metrics data, including ones generated by the traces-to-metric and logs-to-metric pipelines. * *Blaze Tier:* Last 2 hours * *Hot Tier:* Last 28 days * *Cold Tier:* As per your Cold Storage 3. **Retention:** Metrics data is retained for 28 days by default with cold storage for backup. Logs and Traces data is retained for 14 days with cold storage for backup and on-demand rehydration. ## Cold Storage  For your logs and traces, Last9 currently offers an integration with your AWS S3 bucket to store data older than 15 days. This data will be available for on-demand rehydration to run queries and report on. Read the [Cold Storage](/docs/control-plane-cold-storage/) guide for more details. ## Rehydration The historical logs can be rehydrated for later consumption as needed. You can rehydrate based on a time range filter. Additionally for live debugging use cases, Last9 performs automatic rehydration of logs upto 10M log lines when the requested time range is beyond the retention period. Read the [Rehydration](/docs/control-plane-rehydration/) guide for more details. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Create a GCP service account with read-only access > Step by step guide to create a GCP service account with read-only access for monitoring ## Objective A service account is required to access GCP environment resources for monitoring. This doc provides step by step information on creating a GCP service account with monitoring read-only access. Once you have created the account, share the configuration with Last9 team so that the monitoring data can be sent to [Last9](https://last9.io/). ## Prerequisites * Go to the Google Cloud Console ([console.cloud.google.com](https://console.cloud.google.com/)) account * Select the project in which you want to create the service account * Click on the “IAM & Admin” tab in the left navigation menu * Click on the “Service Accounts” tab  Note For a GCP Project, ensure that you have access to create credentials and grant permissions. ## Creating Service Account * Click on the “Create Service Account” button * Enter following details 1. Service Account Name: `last9-monitor` 2. Service Account ID: `last9-monitor` 3. Service Account Description: *Allows Last9 API access to read resource metadata and monitoring data* * Click on the “Create and Continue” button  ## Monitoring Viewer Role Grant Permissions to this Service Account with Role as `Monitoring Viewer`.  Grant other users internal to your organization access to this Service Account(Optional)  ## Generate Credentials * Click on the newly created Service Account to view more details  * Create a new Service Account Key   * Share the downloaded key with your Last9 team # Creating Log Analytics Dashboards > Guide to creating Log Analytics Dashboards from aggegragated queries in Logs Explorer ## Introduction Creating custom log analytics dashboards in Last9 allows you to visualize and monitor log data through aggregated metrics. This guide explains how to create and promote log queries into dashboard visualizations. ## Starting with Logs Explorer ### Using Editor Mode 1. Navigate to the [Logs Explorer](https://app.last9.io/logs) in Last9 2. Switch to Editor Mode — this enables writing advanced LogQL-compatible queries 3. Write a normal query to explore your data ```sql {service="adservice"} ``` 4. Convert it into an aggregation query by adding an aggregation function ```sql sum by (severity) (count_over_time({service="adservice"} [1m])) ``` 5. Promote the query to a dashboard by clicking the **Add to Dashboard** button  6. Create a new dashboard or add it to an existing dashboard by adding a unique panel name  7. You will be redirected to the new dashboard with your query added as a panel  8. You can edit the panel by clicking the **⋮** button and then the **edit** button  9. Add multiple panels to the dashboard by following the same steps as above ### Supported Aggregation Functions Last9 supports several aggregation functions for creating meaningful visualizations: * Time-based aggregations: * **`count_over_time`**: Counts the number of logs over time * **`sum_over_time`**: Sums the values of a numeric field over time * **`avg_over_time`**: Averages the values of a numeric field over time * **`max_over_time`**: Finds the maximum value of a numeric field over time * **`min_over_time`**: Finds the minimum value of a numeric field over time * **`rate`**: Calculates the rate of change of a numeric field over time * Statistical aggregations: * **`sum`**: Sums the values of a numeric field * **`avg`**: Averages the values of a numeric field * **`count`**: Counts the number of logs * **`max`**: Finds the maximum value of a numeric field * **`min`**: Finds the minimum value of a numeric field * **`stddev`**: Calculates the standard deviation of a numeric field * **`median`**: Finds the median value of a numeric field * **`stdvar`**: Calculates the standard variance of a numeric field ## Query Construction Guidelines ### Time Windows Specify time windows using the following formats: * Minutes: **`[1m]`** * Hours: **`[1h]`** * Days: **`[1d]`** ### Query Examples Basic severity-based aggregation: ```sql sum by (severity) (count_over_time({service!="user-service"} [1m])) ``` Complex bucket-based aggregation: ```sql sum by (bucket) (count_over_time({ service="unknown", ingestor="s3", bucket=~"elb-logs", log.file.path=~".*api.*" } [3h])) ``` ## Best Practices ### Time Range Selection * Match window size to query requirements * For instant queries, set time range equal to window size * Consider data retention and query performance when selecting time ranges ### Query Performance * Leverage accelerated queries by including Service or Severity filters * Use specific filters to reduce data scanning * Test queries with smaller time ranges before expanding to larger windows ### Dashboard Organization * Group related visualizations together * Use clear, descriptive titles * Include context in dashboard descriptions * Set appropriate refresh intervals based on data update frequency *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Delegate Subdomain between two AWS Accounts using Route 53 > Step-by-step guide on how to delegate Subdomain between two AWS Accounts using Route 53 Note This is specifically useful for Last9 BYOC deployments ## Assumptions 1. AWS Account A is the Primary Account 2. AWS Account B is the Sub Account 3. `example.com` is an arbitrary domain used purely for easy understanding 4. You have enough permissions granted by your AWS Admin to add/modify Route53 ## Premise To set up `subdomain.example.com` as a hosted zone in **AWS Account B** and extend it for internal usage (e.g., `internal.subdomain.example.com`), you need to delegate authority for the subdomain from **AWS Account A** to **AWS Account B**. This process involves creating a new hosted zone in **Account B** for the subdomain and then updating the parent hosted zone in **Account A** to delegate DNS resolution to the nameservers for the subdomain in **Account B**.  ## Step-by-Step Procedure ### Step 1: Create a Hosted Zone for the Subdomain in Account B 1. **Sign in to the AWS Management Console for Account B** 2. **Create a Hosted Zone**: * Navigate to the Route 53 console * Click on **“Hosted zones”** in the left navigation pane * Click the **“Create hosted zone”** button * Enter `subdomain.example.com` as the domain name * Choose the type as “**Public hosted zone”** (or **Private hosted zone for Amazon VPC** if it’s for internal usage) * Click **Create hosted zone** 3. **Note the Nameservers**: * After the hosted zone is created, note the nameservers (NS records) provided by Route 53 for the new hosted zone. You will need these nameservers to delegate the subdomain from Account A ### Step 2: Delegate the Subdomain from Account A to Account B 1. **Sign in to the AWS Management Console for Account A** 2. **Navigate to the Hosted Zone for `example.com`**: * Go to the Route 53 console * Click on **Hosted zones** in the left navigation pane * Click on the hosted zone for `example.com` 3. **Create NS Record for the Subdomain**: * Click **Create record** * Choose **Simple routing** and click **Next** * For **Record name**, enter `subdomain` (to delegate `subdomain.example.com`) * Choose **Record type** as **NS - Name Server** * In the **Value** field, enter the nameservers for `subdomain.example.com` provided by Account B * Click **Create records** ### Step 3: Create a Hosted Zone for Internal Usage in Account B 1. **Sign in to the AWS Management Console for Account B** 2. **Create a Hosted Zone for `internal.subdomain.example.com`**: * Navigate to the Route 53 console * Click on **Hosted zones** in the left navigation pane * Click the **Create hosted zone** button * Enter `internal.subdomain.example.com` as the domain name * Choose the type as **Private hosted zone for Amazon VPC** * Select the appropriate VPCs * Click **Create hosted zone** ## Verification **Public Subdomain Delegation**: * You can verify that `subdomain.example.com` is correctly delegated by using the `dig` or `nslookup` commands: ```plaintext dig ns subdomain.example.com ``` **Internal Subdomain Resolution**: * For `internal.subdomain.example.com`, ensure that your VPC’s DNS settings are configured correctly and that Route 53 resolver endpoints are set up if necessary ## Summary 1. **Create a hosted zone for `subdomain.example.com` in Account B** and note the nameservers 2. **Delegate the subdomain** from Account A to Account B by creating an NS record in the `example.com` hosted zone in Account A pointing to the nameservers for `subdomain.example.com` in Account B 3. **(Optional) Create a hosted zone for `internal.subdomain.example.com`** in Account B for internal DNS resolution *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # FAQs > Frequently Asked Questions ## What are the challenges of running your own Prometheus? The key challenge faced by an enterprise: * Modern Time Series systems don’t have to grow along a single axis of Cardinality, Coverage, or Retention alone. Instead, the rate of ingestion and exploration warrants an expansion on all three axes. There is a constant need to scale TimeSeries Database vertically * Data are abundant, but it’s not being used. **Growing costs of storing and querying this data** lead to time and effort in auditing what metrics are used, how much data retention is necessary, and trimming from your database * **Enterprises must dedicate full-time engineering resources to manage their time series database**. Automation to scale with rapid changes in data shape and recovery from downtime must be implemented and maintained. To support needs across the business, teams create multiple database instances to handle concurrency and implement query sharding to improve performance. Orgs will also implement a solution like Thanos or Cortex to enable long-term storage of metric data. These implementations will cost significant engineering time to create and maintain and make things operationally confusing for developers sending and querying metrics ## Is Last9 fully managed? You can set up a cluster and get going in under two minutes. How do you ask? Read our [How to onboard to Last9](/docs/onboard/) guide for more details. ## Should an enterprise run Prometheus internally, or can Last9 support all internal enterprise requirements related to Prometheus? An enterprise can run Prometheus internally or offload all metrics storage to [Last9](https://last9.io/). We provide flexibility in our offerings per the requirements most suited for our customers. Prometheus is open-source software that collects metrics from targets by “**scraping**” metrics HTTP endpoints and **stores** the metrics as time-series data. ## Is Last9 SOC2 compliant? Last9 cares deeply about its customer’s data and is SOC2 Type II certified. ## Can Last9 be deployed on infrastructure owned by the enterprise? Yes, Last9 can also run on any cloud provider of your choice. ## What features are available on SaaS vs. on-prem instances? All features, including retention, are available on SaaS and on-prem instances. ## How do enterprises ingest data from GCP, Azure & AWS? All cloud providers have a Prometheus-compatible exporter. This can be run to scrape metrics from all infrastructure resources and ingested into Last9. ## Does the data go over an open internet? Yes, by default, it does, and securely. Alternatively, you could do VPC Peering and remote write data to Last9 over HTTP. ## How to retrieve data for internal consumption? This can be done with simple querying as one will with Grafana and other visualization tools to query TSDB data. Read our [How to configure Grafana for Last9](/docs/grafana-config/) guide for more details. # Getting started with API > Step-by-step walkthrough on how to obtain the API tokens for performing various operations with Last9 The API provides a programmatic method to access and operate Last9. This exposes a subset of features and actions that can be performed on Last9 as REST APIs. For example, you can send [change events](/docs/change-events/) to Last9 using these APIs or you can [generate alert rules](/docs/alerting-via-iac/). Tip These APIs differ from the instrumentation and configurations required for data ingestion into Last9 clusters. They also differ from querying APIs provided by Last9 to read data from a Last9 cluster. ## Base URL The base API URL can be obtained from the [API Access](https://app.last9.io/apiaccess) page. It is in the following format: ```text https://{domain}/api/{version}/organizations/{org}/{endpoint} ``` The `{org}` parameter is your unique organization slug.  ## Tokens The Authentication is performed using Bearer access tokens. Tokens are generated for a logged-in user. Click “Generate Tokens” to create the tokens. A total of 3 pairs of tokens for access and refresh are generated with authentication-specific claims for read, write, and delete operations.  ### Token Expiry The Tokens expire in 24 hours; the user or application that uses these tokens should account for expirations and incorporate a refresh mechanism using the refresh token issued along with the access token. The following error will be raised when the access token expires. ```json { "error": "Authorization token is expired" } ``` In such scenarios, you can generate a new access token using the refresh token as follows. ```text POST /v4/oauth/access_token ``` Request Body: ```json { "refresh_token": "eyJhbGciOiXXXXXXXXXXXXX.eyJleHXXXXXXXXX.XXXXXXXXXOwuvUNA" } ``` The response of this endpoint will contain a pair of access tokens and refresh tokens if the refresh token in the request body is valid. Response ```json { "access_token": "eyJhbGciOiXXXXXXXXXXXXXX.eyJleHXXXXXXXXX.XXXXXXXXXOwuvUNA", "expires_at": 1587412870, "issued_at": 1587240070, "refresh_token": "eyJhbGciOiXXXXXXXXXXXXX.eyJleHXXXXXXXXX.XXXXXXXXXOwuvUNA", "type": "Bearer", "scopes": ["read", "write", "delete"] } ``` ## Usage The tokens are specifically separated based on the scopes they are authorized to perform based on the impact they might have on the system’s overall behavior. * **Read Tokens**: Have a minimum impact on the performance of the Last9 application. These are to be specifically used for reading the current state of the data * **Write Tokens**: Use this token to create or modify data in any supported entity. This could change the behavior of your usage of Last9 * **Delete Tokens**: Use this token judiciously. This could break the processes and cause an irrevocable state through missing data ## Authentication & Authorization All public API endpoints require a Token to be supplied as an authorization header for all requests. The token is used to identify the user/application and authenticate the requests to API. The header name must be **X-LAST9-API-TOKEN**. ## Making your first API request Please follow the steps below to create our first API request for a change event. ### Step 1: Generate Tokens Refer to the Tokens section above and generate the tokens from the API Access page. For your first request, copy the Write Access token. ### Step 2: Base URL The base URL of your instance can be obtained as specified in the Base URL section above. ### Step 3: Making the API request The endpoint for creating change events is ```text PUT /change_events ``` ```json { "timestamp": "2024-01-15T17:57:22+05:30", "event_name": "new_deployment", "event_state": "start", "attributes": { "env": "production", "k8s_cluster": "prod-us-east-1", "app": "backend-api" } } ``` The cURL request looks as follows: ```shell curl --location --request PUT 'https://app.last9.io/api/v4/organizations/github-prathamesh-sonpatki/change_events' \ --header 'X-LAST9-API-TOKEN: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "timestamp": "2024-01-15T17:57:22+05:30", "event_name": "new_deployment", "event_state": "start", "attributes": { "env": "production", "k8s_cluster": "prod-us-east-1", "app": "backend-api" } }' ``` ### Step 4: Verify the response The API will return the following response in case of success with HTTP status code 200. ```json { "message": "success" } ``` *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Getting Started with Events > This document explains how to send events to Last9 and different ways to consume them such as via streaming aggregation. Last9 time series data warehouse is a powerful tool for storing and analyzing time series data. Last9 also supports ingesting real-time events and converting them into metrics so that they can be consumed in myriad ways engineers are already familiar with. This document outlines the steps necessary to send events to Last9 and the different ways of consuming them. ## Events Generally, there are two types of events. 1. They happen over time, and their performance, like frequency, presence, or absence, is interesting * Example: Average hourly take-offs from the San Francisco Airport in the last week. 2. The event and its data are of interest * Example: When was the last time Arsenal won in EPL? The first example is asking questions based on specific aggregations performed on raw events, the individual events may not be necessary, but their aggregations and insights captured using them are relevant for business. The second example is about the event and gives insights based on the event data. Last9 supports extracting both kinds of information from the events. ## Structure of Events Last9 supports accepting events in the following JSON format. Every event has a unique name defined by the `event` key and a list of `properties`. Any extra keys apart from `event` and `properties` are not retained by Last9. ```json { "event": "heartbeat", "properties": { "server": "ip_address", "environment": "staging" }, "extra_fields": "will be dropped" } ``` ## Sending Events to Last9 ### Prerequisites Grab the [Prometheus Remote Write](https://last9.io/blog/what-is-prometheus-remote-write/) URL, cluster ID, and the write token of the Last9 cluster, which you want to use as an Event store. Follow the instructions [here](/docs/onboard/) if you haven’t created the cluster and its write token. ### Sending data Grab the Prometheus Remote Write URL for your Last9 Cluster, and make following changes to the URL. If your Prometheus URL is `https://username:token@app-tsdb.last9.io/v1/metrics/{uuid}/sender/acme/write` The Event URL would be `https://username:token@app-events-tsdb.last9.io/v1/events/{uuid}/sender/acme/publish` Note Note down the table below to get the hostname for the Events gateway depending on the region in which the Last9 cluster exists.  | Cluster Region | Host | | :----------------- | :---------------------------- | | Virginia us-east-1 | app-events-tsdb-use1.last9.io | | India ap-south-1 | app-events-tsdb.last9.io | Events must be sent with the `Content-Type: application/json` header. ```bash curl --location --request XPOST 'https://username:token@app-events-tsdb.last9.io/v1/events/{uuid}/sender/acme/publish' \ --header 'Content-Type: application/json' \ --data-raw '[{ "event": "sign_up", "properties": { "currentAppVersion": "4.0.1", "deviceType": "iPhone 14", "dataConnectionType": "wifi", "osType": "iOS", "platformType": "mobile", "mobileNetworkType": "wifi", "country": "US", "state": "CA" } }]' ``` The API endpoint accepts an array of events in the payload so one or more events can be sent in the same packet. The API is greedy and allows partial ingestion. If one or more events in the packet have a problem, they are returned to the response body; everything else is ingested into the system. ## Consuming events as metrics Events are converted to Metrics at 1 DPM (Data Point per minute) and combined to emit `gauges` per combinations of `event name + properties` for the previous minute. We use **[Tumbling Windows](https://learn.microsoft.com/en-us/stream-analytics-query/tumbling-window-azure-stream-analytics)** to represent the Event stream’s consistent and disjoint time intervals. All published events are partitioned into buckets of 1 minute each and then grouped by event Name and properties. For example, the elements with timestamp values `\[0:00:00-0:01:00)` are in the first window. Elements with timestamp values `\[0:01:00-0:02:00)` are in the second window. And so on.  The following events will produce the following metrics: ```bash event_name_count{properties...1} 5 event_name_count{properties...2} 2 ... event_name_count{properties...n} 5 event_name_count{properties...1} 5 event_name_count{properties...2} 1 ... event_name_count{properties...n} 3 event_name_count{properties...1} 2 event_name_count{properties...2} 4 ... event_name_count{properties...n} 3 ``` ### Define Streaming Aggregations You need to define streaming aggregations to query the metrics converted from events. Last9 allows defining a Streaming Aggregation as a PromQL to emit an aggregated metric that alerts or dashboards can then consume. ```yaml - promql: "sum(device_health_total{version='1.0.1'})[5m] by (os)" as: total_devices_by_os_5m with_name: total - promql: "sum(device_health_total{os='ios'})[1m] by (version)" as: concurrency_by_ios_version with_name: concurrency ``` Please refer to the PromQL-powered [Streaming Aggregations](/docs/streaming-aggregations/) to understand the workflow of where and how to define the Streaming Aggregation Pipelines. This feature enables the folding of all metrics that would otherwise explode in cardinality and allows for the emission of meaningful aggregations and views. It is also available for Last9 Metrics, not just limited to Events. ## Events to Gauge Metrics Consider an event named `memoryUsageSpikeAlert` with the following properties: * `increaseInBytes` indicating an increase in memory usage by 1,610,612,736 bytes * `host` represents the host’s IP address associated with the event * `osType` specifying the operating system type as “linux” ```json { "event": "memoryUsageSpikeAlert", "properties": { "increaseInBytes": "1610612736", "host": "10.1.6.14", "osType": "linux" } } ``` Define the streaming aggregation configuration for `max` of `memoryUsageSpikeAlert` as follows: ```yaml - promql: "max by (host) (memoryUsageSpikeAlert_maximum)" as: "max_memory_usage_spike" with_value: "increaseInBytes" with_name: "maximum" ``` Let’s break the example: * `max` is the aggregation function * `maximum` is the `with_name` value appended to the intermediate metric name to maintain uniqueness. It can be any string. * `with_value` is the event’s property name on which the gauge aggregation has to be applied. In this case, it is `increasedBytes`. * `max_memory_usage_spike` will be the final output metric you are exposed to query against. > You can also use the `min` and `sum` aggregations. ## Querying Events Once Events have been converted to Metrics, they can be queried like metrics. This could be a Grafana Dashboard or any other Prometheus Query API client.  You may also set alerts on these events, converted to metrics, using Prometheus-compatible Alertmanager. ## Conclusion Here’s a link to the sample repository that brings this all together. It contains some example schemas and aggregation pipelines. ## FAQs **Q: Why is time not accepted as a first-class property?** A: Accepting a User-provided timestamp is extremely risky. A timestamp may not be formatted correctly, or instead of a UnixMilli, one may send Unix alone. Or send January 1st, 1970 as ***testing* data.** Such precision is unfair to expect from developers who want easier integration that is not prone to fragility. Besides, since the system is optimized for time, a malformed timestamp will result in undesired results of dropped packets or the system having to backfill data in frozen shards. Hence, we accept timestamps as when the event is received. This also means that Last9 works well with real-time events! **Q: What happens to the timestamp if there are delays in arrival?** A: Last9 Gateways are designed to be extremely fast and lightweight. They write data as early as they receive it and are highly available across multiple availability zones. They need more processing to ensure messages are not lost or delayed upon arrival. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Configure Grafana with Last9 as Datasource > How to configure Grafana with Last9 cluster as datasource and visualize the metrics stored in Last9 ## How to configure Grafana for Last9? Create a new data source with the appropriate URL.  Each Last9 Cluster comes with a Read URL which needs to be used when creating a Data Source in Grafana.  Here’s an example of creating a Data Source using the Read URL. You can grab the Read URL either: * While creating your cluster → Read Data → Bring Your Own Visualization * Or, by going to the Last9 Cluster → Settings → Read Data → Bring Your Own Visualization  Note Last9 Read URL mandates authenticated access. Create a Read Token for your cluster and use it as `password`. Use the cluster id as `user`. Add the `user` and `password` in the `Basic Auth Details` section while creating data source in Grafana. Make sure that the status of the data source appears as all ✅.  After this, try exploring data or create a new dashboard in Grafana based on metrics in Last9. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Grafana Loki in Last9 > Use Last9's embedded Grafana Loki to view logs. ## Using Grafana Loki Last9 provides a Grafana Loki interface using LogQL to explore your logs data.  * Access the Loki UI by visiting [Grafana Explore](https://app.last9.io/explore/query) and selecting Loki as the datasource. * You can perform [LogQL queries](https://grafana.com/docs/loki/latest/query/) to explore logs in this interface. This is useful for structured exploration of logs data for people who are familiar with Grafana and Loki. Note You can also use [Editor Mode](/docs/logs-explorer/#editor-mode) in Last9’s Logs Explorer to perform LogQL queries as well. ## LogQL Compatibility Following functions in LogQL are supported: * **`RATE`** * **`COUNT_OVER_TIME`** * **`SUM_OVER_TIME`** * **`AVG_OVER_TIME`** * **`MAX_OVER_TIME`** * **`MIN_OVER_TIME`** * **`SUM`** * **`AVG`** * **`COUNT`** * **`MAX`** * **`MIN`** * **`STDDEV`** * **`MEDIAN`** * **`STDVAR`** Following parsers in LogQL are supported: * **`json`** * **`regexp`** Read more about the documentation for each function [here](https://grafana.com/docs/loki/latest/query/). ## Creating Dashboards ### Accessing Grafana 1. Navigate to the Grafana section in Last9 2. Create a new dashboard by clicking **Create Dashboard** 3. Add a new panel to begin visualizing your data ### Selecting Loki Data Source The Loki data source comes pre-configured in Last9’s embedded Grafana, so you can start querying immediately. ### Query Construction Methods #### Using Builder Mode Builder mode provides a visual interface for constructing Loki queries without writing LogQL. Here’s how to use it: 1. Label Selection * Click **Add label** to start building your query * Select labels (e.g., service, severity) from the dropdown * Choose operators (=, !=, =~~, !~~) * Select or type values for the labels 2. Operations * Add operations using the **Operations** button * Common operations include: * Line contains * Line does not contain * Line contains regex * Line does not contain regex * JSON 3. Aggregations * Click **Add range function** * Select functions like: * Rate * Count over time * Sum over time * Avg over time * Set time windows (\[1m], \[5m], \[1h]) 4. Examples Using Builder Mode: Basic Query: * Label: **`service = "auth-service"`** * Operation: **`Line contains "error"`** * Range: **`count_over_time [5m]`** Advanced Query: * Label: **`service =~ "api.*"`** * Label: **`severity = "error"`** * Operation: **`JSON`** * Operation: **`Line contains "timeout"`** * Range: **`sum by (status_code)`** 5. Builder to Code Mode * Switch between modes to see the LogQL equivalent * Learn LogQL syntax through the Builder interface * Fine-tune queries in Code mode #### Writing LogQL Queries For advanced users or complex queries, you can write LogQL directly: Basic Query Structure: ```sql {service="your-service"} ``` Common Aggregation Patterns: ```sql sum by (severity) (count_over_time({service="your-service"}[5m])) ``` ### Key Query Components * Label matchers: **`{label="value"}`** * Line filters: **`|= "error"`** * Aggregation functions: **`sum`**, **`avg`**, **`max`** * Time windows: **`[1m]`**, **`[1h]`**, **`[1d]`** ### Understanding Window Behavior Remember that Last9’s window behavior differs from standard Loki: * Last9 uses tumbling windows (window size = step size) * Both window and step size are defined by the **`[]`** parameter * For instant queries, match time range to window size ### Creating Visualizations #### Panel Types 1. Time Series * Best for tracking metrics over time * Suitable for rate and count queries 2. Bar Charts * Good for comparing values across categories * Works well with **`sum by`** aggregations 3. Tables * Useful for detailed log analysis * Can show multiple columns of log data #### Panel Configuration 1. Set appropriate panel title and description 2. Configure axes and legends 3. Set up thresholds and alerts if needed 4. Choose color scheme for better visibility ### Advanced Query Techniques #### Using Multiple Queries ```sql sum(rate({service="auth-service"} |= "error" [5m])) by (severity) sum(rate({service="auth-service"} |= "warning" [5m])) by (severity) ``` #### Pattern Matching ```sql {service=~"auth.*"} |= "error" != "timeout" ``` #### Metric Extraction ```sql sum by (status_code) (count_over_time({service="api"} | json | status_code != "" [5m])) ``` ### Dashboard Organization #### Best Practices * Group related panels logically * Use consistent time ranges across related panels * Add descriptive titles and documentation * Consider user permissions and sharing settings #### Layout Tips * Arrange panels in order of importance * Use rows to group related visualizations * Consider different screen sizes and resolutions ### Performance Optimization #### Query Efficiency 1. Use label filters before line filters 2. Start with Service and Severity filters for better performance 3. Avoid processing unnecessary data #### Time Range Considerations * Start with smaller time ranges during development * Consider data retention policies * Use appropriate aggregation intervals *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Grafana Tempo in Last9 > Use Last9's embedded Grafana Tempo to view traces. ## Using Grafana Tempo Last9 provides a Grafana Tempo interface to explore your traces data.  * Access the Tempo UI by visiting [Explore](https://app.last9.io/explore/query) and selecting Tempo as the datasource. * You can perform [TraceQL queries](https://grafana.com/docs/tempo/latest/traceql/) to explore traces in this interface. This is useful for structured exploration of traces data for people who are familiar with Grafana and Tempo. \##last9-traces-tempo.png Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Calculate usage patterns and data volume in New Relic > Sample queries to understand data volume of spans, metrics and transactions in New Relic This document lists queries which you can run in your New Relic and share with Last9 as process of migration from New Relic to Last9. ## Calculating Data in New Relic This following NRQL queries will calculate calculate total no. of transactions, spans and mertics over the last day. 1. Total no. of spans over a week ```sql SELECT count(*) FROM Span SINCE 1 week ago ``` 2. Total transactions ```sql SELECT count(*) FROM Transaction SINCE 1 week ago ``` ```sql SELECT count(*) FROM Transaction FACET dateOf(timestamp) SINCE 1 week ago TIMESERIES 1 day ``` 3. Total Metrics ```sql SELECT count(*) FROM Metric SINCE 1 week ago ``` 4.Total Event ```sql SELECT count(*) FROM Event SINCE 1 week ago ``` 5. Total Logs ```sql SELECT count(*) FROM Log SINCE 1 week ago ``` Additionally, you can share how much data is getting ingested in New Relic from the data management tab. # Calculate usage patterns and data volume in Prometheus > Sample PromQL queries to understand ingestion rate, read query rate and total time series ## Calculating Ingestion Rate This query will calculate the per-minute ingestion rate by averaging the per-second ingestion rate over the past minute, as measured by the `prometheus_tsdb_head_samples_appended_total` metric. The result will be a single value representing the average number of samples ingested per minute over the past minute. ```sql rate(prometheus_tsdb_head_samples_appended_total[1m]) * 60 ``` ## Calculating Read Query Rate This query will calculate the per-minute query rate by averaging the per-second query rate over the past minute, as measured by the prometheus\_http\_requests\_total metric for GET requests to the /api/v1/query endpoint. This endpoint is used for executing queries against the Prometheus database, so this metric represents the number of read queries executed by the server. ```sql sum by (handler) (rate(prometheus_http_requests_total{handler="/api/v1/query"}[1m]) * 60) ``` ## Calculating Total Time Series This query will count the number of distinct time series in the database, regardless of the metric or label values. The regular expression ”.+” matches all series names, so this query effectively counts all series. ```sql count({__name__=~".+"}) ``` Tip Note that counting the total number of time series can be resource-intensive for large databases, and may take some time to complete. Additionally, this query may return inaccurate results if the database is actively ingesting or deleting time series during the query. # Create a AWS STS Role > This tutorial walks through setting up a AWS STS (Secure Token Service) role for discovering resources via cloudwatch ## Creating trusted role without external id 1. Visit [AWS Console/Roles](https://console.aws.amazon.com/iam/home#/roles) 2. Click [Create Role](https://console.aws.amazon.com/iam/home#/roles$new?step=type)  3. Select **[Another AWS Account](https://console.aws.amazon.com/iam/home#/roles$new?step=type\&roleType=crossAccount)** tab * **Account ID**: `652845092827` * **Next Permissions**  4. Attach policies Tip Last9 requires Cloudwatch Read Only Access to fetch metrics for components that are being used. Last9 also requests Security Audit Access as it enables us to identify components by name and tags. The AWS-managed SecurityAudit policy grants read-only access to logs, events and configuration detail for current and future AWS services. This policy allows Last9 to perform list and describe api calls to AWS to fetch component details such as name, ARNs and tags etc. And as Last9 provides auto detection of infrastructure components, this policy helps to minimize the access request burden on customers as they introduce new services to their infrastructure and as Last9 introduces new services to our platform. a. **SecurityAudit** Policy  b. **CloudWatchReadOnlyAccess** Policy  c. Proceed to Next Steps 5. Add tags if needed  6. Review 1. **Role name:** `${business_name}_last9_role` 2. **Role description**: Security Audit Access to Last9 3. **Verify Last9 AWS Account Number** 4. **Verify Granted Policy** 5. **Create Role**  7. After the role is created, Go to Role → Trust Relationships → Edit Trust Relationship  8. Update the JSON to the following and click “**Update Trust Policy**” ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::652845092827:root" }, "Action": "sts:AssumeRole", "Condition": {} } ] } ``` 9. Edit the role and update “**Maximum session duration**” to 3 hours if your security policy permits it. Else leave it as 1 hour.  10. Share the created role ARN with your Last9 point of contact *** ## Creating trusted role with external id 1. Visit [AWS Console/Roles](https://console.aws.amazon.com/iam/home#/roles) 2. Click [Create Role](https://console.aws.amazon.com/iam/home#/roles$new?step=type)  3. Select “**[Another AWS Account](https://console.aws.amazon.com/iam/home#/roles$new?step=type\&roleType=crossAccount)**” tab with external ID as a random string. It has to be something other than “somerandomstring” and share it with Last9 * **Account ID**: `652845092827`  4. Attach policies Tip Last9 requires Cloudwatch Read Only Access to fetch metrics for components that are being used. Last9 also requests Security Audit Access as it enables us to identify components by name and tags. The AWS-managed SecurityAudit policy grants read-only access to logs, events and configuration detail for current and future AWS services. This policy allows Last9 to perform list and describe api calls to AWS to fetch component details such as name, ARNs and tags etc. And as Last9 provides auto detection of infrastructure components, this policy helps to minimize the access request burden on customers as they introduce new services to their infrastructure and as Last9 introduces new services to our platform. a. **SecurityAudit** Policy  b. **CloudWatchReadOnlyAccess** Policy  c. Proceed to Next Steps 5. Add tags if needed  6. Review 1. **Role name:** `${business_name}_last9_role` 2. **Role description**: Security Audit Access to Last9 3. **Verify Last9 AWS Account Number** 4. **Verify Granted Policy** 5. **Create Role**  7. After the role is created, Go to Role → Trust Relationships → Edit Trust Relationship  8. Update the JSON to the following and click “**Update Trust Policy**”. Ensure that the value for `sts:ExternalId` matches the value set earlier for External-ID ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::652845092827:root" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "somerandomstring" } } } ] } ``` 9. Edit the role and update “**Maximum session duration**” to 3 hours if your security policy permits it. Else leave it as 1 hour  10. Share the created role ARN and external ID string with your Last9 point of contact # Enable EC2 Service Discovery with vmagent > This tutorial walks through setting up service disocvery for EC2 instances with vmagent. ## Service Discovery In the context of monitoring, Service Discovery refers to automatically detecting devices, services, or systems in a network that need to be monitored. Service discovery is significant in cloud environments that use auto-scaling and EC2 instances. These environments often have instances that change rapidly, making manual tracking infeasible from a monitoring point of view. This document lists steps to enable service discovery of EC2 instances so new instances can be monitored as they are created and decommissioned instances can be removed from monitoring, tackling false alerts. This document assumes that the EC2 instance service discovery will be set up for vmagent to send metrics to Last9 via Remote Write. Note Follow our guide for [setting up vmagent on an EC2 machine](/docs/how-to-setup-vmagent-on-ubuntu/). Given that vmagent is successfully running on an EC2 Instance, we need to make provisions for vmagent to discover other EC2 instances, that is, scrape targets based on [ec2\_sd\_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ec2_sd_config). ## Create `ec2-trustee` IAM role with assume role policy Go to AWS Console → IAM → Roles → Create Role * Select Trusted Entity  * Do NOT add any permissions and click next  * Name, Review and Create   ## Attach `ec2-trustee` IAM role to vmagent EC2 Host EC2 Instances > Select vmagent Instance > Actions > Instance Settings * Modify IAM Role  * Select `ec2-trustee` IAM role and Update  ## Create `vmagent-sd-role` IAM role Go to AWS Console → IAM → Roles → Create Role * Select Trusted Entity > Custom Trust Policy with below trust policy  ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::AWS_ACCOUNT_ID:role/ec2-trustee" }, "Action": "sts:AssumeRole" } ] } ``` * Add Permissions 1. Create Custom Policy with the below policy ````plaintext ```json { "Statement": [ { "Action": "ec2:Describe*", "Effect": "Allow", "Resource": "*" } ], "Version": "2012-10-17" } ``` ```` 2. Select Policy and click next   3. Name, Review and Create * Add `vmagent-sd-role` as the name of the role, review permissions and trusted entities and create role  ## Use the `vmagent-sd-role` ARN in vmagent configuration Update the `scrape_configs` stanza in your `vmagent.yaml` with the `ec2_sd_configs` stanza as follows and restart vmagent. ```yaml # vmagent.yaml # Check https://prometheus.io/docs/prometheus/latest/configuration/configuration for more details scrape_configs: - job_name: "node-exporter-sd" ec2_sd_configs: - region: ap-south-1 role_arn: "__role_arn_with_ec2_read_access__" filters: - name: tag:namespace values: - node-exporter port: 9100 ``` This will discover new EC2 instances automatically using the Service Discovery mechanism and their metrics will sent to Last9 from vmagent. # Scrape selective metrics in Prometheus > Recipe to only scrape selective metrics in Prometheus to reduce cardinality Prometheus provides the ability to filter specific metrics post-scraping, via the `metrics_relabel_config` stanza. Tip This is useful in reducing the number of metrics before they are consumed and sent to a remote write long term storage. We will use the `node_exporter` Prometheus exporter as an example, and send only metrics matching the regex `node_cpu.*`. ## Prometheus scrape config without filtering ```shell global: scrape_interval: 1m scrape_configs: - job_name: 'node-exporter-01' static_configs: - targets: [ 'localhost:9100' ] ``` This scrapes and stores all node exporter metrics. ## Prometheus scrape config with filtering ```bash global: scrape_interval: 1m scrape_configs: - job_name: 'node-exporter-01' static_configs: - targets: [ 'localhost:9100' ] metric_relabel_configs: - source_labels: [__name__] action: keep regex: '(node_cpu)' ``` This will scrape all metrics, but drop anything that does not match the entries in the `regex` section. # Scrape selective kube state metrics > This document describes how to scrape selective kube state metrics ## Obtain the list of kube state metrics from your Last9 cluster ```json curl -XGET 'https://:@read-app-tsdb.last9.io/hot/v1/metrics//sender//api/v1/label/__name__/values' | jq | grep -e "kube_" "kube_apiserver_pod_logs_pods_logs_backend_tls_failure_total", "kube_apiserver_pod_logs_pods_logs_insecure_backend_total", "kube_certificatesigningrequest_annotations", "kube_certificatesigningrequest_cert_length", "kube_certificatesigningrequest_condition", "kube_certificatesigningrequest_created", "kube_certificatesigningrequest_labels", "kube_configmap_annotations", "kube_configmap_created", "kube_configmap_info", "kube_configmap_labels", "kube_configmap_metadata_resource_version", "kube_daemonset_annotations", "kube_daemonset_created", "kube_daemonset_labels", "kube_daemonset_metadata_generation", "kube_daemonset_status_current_number_scheduled", "kube_daemonset_status_desired_number_scheduled", "kube_daemonset_status_number_available", "kube_daemonset_status_number_misscheduled", "kube_daemonset_status_number_ready", "kube_daemonset_status_number_unavailable", "kube_daemonset_status_observed_generation", "kube_daemonset_status_updated_number_scheduled", "kube_deployment_annotations", "kube_deployment_created", "kube_deployment_labels", "kube_deployment_metadata_generation", "kube_deployment_spec_paused", "kube_deployment_spec_replicas", "kube_deployment_spec_strategy_rollingupdate_max_surge", "kube_deployment_spec_strategy_rollingupdate_max_unavailable", "kube_deployment_status_condition", "kube_deployment_status_observed_generation", "kube_deployment_status_replicas", "kube_deployment_status_replicas_available", "kube_deployment_status_replicas_ready", "kube_deployment_status_replicas_unavailable", "kube_deployment_status_replicas_updated", "kube_endpoint_address", "kube_endpoint_address_available", "kube_endpoint_address_not_ready", "kube_endpoint_annotations", "kube_endpoint_created", "kube_endpoint_info", "kube_endpoint_labels", "kube_endpoint_ports", "kube_ingress_annotations", "kube_ingress_created", "kube_ingress_info", "kube_ingress_labels", "kube_ingress_metadata_resource_version", "kube_ingress_path", "kube_job_annotations", "kube_job_complete", "kube_job_created", "kube_job_info", "kube_job_labels", "kube_job_owner", "kube_job_spec_completions", "kube_job_spec_parallelism", "kube_job_status_active", "kube_job_status_completion_time", "kube_job_status_failed", "kube_job_status_start_time", "kube_job_status_succeeded", "kube_lease_owner", "kube_lease_renew_time", "kube_mutatingwebhookconfiguration_created", "kube_mutatingwebhookconfiguration_info", "kube_mutatingwebhookconfiguration_metadata_resource_version", "kube_namespace_annotations", "kube_namespace_created", "kube_namespace_labels", "kube_namespace_status_phase", "kube_node_annotations", "kube_node_created", "kube_node_deletion_timestamp", "kube_node_info", "kube_node_labels", "kube_node_spec_taint", "kube_node_spec_unschedulable", "kube_node_status_allocatable", "kube_node_status_capacity", "kube_node_status_condition", "kube_persistentvolume_annotations", "kube_persistentvolume_capacity_bytes", "kube_persistentvolume_claim_ref", "kube_persistentvolume_created", "kube_persistentvolume_info", "kube_persistentvolume_labels", "kube_persistentvolume_status_phase", "kube_persistentvolumeclaim_access_mode", "kube_persistentvolumeclaim_annotations", "kube_persistentvolumeclaim_created", "kube_persistentvolumeclaim_info", "kube_persistentvolumeclaim_labels", "kube_persistentvolumeclaim_resource_requests_storage_bytes", "kube_persistentvolumeclaim_status_phase", "kube_pod_annotations", "kube_pod_completion_time", "kube_pod_container_info", "kube_pod_container_resource_limits", "kube_pod_container_resource_requests", "kube_pod_container_state_started", "kube_pod_container_status_last_terminated_exitcode", "kube_pod_container_status_last_terminated_reason", "kube_pod_container_status_ready", "kube_pod_container_status_restarts_total", "kube_pod_container_status_running", "kube_pod_container_status_terminated", "kube_pod_container_status_terminated_reason", "kube_pod_container_status_waiting", "kube_pod_container_status_waiting_reason", "kube_pod_created", "kube_pod_deletion_timestamp", "kube_pod_info", "kube_pod_init_container_info", "kube_pod_init_container_status_ready", "kube_pod_init_container_status_restarts_total", "kube_pod_init_container_status_running", "kube_pod_init_container_status_terminated", "kube_pod_init_container_status_terminated_reason", "kube_pod_init_container_status_waiting", "kube_pod_init_container_status_waiting_reason", "kube_pod_ips", "kube_pod_labels", "kube_pod_owner", "kube_pod_restart_policy", "kube_pod_spec_volumes_persistentvolumeclaims_info", "kube_pod_spec_volumes_persistentvolumeclaims_readonly", "kube_pod_start_time", "kube_pod_status_container_ready_time", "kube_pod_status_phase", "kube_pod_status_qos_class", "kube_pod_status_ready", "kube_pod_status_ready_time", "kube_pod_status_reason", "kube_pod_status_scheduled", "kube_pod_status_scheduled_time", "kube_pod_status_unschedulable", "kube_pod_tolerations", "kube_poddisruptionbudget_annotations", "kube_poddisruptionbudget_created", "kube_poddisruptionbudget_labels", "kube_poddisruptionbudget_status_current_healthy", "kube_poddisruptionbudget_status_desired_healthy", "kube_poddisruptionbudget_status_expected_pods", "kube_poddisruptionbudget_status_observed_generation", "kube_poddisruptionbudget_status_pod_disruptions_allowed", "kube_replicaset_annotations", "kube_replicaset_created", "kube_replicaset_labels", "kube_replicaset_metadata_generation", "kube_replicaset_owner", "kube_replicaset_spec_replicas", "kube_replicaset_status_fully_labeled_replicas", "kube_replicaset_status_observed_generation", "kube_replicaset_status_ready_replicas", "kube_replicaset_status_replicas", "kube_secret_annotations", "kube_secret_created", "kube_secret_info", "kube_secret_labels", "kube_secret_metadata_resource_version", "kube_secret_type", "kube_service_annotations", "kube_service_created", "kube_service_info", "kube_service_labels", "kube_service_spec_type", "kube_service_status_load_balancer_ingress", "kube_storageclass_annotations", "kube_storageclass_created", "kube_storageclass_info", "kube_storageclass_labels", "kube_validatingwebhookconfiguration_created", "kube_validatingwebhookconfiguration_info", "kube_validatingwebhookconfiguration_metadata_resource_version", ``` ## Let’s decide to omit `kube_certificatesigningrequest_*` metrics 1. Prepare a list of metrics you want to omit. This needs to be a comma separated array of strings. ```json [ "kube_certificatesigningrequest_annotations", "kube_certificatesigningrequest_cert_length", "kube_certificatesigningrequest_condition", "kube_certificatesigningrequest_created", "kube_certificatesigningrequest_labels" ] ``` 2. Find the `metricDenylist` configuration in your Kube State Metrics Helm chart and append this list to that config. ```json metricDenylist: ["kube_certificatesigningrequest_annotations", "kube_certificatesigningrequest_cert_length", "kube_certificatesigningrequest_condition", "kube_certificatesigningrequest_created", "kube_certificatesigningrequest_labels"] ``` 3. Now deploy your Kube State Metrics Helm chart as usual 4. Run the below command after the deployment to verify that it is in effect. You should not find any metrics with `kube_certificatesigningrequest` prefix being emitted anymore ```json curl -XGET 'https://:@read-app-tsdb.last9.io/hot/v1/metrics//sender//api/v1/label/__name__/values' | jq | grep -e "kube_certificatesigningrequest_" ``` # Install VictoriaMetrics VMAgent on Ubuntu > This tutorial walks through setting up VMAgent on Ubuntu as a standalone process and monitor itself. [VMAgent](https://docs.victoriametrics.com/vmagent.html#vmagent) is a tiny agent which helps you collect metrics from various sources, relabel and filter the collected metrics and store them in Prometheus compatible remote storage such as Last9 using the [Prometheus remote write](https://last9.io/blog/what-is-prometheus-remote-write/) protocol. In this tutorial, we’ll cover how to install VMAgent on your Ubuntu server, manage the VMAgent process, and set up scrape configs to collect metrics from VMAgent itself. ## Prerequisites Before you begin this guide, you should have a regular, non-root user with sudo privileges configured on your server and also basic dependencies such as `wget`. Create a Last9 cluster by following the [Quick Start Guide](/docs/onboard/). Keep the following information handy after creating the cluster: * `$levitate_remote_write_url` - Last9 cluster’s Remote write endpoint * `$levitate_remote_write_username` - Cluster ID * `$levitate_remote_write_password` - Write token created for the cluster ## Download & extract VMAgent VMAgent is available in as part of the VictoriaMetrics repository’s [latest releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest). First obtain the required binary. ```bash $ ARCH=$(uname -m) case $ARCH in x86_64 | amd64) ARCH="amd64" ;; aarch64 | arm64) ARCH="arm64" ;; *) echo "Unsupported architecture: $ARCH" exit 1 ;; esac $ wget -O /var/tmp/vmutils.tar.gz "https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.96.0/vmutils-linux-${ARCH}-v1.96.0.tar.gz" ``` Extract VMAgent from the compressed file i.e `/var/tmp/vmutils.tar.gz` ```bash $ sudo mkdir -p /opt/vmagent $ tar -xzf /var/tmp/vmutils.tar.gz -C /opt/vmagent ``` ## Setup VMAgent scrape config Create a scrape config in the same directory as the VMAgent binary. ```bash $ sudo cat > /opt/vmagent/vmagent.yaml </etc/systemd/system/vmagent.service < Step by step guide on how to setup vmoperator with only vmagent and vmservicescrape to scrape your Kubernetes svcs and remote write metrics to Last9 The [vmoperator](https://github.com/VictoriaMetrics/operator) streamlines the deployment and management of vmagent on Kubernetes, optimizing for ease of use while retaining native configuration options inherent to Kubernetes environments. It achieves this by introducing various custom resource definitions (CRDs) into the Kubernetes ecosystem and send metrics semalessly to Last9. **Custom Resource Definitions (CRDs)** * vmagent * vmnodescrapes * vmservicescrapes * vmprobes * vmpodscrapes * vmstaticscrapes These CRDs empower users to effortlessly create and manage vmagent instances along with scrape configurations like VMServiceScrape, VMPodScrape, etc. These configurations closely resemble the [Prometheus Operator’s](https://prometheus-operator.dev/) ServiceMonitor and PodMonitor. This eliminates the need for manual setup of vmagent deployment, image configuration, and other intricate details. ## Prerequisites Make sure you have the following prerequisites installed: * [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl) - Kubernetes command-line tool * Clone this repository to your local machine: ```bash git clone https://github.com/last9/vmagent-operator-levitate.git cd vmagent-operator-levitate ``` ## Install Custom Resource Definitions (CRDs) 1. Navigate to the `crds/` directory: ```bash cd ./crds/ ``` 2. Install the CRDs using `kubectl`: ```bash $ kubectl apply -f ./crd.yaml ``` 3. Verify that the CRDs are successfully installed: ```bash $ kubectl get crd --sort-by=.metadata.creationTimestamp # Output NAME CREATED AT vmagents.operator.victoriametrics.com 2023-12-26T12:02:48Z vmnodescrapes.operator.victoriametrics.com 2023-12-26T12:02:49Z vmservicescrapes.operator.victoriametrics.com 2023-12-26T12:02:50Z vmprobes.operator.victoriametrics.com 2023-12-26T12:02:50Z vmpodscrapes.operator.victoriametrics.com 2023-12-26T12:02:50Z vmstaticscrapes.operator.victoriametrics.com 2023-12-26T12:02:51Z ``` You should see the names of the installed CRDs in the output. ## Install vmoperator 1. Navigate to the `operator/` directory: ```bash cd ./operator/ ``` 2. Install the operator and rbac using `kubectl`: ```bash $ kubectl apply -f ./manager.yaml -f rbac.yaml ``` Note This creates a separate `last9-monitoring` namespace to not clash with your existing namespaces. 3. Verify the status of the operator: ```bash $ kubectl get pods -n monitoring-system # Output NAME READY STATUS RESTARTS AGE vm-operator-667dfbff55-cbvkf 1/1 Running 0 101s 2023-12-26T12:02:51Z ``` You should see the vmoperator pod in running status. ## Install vmagent 1. Navigate to the `vmagent/` directory: ```bash cd ./vmagent/ ``` 2. You will need to obtain your Last9 cluster’s Remote Write URL and its credentials. [Here](/docs/onboard/) is a quick way to create your cluster and obtain your credentials. Run the below command to list all the placeholder values in this file [vmagent.yaml](https://github.com/last9/vmagent-operator-levitate/blob/main/vmagent/vmagent.yaml) ```bash $ cat ./vmagent.yaml | grep -n "Todo" # Output 11: levitate_cluster_username: "" # Todo: append levitate cluster username 12: levitate_cluster_password: "" # Todo: append levitate cluster password 31: via_cluster: # Todo: add a relevant cluster name. e.g: k8s cluster name 33: - url: # Todo: append levitate remote write URL 55: storage: 20Gi # Todo: Default is 20Gi. Scale up after you have provisioned more if you need more 58:# Todo: Below configs need to be enabled depending upon your affinity towards nodegroups. 59:# Todo: Ensure that the below selector terms and tolerations are exactly same as the metadata of the nodegroups itself. ``` 3. Proceed to installation once you have replaced the placeholder values with actual values 4. Install vmagent in the `last9-monitoring` namespace using `kubectl`: ```bash $ kubectl apply -f ./vmagent.yaml -n last9-monitoring ``` 5. Verify the status of the vmagent and ensure that it’s running: ```bash $ kubectl get pods -n last9-monitoring -l "last9_monitoring_agent=vmagent" # Output NAME READY STATUS RESTARTS AGE vmagent-demo-6785f7d7b9-zpbv6 2/2 Running 0 72s ``` ## Install VMServiceScrape (i.e ServiceMonitor and PodMonitor) Navigate to the `vmservicescrape/` directory: ````plaintext ```bash cd ./vmservicescrape/ ``` ```` **Caveats** VMServiceScrape works similar to Prometheus Operator’s ServiceMonitor and PodMonitor where you can define scrape selectors to do service and pod discovery. In this file [vmservicescrape.yaml](https://github.com/last9/vmagent-operator-levitate/blob/main/vmservicescrape/vmservicescrape.yaml) you can override default scrape selectors to suit your requirements. By default this assumes the default labels that are applied to the exporter as part of their installations. Below is the generic command to find you the labels of your K8s Services. ````plaintext ```bash $ kubectl get services -n -o jsonpath='{.metadata.labels}' ``` ```` Once, you have inspected the labels for the services that you chose to scrape, you can then proceed to modify this file [vmservicescrape.yaml](https://github.com/last9/vmagent-operator-levitate/blob/main/vmservicescrape/vmservicescrape.yaml) and match the scrape selector labels with the labels of your services. Another caveat to note here is to ensure that the namespaces are also declared correctly for the scrape selectors to correctly perform service discovery. This file also includes scrape configs for Common Exporters such as Kafka, Redis, Node, RabbitMQ, Prometheus Pushgateway etc. Run this command to list all the `Todo` comments which will guide you to customize this file [vmservicescrape.yaml](https://github.com/last9/vmagent-operator-levitate/blob/main/vmservicescrape/vmservicescrape.yaml) as required. ````plaintext ```bash $ cat ./vmservicescrape.yaml | grep -n "Todo" # Output 48: matchNames: [ "kube-system" ] # Todo: append namespaces here 63: matchNames: [ "kube-system" ] # Todo: append namespaces here 86:# Todo: Uncomment this if you have custom application enabled svcs running and you want to scrape them 97:# matchNames: [ ] # Todo: Append more namespaces here 105:# app: "" # Todo: Append app name label here 107:# Todo: Uncomment this if you have node exporters svcs running and you want to scrape them 118:# matchNames: [ ] # Todo: Append more namespaces here 129:# Todo: Uncomment this if you have rabbitMQ exporters svcs running and you want to scrape them 140:# matchNames: [ ] # Todo: Append more namespaces here 151:# Todo: Uncomment this if you have kafka exporters svcs running and you want to scrape them 162:# matchNames: [ ] # Todo: Append more namespaces here 173:# Todo: Uncomment this if you have redis exporters svcs running and you want to scrape them 184:# matchNames: [ ] # Todo: Append more namespaces here 195:# Todo: Uncomment this if you have pushgateway svcs running and you want to scrape them 206:# matchNames: [ ] # Todo: Append more namespaces here ``` ```` Install VMServiceScrape using `kubectl`: ````plaintext ```bash $ kubectl apply -f ./vmservicescrape.yaml -n last9-monitoring # Output vmservicescrape.operator.victoriametrics.com/last9-vmservicescrape-vmagent-01 created vmservicescrape.operator.victoriametrics.com/last9-servicescrape-k8s-01 created vmservicescrape.operator.victoriametrics.com/last9-servicescrape-metrics-server-01 created ``` ```` *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Tutorials > Find best practices recipes and tutorials for monitoring and observability curated by the Last9 team. 1. [Calculate usage patterns and data volume in Prometheus](/docs/how-to-calculate-usage-patterns-and-data-volume-in-prometheus/) 2. [Scrape selective metrics in Prometheus](/docs/how-to-scrape-only-selective-metrics-in-prometheus/) 3. [Using OpenTelemetry Exporter for Prometheus Remote Write](/docs/using-open-telemetry-exporter-for-prometheus-remote-write/) 4. [Scrape selective kube state metrics](/docs/how-to-scrape-selective-kube-state-metrics/) 5. [Setting up Docker and Docker Compose](/docs/setting-up-docker-and-docker-compose-on-linux/) 6. [Install VictoriaMetrics VMAgent on Ubuntu](/docs/how-to-setup-vmagent-on-ubuntu/) 7. [Create a GCP service account with read-only access for monitoring](/docs/create-gcp-service-account-with-read-only-access/) 8. [Setup Kubernetes monitoring using kube-state-metrics(KSM) and Prometheus](/docs/ingest-kubernetes-metrics-via-prometheus/) 9. [Setup vmoperator with only vmagent and vmservicescrape](/docs/how-to-setup-vmoperator-with-vmagent-in-kubernetes/) 10. [Enable EC2 Service Discovery with vmagent](/docs/how-to-enable-ec2-service-discovery-with-vmagent/) 11. [Create AWS STS (Secure Token Service) Role](/docs/how-to-create-aws-sts-role/) 12. [Monitor RabbitMQ using Last9](/docs/integrations-rabbitmq/) 13. [Delegate Subdomain between two AWS Accounts using Route 53](/docs/delegate-subdomain-between-aws-accounts-using-route-53/) # Querying Last9 using HTTP API > How to query metrics from Last9 using HTTP API This step-by-step guide explains how to query Last9 using HTTP API. ## Last9 Read URL Create a Last9 cluster by following [Getting Started](/docs/onboard/). Each Last9 Cluster comes with a Read URL which needs to be used when querying metrics data from Last9.  You can grab the Read URL by going to the Last9 Cluster → Settings → Read Data → Bring Your Own Visualization. Note Last9 Read URL mandates authenticated access. Create a Read Token for your cluster and use it as `password`. Use the cluster id as `username`. Keep the following information handy after creating the Last9 cluster: * `$levitate_read_url` - Last9’s Read endpoint * `$levitate_username` - Cluster ID * `$levitate_password` - Read token created for the cluster ## Authentication Generate the authorization header for authenticated access to Last9 metrics API as follows. ```bash USERNAME="$levitate_ cluster_username" PASSWORD="$levitate_cluster_password" BASIC_AUTH_HEADER=$(echo -n "$USERNAME:$PASSWORD" | base64) AUTH_HEADER="Authorization: Basic $BASIC_AUTH_HEADER" ``` Note Last9 is Prometheus compatible TSDB. You can query metrics stored in Last9 using Prometheus HTTP API. ## Instant Query The simplest way to query Last9 is to use `$levitate_read_url` along with `$AUTH_HEADER`, which allows you to execute an instant query. ```bash curl -XPOST "$levitate_read_url/api/v1/query?query=" -H "$AUTH_HEADER" ``` **Example**: Query the current CPU usage: ```bash curl -XPOST "$levitate_read_url/api/v1/query?query=node_cpu_seconds_total{}" -H "$AUTH_HEADER" ``` ## Range Query Use the `query_range` endpoint to retrieve data over a time range. You need to specify the start time, end time, and step duration. ```bash curl -XPOST '$levitate_read_url/api/v1/query_range?query=&start=&end=&step=' -H "$AUTH_HEADER" ``` **Example**: Query CPU usage over the last hour with a 1-minute step: ```bash curl -XPOST '$levitate_read_url/api/v1/query_range?query=node_cpu_seconds_total&start=$(date -d "1 hour ago" +%s)&end=$(date +%s)&step=60' -H "$AUTH_HEADER" ``` Tip For more details, refer to the [Prometheus HTTP API documentation](https://prometheus.io/docs/prometheus/latest/querying/api/#http-api). *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Indicators > Overview of Indicators ## Indicator Overview Indicators are PromQL queries saved as a part of an Alert Group. Alert Rules evaluate these Indicators against thresholds to generate alerts. ## Creating an Indicator Before you configure an Alert Rule, you need at least one indicator to be created in the Alert Group. To create an Indicator: 1. Navigate to the Alert Group in which you would like to create the indicator: **Home** → **Alert Studio** → **Alert Groups** → *Select an Alert Group* → **Indicators** Tab and Click on the **Create New Indicator**   2. The following details are required for an Indicator: 1. **Indicator Name**: Use a descriptive name so that the Indicator is easily identified 2. **Indicator Description** *(Optional)*: Helps your team members identify the purpose of the Indicator 3. **Query**: The PromQL query for the indicator. This can be a query that returns multiple timeseries (as seen in the example below) but cannot contain any variables (for example, `$instance`) 4. **Unit**: The unit that which want to assign to the indicator 5. **Data Source** *(Advanced)*: By default, Indicators inherit the data source from their Alert Group. That is if you change the data source for the Alert Group the same data source will be used for the indicator. You can also override this behavior and assign a different data source for the Indicator. Once you override the Indicator’s data source, the configured data source will now take precedence over the data source configured at the Alert Group level  After entering the query, you will need to validate it to ensure that it has no syntax errors. If the query is validated successfully, a preview will be generated.  Click **Create Indicator** to save this Indicator.  3. This Indicator is now ready to be used in Alert Rules. To edit/duplicate or delete this indicator can click the **…** button  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Setup Kubernetes monitoring using kube-state-metrics(KSM) and Prometheus Agent > Step by step guide to enable ingesting Kubernetes metrics via Prometheus Agent and send to Last9 via remote write. ## Pre-requisites 1. Ensure that your [kubectl configuration](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) is pointing to the right Kubernetes cluster 2. Create a Last9 cluster by following [Quick start guide](/docs/onboard/) ## What is kube-state-metrics(KSM) `kube-state-metrics` (KSM) is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. It is not focused on the health of the individual Kubernetes components but on the health of the various objects inside, such as deployments, nodes, and pods. The metrics are exported by default to the port’s HTTP endpoint `/metrics` on port 8080. They are served as plaintext. They are designed to be consumed either by Prometheus itself or by a scraper compatible with scraping a Prometheus client endpoint. You can also open `/metrics` in a browser to see the raw metrics. Note that the metrics exposed on the `/metrics` endpoint reflect the current state of the Kubernetes cluster. When Kubernetes objects are deleted, they are no longer visible on the `/metrics` endpoint. Tip The documentation for the metrics exposed by KSM can be found [here](https://github.com/kubernetes/kube-state-metrics/tree/main/docs/). ## Automated installation (Preferred) ### Step 1: Copy the installation command ### Step 2: Run the installation command Before running the command, update it to use the write token of the Last9 cluster. ```text ``` Running the command will download the manifest yaml in the current working directory. It is strongly recommended that you check the manifest file in git so that it can be extended later. You can just follow the video to see the end-to-end setup. ## Manual Installation 1. Clone the GitHub repo ```shell git clone https://github.com/kubernetes/kube-state-metrics.git ``` 2. Deployment steps To deploy this project, you can simply run `kubectl apply -f examples/standard`, and a Kubernetes service and deployment will be created. ```shell kubectl apply -f examples/standard ``` Read for more details on deployment [here](https://github.com/kubernetes/kube-state-metrics?tab=readme-ov-file#kubernetes-deployment). 3. Validate corresponding deployment ```shell kubectl get deployments kube-state-metrics -n kube-system ``` This is the sample output that you should see. ```shell NAME READY UP-TO-DATE AVAILABLE AGE kube-state-metrics 1/1 1 1 6d1h ``` ## Configure remote write to Last9 If you already have a running Prometheus setup, add the attached scrape configs, and remote write setup to your Prometheus config file to send data to Last9. ```yaml # prometheus.yaml scrape_configs: - job_name: "node-exporter" kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_endpoints_name] regex: "node-exporter" action: keep - job_name: "kubernetes-apiservers" kubernetes_sd_configs: - role: endpoints scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs: - source_labels: [ __meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name, ] action: keep regex: default;kubernetes;https - job_name: "kubernetes-nodes" scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics - job_name: "kubernetes-pods" kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name - job_name: "kube-state-metrics" static_configs: - targets: ["kube-state-metrics.kube-system.svc.cluster.local:8080"] - job_name: "kubernetes-cadvisor" scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: kubernetes.default.svc:443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor - job_name: "kubernetes-service-endpoints" kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: __scheme__ regex: (https?) - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] action: replace target_label: kubernetes_name remote_write: - url: remote_timeout: 60s queue_config: capacity: 10000 max_samples_per_send: 3000 batch_send_deadline: 20s min_shards: 4 max_shards: 200 min_backoff: 100ms max_backoff: 10s basic_auth: username: password: ``` * Replace the `cluster` variable in `external_labels` as per the description ```yaml external_labels: # TODO - replace xyz.acme.io with a logical name for the cluster being scraped. # by Prometheus e.g. prod1.xyz.com cluster: "xyz.acme.io" ``` Tip If you do not have a Prometheus setup, you can [setup vmagent](/docs/how-to-setup-vmagent-on-ubuntu/) as well. ## Steps to uninstall the KSM setup ### For automated setup To uninstall the Kubernetes resources that were created by the automated installation, you can use the `kubectl delete` command with the `-f` flag pointing to the same YAML file. This will delete all the resources defined in the file. ```bash kubectl delete -f kube-state-metrics.yml ``` This command will remove the namespaces, deployments, services, service accounts, and any other resources defined in the `kube-state-metrics.yml` file. ### For manual setup Delete the created kube-state-metrics objects. ```shell kubectl delete -f examples/standard ``` # Ingestion Tokens > Create and manage tokens for sending telemetry data to Last9, including RUM and Prometheus metrics.  [Ingestion Tokens](https://app.last9.io/control-plane/ingestion-tokens) authenticate your applications and services when sending telemetry data to Last9. These tokens control what data can be sent and from which origins, ensuring secure data collection. ## Creating Ingestion Tokens  1. **Select Token Type**: * **Client**: For client-based data collection (eg: RUM) * **Prometheus Remote-Write**: For server-side Prometheus-compatible metrics 2. **Configure Client Type** (if Client selected): * **Web Browser**: Default and currently available option for web applications * **Mobile**: Coming soon for mobile app monitoring 3. **Set Origins** (for Client tokens): * Add the domains from which your application will send data * Example: `https://www.example.com` * Multiple origins can be added for multi-domain applications * Subdomain matching is not automatic, each subdomain needs separate entries Caution Data sent from origins not listed here will be rejected. ```plaintext ✅ Correct Origins: https://app.example.com https://www.example.com http://localhost:3000 ❌ Incorrect Origins: example.com (missing protocol) *.example.com (wildcards not supported) app.example.com (missing protocol) ``` 4. **Name Your Token**: * Use a descriptive name to identify the token’s purpose * Example: “Production RUM Token” or “Staging Web Monitoring” 5. Click **CREATE TOKEN** to generate your token *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Instrumentation > Instrumentation Overview [ Prometheus ](/docs/integrations-prometheus/)[ vmagnet ](/docs/integrations-vmagent/)[ OpenTelemetry ](/docs/integrations-opentelemetry-collector/)[ Express.js ](/docs/integrations-opentelemetry-expressjs/)[ NestJS ](/docs/integrations-opentelemetry-nestjs/)[ Next.js ](/docs/integrations-opentelemetry-nextjs/)[ Koa ](/docs/integrations-opentelemetry-koa/)[ Django ](/docs/integrations-opentelemetry-django/)[ Flask ](/docs/integrations-opentelemetry-flask/)[ Kong ](/docs/integrations-opentelemetry-kong-gateway/)[ Phoenix Framework ](/docs/integrations-opentelemetry-phoenix/)[ Ruby on Rails ](/docs/integrations-opentelemetry-ruby-on-rails/)[ Sinatra ](/docs/integrations-opentelemetry-ruby-on-rails/)[ Roda ](/docs/integrations-opentelemetry-ruby-on-rails/)[ Fluent Bit ](/docs/integrations-opentelemetry-fluent-bit/)[ Ubuntu ](/docs/integrations-opentelemetry-ubuntu/)[ Datadog Agent ](/docs/integrations-opentelemetry-datadog-agent/)[ Ubuntu Host Metrics ](/docs/integrations-opentelemetry-ubuntu-host-metrics/)[ Gin ](/docs/integrations-opentelemetry-gin/)[ gRPC ](/docs/integrations-opentelemetry-grpc/)[ FastHTTP ](/docs/integrations-opentelemetry-fasthttp/)[ Iris ](/docs/integrations-opentelemetry-iris/)[ Gorilla Mux ](/docs/integrations-opentelemetry-gorilla-mux/)[ Logs from Kubernetes Cluster ](/docs/integrations-opentelemetry-kubernetes-logs/)[ Kubernetes Audit Logs ](/docs/integrations-opentelemetry-kubernetes-audit-logs/)[ AWS EC2 Instance ](/docs/integrations-opentelemetry-aws-ec2/)[ Logs from AWS S3 ](/docs/integrations-opentelemetry-aws-s3/)[ MariaDB ](/docs/integrations-opentelemetry-mariadb/)[ statsd ](/docs/integrations-statsd/)[ AWS Cloudwatch Metric Stream ](/docs/integrations-aws-cloudstream/)[ Telegraf ](/docs/integrations-telegraf/)[ jmxtrans ](/docs/integrations-jmxtrans/)[ Confluent ](/docs/integrations-confluent-cloud/)[ Loki ](/docs/integrations-grafana-loki-ruler/)[ Akamai ](/docs/integrations-akamai/)[ Keda ](/docs/integrations-keda/)[ Prodvana ](/docs/integrations-prodvana/)[ LaunchDarkly ](/docs/integrations-launchdarkly/)[ Apache APISIX ](/docs/integrations-apache-apisix/)[ Fastly ](/docs/integrations-fastly/)[ Cloudflare Logs ](/docs/integrations-cloudflare-logs/)[ Cloudflare Workers](/docs/integrations-cloudflare-workers/) # Akamai > Send logs and metrics to Last9 from Akamai for CDN monitoring This document lists step-by-step instructions for Akamai CDN monitoring with Last9. ## Prerequisites Create a Last9 cluster by following [Getting Started](/docs/onboard/). Keep the following information handy after creating the cluster: * `$levitate_remote_write_url` - Last9’s Remote write endpoint * `$levitate_remote_write_username` - Cluster ID * `$levitate_remote_write_password` - Write token created for the cluster ## Setup Last9 supports ingesting logs and metrics from Akamai. ### Logs Last9 leverages [Akamai Datastream V2](https://techdocs.akamai.com/datastream2/docs/welcome-datastream2) Custom HTTPS endpoint integration to push logs from Akamai to Last9. Datastream can gather performance and security data for your global Akamai edge platform properties and stream them Last9. #### Configuration Add Last9 ingestion endpoint for Akamai logs as Custom HTTPS endpoint. 1. In Destination, select Custom HTTPS. 2. Enter a human-readable description for the destination as a name. 3. In Endpoint URL, enter the ingestion endpoint from [Akamai integration](https://app.last9.io/integrations?category=all\&integration=Akamai+Logs). 4. In Authentication, select: None for no authentication. The authentication is handled as **Basic Authorization** within the URL itself. 5. Last9 Akamai ingestion endpoint supports JSON payloads. Enable `application/json` as the content type and log format as JSON instead of structured logs. 6. Click Validate & Save to validate the connection to Last9. ## Verification Visit [Logs Panel](https://app.last9.io/logs) to see the Akamai logs. > More details on the custom HTTPS endpoint can be found [here](https://techdocs.akamai.com/datastream2/docs/stream-custom-https). ### Metrics Last9 uses the [Akamai Reporting API](https://techdocs.akamai.com/reporting/reference/get-report-version-data) to fetch the relevant metrics and push them to Last9. [Create an Akamai API Client](https://techdocs.akamai.com/developer/docs/set-up-authentication-credentials) for the relevant reporting APIs, share the following credentials with the Last9 team. Tip The complete list of available reports can be found [here](https://techdocs.akamai.com/reporting/reference/available-reports). * `host` * `client_token` * `client_secret` * `access_token` The data flow is as follows.  #### CP Codes Content Provider codes (CP codes) identify your traffic on the ​Akamai​ network for reporting, billing, and monitoring purposes. The Last9 Akamai integration needs access to the CP Codes from your account to map the metrics underlying the CP Code correctly. Please share the CP Codes in the following CSV format. ```csv Network(ESSL/FF),CPCode,Expected peak hits/s for the CPCode,List of hostnames,Path ESSL,1395076,65000,playback-content.mystreamingservice.com,/stream/v1/users ``` #### Standard Akamai Metrics This integration collects the following standard metrics from Akamai for given CP codes. * `edgeHits` * `hitsOffload` * `originHits` * `bytesOffload` * `edgeBytes` * `midgressBytes` * `originBytes` * `edgeHitsTotal` * `originHitsTotal` * `hitsOffload` * `bytesOffload` * `edgeBytesTotal` * `midgressBytesTotal` * `originBytesTotal` Tip The Akamai reporting API reports metrics at 5-minute intervals. ## Next steps Once the credentials are shared with the Last9 team, the Akamai connector will be enabled for your account. Moreover, metrics will start flowing into your Last9 cluster. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Apache APISIX > Send APISIX metrics and traces to Last9 using Prometheus Remote Write and OpenTelemetry This document will showcase how to send metrics and traces from [Apache APISIX](https://apisix.apache.org/) to Last9. ## Pre-requisites 1. You have created a Last9 cluster by following the [getting started guide](/docs/onboard/) 2. You have Apache APISIX set up and running. ## Setup Metrics  ### Enable Prometheus Plugin in APISIX The APISIX Prometheus Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats). It can be enabled with following configuration. ```yaml plugin_attr: prometheus: export_uri: /apisix/prometheus/metrics ``` The exposed metrics can now be fetched as follows: ```yaml curl -i http://127.0.0.1:9091/apisix/prometheus/metrics ``` Follow the official documentation for more details and configuration options for the Prometheus Plugin [here](https://apisix.apache.org/docs/apisix/plugins/prometheus/). ### Setup Prometheus Agent Follow [this](/docs/integrations-prometheus/) guide to setup Prometheus agent which will remote write metrics to Last9. ### Configure Prometheus Agent to fetch metrics from APISIX Modify the `prometheus.yaml` as follows. ```yaml scrape_configs: - job_name: "apisix" scrape_interval: 15s # This value will be related to the time range of the rate function in Prometheus QL. The time range in the rate function should be at least twice this value. metrics_path: "/apisix/prometheus/metrics" static_configs: - targets: ["127.0.0.1:9091"] ``` Note The IP Address in the documentation is `127.0.0.1`, which may not be the same in your case. This IP address should ideally be the internal IP address of the instance in which APISIX is running. ### Grafana Dashboard You can import the Grafana dashboard for APISIX from the repo [here](https://github.com/apache/apisix/blob/master/docs/assets/other/json/apisix-grafana-dashboard.json) ### Available Metrics The following metrics are exported by the APISIX: * Status code: HTTP status code returned from Upstream services. They are available for a single service and across all services. * Bandwidth: Total amount of traffic (ingress and egress) flowing through APISIX. Total bandwidth of a service can also be obtained. * `etcd reachability`: A gauge type representing whether etcd can be reached by APISIX. A value of 1 represents reachable, and 0 represents unreachable. * Connections: Nginx connection metrics like active, reading, writing, and number of accepted connections. * Batch process entries: A gauge type useful when Plugins like syslog, http-logger, tcp-logger, udp-logger, and zipkin use batch process to send data. Entries that hasn’t been sent in batch process will be counted in the metrics. * Latency: Histogram of the request time per service in different dimensions. * Info: Information about the APISIX node. * Shared dict: The capacity and free space of all `nginx.shared.DICT` in APISIX. * `apisix_upstream_status`: Health check result status of upstream nodes. A value of 1 represents healthy and 0 represents unhealthy. Detailed information on the exposed metrics and their dimensions can be found [here](https://apisix.apache.org/docs/apisix/plugins/prometheus/#available-http-metrics) ## Setup Traces The APISIX OpenTelemetry plugin can be used to report tracing data according to the [OpenTelemetry specification](https://opentelemetry.io/docs/specs/otel/).  ### Enable OpenTelemetry Plugin in APISIX Enable the OpenTelemetry plugin as follows: ```yaml plugins: - opentelemetry plugin_attr: opentelemetry: resource: service.name: APISIX tenant.id: business_id collector: address: request_timeout: 3 request_headers: foo: bar batch_span_processor: drop_on_queue_full: false max_queue_size: 6 batch_timeout: 2 inactive_timeout: 1 max_export_batch_size: 2 ``` You can follow the official documentation on enabling the OpenTelemetry plugin [here](https://apisix.apache.org/docs/apisix/plugins/opentelemetry/#enable-plugin). This will send traces from APISIX to Last9 with [OpenTelemetry Specification](https://opentelemetry.io/docs/specs/otel/). *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # AWS Cloudwatch Metric Stream > AWS Cloudwatch Metric Stream enables customers to send their Cloudwatch metrics to Last9. ## Pre-requisites Obtain the following and copy it to your clipboard from the [Home > Integrations > Cloudwatch](https://app.last9.io/integrations?category=cloudwatch) section. 1. `HTTP Endpoint URL` 2. `Username` 3. `Password`  ## Setting up required IAM policy Note Ensuring your AWS Identity and Access Management (IAM) user account has access permissions is crucial. The following access policy is specifically crafted to enable actions related to creating Kinesis Data Streams and CloudWatch Metric Streams. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudwatch:StartMetricStreams", "cloudwatch:PutMetricStream", "cloudwatch:GetMetricStream", "cloudwatch:GetMetricData", "cloudwatch:ListMetrics", "cloudwatch:ListMetricStreams" ], "Resource": ["*"] }, { "Effect": "Allow", "Action": [ "firehose:PutRecord", "firehose:CreateDeliveryStream", "firehose:DescribeDeliveryStream", "firehose:PutRecordBatch", "firehose:UpdateDestination", "firehose:ListDeliveryStreams" ], "Resource": ["*"] }, { "Effect": "Allow", "Action": [ "s3:ListAllMyBuckets", "s3:CreateBucket", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:ListBucketMultipartUploads", "s3:PutObject" ], "Resource": ["arn:aws:s3:::*"] }, { "Effect": "Allow", "Action": [ "iam:CreateRole", "iam:CreatePolicy", "iam:AttachRolePolicy", "iam:CreatePolicyVersion", "iam:DeletePolicyVersion", "iam:PassRole" ], "Resource": [ "arn:aws:iam:::policy/*", "arn:aws:iam:::role/*" ] }, { "Effect": "Allow", "Action": ["logs:CreateLogGroup", "logs:CreateLogStream"], "Resource": [ "arn:aws:logs:::log-group:*:log-stream:*" ] } ] } ``` ## Creating an AWS Kinesis Delivery Stream 1. Open the AWS Kinesis homepage ([console.aws.amazon.com/kinesis/home](https://console.aws.amazon.com/kinesis/home)) 2. Open the left sidebar (click on the ☰ icon, if it is not expanded already) 3. Click on `Delivery Streams`  4. Click on `Create delivery stream`  5. Choose `Direct PUT`  6. Delivery stream name = `last9-$your_organization_name`  7. Set the copied write HTTP Endpoint URL from the Last9 cluster as an HTTP endpoint.  8. Add `username` and `password`.  9. Choose or create an S3 bucket to save data the stream failed to deliver  10. Click on `Create delivery stream`  ## Sending data from Cloudwatch to the delivery stream 1. Open the Cloudwatch console and click on `Metrics -> Streams`  2. Click on `Create metric stream`  3. Choose `All metrics` to send all Cloudwatch metrics. Optionally, you can also select the metrics you want to stream. You can include or exclude specific namespaces and metrics you want to send by using `Select metrics` option.  4. Ensure that you use the delivery stream created in the earlier step and that the output format is `OpenTelemetry 0.7`.  5. Enter the Custom Metric stream name as `last9-$your_organization_name` and then click on `Create metric stream`  ## Verification Once the Cloudwatch metric stream is enabled, it sends metrics with the prefix `amazonaws_com_AWS prefix`. They can be observed in the Hosted Grafana in the Grafana tab in Last9. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Cloudflare Logs > Send logs to Last9 from Cloudflare for CDN monitoring and worker monitoring via Logpush This document lists step-by-step instructions for pushing logs from Cloudflare to Last9. ## Prerequisites 1. Create a Last9 account by following [Getting Started](/docs/onboard/). 2. Keep the following information handy from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `$last9_otlp_endpoint`: Last9’s OTLP endpoint copies from Cloudflare Integration section from [Integrations](https://app.last9.io/integrations/) page. * `$last9_basic_auth_header`: OTLP Basic authorization header ## Setup You can use Cloudflare’s [Logpush](https://developers.cloudflare.com/logs/about/) Custom [HTTPS endpoint integration](https://developers.cloudflare.com/logs/get-started/enable-destinations/http/) to push logs from Cloudflare to Last9. Logpush delivers logs in batches as quickly as possible, with no minimum batch size, potentially delivering files more than once per minute. This capability enables Cloudflare to provide information almost in real time, in smaller file sizes. Logpush does not offer storage or search functionality for logs; its primary aim is to send logs as quickly as they arrive which makes it perfect candidate to send logs to [Last9 Log Management](https://last9.io/logs/). ### Via Cloudflare dashboard 5. In **Select a destination**, choose **HTTP destination**. 6. Enter the **HTTP endpoint** as the Last9 Endpoint, and select **Continue**. ```shell https://$last9_otlp_endpoint/cloudflare?header_Authorization= ``` 7. Select the dataset to push to the storage service. 8. In the next step, you need to configure your logpush job: * Enter the **Job name** as Last9. * Under **If logs match**, you can select the events to include and/or remove from your logs. Refer to [Filters](https://developers.cloudflare.com/logs/reference/filters/) for more information. Not all datasets have this option available. * In **Send the following fields**, you can choose to either push all logs to Last9 or selectively choose which logs you want to push. 9. In **Advanced Options**: * Choose the format of timestamp fields in your logs to be `UnixNano`. 10. Select **Submit** once you are done configuring your logpush job. ### Via API To create a Logpush job, make a `POST` request to the [Logpush job creation endpoint URL](https://developers.cloudflare.com/logs/get-started/api-configuration/) with the appropriate parameters. #### Example curl request ```bash curl https://api.cloudflare.com/client/v4/zones/{zone_id}/logpush/jobs \ --header "X-Auth-Email: " \ --header "X-Auth-Key: " \ --header "Content-Type: application/json" \ --data '{ "name": "last9", "output_options": { "field_names": ["EdgeStartTimestamp", "RayID"], "timestamp_format": "unixnano" }, "destination_conf": "https://$last9_otlp_endpoint/cloudflare?header_Authorization=", "dataset": "http_requests", "enabled": true }' ``` ## Verification Visit [Log Explorer](https://app.last9.io/logs) to see the Cloudflare logs in action. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Cloudflare Workers > Monitor Cloudflare Workers using OpenTelemetry and Last9 This document lists step-by-step instructions for setting up monitoring for Cloudflare Workers with Last9. ## Prerequisites 1. Create a Last9 account by following [Getting Started](/docs/onboard/). 2. Keep the following information handy from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `$last9_otlp_endpoint`: Last9’s OTLP endpoint copies from Cloudflare Integration section from [Integrations](https://app.last9.io/integrations/) page. * `$last9_basic_auth_header`: OTLP Basic authorization header ## Setup Instrument your [Cloudflare Worker](https://developers.cloudflare.com/workers/) applications with [OpenTelemetry](https://opentelemetry.io/) using the the [otel-cf-workers](https://github.com/evanderkoogh/otel-cf-workers) SDK. ### Step 1: Install the SDK Install `@microlabs/otel-cf-workers` in your project. ```bash npm i @microlabs/otel-cf-workers ``` ### Step 2: Add Node.js Compatibility Flags OpenTelemetry requires the Node.js Compatibility flag is enabled at the top level of your `wrangler.toml` file. ```toml compatibility_flags = [ "nodejs_compat" ] ``` ### Step 3: Configure the tracer In your Cloudflare worker file, add the following configuration code to configure OpenTelemetry. ```typescript import { instrument, ResolveConfigFn } from "@microlabs/otel-cf-workers"; export interface Env { LAST9_BASIC_AUTH: string; // Last9 Basic Auth Header SERVICE_NAME: string; // Your service name } const handler = { async fetch( request: Request, env: Env, ctx: ExecutionContext, ): Promise { // your cloudflare worker code }, }; const config: ResolveConfigFn = (env: Env, _trigger) => { return { exporter: { url: `$last9_otlp_endpoint/v1/traces`, headers: { Authorization: env.LAST9_BASIC_AUTH }, }, service: { name: env.SERVICE_NAME }, }; }; export default instrument(handler, config); ``` ### Step 4: Set the Last9 environment variables In your [Cloudflare Workers Secret Configuration](https://developers.cloudflare.com/workers/configuration/secrets/) add the `LAST9_BASIC_AUTH`. To enable tracing for local dev add your `LAST9_BASIC_AUTH` to your `.dev.vars` file ```js LAST9_BASIC_AUTH = $last9_basic_auth_header; ``` In your `wrangler.toml` file set the `SERVICE_NAME` variable ```toml [vars] SERVICE_NAME = "my-service-name" ``` Once these steps are completed, distributed traces from your Cloudflare Workers application should be available in [Last9 Trace Explorer](https://app.last9.io/traces). ## Adding custom OpenTelemetry spans To add custom spans to your OpenTelemetry traces, install the `@opentelemetry/api` package. ```bash npm i @opentelemetry/api ``` And manually add spans to your traces. ```typescript import { trace } from "@opentelemetry/api"; const tracer = trace.getTracer("custom-traces"); const handler = { async fetch( request: Request, env: Env, ctx: ExecutionContext, ): Promise { const span = trace.getActiveSpan(); span.setAttribute("search", search); const result = await tracer.startActiveSpan( `transaction-started`, async (span) => { // your business logic const input = { search }; span.setAttributes(input); const result = await transactionLogic(input); span.setAttributes(result); return result; }, ); }, }; ``` ## Verification Visit [Trace Explorer](https://app.last9.io/traces) to see the Cloudflare traces in action. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Confluent Cloud > Send Kafka metrics from Confluent Cloud to Last9 using Prometheus Remote Write This document will showcase how to send Kafka metrics from Confluent Cloud to Last9 via the `vmagent` or Prometheus. ## Pre-requisites 1. You have created a Last9 cluster by following the [getting started guide](/docs/onboard/) 2. You have an active Confluent cloud account ## Setup  The setup primarily has three components. 1. Confluent Cloud, which has the required metrics that you want to send to Last9. Confluent Cloud exposes metrics on `/export` GET endpoint. 2. `vmagent` or Prometheus will scrape the metrics from the `/export` endpoint and [remote write](https://last9.io/blog/what-is-prometheus-remote-write/) to Last9 The endpoint is as follows: This endpoint requires a[ Cloud API key for authentication](https://api.telemetry.confluent.cloud/docs?&_ga=2.51091241.1213369535.1676543952-257140517.1676543951#section/Authentication). The Cloud API key can be created using the [Confluent Cloud CLI](https://docs.confluent.io/current/cloud/cli/). Caution This API endpoint does not export [health metrics dataset](https://api.telemetry.confluent.cloud/docs/descriptors/datasets/health-plus). Once you have the key, use it in the `vmagent` configuration as follows: ```yaml scrape_configs: - job_name: Confluent Cloud scrape_interval: 1m scrape_timeout: 1m honor_timestamps: true static_configs: - targets: - api.telemetry.confluent.cloud scheme: https basic_auth: username: password: metrics_path: /v2/metrics/cloud/export params: "resource.kafka.id": - lkc-1 - lkc-2 ``` A sample docker-compose for`vmagent` can be as follows: ```yaml version: "3.5" services: vmagent: container_name: vmagent image: victoriametrics/vmagent ports: - 8429:8429 volumes: - vmagentdata:/vmagentdata - /var/tmp:/var/tmp - ./vmagent.yaml:/etc/vmagent/vmagent.yaml command: - "--promscrape.config=/etc/vmagent/vmagent.yaml" - "--remoteWrite.tmpDataPath=/var/tmp/vmagent/" - "--remoteWrite.maxDiskUsagePerURL=10737418240" - "--remoteWrite.url=$levitate_remote_write_url" - "--remoteWrite.basicAuth.username=$levitate_remote_write_username" - "--remoteWrite.basicAuth.password=$levitate_remote_write_password" restart: always network_mode: "host" volumes: vmagentdata: {} ``` This will start scraping metrics from Confluent Cloud and [remote write](https://last9.io/blog/what-is-prometheus-remote-write/) to Last9. Find more details on the export metrics API specification of Confluent Cloud [here](https://api.telemetry.confluent.cloud/docs?&_ga=2.51091241.1213369535.1676543952-257140517.1676543951#tag/Version-2/paths/~1v2~1metrics~1%7Bdataset%7D~1export/get). The doc for all metrics exposed by Confluent Cloud is [here](https://api.telemetry.confluent.cloud/docs/descriptors/datasets/cloud). ## Next steps Create a read token for your Last9 Cluster and follow our guide to [Configure Grafana](/docs/grafana-config/) to visualize the time series data getting sent to Last9. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Fastly > Send logs to Last9 from Fastly for CDN monitoring This document lists step-by-step instructions for pushing logs from Fastly CDN to Last9. ## Prerequisites 1. Create a Last9 cluster by following [Getting Started](/docs/onboard/). 2. Keep the following information handy after creating the cluster from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `$last9_otlp_endpoint`: Last9’s OTLP endpoint copies from Fastly Integration section from [Integrations](https://app.last9.io/integrations) page. * `$last9_basic_auth_header`: OTLP Basic authorization header ## Setup Last9 leverages Fastly’s [real time log streaming](https://docs.fastly.com/en/guides/about-fastlys-realtime-log-streaming-features) Custom HTTPS endpoint integration to push logs from Fastly to Last9. Add Last9 ingestion endpoint for Fastly logs as Custom HTTPS endpoint by following next steps. 1. Select the Service for which you want to send logs to Last9 and click on **Logging**.  2. Select HTTPS Endpoint.  3. Add the Last9 endpoint in the **URL** field. Use the `$last9_otlp_endpoint` copied in the earlier step. Ensure that you add following query parameters. These are mandatory parameters for the Last9 integration. ```bash ?source=fastly&service_id= ``` Tip You can also add any other additional query parameters to the URL. Make sure to URL encode the query parameters.  4. Add Authorization header under Advanced Configuration. Use the `$last9_basic_auth_header` copied in the earlier step.  Click `Create` to finish setting up Last9 logs ingestion endpoint. ## Verification Visit [Log Explorer](https://app.last9.io/logs) to see the Fastly logs in action. The service name for all Fastly logs is `fastly` unless overriden in [Last9 Control Plane](https://last9.io/control-plane/). *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Grafana Loki > Send data to Last9 using Grafana Loki’s recording rules’ remote write feature We make use of the Loki Ruler’s [remote-write](https://grafana.com/docs/loki/latest/alert/#remote-write) feature to allow Loki Ruler to write to an external remote endpoint i.e. Last9’s remote-write endpoint. ## Prerequisites Create a Last9 cluster by following [Getting Started](/docs/onboard/). Keep the following information handy after creating the cluster: * `$levitate_remote_write_url` - Last9’s Remote write endpoint * `$levitate_remote_write_username` - Cluster ID * `$levitate_remote_write_password` - Write token created for the cluster ## Setup  The setup is pretty straight forward. Grafana Loki has the required recording rules that are evaluated by the ruler component. This ruler component has the ability to remote write the declared recording rules as metrics. We need to configure the ruler component to remote write to Last9. This can be done as follows. Update the existing Loki configuration to include the remote write config under the ruler section. ```yaml ruler: ... remote_write: enabled: true client: url: "$levitate_remote_write_url" basic_auth: username: "$levitate_remote_write_username" password: "$levitate_remote_write_password" ``` After that, restart Loki, and data will be written to Last9. Find more details on the remote write configurations [here](https://grafana.com/docs/loki/latest/operations/recording-rules/#remote-write) ## Next steps Explore metric data using embedded Grafana by querying for the recording rule itself. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # jmxtrans > How to send metrics from Java-based applications to Last9 using jmxtrans [jmxtrans](https://github.com/jmxtrans/jmxtrans) is one of the common tools used to extract metrics from JVM via JMX and translate them into a variety of output metric formats e.g. statsd, telegraf, etc. This document will showcase how to use your existing `jmxtrans` setup to emit metrics to Last9. ## Pre-requisites 1. You have created a Last9 cluster by following the [getting started guide](/docs/onboard/) 2. Clone [last9-integrations](https://github.com/last9/last9-integrations/tree/master/levitate/remote-write/telegraf) GitHub repository contains the sample code with different approaches discussed in this article ## Setup  The setup primarily has 3 components: 1. The Java Application is the application under observation for monitoring. It exposes a JMX interface 2. A `jmxtrans` container queries the `jmx` endpoint exposed by the Java application for metrics and converts them into graphite output format. These metrics are pushed to `vmagent` 3. `vmagent` reads the data in the graphite format converts it into Prometheus-compatible format and remote writes to Last9 A complete example using this approach can be found [here](https://github.com/last9/last9-integrations/tree/master/levitate/remote-write/jmxtrans). ## Next steps Create a read token for your Last9 Cluster and follow our guide to [Configure Grafana](/docs/grafana-config/) to visualize the time series data getting sent to Last9. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # KEDA > Setup autoscaling with KEDA and Last9 This document lists step-by-step instructions for setting up auto scaling with [KEDA](https://KEDA.sh/) and Last9. ## What is KEDA KEDA is a Kubernetes-based Event Driven Autoscaler. With KEDA, you can drive the scaling of any container in Kubernetes based on the number of events needing to be processed. KEDA works by integrating with various event sources and metrics sources such as Last9 to dynamically adjust the number of replicas of Kubernetes deployments, StatefulSets, or any other scalable resources. ## Prerequisites Create a Last9 cluster by following [Getting Started](/docs/onboard/). Keep the following information handy after creating the cluster: * `$levitate_read_url` - Last9’s Read endpoint * `$levitate_username` - Cluster ID * `$levitate_password` - Read token created for the cluster ## KEDA Installation Refer to the KEDA [docs](https://KEDA.sh/docs/2.13/deploy/#helm) to install KEDA in your Kuberentes Cluster. Check the default `values.yaml` and modify as required. Below is a command to derive the default `values.yaml` ```yaml helm show values kedacore/keda > values.yaml ``` Ensure to install the following. 1. KEDA Operator - Responsible for activation and deactivation of Kubernetes [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) to scale to and from zero on no events 2. KEDA Metric Server - This is a [Kubernetes metrics server](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-custom-metrics) that exposes rich event data like queue length or stream lag to the Horizontal Pod Autoscaler to drive scale out It is up to the Deployment to consume the events directly from the source. This preserves rich event integration and enables gestures like completing or abandoning queue messages to work out of the box. The metric serving is the primary role of the `keda-operator-metrics-apiserver` container that runs when you install KEDA. ## KEDA Scalers KEDA scalers are the components within KEDA that interact with different event sources or metrics services to determine the current demand for an application and scale it accordingly. Each scaler is responsible for a specific type of event source or metric. For example, there are scalers for Prometheus Azure Service Bus, RabbitMQ, Kafka, HTTP requests, and many others. Scalers work by polling the event source at a specified interval to retrieve metrics that indicate the current load or demand. They then use this information to scale the application in or out. The scaling parameters and the thresholds for scaling can be customized through the scaler’s configuration. Keep in handy the following parameters which are variables that act as levers to define scale strategies in KEDA. * `serverAddress` - Read URL of Last9 Cluster * `metricName` - Name of the metric on which scaling decision will be based * `threshold` - Value to start scaling for. (This value can be a float) * `query` - PromQL Query to run #### ScaledObject `ScaledObject` is a CRD that KEDA uses as rule sets that define [scale strategies](https://keda.sh/docs/2.13/concepts/scaling-deployments/). Sample configuration for a `ScaledObject` which can be applied via `kubectl`. ```yaml # scaled-object.yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: template-deployment-traffic namespace: production spec: scaleTargetRef: name: rails-app pollingInterval: 30 # Must be seconds minReplicaCount: 2 maxReplicaCount: 3 triggers: - type: prometheus metadata: serverAddress: https://<$levitate_username>:<$levitate_password>@<$levitate_read_url> metricName: rails_requests_total threshold: "100000" query: sum(increase(rails_requests_total{kubernetes_namespace="production", app="web"}[4m])) ``` Let’s drill down the trigger configuration. #### type Last9 is a Prometheus-compatible telemetry data platform so, you must use the trigger with type `prometheus` in the YAML configuration to trigger scaling. #### serverAddress This is the URL of the metrics source which is the Last9 cluster’s READ URL where the metrics will be read from. #### metricName Name of the metric which will be used to evaluate the trigger condition. #### threshold The threshold for value of the query when the scaling will trigger in the form of creating a new pod. #### query This query is designed to check the number of requests served by the application. The threshold is 100K. When the request count reaches 100K, a new pod is created. Similarly, when the usage drops to 50K, a pod is deleted. To determine the value required to reduce the number of pods, KEDA waits till the result of the query becomes half of the threshold. ### Applying the ScaledObject Apply the configuration as follows. ```yaml kubectl apply -f ./scaled-object.yaml -n production --kubeconfig=$KUBECONFIG ``` This will setup the trigger for autoscaling the pods depending on real time request traffic from the Rails application. Tip You can read more about scaling triggers that KEDA supports [here](https://KEDA.sh/docs/2.13/concepts/scaling-deployments/#triggers) *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # LaunchDarkly > Send LaunchDarkly feature flag change events to Last9 and visualize them as annotations in Last9's Alert Details and managed Grafana Last9 Last9’s integration with LaunchDarkly enables change intelligence with feature flag events. Visualize the feature flag events as Change Event annotations in Last9 to correlate system health. ## Prerequisites To start sending feature flag events to Last9 Change Events, you’ll need: * [Last9 account](https://app.last9.io), and create a Last9 cluster by following [Getting Started](/docs/onboard/) * [Last9 API write token](https://app.last9.io/apiaccess) / [(docs)](/docs/getting-started-with-api/) * A [default Change Events](https://app.last9.io/change-events/) cluster / [(docs)](/docs/change-events/#change-events-storage) ## Set up the Last9 integration in the LaunchDarkly dashboard 1. Navigate to the [LaunchDarkly Integrations](https://app.launchdarkly.com/default/integrations?q=last9) page and find “Last9” 2. Click on “Add Integration” to view the “Create Last9 configuration” side panel 3. (Optional) Provide a Name to the integration that’s human-readable and easy to identify, especially if this instance of the integration is scoped to certain policy filters 4. Enter the Last9 API’s base URL. Do not include a trailing `/`. For the `https://{host}/api/v4/organizations/{org}` format, use `app.last9.io` as the `{host}` and you can use your organization slug from the Last9 dashboard URL (the `demo` in app.last9.io/v2/organizations/demo/apps) as the `{org}`  5. Enter the Last9 API write refresh token. If you have not saved the write token earlier while generating it, you can click on “Generate Tokens” on the [API Access](https://app.last9.io/apiaccess) page in the Last9 dashboard to get a new write refresh token  6. (Optional) Provide a Last9 tag. If provided, all feature flag events matching the policy filter (defined in the next step) will be only associated with Last9 entities with the same tag. This is particularly helpful if this instance of the integration is scoped to certain LaunchDarkly policy filters 7. (Optional) Configure a custom LaunchDarkly policy to control which events LaunchDarkly sends to Last9 8. Once you’ve read the [Integration Terms and Conditions](https://launchdarkly.com/policies/integrations/), click on the “I have read and agree to the Integration Terms and Conditions” checkbox 9. Click on “Save Configuration”. This new integration now appears on the Integrations page under Last9. It is switched on by default 10. Last9 will now start receiving relevant change events whenever a feature flag event is triggered 11. To verify if the integration is working as expected: 1. Click on the `...` icon next to the newly created integration and click on “Edit Integration Configuration” 2. Click on “Validate Connection”. This will trigger a test event from LaunchDarkly 3. To view the test event in Last9, visit [Grafana](https://app.last9.io/grafana/query) 4. Run a query against the `last9_change_events{}` metric to visualize it on the chart 5. You’ll see a data point on which you can hover to see the test event’s labels ## Visualize LaunchDarkly events as Annotations in Last9’s managed Grafana 1. Open any [Dashboard](https://app.last9.io/grafana/dashboards) → Settings → Annotations 2. Click on “Add Annotation Query” 3. Give it a name, eg: LaunchDarkly All Events 4. Select the “Data source” to be the same as the default cluster set in the Change Events settings 5. In “Query”, expand the “Metrics browser”, and select the `last9_change_events` metric 6. Optionally, you can select one more label and their values if you want to filter the visualized annotations. For eg, selecting `environment_key` and `event_state` labels followed by `production` and `start` values respectively. Only start annotations of the production environment will be visualized 7. Click on “Use query” to construct the final query 8. Give an appropriate name to be visualized while viewing the annotation. For eg, “Event on {{environment\_key}} started” will result in “Event on production started” 9. Click on “Apply” to save the configuration. You can also click on “Preview in dashboard” to save and preview 10. Do remember to click on “Save dashboard” to persist the new annotation ## Visualize LaunchDarkly events as Annotations in Last9’s Alert Details modal 1. Click on an alert to view the Alert Details modal * You can receive the alert as a notification in Slack or another tool of choice * You can click on an alert while viewing Alert Monitor * You can click on an alert while viewing an Alert Group’s Health tab 2. While viewing the Alert Details modal, click on an “Impacted Label” 3. If there are any associated Change Events: * You‘ll see “x events for selected label” * You‘ll see a purple annotation on the chart with a timestamp * Hovering on the annotation will let you view details of the change event Note: * While Change Events can be sent to any data source while using Last9’s Change Events API, in case of the LaunchDarkly integration, all feature flag events are sent ***only*** to the default cluster selected in the Change Events settings in the Last9 dashboard * Change Events are only visualized when an Alert Group’s data source is the same as well. In the case of the LaunchDarkly integration, the Alert Group’s data source should be the same as the default cluster selected in the Change Event settings *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # OpenTelemetry > Start sending metrics, logs, and trace data from your applications using OpenTelemetry for a correlated single pane. ## Send data using OpenTelemetry Last9 is fully [OpenTelemetry](https://opentelemetry.io/) compatible. OpenTelemetry is an open-source observability framework. It provides a standardized way to collect and export telemetry data — metrics, logs, and traces — from your applications and infrastructure. If you are instrumenting code for the first time, we recommend using OpenTelemetry. You can either run OpenTelemetry Collector or send the telemetry directly from applications using the SDKs. Last9 supports both gRPC and HTTP endpoints for OpenTelemetry. ## Using OTel Collector If your code is already instrumented using Prometheus, statsD, Influx, Jaeger, Zipkin, etc, please refer to [OpenTelemetry Collector documentation](/docs/integrations-opentelemetry-collector/). OTel Collector supports sending metrics, logs, and trace data irrespective of source. ## Using OTel SDKs [ Express.js ](/docs/integrations-opentelemetry-expressjs/)[ NestJS ](/docs/integrations-opentelemetry-nestjs/)[ Next.js ](/docs/integrations-opentelemetry-nextjs/)[ Koa ](/docs/integrations-opentelemetry-koa/)[ Django ](/docs/integrations-opentelemetry-django/)[ Flask ](/docs/integrations-opentelemetry-flask/)[ Kong ](/docs/integrations-opentelemetry-kong-gateway/)[ Phoenix Framework ](/docs/integrations-opentelemetry-phoenix/)[ Ruby on Rails ](/docs/integrations-opentelemetry-ruby-on-rails/)[ Sinatra ](/docs/integrations-opentelemetry-ruby-on-rails/)[ Roda ](/docs/integrations-opentelemetry-ruby-on-rails/)[ Fluent Bit ](/docs/integrations-opentelemetry-fluent-bit/)[ Ubuntu ](/docs/integrations-opentelemetry-ubuntu/)[ Datadog Agent ](/docs/integrations-opentelemetry-datadog-agent/)[ Ubuntu Host Metrics ](/docs/integrations-opentelemetry-ubuntu-host-metrics/)[ Gin ](/docs/integrations-opentelemetry-gin/)[ gRPC ](/docs/integrations-opentelemetry-grpc/)[ FastHTTP ](/docs/integrations-opentelemetry-fasthttp/)[ Iris ](/docs/integrations-opentelemetry-iris/)[ Gorilla Mux ](/docs/integrations-opentelemetry-gorilla-mux/)[ Logs from Kubernetes Cluster ](/docs/integrations-opentelemetry-kubernetes-logs/)[ Kubernetes Audit Logs ](/docs/integrations-opentelemetry-kubernetes-audit-logs/)[ AWS EC2 Instance ](/docs/integrations-opentelemetry-aws-ec2/)[ Logs from AWS S3 ](/docs/integrations-opentelemetry-aws-s3/)[ MariaDB](/docs/integrations-opentelemetry-mariadb/) ## Obtain OTLP Endpoint & Credentials 1. Visit the [Integrations](https://app.last9.io/integrations/) page 2. Select [OpenTelemetry](https://app.last9.io/integrations?integration=OpenTelemetry) 3. Copy the Endpoint URL and the Authorization Header  Last9’s OTLP compatible endpoint formats are as follows: #### HTTP ```yaml https://otlp.last9.io #for US-EAST-1 region https://otlp-aps1.last9.io #for AP-SOUTH-1 region ``` #### gRPC ```yaml otlp.last9.io:443 #for US-EAST-1 region otlp-aps1.last9.io:443 #for AP-SOUTH-1 region ``` Note The endpoints are region specific. Please refer to the [Integration](https://app.last9.io/integrations?integration=OpenTelemetry) page for your specific region. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # EC2 Instance > Send logs and hostmetrics from AWS EC2 instance using OpenTelemetry This guide will help you instrument your AWS EC2 instance with OpenTelemetry and smoothly send the logs and host metrics to a Last9. ## Pre-requisites 1. You have a AWS EC2 instance and workload running in it. 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` 3. Optional: Attach an IAM policy to the EC2 instance with `ec2:DescribeTags` permission. This is needed for resource detection processor to fetch additional tags associated with the EC2 instance which can be used as additional resource attributes. 4. Install Otel Collector. There are multiple ways to install the Otel Collector. One possible way of installing it using rpm is as follows. Every Collector release includes APK, DEB and RPM packaging for Linux amd64/arm64/i386 systems. > Note: systemd is required for automatic service configuration. ```sh sudo rpm -ivh otelcol-contrib_0.103.0_linux_amd64.rpm ``` More installation options can be found [here](https://opentelemetry.io/docs/collector/installation/#linux). > Note: We recommend installing `otel-collector-contrib` version `0.103.0`. ## Sample Otel Collector Configuration The default path for otel config is `/etc/otelcol-contrib/config.yaml`. You can edit it and update it with below configuration. The configuration is annotated with comments which should be addressed before applying the configuration. The configuration for operators is especially important to extract the `timestamp` and `severity`. For JSON logs, you can use `json_parser` and use its keys for log attributes. For non-structured logs, use the `regex_parser`. The configuration provdies sample example of both JSON parser and regex parsers. ```yaml receivers: hostmetrics: collection_interval: 30s scrapers: cpu: metrics: system.cpu.logical.count: enabled: true memory: metrics: system.memory.utilization: enabled: true system.memory.limit: enabled: true load: disk: filesystem: metrics: system.filesystem.utilization: enabled: true network: paging: processes: process: mute_process_user_error: true metrics: process.cpu.utilization: enabled: true process.memory.utilization: enabled: true process.threads: enabled: true process.paging.faults: enabled: true otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 # Detailed configuration options can be found at https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver filelog: # File path pattern to read logs from. Update this to the destination from where you want to read logs. include: [/tmp/*.log] exclude: [/home/ubuntu/exclude/*.log] include_file_path: true # attributes: # A map of key: value pairs to add to the entry's attributes. # resource: # A map of key: value pairs to add to the entry's resource. retry_on_failure: enabled: true operators: # For logs in JSON format - type: json_parser severity: parse_from: attributes.level timestamp: parse_from: attributes.time layout: "%Y-%m-%d %H:%M:%S" # For plain text logs - type: regex_parser regex: '(?P^[A-Za-z]+) (?P[0-9]{4}-[0-9]{2}-[0-9]{2}.*[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{1,3})?)' timestamp: parse_from: attributes.time layout: "%Y-%m-%d-%H:%M:%S" severity: parse_from: attributes.level processors: batch: timeout: 5s send_batch_size: 10000 send_batch_max_size: 10000 resourcedetection/ec2: detectors: ["ec2"] ec2: # A list of regex's to match tag keys to add as resource attributes can be specified tags: # This means you have a tag `Name` associated with the EC2 Instance. - ^Name$ # This means you have a tag `app` associated with the EC2 Instance. - ^app$ transform/ec2: error_mode: ignore log_statements: - context: resource statements: # Set Service name as the `Name` tag associated with the EC2 Instance. The format is `ec2.tag.`. - set(attributes["service.name"], attributes["ec2.tag.Name"]) resourcedetection/system: detectors: ["system"] system: hostname_sources: ["os"] transform/hostmetrics: metric_statements: - context: datapoint statements: - set(attributes["host.name"], resource.attributes["host.name"]) - set(attributes["process.command"], resource.attributes["process.command"]) - set(attributes["process.command_line"], resource.attributes["process.command_line"]) - set(attributes["process.executable.name"], resource.attributes["process.executable.name"]) - set(attributes["process.executable.path"], resource.attributes["process.executable.path"]) - set(attributes["process.owner"], resource.attributes["process.owner"]) - set(attributes["process.parent_pid"], resource.attributes["process.parent_pid"]) - set(attributes["process.pid"], resource.attributes["process.pid"]) exporters: debug: verbosity: detailed otlp/last9: endpoint: "" headers: "Authorization": "" service: pipelines: logs: receivers: [filelog] processors: [resourcedetection/ec2, transform/ec2, batch] exporters: [otlp/last9] metrics: receivers: [hostmetrics] processors: [resourcedetection/system, transform/hostmetrics, batch] exporters: [otlp/last9] ``` ## Running otel collector Run the otel collector using `systemctl` command. ```sh sudo systemctl start otelcol-contrib sudo systemctl status otelcol-contrib sudo systemctl restart otelcol-contrib ``` ## Checking logs of otel collector ```sh sudo journalctl -u otelcol-contrib -f ``` This will enable logs and host metrics to be sent to Last9. You can visit the [Logs Dashboard](https://app.last9.io/logs) and [Metrics Dashboard](https://app.last9.io/grafana) to see the data in action. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Logs from S3 Bucket > Ingest logs from AWS S3 bucket to Last9 This guide will walk through the steps required to grant necessary permissions to Last9 to read logs from an S3 bucket. Last9 can ingest logs from an AWS S3 bucket using evented architecture as follows.  This architecture ensures that * Last9 automatically receives notifications for new files being added or existing files being updated. * Last9 starts ingesting logs for the files in the S3 bucket that are created or updated after the integration is enabled. * For past data, Last9 can do one time ingestion of files in the S3 bucket based on filters similar to rehydration. ## Pre-requisites 1. You have an AWS S3 bucket. 2. You have signed up for [Last9](https://app.last9.io). ## Notify Last9 when new files are created in the S3 bucket * The S3 ingestion works on evented arhictecture using SQS queue. * Last9 team will share ARN of the SQS queue. * Enable event notification in the S3 bucket for new objects.  * Choose destination for event notification to be ARN of the Last9 SQS queue.  > Read more about S3 Event notification [here](https://aws.amazon.com/blogs/aws/s3-event-notification/). ## AWS STS Role Attach the following Trust relationship to a role with Last9 prinicpal user ARN. You can create the role to delegate permissions as per the [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html). ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "", "Service": "s3.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` Note Get the `` from Last9 team. ## S3 Access policy Attach the following IAM policy to the role created in above step so that Last9 ingestion pipeline can access S3 bucket contents. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": ["s3:GetObject", "s3:ListBucket"], "Resource": ["arn:aws:s3:::bucket_name", "arn:aws:s3:::bucket_name/*"] } ] } ``` Share the role ARN with the Last9 team and the S3 bucket ARN for seamless ingestion of logs. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # OpenTelemetry Collector > This document describes a sample setup for sending metrics to Last9 via OpenTelemetry Collector. ## Prerequisites 1. Create a Last9 cluster by following [Getting Started](/docs/onboard/) 2. Keep the following information handy after creating the cluster: * `$levitate_remote_write_url`: Last9’s Remote write endpoint * `$levitate_remote_write_username`: Cluster ID * `$levitate_remote_write_password`: Write token created for the cluster ## OpenTelemetry Collector Basics  OpenTelemetry collectors consist of three parts: 1. Receivers * A receiver, which can be push or pull-based sends data to the collector * In this doc, we will use Prometheus receiver, as an example, which will receive metrics 2. Processors * Processors are run on data between being received and being exported * Processors are not mandatory for sending data to Last9 3. Exporters * An exporter, which can be push or pull-based, sends data to one or more backends/destinations * We will use Last9 as an exporter destination ## Setup 1. Create a `docker-compose.yaml` file with the following config: Note that the image used for OpenTelemetry Collector is `otel/opentelemetry-collector-contrib:0.64.0` which has support for the Basic authentication extension. ```yaml version: "3.5" services: otel-collector: container_name: otel-collector-contrib image: otel/opentelemetry-collector-contrib:0.64.0 command: ["--config=/etc/otelcol-config.yaml"] ports: - "8888:8888" # Prometheus metrics exposed by the collector - "13133:13133" # health_check extension volumes: - ./otel-config.yaml:/etc/otelcol-config.yaml restart: always networks: - monitoring networks: monitoring: driver: bridge ``` 2. Create `otel-config.yaml` with a sample scrape config. Replace this scrape config by your desired config: ```plaintext extensions: basicauth/prw: client_auth: # Replace these variables by their actual values username: $levitate_remote_write_username password: $levitate_remote_write_password receivers: prometheus: config: scrape_configs: - job_name: 'otel-collector-01' scrape_interval: 60s static_configs: - targets: ['otel-collector:8888'] exporters: prometheusremotewrite: auth: authenticator: basicauth/prw endpoint: "$levitate_remote_write_url" # Replace this variable by its actual value service: extensions: [basicauth/prw] pipelines: metrics: receivers: [prometheus] exporters: [prometheusremotewrite] ``` 3. Deploy the setup using: ```plaintext docker-compose up ``` As per the config, the entire flow is combined in the `pipelines` section under the `service` where the data flows from OpenTelemetry Collector to Last9 using the basic auth mechanism. Note The same config file `otel-config.yaml` can be used when using another compute layer e.g. K8s. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Datadog Agent > This document describes a sample setup for sending metrics and traces to Last9 from Datadog Agent [Datadog Agent](https://docs.datadoghq.com/agent/) collects events and metrics from hosts and sends them to Datadog, where you can analyze your monitoring and performance data. The Datadog Agent is open source, and its source code is available on GitHub at DataDog/datadog-agent. ## Prerequisites 1. Create a Last9 cluster by following [Getting Started](/docs/onboard/) guide. 2. Keep the following information handy after creating the cluster from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `$last9_otlp_endpoint`: Last9’s OTLP endpoint * `$last9_basic_auth_header`: OTLP Basic authorization header * `$last9_username`: OTLP username * `$last9_password`: OTLP password 3. You have access to the Datadog Agent configuration. ## Configuration Last9 leverages [“Dual shipping”](https://docs.datadoghq.com/agent/configuration/dual-shipping/) capability of Datadog Agent to ship telemetry data to Last9 in parallel to Datadog. This requires the following changes in the Datadog agent configuration. Set the Last9 ingestion endpoint as the following. ```yml ######################### ## Basic Configuration ## ######################### api_key: # For Metrics additional_endpoints: "https://{$last9_username}:{$last9_password}@{$last9_otlp_endpoint}/datadog": - "" # For Traces apm_config: enabled: true additional_endpoints: "https://{$last9_username}:{$last9_password}@{$last9_otlp_endpoint}": - ``` Alternatively, you can also configure the Datadog agent to ship metrics and traces by setting the following environment variables. ```shell DD_ADDITIONAL_ENDPOINTS=https://{$last9_username}:{$last9_password}@{$last9_otlp_endpoint/datadog DD_APM_ADDITIONAL_ENDPOINT=https://{$last9_username}:{$last9_password}@{$last9_otlp_endpoint} ``` Restarting the Datadog agent will start shipping traces and metrics to Last9. You will see the following in the Datadog agent logs. ```shell 2024-10-09 09:19:39 IST | CORE | INFO | (comp/forwarder/defaultforwarder/default_forwarder.go:391 in Start) | Forwarder started, sending to 1 endpoint(s) with 1 worker(s) each: "https://***:********@otlp.last9.io/datadog" (2 api key(s)) ``` *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Sending OpenTelemetry Demo data to Last9 > Learn how to connect the OpenTelemetry demo microservices application with Last9 to gain immediate insights through traces, metrics, and logs visualization. The OpenTelemetry Demo is a fully instrumented microservices application that simulates an e-commerce platform. By connecting this demo to Last9, you’ll experience how comprehensive observability works across distributed systems without modifying application code. This integration takes approximately 5-10 minutes to complete and requires basic familiarity with command line operations. ## Prerequisites 1. **Last9 Account**: [Create an account](https://app.last9.io/) to access the observability platform 2. **Last9 Cluster**: Set up your monitoring environment by following the [Getting Started guide](/docs/onboard/) 3. **OpenTelemetry Credentials**: Retrieve your unique authorization details from the [Integrations page](https://app.last9.io/integrations?integration=OpenTelemetry) (you’ll need these for the collector configuration) **System Requirements**: Docker (or Kubernetes), 4GB+ available RAM, and Git installed on your local system. ## Steps 1. ### Clone the OpenTelemetry Demo Repository First, clone the [OpenTelemetry demo repository](https://github.com/open-telemetry/opentelemetry-demo), which contains a pre-instrumented e-commerce application with 12+ microservices that generate traces, metrics, and logs. ```bash git clone https://github.com/open-telemetry/opentelemetry-demo.git cd opentelemetry-demo ``` This demo simulates a complete shopping application with frontend, product catalog, cart, checkout, shipping, and payment services—all generating realistic telemetry data. 2. ### Configure OpenTelemetry Collector with Last9 Integration Now, configure the OpenTelemetry Collector to forward telemetry data to your Last9 account. Create or edit the file `src/otel-collector/otel-config-extras.yml` with the following configuration. Replace `` and `` with the credentials you obtained from the Last9 Integrations page: ```yaml exporters: otlp/last9: endpoint: "" headers: "Authorization": "Basic " service: pipelines: traces: exporters: [spanmetrics, otlp/last9] metrics: exporters: [otlp/last9] logs: exporters: [otlp/last9] ``` **What this does:** The collector automatically combines this configuration with the default one in `otel-config.yml`. Your configuration adds Last9 as an export destination while preserving the demo’s original functionality. The three pipeline sections ensure all telemetry types (traces, metrics, and logs) are sent to Last9. 3. ### Launch the Demo Application The demo application runs a complete microservices architecture with a web frontend, multiple backend services, and databases—all pre-instrumented with OpenTelemetry. * **Using Docker** (recommended for first-time setup) ```bash docker compose up --force-recreate --remove-orphans --detach ``` This command starts all services locally. The first run will take 5-10 minutes as it downloads and builds containers. Subsequent starts will be faster. Resource usage: This demo runs 15+ containers and requires approximately 4GB of RAM. * **Using Kubernetes**: For production-like deployments, refer the [OTel Demo K8s deployment guide](https://opentelemetry.io/docs/demo/kubernetes-deployment/) and follow the configuration in the [Last9 Kubernetes Integration guide](https://app.last9.io/integrations?integration=Last9+Otel+Collector+Setup+for+Kubernetes) **How to verify:** After startup completes, you should see all containers running with docker ps. The frontend will be available at . 4. ### Verify Data Flow in Last9 * Access the demo application at to see the frontend * Wait approximately 2-3 minutes for the first data to appear in Last9 (the demo’s built-in load generator automatically creates traffic) * Open [Traces Explorer](https://app.last9.io/traces/) in Last9 and look for: * Service names like `frontend`, `productcatalogservice`, and `checkoutservice` * Operation names such as `Get /product` or `Cart/AddItem` **Troubleshooting:** If no data appears after 5 minutes, check your authorization credentials and verify all containers are running with `docker ps`. 5. ### Explore the Demo in Last9 Now that your demo is sending data to Last9, explore these key observability features: * [Traces Explorer](https://app.last9.io/traces/): View end-to-end request flows across services * Try filtering for frontend service to see customer-facing requests * Look for traces with errors to see how failures propagate through services * [Logs Explorer](https://app.last9.io/logs/): Search and analyze application logs * Filter by service name to see logs from specific components * Search for “error” to identify issues without browsing traces * [Dashboards](https://app.last9.io/dashboards/): Monitor key performance metrics * Create a dashboard showing request rates and latencies for critical services * Set up alerts for abnormal patterns in error rates or response times * [API Catalog](https://app.last9.io/api-catalog/): Track API performance * Identify which endpoints experience the highest latency * Monitor error rates across different API versions **Experiment:** Try adding items to the cart in the demo app, then search for your session in the Traces Explorer to see your actual user journey. **Next Steps:** When you’re ready to integrate your own applications, refer to the [Last9 OpenTelemetry Integration Docs](/docs/integrations-opentelemetry/) for complete instructions. *** ## Troubleshooting ### Common Issues * No data appearing in Last9: * Verify your Authorization Header is correctly copied from the Integrations page * Check that all containers are running with `docker ps | grep otel` * Ensure your network allows outbound HTTPS connections to otlp.last9.io * Demo application not loading: * Verify all containers are running with `docker ps` * Check container logs with `docker logs otelcol` * Ensure you have sufficient system resources (at least 4GB free RAM) * Partial data appearing: * Some services may take longer to initialize and start sending telemetry * Wait 5-10 minutes for all services to fully report data ### Getting Help If you encounter issues not covered above, please contact our support team with: * The specific error messages you’re seeing * Output from docker ps showing running containers * The collector logs from docker logs otelcol Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Django > Send traces to Last9 from a Django app using OpenTelemetry Django is a high-level Python web framework that simplifies the development of secure and maintainable web applications. This comprehensive guide will help you instrument your Django application with OpenTelemetry and smoothly send the traces to a Last9 cluster. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/python/django). ## Pre-requisites 1. You have a Django application 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` ## Install OpenTelemetry packages To install the required packages, run the following command: ```bash pip install python-dotenv opentelemetry-distro opentelemetry-exporter-otlp ``` After, installing the required packages, run the following command to install the required instrumentation packages: ```bash opentelemetry-bootstrap -a install ``` Tip To know more about them, you can check: ## Set the environment variables Now, that you’ve installed all the packages. Run the following command to initialize environment variables: ```bash export OTEL_SERVICE_NAME=django-app export OTEL_EXPORTER_OTLP_ENDPOINT= export OTEL_EXPORTER_OTLP_HEADERS="Authorization=" export OTEL_TRACES_EXPORTER=otlp export OTEL_METRICS_EXPORTER=otlp ``` > Note: Replace `` with the URL encoded value of the basic auth header. ## Run the application After setting the environment variables, run the Django application: ```bash opentelemetry-instrument uwsgi --http 8000 --ini uwsgi.ini ``` ## Visualize data After running the Django app, you can visualize the traces in the Last9’s APM dashboard: [Link](https://app.last9.io/traces) *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Express.js > Send distributed traces to Last9 from an Express.js app using OpenTelemetry Express.js is widely recognized as the most popular web framework for Node.js. This comprehensive guide will help you instrument your Express.js application with OpenTelemetry, seamlessly sending the traces to Last9. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/javascript/express). ## Pre-requisites 1. You have an Express.js application 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` ## Install OpenTelemetry packages To perform the instrumentation, install the following OpenTelemetry packages. ```bash npm install --save @opentelemetry/api \ @opentelemetry/instrumentation \ @opentelemetry/tracing \ @opentelemetry/exporter-trace-otlp-http \ @opentelemetry/resources \ @opentelemetry/semantic-conventions \ @opentelemetry/auto-instrumentations-node \ @opentelemetry/sdk-node ``` To know more about them, you can check: ## Setup auto-instrumentation using OpenTelemetry Add the necessary environment variables that can be obtained from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page. ```bash OTLP_ENDPOINT= OTLP_AUTH_HEADER="Basic " ```  Next, create a file named `instrumentation.ts` and add the following code: ```ts import { NodeTracerProvider, TracerConfig, } from "@opentelemetry/sdk-trace-node"; import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"; // Use BatchSpanProcessor for better performance import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; // Import OTLPTraceExporter import { registerInstrumentations } from "@opentelemetry/instrumentation"; import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node"; import { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_DEPLOYMENT_ENVIRONMENT, } from "@opentelemetry/semantic-conventions"; import { Resource } from "@opentelemetry/resources"; const providerConfig: TracerConfig = { resource: new Resource({ [SEMRESATTRS_SERVICE_NAME]: "", [SEMRESATTRS_DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV, }), }; // Initialize and register the tracer provider const provider = new NodeTracerProvider(providerConfig); const otlp = new OTLPTraceExporter({ url: process.env.OTLP_ENDPOINT, headers: { Authorization: process.env.OTLP_AUTH_HEADER, }, }); // Configure the OTLP exporter provider.addSpanProcessor(new BatchSpanProcessor(otlp)); provider.register(); // Automatically instrument HTTP and Express (additional instrumentations can be added) registerInstrumentations({ instrumentations: [ getNodeAutoInstrumentations({ // instrumentation-fs is disabled to reduce the noise of spans related to file operations "@opentelemetry/instrumentation-fs": { enabled: false, }, }), ], }); ``` This above code performs the following steps: 1. Set up Trace Provider with the application’s name as Service Name. 2. Set up OTLP Exporter with Last9 OTLP endpoint. 3. Set up auto instrumentation. Next, this script must be imported at the application’s **entry point**. For instance, if you have `server.ts` as your entry file, then import it at the top of the file as follows: ```ts // server.ts import "./instrumentation.ts"; // Application Code... ``` Tip You can also use `instrumentation.js` if you are not using Typescript. ## Visualize the traces in Last9 Once the application runs with the above code, it will start pushing traces to Last9. You can see the result in action by looking at the APM dashboard in Last9.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # fasthttp > Send distributed traces and metrics to Last9 from Golang fasthttp application using OpenTelemetry [fasthttp](https://github.com/valyala/fasthttp) is a fast, flexible HTTP server framework written in Go (Golang). This comprehensive guide will help you instrument your fasthttp application with OpenTelemetry and smoothly send the traces to a Last9 cluster. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/go/fasthttp). ## Pre-requisites 1. You have a fasthttp application. 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` ## Install OpenTelemetry packages To install the required packages, run the following command: ```bash go get -u go.opentelemetry.io/otel go get -u go.opentelemetry.io/otel/sdk go get -u go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc go get -u go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc go get -u go.opentelemetry.io/otel/sdk/metric go get -u go.nhat.io/otelsql ``` For setting up Redis instrumentation, first verify which `go-redis` version you are using. If you are using `go-redis` v8 then ```bash go get -u github.com/go-redis/redis/extra/redisotel ``` If you are using `go-redis` v9 then ```bash go get -u github.com/redis/go-redis/extra/redisotel ``` Using these packages, you can instrument your fasthttp application to send traces to Last9. ### Traces This application generates traces for the following: * HTTP requests using [last9/otelMiddleware](https://github.com/last9/opentelemetry-examples/blob/master/go/fasthttp/last9/otelMiddleware.go) * Database queries using [otelsql](https://github.com/nhatthm/otelsql) * Redis commands using [redisotel](https://github.com/redis/redis-go-cluster/tree/main/redisotel) For HTTP requests, wrap the fasthttp router with the `last9.otelMiddleware` middleware. > Refer to [main.go](https://github.com/last9/opentelemetry-examples/blob/master/go/fasthttp/main.go) for more details. For database queries, use the `otelsql` package to wrap the `sql.DB` object. > Refer to `initDB()` in [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/master/go/fasthttp/users/controller.go) for more details. For Redis commands, use the `redisotel` package to wrap the `redis.Client` object. > Refer to `initRedis()` in [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/master/go/fasthttp/users/controller.go) for more details. ## Set the environment variables Set the following environment variables: ```bash export OTEL_SERVICE_NAME=fasthttp-app-service export OTEL_EXPORTER_OTLP_ENDPOINT= export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic " ``` These environment variables are used to configure the OpenTelemetry SDK to send traces and metrics to Last9. ## Instrument HTTP requests Setup instrumentation for fasthttp application using [last9/instrumentation](https://github.com/last9/opentelemetry-examples/blob/master/go/fasthttp/last9/instrumentation.go). ```go package last9 import ( "context" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/propagation" "go.opentelemetry.io/otel/sdk/resource" sdktrace "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.26.0" "go.opentelemetry.io/otel/trace" ) type Instrumentation struct { TracerProvider *sdktrace.TracerProvider Tracer trace.Tracer } func initTracerProvider() *sdktrace.TracerProvider { exporter, err := otlptracehttp.New(context.Background()) if err != nil { panic(err) } attr := resource.WithAttributes( semconv.DeploymentEnvironmentKey.String("production"), // replace with actual environment semconv.ServiceNameKey.String("fasthttp-server"), // replace with service name ) resources, err := resource.New(context.Background(), resource.WithFromEnv(), resource.WithTelemetrySDK(), resource.WithProcess(), resource.WithOS(), resource.WithContainer(), resource.WithHost(), attr) if err != nil { panic(err) } tp := sdktrace.NewTracerProvider( sdktrace.WithBatcher(exporter), sdktrace.WithResource(resources), ) otel.SetTracerProvider(tp) otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})) return tp } func NewInstrumentation() *Instrumentation { tp := initTracerProvider() return &Instrumentation{ TracerProvider: tp, Tracer: tp.Tracer("fasthttp-server"), } } ``` The above code configures the OpenTelemetry SDK to use the OTLP exporter and initializes the TracerProvider. Next, at the entry point of your application, add the following code to instrument your application: ```go // main.go func main() { i := NewInstrumentation() defer func() { if err := i.TracerProvider.Shutdown(context.Background()); err != nil { log.Printf("Error shutting down tracer provider: %v", err) } }() // other code } ``` Now, add the otel fasthttp middleware to your application: ```go // main.go func main() { i := last9.NewInstrumentation() defer func() { if err := i.TracerProvider.Shutdown(context.Background()); err != nil { log.Printf("Error shutting down tracer provider: %v", err) } }() // other code // Routes r.GET("/users", h.GetUsers) r.GET("/users/{id}", h.GetUser) r.POST("/users", h.CreateUser) r.PUT("/users/{id}", h.UpdateUser) r.DELETE("/users/{id}", h.DeleteUser) r.GET("/joke", func(ctx *fasthttp.RequestCtx) { getRandomJoke(ctx, i) }) handler := last9.OtelMiddleware("fasthttp-server") // replace with name of the service // other code } ``` ## Instrument database operations ### Instrumenting with `sql.DB` Add the following code to instrument the database queries. It uses the `otelsql` package to wrap the `sql.DB` object and emit traces and metrics for database queries and connections. Refer to [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/master/go/fasthttp/users/controller.go) for more details. ```go func initDB() (*sql.DB, error) { driverName, err := otelsql.Register("postgres", // Read more about the options here: https://github.com/nhatthm/otelsql?tab=readme-ov-file#options otelsql.AllowRoot(), otelsql.TraceQueryWithoutArgs(), otelsql.TraceRowsClose(), otelsql.TraceRowsAffected(), otelsql.WithDatabaseName("otel_demo"), // database name otelsql.WithSystem(semconv.DBSystemPostgreSQL), ) if err != nil { return nil, fmt.Errorf("failed to register driver: %v", err) } db, err := sql.Open(driverName, dsnName) if err != nil { return nil, fmt.Errorf("failed to connect to database: %v", err) } // Record stats to expose metrics if err := otelsql.RecordStats(db); err != nil { return nil, err } return db, nil } ``` ### Instrumenting with `pgx` For database instrumentation where [pgx](https://github.com/jackc/pgx) is used, we use [otelpgx](https://github.com/exaring/otelpgx) to wrap the pgx connection pool. Add the following code to instrument the database queries. ```go var connString = os.Getenv("DATABASE_URL") cfg, err := pgxpool.ParseConfig(connString) if err != nil { fmt.Fprintf(os.Stderr, "create connection pool: %w", err) os.Exit(1) } // Add the tracer to the connection pool configuration cfg.ConnConfig.Tracer = otelpgx.NewTracer() // Create a new connection pool with the tracer conn, err = pgxpool.NewWithConfig(context.Background(), cfg) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connection to database: %v\n", err) os.Exit(1) } ``` > Refer to the complete example using pgx adapter and Otel instrumentation [here](https://github.com/last9/opentelemetry-examples/tree/master/go/pgx). ## Instrument Redis operations Tip Note that between `redis-go` versions v8 and v9 the import paths are changed. Add the following code to instrument the Redis operations. It uses the `redisotel` package to wrap the `redis.Client` object and emit traces for Redis commands. ### go-redis v9 If you are using `go-redis` v9 then use the following code. > Refer to [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/master/go/fasthttp/users/controller.go) for more details. ```go func initRedis() *redis.Client { rdb := redis.NewClient(&redis.Options{ Addr: "localhost:6379", // Update this with your Redis server address }) // Setup traces for redis instrumentation if err := redisotel.InstrumentTracing(rdb); err != nil { log.Fatalf("failed to instrument traces for Redis client: %v", err) return nil } return rdb } ``` ### go-redis v8 If you are using `go-redis` v8 then use the following code. ```go func initRedis() *redis.Client { rdb := redis.NewClient(&redis.Options{ Addr: "localhost:6379", // Update this with your Redis server address }) // Setup traces for redis instrumentation rdb.AddHook(redisotel.NewTracingHook()) return rdb } ``` > Refer to [main.go](https://github.com/last9/opentelemetry-examples/blob/master/go/fasthttp/users/main.go) for more details. ## Run the application Start your fasthttp application by running the following command: ```bash go run main.go ``` This will start the fasthttp application and the OpenTelemetry SDK will automatically collect traces and metrics from the fasthttp application. These traces will be sent to Last9 automatically based on the environment variables set. ## View traces in Last9 After running the fasthttp app, you can visualize the traces in Last9’s APM dashboard. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Flask > Send traces to Last9 from a Flask app using OpenTelemetry Flask is a lightweight WSGI web application framework. This comprehensive guide will help you instrument your Flask application with OpenTelemetry and smoothly send the traces to a Last9 cluster. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/python/flask). ## Pre-requisites 1. You have a Flask application 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` ## Install OpenTelemetry packages To install the required packages, run the following command: ```bash pip install python-dotenv opentelemetry-distro opentelemetry-exporter-otlp ``` After, installing the required packages, run the following command to install the required instrumentation packages: ```bash opentelemetry-bootstrap -a install ``` Tip To know more about them, you can check: ## Set the environment variables Now, that you’ve installed all the packages. Run the following command to initialize environment variables: ```bash export OTEL_SERVICE_NAME=flask-app export OTEL_EXPORTER_OTLP_ENDPOINT= export OTEL_EXPORTER_OTLP_HEADERS="Authorization=" export OTEL_TRACES_EXPORTER=otlp export OTEL_METRICS_EXPORTER=otlp ``` > Note: Replace `` with the URL encoded value of the basic auth header. ## Run the application After setting the environment variables, run the Flask application: ```bash opentelemetry-instrument flask run ``` ## Visualize data After running the Flask app, you can visualize the traces in the Last9’s APM dashboard.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Fluent Bit > This document describes a sample setup for sending logs to Last9 via Fluent bit [Fluent Bit](https://docs.fluentbit.io/manual) is a fast and lightweight telemetry agent for logs, metrics, and traces. This document provides step by step instructions to send logs from Fluent Bit to Last9. ## Prerequisites 1. Create a Last9 cluster by following [Getting Started](/docs/onboard/) guide. 2. Keep the following information handy after creating the cluster from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `$last9_otlp_endpoint`: Last9’s OTLP endpoint * `$last9_basic_auth_header`: OpenTelemetry Basic authorization header * `$last9_username`: OpenTelemetry username * `$last9_password`: OpenTelemetry password ## Fluent Bit Configuration Fluent bit supports shipping logs to Last9 using the `opentelemetry` output plugin. Add following output stanza in the Fluent Bit configuration file. ```plaintext [OUTPUT] Name opentelemetry Match kube.* Host otlp.last9.io http_user http_passwd Port 443 Logs_uri /v1/logs Tls On Tls.verify On logs_body_key_attributes true ``` Read more on the configuration options provided by the `opentelemetry` plugin [here](https://docs.fluentbit.io/manual/pipeline/outputs/opentelemetry). Note Please note that the `Match` field must match the value of Match field in the `Input` or `Filter` sections in the Fluent Bit configuration. This is the bare minimum configuration that you can add in the Fluent Bit config to start sending logs to Last9. ## Advanced Configuration You can add advanced lua functions to parse the logs before sending to Last9. ```lua function set_fields(tag, timestamp, record) local new_record = {} local function flatten_table(prefix, t) for k, v in pairs(t) do if type(v) == "table" then flatten_table(prefix .. k .. ".", v) else new_record[prefix .. k] = v end end end flatten_table("", record) return 1, timestamp, new_record end ``` As an example, above lua function flattens the nested maps present in logs. Additionally, it is highly recommended to add a unique tag to the log records per cluster or environment. It can be done by adding a modify Filter. ```plaintext [FILTER] Name modify Match kube.* Add env production ``` Read more on Fluent bit modify filter [here](https://docs.fluentbit.io/manual/pipeline/filters/modify). ## Verification Login to Last9 and visit the [Logs](https://app.last9.io/logs) panel.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Gin > Send distributed traces and metrics to Last9 from Golang Gin application using OpenTelemetry Gin is a web framework written in Go (Golang). This comprehensive guide will help you instrument your Gin application with OpenTelemetry and smoothly send the traces to Last9. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/go/gin). ## Pre-requisites 1. You have a Gin application. 2. You have signed up for [Last9](https://app.last9.io), and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` ## Install OpenTelemetry packages To install the required packages, run the following command: ```bash go get -u go.opentelemetry.io/otel go get -u go.opentelemetry.io/otel/sdk go get -u go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc go get -u go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin go get -u go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc go get -u go.opentelemetry.io/otel/sdk/metric go get -u go.nhat.io/otelsql ``` For setting up Redis instrumentation, first verify which `go-redis` version you are using. If you are using `go-redis` v8 then ```bash go get -u github.com/go-redis/redis/extra/redisotel/v8 ``` If you are using `go-redis` v9 then ```bash go get github.com/redis/go-redis/extra/redisotel/v9 ``` Using these packages, you can instrument your Gin application to send traces and metrics to Last9. ### Traces This application generates traces for the following: * HTTP requests using [otelgin](https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/github.com/gin-gonic/gin/otelgin) * Database queries using [otelsql](https://github.com/nhatthm/otelsql) * Redis commands using [redisotel](https://github.com/redis/redis-go-cluster/tree/main/redisotel) For HTTP requests, wrap the Gin router with the `otelgin.Middleware` middleware. > Refer to [main.go](https://github.com/last9/opentelemetry-examples/blob/main/go/gin/main.go) for more details. For database queries, use the `otelsql` package to wrap the `sql.DB` object. > Refer to `initDB()` in [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/main/go/gin/users/controller.go) for more details. For Redis commands, use the `redisotel` package to wrap the `redis.Client` object. > Refer to `initRedis()` in [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/main/go/gin/users/controller.go) for more details. ### Metrics The application also generates metrics for the following: * Database queries using [otelsql](https://github.com/nhatthm/otelsql) ## Set the environment variables Set the following environment variables: ```bash export OTEL_SERVICE_NAME=gin-app-service export OTEL_EXPORTER_OTLP_ENDPOINT= export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic " ``` These environment variables are used to configure the OpenTelemetry SDK to send traces and metrics to Last9. ## Instrument HTTP requests In your Gin application, create a new file `instrumentation.go` and add the following code: Refer to [main.go](https://github.com/last9/opentelemetry-examples/blob/main/go/gin/main.go) for more details. ```go package main import ( "context" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" "go.opentelemetry.io/otel/propagation" sdktrace "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.26.0" "go.opentelemetry.io/otel/trace" ) type Instrumentation struct { TracerProvider *sdktrace.TracerProvider Tracer trace.Tracer } func initTracerProvider() *sdktrace.TracerProvider { exporter, err := otlptracegrpc.New(context.Background()) if err != nil { panic(err) } attr := resource.WithAttributes( semconv.DeploymentEnvironmentKey.String("production"), // You can change this value to "development" or "staging" or you can get the value from the environment variables semconv.ServiceNameKey.String("gin-server"), // You can change this value to the name of your service // You can add more attributes here ) resources, err := resource.New(context.Background(), resource.WithFromEnv(), resource.WithTelemetrySDK(), resource.WithProcess(), resource.WithOS(), resource.WithContainer(), resource.WithHost(), attr) if err != nil { panic(err) } tp := sdktrace.NewTracerProvider( sdktrace.WithBatcher(exporter), sdktrace.WithResource(resources), ) otel.SetTracerProvider(tp) otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})) return tp } func NewInstrumentation() *Instrumentation { tp := initTracerProvider() return &Instrumentation{ TracerProvider: tp, Tracer: tp.Tracer("gin-server"), } } ``` The above code configures the OpenTelemetry SDK to use the OTLP exporter and initializes the TracerProvider. Next, at the entry point of your application, add the following code to instrument your application: Refer to [main.go](https://github.com/last9/opentelemetry-examples/blob/main/go/gin/main.go) for more details. ```go i := NewInstrumentation() defer func() { if err := i.TracerProvider.Shutdown(context.Background()); err != nil { log.Printf("Error shutting down tracer provider: %v", err) } }() ``` Now, add the otel gin middleware to your application: ```go import ( // other imports "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin" ) // other code r := gin.Default() // Make sure this is added before any routes are defined r.Use(otelgin.Middleware("gin-server")) // other code ``` > Refer to [main.go](https://github.com/last9/opentelemetry-examples/blob/main/go/gin/main.go) for more details. Use the `tracer` object to create spans in your application as follows with custom attributes such as user id: > Refer to [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/main/go/gin/users/controller.go) for more details. ```go _, span := u.tracer.Start(c.Request.Context(), "UpdateUser", oteltrace.WithAttributes( attribute.String("user.id", c.Param("id")), )) defer span.End() ``` ## Instrument database operations ### Instrumenting with `sql.DB` Add the following code to instrument the database queries. It uses the `otelsql` package to wrap the `sql.DB` object and emit traces and metrics for database queries and connections. Refer to [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/main/go/gin/users/controller.go) for more details. ```go func initDB() (*sql.DB, error) { driverName, err := otelsql.Register("postgres", // Read more about the options here: https://github.com/nhatthm/otelsql?tab=readme-ov-file#options otelsql.AllowRoot(), otelsql.TraceQueryWithoutArgs(), otelsql.TraceRowsClose(), otelsql.TraceRowsAffected(), otelsql.WithDatabaseName("otel_demo"), // database name otelsql.WithSystem(semconv.DBSystemPostgreSQL), ) if err != nil { return nil, fmt.Errorf("failed to register driver: %v", err) } db, err := sql.Open(driverName, dsnName) if err != nil { return nil, fmt.Errorf("failed to connect to database: %v", err) } // Record stats to expose metrics if err := otelsql.RecordStats(db); err != nil { return nil, err } return db, nil } ``` ### Instrumenting with `pgx` For database instrumentation where [pgx](https://github.com/jackc/pgx) is used, we use [otelpgx](https://github.com/exaring/otelpgx) to wrap the pgx connection pool. Add the following code to instrument the database queries. ```go var connString = os.Getenv("DATABASE_URL") cfg, err := pgxpool.ParseConfig(connString) if err != nil { fmt.Fprintf(os.Stderr, "create connection pool: %w", err) os.Exit(1) } // Add the tracer to the connection pool configuration cfg.ConnConfig.Tracer = otelpgx.NewTracer() // Create a new connection pool with the tracer conn, err = pgxpool.NewWithConfig(context.Background(), cfg) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connection to database: %v\n", err) os.Exit(1) } ``` > Refer to the complete example using pgx adapter and Otel instrumentation [here](https://github.com/last9/opentelemetry-examples/tree/master/go/pgx). ## Instrument Redis operations Tip Note that between `redis-go` versions v8 and v9 the import paths are changed. Add the following code to instrument the Redis operations. It uses the `redisotel` package to wrap the `redis.Client` object and emit traces for Redis commands. ### go-redis v9 If you are using `go-redis` v9 then use the following code. > Refer to [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/main/go/gin/users/controller.go) for more details. ```go func initRedis() *redis.Client { rdb := redis.NewClient(&redis.Options{ Addr: "localhost:6379", // Update this with your Redis server address }) // Setup traces for redis instrumentation if err := redisotel.InstrumentTracing(rdb); err != nil { log.Fatalf("failed to instrument traces for Redis client: %v", err) return nil } return rdb } ``` ### go-redis v8 If you are using `go-redis` v8 then use the following code. ```go func initRedis() *redis.Client { rdb := redis.NewClient(&redis.Options{ Addr: "localhost:6379", // Update this with your Redis server address }) // Setup traces for redis instrumentation rdb.AddHook(redisotel.NewTracingHook()) return rdb } ``` > Refer to [main.go](https://github.com/last9/opentelemetry-examples/blob/main/go/gin/main.go) for more details. ## Run the application Start your Gin application by running the following command: ```bash go run main.go ``` This will start the Gin application and the OpenTelemetry SDK will automatically collect traces and metrics from the Gin application. These traces and metrics will be sent to Last9 automatically based on the environment variables set. ## View traces and metrics in Last9 After running the Gin app, you can visualize the traces and metrics in Last9’s APM dashboard. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Gorilla Mux > Send distributed traces and metrics to Last9 from Golang Gorilla Mux application using OpenTelemetry [Gorilla Mux](https://github.com/gorilla/mux) is a powerful HTTP router and dispatcher for Go. This comprehensive guide will help you instrument your Gorilla Mux application with OpenTelemetry and smoothly send the traces to Last9. Tip You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/go/gorilla-mux). ## Pre-requisites 1. You have a Gorilla Mux application. 2. You have signed up for [Last9](https://app.last9.io), and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` ## Install OpenTelemetry packages To install the required packages, run the following command: ```bash go get -u go.opentelemetry.io/otel go get -u go.opentelemetry.io/otel/sdk go get -u go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc go get -u go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux go get -u go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc go get -u go.opentelemetry.io/otel/sdk/metric ``` ### Redis instrumentation For setting up Redis instrumentation, first verify which `go-redis` version you are using. If you are using `go-redis` v8 then ```bash go get -u github.com/go-redis/redis/extra/redisotel/v8 ``` If you are using `go-redis` v9 then ```bash go get github.com/redis/go-redis/extra/redisotel/v9 ``` ### Database instrumentation If you are using `sql.DB` then use [otelsql](https://github.com/nhatthm/otelsql) to instrument the database queries. ```shell go get -u go.nhat.io/otelsql ``` If you are using `pgx` then use [otelpgx](https://github.com/exaring/otelpgx) to instrument the database queries. ```shell go get -u github.com/exaring/otelpgx ``` ### Traces This application generates traces for the following: * HTTP requests using [otelmux](https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/github.com/gorilla/mux/otelmux) * Database queries using [otelsql](https://github.com/nhatthm/otelsql) * redis operations using [redisotel](https://github.com/redis/go-redis/tree/master/extra/redisotel) ### Metrics The application also generates metrics for the following: * Database queries using [otelsql](https://github.com/nhatthm/otelsql) ## Environment variables Set the following environment variables to send traces and metrics to Last9: ```bash export OTEL_SERVICE_NAME=gorilla-mux-app-service // Change this value to the name of your service export OTEL_EXPORTER_OTLP_ENDPOINT= // Last9 OTLP endpoint export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic " // Last9 OTLP auth header export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=,component=api // Add more attributes as needed as comma separated key=value pairs. ``` ## Instrument HTTP requests In your Gorilla Mux application, create a new file `last9/instrumentation.go` and add the following code: ```go package last9 import ( "context" "fmt" "time" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/propagation" "go.opentelemetry.io/otel/sdk/metric" "go.opentelemetry.io/otel/sdk/resource" sdktrace "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.26.0" "go.opentelemetry.io/otel/trace" ) type Instrumentation struct { TracerProvider *sdktrace.TracerProvider Tracer trace.Tracer } func initMetrics() (*metric.MeterProvider, error) { exporter, err := otlpmetricgrpc.New(context.Background()) if err != nil { fmt.Println("Error creating metrics exporter:", err) return nil, err } resources, err := resource.New(context.Background(), resource.WithFromEnv(), resource.WithTelemetrySDK(), resource.WithProcess(), resource.WithOS(), resource.WithContainer(), resource.WithHost()) if err != nil { return nil, err } // Set up the meter provider with the exporter and resource and a periodic reader that flushes every minute mp := metric.NewMeterProvider(metric.WithResource(resources), metric.WithReader(metric.NewPeriodicReader(exporter, metric.WithInterval(1*time.Minute)))) return mp, nil } func initTracerProvider() *sdktrace.TracerProvider { exporter, err := otlptracehttp.New(context.Background()) if err != nil { panic(err) } resources, err := resource.New(context.Background(), resource.WithFromEnv(), resource.WithTelemetrySDK(), resource.WithProcess(), resource.WithOS(), resource.WithContainer(), resource.WithHost()) if err != nil { panic(err) } tp := sdktrace.NewTracerProvider( sdktrace.WithBatcher(exporter), sdktrace.WithResource(resources), ) otel.SetTracerProvider(tp) otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})) return tp } func NewInstrumentation() *Instrumentation { tp := initTracerProvider() return &Instrumentation{ TracerProvider: tp, Tracer: tp.Tracer("gorilla-mux-server"), // Change this value to the name of your service } } ``` The above code configures the OpenTelemetry SDK to use the OTLP exporter and initializes the TracerProvider. Next, at the entry point of your application, add the following code to instrument your application: ```go import ( // other imports "github.com/gorilla/mux" "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux" ) func main() { i := last9NewInstrumentation() defer func() { if err := i.TracerProvider.Shutdown(context.Background()); err != nil { log.Printf("Error shutting down tracer provider: %v", err) } }() r := mux.NewRouter() // Add the middleware before adding any routes r.Use(otelmux.Middleware("gorilla-mux-server")) // Change this value to the name of your service // Add your routes here r.HandleFunc("/users/{id}", UpdateUser).Methods("PUT") // Start the server log.Fatal(http.ListenAndServe(":8080", r)) } ``` ## Instrument database operations ### Instrumenting with `sql.DB` Add the following code to instrument the database queries. It uses the `otelsql` package to wrap the `sql.DB` object and emit traces and metrics for database queries and connections. ```go func initDB() (*sql.DB, error) { // Register the driver with tracing driverName, err := otelsql.Register("postgres", // Read more about the options here: https://github.com/nhatthm/otelsql?tab=readme-ov-file#options otelsql.AllowRoot(), otelsql.TraceQueryWithoutArgs(), otelsql.TraceRowsClose(), otelsql.TraceRowsAffected(), otelsql.WithDatabaseName("otel_demo"), // your database name otelsql.WithSystem(semconv.DBSystemPostgreSQL), ) if err != nil { return nil, fmt.Errorf("failed to register driver: %v", err) } db, err := sql.Open(driverName, dsnName) if err != nil { return nil, fmt.Errorf("failed to connect to database: %v", err) } // Record stats to expose metrics if err := otelsql.RecordStats(db); err != nil { return nil, err } return db, nil } ``` ### Instrumenting with `pgx` For database instrumentation where [pgx](https://github.com/jackc/pgx) is used, use [otelpgx](https://github.com/exaring/otelpgx) to wrap the pgx connection pool. Add the following code to instrument the database queries. ```go var connString = os.Getenv("DATABASE_URL") cfg, err := pgxpool.ParseConfig(connString) if err != nil { fmt.Fprintf(os.Stderr, "create connection pool: %w", err) os.Exit(1) } // Add the tracer to the connection pool configuration cfg.ConnConfig.Tracer = otelpgx.NewTracer() // Create a new connection pool with the tracer conn, err = pgxpool.NewWithConfig(context.Background(), cfg) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connection to database: %v\n", err) os.Exit(1) } ``` Note Refer to the complete example using pgx adapter and Otel instrumentation [here](https://github.com/last9/opentelemetry-examples/tree/master/go/pgx). ## Instrument Redis operations Tip Note that between `redis-go` versions v8 and v9 the import paths are changed. Add the following code to instrument the Redis operations. It uses the `redisotel` package to wrap the `redis.Client` object and emit traces for redis operations. ### go-redis v9 If you are using `go-redis` v9 then use the following code. ```go func initRedis() *redis.Client { rdb := redis.NewClient(&redis.Options{ Addr: "localhost:6379", // Update this with your Redis server address }) // Setup traces for redis instrumentation if err := redisotel.InstrumentTracing(rdb); err != nil { log.Fatalf("failed to instrument traces for Redis client: %v", err) return nil } return rdb } ``` ### go-redis v8 If you are using `go-redis` v8 then use the following code. ```go func initRedis() *redis.Client { rdb := redis.NewClient(&redis.Options{ Addr: "localhost:6379", // Update this with your Redis server address }) // Setup traces for redis instrumentation rdb.AddHook(redisotel.NewTracingHook()) return rdb } ``` ## Run the application Start your Gorilla Mux application by running the following command: ```bash go run main.go ``` This will start the Gorilla Mux application and the OpenTelemetry SDK will automatically collect traces and metrics from the application. These traces and metrics will be sent to Last9 automatically based on the environment variables set. ## View traces and metrics in Last9 After running the Gorilla Mux app, you can visualize the traces and metrics in [Trace Explorer](https://app.last9.io/traces). *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # gRPC > Send distributed traces to Last9 from a Golang gRPC using OpenTelemetry gRPC is a high-performance, open-source universal RPC framework. This comprehensive guide will help you instrument your gRPC application with OpenTelemetry and smoothly send the traces to a Last9 cluster. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/go/grpc). ## Pre-requisites 1. You have a gRPC application. For this document, we will use a simple gRPC server and client application. You can use any gRPC application and follow the same steps. 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` ## Install OpenTelemetry packages To install the required packages, run the following command: ```bash go get -u go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc go get -u go.opentelemetry.io/otel go get -u go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc go get -u go.opentelemetry.io/otel/sdk ``` ## Setup the OpenTelemetry SDK To setup the OpenTelemetry SDK, you need to add the following code to your application: ```go package instrumentation // this can be changed to any name import ( "context" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" "go.opentelemetry.io/otel/propagation" "go.opentelemetry.io/otel/sdk/resource" sdktrace "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.17.0" ) // InitTracer initializes the OpenTelemetry tracer func InitTracer(serviceName string) func(context.Context) error { // Initialize the exporter exporter, err := otlptracegrpc.New(context.Background()) if err != nil { panic(err) } attr := resource.WithAttributes( // Set the deployment environment semantic attribute semconv.DeploymentEnvironmentKey.String("production"), // You can change this value to "development" or "staging" or you can get the value from the environment variables // You can add more attributes here ) // Create a new resource with the attributes resources, err := resource.New(context.Background(), resource.WithFromEnv(), resource.WithTelemetrySDK(), resource.WithProcess(), resource.WithOS(), resource.WithContainer(), resource.WithHost(), attr) if err != nil { panic(err) } // Create a new tracer provider with the exporter and resource tp := sdktrace.NewTracerProvider( // Batch the traces to the exporter using BatchSpanProcessor sdktrace.WithBatcher(exporter), sdktrace.WithResource(resources), ) otel.SetTracerProvider(tp) otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})) return tp.Shutdown } ``` ## Instrument the gRPC server application To instrument the gRPC server application, you need to add the following code to your application: ```go func main() { // Initialize the tracer shutdown := instrumentation.InitTracer("grpc-server") defer shutdown(context.Background()) lis, err := net.Listen("tcp", ":50051") if err != nil { log.Fatalf("failed to listen: %v", err) } s := grpc.NewServer( grpc.StatsHandler(otelgrpc.NewServerHandler()), ) pb.RegisterGreeterServer(s, &server{}) log.Printf("server listening at %v", lis.Addr()) if err := s.Serve(lis); err != nil { log.Fatalf("failed to serve: %v", err) } } ``` ## Instrument the gRPC client application To instrument the gRPC client application, you need to add the following code to your application: ```go func main() { // Initialize the tracer shutdown := instrumentation.InitTracer("grpc-client") defer shutdown(context.Background()) conn, err := grpc.Dial("localhost:50051", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithStatsHandler(otelgrpc.NewClientHandler())) if err != nil { log.Fatalf("did not connect: %v", err) } defer conn.Close() c := pb.NewGreeterClient(conn) name := "World" if len(os.Args) > 1 { name = os.Args[1] } ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() r, err := c.SayHello(ctx, &pb.HelloRequest{Name: name}) if err != nil { log.Fatalf("could not greet: %v", err) } log.Printf("Greeting: %s", r.GetMessage()) } ``` ## Run the application Set the `endpoint` and `auth_header` environment variables with the values you obtained from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page. ```bash export OTEL_SERVICE_NAME=grpc-server-app // this is name of the server side application export OTEL_EXPORTER_OTLP_ENDPOINT= export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic " ``` Run the server application: ```bash go run server/main.go ``` Run the client application: ```bash export OTEL_SERVICE_NAME=grpc-client-app // this is name of the client side application export OTEL_EXPORTER_OTLP_ENDPOINT= export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic " ``` ```bash go run client/main.go ``` Note Note that the service name should be different for the server and client applications to distinguish between them. ## View traces in Last9 After running the gRPC apps, you can visualize the traces in Last9’s APM dashboard. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Iris > Send distributed traces and metrics to Last9 from Golang Iris application using OpenTelemetry [Iris](https://github.com/kataras/iris) is a fast, flexible HTTP server framework written in Go (Golang). This comprehensive guide will help you instrument your Iris application with OpenTelemetry and smoothly send the traces to a Last9 cluster. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/go/iris). ## Pre-requisites 1. You have a Iris application. 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` ## Install OpenTelemetry packages To install the required packages, run the following command: ```bash go get -u go.opentelemetry.io/otel go get -u go.opentelemetry.io/otel/sdk go get -u go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc go get -u go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc go get -u go.opentelemetry.io/otel/sdk/metric go get -u go.nhat.io/otelsql ``` For setting up Redis instrumentation, first verify which `go-redis` version you are using. If you are using `go-redis` v8 then ```bash go get -u github.com/go-redis/redis/extra/redisotel ``` If you are using `go-redis` v9 then ```bash go get -u github.com/redis/go-redis/extra/redisotel ``` Using these packages, you can instrument your iris application to send traces to Last9. ### Traces This application generates traces for the following: * HTTP requests using [last9/otelMiddleware](https://github.com/last9/opentelemetry-examples/blob/master/go/iris/last9/otelMiddleware.go) * Database queries using [otelsql](https://github.com/nhatthm/otelsql) * Redis commands using [redisotel](https://github.com/redis/redis-go-cluster/tree/main/redisotel) For HTTP requests, wrap the iris router with the `last9.otelMiddleware` middleware. > Refer to [main.go](https://github.com/last9/opentelemetry-examples/blob/master/go/iris/main.go) for more details. For database queries, use the `otelsql` package to wrap the `sql.DB` object. > Refer to `initDB()` in [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/master/go/iris/users/controller.go) for more details. For Redis commands, use the `redisotel` package to wrap the `redis.Client` object. > Refer to `initRedis()` in [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/master/go/iris/users/controller.go) for more details. ## Set the environment variables Set the following environment variables: ```bash export OTEL_SERVICE_NAME=iris-app-service export OTEL_EXPORTER_OTLP_ENDPOINT= export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic " ``` These environment variables are used to configure the OpenTelemetry SDK to send traces and metrics to Last9. ## Instrument HTTP requests Setup instrumentation for iris application using [last9/instrumentation](https://github.com/last9/opentelemetry-examples/blob/master/go/iris/last9/instrumentation.go). ```go package last9 import ( "context" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/propagation" "go.opentelemetry.io/otel/sdk/resource" sdktrace "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.26.0" "go.opentelemetry.io/otel/trace" ) type Instrumentation struct { TracerProvider *sdktrace.TracerProvider Tracer trace.Tracer } func initTracerProvider() *sdktrace.TracerProvider { exporter, err := otlptracehttp.New(context.Background()) if err != nil { panic(err) } attr := resource.WithAttributes( semconv.DeploymentEnvironmentKey.String("production"), // replace with actual environment semconv.ServiceNameKey.String("iris-server"), // replace with service name ) resources, err := resource.New(context.Background(), resource.WithFromEnv(), resource.WithTelemetrySDK(), resource.WithProcess(), resource.WithOS(), resource.WithContainer(), resource.WithHost(), attr) if err != nil { panic(err) } tp := sdktrace.NewTracerProvider( sdktrace.WithBatcher(exporter), sdktrace.WithResource(resources), ) otel.SetTracerProvider(tp) otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})) return tp } func NewInstrumentation() *Instrumentation { tp := initTracerProvider() return &Instrumentation{ TracerProvider: tp, Tracer: tp.Tracer("iris-server"), } } ``` The above code configures the OpenTelemetry SDK to use the OTLP exporter and initializes the TracerProvider. Next, at the entry point of your application, add the following code to instrument your application: ```go // main.go func main() { i := NewInstrumentation() defer func() { if err := i.TracerProvider.Shutdown(context.Background()); err != nil { log.Printf("Error shutting down tracer provider: %v", err) } }() // other code } ``` Now, add the otel iris middleware to your application: ```go // main.go func main() { i := last9.NewInstrumentation() defer func() { if err := i.TracerProvider.Shutdown(context.Background()); err != nil { log.Printf("Error shutting down tracer provider: %v", err) } }() // other code // Use the OtelMiddleware app.Use(last9.OtelMiddleware("iris-server")) // replace with name of the service // other code } ``` ## Instrument database operations ### Instrumenting with `sql.DB` Add the following code to instrument the database queries. It uses the `otelsql` package to wrap the `sql.DB` object and emit traces and metrics for database queries and connections. Refer to [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/master/go/iris/users/controller.go) for more details. ```go func initDB() (*sql.DB, error) { driverName, err := otelsql.Register("postgres", // Read more about the options here: https://github.com/nhatthm/otelsql?tab=readme-ov-file#options otelsql.AllowRoot(), otelsql.TraceQueryWithoutArgs(), otelsql.TraceRowsClose(), otelsql.TraceRowsAffected(), otelsql.WithDatabaseName("otel_demo"), // database name otelsql.WithSystem(semconv.DBSystemPostgreSQL), ) if err != nil { return nil, fmt.Errorf("failed to register driver: %v", err) } db, err := sql.Open(driverName, dsnName) if err != nil { return nil, fmt.Errorf("failed to connect to database: %v", err) } // Record stats to expose metrics if err := otelsql.RecordStats(db); err != nil { return nil, err } return db, nil } ``` ### Instrumenting with `pgx` For database instrumentation where [pgx](https://github.com/jackc/pgx) is used, we use [otelpgx](https://github.com/exaring/otelpgx) to wrap the pgx connection pool. Add the following code to instrument the database queries. ```go var connString = os.Getenv("DATABASE_URL") cfg, err := pgxpool.ParseConfig(connString) if err != nil { fmt.Fprintf(os.Stderr, "create connection pool: %w", err) os.Exit(1) } // Add the tracer to the connection pool configuration cfg.ConnConfig.Tracer = otelpgx.NewTracer() // Create a new connection pool with the tracer conn, err = pgxpool.NewWithConfig(context.Background(), cfg) if err != nil { fmt.Fprintf(os.Stderr, "Unable to connection to database: %v\n", err) os.Exit(1) } ``` > Refer to the complete example using pgx adapter and Otel instrumentation [here](https://github.com/last9/opentelemetry-examples/tree/master/go/pgx). ## Instrument Redis operations Tip Note that between `redis-go` versions v8 and v9 the import paths are changed. Add the following code to instrument the Redis operations. It uses the `redisotel` package to wrap the `redis.Client` object and emit traces for Redis commands. ### go-redis v9 If you are using `go-redis` v9 then use the following code. > Refer to [users/controller.go](https://github.com/last9/opentelemetry-examples/blob/master/go/iris/users/controller.go) for more details. ```go func initRedis() *redis.Client { rdb := redis.NewClient(&redis.Options{ Addr: "localhost:6379", // Update this with your Redis server address }) // Setup traces for redis instrumentation if err := redisotel.InstrumentTracing(rdb); err != nil { log.Fatalf("failed to instrument traces for Redis client: %v", err) return nil } return rdb } ``` ### go-redis v8 If you are using `go-redis` v8 then use the following code. ```go func initRedis() *redis.Client { rdb := redis.NewClient(&redis.Options{ Addr: "localhost:6379", // Update this with your Redis server address }) // Setup traces for redis instrumentation rdb.AddHook(redisotel.NewTracingHook()) return rdb } ``` > Refer to [main.go](https://github.com/last9/opentelemetry-examples/blob/master/go/iris/users/main.go) for more details. ## Run the application Start your iris application by running the following command: ```bash go run main.go ``` This will start the iris application and the OpenTelemetry SDK will automatically collect traces and metrics from the iris application. These traces will be sent to Last9 automatically based on the environment variables set. ## View traces in Last9 After running the iris app, you can visualize the traces in Last9’s APM dashboard. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Koa > Send distributed traces to Last9 from a Koa app using OpenTelemetry Koa is a web framework for Node.js. This comprehensive guide will help you instrument your Koa application with OpenTelemetry, seamlessly sending the traces to Last9. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/javascript/koa). ## Pre-requisites 1. You have a Koa application 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` ## Install OpenTelemetry packages To perform the instrumentation, install the following OpenTelemetry packages. ```bash npm install --save @opentelemetry/api \ @opentelemetry/instrumentation \ @opentelemetry/tracing \ @opentelemetry/exporter-trace-otlp-http \ @opentelemetry/resources \ @opentelemetry/semantic-conventions \ @opentelemetry/auto-instrumentations-node \ @opentelemetry/sdk-node ``` To know more about them, you can check: ## Setup auto-instrumentation using OpenTelemetry Add the necessary environment variables that can be obtained from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page. ```bash OTET_EXPORTER_OTLP_ENDPOINT=/v1/traces OTLP_AUTHORIZATION_HEADER="Basic " ```  Next, create a file named `instrumentation.ts` and add the following code: ```ts import * as api from "@opentelemetry/api"; import { NodeTracerProvider, TracerConfig, } from "@opentelemetry/sdk-trace-node"; import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"; import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; import { registerInstrumentations } from "@opentelemetry/instrumentation"; import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node"; import { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_DEPLOYMENT_ENVIRONMENT, } from "@opentelemetry/semantic-conventions"; import { Resource } from "@opentelemetry/resources"; // For troubleshooting, set the log level to DiagLogLevel.DEBUG // import { diag, DiagConsoleLogger, DiagLogLevel } from "@opentelemetry/api"; // diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG); export const setupTracing = (serviceName: string) => { const providerConfig: TracerConfig = { resource: new Resource({ [SEMRESATTRS_SERVICE_NAME]: serviceName, [SEMRESATTRS_DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV, }), }; // Initialize and register the tracer provider const provider = new NodeTracerProvider(providerConfig); const otlp = new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT, headers: { Authorization: process.env.OTEL_AUTHORIZATION_HEADER, }, }); provider.addSpanProcessor(new BatchSpanProcessor(otlp)); // Automatically instrument HTTP and Koa registerInstrumentations({ instrumentations: [ getNodeAutoInstrumentations({ "@opentelemetry/instrumentation-fs": { enabled: false, }, }), ], tracerProvider: provider, }); provider.register(); return api.trace.getTracer(serviceName); }; ``` This above code performs the following steps: 1. Set up Trace Provider with the application’s name as Service Name. 2. Set up OTLP Exporter with Last9 OTLP endpoint. 3. Set up auto instrumentation for all the supported libraries. Next, this script must be imported at the application’s **entry point**, even before importing `koa` package. For instance, if you have `server.ts` as your entry file, then import it at the top of the file as follows: ```ts import { setupTracing } from "./instrumentation"; // Do this before requiring koa package setupTracing("koa-api-server"); import Koa from "koa"; // other imports... ``` Tip You can also use `instrumentation.js` if you are not using Typescript. ## Visualize the traces in Last9 Once the application runs with the above code, it will start pushing traces to Last9. You can see the result in action by looking at the APM dashboard in Last9.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Kong > Send distributed traces from Kong to Last9 using OpenTelemetry This guide will help you integrate [Kong](https://konghq.com/) with Last9 to monitor and improve the observability of your API gateway. This document provides step by step instructions to start collecting traces using Kong’s OpenTelemetry plugin. ## Prerequisites 1. Kong (version 2.0 or above) 2. Last9 OpenTelemetry endpoint credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page * `levitate_otlp_endpoint` * `levitate_otlp_auth_header` ## Enable Distributed Tracing in Kong 1. **Enable tracing instrumentation** Set the environment variable `KONG_TRACING_INSTRUMENTATIONS="all"` and restart Kong. More Kong configuration options available [here](https://docs.konghq.com/gateway/3.7.x/reference/configuration/#tracing_instrumentations). 2. **Enable the OpenTelemetry Plugin** Add the OpenTelemetry plugin to your Kong configuration file or enable it via the Admin API. Here’s an example of how to enable the plugin globally via the Kong Admin API: ```bash curl -i -X POST http://:8001/plugins \ --data "name=opentelemetry" \ --data "config.endpoint=http://" \ --data "config.headers.Authorization=" \ --data "config.service_name=kong" ``` Note The `opentelemetry` plugin sends trace to an OpenTelemetry compatible backend such as Last9. 3. **Verify the Plugin is Enabled** You can verify that the plugin is enabled by checking the Kong logs for trace data being sent to Last9. Tip Set environment variable `KONG_LOG_LEVEL="debug"` to verify traces are flowing in correctly to Last9 if needed. ## Verify Traces in Last9  * Log in to your Last9 account and navigate to the Grafana dashboard * Navigate to the traces dashboard to see the traces collected from Kong. You should see traces corresponding to your API requests, with detailed information about the request flow and latency *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Kubernetes Audit Logs > Send Kubernetes audit logs to Last9 using OpenTelemetry This guide will help you ingest audit logs from a Kubernetes cluster to Last9. It is useful when you have set up a Kubernetes cluster using tools like kOps and enabled audit logs. If you use EKS and want to ingest audit logs to Last9, please use the AWS Cloudwatch integration. ## Pre-requisites 1. You have a Kubernetes Cluster and workload running in it. 2. You have enabled audit logs for your Kubernetes cluster to be written to a file. 3. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` 4. You have an Otel collector agent dameonset running as per the [Last9 Kubernetes logs integration](/docs/integrations-opentelemetry-kubernetes-logs/) ## Enable the volume mounts for Audit Logs Add the following volumes to the `values.yaml` to enable the Collector Agent dameonset installed in step 4 above to read the audit logs. ```yaml extraVolumes: - name: audit-logs hostPath: path: /var/log type: Directory extraVolumeMounts: - name: audit-logs mountPath: /var/log readOnly: true ``` ## Update the Otel Pipeline to read audit logs Update the `receiver` section with the following configuration. ```yaml receivers: filelog: include: # Read logs from all pods on the node - /var/log/pods/*/*/*.log # Update this according to the path you have set for the audit log path. - /var/log/kube-apiserver-audit.log include_file_name: false include_file_path: true operators: - id: container-parser max_log_size: 102400 type: container retry_on_failure: enabled: true start_at: end ``` Keep all the other settings in `values.yaml` the same. These two changes will enable the Otel Collector Agent to read audit logs and the logs from all the pods and send them to Last9. ## Verification Login to Last9 and visit the [Logs](https://app.last9.io/logs) panel.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Logs from Kubernetes Cluster > Send logs from Kubernetes Cluster using OpenTelemetry This guide will help you instrument your Kubernetes Cluster with OpenTelemetry and smoothly send the logs to a Last9 from all containers in the cluster. ## Pre-requisites 1. You have a Kubernetes Cluster and workload running in it. 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` 3. Install [helm](https://helm.sh/). 4. Create a namespace `last9` in your kubernetes cluster. We will use this to setup the OpenTelemetry Collector agent. ## Helm Chart Installation of OpenTelemetry Collector Copy the Helm chart locally as `last9-otel-collector-values.yaml` ```yaml # Default values for opentelemetry-collector. # This is a YAML-formatted file. # Declare variables to be passed into your templates. nameOverride: "last9-otel-collector" fullnameOverride: "" # Valid values are "daemonset", "deployment", and "statefulset". mode: "daemonset" # Specify which namespace should be used to deploy the resources into namespaceOverride: "last9" # Handles basic configuration of components that # also require k8s modifications to work correctly. # .Values.config can be used to modify/add to a preset # component configuration, but CANNOT be used to remove # preset configuration. If you require removal of any # sections of a preset configuration, you cannot use # the preset. Instead, configure the component manually in # .Values.config and use the other fields supplied in the # values.yaml to configure k8s as necessary. presets: # Configures the collector to collect logs. # Adds the filelog receiver to the logs pipeline # and adds the necessary volumes and volume mounts. # Best used with mode = daemonset. # See https://opentelemetry.io/docs/kubernetes/collector/components/#filelog-receiver for details on the receiver. logsCollection: enabled: true includeCollectorLogs: false # Enabling this writes checkpoints in /var/lib/otelcol/ host directory. # Note this changes collector's user to root, so that it can write to host directory. storeCheckpoints: false # The maximum bytes size of the recombined field. # Once the size exceeds the limit, all received entries of the source will be combined and flushed. maxRecombineLogSize: 102400 # Configures the collector to collect host metrics. # Adds the hostmetrics receiver to the metrics pipeline # and adds the necessary volumes and volume mounts. # Best used with mode = daemonset. # See https://opentelemetry.io/docs/kubernetes/collector/components/#host-metrics-receiver for details on the receiver. hostMetrics: enabled: true # Configures the Kubernetes Processor to add Kubernetes metadata. # Adds the k8sattributes processor to all the pipelines # and adds the necessary rules to ClusteRole. # Best used with mode = daemonset. # See https://opentelemetry.io/docs/kubernetes/collector/components/#kubernetes-attributes-processor for details on the receiver. kubernetesAttributes: enabled: true # When enabled the processor will extra all labels for an associated pod and add them as resource attributes. # The label's exact name will be the key. extractAllPodLabels: true # When enabled the processor will extra all annotations for an associated pod and add them as resource attributes. # The annotation's exact name will be the key. extractAllPodAnnotations: false # Configures the collector to collect node, pod, and container metrics from the API server on a kubelet.. # Adds the kubeletstats receiver to the metrics pipeline # and adds the necessary rules to ClusteRole. # Best used with mode = daemonset. # See https://opentelemetry.io/docs/kubernetes/collector/components/#kubeletstats-receiver for details on the receiver. kubeletMetrics: enabled: true # Configures the collector to collect kubernetes events. # Adds the k8sobject receiver to the logs pipeline # and collects kubernetes events by default. # Best used with mode = deployment or statefulset. # See https://opentelemetry.io/docs/kubernetes/collector/components/#kubernetes-objects-receiver for details on the receiver. kubernetesEvents: enabled: false # Configures the Kubernetes Cluster Receiver to collect cluster-level metrics. # Adds the k8s_cluster receiver to the metrics pipeline # and adds the necessary rules to ClusteRole. # Best used with mode = deployment or statefulset. # See https://opentelemetry.io/docs/kubernetes/collector/components/#kubernetes-cluster-receiver for details on the receiver. clusterMetrics: enabled: false configMap: # Specifies whether a configMap should be created (true by default) create: true # Specifies an existing ConfigMap to be mounted to the pod # The ConfigMap MUST include the collector configuration via a key named 'relay' or the collector will not start. existingName: "" # Base collector configuration. config: exporters: # Use when you need to debug output # debug: # verbosity: detailed # sampling_initial: 5 # sampling_thereafter: 200 otlp/last9: endpoint: "" headers: "Authorization": "Basic " extensions: # The health_check extension is mandatory for this chart. # Without the health_check extension the collector will fail the readiness and liveliness probes. # The health_check extension can be modified, but should never be removed. health_check: endpoint: ${env:MY_POD_IP}:13133 processors: transform/logs/last9: error_mode: ignore log_statements: - context: resource statements: # set the service name as app label of the container # - set(attributes["service.name"], attributes[""]) where attributes["service.name"] == "" # Additional resource attributes can be set as follows. # - set(attributes[""], value) # - set(attributes["deployment_environment"], "staging") batch: send_batch_size: 50000 send_batch_max_size: 50000 timeout: 5s # Default memory limiter configuration for the collector based on k8s resource limits. # This configuration is best suited when you are only shipping logs via the Otel Collector. # If you want to ship more telemerty signals, then consult with Last9 team to twaek the memory # limiter settings. memory_limiter/logs: # check_interval is the time between measurements of memory usage. check_interval: 5s # By default limit_mib is set to 85% of ".Values.resources.limits.memory" limit_percentage: 85 # By default spike_limit_mib is set to 15% of ".Values.resources.limits.memory" spike_limit_percentage: 15 receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 max_recv_msg_size_mib: 16 http: endpoint: 0.0.0.0:4318 service: extensions: - health_check pipelines: logs: receivers: - otlp - filelog processors: - memory_limiter/logs - k8sattributes - transform/logs/last9 - batch exporters: - otlp/last9 metrics: receivers: - otlp - prometheus - hostmetrics - kubeletstats exporters: - otlp/last9 traces: receivers: - otlp exporters: - otlp/last9 image: # If you want to use the core image `otel/opentelemetry-collector`, you also need to change `command.name` value to `otelcol`. repository: "otel/opentelemetry-collector-contrib" pullPolicy: IfNotPresent # Overrides the image tag whose default is the chart appVersion. tag: "0.107.0" # When digest is set to a non-empty value, images will be pulled by digest (regardless of tag value). digest: "" imagePullSecrets: [] # OpenTelemetry Collector executable command: name: "" extraArgs: [] serviceAccount: # Specifies whether a service account should be created create: true # Annotations to add to the service account annotations: {} # The name of the service account to use. # If not set and create is true, a name is generated using the fullname template name: "" clusterRole: # Specifies whether a clusterRole should be created # Some presets also trigger the creation of a cluster role and cluster role binding. # If using one of those presets, this field is no-op. create: false # Annotations to add to the clusterRole # Can be used in combination with presets that create a cluster role. annotations: {} # The name of the clusterRole to use. # If not set a name is generated using the fullname template # Can be used in combination with presets that create a cluster role. name: "" # A set of rules as documented here : https://kubernetes.io/docs/reference/access-authn-authz/rbac/ # Can be used in combination with presets that create a cluster role to add additional rules. rules: [] # - apiGroups: # - '' # resources: # - 'pods' # - 'nodes' # verbs: # - 'get' # - 'list' # - 'watch' clusterRoleBinding: # Annotations to add to the clusterRoleBinding # Can be used in combination with presets that create a cluster role binding. annotations: {} # The name of the clusterRoleBinding to use. # If not set a name is generated using the fullname template # Can be used in combination with presets that create a cluster role binding. name: "" podSecurityContext: {} securityContext: {} nodeSelector: {} tolerations: [] affinity: {} topologySpreadConstraints: [] # Allows for pod scheduler prioritisation priorityClassName: "" extraEnvs: [] extraEnvsFrom: [] # This also supports template content, which will eventually be converted to yaml. extraVolumes: [] # This also supports template content, which will eventually be converted to yaml. extraVolumeMounts: [] # Configuration for ports # nodePort is also allowed ports: otlp: enabled: false containerPort: 4317 servicePort: 4317 hostPort: 4317 protocol: TCP # nodePort: 30317 appProtocol: grpc otlp-http: enabled: false containerPort: 4318 servicePort: 4318 hostPort: 4318 protocol: TCP metrics: # The metrics port is disabled by default. However you need to enable the port # in order to use the ServiceMonitor (serviceMonitor.enabled) or PodMonitor (podMonitor.enabled). enabled: false containerPort: 8888 servicePort: 8888 protocol: TCP # When enabled, the chart will set the GOMEMLIMIT env var to 80% of the configured resources.limits.memory. # If no resources.limits.memory are defined then enabling does nothing. # It is HIGHLY recommend to enable this setting and set a value for resources.limits.memory. useGOMEMLIMIT: true # Resource limits & requests. # It is HIGHLY recommended to set resource limits. resources: {} # resources: # limits: # cpu: 250m # memory: 512Mi podAnnotations: {} podLabels: {} # Common labels to add to all otel-collector resources. Evaluated as a template. additionalLabels: {} # app.kubernetes.io/part-of: my-app # Host networking requested for this pod. Use the host's network namespace. hostNetwork: false # Adding entries to Pod /etc/hosts with HostAliases # https://kubernetes.io/docs/tasks/network/customize-hosts-file-for-pods/ hostAliases: [] # - ip: "1.2.3.4" # hostnames: # - "my.host.com" # Pod DNS policy ClusterFirst, ClusterFirstWithHostNet, None, Default, None dnsPolicy: "" # Custom DNS config. Required when DNS policy is None. dnsConfig: {} # only used with deployment mode replicaCount: 1 # only used with deployment mode revisionHistoryLimit: 10 annotations: {} # List of extra sidecars to add. # This also supports template content, which will eventually be converted to yaml. extraContainers: [] # extraContainers: # - name: test # command: # - cp # args: # - /bin/sleep # - /test/sleep # image: busybox:latest # volumeMounts: # - name: test # mountPath: /test # List of init container specs, e.g. for copying a binary to be executed as a lifecycle hook. # This also supports template content, which will eventually be converted to yaml. # Another usage of init containers is e.g. initializing filesystem permissions to the OTLP Collector user `10001` in case you are using persistence and the volume is producing a permission denied error for the OTLP Collector container. initContainers: [] # initContainers: # - name: test # image: busybox:latest # command: # - cp # args: # - /bin/sleep # - /test/sleep # volumeMounts: # - name: test # mountPath: /test # - name: init-fs # image: busybox:latest # command: # - sh # - '-c' # - 'chown -R 10001: /var/lib/storage/otc' # use the path given as per `extensions.file_storage.directory` & `extraVolumeMounts[x].mountPath` # volumeMounts: # - name: opentelemetry-collector-data # use the name of the volume used for persistence # mountPath: /var/lib/storage/otc # use the path given as per `extensions.file_storage.directory` & `extraVolumeMounts[x].mountPath` # Pod lifecycle policies. lifecycleHooks: {} # lifecycleHooks: # preStop: # exec: # command: # - /test/sleep # - "5" # liveness probe configuration # Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ ## livenessProbe: # Number of seconds after the container has started before startup, liveness or readiness probes are initiated. # initialDelaySeconds: 1 # How often in seconds to perform the probe. # periodSeconds: 10 # Number of seconds after which the probe times out. # timeoutSeconds: 1 # Minimum consecutive failures for the probe to be considered failed after having succeeded. # failureThreshold: 1 # Duration in seconds the pod needs to terminate gracefully upon probe failure. # terminationGracePeriodSeconds: 10 httpGet: port: 13133 path: / # readiness probe configuration # Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ ## readinessProbe: # Number of seconds after the container has started before startup, liveness or readiness probes are initiated. # initialDelaySeconds: 1 # How often (in seconds) to perform the probe. # periodSeconds: 10 # Number of seconds after which the probe times out. # timeoutSeconds: 1 # Minimum consecutive successes for the probe to be considered successful after having failed. # successThreshold: 1 # Minimum consecutive failures for the probe to be considered failed after having succeeded. # failureThreshold: 1 httpGet: port: 13133 path: / service: # Enable the creation of a Service. # By default, it's enabled on mode != daemonset. # However, to enable it on mode = daemonset, its creation must be explicitly enabled # enabled: true type: ClusterIP # type: LoadBalancer # loadBalancerIP: 1.2.3.4 # loadBalancerSourceRanges: [] # By default, Service of type 'LoadBalancer' will be created setting 'externalTrafficPolicy: Cluster' # unless other value is explicitly set. # Possible values are Cluster or Local (https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip) # externalTrafficPolicy: Cluster annotations: {} # By default, Service will be created setting 'internalTrafficPolicy: Local' on mode = daemonset # unless other value is explicitly set. # Setting 'internalTrafficPolicy: Cluster' on a daemonset is not recommended # internalTrafficPolicy: Cluster ingress: enabled: false # annotations: {} # ingressClassName: nginx # hosts: # - host: collector.example.com # paths: # - path: / # pathType: Prefix # port: 4318 # tls: # - secretName: collector-tls # hosts: # - collector.example.com # Additional ingresses - only created if ingress.enabled is true # Useful for when differently annotated ingress services are required # Each additional ingress needs key "name" set to something unique additionalIngresses: [] # - name: cloudwatch # ingressClassName: nginx # annotations: {} # hosts: # - host: collector.example.com # paths: # - path: / # pathType: Prefix # port: 4318 # tls: # - secretName: collector-tls # hosts: # - collector.example.com podMonitor: # The pod monitor by default scrapes the metrics port. # The metrics port needs to be enabled as well. enabled: false metricsEndpoints: - port: metrics # interval: 15s # additional labels for the PodMonitor extraLabels: {} # release: kube-prometheus-stack serviceMonitor: # The service monitor by default scrapes the metrics port. # The metrics port needs to be enabled as well. enabled: false metricsEndpoints: - port: metrics # interval: 15s # additional labels for the ServiceMonitor extraLabels: {} # release: kube-prometheus-stack # Used to set relabeling and metricRelabeling configs on the ServiceMonitor # https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config relabelings: [] metricRelabelings: [] # PodDisruptionBudget is used only if deployment enabled podDisruptionBudget: enabled: false # minAvailable: 2 # maxUnavailable: 1 # autoscaling is used only if mode is "deployment" or "statefulset" autoscaling: enabled: false minReplicas: 1 maxReplicas: 10 behavior: {} targetCPUUtilizationPercentage: 80 # targetMemoryUtilizationPercentage: 80 rollout: rollingUpdate: {} # When 'mode: daemonset', maxSurge cannot be used when hostPort is set for any of the ports # maxSurge: 25% # maxUnavailable: 0 strategy: RollingUpdate prometheusRule: enabled: false groups: [] # Create default rules for monitoring the collector defaultRules: enabled: false # additional labels for the PrometheusRule extraLabels: {} statefulset: # volumeClaimTemplates for a statefulset volumeClaimTemplates: [] podManagementPolicy: "Parallel" # Controls if and how PVCs created by the StatefulSet are deleted. Available in Kubernetes 1.23+. persistentVolumeClaimRetentionPolicy: enabled: false whenDeleted: Retain whenScaled: Retain networkPolicy: enabled: false # Annotations to add to the NetworkPolicy annotations: {} # Configure the 'from' clause of the NetworkPolicy. # By default this will restrict traffic to ports enabled for the Collector. If # you wish to further restrict traffic to other hosts or specific namespaces, # see the standard NetworkPolicy 'spec.ingress.from' definition for more info: # https://kubernetes.io/docs/reference/kubernetes-api/policy-resources/network-policy-v1/ allowIngressFrom: [] # # Allow traffic from any pod in any namespace, but not external hosts # - namespaceSelector: {} # # Allow external access from a specific cidr block # - ipBlock: # cidr: 192.168.1.64/32 # # Allow access from pods in specific namespaces # - namespaceSelector: # matchExpressions: # - key: kubernetes.io/metadata.name # operator: In # values: # - "cats" # - "dogs" # Add additional ingress rules to specific ports # Useful to allow external hosts/services to access specific ports # An example is allowing an external prometheus server to scrape metrics # # See the standard NetworkPolicy 'spec.ingress' definition for more info: # https://kubernetes.io/docs/reference/kubernetes-api/policy-resources/network-policy-v1/ extraIngressRules: [] # - ports: # - port: metrics # protocol: TCP # from: # - ipBlock: # cidr: 192.168.1.64/32 # Restrict egress traffic from the OpenTelemetry collector pod # See the standard NetworkPolicy 'spec.egress' definition for more info: # https://kubernetes.io/docs/reference/kubernetes-api/policy-resources/network-policy-v1/ egressRules: [] # - to: # - namespaceSelector: {} # - ipBlock: # cidr: 192.168.10.10/24 # ports: # - port: 1234 # protocol: TCP # Allow containers to share processes across pod namespace shareProcessNamespace: false ``` ## What does this Helm chart do? This helm chart installs OpenTelemetry Collector in agent mode in each and every Kubernetes node in your cluster using the dameonset mode. It configures the logging pipeline to send logs from all containers for the node to Last9. It also sends health metrics from the Otel Collector to Last9 so that Last9 can monitor the ingestion pipeline as well. ## Setting additional log attributes You can update the additional labels by configuring the transform processor. ```yaml transform/logs/last9: error_mode: ignore log_statements: - context: resource statements: - set(attributes["service.name"], attributes["k8s.container.name"]) where attributes["service.name"] == "" # Additional resource attributes can be set as follows. # Use additional resource attributes to differentiate between clusters and deployments. # set(attributes[""], value) - set(attributes["deployment_environment"], "staging") ``` > Note that, do not remove the service.name transformation as it is needed for accelerated queries to kick in on Last9. ## Installation Make sure that `helm` is installed and up-to-date. ```yaml helm repo update helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm upgrade --install last9-opentelemetry-collector open-telemetry/opentelemetry-collector -f last9-otel-collector-values.yaml ``` It is highly recommended to use a dedicated namespace for the OpenTelemetry Collector and checkin the `values.yaml` file in your version control setup. ## Verification Verify that the helm chart is installed correctly. If all goes well, you should see a message along the following lines. ```yaml Release "last9-opentelemetry-collector" has been upgraded. Happy Helming! NAME: last9-opentelemetry-collector LAST DEPLOYED: Mon Aug 19 13:24:09 2024 NAMESPACE: default STATUS: deployed REVISION: 4 TEST SUITE: None NOTES: [WARNING] No resource limits or requests were set. Consider setter resource requests and limits for your collector(s) via the `resources` field. [WARNING] "useGOMEMLIMIT" is enabled but memory limits have not been supplied so the GOMEMLIMIT env var could not be added. Solve this problem by setting resources.limits.memory or disabling useGOMEMLIMIT ``` You can also visit the [Logs Dashboard](https://app.last9.io/logs) to see the data in action. ## Advanced settings You can configure resource limits for the memory and CPU in the `resources` section as needed. By default it is not set. ```yaml resources: {} ``` ## Testing Your Config You can test your configs before sending to Last9 to ensure they’re working as expected by adding a debug exporter section to your opentelemetry collector. ```yaml exporters: # Add debug exporter along with your Last9 exporter debug: verbosity: detailed sampling_initial: 10 # Sample first 10 items sampling_thereafter: 50 # Then sample 1 in 50 otlp/last9: endpoint: "" headers: "Authorization": "Basic your-auth-token" service: pipelines: logs: receivers: - otlp - filelog processors: - memory_limiter/logs - k8sattributes - transform/logs/last9 # Your transformation processor - batch exporters: - debug # Add debug exporter here - otlp/last9 ``` With this configuration: 1. The debug exporter will print processed telemetry data to the collector’s logs 2. Check transformations by examining the collector logs: ```bash kubectl logs -n last9 -l app.kubernetes.io/name=last9-otel-collector ``` 3. Verify your attributes are set correctly in the log output 4. The sampling parameters control how much data is printed: * `sampling_initial: 10` logs the first 10 items * `sampling_thereafter: 50` logs 1 in every 50 items afterward 5. Adjust `verbosity` level: * `normal`: Summary of the data * `detailed`: Full representation with all attributes For targeted testing: 1. Create a temporary pipeline specifically for testing: ```yaml service: pipelines: logs/test: # Testing pipeline receivers: - otlp processors: - transform/logs/test # Your test transformation exporters: - debug ``` 2. Once verified, remove the debug exporter before deploying to production to avoid performance impact This approach lets you validate your transformations before sending data to Last9, ensuring attributes are correctly set and improving your confidence in the configuration. ## Verification Login to Last9 and visit the [Logs](https://app.last9.io/logs) panel.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # MariaDB > This document describes a sample setup for sending MariaDB logs and metrics to Last9 via OpenTelemetry collector This document lists step-by-step instructions for setting up MariaDB/MySQL monitoring using [`mysqld_exporter`](https://github.com/prometheus/mysqld_exporter) and [OpenTelemetry Collector](https://last9.io/blog/opentelemetry-collector-guide/) with Last9. It collects host metrics, MySQL metrics, and MySQL logs and sends them to Last9. ## Prerequisites 1. Create a Last9 account by following [Getting Started](/docs/onboard/) guide. 2. Keep the following information handy after creating the account from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `$last9_otlp_endpoint`: Last9 OpenTelemetry endpoint * `$last9_basic_auth_header`: OpenTelemetry Basic authorization header ### Supported versions: * MySQL >= 5.6. * MariaDB >= 10.3 ## Installation If you don’t have a MariaDB instance, you can follow the steps below to set it up. If you already have a MariaDB instance, you can skip this step. ### 1. MariaDB Setup ```bash # Install MariaDB sudo apt-get update sudo apt-get install -y mariadb-server mariadb-client # Verify installation sudo systemctl status mariadb # Run security script sudo mysql_secure_installation ``` #### Configure MariaDB: ```bash sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf ``` Add configuration: ```ini [mysqld] bind-address = 0.0.0.0 max_connections = 100 innodb_buffer_pool_size = 256M thread_cache_size = 8 slow_query_log = 1 slow_query_log_file = /var/log/mysql/mariadb-slow.log long_query_time = 2 ``` ```bash # Restart MariaDB sudo systemctl restart mariadb ``` ### 2. Create Monitoring User ```sql sudo mysql -u root -p CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'strong_password'; GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost'; FLUSH PRIVILEGES; EXIT; ``` ### 3. Install mysqld\_exporter ```bash # Create system user sudo useradd --no-create-home --shell /bin/false mysqld_exporter # Download and install cd /tmp curl -s https://api.github.com/repos/prometheus/mysqld_exporter/releases/latest \ | grep browser_download_url \ | grep linux-amd64 \ | cut -d '"' -f 4 \ | wget -qi - tar xvf mysqld_exporter*.tar.gz sudo mv mysqld_exporter-*.linux-amd64/mysqld_exporter /usr/local/bin/ sudo chown mysqld_exporter:mysqld_exporter /usr/local/bin/mysqld_exporter ``` #### Configure exporter: ```bash sudo tee /etc/.mysqld_exporter.cnf << EOF [client] user=exporter password=strong_password EOF sudo chown mysqld_exporter:mysqld_exporter /etc/.mysqld_exporter.cnf sudo chmod 600 /etc/.mysqld_exporter.cnf ``` #### Create service: ```bash sudo tee /etc/systemd/system/mysqld_exporter.service << EOF [Unit] Description=MySQLd Exporter Wants=network-online.target After=network-online.target [Service] User=mysqld_exporter Group=mysqld_exporter Type=simple ExecStart=/usr/local/bin/mysqld_exporter \ --config.my-cnf=/etc/.mysqld_exporter.cnf \ --collect.global_status \ --collect.global_variables \ --collect.info_schema.innodb_metrics \ --collect.info_schema.processlist \ --collect.info_schema.tables \ --collect.info_schema.tables.databases='*' \ --collect.perf_schema.eventsstatements \ --collect.perf_schema.file_events \ --collect.perf_schema.indexiowaits \ --collect.perf_schema.tableiowaits \ --collect.perf_schema.tablelocks \ --collect.slave_status \ --web.listen-address=:9104 [Install] WantedBy=multi-user.target EOF sudo systemctl daemon-reload sudo systemctl start mysqld_exporter sudo systemctl enable mysqld_exporter ``` This step ensures the `mysqld_exporter` is running and emitting metrics from MariaDB on port 9104. ### 4. Install OpenTelemetry Collector ```bash sudo apt-get update sudo apt-get -y install wget systemctl wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.110.0/otelcol-contrib_0.110.0_linux_amd64.deb sudo dpkg -i otelcol-contrib_0.110.0_linux_amd64.deb ``` #### Configure collector: ```bash sudo nano /etc/otelcol-contrib/config.yaml ``` Copy the configuration from [here](https://github.com/last9/opentelemetry-examples/blob/main/otel-collector/mariadb/otel-config.yaml) and update in `/etc/otelcol-contrib/config.yaml`. Make sure to update the OpenTelemetry endpoint to `$last9_otlp_endpoint` and authorization header to `$last9_basic_auth_header`. #### Start collector: ```bash otelcol-contrib --config /etc/otelcol-contrib/config.yaml ``` ## Verification 1. Check MariaDB: ```bash mysql -u root -p -e "SELECT VERSION();" ``` 2. Verify configuration: ```bash mysql -u root -p -e "SHOW VARIABLES LIKE '%max_connections%';" mysql -u root -p -e "SHOW VARIABLES LIKE '%innodb_buffer_pool_size%';" mysql -u root -p -e "SHOW VARIABLES LIKE '%slow_query%';" ``` 3. Test exporter: ```bash mysql -u exporter -p -e "SELECT 1;" curl http://localhost:9104/metrics | grep mysql_up ``` ## Troubleshooting 1. MariaDB issues: ```bash sudo systemctl status mariadb sudo journalctl -u mariadb ``` 2. Exporter issues: ```bash sudo systemctl status mysqld_exporter sudo journalctl -u mysqld_exporter ``` 3. Collector issues: ```bash sudo systemctl status otelcol-contrib sudo journalctl -u otelcol-contrib ``` ## Verification Visit [Last9](https://app.last9.io) to see the MariaDB metrics and logs. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Nest.js > Send distributed traces to Last9 from an Nest.js app using OpenTelemetry Nest.js is a progressive Node.js framework for building efficient, reliable, scalable server-side applications. This comprehensive guide will help you instrument your Nest.js application with OpenTelemetry, seamlessly sending the traces to Last9. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/javascript/nestjs). ## Pre-requisites 1. You have a Nest.js application. 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` ## Install OpenTelemetry packages To perform the instrumentation, install the following OpenTelemetry packages. ```bash npm install --save @opentelemetry/api @opentelemetry/instrumentation @opentelemetry/tracing @opentelemetry/exporter-trace-otlp-http @opentelemetry/resources @opentelemetry/semantic-conventions @opentelemetry/auto-instrumentations-node @opentelemetry/sdk-node ``` To know more about them, you can check: ## Setup auto-instrumentation using OpenTelemetry Add the necessary environment variables that can be obtained from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page. ```bash OTEL_EXPORTER_OTLP_ENDPOINT= OTEL_EXPORTER_OTLP_HEADERS="Authorization=" ```  Next, create a file named `instrumentation.ts` and add the following code: ```ts import { NodeTracerProvider, TracerConfig, } from "@opentelemetry/sdk-trace-node"; import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"; // Use BatchSpanProcessor for better performance import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; import { registerInstrumentations } from "@opentelemetry/instrumentation"; import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node"; import { SEMRESATTRS_DEPLOYMENT_ENVIRONMENT, SEMRESATTRS_SERVICE_NAME, } from "@opentelemetry/semantic-conventions"; import { Resource } from "@opentelemetry/resources"; const providerConfig: TracerConfig = { resource: new Resource({ [SEMRESATTRS_SERVICE_NAME]: "", [SEMRESATTRS_DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV, }), }; // Initialize and register the tracer provider const provider = new NodeTracerProvider(providerConfig); const otlp = new OTLPTraceExporter(); provider.addSpanProcessor(new BatchSpanProcessor(otlp)); provider.register(); // Automatically instrument NestJS (additional instrumentations can be added similarly) registerInstrumentations({ instrumentations: [getNodeAutoInstrumentations()], }); ``` This above code performs the following steps: 1. Set up Trace Provider with the application’s name as Service Name. 2. Set up OTLP Exporter with Last9 OTLP endpoint. 3. Set up auto instrumentation. Next, the script should be “required” in the `main.ts` file to start the instrumentation. ```ts import { NestFactory } from "@nestjs/core"; import { AppModule } from "./app.module"; async function bootstrap() { require("./instrumentation"); // Add this line to start the instrumentation const app = await NestFactory.create(AppModule); await app.listen(3000); } bootstrap(); ``` Tip You can also use `instrumentation.js` if you are not using Typescript. ## Visualize the traces in Last9 Once the application runs with the above code, it pushes traces to Last9. You can see the result in action by looking at the APM dashboard in Last9.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Next.js > Send distributed traces to Last9 from an Next.js app using OpenTelemetry Next.js is a popular React framework that not only helps you generate Server Rendered React applications but also provides a great developer experience writing full-stack applications. ## Pre-requisites 1. You have a Next.js application. 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials: 1. `endpoint` - `https://otlp.last9.io` 2. `auth_header` - Obtain it from the Last9 dashboard. ## Install OpenTelemetry packages For instrumentation, Vercel provides a utility that can be used to instrument Next.js application with OpenTelemetry. To install the package, run the following command: ```bash npm install --save @vercel/otel ``` ## Setup auto-instrumentation using OpenTelemetry In your root directory create a file names `instrumentation.ts` and add the following code: ```ts import { registerOTel } from "@vercel/otel"; export async function register() { registerOTel(); } ``` The `registerOTel` function will have all the necessary OpenTelemetry packages and it will automatically instrument your app. ## Add environment variables Add the necessary OpenTelemetry environment variables to your `.env.local` file: ```js OTEL_SERVICE_NAME=nextjs-app-service OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.last9.io OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic " OTEL_TRACES_EXPORTER=otlp ``` If you don’t have a `.env.local` file, create one using the following command in the root directory: ```bash touch .env.local ``` ## Start the application Start the Next.js application: ```bash npm run dev ``` ## Visualize the traces in Last9 Once the application runs with the above code, it will start pushing traces to Last9. You can see the result in action by looking at the APM dashboard in Last9.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Phoenix > Send distributed traces to Last9 from an Phoenix app using OpenTelemetry Phoenix is a web framework for the Elixir programming language. This comprehensive guide will help you instrument your Phoenix application with OpenTelemetry, seamlessly sending the traces to Last9. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/main/elixir/phoenix). ## Pre-requisites 1. You have a Phoenix application. 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials: 1. `endpoint` - `https://otlp.last9.io` 2. `auth_header` - Obtain it from the Last9 dashboard. ## Install OpenTelemetry packages To perform the instrumentation, add the following packages to the `mix.exs` file: ```elixir defp deps do [ {:opentelemetry, "~> 1.4.0"}, {:opentelemetry_api, "~> 1.3.0"}, {:opentelemetry_ecto, "~> 1.0"}, {:opentelemetry_exporter, "~> 1.7.0"}, {:opentelemetry_phoenix, "~> 1.2.0"} ] end ``` Now, run the following command to install the dependencies: ```bash mix deps.get ``` ## Setup instrumentation using OpenTelemetry Add the necessary environment variables that can be obtained from the Last9 cluster’s write endpoint section details. ```bash export OTEL_SERVICE_NAME=phoenix-app-service export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.last9.io export OTEL_EXPORTER_OTLP_AUTH_HEADER="Basic " ``` Next, in config/config.exs, add the following configuration: ```elixir config :opentelemetry, :processors, otel_batch_processor: %{ exporter: {:opentelemetry_exporter, %{ endpoints: [System.get_env("OTEL_EXPORTER_OTLP_ENDPOINT") || "http://localhost:4317"], headers: [{"Authorization", System.get_env("OTEL_EXPORTER_OTLP_AUTH_HEADER")}] }} } config :opentelemetry, :resource, service: %{name: System.get_env("OTEL_SERVICE_NAME")} config :opentelemetry_ecto, :tracer, repos: [:timeline, :repo] # Add the repositories you want to trace config :opentelemetry_phoenix, :tracer, service_name: System.get_env("OTEL_SERVICE_NAME") ``` Also, add the following in your `endpoint.ex` file: ```elixir defmodule PhoenixAppWeb.Endpoint do use Phoenix.Endpoint, otp_app: :phoenix_app # ... plug Plug.Telemetry, event_prefix: [:phoenix, :endpoint] # ... end ``` This is require by the `opentelemetry_phoenix` package to trace the Phoenix requests. ## Visualize the traces in Last9 Once the application runs with the above code, it will start pushing traces to Last9. You can see the result in action by looking at the APM dashboard in Last9.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Roda > Send traces to Last9 from a Roda app using OpenTelemetry [Roda](https://roda.jeremyevans.net/) is a routing tree web toolkit for Ruby. This guide will help you set up OpenTelemetry instrumentation for your Roda application. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/ruby/roda). ## Pre-requisites Before you begin, make sure you have: 1. A Roda application set up and running. 2. Signed up for [Last9](https://app.last9.io) and created a cluster. 3. Obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` These credentials will be necessary for sending traces to your Last9 cluster. ## Install OpenTelemetry Packages To instrument your Roda application with OpenTelemetry, you need to install several gems. Add the following lines to your `Gemfile`: ```ruby # Gemfile gem 'opentelemetry-sdk' gem 'opentelemetry-exporter-otlp' gem 'opentelemetry-instrumentation-all' ``` Then, run the following command to install the packages: ```bash bundle install ``` ## Set Environment Variables Create a `.env` file in the root directory of your application and add the following environment variables: ```bash OTEL_SERVICE_NAME=roda-api-service OTEL_EXPORTER_OTLP_ENDPOINT=https://{endpoint}/v1/traces OTEL_EXPORTER_OTLP_AUTH_HEADERS="Authorization=Basic {auth_header}" ``` ## Configure Roda Application Add the following code to your Roda application to initialize OpenTelemetry in a new file `instrumentation.rb`: ```ruby # instrumentation.rb require 'opentelemetry/sdk' require 'opentelemetry/exporter/otlp' require 'opentelemetry/instrumentation/all' OpenTelemetry::SDK.configure do |c| c.service_name = 'roda-api-service' c.use_all('OpenTelemetry::Instrumentation::Rack' => { url_quantization: -> (path, env) { path_parts = path.to_s.split('/') quantized_path = path_parts.map.with_index do |part, index| if index > 0 case part when /\A\d+\z/ ":id" # Supports: /user/123 -> /user/:id when /\A[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\z/i ":uuid" # Supports: /user/550e8400-e29b-41d4-a716-446655440000 -> /user/:uuid else part # Keeps other path parts unchanged: /api/v1/users -> /api/v1/users end else part # Keeps the root path unchanged: / -> / end end.join('/') # Examples of supported transformations for request path to span name: # /user/123 -> /user/:id # /api/v1/posts/abcd-1234-5678-efgh -> /api/v1/posts/:uuid # /products/456/reviews/789 -> /products/:id/reviews/:id # /categories/electronics/items/101 -> /categories/electronics/items/:id "#{env['REQUEST_METHOD']} #{quantized_path}" } }) end ``` Update your entry file such as `app.rb` to load the instrumentation: ```ruby # app.rb # Add following line to your Roda app require 'rack' require './instrumentation' class App < Roda # Your Roda app code here end # Add this line to your Roda app App.use ::Rack::Events, [OpenTelemetry::Instrumentation::Rack::Middlewares::EventHandler.new] ``` ## Verify the setup Start the Roda application and make some requests to it. You should see the application traces in your Last9 dashboard. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Ruby on Rails > Send distributed traces to Last9 from a Ruby on Rails app using OpenTelemetry Ruby on Rails is a server-side web application framework written in Ruby. This comprehensive guide will help you instrument your Ruby on Rails application with OpenTelemetry and smoothly send the traces to a Last9 cluster. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/ruby/rails-api). ## Pre-requisites 1. You have a Ruby on Rails application. 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` ## Install OpenTelemetry packages To install the required packages, add the following lines to your Gemfile: ```ruby gem 'opentelemetry-sdk' gem 'opentelemetry-exporter-otlp' gem 'opentelemetry-instrumentation-all' gem 'dotenv-rails', groups: [:development, :test] ``` Then, run the following command to install the packages: ```bash bundle install ``` ## Set the environment variables Create a `.env` file in the root directory of your application and add the following environment variables: ```bash OTEL_SERVICE_NAME=ruby-on-rails-api-service OTEL_EXPORTER_OTLP_ENDPOINT= OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic " OTEL_TRACES_EXPORTER=otlp ``` > Note: Replace `` with the URL encoded value of the basic auth header. ## Instrument your application In `config/initializers/opentelemetry.rb`, add the following code to instrument your application: ```ruby require 'opentelemetry/sdk' require 'opentelemetry/exporter/otlp' require 'opentelemetry/instrumentation/all' # Exporter and Processor configuration otel_exporter = OpenTelemetry::Exporter::OTLP::Exporter.new processor = OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(otel_exporter) OpenTelemetry::SDK.configure do |c| # Exporter and Processor configuration c.add_span_processor(processor) # Created above this SDK.configure block # Resource configuration c.resource = OpenTelemetry::SDK::Resources::Resource.create({ OpenTelemetry::SemanticConventions::Resource::SERVICE_NAME => 'ruby-on-rails-api-service', OpenTelemetry::SemanticConventions::Resource::SERVICE_VERSION => "0.0.0", OpenTelemetry::SemanticConventions::Resource::DEPLOYMENT_ENVIRONMENT => Rails.env.to_s }) c.use_all() # enables all instrumentation! end ``` This code snippet configures the OpenTelemetry SDK to use the OTLP exporter and enables instrumentation for all standard components of a Rails application. ## Run the application Start your Ruby on Rails application by running the following command: ```bash bin/rails server ``` ## Visualize data After running the Ruby on Rails app, you can visualize the traces in the Last9’s APM dashboard.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Sinatra > Send traces to Last9 from a Sinatra app using OpenTelemetry Sinatra is a lightweight web application framework written in Ruby. This comprehensive guide will help you instrument your Sinatra application with OpenTelemetry and smoothly send the traces to a Last9 cluster. You can also check out the example application on [GitHub↗](https://github.com/last9/opentelemetry-examples/tree/master/ruby/sinatra). ## Pre-requisites 1. You’ve a Sinatra application. 2. You have signed up for [Last9](https://app.last9.io), created a cluster, and obtained the following OTLP credentials from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page: * `endpoint` * `auth_header` ## Install OpenTelemetry packages To install the required packages, add the following lines to your Gemfile: ```ruby # Gemfile source 'https://rubygems.org' gem 'sinatra' gem 'dotenv' gem 'opentelemetry-sdk' gem 'opentelemetry-exporter-otlp' gem 'opentelemetry-instrumentation-all' ``` Then, run the following command to install the packages: ```bash bundle install ``` ## Set the environment variables Create a `.env` file in the root directory of your application and add the following environment variables: ```bash OTEL_SERVICE_NAME=sinatra-api-service OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.last9.io OTEL_EXPORTER_OTLP_HEADERS="Authorization=" OTEL_TRACES_EXPORTER=otlp ``` > Note: Replace `` with the URL encoded value of the basic auth header obtained from the [Integrations](https://app.last9.io/integrations?integration=OpenTelemetry) page. In `app.rb`, add the following code to load the environment variables: ```ruby # app.rb require 'dotenv/load' # Rest of the code... ``` Note You can also other mechanisms to load environment variables and configuration such as loading a YAML file. ## Instrument your application Create a new file `instrumentation.rb` and add the following code: ```ruby require 'opentelemetry/sdk' require 'opentelemetry/exporter/otlp' require 'opentelemetry/instrumentation/all' OpenTelemetry::SDK.configure do |c| c.service_name = 'sinatra-api-service' c.use_all() # enables all instrumentation! end ``` This code snippet configures the OpenTelemetry SDK to use the OTLP exporter and enables all instrumentation. We will require this file in `app.rb` to instrument the application. In `app.rb`, require the `instrumentation.rb` file: ```ruby # app.rb require_relative 'instrumentation' # Rest of the code... ``` ## Run the application Run the Sinatra application using the following command: ```bash ruby app.rb ``` ## Visualize the traces After running the Sinatra app, you can visualize the traces in the Last9’s APM dashboard.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Ubuntu > This document describes a sample setup for sending logs to Last9 via OpenTelemetry collector from Ubuntu instance ## Installing OpenTelenetry collector on Ubuntu instance Use Last9’s OpenTelemetry endpoint to ingest logs from Ubuntu instances using Otel Collector. ## Prerequisites: There are multiple ways to install the Otel Collector. One possible way is to use the package. Every Collector release includes APK, DEB, and RPM packaging for Linux amd64/arm64/i386 systems. ```bash sudo apt-get update sudo apt-get -y install wget systemctl wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.110.0/otelcol-contrib_0.110.0_linux_amd64.deb sudo dpkg -i otelcol-contrib_0.110.0_linux_amd64.deb ``` More installation options can be found [here](https://opentelemetry.io/docs/collector/installation/). ## Sample Otel collector Configuration: The default path for otel config is `/etc/otelcol-contrib/config.yaml`. You can edit it and update it using the configuration below. The configuration for operators is essential to extract the timestamp and severity. For JSON logs, you can use `json_parser` and use its keys for log attributes. For non-structured logs, use the `regex_parser`. The configuration provides a sample example of a JSON parser. ```yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 # Detailed configuration options can be found at https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver filelog: # File path to read logs from. You can add multiple files here. include: [/var/log/app/*.log] include_file_path: true operators: - type: json_parser - type: severity_parser parse_from: attributes.level mapping: critical: 50 error: 40 warning: 30 info: 20 debug: 10 processors: transform/add_timestamp: log_statements: - context: log statements: - set(observed_time, Now()) - set(time, Now()) attributes: actions: - key: test.name value: "jammytest" action: insert - key: deployment.environment value: "production" action: insert - key: otel.processed value: true action: insert batch: timeout: 1s send_batch_size: 1024 exporters: debug: verbosity: detailed otlp/last9: endpoint: headers: "Authorization": service: pipelines: logs: receivers: [filelog] processors: [attributes, batch, transform/add_timestamp] exporters: [debug, otlp/last9] ``` ## Run the otel collector using `systemctl` command ```bash sudo systemctl start otelcol-contrib sudo systemctl status otelcol-contrib sudo systemctl restart otelcol-contrib ``` ## Checking logs of otel collector ```bash sudo journalctl -u otelcol-contrib -f ``` This will enable logs from the Ubuntu machine to be sent to Last9. To see the data in action, visit the [Logs Dashboard](https://app.last9.io/logs). ## Verification Login to Last9 and visit the [Logs](https://app.last9.io/logs) panel.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Host Metrics > This document describes a sample setup for sending host metrics to Last9 via OpenTelemetry collector from Ubuntu host machines ## Installing OTEL Collector to scrape Ubuntu Host metrics This guide explains using the Last9 OpenTelemetry metrics endpoint to ingest metrics from Ubuntu using OpenTelemetry Collector. ## Prerequisites 1. Install Otel Collector. There are multiple ways to install the Otel Collector. One possible way is to use the package. Every Collector release includes APK, DEB, and RPM packaging for Linux amd64/arm64/i386 systems. 2. Tested the following configuration with Ubuntu 20.04/22.04 (LTS) versions. ## Installation Steps ### 1. Install OpenTelemetry Collector Tested on Ubuntu 22.04 (LTS): ```bash sudo apt-get update sudo apt-get -y install wget systemctl wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.110.0/otelcol-contrib_0.110.0_linux_amd64.deb sudo dpkg -i otelcol-contrib_0.110.0_linux_amd64.deb ``` For more installation options, refer to the [official documentation](https://opentelemetry.io/docs/collector/installation/). ## Configuration ### OpenTelemetry Collector Configuration Edit the configuration file at `/etc/otelcol-contrib/config.yaml`: ```yaml receivers: hostmetrics: collection_interval: 30s scrapers: cpu: metrics: system.cpu.time: enabled: true system.cpu.utilization: enabled: true system.cpu.logical.count: enabled: true memory: metrics: system.memory.usage: enabled: true system.memory.utilization: enabled: true load: metrics: system.cpu.load_average.1m: enabled: true system.cpu.load_average.5m: enabled: true system.cpu.load_average.15m: enabled: true disk: metrics: system.disk.io: enabled: true system.disk.operations: enabled: true filesystem: metrics: system.filesystem.usage: enabled: true system.filesystem.utilization: enabled: true network: metrics: system.network.io: enabled: true system.network.packets: enabled: true system.network.errors: enabled: true paging: metrics: system.paging.usage: enabled: true system.paging.operations: enabled: true processes: metrics: system.processes.count: enabled: true system.processes.created: enabled: true process: mute_process_user_error: true metrics: process.cpu.time: enabled: true process.cpu.utilization: enabled: true process.memory.usage: enabled: true process.memory.utilization: enabled: true process.disk.io: enabled: true process.threads: enabled: true process.paging.faults: enabled: true processors: batch: timeout: 5s send_batch_size: 10000 send_batch_max_size: 10000 resourcedetection/system: detectors: ["system"] system: hostname_sources: ["os"] transform/hostmetrics: metric_statements: - context: datapoint statements: - set(attributes["host.name"], resource.attributes["host.name"]) - set(attributes["process.command"], resource.attributes["process.command"]) - set(attributes["process.command_line"], resource.attributes["process.command_line"]) - set(attributes["process.executable.name"], resource.attributes["process.executable.name"]) - set(attributes["process.executable.path"], resource.attributes["process.executable.path"]) - set(attributes["process.owner"], resource.attributes["process.owner"]) - set(attributes["process.parent_pid"], resource.attributes["process.parent_pid"]) - set(attributes["process.pid"], resource.attributes["process.pid"]) exporters: debug: verbosity: detailed otlp/last9: endpoint: headers: Authorization: service: pipelines: metrics: receivers: [hostmetrics] processors: [resourcedetection/system, transform/hostmetrics, batch] exporters: [debug, otlp/last9] ``` Run the OpenTelemetry Collector: ```bash otelcol-contrib --config /etc/otelcol-contrib/config.yaml ``` ## Run the otel collector using `systemctl` command ```bash sudo systemctl start otelcol-contrib sudo systemctl status otelcol-contrib sudo systemctl restart otelcol-contrib ``` ## Checking logs of otel collector ```bash sudo journalctl -u otelcol-contrib -f ``` ## Verifying Metrics This will push the metrics from the host machine to Last9. To see the data in action, visit the [Grafana Dashboard](https://app.last9.io/grafana). *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Prodvana > Send Prodvana deployment change events to Last9 This document lists step-by-step instructions for setting up [Prodvana’s](https://www.prodvana.io/) Last9 integration to send deployment change events to Last9. Learn more about [Change Events in Last9](/docs/change-events/). ## Prerequisites 1. Create a Last9 cluster by following [Getting Started](/docs/onboard/) 2. Keep the following information handy after creating the cluster: * `$levitate_remote_write_url`: Last9’s Remote write endpoint * `$levitate_remote_write_username`: Cluster ID * `$levitate_remote_write_password`: Write token created for the cluster 3. Create a Prodvana account and set up an application deployment * You can find a sample application code for performing a deployment on Prodvana [here](https://github.com/last9/levitate-prodvana-integration-sample-app) ## Setup Last9 integration in Prodvana 1. Go to the Integrations page in the Prodvana Dashboard.  2. Click on Connect under the Last9 integration.  3. Follow the [Getting Started with API](/docs/getting-started-with-api/) guide and copy the org slug, access token and refresh tokens from the Last9 dashboard Tip Copy the write access and refresh tokens as Prodvana integration pushes deployment change events to Last9 which is a “write” operation. 4. Once the connection is successful, you will see a success message  ## Configure your Prodvana application To enable deployment change events to be sent to Last9 automatically when a deployment is triggered in Prodvana, add `annotations` to your Prodvana application config: ```yaml application: name: user-service # replace with your desired app name releaseChannels: - name: staging runtimes: - runtime: emulated-runtime - name: production runtimes: - runtime: emulated-runtime # can be the same runtime as the previous step preconditions: - releaseChannelStable: releaseChannel: staging - manualApproval: {} annotations: last9: # you can find this in your Last9 dashboard at: # https://app.last9.io/v2/organizations//datasources data_source: ``` The `data_source` is the Last9 cluster where you want to send the events. You can copy the `data_source_name` from Last9’s Data Sources page.  Whenever there is a deployment for your Prodvana app called `user-service` (or whatever your specified app name in the config is), it automatically sends deployment events in Last9. ## Visualize the deployment events All deployment events sent by Prodvana when a deployment happens will have the following labels: * application\_name * event\_name * event\_state * release\_channel * rollback * service\_name * service\_version The events are sent for both `start` and `stop` deployment operations.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Prometheus > Send data to Last9 using Prometheus Remote Write We make use of the Prometheus [remote-write](https://prometheus.io/docs/practices/remote_write/) feature to allow Prometheus to write to an external remote endpoint i.e., Last9’s remote-write endpoint. ## Prerequisites Create a Last9 cluster by following [Getting Started](/docs/onboard/). Keep the following information handy after creating the cluster: * `$levitate_remote_write_url` - Last9’s Remote write endpoint * `$levitate_remote_write_username` - Cluster ID * `$levitate_remote_write_password` - Write token created for the cluster ## Writing data to Last9 The information flow is as follows:  Just update the Prometheus configuration with the following changes for the`remote_write`: ```yaml remote_write: - url: "$remote_write_url" basic_auth: username: "$remote_write_username" password: "$remote_write_password" remote_timeout: 60s ``` After that, restart Prometheus, and data will be written to Last9. ## Using Prometheus Agent mode With Last9 as Long-term storage, you don’t need to store time series in your Prometheus. You can run Prometheus only in agent mode and stream data directly to Last9. The agent mode is limited to discovery, scrape, and remote write, which fits our needs nicely.  You don’t need to have the overhead of running full-fledged Prometheus anymore. Here is a sample `docker-compose.yaml` that runs Prometheus in agent mode. ```yaml services: prometheus: container_name: prometheus-01 image: prom/prometheus:v2.39.1 ports: - 9091:9090/tcp volumes: - ./prometheus.yaml:/etc/prometheus/prometheus.yaml command: - "--config.file=/etc/prometheus/prometheus.yaml" - "--enable-feature=agent" network_mode: host restart: always ``` The `prometheus.yaml` will need to have `remote_write`section as follows: ```yaml global: scrape_interval: 1m scrape_timeout: 30s scrape_configs: - job_name: prometheus-self static_configs: - targets: ["127.0.0.1:9090"] remote_write: - url: "$remote_write_url" basic_auth: username: "$remote_write_username" password: "$remote_write_password" remote_timeout: 60s ``` ## Verification The Last9 Cluster will start showing a summary of ingested samples.  ## Next steps Create a read token for your Last9 Cluster and follow our guide to [Configure Grafana](/docs/grafana-config/) to visualize the time series data getting sent to Last9. ## Scaling Prometheus Remote Write Read , where we talk about how to scale [Prometheus Remote write](https://last9.io/blog/what-is-prometheus-remote-write/) for Federated clusters, considering High Availability requirements. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Monitor RabbitMQ using Last9 > Send RabbitMQ metrics to Last9 This document lists step-by-step instructions for setting up monitoring for RabbitMQ using Last9. ## Prerequisites Create a Last9 cluster by following [Getting Started](/docs/onboard/). Keep the following information handy after creating the cluster: * `$levitate_read_url` - Last9’s Read endpoint * `$levitate_username` - Cluster ID * `$levitate_password` - Read token created for the cluster Ensure RabbitMQ is installed and running on each VM. You should have root or administrative access. Ensure that [rabbitmq\_prometheus plugin](https://www.rabbitmq.com/docs/prometheus#enable-rabbitmq_prometheus) is configured as per its documentation. ## Configure Prometheus Agent to Scrape Metrics ### Architecture  ### Setting up Prometheus Agent Install Prometheus Agent on a central server that can access all VMs running RabbitMQ. This script can be run on the server: ```bash #!/bin/bash # Function to determine OS architecture get_architecture() { architecture=$(uname -m) case $architecture in x86_64) arch="amd64" ;; aarch64) arch="arm64" ;; arm*) arch="armv7" ;; *) echo "Architecture $architecture is not supported by this script." exit 1 ;; esac echo $arch } # Define version and architecture VERSION="2.37.0" ARCH=$(get_architecture) # Download and install Prometheus Agent URL="https://github.com/prometheus/prometheus/releases/download/v$VERSION/prometheus-$VERSION.linux-$ARCH.tar.gz" wget $URL -O prometheus.tar.gz tar -xzf prometheus.tar.gz cd prometheus-$VERSION.linux-$ARCH # Move executables to your PATH sudo mv prometheus promtool /usr/local/bin/ sudo mkdir /etc/prometheus sudo mkdir -p /var/lib/prometheus sudo mv consoles/ console_libraries/ prometheus.yml /etc/prometheus/ # Create a service file for Prometheus echo "[Unit] Description=Prometheus Wants=network-online.target After=network-online.target [Service] User=$USER Restart=on-failure ExecStart=/usr/local/bin/prometheus \\ --config.file=/etc/prometheus/prometheus.yml \\ --enable-feature=agent [Install] WantedBy=multi-user.target" | sudo tee /etc/systemd/system/prometheus.service # Reload systemd to pick up the new service and start Prometheus Agent sudo systemctl daemon-reload sudo systemctl start prometheus sudo systemctl enable prometheus ``` ### Configure Scrape Targets Use either `ec2_sd_config` or `file_sd_config` to configure scrape targets. The example uses `file_sd_config` for simplicity. Create a file `targets.json` and add the following: Ensure that the targets are the Private IPs of the RabbitMQ cluster or instances. ```json [ { "targets": ["rbmq-vm-1:15692", "rbmq-vm-1:15692"], "labels": { "job": "rabbitmq" } } ] ``` ### Configure Last9 Remote Write Credentials into Prometheus Agent Add the following remote\_write configuration to the prometheus.yml file: ```shell echo "remote_write: - url: "$remote_write_url" basic_auth: username: "$remote_write_username" password: "$remote_write_password" write_relabel_configs: - source_labels: [__name__] regex: 'rabbitmq_(.*)'" | sudo tee -a /etc/prometheus/prometheus.y ``` Replace `remote_write_*` template variables with your Last9 Credentials that can be obtained by following this documentation ### Restart Prometheus Agent Restart Prometheus Agent to apply changes: ```plaintext sudo systemctl restart prometheus ``` ### Verification * Check Prometheus Agent’ `/targets` and `/graph` web interfaces to ensure it’s scraping the RabbitMQ metrics * Verify that metrics are being received in the remote storage This script-driven setup automates the deployment and configuration of RabbitMQ monitoring using Prometheus Agent. ### Dashboard Download the latest dashboard from [here](https://grafana.com/grafana/dashboards/10991-rabbitmq-overview/) to visualize the metrics.    *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # statsD > This document describes the ways in which a user can send ship metrics via statsD to Last9. [statsD](https://github.com/statsd/statsd) is one of the oldest and most common ways to ship your application or infra metrics to a backend time series database like Graphite, InfluxDB, etc. statsD comprises 3 basic concepts: 1. client: usually a client library that instruments an application to send metrics e.g. [statsd-kafka](https://github.com/airbnb/kafka-statsd-metrics2) 2. server: the statsd daemon accepting metrics from the `client` 3. backend: TSDB accepting metrics from the `server` ## Pre-requisites 1. You have created a Last9 cluster by following [the getting started guide](/docs/onboard/) and have the following info: 1. Remote write endpoint URL = `$LEVITATE_REMOTE_WRITE_URL` 2. Remote write username = `$LEVITATE_REMOTE_WRITE_USERNAME` 3. Remote write password = `$LEVITATE_REMOTE_WRITE_PASSWORD` 2. Clone [last9-integrations](https://github.com/last9/last9-integrations/) GitHub repo which contains the sample code ## statsD graphite backend method (recommended)  1. `statsd_client_application` pushes metrics to `statsd_server` 2. `statsd_server` has enabled built in backend [Graphite](https://github.com/statsd/statsd/blob/master/docs/backend.md) 3. `vmagent` listens on port 2003 to accept metrics in Graphite format 4. `vmagent` remote writes these metrics to Last9/ A complete example using this approach can be found [here](https://github.com/last9/last9-integrations/tree/master/levitate/remote-write/statsd/vmagent-graphite). **This is our recommended approach for pushing statsd metrics to Last9.** ## Relay method  1. `statsd_client_application` pushes metrics to `statsd_exporter` 2. `statsd_exporter` transparently relays the metrics to existing `statsd_server`, which writes them to the `statsd_backend` e.g. Graphite, InfluxDb. 3. `statsd_exporter` exposes the metrics on a `/metrics` port 4. `vmagent` scrapes these metrics and remote writes them to Last9 A complete example using this approach can be found [here](https://github.com/last9/last9-integrations/tree/master/levitate/remote-write/statsd/relay) ## Repeater method  1. statsd\_application pushes metrics to statsd\_server 2. statsd\_server enables [repeater](https://github.com/statsd/statsd/blob/master/docs/backend.md) mode to push metrics to existing `statsd_backend` and `statsd_exporter` 3. `statsd_exporter` exposes metrics on `/metrics` 4. `vmagent` scrapes these metrics and remote writes them to Last9 A complete example using this approach can be found [here](https://github.com/last9/last9-integrations/tree/master/levitate/remote-write/statsd/repeater) ## Comparison | Approach | Pros | Cons | | :--------------- | :---------------------------------------------------------------------- | :------------------------------------ | | Graphite backend | vmagent can push metrics to Last9, no config changes to the applicaiton | Requires changing statsd config | | Relay | No config changes to the statsd. | Requires changing application config. | | Repeater | No config changes to the application. | Requires changing statsd config. | ## Next steps Create a read token for your Last9 Cluster and follow our guide to [Configure Grafana](/docs/grafana-config/) to visualize the time series data getting sent to Last9. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Telegraf > How to configure Telegraf to send data to Last9 using the Prometheus Remote Write feature [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) is an open-source agent that collects metrics and data from various sources and sends them to other services, such as InfluxDB, Graphite, and Prometheus. This document describes how Telegraf can integrate with Last9. Last9 accepts [remote write metrics](https://last9.io/blog/what-is-prometheus-remote-write/) in Prometheus / [OpenTelemetry](https://last9.io/blog/what-is-opentelemetry/) format. Telegraf supports both push-based and pull-based methods for these formats. ## Pre-requisites 1. You have created a Last9 cluster by following the [getting started guide](/docs/onboard/) 2. Clone [last9-integrations](https://github.com/last9/last9-integrations/tree/master/levitate/remote-write/telegraf) GitHub repository contains the sample code with different approaches discussed in this article ## Push via vmagent (recommended)  1. A sample application sends metrics to `telegraf` 2. `telegraf` has an [InfluxDB output plugin](https://github.com/influxdata/telegraf/blob/release-1.14/plugins/outputs/influxdb/README.md), which send metrics to `vmagent` 3. `vmagent` [accepts InfluxDB format](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-influxdb-compatible-agents-such-as-telegraf) as input and outputs Prometheus format metrics to Last9 You can configure the `vmagent` URL as follows in the telegraf config file. ```shell ############################################################################### # OUTPUTS # ############################################################################### [[outputs.influxdb]] urls = ["http://vmagent:8429"] ``` A working sample application can be found [here](https://github.com/last9/last9-integrations/tree/master/levitate/remote-write/telegraf/influxdb-to-vmagent). **This is the recommended approach for pushing metrics from Telegraf to Last9.** ## Push via [Prometheus Remote Write](https://last9.io/blog/how-to-scale-prometheus-remote-write/) plugin  1. A sample application sends metrics to `telegraf` 2. `telegraf` has an [HTTP output plugin](https://github.com/influxdata/telegraf/blob/master/plugins/serializers/prometheusremotewrite/README.md), which supports `prometheusremotewrite` format to convert input metrics to Prometheus protobuf format and push them to [Last9](https://last9.io/) The changes to the telegraf config are as follows: ```shell ## Data format to output. data_format = "prometheusremotewrite" ## Additional HTTP headers [outputs.http.headers] Content-Type = "application/x-protobuf" Content-Encoding = "snappy" X-Prometheus-Remote-Write-Version = "0.1.0" ``` Here is a [sample application](https://github.com/last9/last9-integrations/tree/master/levitate/remote-write/telegraf/http-prometheusremotewrite) with this setup. ## Pull via Prometheus Client plugin  1. A sample application sends metrics to `telegraf` 2. `telegraf` has a [Prometheus output plugin](https://github.com/influxdata/telegraf/blob/master/plugins/outputs/prometheus_client/README.md), which exposes all input metrics on `/metrics` url 3. These metrics are scraped by `vmagent` which remote writes to Last9 This approach also requires a change in the outputs section in the telegraf config as follows. The `9273` port is used in the `vmagent` [config file](https://github.com/last9/last9-integrations/blob/master/levitate/remote-write/telegraf/prometheus-client/vmagent.yaml) which scrapes it for metrics. ```shell ############################################################################### # OUTPUTS # ############################################################################### # Configuration for the Prometheus client to spawn [[outputs.prometheus_client]] ## Address to listen on. listen = ":9273" ``` A complete example using this approach can be found [here](https://github.com/last9/last9-integrations/tree/master/levitate/remote-write/telegraf/prometheus-client). ## Comparison | Approach | Pros | Cons | | :------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Push via vmagent | Transparent relay from Telegraf to InfluxDB format to Last9. | None | | Push via Prometheus remote write | No need to run any additional scraping agent because of the direct push model. | Histogram-based metrics might be impacted if Telegraf pushes partial results in one chunk. This has been mentioned in [prometheusremotewrite](https://github.com/influxdata/telegraf/blob/master/plugins/serializers/prometheusremotewrite/README.md) plugin docs. | | Pull via Client plugin | Metrics are shipped to Last9 using the [standard scrape model](https://last9.io/blog/best-practices-using-and-writing-prometheus-exporters/). | It has an overhead to run `vmagent` or any other scraping tool additionally. | ## Next steps Create a read token for your Last9 Cluster and follow our guide to [Configure Grafana](/docs/grafana-config/) to visualize the time series data getting sent to Last9. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # vmagent > Send data to Last9 via VictoriaMetrics vmagent using Prometheus Remote Write protocol We make use of the VictoriaMetrics’ vmagent [remote-write](https://docs.victoriametrics.com/vmagent.html) to write to an external remote endpoint i.e. Last9’s remote-write endpoint. ## Prerequisites Create a Last9 cluster by following [Getting Started](/docs/onboard/). Keep the following information handy after creating the cluster: * `$levitate_remote_write_url` - Last9’s Remote write endpoint * `$levitate_remote_write_username` - Cluster ID * `$levitate_remote_write_password` - Write token created for the cluster ## Sending data to Last9 The information flow is as follows:  ## Setup Here is a sample`docker-compose.yaml` for the `vmagent` setup: ```yaml version: "3.5" services: vmagent: container_name: vmagent image: victoriametrics/vmagent ports: - 8429:8429 volumes: - vmagentdata:/vmagentdata - /var/tmp:/var/tmp - ./vmagent.yaml:/etc/vmagent/vmagent.yaml command: - "--promscrape.config=/etc/vmagent/vmagent.yaml" - "--remoteWrite.tmpDataPath=/var/tmp/vmagent/" - "--remoteWrite.maxDiskUsagePerURL=10737418240" - "--remoteWrite.url=$levitate_remote_write_url" - "--remoteWrite.basicAuth.username=$levitate_remote_write_username" - "--remoteWrite.basicAuth.password=$levitate_remote_write_password" restart: always network_mode: "host" volumes: vmagentdata: {} ``` Create `vmagent.yaml` to add a sample scrape config. Replace this scrape config with your desired config: ```yaml global: scrape_interval: 1m scrape_timeout: 20s # Check https://prometheus.io/docs/prometheus/latest/configuration/configuration for more details scrape_configs: - job_name: "vmagent" static_configs: - targets: ["localhost:8429"] ``` This will run the `vmagent` container, scrape the desired targets, and remote write metrics to Last9. ## Verification The Last9 Cluster will start showing a summary of ingested samples.  Tip Check out how to install VMAgent on Ubuntu as standalone serivce [here](/docs/how-to-setup-vmagent-on-ubuntu/). ## Next steps Create a read token for your Last9 Cluster and follow our guide to [Configure Grafana](/docs/grafana-config/) to visualize the time series data getting sent to Last9. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # What is Last9? > Everything you need to know about Last9’s single pane for high cardinality observability, the underlying telemetry data platform powering Last9, and the control plane built on top of it. ## What is Last9? Last9 is a high cardinality observability tool for all your telemetry types — logs, traces, and metrics in a single pane, with first-class telemetry lifecycle management and anomaly-detection-based alerting capabilities.  * **Open Standards:** Built for Cloud Native environment, Last9 is OpenTelemetry-native and Prometheus compatible, and also integrates with over 100 sources to collect data across all telemetry types. * **Single Pane:** Correlate and alert on your application & infrastructure metrics, events, logs, and traces in one place. Debug faster and get to the root cause without changing the context. * **Simple Pricing:** Our architectural choices and the decision to build on Open Standards allow us to pass on those savings to you. Predictable “pay for what you use” plan. No per-host billing. No per-user billing. No separate custom metrics billing. ## Telemetry Data Platform, not just a database Modern micro-service architectures demand granular observability, which traditional/legacy observability products find difficult to provide support for. Self-driving cars are only possible because of today’s high-definition cameras and other high-fidelity inputs. Similarly, that’s what engineering teams today and in the future need. Since 2020, Last9 has been focused on solving for high-cardinality metrics. Twelve of the [18 largest live-streaming events](https://www.streamingmediablog.com/2023/10/largestlivestreaminghistory.html) in the world have used Last9 for their observability — the peak being 59 million concurrent viewers on Diney+ Hotstar for the 2023 Cricket World Cup. From a highly performant and scalable time series warehouse, Last9 has now evolved into a telemetry data platform to add support for logs and traces as well — all powered by a first-class Control Plane to manage your telemetry, its settings, and lifecycle. Last9 has decoupled the UI from the telemetry data platform unlocking various use cases and experiences: * Last9’s native explorer UIs for correlated telemetry and first-class search and filtering experience along with LogQL and TraceQL support. * Teams used to and more comfortable with Grafana can use the embedded instance for metrics dashboarding or the Loki and Tempo plugins for logs and traces. * Access telemetry directly via APIs to power your third-party monitoring tools or to build custom visualizations in your products for your customers. ### Instrumentation Last9 is OpenTelemetry-native and fully Prometheus compatible. The telemetry data platform can ingest logs, traces, and metrics at scale from over 100 sources — from infrastructure like AWS and Cloudflare to applications built on JavaScript, Python, Ruby, and more. See our documented [list of integrations](/docs/integrations/) here. ### Control Plane A first-class developer experience to manage your telemetry, its settings, and lifecycle across Ingestion, Storage, Query, and Usage. [Read more about the Control Plane](/docs/control-plane/). * **Ingestion:** Pre-ingestion workflows allow you to manage incoming telemetry during runtime without having to make instrumentation changes or redeploy your application — configure rules for streaming aggregations, filtering, and routing. * **Storage:** Configure cold storage settings and rehydration indexes. * **Query:** Define macros for reusing complex queries, and configure notifications for scheduled search patterns. * **Usage:** Understand usage and metrics cardinality, and view slow query logs. ## Purpose-built for High Cardinality  Here’s what one of our customers said when we asked why they’re picking us over others: > You’ve cracked high cardinality at scale for metrics. We can trust you with logs and traces at scale as well. Solving for high cardinality metrics has enabled our customers to gain granular observability. But also allows us to provide Logs-to-Metrics and Traces-to-Metrics pipelines to power historical reporting and anomaly detection. [Read our guide](https://last9.io/guides/log-data-anatomy/) to the anatomy of Log Data to understand how high cardinality applies to logs and traces as well. * **High-fidelity observability:** Instrument your system for individual users, tenants, sessions, etc. without worrying about cardinality explosion. Unlock granular monitoring and alerting for use cases like tenant-level debugging, and use Changeboards for correlated health. * **Cardinality workflows and superior defaults:** Along with a starting quota of 20M time series per metric, tools like Cardinality Explorer enable you to understand impacted metrics and use Streaming Aggregation to control it without instrumentation changes. ## Operational & Change Intelligence  * `Coming Soon` **Operational Recommendations:** Understanding incoming telemetry and its historical patterns to suggest dashboards, alerting, additional instrumentation, and deviations in telemetry. * **One-click Dashboards and Alerting:** Quickstart your observability and improve coverage of your system with recommendations from our library of standard components for APM and infrastructure monitoring. * **Anomaly Detection and Change Events:** Pattern matching algorithms in Alert Studio that go beyond static thresholds to reduce alert fatigue. Track deployments and feature flags using Change Events. ## Industry Standards * **Deploy in your cloud:** Last9 is available as SaaS or BYOC/on-prem deployments. Take advantage of AWS and GCP marketplaces to ease procurement, avail cloud credits, and forget about egress costs. * **Managed to take the toil away:** We regularly handle 400M samples/min and 35M concurrent users (59M at peak) for one of our biggest customers. * **Compliance ready:** Last9 is SOC2 Type II approved and PCI ready, with access controls, multi-region support, and OAuth-based SSO. * **SLAs with clawback:** We commit to the 99.9% write and 99.5% read availability to our SaaS customers, with higher SLAs for enterprise customers. ## Get Started Follow our [Quick Start Guide](/docs/onboard/) to get set up and start sending data. Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or by [Email](mailto:cs@last9.io) if you have any questions. You can [schedule a demo](https://last9.io/schedule-demo/) with us. # Logs > Everything you need to know to start exploring Logs with Last9. ## Sending Logs to Last9 You can send a variety of logs to Last9 from different sources: * [Using OpenTelemetry](/docs/integrations-opentelemetry/) * and, [other integrations](/docs/integrations/) You can also view the list of integrations in the Last9 app as well under the [Logs category on the Integrations screen](https://app.last9.io/integrations?category=logs). ## Exploring Logs There are 3 primary ways of exploring Logs depending on specific use cases. * **Logs Explorer:** Use Last9’s native UI for logs with first-class search and filters in the Builder mode or the LogQL-compatible Editor mode for advanced queries. Quickly view related logs, traces, and metrics by clicking on any log line. [Learn more](/docs/logs-explorer/). * **Grafana Loki:** For people who are familiar with Loki, use Last9’s embedded instance of Grafana. [Learn more](/docs/grafana-loki-in-last9/). * **SQL Exploration:** Last9 also allows exploring raw logs data using Clickhouse SQL. This is useful for adhoc exploration of Logs for deeper analysis. Contact Last9 team to provision a custom SQL data source for you. ## Accelerated Queries Last9 accelerates queries when a query is made on following attributes. * Service * Severity For this reason, users are encouraged to filter by `Service` or `Severity` first when searching for long time ranges. Tip The Logs Explorer and Loki interface shows only `Service`, `Severity` and `Body` attributes as first filters by default for time ranges greater than 15 minutes. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Logs Explorer > Use Last9's native UI for logs with first-class search and filters — quickly view related logs, traces, and metrics by clicking on any log line. ## Using Logs Explorer You can start exploring logs by visiting [Logs Explorer](https://app.last9.io/logs) in Last9. The Log Explorer allows to filter logs by specific log attributes and resource info to slice and dice by various log dimensions, and view correlated telemetry. ### Builder Mode  Logs Explorer in Builder Mode with a dropdown to select/unselect additional columns Last9 provides a native experience to explore your logs without using any query language. * **Search:** * With auto-complete support, `service`, `severity`, and `body` are first-class attributes * Supported operators are `equal`, `not equal`, `contains`, and `does not contain` * Edit the same attribute chip to add multiple values * **Filters:** Apart from severity and service, attributes and resources from the log lines in the selected time window are also listed. Click on `only` or `all` next to a filter item for quick actions. * **Live Tail:** A live stream of logs matching any filters, if applied. Useful to debug any instant changes to while deploying, etc. We recommened applying filters to narrow the scope of the Live Tail, which also helps improve the overall performance. * **Time Picker:** Select an absolute or relative time range. You can also switch between timezones. * **Volume Chart:** A stacked bar chart for the selected time window, color-coded to severity of the log lines. * **Toggle Columns:** Click on the settings icon on the top right of the table to add dynamic columns based on log attributes. ### Editor Mode  Switch to Editor mode for advanced queries You can write complex queries, with aggregations, that are LogQL-compatible using the Editor mode. * LogQL auto-completion is supported. Auto-complete for aggregation functions is WIP. * Aggregation queries are visualized as timeseries instead of as a volume bar chart. * Switching from Builder to Editor will convert any existing search to LogQL, but not vice versa. * Queries in Editor mode are not auto-run. Please click on the Run Query button or use the cmd/alt + enter keyboard shortcut. Read more about [LogQL compatibility](/docs/grafana-loki-in-last9/#logql-compatibility) and supported functions. ## Log Details  Log Details side panel with Related Logs tab Clicking on a log line in Logs Explorer opens a side panel with additional context and information about the selected log line. * **Content**, with payload size and option to view as raw or JSON * **Attributes** of the selected log line * **Resource Info** of the selected log line * **Related Logs**, based on the service and other attributes of the selected log line * **Related Traces**, based on the service and other attributes of the selected log line * **Related Metrics**, visualizing CPU and memory utilization of the relevant container, instance, and pod resources ## Query Library  Query Library side panel with recent and saved queries * **Recent Queries:** This is a history of queries made by you. * **Saved Queries:** This is a list of queries saved by your team or you. Queries can be shared with the team or be kept private. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Query Builder > Advanced logs query builder mode to parse, filter, transform, and aggregate logs without writing LogQL ## Introduction The Logs Explorer Query Builder provides a visual interface for constructing log analysis queries through a series of stages or operation blocks. Each stage represents a specific data manipulation function.  ## Core Stages ### Filter The FILTER stage allows data filtering based on conditions using various operators: Supported Operators: * **=**: Exact match * **!=**: Not equal to * **contains**: String contains substring * **not contains**: String does not contain substring * **matches**: Matches a pattern/regular expression Parameters: * Field name: The field to apply the filter on * Operator: One of the supported operators * Value: The comparison value or pattern A single field and operator can have multiple values. These values are “ORed”. To get the equivalent of AND, use multiple filter stages. Examples: ```shell resource_source = "kubernetes/infosec/trufflehog" repository != "https://github.com/brightcove/trufflehog.git" message contains "error" user_agent not contains "bot" email matches ".*@company\.com" ``` ### Parse The PARSE stage supports multiple parsing methods: 1. **JSONP** * Parameters: * **from**: Source field to parse. Default is the log body * **fields**: Optional comma-separated list of fields to extract * Example: **json from body fields DetectorName as credential\_type** 2. **REGEXP** * Parameters: * **from**: Source field to apply regex on. Default is the log body * **pattern**: Regular expression pattern with one or more capture groups * **into**: Target field names, one for each capture group in the regex pattern * Examples: ```sql regexp from finding ((?i)client[_-]?id[=\s\'\"]+([\'\"]*(\w|<|>|;|${})+)) into client_id ``` ```sql regexp from finding ((?i)[^A-Za-z0-9]([A-Za-z0-9]{40}?)[^A-Za-z0-9]) into liveapikey ``` ### Transform The TRANSFORM stage provides several data transformation methods: 1. **splitByString** * Parameters: * **from**: Source field * **on**: Delimiter character * **select part**: Part number to select * **as**: Output field name * Example: ```sql splitByString from link on # select part 1 as real_link ``` 2. **replaceRegex** * Parameters: * **from**: Source field * **pattern**: Regex pattern to match * **with**: Replacement string * **as**: Output field name * Example: ```sql replaceRegex from user_email /[<>]/, with - as user_email ``` 3. **concat** * Parameters: * **from**: First field to concatenate * Additional fields/literals to concatenate * **as**: Output field name * Example: ```sql concat real_link, '#', line_number as link ``` 4. **if** * Parameters: * **Condition**: `isEqual`, `!isEqual`, `isEmpty`, `!isEmpty` * **then**: Value if condition is true * **else**: Value if condition is false * **as**: Output field name * Example: ```sql if !isEmpty client_id then client_id else finding as finding ``` ### Aggregate The AGGREGATE stage supports various statistical computations with optional grouping: 1. Zero or One Argument Functions: * **count** * Usage without argument: **count as total\_count** * Usage with argument: **count field\_name as field\_count** 2. Single Argument Functions: * **sum**: Calculate sum of values * **min**: Find minimum value * **max**: Find maximum value * **avg**: Calculate average * **median**: Find median value * **stddev**: Calculate sample standard deviation * **stddev\_pop**: Calculate population standard deviation * **variance**: Calculate sample variance * **variance\_pop**: Calculate population variance * Usage Syntax: **function\_name field as result\_name** 3. Two Argument Functions: * **quantile**: Calculate approximate quantile. Quantile value is between 0 to 1. * **quantile\_exact**: Calculate exact quantile. Quantile value is between 0 to 1. * Usage: **quantile(0.99, bytes\_transferred as bytes\_p99)** Additional Features: * **groupby**: Group results by specified fields. Similar to a SQL groupby * Example: **groupby user\_agent as user\_agent, service as service** #### Note: Each aggregation stage limits available fields to those defined in the `as` clause of the previous stage. ## Field Handling ### Field Aliasing Use the `as` keyword to create aliases for fields: ```plaintext SourceName as event_id Raw as finding ``` ### Nested Fields Access nested data using dot notation: ```json SourceMetadata.Data.Github.email SourceMetadata.Data.Github.repository SourceMetadata.Data.Github.file ``` ## Query Construction Best Practices 1. **Operation Order** * Start with FILTER operations to reduce data volume * Follow with PARSE operations to extract needed fields * Apply TRANSFORM operations last for final data shaping * Apply AGGREGATES operations last for final data shaping * Applying FILTER after TRANSFORM and AGGREGATE, is a common pattern 2. **Field Naming** * Use descriptive names for output fields * Maintain consistent naming conventions * Avoid special characters in field names 3. **Error Prevention** * Use **!isBlank** checks before transformations * Validate regex patterns before using them (You can use something like ) * Check field existence before accessing nested data 4. **Performance Optimization** * Keep time ranges reasonable * Use **=** on Service for much faster query executions * Filter early in the query chain * Prefer querying on pre-extracted attributes rather than operations on Body. Use the control plane to extract from body during ingestion. Note Switching from Builder to Editor doesn’t convert the query to LogQL. Both states are preserved, so you can switch modes without losing your query. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Data Tiering > Last9 supports automatic data tiering of metrics based on retention policies and query range limits which allows long term retention of time series data without any downsampling or rollups. Last9 supports automatic data tiering of the metrics based on retention policies. Leveraging this underlying storage capability, Last9 can store data for long-term durations without downsampling or rollups. It enables traffic shaping of time series data by creating access policies tailored for specific use cases. It is a fundamental differentiator, making Last9 a time series data warehouse built for long-term retention and superior performance. Each Data Tier has a retention period, which dictates how long it will retain the data, and a query range, which dictates how far in the past one can query. Additionally, its tier has a query depth, which decides how much one query is in a single query. ## Blaze Tier Ideal for Alert Manager, real-time, shallow consumption of data. Retains data for two hours. It has delta caching enabled for faster queries. ## Hot Tier Ideal for quick daily trends and impact history of recent changes and fixes. Retains data for six months. Best for exploratory workloads such as Grafana dashboards. ## Cold Tier Ideal for long-term data exploration and analysis. Last9 offers long-term retention of a year of data retention without any rollups or downsampling out of the box. This is customizable as per customers’ requirements and reliability requirements and can be extended even further. ***  You can access the tiers from a Last9 cluster’s data tiers tab. ## How does it work under the hood? Last9 automatically tiers the data. Users do not have to do anything special. You can follow the instructions in the Last9 onboarding guide and start sending metrics, which will be automatically tiered based on the retention policies. After that, you can create read tokens for specific tiers from the *Policies* section. More about reading data from different tiers can be found in the [Read data from Last9](/docs/grafana-config/) guide. # Maintenance Windows > Manage scheduled software upgrades on your Last9 clusters A maintenance window is a repeating window during which automatic maintenance is permitted. Maintenance windows give fine-grained control over when Last9 can perform scheduled software upgrades on your Last9 clusters. For example, you can prevent maintenance from happening before a critical event or make sure it happens on your weekdays instead of weekends. ## Configuration A maintenance window can be configured for each cluster from the cluster’s settings tab.   ## Important considerations * Maintenance windows will be considered for software upgrades, including deployment operations * The maintenance operation will run for a maximum of an hour * Critical security patches and bug fixes will be rolled out immediately without waiting for a maintenance window *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Managing High Cardinality > This document explains how Last9 provides visibility, superior defaults, and control levers to tame high cardinality ## Last9 Limits | Quota Type | Limit | Reset Period | Possible Actions | | -------------------------------------------- | -------- | ------------ | ------------------------- | | **Writes** | | | | | Per Time Series Cardinality | 1M | Per Hour | Can be raised on request | | Per Time Series Cardinality | 20M | Per Day | Can be raised on request | | Streaming Aggregation Cardinality | 3M | Per Hour | Can be raised on request | | Ingestion Concurrency | 20K | Per Second | Can be raised on Request | | Number of Metrics Aggregated in one Pipeline | 1 Metric | Per Query | Cannot be changed for now | | **Reads** | | | | | Time Series Scanned Per Query — Blaze Tier | 5M | Per Query | Cannot be changed | | Time Series Scanned Per Query — Hot Tier | 10M | Per Query | Cannot be changed | | Samples Scanned Per Query | 100M | Per Query | Cannot be changed | | Query Time Range | 35 Days | Per Query | Can be raised on request | Tip Even though Last9 can scale to large values of cardinality in terms of the ingest pipeline, large per-metric cardinalities have an adverse effect on read response-times. Beyond a daily cardinality of 3M time series per metric, query response times for a metric start degrading. It is advised to keep daily per-metric cardinality within this limit by using Last9’s streaming aggregation pipeline. Read more about using [Cardinality Explorer](/docs/cardinality-explorer/) to identify impacted metrics and labels and how to PromQL-powered [Streaming Aggregations](/docs/streaming-aggregations/) to reduce high cardinality metrics data. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Last9 MCP > Ask your agent to bring production context to your local environment, debug issues, and fix them. ## What is Model Context Protocol? [MCP](https://modelcontextprotocol.io/introduction) is an open protocol that standardizes how applications provide context to LLMs. Think of MCP like a USB-C port for AI applications. Just as USB-C provides a standardized way to connect your devices to various peripherals and accessories, MCP provides a standardized way to connect AI models to different data sources and tools. Using MCP, the agent in your preferred IDE (Cursor, Windsurf, VS Code) or Claude desktop app can talk to Last9 (or other various products) to fetch additional information. ## Why use Last9 MCP? The core idea was, how do we help with developer productivity? The Last9 MCP server lets developers bring in context of real-time production issues to their local dev environment and let the agent auto-suggest fixes to their codebase based on the production obeservability data. Gone are the days of “but this works on my machine”. Read more in our [launch blogpost](/blog/launching-last9-mcp-server/). ## Get Started ### Environment Variables The Last9 MCP requires the following environment variables: * `LAST9_BASE_URL`: (required) Last9 API URL from [OTel integration](https://app.last9.io/integrations?integration=OpenTelemetry) * `LAST9_AUTH_TOKEN`: (required) Authentication token for Last9 MCP server from [OTel integration](https://app.last9.io/integrations?integration=OpenTelemetry) * `LAST9_REFRESH_TOKEN`: (required) Refresh Token with Write permissions, needed for accessing control plane APIs from [API Access](https://app.last9.io/settings/api-access) ### Setup MCP 1. Send logs, metrics, & traces to Last9 using the [OpenTelemetry integration](https://app.last9.io/integrations?integration=OpenTelemetry) 2. Install the [Last9 MCP server](https://github.com/last9/last9-mcp-server) * Homebrew ```plaintext # Add the Last9 tap brew tap last9/tap # Install the Last9 MCP CLI brew install last9-mcp ``` * NPM ```plaintext # Install globally npm install -g @last9/mcp-server # Or run directly with npx npx @last9/mcp-server ``` 3. Connect to the Last9 MCP server * Claude 1. Open the Claude Desktop app, go to Settings, then Developer 2. Click Edit Config 3. Open the claude\_desktop\_config.json file 4. Copy and paste the server config to your existing file, then save 5. Restart Claude ```json { "mcpServers": { "last9": { "command": "/opt/homebrew/bin/last9-mcp", "env": { "LAST9_BASE_URL": "", "LAST9_AUTH_TOKEN": "", "LAST9_REFRESH_TOKEN": "" } } } } ``` * Cursor 6. Open Cursor, go to Settings, then Cursor Settings 7. Select MCP on the left 8. Click Add “New Global MCP Server” at the top right 9. Copy and paste the server config to your existing file, then save 10. Restart Cursor ```json { "mcpServers": { "last9": { "command": "/opt/homebrew/bin/last9-mcp", "env": { "LAST9_BASE_URL": "", "LAST9_AUTH_TOKEN": "", "LAST9_REFRESH_TOKEN": "" } } } } ``` * Windsurf 11. Open Windsurf, go to Settings, then Developer 12. Click Edit Config 13. Open the windsurf\_config.json file 14. Copy and paste the server config to your existing file, then save 15. Restart Windsurf ```json { "mcpServers": { "last9": { "command": "/opt/homebrew/bin/last9-mcp", "env": { "LAST9_BASE_URL": "", "LAST9_AUTH_TOKEN": "", "LAST9_REFRESH_TOKEN": "" } } } } ``` * VS Code > Note: MCP support in VS Code is available starting v1.99 and is currently in preview. For advanced configuration options and alternative setup methods, [view the VS Code MCP documentation](https://code.visualstudio.com/docs/copilot/chat/mcp-servers). 16. Open VS Code, go to Settings, select the User tab, then Features, then Chat 17. Click “Edit settings.json” 18. Copy and paste the server config to your existing file, then save 19. Restart VS Code ```json { "mcp": { "servers": { "last9": { "type": "stdio", "command": "/opt/homebrew/bin/last9-mcp", "env": { "LAST9_BASE_URL": "", "LAST9_AUTH_TOKEN": "", "LAST9_REFRESH_TOKEN": "" } } } } } ``` 4. Vibe ## MCP Tools > Tools are a powerful primitive in the Model Context Protocol (MCP) that enable servers to expose executable functionality to clients. Through tools, LLMs can interact with external systems, perform computations, and take actions in the real world. The agent will have to context to call these tools on its own. Currently supported tools/use cases: 1. **`get_exceptions`:** Retrieves server-side exceptions over a specified time range. Parameters: * `limit` (integer, optional): Maximum number of exceptions to return. Default: 20 * `lookback_minutes` (integer, recommended): Number of minutes to look back from now. Default: 60. Examples: 60, 30, 15 * `start_time_iso` (string, optional): Start time in ISO format (YYYY-MM-DD HH:MM:SS). Leave empty to use lookback\_minutes * `end_time_iso` (string, optional): End time in ISO format (YYYY-MM-DD HH:MM:SS). Leave empty to default to current time * `span_name` (string, optional): Name of the span to filter by 2. **`get_service_graph`:** Gets the upstream and downstream services for a given span name, along with the throughput for each service. Parameters: * `span_name` (string, required): Name of the span to get dependencies for * `lookback_minutes` (integer, recommended): Number of minutes to look back from now. Default: 60. Examples: 60, 30, 15 * `start_time_iso` (string, optional): Start time in ISO format (YYYY-MM-DD HH:MM:SS). Leave empty to use lookback\_minutes 3. **`get_logs`:** Gets logs filtered by optional service name and/or severity level within a specified time range. Parameters: * `service` (string, optional): Name of the service to get logs for * `severity` (string, optional): Severity of the logs to get * `lookback_minutes` (integer, recommended): Number of minutes to look back from now. Default: 60. Examples: 60, 30, 15 * `start_time_iso` (string, optional): Start time in ISO format (YYYY-MM-DD HH:MM:SS). Leave empty to use lookback\_minutes * `end_time_iso` (string, optional): End time in ISO format (YYYY-MM-DD HH:MM:SS). Leave empty to default to current time * `limit` (integer, optional): Maximum number of logs to return. Default: 20 4. **`get_drop_rules`:** Gets drop rules for logs, which determine what logs get filtered out from reaching Last9. 5. **`add_drop_rule`:** Adds a new drop rule to filter out specific logs at Last9 Control Plane Parameters: * `name` (string, required): Name of the drop rule * `filters` (array, required): List of filter conditions to apply. Each filter has: * `key` (string, required): The key to filter on. Only attributes and resource.attributes keys are supported. For resource attributes, use format: `resource.attributes[key_name]` and for log attributes, use format: `attributes[key_name]`. Double quotes in key names must be escaped * `value` (string, required): The value to filter against * `operator` (string, required): The operator used for filtering. Valid values: * “equals” * “not\_equals” * conjunction (string, required): The logical conjunction between filters. Valid values: * “and” 6. More coming soon… ## Demos [YouTube video player](https://www.youtube.com/embed/AQH5xq6qzjI?si=FTrs41Ph2PQkZ1Jq) [YouTube video player](https://www.youtube.com/embed/0M-XG63ILXk?si=yHXYMugA7v94DIaK) *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Prometheus alertmanager Migrations > Guide to migrating Prometheus alertmanager YAML to Last9 ## Migrating from alertmanager It is possible to migrate the alert rules YAML file compatible with Prometheus alertmanager to Last9 compatible alert configuration via an HTTP endpoint. `POST /v4/organizations/{org_slug}/entities/migrate/alertmanager` Request Body - The YAML config of alertmanager. Response - The YAML config is compatible with Last9. Example: ```bash # sample.yaml contains the above YAML curl -XPOST https://app.last9.io/api/v4/organizations/{org_slug}/entities/migrate/alertmanager \ --data-binary @sample.yaml \ -H "X-LAST9-API-TOKEN: Bearer $TOKEN" ``` Response: ```yaml entities: - name: payment service type: alert-manager external_ref: payment service-alert-manager-alert-manager entity_class: alert-manager indicators: - name: "EXPR: HighRequestLatency - breach" query: job:request_latency_seconds:mean5m{service="payment"} > 0.5 - name: "EXPR: HighRequestLatency - threat" query: job:request_latency_seconds:mean5m{service="payment"} > 0.1 alert_rules: - name: High request latency indicator: "EXPR: HighRequestLatency - breach" total_minutes: 10 bad_minutes: 10 greater_than: 0 - name: HighRequestLatency - threat indicator: "EXPR: HighRequestLatency - threat" total_minutes: 10 bad_minutes: 10 greater_than: 0 ``` *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Notification Channels > How to use Slack, PagerDuty, Opsgenie, webhooks, and email for getting alert notifications.  [Notification Channels](https://app.last9.io/alerting/notification-channels/) are destinations for Last9 to send alert notifications. We currently support Slack, PagerDuty, Opsgenie, Webhook, and Email integrations. Note Email as a channel is supported for [Scheduled Search](/docs/scheduled-search/) alerts only. ## Adding a Notification Channel 1. Navigate to [**Notification Channels**](https://app.last9.io/alerting/notification-channels/) and click **Add** 2. Provide the following details:  * **Channel Name**: Descriptive name to easily identify channel destination * **Channel**: Choose a channel from the supported integrations list * **Webhook/API Key/Email**: Provide the relevant details for the selected channel * **Slack:** See how to send messages using [Incoming Webhooks](https://api.slack.com/messaging/webhooks) * **PagerDuty:** See how to use the [PagerDuty Events V2 API](/docs/pagerduty-integration/) * **Email:** Enter the email IDs to be notified * **Opsgenie:** Enter the API key * **Webhook:** Enter the webhook to be triggered, and headers (like auth), if any * **Send Resolved:** Enable if you want to be notified when an alert has been resolved, useful for automation of incident management 3. If you’ve selected Slack as the channel, you can also send a test message to the configured channel by clicking on **Test Config**  4. Click on **Save** to enable this channel. This channel can now be used in Alert Groups or Scheduled Search to start receiving notification ## Usage of a Notification Channel  In [Notification Channels](https://app.last9.io/alerting/notification-channels/), you can quickly understand which configured channels are being used and by how many alert groups or scheduled searches. To view which alert groups or scheduled searches are using a particular channel, click on the relevant channel’s Usage link to view the list. From here, you can also jump to any of the alert groups or scheduled searches. *** ## Notification Payloads In case you want to further use the generated JSON payloads for custom incident details, automation and workflows, alert enrichment, or integrating with other tools, refer to the following mapping for PagerDuty and Opsgenie. ### PagerDuty | PagerDuty field | Type | Description | | ----------------------- | ---------------- | ------------------------------------------------------------------------ | | payload | object | | | payload.summary | string | Title for the incident | | payload.timestamp | timestamp | The ending time of this alert, in ISO 8601 format | | payload.severity | string | critical / warning for alerts marked as breach/threat in alert rule | | payload.source | string | Dedup key for the incident | | payload.component | string | Empty | | payload.group | string | Dedup key for the incident | | payload.class | string | Alert Rule Type | | payload.custom\_details | object | Described below | | routing\_key | string | PagerDuty integration key | | event\_action | string | ’trigger’ for active notifications, ‘resolve’ for resolved notifications | | dedup\_key | string | Dedup key for the incident | | client | string | ”Last9 Dashboard” | | client\_url | string | Link to health dashboard for the alert in Last9 | | links | array of objects | Empty array | | images | array of objects | Empty array | #### Custom Details * `alert_condition` - Condition set on alert. Static alerts, it is of the format.`expr > 10` based on the threshold configured. For pattern-based alerts, it is of the format `algo_type(tunable, expr)`. For example, for a high spike alert set with tunable 3, this would be `high_spike(3, expr)` * `algo_type` - Type of alert (`static_threshold`, `increasing_changepoint` etc) * `client_url` - Link to the health dashboard for this alert on Last9 * `description` - Description of the alert. If a description is provided while configuring the rule, it appears here. Otherwise, a default description based on the algorithm, indicator, and entity is shown * `start` - Starting time of this alert, in ISO 8601 format * `end`- Ending time of this alert, in ISO 8601 format * `expression` - Name of the indicator * `entity_name` - Entity name * `entity_type` - Entity type * `entity_team` - Entity team. Is `None` if not assigned * `entity_tier` - Entity tier. Is `None` if not assigned * `entity_workspace` - Entity workspace. Is `None` if not assigned * `entity_namespace` - Entity namespace. Is `None` if not assigned * `severity` - Severity of the alert (`breach`/ `threat`) * `notification_call` - Whether this alert is sent for the first time or repeated (`first`/ `repeat`) * `runbook` - Link to the runbook for this alert (has to be configured while setting up alert). This key is omitted if the runbook isn’t configured * If the entity under alert has `tags` associated with it, they are included in custom details as `tag_` = `true` * `time_in_alert` - Duration for which this alert was observed. E.g., 8 in 10 minutes. ### Opsgenie | Opsgenie field | Type | Description | | -------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | message | string | Title for the incident | | alias | string | Dedup key for the incident | | description | string | Description of the alert. If a description is provided while configuring the rule, it appears here. Otherwise, this field is omitted | | tags | array of strings | Tags associated with the entity | | actions | array of strings | \[“Debug”] | | details | object | Described below | | entity | string | null | | source | string | Last9 Dashboard | | note | string | A string description of the alert, along with the health dashboard link for the alert | | responders | array of objects | Not used | | visibleTo | array of objects | Not used | | priority | string | Not used | | user | string | Not used | #### Details * `alert_condition`: Condition set on alert. Static alerts, it is of the format.`expr > 10` based on the threshold configured. For anomaly alerts, it is of the format `algo_name(tunable, expr)`. For example, for a high spike alert set with tunable 3, this would be `high_spike(3, expr)` * `algorithm`: Type of alert (`static_threshold`, `increasing_changepoint` etc) * `component`: `null` * `last9_dashboard`: Link to the health dashboard for this alert * `expression`: Name of the indicator * `service`: Name and type of the entity * `source`: Dedup key for this incident * `entity_name`: Entity name * `entity_type`: Entity type * `entity_team`: Entity team. Is `None` if not assigned * `entity_tier`: Entity tier. Is `None` if not assigned * `entity_workspace`: Entity workspace. Is `None` if not assigned * `entity_namespace`: Entity namespace. Is `None` if not assigned * `severity`: Severity of the alert (`breach`/ `threat`) * `notification_call`: Whether this alert is sent for the first time or repeated (`first`/ `repeat`) * `runbook`: Link to the runbook for this alert (has to be configured while setting up alert). This key is omitted if the runbook isn’t configured ### Webhook | Field | Type | Description | | ----------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | routing\_key | string | The full generic webhook URL generated in Last9. This is used to send alerts to your configured endpoint | | event\_action | string | The type of event. Can be `trigger`, `acknowledge` or `resolve` | | dedup\_key | string | Deduplication key for correlating triggers and resolves. The maximum permitted length of this property is 255 characters | | payload.summary | string | A brief text summary of the event, used to generate the summaries/titles of any associated alerts | | payload.source | string | The unique location of the affected system, preferably a hostname or FQDN | | payload.severity | string | The perceived severity of the status the event is describing with respect to the affected system. This can be `critical`, `error`, `warning` or `info` | | payload.timestamp | timestamp | The time at which the emitting tool detected or generated the event | | payload.component | string | Component of the source machine that is responsible for the event, for example `mysql` or `eth0` | | payload.group | string | Logical grouping of components of a service, for example `app-stack` | | payload.class | string | The class/type of the event, for example `ping failure` or `cpu load` | | payload.custom\_details | object | Additional details about the event and affected system | | images | array of objects | List of images to include | | links | array of objects | List of links to include | #### Sample Payloads * Trigger ```plaintext { "payload": { "summary": "Scheduled Search Triggered - Dashboard logs", "timestamp": "2025-04-07T04:32:00.000+0000", "severity": "critical", "component": null, "source": "3eee4fb7:241fb9df-8d31-4f69-8122-38d16bbe16df:breach-static_threshold", "group": "3eee4fb7:241fb9df-8d31-4f69-8122-38d16bbe16df:breach-static_threshold", "class": "Static Threshold", "custom_details": { "description": "Scheduled Search Alert is Triggered. Please check the Last9 Dashboard for more details.", "severity": "Breach", "client_url": "https://app.last9.io/logs?logql=%7Bservice%3D%22dashboard%22%7D&from=1743999960&to=1744000320", "rule_name": "Dashboard logs", "telemetry": "logs", "query": "{service=\"dashboard\"}", "evaluation_frequency": "Every 5 Minutes", "alert_condition": "expr > 7", "metric_name": "last9_change_events" } }, "routing_key": "https://app.last9.io/api/v4/generic_webhook?email=apps@last9.io", "dedup_key": "3eee4fb7:241fb9df-8d31-4f69-8122-38d16bbe16df:breach-static_threshold", "images": [], "links": [], "event_action": "trigger", "client": "Last9 Dashboard", "client_url": "https://app.last9.io/logs?logql=%7Bservice%3D%22dashboard%22%7D&from=1743999960&to=1744000320" } ``` * Resolve ```plaintext { "routing_key": "https://alpha.last9.io/api/v4/generic_webhook?email=apps@last9.io", "dedup_key": "3eee4fb7:241fb9df-8d31-4f69-8122-38d16bbe16df:breach-static_threshold", "event_action": "resolve" } ``` *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Quick Start Guide > Onboard with Last9, create your first cluster, and start writing metrics data under just two minutes Last9 takes under two minutes to sign up and get started. After signing up, you can create a cluster, grab a remote write URL, and plug it into your existing Prometheus or OpenTelemetry agents, and send metrics to Last9. Here is a walkthrough to sign up, create your first Last9 cluster, and get a fully functional managed time series database and Grafana in under two minutes. 1. ## Sign up on Last9 Sign up via your Google/Microsoft Business account [here](https://app.last9.io/). Note that Last9 only allows work/business email accounts to sign up. Using personal emails is not allowed. You can use GitHub login for personal emails instead.   2. ## Set up Last9 Cluster  Provide details of the cluster you are creating: * Region → Asia/Mumbai(ap-south-1) OR Europe/Frankfurt(eu-west-1) OR North America/Virginia(us-east-2) * Cluster Name → Identify your cluster with a name * Description → Some additional info on what purpose this cluster serves  3. ## Copy the Read and Write Token Once you hit `Create`, Last9 automatically creates a token to access the metrics data. Just copy it and store in a secure place to be used in Prometheus or OpenTelemetry configurations.  4. ## Start writing data to Last9 Copy the Write Configuration and paste it to your Prometheus remote write config. Then, reload Prometheus to start sending data to your Last9 cluster.  5. ## Start reading data from Last9 Use Managed Grafana, part of Last9, or read our [How to configure Grafana for Last9](/docs/grafana-config/) guide.   With these simple steps, you have a fully operational Last9 cluster to be used as a metrics database. Invite your team members to Last9, create new clusters, or set up alerts. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # OpenAPM > OpenAPM is an Application Performance Monitoring toolkit based on Prometheus, and Grafana. ## What is OpenAPM The OpenAPM project was created to make monitoring the applications using Open Standards such as Prometheus and Grafana easier. It consists of auto-instrumentation libraries for popular languages, standard Grafana dashboards for visualization and alerting rules. ## Instrumentation OpenAPM supports auto-instrumentation of Node.js, Ruby, Golang and Python applications. Once the OpenAPM instrumentation package is set up in your application, you can scrape the metrics via the Prometheus agent and push it to Prometheus or a Prometheus-compatible long-term storage like [Last9](/docs/introduction/). Note The metrics can be sent to Prometheus using [Prometheus Remote Write Protocol](https://last9.io/blog/what-is-prometheus-remote-write/). ## Advanced Capabilities The OpenAPM instrumentation libraries support following additional capabilities over instrumenting RED metrics. ### Multi-Tenancy support * Extract tenant information from the URL and emit as a label * Additional labels and metadata can be extracted by using Regular expressions ### Default labels * Add any number of constant default labels apart from the request metrics ### Track application lifecycle events * Support for tracking [change events](/docs/change-events/) such as `application_started`. ## Querying OpenAPM data is stored in Prometheus compatible systems, so one can use PromQL to query the data. ## Visualization OpenAPM collection ships with ready to import Grafana dashboards for applications to understand key performance indicators such as Apdex score, slow endpoints and slow database queries among many others. ### Steps to import default dashboard 1. Import [this](https://github.com/last9/openapm-nodejs/blob/main/APM-Grafana-Dashboard.json) dashboard into Grafana 2. Set up the data source to the one where metrics from OpenAPM are getting sent 3. Save the dashboard    ## Alerting OpenAPM comes with standard alert definitions that can be used to monitor the applications. ## Apdex score calculation Apdex stands for Application Performance Index. It is a measure of application performance with respect to user satisfaction. ### Why is the Apdex score a better measure of application performance? The definition of application performance varies as per different personas. For the engineer, it’s response time; for SRE, it’s uptime; for the product manager, it is user retention on the product. These definitions vary as per the roles or priorities of each role. In such cases, we have to find a uniform way to measure the application performance across multiple services, applications, and teams. How to do that? **Apdex** solves that problem by quantifying **user experience** to a number, as at the end of any business all roles are trying to find user experience by using different measurements. To define Apdex in terms of user experiences, we take three different categories of users. 1. **Satisfied**: One who enjoys the application experience without hiccups or slowness 2. **Tolerating**: One who faces a lag or slowness but keeps using the application without complaining 3. **Frustrated**: One who abandoned the application due to lag or slowness Based on application response time, the Apdex score measures satisfied, tolerating, and frustrated users. ### How to find an Apdex score for an HTTP application? * Define a satisfied user response time threshold, for example, a happy customer would get a response within 1 second * 4 times the satisfied user threshold defines the threshold for tolerating users. So, using the above example, anything from 1 second to 4 seconds defines the tolerating user response time. Anything above that defines frustrated user response time * Find the Apdex score by: ```text Apdex score = ( No. of satisfied users + (0.5) * No. of tolerating users + 0 * No. of frustrated users ) / Total users ``` For any application, the number of users can be quantified by the number of requests, which is throughput. Assumptions: * Application received 1000 requests in total * The satisfied user threshold is 1 second * No. of requests finished within 1 second = 600 * No. of requests finished within 1 to 4 seconds = 200 * No. of requests finished greater than 4 seconds = 200 Apdex Score will be: * Satisfied users: (600 \* 1 = 600) * Tolerating users: (200 \* 0.5 = 100) * Frustrating users: (200 \* 0 = 0) ```plaintext Apdex score = (600 + 100 + 0 ) / 1000 = 0.7 ``` ### PromQL to calculate Apdex score For calculating the Apdex score, OpenAPM utilizes following metrics: * `http_requests_duration_milliseconds_bucket` which denotes the latency of the application * `http_requests_duration_milliseconds_count` which denotes the throughput of the application * OpenAPM uses P50 as the measure of satisfied users. This is dynamic **threshold** which measures user experience in real time ```text histogram_quantile(0.50, sum by (le) (rate(http_requests_duration_milliseconds_bucket{}[4m])*60)) ``` * Satisfied Users as **satisfied\_users** ```plaintext sum( topk(1, sum by (le) (rate(http_requests_duration_milliseconds_bucket{}[4m]) * 60) and label_value(sum by (le)(http_requests_duration_milliseconds_bucket{}), "le") < threshold)) ``` * Tolerating Users as **tolerating\_users** ```plaintext sum( topk(1, sum by (le) (rate(http_requests_duration_milliseconds_bucket{}[4m]) * 60) and label_value(sum by (le)(http_requests_duration_milliseconds_bucket{}), "le") < 4 * threshold)) - sum( topk(1, sum by (le) (rate(http_requests_duration_milliseconds_bucket{}[4m]) * 60) and label_value(sum by (le)(http_requests_duration_milliseconds_bucket{}), "le") < threshold)) ``` * Total Users as **total\_users** which considers total requests ```plaintext sum (rate(http_requests_duration_milliseconds_count{}[4m]) * 60)) ``` * Final PromQL ```text Apdex score = ( satisfied_users + ( tolerating_users / 2 ) ) / total_users ``` The final PromQL to calculate the Apdex score is as follows. ```text ((sum(topk(1,sum by (le)(rate(http_requests_duration_milliseconds_bucket{program=~"$program", version=~"$version", environment=~"$environment"}[4m])*60) and label_value(sum by (le)(http_requests_duration_milliseconds_bucket{program=~"$program", version=~"$version", environment=~"$environment"}), "le") < histogram_quantile(0.50, sum by (le)(rate(http_requests_duration_milliseconds_bucket{program=~"$program", version=~"$version", environment=~"$environment"}[4m])*60)))) + (sum(topk(1,sum by (le)(rate(http_requests_duration_milliseconds_bucket{program=~"$program", version=~"$version", environment=~"$environment"}[4m])*60) and label_value(sum by (le)(http_requests_duration_milliseconds_bucket{program=~"$program", version=~"$version", environment=~"$environment"}), "le") < histogram_quantile(0.90, sum by (le)(rate(http_requests_duration_milliseconds_bucket{program=~"$program", version=~"$version", environment=~"$environment"}[4m])*60)))) - sum(topk(1,sum by (le)(rate(http_requests_duration_milliseconds_bucket{program=~"$program", version=~"$version", environment=~"$environment"}[4m])*60) and label_value(sum by (le)(http_requests_duration_milliseconds_bucket{program=~"$program", version=~"$version", environment=~"$environment"}), "le") < histogram_quantile(0.50, sum by (le)(rate(http_requests_duration_milliseconds_bucket{program=~"$program", version=~"$version", environment=~"$environment"}[4m])*60)))))/2) / sum (rate(http_requests_duration_milliseconds_count{program=~"$program", version=~"$version", environment=~"$environment"}[4m])*60)) ``` Tip You don’t need to remember this complex PromQL. Last9 clusters are equipped with [PromQL macros](/docs/promql-macros/) and you just have to use `apdex_score(0.5, "", http_requests_duration_milliseconds_bucket, http_requests_duration_milliseconds_count)` ### Explaining Functions used by the PromQL #### `label_value` ```plaintext label_value(sum by (le)(http_requests_duration_milliseconds_bucket{}), "le") < threshold ``` `label_value` is used to get any label value from a query. Whatever labels are coming as part of output series from query, we can select which label values we are interested in by providing it’s label as 2nd argument to **label\_value** Here threshold is latency (P50) so it must be either in seconds or milliseconds. On receiving label value **le** which gives time buckets, we are filtering all buckets which has value less than threshold. #### `and` operator ```plaintext sum by (le) (rate(http_requests_duration_milliseconds_bucket{}[4m]) * 60) and label_value(sum by (le)(http_requests_duration_milliseconds_bucket{}), "le") ``` `and` operator is used to finding intersection between first query and second query. Here it will give all those timeseries where ouput timeseries of first query is intersecting with output timeseries of second query. #### `topk` ```plaintext topk(1, sum by (le) (rate(http_requests_duration_milliseconds_bucket{}[4m]) * 60) and label_value(sum by (le)(http_requests_duration_milliseconds_bucket{}), "le") < threshold) ``` `topk` is used to get top k number of series with highest value from query. #### Understaing reason behind \[4m] The Counter is one of the metric types supported by Prometheus-compatible systems. The nature of these metrics is monotonically increasing. Instant values of any such metric are barely helpful. So, to find meaningful information, we have to find the rate of change of these metrics. Prometheus-compatible systems provide three functions to find the `rate` of change of counters: * rate * increase * irate To use any of the above functions, Prometheus needs at least 2 data points; otherwise, it will return the single point itself. The recommended duration that specifies a sufficient number of data points to be received is four times the scrape interval so that we have at least >= 2 data points. rate is per second average of metric value over the duration specified (for the example, it is \[4m], Prometheus finds this value by finding the slope from the start value to the end value for the duration. As Last9 is Prometheus compatible TDSB, same duration is used in the Apdex score calculation. ## Getting Started Follow the [quickstart guide for nodejs applications](/docs/openapm/nodejs/) to know more. # Monitor Node.js applications > Quickstart guide for monitoring Node.js applications using OpenAPM To quickly get started with the OpenAPM in your Node.js application, follow the steps below: ## Installation ```shell npm install --save @last9/openapm@latest ``` OR ```shell yarn add @last9/openapm@latest ``` ## Usage OpenAPM needs to be initialized in the application as early as possible so that it starts a metrics server and can monitor your application from the get-go. To initialize OpenAPM, import the `OpenAPM` class from the package and create its instance. This will create a metrics server running by default on `http://localhost:9097`. The metrics will be available on `http://localhost:9097/metrics`. ```js import { OpenAPM } from "@last9/openapm"; const openapm = new OpenAPM(); ``` Now, OpenAPM is ready to instrument the application packages using the `.instrument()` method. ## Instrumentation Currently, OpenAPM supports the following frameworks and databases. 1. [Express](#express) 2. [MySQL](#mysql) 3. [NestJS](#nestjs) Note We will keep adding support for more packages. Feel free to request a feature [here](https://github.com/last9/docs/issues/new?title=feat%3A). ### Express Call the `instrument` method before initializing the Express application as below. ```js openapm.instrument("express"); // Create application after the openapm initialization const app = express(); // ... app.listen(3000); ``` This will instrument all HTTP requests from the Express application and emit the **Rate, Errors, Duration (RED)** metrics. These metrics will be available on `http://localhost:9097/metrics`. ### MySQL OpenAPM supports instrumenting the [mysql2](https://www.npmjs.com/package/mysql2) package. This means all the ORMs that use the `mysql2` package internally will get instrumented. ```js openapm.instrument("mysql"); ``` The database query metrics will be exposed on the `/metrics` route on the metrics server. ### NestJS OpenAPM supports instrumenting [@nestjs/core](https://www.npmjs.com/package/@nestjs/core) v4 and above. ```js import { NestFactory } from "@nestjs/core"; import { AppModule } from "./app.module"; import { OpenAPM } from "@last9/openapm"; async function bootstrap() { const openapm = new OpenAPM(); openapm.instrument("nestjs"); const app = await NestFactory.create(AppModule); await app.listen(3000); } bootstrap(); ``` The **Rate, Errors, Duration (RED)** metrics for all the HTTP requests in the application will be emitted by default. ## Configuration You can customize the default behavior of OpenAPM by providing specific options described below. ```js const openapm = new OpenAPM({ // Options go here }); ``` 1. `path`: The path at which the metrics will be served. Defaults to `/metrics`. 2. `metricsServerPort`: (Optional) The port at which the metricsServer will run. Defaults to `9097`. 3. `environment`: (Optional) The application environment. Defaults to `production`. 4. `defaultLabels`: (Optional) Any default labels to be included. 5. `requestsCounterConfig`: (Optional) Requests counter configuration, same as [Counter](https://github.com/siimon/prom-client#counter) in `prom-client`. Defaults to ```js { name: 'http_requests_total', help: 'Total number of requests', labelNames: ['path', 'method', 'status'], } ``` 6. `requestDurationHistogramConfig`: (Optional) Requests Duration histogram configuration, the same as [Histogram](https://github.com/siimon/prom-client#histogram) in `prom-client`. Defaults to ```js { name: 'http_requests_duration_milliseconds', help: 'Duration of HTTP requests in milliseconds', labelNames: ['path', 'method', 'status'], buckets: promClient.exponentialBuckets(0.25, 1.5, 31), } ``` 7. `extractLabels`: (Optional) Extract labels from URL path params. ```js // To extract from the URL params { ... extractLabels: { tenant: { // Here, 'tenant' is the label name from: 'params', key: 'org' // Which key to extract from the params mask: ':org' // Replacement string } } } ``` 8. `excludeDefaultLabels`: (Optional) Provide labels to exclude from the default labels emitted by OpenAPM. ```js { ... excludeDefaultLabels: ['environment', 'version'] } ``` 1. `levitateConfig`: (Optional) Configuration for [Last9](https://last9.io/). Adding this configuration will enable the [Change Events](/docs/change-events/) such as application lifecycle events. ```js { ... levitateConfig: { host: 'https://play.last9.io', orgSlug: 'acme', /** The slug can be obtained from the Last9 dashboard.*/ dataSourceName: 'data-source', /** The data source can be obtained from the data source pages in the Last9 dashboard*/ refreshTokens: { write: '0d2a1a9aXXX' /** You can get this from the API access page on Last9 dashboard*/ } } } ``` ## Storing Metrics Metrics emitted by the OpenAPM library can be scraped by the Prometheus agent and sent to remote write storage like Last9. Follow the guide here to [know more](/docs/integrations-prometheus/). ## Visualization OpenAPM collection ships with ready-to-import Grafana dashboards for applications to understand key performance indicators such as Apdex score, slow endpoints and slow database queries, among many others. ### Steps to import default dashboards 1. Import [this](https://github.com/last9/openapm-nodejs/blob/main/APM-Grafana-Dashboard.json) dashboard into Grafana. 2. Set up the data source to the one where metrics from OpenAPM are getting sent. 3. Save the dashboard.    # Receive Alert Notifications via Pagerduty > Setup Pagerduty integration and receive alert notifications from Last9. ## Getting started Last9 can send alert notifications and resolutions to Pagerduty. This document lists the step by step instructions on how to setup Pagerduty integration with Last9 and start receiving alert notifications. Tip Last9 integrates with PagerDuty [Events V2 API](https://developer.pagerduty.com/docs/events-api-v2/overview/) to send alert notifications to PagerDuty. The Pagerduty Events V2 API is a highly reliable, highly available asynchronous API that ingests machine events from monitoring tools and other systems like Last9. Events sent to this API are ultimately routed to a PagerDuty service and processed. ## Setting up an Events API V2 integration in Pagerduty [Create an integration on any PagerDuty service](https://support.pagerduty.com/docs/services-and-integrations/#section-events-api-v2)  Select **Events API V2** as the Integration Type.  Copy the integration key and keep it handy as we will use it while creating a notification channel in Last9.  ## Setting up a notification channel in Last9 1. In [Notification Channels](https://app.last9.io/channels), add a new PagerDuty channel  2. Add the API key copied from Pagerduty integration in the `Integration Key` field  3. By default it is not assigned to any alert group. You can assign it to the alert group either via IaC flow or manually once alert rules are added to an alert group.  ## Assigning a notification channel to an alert group First navigate to the alert group and click on Pagerduty icon.  Select from the dropdown which Notification channel you want to select from the dropdown.  Tip You can set different Pagerduty channels for `threat` and `breach` notifications depending on the severity of the alerts. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Physical Indexes > How to create and use Physical Indexes to improve query performance and organize specific data in Last9 ## What are Physical Indexes? [Physical Indexes](https://app.last9.io/control-plane/physical-indexes) allow you to create specialized query spaces for specific log data based on custom filters you define. By narrowing the search scope, these indexes significantly improve performance for queries targeting specific data subsets. While Last9’s core value comes from providing correlated telemetry across your entire dataset, there are scenarios where isolating certain data can improve performance and workflow efficiency. ## When to Use Physical Indexes Physical Indexes are most valuable when: 1. **Team or Environment Isolation**: Separate production from staging data, or create team-specific indexes in organizations with siloed responsibilities. 2. **Service-Specific Workflows**: Create separate indexes logs from services that don’t typically require correlation within themselves or with other systems (like CDN logs, proxy logs). 3. **High-Volume Data Management**: Create dedicated indexes for extremely high-volume services that might otherwise impact query performance. 4. **Specialized Analysis**: Support focused workflows for security events, authentication logs, or error tracking. Tip The power of Last9 comes from correlated telemetry. Use Physical Indexes selectively where broad correlation isn’t essential to your analysis. ## Creating a Physical Index  Note Only account administrators can create and manage Physical Indexes. Contact [support](mailto:cs@last9.io) to enable this feature for your account. 1. Navigate to [Physical Indexes](https://app.last9.io/control-plane/physical-indexes) and click “New Physical Index” 2. Define filters to determine which logs will be included: * Logs are currently the only supported telemetry type * Configure filter conditions using attributes or resource attributes * Click on “View Logs” to confirm your filter works as expected 3. Specify index details: * Retention period matches your organization’s default retention, but can be changed by contacting [support](mailto:cs@last9.io) * Give a descriptive name (cannot be changed after creation) * Add an optional description (see [naming best practices](#index-naming-best-practices)) * Choose whether to also write logs to the default index (see billing impact in [important considerations](#important-considerations)) 4. Click “Create Physical Index” ## Index Naming Best Practices While there are no strict rules for naming indexes, consider these patterns for clarity and organization: * **Environment-based**: `prod-logs`, `staging-logs`, `dev-logs` * **Team-based**: `team-payments`, `team-auth`, `team-frontend` * **Service-based**: `service-cdn`, `service-api`, `service-db` * **Purpose-based**: `security-events`, `error-logs`, `performance-metrics` ## Important Considerations 1. **Billing Impact**: Logs stored in multiple indexes (both a Physical Index and the default index) are counted separately for billing purposes. Last9 billing is based on events stored, not deduplicated across indexes. 2. **Correlation Trade-offs**: Physical Indexes create separate query spaces. While this improves performance for targeted queries, it may limit correlation capabilities across your entire dataset. 3. **Retention Periods**: Physical Indexes share the same retention period as your organization’s default index. To modify retention for a specific index, contact support. 4. **Filter Design**: Create precise filters to include only the data you need. Overly broad filters may reduce the performance benefits of separate indexes. ## Example Use Cases ### Environment Separation Create separate indexes for production and non-production environments: ```plaintext Filter: resource.kubernetes.namespace.name == "production-*" Name: prod-environment ``` ### Team-Specific Indexes Isolate data for team-specific workflows: ```plaintext Filter: resource.service.name IN ("payment-service", "billing-service", "invoice-service") Name: team-finance ``` ### High-Volume Service Isolation Create dedicated indexes for services generating large volumes of logs: ```plaintext Filter: resource.service.name == "cdn-edge" Name: high-volume-cdn ``` ### Security Monitoring Isolate authentication and security events: ```plaintext Filter: attributes.log.level IN ("ERROR", "CRITICAL") AND attributes.event.type IN ("auth", "security") Name: security-events ``` *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Macros > Everything you need to know to start harnessing power of PromQL-based Macros. ## Why Macros Today, if you want to set a 1-day request SLO for your service`foo`, this is what your expression would like: ```text sum(rate(http_requests_to tal{handler!~"/metrics", method="GET, service="foo", code=~"2.*"}[1d])) / sum(rate(http_requests_total{handler!~"/metrics", method="GET, service="foo"}[1d])) ``` Now: * You want to set up a 7-day SLO as well * You want to ensure the same SLO is used across different services for their primary indicators. You can see how quickly things will get out of hand, and you’ll start running into the following issues: * Repetition of code * Reduced abstractions and readability * Error-prone large text blobs * Modifications are hard and tiresome To implement this fix across all consumers, dashboards, and alerts, it will take: * Weeks of migrations and trial and error to roll out * A week of distraction away from feature development The solution to this is Macros. They work in a way that is similar to how SQL developers use stored procedures. Macros take full advantage of the time-tested best practices of functions, abstractions, and reusability to replace cumbersome and error-prone methods. ## Configuration Template and Examples Macros are written as functions to which a prop is passed, can contain variables, and returns a PromQL. ```text function macro_name(props) { ... return promql } ``` Continuing with the above Availability PromQL example, here’s what an abstracted macro for it would look like: ```text function availability (metric, window, service) { let exclude = "/metrics" let good_codes = "2.*" let all_codes = "2.*|5.*" return sum(rate(metric{handler!~exclude, method="GET", service=service, code=~good_codes}[window])) / sum(rate(metric{handler!~exclude, method="GET, service=service, code=~all_codes}[window])) } ``` Now you can use it as.`availability(http_requests_total,7d,foo)` wherever needed. ## Setting Up Macros  1. Access the UI by navigating: Cluster → Settings → Macros 2. Define the macro function(s). See the [Configuration Template and Examples](/docs/promql-macros/#configuration-template-and-examples) section for reference 3. On clicking Save, we’ll check to ensure no syntax errors Tip Please allow up to 5 mins before the updated macros are available to query. Similarly, after deleting a macro, it may still be available for querying up to 5 mins. ## Calling the Macro Once the macro is defined, you can call it just like a function call by passing values. ```plaintext availability(http_requests_total,7d,foo) ``` Note that the Macro evaluation engine will replace all occurrences of variables inside the body of the Macro definition with the values passed when the Macro was called. This can be tricky when you use `by` clause inside the Macro definition uses the same variable names as the ones defined in the Macro definition. ```plaintext function availability (metric, window, service) { let exclude = "/metrics" let good_codes = "2.*" let all_codes = "2.*|5.*" return sum(rate(metric{handler!~exclude, method="GET", service=service, code=~good_codes}[window]) by (metric)) / sum(rate(metric{handler!~exclude, method="GET, service=service, code=~all_codes}[window])) } ``` In this case, the Macro engine will evaluate this to ```plaintext function availability (metric, window, service) { let exclude = "/metrics" let good_codes = "2.*" let all_codes = "2.*|5.*" return sum(rate(metric{handler!~exclude, method="GET", service=service, code=~good_codes}[window]) by ('http_requests_total')) / sum(rate(metric{handler!~exclude, method="GET, service=service, code=~all_codes}[window])) } ``` To mitigate this, you can rename the variables either in by clause or in the Macro definition. ```plaintext function availability (metric1, window, service) { let exclude = "/metrics" let good_codes = "2.*" let all_codes = "2.*|5.*" return sum(rate(metric{handler!~exclude, method="GET", service=service, code=~good_codes}[window]) by (metric)) / sum(rate(metric{handler!~exclude, method="GET, service=service, code=~all_codes}[window])) } ``` *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Logs Query API > Use Logs Query API for searching and retrieving logs programmatically from your services. Last9 provides a powerful API for querying logs from your services. This document explains how to use the Logs Query API to search and retrieve logs programmatically. The Logs Query API is LogQL and Loki compatible. For list of supported functions and parsers, [refer to this](/docs/grafana-loki-in-last9/#logql-compatibility). ## API Access You can find necessary credentials in the [OpenTelemetry](https://app.last9.io/integrations?category=all\&integration=OpenTelemetry) integration page. ### Endpoint The endpoint for querying logs is: ```plaintext GET /loki/logs/api/v2/query_range ``` ### API Host The API host is the same as your Last9 OpenTelemetry endpoint. For example: * `https://otlp.last9.io` * `https://otlp-aps1.last9.io` ### Authentication The API requires basic authentication using your organization credentials: * `$last9_username`: OTLP username * `$last9_password`: OTLP password ### Query Parameters The endpoint accepts the following parameters: | Parameter | Description | Example | | --------- | --------------------------------------------- | ------------------------- | | `query` | The LogQL query to search for logs | `{service="api-gateway"}` | | `start` | The start time of the query in Unix timestamp | `1743500000` | | `end` | The end time of the query in Unix timestamp | `1743510000` | | `limit` | The maximum number of logs to return | `100` | ## Example Queries ### Basic Query This example queries logs from a service named “api-gateway” with a limit of 100 results: ```bash curl -X GET 'https://otlp.last9.io/loki/logs/api/v2/query_range?query=%7Bservice%3D%22api-gateway%22%7D&start=1743500000&end=1743510000&limit=100' \ -H 'Authorization: Basic $(echo -n $last9_username:$last9_password | base64)' ``` Note: The query parameter is URL-encoded. `{service="api-gateway"}` becomes `%7Bservice%3D%22api-gateway%22%7D`. ### With Text Search This example searches for logs containing the word “error” from a specific service: ```bash curl -X GET 'https://otlp.last9.io/loki/logs/api/v2/query_range?query=%7Bservice%3D%22api-gateway%22%7D%20|%3D%20%22error%22&start=1743500000&end=1743510000&limit=50' \ -H 'Authorization: Basic $(echo -n $last9_username:$last9_password | base64)' ``` ### With Multiple Labels This example queries logs with multiple label conditions: ```bash curl -X GET 'https://otlp.last9.io/loki/logs/api/v2/query_range?query=%7Bservice%3D%22payment-service%22%2C%20env%3D%22production%22%7D&start=1743500000&end=1743510000&limit=100' \ -H 'Authorization: Basic $(echo -n $last9_username:$last9_password | base64)' ``` ## Response Format A successful response will return a JSON object with the following structure: ```json { "status": "success", "data": { "resultType": "streams", "result": [ { "stream": { "service": "api-gateway", "level": "info", "env": "production" }, "values": [ ["1743505000000000000", "Log message 1"], ["1743504990000000000", "Log message 2"], ["1743504980000000000", "Log message 3"] ] } ], "stats": { "summary": { "bytesProcessedPerSecond": 1048576, "linesProcessedPerSecond": 500, "totalBytesProcessed": 2097152, "totalLinesProcessed": 1000, "execTime": 0.25 } } } } ``` If no logs are found, the `result` field will be `null`: ```json { "status": "success", "data": { "resultType": "streams", "result": null, "stats": { "summary": { "bytesProcessedPerSecond": 0, "linesProcessedPerSecond": 0, "totalBytesProcessed": 0, "totalLinesProcessed": 0, "execTime": 0 } } } } ``` ## Advanced Usage ### Discovering Services To discover what service names are available in your logs, you can use the label values API: ```bash curl -X GET 'https://otlp.last9.io/loki/logs/api/v1/label/service/values?start=1743000000&end=1743600000' \ -H 'Authorization: Basic $(echo -n $last9_username:$last9_password | base64)' ``` This will return a list of all service names that have logs in the specified time range. ### Time Range Conversion Unix timestamps can be generated using various tools: * Current time: `date +%s` * Time from 1 hour ago: `date -d "1 hour ago" +%s` * Converting a specific date: `date -d "2025-04-01 12:00:00" +%s` ### Querying From Specific Indices By default, the API queries the default index. To query logs from a specific index, use the `index` parameter with the appropriate prefix: #### For Physical Indices ```bash curl -X GET 'https://otlp.last9.io/loki/logs/api/v2/query_range?query=%7Bservice%3D%22api-gateway%22%7D&start=1743500000&end=1743510000&limit=100&index=physical_index:Pt_prod_k8s' \ -H 'Authorization: Basic $(echo -n $last9_username:$last9_password | base64)' ``` #### For Rehydration Indices ```bash curl -X GET 'https://otlp.last9.io/loki/logs/api/v2/query_range?query=%7Bservice%3D%22api-gateway%22%7D&start=1743500000&end=1743510000&limit=100&index=rehydration_index:Rh_prod_archive' \ -H 'Authorization: Basic $(echo -n $last9_username:$last9_password | base64)' ``` The format for the index parameter is: * `physical_index:` for physical indices * `rehydration_index:` for rehydration indices ### Filtering by Log Level ```bash curl -X GET 'https://otlp.last9.io/loki/logs/api/v2/query_range?query=%7Bservice%3D%22api-gateway%22%2C%20level%3D%22error%22%7D&start=1743500000&end=1743510000&limit=100' \ -H 'Authorization: Basic $(echo -n $last9_username:$last9_password | base64)' ``` ### RegEx Queries ```bash curl -X GET 'https://otlp.last9.io/loki/logs/api/v2/query_range?query=%7Bservice%3D%22api-gateway%22%7D%20|~%20%22error.*timeout%22&start=1743500000&end=1743510000&limit=100' \ -H 'Authorization: Basic $(echo -n $last9_username:$last9_password | base64)' ``` ### Rate Limiting Be aware that the API may have rate limits to prevent abuse. If you’re making frequent queries, consider implementing backoff strategies in your application. ## Debugging Common Scenarios ### No Data Returned If your query returns no data (`"result": null`), check the following: 1. **Service name**: Verify the service name is correct. Service names are case-sensitive and must match exactly. 2. **Time range**: Ensure your `start` and `end` timestamps cover a period where logs exist. 3. **Data retention**: Check if the queried time range is within your organization’s data retention period. 4. **Query syntax**: Make sure your LogQL query is correctly formatted and URL-encoded. ### Invalid LogQL Error If you receive `{"detail":"parse error: invalid LogQL"}`, check: 1. **URL encoding**: Ensure all special characters in your query are properly URL-encoded. 2. **Query syntax**: Verify that your LogQL query follows the correct syntax. 3. **Quotation marks**: Make sure all quotation marks in your query are properly escaped. ### Parsing Time Range Error If you receive `{"detail":"parsing time range"}`, ensure: 1. **Valid timestamps**: Confirm that your `start` and `end` parameters are valid Unix timestamps. 2. **Chronological order**: The `start` timestamp must be earlier than the `end` timestamp. 3. **Required parameters**: Some API endpoints require both `start` and `end` parameters even for non-time-specific queries. ### Authorization Issues If you receive authentication errors: 1. **Credentials**: Verify your organization name and basic auth key are correct. 2. **Base64 encoding**: Ensure the authorization header is properly formatted with correct Base64 encoding. 3. **API access**: Confirm your organization has access to the logs API. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Query Tokens > Create and manage tokens for reading telemetry data from Last9 using external visualization tools like Grafana.  [Query Tokens](https://app.last9.io/control-plane/query-tokens) provide read-only access to your telemetry data for external visualization tools like Grafana, alerting systems, and custom applications. ## Creating Query Tokens  1. **Token Type**: * **Prometheus**: Currently the only available option for querying data 2. **Token Name**: * Use a descriptive name that identifies the token’s purpose * Examples: “Grafana Dashboard”, “Production Alerts” 3. Click **CREATE TOKEN** to generate your token *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Real User Monitoring (RUM) > Monitor your web application's performance from your users' perspective with Web Vitals and real-time insights.  Real User Monitoring (RUM) gives you real-time insights into your web application’s performance from your actual users’ perspective. Track Core Web Vitals, analyze user journeys, and identify performance bottlenecks that impact user experience. Unlike synthetic monitoring that tests from controlled environments, RUM captures how your application actually performs for real users across different devices, networks, and locations. This gives you the complete picture of user experience and helps you prioritize optimizations that matter most. ## What You Can Monitor 1. **Core Web Vitals**: Track Google’s official user experience metrics including Time to First Byte (TTFB), Largest Contentful Paint (LCP), First Contentful Paint (FCP), Cumulative Layout Shift (CLS), and Interaction to Next Paint (INP). 2. **User Traffic Patterns**: Understand when and how users interact with your application, identify peak usage periods, and see which pages receive the most traffic. 3. **Performance Distribution**: Analyze performance at different percentiles (P75, P90, P99) to understand how your application performs for different user segments. 4. **Real-Time Filtering**: Filter by specific pages, user attributes, environments, and application versions to drill down into performance issues. ## Key Benefits * **Identify Real Issues**: See performance problems that actually affect your users, not just lab conditions * **Prioritize Optimizations**: Focus on pages and issues that impact the most users * **Track Improvements**: Measure the impact of performance optimizations over time * **Environment Comparison**: Compare performance across different deployment environments * **Version Analysis**: Understand how new releases affect user experience ## Setup and Usage [code](/docs/rum-getting-started/) ## [Getting Started](/docs/rum-getting-started/) [Set up the RUM SDK in your React or vanilla JavaScript application](/docs/rum-getting-started/) [monitor\_heart](/docs/using-rum/) ## [Using RUM](/docs/using-rum/) [Navigate the dashboard, filter data, and analyze Core Web Vitals performance](/docs/using-rum/) *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Getting Started > Set up Last9's RUM SDK in your React or vanilla JavaScript application to start monitoring real user performance. Before you can view performance data in the RUM dashboard, you need to integrate the RUM SDK into your web application. The SDK automatically collects Core Web Vitals and user interaction data without impacting your application’s performance. ## Prerequisites 1. **Token**: A `Client — Web Browser` token from [Ingestion Tokens](https://app.last9.io/control-plane/ingestion-tokens), read [the docs](/docs/ingestion-tokens/) for more details. 2. **Endpoint**: A endpoint URL from the [RUM integration](https://app.last9.io/integrations?category=all\&integration=RUM) page ## Installation ### Latest RUM SDK Version **1.3.2** ### React Applications 1. Add the RUM SDK script to your `index.html` file in the `` section: ```html ``` 2. Initialize RUM in your application’s entry point: ```jsx useEffect(() => { // Initialize in your app's entry point L9RUM.init({ headers: { clientToken: "your-client-token", }, endpoint: "https://your-rum-endpoint", resourceAttributes: { serviceName: "your-app-name", deploymentEnvironment: "production", appVersion: "0.0.1", }, sampleRate: 50, }); }, []); ``` ### Vanilla JavaScript 1. Add the SDK script and initialization to your HTML: ```html ``` ## Configuration Options ### Required Settings These settings are mandatory for the RUM SDK to function: * **`headers.clientToken`**: Your client authentication token * **`endpoint`**: Your RUM endpoint URL * **`resourceAttributes.serviceName`**: Name of your application or service * **`resourceAttributes.deploymentEnvironment`**: Environment identifier (e.g., ‘production’, ‘staging’, ‘development’) * **`resourceAttributes.appVersion`**: Version identifier (e.g., ‘1.2.3’, git commit hash) ### Optional Settings * **`sampleRate`**: Percentage of user sessions to monitor (1-100, default: 40) Choose your sample rate based on environment and traffic volume: **Production Environments** * High Traffic (>10k daily users): 1-10% to manage data volume and costs * Medium Traffic (1k-10k daily users): 10-25% for good coverage * Low Traffic (<1k daily users): 25-50% for comprehensive insights **Non-Production Environments** * Staging: 50-100% for thorough testing before releases * Development: 100% for debugging and optimization Tip Start with a higher sample rate initially to gather baseline data, then adjust based on your data volume and analysis needs. * **`name`**: Root trace name (default: “View”) ## Automatic Data Collection Once initialized, the RUM SDK automatically collects: ### Browser and Device Information * User agent, browser name and version * Device type (desktop, mobile, tablet) * Screen dimensions and viewport size * Touch screen capability * Browser language and platform ### Network Information * Connection type (4g, 3g, etc.) * Effective connection speed ### Page Context * Entry page URL, path, and referrer * Page hostname and search parameters * Hash fragments and route information ### Core Web Vitals * Time to First Byte (TTFB) * Largest Contentful Paint (LCP) * First Contentful Paint (FCP) * Cumulative Layout Shift (CLS) * Interaction to Next Paint (INP) Each metric includes detailed attribution data to help you understand performance bottlenecks. ## Verification After implementing the SDK, verify it’s working correctly: 1. **Check Browser Console**: Look for any RUM SDK initialization errors 2. **Network Tab**: Verify traces are being sent to your Last9 RUM endpoint 3. **RUM Dashboard**: Visit and confirm data appears within a few minutes ## Next Steps With the RUM SDK installed and configured, you’re ready to start analyzing your application’s performance. Head over to the [Using RUM doc](/docs/using-rum/) to learn how to navigate the dashboard and interpret your performance data. *** ## Troubleshooting **No Data Appearing**: * Verify your `clientToken` and `endpoint` are correct * Check browser console for JavaScript errors * Ensure your sample rate isn’t too low **Missing Web Vitals**: * Some metrics only appear after user interactions (like INP) * LCP requires visible content elements * Check that your pages have sufficient content to trigger measurements **High Data Volume**: * Reduce your sample rate in production environments * Consider filtering specific pages or user segments Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Scheduled Search > Create periodic searches on logs data and set alerts when patterns are found or missing. [Scheduled Search](https://app.last9.io/control-plane/scheduled-search) allows you to configure periodic searches on your logs data and receive alerts based on specific conditions. You can set up alerts for both regular queries and aggregation queries. This capability is valuable for scenarios like: * Detecting error rates exceeding normal thresholds * Identifying unusual patterns in user behavior * Monitoring for missing scheduled jobs or backups * Watching for repeated failed login attempts that could indicate brute force attacks * Tracking performance degradation through aggregated metrics * Monitoring resource consumption trends across services ## Creating a Scheduled Search ### Via Logs Explorer [](/docs/videos/control-plane-scheduled-search-via-logs.mp4) 1. Run a query in [Logs Explorer](https://app.last9.io/logs), either in Builder or Editor mode 2. Click on `⋮` > “Save Query” in the top right 3. Enter a descriptive name for your query 4. Click “Save query & add alert” 5. Define Alert Configuration: * Review the query preview to confirm it’s correct * Set a numerical threshold with an operator (`>`, `<`, `=`, `>=`, `<=`) and a value * Select the appropriate evaluation frequency * Note: If your query uses a timeslice parameter, the evaluation frequency will automatically inherit this value * Select an alert destination from your notification channels * If no preconfigured channel exists, you can set one up via [Notification Channels](https://app.last9.io/alerting/notification-channels) ([How to setup a Notification Channel?](https://last9.io/docs/notification-channels/)) 6. View configured alerts in [Scheduled Search](https://app.last9.io/control-plane/scheduled-search) ### Via Saved Queries [](/docs/videos/control-plane-scheduled-search-via-saved-query.mp4) 1. In [Logs Explorer](https://app.last9.io/logs), click on “Query Library” > “Saved Queries” in the top right 2. Locate the query you want to set a Scheduled Search for 3. Click on “Add Alert” 4. Follow Step #5 onwards as in [Via Logs Explorer](#via-logs-explorer) ## Example Use Cases 1. **Monitoring API Error Rates:** This alerts you when any endpoint experiences more than 10 errors in a 5-minute window. * Query: `sum(count) by (endpoint) | filter status_code >= 500 | timeslice 5m` * Alert when: `sum > 10` 2. **Detecting Latency Spikes:** This alerts you when average response time for any service exceeds 500ms in a 1-minute window. * Query: `avg(response_time) by (service) | timeslice 1m` * Alert when: `avg > 500` 3. **Tracking Authentication Failures:** This alerts you when there are more than 20 authentication failures in a 10-minute period, which might indicate a brute force attack. * Query: `count | filter event_type = "auth_failure" | timeslice 10m` * Alert when: `count > 20` 4. **Monitoring Missing Backup Jobs:** This alerts you when no successful backup jobs have completed within a 24-hour period. * Query: `| filter job_type = "backup" AND status = "completed"` * Alert when: `No Results` * Evaluation frequency: `24h` ## Current Limitations * **Single Aggregate Only:** Scheduled Search cannot be configured on queries with multiple aggregates, as each aggregate is treated as a separate metric * **Default Index Requirement:** Currently, Scheduled Search can only be configured for queries on the Default Index *** ## Troubleshooting If your alerts aren’t triggering as expected: 1. Manually run your query to verify it returns data 2. Check that your threshold isn’t set too high or too low 3. Verify that your notification channels are correctly configured 4. Ensure your query syntax is valid and returns the expected type of results 5. For aggregation queries, confirm that your query is returning numerical values 6. Check your alert history to see if evaluations are happening but not crossing thresholds Need help? Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Setting up Docker and Docker Compose > A step by step tutorial to setup Docker and Docker Compose on Linux systems This document describes the steps for setting up `docker` and `docker-compose` on a Linux system. It supports Ubuntu and CentOS-like flavours. Create a file `/var/tmp/init-docker.sh` with the following contents: ```shell #!/bin/bash set -e # Install docker echo echo "==========================" echo "STATUS: docker" echo "==========================" # Can do command -v but better to be specific if [[ -f '/usr/bin/docker' ]]; then echo "STATUS: docker: installed" else echo "STATUS: docker: installing" set -ex curl -fsSL https://get.docker.com/ | sh -x command -v docker > /dev/null set +ex sudo groupadd docker sudo usermod -aG docker "$USER" echo "STATUS: docker: installed" newgrp - fi # Install docker-compose echo echo "==========================" echo "STATUS: docker-compose" echo "==========================" # Can do command -v but better to be specific if [[ -f '/usr/local/bin/docker-compose' ]]; then echo "STATUS: docker-compose: installed" else echo "STATUS: docker-compose: installing" set -ex sudo curl -L \ "https://github.com/docker/compose/releases/download/2.23.3/docker-compose-$(uname -s)-$(uname -m)" \ -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose command -v docker-compose > /dev/null set +ex echo "STATUS: docker-compose: installed" fi ``` Run this file as a non-root user who can have `sudo` privileges ```shell ./init-docker.sh ``` Enable docker to run at startup For Ubuntu: ```shell systemctl --user enable docker ``` For Centos / Amazon Linux ```shell sudo service docker start sudo chkconfig docker on ``` ## Optional hardening steps * Allow Docker processes to be long-running and not get terminated on User-shell logout Users cannot set user services to run at boot time by default. The admin must enable this on an individual basis for each user. From the [documentation](https://www.freedesktop.org/software/systemd/man/loginctl.html): > Enable/disable user lingering for one or more users. If enabled for a specific user, a user manager is spawned for the user at boot and kept around after logouts. This allows users who are not logged in to run long-running services. Takes one or more user names or numeric UIDs as argument. If no argument is specified, enables/disables lingering for the user of the session of the caller. ```shell sudo loginctl enable-linger $(whoami) ``` * Prevent Docker daemon crashes to terminate the processes. By default, when the Docker daemon terminates, it shuts down running containers. You can configure the daemon so that containers remain running if the daemon becomes unavailable. This functionality is called live restore. The live restore option helps reduce container downtime due to daemon crashes, planned outages, or upgrades # Streaming Aggregation > Transform incoming telemetry data into metrics to improve query performance, optimize storage costs, and manage high cardinality in Last9.  Order of Last9’s pipeline processing. [Streaming Aggregation](https://app.last9.io/control-plane/streaming) transforms incoming telemetry data into metrics to improve query performance and optimize storage costs. It processes data before storage, making it particularly effective for managing high cardinality metrics in time series data while preserving valuable information. For in-depth explanation of high cardinality challenges and conceptual details, check our [guide on Streaming Aggregation](/guides/high-cardinality/how-streaming-aggregation-works/). ## Common Use Cases ### Reducing cardinality while preserving information **Problem:** You have a metric with high-cardinality labels (`pod`, `pod_name`) alongside lower-cardinality labels (`instance`, `country`, `service`). **Solution:** Create two streaming aggregations: 1. One that drops high-cardinality labels but keeps lower-cardinality ones: ```yaml - promql: "sum by (instance, country, service) (my_metric{}[2m])" as: my_metric_by_location_2m ``` 2. Another that preserves only essential high-cardinality information: ```yaml - promql: "sum by (pod, pod_name, service) (my_metric{}[2m])" as: my_metric_by_pod_2m ``` This approach reduces cardinality while allowing you to correlate information between the two aggregated metrics. ### Transforming Logs to Metrics (Last9 LogMetrics) Last9 LogMetrics enables you to create metrics from your logs data, helping you extract key insights and metrics from log-based data sources. * **Monitoring Service Error Rates:** This creates a metric that shows how many errors each service is generating every 5 minutes, allowing you to quickly identify problematic services. * Filter: `level=error` * Aggregate: `count as _count` * Group by: `service` * Timeslice: `5 minutes` * **Tracking API Response Times:** This creates metrics showing average response times per endpoint and status code, helping to identify slow endpoints or failed requests. * Filter: `component=api` * Aggregate: `avg(response_time) as avg_response_time` * Group by: `endpoint, status_code` * Timeslice: `1 minute` * **Detecting Service Unavailability:** By tracking the volume of logs from critical services, you can detect when a service potentially goes down if log volume drops significantly. * Filter: `service in ("payment", "authentication", "database")` * Aggregate: `count as _count` * Group by: `service` * Timeslice: `10 minutes` ### Rolling up data over time Use streaming aggregation to roll up data over longer time windows: ```yaml - promql: "sum by (service, endpoint) (api_calls_total{}[5m])" as: api_calls_total_5m ``` This creates a 5-minute rolled-up version of your metric, which is useful for longer-term trend analysis while reducing storage requirements. ## Getting Started Last9 offers two main approaches to set up streaming aggregations: ### Option 1: Using the UI 1. Navigate to [**Control Plane → Streaming Aggregation**](https://app.last9.io/control-plane/streaming) 2. Click **+ NEW RULE** to open the rule creation form 3. Choose your **Telemetry** source: * **Metrics**: For metric-based aggregation * **Events**: For event-based aggregation * **Logs**: For creating metrics from logs ([Last9 LogMetrics](#transforming-logs-to-metrics-last9-logmetrics)) 4. **For Metrics & Events:** * Enter the metric name, resolution, and aggregation function * Choose labels to include (With) or exclude (Without) * Set the output metric name and rule name 5. **For Logs (Last9 LogMetrics):** * Use the Builder or Editor interface to create your query * Set filter conditions, aggregate functions, and group by dimensions * Define your evaluation frequency (timeslice), auto inherited from Editor mode query * Set the output metric name and rule name Note Currently, logs only from the default index is supported. Support for [Physical Indexes](/docs/physical-indexes/) is coming soon. You can also create LogMetrics directly from [Logs Explorer](https://app.last9.io/logs): * Create and run a query in Logs Explorer * Click the **Create Metric** button next to your visualization * This opens a pre-filled streaming aggregation form with your query * Verify the preview looks correct * Set the output metric name and rule name 6. Click **SAVE** to activate your streaming aggregation rule ### Option 2: GitOps Workflow Note GitOps Workflow only supports metrics-to-metrics pipeline for now. Please use the UI if you want to use the EventMetrics or LogMetrics pipelines as well. For teams who prefer infrastructure-as-code approaches: 1. **Request enabling GitOps workflow** Please reach out to [cs@last9.io](mailto:cs@last.io), or on our shared Slack/Teams channel, to switch you over to the GitOps workflow for Streaming Aggregation. 2. **Define Your Aggregation PromQL** Identify the specific metrics you want to aggregate. Using the [Explore tab in embedded Grafana](https://app.last9.io/explore/query), create a PromQL query that defines how you want to aggregate your data. For example, to aggregate HTTP request durations by stack: ```plaintext sum by (stack) (http_requests_duration_seconds_count{service="pushnotifs"}[1m]) ``` This query reduces cardinality by grouping data by the `stack` label, making it more manageable and queryable. 3. **Configure the Aggregation Rule** Add your aggregation rule to the YAML file for your Last9 cluster. The basic syntax is: ```yaml - promql: 'sum by (stack, le) (http_requests_duration_seconds_bucket{service="pushnotifs"}[2m])' as: pushnotifs_http_requests_duration:2m ``` This configuration: * Takes the metric `http_requests_duration_seconds_bucket` filtered for the `pushnotifs` service * Aggregates it over a 2-minute window * Groups by `stack` and `le` (latency buckets) * Creates a new metric named `pushnotifs_http_requests_duration:2m` 4. **Deploy Using GitOps Workflow** 1. **Create a Pull Request** with your updated rules to the GitHub repository 2. **Wait for CI Tests** to validate your streaming aggregation syntax 3. **Merge the Pull Request** to activate the pipeline in Last9 4. **Query the New Metric** in your Last9 cluster Caution Streaming aggregations are limited to a cardinality of 3M timeseries per hour. If a streaming aggregate generates more data than that, it will be skipped. This limit is *not* customizable. ## Histogram Aggregation for Percentiles Histograms are essential for calculating accurate percentiles (like p95 latencies). When aggregating histogram metrics, you need to maintain their statistical integrity by properly handling all three components. Histograms require three related metrics to function properly: `_bucket`, `_sum`, and `_count`. To create a proper histogram using streaming aggregation, define all three components: ```yaml - promql: 'sum2 by (stack, le) (http_requests_duration_seconds_bucket{service="pushnotifs"}[2m])' as: pushnotifs_http_requests_duration_seconds_bucket - promql: 'sum2 by (stack) (http_requests_duration_seconds_sum{service="pushnotifs"}[2m])' as: pushnotifs_http_requests_duration_seconds_sum - promql: 'sum2 by (stack) (http_requests_duration_seconds_count{service="pushnotifs"}[2m])' as: pushnotifs_http_requests_duration_seconds_count ``` Tip Use the `sum2` aggregator function when dealing with counters instead of `sum`. This aggregator handles counter resets properly. You can then use `histogram_quantile` functions on the aggregated metrics:  ## Supported Functions ### For Metrics The following aggregation functions are available for metric-based Streaming Aggregation: * **sum:** Total to be used for other metric types * **max:** The Maximum value of the samples * **sum2:** Sum, but for counters and reset awareness ### For Logs For log-based aggregations, see the supported functions in the [Logs Query Builder Aggregate Stage](/docs/logs-query-builder/#aggregate) documentation. Note that *only one aggregate function is allowed per query*. *** ## Troubleshooting 1. **Streaming Aggregation Not Appearing** * Check that your rule was properly saved or PR was successfully merged * Verify the syntax of your PromQL or log query * Check [Cardinality Explorer](https://app.last9.io/control-plane/cardinality-explorer) to Ensure the cardinality is below the 3M timeseries per hour limit * For LogMetrics, ensure your query is returning numerical data 2. **Incorrect Aggregation Results** * For counters, ensure you’re using `sum2` instead of `sum` * Check the time window `[Nm]` is appropriate for your data frequency * Verify that your `by` clause or group by includes all necessary labels * For logs, make sure your filter conditions are correctly specified 3. **Performance Issues** * Start with longer time windows like `[5m]` to reduce processing load * Limit the number of labels in your `by` clause * Consider creating multiple targeted aggregations instead of one large one * For log queries, add specific filters to narrow down the data being processed Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Traces Overview > Everything you need to know to start exploring traces with Last9. ## Sending Traces to Last9 You can send a variety of distributed traces to Last9 from different sources: * [Using OpenTelemetry](/docs/integrations-opentelemetry/) * and, [other integrations](/docs/integrations/) You can also view the list of integrations in the Last9 app as well under the [Traces category on the Integrations screen](https://app.last9.io/integrations?category=traces). ## Exploring Traces There are 3 primary ways of exploring Traces depending on specific use cases. * **Traces Explorer:** Use Last9’s native UI for traces with first-class search and filters. Quickly view dependencies with a waterfall time, span attributes and events, and related logs by clicking on any trace. [Learn more](/docs/traces-explorer/). * **Grafana Tempo:** For people who are familiar with Tempo, use Last9’s embedded instance of Grafana. [Learn more](/docs/grafana-tempo-in-last9/). * **SQL Exploration:** Last9 also allows exploring raw traces data using Clickhouse SQL. This is useful for adhoc exploration of Traces for deeper analysis. Contact Last9 team to provision a custom SQL data source for you. ## Accelerated Queries Last9 accelerates queries when a query is made on following attributes. * Service * Span Name For this reason, users are encouraged to filter by `Service` or `Span Name` first when searching for long time ranges. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Traces Explorer > Use Last9's native UI for tarces with first-class search and filters — quickly view dependencies with a waterfall timeline, span attributes and events, and related logs by clicking on any trace span. ## Using Traces Explorer You can start exploring traces by visiting [Traces Explorer](https://app.last9.io/traces) in Last9. The Traces Explorer allows to filter traces by specific traces attributes and resource info to slice and dice by various trace dimensions, and view correlated telemetry.  Traces Explorer with a distrubution heatmap of latency durations Last9 provides a native experience to explore your traces without using any query language. * **Search:** * With auto-complete support, `service` and `span` are first-class attributes * Supported operators are `equal`, `not equal`, `exists`, and `does not exist` * **Time Picker:** Select an absolute or relative time range. You can also switch between timezones. * **Duration Heatmap:** A heatmap of latency distribution. Click and drag to zoom into and filter out specific outliers. ## Trace Details  Trace Details with dependencies and waterfall timeline, and span attributes and events Clicking on a trace in Traces Explorer opens a side panel with additional context and information about the selected trace ID. * **Duration** and **no. of spans** * **Waterfall timeline** with dependencies * Click on the count pill to view any child spans * Click on a span to view span details * **Span Details** with: * Span Name * Span ID * Service Name * Attributes * Span Events * Links * **Span Context** with attributes, resource info, and related logs *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # User Access > Manage user access by creating an allowlist or a blocklist. While any user from your organization can sign up on Last9, we also provide the option of defining an allowlist or a blocklist to manage access. A more fine-grain control based on roles and features is in the works. Visit [User Access](https://app.last9.io/user-access) in the Last9 dashboard to manage these settings.  ## All Users (Default) Based on the email ID of the first user from your organization who signed up, everyone with an email ID matching the same root domain, will be added to the same account in Last9. The same very first user is also the associated admin of the account. You can also reach out to to either change the associated root domain or admin, or even add another root domain in case your organization uses multiple domains for emails. ## Allow Certain Users In case you want to allow only certain users, you can select this option. By default, the admin is auto-included in the allowlist. If no email IDs are added to the allowlist, only the admin will continue to have access. * If anyone not in the allowlist tries to signup or login, they will be prompted to reach out to the associated admin email. * Using the more menu in each row, an email ID can be removed from the list. * Switching to either All Users or Block Certain Users will delete the allowlist and it cannot be restored. Note: Only admins are allowed to to create an allowlist. ## Block Certain Users In case you want to block certain users, you can select this option. The admin cannot be included in the blocklist. If no email IDs are added to the blocklist, all email IDs will continue to have access. * If anyone in the blocklist tries to signup or login, they will be prompted to reach out to the associated admin email. * Using the more menu in each row, an email ID can be removed from the list. * Switching to either All Users or Allow Certain Users will delete the blocklist and it cannot be restored. Note: Only admins are allowed to to create a blocklist. ## Controlling access to Grafana Dashboards In case you want to restrict access to specific dashboards, please reach out to with the detailed requirement. Last9 Access Control layer allows restricting specific users to have access to certain dashboards based on demand. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Using OpenTelemetry Exporter for Prometheus Remote Write > Use OpenTelemetry Prometheus Exporter package in Python applications to send data to Prometheus-compatible systems such as Last9 You can push custom metrics to Prometheus-compatible backends, such as Last9, using OpenTelemetry Exporter for [Prometheus Remote Write](https://last9.io/blog/what-is-prometheus-remote-write/). This tutorial shows how to use the [Otel Exporter](https://pypi.org/project/opentelemetry-exporter-prometheus-remote-write/) with a Python application to send metrics to a Prometheus-compatible backend. Here is a sample Python application that includes the package for [Otel Expoter for Remote Write](https://pypi.org/project/opentelemetry-exporter-prometheus-remote-write/). ```python #!/usr/bin/env python import logging import random import sys import time import json from logging import INFO from opentelemetry import metrics from opentelemetry.exporter.prometheus_remote_write import ( PrometheusRemoteWriteMetricsExporter, ) from opentelemetry.metrics import Observation from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader logging.basicConfig(stream=sys.stdout, level=logging.INFO) logger = logging.getLogger(__name__) exporter = PrometheusRemoteWriteMetricsExporter( endpoint=sys.argv[1], basic_auth={ "username": sys.argv[2], "password": sys.argv[3], }, headers={}, ) reader = PeriodicExportingMetricReader(exporter, 1000) provider = MeterProvider(metric_readers=[reader]) metrics.set_meter_provider(provider) meter = metrics.get_meter(__name__) requests_counter = meter.create_counter( name=sys.argv[4], description="number of requests", ) testing_labels = json.loads(sys.argv[5]) num = random.randint(0, 1000) requests_counter.add(num % 131 + 200, testing_labels) logger.log(level=INFO, msg="completed metrics collection cycle") ``` The script accepts the following CLI arguments. 1. Remote Write URL 2. Username 3. Password 4. Metric name 5. Labels A complete example is as follows: ```shell python app.py LEVITATE_REMOTE_WRITE_URL LEVITATE_CLUSTER_USERNAME LEVITATE_CLUSTER_PASSWORD 'my_metric' '{"environment": "staging", "label": "9999"}' ``` This will insert the metric into the Remote Write backend, and we can confirm this by looking into Grafana.  The complete code example can be found [here](https://github.com/last9/last9-integrations/tree/master/levitate/remote-write/python). # Using RUM > Navigate the RUM dashboard, filter performance data, and analyze Core Web Vitals to optimize user experience.  [Real User Monitoring](https://app.last9.io/rum) provides comprehensive insights into your web application’s real-world performance. Learn how to navigate the interface, filter data effectively, and interpret Core Web Vitals metrics. ## Dashboard Overview The RUM dashboard is organized into three main views: * **Performance**: Core Web Vitals analysis and traffic insights * **Errors** (Coming Soon): JavaScript errors and failed requests * **Sessions** (Coming Soon): User journey and engagement analysis ## Filtering Your Data Effective filtering helps you focus on specific performance issues and user segments. ### Global Filters 1. **Application**: Choose which application to analyze when you have multiple services instrumented with RUM. 2. **Environment**: Filter by deployment environment to compare performance across production, staging, development 3. **Version**: Compare performance between different application versions to measure the impact of releases. 4. **Time Period**: Select your analysis timeframe in the top-right corner. The default is the last 30 minutes, but you can extend to hours or days for trend analysis. ### Path and Attribute Filtering Use the search bar to filter by specific pages or user attributes. #### Path Filtering Filter by URL paths using these operators: * **`=`** (equals): Exact path match * **`exists`**: Pages where the path field is present * **`not exists`**: Pages missing path information * **`starts with`**: Paths beginning with specific text * **`ends with`**: Paths ending with specific text * **`contains`**: Paths containing specific text * **`does not contain`**: Paths excluding specific text *Example*: `path starts with /api` to analyze API endpoint performance #### Attribute Filtering Filter by any collected attribute, including: **Page Attributes**: * **`origin`**: Page origin (protocol + hostname) * **`page.hash`**: Current page hash * **`page.hostname`**: Current page hostname * **`page.route`**: Current page route/path * **`page.search`**: Current page search/query string * **`page.url`**: Current page URL * **`url.path`**: Current page path * **`path`**: Folded path (for navigation) **Web Vitals Attributes**: For each web vital metric (CLS, FCP, LCP, TTFB, INP), the following attributes are collected (where applicable): * **`web_vital..id`**: Unique ID for the metric * **`web_vital..value`**: Value of the metric * **`web_vital..timestamp`**: Timestamp when the metric was recorded * **`web_vital..rating`**: Rating (e.g., ‘good’, ‘needs-improvement’, ‘poor’) * **`web_vital..delta`**: Delta value * **`web_vital..entries_count`**: Number of entries * **`web_vital..start_time`**: Start time (for TTFB) * **`web_vital..duration`**: Duration (for TTFB) * **`web_vital..fetch_start`**: Fetch start time (for TTFB) * **`web_vital..response_start`**: Response start time (for TTFB) * **`web_vital..request_start`**: Request start time (for TTFB) * **`web_vital..waiting_duration`**: Waiting duration (for TTFB, attribution) * **`web_vital..dns_duration`**: DNS duration (for TTFB, attribution) * **`web_vital..connection_duration`**: Connection duration (for TTFB, attribution) * **`web_vital..request_duration`**: Request duration (for TTFB, attribution) * **`web_vital.navigation_type`**: Navigation type (e.g., ‘reload’, ‘navigate’) ## Performance Analysis ### Overview Metrics **Views Over Time**: Track user traffic patterns throughout your selected time period. Use this to identify peak usage times, traffic spikes, or unusual drops that might indicate issues. **Top Paths by Views**: See which pages receive the most traffic. This helps you prioritize optimization efforts on high-impact pages. ### Web Vitals Each Web Vital can be analyzed at different percentiles (P75, P90, P99) to understand performance distribution across your user base. #### Time To First Byte (TTFB) * **Definition**: TTFB measures server responsiveness — the time between a user’s request and when the first byte of response arrives. * **Performance Thresholds**: * 🟢 **Good**: ≤ 800ms * 🟡 **Needs Improvement**: ≤ 1.8s * 🔴 **Poor**: > 1.8s * **What It Means**: High TTFB indicates server-side performance issues. This could be slow database queries, inefficient server processing, or network latency between your server and users. * **Optimization Focus**: Server performance, database optimization, CDN usage, caching strategies. #### Largest Contentful Paint (LCP) * **Definition**: LCP tracks when the main content becomes visible - specifically when the largest content element (image, video, or text block) finishes rendering. * **Performance Thresholds**: * 🟢 **Good**: ≤ 2.5s * 🟡 **Needs Improvement**: ≤ 4s * 🔴 **Poor**: > 4s * **What It Means**: LCP directly impacts perceived loading performance. Users judge page speed based on when they see the main content, not when everything finishes loading. * **Optimization Focus**: Image optimization, lazy loading, critical resource prioritization, server response times. #### First Contentful Paint (FCP) * **Definition**: FCP measures when users first see any content - the time until the first text, image, or other element appears. * **Performance Thresholds**: * 🟢 **Good**: ≤ 1.8s * 🟡 **Needs Improvement**: ≤ 3s * 🔴 **Poor**: > 3s * **What It Means**: FCP indicates how quickly users perceive your page is starting to load. Even if it’s just a small element, it signals that something is happening. * **Optimization Focus**: Critical CSS, font loading strategies, eliminating render-blocking resources. #### Cumulative Layout Shift (CLS) * **Definition**: CLS quantifies visual stability by measuring unexpected layout shifts during page loading. * **Performance Thresholds**: * 🟢 **Good**: ≤ 0.1 * 🟡 **Needs Improvement**: ≤ 0.25 * 🔴 **Poor**: > 0.25 * **What It Means**: High CLS creates frustrating experiences when elements move unexpectedly as users try to interact with your page. This often happens when images load without defined dimensions or ads insert dynamically. * **Optimization Focus**: Size attributes for images/videos, space reservation for dynamic content, font loading optimization. #### Interaction to Next Paint (INP) * **Definition**: INP measures interface responsiveness by tracking the time between user interactions (clicks, taps, keystrokes) and the next visual update. * **Performance Thresholds**: * 🟢 **Good**: ≤ 200ms * 🟡 **Needs Improvement**: ≤ 500ms * 🔴 **Poor**: > 500ms * **What It Means**: INP affects how responsive your application feels. Long delays between user actions and visual feedback make interfaces feel sluggish and unresponsive. * **Optimization Focus**: JavaScript optimization, reducing main thread work, efficient event handlers, code splitting. ## Best Practices * **Regular Monitoring**: Check your RUM dashboard weekly to catch performance regressions early. * **Focus on High-Impact Pages**: Prioritize optimization efforts on pages with high traffic and poor performance. * **Monitor All Percentiles**: Don’t just look at averages - P99 shows what your worst-performing users experience. * **Correlate with Deployments**: Use version filtering to understand how releases affect performance. * **Set Performance Budgets**: Define acceptable thresholds for each metric and monitor against them. *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions. # Visualizing Alerts > Visualizing Alerts Overview ## Visualization Alerts & Health You can visualize how all the Alert Rules in an Alert Group have performed for the last 14 days using the **Health** section ### Health  Each dot in this visualization represents one-hour of the day. The dots change color if there were alerts triggered in that hour. On the right side, you can see a timeline of all the alerts that were triggered for the selected dot. You can select one or multiple dots see the list of all alerts that were triggered in selected duration.  ### Visualizing Individual Alerts Clicking on an alert opens the Alert Visualization view. We also link to this view from notifications that sent on the alert channels. Every alert has its own unique URL, so you can safely use these in your RCAs or to share them with your team members  ### Alert Monitor Alerts that are firing indicate how a system performs at any given point. Last9 provides ways to get that information for the system’s current state. Alert Monitor highlights all alerts that are currently firing. Additionally, you can look back at all alerts firing in the last 15 minutes.  *** ## Troubleshooting Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:cs@last9.io) if you have any questions.