# Clusters

> Overview of Clusters

Source: https://last9.io/docs/cluster-overview/

## Cluster Overview

To start using Last9 you need at least one Cluster, from which you read and write metric data. In this document, we dive deep into all things related to a cluster.

To get up and running fast, see our [Quick Start Guide](/docs/onboard/).

Think of a Cluster as a logically separated, Prometheus API-compatible data source for all your metric data. You can create as many Clusters as you want, the number of clusters has no impact on your billing.

It is typically recommended that you create Clusters for each of your environments. Example: Production Cluster, Staging Cluster, etc.

## Creating a New Cluster

To create a Cluster:

1. Navigate to **Home** → **Levitate**
   ![Creating a Cluster 1](../../../../../assets/content/docs/levitate/overview/cluster/cluster-1.png)

2. Click the _Launch Cluster_ button to launch the setup wizard
   ![Creating a Cluster 2](../../../../../assets/content/docs/levitate/overview/cluster/cluster-2.png)

3. Select the AWS region you would like to deploy the cluster in. This should ideally be the same region as your application
4. Give the Cluster a descriptive name
5. Optionally, add a description which will be displayed on the Cluster Overview screen
   ![Creating a Cluster 3](../../../../../assets/content/docs/levitate/overview/cluster/cluster-3.png)
6. Press the **Create** to create your new Cluster
   As the Cluster gets created, you will be presented with an access token that is automatically created. This token is required to start writing & reading data to the Cluster. Tokens are only shown once, so please copy or download credentials (or you can always create another token from Cluster settings).
   ![Creating a Cluster 4](../../../../../assets/content/docs/levitate/overview/cluster/cluster-4.png)
   Your new Cluster is now ready to receive metrics.
7. To start writing data to this new Cluster, please follow the Write Data steps start writing data from Kubernetes, Prometheus, AWS/CloudStream or quickly try out by running a local demo environment
   ![Creating a Cluster 5](../../../../../assets/content/docs/levitate/overview/cluster/cluster-5.png)
   Using the **Test Config** button you can verify if your Last9 cluster has started receiving data.
   Click the **Next** button to start reading data/querying metrics from your Cluster.

   See our guides on how you can send data from [Prometheus](#), [OpenTelemetry](#), [VMAgent](#), or other various s[Integrations](#) supported. 8. To start reading metrics from your new Cluster you can use Managed Grafana which comes included with every plan.

   Alternatively, you can use the provided **Read URL** to read data using any Prometheus HTTP API Compatible tool like AlertManager, your own Grafana, KEDA, etc. See the guide on [how to connect your own Grafana](/docs/grafana-config/) with a Last9 cluster.
   ![Creating a Cluster 6](../../../../../assets/content/docs/levitate/overview/cluster/cluster-6.png)

---

## Managing a Cluster

### Cluster Usage and Performance

Last9 provides the following tools to observe the Cluster’s performance:

- [Cluster Health Dashboard](#cluster-health-dashboard) - Performance & usage metrics report
- [Query Logs](#query-logs) - Identify slow-running queries

#### Cluster Usage

![Cluster Usage](../../../../../assets/content/docs/levitate/overview/cluster/manage-cluster-1.png)
Usage for each cluster is reported in _Samples Ingested_. A **sample** refers to a single data point in a time series.

Usage for each Cluster can be viewed from the Cluster’s details page. For more granular and historical usage, see the Cluster Health dashboard’s Sample Ingested panel.

#### Cluster Quotas

There are no _per-cluster_ limits in Last9. You are billed for usage across all Clusters combined. The ingestion rate, read query rate, and data retention quotas are applied for all the data across all clusters.

#### Default Cluster Quotas

Last9’s default cluster quotas are fairly generous. In certain cases, keeping in mind performance and cost impacts, we may be able to increase a quota after a discussion with your team.

#### Write Quotas

| Type                                         | Base Quota | Reset Period | Note                     |
| -------------------------------------------- | ---------- | ------------ | ------------------------ |
| Per Time Series Cardinality                  | 1M         | Per Hour     | Can be raised on request |
| Per Time Series Cardinality                  | 20M        | Per Day      | Can be raised on request |
| Streaming Aggregation Cardinality            | 3M         | Per Hour     | Can be raised on request |
| Ingestion Concurrency                        | 20K        | Per Second   | Can be raised on request |
| Number of Metrics Aggregated in one Pipeline | 1 Metric   | Per Query    | Cannot be changed        |

#### Read Quotas

| Type                                       | Base Quota | Note                     |
| ------------------------------------------ | ---------- | ------------------------ |
| Time Series Scanned Per Query — Blaze Tier | 5M         | Cannot be changed        |
| Time Series Scanned Per Query — Hot Tier   | 10M        | Cannot be changed        |
| Samples Scanned Per Query                  | 100M       | Cannot be changed        |
| Query Time Range — Blaze Tier              | 2 Hours    | Can be raised on request |
| Query Time Range — Hot Tier                | 35 Days    | Can be raised on request |

If you wish to change your quotas, please raise a request by emailing us on: [support@last9.io](mailto:support@last9.io)

### Cluster Health Dashboard

Every Last9 Cluster comes with its own Health dashboard. To view the Health dashboard, navigate to the Cluster details page and click on the **View Health** link in the performance card.
![Cluster Health - 1](../../../../../assets/content/docs/levitate/overview/cluster/manage-cluster-2.png)

The following Cluster Performance Metrics are available in the health dashboard:
![Cluster Health - 2](../../../../../assets/content/docs/levitate/overview/cluster/manage-cluster-3.png)

- **Write Success** - Total successful write requests
- **Write Error** - Total failed write requests
- **Samples Ingested** - Total number of samples ingested
- **Write Availability** - Percentage of write requests successful
- **Write Latency** - Write request latency
- **Lag** - Pending samples waiting to be indexed (in bytes)
- **Read Success** - Total successful write requests
- **Read Errors** - Total failed read requests
- **Cardinality Limited** - Metrics whose cardinality has been limited
- **Read Latency** - Query Latency
- **Cardinality Limiter (Early Warning)** - Metrics whose cardinality is about to be limited
- **Bytes Dropped** - Samples permanently failed to be indexed (in bytes)

---

## Query Logs

Query Logs helps identify slow-running queries so that you can debug and optimize your PromQL.

Query Logs displays slow queries in the last 24 hours, which were successfully executed but have taken more than 1000ms (ie one second) to execute.
![Query Logs](../../../../../assets/content/docs/levitate/overview/cluster/query-logs-1.png)

When a slow query is identified the following details are displayed:

- **Timestamp** - Time when the query was executed
- **Query** - PromQL along with the query’s time range and query resolution step width
- **Latency** - approximate time taken for the query to execute
- **Token** Name - the name of the token used to query
- **Tier** - storage tier that was used for this query

---

## Cluster Settings

### Tokens

Tokens provide a mechanism for access management for your clients. We generate a default token when the Cluster is created for the first time

#### Creating a New Token

1. Navigate to the Cluster that you wish to create a token for: **Control Plane** → **Tokens**
   ![Create Token 1](../../../../../assets/content/docs/levitate/overview/cluster/create-token-1.png)
1. Click **New Token**
   ![Create Token 2](../../../../../assets/content/docs/levitate/overview/cluster/create-token-2.png)
1. Provide a descriptive **Token Name** the access **Scope** (Write Only, Read Only, Read & Write) and click **Create**
1. Copy the generated token since it will be visible only once. This token can now be used along with the Read or Write URL (depending on the Scope selected)
   ![Create Token 3](../../../../../assets/content/docs/levitate/overview/cluster/create-token-3.png)

#### Delete a Token

To delete/revoke a token:

1. Navigate to the Cluster that you wish to revoke a token from: **Control Plane → **Tokens\*\*
2. Click the **…** button and select Delete
   ![Delete Token 1](../../../../../assets/content/docs/levitate/overview/cluster/delete-token-1.png)

Note:

- This action cannot be undone, once deleted tokens cannot be recovered
- Tokens can only be deleted by your organization's admin

### Write & Read Data

Refer to the list of available [Integrations](/docs/integrations/) that can be used to start writing and reading data to a Last9 Cluster.

### Access Policies

Last9 has built-in data-tiering capabilities based on retention policies. Access policies let you define policies to control which token or client can query a specified data tier.

See our in-depth guide on how you can leverage this powerful feature - Guide on Access Policies

#### To define a new access policy:

1. Navigate to **Control Plane** → **Access Policies**
   ![Access Token 1](../../../../../assets/content/docs/levitate/overview/cluster/access-tokens-1.png)
   Every cluster comes with a default access policy pre-configured.
2. To define a new policy click the Create button
   ![Access Token 2](../../../../../assets/content/docs/levitate/overview/cluster/access-tokens-2.png)
   Provide the following details:

   - Policy Name: Give a descriptive name for this access policy
   - Token: Select a specific Token for which this access policy is applied or select _Any_
   - Query Client: We can identify traffic from known clients or select Any for the policy to apply from any client
   - Tier: Select the Tier from which the queries will be served for this policy

   And click **Create**

3. Your new access policy will be applied instantly
   ![Access Token 3](../../../../../assets/content/docs/levitate/overview/cluster/access-tokens-3.png)

#### To delete an Access Policy :

1. Select the **…** button beside the access policy you wish to delete
   ![Access Token 4](../../../../../assets/content/docs/levitate/overview/cluster/access-tokens-4.png)
2. Select **Delete** from the menu

Do Note:

- Access policies can only be deleted by the admin user(s) of your org
- Deleting an access policy may limit or lock access for a client or token, please be mindful before deleting

### Macros

Macros lets you define PromQL queries as reusable functions and use them as abstracted metric names across Grafana, Alert Manager, or the CLI

We cover how to define and use Macros in detail in [guide on PromQL Macros](/docs/promql-macros/)

#### Enabling Macros:

1. Navigate to **Control Plane** → **Macros**
   ![Macros 1](../../../../../assets/content/docs/levitate/overview/cluster/macros-1.png)
2. Write/Paste your Macro function and Click Save
   ![Macros 2](../../../../../assets/content/docs/levitate/overview/cluster/macros-2.png)
   We perform validation once you click Save
   ![Macros 3](../../../../../assets/content/docs/levitate/overview/cluster/macros-3.png)
   Once validated, we will save your Macro function. Do note that it will take upto 5 minutes for new Macros to be available for querying
   ![Macros 4](../../../../../assets/content/docs/levitate/overview/cluster/macros-4.png)

#### Deleting Macros:

1. Navigate to **Control Plane** → **Macros**
   ![Macros 5](../../../../../assets/content/docs/levitate/overview/cluster/macros-5.png)
2. Click the delete icon and click confirm

Note:

- Deleted Macros will impact any queries and dashboards where the macro functions were used
- Deleted Macros may be available for queries up to 5 minutes after they have been deleted

### Streaming Aggregation

Streaming aggregation is a powerful metric cardinality that is built-in with Last9.
Refer to our [Guide on Streaming Aggregation](/docs/streaming-aggregations/) for an in-depth tutorial