# Traces Query API

> Query and analyze distributed traces programmatically using the Last9 Traces API

Source: https://last9.io/docs/traces-query-api/

The Traces Query API enables programmatic access to your distributed tracing data stored in Last9. Use these endpoints to query traces, search for spans, manage saved searches, and retrieve trace metadata.

:::tip
This API is for querying trace data. For sending traces to Last9, see the [OpenTelemetry integration](/docs/integrations/observability/opentelemetry/).
:::

## Prerequisites

Before using the Traces Query API, you need:

1. An API access token (see [Getting started with API](/docs/getting-started-with-api/))
2. Your organization slug
3. Your data region (e.g., `us-east-1`)

### Base URL

```text
https://app.last9.io/api/v4/organizations/{org-slug}
```

Replace `{org-slug}` with your organization's unique slug.

### Authentication

All API endpoints require authentication via the `X-LAST9-API-TOKEN` header:

```text
X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>
```

---

## Query Traces

The primary endpoint for querying trace data with pipeline-based filtering.

```text
POST /cat/api/traces/v2/query_range/json
```

### Query Parameters

| Parameter   | Type    | Required | Description                          |
| ----------- | ------- | -------- | ------------------------------------ |
| `region`    | string  | Yes      | Your data region (e.g., `us-east-1`) |
| `start`     | integer | Yes      | Start time in Unix seconds           |
| `end`       | integer | Yes      | End time in Unix seconds             |
| `limit`     | integer | No       | Maximum number of results to return  |
| `order`     | string  | No       | Sort order (`asc` or `desc`)         |
| `direction` | string  | No       | Query direction                      |
| `mode`      | string  | No       | Query mode: `span` or `trace`        |

### Request Body

The request body contains a `pipeline` array with filter stages:

```json
{
  "pipeline": [
    {
      "type": "filter",
      "query": {
        "$and": [
          { "$eq": ["service.name", "api-gateway"] },
          { "$gte": ["http.status_code", "500"] }
        ]
      }
    }
  ]
}
```

### Example Request

```shell
curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/query_range/json?region=us-east-1&start=1727083606&end=1727087206&limit=100&mode=trace" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{
    "pipeline": [
      {
        "type": "filter",
        "query": {
          "$and": [
            { "$eq": ["service.name", "api-gateway"] }
          ]
        }
      }
    ]
  }'
```

### Response

```json
{
  "status": "success",
  "data": {
    "resultType": "stream",
    "result": [
      {
        "Timestamp": "2024-09-23T09:45:13.700906028Z",
        "TraceId": "cb9cd197d727655add1c12c4afe0b476",
        "SpanId": "83fca378bb34f8db",
        "ParentSpanId": "",
        "SpanName": "GET /api/v1/users",
        "SpanKind": "SPAN_KIND_SERVER",
        "ServiceName": "api-gateway",
        "ResourceAttributes": {
          "service.name": "api-gateway",
          "telemetry.sdk.name": "opentelemetry"
        },
        "SpanAttributes": {
          "http.method": "GET",
          "http.status_code": "200",
          "http.target": "/api/v1/users"
        },
        "Duration": 25748892,
        "StatusCode": "STATUS_CODE_UNSET",
        "StatusMessage": ""
      }
    ]
  }
}
```

---

## Search Traces

Search for traces matching specific criteria with span sets. This endpoint is useful for finding traces based on duration thresholds.

```text
POST /cat/api/traces/v2/search/json
```

### Query Parameters

| Parameter     | Type    | Required | Description                                     |
| ------------- | ------- | -------- | ----------------------------------------------- |
| `region`      | string  | Yes      | Your data region                                |
| `start`       | integer | Yes      | Start time in Unix seconds                      |
| `end`         | integer | Yes      | End time in Unix seconds                        |
| `limit`       | integer | No       | Maximum number of traces to return              |
| `span_limit`  | integer | No       | Maximum spans per trace                         |
| `order`       | string  | No       | Sort order                                      |
| `mode`        | string  | No       | Query mode: `span` or `trace`                   |
| `minDuration` | string  | No       | Minimum duration filter (e.g., `100ms`, `1s`)   |
| `maxDuration` | string  | No       | Maximum duration filter (e.g., `5s`, `10000ms`) |

### Example Request

```shell
curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/search/json?region=us-east-1&start=1727083606&end=1727087206&limit=50&minDuration=500ms" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"pipeline": []}'
```

### Response

```json
{
  "status": "success",
  "data": {
    "resultType": "stream",
    "result": [
      {
        "traceID": "7d7de75bd7d3d51cf29d73ad5e36a227",
        "rootServiceName": "api-gateway",
        "rootTraceName": "GET /api/v1/users",
        "startTimeUnixNano": 1727083606684633211,
        "endTimeUnixNano": 1727083606716391435,
        "durationMs": 31.758224,
        "spanSets": [
          {
            "matched": 1,
            "spans": [...]
          }
        ]
      }
    ]
  }
}
```

---

## Get Trace by ID

Retrieve all spans for a specific trace.

```text
GET /cat/api/traces/{traceID}
```

### Path Parameters

| Parameter | Type   | Description                   |
| --------- | ------ | ----------------------------- |
| `traceID` | string | The 32-character hex trace ID |

### Query Parameters

| Parameter | Type    | Required | Description                                       |
| --------- | ------- | -------- | ------------------------------------------------- |
| `region`  | string  | Yes      | Your data region                                  |
| `start`   | integer | Yes      | Start time in Unix seconds                        |
| `end`     | integer | Yes      | End time in Unix seconds                          |
| `limit`   | integer | No       | Maximum number of spans to return (default: 1000) |

### Example Request

```shell
curl -X GET \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/7d7de75bd7d3d51cf29d73ad5e36a227?region=us-east-1&start=1727083606&end=1727087206" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>"
```

### Response

```json
{
  "traces": [
    {
      "Timestamp": "2024-09-23T09:45:13.700906028Z",
      "TraceId": "cb9cd197d727655add1c12c4afe0b476",
      "SpanId": "83fca378bb34f8db",
      "ParentSpanId": "",
      "TraceState": "",
      "SpanName": "GET /api/v1/users",
      "SpanKind": "SPAN_KIND_SERVER",
      "ServiceName": "api-gateway",
      "ResourceAttributes": {
        "service.name": "api-gateway",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.4.1"
      },
      "ScopeName": "OpenTelemetry::Instrumentation::Rack",
      "ScopeVersion": "0.24.5",
      "SpanAttributes": {
        "http.method": "GET",
        "http.status_code": "200",
        "http.target": "/api/v1/users"
      },
      "Duration": 25748892,
      "StatusCode": "STATUS_CODE_UNSET",
      "StatusMessage": "",
      "EventsTimestamp": [],
      "EventsName": [],
      "EventsAttributes": [],
      "LinksTraceId": [],
      "LinksSpanId": [],
      "LinksTraceState": [],
      "LinksAttributes": []
    }
  ]
}
```

---

## Get Trace Heatmap

Retrieve a heatmap of trace durations over time for visualizing latency distribution.

```text
POST /cat/api/traces/v2/heatmap/json
```

### Query Parameters

| Parameter             | Type    | Required | Description                                    |
| --------------------- | ------- | -------- | ---------------------------------------------- |
| `region`              | string  | Yes      | Your data region                               |
| `start`               | integer | Yes      | Start time in Unix seconds                     |
| `end`                 | integer | Yes      | End time in Unix seconds                       |
| `time_resolution_sec` | integer | Yes      | X-axis resolution in seconds                   |
| `buckets`             | integer | Yes      | Y-axis resolution (number of duration buckets) |

### Example Request

```shell
curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/heatmap/json?region=us-east-1&start=1727083606&end=1727087206&time_resolution_sec=60&buckets=20" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"pipeline": []}'
```

### Response

```json
{
  "data": [
    {
      "timestamp": "2024-09-23T09:45:00Z",
      "values": [
        [10.5, 150],
        [50.2, 230],
        [100.8, 85]
      ]
    }
  ]
}
```

---

## Get Trace Tags

Retrieve available tags/labels for filtering traces.

```text
POST /cat/api/traces/v2/series/json
```

### Query Parameters

| Parameter | Type    | Required | Description                |
| --------- | ------- | -------- | -------------------------- |
| `region`  | string  | Yes      | Your data region           |
| `start`   | integer | Yes      | Start time in Unix seconds |
| `end`     | integer | Yes      | End time in Unix seconds   |

### Example Request

```shell
curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/series/json?region=us-east-1&start=1727083606&end=1727087206" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{}'
```

### Response

```json
{
  "status": "success",
  "data": [
    {
      "service.name": "api-gateway",
      "http.method": "GET"
    }
  ]
}
```

---

## Get Tag Values

Retrieve possible values for a specific tag.

```text
POST /cat/api/traces/v2/label/json/{tagName}/values
```

### Path Parameters

| Parameter | Type   | Description  |
| --------- | ------ | ------------ |
| `tagName` | string | The tag name |

### Query Parameters

| Parameter | Type    | Required | Description                |
| --------- | ------- | -------- | -------------------------- |
| `region`  | string  | Yes      | Your data region           |
| `start`   | integer | Yes      | Start time in Unix seconds |
| `end`     | integer | Yes      | End time in Unix seconds   |

### Example Request

```shell
curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/label/json/service.name/values?region=us-east-1&start=1727083606&end=1727087206" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{}'
```

### Response

```json
{
  "status": "success",
  "data": [
    "api-gateway",
    "user-service",
    "payment-service",
    "notification-service"
  ]
}
```

---

## Pipeline Syntax

The Traces API uses a JSON pipeline format for filtering trace data.

### Filter Stage

```json
{
  "type": "filter",
  "query": {
    "$and": [
      { "$eq": ["field.name", "value"] },
      { "$gte": ["numeric.field", "100"] }
    ]
  }
}
```

### Filter Operators

| Operator       | Description                         | Example                                 |
| -------------- | ----------------------------------- | --------------------------------------- |
| `$eq`          | Equal (case-sensitive)              | `{"$eq": ["service.name", "api"]}`      |
| `$ieq`         | Equal (case-insensitive)            | `{"$ieq": ["service.name", "API"]}`     |
| `$neq`         | Not equal (case-sensitive)          | `{"$neq": ["status", "error"]}`         |
| `$ineq`        | Not equal (case-insensitive)        | `{"$ineq": ["status", "ERROR"]}`        |
| `$gt`          | Greater than                        | `{"$gt": ["duration", "1000"]}`         |
| `$gte`         | Greater than or equal               | `{"$gte": ["http.status_code", "400"]}` |
| `$lt`          | Less than                           | `{"$lt": ["duration", "5000"]}`         |
| `$lte`         | Less than or equal                  | `{"$lte": ["http.status_code", "499"]}` |
| `$contains`    | Contains substring (case-sensitive) | `{"$contains": ["path", "/api"]}`       |
| `$icontains`   | Contains (case-insensitive)         | `{"$icontains": ["path", "/API"]}`      |
| `$notcontains` | Does not contain (case-sensitive)   | `{"$notcontains": ["path", "health"]}`  |
| `$regex`       | Matches regex (case-sensitive)      | `{"$regex": ["path", "^/api/v[0-9]+"]}` |
| `$iregex`      | Matches regex (case-insensitive)    | `{"$iregex": ["method", "get\|post"]}`  |
| `$notregex`    | Does not match regex                | `{"$notregex": ["path", "internal"]}`   |

### Logical Operators

Combine conditions using `$and`, `$or`, and `$not`:

#### AND (All conditions must match)

```json
{
  "$and": [
    { "$eq": ["service.name", "api-gateway"] },
    { "$gte": ["http.status_code", "500"] }
  ]
}
```

#### OR (Any condition can match)

```json
{
  "$or": [
    { "$eq": ["http.status_code", "500"] },
    { "$eq": ["http.status_code", "502"] }
  ]
}
```

#### NOT (Negate conditions)

```json
{
  "$not": [{ "$and": [{ "$eq": ["service.name", "health-check"] }] }]
}
```

#### Combining Logical Operators

Nest logical operators to build complex queries:

```json
{
  "$and": [
    { "$eq": ["service.name", "api-gateway"] },
    {
      "$or": [
        { "$eq": ["http.status_code", "500"] },
        { "$eq": ["http.status_code", "502"] }
      ]
    }
  ]
}
```

### Field Prefixes

| Prefix      | Description                      | Example                 |
| ----------- | -------------------------------- | ----------------------- |
| `resource.` | Resource-level attributes        | `resource.service.name` |
| `span.`     | Span-level attributes            | `span.http.method`      |
| (none)      | Auto-resolved based on attribute | `service.name`          |

---

## Common Query Patterns

### Filter by Service and Status Code

```shell
curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/query_range/json?region=us-east-1&start=1727083606&end=1727087206&limit=100&mode=trace" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": [
      {
        "type": "filter",
        "query": {
          "$and": [
            { "$eq": ["service.name", "api-gateway"] },
            { "$gte": ["http.status_code", "500"] }
          ]
        }
      }
    ]
  }'
```

### Duration-Based Filtering

Find slow traces using the Search endpoint's duration parameters:

```shell
curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/search/json?region=us-east-1&start=1727083606&end=1727087206&limit=50&minDuration=1s&maxDuration=10s" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"pipeline": []}'
```

### Regex Pattern Matching

Match spans with paths following a pattern:

```shell
curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/query_range/json?region=us-east-1&start=1727083606&end=1727087206&limit=100&mode=span" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": [
      {
        "type": "filter",
        "query": {
          "$and": [
            { "$regex": ["http.target", "^/api/v[0-9]+/users"] }
          ]
        }
      }
    ]
  }'
```

### Multi-Condition with OR

Find error traces across multiple services:

```shell
curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/query_range/json?region=us-east-1&start=1727083606&end=1727087206&limit=100&mode=trace" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": [
      {
        "type": "filter",
        "query": {
          "$and": [
            { "$or": [
              { "$eq": ["service.name", "api-gateway"] },
              { "$eq": ["service.name", "payment-service"] }
            ]},
            { "$gte": ["http.status_code", "500"] }
          ]
        }
      }
    ]
  }'
```

---

## Response Format

### Success Response Structure

All query endpoints return a JSON envelope with a `status` field and a `data` object containing the results:

```json
{
  "status": "success",
  "data": {
    "resultType": "stream",
    "result": [ ... ]
  }
}
```

The shape of objects inside `result` varies by endpoint — see each endpoint's Response section for details.

### Empty Results

When no traces match your query, the `result` array will be empty:

```json
{
  "status": "success",
  "data": {
    "resultType": "stream",
    "result": []
  }
}
```

---

## Reference

### Trace Modes

| Mode    | Description                                     |
| ------- | ----------------------------------------------- |
| `span`  | Query and filter individual spans               |
| `trace` | Query complete traces (groups of related spans) |

### Span Data Types

#### Span Kinds

| Value                   | Description                            |
| ----------------------- | -------------------------------------- |
| `SPAN_KIND_UNSPECIFIED` | Unspecified span kind                  |
| `SPAN_KIND_INTERNAL`    | Internal operation within a service    |
| `SPAN_KIND_SERVER`      | Server-side handling of an RPC request |
| `SPAN_KIND_CLIENT`      | Client-side of an RPC request          |
| `SPAN_KIND_PRODUCER`    | Producer of an async message           |
| `SPAN_KIND_CONSUMER`    | Consumer of an async message           |

#### Status Codes

| Value               | Description                      |
| ------------------- | -------------------------------- |
| `STATUS_CODE_UNSET` | Status not set                   |
| `STATUS_CODE_OK`    | Operation completed successfully |
| `STATUS_CODE_ERROR` | Operation failed                 |

### Common Span Attributes

#### HTTP Attributes

| Attribute          | Description                   |
| ------------------ | ----------------------------- |
| `http.method`      | HTTP method (GET, POST, etc.) |
| `http.status_code` | HTTP response status code     |
| `http.target`      | Request target/path           |
| `http.host`        | Request host                  |
| `http.scheme`      | URL scheme (http, https)      |
| `http.user_agent`  | User agent string             |

#### Database Attributes

| Attribute      | Description                            |
| -------------- | -------------------------------------- |
| `db.system`    | Database type (mysql, postgres, redis) |
| `db.name`      | Database name                          |
| `db.statement` | Database query/statement               |

#### Network Attributes

| Attribute       | Description        |
| --------------- | ------------------ |
| `net.peer.name` | Remote hostname/IP |
| `net.peer.port` | Remote port        |

#### Code Attributes

| Attribute        | Description          |
| ---------------- | -------------------- |
| `code.function`  | Function/method name |
| `code.namespace` | Code namespace/class |

---

## Saved Searches

Manage saved trace searches for your organization.

### List Saved Searches

```text
GET /traces/searches
```

#### Query Parameters

| Parameter      | Type   | Required | Description                         |
| -------------- | ------ | -------- | ----------------------------------- |
| `query_source` | string | No       | Filter by source: `manual` or `nlp` |

#### Example Request

```shell
curl -X GET \
  "https://app.last9.io/api/v4/organizations/{org-slug}/traces/searches" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>"
```

### Create Saved Search

```text
POST /traces/searches
```

#### Request Body

```json
{
  "name": "Error Traces",
  "pipeline": [
    {
      "type": "filter",
      "query": {
        "$and": [{ "$gte": ["http.status_code", "500"] }]
      }
    }
  ],
  "mode": "trace",
  "shared": false
}
```

| Field      | Type    | Required | Description                              |
| ---------- | ------- | -------- | ---------------------------------------- |
| `name`     | string  | Yes      | Name of the saved search                 |
| `pipeline` | array   | Yes      | JSON pipeline for trace queries          |
| `mode`     | string  | Yes      | Query mode: `span` or `trace`            |
| `shared`   | boolean | No       | Share with organization (default: false) |

### Update Saved Search

```text
PUT /traces/searches/{id}
```

### Delete Saved Search

```text
DELETE /traces/searches/{id}
```

---

## Recent Searches

Access recently executed trace searches (retained for 7 days).

```text
GET /traces/recent-searches
```

### Query Parameters

| Parameter      | Type   | Required | Description                         |
| -------------- | ------ | -------- | ----------------------------------- |
| `query_source` | string | No       | Filter by source: `manual` or `nlp` |

---

## Error Handling

### Error Response Format

```json
{
  "error": {
    "code": "ERROR_CODE",
    "message": "Human-readable error message"
  }
}
```

### Common Errors

| HTTP Status | Error                                | Description                          |
| ----------- | ------------------------------------ | ------------------------------------ |
| 400         | `region query parameter is required` | Missing required region parameter    |
| 401         | `Authorization token is expired`     | Access token has expired             |
| 403         | Forbidden                            | Insufficient permissions             |
| 404         | Not Found                            | Resource not found                   |
| 500         | `ERR_S3_CONFIG_MISSING`              | OTLP configuration not found for org |

### Debugging Common Scenarios

#### No Data Returned

If your query returns an empty `result` array, check the following:

1. **Field names**: Verify the field names are correct. Use [Get Trace Tags](#get-trace-tags) to discover available fields.
2. **Time range**: Ensure your `start` and `end` timestamps cover a period where traces exist. Timestamps are in Unix seconds.
3. **Data retention**: Check if the queried time range is within your organization's data retention period.
4. **Pipeline syntax**: Make sure your JSON pipeline is correctly formatted. Each operator takes an array with exactly 2 elements: `["field", "value"]`.
5. **Mode**: Verify you are using the correct `mode` (`span` vs `trace`) for your use case.

#### Authorization Issues

If you receive authentication errors:

1. **Token validity**: Verify your access token is valid and not expired.
2. **Header format**: The header must be `X-LAST9-API-TOKEN: Bearer <TOKEN>`.
3. **Organization slug**: Ensure the `{org-slug}` in the URL matches your organization.

---

## API Endpoints Summary

| Method | Endpoint                                         | Description                  |
| ------ | ------------------------------------------------ | ---------------------------- |
| POST   | `/cat/api/traces/v2/query_range/json`            | Query traces with pipeline   |
| POST   | `/cat/api/traces/v2/search/json`                 | Search traces with span sets |
| GET    | `/cat/api/traces/{traceID}`                      | Get trace by ID              |
| POST   | `/cat/api/traces/v2/heatmap/json`                | Get trace duration heatmap   |
| POST   | `/cat/api/traces/v2/series/json`                 | Get available tags           |
| POST   | `/cat/api/traces/v2/label/json/{tagName}/values` | Get tag values               |
| GET    | `/traces/searches`                               | List saved searches          |
| POST   | `/traces/searches`                               | Create saved search          |
| PUT    | `/traces/searches/{id}`                          | Update saved search          |
| DELETE | `/traces/searches/{id}`                          | Delete saved search          |
| GET    | `/traces/recent-searches`                        | List recent searches         |

---

## Troubleshooting

Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:support@last9.io) if you have any questions.