Traces Query API

The Traces Query API enables programmatic access to your distributed tracing data stored in Last9. Use these endpoints to query traces, search for spans, manage saved searches, and retrieve trace metadata.

Prerequisites

Before using the Traces Query API, you need:

An API access token (see Getting started with API)
Your organization slug
Your data region (e.g., us-east-1)

Base URL

https://app.last9.io/api/v4/organizations/{org-slug}

Replace {org-slug} with your organization’s unique slug.

Authentication

All API endpoints require authentication via the X-LAST9-API-TOKEN header:

X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>

Query Traces

The primary endpoint for querying trace data with pipeline-based filtering.

POST /cat/api/traces/v2/query_range/json

Query Parameters

Parameter	Type	Required	Description
`region`	string	Yes	Your data region (e.g., `us-east-1`)
`start`	integer	Yes	Start time in Unix seconds
`end`	integer	Yes	End time in Unix seconds
`limit`	integer	No	Maximum number of results to return
`order`	string	No	Sort order (`asc` or `desc`)
`direction`	string	No	Query direction
`mode`	string	No	Query mode: `span` or `trace`

Request Body

The request body contains a pipeline array with filter stages:

{
  "pipeline": [
    {
      "type": "filter",
      "query": {
        "$and": [
          { "$eq": ["service.name", "api-gateway"] },
          { "$gte": ["http.status_code", "500"] }
        ]
      }
    }
  ]
}

Example Request

curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/query_range/json?region=us-east-1&start=1727083606&end=1727087206&limit=100&mode=trace" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{
    "pipeline": [
      {
        "type": "filter",
        "query": {
          "$and": [
            { "$eq": ["service.name", "api-gateway"] }
          ]
        }
      }
    ]
  }'

Response

{
  "status": "success",
  "data": {
    "resultType": "stream",
    "result": [
      {
        "Timestamp": "2024-09-23T09:45:13.700906028Z",
        "TraceId": "cb9cd197d727655add1c12c4afe0b476",
        "SpanId": "83fca378bb34f8db",
        "ParentSpanId": "",
        "SpanName": "GET /api/v1/users",
        "SpanKind": "SPAN_KIND_SERVER",
        "ServiceName": "api-gateway",
        "ResourceAttributes": {
          "service.name": "api-gateway",
          "telemetry.sdk.name": "opentelemetry"
        },
        "SpanAttributes": {
          "http.method": "GET",
          "http.status_code": "200",
          "http.target": "/api/v1/users"
        },
        "Duration": 25748892,
        "StatusCode": "STATUS_CODE_UNSET",
        "StatusMessage": ""
      }
    ]
  }
}

Search Traces

Search for traces matching specific criteria with span sets. This endpoint is useful for finding traces based on duration thresholds.

POST /cat/api/traces/v2/search/json

Query Parameters

Parameter	Type	Required	Description
`region`	string	Yes	Your data region
`start`	integer	Yes	Start time in Unix seconds
`end`	integer	Yes	End time in Unix seconds
`limit`	integer	No	Maximum number of traces to return
`span_limit`	integer	No	Maximum spans per trace
`order`	string	No	Sort order
`mode`	string	No	Query mode: `span` or `trace`
`minDuration`	string	No	Minimum duration filter (e.g., `100ms`, `1s`)
`maxDuration`	string	No	Maximum duration filter (e.g., `5s`, `10000ms`)

Example Request

curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/search/json?region=us-east-1&start=1727083606&end=1727087206&limit=50&minDuration=500ms" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"pipeline": []}'

Response

{
  "status": "success",
  "data": {
    "resultType": "stream",
    "result": [
      {
        "traceID": "7d7de75bd7d3d51cf29d73ad5e36a227",
        "rootServiceName": "api-gateway",
        "rootTraceName": "GET /api/v1/users",
        "startTimeUnixNano": 1727083606684633211,
        "endTimeUnixNano": 1727083606716391435,
        "durationMs": 31.758224,
        "spanSets": [
          {
            "matched": 1,
            "spans": [...]
          }
        ]
      }
    ]
  }
}

Get Trace by ID

Retrieve all spans for a specific trace.

GET /cat/api/traces/{traceID}

Path Parameters

Parameter	Type	Description
`traceID`	string	The 32-character hex trace ID

Query Parameters

Parameter	Type	Required	Description
`region`	string	Yes	Your data region
`start`	integer	Yes	Start time in Unix seconds
`end`	integer	Yes	End time in Unix seconds
`limit`	integer	No	Maximum number of spans to return (default: 1000)

Example Request

curl -X GET \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/7d7de75bd7d3d51cf29d73ad5e36a227?region=us-east-1&start=1727083606&end=1727087206" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>"

Response

{
  "traces": [
    {
      "Timestamp": "2024-09-23T09:45:13.700906028Z",
      "TraceId": "cb9cd197d727655add1c12c4afe0b476",
      "SpanId": "83fca378bb34f8db",
      "ParentSpanId": "",
      "TraceState": "",
      "SpanName": "GET /api/v1/users",
      "SpanKind": "SPAN_KIND_SERVER",
      "ServiceName": "api-gateway",
      "ResourceAttributes": {
        "service.name": "api-gateway",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.4.1"
      },
      "ScopeName": "OpenTelemetry::Instrumentation::Rack",
      "ScopeVersion": "0.24.5",
      "SpanAttributes": {
        "http.method": "GET",
        "http.status_code": "200",
        "http.target": "/api/v1/users"
      },
      "Duration": 25748892,
      "StatusCode": "STATUS_CODE_UNSET",
      "StatusMessage": "",
      "EventsTimestamp": [],
      "EventsName": [],
      "EventsAttributes": [],
      "LinksTraceId": [],
      "LinksSpanId": [],
      "LinksTraceState": [],
      "LinksAttributes": []
    }
  ]
}

Get Trace Heatmap

Retrieve a heatmap of trace durations over time for visualizing latency distribution.

POST /cat/api/traces/v2/heatmap/json

Query Parameters

Parameter	Type	Required	Description
`region`	string	Yes	Your data region
`start`	integer	Yes	Start time in Unix seconds
`end`	integer	Yes	End time in Unix seconds
`time_resolution_sec`	integer	Yes	X-axis resolution in seconds
`buckets`	integer	Yes	Y-axis resolution (number of duration buckets)

Example Request

curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/heatmap/json?region=us-east-1&start=1727083606&end=1727087206&time_resolution_sec=60&buckets=20" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"pipeline": []}'

Response

{
  "data": [
    {
      "timestamp": "2024-09-23T09:45:00Z",
      "values": [
        [10.5, 150],
        [50.2, 230],
        [100.8, 85]
      ]
    }
  ]
}

Get Trace Tags

Retrieve available tags/labels for filtering traces.

POST /cat/api/traces/v2/series/json

Query Parameters

Parameter	Type	Required	Description
`region`	string	Yes	Your data region
`start`	integer	Yes	Start time in Unix seconds
`end`	integer	Yes	End time in Unix seconds

Example Request

curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/series/json?region=us-east-1&start=1727083606&end=1727087206" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{}'

Response

{
  "status": "success",
  "data": [
    {
      "service.name": "api-gateway",
      "http.method": "GET"
    }
  ]
}

Get Tag Values

Retrieve possible values for a specific tag.

POST /cat/api/traces/v2/label/json/{tagName}/values

Path Parameters

Parameter	Type	Description
`tagName`	string	The tag name

Query Parameters

Parameter	Type	Required	Description
`region`	string	Yes	Your data region
`start`	integer	Yes	Start time in Unix seconds
`end`	integer	Yes	End time in Unix seconds

Example Request

curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/label/json/service.name/values?region=us-east-1&start=1727083606&end=1727087206" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{}'

Response

{
  "status": "success",
  "data": ["api-gateway", "user-service", "payment-service", "notification-service"]
}

Pipeline Syntax

The Traces API uses a JSON pipeline format for filtering trace data.

Filter Stage

{
  "type": "filter",
  "query": {
    "$and": [
      { "$eq": ["field.name", "value"] },
      { "$gte": ["numeric.field", "100"] }
    ]
  }
}

Filter Operators

Operator	Description	Example
`$eq`	Equal (case-sensitive)	`{"$eq": ["service.name", "api"]}`
`$ieq`	Equal (case-insensitive)	`{"$ieq": ["service.name", "API"]}`
`$neq`	Not equal (case-sensitive)	`{"$neq": ["status", "error"]}`
`$ineq`	Not equal (case-insensitive)	`{"$ineq": ["status", "ERROR"]}`
`$gt`	Greater than	`{"$gt": ["duration", "1000"]}`
`$gte`	Greater than or equal	`{"$gte": ["http.status_code", "400"]}`
`$lt`	Less than	`{"$lt": ["duration", "5000"]}`
`$lte`	Less than or equal	`{"$lte": ["http.status_code", "499"]}`
`$contains`	Contains substring (case-sensitive)	`{"$contains": ["path", "/api"]}`
`$icontains`	Contains (case-insensitive)	`{"$icontains": ["path", "/API"]}`
`$notcontains`	Does not contain (case-sensitive)	`{"$notcontains": ["path", "health"]}`
`$regex`	Matches regex (case-sensitive)	`{"$regex": ["path", "^/api/v[0-9]+"]}`
`$iregex`	Matches regex (case-insensitive)	`{"$iregex": ["method", "get\|post"]}`
`$notregex`	Does not match regex	`{"$notregex": ["path", "internal"]}`

Logical Operators

Combine conditions using $and, $or, and $not:

AND (All conditions must match)

{
  "$and": [
    { "$eq": ["service.name", "api-gateway"] },
    { "$gte": ["http.status_code", "500"] }
  ]
}

OR (Any condition can match)

{
  "$or": [
    { "$eq": ["http.status_code", "500"] },
    { "$eq": ["http.status_code", "502"] }
  ]
}

NOT (Negate conditions)

{
  "$not": [
    { "$and": [
      { "$eq": ["service.name", "health-check"] }
    ]}
  ]
}

Combining Logical Operators

Nest logical operators to build complex queries:

{
  "$and": [
    { "$eq": ["service.name", "api-gateway"] },
    { "$or": [
      { "$eq": ["http.status_code", "500"] },
      { "$eq": ["http.status_code", "502"] }
    ]}
  ]
}

Field Prefixes

Prefix	Description	Example
`resource.`	Resource-level attributes	`resource.service.name`
`span.`	Span-level attributes	`span.http.method`
(none)	Auto-resolved based on attribute	`service.name`

Common Query Patterns

Filter by Service and Status Code

curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/query_range/json?region=us-east-1&start=1727083606&end=1727087206&limit=100&mode=trace" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": [
      {
        "type": "filter",
        "query": {
          "$and": [
            { "$eq": ["service.name", "api-gateway"] },
            { "$gte": ["http.status_code", "500"] }
          ]
        }
      }
    ]
  }'

Duration-Based Filtering

Find slow traces using the Search endpoint’s duration parameters:

curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/search/json?region=us-east-1&start=1727083606&end=1727087206&limit=50&minDuration=1s&maxDuration=10s" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"pipeline": []}'

Regex Pattern Matching

Match spans with paths following a pattern:

curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/query_range/json?region=us-east-1&start=1727083606&end=1727087206&limit=100&mode=span" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": [
      {
        "type": "filter",
        "query": {
          "$and": [
            { "$regex": ["http.target", "^/api/v[0-9]+/users"] }
          ]
        }
      }
    ]
  }'

Multi-Condition with OR

Find error traces across multiple services:

curl -X POST \
  "https://app.last9.io/api/v4/organizations/{org-slug}/cat/api/traces/v2/query_range/json?region=us-east-1&start=1727083606&end=1727087206&limit=100&mode=trace" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": [
      {
        "type": "filter",
        "query": {
          "$and": [
            { "$or": [
              { "$eq": ["service.name", "api-gateway"] },
              { "$eq": ["service.name", "payment-service"] }
            ]},
            { "$gte": ["http.status_code", "500"] }
          ]
        }
      }
    ]
  }'

Response Format

Success Response Structure

All query endpoints return a JSON envelope with a status field and a data object containing the results:

{
  "status": "success",
  "data": {
    "resultType": "stream",
    "result": [ ... ]
  }
}

The shape of objects inside result varies by endpoint — see each endpoint’s Response section for details.

Empty Results

When no traces match your query, the result array will be empty:

{
  "status": "success",
  "data": {
    "resultType": "stream",
    "result": []
  }
}

Reference

Trace Modes

Mode	Description
`span`	Query and filter individual spans
`trace`	Query complete traces (groups of related spans)

Span Data Types

Span Kinds

Value	Description
`SPAN_KIND_UNSPECIFIED`	Unspecified span kind
`SPAN_KIND_INTERNAL`	Internal operation within a service
`SPAN_KIND_SERVER`	Server-side handling of an RPC request
`SPAN_KIND_CLIENT`	Client-side of an RPC request
`SPAN_KIND_PRODUCER`	Producer of an async message
`SPAN_KIND_CONSUMER`	Consumer of an async message

Status Codes

Value	Description
`STATUS_CODE_UNSET`	Status not set
`STATUS_CODE_OK`	Operation completed successfully
`STATUS_CODE_ERROR`	Operation failed

Common Span Attributes

HTTP Attributes

Attribute	Description
`http.method`	HTTP method (GET, POST, etc.)
`http.status_code`	HTTP response status code
`http.target`	Request target/path
`http.host`	Request host
`http.scheme`	URL scheme (http, https)
`http.user_agent`	User agent string

Database Attributes

Attribute	Description
`db.system`	Database type (mysql, postgres, redis)
`db.name`	Database name
`db.statement`	Database query/statement

Network Attributes

Attribute	Description
`net.peer.name`	Remote hostname/IP
`net.peer.port`	Remote port

Code Attributes

Attribute	Description
`code.function`	Function/method name
`code.namespace`	Code namespace/class

Saved Searches

Manage saved trace searches for your organization.

List Saved Searches

GET /traces/searches

Query Parameters

Parameter	Type	Required	Description
`query_source`	string	No	Filter by source: `manual` or `nlp`

Example Request

curl -X GET \
  "https://app.last9.io/api/v4/organizations/{org-slug}/traces/searches" \
  -H "X-LAST9-API-TOKEN: Bearer <ACCESS_TOKEN>"

Create Saved Search

POST /traces/searches

Request Body

{
  "name": "Error Traces",
  "pipeline": [
    {
      "type": "filter",
      "query": {
        "$and": [
          { "$gte": ["http.status_code", "500"] }
        ]
      }
    }
  ],
  "mode": "trace",
  "shared": false
}

Field	Type	Required	Description
`name`	string	Yes	Name of the saved search
`pipeline`	array	Yes	JSON pipeline for trace queries
`mode`	string	Yes	Query mode: `span` or `trace`
`shared`	boolean	No	Share with organization (default: false)

Update Saved Search

PUT /traces/searches/{id}

Delete Saved Search

DELETE /traces/searches/{id}

Recent Searches

Access recently executed trace searches (retained for 7 days).

GET /traces/recent-searches

Query Parameters

Parameter	Type	Required	Description
`query_source`	string	No	Filter by source: `manual` or `nlp`

Error Handling

Error Response Format

{
  "error": {
    "code": "ERROR_CODE",
    "message": "Human-readable error message"
  }
}

Common Errors

HTTP Status	Error	Description
400	`region query parameter is required`	Missing required region parameter
401	`Authorization token is expired`	Access token has expired
403	Forbidden	Insufficient permissions
404	Not Found	Resource not found
500	`ERR_S3_CONFIG_MISSING`	OTLP configuration not found for org

Debugging Common Scenarios

No Data Returned

If your query returns an empty result array, check the following:

Field names: Verify the field names are correct. Use Get Trace Tags to discover available fields.
Time range: Ensure your start and end timestamps cover a period where traces exist. Timestamps are in Unix seconds.
Data retention: Check if the queried time range is within your organization’s data retention period.
Pipeline syntax: Make sure your JSON pipeline is correctly formatted. Each operator takes an array with exactly 2 elements: ["field", "value"].
Mode: Verify you are using the correct mode (span vs trace) for your use case.

Authorization Issues

If you receive authentication errors:

Token validity: Verify your access token is valid and not expired.
Header format: The header must be X-LAST9-API-TOKEN: Bearer <TOKEN>.
Organization slug: Ensure the {org-slug} in the URL matches your organization.

API Endpoints Summary

Method	Endpoint	Description
POST	`/cat/api/traces/v2/query_range/json`	Query traces with pipeline
POST	`/cat/api/traces/v2/search/json`	Search traces with span sets
GET	`/cat/api/traces/{traceID}`	Get trace by ID
POST	`/cat/api/traces/v2/heatmap/json`	Get trace duration heatmap
POST	`/cat/api/traces/v2/series/json`	Get available tags
POST	`/cat/api/traces/v2/label/json/{tagName}/values`	Get tag values
GET	`/traces/searches`	List saved searches
POST	`/traces/searches`	Create saved search
PUT	`/traces/searches/{id}`	Update saved search
DELETE	`/traces/searches/{id}`	Delete saved search
GET	`/traces/recent-searches`	List recent searches

Troubleshooting

Please get in touch with us on Discord or Email if you have any questions.