Traces Query Builder

The Traces Query Builder provides a visual interface for constructing trace analysis queries through operation stages. Each stage represents a specific data manipulation function, allowing you to build complex queries without writing TraceQL syntax.

Accessing the Query Builder

Navigate to Traces in Last9 and use the Span or Trace tabs to begin building queries. The Query Builder is available by default - simply click ADD STAGE to start adding operation blocks.

Core Stages

Filter

The FILTER stage allows you to narrow down traces and spans based on specific conditions using various operators:

Supported Operators:

• =: Exact match
• !=: Not equal to
• exists: Field exists with any value
• not exists: Field does not exist
• contains: String contains substring
• not contains: String does not contain substring
• matches: Matches a pattern/regular expression

Parameters:

• Field name: The span or trace attribute to filter on
• Operator: One of the supported operators
• Value: The comparison value or pattern (not required for exists operators)

A single field and operator can have multiple values. These values are “ORed” together. To achieve AND logic, use multiple filter stages.

Examples:

Service Name exists
Trace Status = STATUS_CODE_ERROR
http.method != POST
span.name contains "database"
user.id matches "user-[0-9]+"

Aggregate

The AGGREGATE stage supports various statistical computations with optional grouping and timeslicing:

1. Zero or One Argument Functions:

   • count
     • Usage without argument: count as total_count
     • Usage with argument: count field_name as field_count

2. Single Argument Functions:

   • sum: Calculate sum of values

   • min: Find minimum value

   • max: Find maximum value

   • avg: Calculate average

   • median: Find median value

   • stddev: Calculate sample standard deviation

   • stddev_pop: Calculate population standard deviation

   • variance: Calculate sample variance

   • variance_pop: Calculate population variance

   • Usage Syntax: function_name field as result_name

3. Two Argument Functions:

   • quantile: Calculate approximate quantile (value between 0 and 1)

   • quantile_exact: Calculate exact quantile (value between 0 and 1)

   • Usage: quantile(0.99, duration) as p99_duration

Additional Features:

• groupby: Group results by specified fields, similar to SQL GROUP BY
  • Example: groupby Service Name as service_name
• timeslice: Define time bucket intervals for time-series aggregation
  • When specified, creates time-series visualization
  • Without timeslice, displays only table view
  • Example: timeslice 15 minutes

Example Query:

count as _count
groupby Service Name as service_name
timeslice 15 minutes

Transform

The TRANSFORM stage provides data transformation methods for manipulating span and trace attributes:

1. split

   • Parameters:

     • from: Source field
     • on: Delimiter character
     • select part: Part number to select (0-indexed)
     • as: Output field name

   • Example:

     split from http.url on / select part 2 as endpoint

2. splitInto

   • Parameters:

     • from: Source field
     • on: Delimiter character
     • into: Comma-separated list of output field names

   • Example:

     splitInto from trace.id on - into region, service, request_id

3. replaceRegex

   • Parameters:

     • from: Source field
     • pattern: Regex pattern to match
     • with: Replacement string
     • as: Output field name

   • Example:

     replaceRegex from user.email /[<>]/ with - as sanitized_email

4. concat

   • Parameters:

     • from: First field to concatenate
     • Additional fields/literals to concatenate
     • as: Output field name

   • Example:

     concat http.method, ' ', http.route as full_endpoint

5. join

   • Parameters:

     • field: Field to join
     • separator: String to use between values
     • as: Output field name

   • Example:

     join error.type , as error_types

6. if

   • Parameters:

     • Condition: isEqual, !isEqual, isEmpty, !isEmpty
     • then: Value if condition is true
     • else: Value if condition is false
     • as: Output field name

   • Example:

     if isEmpty error.message then 'No Error' else error.message as error_display

Visualization Modes

The Query Builder provides two visualization modes:

Visualization (Chart)

Time-series chart visualization is displayed when your query includes a timeslice parameter in the AGGREGATE stage. This mode shows trends over time and is ideal for monitoring patterns and anomalies.

Table

Table view displays aggregated results in tabular format. This is the only view available when no timeslice is specified, or you can switch to it manually when both modes are available.

Query Construction Best Practices

1. Operation Order

   • Start with FILTER operations to reduce data volume early
   • Apply TRANSFORM operations to prepare fields for aggregation
   • Use AGGREGATE operations last to summarize data
   • Apply additional FILTER stages after AGGREGATE to filter aggregated results

2. Field Naming

   • Use descriptive names for output fields in as clauses
   • Maintain consistent naming conventions (snake_case recommended)
   • Avoid special characters in field names
   • Use meaningful prefixes for grouped fields

3. Performance Optimization

   • Keep time ranges reasonable (avoid queries spanning months)
   • Filter on indexed fields like Service Name for faster execution
   • Use exists operator for field presence checks rather than string matching
   • Filter early in the query chain to minimize data processing
   • Use = operator on Service Name for significantly faster query execution

4. Timeslice Guidelines

   • Match timeslice intervals to your time range:
     • 1-hour range: 1-5 minute intervals
     • 1-day range: 5-30 minute intervals
     • 1-week range: 1-4 hour intervals
   • Shorter intervals provide more detail but slower queries
   • Longer intervals are faster but may miss short-lived patterns

5. Error Prevention

   • Use exists checks before accessing optional fields
   • Validate regex patterns before using them (tools like regex101.com)
   • Test queries on shorter time ranges before expanding
   • Use !isEmpty checks before transformations

Using Query Builder with Streaming Aggregation

Queries built in the Query Builder can be saved as Streaming Aggregation rules to generate TraceMetrics and continuously compute metrics from your trace data. This is particularly useful for creating custom metrics without impacting query performance.

1. Build your query in the Query Builder with FILTER and AGGREGATE stages
2. Click Create Metric to open the Streaming Aggregation configuration
3. Configure the evaluation frequency (timeslice) and provide a descriptive rule name
4. Save the rule to begin generating metrics

Your saved query will now run continuously, creating metrics you can query without processing raw trace data. Learn more about Streaming Aggregation.

Field Handling

Common Trace Attributes

Access standard trace and span attributes using their exact names:

Service Name
Trace Status
http.method
http.status_code
http.route
error.type
error.message
duration

Custom Attributes

Access custom attributes and tags added by your instrumentation:

user.id
tenant.id
feature.flag
environment

Nested Fields

For nested attributes, use dot notation:

http.request.headers.user_agent
resource.attributes.service.version

Keyboard Shortcuts

The Query Builder supports keyboard shortcuts for faster query construction:

• Shift + Enter: Add a new stage
• Shift + Delete: Remove the current stage
• Tab: Navigate between fields in a stage

Troubleshooting

Please get in touch with us on Discord or Email if you have any questions.