Physical Indexes

What are Physical Indexes?

Physical Indexes allow you to create specialized query spaces for specific log data based on custom filters you define. By narrowing the search scope, these indexes significantly improve performance for queries targeting specific data subsets.

While Last9’s core value comes from providing correlated telemetry across your entire dataset, there are scenarios where isolating certain data can improve performance and workflow efficiency.

When to Use Physical Indexes

Physical Indexes are most valuable when:

1. Team or Environment Isolation: Separate production from staging data, or create team-specific indexes in organizations with siloed responsibilities.

2. Service-Specific Workflows: Create separate indexes logs from services that don’t typically require correlation within themselves or with other systems (like CDN logs, proxy logs).

3. High-Volume Data Management: Create dedicated indexes for extremely high-volume services that might otherwise impact query performance.

4. Specialized Analysis: Support focused workflows for security events, authentication logs, or error tracking.

Creating a Physical Index

1. Navigate to Physical Indexes and click “New Physical Index”
2. Define filters to determine which logs will be included:
   • Logs are currently the only supported telemetry type
   • Configure filter conditions using attributes or resource attributes
   • Click on “View Logs” to confirm your filter works as expected
3. Specify index details:
   • Retention period matches your organization’s default retention, but can be changed by contacting support
   • Give a descriptive name (cannot be changed after creation)
   • Add an optional description (see naming best practices)
   • Choose whether to also write logs to the default index (see billing impact in important considerations)
4. Click “Create Physical Index”

Index Naming Best Practices

While there are no strict rules for naming indexes, consider these patterns for clarity and organization:

• Environment-based: prod-logs, staging-logs, dev-logs
• Team-based: team-payments, team-auth, team-frontend
• Service-based: service-cdn, service-api, service-db
• Purpose-based: security-events, error-logs, performance-metrics

Important Considerations

1. Billing Impact: Logs stored in multiple indexes (both a Physical Index and the default index) are counted separately for billing purposes. Last9 billing is based on events stored, not deduplicated across indexes.

2. Correlation Trade-offs: Physical Indexes create separate query spaces. While this improves performance for targeted queries, it may limit correlation capabilities across your entire dataset.

3. Retention Periods: Physical Indexes share the same retention period as your organization’s default index. To modify retention for a specific index, contact support.

4. Filter Design: Create precise filters to include only the data you need. Overly broad filters may reduce the performance benefits of separate indexes.

Example Use Cases

Environment Separation

Create separate indexes for production and non-production environments:

Filter: resource.kubernetes.namespace.name == "production-*"
Name: prod-environment

Team-Specific Indexes

Isolate data for team-specific workflows:

Filter: resource.service.name IN ("payment-service", "billing-service", "invoice-service")
Name: team-finance

High-Volume Service Isolation

Create dedicated indexes for services generating large volumes of logs:

Filter: resource.service.name == "cdn-edge"
Name: high-volume-cdn

Security Monitoring

Isolate authentication and security events:

Filter: attributes.log.level IN ("ERROR", "CRITICAL") AND attributes.event.type IN ("auth", "security")
Name: security-events

Troubleshooting

Please get in touch with us on Discord or Email if you have any questions.