Scheduled Search

Scheduled Search allows you to configure periodic searches on your logs data and receive alerts based on specific conditions. You can set up alerts for both regular queries and aggregation queries.

This capability is valuable for scenarios like:

• Detecting error rates exceeding normal thresholds
• Identifying unusual patterns in user behavior
• Monitoring for missing scheduled jobs or backups
• Watching for repeated failed login attempts that could indicate brute force attacks
• Tracking performance degradation through aggregated metrics
• Monitoring resource consumption trends across services

Creating a Scheduled Search

Via Logs Explorer

1. Run a query in Logs Explorer, either in Builder or Editor mode
2. Click on ⋮ > “Save Query” in the top right
3. Enter a descriptive name for your query
4. Click “Save query & add alert”
5. Define Alert Configuration:
   • Review the query preview to confirm it’s correct
   • Set a numerical threshold with an operator (>, <, =, >=, <=) and a value
   • Select the appropriate evaluation frequency
     • Note: If your query uses a timeslice parameter, the evaluation frequency will automatically inherit this value
   • Select an alert destination from your notification channels
     • If no preconfigured channel exists, you can set one up via Notification Channels (How to setup a Notification Channel?)
6. View configured alerts in Scheduled Search

Via Saved Queries

1. In Logs Explorer, click on “Query Library” > “Saved Queries” in the top right
2. Locate the query you want to set a Scheduled Search for
3. Click on “Add Alert”
4. Follow Step #5 onwards as in Via Logs Explorer

Example Use Cases

1. Monitoring API Error Rates: This alerts you when any endpoint experiences more than 10 errors in a 5-minute window.

   • Query: sum(count) by (endpoint) | filter status_code >= 500 | timeslice 5m
   • Alert when: sum > 10

2. Detecting Latency Spikes: This alerts you when average response time for any service exceeds 500ms in a 1-minute window.

   • Query: avg(response_time) by (service) | timeslice 1m
   • Alert when: avg > 500

3. Tracking Authentication Failures: This alerts you when there are more than 20 authentication failures in a 10-minute period, which might indicate a brute force attack.

   • Query: count | filter event_type = "auth_failure" | timeslice 10m
   • Alert when: count > 20

4. Monitoring Missing Backup Jobs: This alerts you when no successful backup jobs have completed within a 24-hour period.

   • Query: | filter job_type = "backup" AND status = "completed"
   • Alert when: No Results
   • Evaluation frequency: 24h

Current Limitations

• Single Aggregate Only: Scheduled Search cannot be configured on queries with multiple aggregates, as each aggregate is treated as a separate metric
• Default Index Requirement: Currently, Scheduled Search can only be configured for queries on the Default Index

---

Troubleshooting

If your alerts aren’t triggering as expected:

1. Manually run your query to verify it returns data
2. Check that your threshold isn’t set too high or too low
3. Verify that your notification channels are correctly configured
4. Ensure your query syntax is valid and returns the expected type of results
5. For aggregation queries, confirm that your query is returning numerical values
6. Check your alert history to see if evaluations are happening but not crossing thresholds

Need help? Please get in touch with us on Discord or Email if you have any questions.