Skip to content
Last9
Talk to an Expert

Scheduled Search

Create periodic searches on logs data and set alerts when patterns are found or missing.

Scheduled Search allows you to configure periodic searches on your logs data and receive alerts based on specific conditions. You can set up alerts for both regular queries and aggregation queries.

This capability is valuable for scenarios like:

  • Alerting when API error rates exceed 50 errors per minute
  • Detecting when average response times spike above 2 seconds
  • Monitoring for missing critical scheduled jobs (backups, data exports)
  • Identifying potential security threats through repeated failed logins

Via Logs Explorer

  1. Run a query in Logs Explorer, either in Builder or Editor mode
  2. Click on ⋮ > “Save Query” in the top right
  3. Enter a descriptive name for your query
  4. Click “Save query & add alert”
  5. Define Alert Configuration:
    • Review the query preview to confirm it’s correct
    • Set a numerical threshold with an operator (>, <, =, >=, <=) and a value
    • Select the appropriate evaluation frequency
      • Note: If your query uses a timeslice parameter, the evaluation frequency will automatically inherit this value
    • Select an alert destination from your notification channels
  6. View configured alerts in Scheduled Search

Via Saved Queries

  1. In Logs Explorer, click on “Query Library” > “Saved Queries” in the top right
  2. Locate the query you want to set a Scheduled Search for
  3. Click on “Add Alert”
  4. Follow Step #5 onwards as in Via Logs Explorer

Managing Scheduled Search Rules

You can view and manage your scheduled search rules from two locations:

Via Scheduled Search Dashboard

  1. Navigate to Scheduled Search
  2. View all your configured scheduled search rules in one place
  3. Click on any rule to preview its configuration details
  4. Delete rules you no longer need

Via Query Library

  1. In Logs Explorer, click on “Query Library” > “Saved Queries”
  2. Use the “With Alerts” filter to view only saved queries that have scheduled search rules configured
  3. For queries you’ve created with existing alerts, click “Manage Alert” to:
    • Modify thresholds, evaluation frequency, or notification channels
    • View and edit alert rule configuration
    • Enable/disable the alert rule

Example Use Cases

  1. Monitoring API Error Rates: This alerts you when any endpoint experiences more than 10 errors in a 5-minute window.

    • Query: sum(count) by (endpoint) | filter status_code >= 500 | timeslice 5m
    • Alert when: sum > 10

  2. Detecting Latency Spikes: This alerts you when average response time for any service exceeds 500ms in a 1-minute window.

    • Query: avg(response_time) by (service) | timeslice 1m
    • Alert when: avg > 500

  3. Tracking Authentication Failures: This alerts you when there are more than 20 authentication failures in a 10-minute period, which might indicate a brute force attack.

    • Query: count | filter event_type = "auth_failure" | timeslice 10m
    • Alert when: count > 20

  4. Monitoring Missing Backup Jobs: This alerts you when no successful backup jobs have completed within a 24-hour period.

    • Query: | filter job_type = "backup" AND status = "completed"
    • Alert when: No Results
    • Evaluation frequency: 24h

Current Limitations

Query Restrictions:

  • Single aggregate queries only, queries with multiple aggregates aren’t supported since each aggregate becomes a separate metric
  • Default Index only, queries on custom indexes aren’t currently supported

Alert Management:

  • Scheduled Search alerts don’t appear in the centralized Alert Monitor
  • Alert history and management happens at the individual scheduled search level

Troubleshooting

Query Issues:

  • Manually run your query in Logs Explorer to verify it returns expected data
  • Ensure your query syntax is valid and returns the expected data type
  • For aggregation queries, confirm your query returns numerical values (not text or null)

Alert Configuration:

  • Check that your threshold isn’t set too high or too low for your data patterns
  • Verify the evaluation frequency matches your data collection intervals
  • For “No Results” alerts, ensure the query normally returns data when conditions are met

Notification Problems:

  • Verify that your notification channels are correctly configured in Notification Channels
  • Test your notification channels independently to ensure they’re working
  • Check your alert history in the scheduled search details to see if evaluations are happening but not triggering notifications

Need help? Please get in touch with us on Discord or Email if you have any questions.