Investigate an incident with AI

Use Last9 AI to gather evidence for an incident, test an explanation, and prepare a summary for your team. Start with the affected service and time window, and ask for the queries behind each conclusion. Verify recovery separately before closing the incident.

You can use the prompts below in AI Assistant, Slack, or an agent connected through Last9 MCP. Available tools and approval controls depend on the interface and its configuration.

Before you start

Confirm that the relevant logs, metrics, or traces are available in Last9.
Identify the service, environment, and data source you want to investigate. If you only know an application nickname, ask the assistant to find the matching service first.
Record the incident’s start and end time, including the time zone. Use the same window when comparing results from different signals.
Include an alert, request ID, trace ID, or error message when you have one.

Run the investigation

Work through the following steps in order. Each step builds on the evidence the previous one returned.

1. Define the question

A question with a specific service and time window gives you a result you can check:

Investigate elevated HTTP 5xx responses for checkout-api in production between 14:00 and 14:30 UTC on 2026-09-10. Confirm the service and data source first. Summarize the affected routes, error counts, and latency. Include evidence links and identify any data you could not query.

Replace the example service and time window with your incident details. If a query returns no data, verify the service and time window before concluding that the service is healthy.

2. Establish what happened

Ask for the symptom before the explanation:

Which requests failed in the incident window? Show the first and last observed failure, affected routes, status codes, and a representative request or trace ID. Distinguish total counts from sampled records.

Open the returned links in Logs Explorer, Traces Explorer, or Exceptions. Check the service, filters, and time range against the question you asked.

Explain what each query result establishes:

Evidence	What it establishes
A failed request or exception	A specific failure was observed.
Error counts over a fixed window	The count shows how often a failure matched the query’s filters and time window.
A trace with related spans	The trace shows how an instrumented request moved through services and dependencies.
A sample of returned records	The sample shows what the selected records contain, but may not represent the full result set.
A query with no matches	The query returned no matching data. Check the filters, time window, and data availability.

3. Test a possible cause

Ask the assistant to connect the symptom to evidence, and to describe what would disprove its explanation:

For the failing requests, check related traces and logs for a shared dependency or error pattern. Compare with successful requests in the same window. Separate confirmed findings from possible causes and list the missing evidence needed to distinguish them.

If change events are available, compare the incident with deployments and configuration changes. Investigate any deployment near the incident, but do not treat timing alone as proof of a cause.

For latency, ask which spans account for the delay and compare similar successful or faster requests. For errors, inspect the failing operation and its related logs. Report any missing dependency instrumentation or correlation IDs in the conclusion.

4. Verify recovery in a new window

An incident summary describes the data already queried. It does not establish the system’s current condition.

Recheck checkout-api in production for the last 15 minutes. State the exact window you query. Are requests succeeding under normal traffic? Compare error rate and latency with the incident window, and say whether the evidence supports recovery.

Check for successful activity as well as fewer errors. A quiet service may have no errors because it received no traffic. A single successful retry confirms that request succeeded, but leaves sustained recovery unverified.

5. Prepare the handoff

A handoff is a summary another engineer can use to continue the investigation. Ask the assistant to include the findings, completed actions, and remaining questions:

Write an incident handoff with: - Affected services, environments, and exact time windows. - Confirmed findings and their evidence links. - Most likely explanation, supporting evidence, and remaining uncertainty. - Actions actually taken, who took them, and what remains to do. - Latest recovery check and any follow-up validation needed. Do not describe a proposed action as completed or a possible cause as confirmed.

Review any proposed change before approving it in your client. For example, creating or updating a dashboard changes saved configuration. An investigation recommendation does not mean the assistant has applied a fix.

Examples from incident investigations

The examples below are adapted from incident investigations. Names and details are changed, and the prompts are illustrative.

An outage question followed by an RCA request

Operators use Last9 AI to ask about service health, p95 latency, recent errors, and outages, then request a root cause analysis (RCA). In one outage investigation, the assistant queried several signals but stopped at its investigation limit without enough matching telemetry to establish a cause.

Before writing an RCA from an incomplete answer, ask which evidence is missing:

Before writing the RCA, list which queries returned evidence, which found no matching data, and which could not complete. Confirm the service, environment, and incident window. What is the next targeted check that could establish the cause?

An incomplete investigation can still identify the next check. Its unanswered questions belong in the handoff.

An API failure with an unconfirmed downstream cause

An investigation located a failed request and a later successful, slower request. The engineer corrected the service name during the investigation. The available evidence confirmed the failure but did not establish which downstream component caused it.

Use a follow-up that asks for the missing connection:

Use the corrected service name and the same incident window. Can you connect the failed request to a downstream trace or log using its request ID? If the records cannot be connected, explain what we can confirm and what evidence we still need.

The summary should describe the observed failure, verified service and time window, and next evidence to collect. The available data did not support a confirmed cause.

A large query result and an unresponsive browser

An engineer reproduced a query through MCP and inspected a small sample of large records. The assistant initially confused the sizes of log records with the size of the HTTP response, then corrected its interpretation.

Use a follow-up that separates measurements from estimates:

Separate the result count, sizes of sampled records, estimated total size, and measured HTTP response size. Label each as measured, estimated, or unknown. What would we need to measure to determine whether the response caused the browser to become unresponsive?

Large sampled records can justify measuring the response size. To establish why the browser became unresponsive, collect evidence from the response and the browser itself.

A notification incident ready for handoff

Before accepting an RCA summary, an engineer asked whether the problem was still occurring. The assistant queried a fresh window and checked successful deliveries and HTTP responses. Engineers performed the remediation separately.

Use a follow-up that requires a current check:

Before finalizing the RCA, check whether deliveries are succeeding now. State the new query window and show success and failure evidence. Update the status based on the new results and keep the engineer's completed actions separate from proposed follow-ups.

Troubleshooting

Symptom	Next step
The service cannot be found	Ask for matching service names; confirm environment and data source.
Queries return no matches	Check the time zone, retention, filters, and whether that signal is being ingested.
Logs and traces disagree	Align their time windows and filters; check whether a count covers requests, spans, or log records.
A response shows only a sample	Ask for a count of all matching records or use more specific filters before drawing conclusions about the full result set.
A tool fails or access is denied	Resolve the query or access issue, then rerun it. Keep that signal marked as unverified.
A possible cause lacks evidence	Ask what observation would confirm or reject it; collect that evidence before closing the RCA.

For connection problems, see Slack troubleshooting or MCP troubleshooting.

Please get in touch with us on Discord or Email if you have any questions.