# Investigate an incident with AI

> Investigate an affected service, inspect telemetry, test possible causes, and verify recovery with Last9 AI.

Source: https://last9.io/docs/ai/investigate-an-incident/

Use Last9 AI to gather evidence for an incident, test an explanation, and prepare a summary for your team. Start with the affected service and time window, and ask for the queries behind each conclusion. Verify recovery separately before closing the incident.

You can use the prompts below in [AI Assistant](/docs/ai/ai-assistant/), [Slack](/docs/ai/slack/), or an agent connected through [Last9 MCP](/docs/ai/mcp/). Available tools and approval controls depend on the interface and its configuration.

## Before you start

- Confirm that the relevant logs, metrics, or traces are available in Last9.
- Identify the service, environment, and data source you want to investigate. If you only know an application nickname, ask the assistant to find the matching service first.
- Record the incident's start and end time, including the time zone. Use the same window when comparing results from different signals.
- Include an alert, request ID, trace ID, or error message when you have one.

## Run the investigation

Work through the following steps in order. Each step builds on the evidence the previous one returned.

### 1. Define the question

A question with a specific service and time window gives you a result you can check:

Replace the example service and time window with your incident details. If a query returns no data, verify the service and time window before concluding that the service is healthy.

### 2. Establish what happened

Ask for the symptom before the explanation:

Open the returned links in [Logs Explorer](/docs/logs-explorer/), [Traces Explorer](/docs/traces-explorer/), or [Exceptions](/docs/discover-exceptions/). Check the service, filters, and time range against the question you asked.

Explain what each query result establishes:

| Evidence                         | What it establishes                                                                            |
| -------------------------------- | ---------------------------------------------------------------------------------------------- |
| A failed request or exception    | A specific failure was observed.                                                               |
| Error counts over a fixed window | The count shows how often a failure matched the query's filters and time window.               |
| A trace with related spans       | The trace shows how an instrumented request moved through services and dependencies.           |
| A sample of returned records     | The sample shows what the selected records contain, but may not represent the full result set. |
| A query with no matches          | The query returned no matching data. Check the filters, time window, and data availability.    |

### 3. Test a possible cause

Ask the assistant to connect the symptom to evidence, and to describe what would disprove its explanation:

If [change events](/docs/change-events/) are available, compare the incident with deployments and configuration changes. Investigate any deployment near the incident, but do not treat timing alone as proof of a cause.

For latency, ask which spans account for the delay and compare similar successful or faster requests. For errors, inspect the failing operation and its related logs. Report any missing dependency instrumentation or correlation IDs in the conclusion.

### 4. Verify recovery in a new window

An incident summary describes the data already queried. It does not establish the system's current condition.

Check for successful activity as well as fewer errors. A quiet service may have no errors because it received no traffic. A single successful retry confirms that request succeeded, but leaves sustained recovery unverified.

### 5. Prepare the handoff

A handoff is a summary another engineer can use to continue the investigation. Ask the assistant to include the findings, completed actions, and remaining questions:

Review any proposed change before approving it in your client. For example, creating or updating a dashboard changes saved configuration. An investigation recommendation does not mean the assistant has applied a fix.

## Examples from incident investigations

The examples below are adapted from incident investigations. Names and details are changed, and the prompts are illustrative.

### An outage question followed by an RCA request

Operators use Last9 AI to ask about service health, p95 latency, recent errors, and outages, then request a root cause analysis (RCA). In one outage investigation, the assistant queried several signals but stopped at its investigation limit without enough matching telemetry to establish a cause.

Before writing an RCA from an incomplete answer, ask which evidence is missing:

An incomplete investigation can still identify the next check. Its unanswered questions belong in the handoff.

### An API failure with an unconfirmed downstream cause

An investigation located a failed request and a later successful, slower request. The engineer corrected the service name during the investigation. The available evidence confirmed the failure but did not establish which downstream component caused it.

Use a follow-up that asks for the missing connection:

The summary should describe the observed failure, verified service and time window, and next evidence to collect. The available data did not support a confirmed cause.

### A large query result and an unresponsive browser

An engineer reproduced a query through MCP and inspected a small sample of large records. The assistant initially confused the sizes of log records with the size of the HTTP response, then corrected its interpretation.

Use a follow-up that separates measurements from estimates:

Large sampled records can justify measuring the response size. To establish why the browser became unresponsive, collect evidence from the response and the browser itself.

### A notification incident ready for handoff

Before accepting an RCA summary, an engineer asked whether the problem was still occurring. The assistant queried a fresh window and checked successful deliveries and HTTP responses. Engineers performed the remediation separately.

Use a follow-up that requires a current check:

---

## Troubleshooting

| Symptom                          | Next step                                                                                                                  |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| The service cannot be found      | Ask for matching service names; confirm environment and data source.                                                       |
| Queries return no matches        | Check the time zone, retention, filters, and whether that signal is being ingested.                                        |
| Logs and traces disagree         | Align their time windows and filters; check whether a count covers requests, spans, or log records.                        |
| A response shows only a sample   | Ask for a count of all matching records or use more specific filters before drawing conclusions about the full result set. |
| A tool fails or access is denied | Resolve the query or access issue, then rerun it. Keep that signal marked as unverified.                                   |
| A possible cause lacks evidence  | Ask what observation would confirm or reject it; collect that evidence before closing the RCA.                             |

For connection problems, see [Slack troubleshooting](/docs/ai/slack/#troubleshooting) or [MCP troubleshooting](/docs/ai/mcp/#troubleshooting).

Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:support@last9.io) if you have any questions.