Build your own AI SRE on Last9 data

Connect your own agent to Last9 MCP to investigate an alert or answer an operational question using Last9 telemetry. You choose the model, trigger, conversation storage, and destination for the answer. Last9 provides tools for querying logs, metrics, traces, exceptions, alerts, and other supported data.

This page describes how to design that agent. For managed chat in Last9, use AI Assistant or Slack. For team conversations and a framework option for your own shared agent, see Collaboration mode.

Where this guidance comes from

Some guidance on this page comes from an observed integration, where a team connected an internal chat bot to the managed Last9 assistant. The notes marked From an internal chat integration record what failed in that deployment. They omit identifying details, and they describe observed failures rather than a recommended implementation.

Prerequisites

Your agent reaches Last9 telemetry through Last9 MCP, so make the connection work before you design the workflow.

AI access enabled for your Last9 organization. See Enable Last9 AI.
Relevant logs, metrics, or traces flowing into Last9.
An agent framework or SDK that can call a remote MCP server and run a tool-use loop.
An MCP-type Client token from Last9 Query Tokens, kept in your application’s secret storage. See MCP for agents for token setup and SDK examples.
Your organization’s hosted MCP endpoint, which you can check on the MCP page in Last9:
```
https://app.last9.io/api/v4/organizations/<org-slug>/mcp
```
Send the token using the authentication mechanism your MCP client supports. Connect the client and list the available tools, then run one read query against a service you know has data. Use the tool schemas returned by the connection to construct calls, because parameters and availability depend on the connected server.

Design the investigation workflow

Decide what the agent investigates, which tools it can call, and what it must return, before you connect it to an incident channel.

Start with one workflow

Pick a specific job, such as explaining a service alert, investigating increased latency, or summarizing service errors. Divide the work between the following parts of your application:

Part	Responsibility
Trigger	Receive an alert, chat message, or request containing an operational question.
Your application	Authenticate the request, record the affected service and time window, retain conversation context, and run the agent.
Model	Select from the tools you expose, interpret their results, and ask for missing information.
Last9 MCP	Execute supported queries against Last9 and return telemetry results.
Report destination	Show the findings, evidence links, open questions, and next action in your application or incident thread.

First, your application receives an alert, chat message, or request. It then sends the question to the model and runs the model’s Last9 MCP tool calls until it can return an investigation report. Your application controls execution limits and access to tools.

Preserve incident context

Store the incident’s service, environment, and time window with the conversation, separately from the model’s written answer. Follow-ups such as “share the root cause analysis” can then use the same incident details. Identify each conversation by organization, user, and chat or incident thread to keep unrelated investigations separate.

For example, your application could store the following record:

{
  "incident_id": "incident-123",
  "service": "checkout-api",
  "environment": "production",
  "start_time": "2026-09-10T14:00:00Z",
  "end_time": "2026-09-10T14:30:00Z",
  "question": "Why did HTTP 5xx responses increase?"
}

The example illustrates a record in your application, not a Last9 API payload. Store the verified organization and data source alongside it. Add request IDs, trace IDs, alert details, and corrected service names as they become available.

Use the same absolute time window when comparing logs, metrics, and traces. For a follow-up about recovery, query a new window and keep the original incident window for comparison.

Store the original message separately from a follow-up rewritten to include prior context. Preserve the service, host, environment, time window, and requests for logs or root cause analysis. Compare the message your application received with the text sent to the model, especially when your bot also creates a preview or summary for display. For example, rewriting “compare it with last week” should add the previously selected service and time window without dropping the comparison request.

Select tools for the job

Expose a small set of read tools appropriate to the workflow. Confirm the names and schemas in your connected server. The MCP tool reference describes the available categories.

Question	Useful starting points
Which service or data source matches the request?	Use entity and data-source discovery tools.
How much HTTP traffic arrived, and from which callers?	Query access logs for the service and time window, then group by observed path, status, client, or user-agent fields.
What failed?	Inspect exceptions and service logs, then use log queries to summarize matching errors.
Why was a request slow?	Check service performance, traces, and trace waterfalls, then compare groups of requests.
What changed near the incident?	Check change events and alert details against telemetry from the same time window.
Is an AWS resource constrained?	Discover available metrics, confirm the resource, statistic, and unit, then query the selected resource.

Discover field names before building filters. For example, two services may record HTTP status under different attributes. A filter on a guessed field can return no matches even when the failures are present.

Reuse the Last9 agent skills for log, trace, and CloudWatch investigation instructions where your agent supports them. Keep their references available to the agent and use the connected tools’ schemas as the parameter source.

Guide your agent to discover available access log fields and filter by the relevant host, path, method, and time. Use traces to add request detail when available. If one service lookup finds nothing, verify the service and time window, then check other relevant signals before ending the investigation.

Report only dimensions that the returned data supports. For example, a request count alone cannot establish which client made the requests.

For daily p95 or p99 latency, specify which requests to include, the duration unit, and the time zone that defines each day. Keep the service, route, environment, and success or error filters consistent across days. Include request counts so days with little traffic are visible in the comparison.

Calculate each percentile from suitable data for the requests included on each day. Averaging hourly p95 values does not produce a daily p95. If the available metric only exposes precomputed percentiles, explain which requests and time period the series represents before comparing its values.

Give the agent a clear contract

An investigation contract states what the agent must check and return. Use the following requirements as a starting point for your agent’s prompt:

Investigate the supplied incident using the enabled Last9 read tools.

1. Confirm the service, environment, data source, and time window.
2. Discover available fields before constructing filters.
3. Establish the observed symptom, then test possible explanations
   with related telemetry for the same service and time window.
4. Distinguish aggregate measurements from sampled records.
5. Report failed queries, missing data, and execution limits explicitly.
6. Return confirmed findings with evidence links, possible causes
   with supporting evidence, and the next targeted check.
7. Describe a fix as proposed unless execution is separately
   confirmed. Never infer recovery from silence alone.

Implement permissions in your application or MCP client. For a read-only investigator, maintain a list of permitted read tools and check it when exposing or calling tools. Exclude dashboard creation, dashboard updates, and other changes. A prompt asking for read-only behavior is not an access control.

Set limits and return partial results

Set limits on elapsed time, tool calls, query windows, and result sizes in your application. Start with limits that let the workflow complete, then adjust them based on test runs.

When a query fails or the agent exceeds a limit, return the evidence already collected and the next useful check. Record whether each query succeeded, returned no matches, returned a sample, or failed. Do not treat a timeout as a zero count or proof that no matching data exists.

An investigation report should include the following details:

State which service and time window were investigated.
List the confirmed findings and supporting queries or links.
Explain the possible causes, what could not be checked, and what would confirm or reject each cause.
Recommend the engineer’s next step.

Store the relevant tool results or evidence references with the incident. A follow-up can then reuse verified observations without rerunning every query, while still making a fresh query when asked about current conditions.

An interactive chat can ask for missing information. An automated caller needs a predictable outcome. When the request already supplies a service and time window, reuse them rather than asking again.

If an automated request lacks a required service, resource, or time window, return a structured incomplete result or error listing the missing fields. Do not leave the job waiting for a conversational answer the caller cannot provide. Define completed, incomplete, and missing-input outcomes in your application rather than parsing them from the model’s final sentence.

Limit concurrent investigations and record failures before increasing the number of requests. Set retry limits for read queries, and retry only errors that may be temporary. Test concurrent calls, long query windows, and retries with your own workload.

Test and extend the agent

Confirm the agent’s behavior against real telemetry before an engineer depends on it. Add any change to a system as a separate workflow.

Test the workflow with real telemetry

Before connecting the agent to an incident channel, test it with the following cases:

Case	What to verify
A service with known errors	The agent finds the relevant evidence for the intended service and time window.
A misspelled or ambiguous service	It resolves or asks about the service before drawing conclusions.
A service with logs but no traces	It uses the available signal and reports the trace gap.
An empty result	It distinguishes no matching records from a measured zero or a failed query.
A query timeout or execution limit	It returns an incomplete investigation with usable evidence and a next step.
A follow-up requesting a root cause analysis	It retains the incident window and verified findings.
A long prompt or rewritten follow-up	It preserves the full original request, service, environment, and comparison periods.
A recovery check	It queries a fresh window and checks successful activity as well as failures.

Inspect the actual tool calls and results, not just the final wording. Use the incident investigation guide to review the report an engineer will receive.

Add remediation as a separate workflow

Once the investigation is useful, you can add a separately authorized action flow in your application. Show the proposed operation, target, and arguments to a person with permission to approve it. Record the execution result and verify the affected system afterward.

Keep the investigation report clear about who performed each action. Writing the report, proposing a fix, executing a change, and confirming recovery are separate outcomes.

Refer to MCP for agents for connection examples or Last9 MCP for the tool reference.

Troubleshooting

The agent cannot list any Last9 tools

Check the organization slug in the MCP endpoint, and confirm the token is an MCP-type Client token from Query Tokens. Run one read query against a service you know has data.
A follow-up loses the incident details

Store the service, environment, and time window with the conversation, separately from the model’s answer. Pass the stored record into each follow-up.
A filter returns no matches although the failures are present

Have the agent call the discovery tools and build the filter from the returned field names. Two services can record the same value under different attributes.
A query times out and the report states a zero count

Record the outcome of each query separately, then return the evidence already collected and the next useful check. Do not treat a timeout as a measured zero.
The agent calls a tool your workflow does not allow

Maintain a list of permitted read tools in your application or MCP client, and check it when you expose or call a tool. A read-only instruction in the prompt is not an access control.

Please get in touch with us on Discord or Email if you have any questions.