# Verify AI Generated Code in Production with Traces, Metrics and Logs > You can't review every line an AI assistant writes. Verify AI generated code in production by comparing traces, metrics and logs across releases, with a skill your agent can run. Source: https://last9.io/blog/verify-ai-generated-code-with-production-observability/ A reviewer can read a diff. Nobody can read the diff for every change an AI assistant produced this week, and the volume only goes one way. If review is the only gate, AI generated code ships on trust. Two recent Hacker News threads put the gap plainly. One commenter asked whether anyone cares what happens in production once the code is written. Another said AI coding needs a lot of care around observability and verification. Both are pointing at the same missing step: the code ran, and nobody checked what it did. This post covers that step. We tag every signal with the release that shipped, then compare traces, metrics and logs between releases, using OpenTelemetry data. It ends with a skill you can paste into your coding agent so it runs the comparison after each deploy. ## Why reading AI generated code doesn't scale Review time per change stays roughly flat while the number of changes grows. That is arithmetic, not a complaint about reviewers. Past a certain volume, a reviewer skims. The failures that get through are rarely syntax errors. Plausible code tends to fail on the things a diff does not show: - **Load.** A loop that is fine for ten rows and slow for ten thousand. - **Data shape.** A field that is null in production and never in the test fixtures. - **Dependencies.** A new call to a service that has a lower rate limit than the code assumes. - **Query count.** One database call per item instead of one per request. None of these look wrong in a pull request. All of them show up in telemetry within minutes of real traffic. That is why the verification step belongs after deploy as well as before merge. ## Is AI generated code safe? Move the check to production Static checks, tests and review catch a share of defects before merge. Production shows the rest: behavior under real traffic, real data and real dependencies. The move is to define "works" as signals before the change ships. For a service, that usually means four numbers you can compare across releases: 1. Error ratio on server spans. 2. p95 latency on the same spans. 3. Calls per request to each downstream dependency. 4. CPU, memory and connection pool use. Then you let the data answer. If the new release matches the old one on all four, the change behaves. If one moves, you have a specific place to look. This does not make AI generated code safe by itself. Telemetry detects behavior drift. It will not tell you that a query is open to injection or that a discount is calculated wrong but consistently. We cover those limits near the end. ## Tag every signal with the release that shipped A release comparison needs to know which release produced each span, metric and log line. Set it once, as a resource attribute, so it lands on everything the process emits: ```shell export OTEL_RESOURCE_ATTRIBUTES="service.name=checkout,service.version=2026.11.3,deployment.environment.name=production" ``` `service.version` and `deployment.environment.name` come from the OpenTelemetry [resource semantic conventions](https://opentelemetry.io/docs/specs/semconv/resource/). Use a value that maps to one deploy: a release number or a build ID. A commit SHA also works on traces, but keep it out of metric labels, where each new value creates a new series. Then mark the deploy itself. Our [change events API](https://last9.io/docs/change-events/) records a deployment as the `last9_change_events` metric, with the `attributes` you send as labels: ```shell curl -XPUT https://app.last9.io/api/v4/organizations/{org_slug}/change_events \ --header 'Content-Type: application/json' \ --header 'X-LAST9-API-TOKEN: Bearer ' \ --data-raw '{ "event_name": "new_deployment", "event_state": "start", "attributes": { "service_name": "checkout", "deployment_environment": "production", "version": "2026.11.3", "ai_assisted": "true" } }' ``` Set `service_name` to your APM service name exactly, or the event will not show as a marker on charts. It is still stored and queryable. `ai_assisted` is an attribute we made up for this example. If your process records which changes an AI tool wrote, a label like that lets you compare AI-assisted releases against the rest. If it does not, skip it. The comparison works without it. Send a `stop` event when the rollout finishes. The window between the two is your bake window. ## Traces: confirm the new code takes the path you expect Metrics tell you that something moved. Traces tell you what the code did differently. For AI generated code, the useful question is whether the new release takes the path you meant it to take. Filter traces to the new `service.version` in [Traces Explorer](https://last9.io/docs/traces-explorer/) and pick the same endpoint on both versions. Compare the span trees for three shapes: - **New database calls.** A request that used to make two queries now makes twenty. That is the N+1 pattern, and tests with three fixture rows never trigger it. - **Extra fan-out.** One request now calls a downstream service for each item in a list. - **Retries and loops.** The same call repeated three times under one parent, usually a retry added to a call that did not need one. The span count per trace is the quickest comparison. If the median trace for `POST /checkout` went from 14 spans to 60, you do not need to read the code to know something changed. Trace attributes help here too. Our MCP server exposes `get_trace_attribute_deviations`, which ranks the attribute values that differ between slow and fast requests, so you can ask what the slow ones have in common. When `service.version` is the attribute that separates them, you have your answer. The tool needs the trace-analysis capability enabled for your tenant, so check that it appears in your tool list before you rely on it. ## Metrics: compare error rate, latency and saturation across releases You can get per-release request metrics from spans without touching application code. The OpenTelemetry Collector's [spanmetrics connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector) generates call counts and duration histograms from spans. Its `dimensions` setting looks a name up in span attributes and then resource attributes, so the release version can become a label. Our guide to [converting traces to metrics with spanmetrics](https://last9.io/blog/convert-opentelemetry-traces-to-metrics-using-spanconnector/) covers the full setup. For this use, the dimensions are the part that matters: ```yaml connectors: span_metrics: namespace: release dimensions: - name: service.version - name: http.route service: pipelines: traces: receivers: [otlp] exporters: [span_metrics, otlp/last9] metrics/span_metrics: receivers: [span_metrics] exporters: [otlp/last9] ``` The component is named `span_metrics` in current Collector releases. Older releases and some of our integration docs call it `spanmetrics`, which still works as a deprecated alias, so use the name your Collector version accepts. Here `otlp/last9` stands for your existing OTLP exporter to Last9. Keep dimensions to attributes with a small set of values. A route template is fine. A raw URL, a user ID or a commit SHA would create a series per value. With that running, you can compare the two releases directly. Exact metric and label names depend on your exporter's normalization, with `.` in label names rewritten to `_`, so confirm them in [Metrics Explorer](https://last9.io/docs/metrics-explorer/) before you build alerts on them: ```promql # Error ratio on server spans, by release sum by (service_version) ( rate(release_calls_total{service_name="checkout", span_kind="SPAN_KIND_SERVER", status_code="STATUS_CODE_ERROR"}[10m]) ) / sum by (service_version) ( rate(release_calls_total{service_name="checkout", span_kind="SPAN_KIND_SERVER"}[10m]) ) # p95 latency, by release histogram_quantile(0.95, sum by (le, service_version) ( rate(release_duration_bucket{service_name="checkout", span_kind="SPAN_KIND_SERVER"}[10m]) ) ) ``` Request metrics are not enough on their own. A change can leave error ratio and latency alone while it doubles memory use or holds a connection open. Add the saturation side: container CPU and memory, and connection pool use, grouped by the same release label where your exporter provides one. One condition: this comparison needs traffic on both versions at the same time. A canary or a rolling deploy gives you that. If you cut over all at once, compare the new release against the same window a day earlier instead. Our post on [canary monitoring with Argo Rollouts](https://last9.io/blog/argo-rollouts-canary-monitoring-last9/) shows how to turn the same queries into an automated promotion gate. ## Logs: catch new error patterns behind a 200 Some of the worst AI generated code never throws. It catches an exception, logs a warning and returns a default, so the request ends in a 200 and the metrics stay quiet. The evidence is in the logs. Look for log signatures that did not exist in the previous release. A new `WARN` message that appears only on the new version is a strong signal, especially phrases like "falling back", "using default" or "ignoring error". If your log records carry `service.version`, count lines by level for each release. Adjust the label and field names to your log schema: ```text sum by (level) (count_over_time({service="checkout"} | json | service_version="2026.11.3" [10m])) ``` Then pivot from a suspect line to its trace through the `trace_id`. If your logs carry no `trace_id`, no tool can join them to a trace by key. Our post on [correlating logs, traces and metrics](https://last9.io/blog/correlate-logs-traces-metrics-for-faster-root-cause/) walks through those pivots and the checks to run before the next incident. ## How to review AI generated code after it ships: a release checklist Run the same ordered list for every release, whoever or whatever wrote it. The order puts the cheapest checks first. 1. **Errors.** Error ratio on server spans, new release against previous. 2. **Latency.** p95 and p99 on the same spans. 3. **New spans.** Span count per trace and any new database or HTTP client spans. 4. **New logs.** Log signatures present only on the new version. 5. **Resources.** CPU, memory and connection pool use. 6. **Dependencies.** Calls per request to each downstream service, and their error rates. Decide the bake window and the rollback criteria before the deploy. "Roll back if error ratio is above the previous release by more than X for Y minutes" is a rule. "Roll back if it looks bad" is an argument at 2 a.m. We have no universal values for X and Y. A lookup service and a batch exporter tolerate different amounts of drift, so set them from your own history. Where you already use progressive delivery, wire the same checks into the gate so the data decides. Our write-up of [CI/CD observability](https://last9.io/blog/ci-cd-observability/) covers how deploy markers connect a regression back to the pipeline run that shipped it. ## Hand the checklist to your agent The checklist is mechanical, which makes it a good job for the agent that wrote the change. There are two copy-paste pieces below: a prompt for before the merge, so the change can be verified at all, and a prompt plus a skill file for after the deploy. The after-deploy pieces assume your agent is connected to the Last9 MCP server, covered in the next section. ### Before merge: make the change observable A comparison across releases only works if the new code emits something to compare. Ask the agent to add that while it still has the diff in context: ```text Review the change you just wrote and make it observable. Do not change behavior. 1. For each new function that calls a database, an HTTP client, a queue or another service, check that an OpenTelemetry span covers the call. Use the instrumentation already in this repo. Do not add a second tracing setup. 2. List every place the change catches an exception and continues. For each, make sure the error is recorded on the active span and logged at ERROR or WARN with the trace ID, so a fallback cannot hide behind a 200. 3. Confirm service.version is set as a resource attribute, from the release number or build ID the pipeline already has. Do not hard-code a value. 4. Add no metric label whose values are unbounded, such as user IDs, raw URLs or commit SHAs. Report what you changed and what you could not verify. ``` ### After deploy: run the release comparison Paste this into a chat with your coding agent right after a deploy: ```text Verify the release I just deployed, using Last9 MCP tools only. Inputs - service: - environment: - new release (service.version): - previous release: - deploy time (RFC3339), or say "look it up": Steps 1. If I did not give a deploy time, call get_change_events for the service and environment and use the latest deployment. If the service name returns nothing, call did_you_mean first. 2. Call get_apm_service_deviations with the 60 minutes after the deploy as the current window and the 60 minutes before it as the baseline. 3. With prometheus_range_query, compare and on: error ratio, p95 latency, and request rate. Group by service_version. Confirm the metric and label names exist before querying. 4. Call get_trace_attribute_deviations to see which attributes separate slow or failing requests. Call get_trace_waterfall on one slow trace from the new release and report span count and any repeated database or HTTP calls. 5. Call get_service_logs for the new release and list error or warning messages that do not appear for the old release. 6. Check CPU, memory and connection pool saturation for the service. Report - A table: signal, old release, new release, change. - Verdict: PASS, HOLD or ROLL BACK, judged against my criteria: error ratio no more than above the old release, p95 no more than above it, no new error signatures. - Evidence for every claim, with the tool call it came from. - Anything you could not check, and why. Do not treat an empty result as healthy. Do not roll back, restart or change anything. Recommend only. ``` ### Save it as a skill If you want the agent to apply the after-deploy steps every time without being asked, save the same instructions as a skill. The [Agent Skills format](https://skills.sh) is a folder with a `SKILL.md` file, and agents such as Claude Code, Codex and Cursor can load it: ```markdown --- name: verify-release description: Verify a production release with traces, metrics and logs. Use after a deploy, or when asked whether a recent change is safe, healthy or should be rolled back. --- # Verify a release Use the Last9 MCP tools. Do not roll back, restart or change anything. ## Inputs Service, environment, new service.version, previous service.version. If the deploy time is missing, get it from get_change_events. ## Steps 1. Resolve names with did_you_mean if a lookup returns nothing. 2. Compare the post-deploy window with the equal-length window before it using get_apm_service_deviations. 3. Query error ratio, p95 latency and request rate by service_version with prometheus_range_query. Check that the metric names exist first. 4. Use get_trace_attribute_deviations and get_trace_waterfall to find new span shapes: extra database calls, fan-out, retries. 5. Use get_service_logs to find log messages that exist only on the new release. 6. Check CPU, memory and connection pool saturation. ## Report A table of old against new for each signal, then PASS, HOLD or ROLL BACK against the margins the user gave. If no margins were given, ask for them before giving a verdict. Cite the tool call behind every number. List what could not be checked. An empty result is not a healthy result. ``` Two notes on using these. Fill in the margins yourself, because the agent has no way to know what your service tolerates. And read the table before you trust the verdict: the agent reports what the tools returned, and the Last9 docs say your client controls which tool calls need confirmation, so review its permissions first. Our [agent skills page](https://last9.io/docs/ai/agent-skills/) lists the skills we ship, including logs, traces, CloudWatch and Go instrumentation, which you can install alongside this one. `verify-release` is not one of them. It is a starting point to adapt. ## Doing this on Last9 Last9 is OpenTelemetry-native, so the setup above is the setup: - **Send OTLP.** Spans, metrics and logs with `service.version` set as a resource attribute. - **Send change events.** One per deploy, from your pipeline, through the API shown earlier. They appear as markers on Discover service pages, and you can query them with PromQL. - **Filter by version.** Use Traces Explorer for span trees and Metrics Explorer for the per-release queries. - **Connect the agent.** The [Last9 MCP server](https://last9.io/docs/ai/mcp/) gives your coding agent tools for traces, logs, metrics, change events and service deviations, which is what the skill above calls. Our [launch post](https://last9.io/blog/launching-last9-mcp-server/) shows the prompts and the tools they trigger. A tradeoff worth knowing: `get_apm_service_deviations` compares a current time window with an equal-length baseline window. It compares windows, not release labels. For a per-release comparison, use the PromQL queries. For a quick "did something regress after 2 p.m." check, the deviation tool is faster. We keep high-cardinality labels with no sampling, so a label you do need, such as a build ID on traces, is not something you have to drop to control cost. ## What this approach does not catch Production verification has limits, and a clean report is not a guarantee. - **Low-traffic paths.** A branch that runs once a day may not execute inside your bake window. A clean release can still hide a broken rare path. - **Wrong but successful.** Code that returns a valid response with the wrong number looks healthy to every signal above. Catching it takes tests or business-level metrics, such as refunds issued or orders per minute. - **Security flaws.** An injectable query behaves normally until someone exploits it. Use scanning before merge. Telemetry only helps after the exploit shows up, and only if you are looking for it. - **Thin baselines.** If the previous release had little traffic, the comparison is noisy. Widen the window or use a longer baseline. - **Attribution.** A release that changes three things can move a metric for reasons unrelated to the AI-written part. Smaller, more frequent deploys make the signal cleaner. Observability is the last check, not the only one. It is the one that sees the code running against real traffic. ## FAQ ### Is AI generated code safe? It is as safe as the checks around it. AI generated code can look correct and still fail on real traffic, unusual data or a slow dependency, and it can carry security flaws that no metric shows. Treat it like any other change: run tests, static analysis and security scanning before merge, then confirm behavior after deploy by comparing the new release's traces, metrics and logs with the previous release. Production telemetry catches drift in behavior. It does not prove the code is secure or logically correct. ### How to review AI generated code? Review it in two places. Before merge, read the diff for intent, run tests and scan for security issues. After deploy, compare the new release with the previous one: error ratio, p95 latency, new span shapes such as extra database calls, new log signatures, and CPU, memory and connection pool saturation. Tag every signal with `service.version` and send a change event per deploy, so the comparison window is exact and a regression points back to one release.