How do you monitor an LLM system in production?

I would monitor an LLM system at three levels: individual requests, multi-step traces, and aggregate service health. For every request, I would log a request identifier, the prompt or template version, model name, token counts, latency, cost, and a structured outcome. Those fields let me connect a user report to the exact call that produced it. I would avoid storing unnecessary prompt content, and I would redact personally identifiable information before any text reaches a log or observability tool.

For an agent or multi-step workflow, request logs are not enough. I would add distributed traces that show each model call, tool call, retrieval step, input, output, and error in the order they occurred. Dashboards would track error rate, p95 latency, token usage, and cost, while alerts would fire for sustained error increases, p95 latency regressions, and abnormal cost spikes. I would be careful to make alerts actionable so responders know which service and version are affected.

Operational logs and traces alone do not establish whether an output was useful. I would sample conversations for periodic human review, track thumbs-up, thumbs-down, and other user feedback, and add quality signals such as schema validity, refusal rates, or retrieval hit rates where they apply. I would segment these metrics by prompt version and model so I can detect a quality regression that averages hide. Finally, I would document retention limits, verify that redaction works, and restrict access to sensitive logs.