Agent metrics

Six metrics cover a deployed agent, one for each way it can quietly go wrong: cost, speed, tools, user opinion, invention and rule-breaking. Each one comes from the trace (see traceabilityTraceabilityLog the whole reasoning trace, so any answer can be explained afterwards.Opens in a new tab).

Metric	What it tells you	Where it comes from
Token cost, per query and per user	What an answer costs, and who is spending it	Token counts on each model call
Latency, per thought	Where the time goes inside the loop	Duration of each model and tool span
Tool call success rate	Whether tools are called correctly and answer	Tool spans and their errors
User sentiment and feedback	Whether the user thinks it helped	Ratings and comments attached to the trace
Hallucination rate	How often it invents content	User reports, or an LLM judge on a sample
Guardrail violation rate	How often unsafe queries were blocked	The guardrail log

Per thought, per user

Measure per thought, not only per request. A slow answer is rarely one slow call, it is five calls in a loop, and only per-step latency shows which one. Cost works the same way: a total hides that one user, or one query shape, is spending most of it.

The two that need judging

Cost, latency and tool errors are counted straight off the spans. LangSmith’s prebuilt dashboards cover traces, latency, error rates, token and cost totals, tool stats and feedback scores, and OpenTelemetry’s GenAI conventions define agent duration, tool call counts and tool duration as metrics (still in development).

Hallucination rate has no counter. It comes from users reporting wrong answers, or from running a judge over a sample of traces (see LLM as judgeLLM as judgeA second model scores the result against written criteria.Opens in a new tab). Guardrail violations are the other one: log every blocked unsafe query (see guardrailsLLM guardrailsChecks around the model, so a bad input or a bad output is caught by something that does not rely on the model behaving.Opens in a new tab). A rise can mean someone is probing the agent, or that a change made it drift.

The catch

A metric says that something moved, not why. The number points, the trace explains. When one of these shifts, open the traces behind it before deciding what to change.