Monitor Search Quality, Latency & Cost in AI Agents
This guide shows you how to set up practical monitoring for AI agents that perform search tasks, covering quality (are results relevant?), latency (how fast?), and cost (how expensive?). You’ll learn to define metrics, collect data, and build dashboards that help you catch problems early and optimize your agent over time. No prior monitoring experience is needed, but basic familiarity with logging and APIs helps.
What You’ll Need
- Access to your AI agent’s logs or a way to add logging (e.g., console, file, or a logging service).
- Basic knowledge of your agent’s architecture: what search API it calls, how it processes results, and where the user request enters/exits.
- A simple spreadsheet or a dashboard tool (like Grafana, Datadog, or even a Google Sheet) to visualize metrics.
- If your agent uses a managed LLM service, access to its usage dashboard (for token counts and cost) or an API that returns usage data.
Step-by-Step
- Define your metrics. Start with the three core categories:
- Quality: Relevance score (if available), click-through rate (if you have UI), user feedback (thumbs up/down), or a manual rating sample.
- Latency: Time from user request to final response (end-to-end), time spent in the search API call, and time spent in the LLM generation.
- Cost: Cost per search query (or per 1k queries), token usage (input + output), and number of API calls per query.
- Instrument your agent code. Add logging at key points:
- Log the start and end timestamp of each search request.
- Log the search API call duration separately.
- Log the number of tokens used by the LLM (most SDKs return this in the response).
- Log the final answer and the search results (for later quality review).
- Calculate cost per query. If you know the price per 1K tokens for your model, compute cost = (input tokens/1000 * input price) + (output tokens/1000 * output price). Add any search API costs (e.g., per call or per 1K results). Log this number with each query.
- Store metrics in a structured format. Use JSON or CSV lines with fields: timestamp, query_id, latency_ms, search_latency_ms, llm_latency_ms, input_tokens, output_tokens, cost_usd, quality_score (if available). This makes it easy to aggregate later.
- Set up a dashboard. Create visualizations:
- Latency: average and p95 (95th percentile) over time.
- Cost: total cost per day, cost per query over time.
- Quality: average relevance score or feedback rating over time.
- Correlation: overlay latency and cost on the same chart to spot trends.
- Establish baselines and alerts. After a week of data, set thresholds: e.g., alert if p95 latency exceeds 5 seconds, or if cost per query doubles. Use your dashboard tool’s alerting features or a simple script that checks logs.
- Review quality manually. Periodically sample queries (e.g., 50 per week) and rate the relevance of the agent’s answer against the user’s question. Log these ratings to track quality over time.
- Optimize iteratively. Use the data to make changes: reduce the number of search API calls, switch to a cheaper model, or add caching for common queries. Then re-measure to see if quality/latency/cost improved.
Common Mistakes
- Ignoring p95 latency. Average latency can look fine while 5% of users wait too long. Always track percentiles.
- Not separating search API latency from LLM latency. If you only measure end-to-end, you won’t know which part is slow.
- Forgetting to log token usage. Many SDKs return usage, but you must capture it. Without it, you can’t compute cost.
- Using only one quality metric. Click-through rate can be misleading if the UI is poor. Combine automated signals with manual review.
- Not cleaning up logs. Logs grow fast; set up rotation or retention policies to avoid storage costs.
Tips
- Use a unique query ID to correlate logs across different services (e.g., search API, LLM, final response).
- Start with a simple logging library (like Python’s
loggingor Node’sconsole) before investing in a full observability platform. - If you use a managed LLM, check if its API provides a
usagefield—use it directly. - For cost, consider caching frequent queries to avoid repeated LLM calls.
- Review your metrics weekly, not daily, to avoid overreacting to noise.
- If you have a UI, add a simple feedback button (thumbs up/down) to collect quality data at scale.
Troubleshooting
- No logs are appearing: Check that your logging level is set correctly (e.g., DEBUG vs INFO). Ensure the logging handler is writing to the expected destination.
- Token usage is missing: Some SDKs require you to explicitly request usage in the API call (e.g.,
return_usage=True). Check the documentation. - Cost calculation seems wrong: Verify the price per token for your model—prices change. Also check if you’re using the correct unit (per 1K or per 1M tokens).
- Latency spikes are unexplained: Look at the search API’s own status page or logs. Also check if the LLM is hitting rate limits—add retry logic with backoff.
- Quality scores are not changing: Manual review might be too infrequent. Increase the sample size or use a more granular rating scale.
- Dashboard is too slow to load: Aggregate data into hourly or daily summaries instead of querying raw logs every time.
Frequently Asked Questions
What is a good latency target for an AI search agent?
It depends on your use case. For interactive chat, aim for under 2 seconds end-to-end. For batch processing, 10 seconds may be fine. Track p95 and set a target based on user expectations.
How do I measure quality if I don’t have user feedback?
Use manual evaluation: sample queries and rate the relevance of answers. You can also use automated metrics like BLEU or ROUGE if you have reference answers, but they’re imperfect.
What’s the best way to reduce cost?
First, reduce the number of LLM calls per query (e.g., by caching). Then consider a smaller model or using a cheaper search API. Monitor cost per query after each change.
Can I monitor all this with free tools?
Yes. Use a spreadsheet for manual logging, or open-source tools like Grafana + Prometheus. For small volumes, simple log files and a script to compute averages work fine.
How often should I review the metrics?
Weekly is a good starting point. Daily if you’re actively optimizing. Set up alerts for critical thresholds so you’re notified immediately when something breaks.
Should I log every query or sample?
Log every query for latency and cost—it’s cheap. For quality, sample and manually review a subset (e.g., 50 per week) to keep effort manageable.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.