Monitor Token Usage Across AI Agents and Providers

Updated 9 November 2026 · Standard formula, shown below

Monitoring token usage across AI agents and model providers helps you understand what is driving consumption, compare activity consistently, and catch unexpected increases early. You will need access to the agents’ logs or tracing, provider usage data, and a shared place to record and review the results.

What You’ll Need

Access to each agent’s runtime logs, tracing system, or middleware, plus permission to view provider usage dashboards or reports.
A place to collect records, such as a database, observability platform, spreadsheet, or internal reporting system.
A consistent way to identify each request, agent, model provider, model, environment, and time.
Optional but useful: a known token-counting library or tokenizer for each model family, and a budget or alert threshold.

Step-by-Step

Decide what you need to measure. Define whether the goal is operational visibility, usage allocation, capacity planning, or cost control. Track input tokens and output tokens separately, since providers may count or price them differently. If relevant, also record cached input, reasoning, image, audio, or other modality-specific usage as distinct fields. Do not treat every token category as interchangeable.
List every agent and provider path. Inventory agents, tools, model routes, fallback providers, background jobs, and environments. Include calls made through orchestration libraries as well as direct API calls. A single user task may trigger multiple model requests, so monitoring only the top-level request can hide substantial usage.
Choose a common record format. Create one event per model request, with fields such as timestamp, request ID, parent task or trace ID, agent name, environment, provider, model, input tokens, output tokens, and status. Add cached or modality-specific token counts when available. Keep units and field meanings documented; for example, distinguish provider-reported usage from a locally estimated count.
Capture provider-reported usage first. After each response, read the usage metadata returned by the provider and save it alongside the request record. Use the provider’s own reported values for billing-oriented reporting when available. Store the raw response metadata or relevant fields so you can investigate changes later. Check that failed, interrupted, streamed, and retried requests are handled consistently; a response that ends early may still consume tokens.
Instrument calls that do not expose usage clearly. For streaming responses, collect the final usage event or response metadata if the API supports it. If it does not, count input and output locally using a tokenizer appropriate to the selected model, and mark the result as an estimate. For tool-using agents, record each model call separately and attach it to the same parent task. Do not count tool text as model tokens unless it is actually sent to a model.
Normalize model and provider names. Map aliases and deployment-specific names to stable labels while preserving the original value. Record model version or deployment when available. Tokenization can vary by model, and provider-specific features such as caching may change what is billed, so avoid comparing raw totals as if every model uses identical rules.
Validate the data against provider reports. Select a known time window and compare your collected totals with the provider’s usage dashboard or export. Align time zones, project or account scope, model filters, and reporting delays. Investigate differences by checking retries, calls from other applications, missing streaming events, and whether local counts are estimates. Expect small discrepancies when sources use different definitions or aggregation windows.
Build useful views. Show usage over time and group it by agent, provider, model, environment, and request type. Include input and output separately, plus any separately reported cached or modality usage. Add both request counts and token totals so a rise in volume can be distinguished from longer prompts or responses. Provide filters for date range and deployment, and show whether a value is provider-reported or estimated.
Set alerts and ownership. Establish thresholds appropriate to normal traffic, such as an unusual increase over a recent baseline or a team-defined usage limit. Alert the person responsible for the agent or service, and include the affected provider, model, time range, and trend. Treat an alert as a prompt to investigate, not proof of misuse: legitimate traffic changes, retries, or a new feature can raise usage.
Review and maintain the system. Check dashboards and data quality regularly. Revalidate mappings and counting behavior when changing models, providers, SDKs, or agent frameworks. Document retention, access, and aggregation choices, and avoid putting prompts, credentials, or other sensitive content into usage records when counts and identifiers are sufficient.

Common Mistakes

Counting only user requests: agents may make several model calls for one task, including planning, tool selection, retries, and summaries.
Mixing estimated and reported counts: label the source of each measurement, and do not present local tokenizer estimates as billing totals.
Adding unlike categories together: cached input, generated output, and multimodal usage may have different definitions. Preserve separate fields before calculating a combined total.
Ignoring retries and failures: unsuccessful calls can still consume tokens. Record attempts and outcomes rather than assuming failed requests cost nothing.
Assuming a model name tells the whole story: aliases, versions, and provider-specific deployments can differ in tokenization and usage reporting.
Logging too much: full prompts are rarely required for token monitoring and can expose confidential information. Store minimal metadata and apply appropriate access controls.

Tips

Use a trace or parent-task ID to connect an agent’s many model calls without merging them into one measurement.
Keep an immutable request-level record and build charts from it. Aggregated dashboards are easier to correct when the underlying events remain available.
Record the time zone and reporting window used in every comparison; provider dashboards may update later than application logs.
Break down usage by task type or feature when possible. Averages across all agents can conceal one workflow that is responsible for a sharp increase.
Review token totals alongside request volume and latency. Together, these measures help explain whether a change comes from more calls, longer exchanges, or a different model route.

Troubleshooting

Your totals are lower than the provider dashboard: Check for uninstrumented services, retries, batch jobs, or another application using the same account. Confirm that your time zone, account, project, model filters, and reporting window match. Check whether streamed responses emit usage only in a final event.

Your totals are higher than expected: Look for duplicate event ingestion, counting the same response at both middleware and agent levels, or aggregating child calls twice. Verify that input and output fields are not accidentally added more than once.

Some calls have no token counts: Inspect error and streaming paths, SDK response handling, and provider metadata. Mark missing values as unknown rather than zero. If a local estimate is necessary, identify it clearly and validate it against representative provider-reported responses.

Usage is attributed to the wrong agent: Propagate the agent and parent-task identifiers through asynchronous jobs, tool calls, and provider wrappers. Use a fallback label such as “unassigned” rather than silently dropping records, then repair the propagation point.

Alerts fire constantly or miss increases: Confirm the aggregation interval and baseline, then tune thresholds using normal traffic patterns. Ensure delayed provider reports are not being compared with complete application data for the same period.

Frequently Asked Questions

Should I use provider-reported tokens or count them myself?
Use provider-reported usage for provider-side reporting when it is available. Local tokenizers are useful for estimates and debugging, but may differ by model, provider, or feature.

How often should I collect usage?
Capture usage at the end of each model request when possible. Aggregate it into dashboards at an interval that suits your operational needs, while accounting for provider reporting delays.

Can I compare token usage across different providers?
Yes, as a measure of reported token volume if you preserve each provider’s definitions. Do not assume the counts imply equal cost or identical tokenization; compare costs only with the relevant provider pricing and usage categories.

How do I track agents that make several model calls?
Record each call as its own event, then connect events with a shared trace or parent-task ID. This shows both the total per task and the contribution of each step.

Do failed requests count toward usage?
They may. Record provider-reported usage for failed or interrupted calls when available, and check the provider’s reporting definitions rather than treating every failure as zero usage.

What should I do if a provider changes its usage format?
Preserve the raw usage fields, update your parser and documented mapping, and validate the new totals against the provider’s reports before relying on comparisons.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.