LLM Speed: Measure Tokens per Second, Latency, and Throughput

Updated 5 November 2026 · Standard formula, shown below

Measuring an LLM’s speed shows how long a request takes, how quickly it generates tokens, and how many requests it can serve over time. Enter a repeatable prompt and record timing and token counts from your client or server; the results help compare configurations and find bottlenecks. Use the steps below to measure single-request latency and multi-request throughput without confusing the two.

What You’ll Need

Access to the model endpoint or application, plus its usage or timing logs if available.
A way to send repeatable requests, such as a small script or an existing load-testing tool.
A fixed prompt, generation settings, and target output length for fair comparisons.
A clock with millisecond resolution, or client/server timestamps precise enough to measure short requests.
A safe test environment and permission to generate the planned request volume.

Step-by-Step

Decide which speed measures matter. Record time to first token (TTFT), time to completion, output tokens per second, and throughput. TTFT is the interval from sending a request until the first generated token arrives. Completion latency is the time from sending until the full response arrives. Output generation speed describes tokens produced per second, while throughput describes tokens or completed requests served per second across a workload. State whether you are measuring one request or concurrent traffic.
Keep the test conditions fixed. Use the same model and version, prompt text, input length, maximum output length, sampling settings, and hardware or service configuration. Record these details alongside results. Disable unrelated work where possible, and avoid changing settings between runs. If comparing models, use equivalent prompts and output limits, and label each model clearly.
Warm up the system. Send a few requests that you will not include in the results. This can reveal startup, model-loading, connection setup, or cache effects. Do not assume every system behaves alike: note whether caching is enabled and whether the measured run uses a warm or cold state. For a cold-start measurement, report it separately rather than mixing it with warmed requests.
Capture timestamps and token counts. For each request, record the time the client sends it, the arrival time of the first streamed token, and the time the response finishes. Also record input and output token counts when available. Prefer the model’s tokenizer or server-reported counts; character counts and whitespace-separated words are not reliable substitutes for tokens. Use a monotonic clock for elapsed time so wall-clock corrections do not distort measurements.
Measure a single request first. Send one request at a time and repeat it several times, ideally at least 10 for a rough baseline. Calculate TTFT as first-token time minus send time, and completion latency as finish time minus send time. For streamed output, estimate generation speed as output tokens divided by the interval from first token to completion. This is a practical average, not a per-token speed trace; account for the final token and timing conventions used by your client.
Run a controlled concurrency test. Increase simultaneous requests in steps, such as 1, 2, 4, then 8, while keeping the prompt and output settings fixed. Hold each level long enough to collect a stable sample, and record errors, queueing, and latency as well as throughput. Stop if you hit service limits, resource constraints, or unacceptable latency. Do not send uncontrolled load to a shared or production service.
Calculate and label the results. For one response, output tokens per second = output token count ÷ generation time in seconds. For a test window, aggregate output-token throughput = total output tokens completed ÷ elapsed wall-clock seconds. Request throughput = completed requests ÷ elapsed seconds. Say whether the numerator includes input tokens; “tokens per second” is ambiguous unless defined. Do not divide total tokens by the sum of individual request durations when reporting system throughput, because concurrent requests overlap.
Summarize variability and compare fairly. Report the number of runs and use median latency and median generation speed for a typical result. Include a high percentile, such as p95, when tail delays matter, and give the full test conditions. Compare runs only when their workload, concurrency, streaming behavior, and token-count method match. Repeat the test if results vary widely, and investigate queueing, network delays, or changing server load.

Common Mistakes

Calling every measure “tokens per second.” Per-request generation speed and aggregate throughput answer different questions. Define which one you report.
Including prompt processing in generation speed. Input tokens are processed before or during generation, but output tokens per second normally uses output tokens and the generation interval. Report TTFT separately to show prompt-processing and queueing delay.
Timing only the complete response. Completion latency includes TTFT and the time to generate the rest. It cannot reveal whether a slow first token or slow generation caused the delay.
Using words or characters as token counts. Tokenization varies by text and model. Use the matching tokenizer or trustworthy usage metadata.
Mixing warm and cold runs. Startup or cache behavior can make early results unlike steady-state performance. Keep these categories separate.
Overlooking concurrency and failures. A fast isolated response does not prove high-capacity service. Count completed work, timeouts, and errors during load tests.

Tips

Keep a simple results table with test date, model/configuration, concurrency, input tokens, output tokens, TTFT, completion time, and errors.
Use streaming if you need TTFT and generation-rate measurements. With non-streaming responses, you can measure completion latency but not when the first token became available to the client.
Measure from the same point each time. Client-side timings include network and client overhead; server-side timings may exclude them. Label the measurement boundary.
Use the same connection and request format for comparisons. Separate network or queueing effects from model generation when server telemetry allows it.
For a production-like test, use a representative mix of prompt and response lengths rather than only short, identical requests.

Troubleshooting

Numbers change sharply between runs: increase repetitions, check for shared load, and separate warm-up from measured runs. Compare medians and percentiles instead of relying on one request.

Tokens per second looks unexpectedly low: verify the output-token count, generation-time boundaries, streaming behavior, and whether your timer includes queueing or network waits. Confirm that seconds—not milliseconds—are used in the calculation.

TTFT is high but generation speed is normal: investigate request queueing, long prompts, connection setup, or prompt-processing time. If available, compare client and server timestamps to isolate network delay.

Throughput falls as concurrency rises: check for rate limits, queue growth, resource saturation, retries, and timeouts. Throughput may stop increasing even while latency worsens; report the saturation point rather than treating higher concurrency as automatically better.

Token counts do not match: check whether the client and server use the same tokenizer and whether usage data includes hidden formatting or special tokens. State which count you used and avoid comparing unlike counting methods.

Frequently Asked Questions

What is a good LLM tokens-per-second speed?
There is no universal target. The useful rate depends on the model, hardware or service, prompt and response lengths, and whether the result is per request or aggregate.

How do I calculate output tokens per second?
Divide the number of generated output tokens by the generation interval in seconds. Define whether that interval begins at the first token or includes TTFT; the former is commonly used for generation speed.

What is the difference between latency and throughput?
Latency is how long an individual request takes. Throughput is how much work finishes per unit of wall-clock time, often requests or tokens per second across concurrent requests.

Should TTFT be included in tokens-per-second calculations?
Usually report TTFT separately and calculate generation speed from the first token to completion. If you use total request time instead, label it as end-to-end output rate so it is not mistaken for generation speed.

How many runs should I test?
For a basic comparison, collect at least 10 measured single-request runs after warm-up. Use more runs and a sustained test window for variable services or concurrency tests.

Can I measure tokens per second without streaming?
You can measure total completion latency and calculate an end-to-end rate, but you cannot observe TTFT or isolate the generation interval unless the system provides suitable timing data.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.