Llama.cpp Row Split: What Changed and What to Measure

Updated 5 November 2026 · Standard formula, shown below

Llama.cpp’s row-split mode distributes a model’s weight rows across multiple GPUs, changing how work and memory are divided compared with layer-based splitting. To judge whether it helps your setup, measure end-to-end generation speed, prompt-processing speed, per-GPU memory use, and stability—not just whether the model loads.

What row split means

Llama.cpp supports several ways to place model computation across GPUs. In the familiar layer-split approach, different GPUs handle different layers: a token’s activations move from one device to the next as inference proceeds. Row split instead partitions eligible matrix operations by rows, assigning portions of a layer’s work to multiple devices. Their partial results must then be combined.

The practical goal is to use the GPUs more evenly or make use of combined device memory and compute. But row split is not automatically faster. Dividing work can increase communication and synchronization, and the benefit depends on the model, GPU interconnect, backend, batch size, and workload. “Row split” is a placement strategy, not a guarantee of a particular speedup or a way to make every model fit.

What changed in practice

When a llama.cpp build or configuration adds or changes row-split behavior, the important change is how eligible tensors are divided and how that division interacts with the selected backend. Exact support and command-line options can vary across revisions and GPU backends; check the help output and documentation for the specific build rather than assuming an option from another version still applies.

Compared with layer splitting, row splitting may spread matrix work within layers rather than assigning whole layers to devices. That can change GPU utilization and memory balance, but it can also introduce additional cross-device data movement. The result may be better when one GPU is otherwise underused, and worse when communication costs outweigh the parallel work.

Also distinguish model-weight placement from other memory demands. The context cache (often called the KV cache), temporary compute buffers, and backend workspaces consume memory too. A model that fits after distributing its weights may still fail when you increase context length, batch size, or parallel requests.

How to compare modes fairly

Make a controlled comparison between row split and the available alternative, commonly layer split. Keep the same model file, quantization, prompt, generation settings, context length, batch settings, GPU selection, and software build. Record the exact command and environment. If you change multiple settings at once, you will not know which change caused the result.

Prompt processing: Record prompt-evaluation throughput, usually reported in tokens per second. This measures how quickly the model processes the supplied context.
Text generation: Record generation throughput in tokens per second after the prompt is processed. This is often the number that matters most for interactive use.
Time to first token: Measure elapsed time from starting a request to receiving its first generated token. Include the prompt length, since prompt processing affects this delay.
Memory per GPU: Record peak allocated or used memory for every device during the run, not just the total. Uneven allocation or a nearly full device can explain failures.
Stability: Repeat runs and note out-of-memory errors, backend errors, crashes, or unexpected changes in output. A fast run that fails under the intended context size is not a usable improvement.

Run a short warm-up if the backend has initialization or compilation overhead, then take several timed runs. Compare medians as well as the spread; one unusually fast or slow run can mislead. Use a representative prompt and output length, and avoid measuring only a tiny prompt if real use involves long context.

What to measure next

After establishing a baseline, vary one workload dimension at a time. For example, test a short prompt and a long prompt at the same generation settings, then test the context length you actually need. If your use includes concurrent requests, measure that separately: results from one request do not predict multi-request throughput.

Track device utilization and memory over the whole run when monitoring tools are available. High utilization on one GPU and low utilization on another may indicate an imbalance, but utilization alone is not a performance result. Pair it with tokens per second and latency. If row split is slower, check whether data transfers or synchronization are taking time and whether the GPUs are connected in a way that makes those transfers costly.

A useful test record includes the llama.cpp revision or release, backend, GPU models, driver and runtime versions, model and quantization, split configuration, prompt and output token counts, context and batch settings, peak memory by device, and repeated timing results. This makes a result reproducible and helps separate a code change from a hardware or configuration change.

Common problems and misconceptions

Comparing different builds: A performance change may come from backend or kernel changes rather than row split. Keep builds fixed for a placement comparison.
Using only total memory: Combined free memory does not mean every allocation can be placed anywhere. A single device can run out of space even when the system has memory left elsewhere.
Confusing prompt speed with generation speed: These phases have different workloads. Report both when relevant, and state which one drives your use case.
Expecting more GPUs to always mean faster output: Extra devices can add transfer and coordination overhead. More capacity and more speed are separate outcomes.
Changing context or batch size mid-test: These affect memory and throughput. Treat them as separate experiments, not as equivalent runs.

If the model fails to load or a GPU is not used, first verify that the build includes the intended backend and that the option is valid for that revision. Then reduce the workload to a small, known-good test and inspect the startup log for device placement and allocation details. Avoid drawing a performance conclusion from a run that silently fell back to CPU execution or did not use the intended split.

Frequently Asked Questions

Is row split faster than layer split?
Not in every setup. It can improve balance or throughput for some hardware and workloads, but communication overhead can make it slower. Benchmark both using identical settings.

Does row split reduce model memory use?
It distributes eligible model data across devices; it does not make the weights smaller. Total memory demand still depends on the model, quantization, context cache, and temporary buffers.

Which speed should I compare: prompt or generation tokens per second?
Compare both if you care about both phases. For interactive chat, time to first token and generation speed are useful; for processing large prompts, prompt-evaluation speed may matter more.

Why does adding a GPU make inference slower?
The additional device may require more data transfers and synchronization than the workload can offset. Hardware connectivity, model shape, backend support, and workload size all affect the balance.

What should I report so someone can reproduce my result?
Include the software revision and backend, hardware, model and quantization, split settings, prompt and output sizes, context and batch settings, per-device peak memory, and repeated timing results.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.