Non-English Prompts & LLM Costs: Token Counts Explained

Token counts determine the cost of using large language models (LLMs) like GPT, Claude, and Gemini, and they are not the same as word counts—especially for non-English prompts. For English, one token averages about 0.75 words, but for languages like Japanese, Korean, or Thai, a single character can be one or more tokens, making prompts up to 2–3 times more expensive per meaning. This article explains what tokens are, how they differ across languages, and how to estimate and reduce costs.

Formula

Estimated tokens = characters × language factor

Where:

• t = number of characters in the text (including spaces and punctuation)
• l = language factor: 1.0 for English, 1.3 for Spanish/French, 2.0 for Arabic, 2.5 for Japanese/Korean/Thai, 1.5 for Chinese (approx.)

Estimated cost = (tokens / 1000) × price per 1K tokens

The default price of $0.003 per 1K tokens is a typical rate for input tokens on many current LLM APIs; check your provider’s current pricing.

Worked Example

Using the defaults: text length = 100 characters, language factor = 1.3.

1. Tokens = 100 × 1.3 = 130 tokens
2. Cost = (130 / 1000) × 0.003 = 0.00039 USD

So a 100-character Spanish prompt would cost about $0.00039 for input, assuming the same rate.

What Are Tokens and Why Do They Vary by Language?

LLMs don’t read text as letters or words; they break text into tokens—small chunks of characters. For English, a token is often a short word or part of a word (e.g., “token� = 1 token, “tokenization� = 2 tokens). For languages with more characters per word or complex scripts, the tokenizer may split each character or even each syllable into separate tokens. For example:

• English: “Hello worldâ€� → 2 tokens
• Japanese: “ã�“ã‚“ã�«ã�¡ã�¯â€� (5 characters) → often 5 tokens
• Thai: “สวัสดีâ€� (6 characters) → often 6 tokens

This happens because tokenizers are trained primarily on English text, so they are efficient for English but inefficient for other scripts. The result: a non-English prompt may cost 2–3 times more than the same meaning in English.

How Token Counts Affect LLM Costs

API pricing is per token, not per word or character. Both input (your prompt) and output (the model’s response) are charged. A typical pricing model is:

• Input: $0.003 per 1K tokens (e.g., GPT-3.5)
• Output: $0.006 per 1K tokens

So a 1,000-token prompt costs $0.003 for input. If the same prompt in Japanese uses 2,500 tokens, it costs $0.0075—2.5× more. Over thousands of requests, this adds up.

How to Estimate Tokens for Your Language

Most providers offer a tokenizer tool or API endpoint to count tokens exactly. If you can’t use one, use these rough factors:

• English: 1 token ≈ 4 characters (or 0.75 words)
• Spanish, French, Italian: 1 token ≈ 3.5–4 characters
• Chinese (simplified): 1 character ≈ 1–2 tokens
• Japanese, Korean, Thai: 1 character ≈ 1–2 tokens (often 1 character = 1 token)
• Arabic, Hebrew: 1 token ≈ 2–3 characters

For a quick estimate, multiply the number of characters by the factor above. For example, a 500-character Japanese prompt ≈ 500–1000 tokens, whereas English would be ~125 tokens.

Common Mistakes and Misconceptions

One common mistake is assuming that “token� equals “word.� This leads to underestimating costs for non-English input. Another misconception is that all languages cost the same—they don’t. Also, note that the model’s context window (e.g., 8K tokens) counts both input and output; a long non-English prompt can fill the window quickly, forcing you to truncate or use a more expensive model with a larger context.

Another frequent error is forgetting to count the system prompt and any few-shot examples. Those are also tokenized and charged. Always include them in your estimate.

How to Reduce Costs for Non-English Prompts

If you work with non-English text, consider these strategies:

• Translate your prompt to English before sending it, then translate the output back. This may be cheaper if the translation is done by a cheaper model or a local library.
• Use a model with a tokenizer optimized for your language (e.g., some models are trained on multilingual data and have better token efficiency).
• Shorten your prompt: remove unnecessary words, use abbreviations, or compress instructions.
• Use a smaller model for simple tasks—they often have lower per-token prices.

However, translation adds latency and may lose nuance. Test both approaches to see which is more cost-effective for your use case.

Comparing Token Efficiency Across Languages

To give you a concrete sense, here’s an approximate comparison for the same sentence “The quick brown fox jumps over the lazy dog� in different languages:

• English: 9 words → ~10 tokens
• Spanish: 9 words → ~11 tokens
• Japanese: 11 characters → ~11 tokens
• Thai: 40 characters → ~40 tokens

Notice that Thai uses 4× more tokens than English for the same meaning. This is because Thai has no spaces between words and the tokenizer splits each character.

Frequently Asked Questions

Why do non-English prompts cost more?

Because tokenizers are less efficient for non-English scripts, so more tokens are needed to represent the same text. This increases the token count and therefore the cost.

Can I use a tokenizer to count tokens before sending?

Yes, most LLM providers offer a tokenizer tool or API endpoint. You can paste your text and get an exact token count, which helps you estimate cost accurately.

Does the output language also affect cost?

Yes, output tokens are charged at a different rate, and if the model generates text in a non-English language, it will use more tokens than English for the same content, increasing cost.

Is it cheaper to translate my prompt to English?

Often yes, because English tokenization is more efficient. But translation adds latency and may lose nuance. Compare the total cost of translation + English prompt vs. direct non-English prompt.

Are there models with better token efficiency for non-English?

Some multilingual models are trained on diverse languages and have improved tokenizers. Research the specific model’s tokenizer behavior for your language before committing.

How do I know the exact token count for my prompt?

Use the provider’s tokenizer tool or API. If unavailable, use the character-based estimate as a rough guide.

Related

• 1440p 210Hz vs 1080p 240Hz: Which Monitor to Pick
• OLED Gaming Laptop Burn-In, Brightness & Everyday Use
• Balanced Healthy Eating Plan: Step-by-Step Guide

in t | Text length | characters | 100 | 1
in l | Language factor | - | 1.3 | 0
out tokens | Estimated tokens | tokens | t * l | 0
out cost | Estimated cost (USD) | $ | (t * l / 1000) * 0.003 | 4