The Token Tax: Why AI Is More Expensive for Non-English Languages

Quick overview:

If you use ChatGPT, Gemini, or Claude in Hindi, Arabic, or Tamil instead of English, you are almost certainly paying more for the exact same conversation. This is not a pricing decision made by any AI company. It is a side effect of how these models read text in the first place, and it is quietly making AI more expensive, slower, and less accurate for most of the world's population.

Type "How do I fix a leaking tap?" into ChatGPT in English, then ask the same question in Hindi. You will get a similarly useful answer both times. What you will not see, because it happens invisibly behind the scenes, is that the Hindi version can cost two to three times more to process. Multiply that gap across the 544 million Indian users of ChatGPT alone, or across the billions of people worldwide who do not speak English as a first language, and a small technical quirk turns into what researchers are now calling a systematic language tax.

This isn't a conspiracy or a deliberate business decision by any single company. It's a side effect of how large language models were built in the first place, mostly on English-heavy training data, using a text-processing method that happens to favor languages like English by design. But the effect is real, it's measurable, and it's shaping who gets cheap, fast, reliable access to AI and who doesn't.

In this article, we'll break down exactly what causes this "token tax," how big the gap actually is across different languages, what AI companies are doing about it, and what it means for the future of AI access outside the English-speaking world.

What Is the "Token Tax"?

Before getting into the tax itself, it helps to understand what a token actually is.

AI models like GPT, Claude, and Gemini don't read text the way humans do. They can't process raw words directly, so every sentence you type gets broken down into small chunks called tokens before the model can work with it. A token might be a whole word, part of a word, or even a single character, depending on the language and the specific method the model uses to divide up text.

In English, this system works fairly efficiently. On average, one token covers about four characters, or roughly three-quarters of a word, so a thousand tokens works out to around 750 words of English text.

The token tax refers to the fact that this efficient ratio only really holds for English and a handful of other Latin-script languages. Non-English languages, especially those with different scripts like Devanagari, Arabic, or Chinese characters, often require significantly more tokens to represent the exact same meaning. Because every major AI provider charges by the token, and because context windows are also measured in tokens, this creates a real cost and performance gap based purely on what language you happen to type in.

Why Does This Happen?

The root cause comes down to how tokenizers, the systems that break text into tokens, are trained.

Most large language models use a method called byte-pair encoding, or a close variant of it. In simple terms, the tokenizer looks at enormous amounts of training text and learns to group frequently occurring character sequences into single tokens. Common English patterns like "ing," "tion," or "the" become efficient single tokens because they appear constantly in the training data.

Since the majority of internet text, and therefore the majority of training data used to build these tokenizers, is in English, the resulting vocabulary is naturally optimized around English patterns. When the same tokenizer encounters Hindi, Arabic, or Chinese, it hasn't learned enough efficient shortcuts for those languages, so it ends up breaking words into smaller, more fragmented pieces to represent the same idea.

Researchers studying this problem, notably a widely cited paper by Petrov and colleagues, introduced the concept of a tokenization premium: the ratio of tokens needed to encode a sentence in a given language compared to the tokens needed for the equivalent sentence in English. Some findings from this line of research, detailed in a study on reducing tokenization premiums for low-resource languages, are striking. One low-resource language, Shan, was found to need up to nineteen times more tokens than English under an older GPT tokenizer.

How Big Is the Gap, Really?

The numbers vary depending on the language, the script, and which AI model you're using, but the pattern holds consistently across nearly every study on the topic.

• A widely cited independent analysis on LLM tokenization costs found that non-English text typically requires two to three times more tokens than English for the same meaning, and that a multilingual customer support system handling English, Spanish, and Mandarin could end up paying roughly 2.5 times more for non-English queries at identical volume.

• Academic research comparing token efficiency in Sanskrit, Hindi, and English found close to a two-fold difference in token counts between Sanskrit and English or Hindi, and found that translating Sanskrit commentary into English or Hindi could inflate token counts by as much as twenty times.

• A recent multilingual reasoning benchmark found that models like Claude incurred over twice the cost to process Thai compared to English, while simultaneously scoring several percentage points lower on accuracy, meaning some languages are both more expensive and less reliable at the same time.

• Chinese sits in an interesting middle ground. Some studies found older GPT and Llama tokenizers penalized Chinese with around 15% more tokens than equivalent English content, while newer models with expanded vocabularies for Chinese characters have closed much of that gap, and in some cases even reversed it.

It's worth noting that this isn't a fixed, universal ratio. The size of the tax depends heavily on which script a language uses, how much training data existed for it, and which specific tokenizer a given AI model relies on.

The Improvements Nobody Talks About Enough

To be fair to the major AI labs, this problem hasn't been ignored, and there has been real progress.

When OpenAI released GPT-4o in 2024, it shipped with a new tokenizer built specifically to be more efficient across languages. According to OpenAI's own announcement, GPT-4o matched GPT-4 Turbo's performance on English text while showing a significant improvement in non-English languages. Independent analysis of that new tokenizer's performance on Indic languages found even more specific gains: Hindi required roughly 2.9 times fewer tokens than before, Malayalam saw close to a four-times efficiency improvement, and Kannada and Telugu both saw token usage drop by around 77 to 79%.

That's a genuinely meaningful shift for hundreds of millions of Indian language speakers. But it also illustrates the core issue: these improvements happen unevenly and only when a company decides to prioritize a specific language. The same analysis found that Kashmiri saw a much smaller improvement, and Manipuri saw effectively none at all. Efficiency gains tend to flow toward languages with large speaker populations and commercial demand, while lower-resource languages are left waiting.

Why This Matters Beyond Just Pricing

It would be easy to file this under "interesting technical trivia," but the token tax has consequences that go well beyond a slightly higher API bill.

Context windows shrink

Every AI model has a maximum context window, measured in tokens, not words. If a language needs two to three times more tokens to say the same thing, a user working in that language effectively gets a smaller usable context window than an English speaker, even though both are technically working within the same token limit.

Response quality can suffer

Some research has found a real correlation between token cost and accuracy. In other words, the languages that cost more to process in tokens are sometimes also the languages where the model performs worse, meaning users are paying a penalty twice: once in cost, and once in reliability.

It compounds at scale

For an individual asking ChatGPT a single question, the price difference is invisible. But for a business building a multilingual customer support tool, an ed-tech platform serving regional language students, or a government service deploying AI for citizens, that 2 to 3x multiplier turns into a real, ongoing operating cost difference that can make or break a project's economics.

It shapes who gets left out

As AI tools become embedded into search, education, healthcare information, and government services, unequal token costs quietly become an equity issue. The same quality of assistance costs more, moves slower, or gets truncated more often, depending purely on which language someone happens to speak.

What Founders and Users Are Doing About It

The token tax has become enough of a known problem that it's shaping how some companies build AI products in the first place.

Rather than accepting the inefficiency of general-purpose tokenizers, a growing number of startups are building foundational models trained natively on non-English data, with tokenizers designed around the target language's actual structure from day one, instead of retrofitting an English-first system after the fact. This is part of why India-focused AI labs frame their work not as a translation layer on top of ChatGPT, but as building the entire stack, tokenizer included, around Indian languages first.

On the user side, a few practical takeaways have emerged from developers dealing with this in production:

• Removing filler words and redundant phrasing before sending text to an API can meaningfully cut token counts in any language.

• Where possible, comparing tokenizer efficiency across providers for your specific target language before committing to one API can reveal real cost differences.

• Businesses serving multilingual audiences at scale should budget compute costs per language rather than assuming a flat, uniform rate.

The Bigger Picture

The token tax is a reminder that AI systems inherit the biases and blind spots of the data and design choices behind them, often in ways that aren't obvious until someone measures them directly. Nobody sat down and decided that Hindi speakers should pay more than English speakers for the same conversation. It emerged as a side effect of what data was easiest to gather when today's tokenizers were built.

The good news is that this is a solvable, well-understood problem, not some fundamental limitation of AI itself. GPT-4o's tokenizer overhaul proves that meaningful progress is possible when a company decides to prioritize it. The open question is how quickly that same effort reaches the thousands of other languages still waiting for their turn, and whether pricing models will eventually account for the real information content of a message rather than a raw token count that penalizes entire regions of the world by default.

Read Also:

If AI Never Forgets, What Happens to Forgiveness?

Why AI Doesn't Have a Data Problem, It Has a Trust Problem

I Let AI Plan My Entire Week. Here's What Happened

AI Is Diagnosing Patients: What Happens When It Gets It Wrong?

When AI Makes a Mistake That Kills Someone, Who Goes to Prison?