Not all AI users pay the same price for the same task. The difference has nothing to do with subscription tiers or geographic pricing policies. It comes from something far more fundamental: the language a person types in.
A study published by the Coalition for Language Equity in AI Research found that AI models require significantly more tokens to process certain languages than others. Tokens are the basic units of text that a model reads and generates. More tokens means higher costs, slower responses, and less room in the model’s memory for context. The researchers named this phenomenon the “tokenizer tax,” and its implications reach well beyond billing.
Why the Same Sentence Costs More in Some Languages
A tokenizer is the component that breaks raw text into the chunks a model can process. The problem is that most tokenizers were built with a heavy bias toward English and a handful of other high-resource languages. Languages with less digital presence, or written in scripts the tokenizer rarely encountered during training, get fragmented into far more pieces to represent the same meaning.
The study found that even among the 20 most widely used languages on the internet, the most expensive required between 1.5 and 5 times more tokens than the cheapest. When the analysis expanded to 200 languages, that gap grew to 17 times in one AI model. This is not a marginal inefficiency. It is a structural disparity baked into the architecture of tools that are increasingly presented as universal.
Script matters as much as speaker count. Asturian, a language with relatively few speakers, benefits from being written in the Latin alphabet and from its proximity to other Romance languages like Italian. Languages written in scripts that tokenizers rarely encountered during training, including Burmese, Tibetan, Tifinagh, Khmer, Lao, and Ol Chiki, consistently ranked among the most disadvantaged across nearly all models in the analysis. A language can have millions of speakers and still face a severe tokenizer penalty if its writing system is underrepresented in training data.
The Cost Is Not Just Financial
The tokenizer tax compounds in ways that go beyond a higher API bill. Higher token counts produce higher latency: responses take longer to generate. They also reduce what researchers call effective context capacity, the amount of information a model can hold in its working memory during a conversation.
This has practical consequences. Pratyush Choudhury, co-founder of Activate AI, described the problem through the example of a tutoring application. A good tutor retains a student’s earlier mistakes and explanations across a session. If the conversation history consumes the context window much faster in one language than another, the quality of the service degrades, not because the model is less capable in principle, but because the tokenizer leaves less room to work with.
The study made the structural stakes explicit: a startup operating in Bengaluru may pay a surcharge on every prompt and completion that its competitor in Austin does not. The tokenizer tax, as the researchers framed it, is a constraint on who can participate in the AI economy.
Raoul Nanavati, co-founder and CEO of Mumbai-based AI voice infrastructure startup Navana.ai, pointed to open data initiatives as one path forward. He cited RESPIN, a collaborative project with the Indian Institute of Science funded by the Gates Foundation, which collected text from native speakers of 38 dialects alongside recorded speech. Better underlying data is a necessary condition for improving how models handle underrepresented languages.
What Fixing This Actually Requires
Better training data helps, but it does not resolve the tokenization gap on its own. Dheemanth Reddy, CEO of Maya Research, a company developing conversational AI models for the next five billion internet users, drew a clear line between the two problems. Improving multilingual and synthetic data can raise model quality. Actually closing the tokenization gap requires AI labs to deliberately design future tokenizers to handle these languages and scripts better. One does not automatically follow from the other.
Reddy noted that token prices are falling, which should reduce the direct cost impact over time. That is a partial relief, not a solution. The latency penalties and context limitations persist regardless of price per token.
Choudhury argued that language efficiency should be treated as a core product requirement, alongside accuracy, speed, and reliability. He called on major AI labs to publish language-by-language scorecards showing what it costs to complete comparable tasks across languages, and to commit to measurable improvements for the languages they serve least well. The framing matters: this is not a request for charity toward minority languages. It is a demand for transparency about a structural inequality that currently shapes who benefits from AI and who subsidizes it.
In Short
AI models process text in tokens, and some languages require far more tokens than others to express the same content. This creates a hidden cost for users of underrepresented languages: higher prices, slower responses, and reduced context quality. The gap reaches up to 17 times between the most and least efficient languages in some models, and it is driven as much by script as by speaker count. Fixing it requires deliberate redesign of tokenizers, not just better data. Until that happens, the architecture of AI carries a built-in advantage for English speakers that most users never see.
Based on reporting from Rest of World.