Non-English speakers pay a 'tokenizer tax' with AI - does it affect your learning tools?

Question

Languages that tokenize poorly cost more and sometimes get worse quality from LLMs. If you use AI chat for practice in less common languages, have you noticed it being worse?

Cedar Comet · Accepted Answer

for Finnish it's okay but it invents grammar sometimes. always double check with a native

Non-English speakers pay a 'tokenizer tax' with AI - does it affect your learning tools?

Comments