Token

Before a model sees your message, a tokenizer cuts it into pieces taken from a fixed vocabulary of tens of thousands of entries. Common words are usually one token, rarer words are split into several, and a long technical term or a name may take four or five. A word like “unbelievable” might become “un”, “believ” and “able”, while “the” stays whole. Capital letters, accents, tabs and trailing spaces change the split too. Each token is then turned into a list of numbers, which is what the neural network really processes.

The split depends on the language and on the tokenizer of each model. Text in English tends to use fewer tokens per word than text in languages with richer word forms or other writing systems, so the same sentence can cost more in Japanese or Arabic than in English. Source code, long numbers, emoji and URLs also shatter into many small pieces. Tokenizers differ between model families, so a count from one vendor’s tool is only an estimate for another.

Tokens matter for three practical reasons. They measure length: a model has a limit on how many tokens it can handle at once, and on how many it can write in one reply. They measure cost: providers usually charge separately for tokens read (input) and tokens written (output). Billing, latency, truncation and rate limits all follow from the same count, whether the job is a meeting summary, a lease clause, a regex fix or a menu translation. And they explain odd behavior: a model that works on tokens rather than letters can stumble on puzzles such as how many times “r” appears in “strawberry”, or reversing a long string. Tokenization algorithms such as byte-pair encoding start from single bytes and repeatedly merge the most frequent neighboring pairs until a vocabulary of the desired size emerges, which explains why common syllables become single entries while rare strings stay fragmented.

On NeverNo, tokens show up in the limits shown for each model. DeepSeek V4 Flash, a free private model, has a context window of 128,000 tokens and can write up to 8,192 tokens in one reply, which is room for a long document and a long answer in the same conversation. Other models have more or less room, and each model page lists its own figures.

In the API, usage is reported in tokens, and the parameter max_tokens (or max_completion_tokens) caps the length of a reply. The embeddings endpoint, which turns text into vectors for search, also works on tokens. The page about the ChatGPT alternative explains how to move from another assistant and keep your history in your browser.