Mixture of experts
A mixture-of-experts model is a neural network split into many specialized sub-networks called experts, with a router that sends each token to only a few of them. Because only part of the model works on each token, it can be very large in total while costing less to run than a dense model of the same size.
Last updated November 8, 2026
Readable on your devices only.
Your conversation is readable on your device, and on your other devices if you sync them. Nowhere else.
Unreadable to us.
Your message passes through without being written down; your chats are only ever stored encrypted.
The model doesn’t know you.
It gets the question, never your email, your account or your IP address.
Nothing is left along the way.
The reply comes back and is stored here, encrypted; even when you sync, your key never reaches us in the clear.
How it works
In a dense model, every weight takes part in every token. In a mixture-of-experts model, the feed-forward blocks are replaced by a set of experts, say dozens or hundreds, and a small learned router picks a few of them for each token at each layer. The experts do not turn into tidy specialists such as a lawyer or a chemist; they tend to specialize in patterns of tokens, and the router learns the split during training. Published sizes are therefore given twice: total parameters, and active parameters used per token.
The benefit is quality close to that of a much larger dense model, at a fraction of the computing per token, so generation can be fast and cheap. The costs are real as well. All experts must still sit in memory, so the memory needed follows the total count, not the active one. Routing must stay balanced so that some experts are not overloaded while others idle, and training and serving are more complex. Fine-tuning can be trickier too, and quantization often goes hand in hand with these models to make them fit in memory.
Do not read a figure like 35B or 400B as how much computing a reply uses. For a mixture-of-experts model, ask for both numbers: active parameters hint at speed and cost, while total parameters hint at knowledge capacity and memory. Many recent open-weight models and some closed ones are built this way, though makers do not always say so. Engineers also talk about load-balancing losses, capacity limits and shared experts that every token passes through, all of which exist to keep the routing stable. In an open model these design choices are written in the technical report and the configuration file, which list the number of experts, how many are chosen per token and the size of each.
In NeverNo
NeverNo’s catalog does not say which models are mixtures of experts, so check the maker’s own documentation for a given model. What the catalog states is practical: the provider, the plan, whether the model is private or anonymized, its context length, and whether it reads images or reasons.
The private models come from open-weight families such as DeepSeek, GLM, Qwen and Mistral. Qwen 3.5 35B is a free one from Alibaba, with 128,000 tokens of context, that reads images and lets you choose whether it reasons. Private means it is served by providers chosen because they do not keep requests, and the transparency page says plainly that NeverNo cannot inspect their servers.
Worth remembering
A router activates only a few experts per token, so computing per token is lower than the total size suggests.
Memory still follows the total parameters, since all experts must be loaded.
Experts specialize in patterns of tokens, not in neat school subjects.
Read both numbers, total and active, when you compare models.
A model’s configuration or technical report states the number of experts and how many are picked per token.
Frequently asked questions
Is a mixture-of-experts model several models?
No. It is one network trained as a whole: the experts are parts inside its layers, and the router is trained together with them. You send one request to one model and get one answer. Think of a hospital with many wards: a patient visits only the ones needed, yet the whole building must be staffed and kept open.
Why do some models list total and active parameters?
The total shows the size in memory and the capacity of the model, while the active count shows how many weights work on each token, which sets speed and cost. Both matter for a fair comparison. Counting both figures keeps leaderboards honest, because a model advertised by its total size may run as lightly as a far smaller dense one.
Is it better than a dense model?
It depends. It often gives better quality per unit of computing, but it needs more memory and is harder to run. Dense models are simpler and can be the better choice at small sizes. Dense models are easier to fine-tune, quantize and debug, which is why many small and medium releases stay dense while the largest ones lean on experts to keep each reply affordable. Advantages include lower latency and better throughput per watt; drawbacks include cluster complexity and the discipline that routing demands.
Ask your first question
No account and no card. Write in any of the 10 languages NeverNo speaks.
Open the chat