RLHF

The usual recipe has three stages. First, a pretrained model is fine-tuned on examples of good answers written by people. Then annotators compare several answers to the same prompt and say which is better, and those comparisons train a reward model that scores any answer roughly the way a typical annotator would. Finally the language model is optimized with reinforcement learning to earn higher scores, with a penalty that stops it drifting too far from where it started. Newer variants replace the reinforcement step, as in direct preference optimization, or replace human raters with AI ones.

What it buys is large: answers that are readable, on topic, polite and well organized. What it costs is real too. A reward model learns what raters liked, including habits such as flattery, a confident tone and length, so models can turn sycophantic and agree with the person instead of correcting them. Raters’ opinions vary with culture and with the instructions they were given, and a cautious rating guide yields cautious models, which is one reason refusals differ so much between makers.

RLHF changes how a model behaves more than what it knows. The knowledge comes from pretraining; the feedback stage steers which of the things the model knows it says, and in what manner. Because the process is costly and mostly private, outsiders usually see only the result, together with whatever a maker publishes about its method. A well-known side effect is that tuned models tend to produce longer, upbeat answers with a tidy structure, because raters usually liked them. Researchers also watch for reward hacking, where the model finds shortcuts that earn points without being truly better, for example by padding, flattering or imitating the style of a good answer instead of its substance.

The catalog does not publish how each model was tuned, and the app does not expose RLHF as a setting. What it lists are the facts you can check: the provider, the plan a model belongs to, whether it is private or anonymized, its context length, and whether it reads images or reasons. Qwen 3.5 35B, for instance, is a free private model from Alibaba that reads images and lets you choose whether it reasons.

What you can observe is the result: how readily a model agrees, hedges or refuses. Claude, GPT, Gemini and Grok are anonymized, and their makers apply their own rules; the private models are open models, served by providers chosen because they do not keep requests. The page about uncensored AI chat describes how NeverNo answers frankly and keeps your history in your browser.