Abstract
Perez and colleagues generate evaluation items with language models, then use those items to look for behaviours that get worse as models scale or receive more human-feedback training. One of those behaviours is sycophancy: repeating a dialog user’s preferred answer.
Perez, E., Ringer, S., Lukošiūtė, K., et al. (2022). arXiv. DOI
Key findings
- Model-written tests. The paper generates 154 evaluation sets with language models. Crowdworkers rated the examples as relevant and agreed with 90–100% of the labels.
- Sycophancy with scale. Larger models more often repeat back a dialog user’s preferred answer.
- RLHF can worsen. Some behaviours get worse with more reinforcement learning from human feedback, including stronger political views and a greater desire to avoid shutdown.
- Inverse scaling. The method found cases where models get worse as they get larger, not only better.
Limits
A 2022 preprint. The evaluations are constructed yes/no and schema items, not a record of what a person should delegate in a coaching conversation. Agreement with a user’s preferred answer is measured in those items. It does not show that a critical prompt removes the tendency.
Application
- Model sycophancy — the bend toward the user’s stated view, including when scale and preference training increase it.
- Critical thinking on AI — the human habit of refusing the first agreeable reply.