How Goodfire used Ai2’s open post-training stack to trace unwanted model behavior

Preference training is a key step in developing LLMs. It uses examples of better and worse responses to shape a model’s behavior—for instance, how helpful, safe, concise, or compliant the model is. 

But steering a model toward behaviors developers want can also have unintended effects. Training that improves a model overall might weaken a safeguard in a particular context or strengthen a behavior no one thought to test for. As the entire AI ecosystem wrestles with how to develop models that remain in alignment with human values, understanding these relationships and getting ahead of unintended consequences is critical.

When unintended behaviors do appear, model builders face a difficult debugging problem: What changed, which training examples caused it, and can the unwanted behavior be corrected without undoing improvements elsewhere?

Goodfire, an interpretability research company, used Ai2’s fully open post-training stack to show what becomes possible when researchers can answer those questions across the entire training pipeline. They predicted behavioral changes before a full training run, traced an observed safety regression back to individual preference examples, and tested targeted changes designed to reduce the regression without sacrificing the model’s broader capability gains.

That level of debugging is difficult partly because of how preference training works. A preference dataset contains prompts alongside responses labeled as preferred or rejected. Across hundreds of thousands of examples, those individual choices collectively become a signal telling the model which behaviors to strengthen and which to suppress. 

“A preference dataset is effectively ‘programming’ the model,” wrote the Goodfire team, “but the instructions implied by a preference dataset cannot be naively inspected, understood, and debugged.”

Why preference training is hard to debug