How a Georgia Tech team used the open Olmo stack to trace social reasoning

Glenn Matlin, a computer-science PhD candidate at Georgia Tech, and his co-author Chandreyi (Zini) Chakraborty have spent years tracing the roots of a language model’s social reasoning—its ability to interpret people’s beliefs, emotions, intentions, and everyday moral choices. "We set out with one main question: where does [it] come from?” says Chakraborty.

Matlin and Chakraborty designed a study around influence functions, a technique that estimates how much any single training document – an article, post, or page a model trained on – shaped how the model answered a particular benchmark question. Add those estimates up across the data, and you can see which kinds of writing – literary, technical, and so on – support a given ability in the model. But that holds only if you can confirm the model actually trained on the documents in question.

It’s the kind of question they could only answer with a fully open model, which is why they chose to work with Olmo 3. Olmo 3 is one of the few LLMs released in full—its training corpora, checkpoints, and evaluation tools are all public alongside the more commonly released model weights. 

“The vision for my research is to create a map that connects model behavior back to the kinds of training data or dynamics that may have helped produce it,” Matlin says.

Matlin and Chakraborty’s study uses five open pieces of the Olmo ecosystem: Olmo 3, Dolma 3 (the data Olmo 3 was trained on), WebOrganizer (a labeling system for sorting datasets like Dolma into different categories by topic and format), OlmoEval (which supplies benchmarks to measure model abilities), and OLMES (a shared benchmark scoring standard).