Modular LLMs at scale: how FlexOlmo is helping to pool national expertise without pooling sensitive data

When we released FlexOlmo last year, we wanted to show that a language model doesn't have to be a monolith. Different teams could train their own pieces – specialized modules called experts – in isolation and merge them into a shared model, without ever pooling the data underneath.

A project out of Denmark, Danish Foundation Models (DFM), used FlexOlmo as the cornerstone for an architecture of their own. DFM develops open language models for the Danish language on the premise that models for lower-resource languages will fall behind unless independent efforts step in from outside well-resourced commercial labs. The institutions that would benefit most from Danish-language models – hospitals, universities, public-sector organizations, and smaller companies – often hold data that they can't share, whether for regulatory or proprietary reasons. Yet that data is exactly what's needed to train the models that would serve them.

For DFM, FlexOlmo was the right starting point.

"We envisioned a modular system whereby national initiatives like ours can independently train on their respective corpora, and then bring those independently trained models together," says Jacob Nielsen, an Industrial PhD Fellow at Ordbogen A/S and a researcher at the University of Southern Denmark's (SDU) OdenseNLP lab, a research group at SDU led by Peter Schneider-Kamp and Lukas Galke Poech and part of DFM. “More broadly, we aim to contribute to modular multilingual models that are beneficial on an international scale.”