New in Together GPU Clusters: Reliability and control for production GPU clusters

40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...

We’ve spent the last several weeks shipping a set of changes to Together GPU Clusters aimed at the operational reality of running training and inference at scale: hardware fails, schedulers leak, and teams outgrow the single-admin-kubeconfig workflow they started with. This post walks through what we shipped, why we built it the way we did, and what it means for the workloads you’re running on Together today.

The changes group into two themes. The first is platform health: passive health checks, auto node repair, and Slinky 1.0, focused on catching and recovering from the failure modes that actually take down jobs. The second is operational control: a new cluster details view, external OIDC, startup scripts, and an acceptance-test opt-out, focused on giving your team the visibility, access, and customization hooks needed to run clusters as your organization grows.

Catching and fixing failures as they happen

If you’ve run a multi-day training job on a large cluster, you know the pattern. A GPU falls off the PCIe bus. An Xid error takes a node out of rotation. Thermal throttling silently caps a job’s throughput and you don’t notice until the loss curve flattens.

These are steady-state failure modes at scale. What matters is how quickly you catch them and how cleanly you recover.

We already ran active health checks, synthetic tests that exercise the hardware against a known-good baseline. Active checks are useful at provisioning time and on idle nodes. Passive checks extend that coverage to failures that appear while real workloads are running.