NVIDIA's Nemotron TwoTower is a MoE experiment

NVIDIA has quietly published Nemotron TwoTower 30B-A3B Base, an experimental language model that departs from the standard transformer recipe. Released on Hugging Face in BF16, it is a base model — meaning it ships without instruction tuning or alignment, intended as a foundation for further research and fine-tuning.

The architecture is where things get interesting. The model is described as a two-tower design that draws on diffusion and Mamba-style approaches, wrapped in a mixture-of-experts (MoE) layout. With 30 billion total parameters but only about 3 billion active per token, it aims for the efficiency profile that has made sparse MoE models attractive: large capacity at a fraction of the inference cost.

Why it matters

Most frontier open-weight models still lean on dense or MoE transformers. NVIDIA's willingness to publish a hybrid that mixes diffusion and state-space (Mamba) components signals continued exploration of alternatives to attention-heavy designs, particularly for long-context and throughput-sensitive workloads.

• 30B total parameters, roughly 3B active per token
• Mixture-of-experts with a two-tower structure
• Base checkpoint in BF16, no instruction tuning
• Released under a custom NVIDIA license

As an experimental, notable-tier release, this is aimed squarely at researchers rather than production users. The unusual architecture makes it a useful testbed for anyone probing how diffusion and Mamba ideas hold up at scale — but expect rough edges, and check the license terms before building on it.