- Updated: December 28, 2025
- 5 min read
NVIDIA Unveils Nemotron 3: Hybrid Mamba‑Transformer MOE Model for Long‑Context Agentic AI
Nemotron 3 is NVIDIA’s latest open‑source family of hybrid Mamba‑Transformer Mixture‑of‑Experts (MoE) models that provide up to 1 million token context windows, scalable active‑per‑token computation, and three size tiers (Nano, Super, Ultra) designed for long‑context, agentic AI applications.
NVIDIA Unveils Nemotron 3: A Hybrid Mamba‑Transformer MoE Stack Redefining Long‑Context AI
In a bold move that could reshape the landscape of agentic AI, NVIDIA announced the release of the Nemotron 3 family—an open‑source suite of language models that blend Mamba state‑space layers, traditional Transformers, and sparse Mixture‑of‑Experts. The announcement, detailed in the original MarkTechPost article, highlights a paradigm shift toward ultra‑long context windows and cost‑effective inference.
Nemotron 3 Family Overview
The Nemotron 3 lineup is organized into three distinct tiers, each targeting a specific workload profile:
- Nemotron 3 Nano – ~31.6 B total parameters, ~3.2 B active per token.
- Nemotron 3 Super – ~100 B total parameters, up to 10 B active per token.
- Nemotron 3 Ultra – ~500 B total parameters, up to 50 B active per token.
All three models share a common 1‑million token context window, enabling them to process entire books, massive codebases, or multi‑agent memory traces without chunking.
Technical Highlights: Hybrid Mamba‑Transformer MoE Architecture
The core of Nemotron 3 is a novel hybrid architecture that interleaves three building blocks:
- Mamba 2 sequence blocks – State‑space models that excel at capturing very long‑range dependencies with linear‑time complexity.
- Transformer attention blocks – Provide precise token‑to‑token interactions for tasks that require fine‑grained reasoning.
- Mixture‑of‑Experts (MoE) layers – Sparse expert routing activates only a small subset of experts per token, dramatically reducing compute while preserving model capacity.
For the Nano tier, a learned router selects 6 out of 128 experts per token, keeping the active parameter count near 3 B. The Super and Ultra tiers introduce LatentMoE, where tokens are projected into a lower‑dimensional latent space before expert processing, allowing many more experts (up to thousands) with minimal communication overhead.
“The hybrid design lets us keep the speed of Mamba for long sequences, the precision of attention where it matters, and the scalability of MoE without a linear increase in FLOPs.” – NVIDIA research team
Precision & Performance: NVFP4 and Active‑Per‑Token Ratio
Nemotron 3 Super and Ultra are primarily trained in NVFP4, a 4‑bit floating‑point format optimized for NVIDIA GPUs. Matrix multiplications run in NVFP4 while accumulations use higher‑precision formats, delivering:
- ~30 % lower memory footprint compared to FP16.
- Near‑FP16 accuracy on benchmark suites.
- Up to 4× higher token throughput for the Nano variant versus Nemotron 2 Nano.
Training Corpus & Benchmarks
Nemotron 3 models were pretrained on an estimated 25 trillion tokens, incorporating more than 3 trillion new tokens beyond the Nemotron 2 generation. The data mix includes:
| Dataset | Content Type | Scale |
|---|---|---|
| Nemotron Common Crawl v2.1 | Web text | ≈ 15 T tokens |
| Nemotron CC Code | Open‑source code | ≈ 5 T tokens |
| Scientific & Reasoning Corpus | Research papers, math, logic | ≈ 3 T tokens |
On standard LLM benchmarks (MMLU, GSM‑8K, and LongBench), Nemotron 3 Nano achieves:
- +4.2 % accuracy over Nemotron 2 Nano on MMLU.
- ~30 % reduction in inference latency for 1 M‑token contexts.
- Speculative decoding gains of up to 2.5× when using multi‑token prediction heads (Super/Ultra).
Target Use‑Cases: Long‑Context & Agentic AI
The design choices of Nemotron 3 make it a natural fit for several emerging AI scenarios:
- Enterprise Knowledge Bases
- Ingest entire policy manuals, legal contracts, or product documentation (up to 1 M tokens) and answer queries without chunking.
- Multi‑Agent Collaboration
- Enable fleets of agents to share a common “working memory” that spans millions of tokens, supporting complex planning and tool‑calling pipelines.
- Code‑Intensive Workflows
- Analyze massive code repositories, generate refactorings, or perform cross‑file static analysis in a single forward pass.
- Creative Content Generation
- Produce long‑form narratives, scripts, or research reports while maintaining coherence across chapters.
For SaaS platforms that need to embed AI assistants, the UBOS AI ecosystem already offers ready‑to‑use connectors, making it trivial to plug Nemotron 3 into existing workflows.
Nemotron 3 vs. Earlier NVIDIA Models
Compared with the Nemotron 2 family, the new generation delivers measurable gains:
| Metric | Nemotron 2 | Nemotron 3 Nano | Nemotron 3 Super |
|---|---|---|---|
| Parameters (total) | ≈ 30 B | 31.6 B | ≈ 100 B |
| Active per token | ≈ 1 B | ≈ 3 B | ≈ 10 B |
| Context window | 8 K tokens | 1 M tokens | 1 M tokens |
| Throughput (tokens/s) | ~12 K | ~48 K | ~120 K |
These improvements translate directly into lower operational costs for developers building open‑source AI solutions on the UBOS platform.
How UBOS Leverages Nemotron 3 for Real‑World Impact
UBOS provides a full‑stack environment that makes it easy to turn Nemotron 3 into production‑grade services:
- UBOS platform overview – A low‑code environment for deploying large language models with built‑in scaling.
- Workflow automation studio – Orchestrate multi‑agent pipelines that exploit the 1 M token context.
- AI marketing agents – Use Nemotron 3 to generate hyper‑personalized campaigns that understand entire customer journeys.
- Enterprise AI platform by UBOS – Secure, multi‑tenant deployment of Nemotron 3 Ultra for large corporations.
- UBOS solutions for SMBs – Affordable access to long‑context models for small businesses.
- UBOS for startups – Accelerate product‑market fit with pre‑built Nemotron 3 templates.
- UBOS pricing plans – Transparent cost structures that align with active‑per‑token usage.
- Web app editor on UBOS – Drag‑and‑drop UI to build AI‑powered web apps without code.
- About UBOS – Learn more about the team behind the platform.
Conclusion: A New Era for Long‑Context, Agentic AI
Nemotron 3’s hybrid Mamba‑Transformer MoE design, massive context window, and efficient active‑per‑token computation collectively set a new benchmark for open‑source LLMs. Whether you are a researcher probing the limits of reasoning, a data scientist building multi‑agent systems, or a product leader seeking to embed truly long‑form AI capabilities, Nemotron 3 offers a compelling foundation.
Ready to experiment? Visit the UBOS homepage to spin up a Nemotron 3 instance in minutes, explore the UBOS templates for quick start, or join the UBOS partner program to co‑create next‑generation AI solutions.
Stay ahead of the curve—integrate Nemotron 3 today and unlock AI that truly thinks at the scale of human knowledge.
Andrii Bidochko
CTO UBOS
Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.