Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models

/ Authors

Nvidia Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adi Renduchintala, A. Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Mahabaleshwarkar

and 189 more authors

Andrew Tao, Anna Shors, Ashwath Aithal, A. Poojary, Ayush Dattagupta, B. Buddharaju, Bobby Chen, Boris Ginsburg, Boxin Wang, Brandon Norick, Brian Butterfield, Bryan Catanzaro, Carlo del Mundo, Cheng Dong, C. Harvey, Christopher Parisien, Dan Su, Daniel Korzekwa, Hongxu Yin, Daria Gitman, David Mosallanezhad, Deepak Narayanan, Denys Fridman, Dima Rekesh, Ding Ma, Dmytro Pykhtar, Dong Ahn, Duncan Riach, Dusan Stosic, E. Long, Elad Segal, Ellie Evans, Eric Chung, Erick Galinkin, E. Bakhturina, Ewa Dobrowolska, Fei Jia, Fuxiao Liu, G. Prasad, Gerald Shen, Guilin Liu, Guo Chen, Haifeng Qian, Helen Ngo, Hongbin Liu, Hui Li, Igor Gitman, I. Karmanov, Ivan Moshkov, Izik Golan, Jan Kautz, Jane Scowcroft, J. Casper, Jarno Seppanen, Jason Lu, J. Sewall, Jiaqi Zeng, Jiaxuan You, Jimmy Zhang, Jing Zhang, Jining Huang, Jinze Xue, Jocelyn Huang, Joey Conway, John Kamalu, Jon Barker, J. Cohen, Joseph Jennings, Jupinder Parmar, Karan Sapra, Kari Briski, Kateryna Chumachenko, K. Luna, Keshav Santhanam, Kezhi Kong, K. Sivamani, Krzysztof Pawelec, Kumar Anik, Kunlun Li, Lawrence C. McAfee, Leon Derczynski, Lindsey Pavao, Luis Vega, Lukas Voegtle, Maciej Bala, Maer Rodrigues de Melo, Makesh Narsimhan Sreedhar, Marcin Chochowski, Markus Kliegl, Marta M. Stepniewska-Dziubinska, Matthieu Le, Matvei Novikov, Mehrzad Samadi, M. Andersch, Michael Evans, Miguel Martínez, Mike Chrzanowski, Michael Ranzinger, Mikolaj Blaz, Misha Smelyanskiy, M. Fawzy, M. Shoeybi, M. Patwary, Nayeon Lee, Nima Tajbakhsh, Ning Xu, O. Rybakov, Oleksii Kuchaiev, Olivier Delalleau, Osvald Nitski, Parth Chadha, Pasha Shamis, P. Micikevicius, Pavlo Molchanov, Peter Dykas, Philipp Fischer, P. Aquilanti, Piotr Bialecki, Prasoon Varshney, Pritam Gundecha, Przemek Tredak, Rabeeh Karimi, Rahul Kandu, Ran El-Yaniv, Raviraj Joshi, R. Waleffe, Ruoxi Zhang, Sabrina Kavanaugh, Sahil Jain, Samuel Kriman, Sangkug Lym, S. Satheesh, Saurav Muralidharan, Sean Narenthiran, S. Anandaraj, Seonmyeong Bak, Sergey Kashirsky, Seungju Han, Shantanu Acharya, Shaona Ghosh, Sharath Turuvekere Sreenivas, Sharon Clay, S. Thomas, Shrimai Prabhumoye, Shubham Pachori, Shubham Toshniwal, Shyamala Prayaga, Siddhartha Jain, Sirshak Das, Slawomir Kierat, Somshubra Majumdar, Song Han, Soumye Singhal, Sriharsha Niverty, Stefania Alborghetti, Suseella Panguluri, Swetha Bhendigeri, Syeda Nahida Akter, Szymon Migacz, Tal Shiri, Terry Kong, Timo Roman, Tomer Ronen, Trisha Saar, Tugrul Konuk, Tuomas Rintamaki, Tyler Poon, Ushnish De, V. Noroozi, Varun Singh, V. Korthikanti, V. Kurin, W. Ahmad, Wei Du, Wei Ping, Wenliang Dai, Wonmin Byeon, Xiao-Qian Ren, Yao Xu, Yejin Choi, Yian Zhang, Ying Lin, Yoshi Suhara, Zhiding Yu, Zhiqi Li, Zhiyu Li, Zhongbo Zhu, Zhuolin Yang, Zijia Chen

/ Abstract

As inference-time scaling becomes critical for enhanced reasoning capabilities, it is increasingly becoming important to build models that are efficient to infer. We introduce Nemotron-H, a family of 8B and 56B/47B hybrid Mamba-Transformer models designed to reduce inference cost for a given accuracy level. To achieve this goal, we replace the majority of self-attention layers in the common Transformer model architecture with Mamba layers that perform constant computation and require constant memory per generated token. We show that Nemotron-H models offer either better or on-par accuracy compared to other similarly-sized state-of-the-art open-sourced Transformer models (e.g., Qwen-2.5-7B/72B and Llama-3.1-8B/70B), while being up to 3$\times$ faster at inference. To further increase inference speed and reduce the memory required at inference time, we created Nemotron-H-47B-Base from the 56B model using a new compression via pruning and distillation technique called MiniPuzzle. Nemotron-H-47B-Base achieves similar accuracy to the 56B model, but is 20% faster to infer. In addition, we introduce an FP8-based training recipe and show that it can achieve on par results with BF16-based training. This recipe is used to train the 56B model. We are releasing Nemotron-H base model checkpoints with support in Hugging Face and NeMo.