Efficient Pretraining of LLMs in NVFP4 at PyTorchCon
Anjulie Agrusa, Ryan Spring, and Bruce Zitelli of NVIDIA will show how the latest NVFP4 pretraining recipes can accelerate large-scale LLM training while closing the quality gap to BF16 at PyTorch Conference North America 2026.
“Efficient Pretraining of LLMs in NVFP4,” will cover the recipe and PyTorch tooling that make this possible, including recipe design, kernel choices, and the API surface needed to bring NVFP4 training into native PyTorch workflows.
The recipes are being upstreamed into the PyTorch ecosystem through TorchAO and TorchTitan, with dense linear NVFP4 training available in TorchAO today.
Register for PyTorchCon North America: https://hubs.la/Q04v4SL60
“Efficient Pretraining of LLMs in NVFP4,” will cover the recipe and PyTorch tooling that make this possible, including recipe design, kernel choices, and the API surface needed to bring NVFP4 training into native PyTorch workflows.
The recipes are being upstreamed into the PyTorch ecosystem through TorchAO and TorchTitan, with dense linear NVFP4 training available in TorchAO today.
Register for PyTorchCon North America: https://hubs.la/Q04v4SL60
PyTorch
Welcome to the official PyTorch YouTube Channel. Learn about the latest PyTorch tutorials, new, and more.
PyTorch is an open source machine learning framework that is used by both researchers and developers to build, train, and deploy ML systems that so...
torch.compile and Diffusers: A Hands-On Guide to Peak Performance - Sayak Paul, Hugging Face
PyTorch
torch compile and Diffusers - A Hands On Guide to Peak Performance - PyTorch Compiler Series
PyTorch
verl: An Open Source Large Scale LLM RL Framework for Agentic Tasks - Yuxuan Tong, Bytedance
PyTorch
Lightning Talk: Sparsifying Vision Transformers with Minimal Accuracy Loss - Jesse Cai, Meta
PyTorch