The size of generative AI models is constantly increasing, with current models reaching hundreds of billions of parameters and the most advanced ones approaching 2 trillion. Training such large models on modern accelerators necessitates clusters exceeding 10,000 nodes. GKE, currently supporting the world's largest managed Kubernetes clusters with 15,000 nodes, has the capacity to handle these demanding training workloads. Anticipating further advancements and even larger models, we are introducing support for 65,000-node clusters. This expansion, combined with innovations in accelerator computing power, will enable the training of models with 10 trillion parameters or more.
Subscribe to Google Cloud Tech → https://goo.gle/GoogleCloudTech