AI Infrastructure with vLLM and LLM-D
? Try it yourself in the free lab: https://kode.wiki/4hAjYQq
AI infrastructure explained, from a single GPU all the way to a full fleet of model servers running in production.
Every ChatGPT reply hides a stack that is unreasonably hard to build. This pulls that stack apart piece by piece, starting with one GPU and one model file, then scaling it into a cluster that serves the whole world. By the end, the reason ChatGPT sometimes says "at capacity" stops being a mystery and turns into a memory-math problem you can actually reason about.
? What you'll learn:
1️⃣ Why GPUs (not CPUs) run models, and what compute, capacity, and bandwidth each decide
2️⃣ The two halves of every request: prefill (the pause) and decode (the stream)
3️⃣ How the KV cache and prefix caching cut both latency and cost
4️⃣ Why batching hits a hard ceiling, and that ceiling is memory, not compute
5️⃣ How LLM-D routes a whole fleet on Kubernetes so expensive GPUs stop sitting idle
? Free hands-on lab: https://kode.wiki/4hAjYQq
? Start Your AI Journey with KodeKloud: https://kode.wiki/4qsrspX
⏰ Timestamps :
00:00 - Why AI infrastructure is the career move right now
02:05 - How ChatGPT Works?
02:43 - What a Model really is?
05:09 - What is a Transformer?
05:45 - A model is just a file on disk
07:01 - Why you can't run it on your laptop
09:32 - GPU cores, VRAM, and bandwidth
12:57 - The Three numbers that size every GPU
15:36 - PyTorch and the vLLM model server
17:41 - Tokens and the one-token-at-a-time loop
20:34 - Prefill vs decode: TTFT and TPOT
24:53 - The KV cache and prefix caching
28:27 - Batching: many users on one GPU
32:22 - Sharding: splitting giant models across GPUs
34:40 - Why load balancing breaks for LLMs
37:53 - LLM-D: the smart router on Kubernetes
41:16 - LLM-D Well Lit Paths
43:10 - Running LLM-D on Kubernetes
44:56 - Stateful Sets
46:50 - Deployed in Kubernetes
48:45 - Recap
50:39 - Try our Free Labs
? Learn AI from this Playlist: https://www.youtube.com/watch?v=ZaPbP9DwBOE&list=PL2We04F3Y_43f3x3n9pawcEuAwru7bcMG
? Subscribe for more AI infrastructure and DevOps deep dives
#LLMInference #LLMD #vLLM #KodeKloud #AIInfrastructure #Kubernetes #GPU #KVCache #AIEngineering #MLOps #DevOps #Inference #Transformers #ChatGPT #ModelServing
AI infrastructure explained, from a single GPU all the way to a full fleet of model servers running in production.
Every ChatGPT reply hides a stack that is unreasonably hard to build. This pulls that stack apart piece by piece, starting with one GPU and one model file, then scaling it into a cluster that serves the whole world. By the end, the reason ChatGPT sometimes says "at capacity" stops being a mystery and turns into a memory-math problem you can actually reason about.
? What you'll learn:
1️⃣ Why GPUs (not CPUs) run models, and what compute, capacity, and bandwidth each decide
2️⃣ The two halves of every request: prefill (the pause) and decode (the stream)
3️⃣ How the KV cache and prefix caching cut both latency and cost
4️⃣ Why batching hits a hard ceiling, and that ceiling is memory, not compute
5️⃣ How LLM-D routes a whole fleet on Kubernetes so expensive GPUs stop sitting idle
? Free hands-on lab: https://kode.wiki/4hAjYQq
? Start Your AI Journey with KodeKloud: https://kode.wiki/4qsrspX
⏰ Timestamps :
00:00 - Why AI infrastructure is the career move right now
02:05 - How ChatGPT Works?
02:43 - What a Model really is?
05:09 - What is a Transformer?
05:45 - A model is just a file on disk
07:01 - Why you can't run it on your laptop
09:32 - GPU cores, VRAM, and bandwidth
12:57 - The Three numbers that size every GPU
15:36 - PyTorch and the vLLM model server
17:41 - Tokens and the one-token-at-a-time loop
20:34 - Prefill vs decode: TTFT and TPOT
24:53 - The KV cache and prefix caching
28:27 - Batching: many users on one GPU
32:22 - Sharding: splitting giant models across GPUs
34:40 - Why load balancing breaks for LLMs
37:53 - LLM-D: the smart router on Kubernetes
41:16 - LLM-D Well Lit Paths
43:10 - Running LLM-D on Kubernetes
44:56 - Stateful Sets
46:50 - Deployed in Kubernetes
48:45 - Recap
50:39 - Try our Free Labs
? Learn AI from this Playlist: https://www.youtube.com/watch?v=ZaPbP9DwBOE&list=PL2We04F3Y_43f3x3n9pawcEuAwru7bcMG
? Subscribe for more AI infrastructure and DevOps deep dives
#LLMInference #LLMD #vLLM #KodeKloud #AIInfrastructure #Kubernetes #GPU #KVCache #AIEngineering #MLOps #DevOps #Inference #Transformers #ChatGPT #ModelServing
KodeKloud
...