How OpenAI Saved 30,000 CPU Cores! ?
How OpenAI Freed 30,000 CPU Cores With One Config Line!
OpenAI's Kubernetes clusters push around 9 petabytes of logs a day, and Fluent Bit collects them on every node. It watches each log file with inotify, so every new log line fires a notification. The catch is that inotify only says a file changed, not how much data was written, so Fluent Bit runs a stat syscall every time just to read the new file size. At millions of log lines per second, that turns into millions of stat calls, enough to starve Fluent Bit of CPU and start dropping logs.
The fix was one line: turn inotify off and poll each file once per second. That took thousands of stat calls per second down to one. CPU usage dropped 50%, which freed 30,000 cores across the fleet.
Reactive inotify or a one-second poll, which would you have reached for first? Drop your answer below.
#OpenAI #FluentBit #Kubernetes #DevOps #Linux #inotify #syscall #Observability #SRE #CPUOptimization #CloudNative #Logging #KodeKloud
OpenAI's Kubernetes clusters push around 9 petabytes of logs a day, and Fluent Bit collects them on every node. It watches each log file with inotify, so every new log line fires a notification. The catch is that inotify only says a file changed, not how much data was written, so Fluent Bit runs a stat syscall every time just to read the new file size. At millions of log lines per second, that turns into millions of stat calls, enough to starve Fluent Bit of CPU and start dropping logs.
The fix was one line: turn inotify off and poll each file once per second. That took thousands of stat calls per second down to one. CPU usage dropped 50%, which freed 30,000 cores across the fleet.
Reactive inotify or a one-second poll, which would you have reached for first? Drop your answer below.
#OpenAI #FluentBit #Kubernetes #DevOps #Linux #inotify #syscall #Observability #SRE #CPUOptimization #CloudNative #Logging #KodeKloud
KodeKloud
...