Why can't you run an AI model even when you have enough storage? Because an AI model is just billions of numbers, and "8B" means 8 billion parameters, not 8 gigabytes. At full precision (16 bits each), those 8 billion numbers take about 16GB.
Compress them to 4 bits each, called quantization, and you're down to roughly 4GB plus some overhead. The catch: the whole model has to sit in fast memory your GPU can reach, not your SSD, not your hard drive, not even regular system RAM. Having space to download a model doesn't mean you can actually run it.
#techwithtim #ai #machinelearning