Sparsity: A Crash Diet to Reduce Model Size and Latency

State-of-the-art AI models require powerful hardware to run and they have enormous file sizes–leading to high hosting and inference costs. SMEs are unable to integrate these models into their applications because of their limited budget. Sparsity is a technique that prunes a model’s parameters leading to a smaller, faster model that can run on-device. An SME’s IT team can use sparsity to integrate powerful models into their applications while reducing hosting and inference costs.

Mon., 27. January 2025 | 4 min read

Similar to quantization, sparsity reduces a model’s size, which leads to cost savings for hosting and inference. This reduction technique is useful because a model’s size and computational needs usually increase as its performance improves. Using SparseGPT, any GPT model can be pruned by at least 50% with a minimal accuracy decrease. Sparsity prunes a model by removing redundant parameters. Sparsity can allow SMEs to run models on-device for their applications. Meta’s Llama 3.1 and Mistral Large 2 are over 229 GB in file size. This makes it difficult for SMEs to afford cloud computing costs to use large foundational models in their applications. An SME’s IT team can turn to sparsity to reduce costs and energy consumption and improve inference speeds while maintaining a suitable accuracy level.