This talk will demonstrate the algorithmic progress that can exponentially reduce the compute and memory cost of training and inference using "dynamic sparsity" with neural networks. Dynamic sparsity, unlike static sparsity, aligns with Neural Scaling Laws and does not reduce the power of neural networks while reducing the number of FLOPS required by neural models by 99% or more. We will show how data structures, particularly randomized hash tables, can be used to design an efficient "associative memory" that reduces the number of multiplications associated with the training of the neural networks. Current implementations of this idea challenge the common knowledge prevailing in the community that specialized processors like GPUs are significantly superior to CPUs for training large neural networks. The resulting algorithm is orders of magnitude cheaper and energy-efficient. Our careful implementations can train billions of parameter recommendations and Language models on commodity desktop CPUs significantly faster than top-of-the-line TensorFlow alternatives on the most potent A100 GPU clusters, with the same or better accuracies.

We will show some demos, including how to train and fine-tune (with rhfl) a billion-parameter language model on a laptop from scratch for search, discovery, and summarization.