The Best Optimizer Depends on Batch Size
Explore scaling rules, optimizer comparisons, and noisy-quadratic simulations.
Writing
Technical blog posts are collected below. On my Substack, Under the Assumptions, I share personal musings on the role of deep learning theory in modern practice.
The Hidden Infinity in Preference Learning
An illustration of how length normalization aids learning from model-annotated data
Using LESS Data to Tune Models
Data Selection in the Era of LLMs
How to Scale Hyperparameters as Batch Size Increases
Understanding Optimization using Stochastic Differential Equations