Scaling Laws for Neural Language ModelsPDF Download

Kaplan and colleagues study how language-model loss varies with model size, data, and compute. They turn scaling from an intuition into a set of empirical relationships within their experiments.

The curves are a good starting point for thinking about resource allocation. Their assumptions and measurement range matter; a fitted relationship is not a promise about every future model.