In machine learning, a neural scaling law is an empirical scaling law that describes how neural network performance changes as key factors are scaled up or down. These factors typically include the number of parameters, training dataset size,[1][2] and training cost.
- ^ Bahri, Yasaman; Dyer, Ethan; Kaplan, Jared; Lee, Jaehoon; Sharma, Utkarsh (2024). "Explaining neural scaling laws". Proceedings of the National Academy of Sciences. 121 (27): e2311878121. arXiv:2102.06701. Bibcode:2024PNAS..12111878B. doi:10.1073/pnas.2311878121. PMC 11228526. PMID 38913889.
- ^ Hestness, Joel; Narang, Sharan; Ardalani, Newsha; Diamos, Gregory; Jun, Heewoo; Kianinejad, Hassan; Patwary, Md Mostofa Ali; Yang, Yang; Zhou, Yanqi (2017-12-01). "Deep Learning Scaling is Predictable, Empirically". arXiv:1712.00409 [cs.LG].