Field Note #51. On "Deep Computational Burden"

Source: Neil Thompson et al., "The Computational Limits of Deep Learning," avail. online at arXiv (July 2022)

The Main Idea: Computational Gauntlet.

The fate of deep learning is tied to solving the computational burden.  The computational expense comes from two aspects that increase deep learning efficacy: extremely large data sets and overparameterization that enhances model flexibility (and thus improves chances that predictive attributes are captured).

"[T]he growing computational burden of deep learning will soon be constraining for a range of applications, making the achievement of important benchmark milestones impossible if current trajectories hold." (Id. at 15)

The Visuals.

The Details.