Advanced Topics in Computer Science: Efficient Systems for Foundation Models

As models increase in size and training budget, they not only systematically improve in upstream quality, but also exhibit novel emergent capabilities. This increase in scale raises proportionate difficulties for practitioners: foundation model training and inference lie at a unique interdisciplinary crossroad, combining open problems in algorithms, system design, and software engineering. The goal of this course is to give an overview of emerging research questions and challenges associated with foundation model training and inference.