Spring 2017: Theory of Machine Learning

CS 5966/6966: Overview

Learning algorithms are ubiquitous, and are playing an ever increasing role in our daily lives. How are they different from any old algorithm? How can we reason about the ability of an algorithm to "learn from examples", and classify data it has never seen? We will introduce the formal notions of learnability, generalization, supervised vs. unsupervised learning, etc. The goal is also to introduce the fundamental techniques and models that are central in today's ML applications.

When/Where

Hours:       1:25-2:45PM Mon/Wed
Location:  MEB 3105
Office hours:  Tuesdays 10-12, or by appointment (MEB 3120)

Pre-requisites

There are no formal pre-requisites for the class, but in order to appreciate the content, some knowledge of basic machine learning methods (e.g., from CS 6350) is recommended. Background in basic algorithm design, probability and linear algebra will also be assumed.

Textbook/Resources

The main textbook for the course is Understanding Machine Learning: From Theory to Algorithms, by Shai Shalev-Shwartz and Shai Ben-David. A copy of the book (for personal use) is available online from the author's page.

In the later part of the course, we will also be using the excellent lecture notes by Ankur Moitra.

What we will cover

See the lecture schedule for full details. Here's a rough list of topics.

  • Statistical learning theory: background and the PAC model
  • Linear classification, kernels
  • VC dimension, generalization bounds, Rademacher complexity
  • Convex optimization, stochastic gradient descent
  • Online learning and the multiplicative weight update method
  • Neural networks: power and limitations
  • Dimension reduction: LDA, PCA
  • Unsupervised learning; clustering
  • Generative models, the EM algorithm
  • Factor analysis; non-negative matrix factorization, matrix completion, tensor decomposition

Grading/Logistics

There will be four homeworks, one on each of the four main themes in the course (40%). There is also a course project (30%), which involves a 15-30 minute presentation on a topic of your choice. A list of candidate topics will be provided half way into the semester. Finally, every student registered for the course is required to prepare high-quality scribe notes for two of the lectures (30%). These need to be turned in within 7 days of the lecture in order to receive full credit.