Penalized and generalized linear models: Logistic regression, L1 and L2 penalization, elastic net, SCAD penalty, application to high dimensional low sample size problems. Intro to neural networks: Artificial neuron, single hidden layer, multiple hidden layers, backpropagation, momentum, loss functions, relation with support vector machines and penalized logistic regression. Convolutional neural networks: Convolutional layers, pooling layers, drop out, VGGnet, inception modules, residual networks, deconv nets, applications to object recognition. Why deep learning works: Role of depth, the closeness of local minima to global minimal, the predominance of saddle points and ridges vs. local minima.
Recurrent neural networks and LSTMs: lateral connections, LSTM units, gated recurrent networks, applications to NLP.
Probabilistic Graphical Models: Factor graphs and belief networks, Deep belief networks and Boltzmann machines, sampling methods including Gibbs sampling, contrastive divergence, generative adversarial networks.