Michael's Wiki

Trace Trick

The trace is the sum of the diagonals of a matrix. $tr(A) = \sum_i A_{i,i}$

The trace operator has the cyclic permutation property: $tr(x^T A x) = tr(A x^T x) = tr(x^T x A)$

See Kevin Murphy text page 99.

Dealing with http://en.wikipedia.org/wiki/Mahalanobis_distance Mahalanobis terms

Expand out $(x-\mu)^T\Sigma^{-1}(x-\mu)$ into $(x^T\Sigma^{-1}x) - (x^T\Sigma^{-1}\mu) - (\mu^T\Sigma^{-1}x) + (\mu^T\Sigma^{-1}\mu) $

Only need to bring writing instruments.

Test is through Lecture 10, Model Selection With Probabilistic Models

In principle, can only study homeworks and class notes. Additional reading should help.

A d-by-d covariance matrix can be stored using $\frac{d^2}{2}$ parameters due to symmetry.

For $K$ models in problem 3.2, Bayesian Model Averaging.

Go up to basics of HMMs

Review of basic probability

“All models are wrong, but some are useful.” - George Box

conditional independence

Directed Graphical Models

• Common Examples
• Computing in Graphical Models

linear correlation

Multivariate Gaussian Model

• See Murphy text p97-98, Barber p169

The key to learning is writing down a likelihood. Then the full Bayesian model is just icing afterword.

Learning from Data

Review of Computing in Graphical Models: complexity analysis

Maximum Likelihood Estimation

Canceled for MLK Jr. Day

Computing in Graphical Models: binary tree model

Naive Bayes MLE Classification Example

Conditional Likelihood

Conditional Likelihood

Theoretical Properties of Maximum Likelihood

Bayesian Estimation

Conjugacy

Predictive Modeling

Reviewed HW 2

Discuss Midterm

Bayesian Estimation

Predictive Densities:Model Selection With Probabilistic Models

Predictive Modeling

Model Selection With Probabilistic Models

• See chapter on Model Comparison and Occam's Razor from David MacKay's book on

Information Theory and Inference, and video lecture of David lecturing on Bayesian inference.

Regression

Linear Regression:MSE Criterion

Gradient Descent

Midterm Exam

Review Midterm

Review Regression

Probabilistic Interpretation of Regression

Conditional Likelihood for Regression

Cancelled for President's Day holiday

Bayesian Interpretation of Regression

Theoretical Properties of Minimizing MSE

Bias-Variance Tradeoff

Logistic Regression:Conditional Likelihood

• See Murphy text chapter 8

Newton-Raphson Optimization

Multiclass Logistic Regression

Feedforward Neural Networks

Generative Approach / Classification

Unsupervised Learning:Finite Mixture Models

Expectation Maximization

EM for Gaussian Mixtures

EM for Binary Data Mixtures

Online EM

Generalized EM

continued Expectation Maximization:Singular Solutions

Kernel Density Estimation

Hidden Markov Models* (See Murphy text chapter 17)

review of Hidden Markov Models* 10 minutes of Gibbs Sampling