Michael's Wiki

Execute by optimizing the objective function:

$$ minimize \sum_{i=1}^{M} {\left\Vert \hat{f}^{(i)} \right\Vert}_1 = \sum_{i=1}^{M} {\left\Vert \frac{\tilde{f}^{(i)}}{\Vert \tilde{f}^{(i)} \Vert_2} \right\Vert}_1 $$

Input is described not as raw data, but as a feature matrix of features per observation.

This describes a single layer of a feature learning matrix. Can apply to repeated layers in a greedy manner as described below to train a deep neural net.

Ngiam et al execute this for their tests using an out-of-the-box optimizer¹⁾²⁾.

1. Normalize for uniform activity using $l_2$ norms
2. Normalize per example using $l_2$-ball
3. Normalize for sparseness using $l_1$

See introduction publication for detailed derivations³⁾.

Normalizing between features produces implicit competition between features.

• Population Sparsity
  • Observations are represented by sparse number of features.
• Lifetime Sparsity
  • Features are sparsely activated among all observations. The number of observations that each feature is activated in is sparse.
• High Dispersal
  • No feature dominates over the rest; each feature is similarly distributed. This functions as a counterbalance to the previous two.

Sparse filtering, as a framework, doesn't care what created the weights in the feature matrix. Ngiam used greedy layer-wise training⁴⁾.

Sparse filtering was applied to object classification and phone classification⁵⁾.

feature_learning object_classification phone_classification