Sample-wise vs. feature-wise normalization

It is important to be aware of the difference between normalization methods that work column-wise (i.e. per feature) and that work row-wise (per sample).

Feature-wise

If you care about the distribution of values for a given feature, then you should consider feature-wise normalization (i.e. column-wise in a n_samples x n_features matrix). Many classifiers either use or make assumptions on the distribution of feature values and hence it is often important to scale the features (so column-wise) to a normal distribution with zero mean and unit standard deviation. In sklearn you can do this with a StandardScaler.

Be aware that because this operation is column-wise, you should do this after data splitting, otherwise you'll leak information from the test set. See feature selection as source of train test leakage.

Sample-wise

If you apply sample-wise normalization, you will alter the value distributions per feature! You'll keep information on the relative relation between feature values per sample. Whether this is appropriate depends on your application.

For example, a common NLP/IR task is the clustering of documents to find meaningful patterns. It may turn out that a particular type of text is characterized by the frequent usage of verbs over adverbs. In this case, we do not really care that in text X twice as many verbs are used as in text Y, because this may simply be explained by text X being twice as long. In we normalize sample-wise, we mitigate the effect of text length while keeping the information on the relation between the frequency of verbs and adverbs. Similar reasoning applies in applications that rely on distance metrics between feature vectors. We can achieve this e.g. with the sklearn Normalizer.

L2-normalization is quite common and means that the vector is rescaled such that the L2-norm (sum of squares) of the feature vector is 1. This also has additional practical numerical benefits. L2-normalization is for example pretty much a must when using k-means.

Handy links: