Elbow method
The Elbow method can be used to determine a suitable number of clusters in an unsupervised method. This is done by plotting a suitable metric (such as "distortion" or also quite commonly the silhouette score) against the number of clusters. The goal is to identify the point where increasing the amount of clusters has diminishing returns for the cluster quality, in the hope of getting the most meaningful amount of clusters. This is done by finding the first significant turning point in the curve, i.e. its "elbow". This method is only reliable if there is indeed a clear curve.
See the following example (WIP) where I started work on an implementation using sklearn, but also allow a very nice ready made method from the yellowbrick library:
from sklearn.cluster import KMeans
from sklearn.metrics import calinski_harabasz_score, silhouette_score, silhouette_samples
import matplotlib.pyplot as plt
from yellowbrick.cluster import KElbowVisualizer
def kmeans_elbow_curve(X, style=2, max=15, metric='distortion'):
'''
Curve to help automatically define a good amount of clusters
for k-means, based on the inertia score. Does not use the
custom Clusterer but only K-Means.
style: if 1, use plain elbow method using sklearn; if 2, use nice visualization from the Yellowbrick library
max: maximum number of clusters to try
metric: uses 'distortion' by default (sum of squared distances from each point to its assigned cluster center)
another possible value is 'silhouette' or 'calinski_harabasz'
'''
elbow_value = 0 # The elbow value is the point where a curve starts to flatten out
Ks = (2, max) # K clusters to try
if style == 1:
# Computation 'by hand'
Cs = range(*Ks)
if metric.lower() == 'silhouette':
scores = [silhouette_score(X, KMeans(n_clusters=c).fit(X).labels_) for c in Cs]
elif metric.lower() == 'calinski_harabasz':
scores = [calinski_harabasz_score(X, KMeans(n_clusters=c).fit(X).labels_) for c in Cs]
if metric.lower() == 'distortion':
scores = [KMeans(n_clusters=c).fit(X).inertia_ for c in Cs]
plt.plot(Cs, scores)
plt.show()
# TODO We have not yet automatically determined the elbow value
print("WARNING: Automatic determination of elbow value not yet determined for style=1!")
else:
# Very nice yellowbrick visualization
visualizer = KElbowVisualizer(
KMeans(), k=Ks,
metric=metric)
visualizer.fit(X)
visualizer.show()
elbow_value = visualizer.elbow_value_
return elbow_value
