Home / scikit-learn / references/unsupervised_learning.md

references/unsupervised_learning.md

Unsupervised Learning Reference

Overview

Unsupervised learning discovers patterns in unlabeled data through clustering, dimensionality reduction, and density estimation.

Clustering

K-Means

KMeans (sklearn.cluster.KMeans) - Partition-based clustering into K clusters - Key parameters: - n_clusters: Number of clusters to form - init: Initialization method ('k-means++', 'random') - n_init: Number of initializations (default=10) - max_iter: Maximum iterations - Use when: Know number of clusters, spherical cluster shapes - Fast and scalable - Example:

from sklearn.cluster import KMeans

model = KMeans(n_clusters=3, init='k-means++', n_init=10, random_state=42)
labels = model.fit_predict(X)
centers = model.cluster_centers_

# Inertia (sum of squared distances to nearest center)
print(f"Inertia: {model.inertia_}")

MiniBatchKMeans - Faster K-Means using mini-batches - Use when: Large datasets, need faster training - Slightly less accurate than K-Means - Example:

from sklearn.cluster import MiniBatchKMeans

model = MiniBatchKMeans(n_clusters=3, batch_size=100, random_state=42)
labels = model.fit_predict(X)

Density-Based Clustering

DBSCAN (sklearn.cluster.DBSCAN) - Density-Based Spatial Clustering - Key parameters: - eps: Maximum distance between two samples to be neighbors - min_samples: Minimum samples in neighborhood to form core point - metric: Distance metric - Use when: Arbitrary cluster shapes, presence of noise/outliers - Automatically determines number of clusters - Labels noise points as -1 - Example:

from sklearn.cluster import DBSCAN

model = DBSCAN(eps=0.5, min_samples=5, metric='euclidean')
labels = model.fit_predict(X)

# Number of clusters (excluding noise)
n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
n_noise = list(labels).count(-1)
print(f"Clusters: {n_clusters}, Noise points: {n_noise}")

HDBSCAN (sklearn.cluster.HDBSCAN) - Hierarchical DBSCAN with adaptive epsilon - More robust than DBSCAN - Key parameter: min_cluster_size - Use when: Varying density clusters - Example:

from sklearn.cluster import HDBSCAN

model = HDBSCAN(min_cluster_size=10, min_samples=5)
labels = model.fit_predict(X)

OPTICS (sklearn.cluster.OPTICS) - Ordering points to identify clustering structure - Similar to DBSCAN but doesn't require eps parameter - Key parameters: min_samples, max_eps - Use when: Varying density, exploratory analysis - Example:

from sklearn.cluster import OPTICS

model = OPTICS(min_samples=5, max_eps=0.5)
labels = model.fit_predict(X)

Hierarchical Clustering

AgglomerativeClustering - Bottom-up hierarchical clustering - Key parameters: - n_clusters: Number of clusters (or use distance_threshold) - linkage: 'ward', 'complete', 'average', 'single' - metric: Distance metric - Use when: Need dendrogram, hierarchical structure important - Example:

from sklearn.cluster import AgglomerativeClustering

model = AgglomerativeClustering(n_clusters=3, linkage='ward')
labels = model.fit_predict(X)

# Create dendrogram using scipy
from scipy.cluster.hierarchy import dendrogram, linkage
Z = linkage(X, method='ward')
dendrogram(Z)

Other Clustering Methods

MeanShift - Finds clusters by shifting points toward mode of density - Automatically determines number of clusters - Key parameter: bandwidth - Use when: Don't know number of clusters, arbitrary shapes - Example:

from sklearn.cluster import MeanShift, estimate_bandwidth

# Estimate bandwidth
bandwidth = estimate_bandwidth(X, quantile=0.2, n_samples=500)
model = MeanShift(bandwidth=bandwidth)
labels = model.fit_predict(X)

SpectralClustering - Uses graph-based approach with eigenvalues - Key parameters: n_clusters, affinity ('rbf', 'nearest_neighbors') - Use when: Non-convex clusters, graph structure - Example:

from sklearn.cluster import SpectralClustering

model = SpectralClustering(n_clusters=3, affinity='rbf', random_state=42)
labels = model.fit_predict(X)

AffinityPropagation - Finds exemplars by message passing - Automatically determines number of clusters - Key parameters: damping, preference - Use when: Don't know number of clusters - Example:

from sklearn.cluster import AffinityPropagation

model = AffinityPropagation(damping=0.9, random_state=42)
labels = model.fit_predict(X)
n_clusters = len(model.cluster_centers_indices_)

BIRCH - Balanced Iterative Reducing and Clustering using Hierarchies - Memory efficient for large datasets - Key parameters: n_clusters, threshold, branching_factor - Use when: Very large datasets - Example:

from sklearn.cluster import Birch

model = Birch(n_clusters=3, threshold=0.5)
labels = model.fit_predict(X)

Clustering Evaluation

Metrics when ground truth is known:

from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
from sklearn.metrics import adjusted_mutual_info_score, fowlkes_mallows_score

# Compare predicted labels with true labels
ari = adjusted_rand_score(y_true, y_pred)
nmi = normalized_mutual_info_score(y_true, y_pred)
ami = adjusted_mutual_info_score(y_true, y_pred)
fmi = fowlkes_mallows_score(y_true, y_pred)

Metrics without ground truth:

from sklearn.metrics import silhouette_score, calinski_harabasz_score
from sklearn.metrics import davies_bouldin_score

# Silhouette: [-1, 1], higher is better
silhouette = silhouette_score(X, labels)

# Calinski-Harabasz: higher is better
ch_score = calinski_harabasz_score(X, labels)

# Davies-Bouldin: lower is better
db_score = davies_bouldin_score(X, labels)

Elbow method for K-Means:

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

inertias = []
K_range = range(2, 11)
for k in K_range:
    model = KMeans(n_clusters=k, random_state=42)
    model.fit(X)
    inertias.append(model.inertia_)

plt.plot(K_range, inertias, 'bo-')
plt.xlabel('Number of clusters')
plt.ylabel('Inertia')
plt.title('Elbow Method')

Dimensionality Reduction

Principal Component Analysis (PCA)

PCA (sklearn.decomposition.PCA) - Linear dimensionality reduction using SVD - Key parameters: - n_components: Number of components (int or float for explained variance) - whiten: Whiten components to unit variance - Use when: Linear relationships, want to explain variance - Example:

from sklearn.decomposition import PCA

# Keep components explaining 95% variance
pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X)

print(f"Original dimensions: {X.shape[1]}")
print(f"Reduced dimensions: {X_reduced.shape[1]}")
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Total variance explained: {pca.explained_variance_ratio_.sum()}")

# Or specify exact number of components
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)

IncrementalPCA - PCA for large datasets that don't fit in memory - Processes data in batches - Key parameter: n_components, batch_size - Example:

from sklearn.decomposition import IncrementalPCA

pca = IncrementalPCA(n_components=50, batch_size=100)
X_reduced = pca.fit_transform(X)

KernelPCA - Non-linear dimensionality reduction using kernels - Key parameters: n_components, kernel ('linear', 'poly', 'rbf', 'sigmoid') - Use when: Non-linear relationships - Example:

from sklearn.decomposition import KernelPCA

pca = KernelPCA(n_components=2, kernel='rbf', gamma=0.1)
X_reduced = pca.fit_transform(X)

Manifold Learning

t-SNE (sklearn.manifold.TSNE) - t-distributed Stochastic Neighbor Embedding - Excellent for 2D/3D visualization - Key parameters: - n_components: Usually 2 or 3 - perplexity: Balance between local and global structure (5-50) - learning_rate: Usually 10-1000 - n_iter: Number of iterations (min 250) - Use when: Visualizing high-dimensional data - Note: Slow on large datasets, no transform() method - Example:

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, perplexity=30, learning_rate=200, n_iter=1000, random_state=42)
X_embedded = tsne.fit_transform(X)

# Visualize
import matplotlib.pyplot as plt
plt.scatter(X_embedded[:, 0], X_embedded[:, 1], c=labels, cmap='viridis')
plt.title('t-SNE visualization')

UMAP (not in scikit-learn, but compatible) - Uniform Manifold Approximation and Projection - Faster than t-SNE, preserves global structure better - Install: uv pip install umap-learn - Example:

from umap import UMAP

reducer = UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42)
X_embedded = reducer.fit_transform(X)

Isomap - Isometric Mapping - Preserves geodesic distances - Key parameters: n_components, n_neighbors - Use when: Non-linear manifolds - Example:

from sklearn.manifold import Isomap

isomap = Isomap(n_components=2, n_neighbors=5)
X_embedded = isomap.fit_transform(X)

Locally Linear Embedding (LLE) - Preserves local neighborhood structure - Key parameters: n_components, n_neighbors - Example:

from sklearn.manifold import LocallyLinearEmbedding

lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10)
X_embedded = lle.fit_transform(X)

MDS (Multidimensional Scaling) - Preserves pairwise distances - Key parameter: n_components, metric (True/False) - Example:

from sklearn.manifold import MDS

mds = MDS(n_components=2, metric=True, random_state=42)
X_embedded = mds.fit_transform(X)

Matrix Factorization

NMF (Non-negative Matrix Factorization) - Factorizes into non-negative matrices - Key parameters: n_components, init ('nndsvd', 'random') - Use when: Data is non-negative (images, text) - Interpretable components - Example:

from sklearn.decomposition import NMF

nmf = NMF(n_components=10, init='nndsvd', random_state=42)
W = nmf.fit_transform(X)  # Document-topic matrix
H = nmf.components_  # Topic-word matrix

TruncatedSVD - SVD for sparse matrices - Similar to PCA but works with sparse data - Use when: Text data, sparse matrices - Example:

from sklearn.decomposition import TruncatedSVD

svd = TruncatedSVD(n_components=100, random_state=42)
X_reduced = svd.fit_transform(X_sparse)
print(f"Explained variance: {svd.explained_variance_ratio_.sum()}")

FastICA - Independent Component Analysis - Separates multivariate signal into independent components - Key parameter: n_components - Use when: Signal separation (e.g., audio, EEG) - Example:

from sklearn.decomposition import FastICA

ica = FastICA(n_components=10, random_state=42)
S = ica.fit_transform(X)  # Independent sources
A = ica.mixing_  # Mixing matrix

LatentDirichletAllocation (LDA) - Topic modeling for text data - Key parameters: n_components (number of topics), learning_method ('batch', 'online') - Use when: Topic modeling, document clustering - Example:

from sklearn.decomposition import LatentDirichletAllocation

lda = LatentDirichletAllocation(n_components=10, random_state=42)
doc_topics = lda.fit_transform(X_counts)  # Document-topic distribution

# Get top words for each topic
feature_names = vectorizer.get_feature_names_out()
for topic_idx, topic in enumerate(lda.components_):
    top_words = [feature_names[i] for i in topic.argsort()[-10:]]
    print(f"Topic {topic_idx}: {', '.join(top_words)}")

Outlier and Novelty Detection

Outlier Detection

IsolationForest - Isolates anomalies using random trees - Key parameters: - contamination: Expected proportion of outliers - n_estimators: Number of trees - Use when: High-dimensional data, efficiency important - Example:

from sklearn.ensemble import IsolationForest

model = IsolationForest(contamination=0.1, random_state=42)
predictions = model.fit_predict(X)  # -1 for outliers, 1 for inliers

LocalOutlierFactor - Measures local density deviation - Key parameters: n_neighbors, contamination - Use when: Varying density regions - Example:

from sklearn.neighbors import LocalOutlierFactor

lof = LocalOutlierFactor(n_neighbors=20, contamination=0.1)
predictions = lof.fit_predict(X)  # -1 for outliers, 1 for inliers
outlier_scores = lof.negative_outlier_factor_

One-Class SVM - Learns decision boundary around normal data - Key parameters: nu (upper bound on outliers), kernel, gamma - Use when: Small training set of normal data - Example:

from sklearn.svm import OneClassSVM

model = OneClassSVM(nu=0.1, kernel='rbf', gamma='auto')
model.fit(X_train)
predictions = model.predict(X_test)  # -1 for outliers, 1 for inliers

EllipticEnvelope - Assumes Gaussian distribution - Key parameter: contamination - Use when: Data is Gaussian-distributed - Example:

from sklearn.covariance import EllipticEnvelope

model = EllipticEnvelope(contamination=0.1, random_state=42)
predictions = model.fit_predict(X)

Gaussian Mixture Models

GaussianMixture - Probabilistic clustering with mixture of Gaussians - Key parameters: - n_components: Number of mixture components - covariance_type: 'full', 'tied', 'diag', 'spherical' - Use when: Soft clustering, need probability estimates - Example:

from sklearn.mixture import GaussianMixture

gmm = GaussianMixture(n_components=3, covariance_type='full', random_state=42)
gmm.fit(X)

# Predict cluster labels
labels = gmm.predict(X)

# Get probability of each cluster
probabilities = gmm.predict_proba(X)

# Information criteria for model selection
print(f"BIC: {gmm.bic(X)}")  # Lower is better
print(f"AIC: {gmm.aic(X)}")  # Lower is better

Choosing the Right Method

Clustering:

  • Know K, spherical clusters: K-Means
  • Arbitrary shapes, noise: DBSCAN, HDBSCAN
  • Hierarchical structure: AgglomerativeClustering
  • Very large data: MiniBatchKMeans, BIRCH
  • Probabilistic: GaussianMixture

Dimensionality Reduction:

  • Linear, variance explanation: PCA
  • Non-linear, visualization: t-SNE, UMAP
  • Non-negative data: NMF
  • Sparse data: TruncatedSVD
  • Topic modeling: LatentDirichletAllocation

Outlier Detection:

  • High-dimensional: IsolationForest
  • Varying density: LocalOutlierFactor
  • Gaussian data: EllipticEnvelope
← Back to scikit-learn