Unsupervised learning: clustering assets and correlation structure

ML · Chapter 412 min readAsked at Two Sigma, AQR, QuantCo, Citadel

Assumes Other supervised methods: kNN, SVMs and the kernel trick.

After this lesson you should be able to

  • Choose a clustering method from the shape you expect.
  • Convert a correlation into a valid distance.
  • Say what EM does and where mixture models are used in finance.

Unsupervised methods look for structure without a target, and in finance the structure of interest is almost always the correlation matrix — which assets move together, and whether those groups are stable. Getting the distance right matters more than the choice of algorithm.

MethodAssumesGood for
k-meansRound, similar-sized, Euclidean clustersQuick partitions of well-separated data
HierarchicalA nested structure; no fixed kkCorrelation matrices, where nesting is natural
Gaussian mixtureClusters are Gaussian, possibly elongatedRegime identification, soft assignment
DBSCANClusters are dense regions of any shapeOutlier detection; no need to fix kk
PCAVariance is the structureFactor extraction, dimensionality reduction
Table 4.1 · The methods. Hierarchical clustering is the usual choice on a correlation matrix: it needs no kk, it produces a dendrogram you can inspect, and the nesting matches how markets are actually organised into sectors within regions.

Equation 4.2

Correlation as a distance

The standard conversion. It is a proper metric — it satisfies the triangle inequality — which naive alternatives like 1−ρ1 - \rho do not.

dij=2(1−ρij)d_{ij} = \sqrt{2(1 - \rho_{ij})}
ρ=1\rho = 1
Distance zero: identical.
ρ=−1\rho = -1
Distance 2: maximally opposed.
-101012√(2(1−ρ))Correlation ρDistance
Figure 4.3 · Turning correlation into a distance. Perfectly correlated names sit on top of each other, uncorrelated ones are 2\sqrt{2} apart, and perfectly anticorrelated ones are as far apart as the metric allows. Clustering on 1−ρ1-\rho instead breaks the triangle inequality, and every algorithm that assumes a metric then misbehaves quietly.

Why the square root matters. It comes from geometry: if returns are standardised, the Euclidean distance between two assets’ return vectors is exactly 2(1−ρ)\sqrt{2(1-\rho)} times a constant. So this is not a convenient transformation but the actual distance in the space where the data live — which is why it satisfies the triangle inequality and why clustering with it produces groupings that behave sensibly. Feeding 1−ρ1 - \rho to a method that assumes a metric gives results that can violate transitivity in ways that are hard to notice and hard to defend.

The rest of this lesson is in Premium

You have read the opening. 10 more sections follow, including 4 worked examples and 3 quick checks.

Start the free 7-day trialSign in

Nothing is charged for 7 days, and you can cancel before then. Or read The law of large numbers and the central limit theorem in full, free.

Unsupervised learning: clustering assets and correlation structure · QuantMax