Data leakage is using information at fit time that a real deployment would not have yet. The most common source is preprocessing: fit a scaler or a feature selector on the whole dataset, then split, and the transform has already read the rows you later score on. Fit every transform on the training folds only, which a Pipeline does for you because the split happens first.

A cross-validation score is meant to estimate how a model does on data it has not seen. That estimate quietly breaks when a preprocessing step is fit on the full dataset before the data is split into training and test parts. The scikit-learn docs give the general definition: “Data leakage occurs when information that would not be available at prediction time is used when building the model.” The effect they name is the one that matters here, because it looks like success — leakage produces “overly optimistic performance estimates.”

Feature selection is a clean way to see it happen, because the leak is large and easy to reproduce.

A run on pure noise

Take a dataset with no signal in it at all: 200 rows, 10,000 features drawn from a normal distribution, and labels assigned at random. There is nothing for any model to learn, so the only honest score is chance, about 0.5.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, KFold
from sklearn.pipeline import make_pipeline

rng = np.random.RandomState(0)
X = rng.normal(size=(200, 10000))   # pure noise: 10000 random features
y = rng.randint(0, 2, size=200)     # random 0/1 labels — NO real signal
cv = KFold(n_splits=5, shuffle=True, random_state=0)

# WRONG: pick the 20 "best" features using the whole dataset, THEN cross-validate
sel = SelectKBest(f_classif, k=20).fit(X, y)
leaked = cross_val_score(LogisticRegression(), sel.transform(X), y, cv=cv).mean()

# RIGHT: selection lives in the pipeline, re-fit on each training fold only
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())
honest = cross_val_score(pipe, X, y, cv=cv).mean()

SelectKBest(f_classif, k=20) scores each of the 10,000 features against the labels — an ANOVA F-statistic per feature — and keeps the 20 with the highest scores. In the first version that .fit(X, y) runs across all 200 rows. Then cross_val_score splits the already-reduced 20-column matrix into five folds, trains a logistic regression on four of them, and scores it on the fifth.

That first version returns about 0.815. The pipeline version returns about 0.545. Same noise, same labels, same model, same five folds — the only difference is where the feature selection was fit.

Why the number is inflated

The selection step in the first version was fit on X and y together, all 200 rows. When a fold later holds out 40 rows to score itself, those 40 rows were already part of choosing the 20 columns. Their labels helped decide which features survived. So the held-out rows are not really held out: the model is scored on rows that already contributed to how the features were picked.

With 10,000 random columns and 200 rows, a handful will line up with the random labels by chance closely enough to look predictive across the whole set. Selecting on the full data keeps exactly those, and every fold inherits them. The 0.815 is the score of a model that found 20 columns correlated with the labels in the same data it is then measured on. There is nothing to learn, and it still scores well above chance.

Fit on the training folds only

The pipeline version fits the same SelectKBest inside make_pipeline, so on each fold the selection is re-fit on that fold’s training rows alone and then applied to the held-out rows without ever seeing their labels. The 20 columns are chosen five separate times, from four folds each time, and the fifth fold is scored on columns picked without it. The result lands near chance, which is what pure noise should score.

This is the rule the scikit-learn docs state: “Always split the data into train and test subsets first, particularly before any preprocessing steps,” and preprocessing “transformations are only learnt from the training data.” The general rule they give is “to never call” fit on the test data — and the same holds for the held-out fold during cross-validation. Wrapping the transform in a Pipeline is their recommended way to enforce it, because the pipeline fits on the training part and only transforms the test part. Kaggle’s write-up on leakage puts the same failure in terms of the target: it “happens when your training data contains information about the target, but similar data will not be available when the model is used for prediction.”

Any transform with a fit step can do this, not just feature selection. The size of the leak varies: a supervised selector reads the labels, so it leaks a lot, while something like a StandardScaler reads only the feature columns and usually leaks very little in a setup like this. The habit is the same either way — fitting before the split lets the transform read rows the model is later graded on, so the estimate comes back higher than it should. Fit transforms on the training folds only; put the preprocessing in a pipeline so the split comes first.

That’s the whole demonstration — a small script you can run yourself. If I’ve got something wrong here, tell me and I’ll fix it.

Sources

• scikit-learn, Common pitfalls and recommended practices — Data leakage
• scikit-learn, Cross-validation
• Kaggle, Data Leakage

Run with scikit-learn 1.9.0 and numpy 2.4.6; the seeds make it deterministic.