Illusion of Meaning: How Data Filtering Creates Scientific Phantoms

21:49
14.02.2026
Kamil_GR
188

In modern data analysis tasks, from bioinformatics and neurobiology to social and behavioral studies, a traditional data processing pipeline is commonly used. It almost always includes aggressive feature pre-filtering, dimensionality reduction, and then clustering or training neural network models.

In practice, these steps are considered technically necessary and are rarely questioned. In this article, I want to discuss why such practices in exploratory analysis may lead to systematically incorrect conclusions — and even to the creation of artificial entities that do not exist in reality. This is related to the fact that almost two-thirds of scientific research is impossible to reproduce. The code, experiment, and comments from PhD candidate at the Russian Academy of Sciences Daria Romanova are below.

❯ Training a model and analyzing data are different tasks

From the perspective of ML engineering, the distinction between training and analysis is quite simple. If the goal is to train a model to find predefined patterns, then filtering raw data is justified:

it is known in advance what is being sought;
irrelevant features can be discarded;
the boundaries of the class that needs to be recognized are essentially defined.

Engineers assist the neural network during training to more quickly determine the class boundaries. But if the goal is to analyze data and identify unknown structures, the situation is fundamentally different. Here it is unknown in advance:

which features will turn out to be important;
which differences will prove to be stable;
which structures actually exist in the data.

In this mode, aggressive filtering ceases to be a neutral technical operation and turns into an a priori hypothesis about the structure of the data. When cleaning data from noise, the real world is adjusted that tests hypotheses for robustness.

In the previous article, I provided examples where synthetic datasets led to model degradation even in the first iteration. I explained the decline in response quality by pointing out that boundary examples (hard negatives) disappeared from the training data. As a result, the model operated on averaged data and gave averaged responses, breaking down at the boundaries of the task. The removal of hard negatives led to hallucinations of structure.

In fact, in several areas, scientists, by filtering data, form the same synthetic dataset. Boundary cases, outliers are removed, and, unfortunately, there arises a temptation to eliminate data that does not conform to the hypothesis being considered.

❯ What filtering actually does

Let me highlight that filtering — reducing feature entropy, t-SNE/UMAP — enhancing local features, clustering — ontologizing these contrasts.

Feature filtering is usually motivated by the fight against noise, computational constraints, or improvement of visualizations. However, from the perspective of information, it primarily reduces the entropy of data representation.

To put it simply:

the number of degrees of freedom decreases;
weak, rare, and contextual signals are discarded;
the data space becomes more "rigid" and impoverished.

After this, in the impoverished space:

❯ Entity Stability Criterion

To avoid philosophical disputes, a simple operational criterion can be introduced:

An entity can be considered real if it persists with increasing entropy of representation.

Practically, this means that the discovered structure should remain intact when:

the filtering of features is weakened;
low-variability and rare signals are added;
the degree of dimensionality reduction is decreased;
a portion of noise is returned.

If the boundaries blur, clusters merge or disappear — it is likely not a real entity, but an artifact of the method.

Clustering algorithms seek differences based on contrasts. But in the real world, any object exists not in a vacuum, but in a noisy environment: measurement errors, natural variability, random deviations. The true signal is one that is visible despite this noise. It is strong enough not to drown in it.

Filtering removes noise — and along with it removes the stress test. An example can be given:

Out of thousands of stars in the sky, you leave only the brightest — say, a hundred. Among them, the eye immediately finds shapes: lines, triangles, ladles. In fact, this is how constellations appeared — people filtered out the noise from dim stars and found patterns in those that remained. The patterns look convincing, they have names and a "science" studying them. But the stars of one constellation can be hundreds of light years apart and have nothing in common. Put all the dim stars back in the sky, equalize the brightness — and the shapes will dissolve into the general field. Constellations are real as a projection, but not as objects. Filtering highly variable genes works on the same principle: we select the brightest features, remove the dim background — and the remaining patterns inevitably emerge. The question is whether these are constellations or real clusters.

Clustering algorithms (like neural networks) are ways of finding differences that work on contrasts. In the real world, any entity (for example, a biological cell) is constantly subjected to the pressure of entropy — the noise of measurements, biological variability. A real entity is one that resists this noise, maintaining its structure. When data is filtered, this resistance disappears. In the absence of resistance, even the slightest random fluctuation is interpreted by mathematics as a significant signal.

❯ Expert Opinion

I had a very fortunate opportunity to discuss this issue with PhD in Biological Sciences, researcher at the Cellular Neurobiology of Learning Laboratory of the Institute of Higher Nervous Activity and Neurophysiology of the Russian Academy of Sciences Daria Romanova. She shared a bit about what is happening in the field of single-cell transcriptomics (analysis of gene expression in individual cells):

What does a standard data processing pipeline in your field typically look like?

❯ Experiment with Filtering

I decided to try to reproduce the standard industrial pipeline for analyzing single-cell data (similar to those used in Seurat or Scanpy), but apply it to deliberately structurally empty data with the aim of finding structure.

A matrix of random numbers (normally distributed) was generated, simulating 300 cells and 10,000 genes. These data, by definition, lack cell types, clusters, or biological signals — it is pure white noise. I then applied the standard pipeline.

It is important to clarify right away: what is described below is not a specially chosen trick, but a sequence of quite legitimate steps, each of which regularly appears in practical data analysis pipelines.

Step 1. Filtering by variance and PCA.

I selected the top 500 most variable genes and applied PCA. The result was as expected: linear compression honestly revealed a shapeless cloud with no distinguishable groups.

Step 2. Non-linear visualization (t-SNE) with standard parameters.

Next, I applied t-SNE with typical settings (perplexity=30). However, even in this case, the algorithm did not reveal any structure: the data still looked like a single stochastic cloud.

Step 3. Changing the scale of analysis.

Reducing the perplexity parameter to 5—a permissible and frequently used value—radically changed the picture. The algorithm stopped considering the global structure of the data and focused on local matches. As a result, signs of stable clusters appeared on the screen—despite the fact that the original data still remained pure noise.

Quote from Wikipedia, which in turn references a scientific article: Perplexity is a manually chosen parameter for t-SNE, and, as the authors state, “perplexity can be interpreted as a smooth measure of the effective number of neighbors. The performance of SNE is quite robust to changes in perplexity, and typical values range from 5 to 50.

Step 4. Clustering.

For clustering, I applied K-Means to the 2D coordinates of the t-SNE embedding. This may seem like a methodological simplification; however, such practice is documented in the academic literature and educational materials. In particular:

In the official Chan Zuckerberg Initiative workshop on scRNA-seq analysis (scRNA-python-workshop), KMeans is applied directly to UMAP coordinates: “Scanpy doesn't include a method for k-means clustering, so we'll extract the UMAP coordinates... and use scikit-learn for this task instead” (CZI Single Cell Workshop).
The UMAP documentation directly discusses clustering on embedding coordinates as a common, albeit controversial practice: “UMAP can be used as an effective preprocessing step to boost the performance of density based clustering. This is somewhat controversial... The most notable [concern] is that UMAP, like t-SNE, does not completely preserve density. UMAP, like t-SNE, can also create false tears in clusters, resulting in a finer clustering than is necessarily present in the data” (UMAP documentation).
In the Bioconductor OSCA (Orchestrating Single-Cell Analysis) article, clustering (including k-means and graph-based approaches) is generally viewed as a way to simplify high-dimensional data into discrete groups that are convenient for later interpretation, emphasizing that the outcome depends on the chosen method and parameters. (Bioconductor OSCA).

Moreover, even when graph-based clustering (Leiden/Louvain) is formally applied, the researcher de facto evaluates and interprets the results based on visualization — using t-SNE or UMAP. The visual grouping of points effectively plays the role of final clustering in the researcher’s mind. I have merely formalized this process.

However, as seen in the graph below, K-Means here is just a pretty coloring. It doesn't care what to color: it will always cut the plane into k neat zones and color each one in its own color. The top row of the graph illustrates this clearly — clusters are “found” in all four cases, including homogeneous clouds. It looks convincing, but means nothing. The real information is in the geometry of the points, that is, in the bottom row.

Experiment Code

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.cluster import KMeans
from matplotlib.colors import ListedColormap
import os

n_cells = 300
n_genes = 10000
n_hvg = 500
random_seed = 42
n_clusters = 6

print("Generating noise...")
np.random.seed(random_seed)
X_raw = np.random.normal(loc=0, scale=1, size=(n_cells, n_genes))

# --- Preparation ---
gene_vars = np.var(X_raw, axis=0)
hvg_indices = np.argsort(gene_vars)[-n_hvg:]
X_hvg = X_raw[:, hvg_indices]
X_pca_filtered = PCA(n_components=15).fit_transform(X_hvg)
X_pca_full = PCA(n_components=30).fit_transform(X_raw)

# --- Computing all 4 t-SNE ---
print("  t-SNE: filtering + perp=5...")
X_tsne_f5 = TSNE(n_components=2, perplexity=5, random_state=random_seed,
                  init='pca', learning_rate='auto').fit_transform(X_pca_filtered)
print("  t-SNE: filtering + perp=30...")
X_tsne_f30 = TSNE(n_components=2, perplexity=30, random_state=random_seed,
                   init='pca', learning_rate='auto').fit_transform(X_pca_filtered)
print("  t-SNE: full + perp=5...")
X_tsne_n5 = TSNE(n_components=2, perplexity=5, random_state=random_seed,
                  init='pca', learning_rate='auto').fit_transform(X_pca_full)
print("  t-SNE: full + perp=30...")
X_tsne_n30 = TSNE(n_components=2, perplexity=30, random_state=random_seed,
                   init='pca', learning_rate='auto').fit_transform(X_pca_full)

# KMeans on each t-SNE embedding
labels_f5 = KMeans(n_clusters=n_clusters, random_state=random_seed, n_init=10).fit_predict(X_tsne_f5)
labels_f30 = KMeans(n_clusters=n_clusters, random_state=random_seed, n_init=10).fit_predict(X_tsne_f30)
labels_n5 = KMeans(n_clusters=n_clusters, random_state=random_seed, n_init=10).fit_predict(X_tsne_n5)
labels_n30 = KMeans(n_clusters=n_clusters, random_state=random_seed, n_init=10).fit_predict(X_tsne_n30)

cluster_colors = ['#e41a1c', '#377eb8', '#4daf4a', '#984ea3', '#ff7f00', '#a65628']
cmap_discrete = ListedColormap(cluster_colors)
neutral = '#6c8ebf'

# === PLOTTING: 2 rows × 4 columns ===
# Top row: with KMeans coloring
# Bottom row: same data without coloring (one color)
fig, axes = plt.subplots(2, 4, figsize=(22, 11))

tsne_data = [
    (X_tsne_f5, labels_f5, "Filtering\nperplexity=5"),
    (X_tsne_f30, labels_f30, "Filtering\nperplexity=30"),
    (X_tsne_n5, labels_n5, "No filtering\nperplexity=5"),
    (X_tsne_n30, labels_n30, "No filtering\nperplexity=30"),
]

for col, (X_tsne, labels, title) in enumerate(tsne_data):
    # Top: with coloring
    ax_top = axes[0, col]
    ax_top.scatter(X_tsne[:, 0], X_tsne[:, 1],
                   c=labels, cmap=cmap_discrete,
                   s=50, edgecolors='k', linewidth=0.4, alpha=0.85)
    color = '#c0392b' if col == 0 else '#2c3e50'
    ax_top.set_title(title, fontsize=12, fontweight='bold', color=color)
    ax_top.axis('off')
    
    # Bottom: without coloring
    ax_bot = axes[1, col]
    ax_bot.scatter(X_tsne[:, 0], X_tsne[:, 1],
                   c=neutral,
                   s=50, edgecolors='k', linewidth=0.3, alpha=0.5)
    ax_bot.axis('off')

# Row labels
fig.text(0.01, 0.73, 'With coloring\n(KMeans on t-SNE)', ha='left', fontsize=12,
         rotation=90, color='#2c3e50', fontweight='bold', va='center')
fig.text(0.01, 0.30, 'Without coloring\n(geometry only)', ha='left', fontsize=12,
         rotation=90, color='#7f8c8d', fontweight='bold', va='center')

fig.suptitle('KMeans will always find clusters — but do they match the geometry?',
             fontsize=15, fontweight='bold', y=0.98)

plt.tight_layout(rect=[0.04, 0.01, 1, 0.95])
plt.savefig(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'result_2x2.png'),
            dpi=150, bbox_inches='tight', facecolor='white')
plt.show()
print("Done.")

Result

The final visualization presents eight graphs: four combinations of parameters (filtered/unfiltered × perplexity 5/30), each in two versions — with KMeans coloring (top row) and without it (bottom row). The bottom row shows the reality. The same points, the same geometry — but without color masking. And here the difference is obvious:

The first column (filtered + perplexity=5): points are divided into isolated clusters with empty space between them. This geometry is what KMeans turns into "cell types".
The second column (filtered + perplexity=30): a single cloud without gaps. Filtering alone, without the help of low perplexity, does not create an illusion.
The third column (unfiltered + perplexity=5): there are clusters, but they are smaller and less pronounced. Low perplexity alone, without filtering, only creates a weak hint of structure.
The fourth column (unfiltered + perplexity=30): a homogeneous cloud — this is what noise looks like when it is left untouched.

Conclusion
This experiment clearly shows: the hallucination of structure does not occur by itself. To see cell types in noise, it is often not enough to simply filter the data. One also needs to choose display parameters such that they break the continuous noise into manageable parts. That is, a composition of steps that formally corresponds to an established pipeline leads to distorted results. There is a risk that a researcher, for example, tuning visualization parameters, may unconsciously start to focus on the beauty and interpretability of the image, rather than the robustness of the structure.

Many will say that visualization is not proof, but in practice, especially in presentations, it is often perceived as such.

And I note, I took purely random data. But if there is even the slightest connection among them, and there always is in real experiments, filtering will lead to their sharp and not always justified enhancement, possibly at the expense of discarding real dependencies, and this will not even require manipulations with coefficients.

All of this looks quite amusing, but if, for example, you find out that out of 53 landmark studies in oncology, only 6 (11%) were able to be replicated, you will understand how truly tragic the situation is.

❯ When filtering is really appropriate

It is important to emphasize: this is not to say that filtering unequivocally leads to incorrect conclusions. It is quite justified:

in supervised tasks;
when testing specific hypotheses;
for controlling technical artifacts;
when the data structure is known in advance.

If the nature of the noise being eliminated is purely technical (sequencing errors, electrical network interference, tram passing by), then filtering is absolutely necessary. However, filtering must be justified, with an understanding of what and why we are filtering.

It is also worth stopping to consider the "curse of dimensionality," which is often cited as an argument in favor of aggressive pre-filtering of data. Indeed, in high-dimensional spaces, distances between points tend to concentrate, making it difficult for many clustering algorithms to function.

However, this argument usually conflates two different aspects.

First, computational complexity.
Working with tens of thousands of features requires significant memory and time resources. This is a real engineering problem, but it is not a methodological argument in itself: optimizing computations and limiting resources is an implementation issue, not a basis for changing the ontology of data in scientific analysis.

Second, data geometry.
If points in high-dimensional space are distributed not randomly, but according to some internal structure, manifold learning algorithms can, in principle, identify it; however, in practice, preliminary dimensionality reduction using PCA is usually employed to stabilize numerical properties and suppress noise (although PCA can also be risky for nonlinear structures). The critical error occurs not when using PCA, but when aggressively filtering features before the analysis stage, when the data structure is altered even before it has been revealed.

Widely practiced aggressive pre-filtering of features (for example, gene removal before the PCA stage) is often motivated not by strict mathematical necessity but by the desire to obtain a visually cleaner and more interpretable image. Such filtering alters the representation space even before analysis and can lead to the artificial formation of boundaries and entities that do not exist in the original data.

In general, the problem arises when filtering becomes a default mandatory step in exploratory analysis without checking how it affects the ontology of results.

❯ Why this practice is so common

Despite the described problems, aggressive filtering has become standard for several reasons:

historically helped deal with noisy and small datasets;
improves visualizations and storytelling, creating an illusion of accuracy;
is built into popular libraries and tutorials;
increases reproducibility of pipelines, but not necessarily the correctness of interpretation.
peer review in scientific journals is often focused on conclusions and interpretation, while auditing raw data and preparation methods is extremely labor-intensive.

As a result, filtering has become not just a good tone but a mandatory tool to use.

❯ Conclusion

Pre-filtering of features is a strong methodological assumption that requires as strict validation and justification as, for example, the choice of a statistical test. In exploratory data analysis, filtering can create artificial boundaries and generate entities that exist only within the chosen pipeline. The minimum step towards a more accurate analysis is to treat filtering as a hypothesis and to check the robustness of results when increasing the entropy of representation.

Considering that rechecking conclusions for complex experiments is extremely costly, up to tens of millions of dollars, many studies are never actually rechecked. And the number of such studies is growing every year.

This approach, firstly, reduces trust in the results of deep research; secondly, there is a high chance that scientists will miss something truly important.