Prediction Is Not Understanding

We built systems that predict extraordinarily well, and somewhere in the celebrating we forgot to ask whether prediction and understanding are the same thing. They are not. But the confusion is understandable — for most of history, they arrived together. You understood something, and so you could anticipate it. The two felt inseparable.

They are not inseparable. And that rupture is the quiet crisis of this moment. The philosophical essays of our past have plenty of emotional depth, but they often lack empirical evidence; our machine learning frameworks have formal rigor, but they lack a soul. To truly understand the current state of artificial intelligence, we must let each mode justify the other’s existence. The philosophy makes the engineering urgent. The engineering makes the philosophy honest.

Below is the transcript of an interrogation — a series of empirical autopsies performed on our most successful models, proving that our systems know everything and understand nothing.

You can verify the empirical code and models in this essay by running them yourself in the accompanying script and notebook environment.

The Fracture

Imagine two men who share an observatory but little else. One predicts eclipses with astonishing accuracy — centuries of celestial records compressed into a mathematical method. The other traces gravitational mechanics, the invisible choreography underneath the light. They often reach the same answer. They live in entirely different worlds.

Ancient astronomy predicted motion long before it understood causes. And there is something elegiac about that — all that careful watching, all those correct answers, in the absence of knowing why. Aristotle noticed this divide and named it: knowing that something occurs ($to\ hoti$), and knowing why it occurs ($to\ dioti$). Prediction belongs to the first. Understanding demands the second.

“We suppose ourselves to possess unqualified scientific knowledge of a thing… when we think that we know the cause on which the fact depends, as the cause of that fact and of no other, and, further, that the fact could not be other than it is.” — Aristotle (from Posterior Analytics, c. 350 BCE)

Machine learning unexpectedly returns us to this ancient divide. We have become very good at building the first astronomer. The second one’s chair sits empty.

Underdetermination — The First Wound

What makes predictive success so philosophically dangerous is that it feels like evidence of understanding. When something works, we assume comprehension behind it. Yet philosophy has long warned us of a trap: the Duhem-Quine thesisDuhem–Quine ThesisScientific hypotheses cannot be tested in isolation; empirical evidence underdetermines which theory is true. Read more →, which states that a body of evidence can always underdetermine the theory used to explain it. Multiple incompatible accounts of reality can produce identical observations.

“Any statement can be held true come what may, if we make drastic enough adjustments elsewhere in the system.” — W. V. O. Quine (from Two Dogmas of Empiricism, 1951)

In modern machine learning, this is no longer an abstract thought experiment; it is a demonstrable failure point when training deep networks. Let us build a non-linear physical system in code: Voltage ($X$) causes Current ($Y$), which subsequently causes Heat/Temperature ($Z$). Critically, $X$ acts as a confounder—directly affecting both the current $Y$ and the final temperature $Z$.

# mlp_intervention_collapse.py
import torch
import torch.nn as nn
import numpy as np

# --- Observational Data Generation ---
np.random.seed(42)
n_samples = 5000
x_obs = np.random.normal(0, 1.0, n_samples)
y_obs = 2.0 * x_obs + np.random.normal(0, 0.1, n_samples)
z_obs = y_obs**2 + 3.0 * x_obs + np.random.normal(0, 0.1, n_samples)

# Train a Deep MLP to predict Z from Y observationally (correlation-only)
class MLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(1, 16),
            nn.Tanh(),
            nn.Linear(16, 16),
            nn.Tanh(),
            nn.Linear(16, 1)
        )
    def forward(self, x): return self.net(x)

# [Training code omitted for brevity; converges to R^2 > 0.95]

The Analysis of the Collapse

If you execute this code, the MLP achieves an outstanding observational $R^2$ of 0.954. In observation, $X \approx Y/2$, meaning the network successfully fits the empirical curve: $E[Z \mid Y] = Y^2 + 1.5 Y$

But look at what happens when you intervene. If you physically set the current to $Y = 2.0$ while holding voltage $X$ at its baseline distribution, the true physical temperature should be determined by the causal laws: $\mathbb{E}[Z \mid do(Y=2.0)] = 2.0^2 + 3.0 \mathbb{E}[X] = 4.0$

Yet, the MLP predicts an expected temperature of 6.758 (an absolute error of 2.758). Because the predictive model was trained on correlation alone, it assumed that forcing $Y=2.0$ implied the voltage $X$ was also high. It cannot separate the circumstances under which an event occurs (observational correlation) from the physical mechanism that produces it (interventional causal effect). The numbers survive; the physics escapes.

Representations Don’t Mean Anything

The internal state of a neural network is often celebrated as a “conceptual representation.” But history keeps disabusing us of the idea that functional representation equals true reference. Philosopher Hilary Putnam famously illustrated this with his reference problemPutnam's Reference ProblemThe argument that words do not inherently attach to their referents without causal/environmental context. Read more →: symbols do not carry intrinsic meaning just because they successfully map to an external object.

“Words do not stand for things by themselves, nor does a mind stand for things by itself… Symbols do not correspond to things.” — Hilary Putnam (from Reason, Truth and History, 1981)

We see the tragedy of Putnam’s problem play out when training a Variational Autoencoder (VAE)Variational AutoencoderA generative neural network that maps data to a probabilistic latent space to generate new samples. Read more → to compress high-dimensional variables into independent latent dimensions. If two physical causes—such as the size and rotation of an object—are correlated in the pre-training data, the representation-learning objective conflates them:

# vae_causal_conflation.py
import numpy as np

# Generate independent causes (Size, Rotation) correlated in dataset (corr = 0.85)
cov = [[1.0, 0.85], [0.85, 1.0]]
z_true = np.random.multivariate_normal([0, 0], cov, 2000)
z_size, z_rotation = z_true[:, 0], z_true[:, 1]

# Project to 4D observations (corrupted by noise)
x = np.zeros((2000, 4))
x[:, 0] = z_size + 0.15 * z_rotation
x[:, 1] = z_rotation - 0.2 * z_size
x[:, 2] = np.sin(z_size) + z_rotation
# [VAE Trained to compress 4D inputs into 2 bottleneck variables]

The Conflated Concept Space

When we evaluate the resulting latent space, we discover that the VAE fails to disentangle the true independent causes. Instead of allocating one dimension to size and one to rotation, it conflates them.

Learned Latent Dimension 1 exhibits a 0.998 correlation with Size and a 0.868 correlation with Rotation. Because the network minimizes reconstruction error under observational distributions, it maps the data along the dominant axis of covariance, blending size and rotation into a single concept direction.

There are no canonical “conceptual dimensions.” There are only coordinates of statistical utility. The representation does not point to the structural joints of reality; it points only to the correlation of its observations.

Compression is Not Explanation

Machine learning operates heavily on the principle of the Minimum Description Length (MDL)Minimum Description LengthAn information-theoretic principle stating the best model is the one that achieves the most data compression. Read more →. The goal is elegant compression: find the shortest program or the tightest weight configuration that can reproduce the data. In the era of massive LLM scaling laws, we treat this compression as the ultimate proxy for intelligence.

But compression merely finds statistical regularities. It is not explanation. Explanation requires constraints over possible worlds — it describes what could not have been, not just what currently is.

# compression_vs_laws.py

# The model constructs an implicit lookup boundary through millions of parameters
def parameter_heavy_compression(mass, radius):
    # Imagine this simulating an overparameterized deep network mapping coordinates
    # to a high-dimensional probability density landscape.
    # It requires terabytes of text and observation to know what happens.
    pass

# The human system leverages a compact, invariant structural law
# Derived from active principles of energy conservation
def structural_explanation(M, R, G=6.674e-11):
    return np.sqrt((2 * G * M) / R)

When a scaling experiment shows that a model requires billions of parameters to “learn” a physical law that a human can express in five symbols, the model hasn’t found an explanation. It has built an incredibly sophisticated lookup table of probabilities. It has recorded the appearance of rain through pressure gradients and cloud formations, but it has no grasp of the thermodynamic transitions of water states. It has mapped the occurrence, but skipped the production.

The OOD Test — Prediction’s Reckoning

The empirical heart of this argument is found in the phenomenon of shortcut learningShortcut LearningWhen models rely on spurious statistical correlations (like background color) instead of actual causal features. Read more →, best illustrated by the famous Colored MNIST benchmark. Here, we build a neural network that achieves near-perfect training performance by identifying a spurious correlation, only to watch it collapse completely when the world shifts.

# shortcut_learning.py
import torch
import torch.nn as nn
import torch.optim as optim

class SimpleCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 8, kernel_size=3, padding=1)
        self.pool = nn.AdaptiveAvgPool2d((1, 1))
        self.fc = nn.Linear(8, 2)
        
    def forward(self, x):
        x = torch.relu(self.conv(x))
        x = self.pool(x).view(x.size(0), -1)
        return self.fc(x)

# --- THE AUTOPSY ---
# Environment 1 (In-Distribution): Shapes are correlated 95% with background color.
# Environment 2 (Out-of-Distribution): Shape-color correlation is inverted to 5%.

The Deception of Success

When you run this exact experiment, the convolutional network converges with a stunning 94.9% training accuracy and 93.2% in-distribution test accuracy. It appears to have mastered the abstract concept of bars and shapes. Yet, when evaluated on the Out-of-Distribution (OOD)Out-of-DistributionData that comes from a different distribution than the training data, testing a model's generalization. Read more → test set where the color correlations are inverted, its accuracy collapses to exactly 4.8%.

This is not a random coin flip ($50\%$). It is a systematic, predictable failure. The model has learned to cheat so thoroughly—relying on the color shortcut because color is mathematically simpler to compress than shape geometry—that when the cheat is inverted, it fails with absolute certainty. It uses color as an active negative predictor.

Philosopher Karl Popper argued that scientific theories are validated by how well they survive severe, risky attempts at falsificationFalsificationismKarl Popper's theory that scientific ideas must be framed so they can, in principle, be proven false. Read more →. The model did not pass Popper’s test.

“Confirmations should count only if they are the result of risky predictions; that is to say, if, undefended by the theory in question, we should have expected an event which was incompatible with the theory—an event which, had it occurred, would have refuted the theory.” — Karl Popper (from Conjectures and Refutations, 1963)

The 88.4% drop in performance is the exact moment the machine’s “habit and hope” collides with an altered reality. It reveals that our system was blind the entire time it was succeeding.

We Can’t Read Our Own Minds

We often turn to the field of Mechanistic InterpretabilityMechanistic InterpretabilityA subfield of AI safety seeking to understand neural networks by reverse-engineering their internal components. Read more → to save us, hoping to peer into the black box and extract its logic. John Searle’s classic Chinese RoomChinese RoomA thought experiment showing a system can manipulate symbols perfectly without actually understanding them. Read more → argument suggested that a system manipulating symbols lacks understanding because it blindly follows a rulebook.

“The reason that no computer program can ever be a mind is simply that a computer program is only syntactical, and minds are more than syntactical. Minds are semantical, in the sense that they have more than a formal structure, they have a content.” — John Searle (from Minds, Brains, and Programs, 1980)

But mechanistic interpretability reveals a twist that Searle didn’t anticipate: the rulebook itself is illegible.

When we examine the hidden representations of modern transformers, we find that individual neurons do not correspond to clean, isolated human concepts. Instead, they operate in a state of superpositionSuperpositionA neural network state where more features than neurons are represented using high-dimensional space. Read more →, acting as polysemanticPolysemantic NeuronA single neuron that fires in response to multiple unrelated concepts, making it difficult to interpret. Read more → entities.

# superposition_simulation.py
# Reconstructing 5 sparse features using a 2D bottleneck with a ReLU activation:
# h_hat = ReLU(W^T * W * h)
# We optimize W via gradient descent to maximize sparse recovery

The Statistical Soup

When we run a toy model compressing 5 sparse features into 2 bottleneck dimensions, the learned weight columns arrange themselves into a near-perfect symmetric pentagon with an average angle of 72.0 degrees (matching the theoretical symmetric maximum of 72.0 degrees exactly). Under this configuration, every single neuron acts polysemantically. Neuron 0 responds strongly ($|weight| > 0.45$) to exactly 3 separate, completely unrelated features simultaneously.

Because the network compresses more features than it has physical dimensions, it assigns features to linear combinations of directions in space rather than individual dedicated neurons. We find a statistical soup that resists translation. The network’s operations remain mysterious even to the engineers who wrote the architecture.

The Intervention Gap — Pearl’s Ladder

The most formal, mathematically precise expression of the prediction-versus-understanding divide belongs to Judea Pearl and his Ladder of CausationLadder of CausationJudea Pearl's 3-level causal hierarchy: Association (seeing), Intervention (doing), and Counterfactuals (imagining). Read more →. Pure prediction lives entirely on the first rung: Association. It estimates the conditional probability of $Y$ given $X$, written as $P(Y \mid X)$. True understanding requires moving to the second and third rungs: Intervention ($P(Y \mid \text{do}(X))$) and Counterfactuals.

“If I could sum up the message of this book in one pithy phrase, it would be that you are smarter than your data. Data do not understand causes and effects; humans do.” — Judea Pearl (from The Book of Why: The New Science of Cause and Effect, 2018)

We can see this gap clearly in experiments using the IHDP (Infant Health and Development Program)IHDP DatasetInfant Health and Development Program benchmark data, used for evaluating causal inference models. Read more → dataset, a standard benchmark for causal inference.

# intervention_gap.py
import numpy as np
import sklearn
from sklearn.linear_model import LinearRegression

# Simulated IHDP data: Confounding variable W represents baseline health status
# Treatment X is assigned non-randomly based on health (doctors treat sicker infants)
w = np.random.beta(2, 5, size=10000)
x = np.random.binomial(1, p=np.clip(0.8 - w, 0, 1))

# True outcome Y has a constant, true treatment effect of +2.0, but is driven by W
y = 2.0 * x + 35.0 * w + np.random.normal(0, 0.1, size=10000)

Trapped on the First Rung

If you run this simulation, the naive predictive estimator outputs a negative coefficient of -1.546 for the treatment variable $X$. The model concludes that infants who receive the treatment have worse health outcomes than those who do not. It predicts the historical data beautifully, yet it completely fails to calculate individual treatment effects.

Why? Because it confounds selection bias with true causal impact. It confuses the circumstances under which a doctor chooses to give a treatment (sicker infants get treated) with the mechanistic effect of the treatment itself. When we adjust for the baseline health confounderConfoundingAn unmeasured factor that influences both treatment and outcome, causing a spurious correlation. Read more → $W$, the coefficient immediately corrects to 2.001, matching the true causal treatment effect of 2.000 almost perfectly.

Without the explicit mathematical framework of the $\text{do}$-operator, a model cannot bridge this gap. It remains trapped on the bottom rung, observing everything and controlling nothing.

The Frontiers of Reference

This is not to say that the machine learning community is blind to these limitations. Two major contemporary paradigms are actively attempting to bridge the gap between predictive association and true reference:

1. Causal Representation Learning (CRL): Led by Bernhard Schölkopf and others, CRL attempts to move beyond standard representation learning by learning variables that are causally disentangled. Rather than relying on simple reconstruction objectives, CRL methods enforce invariance across multiple environments or temporal shifts, aiming to recover the true underlying structural causal models (SCMs) from high-dimensional observational data.
2. Mechanistic Interpretability and Sparse Autoencoders (SAEs): In the realm of large language models, researchers are using SAEs to decompose the highly polysemantic, compressed representations of neural networks into clean, monosemantic concepts. By enforcing high sparsity, SAEs act as a dictionary that isolates distinct concepts, attempting to map the arbitrary inner world of neural activations back to human-understandable semantic coordinates.

Yet, these remain active, difficult frontiers. The vast majority of deployed systems still rely on standard empirical risk minimization—leaving them structurally vulnerable to the blind spots of correlation.

The Conceptual Spectrum

To bridge these technical diagnostics with our philosophical intuition, we can organize the relationship between machine learning paradigms and epistemic concepts across three distinct levels of cognitive depth: