The Topography of Loss

Imagine opening an old photograph — one you haven’t touched in years. You expect recognition to arrive cleanly, the way light arrives. Instead, something stranger happens: fragments surface first, emotions before chronology, and details that feel half-invented. You do not recover a memory. You generate an approximation of one.

This is not a failure of storage. It is the nature of the operation itself. And modern machine learning, with its strange vocabulary of latent spacesLatent SpaceAn abstract multi-dimensional space where similar data points are clustered close to one another. Read more → and compressed representationsCompressionThe mapping of high-dimensional input into a lower-dimensional bottleneck, discarding non-essential details. Read more →, may be offering us the closest formal language we have ever had for what it actually feels like to remember — and to lose.

What follows is not a technical treatise. It is an attempt to use the architecture of representation learning as an epistemic object — a lens that brings into focus something we have long felt but rarely been able to describe precisely. That loss is not deletion. That forgetting is geometry, not erasure. That to remember is to reconstruct from coordinates that may have quietly shifted.

You can verify the empirical code and models in this essay by running them yourself in the accompanying script and notebook environment.

Memory Is Probably Not a Library

The intuition most of us carry is that experience is stored: filed, indexed, available for retrieval. We imagine the mind as a vast archive, every moment encoded somewhere in a drawer, waiting only for the right key. When retrieval fails, we assume the file has been misplaced or damaged. Grief feels like a room where something essential has been removed.

Modern cognitive science has been quietly dismantling this picture for decades. Memory, at the neural level, does not appear to operate as retrieval from stable storage. It appears closer to a process of reconstructionReconstructionThe generative process of decoding compressed latent coordinates back to high-dimensional data. Read more → — an active generation performed anew each time, from compressed traces, context, expectation, and the particular mood of the moment in which the attempt is made.

“But, in truth, the past is preserved of itself, automatically.” — Henri Bergson (from Matter and Memory, 1896)

The philosopher Henri Bergson argued that this automatic preservation means memory is not a collection of snapshots preserved behind glass. It is the persistence of the past into the present — not stored elsewhere, but folded into every act of attention, every perception of now. Memory, on this account, is not a record. It is the weight of duration bearing down on the present moment.

What happens when we take this intuition seriously — not just philosophically, but formally — is that memory becomes generative. Not archival.

Latent Space as a Theory of Memory

In representation learning, a simple autoencoder takes high-dimensional observations — an image of a thousand pixels, a sentence of ten thousand possible tokens — and compresses them into a much smaller set of coordinates. However, deterministic compression maps points to isolated spots in a fractured landscape. It has no reason to make the empty spaces meaningful. To solve this, a Variational Autoencoder (VAE)Variational AutoencoderA generative neural network that maps data to a probabilistic latent space to generate new samples. Read more → enforces a smooth probability manifold. It does not map an event to a single static point. It maps it to a probability distribution—defined by a mean $\mu$ and a variance $\sigma^2$.

This is where we meet the structure of the human mind. The dimensions of this space are not individually interpretable. You cannot point to coordinate $47$ and say: this is the concept of ocean. Yet if you train such a model on photographs of coasts, cities, and forests, images of oceans cluster together in latent space. The structure of the world — its similarities, its categorical textures — impresses itself onto the geometry without being explicitly named.

“If I uncover the sheet, the writing vanishes… The wax slab, however, retains the permanent trace of the writing in its deep grooves, even though these are not visible in themselves.” — Sigmund Freud (from A Note upon the “Mystic Writing-Pad”, 1925)

This distributed preservation reveals a striking synthesis between two historically opposed philosophies of memory: Henri Bergson’s temporal flow (durée) and Sigmund Freud’s spatial recording device.

While Bergson argued that memory is a continuous, non-spatial flow of the past folding into the present (which we see mirrored in the unfragmented, smooth geometry of the VAE’s latent space), Freud described memory as a two-layer spatial system—The Mystic Writing Pad (1925). Write upon the celluloid sheet, and the markings transfer to the wax slab. Wipe the sheet, and the writing vanishes, yet its physical impressions remain scored permanently in the wax below.

In a generative model, we find both models harmonized. The transient inputs and reconstructions act like Bergson’s fluid act of attention, cleared instantly from memory. The underlying model parameters (the weights), however, function exactly like Freud’s wax slab: they are scored and warped by every gradient that passes through them, carrying permanent, sub-threshold traces of what they once touched.

# vae_memory_manifold.py
import torch.nn as nn

class VariationalMemoryManifold(nn.Module):
    def __init__(self, input_dim=256, latent_dim=2):
        super().__init__()
        self.encoder_base = nn.Linear(input_dim, 64)
        
        # Mapping experience to a continuous probability distribution
        self.fc_mu = nn.Linear(64, latent_dim)      # Representational Center
        self.fc_logvar = nn.Linear(64, latent_dim)  # Ambiguity/Fuzziness range
        
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 64),
            nn.ReLU(),
            nn.Linear(64, input_dim),
            nn.Sigmoid()
        )

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std) # Incorporating active noise
        return mu + eps * std

# latent_distance_math.py
import numpy as np

def gaussian_kl_divergence(mu_1, logvar_1, mu_2, logvar_2):
    # Calculate difference between two continuous memory states
    var_1 = np.exp(logvar_1)
    var_2 = np.exp(logvar_2)
    
    kl = 0.5 * (
        np.sum(var_1 / var_2) + 
        np.sum((mu_2 - mu_1) ** 2 / var_2) - 
        len(mu_1) + 
        np.sum(logvar_2 - logvar_1)
    )
    return kl # Higher values indicate irreversible drift

// conceptual_memory_drift.js
const memoryManifold = {
  anchor: [0.12, -0.45], // Raw coordinates
  fuzziness: 0.15,               // Current emotional distortion
  
  reconstruct: function(attentionalState) {
    const distance = Math.sqrt(
      attentionalState.reduce((sum, val, i) => sum + (val - this.anchor[i])**2, 0)
    );
    // Signal degrades exponentially with spatial distance
    const clarity = Math.exp(-distance / this.fuzziness);
    return clarity > 0.5 ? "Vivid Representation" : "Generic Approximation";
  }
};

What if memory behaves similarly? Events become regions of probability in a high-dimensional space. Recall becomes traversal — an attempt to navigate back to coordinates close enough that reconstruction becomes possible again.

The remembered object is not stored intact. It is approached. And whether the approach succeeds depends on where you are standing now, and how far you have drifted from where the memory was last clearly felt.

The Sandbox Manifold

To understand this geometry, interact with it. Below is a raw, two-dimensional slice of a latent space. The colored points are embedded experiential clusters. The glowing golden node represents your Attention Anchor (the “Present State”).

As you pull your Attention Anchor through the coordinates, notice how reconstruction clarity is local. When you sit at the heart of an experience, the signal is clean. Drift slightly away, and details distort. If you move into the dead spaces between clusters, the decoder hallucinations occur — attempting to synthesize something that never happened, merging disparate memories into a strange, dreamlike composite.

Loss Is Geometric

If memory is position in a latent space, then loss — of access, of vividness, of the ability to reconstruct — is not deletion. It is increasing distance.

The memory still exists, in some formal sense. The weights that encode it have not been zeroed out. But the path has deteriorated. The coordinates from which reconstruction was once possible have shifted, and the machinery of retrieval can no longer navigate close enough.

Consider the phenomenology of the tip-of-the-tongue state. The word is there — you can feel its shape, its syllable count, the register it belongs to. You know you know it. The representation has not been erased. What has failed is traversal: the path between your current position and the coordinate where the reconstruction becomes complete. You circle the target. You feel its gravity. You cannot land.

This geometry also makes sense of the stranger facts of memory: why emotional recollection can survive the loss of detail, why déjà vu feels like activation without reconstruction. The emotional signature and the episodic content are not the same coordinate. They are nearby. When you reach one, the other may or may not follow.

Marcel Proust understood this before the mathematics existed to describe it. The remembered object, he wrote, is inaccessible through direct will. Only a particular sensation — the taste of a madeleine soaked in lime-flower tea, the uneven cobblestones underfoot — could serve as the retrieval key, because only those sensory coordinates were close enough to what remained. The emotion arrived before the event. The geometry always gets there first.

Regularization, Compression, and Decay

In variational autoencoders, the loss function balances two competing pressures. One is reconstruction lossReconstruction LossThe mathematical penalty assessing how much the reconstructed output differs from the original input. Read more → — how accurately the decoded output matches the input. The other is the KL divergenceKL DivergenceKullback-Leibler Divergence. A metric of how much one probability distribution diverges from a second one. Read more → — a regularization term that keeps the latent space smooth, preventing the model from collapsing to a single point.

$\mathcal{L}(\theta, \phi; x) = -\mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)] + \beta D_{\text{KL}}(q_\phi(z|x) \parallel p(z))$

The parameter $\beta$ acts as a dial of abstraction. If $\beta$ is zero, the model memorizes each event precisely but cannot generalize; it becomes rigid, fragile. If $\beta$ is too high, the latent space collapses, washing away individual details to leave only flat, universal approximations.

This destruction is not a failure of the system; it is the mechanism that produces generalization. Without it, there is no latent space, only memorization. Without loss, no abstraction.

The question this raises, quietly but insistently, is whether forgetting might be structured this way too. Not a failure of the mind, but an active regularization — a necessary destruction that prevents us from collapsing under the weight of total recall, that forces experience into a compressed form capable of organizing new encounters.

“Forgetting is not simply a vis inertiae as the superficial believe; it is rather an active and in the strictest sense positive faculty of repression…” — Friedrich Nietzsche (from On the Genealogy of Morals, 1887)

Nietzsche wrote that forgetting is not a passive failure of retention, but an active, positive faculty of containment—the boundary condition that allows us to remain conscious in the present. If this is true, then the memories we cannot release — the ones that stay too sharp, too detailed — are not signs of neurological strength. They are signs that the regularization has failed. The emotional $\beta$ term was set too low. The compression refused to happen.

We regularize to survive. But when the system refuses to generalize—when we demand exact, unyielding reconstruction of a coordinate that the universe has already cleared—the topology breaks down. This is the coordinate space of bereavement.

The Geometry of Grief and Machine Unlearning

Suppose someone disappears from your life. Not through a decision — through the irrevocable kind of absence that arrives without warning and without recourse. What happens to the latent manifold?

The representation does not delete. Their position in your encoded world does not simply zero out. It remains — or more precisely, the shape of the space around it remains. Other coordinates have been organized in relation to theirs for years. Experiences have been encoded relative to their presence. The geometry of your world has been structured, in countless small ways, around a point that is now unreachable.

This is why grief is so spatially disorienting. Not because something has been removed, but because the surrounding structure persists. The pathways into those regions of the latent space are still open. Only the reconstruction fails.

In machine learning, this distributed nightmare has a name: the problem of Machine UnlearningMachine UnlearningThe process of surgically erasing specific training points from a model without catastrophic interference. Read more →. If you want to erase a specific, sensitive training point from a model’s weights, you cannot simply hit “delete.” The model’s weights are distributed; that single training point influenced millions of parameters, subtly bending the entire latent landscape around itself.

To erase it cleanly, researchers must calculate complex influence functions or run “unlearning gradients.” Yet, these unlearning passes almost always cause damage to surrounding concepts — a phenomenon known as catastrophic collateral interference. In trying to force the network to forget one specific truth, we inadvertently distort the coordinates of everything else.

We see this in ourselves. To try and excise a memory—to unlearn a presence—is to alter our entire orientation. You cannot erase a single coordinates-set without fracturing the entire manifold. To forget them is to warp the world they lived in.

Identity as Drift Through Latent Space

We tend to imagine identity as something stable — a fixed point around which experience moves. I was that person then. I am this person now. There is a self that persists, unchanged at the core, even if the surface has shifted.

Representation learning suggests a different picture. If memory is not a library but a latent geometry — if the self is not a stored object but an encoded position — then identity may be better understood as a trajectory rather than a state. Not a point, but a path.

$x_t \to z_t \quad \text{where} \quad z_1 \to z_2 \to z_3 \to \dots$

Each experience encodes into a new coordinate. The sequence of coordinates — the trajectory through the latent manifold over a lifetime — is what we mean when we say “a person.” Not the current position. Not any single point on the path. The movement itself.

“A becoming is not a correspondence between relations. But neither is it a resemblance, an imitation, or, at the limit, an identification.” — Gilles Deleuze & Félix Guattari (from A Thousand Plateaus, 1980)

Gilles Deleuze argued something similar: that identity is becoming, not being. There is no essence hidden behind the accumulation of experience, waiting to be uncovered. There is only the process — always already in motion, never arriving at a final coordinate.

Which raises a question that has no comfortable answer: which coordinate was ever the “real” you? The person before the loss, or after? Before the decade that changed everything? The latent trajectory cannot be compressed to a single point without destroying the information that makes it meaningful. And yet we keep trying. We keep asking: what am I, at bottom? As if the answer could be still.

Why Some Things Never Return: Concept Drift

There is a property of compression that feels, when you encounter it for the first time, like a quiet revelation: it is irreversible. When you compress a signal — reduce its dimensions, throw away the information that is least necessary for reconstruction — you cannot, in general, recover what was discarded. The mathematics does not allow it. The information is gone.

This means that when you reconstruct a compressed memory, you are not recovering the original. You are generating the best approximation the compressed coordinates can produce, given your current decoder — which is itself not the same decoder it was the last time you attempted reconstruction.

In statistics, this is known as Concept DriftConcept DriftThe decay of predictive accuracy when target statistical properties change over time, altering context. Read more → or Covariate Shift. The parameters of the environment change over time. The historical coordinates remain static in the latent vault, but our decoders — the neural weights shaped by our present contexts, values, and age — have quietly re-tuned themselves. When we feed those old coordinates into our current, updated decoder, the output is entirely transformed.

You cannot reconstruct childhood. You reconstruct your current model of childhood — filtered through everything that has happened since, shaped by who you are at the moment of attempting the reconstruction, colored by the mood of the particular Tuesday afternoon in which you sit trying to remember.

“As long as Dasein exists, it is never past, but it is always indeed ‘having-been’…” — Martin Heidegger (from Being and Time, 1927)

Martin Heidegger argued that this “having-been” is not something behind us — not something we have left, available in principle for revisitation. It is folded into present existence, conditioning every act of perception, shaping the horizon from within which the present discloses itself. We do not stand outside time, reaching backward to retrieve what was. We are always already inside a history that structures what we can currently see.

This is not pessimism. It is something closer to a formal acceptance. The past changes not because it was never real, but because we change in our relation to it. The coordinates from which we approach it are always moving. What we reconstruct always carries the mark of the reconstruction — which is to say, the mark of the present in which it was attempted.

This is also, perhaps, what it means to heal. Not to recover the original signal. Not to undo the compression. But to develop a new decoder — a new set of weights — from which the approach to those coordinates produces something livable. Something that can be held without collapsing.