The Devil is in the Data

How our attempts to fix biased data could deepen distortion

At 5:24 AM, I rolled onto my side and opened my eyes to see four piercing blue eyes staring back at me. My son giggled with his tooth-filled grin, my husband cracking a smile at the sound of his giggles. Savoring a few moments of "comfy-womfy" before we begin our day in earnest. I want to remember this moment forever.

But the desire to remember does not guarantee memory. I feel the pull of desire to remember far more times in a day than seems plausible to achieve. Memory is tricky in this way. Some moments are indelibly seared into our memories, for better or worse, while others slip away, also for better or worse.

Remembering and forgetting are in constant tension. What do we remember or forget? When do we remember or forget specific memories? How accurately do we actually remember something? In times of conflict and especially after, what should we remember or forget? When we remember something, what context is relevant?

These are things we wrestle with as individuals, as a society, and now with AI. In the pursuit of building models that are as multifunctional as possible, companies have gone to great lengths to acquire as much training data as possible. But inevitably, this collection of data is biased and contains falsehoods as well. It’s obvious that this is a problem; what is less obvious is how to solve it.

Most forms of AI today are designed as a mimicry of ourselves. Sure, we are trying to make it more powerful than our individual capacities, but we also expect that technology interfaces cleanly with our umwelt. What can the human and social experience of memory teach us about the challenges ahead?

What it Means to Remember

Remembering is so baked into the human experience that we take it for granted. And we remember all sorts of things. Birthdates. Security codes. Our best friend's favorite drink order. A funny line from a movie. The heroic last play of a game. The secluded beach we had all to ourselves.

Evolutionarily, memory helps us to survive. Remembering where predators hide, what water sources persist through drought, and what berries are poisonous are all critical. Stories are the glue for societal cohesion, and as human tribes grew larger and larger, collectively remembering these stories became critical for our survival, too.

As much as we bemoan the things we forget, we don’t want to remember everything. The few individuals with hyperthymesia describe it as a burden to remember so much. So what should we forget?

The Value of Forgetting

Childhood is a great example of the balance between remembering and forgetting. I watch this drama play out daily in my toddler¹. Learning to walk requires hundreds of falls, large and small. Every day, he wakes up and enthusiastically tries to emulate the people around him, with little regard for the failed results of the day before.

Part of this resilience is thought to come from forgetting. By forgetting the pain of small falls, it’s easier to remain eager to try again. The process of effective learning seems to involve outsized reinforcement of small successes that accumulate into dramatic new skills. Most negative outcomes are quickly forgotten, though importantly not all. My son can still tell me about the "owie" he got on his forehead over a month ago.

The tool of forgetting can serve us beyond learning. Sometimes we experience something that is so painful to experience that we completely block the memory. This is an extreme response to trauma, but one that can be a blessing. As much as the absence of memory can itch like a phantom limb, those who retain all of their traumatic memories often wish that they could forget.

As individuals, we don't have a lot of control over what we remember or forget. Yes, there are techniques to help us intentionally remember specific facts, like names, but those approaches won't suddenly fill your mind with forgotten early childhood memories. Similarly, one of the best ways to fixate on something is to intentionally try to forget it.

We see a similar version of this play out on the societal scale, but with a key difference. It is possible, with the help of time, for one group of people to force the eradication of a societal memory.

How Societies Remember and Forget

For the most part, societal memory emerges from the stories individuals share and reinforce. The stories that are told at family dinners. The events that are commemorated with holidays. The narratives that are taught in schools. This forms the base of collective memory, though not its entirety.

Like individual memory, what is forgotten can be a process of innocence or more purposeful. When forgetting becomes deliberate, we see the emergence of "history is written by the victor" in its most literal form. This happens through the intentional destruction of culture: the burning of Mayan codices by Spanish conquistadors, forcing North American indigenous tribes to learn and speak only in English, and banning the discussion or publication of topics deemed dangerous.

A hallmark of forced forgetting is that a small number of people dictate what is remembered versus forgotten. Languages, cultures, and millennia-worth of traditions can be lost in a single generation.

When books are burned, the process is obvious to identify, but subtler dynamics can produce results that are just as dramatic. One of the more open examples of the tension between remembering and forgetting can be seen in how different societies approach remembering the Holocaust. There is a concerted effort across many nations to ensure memory of the mass-scale murder, the attempted erasure of entire segments of the European population, and the tragic consequences of dangerous rhetoric. These are the elements deemed essential to remember.

But a similar effort goes into forgetting and erasing other aspects. Nations around the world ignored these atrocities for years, which now makes them look complicit. Hitler's political rhetoric is often kept from public discourse out of fear that it will still resonate and lead to the same dangerous consequences. The complex reasons for widespread complicity are often reduced to simple stories of heroes and villains.

Different governments have taken different approaches to managing this balance. Germany has strict laws governing the use of Nazi symbols in public, as well as prohibiting the denial or glorification of the Holocaust. These laws reflect a society grappling with how to remember responsibly while preventing the repetition of past horrors.

The United States presents itself as taking a more open approach to these topics, but there’s a hypocrisy in this stance. It wasn't until 1988 that the US government admitted to the internment camps they established during World War II. More than 120,000 people were imprisoned in these camps, two-thirds of whom were US citizens. To this day, they are rarely talked about, and many in my generation remain clueless about their existence. Even self-proclaimed open societies curate what is remembered and when.

The uneven distribution of memory is stark, even though for most of the last 80-90 years, information sources were relatively consolidated. The last few decades have seen that change, as news outlets routinely cite posts on X as part of their breaking news coverage. What will the future hold as news sources increasingly fragment?

A Deluge of Digital Content and Its Consequences

In this digital age, we are living increasingly online. With this transition, social media and AI systems increasingly play roles analogous to those of traditional storytellers and textbooks, shaping what we remember and how we remember it.

Social media has created an unprecedented explosion of content, fundamentally changing how we process and preserve information. This includes vast quantities of misinformation and disinformation. While most of us believe we can spot misinformation, we are all vulnerable in some way.

This explosion of convincing but false content is thought to underlie a troubling trend: members of Generation Z and Generation Alpha are more than twice as likely to deny the Holocaust as older generations, including Millennials. When false narratives spread with the same velocity and as convincingly as factual accounts, it disrupts collective memory.

The challenge isn't just the volume of misinformation, but its sophistication. The outputs of LLMs are, by design, highly convincing - more convincing than humans. When AI is used to develop disinformation campaigns, it outperforms human-generated disinformation campaigns. The exploitation of our cognitive biases, tribal loyalties, and desire for simple explanations results in content that feels true even when it's demonstrably false.

When every perspective can find supporting "evidence" online, the distinction between rigorous historical scholarship and conspiracy theories begins to blur.

Biased Inputs → Biased Outputs

This blurring of lines is already underway, and is only accelerating with the rise of AI summaries over traditional web search. Earlier this year, I wrote about some of the flaws and risks of AI-generated summaries, including the aspect of source material. Here are two relevant excerpts:

In the past, whether cognizant of it or not, you were vetting your sources as you selected which to click and review to find your answer. AI summaries, by contrast, abstract that away. It’s one thing to know there’s a risk of an unreliable answer, the question is how can you tell?

There is a tendency across all of the major AI-answer platforms to rely extensively on just 1-3 of the cited sources8. You may see 7 sources cited, but the information from all of those sources is rarely included. So what looks like a well-balanced summary may in fact be heavily reliant on a single source. Furthermore, the single source leveraged may not be a very reliable one.

Even when AI systems represent source data accurately, that data still reflects the historical and cultural biases built into society. These include history being told from the perspective of the powerful, the overrepresentation of dominant voices, and the persistent spread of misinformation.

To return to the example of the holocaust, as of July 28th, when I googled “how many people died in the holocaust?” the AI summary answer opens with “The Holocaust resulted in the deaths of approximately 6 million Jews and millions of other victims persecuted by the Nazis.” This is technically accurate, but it minimizes the scale of death.

In total, it’s estimated that 11 million people were murdered, with Nazi policies seeking the erasure of Jews, yes, and also the erasure of the Roma, homosexuals, persons with disability, and political dissidents. We should not forget the scale of persecution, nor the fact that all of these groups and more were persecuted.

Biased training data impacts our present decisions, too. A recent study found that Claude recommended significantly lower salaries for women than for men with identical resumes². This is not a technical issue; it’s an all-too-accurate reflection of long-standing structural inequities.

Imagine an employer pasting a resume into an AI model and asking for a fair salary. The system, trained on biased data, outputs a lower number for a woman. The employer assumes the result is neutral and objective. Meanwhile, the applicant, aware of industry pay gaps, asks for the salary her male colleagues are earning. Now they are at an impasse. The employer believes she is asking too much. She believes he is underpaying. Both are convinced they are being fair.

Exploring Solutions for Biased Data

The seemingly obvious solution is to remove false or biased content from the training data, but this idea breaks when it meets reality. How do you discern what is false from what is a minority opinion? Where is the line between factual truth and different lived experiences? How can you remove bias when we don’t culturally agree on the direction of bias?

If AI systems are trained only on the majority perspective, they will perpetuate existing power structures and potentially erase marginalized voices. But if they're trained on all available content without curation, they will amplify conspiracy theories and dangerous misinformation alongside legitimate alternative perspectives. The loudest/most repetitious voice in the room is a poor heuristic for truth.

If every platform takes a different approach to defining what is true, then the fight for market share will also be a fight for being the authoritative source of truth. In practice, this is something we are already familiar with: the competition between news organizations, political parties, and different religions today are in part a fight for the authority to define reality.

But in this sea of bad options, the worst possible outcome could be if all AI platforms adopt the same framework for truth. With a uniformity of truth comes oppression. Dissenting voices, even legitimate ones, would be systematically excluded from the collective memory being encoded in our information systems.

AI tools are being integrated into educational tools, search engines, and decision-making systems across society. It has already become inevitable that these tools will be, and arguably already are, the primary filter of information for billions of people around the world. What remains open for influence is how these systems develop moving forward. The frameworks we establish now for what these systems remember and forget will shape collective memory for generations.

So, Who Gets to Decide?

While my toddler continues to learn when to try again and when to learn from the “big owie,” we are collectively experiencing a digital toddlerhood. The systems we build now are like toddlers: absorbing everything, mimicking what they observe, and reinforcing what is repeated most. Like parents, we are responsible for what they see, hear, and encode.

The scale we are facing is unprecedented: these systems can influence billions of people simultaneously. The speed is unprecedented: changes to training data or algorithms can shift global access to information in moments. The concentration of power is unprecedented: it takes far fewer people to change an algorithm than it took to burn an empire of books.

We’ve spent the past decades digitizing our history and lives, hoping for something more comprehensive and durable than our family photo albums and history texts. Now we must face the fragility of digital memory. The choices about what data to include, how to weight different sources, and what outputs to promote are more than technical problems; they are defining what we understand it means to be human. These decisions need the input of more than a team of engineers.

In an age where artificial intelligence increasingly shapes collective memory, who should decide what we remember and what we forget?

1. Forget the stereotypes, toddlers are the patron saints of patience and acceptance. I'm not kidding, I wish I had my son's resilience to frustration.

2. Awaiting peer review, Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models