The Pattern Trap: Why Our AI Systems Are Destined to Repeat Healthcare’s Mistakes
Joshua Tamayo-Sarver, MD, PhD, FACEP, FAMIA
Large language models possess extraordinary pattern recognition capabilities, yet this power comes with a fundamental limitation: they’re designed not to learn from history, but to repeat it with perfect accuracy. As AI continues to transform healthcare delivery, we face a critical inflection point in how we train these systems and how we implement them to ensure they don’t hardcode our past failures into our future.
The Prediction Paradox
The fundamental architecture of large language models creates what I’ve come to call the “prediction paradox.” These systems achieve success by correctly predicting what happens next based on historical patterns. It’s as if they’re reading the story of healthcare’s past, predicting the next chapter with remarkable accuracy — regardless of whether that next chapter represents the future we actually want to create.
This mathematical reality stands in stark contrast to how humans interact with history. When we study medical history, we don’t just memorize what happened, we evaluate outcomes, identify mistakes, and actively work to create better futures. Recent research in AI ethics confirms this fundamental difference between human and machine learning, noting that AI systems lack the normative reasoning that allows humans to distinguish between descriptive patterns (what did happen) and prescriptive patterns (what should happen).
This is not just a theoretical problem: using the remarkably poor-quality clinical documents produced by our modern EMRs to teach AI how to produce clinical documents results in a document that no physician would ever want to see. The machine is replicating our failure, but faster and at scale.
Encoded Biases, Amplified Outcomes
The implications of this pattern-replication approach extend far beyond stylistic quirks in documentation. When applied to healthcare decision-making, these systems risk amplifying historical disparities in care delivery that reflect decades of structural inequality.
This observation cuts to the heart of the challenge. The very essence of innovation in healthcare is to create new and better approaches than what came before. Yet our most advanced AI tools are designed with a fundamental orientation toward reproducing what already exists rather than imagining what could be.
The Shadow of History
The challenge becomes even more complex when we consider the makeup of training data itself. A comprehensive analysis of medical AI training sets found that 86% of healthcare-specific training data comes from just three countries, with significant underrepresentation of populations from the Global South and rural healthcare settings.
This creates what researchers have termed “data shadows” — areas where certain populations or clinical scenarios are underrepresented or entirely absent from the data used to train these systems. When deployed in these underrepresented contexts, AI systems don’t just perform poorly — they actively impose patterns learned elsewhere, potentially disrupting locally appropriate care practices.
Reimagining AI’s Learning Path
If we accept that large language models are fundamentally designed to replicate history, how do we ensure they help create the healthcare future we want rather than perpetuating the patterns we’re trying to change? The answer requires both technical innovation and a philosophical shift in how we approach AI development.
First, we must be intentional about curating training data that reflects our aspirational future rather than just our flawed past. This means developing synthetic datasets that represent ideal care patterns and outcomes while maintaining clinical realism. It also means ensuring that the historical data we do use is appropriately contextualized and balanced to mitigate historical biases.
Recent advances in reinforcement learning from human feedback (RLHF) show promise in this direction, allowing models to be fine-tuned based on human evaluations that incorporate ethical and clinical judgments beyond simple pattern recognition. However, the effectiveness of these approaches depends entirely on whose feedback is used for reinforcement, making diversity in clinical AI development teams not just a nice-to-have but an essential safeguard.
Guardrails for the Future
Beyond curating better training data, we need robust mechanisms to prevent AI systems from perpetuating harmful historical patterns, even when those patterns appear in their training. This requires that first and foremost, we approach each problem with an understanding of the problem and select a solution that will make it better, not worse. LLMs are incredibly powerful because they are so very easy to use for a proof of concept from a software engineering perspective. But they are often the wrong tool for the job if we are trying to shape what we want to be instead of following the patterns of what has been.
Moving Beyond the Mirror
Instead of measuring success solely by how accurately an AI system predicts historical outcomes, we need evaluation frameworks that assess alignment with evidence-based best practices and equity goals. This means developing robust metrics for fairness, accessibility, and clinical appropriateness that carry as much weight in model evaluation as traditional accuracy measures.
I’ve found that asking one simple question can cut through much of the complexity: “If this AI system achieved perfect pattern recognition, would it help create the healthcare system we want to build?” If the answer is no, no amount of technical refinement will solve the fundamental problem.
Dr. Joshua Tamayo-Sarver, MD, PhD, FACEP, FAMIA, develops and deploys technology solutions in the healthcare ecosystem as a clinician, business leader, software engineer, statistician, and social justice researcher. As the Vice President of Innovation at Inflect Health and Vituity, his unique formula of skills has helped develop over 35 solutions and scale multiple new healthcare products, including the first AI occult sepsis tool with FDA breakthrough designation. Dr. Tamayo-Sarver oversees corporate venture, internal incubation, and advisory services for AI-driven healthcare solutions, blending consumerism and clinical quality to fit the delicate balance of patient desire, user experience and quality medical care. A Harvard graduate, he holds degrees in biochemistry, epidemiology, and biostatistics, as well as a medical degree from Case Western Reserve University. He is a Mentor in the Emergence Program at Stanford University.
