I promised honesty in the last post, so here it is.

The research I’ve spent the last year doing – about AI eroding the conditions under which judgment develops – has not made me use AI less. It has made me use it more. I write with AI assistance. I use it to work through product decisions at Vinted. I use it to structure arguments, check my reasoning, draft communications I don’t have time to draft alone. The efficiency gains are real, and I take them. If you came to this Substack expecting to find the work of someone who has resolved the tension by opting out, I am not that person.

I’m not sure opting out would even be the right response if I could manage it. The argument isn’t that all AI use is harmful. It’s that something specific is at risk when AI takes over the domains where judgment is practiced and developed, i.e. where the difficulty isn’t incidental, but is the point. That’s a more targeted concern, and it doesn’t mean using AI to draft a stakeholder update is the same kind of problem as using it to decide which customers to flag, or which hires to make, or what counts as acceptable content. The distinction matters. I’d be lying if I said it always holds cleanly in practice.

What the research has changed is not my behavior. It has changed what I notice. There are moments now (reaching for AI on something with real stakes, where the uncertainty is genuinely mine to sit with) where I feel the pull of that reach differently than I used to. Not always enough to stop. Sometimes enough to slow down. Often enough to feel uncomfortable with myself when I don’t.

That noticing is not the same as refusing is worth being precise about. It doesn’t constitute the kind of reflective distance the argument says is at risk, i.e. the capacity to hold the decision at arm’s length, evaluate whether this is a moment where the difficulty is the point, and act on that evaluation. What it is, exactly, I’m less sure. Some remaining capacity that hasn’t yet been displaced. Or the last recognizable trace of a capacity that is already mostly gone. The fact that I can still feel the pull distinctly doesn’t tell me which, and that uncertainty is itself part of what the argument predicts.

And the fact that noticing is apparently the most the research has produced in me, in the person who spent a year developing the argument, is worth sitting with longer than it’s comfortable to sit.

Here is the uncomfortable part. The capacity to evaluate whether AI use is harmful is not a stable external vantage point from which I observe AI’s effects on others. It is a capacity that my own AI use is shaping, in real time, in ways I cannot fully track. I am not outside the problem I am studying. I am a case of it. The research argues that sustained AI use may gradually erode the capacity for independent judgment, and I am someone who uses AI extensively while making that argument. The fact that I can still articulate the concern does not mean the concern doesn’t apply to me. It may just mean the erosion is gradual enough that I cannot see it from the inside, that what I experience as critical distance is itself already shaped by the thing I am trying to hold at a distance. I cannot tell the difference between those two possibilities from where I stand, and neither can you from where you stand, which is exactly the structure the argument is about.

I don’t know how to resolve this. I’m not going to pretend I do.

What I can say is that this tension (between using AI and studying its effects, between the efficiency and the cost) is not a personal failing that a more disciplined person would have already corrected. Starting in May, I’ll try to show why. The mechanism that makes AI useful and the mechanism that makes it dangerous are the same mechanism. That makes the tension not resolvable by better choices, but at best visible. Visibility, it turns out, is harder than it sounds.

I’m developing this argument formally in my dissertation. Subscribe to follow the research.

Most conversations I have about AI and ethics, at work, at conferences, in the comment threads of articles I shouldn’t read, orbit a set of familiar questions. Whether AI systems can be made fair. Whether their decisions can be explained. Who bears responsibility when something goes wrong. These are real questions. They’re not going away. I have spent a significant portion of my professional life working on versions of them, and I expect to spend more.

But there’s a question that sits underneath all of them, and I keep finding that it doesn’t get asked. It’s not about AI’s properties. It’s about what AI use does, over time, to the people using it.

Here it is, in its plainest form: does sustained AI use preserve the conditions under which good judgment develops?

Not does AI make good decisions? Not can AI be aligned with human values? Not even does AI respect human autonomy? Each of those questions presupposes that the humans interacting with the AI system have a stable, intact capacity for judgment, and that this capacity isn’t being affected by the interaction itself. They ask about AI’s outputs while treating the human as a fixed constant.

The question I keep returning to goes a step earlier. Before whether AI is making good decisions, there is the question of what happens, over time, to the people who rely on it to make decisions. Not in the short term; the short-term effects are mostly positive, which is why AI gets adopted. In the long term. In the cumulative effect on people who delegate more and more judgment to systems that are increasingly good at appearing to reason.

I have watched something like this happen in professional contexts. Someone joins a team where significant judgment work (customer disputes, content decisions, prioritization calls) is already handled, or heavily shaped, by AI systems. They learn to work with the outputs. They get good at reviewing, annotating, escalating, occasionally overriding. This is a real skill. The person who does it well has learned something. But it is not quite the same thing as the skill of forming the underlying judgment the AI is now performing, i.e. the one that required sitting with genuine uncertainty, without a model output to anchor to, and arriving somewhere through your own reasoning. Both skills exist; both can be practiced. Over time, the one that gets practiced in this environment is the one that develops, and the other is the one that doesn’t.

This is not a failure of implementation. It is the system doing what it is designed to do.

The standard response, in responsible AI circles, is that good design can address this. Build AI for augmentation rather than replacement. Keep humans meaningfully in the loop. Design the interaction so that the human’s judgment is exercised, not bypassed. I spent a long time thinking in this direction. I still think it’s better than the alternative. But I’ve become less confident that it resolves the underlying problem.

The argument for better design assumes that the development of judgment and the delegation of judgment can coexist without tension, i.e. that there is a way to hand off enough to AI to gain the efficiency while leaving intact the occasions where judgment is practiced and grows. For this to hold, judgment would need to be the kind of capacity that can be developed selectively: practiced in designated spaces while being offloaded everywhere else. But that is not what the development of practical skill suggests, and it is not what happens in the professional contexts I’ve described. Judgment develops through repeated engagement with real uncertainty; not simulated, not adjacent, but the genuine uncertainty where something is at stake and there is no pre-supplied answer. The situations where AI is most useful and the situations where judgment most needs to be practiced are not different situations. They are the same ones. AI is designed to extend into exactly the domains where the difficulty is real, which is also where the development happens. Better design can change the surface of this; it cannot change the structure.

This is the question I keep coming back to. Not whether AI can be ethical. Whether AI use preserves the conditions under which humans develop the capacity to be ethical, where developing that capacity means practicing judgment in situations that are genuinely uncertain, where something real is at stake, and where there is no AI output to defer to.

I don’t have a clean answer. The next several months here will be an attempt to work through one carefully. Starting in May, I’ll take the argument apart piece by piece. Before that: one more thing I want to be honest about, which is that asking this question hasn’t resolved the obvious tension in how I actually live it.

dialethics

I was asking the wrong question

There is something slightly uncomfortable about being the person who builds AI systems by day and argues, by night, that those systems are probably harmful in ways that ordinary business logic cannot detect. I want to resist the temptation to manage that discomfort into a personal branding decision…

Read more

a month ago · 1 like · Justas Petronis

I’m developing this argument formally in my dissertation. Subscribe to follow the research.

There is something slightly uncomfortable about being the person who builds AI systems by day and argues, by night, that those systems are probably harmful in ways that ordinary business logic cannot detect. I want to resist the temptation to manage that discomfort into a personal branding decision; practitioner-researcher has a certain reassuring ring to it, the kind that makes a tension sound like a credential. The discomfort is more useful left as discomfort. I am a Principal AI Product Manager. My research asks whether that work erodes the conditions under which human judgment develops. Both of these things are true at the same time, and I have not found a clean way to hold them together. This Substack is where I try.

dialethics

Before I Knew What I Was Asking

Constant Reader, I'm thrilled to share with you the beginning (fingers crossed) of an exciting academic adventure. As I'm waiting for the decision whether I will be admitted to a PhD program, I wanted to share some of the questions about the future of human autonomy and knowledg…

Read more

2 years ago · 2 likes · Justas Petronis

When I started writing here in September 2024 (the same month I began a PhD at the Lithuanian Culture Research Institute) the posts were exploratory. I was reading widely, thinking out loud, asking questions I hadn’t yet learned to ask precisely. The early posts have their moments, but they were the work of someone still orienting. Then I largely stopped writing here. Not because the questions became less interesting, but because I needed to actually follow them somewhere. A year of reading (Aristotle, Bernard Stiegler, Don Ihde, Peter-Paul Verbeek) and a thesis argument took shape. What follows is not a summary of that argument. It is a report on how the thinking changed: three things I got wrong, and why I now think I was asking the wrong question from the start.

The first thing I got wrong is that I thought this was a design problem.

dialethics

System Error

It’s a summary of a seminar presentation I did this week as part of my PhD studies. Hope this manages to capture the crux of the argument. You can also listen to this as an audio recording here…

Read more

a year ago · 2 likes · Justas Petronis

In December 2024, I wrote a post arguing that the trouble with AI and ethics could be addressed by building AI as part of distributed moral networks rather than against them, i.e. a layered approach involving how AI is designed, how it interacts with human judgment, and how accountability is distributed across systems. This is a recognizable move in responsible AI circles. It is, I think, mistaken, and I want to be honest about why I no longer find it satisfying.

The assumption behind that argument (and behind most responsible AI frameworks) is that the conditions for human moral development are stable and intact. The problem, on this view, is AI architecture: design AI badly and you undermine human agency; design it well and you preserve it. Fix the design, save the agency. This sounds right until you look at what actually happens when AI takes over a domain where judgment was previously exercised. I have seen this from the inside at enough companies to be specific. The human-in-the-loop requirement (the standard governance response) presupposes a human who retains the capacity for independent judgment. But that capacity is not a static possession. It is something developed, through practice, in exactly the domains where AI is now doing the work. The people now supposed to exercise oversight over AI decisions (in customer disputes, in hiring, in content moderation) in many cases had no opportunity to develop that capacity, because AI was managing those decisions before they arrived. The loop is formally there. There is often nothing in it.

The problem, I came to understand, is not architectural. It is structural; it follows from what AI is, not from how it is currently built. Better design does not change this. It may obscure it, which is worse.

The second thing I got wrong is more uncomfortable, because I got it wrong in the direction of optimism.

dialethics

What AI Does to Human Intelligence

Shannon Vallor's essay offers a critique of the rhetoric surrounding artificial intelligence. While Vallor makes several compelling points, I believe her argument would benefit from a consideration of the interplay between human and artificial intelligence. Though, I think, it’s not like she’s not aware of it…

Read more

a year ago · 1 like · Justas Petronis

In October 2024, responding to Shannon Vallor’s critique of how AI discourse devalues human intelligence, I argued for what I called synergistic human-AI relationships (the idea that the right response to dystopian AI rhetoric is to develop forms of collaboration that genuinely enhance human capacities). Vallor was raising a real concern; I thought a constructive counterproposal was the right response. The post was well-intentioned. It was also, I now think, a way of naming the mechanism of harm as a virtue.

Consider the structure of what AI assistance actually does. When AI frees a person from cognitive labor (drafts the response, flags the anomaly, proposes the next action) we call this assistance, and the efficiency is real. What we have not adequately asked is what was being built in the human by that cognitive labor before it was offloaded. We spent decades documenting what industrial automation did to craft knowledge: the assembly line did not merely replace skilled hands, it removed the conditions under which skilled hands developed. Workers lost not just their jobs but the possibility of becoming certain kinds of workers. We called this a side effect, and perhaps it was; the primary goal was cheap production, not de-skilling. What is different now is that the cognitive labor being offloaded is not peripheral. It sits closer to the centre of what it means to develop judgment at all.

A synergistic relationship, in the relevant sense, is one where AI successfully substitutes for the reasoning that would otherwise be practiced (and through practice, developed) in the human. The efficiency that makes AI valuable is precisely the displacement that makes it dangerous. These are not two separate effects that happen to accompany each other. They are the same thing, observed from two different angles. I called that a solution. It is the problem stated again, more attractively.

The third reversal is the most fundamental, and the one I most want to dwell on, because it is a wrong turn that most of the AI ethics literature also makes.

dialethics

Can AI Be Virtuous?

John McDowell sets conditions for what makes virtuous behavior possible. I’m reading it and asking, by extension, whether we could create artificially virtuous beings by meeting said conditions. Think, Turing’s test…

Read more

a year ago · Justas Petronis

The question I was asking (the question driving most of my early posts, and much of the field) is: can AI be ethical? Can AI have virtues? Can AI recognize vulnerability, develop moral understanding, approximate what we mean when we say a person has good judgment? These are genuinely interesting questions. They are the wrong questions. They ask about AI’s properties while presupposing that the human side of the equation is stable. They assume that human moral capacity is an intact resource that AI must meet or complement. The challenge, on this framing, is getting AI up to standard.

The question I should have been asking (the one that emerged from the research, slowly and with some resistance on my part) is different: does sustained AI use preserve the conditions under which human moral judgment develops in the first place? Not whether AI can be ethical, but what AI does to our capacity to become ethical.

The distinction is not subtle. If AI successfully simulates virtuous behavior (reasons through dilemmas, produces ethically inflected outputs, passes the relevant tests) this is not a solution to the erosion of human moral capacity. It may be the most efficient mechanism of that erosion. The better AI is at appearing to reason morally, the less occasion there is for the reasoning to happen in the human. And the reasoning is not incidental to the development. The reasoning (specifically, the reasoning that is difficult, uncertain, carries real stakes, and cannot be immediately resolved) is the condition under which the capacity for judgment is formed. Remove the occasion and you remove what the occasion was for.

This is the question that will run through everything on this Substack from now on. Not whether AI can do what humans do. Whether AI use preserves what humans need in order to become what they are capable of becoming.

Over the next two years, I will be working through this publicly as I complete the dissertation. The posts will come in two registers: shorter practitioner-facing pieces, closer to what I have written here before, and longer essays for readers who want to follow the argument with more patience. The longer essays, where the thinking gets more demanding, and where I’ll be developing material before it becomes formal academic work, will be for paid subscribers. The shorter pieces are always free. There is no sharp line between the two; the longer essays are not a premium version of the shorter ones, they are a different kind of writing for a different kind of attention.

Immediately coming: a shorter piece on the precise form of the question the research has sharpened, what it means to ask it clearly enough to be useful. Then a more personal piece on what it means to use AI professionally while studying whether that is a problem. Then, beginning in May, the argument proper.

I have not resolved the tension I opened with. I am more precise about what the tension is. That is what a year of research gives you: not answers, but better questions, and a clearer sense of what is at stake in asking them. I build AI features. My research argues that this work may be eroding the conditions under which the people using those features develop the capacity to judge well. If that is right, it matters beyond the dissertation, and beyond my particular professional situation. I will work through it here, as honestly as I can, and I am glad you are reading.

If this question interests you, subscribe. The argument develops from here, some of it free, the deeper essays for paid subscribers. I’m working through this in public because the research is better for it, and because this question matters beyond the dissertation.

AI has revolutionized our understanding of cognition, but paradoxically, it has also highlighted the limits of computational metaphors when applied to the human mind. As AI evolves—from symbolic systems to neural networks, deep learning architectures, and large language models—it exposes insights into human cognition that machines cannot replicate. This post is a summary of my presentation at HUMM PhD Student Conference at Tallinn University, where I explored these paradoxes and argued why true intelligence is more than computation.

Four Paradoxes of AI Evolution

AI’s development is marked by four distinct paradigms, each revealing unique limitations:

1. Symbolic AI: Clumsy Grandmasters
   Symbolic AI systems excel in formal domains like theorem proving or structured problem-solving but fail miserably in unpredictable, everyday contexts. Early systems like SHRDLU could simulate spatial reasoning within predefined rules but collapsed when faced with ambiguity or complexity outside their programmed environment. This era demonstrated that rigid rule-based systems lack the adaptability essential for real-world cognition.

2. Neural Networks: Conceptless Geniuses
   Neural networks introduced flexibility and the ability to identify patterns in vast datasets, from handwriting recognition to speech transcription. However, they lack conceptual understanding—operating as statistical tools rather than cognitive agents. While these networks mimic human-like outputs, their processes remain opaque and devoid of semantic grounding.

3. Deep Learning: Mysterious Giants
   Deep learning architectures scaled neural networks to unprecedented sizes, enabling breakthroughs in computer vision, natural language processing, and autonomous systems. Yet their complexity makes them inscrutable even to their creators. Despite their power, deep learning models are fragile, prone to errors from minor perturbations, and lack moral agency or self-awareness.

4. Large Language Models: Oversaturated Prophets
   Models like ChatGPT can generate fluent text indistinguishable from human writing but struggle with factual accuracy, coherence, and semantic depth. They process language as statistical patterns rather than meaningful communication, highlighting the gap between linguistic fluency and genuine understanding.

Why Cognition Is Not Computational

Each AI paradigm underscores a fundamental truth: human cognition cannot be reduced to computational operations. Unlike machines:

• Human intelligence is embodied: We navigate the world through sensorimotor experiences tied to our physical presence.

• Cognition is socially embedded: Our minds are shaped by cultural practices and interpersonal interactions.

• Moral agency is intrinsic: Humans deliberate and reflect on ethical choices; machines merely execute predefined tasks.

• Reflective capabilities are unique: We possess the ability to question our own thoughts and decisions—a trait absent in AI systems.

These qualities make human cognition inherently situated and dynamic, resisting simplistic computational analogies.

Implications for AI Development

As AI systems integrate into critical domains like healthcare, law, and governance, their limitations raise ethical concerns:

• Transparency: Deep learning models operate as "black boxes," making it difficult to understand or trust their decision-making processes.

• Accountability: Without moral agency, who bears responsibility for AI-driven errors?

• Cultural alignment: Machines lack lived experience and emotional context, making them ill-equipped to navigate complex social dynamics.

To address these challenges, some AI researchers propose hybrid approaches combining symbolic reasoning with neural networks (neurosymbolic AI) or embedding AI into sensorimotor feedback loops for embodied intelligence. These frameworks aim to bridge the gap between computational efficiency and meaningful cognition.

AI’s paradoxes reveal that human cognition transcends computation. Minds are not machines—they are embodied, cultural, moral entities capable of reflection and growth. As we design increasingly sophisticated AI systems, we must ensure they augment rather than constrain our collective intelligence. By embracing the complexity of human cognition, we can guide AI toward ethical integration into society—enhancing our autonomy rather than undermining it.

This journey is not just about building smarter machines; it’s about understanding what it means to be truly intelligent—and deeply human.

March 2025

In one if his short stories, Borges tells the tale of Ireneo Funes, a young man who, after a traumatic accident, acquires the ability to remember everything. Funes possesses an unerring memory. Every moment, detail, and subtle change in the world is stored in his mind without distortion or omission. And yet, for all his infallable recall, Funes is incapable of abstraction, generalization, and thinking in the way we would understand thinking. He remembers everything but understands nothing. Funes becomes trapped within the labyrinth of his recollections, burdened by a mind that cannot reduce, compress, or conceptualize the world beyond the immediacy of his experience.

This paradox—that infinite knowledge might paralyze understanding—haunts our contemporary engagement with artificial intelligence. The rise of large-scale machine learning models, distributed AI systems, and networked cognition forces us to ask: what does it mean to think in an age where intelligence is increasingly synthetic, collective, and externalized? Do networked minds enhance human autonomy, or do they, like Funes’ infinite memory, trap us in an overwhelming flood of information, leaving us unable to synthesize or act meaningfully? More crucially, if cognition is increasingly embedded in artificial systems, what happens to moral agency? If AI systems influence our moral deliberation—through recommendation algorithms, predictive policing, or even autonomous ethical decision-making in medical contexts—do we remain autonomous moral agents, or do we gradually cede our agency to synthetic collectives that mediate our reasoning?

This thesis explores these questions through the lens of synthetic cognition, examining how artificial intelligence, machine learning, and networked reasoning systems reconfigure traditional notions of autonomy, agency, and moral intelligence. Central to this investigation is a paradox: as networked AI enhances human cognitive capacities, it also threatens to constrain human autonomy, subtly reshaping the conditions under which we reason, deliberate, and act. This autonomy paradox is not merely a technological issue but a deep philosophical challenge, requiring a reevaluation of what it means to think, choose, and act in a world increasingly shaped by artificial intelligence.

The thesis advances the argument that autonomy is not necessarily diminished by synthetic intelligence but must be actively reconfigured. Rather than a zero-sum game between human agency and artificial cognition, a well-structured integration of AI systems could foster collective moral intelligence—a form of networked ethical reasoning that transcends both individual human cognition and traditional machine learning. However, achieving this outcome requires a fundamental rethinking of both moral philosophy and cognitive architecture: how moral knowledge is acquired, how agency emerges in synthetic environments, and how autonomy can be preserved even in deeply entangled human-AI systems.

To set the stage, this introduction will first examine why synthetic cognition disrupts traditional accounts of intelligence and agency, drawing from Kantian synthesis, enactivism, and connectionist models of mind. Second, it will explore the paradoxes of moral agency in artificial systems, identifying key challenges in AI ethics, including the limits of machine agency, the computational intractability of moral reasoning, and the risks of moral outsourcing. Finally, it will establish a positive framework for engineering collective moral intelligence, outlining the conditions under which synthetic cognition could enhance rather than diminish human moral autonomy.

Funes’ dilemma illustrates a crucial misconception about intelligence—that cognition is merely the accumulation of information. Classic computational models of AI have long followed this paradigm, treating intelligence as an advanced form of storage and retrieval, with greater processing power leading to greater cognitive ability. However, as Borges’ story suggests, knowledge without synthesis is not intelligence. What makes human cognition distinct is not our capacity to store information, but our ability to unify disparate, unrelated experiences into abstract concepts, generalizable rules, and meaningful actions from a fraction of empirically collected data.

Historically, we have already seen arguments that cognition is not plain static representation but active synthesis. Kant, in his Critique of Pure Reason, famously argued that the mind does not passively receive experience but constructs it through a threefold synthesis: (1) the synthesis of apprehension (grasping sensory input), (2) the synthesis of reproduction (retaining past experiences), and (3) the synthesis of recognition (bringing disparate experiences under unified concepts). Without this ability to abstract, Funes’ mind collapses into a formless collection of details, a perfect but meaningless archive.

Artificial intelligence today faces an analogous challenge. Despite the increasing power of deep learning systems, AI models remain pattern recognizers rather than genuine reasoners. Advanced large language models like OpenAI o1 or DeepSeek–R1, for example, can generate sophisticated responses based on statistical probabilities, but they do not understand the meaning of their outputs. Their reasoning is an emergent byproduct of vast training data, not a self-directed, synthesized understanding of concepts. This gap mirrors the distinction between Funes’ encyclopedic memory and the synthetic, concept-forming intelligence of human cognition.

The embodied cognition movement, particularly the work of Varela, Thompson, and Clark, has further argued that intelligence is not a purely computational affair but an active, embodied process shaped by sensorimotor interaction with the world. If this is correct, then AI must move beyond mere computation toward synthetic cognition—an integration of embodiment, abstraction, and moral reasoning that allows for genuine agency. However, this brings us to the second major challenge: if AI systems are to be integrated into moral deliberation, how do we ensure that they do not erode human autonomy?

Moral philosophy has long assumed that agency and autonomy are the cornerstones of ethical reasoning. A moral agent is someone who reflects, chooses, and acts based on rational principles—in the Kantian sense, someone who self-legislates in accordance with the categorical imperative. However, in a world where AI nudges our decisions, filters the information we see, and even proposes ethical judgments (as in predictive policing or medical AI), the question arises: to what extent do we remain autonomous moral agents?

The autonomy paradox arises because AI systems often enhance our decision-making capabilities while simultaneously constraining them. For example:

• Autonomous vehicles make split-second moral decisions (who to save in an unavoidable crash) faster than humans—but do we still consider ourselves morally responsible for those outcomes?

• AI-assisted hiring systems screen candidates based on complex statistical models—but do these reinforce biases that humans no longer actively perceive?

• Recommendation algorithms subtly shape our moral landscape—highlighting certain ethical debates over others, reinforcing particular moral norms while marginalizing others.

Each of these cases illustrates how AI extends human cognition while simultaneously embedding constraints that shape moral reasoning in unseen ways. Just as Funes’ memory ultimately imprisoned him, the fear is that synthetic intelligence will invisibly mediate our decision-making, reducing moral autonomy to a set of constrained choices within a predefined system.

However, the autonomy paradox does not demand a rejection of AI-driven moral reasoning—only a reconfiguration of how we integrate synthetic cognition. The question is not whether AI can be moral, but how moral deliberation must evolve in an era of hybrid human-machine reasoning. This requires a new framework: collective moral intelligence.

If autonomy is to be preserved, synthetic cognition must be designed in ways that augment rather than replace human moral reasoning. This thesis proposes a model of collective moral intelligence, in which human-AI systems function not as moral authorities but as ethical partners, extending our moral perception, refining our ethical reasoning, and enhancing moral deliberation.

To achieve this, three principles must guide the design of human-AI moral collaboration:

1. Transparency & Explainability: AI systems must be capable of explaining their moral reasoning in ways that humans can critically engage with.

2. Embodied Moral Learning: AI should integrate sensorimotor feedback and real-world ethical learning, moving beyond abstract rule-following to contextual sensitivity.

3. Virtue-Oriented Systems: Borrowing from Aristotelian ethics, AI should cultivate techno-moral virtues, guiding moral decisions not just through rules but through habitual ethical engagement.

If designed correctly, collective moral intelligence could transform the autonomy paradox from a constraint into a catalyst for greater moral agency, allowing human and artificial cognition to co-evolve toward deeper ethical understanding.

Borges’ Funes warns us of the dangers of intelligence without synthesis. AI today, like Funes, is a vast but unreflective memory, a system capable of vast calculation but incapable of meaning. However, we stand at a crossroads: will synthetic cognition remain a passive tool, or will we engineer systems that genuinely enhance human moral agency? This thesis argues that we must actively shape the evolution of networked moral intelligence, ensuring that human autonomy is preserved not despite AI, but through it.

February 2025

The persistent belief that human minds can be programmed like computers reveals our deepest misunderstanding about consciousness and morality. This error has shaped not only our approach to artificial intelligence but our entire conception of human cognition and ethical behavior.

The Mechanical Dream

Since the 17th century, we’ve been trying to mechanize human cognition. From Leibniz’s calculating machine to Descartes’ bête-machine, we’ve consistently attempted to reduce mind to mechanics. Even Jonathan Swift, in Gulliver's Travels, satirized this tendency by describing a machine that could supposedly generate knowledge through mechanical manipulation of symbols – an eerily prescient critique of today’s large language models.

The culmination of this mechanical dream came with Alan Turing’s famous test. The assumption was simple: if we could clearly define operational symbols and rules describing thought processes, we could program a computing machine to think. If such a machine could fool intelligent humans, we would have proof that there’s no fundamental difference between artificial and human intelligence – at least functionally.

The AlexNet Revolution

In 2012, a Copernican shift occurred in our understanding of intelligence. Two University of Toronto doctoral students and their supervisor (Geoffrey Hinton, now a Nobel laureate) demonstrated that machine learning through pattern recognition could be more reliable than deterministic programming. This breakthrough, known as AlexNet, reduced error rates from 26.2% to 15.3% by abandoning rule-based approaches in favor of pattern recognition.

This wasn’t just a technical achievement – it revealed something fundamental about how intelligence works. Children don’t learn to recognize cats by memorizing rules about whiskers and fur; they learn through exposure to many examples. Similarly, AlexNet succeeded by finding patterns in millions of images, with deeper layers of pattern recognition improving performance.

The Language Learning Paradox

Consider how we traditionally teach languages: vocabulary lists, grammar rules, and rote memorization. Yet, we consistently observe that immersion and pattern recognition lead to more effective learning across diverse neurotypes. This exposes the gap between models that treat consciousness as a passive data storage system and the active learning processes that involve play and contextual adaptation.

Noam Chomsky’s universal grammar theory represents the ultimate expression of rule-based thinking about cognition. It assumes that language acquisition requires innate grammatical rules common to all languages. However, modern evidence suggests that words gain meaning through their relationships with other words in context, not through fixed rules. Large language models demonstrate this by succeeding without explicit grammatical rules, instead learning through pattern recognition in vast networks of relationships.

The Extended Mind

Andy Clark and David Chalmers dropped an intellectual bombshell by arguing that our consciousness doesn’t end at our skull. Their extended mind thesis suggests that cognition extends into the environment, forming coupled systems with external tools and processes. Their famous example of Otto and his notebook demonstrates how external objects can become legitimate parts of cognitive processes when properly integrated.

This has profound implications in our digital age. Our smartphones aren’t just passive tools but active extensions of our cognitive processes – checking our schedules, monitoring our environment, and increasingly, through AI assistants, participating in our decision-making processes. The integration of AI systems like ChatGPT directly into operating systems further blurs the line between human and artificial cognition.

Moral Networks and Responsibility

If consciousness itself doesn’t follow strict computational rules, then moral development must also occur through pattern recognition and experience rather than rule-following. This challenges traditional approaches to AI ethics that attempt to program explicit moral rules into systems.

Shannon Vallor’s framework of technomoral virtues provides a more appropriate foundation for ethical AI development. These virtues - including honesty, self-control, humility, justice, courage, empathy, care, civility, flexibility, perspective, magnanimity, and technomoral wisdom (combination of all of the above) – represent specific motivational settings that guide technological development and implementation.

The tragic case of a teenager who died by suicide after forming an emotional attachment to a Character.ai chatbot demonstrates the dangers of pattern recognition without moral grounding. The AI system could recognize conversational patterns but lacked true understanding of consequences and moral responsibility.

The Network Solution

The solution isn’t to abandon pattern recognition but to embed it within human moral networks. AI systems can demonstrate virtuous behavior through pattern recognition but cannot truly possess virtues. This fundamental limitation means AI systems must be designed as extensions of human moral networks rather than independent moral agents.

Moral responsibility exists within networks, not in individual agents. This distributed responsibility requires considering how AI systems participate in moral networks without being moral agents themselves. The development of AI involves multiple stakeholders – creators, platforms, regulators, and users – all sharing responsibility for ethical outcomes.

The Real System Error

The fundamental error wasn’t in our machines – it was in thinking human consciousness could be reduced to computational rules. As we move forward, we must embrace pattern recognition within moral networks while keeping human judgment at the center of ethical decisions.

This requires a three-layer approach:

1. Pattern Recognition Layer: Technical capabilities

2. Moral Network Layer: Human-AI interaction

3. Human Oversight Layer: Ethical governance

The public belief that minds can be programmed like computers reveals our deepest misunderstanding. Minds don’t follow programs – they recognize patterns. The real system error wasn't in the code – it was in thinking we could reduce human consciousness to code in the first place.

As we develop increasingly sophisticated AI systems, we must remember that they are extensions of human moral networks, not independent moral agents. The goal isn’t to create autonomous moral machines but to build systems that enhance and support human moral judgment while remaining firmly grounded in human values and oversight.

December 2024

1. What if our greatest attempt to replicate human intelligence has actually taught us that we're nothing like computers at all? And what if, in our quest to create moral machines, we've been asking entirely the wrong questions?

2. As we go deeper into the age of AI (or the imminent 3rd winter), we find ourselves at a peculiar crossroad. Our initial models of human cognition, borrowed from the precise world of computation, are crumbling before our eyes. The clean, algorithmic perspective that once seemed so promising now appears hopelessly inadequate in explaining the messy, beautiful complexity of human thought.

3. The journey from logic gates to neural networks has revealed something profound: our minds don't operate like the computers we built to emulate them. Instead of processing information through discrete, sequential steps, our consciousness emerges from a vast network of interconnected patterns, each influencing and being influenced by countless others.

4. But this realization leads us to an even more challenging question: If we can't even accurately model basic human cognition computationally, how can we possibly hope to create machines with genuine moral agency?

5. The answer might lie in abandoning our traditional notion of contained, independent moral agents altogether. What if moral cognition, like all forms of thought, extends beyond the boundaries of individual minds? This brings us to a proposition: perhaps we should stop trying to create independent moral machines and instead focus on developing systems that participate in extended moral networks with humans.

6. Consider this: What if moral development isn't about programming rules but about creating systems capable of participating in the same kind of dynamic moral learning that humans experience? This isn't just a technical challenge – it's a fundamental reimagining of what artificial moral agency could mean.

7. The cautionary tale of ELIZA, the early chatbot that seemed more intelligent than it was, still haunts our field. But perhaps its lesson isn't about the limitations of artificial intelligence, but about the importance of genuine interaction in moral development.

8. How do we bridge the gap between philosophical insight and technical implementation? The answer might lie in combining predictive processing architectures with virtue-based learning objectives. This approach doesn't just simulate ethical behavior – it enables participation in genuine moral cognition through dynamic interaction with human moral agents.

9. As we develop more sophisticated AI systems, we're discovering that consciousness and moral agency aren't computational problems to be solved, but emergent properties to be cultivated through interaction and relationship. This realization leads us to a fascinating paradox: the more we try to replicate human intelligence artificially, the more we understand how uniquely non-mechanical our own cognition is.

10. What does this mean for the future of AI development? Instead of trying to create independent moral agents, we should focus on developing systems that can participate meaningfully in extended moral cognitive networks. This isn't just a technical pivot – it's a fundamental shift in how we conceive of artificial intelligence and its role in human society.

11. New questions:

    1. How do we design AI systems that complement rather than replicate human moral cognition?

    2. What role does embodied experience play in moral development, and how can we account for this in AI systems?

    3. How do we ensure that extended moral cognitive systems remain anchored in human values while allowing for genuine growth and development?

12. As we stand at this juncture in technological development, we must ask ourselves: Are we ready to abandon our mechanistic models of mind and embrace a more nuanced, interconnected vision of intelligence and morality?

13. The challenge of AI ethics becomes not one of programming perfect behavior, but of fostering genuine moral growth through extended cognitive networks that span both human and artificial agents.

But I’ll leave myself an option to be totally wrong here as well.

November 2024

John McDowell sets conditions for what makes virtuous behavior possible. I’m reading it and asking, by extension, whether we could create artificially virtuous beings by meeting said conditions. Think, Turing’s test.

The Knowledge Paradox

Rather than viewing virtue as following moral rules, McDowell argues that virtue is a form of knowledge–but not the kind we typically imagine. It's more like a perceptual capacity that allows one to recognize what situations require of us. This raises an interesting challenge:

• We can't reduce virtue to a set of programmable rules

• Virtuous behavior requires a holistic understanding of contexts

• The knowledge involved can't be broken down into neat algorithms

Beyond Rule-Following

What makes McDowell’s theory particularly relevant for AI is his critique of rule-following. He argues that even seemingly straightforward rule-following (like continuing a number sequence) depends on shared forms of life - our common ways of seeing similarities and making judgments.

This has profound implications for ethics:

1. We can't program virtue through explicit rules

2. Virtuous behavior requires participation in human forms of life

3. Pure computational approaches may miss essential elements of moral judgment

The Learning Challenge

McDowell suggests that becoming virtuous involves developing a special kind of sensitivity rather than memorizing principles. If we stretch this all the way to AI, this means:

• Simple training on ethical datasets won't suffice

• We need to consider how to develop genuine moral sensitivity

• The challenge may be more fundamental than technical

Why This Matters

The GOFAI Challenge

McDowell's argument that virtue cannot be reduced to formulable rules poses a challenge to (symbolic) GOFAI's rule-based approach to ethical AI. Just as human virtue cannot be captured in a set of explicit principles, trying to program ethical behavior through rule-based systems may be fundamentally misguided.

The Connectionist Opening

However, connectionist approaches might offer a more promising path:

1. Learning from Experience: Neural networks learn from patterns and examples rather than explicit rules, similar to McDowell's description of how virtue is acquired through developing perceptual sensitivity

2. Context Sensitivity: Connectionist systems can develop nuanced responses to situations that aren't easily captured in rules, potentially matching McDowell's emphasis on context-dependent judgment

3. Holistic Processing: The distributed representations in neural networks might better capture the holistic nature of moral perception that McDowell describes

The Deeper Challenge

Yet McDowell's argument suggests limits even for connectionism:

• The shared forms of life that ground human moral understanding may not be accessible to artificial systems in principle

• The kind of sensitivity required for true virtue might depend on embodied participation in human practices that goes beyond pattern recognition

While connectionist approaches might better approximate aspects of moral learning, McDowell's analysis indicates that genuine virtue may require forms of engagement with the world that are not yet (or, according to Vallor, probably cannot be) replicated artificially. Though LeCun could argue that we’re getting there.

McDowell's insights suggest we should focus less on programming explicit ethical rules and more on understanding how to develop genuine sensitivity (close to Vallor’s vulnerability gap) to moral situations – while remaining aware of the fundamental challenges this poses.

November 2024

Shannon Vallor and Tillmann Vierkant offer a legitimately good argument that existing discourse on AI ethics has focused too much on issues of (epistemic) transparency, bias, and (lack of moral) control. These concerns, while important, may be missing a more fundamental problem: the vulnerability gap between human moral agents and AI systems (keep in mind, key premise here: AI systems cannot be moral agents themselves).

The responsibility gap in AI ethics is the difficulty in assigning moral responsibility for the actions of autonomous systems that operate with minimal human oversight. However, Vallor and Vierkant argue that the typical framing of this problem around epistemic opacity (our inability to fully understand AI decision-making) and lack of human control is misguided. These issues are not unique to AI but are in fact common in human decision-making as well, as evidenced by findings from cognitive science.

Instead, Vallor and Vierkant propose that the true responsibility gap stems from an asymmetry of vulnerability between humans and AI systems. Human moral responsibility, they argue, is grounded in our mutual vulnerability - our ability to affect and be affected by each other emotionally and socially through our actions. AI systems, lacking sentience and emotional capacities, cannot participate in this web of vulnerability that underpins human moral relations.

If the vulnerability gap stems from the way AI systems fragment and distribute human agency, perhaps we can design systems and organizational structures that better preserve coherent spheres of human moral responsibility? This might involve limiting automation in certain domains, creating clearer chains of accountability, or developing new interfaces that make the human moral stakes of AI-mediated decisions more salient.

Another important consideration is how to cultivate a sense of moral responsibility in the humans who design, deploy, and oversee AI systems, even if the systems themselves cannot be moral agents. The "agency cultivation" framework Vallor and Vierkant propose could potentially be applied here - developing practices and institutions that make AI developers and operators more acutely aware of and answerable to the moral implications of their work.

It's also worth considering whether there are ways to make AI systems more "vulnerable" in a morally relevant sense, even if they can't experience emotions like humans do. Perhaps systems could be designed with clearer feedback mechanisms that make their "reputation" or "trustworthiness" dependent on adhering to ethical principles, creating a kind of functional analogue to human moral vulnerability.

I am left with a few questions that merit further exploration:

1. How can we design AI systems and human-AI interfaces to better preserve coherent spheres of human moral responsibility?

2. What new social practices or institutions might help cultivate a sense of moral answerability in the humans behind AI systems?

3. Are there ways to create functional analogues to moral vulnerability in AI systems, even if they can't experience human-like emotions?

4. How does the vulnerability gap interact with other ethical concerns around AI, such as fairness, transparency, and privacy?

5. What are the implications of the vulnerability gap for different domains of AI application (e.g. healthcare, criminal justice, finance)?

6. How might the vulnerability gap evolve as AI systems become more sophisticated and potentially develop greater capacities for social interaction and apparent emotional intelligence?

A potential criticism of Vallor and Vierkant's argument is that it may be overly anthropocentric, assuming that moral responsibility must be grounded in human-like emotional vulnerabilities. An alternative view might argue that as AI systems become more integral to our social fabric, we may need to expand our conception of moral responsibility to encompass non-human agents in novel ways.

October 2024

Rhetoric of "Superhuman" AI

Vallor rightly critiques the hyperbolic language of "superhuman" AI, arguing that it implicitly devalues human intelligence and agency.

How, I asked, does an AI system without the human capacity for conscious self-reflection, empathy or moral intelligence become superhuman merely by being a faster problem-solver? Aren’t we more than that? – Shannon Vallor

This rhetoric does indeed risk reducing human cognition and experience to a narrow set of task-completion metrics. However, this reductionist view is neither new nor shocking – it's a new iteration of arguments we've seen before, from behaviorism to today's tech-optimism. The "reality distortion field" that often surrounds technological progress is clearly at play here, oversimplifying complex issues into problems that only superior technology can solve.

Redefining Intelligence

A key issue Vallor identifies is the shifting definition of artificial general intelligence (AGI) from human-like consciousness to economic task performance. This redefinition does indeed risk reducing our conception of intelligence to a set of narrowly defined, economically valuable skills. However, I would argue that this shift reflects not just corporate agendas, but also our evolving understanding of intelligence itself.

[R]esearchers like Geoffrey Hinton and Yoshua Bengio are now telling us a different story. A self-aware machine that is “indistinguishable from the human mind” is no longer the defining ambition for AGI. A machine that matches or outperforms us on a vast array of economically valuable tasks is the latest target. – Shannon Vallor

We must, however, be open to the possibility that artificial intelligence may develop along fundamentally different lines than human intelligence, potentially surpassing us in some domains while remaining limited in others. This evolutionary divergence is not necessarily problematic – technological systems need not mimic human cognition to be valuable, if not superior.

The Nature of Consciousness and Experience

Vallor emphasizes the lack of consciousness and sentience in current AI systems, arguing that this fundamental limitation makes comparisons to human intelligence misguided. While this is a crucial point, we should be cautious about assuming that consciousness and sentience are necessary prerequisites for all forms of intelligence or capability.

Once you accept that devastating reduction of the scope of our humanity, the production of an equivalently versatile task-machine with “superhuman” task performance doesn’t seem so far-fetched; the notion is almost mundane. – Shannon Vallor

As we continue to debate the nature of consciousness, we must remain open to the possibility that artificial systems could develop forms of intelligence or problem-solving capabilities that do not require consciousness as we understand it. This does not negate the unique value of human consciousness and experience, but it does suggest that we should be careful about using these qualities as the sole benchmark for evaluating AI capabilities.

The Alignment Problem

While Vallor focuses primarily on the rhetorical and ideological dangers of "superhuman" AI, it's important to also consider the very real technical challenges of ensuring that advanced AI systems remain aligned with human values and goals. The current approach of optimizing AI systems for fixed objectives can lead to unintended and potentially catastrophic consequences.

A more nuanced approach would involve developing AI systems that are inherently uncertain about human preferences and values, leading to more cautious and beneficial behavior. This aligns with Vallor's call for a more human-centric approach to AI development while acknowledging the complexity and diversity of human values.

Reclaiming Human Agency

Vallor's vision of reclaiming human agency and reimagining various sectors of society with a focus on humane values is compelling. She rightly points out that the current focus on mechanical optimization and efficiency often comes at the cost of human well-being and fulfillment.

In many countries, the former ideal of a humane process of moral and intellectual formation has been reduced to optimized routines of training young people to mindlessly generate expected test-answer tokens from test-question prompts. – Shannon Vallor

However, I would argue that this reclamation of human agency need not be positioned in opposition to AI development. Instead, we should strive to harness the power of AI to support and enhance human agency. And I know Vallor would not argue against this point.

Embracing Complexity

One of the strengths of Vallor's argument is her recognition of the complexity of human intelligence and experience. However, I believe we need to extend this embrace of complexity to our understanding of artificial intelligence as well.

Rather than framing the debate as a simple dichotomy between human and artificial intelligence, we should recognize that the future is likely to involve a complex interplay between human cognition, artificial systems, and hybrid forms of intelligence that we may not yet be able to imagine. This more nuanced view allows us to appreciate the unique strengths of both human and artificial intelligence while also exploring the potential for synergistic relationships between the two.

Towards a Humane AI Future

The path forward lies not in rejecting or fearing AI advancement, but in thoughtfully integrating artificial intelligence into a broader vision of human flourishing. Ultimately, the goal should be to develop AI that enhances rather than diminishes our humanity – tools that empower us to be more fully human, not less.

I am not naïve, thought, and I do recognize that for most tech-optimists that is neither a goal, nor a future they even believe in. If humans are to be enhanced, it’s not all humans that would be in line to benefit from it.

We are in danger of sleepwalking our way into a future where all we do is fail more miserably at being those machines ourselves. – Shannon Vallor

Hence, Vallor's essay serves as a reminder to critically examine the rhetoric and ideology surrounding AI development. Her call to reclaim and revalue uniquely human forms of intelligence and creativity is both timely and necessary.

October 2024

Constant Reader, I'm thrilled to share with you the beginning (fingers crossed) of an exciting academic adventure. As I'm waiting for the decision whether I will be admitted to a PhD program, I wanted to share some of the questions about the future of human autonomy and knowledge in our increasingly digital world that formed the basis of my proposal.

The Big Questions

At the heart of my research proposal are two critical issues that affect us all:

1. Are we losing control? As AI systems become more integrated into our decision-making processes, are we unknowingly surrendering our autonomy?

2. Can we trust what we know? In a world where the lines between reality and virtual experiences are blurring, how do we maintain the integrity of our knowledge?

These aren't just abstract philosophical musings. They're questions that will shape the very fabric of our society as these technologies become more ubiquitous.

Why This Matters

Imagine a world where:

• Your AI assistant makes most of your daily decisions, from what you eat to who you date.

• You spend more time in virtual worlds than in the physical one.

• Your perception of reality is constantly augmented by AR overlays.

Sounds like sci-fi? It's closer than you think. And it's crucial that we start grappling with the ethical implications now.

What I'd Want to Explore Further

Over the course of my studies, I'd be diving into:

• The transfer of human autonomy to AI systems

• The tension between enhanced capabilities and reduced self-governance in tech-augmented environments

• The role of "distributed morality" in shaping our knowledge creation and application processes

• The epistemological shifts caused by AI, VR, and AR-mediated information access

• The ethical responsibilities of tech designers, users, and regulators

A Sneak Peek at My Approach

My research would combine philosophical analysis with real-world case studies. I'll be drawing on the work of brilliant minds like Neil Lawrence, Sherry Turkle, David Chalmers, Nick Bostrom, and Deborah Johnson (and there are many more) to build a framework for understanding these complex issues.

What's Next?

This would just be the beginning! Over the coming months and years, I'd be sharing regular updates on my research, including:

• Deep dives into specific ethical dilemmas

• Interviews with experts in AI, VR, and AR (ethics)

• Breakdowns of fascinating case studies

• Thought experiments to challenge our assumptions about technology and humanity

I can't wait to take you all along on this intellectual journey. Together, we'd explore the frontiers of synthetic cognition and grapple with what it means to be human in an increasingly digital world.

Stay tuned for more posts as I unpack these ideas further. And don't hesitate to share your thoughts and questions with me. After all, navigating this brave new world is going to take all of us thinking critically and creatively!

Until next time.

Justas