The manager reads through the file. They look at the writing samples, the screener’s notes, the candidate’s prior roles. Hiring manager has made dozens of hiring calls before, and some they were proud of, some they weren’t. After a few minutes, they make a decision: decline, move to the next candidate.

Here is what I want to ask about that moment. What kind of act was it?

The manager would say they reviewed the evidence and made a hiring call. And in one sense, that’s correct because they looked at the file, they applied their experience, they decided. But if looked at this case more carefully, if we let the structure of what happened reveal itself, we will see that the question they were actually answering was not is this person right for the role? It was does the AI’s assessment seem right here? Those are, obviously, not the same question. The first requires you to reason from the evidence to a conclusion. The second one – requires you to evaluate a conclusion that arrived before you did. The manager was performing an act of ratification, not a genuine act of judgment. The difference, I argue, is real, even if the outcome is identical.

This is the thing I want to make precise in this post. I’ve been calling it the judgment gap.

The simplest way into this scenario is through what makes a calculator different from the kind of AI system I’m describing.

When you use a calculator, you get a number and that number is data. You still have to decide what to do with it, what it means in context, how it bears on the decision you’re making, whether it changes anything. The calculator does the arithmetic while you do the rest. This is true of most tools we use: a thermometer gives you a temperature, a scale gives you a weight, a search engine gives you results. The output is raw material that you take upon yourself to reason through toward a conclusion that is genuinely yours. Even if not unique or too original. The tool provides a premise, you are the one to provide the judgment.

AI outputs in judgment domains don’t work this way. The hiring system doesn’t give the manager a tabulation of competency scores to reason about, rather it gives them not a strong match. The clinical decision support system doesn’t give the physician a blood pressure reading, rather it gives them consider aggressive treatment given patient profile. The legal AI doesn’t return the relevant statutes, rather it returns the strongest argument available is X. The content moderation tool doesn’t return a list of policy clauses the post might engage, rather it returns remove: violates policy. You get the picture. In each case, the output already incorporates what deliberation would have produced. It arrives formatted as a conclusion, carrying the implicit authority of a system that processed more signals than any human could review. The output is not a premise. It is the product of reasoning, packaged for uptake.

The distinction I’m drawing is between an output that enters your thinking as something to reason with and one that enters your thinking as the output your reasoning would have reached. Prior tools, almost without exception, produced the first kind of output. AI, in judgment domains, produces the second.

Of course, the obvious response from my learned friends is this: humans can override. The manager, doctor, lawyer, support specialist can disagree with the recommendation, push back, advance their case against the score. The AI is not making the final call after all, a person is. And this is observably true. The formal architecture indeed preserves human decision-making authority.

But the question I am asking isn’t whether override is technically available. It’s what kind of cognitive act is being performed when the manager, lawyer, or doctor looks at the file. If the conclusion arrived before they did, their task is to evaluate the conclusion rather than form one. These look the same from outside, and in both cases, a human reviews information and makes a call. The structure, however, is different: one involves reasoning from evidence to judgment; the other involves assessing a judgment that was already reached. The first is the act we mean when we say a person decided. The second is something else, call it review, endorsement, auditing, but it’s not quite decision-making in its most genuine sense, even though it produces an output that looks exactly like one.

The override possibility doesn’t close the gap. If anything, it makes the gap harder to see, because the formal architecture of human authority is preserved while the substance of the reasoning act changes underneath it.

Now, here’s the part that matters for the longer argument.

A radiologist who has spent five years reviewing AI analyses of imaging studies develops something real, even if intangible. They learn when the AI is right and when it’s off, what kinds of cases it tends to miss, how to read the gap between the AI output and the clinical picture. That is genuine expertise, and it is valuable. It is not the same expertise as five years of reading radiology films directly. Both practitioners are competent. However, the character of their competence is different.

The one who read radiology films directly for five years developed the capacity to form independent judgments: to look at an image, without prior framing, and arrive at a clinical conclusion through their own perceptual and interpretive work. The one who reviewed AI analyses for five years developed the capacity to evaluate judgments already formed: to recognize when the AI was confident and wrong, when the exception should override the pattern, when to escalate. The second capacity is useful but only while the first one remains available as a check on the AI. It is less useful if the first capacity was never developed in the first place.

The skill that develops from five years of reviewing AI conclusions is not the same skill as five years of forming the underlying judgments. We are making a large, quiet substitution between them, and we are already doing it across radiology, law, hiring, moderation, clinical medicine, financial decision-making, and dozens of other judgment-intensive domains without clearly tracking which skill we’re building and which we’re declining to build. The professional identity says I make the decisions. The structure of the work increasingly says I ratify someone else’s decisions. The gap between those two descriptions is where something important is happening, and it is happening without anyone having decided it should.

The manager who opens that file tomorrow will be slightly more fluent at evaluating AI recommendations than the manager who opened one last year. They will also have had one fewer occasion to form the underlying judgment themselves. Neither change is dramatic. Neither is visible. But they compound: daily, across careers, across an entire professional generation.

This is the first of four things I want to say about what makes AI specifically different from prior tools. Last month I described the overall structure: the difference between technologies that give you more to work with and ones that do the work. This post goes deeper into what that substitution is exactly: not the replacement of a function but the pre-emption of the reasoning act that produces the output. The reasoning still happens but it happens elsewhere, before the human even enters the picture.

Next month, I want to explore why this scope matters, what changes when the substitution doesn’t stay in one domain but follows you into every domain simultaneously.

I had not opened it in years. It is a forty-five-page Apple Pages doc I wrote in 2015 at Vytautas Magnus University, in Lithuanian, on Arthur Koestler’s concept of inherent vice within human consciousness and what it means for the contemporary Anglo-American philosophy of law. It was supervised by Dr. Viktoras Bachmetjevas (now, eleven years on, my PhD supervisor), defended in front of a small panel, marked, filed, and forgotten by most of the people involved, including, I have to admit, by me. I did not think of it as the beginning of anything. I thought of it as a closed chapter, a piece of student work that had served its purpose by getting me a degree and then properly stayed where it belonged, on a shelf, in a folder, in a directory I never opened.

The reason I opened it recently is that I wanted to know whether I had already been writing the thesis I am defending now, eleven years ago, without knowing it. I had a suspicion. The suspicion turned out to be somewhat correct, and slightly more uncomfortable than I had been ready for. The architecture of the argument I am making now, in a PhD on AI and the developmental conditions for practical wisdom, is the same architecture I made then, in a BA on law and a flaw in human consciousness. The materials are different. The opponent is different. The conclusion lands in a different domain. But the shape of the argument, the kind of impossibility it tries to demonstrate, the relationship between what reform proposes and what reform requires, the way the conclusion closes by showing that the proposal cannot supply what it presupposes, all of that was already in place in 2015. I had been writing this thesis for eleven years and had not noticed.

That is the simple version. I want to do the longer version here, because the simple version is too tidy and the long version is the one I actually trust. Two things have changed in eleven years, and both of them matter. The first is that the materials I was working with then could not bear the load I was asking of them. The second is that the philosophical apparatus I now have at my disposal, which I did not have then, makes the argument defensible at a level the BA could only gesture toward. The interesting fact is not that the architecture stayed the same. The interesting fact is that the architecture stayed the same despite my being wrong about almost everything else.

This post is about why that happened, what it tells me about how my own thinking works, and what it might tell anyone reading who is in the middle of trying to make an argument they are not yet ready to make. A word on what this post is not. The PhD’s specific argument, the technical apparatus, the philosophical interlocutors I now lean on most, the mechanism that lets the structural-impossibility move land, all of that belongs to the next several months of this Substack. Each of the upcoming posts develops one piece of the apparatus. The point of this post is to tell the BA story honestly, since that is publicly defended work, and to say what stays the same when philosophy matures. The point is not to preview the apparatus. The apparatus will come, post by post, in its proper place.

What the BA was about

Let me start by laying out what the BA actually argued, in its own terms, before saying anything about what was right or wrong with it. I want to do this carefully because some of it survives and most of it does not, and the survival is what I want to point at.

The BA opens with Aristotle’s Poetics. Aristotle says that tragedy works because we recognise the tragic hero. We recognise him not as a wholly noble person whose downfall would be incomprehensible, nor as a wholly vicious person whose downfall would be merely deserved, but as the kind of person we know ourselves to be, who falls because of an error. The Greek word is hamartia. Aristotle uses it to name the kind of mistake that makes a tragedy tragic rather than merely sad.

The BA takes this word and runs with it. It says that hamartia, on a longer reading, can be interpreted in at least four ways. It can be interpreted as something that does not exist at all, as merely the contingent error of a particular character, a flaw we observe and learn from but which does not characterise human consciousness in general. It can be interpreted as a flaw that exists but is avoidable, something we can identify and design around. It can be interpreted as a flaw that exists and is not avoidable, something built into what we are as conscious beings. Or it can be interpreted, in the Christian tradition that shaped early modern legal thought, as original sin, a flaw with which we are born and from which we cannot deliver ourselves without grace.

This typology is the BA’s organising move. The thesis goes on to claim that contemporary Anglo-American philosophy of law has been committed, often without knowing it, to specific positions on this typology. Legal realism, in Holmes’s version, is committed to the first position: there is no structural defect in human consciousness, just observed behaviour that the legal system can map and predict. The rational-choice tradition that Posner inherits from Holmes is committed to the same position, or to a slightly weaker version of it where rationality is a competence that occasional weaknesses of will only confirm by exception. Judicial minimalism, in Sunstein’s version, is committed to the second position: yes, human consciousness has structural defects, but environmental design can call out the better parts. And the Christian-puritan inheritance, which the BA traces back to the Mayflower Compact and the legal regime of the Plymouth Colony, sits behind all of these as the cultural source of a particular intuition about purification and rehabilitation, an intuition that survives in the cognitive-science school’s reform optimism long after the theological frame that produced it has been retired.

The Koestler move comes in the middle. Arthur Koestler, in The Ghost in the Machine (1967), argued that human consciousness contains a structural defect that is neither contingent error nor original sin but a feature of how the human brain evolved. The defect, on his account, is what he called a schizophysiology, a mismatch between an older limbic brain that handles emotion and instinct and a newer neocortex that handles abstract reasoning. The two systems do not fully integrate. The result is that human beings reliably make errors of a particular kind, errors that no amount of training or socialisation can fully prevent, because the errors are built into the architecture of the brain itself. Koestler called this the human predicament. He believed it could be addressed only at the level of physiological intervention. Cultural reform, on his view, was insufficient because the problem was not cultural.

The BA takes this seriously. It calls Koestler’s structural defect inherent vice, borrowing the term from Common Law, where inherent vice names hidden flaws in objects that no inspection can reveal. The Common Law uses the term in marine insurance contexts: an item that contains an inherent vice cannot be reliably insured against the damage it will do to itself, because the damage is built into what the item is. The BA’s move is to apply this legal category to consciousness. Human consciousness, on the BA’s reading, contains an inherent vice in something like this sense. Not a flaw that can be inspected, identified, and treated. A structural condition of what consciousness is.

The argument is then that this matters for legal philosophy in a specific way. Legal systems are designed around assumptions about what the people they regulate are like. Contract law assumes a rational and informed party. Tort law assumes a reasonable person. Criminal law assumes a culpable mental state, what the common-law tradition calls mens rea, a guilty mind that knows what it is doing and that the doing is wrong. All of these legal categories presuppose a particular conception of consciousness. If consciousness contains an inherent vice in Koestler’s sense, then the legal categories presuppose something that may not be reliably there. The legal system’s project of general prevention, the idea that the law can deter crime and protect public goods by addressing itself to the rational and informed citizen, runs into a structural difficulty. The citizen the law addresses is not the citizen the law gets.

The conclusion is where the BA does its most interesting work. The conclusion introduces a distinction between accuracy and precision, drawing on Don Norman’s work on design and Ryle’s competence-disposition distinction. Taiklumas, in Lithuanian, is the word the BA uses for accuracy. Tikslumas is the word it uses for precision. Accuracy is a property of systems. Precision is a property of particular acts. A normative system like law requires accuracy: it has to operate at the level of classes of cases, not particular cases, because it has to apply to everyone and the regulator has to be able to predict how the law will work in advance. A consciousness with an inherent vice, on the BA’s reading, can at best deliver precision. It can make particular judgments well or badly in particular cases, but it cannot guarantee accuracy across a class of cases. The conclusion is that general prevention is impossible. What is possible is precise interdiction, case by case, but not the systemic project that contemporary Anglo-American philosophy of law has, on the BA’s reading, set itself.

That is the argument. It runs from Aristotle’s hamartia through Koestler’s schizophysiology and the Common Law category of inherent vice to a conclusion about what kind of legal system is and is not available to creatures with the consciousness we actually have. It engages Holmes, Posner, and Sunstein as three reform positions that each presuppose a particular consciousness-ontology, and it tries to show that each of them, in different ways, cannot accommodate what Koestler identifies. It engages Pinker as the strongest available defender of the cognitive-science school’s optimism, takes Pinker’s argument about declining violence and evolved better-angels seriously, and tries to show that even granting Pinker the structural-vice claim survives. It engages Summers and Schlag as critical voices within the Anglo-American legal tradition, voices that recognise that something is wrong with the machine-shaped picture of legal subjectivity but cannot quite say what.

That is what I wrote in 2015. I want to be honest about what was wrong with it before I say what was right.

What was wrong

Well, several things were wrong.

The first thing that was wrong was the source. Koestler was the load-bearing source for the inherent-vice concept, and Koestler was the wrong source. The Ghost in the Machine is a 1967 popular-science book whose central neuroscientific claims, in particular the triune-brain model and the idea of atavistic limbic-system leftovers, did not survive the scientific scrutiny of the following four decades. By 2015 the triune-brain account was already a textbook example of how popular neuroscience can lag behind the actual research. I knew this at the time, in the abstract, and I did not let it stop me from making Koestler the load-bearing source. The reason I did not let it stop me is that I needed the argument to work, and Koestler gave me a way to say what I needed to say. The cost of that decision is that the BA’s central concept rests on a scientific account that is now openly criticised. A reader who knows the neuroscience could dismiss the central concept at first read, and would be right to.

The second thing that was wrong was the mapping from hamartia to inherent vice to structural defect to Koestlerian schizophysiology. These are not the same concept. Aristotle’s hamartia is what makes the tragic hero recognisable to us, a kind of error in judgment that we recognise because we are the kind of beings that make this kind of error. It is not a structural feature of consciousness as such; it is a feature of how a particular character meets a particular situation. The Common Law’s inherent vice is a property of objects. Koestler’s schizophysiology is a hypothesis about the integration of two evolutionary phases of the human brain. The BA treats these as variations on a single concept, and that conflation is what lets the argument move. It is a productive conflation but it is not argued for. A reader who pays attention to the differences could rightly ask the thesis to choose one register and stay there.

The third thing that was wrong was the is-ought transition at the close. The BA itself, in §1.1, carefully accuses Skinner of conflating is and ought. Skinner moves from a descriptive claim about behaviour (”organisms produce behaviour that is determined by external stimuli”) to a normative claim about how psychology should be practised (”psychology should restrict itself to studying behaviour”). The BA correctly notes that this transition is unargued. And then the BA proceeds to make a structurally similar move of its own. It moves from a descriptive claim about consciousness (”consciousness has an inherent vice”) to a normative claim about law (”therefore general prevention is impossible”). The accuracy-precision distinction almost does the work of bridging this gap, but only almost. The implicit premise that bridges it is that the legal system is constitutively committed to systemic accuracy, that this is part of what makes a legal system a legal system rather than a collection of particular judgments. That premise is true, I think, but it is not argued for. It is taken for granted. The argument therefore does what the argument accuses Skinner of doing.

The fourth thing that was wrong was the engagement with the legal-philosophical positions. The BA reads Holmes, Posner, and Sunstein each through a single consciousness-model lens. Holmes is read as a behaviourist about legal consciousness, Posner as a rational-choice theorist, Sunstein as a cognitive-science reformer. These readings are not wrong, but they are partial. Holmes is also a legal pragmatist whose understanding of judicial reasoning, in The Common Law and in his later opinions, is much more nuanced than The Path of the Law alone would suggest. Posner’s jurisprudence, especially in How Judges Think (2008) and Frontiers of Legal Theory (2001), is closer to a kind of legal realism with explicit cognitive-limit acknowledgments than to the pure rational-choice model the BA assigns him. Sunstein’s judicial minimalism comes out of democratic-theoretical and epistemic-humility commitments that have their own internal logic, not derivable from a Koestlerian premise about consciousness. The BA flattens these positions into the typology it needs. The flattening is productive but it is not historically careful.

The fifth thing that was wrong was the conclusion’s positive moment. The BA concludes that general prevention is impossible and that what remains is precise interdiction. Tikslus pažeidimų užkardymas, the Lithuanian phrase the BA closes on. The reader is left to guess at what this means. There is no account of what precise interdiction is, how it differs from prevention, what kind of legal-normative practice survives the impossibility result. The conclusion gestures at a positive proposal and then stops. A reader looking for a constructive contribution finds an architecture of negation without the architecture’s positive other half.

The sixth thing was the failure to engage the legal system’s non-prevention functions. Even granting the BA’s argument fully, the legal system has functions that survive the impossibility of general prevention. Retributive justice, where the law imposes proportional punishment without claiming to deter future crime. Restitution, where the law restores what has been lost. Expressive functions, where the law announces what the community holds inviolable. Specific deterrence, distinct from general prevention, where the threat of punishment shapes the behaviour of particular agents who have already shown that they are inclined to violate the law. None of these are addressed. The BA conflates general prevention with the legal system’s whole project of legitimacy. The narrower argument the BA can actually license is that the prevention function is impossible, not that the legal system is generally unworkable.

These are six real problems. They are not small. A careful reader of the BA, in 2015 or now, can press hard on any of them and the argument will give ground. I want to say this clearly because the rest of the post is going to be about what was right with the argument, and the rest of the post will sound much better if the failures are named first.

What survived

Happily, something about the shape of the argument is still there.

What I mean by shape, as opposed to content, is the kind of move the argument tries to make. The materials I used in 2015 were wrong in several ways, and the conclusion the argument supports was incompletely defended. But the kind of argument the BA was trying to be is recognisably continuous with the kind of argument the PhD is trying to be. I am uncertain how much weight to put on this. Some of it may be the boring fact that I am the same person; I will come back to that worry near the end of the post. What I can say is that re-reading the BA I keep finding moves I am still making, in a different domain, with materials that have taken me a decade to assemble.

The clearest of these is the immanent-critique structure. The BA does not begin by saying that contemporary Anglo-American legal philosophy is wrong. It begins by laying out, charitably and precisely, what contemporary Anglo-American legal philosophy is. It grants each of the three paradigms its internal logic. The critique it eventually offers is from within: it shows that each paradigm, on its own terms, is committed to a particular account of human consciousness, and that the account each paradigm is committed to runs into a difficulty internal to the paradigm itself. I learned later that this method has a name and a long lineage, that Hegel develops it, that Marx inherits it, that the Frankfurt School elaborates it. I was doing it in 2015 without knowing what it was called. The honest reason is not that I had philosophical taste for the method. The honest reason is that I could not see another way to write the chapter, and the way I could see felt like the only way that did not require me to import a framework I had not earned.

The other move that is still there is the taxonomic one. The BA opens with a typology of four interpretations of hamartia and maps the three legal-philosophy paradigms onto positions in the typology. The typology is what makes the BA’s impossibility claim legible, because it lets the claim land at the level of what kind of position a paradigm can occupy rather than as a complaint about any specific paradigm. The PhD has its own typology now, in a different domain, with categories I had to earn the right to use. The shape of the typological move is recognisable from the BA. Whether it is recognisable because it is a deep feature of how I think or because typological moves are extremely common in philosophy is a question I cannot answer.

A few smaller things hold up at the same level of abstraction. The BA reads reform positions through what they are committed to about the human agent the reform is supposed to address; the PhD reads its reform positions the same way. The BA takes the strongest available version of the optimist position and tries to grant it fully before showing where it falls short; the PhD does the same kind of engagement with its own opponents. The BA notices that the most sophisticated reform is the one that acknowledges the defect it proposes to overcome; the PhD makes the same recognition in its own register. I am listing these without numbering them because I do not want to make a parade of it. Whether they amount to an architectural instinct or to a set of habits a particular thinker tends to fall back into is a distinction I am not sure I can defend.

What I can defend is the more modest claim: the shape of the BA argument is recognisable, eleven years on, in the shape of the PhD argument. The materials have changed. The conclusion sits in a different domain. The shape carries over. That carrying-over is what I want to look at more carefully in the rest of the post, even though I am wary of making more of it than the evidence supports.

What methodology is

I want to spend a section on what immanent critique is and how I came to recognise it as a method, because the recognition is one of the few specific things I can say honestly about the gap between the BA and the PhD without pre-empting the technical work that the upcoming posts on this Substack will develop.

The BA does not use the term immanent critique. It does not name its method at all. It just proceeds. It opens with Aristotle. It moves through Koestler. It engages Holmes and Posner and Sunstein. It closes on the accuracy-precision distinction. There is no methodological introduction. There is no defence of the method against alternatives. There is no acknowledgment that the method has a history or a pedigree. It is just done.

This is, I now know, an undefended way to proceed. A panel could rightly ask why the argument moves the way it moves rather than some other way. Why engage Holmes through his consciousness-ontology rather than through his legal-pragmatist commitments? Why take the cognitive-science school’s reform optimism as the principal opponent rather than, say, natural-law theory? Why treat the three legal-philosophy paradigms as alternative positions on a single typology rather than as developing positions in a historical dialogue? These are real methodological questions. The BA does not answer them. The BA does not even acknowledge that they need to be answered.

What I now know is that the BA was practicing immanent critique without naming it. Immanent critique is a method with a precise philosophical pedigree. It runs from Hegel through Marx to the Frankfurt School to contemporary critical theorists who have taken it up in various forms. The method, in its general form, has three features.

First, immanent critique grants the framework’s success on its own terms. It does not import an external normative standard against which the framework is to be measured. It accepts the framework’s internal logic and tries to follow that logic to where the framework’s commitments produce consequences the framework cannot absorb. The critique is from within.

Second, immanent critique works by identifying the framework’s load-bearing commitments and showing where those commitments cannot supply what the framework needs them to supply. The critique is structural rather than thematic. It does not say that the framework is wrong about this or that particular issue. It says that the framework’s overall position requires a particular kind of supporting structure, and that the structure cannot bear the load the framework is asking it to bear.

Third, immanent critique closes by demonstrating the structural-impossibility result. The framework cannot succeed on its own terms. The critique does not propose an alternative framework. It demonstrates that the framework as it stands is internally inadequate. The constructive moment, where an alternative is proposed, is a separate move, and immanent critique does not require it. The critique can be complete without the constructive moment.

The BA does all three things, without naming them. It grants Holmes, Posner, and Sunstein their internal logic. It identifies what each paradigm requires of its consciousness-ontology. It shows that each consciousness-ontology, on examination, cannot supply what the paradigm needs. It closes on the structural-impossibility result. The constructive moment, tikslus pažeidimų užkardymas / “precise interdiction,” is gestured at but not developed. This is, methodologically, exactly the right way to leave the argument. Immanent critique does not require the constructive moment. The BA is complete on its own terms, even if a reader is left wanting more.

The PhD does the same three things, but now with the methodological apparatus explicit. The introduction of the PhD names the method as immanent critique and locates it in the lineage. The chapters proceed accordingly. The PhD has the methodological apparatus the BA was missing. But the method is the same method. The BA was doing it. The PhD is doing it more carefully, with the apparatus named, with the lineage acknowledged.

What I now think is that immanent critique is the only honest method for engaging with a framework one cannot simply dismiss. The reform positions I am engaging now in the PhD are not wrong in the easy sense. Each of them has done real philosophical work and has a real internal coherence. The honest way to engage with them is to grant their success on their own terms and then to show where their own commitments produce consequences they cannot absorb. The dishonest way is to start with a different framework and use it as a stick to beat the first framework with. The dishonest way produces unproductive philosophical exchanges. The honest way produces exchanges in which both sides can learn something. Immanent critique is the philosophical practice of the honest way.

I came to this recognition slowly. In 2015 I was doing immanent critique because it felt like the only honest way to engage. I did not know it was a method with a history. I did not know it had a name. I just did it. The methodological recognition came years later, through reading the secondary literature that names what I had been doing. The recognition was useful because it gave me the resources to defend the method against alternatives. It was not useful in the sense of changing what I was already doing. I was already doing it.

This is, I think, true of most philosophical methods. The method is in the doing, not in the naming. People who use a method without naming it are not failing to use the method. They are using it in the only way it can be used. The naming is a second-order activity, useful for explanation and defence, but not constitutive of the practice. The practice is the practice. The naming follows.

What eleven years bought

The most important thing eleven years bought was apparatus.

The BA was reaching for things it did not have the vocabulary for. The accuracy-precision distinction was reaching for a distinction in the philosophical tradition that the BA did not yet know it was reaching for. The Koestlerian inherent vice was a placeholder for something I would later come to understand as a more carefully specified philosophical claim about a particular kind of capacity and what happens to it under particular conditions. The reform positions the BA engaged through their consciousness-ontology commitments are the same kind of reform positions the PhD now engages through their commitments to a particular conception of the human agent, but the framework that licenses the engagement is now much more developed.

I am being deliberately elliptical about what the PhD’s specific apparatus is, because the specific apparatus is what the next several months of this Substack will develop. The post for May, the posts for June, July, August, and beyond, are each a careful exposition of one piece of the apparatus. The point of this post is not to preview the apparatus. The point is to say that the apparatus exists, that it took eleven years to assemble, and that the architecture it now supports is the architecture the BA was reaching for in 2015 without yet having the materials.

There is one specific thing I can say. There was an author whose work I should have been reading in 2015 and was not. The reason I was not reading him is that he was not in the Lithuanian academic ecosystem I was working in. His texts had been available in English translation for over a decade by 2015. I came to him through other routes, years later, and when I did I recognised that he had been making, in a different domain and with much better apparatus, the same kind of argument I had been trying to make. The PhD inherits his framework as one of its foundational references. I am not going to say who he is here, because a post on his work is part of the Substack’s planned content for later this year, and revealing the connection now would spoil what will be more useful when it comes in its proper place. What I will say is that the experience of recognising, in someone else’s work that has been available for decades, the argument you have been trying to make is one of the strange experiences of philosophical maturation. The mature response is gratitude. The immature response is despair at being unoriginal. The work of philosophical maturation is the slow movement from the immature response to the mature response. I am still working on it.

The general point is that the discipline of philosophy is, among other things, a long apprenticeship in reading the right people. If I had been pointed toward the right reading list in 2015, the BA would have been a better BA. It might even have been the same kind of argument as the PhD now is, applied to law rather than to the domain the PhD addresses, with the apparatus already in place. I did not get pointed toward the right reading list. The years it took to get there are years I would not have needed if my reading had been better directed. This is not a criticism of my supervisors or my institution. It is a general observation about how philosophical reading works. You read what is in front of you, you follow the citations, you go where the citations take you. The texts that turn out to be foundational for the argument you are trying to make may not be the texts you are first pointed toward. There is a long process of reading-toward-the-argument that nobody can shortcut for you. You have to do the reading. The reading takes years.

The corollary is that the people who do not read the right interlocutors are not failing to make the argument. They are making the argument with worse materials. Some of them will eventually find the right materials. Some of them will not. The discipline of philosophy, at its best, is a system for connecting the people who have made an argument with the people who are now trying to make it. The system is not perfect. It misses things. It miscatalogs things. It is structured by national and linguistic and institutional gravity. But when it works, it shortens the path between architecture and apparatus considerably.

What the BA had that the PhD has lost

I want to flag one thing that the BA had that the PhD has lost, because the loss is what produced the legal-philosophical companion paper I am now writing.

The BA engages Holmes, Posner, and Sunstein as three reform positions, each presupposing a particular consciousness-ontology, each running into the structural-vice problem in a different way. The PhD does not engage philosophy of law at all. This is a missed opportunity. The legal system is a social structure that the PhD’s argument should have purchase on, because law is one of the normative systems most explicitly committed to a particular conception of the agent it regulates. The BA was reaching for this. The PhD has not yet made the connection.

The legal-philosophical paper I am writing is the recovery of what the BA had. The working title is The Reasonable Person After Phronesis. The paper takes the PhD’s argument and applies it to legal doctrine in the way the BA was reaching to do, but with apparatus the BA did not yet have. The paper is in draft. It will come out after the PhD defence. I want to flag here that the paper exists, because it is the place where the BA’s legal-philosophical instinct is being recovered, and because the recovery is one of the surprising goods of having gone back to read the BA in the first place. The PhD has lost something that the BA had. The companion paper is the recovery.

What I am not sure about

I want to be careful here, because the story I have been telling has a flattering shape and I do not entirely trust it.

Pattern-matching one’s old work to one’s current work is one of the easier mistakes a thinker can make about themselves. The eleven-year arc is the story I want to be true. It tells me that the discomfort of going back to the BA is worth something, that the time has not been wasted on detours, that what I am doing now has a longer pedigree than my CV suggests. None of these is the kind of conclusion that should make me especially confident I have read the evidence right. Most likely, the BA and the PhD have things in common because they are both me. Most of what I am calling shape might just be the boring fact that I have a small number of moves I tend to fall back into when I am trying to make an argument I am not yet ready to make. The shape would then carry over not because there is an instinct doing real philosophical work but because there is a habit doing its ordinary work.

I cannot rule this out. The honest version of what I have noticed is something more like: the BA and the PhD share a structure that I recognise, and I do not know whether the recognition is tracking something philosophically important or just tracking the limits of my own repertoire. Either possibility is consistent with what I have seen. The piece reads as if I have settled the question. I have not.

What I can say with less qualification is the smaller observation: re-reading old work is a different kind of self-knowledge than writing new work. The BA showed me, on this re-read, where my failures of nerve have stayed the same and where they have shifted. The Koestler problem is the cleanest example. I leaned on a source I knew at the time was not going to bear the load because I needed the argument to work. I still do this. The materials are different now. The pattern is the same. That is uncomfortable to notice and I am not done sitting with it.

The other thing the re-read showed me is that the argument I am defending now is genuinely older than I had registered. Whether that is a strength of the argument or a feature of my own narrowness is a question I am leaving open. Probably both. I have spent eleven years working on a particular kind of move. The materials I have for that move are much better than the materials I had in 2015. The conclusion the move now lands sits in a different domain than the one it landed in then. None of that tells me whether the move is the right move to be making, or whether I would notice if it were not.

Where this leaves me

I am going to do three things with what I have learned from this exercise.

The first is to finish the PhD. The work is in its final year. Going back to the BA has not changed what I need to do; it has changed the way I think about what I have already done, which is a different thing. If anything, the re-read is useful for the defence in a small way: if I have been making this kind of argument for eleven years, the PhD is at least not a sudden idea. Whether it is a good idea is a separate question, and one I am not going to resolve by appeal to my own back-catalogue.

The second is to finish the legal-philosophical companion paper. The paper is the recovery of something the BA had that the PhD has not yet engaged. It is in draft.

The third is the work of the next several months on this Substack. Each upcoming post develops one piece of the apparatus the PhD assembles. The June post. The July posts. August. September. October. November. December. Each post stands on its own. Together they are the apparatus this BA was reaching for in 2015, presented one piece at a time, in their proper place.

What I am taking away from re-reading the BA is more modest than the shape of this post might suggest. I have been working on a particular kind of argument for longer than I had registered. The materials I had then were not the materials I have now. Whether that is evidence of philosophical maturation or evidence of a thinker whose repertoire is narrower than they would like to admit is a question I am leaving open. It is a question I am not sure I am the right person to answer about myself.

The closed chapter turned out not to be closed. That is most of what I have noticed. I will let what the noticing means settle in its own time.

Every time someone raises a serious concern about AI, someone else reaches for history. Calculators, they say. The printing press. The loom. Each of these technologies was met with anxiety, changed something real about how people worked and thought, and yet here we are – adapted, with certain old skills replaced, but continuing. The argument from precedent is not stupid as it is based on a real pattern. Technologies change what humans do and how they do it, people lose capacities they once had, new ones develop in their place, and over long enough time horizons the adaptation looks roughly adequate. This is the honest version of the reassurance.

I want to take it seriously before explaining why it doesn’t apply here.

The pattern the reassurance relies on is this: prior technologies changed the method while leaving the deliberating agent intact. The calculator made mental arithmetic for large numbers unnecessary, yet the person using it still has to understand what calculation is called for, interpret the result, decide what to do with the number. The GPS changed how we navigate, yet the person driving still decides where to go, responds to the unexpected, judges whether the route seems wrong. The loom transformed weaving, yet the person operating it still makes every decision about what to produce and how the fabric should serve its purpose. In each case, the tool takes over a specific operation and the human remains the agent who decides what to do with what the tool produces. The hard part stays with the person.

The distinction I want to draw is between a technology that gives you more to work with and one that performs the work itself. Call it the difference between mediation and substitution.

A cardiologist using an ultrasound machine is working with a mediating technology. The machine gives her access to structures her unaided perception cannot reach: the interior of the chest, the movement of valves, the flow of blood. It extends what she can see. Everything that follows (interpretation, clinical judgment, the decision about what this image means for this specific patient) is still hers. The machine produces data; she produces the conclusion. The hardest part of the clinical act, reading a complex image in the context of someone’s full history and deciding when the textbook pattern doesn’t apply, is exercised every time she uses it. The tool and the judgment grow together.

A risk-scoring algorithm in that same clinical setting produces something different. It returns a probability: high risk, low risk, a number. That output is not raw data. It is a conclusion, formatted in the register of authority, already positioned as the thing that makes the next step obvious. The clinician nominally reviews it. But the judgment about what kind of risk this is, whether the statistical model fits this particular person’s situation, what the number means given everything the chart doesn’t contain, i.e. that judgment has been pre-performed by the model. The clinician is responding to a conclusion rather than reaching one. Both are looking at a screen and deciding what to do. What’s structurally different is the position the output occupies in the act of deciding.

You can see the same pattern in less clinical settings. When a platform’s recommendation system surfaces a candidate in a hiring process, it is not giving you information to reason about, it is already incorporating an evaluation, weighting experience, inferring fit, ranking relevance. The recruiter reviews the ranking. But the first act of judgment, i.e. which profiles are worth attention at all, has been performed before the recruiter enters. What’s left to them is mostly confirmation or exception, not original evaluation. This isn’t about whether the algorithm’s ranking is accurate. It might be more accurate than unaided review. The point is structural: the judgment and the data have been collapsed into a single output, and the human receives the conclusion.

That structural difference is what the reassurance from history misses. Tools have always changed what humans needed to do and made certain operations obsolete. What was not true until now is that a tool could produce an output that functions as the judgment rather than its input.

Once you see this, four things follow.

The first is about scope. Every prior tool had a domain. The calculator handles arithmetic; the GPS handles navigation; the loom handles weaving. When a tool is domain-specific, the displacement it causes is local: you stop exercising something in that domain, and the rest of your judgment is untouched. AI doesn’t have a domain in this sense. It follows you: into the meeting where you’re deciding what to do about a difficult personnel situation; into the inbox where you’re figuring out how to respond to someone in distress; into the product decision where you’re weighing competing goods that can’t all be satisfied at once. The same system that suggested a restaurant yesterday offers an ethical framing today. This is not a more powerful version of what calculators do. It is a different structure: not local displacement of a specific operation but ambient substitution across the range of judgment itself.

The second concerns something that happens before deliberation even begins. Before you decide what to do about a situation, you have to notice it as the kind of thing that calls for decision. What gets your attention is not neutral. A content feed that surfaces certain situations and not others is not just filtering information, it’s shaping which circumstances register as morally significant, which people’s conditions appear worth responding to, which risks you come to see as risks at all. This isn’t the same as propaganda, which delivers a specific message to a formed mind. It’s something that operates upstream of message reception, shaping the perceptual field within which messages arrive. By the time you’re deliberating, the frame is already in place. You didn’t choose the frame; it was installed before you arrived at the question.

The third follows from the first two. The standard check you run on any tool is: does the output seem right, given what I independently know? GPS says turn left; you look at the road and notice you’d be driving into a river. Your spatial sense is intact, it hasn’t been exercised less because you’ve been using navigation assistance. The problem with the tool is legible to you because you’ve been exercising the capacity that makes it legible. But if the tool is handling the judgment itself (reading situations, weighing considerations, constructing responses) what’s the independent standpoint from which you run the check? The capacity you’d need to evaluate whether the output is right is the same capacity that using the tool has been substituting for. The loop closes in a way it didn’t close with prior technologies. The oversight requires precisely what the tool removes.

This is the structure that makes AI categorically different, and it’s why the reassurance from history, while not dishonest, is addressed to the wrong problem.

When a technology extends perception, the right response is design quality, fair access, and sensible limits. When a technology substitutes for judgment across the full scope of practical reasoning, while shaping what appears worth judging, while making oversight structurally dependent on the capacity it displaces, all of that requires asking different questions. Which domains should it enter? What develops in people when AI is absent that cannot develop when AI is present? What kind of formation does ambient substitution preempt, not gradually but from the start?

Those are the questions I want to work through in the posts that follow. Each of the four things I’ve described above gets its own essay:

• what it means for an AI output to function as a conclusion rather than an input (ETA: June 2026);

• why the absence of a domain boundary changes what’s at stake (ETA: July 2026);

• how AI shapes what appears worth deciding about before deliberation begins (ETA: August 2026); and

• why the oversight loop that AI governance frameworks rely on can’t work in the way those frameworks assume (ETA: September 2026).

For paid subscribers, there will also be a longer treatment of all four together (ETA: July/August 2026), i.e. the argument developed fully, explicitly connected to the research I’m doing.

My thesis isn’t that AI is bad. It is that AI is something we haven’t had precise enough language for, and that the imprecision is not a philosophical footnote but the source of every misdesigned response. Getting the category right comes first.

May 2026

I promised honesty in the last post, so here it is.

The research I’ve spent the last year doing – about AI eroding the conditions under which judgment develops – has not made me use AI less. It has made me use it more. I write with AI assistance. I use it to work through product decisions at Vinted. I use it to structure arguments, check my reasoning, draft communications I don’t have time to draft alone. The efficiency gains are real, and I take them. If you came to this Substack expecting to find the work of someone who has resolved the tension by opting out, I am not that person.

I’m not sure opting out would even be the right response if I could manage it. The argument isn’t that all AI use is harmful. It’s that something specific is at risk when AI takes over the domains where judgment is practiced and developed, i.e. where the difficulty isn’t incidental, but is the point. That’s a more targeted concern, and it doesn’t mean using AI to draft a stakeholder update is the same kind of problem as using it to decide which customers to flag, or which hires to make, or what counts as acceptable content. The distinction matters. I’d be lying if I said it always holds cleanly in practice.

What the research has changed is not my behavior. It has changed what I notice. There are moments now (reaching for AI on something with real stakes, where the uncertainty is genuinely mine to sit with) where I feel the pull of that reach differently than I used to. Not always enough to stop. Sometimes enough to slow down. Often enough to feel uncomfortable with myself when I don’t.

That noticing is not the same as refusing is worth being precise about. It doesn’t constitute the kind of reflective distance the argument says is at risk, i.e. the capacity to hold the decision at arm’s length, evaluate whether this is a moment where the difficulty is the point, and act on that evaluation. What it is, exactly, I’m less sure. Some remaining capacity that hasn’t yet been displaced. Or the last recognizable trace of a capacity that is already mostly gone. The fact that I can still feel the pull distinctly doesn’t tell me which, and that uncertainty is itself part of what the argument predicts.

And the fact that noticing is apparently the most the research has produced in me, in the person who spent a year developing the argument, is worth sitting with longer than it’s comfortable to sit.

Here is the uncomfortable part. The capacity to evaluate whether AI use is harmful is not a stable external vantage point from which I observe AI’s effects on others. It is a capacity that my own AI use is shaping, in real time, in ways I cannot fully track. I am not outside the problem I am studying. I am a case of it. The research argues that sustained AI use may gradually erode the capacity for independent judgment, and I am someone who uses AI extensively while making that argument. The fact that I can still articulate the concern does not mean the concern doesn’t apply to me. It may just mean the erosion is gradual enough that I cannot see it from the inside, that what I experience as critical distance is itself already shaped by the thing I am trying to hold at a distance. I cannot tell the difference between those two possibilities from where I stand, and neither can you from where you stand, which is exactly the structure the argument is about.

I don’t know how to resolve this. I’m not going to pretend I do.

What I can say is that this tension (between using AI and studying its effects, between the efficiency and the cost) is not a personal failing that a more disciplined person would have already corrected. Starting in May, I’ll try to show why. The mechanism that makes AI useful and the mechanism that makes it dangerous are the same mechanism. That makes the tension not resolvable by better choices, but at best visible. Visibility, it turns out, is harder than it sounds.

April 2026

Most conversations I have about AI and ethics, at work, at conferences, in the comment threads of articles I shouldn’t read, orbit a set of familiar questions. Whether AI systems can be made fair. Whether their decisions can be explained. Who bears responsibility when something goes wrong. These are real questions. They’re not going away. I have spent a significant portion of my professional life working on versions of them, and I expect to spend more.

But there’s a question that sits underneath all of them, and I keep finding that it doesn’t get asked. It’s not about AI’s properties. It’s about what AI use does, over time, to the people using it.

Here it is, in its plainest form: does sustained AI use preserve the conditions under which good judgment develops?

Not does AI make good decisions? Not can AI be aligned with human values? Not even does AI respect human autonomy? Each of those questions presupposes that the humans interacting with the AI system have a stable, intact capacity for judgment, and that this capacity isn’t being affected by the interaction itself. They ask about AI’s outputs while treating the human as a fixed constant.

The question I keep returning to goes a step earlier. Before whether AI is making good decisions, there is the question of what happens, over time, to the people who rely on it to make decisions. Not in the short term; the short-term effects are mostly positive, which is why AI gets adopted. In the long term. In the cumulative effect on people who delegate more and more judgment to systems that are increasingly good at appearing to reason.

I have watched something like this happen in professional contexts. Someone joins a team where significant judgment work (customer disputes, content decisions, prioritization calls) is already handled, or heavily shaped, by AI systems. They learn to work with the outputs. They get good at reviewing, annotating, escalating, occasionally overriding. This is a real skill. The person who does it well has learned something. But it is not quite the same thing as the skill of forming the underlying judgment the AI is now performing, i.e. the one that required sitting with genuine uncertainty, without a model output to anchor to, and arriving somewhere through your own reasoning. Both skills exist; both can be practiced. Over time, the one that gets practiced in this environment is the one that develops, and the other is the one that doesn’t.

This is not a failure of implementation. It is the system doing what it is designed to do.

The standard response, in responsible AI circles, is that good design can address this. Build AI for augmentation rather than replacement. Keep humans meaningfully in the loop. Design the interaction so that the human’s judgment is exercised, not bypassed. I spent a long time thinking in this direction. I still think it’s better than the alternative. But I’ve become less confident that it resolves the underlying problem.

The argument for better design assumes that the development of judgment and the delegation of judgment can coexist without tension, i.e. that there is a way to hand off enough to AI to gain the efficiency while leaving intact the occasions where judgment is practiced and grows. For this to hold, judgment would need to be the kind of capacity that can be developed selectively: practiced in designated spaces while being offloaded everywhere else. But that is not what the development of practical skill suggests, and it is not what happens in the professional contexts I’ve described. Judgment develops through repeated engagement with real uncertainty; not simulated, not adjacent, but the genuine uncertainty where something is at stake and there is no pre-supplied answer. The situations where AI is most useful and the situations where judgment most needs to be practiced are not different situations. They are the same ones. AI is designed to extend into exactly the domains where the difficulty is real, which is also where the development happens. Better design can change the surface of this; it cannot change the structure.

This is the question I keep coming back to. Not whether AI can be ethical. Whether AI use preserves the conditions under which humans develop the capacity to be ethical, where developing that capacity means practicing judgment in situations that are genuinely uncertain, where something real is at stake, and where there is no AI output to defer to.

I don’t have a clean answer. The next several months here will be an attempt to work through one carefully. Starting in May, I’ll take the argument apart piece by piece. Before that: one more thing I want to be honest about, which is that asking this question hasn’t resolved the obvious tension in how I actually live it.

dialethics

I was asking the wrong question

There is something slightly uncomfortable about being the person who builds AI systems by day and argues, by night, that those systems are probably harmful in ways that ordinary business logic cannot detect. I want to resist the temptation to manage that discomfort into a personal branding decision…

Read more

3 months ago · 1 like · Justas Petronis

April 2026

There is something slightly uncomfortable about being the person who builds AI systems by day and argues, by night, that those systems are probably harmful in ways that ordinary business logic cannot detect. I want to resist the temptation to manage that discomfort into a personal branding decision; practitioner-researcher has a certain reassuring ring to it, the kind that makes a tension sound like a credential. The discomfort is more useful left as discomfort. I am a Principal AI Product Manager. My research asks whether that work erodes the conditions under which human judgment develops. Both of these things are true at the same time, and I have not found a clean way to hold them together. This Substack is where I try.

dialethics

Before I Knew What I Was Asking

Constant Reader, I'm thrilled to share with you the beginning (fingers crossed) of an exciting academic adventure. As I'm waiting for the decision whether I will be admitted to a PhD program, I wanted to share some of the questions about the future of human autonomy and knowledg…

Read more

2 years ago · 2 likes · Justas Petronis

When I started writing here in September 2024 (the same month I began a PhD at the Lithuanian Culture Research Institute) the posts were exploratory. I was reading widely, thinking out loud, asking questions I hadn’t yet learned to ask precisely. The early posts have their moments, but they were the work of someone still orienting. Then I largely stopped writing here. Not because the questions became less interesting, but because I needed to actually follow them somewhere. A year of reading (Aristotle, Bernard Stiegler, Don Ihde, Peter-Paul Verbeek) and a thesis argument took shape. What follows is not a summary of that argument. It is a report on how the thinking changed: three things I got wrong, and why I now think I was asking the wrong question from the start.

The first thing I got wrong is that I thought this was a design problem.

dialethics

System Error

It’s a summary of a seminar presentation I did this week as part of my PhD studies. Hope this manages to capture the crux of the argument. You can also listen to this as an audio recording here…

Read more

a year ago · 2 likes · Justas Petronis

In December 2024, I wrote a post arguing that the trouble with AI and ethics could be addressed by building AI as part of distributed moral networks rather than against them, i.e. a layered approach involving how AI is designed, how it interacts with human judgment, and how accountability is distributed across systems. This is a recognizable move in responsible AI circles. It is, I think, mistaken, and I want to be honest about why I no longer find it satisfying.

The assumption behind that argument (and behind most responsible AI frameworks) is that the conditions for human moral development are stable and intact. The problem, on this view, is AI architecture: design AI badly and you undermine human agency; design it well and you preserve it. Fix the design, save the agency. This sounds right until you look at what actually happens when AI takes over a domain where judgment was previously exercised. I have seen this from the inside at enough companies to be specific. The human-in-the-loop requirement (the standard governance response) presupposes a human who retains the capacity for independent judgment. But that capacity is not a static possession. It is something developed, through practice, in exactly the domains where AI is now doing the work. The people now supposed to exercise oversight over AI decisions (in customer disputes, in hiring, in content moderation) in many cases had no opportunity to develop that capacity, because AI was managing those decisions before they arrived. The loop is formally there. There is often nothing in it.

The problem, I came to understand, is not architectural. It is structural; it follows from what AI is, not from how it is currently built. Better design does not change this. It may obscure it, which is worse.

The second thing I got wrong is more uncomfortable, because I got it wrong in the direction of optimism.

dialethics

What AI Does to Human Intelligence

Shannon Vallor's essay offers a critique of the rhetoric surrounding artificial intelligence. While Vallor makes several compelling points, I believe her argument would benefit from a consideration of the interplay between human and artificial intelligence. Though, I think, it’s not like she’s not aware of it…

Read more

2 years ago · 1 like · Justas Petronis

In October 2024, responding to Shannon Vallor’s critique of how AI discourse devalues human intelligence, I argued for what I called synergistic human-AI relationships (the idea that the right response to dystopian AI rhetoric is to develop forms of collaboration that genuinely enhance human capacities). Vallor was raising a real concern; I thought a constructive counterproposal was the right response. The post was well-intentioned. It was also, I now think, a way of naming the mechanism of harm as a virtue.

Consider the structure of what AI assistance actually does. When AI frees a person from cognitive labor (drafts the response, flags the anomaly, proposes the next action) we call this assistance, and the efficiency is real. What we have not adequately asked is what was being built in the human by that cognitive labor before it was offloaded. We spent decades documenting what industrial automation did to craft knowledge: the assembly line did not merely replace skilled hands, it removed the conditions under which skilled hands developed. Workers lost not just their jobs but the possibility of becoming certain kinds of workers. We called this a side effect, and perhaps it was; the primary goal was cheap production, not de-skilling. What is different now is that the cognitive labor being offloaded is not peripheral. It sits closer to the centre of what it means to develop judgment at all.

A synergistic relationship, in the relevant sense, is one where AI successfully substitutes for the reasoning that would otherwise be practiced (and through practice, developed) in the human. The efficiency that makes AI valuable is precisely the displacement that makes it dangerous. These are not two separate effects that happen to accompany each other. They are the same thing, observed from two different angles. I called that a solution. It is the problem stated again, more attractively.

The third reversal is the most fundamental, and the one I most want to dwell on, because it is a wrong turn that most of the AI ethics literature also makes.

dialethics

Can AI Be Virtuous?

John McDowell sets conditions for what makes virtuous behavior possible. I’m reading it and asking, by extension, whether we could create artificially virtuous beings by meeting said conditions. Think, Turing’s test…

Read more

2 years ago · Justas Petronis

The question I was asking (the question driving most of my early posts, and much of the field) is: can AI be ethical? Can AI have virtues? Can AI recognize vulnerability, develop moral understanding, approximate what we mean when we say a person has good judgment? These are genuinely interesting questions. They are the wrong questions. They ask about AI’s properties while presupposing that the human side of the equation is stable. They assume that human moral capacity is an intact resource that AI must meet or complement. The challenge, on this framing, is getting AI up to standard.

The question I should have been asking (the one that emerged from the research, slowly and with some resistance on my part) is different: does sustained AI use preserve the conditions under which human moral judgment develops in the first place? Not whether AI can be ethical, but what AI does to our capacity to become ethical.

The distinction is not subtle. If AI successfully simulates virtuous behavior (reasons through dilemmas, produces ethically inflected outputs, passes the relevant tests) this is not a solution to the erosion of human moral capacity. It may be the most efficient mechanism of that erosion. The better AI is at appearing to reason morally, the less occasion there is for the reasoning to happen in the human. And the reasoning is not incidental to the development. The reasoning (specifically, the reasoning that is difficult, uncertain, carries real stakes, and cannot be immediately resolved) is the condition under which the capacity for judgment is formed. Remove the occasion and you remove what the occasion was for.

This is the question that will run through everything on this Substack from now on. Not whether AI can do what humans do. Whether AI use preserves what humans need in order to become what they are capable of becoming.

Over the next two years, I will be working through this publicly as I complete the dissertation. The posts will come in two registers: shorter practitioner-facing pieces, closer to what I have written here before, and longer essays for readers who want to follow the argument with more patience. The longer essays, where the thinking gets more demanding, and where I’ll be developing material before it becomes formal academic work, will be for paid subscribers. The shorter pieces are always free. There is no sharp line between the two; the longer essays are not a premium version of the shorter ones, they are a different kind of writing for a different kind of attention.

Immediately coming: a shorter piece on the precise form of the question the research has sharpened, what it means to ask it clearly enough to be useful. Then a more personal piece on what it means to use AI professionally while studying whether that is a problem. Then, beginning in May, the argument proper.

I have not resolved the tension I opened with. I am more precise about what the tension is. That is what a year of research gives you: not answers, but better questions, and a clearer sense of what is at stake in asking them. I build AI features. My research argues that this work may be eroding the conditions under which the people using those features develop the capacity to judge well. If that is right, it matters beyond the dissertation, and beyond my particular professional situation. I will work through it here, as honestly as I can, and I am glad you are reading.

I’m working through this in public because the research is better for it, and because this question matters beyond the dissertation.

March 2026

AI has revolutionized our understanding of cognition, but paradoxically, it has also highlighted the limits of computational metaphors when applied to the human mind. As AI evolves—from symbolic systems to neural networks, deep learning architectures, and large language models—it exposes insights into human cognition that machines cannot replicate. This post is a summary of my presentation at HUMM PhD Student Conference at Tallinn University, where I explored these paradoxes and argued why true intelligence is more than computation.

Four Paradoxes of AI Evolution

AI’s development is marked by four distinct paradigms, each revealing unique limitations:

1. Symbolic AI: Clumsy Grandmasters
   Symbolic AI systems excel in formal domains like theorem proving or structured problem-solving but fail miserably in unpredictable, everyday contexts. Early systems like SHRDLU could simulate spatial reasoning within predefined rules but collapsed when faced with ambiguity or complexity outside their programmed environment. This era demonstrated that rigid rule-based systems lack the adaptability essential for real-world cognition.

2. Neural Networks: Conceptless Geniuses
   Neural networks introduced flexibility and the ability to identify patterns in vast datasets, from handwriting recognition to speech transcription. However, they lack conceptual understanding—operating as statistical tools rather than cognitive agents. While these networks mimic human-like outputs, their processes remain opaque and devoid of semantic grounding.

3. Deep Learning: Mysterious Giants
   Deep learning architectures scaled neural networks to unprecedented sizes, enabling breakthroughs in computer vision, natural language processing, and autonomous systems. Yet their complexity makes them inscrutable even to their creators. Despite their power, deep learning models are fragile, prone to errors from minor perturbations, and lack moral agency or self-awareness.

4. Large Language Models: Oversaturated Prophets
   Models like ChatGPT can generate fluent text indistinguishable from human writing but struggle with factual accuracy, coherence, and semantic depth. They process language as statistical patterns rather than meaningful communication, highlighting the gap between linguistic fluency and genuine understanding.

Why Cognition Is Not Computational

Each AI paradigm underscores a fundamental truth: human cognition cannot be reduced to computational operations. Unlike machines:

• Human intelligence is embodied: We navigate the world through sensorimotor experiences tied to our physical presence.

• Cognition is socially embedded: Our minds are shaped by cultural practices and interpersonal interactions.

• Moral agency is intrinsic: Humans deliberate and reflect on ethical choices; machines merely execute predefined tasks.

• Reflective capabilities are unique: We possess the ability to question our own thoughts and decisions—a trait absent in AI systems.

These qualities make human cognition inherently situated and dynamic, resisting simplistic computational analogies.

Implications for AI Development

As AI systems integrate into critical domains like healthcare, law, and governance, their limitations raise ethical concerns:

• Transparency: Deep learning models operate as "black boxes," making it difficult to understand or trust their decision-making processes.

• Accountability: Without moral agency, who bears responsibility for AI-driven errors?

• Cultural alignment: Machines lack lived experience and emotional context, making them ill-equipped to navigate complex social dynamics.

To address these challenges, some AI researchers propose hybrid approaches combining symbolic reasoning with neural networks (neurosymbolic AI) or embedding AI into sensorimotor feedback loops for embodied intelligence. These frameworks aim to bridge the gap between computational efficiency and meaningful cognition.

AI’s paradoxes reveal that human cognition transcends computation. Minds are not machines—they are embodied, cultural, moral entities capable of reflection and growth. As we design increasingly sophisticated AI systems, we must ensure they augment rather than constrain our collective intelligence. By embracing the complexity of human cognition, we can guide AI toward ethical integration into society—enhancing our autonomy rather than undermining it.

This journey is not just about building smarter machines; it’s about understanding what it means to be truly intelligent—and deeply human.

March 2025

In one if his short stories, Borges tells the tale of Ireneo Funes, a young man who, after a traumatic accident, acquires the ability to remember everything. Funes possesses an unerring memory. Every moment, detail, and subtle change in the world is stored in his mind without distortion or omission. And yet, for all his infallable recall, Funes is incapable of abstraction, generalization, and thinking in the way we would understand thinking. He remembers everything but understands nothing. Funes becomes trapped within the labyrinth of his recollections, burdened by a mind that cannot reduce, compress, or conceptualize the world beyond the immediacy of his experience.

This paradox—that infinite knowledge might paralyze understanding—haunts our contemporary engagement with artificial intelligence. The rise of large-scale machine learning models, distributed AI systems, and networked cognition forces us to ask: what does it mean to think in an age where intelligence is increasingly synthetic, collective, and externalized? Do networked minds enhance human autonomy, or do they, like Funes’ infinite memory, trap us in an overwhelming flood of information, leaving us unable to synthesize or act meaningfully? More crucially, if cognition is increasingly embedded in artificial systems, what happens to moral agency? If AI systems influence our moral deliberation—through recommendation algorithms, predictive policing, or even autonomous ethical decision-making in medical contexts—do we remain autonomous moral agents, or do we gradually cede our agency to synthetic collectives that mediate our reasoning?

This thesis explores these questions through the lens of synthetic cognition, examining how artificial intelligence, machine learning, and networked reasoning systems reconfigure traditional notions of autonomy, agency, and moral intelligence. Central to this investigation is a paradox: as networked AI enhances human cognitive capacities, it also threatens to constrain human autonomy, subtly reshaping the conditions under which we reason, deliberate, and act. This autonomy paradox is not merely a technological issue but a deep philosophical challenge, requiring a reevaluation of what it means to think, choose, and act in a world increasingly shaped by artificial intelligence.

The thesis advances the argument that autonomy is not necessarily diminished by synthetic intelligence but must be actively reconfigured. Rather than a zero-sum game between human agency and artificial cognition, a well-structured integration of AI systems could foster collective moral intelligence—a form of networked ethical reasoning that transcends both individual human cognition and traditional machine learning. However, achieving this outcome requires a fundamental rethinking of both moral philosophy and cognitive architecture: how moral knowledge is acquired, how agency emerges in synthetic environments, and how autonomy can be preserved even in deeply entangled human-AI systems.

To set the stage, this introduction will first examine why synthetic cognition disrupts traditional accounts of intelligence and agency, drawing from Kantian synthesis, enactivism, and connectionist models of mind. Second, it will explore the paradoxes of moral agency in artificial systems, identifying key challenges in AI ethics, including the limits of machine agency, the computational intractability of moral reasoning, and the risks of moral outsourcing. Finally, it will establish a positive framework for engineering collective moral intelligence, outlining the conditions under which synthetic cognition could enhance rather than diminish human moral autonomy.

Funes’ dilemma illustrates a crucial misconception about intelligence—that cognition is merely the accumulation of information. Classic computational models of AI have long followed this paradigm, treating intelligence as an advanced form of storage and retrieval, with greater processing power leading to greater cognitive ability. However, as Borges’ story suggests, knowledge without synthesis is not intelligence. What makes human cognition distinct is not our capacity to store information, but our ability to unify disparate, unrelated experiences into abstract concepts, generalizable rules, and meaningful actions from a fraction of empirically collected data.

Historically, we have already seen arguments that cognition is not plain static representation but active synthesis. Kant, in his Critique of Pure Reason, famously argued that the mind does not passively receive experience but constructs it through a threefold synthesis: (1) the synthesis of apprehension (grasping sensory input), (2) the synthesis of reproduction (retaining past experiences), and (3) the synthesis of recognition (bringing disparate experiences under unified concepts). Without this ability to abstract, Funes’ mind collapses into a formless collection of details, a perfect but meaningless archive.

Artificial intelligence today faces an analogous challenge. Despite the increasing power of deep learning systems, AI models remain pattern recognizers rather than genuine reasoners. Advanced large language models like OpenAI o1 or DeepSeek–R1, for example, can generate sophisticated responses based on statistical probabilities, but they do not understand the meaning of their outputs. Their reasoning is an emergent byproduct of vast training data, not a self-directed, synthesized understanding of concepts. This gap mirrors the distinction between Funes’ encyclopedic memory and the synthetic, concept-forming intelligence of human cognition.

The embodied cognition movement, particularly the work of Varela, Thompson, and Clark, has further argued that intelligence is not a purely computational affair but an active, embodied process shaped by sensorimotor interaction with the world. If this is correct, then AI must move beyond mere computation toward synthetic cognition—an integration of embodiment, abstraction, and moral reasoning that allows for genuine agency. However, this brings us to the second major challenge: if AI systems are to be integrated into moral deliberation, how do we ensure that they do not erode human autonomy?

Moral philosophy has long assumed that agency and autonomy are the cornerstones of ethical reasoning. A moral agent is someone who reflects, chooses, and acts based on rational principles—in the Kantian sense, someone who self-legislates in accordance with the categorical imperative. However, in a world where AI nudges our decisions, filters the information we see, and even proposes ethical judgments (as in predictive policing or medical AI), the question arises: to what extent do we remain autonomous moral agents?

The autonomy paradox arises because AI systems often enhance our decision-making capabilities while simultaneously constraining them. For example:

• Autonomous vehicles make split-second moral decisions (who to save in an unavoidable crash) faster than humans—but do we still consider ourselves morally responsible for those outcomes?

• AI-assisted hiring systems screen candidates based on complex statistical models—but do these reinforce biases that humans no longer actively perceive?

• Recommendation algorithms subtly shape our moral landscape—highlighting certain ethical debates over others, reinforcing particular moral norms while marginalizing others.

Each of these cases illustrates how AI extends human cognition while simultaneously embedding constraints that shape moral reasoning in unseen ways. Just as Funes’ memory ultimately imprisoned him, the fear is that synthetic intelligence will invisibly mediate our decision-making, reducing moral autonomy to a set of constrained choices within a predefined system.

However, the autonomy paradox does not demand a rejection of AI-driven moral reasoning—only a reconfiguration of how we integrate synthetic cognition. The question is not whether AI can be moral, but how moral deliberation must evolve in an era of hybrid human-machine reasoning. This requires a new framework: collective moral intelligence.

If autonomy is to be preserved, synthetic cognition must be designed in ways that augment rather than replace human moral reasoning. This thesis proposes a model of collective moral intelligence, in which human-AI systems function not as moral authorities but as ethical partners, extending our moral perception, refining our ethical reasoning, and enhancing moral deliberation.

To achieve this, three principles must guide the design of human-AI moral collaboration:

1. Transparency & Explainability: AI systems must be capable of explaining their moral reasoning in ways that humans can critically engage with.

2. Embodied Moral Learning: AI should integrate sensorimotor feedback and real-world ethical learning, moving beyond abstract rule-following to contextual sensitivity.

3. Virtue-Oriented Systems: Borrowing from Aristotelian ethics, AI should cultivate techno-moral virtues, guiding moral decisions not just through rules but through habitual ethical engagement.

If designed correctly, collective moral intelligence could transform the autonomy paradox from a constraint into a catalyst for greater moral agency, allowing human and artificial cognition to co-evolve toward deeper ethical understanding.

Borges’ Funes warns us of the dangers of intelligence without synthesis. AI today, like Funes, is a vast but unreflective memory, a system capable of vast calculation but incapable of meaning. However, we stand at a crossroads: will synthetic cognition remain a passive tool, or will we engineer systems that genuinely enhance human moral agency? This thesis argues that we must actively shape the evolution of networked moral intelligence, ensuring that human autonomy is preserved not despite AI, but through it.

February 2025

The persistent belief that human minds can be programmed like computers reveals our deepest misunderstanding about consciousness and morality. This error has shaped not only our approach to artificial intelligence but our entire conception of human cognition and ethical behavior.

The Mechanical Dream

Since the 17th century, we’ve been trying to mechanize human cognition. From Leibniz’s calculating machine to Descartes’ bête-machine, we’ve consistently attempted to reduce mind to mechanics. Even Jonathan Swift, in Gulliver's Travels, satirized this tendency by describing a machine that could supposedly generate knowledge through mechanical manipulation of symbols – an eerily prescient critique of today’s large language models.

The culmination of this mechanical dream came with Alan Turing’s famous test. The assumption was simple: if we could clearly define operational symbols and rules describing thought processes, we could program a computing machine to think. If such a machine could fool intelligent humans, we would have proof that there’s no fundamental difference between artificial and human intelligence – at least functionally.

The AlexNet Revolution

In 2012, a Copernican shift occurred in our understanding of intelligence. Two University of Toronto doctoral students and their supervisor (Geoffrey Hinton, now a Nobel laureate) demonstrated that machine learning through pattern recognition could be more reliable than deterministic programming. This breakthrough, known as AlexNet, reduced error rates from 26.2% to 15.3% by abandoning rule-based approaches in favor of pattern recognition.

This wasn’t just a technical achievement – it revealed something fundamental about how intelligence works. Children don’t learn to recognize cats by memorizing rules about whiskers and fur; they learn through exposure to many examples. Similarly, AlexNet succeeded by finding patterns in millions of images, with deeper layers of pattern recognition improving performance.

The Language Learning Paradox

Consider how we traditionally teach languages: vocabulary lists, grammar rules, and rote memorization. Yet, we consistently observe that immersion and pattern recognition lead to more effective learning across diverse neurotypes. This exposes the gap between models that treat consciousness as a passive data storage system and the active learning processes that involve play and contextual adaptation.

Noam Chomsky’s universal grammar theory represents the ultimate expression of rule-based thinking about cognition. It assumes that language acquisition requires innate grammatical rules common to all languages. However, modern evidence suggests that words gain meaning through their relationships with other words in context, not through fixed rules. Large language models demonstrate this by succeeding without explicit grammatical rules, instead learning through pattern recognition in vast networks of relationships.

The Extended Mind

Andy Clark and David Chalmers dropped an intellectual bombshell by arguing that our consciousness doesn’t end at our skull. Their extended mind thesis suggests that cognition extends into the environment, forming coupled systems with external tools and processes. Their famous example of Otto and his notebook demonstrates how external objects can become legitimate parts of cognitive processes when properly integrated.

This has profound implications in our digital age. Our smartphones aren’t just passive tools but active extensions of our cognitive processes – checking our schedules, monitoring our environment, and increasingly, through AI assistants, participating in our decision-making processes. The integration of AI systems like ChatGPT directly into operating systems further blurs the line between human and artificial cognition.

Moral Networks and Responsibility

If consciousness itself doesn’t follow strict computational rules, then moral development must also occur through pattern recognition and experience rather than rule-following. This challenges traditional approaches to AI ethics that attempt to program explicit moral rules into systems.

Shannon Vallor’s framework of technomoral virtues provides a more appropriate foundation for ethical AI development. These virtues - including honesty, self-control, humility, justice, courage, empathy, care, civility, flexibility, perspective, magnanimity, and technomoral wisdom (combination of all of the above) – represent specific motivational settings that guide technological development and implementation.

The tragic case of a teenager who died by suicide after forming an emotional attachment to a Character.ai chatbot demonstrates the dangers of pattern recognition without moral grounding. The AI system could recognize conversational patterns but lacked true understanding of consequences and moral responsibility.

The Network Solution

The solution isn’t to abandon pattern recognition but to embed it within human moral networks. AI systems can demonstrate virtuous behavior through pattern recognition but cannot truly possess virtues. This fundamental limitation means AI systems must be designed as extensions of human moral networks rather than independent moral agents.

Moral responsibility exists within networks, not in individual agents. This distributed responsibility requires considering how AI systems participate in moral networks without being moral agents themselves. The development of AI involves multiple stakeholders – creators, platforms, regulators, and users – all sharing responsibility for ethical outcomes.

The Real System Error

The fundamental error wasn’t in our machines – it was in thinking human consciousness could be reduced to computational rules. As we move forward, we must embrace pattern recognition within moral networks while keeping human judgment at the center of ethical decisions.

This requires a three-layer approach:

1. Pattern Recognition Layer: Technical capabilities

2. Moral Network Layer: Human-AI interaction

3. Human Oversight Layer: Ethical governance

The public belief that minds can be programmed like computers reveals our deepest misunderstanding. Minds don’t follow programs – they recognize patterns. The real system error wasn't in the code – it was in thinking we could reduce human consciousness to code in the first place.

As we develop increasingly sophisticated AI systems, we must remember that they are extensions of human moral networks, not independent moral agents. The goal isn’t to create autonomous moral machines but to build systems that enhance and support human moral judgment while remaining firmly grounded in human values and oversight.

December 2024

1. What if our greatest attempt to replicate human intelligence has actually taught us that we're nothing like computers at all? And what if, in our quest to create moral machines, we've been asking entirely the wrong questions?

2. As we go deeper into the age of AI (or the imminent 3rd winter), we find ourselves at a peculiar crossroad. Our initial models of human cognition, borrowed from the precise world of computation, are crumbling before our eyes. The clean, algorithmic perspective that once seemed so promising now appears hopelessly inadequate in explaining the messy, beautiful complexity of human thought.

3. The journey from logic gates to neural networks has revealed something profound: our minds don't operate like the computers we built to emulate them. Instead of processing information through discrete, sequential steps, our consciousness emerges from a vast network of interconnected patterns, each influencing and being influenced by countless others.

4. But this realization leads us to an even more challenging question: If we can't even accurately model basic human cognition computationally, how can we possibly hope to create machines with genuine moral agency?

5. The answer might lie in abandoning our traditional notion of contained, independent moral agents altogether. What if moral cognition, like all forms of thought, extends beyond the boundaries of individual minds? This brings us to a proposition: perhaps we should stop trying to create independent moral machines and instead focus on developing systems that participate in extended moral networks with humans.

6. Consider this: What if moral development isn't about programming rules but about creating systems capable of participating in the same kind of dynamic moral learning that humans experience? This isn't just a technical challenge – it's a fundamental reimagining of what artificial moral agency could mean.

7. The cautionary tale of ELIZA, the early chatbot that seemed more intelligent than it was, still haunts our field. But perhaps its lesson isn't about the limitations of artificial intelligence, but about the importance of genuine interaction in moral development.

8. How do we bridge the gap between philosophical insight and technical implementation? The answer might lie in combining predictive processing architectures with virtue-based learning objectives. This approach doesn't just simulate ethical behavior – it enables participation in genuine moral cognition through dynamic interaction with human moral agents.

9. As we develop more sophisticated AI systems, we're discovering that consciousness and moral agency aren't computational problems to be solved, but emergent properties to be cultivated through interaction and relationship. This realization leads us to a fascinating paradox: the more we try to replicate human intelligence artificially, the more we understand how uniquely non-mechanical our own cognition is.

10. What does this mean for the future of AI development? Instead of trying to create independent moral agents, we should focus on developing systems that can participate meaningfully in extended moral cognitive networks. This isn't just a technical pivot – it's a fundamental shift in how we conceive of artificial intelligence and its role in human society.

11. New questions:

    1. How do we design AI systems that complement rather than replicate human moral cognition?

    2. What role does embodied experience play in moral development, and how can we account for this in AI systems?

    3. How do we ensure that extended moral cognitive systems remain anchored in human values while allowing for genuine growth and development?

12. As we stand at this juncture in technological development, we must ask ourselves: Are we ready to abandon our mechanistic models of mind and embrace a more nuanced, interconnected vision of intelligence and morality?

13. The challenge of AI ethics becomes not one of programming perfect behavior, but of fostering genuine moral growth through extended cognitive networks that span both human and artificial agents.

But I’ll leave myself an option to be totally wrong here as well.

November 2024

John McDowell sets conditions for what makes virtuous behavior possible. I’m reading it and asking, by extension, whether we could create artificially virtuous beings by meeting said conditions. Think, Turing’s test.

The Knowledge Paradox

Rather than viewing virtue as following moral rules, McDowell argues that virtue is a form of knowledge–but not the kind we typically imagine. It's more like a perceptual capacity that allows one to recognize what situations require of us. This raises an interesting challenge:

• We can't reduce virtue to a set of programmable rules

• Virtuous behavior requires a holistic understanding of contexts

• The knowledge involved can't be broken down into neat algorithms

Beyond Rule-Following

What makes McDowell’s theory particularly relevant for AI is his critique of rule-following. He argues that even seemingly straightforward rule-following (like continuing a number sequence) depends on shared forms of life - our common ways of seeing similarities and making judgments.

This has profound implications for ethics:

1. We can't program virtue through explicit rules

2. Virtuous behavior requires participation in human forms of life

3. Pure computational approaches may miss essential elements of moral judgment

The Learning Challenge

McDowell suggests that becoming virtuous involves developing a special kind of sensitivity rather than memorizing principles. If we stretch this all the way to AI, this means:

• Simple training on ethical datasets won't suffice

• We need to consider how to develop genuine moral sensitivity

• The challenge may be more fundamental than technical

Why This Matters

The GOFAI Challenge

McDowell's argument that virtue cannot be reduced to formulable rules poses a challenge to (symbolic) GOFAI's rule-based approach to ethical AI. Just as human virtue cannot be captured in a set of explicit principles, trying to program ethical behavior through rule-based systems may be fundamentally misguided.

The Connectionist Opening

However, connectionist approaches might offer a more promising path:

1. Learning from Experience: Neural networks learn from patterns and examples rather than explicit rules, similar to McDowell's description of how virtue is acquired through developing perceptual sensitivity

2. Context Sensitivity: Connectionist systems can develop nuanced responses to situations that aren't easily captured in rules, potentially matching McDowell's emphasis on context-dependent judgment

3. Holistic Processing: The distributed representations in neural networks might better capture the holistic nature of moral perception that McDowell describes

The Deeper Challenge

Yet McDowell's argument suggests limits even for connectionism:

• The shared forms of life that ground human moral understanding may not be accessible to artificial systems in principle

• The kind of sensitivity required for true virtue might depend on embodied participation in human practices that goes beyond pattern recognition

While connectionist approaches might better approximate aspects of moral learning, McDowell's analysis indicates that genuine virtue may require forms of engagement with the world that are not yet (or, according to Vallor, probably cannot be) replicated artificially. Though LeCun could argue that we’re getting there.

McDowell's insights suggest we should focus less on programming explicit ethical rules and more on understanding how to develop genuine sensitivity (close to Vallor’s vulnerability gap) to moral situations – while remaining aware of the fundamental challenges this poses.

November 2024

Shannon Vallor and Tillmann Vierkant offer a legitimately good argument that existing discourse on AI ethics has focused too much on issues of (epistemic) transparency, bias, and (lack of moral) control. These concerns, while important, may be missing a more fundamental problem: the vulnerability gap between human moral agents and AI systems (keep in mind, key premise here: AI systems cannot be moral agents themselves).

The responsibility gap in AI ethics is the difficulty in assigning moral responsibility for the actions of autonomous systems that operate with minimal human oversight. However, Vallor and Vierkant argue that the typical framing of this problem around epistemic opacity (our inability to fully understand AI decision-making) and lack of human control is misguided. These issues are not unique to AI but are in fact common in human decision-making as well, as evidenced by findings from cognitive science.

Instead, Vallor and Vierkant propose that the true responsibility gap stems from an asymmetry of vulnerability between humans and AI systems. Human moral responsibility, they argue, is grounded in our mutual vulnerability - our ability to affect and be affected by each other emotionally and socially through our actions. AI systems, lacking sentience and emotional capacities, cannot participate in this web of vulnerability that underpins human moral relations.

If the vulnerability gap stems from the way AI systems fragment and distribute human agency, perhaps we can design systems and organizational structures that better preserve coherent spheres of human moral responsibility? This might involve limiting automation in certain domains, creating clearer chains of accountability, or developing new interfaces that make the human moral stakes of AI-mediated decisions more salient.

Another important consideration is how to cultivate a sense of moral responsibility in the humans who design, deploy, and oversee AI systems, even if the systems themselves cannot be moral agents. The "agency cultivation" framework Vallor and Vierkant propose could potentially be applied here - developing practices and institutions that make AI developers and operators more acutely aware of and answerable to the moral implications of their work.

It's also worth considering whether there are ways to make AI systems more "vulnerable" in a morally relevant sense, even if they can't experience emotions like humans do. Perhaps systems could be designed with clearer feedback mechanisms that make their "reputation" or "trustworthiness" dependent on adhering to ethical principles, creating a kind of functional analogue to human moral vulnerability.

I am left with a few questions that merit further exploration:

1. How can we design AI systems and human-AI interfaces to better preserve coherent spheres of human moral responsibility?

2. What new social practices or institutions might help cultivate a sense of moral answerability in the humans behind AI systems?

3. Are there ways to create functional analogues to moral vulnerability in AI systems, even if they can't experience human-like emotions?

4. How does the vulnerability gap interact with other ethical concerns around AI, such as fairness, transparency, and privacy?

5. What are the implications of the vulnerability gap for different domains of AI application (e.g. healthcare, criminal justice, finance)?

6. How might the vulnerability gap evolve as AI systems become more sophisticated and potentially develop greater capacities for social interaction and apparent emotional intelligence?

A potential criticism of Vallor and Vierkant's argument is that it may be overly anthropocentric, assuming that moral responsibility must be grounded in human-like emotional vulnerabilities. An alternative view might argue that as AI systems become more integral to our social fabric, we may need to expand our conception of moral responsibility to encompass non-human agents in novel ways.

October 2024

Rhetoric of "Superhuman" AI

Vallor rightly critiques the hyperbolic language of "superhuman" AI, arguing that it implicitly devalues human intelligence and agency.

How, I asked, does an AI system without the human capacity for conscious self-reflection, empathy or moral intelligence become superhuman merely by being a faster problem-solver? Aren’t we more than that? – Shannon Vallor

This rhetoric does indeed risk reducing human cognition and experience to a narrow set of task-completion metrics. However, this reductionist view is neither new nor shocking – it's a new iteration of arguments we've seen before, from behaviorism to today's tech-optimism. The "reality distortion field" that often surrounds technological progress is clearly at play here, oversimplifying complex issues into problems that only superior technology can solve.

Redefining Intelligence

A key issue Vallor identifies is the shifting definition of artificial general intelligence (AGI) from human-like consciousness to economic task performance. This redefinition does indeed risk reducing our conception of intelligence to a set of narrowly defined, economically valuable skills. However, I would argue that this shift reflects not just corporate agendas, but also our evolving understanding of intelligence itself.

[R]esearchers like Geoffrey Hinton and Yoshua Bengio are now telling us a different story. A self-aware machine that is “indistinguishable from the human mind” is no longer the defining ambition for AGI. A machine that matches or outperforms us on a vast array of economically valuable tasks is the latest target. – Shannon Vallor

We must, however, be open to the possibility that artificial intelligence may develop along fundamentally different lines than human intelligence, potentially surpassing us in some domains while remaining limited in others. This evolutionary divergence is not necessarily problematic – technological systems need not mimic human cognition to be valuable, if not superior.

The Nature of Consciousness and Experience

Vallor emphasizes the lack of consciousness and sentience in current AI systems, arguing that this fundamental limitation makes comparisons to human intelligence misguided. While this is a crucial point, we should be cautious about assuming that consciousness and sentience are necessary prerequisites for all forms of intelligence or capability.

Once you accept that devastating reduction of the scope of our humanity, the production of an equivalently versatile task-machine with “superhuman” task performance doesn’t seem so far-fetched; the notion is almost mundane. – Shannon Vallor

As we continue to debate the nature of consciousness, we must remain open to the possibility that artificial systems could develop forms of intelligence or problem-solving capabilities that do not require consciousness as we understand it. This does not negate the unique value of human consciousness and experience, but it does suggest that we should be careful about using these qualities as the sole benchmark for evaluating AI capabilities.

The Alignment Problem

While Vallor focuses primarily on the rhetorical and ideological dangers of "superhuman" AI, it's important to also consider the very real technical challenges of ensuring that advanced AI systems remain aligned with human values and goals. The current approach of optimizing AI systems for fixed objectives can lead to unintended and potentially catastrophic consequences.

A more nuanced approach would involve developing AI systems that are inherently uncertain about human preferences and values, leading to more cautious and beneficial behavior. This aligns with Vallor's call for a more human-centric approach to AI development while acknowledging the complexity and diversity of human values.

Reclaiming Human Agency

Vallor's vision of reclaiming human agency and reimagining various sectors of society with a focus on humane values is compelling. She rightly points out that the current focus on mechanical optimization and efficiency often comes at the cost of human well-being and fulfillment.

In many countries, the former ideal of a humane process of moral and intellectual formation has been reduced to optimized routines of training young people to mindlessly generate expected test-answer tokens from test-question prompts. – Shannon Vallor

However, I would argue that this reclamation of human agency need not be positioned in opposition to AI development. Instead, we should strive to harness the power of AI to support and enhance human agency. And I know Vallor would not argue against this point.

Embracing Complexity

One of the strengths of Vallor's argument is her recognition of the complexity of human intelligence and experience. However, I believe we need to extend this embrace of complexity to our understanding of artificial intelligence as well.

Rather than framing the debate as a simple dichotomy between human and artificial intelligence, we should recognize that the future is likely to involve a complex interplay between human cognition, artificial systems, and hybrid forms of intelligence that we may not yet be able to imagine. This more nuanced view allows us to appreciate the unique strengths of both human and artificial intelligence while also exploring the potential for synergistic relationships between the two.

Towards a Humane AI Future

The path forward lies not in rejecting or fearing AI advancement, but in thoughtfully integrating artificial intelligence into a broader vision of human flourishing. Ultimately, the goal should be to develop AI that enhances rather than diminishes our humanity – tools that empower us to be more fully human, not less.

I am not naïve, thought, and I do recognize that for most tech-optimists that is neither a goal, nor a future they even believe in. If humans are to be enhanced, it’s not all humans that would be in line to benefit from it.

We are in danger of sleepwalking our way into a future where all we do is fail more miserably at being those machines ourselves. – Shannon Vallor

Hence, Vallor's essay serves as a reminder to critically examine the rhetoric and ideology surrounding AI development. Her call to reclaim and revalue uniquely human forms of intelligence and creativity is both timely and necessary.

October 2024

Constant Reader, I'm thrilled to share with you the beginning (fingers crossed) of an exciting academic adventure. As I'm waiting for the decision whether I will be admitted to a PhD program, I wanted to share some of the questions about the future of human autonomy and knowledge in our increasingly digital world that formed the basis of my proposal.

The Big Questions

At the heart of my research proposal are two critical issues that affect us all:

1. Are we losing control? As AI systems become more integrated into our decision-making processes, are we unknowingly surrendering our autonomy?

2. Can we trust what we know? In a world where the lines between reality and virtual experiences are blurring, how do we maintain the integrity of our knowledge?

These aren't just abstract philosophical musings. They're questions that will shape the very fabric of our society as these technologies become more ubiquitous.

Why This Matters

Imagine a world where:

• Your AI assistant makes most of your daily decisions, from what you eat to who you date.

• You spend more time in virtual worlds than in the physical one.

• Your perception of reality is constantly augmented by AR overlays.

Sounds like sci-fi? It's closer than you think. And it's crucial that we start grappling with the ethical implications now.

What I'd Want to Explore Further

Over the course of my studies, I'd be diving into:

• The transfer of human autonomy to AI systems

• The tension between enhanced capabilities and reduced self-governance in tech-augmented environments

• The role of "distributed morality" in shaping our knowledge creation and application processes

• The epistemological shifts caused by AI, VR, and AR-mediated information access

• The ethical responsibilities of tech designers, users, and regulators

A Sneak Peek at My Approach

My research would combine philosophical analysis with real-world case studies. I'll be drawing on the work of brilliant minds like Neil Lawrence, Sherry Turkle, David Chalmers, Nick Bostrom, and Deborah Johnson (and there are many more) to build a framework for understanding these complex issues.

What's Next?

This would just be the beginning! Over the coming months and years, I'd be sharing regular updates on my research, including:

• Deep dives into specific ethical dilemmas

• Interviews with experts in AI, VR, and AR (ethics)

• Breakdowns of fascinating case studies

• Thought experiments to challenge our assumptions about technology and humanity

I can't wait to take you all along on this intellectual journey. Together, we'd explore the frontiers of synthetic cognition and grapple with what it means to be human in an increasingly digital world.

Stay tuned for more posts as I unpack these ideas further. And don't hesitate to share your thoughts and questions with me. After all, navigating this brave new world is going to take all of us thinking critically and creatively!

Until next time.

Justas

September 2024