§1. Capability arguments about the limitations of LLMs/AI aim to make the case that AI systems cannot in-principle do something that humans generally can. Most prove embarrassingly short-sighted—between 2024 and 2025 refutations were sometimes a single frontier model release away—and are best read as retreats into what Leif Weatherby calls remainder humanism. The stronger variants suggest that there is something about embodiment that is necessary for human-parity (or as is the trend: human-surpassing) in an important set of abilities. It's an open question about whether this embodiment requirement can be realised in non-biological substrates.
§2. A relatively under-discussed variant finds its basis in the realm of the social, and is, regrettably, best captured in this principle from a 1979 IBM training manual:
A computer can never be held accountable; therefore, a computer must never make a management decision.
I'm not sure much attention has been given to this kind of argument in popular/public fora. It's not as exciting as questions about whether LLMs can conduct original scientific research or write a literary masterpiece, but it's certainly at least as pressing: there are many cases out in the world today where an answer has been assumed one way or the other without too much examination. This past week, for example, Anthropic introduced Claude Tag, which is roughly a single instance of Claude (plus associated machinery) that is shared across a company and is contactable via Slack, or a similar team-messaging platform. Can Claude in Claude Tag form be held accountable for its actions? Probably not; but why not—and is it a substrate-dependent 'not'?
(There's probably some interesting—or at least relevant—legal discourse on this subject taking place in court cases dealing with chatbot advice leading to death or harm.)
§3. It'll help to look at the normative structure of the sort of practice the IBM manual is concerned with. When we make a decision that affects a group of people with whom we have some relationship, we implicitly pick up—or rather are implicitly attributed by the group—some normative statuses, e.g., a commitment to the justifiability of the decision. The nature of these statuses depends on the type of relationship we have with the group—in this case, we might have some professional obligations or responsibilities towards them.
What makes it possible for us to bear these statuses, or indeed have these relationships? Is it that we have a particular set of traits, like a responsiveness to reasons, or the desire to maintain a certain social standing, or the capacity for empathy? We already have a sufficiently reasonable Claude, can we not imagine a Claude that cares, in some sense, for its social reputation; or a Claude that is human-aligned enough to act empathetically? Or—and this is not necessarily independent from trait-accounting, but might be if taken as irreducible—is it simply that we recognise one another as belonging to the human form of life?
§4. We've returned to a discussion of capabilities. Here too then our inquiry might bottom out in questions about substrate-dependence. But I hope you can somewhat see now that the role of this inquiry, or at least its implications, are at least as significant as those of the permanent-underclass discourse, and certainly more interesting than those of remainder humanism talk, because its end is ostensibly the (re-)definition of the we-concept, and thus the delineation of our form of life.