Automation and Autonomy

04 Nov, 2026

The Economics of Automation

The economic promise of LLM-based AI systems seems to be based almost entirely on the idea that they will allow us to automate processes which up to now have required slow and expensive humans to deliver them. The hope is that this will have a similar economic impact as the Industrial Revolution.

I suspect there is more optimism than realism in the economics. However, what I want to focus on here is the very idea of automation and what that means if the dream comes true of AI automating vast numbers of processes we engage with on a day to day basis. The human activity which was automated during the Industrial Revolution was almost exclusively activity where the human was interacting with material objects. The more recent Computational Revolution also saw automation of human activity, which was not an interaction with material objects but with informational objects, with items of information and data. Digital computers and the Internet replaced much human activity in the processing and communication of information. There is a good reason it is called IT or ICT.

What the current generation of AI, largely driven by LLMs, is offering is the possibility of automating a sphere of human activity where those humans are interacting with other humans. The most familiar cases of this are the automation of financial decisions, from welfare payments to loans, of customer service, clinical assessment, and even of teaching.

There are also uses to further automate the information and communication processes that had already to a large degree been automated by the computational revolution. However, it is unclear what the economic benefits of this further automation are, because it is rapidly producing an absurd situation where humans are cut completely out of the system. A brief prompt is expanded by the sender's AI into a 500 word email, which is then summarized back into a single sentence or two by the recipient's AI. Or a student writes an essay with AI, which the teacher marks with AI. No one could reasonably believe that such automation is of economic value.¹

So we can reasonably assume that the uses of AI which will actually have a chance of producing the economic benefits which are promised are cases where we automate the human activity of making a decision on behalf of some organization about some particular human. This may be what to give or deny them, what to advise them or what to teach them.

Categorisation

The automation of the human interaction with or production of material objects during the Industrial Revolution produced uniformity of output. Handcrafting can produce highly uniform outputs, but that is the decision made on an item by item basis. Automation defaults to uniform output and variation only comes from setting up a new uniform process.

The reason for this is simple: Automation involves doing the same thing again and again.

So what happens when we automate human activity which involves interacting with other humans? Obviously, if we wanted to treat each and every person in exactly the same way, we wouldn't need AI for that. Equally obviously, if we wanted to treat people in arbitrarily different ways, we wouldn't need AI, just some dice. What we are automating here are processes which seek to treat people differently upon the basis of differences between them.

The output of the process, the decision, involves selection between a range of options. The input to the process is information about the person which is deemed to be relevant to the decision. When a human is making the decision, what they're looking for is information about the individual whose fate they are deciding, which will justify one of the range of options they have available.

Of course, since the human decision-maker is making the decision on behalf of some institution or organisation, that institution or organisation will likely have set some rules for that human decision-maker about what evidence is necessary for selecting certain of the available options. They may also structure the employment context so that the human decision maker is strongly disincentivised from challenging those rules when they seem to result in a decision that cannot be justified. What is ineliminable from the human process is that the decision-maker is sensitive to the question of whether the information provided really justifies the decision or not.

Automation fundamentally changes this. The information about the individual is not used to justify a decision at all. Rather, it is used to assign the individual to a category, and then each category is paired to a decision output. It follows that the only way you can interact with such an automation is as an instance of a general category. There is no possibility of being treated as an individual, only as a typical, or stereotypical, member of some group.

Personalisation

One response to the last point is to argue that in fact we are facing the opposite problem: extreme personalisation (also often driven by 'AI' algorithms). Social media, news, advertising, prices and offers: everything is tailored to you based on sophisticated analysis of data about the individual. Does this not suggest that AI driven automation of human activities which involve engaging with other humans can also be personalised and thus be tailored to the individual?

Let us set aside the amount of data that needs to be extracted about a person and their lives in order to achieve this personalisation and the question of whether you would really want the hospital or social security or insurance company or bank to have all that data. Instead let us distinguish between:

1. Personalisation
2. Treating someone as an individual

Consider the way that we personalise our phones with wallpapers, homescreen layouts, a particular set of apps. This may turn out to be unique to you, but what has actually happened is that there is a very large range of variables determining how your phone can be configured and you have selected values for many of them. The more you select away from their defaults, the more likely your particular configuration of personalisation options is to be unique. This does not change the fact that you are being treated as an instance of a category, it just means that you are being treated as an instance of 100s or 1000s of categories.

We get a similar situation, though rather more obvious, with the clinical triage software used by some primary care providers: as the patient ploughs through pages and pages selecting the options that seem closest to their symptoms they are brutally aware that they are being lined up to be dropped into a pigeonhole. All in the hope that rather than an automated text message with advice ('take some paracetamol and rest') they will get into the pigeonhole which creates a human interaction. The number of categories does not make this any less a case of having your individuality - as a patient - converted into an instance of those categories.

To treat someone as an individual is different in that it starts from respecting their self-representation and only comes to placing them into an existing category if that does no harm to their interests, as they express them. This is something which it is impossible to automate because it is essentially a process of treating each person as a separate case - it stands to automation as handcrafting stands to industrialisation.

AI and Autonomy

Sometimes the automation of activities which are interactions with humans has significant benefits and few, if any, harms. We do not need to be treated as an individual all the time and often we benefit from the efficiencies of being treated as an instance of a category. And if greater computational complexity allows greater categorial complexity, all the better.

My worry is that if we push AI-driven automation to the extremes that are necessary to generate the economic benefits required to justify the investment, then we will be forced to settle for personalisation instead of individuality in every aspect of our lives.

This strikes me as a really fundamental loss of autonomy, because the categories used in automation are always chosen by someone else with a view to their interests - usually profit - not ours. A future in which LLMs automate almost all interactions between institutions or corporations and individual humans is a world where we can, at best, choose which stereotypes we live under.

Even worse, those stereotypes we are forced to live under will also be imposed on our personal relationships, as LLMs become intermediaries in (digital) person to person interactions, by drafting messages, editing images, summarising group chats etc.

Perhaps that is what many people want or perhaps they think it is a reasonable trade-off for convenience. But I don't want it and nor do I find treating someone as an individual to be inconvenient. Join me. Resist AI.

Ending with a poem: Not by Erin Hanson

You are not your age, nor the size of clothes you wear,
You are not a weight, or the colour of your hair.
You are not your name, or the dimples in your cheeks.
You are all the books you read, and all the words you speak.
You are your croaky morning voice, and the smiles you try to hide.
You’re the sweetness in your laughter, and every tear you've cried.
You're the songs you sing so loudly when you know you're all alone.
You're the places that you've been to, and the one that you call home.
You're the things that you believe in, and the people whom you love.
You're the photos in your bedroom, and the future you dream of.
You're made of so much beauty, but it seems that you forgot
When you decided that you were defined by all the things you're not.