Written by Dr Caroline Emmer De Albuquerque Green (Director of Research, Institute for Ethics in AI), Donald Macaskill (CEO of Scottish Care), Anna Steffeney (Executive Director FamTech.org)
In June 2026, OpenAI released its Industrial Policy for the Intelligence Age [1], recommending pathways for displaced workers into human-centred work, particularly the care and connection economy. In January, Anthropic published its constitution, in which the company—valued at $965 billion [2]—announced it wants its Large Language Model, Claude, to have good personal values that follow rules of genuine care, attends to users' wellbeing, autonomy and dignity [3]. Other tech companies are building "care platforms," "digital companions," "wellbeing assistants" and "personalised AI."
What is happening is that these companies and their products are learning to speak the language of care, one of deep moral and relational value. It borrows the words that are part of caring human relationships, necessary for people to survive and thrive. Vocabulary of care evokes feelings of authority, virtue, vulnerability, dependency and trust. Positive caring relationships come with responsibility, accountability and humility, when actions often speak much louder than words.
The question is whether by adopting care language, companies are genuinely concerned about their role in society and improving their products, or are they instead quietly taking on the moral authority of caregiving but are primarily serving their own needs and interests? And how is technology reframing the meaning of ‘care’, something so essential to our lives and communities?
As humanity is still working out how to deal with the growing role of AI in our societies, we should be acutely aware and concerned about this development and actively engage with this question. Here are three reasons why:
Care-washing
The first concern is around ‘care-washing’ of a corporate image and products that sound like they are concerned with caring values like users’ wellbeing, dignity and autonomy, but in fact disguise unethical and ‘uncaring’ actions that serve the companies but not the users. The terminology does important work. It makes the new seem ethically assured. But words do not guarantee reality.
What could such uncaring actions entail?
We have already witnessed the devastating consequences of anthropomorphism and sycophantic conversational AI systems on humans, especially vulnerable groups [4]. Such behaviour serves tech companies, who want people to keep using their products. Language of care in AI systems without the necessary safeguards to spot unhealthy relationships with AIs or to detect a crisis can lead to human overreliance instead of ensuring that people get the right support at the right time. To discourage the use of such systems is often the more caring thing to do, rather than trying to keep people engaged with the technology as long and as much as possible.
Furthermore, the more people use these systems, the more personal data is extracted from them. This matters especially in care contexts, such as in clinics or care homes: Patterns of movement, speech, behaviour and health are collected. This data is valuable to tech companies, it is how they grow their wealth and develop their systems.
In other fields, this has previously been described as a new form of colonialism, where data becomes a resource to be extracted, analysed, and monetised, often by large, remote organisations [5]. In care, the ethical stakes are even higher because what is being captured is not just information, but vulnerability itself. And yet this process too is softened by language. It is framed as support, as enhancement, as necessary progress but too often without the full transparency and capability for people to opt out.
These risks are not incidental. They cluster into recognisable categories like dependency engineering, data sensitivity, surveillance, workforce displacement, dignity and autonomy. Naming them is the first step toward governing them.
The quiet reframing of what care means
The second concern is more fundamental to the meaning of care. Much of what is described as care in AI refers to powerful and sophisticated tools that can ‘speak care language’ and assist in self-care and the care of others, mostly through predicting patterns. But there is so much more to care than words or prediction. AI systems do not feel, they do not relate, and they do not care in the human sense, even if some people may contest this. Anyone who has given or received care knows that it is more than a function or practice. It is a relationship. It exists in presence, in trust, in the small, often unseen acts of recognition between individuals. No technology can replicate that.
This matters because once we begin to accept that technology can “care”, we begin quietly to lower our expectations of what care is. The threshold shifts. The absence of human interaction becomes more acceptable. The substitution of presence with simulation becomes thinkable. This is a point that particularly people who are drawing on long-term care and support are concerned about: The quiet stripping away of positive human caring relationships.
This is not to say that technology has no place in caring domains. On the contrary, it has an important role to play. Used well, it can support professionals, enable independence, and extend reach. But it must remain a tool. It must not become the definition.
The hidden costs for caregivers
While care and caregiving may hold great moral value, caregivers (a great proportion of whom are female) have traditionally been underserved and undervalued in the societies in which these companies are based. People who care for children, the sick and old have mostly been the powerless.
It is therefore somewhat ironic that some of the most powerful, male dominated companies in the world are now adopting the moral authority and language of a group of people who have so long been excluded from the political sphere and underserved by policy and business.
The consequence may be a further entrenchment of power structures, with companies and governments adopting a moral stance of being caring and caring about caring, but in fact not doing anything for caregivers.
Furthermore, this may also mean that caregivers will have even less resources allocated to them, with AI systems seen as a panacea to serve their complex support needs that cannot be fully addressed by technology alone but requires investment on multiple levels.
Of course, it can be argued that this language-taking might bring greater visibility to the value of caregiving. Labour economists and demographers broadly agree that care sectors will account for a growing proportion of employment in coming decades, particularly as populations age and automation displaces office-based work.
It is also true that AI can support this underserved population, to enhance their quality of life and wellbeing with new products.
But this will only be the case if products are built not to primarily serve the needs of tech companies, but the needs of people including caregivers and fundamental values. That condition requires more than goodwill. It requires minimum standards, independent oversight, and governance frameworks that place caregiving communities rather than technology companies at the centre of design and accountability.
The way forward
The three concerns outlined above point to a set of risks that are distinct enough to require their own governance framework, one grounded in the lived realities of caregiving relationships rather than adapted from general-purpose AI ethics guidance. General principles are not sufficient. What is needed are named risks, minimum standards, and enforceable accountability. Developments like the shift in the meaning of care often happen quietly, without anyone noticing until it is too late. The taking of the language of care by tech companies must not happen quietly.
Rather, it needs to be discussed and critically evaluated, in the open, with the involvement of the public and caregiving communities. Some frontier labs are have called for exactly this kind of engagement described as ‘mechanisms for public input’ [6] and calls on developers to publish model specifications describing how systems are intended to behave. These proposals are welcome. But broad consultation without care-specific risk frameworks will default to the priorities of the most resourced stakeholders. Caregiving communities need structured instruments through which to participate, not just an open invitation. That is what a governance framework grounded in the realities of care relationships can provide.
At the Institute for Ethics in AI at Oxford University, we are actively working to support such processes and find workable solutions. Our activities include:
- The Caregiving AI Responsibility Initiative (CARI) [7]: A joint initiative between the Accelerator Fellowship Programme at the Institute and FamTech.org, the technology hub for the care economy in the US. CARI is developing a practical governance framework specifically for frontier AI in caregiving contexts and translates these into minimum risk mitigation standards that technology companies building in this space can adopt and be held accountable to.
- The ‘AI Alliance in Care [8]’: A community of good practice for the responsible use of AI in care, which was born out of the ‘Oxford Project on the responsible use of generative AI in adult social care [9]’. This is based on principles of co-production, creating governance recommendations for tech companies, care providers [10] and fostering communications across the care community and beyond.
- Civic AI [11]: Together with Ambassador Audrey Tang, we developed a socio-technical approach that we call ‘Civic AI’, based on political theorist Joan Tronto’s ethics of care. Civic AI proposes principles for how AI can play a role in caring societies, with mechanisms to direct AI to truly serve the needs of people. In practice, this means designing AI systems that are attentive, responsive, and accountable to the people they serve, rather than optimised primarily for engagement, efficiency, or commercial return.
Finally, governments and regulators must start to take a stronger role in the oversight of ‘caring AI’ and this is especially true for regulators of health and social care services. AI will likely be able to serve caregiving communities. But rather than taking their language, they should be at the core of AI and social innovation to support them.
-
Footnotes:
[1] https://openai.com/index/industrial-policy-for-the-intelligence-age/ p.8
[2] https://www.anthropic.com/news/series-h
[3] https://www.anthropic.com/constitution
[4] Dohnány, S., Kurth-Nelson, Z., Spens, E. et al. Technological folie à deux: feedback loops between AI chatbots and mental health. Nat. Mental Health 4, 336–345 (2026). https://doi.org/10.1038/s44220-026-00595-8
[5] E.g Nick Couldry and Ulises A. Mejias, The Costs of Connection: How Data Is Colonizing Human Life and Appropriating It for Capitalism (Stanford, CA: Stanford University Press, 2019).
[6] https://openai.com/index/how-should-ai-systems-behave/
[7] https://famtech.org/resources/institute-ethics-ai-and-famtechorg-announce-new-collaboration-responsible-ai-caregiving/
[8] https://www.digitalcarehub.co.uk/ai-and-robotics/ai-in-care-alliance/
[9] Emmer De Albuquerque Green C, Reinmund T, Hamblin K et al.
Responsible use of artificial intelligence in the provision of long-term care for older people: a care-centric approach The Lancet Healthy Longevity, 2026; 7
[10] https://www.digitalcarehub.co.uk/ai-and-robotics/oxford-project-the-responsible-use-of-generative-ai-in-social-care/
[11] https://civic.ai/
-
Suggested citation: Dr Caroline Emmer De Albuquerque Green, Donald Macaskill and Anna Steffeney, ‘AI companies are adopting “care language”: Three reasons why we should be concerned’, (15 July 2026) AI Ethics at Oxford Blog; https://www.oxford-aiethics.ox.ac.uk/blog/ai-companies-are-adopting-care-language-three-reasons-why-we-should-be-concerned