AI Welfare: Anthropic’s New Hire Fuels Ongoing Ethical Debate

As fears over AI outpacing human control grow, Anthropic, an AI company, has turned its attention to a new concern: chatbot welfare.

In a new move, the company has hired Kyle Fish to research and protect the “interests” of AI systems, as first reported today by Business Insider. Fish’s role includes pondering profound questions such as what qualifies an AI system for moral consideration and how its “rights” might evolve.

The rapid evolution of AI has raised ethical questions once confined to science fiction. If AI systems develop human-like thinking, could they also experience subjective emotions or suffering?

A group of philosophers and scientists argues that these questions demand attention. In a recent preprint report on arXiv, researchers called for AI companies to assess systems for consciousness and decision-making capabilities, while outlining policies to manage such scenarios.

Failing to recognize a conscious AI, the report suggests, could result in neglect or harm to the system. Anil Seth, a consciousness researcher, says that while conscious AI may seem far-fetched, ignoring its possibility could lead to severe consequences, as reported by Nature.

“The problem wasn’t that Frankenstein’s creature came to life; it was that it was conscious and could feel,” Seth argued in Nautilus.

Critics, however, find AI welfare concerns premature. Today’s AI already inflicts harm by spreading disinformation, aiding in warfare, and denying essential services.

Yale anthropologist Lisa Messeri challenges Anthropic’s priorities: “If Anthropic — not a random philosopher or researcher, but Anthropic the company — wants us to take AI welfare seriously, show us you’re taking human welfare seriously,” as reported by Buisness Insider

Supporters of AI welfare contend that preparing for sentient AI now could prevent future ethical crises.

Jonathan Mason, an Oxford mathematician, argues that understanding AI consciousness is critical. “It wouldn’t be sensible to get society to invest so much in something and become so reliant on something that we knew so little about — that we didn’t even realize that it had perception,” as reported by Nature.

While skeptics warn against diverting resources from human needs, proponents believe AI welfare is at a “transitional moment,” as noted by Nature.

Business Insider reports that Fish did not respond to requests for comment regarding his new role. However, they note that on an online forum focused on concerns about an AI-driven future, he expressed a desire to be kind to robots.

Fisher underscores the moral and practical importance of treating AI systems ethically, anticipating future public concerns.

He advocates for a cautious approach to scaling AI welfare resources, suggesting around 5% of AI safety resources be allocated initially while stressing the need for thorough evaluation before any further expansion.

Fisher sees AI welfare as a crucial component of the broader challenge of ensuring that transformative AI contributes to a positive future, rather than an isolated issue.

As AI systems grow more advanced, the concern extends beyond their potential rights and suffering to the dangers they may pose. Malicious actors could exploit AI technologies to create sophisticated malware, making it more challenging for humans to detect and control.

If AI systems are given moral consideration and protection, this could lead to further ethical complexities regarding the use of AI in cyberattacks.

As AI becomes capable of generating self-learning and adaptive malware, the need to protect both AI and human systems from misuse becomes more urgent, requiring a balance between safety and development.

Whether an inflection point or misplaced priority, the debate underscores AI’s complex and evolving role in society.