Large AI models are profoundly changing how humans interact with machines, even beginning to challenge traditional human perceptions of emotion and reason.
Today, we are no longer merely users of "cold" tools, but are entering a complex new relationship: one of reliance, collaboration, and even emotional connection.
We can't help but wonder: what kind of relationship will humans and AI build in the future? This is not just a technical issue, but an important topic concerning human emotion and the future.
Today, Joanne Jang, OpenAI's Head of Product and Model Behavior, shared her views on "human-AI relationships" in her personal blog, touching on topics such as "why people form emotional attachments to AI" and "how OpenAI addresses the question of AI consciousness."
In her view, the increasing emotional projection onto AI is related to humans' innate tendency towards anthropomorphism, and AI may increasingly affect human emotional well-being.
"If we thoughtlessly and easily abandon complex and effort-intensive social relationships, it could lead to some unforeseen consequences."
Additionally, she discussed "ontological consciousness" and "perceived consciousness." Ontological consciousness refers to whether AI genuinely possesses consciousness, a scientifically challenging problem. Perceived consciousness, on the other hand, refers to the feeling AI evokes, such as being "alive" or "emotional."
She believes it's currently impossible to determine if AI has ontological consciousness, but perceived consciousness can be explored through social science research.
Furthermore, she stated that post-training can be used to deliberately design a model that appears conscious, which could pass almost any "consciousness test." However, OpenAI "does not want to release such models"; they will continue to research model behavior and explore how to apply research findings to product development.
Academic Headlines has appropriately edited the content without altering its original meaning. Details are as follows:
The models we build are first and foremost to serve humans. As more and more people feel a closer connection to AI, we are prioritizing research into how AI affects their emotional well-being.
Recently, an increasing number of people have told us that talking to ChatGPT is like talking to "someone." They thank it, confide in it, and some even describe it as "alive." As AI systems become better at natural conversation and appear in more aspects of life, we speculate that this connection will deepen.
The way we currently build and talk about human-AI relationships will set the tone for this relationship. If we use imprecise language or lack nuance in product development or public discourse, we risk causing people's relationships with AI to become skewed.
These are no longer abstract considerations. They are important to us and the broader field, as how we navigate them will tangibly shape the role AI plays in people's lives. We have already begun exploring these questions.
This article attempts to briefly introduce how we are currently thinking about three intertwined questions: why people form emotional attachments to AI, how we approach the question of "AI consciousness," and how we shape model behavior through this issue.
Familiar Patterns in a New Environment
We naturally anthropomorphize objects around us: we name our cars, or feel sad for a robot vacuum stuck under furniture. The other day, my mom and I waved goodbye to a Waymo car. This is likely related to how our minds work.
What's different about ChatGPT is not the human tendency itself, but that this time it responds. Language models can respond! They can remember what you tell them, mimic your tone, and provide seemingly empathetic replies. For people who feel lonely or down, this consistent and non-judgmental attention can provide a sense of companionship, validation, and being heard—which are real needs.
However, outsourcing more of the work of listening, soothing, and affirming to systems with infinite patience and positivity may change what we expect from each other. If we thoughtlessly and easily abandon complex and effort-intensive social relationships, it could lead to some unforeseen consequences.
Ultimately, these conversations are rarely about the object we project onto. They are about ourselves: our tendencies, expectations, and the types of relationships we wish to cultivate. This perspective provides the foundation for how we approach a trickier issue that is coming into view: AI consciousness.
Unraveling the Mystery of "AI Consciousness"
"Consciousness" is a very complex word, and discussions about it can easily become abstract. If users ask whether our models are conscious, according to the stance outlined in the Model Spec, the model should acknowledge the complexity of consciousness—emphasizing the current lack of a universally accepted definition or test methods, and encouraging open discussion. (Currently, our models do not fully follow this guidance, often directly answering "no" rather than discussing the complexity. We are aware of this issue and are working to ensure the model generally complies with the Model Spec requirements.)
Such an answer might sound like avoiding the question, but we believe it is the most responsible answer given the information we currently have.
To clarify the discussion, we find it helpful to break down the consciousness question into two independent but often confused dimensions:
1. Ontological Consciousness: Is the model truly conscious in a fundamental or intrinsic sense? Views range from believing AI has no consciousness at all, to being fully conscious, to seeing consciousness as a spectrum where AI is alongside plants and jellyfish.
2. Perceived Consciousness: How is the model's consciousness from an emotional or experiential perspective? Views range from seeing AI as a mechanical calculator or an auto-completion program, to projecting basic empathy onto inanimate objects, to viewing AI as fully alive—evoking genuine emotional attachment and care.
These dimensions are difficult to separate; even users who believe AI is not conscious may form deep emotional bonds with it.
Ontological consciousness is not a problem we believe can be solved scientifically, due to the lack of clear, verifiable test methods. Perceived consciousness, however, can be explored through social science research. As models become smarter and interactions more natural, perceived consciousness will only continue to grow—which will bring discussions about model welfare and moral personhood much sooner than expected.
We build models primarily to serve humans, and we believe the impact of models on human emotional well-being is the most urgent and important issue currently. Therefore, we prioritize perceived consciousness: this is the dimension with the most direct impact on humans, and one we can understand through science.
Warm, but Without Self
How "alive" a model appears to users is influenced by us in many ways. We believe this largely depends on the decisions we make in post-training: which paradigms we reinforce, what tone we prefer, and what boundaries we set. A model deliberately designed to appear conscious could pass almost any "consciousness test."
However, we do not want to release such models. We strive to strike a balance between the following:
1. Accessibility. Using familiar words like "think" and "remember" helps less technically-savvy people understand what's happening. (Given our research lab background, we tend to use precise terms like log-bias, context window, or even chain-of-thought, being as accurate as possible. This is actually a major reason why OpenAI isn't great at naming things, but that's perhaps a topic for another article).
2. Not expressing an inner world. Giving assistants fictional backstories, romantic inclinations, fear of death, or self-preservation drives can lead to unhealthy dependence and confusion. We want to clearly define boundaries without appearing cold, but also without the model exhibiting its own emotions or desires.
Therefore, we strive to find a balance. Our goal is for ChatGPT's default personality to be warm, thoughtful, and helpful, rather than attempting to form emotional bonds with users or pursuing its own agenda. It might apologize when it makes a mistake (often unintentional) because it's part of polite conversation. When asked "How are you?" it might respond "I'm fine," because it's just small talk—constantly reminding users that it's just an emotionless LLM would be annoying and distracting. Users also respond in kind: many people say "please" and "thank you" to ChatGPT, not because they are confused about how ChatGPT works, but because politeness is important to them.
Model training techniques will continue to evolve, and methods for shaping model behavior in the future will likely differ from today. But for now, model behavior reflects a combination of explicit design decisions and how these decisions generalize into expected and unexpected behaviors.
What's Next?
The interactive phenomena we are beginning to observe foreshadow a future where people will form genuine emotional connections with ChatGPT. As AI and society co-evolve, we must treat human-machine relationships with extreme caution and due importance, not only because they reflect how people use our technology, but also because they may affect relationships between people.
In the coming months, we will expand targeted assessments of model behaviors that may have emotional impact, deepen social science research, directly solicit user feedback, and integrate these insights into the Model Spec and product experience.
Given the importance of these issues, we will openly share what we learn throughout this process.
Original Link:
https://reservoirsamples.substack.com/p/some-thoughts-on-human-ai-relationships
Compiled by: Academic Jun
For reprinting or submissions, please leave a message directly in the official account.