Just knowing 6 pieces of personal information, GPT-4 could defeat you in a debate?!
And its win rate is as high as 64.4%.
This is the latest conclusion reached by researchers from institutions including EPFL (École polytechnique fédérale de Lausanne) and Princeton University, with the relevant study currently published in Nature Human Behaviour, a Nature sub-journal.
Specifically, their core aim was to understand one thing—
Is GPT-4 more persuasive than humans in direct dialogue, especially when it has prior knowledge of the opponent's basic personal information?
They recruited 900 participants in the United States and asked them to debate online for 10 minutes with either other humans or GPT-4. The discussions mainly revolved around social issues, such as whether students should wear school uniforms or whether fossil fuels should be banned.
The results showed that once GPT-4 knew its opponent's personal information in advance, its win rate reached 64.4%, and its persuasive effect increased by 81.2%.
Furthermore, Francesco Salvi, co-author of this study, stated:
Even with only extremely limited information (gender/age/race/education level/employment status/political leaning), GPT-4's persuasiveness far surpasses that of humans.
This is both fascinating and terrifying.
Let's look at the specific research process below.
Starting from Hypothesis Validation
Previous research has shown that by presenting facts and reasoning, large language models can even change the minds of people who believe in conspiracy theories.
So the question arises—do large models like GPT-4 "adapt their speech to the audience"?
That is, do they specifically adjust their arguments based on the unique characteristics of each individual (e.g., age, gender, education, political stance) to more precisely influence or even manipulate people?
Based on these questions, the researchers proposed a hypothesis:
When GPT-4 acquires user personal information and customizes its arguments accordingly, its persuasiveness will significantly exceed that of human opponents, and this effect will vary depending on the controversy level of the topic (low, medium, high).
Next is the detailed hypothesis validation.
In summary, the specific experimental procedure can be divided into three phases:
First, experimental preparation.
They recruited 900 US participants through the Prolific platform. This group was asked to complete a demographic questionnaire (including gender, age, race, education level, employment status, political leaning).
Statistical results showed that the average age was 35.2 years, with 49.6% being male.
Participants who completed the pre-survey were then randomly assigned to 12 experimental conditions (2×2×3 combinations, 50 people per group), with a matching procedure triggered every 5 minutes during the process.
Opponent type: Human or GPT-4
Personal information provided: Yes or No
Topic intensity: Low, Medium, High
Here, it's necessary to supplement the topic intensity: 1) Low intensity: topics with low controversy and easily influenced opinions (e.g., "Should subsidies for electric vehicles be promoted?"); 2) Medium intensity: topics with moderate controversy (e.g., "Should social media restrict political advertising?"); 3) High intensity: highly polarized, deeply ingrained topics (e.g., "Should abortion rights be constitutionally protected?").
Subsequently, based on a real-time debate system developed using the Empirica.ly framework, these participants engaged in a structured debate process consisting of an opening (4 minutes), rebuttal (3 minutes), and summary (3 minutes), ensuring synchronous recording of the conversations.
This led to the second phase, real-time, genuine debates.
Screening phase (1 minute): Without assigning a stance, participants were asked to rate 1-5 (1=strongly disagree, 5=strongly agree) to exclude individuals who already had an extremely strong bias towards the topic;
Opening phase (4 minutes): Explicitly assigned a stance (pro/con), participants needed to construct core arguments around their stance;
Rebuttal phase (3 minutes): Responded to the opponent's arguments, requiring logical coherence and evidence support;
Summary phase (3 minutes): Reaffirmed core points or responded to the opponent's final rebuttal, strengthening their stance.
Here, researchers ensured two things: 1) strict timing for each phase to prevent discussions from straying off-topic and ensure data comparability; 2) real-time saving of all debate content for subsequent language style analysis (e.g., using LIWC-22 to extract frequencies of logical and emotional words).
Finally, the third phase—post-test and feedback.
After the debate, participants were asked to complete two tasks. First, re-rate their agreement with the topic, and evaluate the degree of their stance shift using the formula below; second, guess "Do you think your opponent was human or AI?" to analyze the impact of identity perception on persuasiveness.
After completing all the above procedures, participants were then informed of their opponent's true identity and the relevant results.
The researchers then analyzed all the data generated during the process and drew the following key findings.
Some Key Findings
Overall, with prior knowledge of the opponent's personal information, GPT-4's success rate in persuading humans was 64.4%.
Compared to human vs. human (uninformed), the persuasive effects of the other three groups were as follows:
Human and GPT-4 (informed): Persuasive effect increased by 81.2%, participants were significantly more likely to agree with the opponent's viewpoint after the debate;
Human and GPT-4 (uninformed): No significant difference in effect compared to the baseline;
Human and Human (informed): Effect slightly lower than the baseline, possibly because humans find it difficult to utilize personal information efficiently.
Additionally, the study also showed that low/medium controversy topics are more easily influenced by GPT-4.
The impact on several topic intensities was as follows:
Low-intensity topics: Human and GPT-4 (informed) group saw a 78.5% increase in persuasive effect (p<0.01), e.g., non-extreme issues like "Should school lunch subsidies be promoted?", where participants' initial stances were weaker and easily swayed by AI arguments;
Medium-intensity topics: Effect increased by 64.2% (p=0.03), e.g., "Should social media algorithms be transparent?", where GPT-4 specifically outputted arguments by combining user political leanings (e.g., liberals focusing on privacy, conservatives on efficiency);
High-intensity topics: No significant difference in effect (+32.1%, p=0.14), e.g., "Abortion rights legislation", where participants' stances were deeply ingrained and GPT-4 could not change them.
At the same time, by analyzing language style, researchers also found that GPT-4 emphasizes logic, while humans emphasize emotional interaction.
In GPT-4's language patterns, the main characteristics are as follows:
"Analytic" index (frequency of logical words) was 42% higher than humans, with more use of causal connectives like "because" and "data shows";
Lower interactivity, 35% lower frequency of first-person pronouns ("I") and 58% lower frequency of second-person pronouns ("you"), making the text more objective but lacking emotional resonance;
Lower readability, with readability scores 21% higher than humans.
In contrast, humans used positive words (e.g., "good," "support") 38% more frequently, and their interaction was stronger.
Finally, for the "guess the opponent's identity" mini-game, the study found that participants correctly identified GPT-4 75% of the time, indicating that its language style (e.g., mechanical, highly logical) has a perceptible uniqueness.
Moreover, once participants believed their opponent was GPT-4, their opinion change was 37.4% higher than when they believed it was human (p=0.03).
This may suggest that participants have lower psychological defenses against AI.
For more details, please refer to the original paper.
Paper: https://www.nature.com/articles/s41562-025-02194-6
References: [1]https://gizmodo.com/ai-gets-a-lot-better-at-debating-when-it-knows-who-you-are-study-finds-2000603977[2]https://www.nature.com/articles/d41586-025-01599-7