Authors:
Eunhae Lee、Pat Pataranutaporn、Judith Amores、Pattie Maes
Paper:
https://arxiv.org/abs/2408.06602
Introduction
The rapid advancements in artificial intelligence (AI) have sparked a range of public perceptions, from utopian to dystopian visions. This study investigates the psychological factors influencing belief in AI predictions about personal behavior, comparing it to belief in astrology and personality-based predictions. The research aims to understand how cognitive style, paranormal beliefs, AI attitudes, personality traits, and other factors affect the perceived validity, reliability, usefulness, and personalization of predictions from different sources.
Results
People who are more likely to believe in astrology and personality-based predictions are more likely to believe in AI predictions
The study found a significant positive correlation between belief in AI predictions and belief in astrology and personality-based predictions. The multiple linear regression analysis explained a significant proportion of the variance in the AI overall score, supporting the hypothesis that belief in astrology and personality-based predictions is positively associated with belief in AI predictions.
People generally find fictitious AI predictions about their personal behavior convincing
The mixed-effects model analysis revealed that people generally found fictitious predictions from AI, astrology, and personality convincing. The perceived validity of astrology-based predictions was lower compared to AI predictions, while the difference was not statistically significant for personality-based predictions. The analysis also showed that perceived personalization was rated higher than perceived validity, while perceived reliability and usefulness were rated lower.
There is no evidence of correlation between belief in predictions and cognitive style
Contrary to the hypothesis, the composite cognitive score did not significantly increase the perceived validity of AI predictions. The interactions between cognitive style and prediction sources showed no significant effects, suggesting a lack of evidence that cognitive style is an influential factor in how people perceive predictions based on AI, astrology, and personality.
Higher paranormal beliefs increase perceived validity, reliability, usefulness, and personalization of AI predictions
Paranormal beliefs were positively associated with belief in AI predictions. The mixed-effects model showed that the paranormal beliefs score significantly increased perceived validity, reliability, and usefulness of AI predictions. The effect was stronger for astrology-based predictions, while differences for personality-based predictions were not significant compared to the AI prediction baseline.
Positive AI attitudes increase belief in AI predictions, especially perceived reliability
Individuals with more positive attitudes towards AI found predictions based on AI more believable. The AI attitude score significantly increased perceived validity and reliability of AI predictions. The positive impact of AI attitudes on perceived validity was mostly reversed for astrology-based predictions, while the effect did not differ significantly for personality-based predictions.
People with high conscientiousness are less likely to believe in predictions about personal behavior
Conscientiousness was negatively associated with perceived validity of predictions across all sources. The other domains of the five-factor model did not have significant influence, with some variation in interaction terms.
Level of interest in the prediction topic increases perceived validity, reliability, usefulness, and personalization
Individuals’ interest in the topic of prediction was a strong predictor of perceived validity, reliability, usefulness, and personalization of predictions. The more people are interested in the topic, the more likely they will perceive fictitious predictions to be valid, reliable, useful, and personalized.
There is no evidence that the level of familiarity with the prediction sources influences belief in predictions
Familiarity with the prediction sources did not have a significant effect on perceived validity across all sources. The interaction between familiarity and subscales showed slight but significant increases in perceived personalization, while differences were not statistically significant for perceived reliability and usefulness.
Other results
Gullibility was not found to be a significant predictor of perceived validity of AI predictions. Older age was associated with a decrease in belief in predictions across all sources. Gender interactions revealed that male participants were more likely to perceive AI predictions as reliable than female participants. There were no significant main effects for the level of education.
Discussion
Cognitive Style
The study did not find significant evidence to support the hypothesis that an analytic cognitive style leads to more skepticism in predictions based on astrology and personality. This suggests that analytic cognitive style may not necessarily lead to rational skepticism of fictitious predictions.
Paranormal Beliefs
Paranormal beliefs were found to be an influential predictor of belief in predictions across all sources, including AI predictions. This result points to the existence of the phenomenon of “rational superstition” in AI, where belief is driven more by mental heuristics and intuition than critical evaluation.
AI Attitude/Trust in AI
Positive attitudes towards AI were associated with higher perceived validity, reliability, usefulness, and personalization of AI predictions. This supports findings from prior literature that positive attitudes toward AI lead to higher reliance on AI predictions.
Personality
Conscientiousness had a negative influence on the perception of validity, personalization, reliability, and usefulness, while other traits did not have significant effects. This contrasts with previous findings that suggested other personality traits influence trust in AI.
Interest in the Topic of Prediction
Interest in the topic of prediction was a strong predictor of belief in predictions. This suggests that the more interested one is in a topic, the more likely they may be susceptible to cognitive biases that influence their belief.
Familiarity/Prior Knowledge
Familiarity with the prediction source was not found to be a significant predictor of belief in predictions. This suggests that simply knowing more about the prediction source does not predict whether one finds a prediction more or less believable.
Gullibility
The study showed inconclusive results on whether self-reported gullibility had a positive or negative effect on the belief of fictitious predictions based on AI, astrology, and personality.
Demographic Factors
Older people were more skeptical of predictions about personal behavior, across AI, astrology, and personality sources. Male participants were more likely to perceive AI predictions as reliable than female participants. The level of education did not have significant main effects.
General Discussion
The study highlights the irrational side of how humans perceive and believe in AI predictions. The findings suggest that highly accurate and reliable performance is not a prerequisite for people to put high trust in predictions. The study underscores the importance of mental models and cognitive biases in shaping people’s perceptions of AI predictions and the need to include psychological and contextual factors in studying human-AI interaction.
Limitations and Future Work
The study has several limitations, including the specific context of personal behavior predictions and the reliance on self-reported data. Future studies could explore how people perceive AI predictions in different contexts and prediction topics, and validate the findings using different methods and approaches.
Conclusion
The study empirically investigated the phenomenon of “rational superstition” in AI by comparing people’s perceptions of fictitious predictions based on AI, astrology, and personality. The findings showed that people’s belief in AI predictions was positively associated with paranormal beliefs, positive attitude towards AI, and interest in the topic of prediction, and negatively associated with conscientiousness and age. The study highlights the role of mental models and cognitive biases in shaping people’s perceptions of AI predictions and the importance of including psychological and contextual factors in studying human-AI interaction.
Methods
Participants
Data were collected from 300 participants recruited through Prolific, with 238 participants included in the final analysis after excluding incomplete responses or failed attention checks.
Experiment Protocol
Participants completed a zodiac-related questionnaire, a personality test, and a simulated investment game. They received fictitious predictions from three sources (AI, astrology, personality) and evaluated their perceived validity, personalization, reliability, and usefulness.
Predictions
Participants were randomly assigned to either the “Positive” or “Negative” prediction group. Examples of predictions received are shown below.
Post-study Questionnaire
Participants responded to questionnaires on cognitive style, paranormal beliefs, gullibility, trust in AI, personality, and demographic information.
Analysis
The analysis included multiple linear regression and mixed-effects models to examine the effects of various factors on the perceived validity, reliability, usefulness, and personalization of predictions.
Acknowledgments
We thank Jinjie Liu, Data Science Specialist at the Institute for Quantitative Social Science at Harvard University, for sharing her valuable perspective on the analysis.
