Large Language Models in Social Science Research

The field of large language models (LLMs) is rapidly advancing, with a focus on improving their ability to understand and simulate human behavior in various social science contexts. Recent developments have explored the use of LLMs in simulating voting behavior, predicting survey responses, and analyzing online user feedback. Notably, LLMs have been shown to effectively simulate voting behavior of Members of the European Parliament and predict individual voting decisions with reasonable accuracy. Additionally, LLMs have been used to classify quality characteristics in online user feedback, with promising results in low-data settings. However, concerns about the potential role of LLMs in exacerbating ideological polarization and manipulating public opinion have also been raised. Furthermore, studies have highlighted the importance of evaluating the misalignment between LLM-simulated and actual human behaviors in multiple-choice survey settings. Some noteworthy papers in this area include:

  • Persona-driven Simulation of Voting Behavior in the European Parliament with Large Language Models, which demonstrates the ability of LLMs to simulate voting behavior with reasonable accuracy.
  • Passing the Turing Test in Political Discourse: Fine-Tuning LLMs to Mimic Polarized Social Media Comments, which raises significant ethical concerns about the use of AI in political discourse.
  • Enhancement Report Approval Prediction: A Comparative Study of Large Language Models, which evaluates the performance of various LLM variants in predicting enhancement report approval and demonstrates their potential to streamline software maintenance workflows.

Sources

AssertBench: A Benchmark for Evaluating Self-Assertion in Large Language Models

Classification of Quality Characteristics in Online User Feedback using Linguistic Analysis, Crowdsourcing and LLMs

Persona-driven Simulation of Voting Behavior in the European Parliament with Large Language Models

Revealing Political Bias in LLMs through Structured Multi-Agent Debate

SimSpark: Interactive Simulation of Social Media Behaviors

AIn't Nothing But a Survey? Using Large Language Models for Coding German Open-Ended Survey Responses on Survey Motivation

Passing the Turing Test in Political Discourse: Fine-Tuning LLMs to Mimic Polarized Social Media Comments

Hypothesis Testing for Quantifying LLM-Human Misalignment in Multiple Choice Settings

Enhancement Report Approval Prediction: A Comparative Study of Large Language Models

Algorithmic resolution of crowd-sourced moderation on X in polarized settings across countries

Built with on top of