Advances in Fairness and Bias Evaluation in Large Language Models

The field of natural language processing is rapidly evolving, with a growing focus on fairness and bias evaluation in large language models (LLMs). Recent studies have highlighted the importance of addressing biases in LLMs, particularly in sensitive areas such as healthcare, finance, and law. Researchers are developing innovative methods to identify and mitigate biases, including metamorphic testing, causal reasoning analysis, and debiasing techniques.

One key direction in this field is the development of multimodal approaches to analyze and detect biases in LLMs. This includes the use of multimodal datasets, such as videos and images, to evaluate the performance of LLMs in detecting sexist and biased content. Additionally, researchers are exploring the use of causal inference and information theory to develop autonomous debiasing methods for LLMs.

The evaluation of social bias in LLMs is also an active area of research, with studies focusing on the analysis of bias in low-resource languages and dialects. This includes the development of new datasets and evaluation metrics to assess the performance of LLMs in detecting biases against marginalized groups.

Notable papers in this area include: Metamorphic Testing for Fairness Evaluation in Large Language Models, which introduces a metamorphic testing approach to systematically identify fairness bugs in LLMs. BiasCause: Evaluate Socially Biased Causal Reasoning of Large Language Models, which evaluates the causal reasoning process of LLMs when answering questions that elicit social biases. UoB-NLP at SemEval-2025 Task 11: Leveraging Adapters for Multilingual and Cross-Lingual Emotion Detection, which demonstrates the effectiveness of adapter-based fine-tuning for multilingual emotion detection. Information Gain-Guided Causal Intervention for Autonomous Debiasing Large Language Models, which proposes an information gain-guided causal intervention debiasing framework to autonomously debias LLMs.

Sources

Metamorphic Testing for Fairness Evaluation in Large Language Models: Identifying Intersectional Bias in LLaMA and GPT

Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance

BiasCause: Evaluate Socially Biased Causal Reasoning of Large Language Models

UoB-NLP at SemEval-2025 Task 11: Leveraging Adapters for Multilingual and Cross-Lingual Emotion Detection

Name of Thrones: Evaluating How LLMs Rank Student Names, Race, and Gender in Status Hierarchies

Can LLMs Leverage Observational Data? Towards Data-Driven Causal Discovery with LLMs

MuSeD: A Multimodal Spanish Dataset for Sexism Detection in Social Media Videos

Bias Beyond English: Evaluating Social Bias and Debiasing Methods in a Low-Resource Setting

Robustness and sex differences in skin cancer detection: logistic regression vs CNNs

Masculine Defaults via Gendered Discourse in Podcasts and Large Language Models

Word Embeddings Track Social Group Changes Across 70 Years in China

Reimagining Urban Science: Scaling Causal Inference with Large Language Models

Out of Sight Out of Mind, Out of Sight Out of Mind: Measuring Bias in Language Models Against Overlooked Marginalized Groups in Regional Contexts

Information Gain-Guided Causal Intervention for Autonomous Debiasing Large Language Models

Built with on top of