The field of natural language processing is moving towards a more culturally aware and inclusive approach. Researchers are recognizing the importance of evaluating large language models (LLMs) in a more nuanced and contextualized manner, taking into account the complexities of culture and its impact on language. This shift is driven by the need to address the limitations of current evaluation methods, which often rely on static and isolated assessments of cultural knowledge. Instead, researchers are advocating for a more intentional and systematic approach to cultural evaluation, one that considers the cultural assumptions embedded in all aspects of evaluation and involves communities in the design of evaluation methodologies. Noteworthy papers in this area include:

• Culture is Everywhere: A Call for Intentionally Cultural Evaluation, which argues for a more inclusive and culturally aligned approach to NLP research.
• On the Alignment of Large Language Models with Global Human Opinion, which presents a comprehensive investigation of the alignment of LLMs with human opinions across different countries, languages, and historical periods.
• SESGO: Spanish Evaluation of Stereotypical Generative Outputs, which introduces a novel framework for detecting social biases in instruction-tuned LLMs in the Spanish language.
• Where Should I Study? Biased Language Models Decide!, which empirically examines geographic, demographic, and economic biases in university and program suggestions from LLMs.
• Acquiescence Bias in Large Language Models, which investigates the presence of acquiescence bias in LLMs across different models, tasks, and languages.