The field of large language models is moving towards a deeper understanding of their internal workings and behavior. Researchers are developing new methods to evaluate and analyze these models, focusing on aspects such as epistemological complexity, trustworthiness, and low-dimensional structure. These innovations aim to address limitations in current evaluation metrics and provide a more comprehensive understanding of language model dynamics. Notably, studies have found consistent patterns in language model behavior across different architectures, training data, and scales. Furthermore, new approaches are being explored to assess the trustworthiness of generated text and to leverage the low-rank structure of language models for improved generation capabilities. Some noteworthy papers in this regard include:

• Correlation Dimension of Auto-Regressive Large Language Models, which introduces a fractal-geometric measure to quantify the complexity of text generated by language models.
• Embedding Trust: Semantic Isotropy Predicts Nonfactuality in Long-Form Text Generation, which proposes a method to assess the trustworthiness of long-form responses generated by language models using semantic isotropy.
• Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale, which shows that language models exhibit consistent behavioral phases across different architectures, training data, and scales.
• Sequences of Logits Reveal the Low Rank Structure of Language Models, which demonstrates the low-rank structure of language models and explores its implications for generation.