The field of natural language processing is moving towards a deeper understanding of the internal workings of Transformer-based language models. Researchers are exploring the linguistic interpretability of these models, aiming to uncover how they encode and utilize linguistic knowledge. This involves analyzing the models' internal representations and investigating the relationship between pretraining data and the formation of linear representations. The development of novel frameworks and methods, such as the interpretation of Transformers as probabilistic left context-sensitive languages generators, is providing new insights into the mechanisms driving these models. Furthermore, the creation of benchmarks and evaluation standards, like the Mechanistic Interpretability Benchmark, is enabling the comparison of different interpretability methods and driving progress in the field. Noteworthy papers in this area include: Moving Beyond Next-Token Prediction, which presents a novel framework for interpreting LLMs, and On Linear Representations and Pretraining Data Frequency in Language Models, which investigates the connection between pretraining data frequency and models' linear representations. SMARTe: Slot-based Method for Accountable Relational Triple extraction is also a significant contribution, introducing intrinsic interpretability through a slot attention mechanism. MIB: A Mechanistic Interpretability Benchmark provides a meaningful evaluation standard for mechanistic interpretability methods.