The field of natural language processing is moving towards a deeper understanding of transformer architectures and their capabilities in language learning. Recent research has focused on the role of memory in language learning, with studies showing that human-like fleeting memory can improve language learning in transformer models, but may also impair reading time prediction. Additionally, there is a growing interest in the development of more expressive and efficient transformer architectures, such as pushdown reward machines and two-layer transformers. These advancements have the potential to improve the performance of language models and enable them to learn more complex tasks. Noteworthy papers in this area include the work on pushdown reward machines, which can recognize and reward temporally extended behaviors, and the study on two-layer transformers, which can represent any conditional k-gram. Furthermore, research on in-context learning and vector arithmetic has shown promising results, with transformers able to perform tasks such as factual-recall and cryptographic function learning. Overall, the field is moving towards a more nuanced understanding of transformer architectures and their applications in language learning.