The field of representation learning in deep neural networks is witnessing significant advancements, with a growing focus on understanding the relationships between pre-training, task correlation, and representational similarity. Researchers are exploring the impact of pre-training corpora on the quality of learned transformer representations, as well as the factors that contribute to the success of reusing pre-trained models for different tasks. A key area of investigation is the correlation between tasks and features, with studies demonstrating that even when tasks are provably uncorrelated, reusing pre-trained networks can still yield significantly better performance than random. Furthermore, new methods are being developed to quantify task-relevant representational similarity, such as decision variable correlation, which can capture the similarity of decision strategies between models and brains. However, results also suggest that there may be a fundamental divergence between the representations learned by models and those used by biological systems, highlighting the need for further research into the underlying mechanisms of representation learning. Noteworthy papers include:

• one that introduces an experimental setup to study the factors contributing to the success of reusing pre-trained models, and demonstrates that task accuracy increases monotonically with task correlation.
• another that proposes a new approach to characterize the similarity of decision strategies between models and brains using decision variable correlation, and finds that model--monkey similarity is consistently lower than model--model similarity.