Advances in Unlearning and Data Pruning for Large Language Models and Recommender Systems

Introduction

The field of large language models and recommender systems is moving towards developing more robust and efficient methods for unlearning and data pruning. This is driven by the need to remove unwanted knowledge and noisy interactions from models, while preserving their performance and accuracy.

General Direction

The general direction of the field is towards more precise and effective methods for unlearning and data pruning. This includes the development of novel frameworks and techniques that can automatically identify and remove unwanted knowledge and noisy interactions from models. Additionally, there is a focus on evaluating the efficacy of unlearning algorithms and developing comprehensive auditing frameworks.

Noteworthy Papers

Some papers are particularly noteworthy in this regard. For example, the paper on LURK presents a novel framework for probing hidden retained knowledge in unlearned large language models. The paper on Shapley Value-driven Data Pruning proposes a framework that evaluates interactions based on their objective impact on model training, rather than subjective intent assumptions. The paper on PISCES presents a novel framework for precisely erasing entire concepts from model parameters by directly editing directions that encode them in parameter space.

Sources

Harry Potter is Still Here! Probing Knowledge Leakage in Targeted Unlearned Large Language Models via Automated Adversarial Prompting

Shapley Value-driven Data Pruning for Recommender Systems

ConsRec: Denoising Sequential Recommendation through User-Consistent Preference Modeling

Precise In-Parameter Concept Erasure in Large Language Models

Pre-training for Recommendation Unlearning

Does Machine Unlearning Truly Remove Model Knowledge? A Framework for Auditing Unlearning in LLMs

Built with on top of