Advances in Large Language Model Editing and Robustness

The field of large language models (LLMs) is moving towards improving their editing and robustness capabilities. Researchers are exploring various methods to efficiently update LLMs without compromising their performance, including model editing techniques and robustness evaluation protocols. A key challenge is ensuring that edited knowledge is retained during fine-tuning, and studies have shown that current editing approaches have limitations in this regard. Innovative solutions, such as layer-aware model editing and neural KV databases, are being proposed to address these challenges. Noteworthy papers include:

  • Improving Code LLM Robustness to Prompt Perturbations via Layer-Aware Model Editing, which introduces a novel approach to enhance LLM robustness through targeted parameter updates.
  • NeuralDB: Scaling Knowledge Editing in LLMs to 100,000 Facts with Neural KV Database, which proposes a neural database editing framework that preserves the general abilities of LLMs while allowing for efficient knowledge editing.

Sources

Retention analysis of edited knowledge after fine-tuning

Small Edits, Big Consequences: Telling Good from Bad Robustness in Large Language Models

Improving Code LLM Robustness to Prompt Perturbations via Layer-Aware Model Editing

NeuralDB: Scaling Knowledge Editing in LLMs to 100,000 Facts with Neural KV Database

Built with on top of