The field of large language models (LLMs) is rapidly advancing, with a growing focus on identifying and addressing potential vulnerabilities. Recent research has highlighted the risks of prompt injection attacks, which can be used to manipulate LLMs into performing unintended tasks. To counter these threats, researchers are developing innovative defense strategies, including prompt sanitization techniques, statistical anomaly detection, and architectural modifications to LLM deployment. Another area of concern is the potential for LLMs to be used in malicious ways, such as generating Trojanized prompts or exploiting vulnerabilities in web-based applications. To mitigate these risks, researchers are proposing new frameworks and tools, such as EDU-Prompting, which aims to promote critical thinking and bias awareness in LLM-based educational systems. Noteworthy papers in this area include Paper Summary Attack, which proposes a novel jailbreaking method for LLMs, and PromptArmor, which presents a simple yet effective defense against prompt injection attacks. Additionally, the paper on Multi-Stage Prompt Inference Attacks highlights the need for a holistic approach to securing LLMs in enterprise settings.
Vulnerabilities and Defenses in Large Language Models
Sources
Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
TopicAttack: An Indirect Prompt Injection Attack via Topic Transition
Mitigating Trojanized Prompt Chains in Educational LLM Use Cases: Experimental Findings and Detection Tool Design
Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree
EduThink4AI: Translating Educational Critical Thinking into Multi-Agent LLM Systems
PromptArmor: Simple yet Effective Prompt Injection Defenses
Multi-Stage Prompt Inference Attacks on Enterprise LLM Systems
Depth Gives a False Sense of Privacy: LLM Internal States Inversion
When LLMs Copy to Think: Uncovering Copy-Guided Attacks in Reasoning LLMs
CASCADE: LLM-Powered JavaScript Deobfuscator at Google
BadReasoner: Planting Tunable Overthinking Backdoors into Large Reasoning Models for Fun or Profit