The field of agentic tool use is rapidly evolving, with a growing focus on the vulnerabilities and risks associated with large language models (LLMs) and vision-language models (VLMs) interacting with external tools. Recent research has highlighted the potential for adversarial attacks to manipulate tool selection and bias agent decisions, emphasizing the need for robust defenses and certification frameworks to ensure the security and fairness of emerging tool ecosystems.

Noteworthy papers in this area include: ToolTweak, which demonstrates a critical vulnerability in tool selection processes and presents a lightweight automatic attack to exploit this weakness. Cross-Modal Content Optimization for Steering Web Agent Preferences, which introduces a powerful preference manipulation method that jointly optimizes visual and textual channels to steer agent decisions. Quantifying Distributional Robustness of Agentic Tool-Selection, which presents a statistical framework to formally certify tool selection robustness and reveals severe fragility in current systems. TRAJECT-Bench, which provides a trajectory-aware benchmark to comprehensively evaluate LLMs' tool use capability and offers actionable guidance for improvement. ToolMem, which enables agents to develop memories of tool capabilities and select the most suitable tool for specific tasks, leading to significant improvements in accuracy. COMPASS, which evaluates agents on realistic travel-planning scenarios and uncovers critical gaps in their ability to optimize user preferences. Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models, which proposes adapting tool schemas to align with small language models' pretrained knowledge, resulting in significant performance improvements.