Advancements in Large Language Models

The field of natural language processing is witnessing significant advancements in large language models (LLMs), with a focus on improving their reliability, diversity, and efficiency. Researchers are exploring innovative approaches to enhance text-to-SQL reasoning, model fusion, and query routing. Notably, there is a growing emphasis on developing cost-effective and scalable solutions that can optimize performance without sacrificing accuracy. Techniques such as data synthesis, preference optimization, and guarded query routing are being proposed to address the limitations of existing methods. Furthermore, the development of novel frameworks and architectures, such as InfiGFusion, InfiFPO, and LightRouter, is enabling more efficient and effective collaboration between LLMs. Overall, the field is moving towards more robust, adaptable, and resource-efficient LLMs that can be applied to a wide range of tasks and domains. Noteworthy papers include: SQLForge, which achieves state-of-the-art performance on text-to-SQL benchmarks through data synthesis and augmentation. InfiGFusion, which proposes a novel graph-on-logits distillation approach for model fusion, demonstrating improved fusion quality and stability across multiple benchmarks. InfiFPO, which introduces a preference optimization method for implicit model fusion, outperforming existing methods on 11 benchmarks. Cheaper, Better, Faster, Stronger, which presents a cost-efficient text-to-SQL approach that achieves similar performance to more expensive methods. JOLT-SQL, which streamlines text-to-SQL training through a unified loss function and achieves state-of-the-art execution accuracy. Guarded Query Routing, which studies the problem of routing queries to different LLM endpoints while handling out-of-distribution queries. Abacus, which presents a cost-based optimizer for semantic operator systems and demonstrates improved quality, cost, and latency. Cost-aware LLM-based Online Dataset Annotation, which proposes a novel framework for efficient and accurate dataset annotation. Causal LLM Routing, which learns routing policies by minimizing decision-making regret from observational data. LightRouter, which systematically selects and integrates a small subset of LLMs to optimize task performance and cost efficiency. INFERENCEDYNAMICS, which proposes a flexible and scalable multi-dimensional routing framework for navigating a diverse landscape of LLMs.

Sources

SQLForge: Synthesizing Reliable and Diverse Data to Enhance Text-to-SQL Reasoning in LLMs

InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion

InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models

Cheaper, Better, Faster, Stronger: Robust Text-to-SQL without Chain-of-Thought or Fine-Tuning

JOLT-SQL: Joint Loss Tuning of Text-to-SQL with Confusion-aware Noisy Schema Sampling

Guarded Query Routing for Large Language Models

Abacus: A Cost-Based Optimizer for Semantic Operator Systems

Cost-aware LLM-based Online Dataset Annotation

Causal LLM Routing: End-to-End Regret Minimization from Observational Data

LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead

INFERENCEDYNAMICS: Efficient Routing Across LLMs through Structured Capability and Knowledge Profiling

Built with on top of