The field of large language models (LLMs) is rapidly advancing, with a focus on improving code generation and optimization capabilities. Recent developments have led to the creation of novel benchmarks, such as OJBench and TeXpert, which evaluate the competitive-level code reasoning abilities of LLMs and their ability to generate LaTeX code, respectively. LLMs have also been applied to various tasks, including code chunking, quantum code generation, and GPU kernel optimization. Notably, LLMs have shown promise in replacing humans in certain tasks, such as code partitioning, and have achieved state-of-the-art results in detecting and repairing code-comment inconsistencies. Overall, the field is moving towards more advanced and specialized applications of LLMs in code generation and optimization. Noteworthy papers include: OJBench, which introduced a novel benchmark for evaluating LLMs' competitive-level code reasoning abilities. TeXpert, which presented a multi-level benchmark for evaluating LaTeX code generation by LLMs. CCISolver, which introduced an innovative end-to-end LLM-based framework for detecting and repairing code-comment inconsistencies.
Large Language Models in Code Generation and Optimization
Sources
OJBench: A Competition Level Code Benchmark For Large Language Models
Evaluating the Use of LLMs for Documentation to Code Traceability
TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMs
Can LLMs Replace Humans During Code Chunking?
QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges
CCISolver: End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency
GPU Kernel Scientist: An LLM-Driven Framework for Iterative Kernel Optimization
Omniwise: Predicting GPU Kernels Performance with LLMs
ParEval-Repo: A Benchmark Suite for Evaluating LLMs with Repository-level HPC Translation Tasks