The field of large language models is rapidly advancing, with a growing focus on improving the reliability and performance of these models in production environments. Recent research has highlighted the challenges of troubleshooting performance problems in large-scale model training, as well as the need for effective debugging and diagnosis tools for distributed training and inference frameworks. Notable papers have proposed innovative solutions, such as online performance troubleshooting systems and lightweight error checking and diagnosis tools, which have shown promising results in detecting and localizing bugs in distributed training. Empirical studies have also shed light on the characteristics of bugs in large language model inference engines and distributed training frameworks, providing valuable insights for improving the reliability of these systems. Some notable papers include:

• PerfTracker, which presents an online troubleshooting system for large-scale model training in production, effectively diagnosing performance issues rooted in both hardware and software.
• TTrace, which proposes a lightweight error checking and diagnosis tool for distributed training, capable of detecting and localizing silent bugs in distributed training.
• A First Look at Bugs in LLM Inference Engines, which conducts an empirical study on bugs in large language model inference engines, providing a comprehensive dataset of real-world bugs and shedding light on the key challenges in bug detection and location.