Advancements in Workflow Orchestration and Scheduling

The field of workflow orchestration and scheduling is moving towards more intelligent, distributed, and scalable solutions. Researchers are focusing on developing systems that can automate complex and dynamic processing pipelines, while also reducing operational overhead and enabling reproducible, high-throughput workflows across heterogeneous infrastructures. Noteworthy papers include: iDDS, which presents a versatile workflow orchestration system that integrates data-aware execution, conditional logic, and programmable workflows. PARS, which introduces a prompt-aware LLM task scheduler that improves serving efficiency by approximating shortest-job-first scheduling. SchedMate, which proposes a framework that bridges the semantic gap in deep learning schedulers by systematically extracting deep insights from overlooked data sources.

Sources

iDDS: Intelligent Distributed Dispatch and Scheduling for Workflow Orchestration

PARS: Low-Latency LLM Serving via Pairwise Learning-to-Rank

Semantic-Aware Scheduling for GPU Clusters with Large Language Models

The LCLStream Ecosystem for Multi-Institutional Dataset Exploration

Built with on top of