Optimization and Data Transfer Advances

The field is moving towards developing innovative solutions to optimize data transfer and processing. Researchers are exploring new architectures and algorithms to improve the efficiency and reliability of data transfers, particularly in high-performance applications. Notable advancements include the use of generative models, reinforcement learning, and modular architectures to optimize concurrency levels and reduce transfer completion times.

Some noteworthy papers in this area include: Evolutionary Generative Optimization, which proposes a fully data-driven framework for evolutionary optimization. AutoMDT, a novel modular data transfer architecture that employs deep reinforcement learning to optimize concurrency levels. FastBioDL, a parallel file downloader designed for large biological datasets, featuring an adaptive concurrency controller.

Sources

Benchmarking XRootD-HTTPS on 400Gbps Links with Variable Latencies

Sequential, Parallel and Consecutive Hybrid Evolutionary-Swarm Optimization Metaheuristics

Evolutionary Generative Optimization: Towards Fully Data-Driven Evolutionary Optimization via Generative Learning

Wind Power Scenario Generation based on the Generalized Dynamic Factor Model and Generative Adversarial Network

Data Movement Manager (DMM) for the SENSE-Rucio Interoperation Prototype

Case Studies of Generative Machine Learning Models for Dynamical Systems

Adaptive Parallel Downloader for Large Genomic Datasets

Modular Architecture for High-Performance and Low Overhead Data Transfers

Built with on top of