DR-SWDF: A Dynamically Reconfigurable Framework for Scientific Workflows Deployment in the Cloud

Abstract

Workflows management systems (WfMS) are aimed for designing, scheduling, executing, reusing, and sharing workflows in distributed environments like the Cloud computing. With the emergence of e-science workflows, which are used in different domains like astronomy, life science, and physics, to model and execute vast series of dependents functionalities and a large amount of manipulated data, the workflow management systems are required to provide customizable programming environments to ease the programming effort required by scientists to orchestrate a computational science experiment. A key issue for e-science WfMS is how to deal with the change of the execution environment constraints and the variability and confliction of end users and cloud providers objectives for the execution of the same workflow or sub-workflow. They have to customize their management processes to insure the adaptability of the execution environment to the scientific workflows specificities, especially when dealing with large-scale (data, computing, I/O)-intensive workflows. In this paper, we propose a dynamically re-configurable framework for the deployment of scientific workflows in the Cloud (called DR-SWDF) that allows customizing the workflow deployment process according to a set of objectives and constraints of end users or cloud providers defined differently for the tasks or partitions of the same workflow. The DR-SWDF framework offers a K-means based algorithm that allows dynamically clustering the input workflows or sub-workflows in order to identify the most convenient techniques or algorithms to be applied for their scheduling and deployment in the cloud. The simulations results run on three examples large-scale scientific workflows show that our proposed framework can achieve better results than the use of a generic purpose approach.