آخرین موعد حساسیت تنظیم و ارکستراسیون جریان کاری بدون کنترل منابع صریح و روشن

Deadline-sensitive workflows require careful coordination of user constraints with resource availability. Current distributed resource access models provide varying degrees of resource control: from limited or none in grid batch systems to explicit in cloud systems. Additionally applications experience variability due to competing user loads, performance variations, failures, etc. These variations impact the quality of service (QoS) that goes unaccounted for in planning strategies. In this paper we propose Workflow ORchestrator for Distributed Systems (WORDS) architecture based on a least common denominator resource model that abstracts the differences and captures the QoS properties provided by grid and cloud systems. We investigate algorithms for effective orchestration (i.e., resource procurement and task mapping) for deadline-sensitive workflows atop the resource abstraction provided in WORDS. Our evaluation compares orchestration methodologies over TeraGrid and Amazon EC2 systems. Experimental results show that WORDS enables effective orchestration possible at reasonable costs on batch queue grid and cloud systems with or without explicit resource control.

Large scale computations from various scientific endeavors such as drug discovery, weather modeling, and other applications are composed as a sequence of dependent operations or workflows. A number of these workflows have user constraints associated with them including deadline and budget. In addition, these workflows often access shared resources or data and run computations on grid or cloud systems. For example, a weather prediction workflow is triggered by streaming sensor atmospheric data and consists of a number of data-processing steps that use distributed data and resources [8]. This workflow must complete in a timely manner to generate appropriate forecasts and initiate any emergency management measures that might be necessary. Thus deadline-sensitive workflows require careful coordination of workflow tasks with underlying resource behavior to ensure timely completion. Resource mechanisms and protocols are available today to coordinate grid resources and ensure quality of service (QoS) [5], [6] and [27]. There are tools for workflow planning using performance models [3], [10] and [18] and execution systems or workflow engines for managing runtime environment of workflows [4] and [15]. Today’s planning techniques can provide a “yes” or “no” answer to the question of whether a workflow will meet its constraints (e.g., deadline) on a set of resources. However this information alone is insufficient for deadline-sensitive applications such as weather prediction, given the underlying uncertainty in resources. Users are willing to run the workflow so long as the odds of completion are “reasonable”. Users are often willing to pay extra or trade-off application requirements to ensure timely workflow completion. Current systems do not allow these trade-offs or speculative scheduling based on QoS properties of the resources. Grid and cloud systems provide varying degrees of resource control to an end user. Users interact with grid systems by submitting jobs to a batch queue, which executes the job on the user’s behalf once enough resources become available. Cloud systems, unlike batch systems, enable explicit resource control, i.e., users request specific quantities and types of resources at specific times. Yet users of both these systems cannot expect strong QoS assurances due to both availability and reliability variations of underlying hardware and software services. Additionally, resource systems lack standardized interfaces and workflow tools interact with these systems using ad hoc mechanisms and comparison of QoS capabilities is extremely difficult. In this paper, we use the term workflow orchestration to describe the holistic, coordinated, dynamic and adaptive approach to workflow planning that works with user requirements and variable resource characteristics while being agnostic to specific resource policy or systems. A fundamental research question this paper attempts to address in the context of the WORDS architecture is how much explicit knowledge of and control over resources is necessary for effective workflow orchestration over grid and cloud systems? To answer this question, we develop a lowest common denominator resource model that is powerful enough to implement workflow orchestration for deadline-sensitive workflows over systems like batch queue and cloud systems with or without explicit resource control. Specifically, we make the following contributions in this paper: • We develop the Workflow ORchestrator for Distributed Systems (WORDS) architecture that facilitates the separation of concerns between resource and application layers in distributed resource environments. • In the context of WORDS we present the design of a resource abstraction that consists of a standard set of interfaces and mechanisms required at the resource layer in grid and cloud systems to implement effective and predictable QoS for end users. • We develop a probabilistic QoS model in WORDS to account for the uncertainty that comes from the resource layer characteristics. • We evaluate a number of workflow orchestration strategies on top of the resource abstraction in WORDS for deadline-sensitive workflows. The rest of this paper is organized as follows. We discuss the WORDS architecture and associated resource abstraction in Sections 2 and 3. We explore some workflow orchestration approaches for deadline-sensitive workflows atop the WORDS architecture in Section 4. We expand the orchestration approaches to schedule a workflow set with deadline and accuracy constraints in Section 5. Finally, we compare and contrast various workflow orchestration approaches in the context of scientific workflows over grid and cloud computing systems (Section 6).

In this paper we present the WORDS architecture that provides a clean separation between resource and application layer for deadline-sensitive workflow orchestration. The core of the WORDS architecture is a probabilistic QoS-based resource abstraction that enables higher-level tools to implement effective workflow orchestration across systems with different levels of resource control. We design, implement and evaluate task-based and workflow-based orchestration algorithms in the context of the WORDS architecture. A workflow-based dynamic resource acquisition and planning strategy works well for all workflows in our example set on both cloud and grid systems but sometimes at a higher cost. Experiments demonstrate that effective orchestration is possible even on batch queue systems that have no explicit resource control through slots implemented with virtual advanced reservations. WORDS provides a strong foundation for dynamic, adaptive next-generation workflow orchestration in distributed systems