Using Kestrel and XMPP to Support the STAR Experiment in the Cloud
This paper presents the results and experiences of adapting and improving the Many-Task Computing (MTC) framework Kestrel for use with bag of tasks applications and the STAR experiment in particular. Kestrel is a lightweight, highly available job scheduling framework for Virtual Organization Cluster...
| Main Authors: | , , , , |
|---|---|
| Other Authors: | |
| Format: | Article |
| Language: | English |
| Published: |
Springer Netherlands
2016
|
| Online Access: | http://hdl.handle.net/1721.1/103623 |
| _version_ | 1826193309367795712 |
|---|---|
| author | Stout, Lance Walker, Matthew Lauret, Jérôme Goasguen, Sebastien Murphy, Michael A. |
| author2 | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science |
| author_facet | Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Stout, Lance Walker, Matthew Lauret, Jérôme Goasguen, Sebastien Murphy, Michael A. |
| author_sort | Stout, Lance |
| collection | MIT |
| description | This paper presents the results and experiences of adapting and improving the Many-Task Computing (MTC) framework Kestrel for use with bag of tasks applications and the STAR experiment in particular. Kestrel is a lightweight, highly available job scheduling framework for Virtual Organization Clusters (VOCs) constructed in the cloud. Kestrel uses the Extensible Message and Presence Protocol (XMPP) for increasing MTC platform scalability and mitigating faults in Wide Area Network (WAN) communications. Kestrel’s architecture is based upon pilot job frameworks used extensively in Grid computing, with fault-tolerant communications inspired by command-and-control botnets. The extensibility of XMPP has allowed development of protocols for identifying manager nodes, discovering the capabilities of worker agents, and for distributing tasks. Presence notifications provided by XMPP allow Kestrel to monitor the global state of the pool and to perform task dispatching based on worker availability. Since its inception, Kestrel has been modified based on its performance managing operational scientific workloads from the STAR group at Brookhaven National Laboratories. STAR provided a virtual machine image with applications for simulating proton collisions using PYTHIA and GEANT3. A Kestrel-based Virtual Organization Cluster, created on top of Clemson University’s Palmetto cluster, CERN, and Amazon EC2, was able to provide over 400,000 CPU hours of computation over the course of a month using an average of 800 virtual machine instances every day, generating nearly seven terabytes of data and the largest PYTHIA production run that STAR has achieved to date. |
| first_indexed | 2024-09-23T09:36:56Z |
| format | Article |
| id | mit-1721.1/103623 |
| institution | Massachusetts Institute of Technology |
| language | English |
| last_indexed | 2024-09-23T09:36:56Z |
| publishDate | 2016 |
| publisher | Springer Netherlands |
| record_format | dspace |
| spelling | mit-1721.1/1036232022-04-01T16:26:06Z Using Kestrel and XMPP to Support the STAR Experiment in the Cloud Stout, Lance Walker, Matthew Lauret, Jérôme Goasguen, Sebastien Murphy, Michael A. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory Walker, Matthew This paper presents the results and experiences of adapting and improving the Many-Task Computing (MTC) framework Kestrel for use with bag of tasks applications and the STAR experiment in particular. Kestrel is a lightweight, highly available job scheduling framework for Virtual Organization Clusters (VOCs) constructed in the cloud. Kestrel uses the Extensible Message and Presence Protocol (XMPP) for increasing MTC platform scalability and mitigating faults in Wide Area Network (WAN) communications. Kestrel’s architecture is based upon pilot job frameworks used extensively in Grid computing, with fault-tolerant communications inspired by command-and-control botnets. The extensibility of XMPP has allowed development of protocols for identifying manager nodes, discovering the capabilities of worker agents, and for distributing tasks. Presence notifications provided by XMPP allow Kestrel to monitor the global state of the pool and to perform task dispatching based on worker availability. Since its inception, Kestrel has been modified based on its performance managing operational scientific workloads from the STAR group at Brookhaven National Laboratories. STAR provided a virtual machine image with applications for simulating proton collisions using PYTHIA and GEANT3. A Kestrel-based Virtual Organization Cluster, created on top of Clemson University’s Palmetto cluster, CERN, and Amazon EC2, was able to provide over 400,000 CPU hours of computation over the course of a month using an average of 800 virtual machine instances every day, generating nearly seven terabytes of data and the largest PYTHIA production run that STAR has achieved to date. 2016-07-15T18:01:12Z 2016-07-15T18:01:12Z 2013-04 2011-12 2016-05-23T12:07:39Z Article http://purl.org/eprint/type/JournalArticle 1570-7873 1572-9184 http://hdl.handle.net/1721.1/103623 Stout, Lance et al. “Using Kestrel and XMPP to Support the STAR Experiment in the Cloud.” Journal of Grid Computing 11.2 (2013): 249–264. en http://dx.doi.org/10.1007/s10723-013-9253-8 Journal of Grid Computing Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. Springer Science+Business Media Dordrecht application/pdf Springer Netherlands Springer Netherlands |
| spellingShingle | Stout, Lance Walker, Matthew Lauret, Jérôme Goasguen, Sebastien Murphy, Michael A. Using Kestrel and XMPP to Support the STAR Experiment in the Cloud |
| title | Using Kestrel and XMPP to Support the STAR Experiment in the Cloud |
| title_full | Using Kestrel and XMPP to Support the STAR Experiment in the Cloud |
| title_fullStr | Using Kestrel and XMPP to Support the STAR Experiment in the Cloud |
| title_full_unstemmed | Using Kestrel and XMPP to Support the STAR Experiment in the Cloud |
| title_short | Using Kestrel and XMPP to Support the STAR Experiment in the Cloud |
| title_sort | using kestrel and xmpp to support the star experiment in the cloud |
| url | http://hdl.handle.net/1721.1/103623 |
| work_keys_str_mv | AT stoutlance usingkestrelandxmpptosupportthestarexperimentinthecloud AT walkermatthew usingkestrelandxmpptosupportthestarexperimentinthecloud AT lauretjerome usingkestrelandxmpptosupportthestarexperimentinthecloud AT goasguensebastien usingkestrelandxmpptosupportthestarexperimentinthecloud AT murphymichaela usingkestrelandxmpptosupportthestarexperimentinthecloud |