e-Clouds: A Platform and Marketplace to Access and Publish Scientific Applications as a Service 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País Mario José Villamizar Cano Project Assistant – Cloud Computing Advisor Universidad de los Andes Bogotá, Colombia

THE PROBLEM RELATED WORK SOLUTION REQUIREMENTS SOLUTION ARCHITECTURE IMPLEMENTATION FUTURE WORK 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

THE PROBLEM When Small and Medium research groups have projects that requires large processing capabilities they regularly have to use dedicated cluster and/or grid computing infrastructures. However … 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

THE PROBLEMS WITH CLUSTER AND GRID 1)The total cost of ownership (TCO). 2)Complex processes related to IT management. 3)There are different models to parallelize applications. 4)Researchers require large computing capabilities during peak periods. 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

THE CLOUD COMPUTING MODEL - SaaS Users pay a subscription to access the service (not to buy the license or to buy and maintain hardware) 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

THE CLOUD COMPUTING MODEL 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

THE CLOUD COMPUTING MODEL - IaaS VM TypesvCPUsECUsMemoryStoragePrice/Hour USD m1.small111,701 x 1600,044 m1.medium123,751 x 4100,087 m1.large247,502 x 4200,175 m1.xlarge4815,004 x 4200,350 General Purpose Instances VM TypesvCPUsECUsMemoryStoragePrice/Hour USD c3.large273,752 x 16 SSD0,105 c3.xlarge4147,502 x 40 SSD0,210 c3.2xlarge82815,002 x 80 SSD0,420 c3.4xlarge165530,002 x 160 SSD0,840 c3.8xlarge ,002 x 320 SSD1,680 Compute Optimized Instances 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

THE CLOUD COMPUTING MODEL 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

THE PROBLEM Research groups of different sizes are adopting IaaS solutions. The adoption of cloud computing is a complex process, where non-IT researchers have to face similar or more complex processes as those found in cluster or grid infrastructures. 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

THE PROBLEM At enterprise level, the Software as a Service (SaaS) model allows that small businesses can easily adopt complex and ready-to-use applications. Problems associated to the management and the configuration of the underlying (PaaS, IaaS) infrastructure are abstracted. 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

RELATED WORK – e-Clouds GOAL e-Clouds: A SaaS marketplace where researches can access to on- demand and ready-to-use scientific applications via a Web Browser using a pay-per-use model. We believe that cloud computing may provide more opportunities for a large number of small and medium research groups. 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

RELATED WORK Some commercial scientific SaaS marketplaces such as Cyclone and Cloud Numbers are offering scientific SaaS application on public IaaS. 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

RELATED WORK 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

SOLUTION REQUIREMENTS In most in-house HPC/HTC environments, clusters are the core element used to execute scientific applications. 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

SOLUTION REQUIREMENTS A multi-tenant SaaS solution for scientific computing should allow that researchers ca access to different ready-to-use-applications, execute and monitor jobs, and the movement of data from and to the cloud; we call these three elements the workspace. In-house Cluster End-user interfaces Batch Applications Distributed File Systems External Libraries Cluster middleware Operating Systems Virtualization Servers Storage Networking Public Scientific SaaS (e-Clouds) End-user interfaces Batch Applications Cloud Storage Systems External Libraries Cluster middleware Operating Systems Virtualization Servers Storage Networking 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País Managed by researchers / groups

SOLUTION REQUIREMENTS Blob Storage Scalable storage systems that can grow to any size App Management On-demand installation of hundreds of scientific applications Scheduling Servers can be turned on or turned off as needed. 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

SOLUTION ARCHITECTURE e-Clouds is based on a new architecture that abstract the above problems, allowing researchers to focus their efforts in result analysis and experimental designs, and not on technical IT issues. 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País Application management Data management Job execution and monitoring PricingSecurity Workflow based execution Private/Public IaaS

SOLUTION ARCHITECTURE 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País  e-Clouds provides mechanisms that allow users to act as application configurators, facilitate the installation and configuration of new applications: cluster creation on demand.  Queue based communications and different components of e-Clouds scale automatically and allows e- Clouds to support the execution of large workloads.  Costs Efficiency.  Traditional directory structure based on scalable blob systems.

SOLUTION ARCHITECTURE 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País Researchers/users: 1.Choose the application. 2.Configure the execution. 3.Monitor the jobs/executions. 4.Download the results. 5.Pay only for what they use. e-Clouds is in charge of interacting with the public/private IaaS in all of the steps.

IMPLEMENTATION 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País Amazon Web Services (AWS) Ruby on Rails / Heroku. PostgreSQL. EC2, S3. e-Clouds Agent (Ruby). Script application in R of Von Humboldt Institute.

1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País IMPLEMENTATION

1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

IMPLEMENTATION 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

IMPLEMENTATION 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

IMPLEMENTATION 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

WHEN TO USE CLOUD SOLUTIONS SUCH AS e-Clouds Cases where Private and Commercial Clouds may be Cost Effective (Magellan R.): Unknown Demand. In the case of a new project or a new application where the potential demand is still poorly understood. Sporadic Demand. One of the more common cases for using commercial cloud offerings is when the demand is highly variable, especially if there are also time sensitive requirements for the service. Facility Constrained. Some sites are severely infrastructure limited. 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

FUTURE WORK Usability tests with more researchers. Large scale performance tests. Auto scaling and queues mechanisms. More Bag of Tasks applications. e-Clouds as an Open Source project. Shared workspaces. Different public IaaS. Types of applications (MPI, GPUs). Stronger security mechanisms. 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País

WE ARE LOOKING FOR BETA USERS/RESEARCHERS 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País Más información en: eclouds.uniandes.edu.co

THANKS FOR YOUR ATTENTION! Mario Villamizar - Universidad de los Andes 1er Foro Internacional de Bioinformática Ciencia y Tecnología para el Futuro del País