Äîêóìåíò âçÿò èç êýøà ïîèñêîâîé ìàøèíû. Àäðåñ îðèãèíàëüíîãî äîêóìåíòà : http://www.cosmos.ru/seminar/20151111719/presentation/Kryukov.pdf Äàòà èçìåíåíèÿ: Thu Nov 19 22:21:13 2015 Äàòà èíäåêñèðîâàíèÿ: Sun Apr 10 12:01:14 2016 Êîäèðîâêà: Ïîèñêîâûå ñëîâà: þæíàÿ àòëàíòè÷åñêàÿ àíîìàëèÿ

Development of Web platform for provision of application software as a Web service (SaaS) in open market approach
A. Kryukov, A. Demichev SINP MSU
Supported by RFBR grant No.15-07-09309 Tarusa, Nov. 17-19 , 2015 A.Kryukov, SINP MSU 1/33


Application software and distributed computing
Application software (especially simulation of complex systems) is a time expensive and resource consuming task simulation models require many simulation runs (sweep calculation) a viable solution to speed up the process is to run the simulations on a distributed system
a good example: WLCG project often local resources are insufficient

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

2/33


Success story of distributed computing (1/3)
The Worldwide LHC Computing Grid (WLCG)
an international collaborative project grid-based computer network infrastructure incorporating over 170 computing centers in 36 countries It was designed by CERN to handle the prodigious volume of data produced by Large Hadron Collider (LHC) experiments
approximately 25 petabytes per year per experiment
LHCb ATLAS CMS ALICE

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

3/33


Success story of distributed computing (2/3)
Inspired by the WLCG success, grid computing became popular beyond the high-energy physics
bioinformatics, nanotechnology, geophysics, etc.
to share and combine the power of computers and sophisticated, often unique, scientific instruments

The most ambitious attempt to extend the grid technology to other scientific areas undertaken in the framework of a series of projects DataGrid EGEE EGI (2001 - 2015)

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

4/33


Grid Web platforms
With the growth of performance of individual resources (supercomputers, data storages, cloud systems, etc.) the grid conception began to lose a significant part of its appeal
a large-scale distributed computing grid infrastructure high overheads
requires powerful unifying organizational structure, maintaining a cumbersome grid infrastructure (WLCG -- CERN)

the focus in the development of a new generation of middleware shifts to building convenient and efficient means for accessing individual computing resources
web platforms

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

5/33


Remote access to HPC resources (1/2)
direct access using the ssh protocol remains quite popular
provi the users the des the user with the flexibility in using the capabilities of resources (+) have to learn many subtleties of the environment in which applications are launched (-)

in many cases researchers need to run a large number of similar computing tasks

The model Software as a Service, SaaS Web platform (WP) for remote access to computing resources: a set of specialized web services + web application interfaces

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

6/33


Remote access to HPC resources (2/2)
A pre-arranged task (a launch of an application package, an access to a data storage, etc) = "a tool" A series of tasks depending on each other in a hierarchical way are called workflows Having in their disposal a set of pre-configured "tools", users only needs to formulate the essence of a specific request in a natural language Web platforms can offer specific services for deploment by authorized user of new software packages (tools)

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

7/33


Types of Web Platforms (WP)
1.WPs for job submission: remote submission, monitoring, and obtaining the job results; 2.WPs for job submission and software installation: item 1 + remote installation and configuring of application packages (tools); 3.Web hubs: items 1, 2 + providing the features of professional social networks (sharing experiences, rating of tools, etc); 4.WPs of application software market: items 1, 2, 3 + services for interaction between the providers and consumers of «tools» on market principles.

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

8/33


Basic Functional Requirements to WPs (1/2)
management of user credentials granting the right to use the available resources remote administration of the web platform via a web browser
Authentication, authorization, delegation

Accounts, granting the rights to use of sets of tools; accounts of tool providers

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

9/33


Basic Functional Requirements to WPs (2/2)
job execution management
creation of composite jobs (workflow); representation of the job description in the format of resource; job submission; monitoring; recive results and visualization of it Data transfere from/to local computer or storage service Register of new tools, building templates, handling of access grants to resources

data files transfer management tools (services) management

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

10/33


Administration of the WPs (1/2)
The most flexible approach to distributed resources managing is based on the concept of virtual organization (VO) a dynamic set of individuals or institutions defined around a set of resource-sharing rules and conditions VO may be responsible for the development/installation of the tools, day-by-day interaction with the resource owners/VO members: within the VO both consumers and owners of the resources cooperate. administration of resources gets easier because their owners only deal with VO managers, not with each user individually.

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

11/33


Administration of the WPs (2/2)
Users can just:
compose the tools available for their VO into a workflow provide files with input data and execution parameters for the tool of the workflow.

VO administrators' objective should include the creation and completion of a repository of VO's tools + installation on the widest possible range of resources. Repository of tools can be filled up both by open source software suitable for use in the area of interest of the VO and by the software developed by the members of the VO.

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

12/33


Examples of WPs Implementations (1/4)
web platform of educational-methodical software package «Multiscale modeling in nanotechnology» (Photochemistry Center of the Russian Academy of Sciences (RAS); http://www.nanomodel.ru); «Personal virtual computer» system (South Ural State University; http://supercomputer.susu.ac.ru/pvc); UniHUB, the technological platform of the National «University Cluster» program [13] (Institute for System Programming of RAS; https://unihub.ru); computing cloud platform of the Ural Branch of the RAS (Instit. of Math. and Mechanics, UrB RAS);

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

13/33


Examples of WPs Implementations (2/4)
web portal of the supercomputer management system (Glushkov Institute of Cybernetics of NAS of Ukraine; http://melkon.com.ua/ru/cms); Everest web platform (Institute for Information Transmission Problems of the RAS; http://everest.distcomp.org) multifunctional instrumental and technological platform for cloud computing support CLAVIRE (SPbg State University of Information Technologies, Mechanics and Optics; http://clavire.ru); nanoHUB, web hub in nanotechnology (consortium of the US universities; http://www.nanohub.org);

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

14/33


Examples of WPs Implementations (3/4)
eQUEUE, web platform for the remote job submission (Advanced Clustering Technologies, Inc.; http://www.advancedclustering.com/products/software/eque ue); Nucleonica, scientific web portal (Institute for Transuranium Elements; http://www.nucleonica.net); WebMO web platform (Hope College, Holland, USA; http://www.webmo.net); - e-Science Central (Newcastle University, UK; http://www.esciencecentral.co.uk). Yabi web platform (Centre for Comparative Genomics, Murdoch, Australia; https://ccg.murdoch.edu.au/yabi)

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

15/33


Examples of WPs Implementations (4/4)
almost all of these developments ensure the provision of end-users in advance preset simulation tools they are still insufficient to ensure the creation of a web platform capable of performing the whole range of tasks characteristic for a free open market

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

16/33


Typical example: Yabi Project
provides the execution of workflows consisting of successive tasks has well developed and convenient administrative interface for
configuring the "tools" controlling the user access to these tools accesses to databases, use of the results as input for computing tasks etc.

GNU GPL v3 (noncommercial use)

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

17/33


Yabi Project: architecture

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

18/33


Yabi Project: Resources
Yabi resource manager provides the two types of services: both types have plug-in architecture
compute services data services,

for the interaction with the external services

allows them to communicate with various compute resources and file storage systems credentials encrypted on behalf of the users (by using the login/password pair)

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

19/33


Yabi Project: Web Interface
Web GUI (browser) + CLI Web GUI -- three tabs:

the design view which allows construction of new workflows, the jobs view which shows previously submitted workflows along with their parameters and input files the files view that represents the data storages accessible by the user

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

20/33


Yabi Project: Web GUI

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

21/33


Yabi Project: administration (1/3)

administration module provides web interface that allows an authorized user (Yabi administrator) to manage all aspects of the web platform operation including

the creation of new tools from application software preinstalled on the resources controlling user access to various tools

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

22/33


Yabi Project: administration (2/3)
Creation of tools

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

23/33


Yabi Project: administration (3/3)
Information on completed jobs

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

24/33


Peculiarities of WPs with services of remote deployment of application software
various types of resources provided that application software is deployed by the resources administrators
WP may contain tools to facilitate interaction between software developers and resources administrators NanoHUB/HUBZero
these tools can include means for converting application software into S aaS

remote software installation by users themselves

resources is practically limited to the cloud systems (virtual machines) model Platform as a service (PaaS).

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

25/33


AppStore Paradigm (1/3)

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

26/33


AppStore Paradigm (2/3)
Software developers interact with the cloud management system via the WP WP contains modules needed for market-like interrelations between providers and users of tools
monitoring logging and billing PaaS model software deployment + convertation into the web tool

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

27/33


AppStore Paradigm (3/3)
A prototype of such a platform is currently being developed at SINP MSU From a technological point of view, the WP prototype be implemented as a set of web services
with extensive use of architectural style REST data exchange between the services - in the JSON format Supported by RFBR grant No.15-07-09309

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

28/33


Summary(1/3)
Further line of development of the web toolkit may be related not only with the quantitative increase in the number of web-based platforms for remote access and the expansion of scientific, engineering, and manufacturing areas in which they are used, but also with the improvement of the technology of remote deployment of new application software on resources interacting with the web platforms
This approach will help to overcome an important problem associated with the use of the SaaS model in scientific areas, namely, limited set of application packages offered by SaaS providers
Tarusa, Nov. 17-19 , 2015 A.Kryukov, SINP MSU 29/33


Summary (2/3)
Currently, the provision of services for providers of application software in the context of scientific-oriented web platforms is not developed enough. Although some implementations (for example, e-Science Central) have services for remote application software deployment, they are still insufficient to ensure the creation of a web platform capable of performing the whole range of tasks characteristic for a free open market.

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

30/33


Summary (3/3)
The technology of creating such web platforms market of application software can be based both on the original solutions and on the synthesis and adaptation of the solutions used in research hubs (e.g., nanoHUB; nanohub.org), cloud and grid systems, supercomputers, as well as in on-line app stores. However, it seems that unlike the on-line app stores, the platform should not only provide information services for searching the tools needed by users, but also provide the feasibility of direct using of the necessary tools. Thus, the future web platforms will provide a single entry point both for web service providers and for their customers.
Tarusa, Nov. 17-19 , 2015 A.Kryukov, SINP MSU 31/33


Conclusions
In the SINP MSU was started a new project for research and development of Web platform for scientific software application called eSciMarket which has next primary targets: Provides the users of software (SaaS) and provider of these software by the common Web platforms where they will be interact with each others. The basic principles of such interaction should be principles of free open market. The platform should be a tools itself to development of new SaaS, for example in the form "Workflow as a Service"
Tarusa, Nov. 17-19 , 2015 A.Kryukov, SINP MSU 32/33


Thank you for attention! Questions?

Tarusa, Nov. 17-19 , 2015

A.Kryukov, SINP MSU

33/33