Introduction · Web viewPhilips Gregor von Laszewski Indiana University Badi Abdhul Wahid Indiana...

NIST Special Publication 1500-9r1

DRAFT NIST Big Data Interoperability Framework:

Volume 8, Reference Architecture Interfaces

NIST Big Data Public Working GroupReference Architecture Subgroup

Version 3February 25, 2019

http://dx.doi.org/10.6028/---------

1

2

3

4

5

6

7

8

9

10

11

12

131415

161718

19

20

21

22

23

24

25

NIST Special Publication 1500-9r1Information Technology Laboratory

DRAFT NIST Big Data Interoperability Framework:

Volume 8, Reference Architecture InterfacesVersion 3

NIST Big Data Public Working Group (NBD-PWG)Definitions and Taxonomies Subgroup

National Institute of Standards and TechnologyGaithersburg, MD 20899

This draft publication is available free of charge from:http://dx.doi.org/10.6028/----------

February 2019

U. S. Department of CommerceWilbur L. Ross, Jr., Secretary

National Institute of Standards and TechnologyDr. Walter Copan Under Secretary of Commerce for Standards and Technology

and NIST Director

27

2829303132

33

34

3536

3738394041424344454647484950515253545556

5758596061626364

DRAFT NIST BIG DATA INTEROPERABILITY FRAMEWORK: VOLUME 8, REFERENCE ARCHITECTURE INTERFACES

National Institute of Standards and Technology (NIST) Special Publication 1500-9119 pages (February 25, 2019)

NIST Special Publication series 1500 is intended to capture external perspectives related to NIST standards, measurement, and testing-related efforts. These external perspectives can come from industry, academia, government, and others. These reports are intended to document external perspectives and do not represent official NIST positions.

Certain commercial entities, equipment, or materials may be identified in this document to describe an experimental procedure or concept adequately. Such identification is not intended to imply recommendation or endorsement by NIST, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose.

There may be references in this publication to other publications currently under development by NIST in accordance with its assigned statutory responsibilities. The information in this publication, including concepts and methodologies, may be used by federal agencies even before the completion of such companion publications. Thus, until each publication is completed, current requirements, guidelines, and procedures, where they exist, remain operative. For planning and transition purposes, federal agencies may wish to closely follow the development of these new publications by NIST.

Organizations are encouraged to review all publications during public comment periods and provide feedback to NIST. All NIST publications are available at http://www.nist.gov/publication-portal.cfm .

Comments on this publication may be submitted to Wo Chang

National Institute of Standards and TechnologyAttn: Wo Chang, Information Technology Laboratory

100 Bureau Drive (Mail Stop 8900) Gaithersburg, MD 20899-8930Email: [email protected]

iii

65

6667

68

69707172

73

74757677

787980818283

8485

86

87

888990919293949596

97

98

mailto:[email protected]

http://www.nist.gov/publication-portal.cfm


Reports on Computer Systems TechnologyThe Information Technology Laboratory (ITL) at NIST promotes the U.S. economy and public welfare by providing technical leadership for the Nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analyses to advance the development and productive use of information technology. ITL’s responsibilities include the development of management, administrative, technical, and physical standards and guidelines for the cost-effective security and privacy of other than national security-related information in federal information systems. This document reports on ITL’s research, guidance, and outreach efforts in Information Technology and its collaborative activities with industry, government, and academic organizations.

AbstractThis document summarizes interfaces that are instrumental for the interaction with Clouds, Containers, and High Performance Computing (HPC) systems to manage virtual clusters to support the NIST Big Data Reference Architecture (NBDRA). The REpresentational State Transfer (REST) paradigm is used to define these interfaces, allowing easy integration and adoption by a wide variety of frameworks.

Big Data is a term used to describe extensive datasets, primarily in the characteristics of volume, variety, velocity, and/or variability. While opportunities exist with Big Data, the data characteristics can overwhelm traditional technical approaches, and the growth of data is outpacing scientific and technological advances in data analytics. To advance progress in Big Data, the NIST Big Data Public Working Group (NBD-PWG) is working to develop consensus on important fundamental concepts related to Big Data. The results are reported in the NIST Big Data Interoperability Framework (NBDIF) series of volumes. This volume, Volume 8, uses the work performed by the NBD-PWG to identify objects instrumental for the NIST Big Data Reference Architecture (NBDRA) which is introduced in the NBDIF: Volume 6, Reference Architecture.

KeywordsAdoption; barriers; implementation; interfaces; market maturity; organizational maturity; project maturity; system modernization.

iv

99

100101102103104105106107108

109

110

111

112113114115

116117118119120121122123124

125

126

127

128129

130

131

132


AcknowledgementsThis document reflects the contributions and discussions by the membership of the NBD-PWG, cochaired by Wo Chang (NIST ITL), Bob Marcus (ET-Strategies), and Chaitan Baru (San Diego Supercomputer Center; National Science Foundation). For all versions, the Subgroups were led by the following people: Nancy Grady (SAIC), Natasha Balac (San Diego Supercomputer Center), and Eugene Luster (R2AD) for the Definitions and Taxonomies Subgroup; Geoffrey Fox (Indiana University) and Tsegereda Beyene (Cisco Systems) for the Use Cases and Requirements Subgroup; Arnab Roy (Fujitsu), Mark Underwood (Krypton Brothers; Synchrony Financial), and Akhil Manchanda (GE) for the Security and Privacy Subgroup; David Boyd (InCadence Strategic Solutions), Orit Levin (Microsoft), Don Krapohl (Augmented Intelligence), and James Ketner (AT&T) for the Reference Architecture Subgroup; and Russell Reinsch (Center for Government Interoperability), David Boyd (InCadence Strategic Solutions), Carl Buffington (Vistronix), and Dan McClary (Oracle), for the Standards Roadmap Subgroup.

The editors for this document were the following:

Version 1: This volume resulted from Stage 2 work and was not part of the Version 1 scope. Version 2: Gregor von Laszewski (Indiana University) and Wo Chang (NIST). Version3: Gregor von Laszewski (Indiana University) and Wo Chang (NIST).

Laurie Aldape (Energetics Incorporated) and Elizabeth Lennon (NIST) provided editorial assistance across all NBDIF volumes.

NIST SP 1500-9, Draft NIST Big Data Interoperability Framework: Volume 8, Reference Architecture Interfaces, Version 2 has been collaboratively authored by the NBD-PWG. As of the date of publication, there are over six hundred NBD-PWG participants from industry, academia, and government. Federal agency participants include the National Archives and Records Administration (NARA), National Aeronautics and Space Administration (NASA), National Science Foundation (NSF), and the U.S. Departments of Agriculture, Commerce, Defense, Energy, Census, Health and Human Services, Homeland Security, Transportation, Treasury, and Veterans Affairs.

NIST would like to acknowledge the specific contributionsa to this volume, during Version 1, Version 2, and/or Version 3 activities, by the following NBD-PWG members:

Wo ChangNational Institute of Standard and Technology

Geoffrey C. FoxIndiana University

Pratik ThakkarPhilips

Gregor von LaszewskiIndiana University

Badi Abdhul WahidIndiana University

Fugang WangIndiana University

Robert C. WhetselDISA/NBIS

Alicia Zuniga-AlvaradoConsultant

a “Contributors” are members of the NIST Big Data Public Working Group who dedicated great effort to prepare and gave substantial time on a regular basis to research and development in support of this document.

v

133

134135136137138139140141142143144

145

146147148

149150

151152153154155156157

158159

160

161

12


TABLE OF CONTENTSEXECUTIVE SUMMARY..................................................................................................................................... VIII

1 INTRODUCTION.......................................................................................................................................... 1

1.1 BACKGROUND.................................................................................................................................................11.2 SCOPE AND OBJECTIVES OF THE REFERENCE ARCHITECTURES SUBGROUP...................................................................31.3 REPORT PRODUCTION......................................................................................................................................31.4 REPORT STRUCTURE.........................................................................................................................................4

2 NBDRA INTERFACE REQUIREMENTS............................................................................................................ 5

2.1 HIGH-LEVEL REQUIREMENTS OF THE INTERFACE APPROACH....................................................................................62.1.1 Technology- and Vendor-Agnostic.........................................................................................................62.1.2 Support of Plug-In Compute Infrastructure............................................................................................62.1.3 Orchestration of Infrastructure and Services..........................................................................................72.1.4 Orchestration of Big Data Applications and Experiments.......................................................................72.1.5 Reusability..............................................................................................................................................72.1.6 Execution Workloads..............................................................................................................................72.1.7 Security and Privacy Fabric Requirements..............................................................................................7

2.2 COMPONENT-SPECIFIC INTERFACE REQUIREMENTS................................................................................................82.2.1 System Orchestrator Interface Requirements........................................................................................82.2.2 Data Provider Interface Requirements...................................................................................................92.2.3 Data Consumer Interface Requirements................................................................................................92.2.4 Big Data Application Interface Provider Requirements..........................................................................92.2.5 Big Data Provider Framework Interface Requirements........................................................................112.2.6 Big Data Application Provider to Big Data Framework Provider Interface...........................................12

3 SPECIFICATION PARADIGM....................................................................................................................... 13

3.1 HYBRID AND MULTIPLE FRAMEWORKS..............................................................................................................133.2 DESIGN BY RESOURCE-ORIENTED ARCHITECTURE.................................................................................................133.3 DESIGN BY EXAMPLE......................................................................................................................................133.4 VERSION MANAGEMENT.................................................................................................................................133.5 INTERFACE COMPLIANCY.................................................................................................................................13

4 EXAMPLE SPECIFICATION.......................................................................................................................... 15

4.1 LIST OF SPECIFICATIONS..................................................................................................................................154.2 IDENTITY......................................................................................................................................................17

4.2.1 Authentication.....................................................................................................................................174.2.2 Organization........................................................................................................................................184.2.3 User......................................................................................................................................................214.2.4 PublicKeyStore.....................................................................................................................................25

4.3 GENERAL RESOURCES.....................................................................................................................................284.3.1 Timestamp...........................................................................................................................................284.3.2 Alias.....................................................................................................................................................304.3.3 Variables..............................................................................................................................................334.3.4 Keyvaluestore.......................................................................................................................................354.3.5 Default.................................................................................................................................................38

4.4 DATA MANAGEMENT.....................................................................................................................................414.4.1 File.......................................................................................................................................................41

vi

162

163

164

165166167168

169

170171172173174175176177178179180181182183184

185

186187188189190

191

192193194195196197198199200201202203204205


4.4.2 Replica.................................................................................................................................................434.4.3 Database..............................................................................................................................................464.4.4 Virtual Directory...................................................................................................................................49

4.5 COMPUTE MANAGEMENT - VIRTUAL CLUSTERS...................................................................................................524.5.1 Virtual Cluster......................................................................................................................................524.5.2 Scheduler.............................................................................................................................................57

4.6 COMPUTE MANAGEMENT - VIRTUAL MACHINES.................................................................................................604.6.1 Image...................................................................................................................................................604.6.2 Flavor...................................................................................................................................................634.6.3 VM.......................................................................................................................................................674.6.4 Secgroup..............................................................................................................................................694.6.5 Nic........................................................................................................................................................75

4.7 COMPUTE MANAGEMENT - CONTAINERS...........................................................................................................784.7.1 Containers............................................................................................................................................78

4.8 COMPUTE MANAGEMENT - FUNCTIONS.............................................................................................................814.8.1 Microservice.........................................................................................................................................81

4.9 BATCH PROCESSING.......................................................................................................................................834.9.1 Batchjob...............................................................................................................................................834.9.2 Slurmjob...............................................................................................................................................86

4.10 RESERVATION...............................................................................................................................................894.10.1 Reservation......................................................................................................................................89

4.11 DATA STREAMS.............................................................................................................................................924.11.1 Stream.............................................................................................................................................924.11.2 Filter................................................................................................................................................95

4.12 MAPREDUCE................................................................................................................................................974.12.1 Hadoop............................................................................................................................................97

4.13 DEPLOYMENT.............................................................................................................................................1004.13.1 Deployment...................................................................................................................................100

5 STATUS CODES AND ERROR RESPONSES..................................................................................................104

APPENDIX A: ACRONYMS AND TERMS.............................................................................................................105

APPENDIX B: BIBLIOGRAPHY........................................................................................................................... 107

FIGURES

FIGURE 1: NIST BIG DATA REFERENCE ARCHITECTURE (NBDRA)...........................................................................5

FIGURE 2: PROVIDER VIEW................................................................................................................................ 16

FIGURE 3: RESOURCE VIEW............................................................................................................................... 17

TABLES

TABLE 1: LIST OF SPECIFICATIONS...................................................................................................................... 15

TABLE 2: HTTP RESPONSE CODES..................................................................................................................... 104

vii

206207208209210211212213214215216217218219220221222223224225226227228229230231232233

234

235

236

237

238

239

240

241

242

243

244

245

246

247


Executive SummaryThe NIST Big Data Interoperability Framework (NBDIF): Volume 8, Reference Architecture Interfaces document was prepared by the NIST Big Data Public Working Group (NBD-PWG) Reference Architecture Subgroup to identify interfaces in support of the NIST Big Data Reference Architecture (NBDRA) The interfaces contain two different aspects:

The definition of resources that are part of the NBDRA. These resources are formulated in JavaScript Object Notation (JSON) format and can be easily integrated into a REpresentational State Transfer (REST) framework or an object-based framework.

The definition of simple interface use cases that allow us to illustrate the usefulness of the resources defined.

The resources were categorized in groups that are identified by the NBDRA set forward in the NBDIF: Volume 6, Reference Architecture document. While the NBDIF: Volume 3, Use Cases and General Requirements document provides application-oriented high-level use cases, the use cases defined in this document are subsets of them and focus on interface use cases. The interface use cases are not meant to be complete examples, but showcase why the resource has been defined. Hence, the interfaces use cases are only representative, and do not encompass the entire spectrum of Big Data usage. All the interfaces were openly discussed in the working group. Additions to the interfaces are welcome and the NBD-PWG is open to discuss any contributions.

The NIST Big Data Interoperability Framework consists of nine volumes, each of which addresses a specific key topic, resulting from the work of the NBD-PWG. The nine volumes are as follows:

Volume 1: Definitions [1] Volume 2: Taxonomies [2] Volume 3: Use Cases and General Requirements [3] Volume 4: Security and Privacy [4] Volume 5: Architectures White Paper Survey [5] Volume 6: Reference Architecture [6] Volume 7: Standards Roadmap [7] Volume 8: Reference Architecture Interfaces (this volume) Volume 9: Adoption and Modernization [8]

The NBDIF will be released in three versions, which correspond to the three development stages of the NBD-PWG work. The three stages aim to achieve the following with respect to the NBDRA.

Stage 1. Identify the high-level Big Data reference architecture key components, which are technology-, infrastructure-, and vendor-agnostic.

Stage 2. Define general interfaces between the NBDRA components. Stage 3. Validate the NBDRA by building Big Data general applications through the general

interfaces.

This document is the result of Stage 3 work of the NBD-PWG. Coordination of the group is conducted on the NBD-PWG web page (https://bigdatawg.nist.gov).

viii

248

249250251252

253254255256257

258259260261262263264265

266267

268269270271272273274275276

277278

279280281282283

284285

286

https://bigdatawg.nist.gov/


1 INTRODUCTION1.1 BACKGROUNDThere is broad agreement among commercial, academic, and government leaders about the remarkable potential of Big Data to spark innovation, fuel commerce, and drive progress. Big Data is the common term used to describe the deluge of data in today’s networked, digitized, sensor-laden, and information-driven world. The availability of vast data resources carries the potential to answer questions previously out of reach, including the following:

How can a potential pandemic reliably be detected early enough to intervene? Can new materials with advanced properties be predicted before these materials have ever been

synthesized? How can the current advantage of the attacker over the defender in guarding against cybersecurity

threats be reversed?

There is also broad agreement on the ability of Big Data to overwhelm traditional approaches. The growth rates for data volumes, speeds, and complexity are outpacing scientific and technological advances in data analytics, management, transport, and data user spheres.

Despite widespread agreement on the inherent opportunities and current limitations of Big Data, a lack of consensus on some important fundamental questions continues to confuse potential users and stymie progress. These questions include the following:

How is Big Data defined? What attributes define Big Data solutions? What is new in Big Data? What is the difference between Big Data and bigger data that has been collected for years? How is Big Data different from traditional data environments and related applications? What are the essential characteristics of Big Data environments? How do these environments integrate with currently deployed architectures? What are the central scientific, technological, and standardization challenges that need to be

addressed to accelerate the deployment of robust, secure Big Data solutions?

Within this context, on March 29, 2012, the White House announced the Big Data Research and Development Initiative (The White House Office of Science and Technology Policy, “Big Data is a Big Deal,” OSTP Blog, accessed February 21, 2014 [9]. The initiative’s goals include helping to accelerate the pace of discovery in science and engineering, strengthening national security, and transforming teaching and learning by improving analysts’ ability to extract knowledge and insights from large and complex collections of digital data.

Six federal departments and their agencies announced more than $200 million in commitments spread across more than 80 projects, which aim to significantly improve the tools and techniques needed to access, organize, and draw conclusions from huge volumes of digital data. The initiative also challenged industry, research universities, and nonprofits to join with the federal government to make the most of the opportunities created by Big Data.

Motivated by the White House initiative and public suggestions, the National Institute of Standards and Technology (NIST) accepted the challenge to stimulate collaboration among industry professionals to further the secure and effective adoption of Big Data. As a result of NIST’s Cloud and Big Data Forum held on January 15–17, 2013, there was strong encouragement for NIST to create a public working group

1

287

288

289290291292293

294295296297298

299300301

302303304

305306307308309310311312313

314315316317318319

320321322323324

325326327328


for the development of a Big Data Standards Roadmap. Forum participants noted that this roadmap should define and prioritize Big Data requirements, including interoperability, portability, reusability, extensibility, data usage, analytics, and technology infrastructure. In doing so, the roadmap would accelerate the adoption of the most secure and effective Big Data techniques and technology.

On June 19, 2013, the NIST Big Data Public Working Group (NBD-PWG) was launched with extensive participation by industry, academia, and government from across the nation. The scope of the NBD-PWG involves forming a community of interests from all sectors—including industry, academia, and government—with the goal of developing consensus on definitions, taxonomies, secure reference architectures, security and privacy, and, from these, a standards roadmap. Such a consensus would create a vendor-neutral, technology- and infrastructure-independent framework that would enable Big Data stakeholders to identify and use the best analytics tools for their processing and visualization requirements on the most suitable computing platform and cluster, while also allowing added value from Big Data service providers.

The NIST Big Data Interoperability Framework (NBDIF) will be released in three versions, which correspond to the three stages of the NBD-PWG work. The three stages aim to achieve the following with respect to the NIST Big Data Reference Architecture (NBDRA).

Stage 1: Identify the high-level Big Data reference architecture key components, which are technology, infrastructure, and vendor agnostic.

Stage 2: Define general interfaces between the NBDRA components. Stage 3: Validate the NBDRA by building Big Data general applications through the general

interfaces.

On September 16, 2015, seven NBDIF Version 1 volumes were published (http://bigdatawg.nist.gov/V1_output_docs.php), each of which addresses a specific key topic, resulting from the work of the NBD-PWG. The seven volumes are as follows:

Volume 1, Definitions [1] Volume 2, Taxonomies [2] Volume 3, Use Cases and General Requirements [3] Volume 4, Security and Privacy [4] Volume 5, Architectures White Paper Survey [5] Volume 6, Reference Architecture [6] Volume 7, Standards Roadmap [7]

The NBD-PWG worked on Stage 2 with the goals to enhance the Version 1 content, define general interfaces between the NBDRA components by aggregating low-level interactions into high-level general interfaces, and demonstrate how the NBDRA can be used. As a result of the Stage 2 work, the following two additional NBDIF volumes were developed.

Volume 8, Reference Architecture Interfaces (this volume) Volume 9, Adoption and Modernization [8]

Version 2 of the NBDIF volumes, resulting from Stage 2 work, can be downloaded from the NBD-PWG website (https://bigdatawg.nist.gov/V2_output_docs.php). The current effort documented in this volume reflects concepts developed within the rapidly evolving field of Big Data.

1.2 SCOPE AND OBJECTIVES OF THE REFERENCE ARCHITECTURES SUBGROUP

Reference architectures provide “an authoritative source of information about a specific subject area that guides and constrains the instantiations of multiple architectures and solutions” [10]. Reference

2

329330331332

333334335336337338339340341

342343344

345346347348349

350351352

353354355356357358359

360361362363

364365

366367368

369

370

371372

https://bigdatawg.nist.gov/V2_output_docs.php

http://bigdatawg.nist.gov/V1_output_docs.php


architectures generally serve as a foundation for solution architectures and may also be used for comparison and alignment of instantiations of architectures and solutions.

The goal of the NBD-PWG Reference Architecture Subgroup is to develop an open reference architecture for Big Data that achieves the following objectives:

Provides a common language for the various stakeholders; Encourages adherence to common standards, specifications, and patterns; Provides consistent methods for implementation of technology to solve similar problem sets; Illustrates and improves understanding of the various Big Data components, processes, and

systems, in the context of a vendor- and technology-agnostic Big Data conceptual model; Provides a technical reference for U.S. government departments, agencies, and other consumers

to understand, discuss, categorize, and compare Big Data solutions; and Facilitates analysis of candidate standards for interoperability, portability, reusability, and

extendibility.

The NBDRA is a high-level conceptual model crafted to serve as a tool to facilitate open discussion of the requirements, design structures, and operations inherent in Big Data. The NBDRA is intended to facilitate the understanding of the operational intricacies in Big Data. It does not represent the system architecture of a specific Big Data system, but rather is a tool for describing, discussing, and developing system-specific architectures using a common framework of reference. The model is not tied to any specific vendor products, services, or reference implementation, nor does it define prescriptive solutions that inhibit innovation.

The NBDRA does not address the following:

Detailed specifications for any organization’s operational systems; Detailed specifications of information exchanges or services; and Recommendations or standards for integration of infrastructure products.

The goals of the Subgroup were realized throughout the three planned phases of the NBD-PWG work, as outlined in Section 1.3.

1.3 REPORT PRODUCTIONThe NBDIF: Volume 8, References Architecture Interfaces is one of nine volumes, whose overall aims are to define and prioritize Big Data requirements, including interoperability, portability, reusability, extensibility, data usage, analytic techniques, and technology infrastructure to support secure and effective adoption of Big Data. The overall goals of this volume are to define and specify interfaces to implement the Big Data Reference Architecture. This volume arose from discussions during the weekly NBD-PWG conference calls. Topics included in this volume began to take form in Phase 2 of the NBD-PWG work. During the discussions, the NBD-PWG identified the need to specify a variety of interfaces.

Phase 3 work, which built upon the groundwork developed during Phase 2, included an early specification based on resource object specifications that provided a simplified version of an API interface design. To achieve technical and high-quality document content, this document will go through a public comment period along with NIST internal review.

The following milestone were achieved in the versions produced listed below:

Version 2.1: A previous volume used just the definition of the schema based on examples. It was easier to read but only included the definition of the resources and not the interaction with the resources. This volume was in place until June 2018.

Version 2.2: This version was significantly changed and now uses OpenAPI to specify the Interfaces between the various services and components.

3

373374

375376

377378379380381382383384385

386387388389390391392

393

394395396

397398

399

400401402403404405406

407408409410

411

412413414415416


Version 3.1.0: The version includes the significant improvements of the object specifications.

1.4 REPORT STRUCTURETo enable interoperability between the NBDRA components, a list of well-defined NBDRA interfaces is needed. These interfaces are documented in this volume. To introduce them, the NBDRA structure will be followed, focusing on interfaces that allow bootstrapping of the NBDRA. The document begins with a summary of requirements that will be integrated into our specifications. Subsequently, each section will introduce a number of objects that build the core of the interface addressing a specific aspect of the NBDRA. A selected number of interface use cases will be showcased to outline how the specific interface can be used in a reference implementation of the NBDRA. Validation of this approach can be achieved while applying it to the application use cases that have been gathered in the NBDIF: Volume 3, Use Cases and Requirements document. These application use cases have considerably contributed towards the design of the NBDRA. Hence the expectation is that: (1) the interfaces can be used to help implement a Big Data architecture for a specific use case; and (2) the proper implementation. This approach can facilitate subsequent analysis and comparison of the use cases.

The organization of this document roughly corresponds to the process used by the NBD-PWG to develop the interfaces. Following the introductory material presented in Section 1, the remainder of this document is organized as follows:

Section 2 presents the interface requirements; Section 3 presents the specification paradign the we use; Section 4 presents several objects grouped by functional use while providing a summary table of

selected proposed objects in Section 4.1.

4

417

418

419420421422423424425426427428429430

431432433

434435436437

438


2 NBDRA INTERFACE REQUIREMENTSThe development of a Big Data reference architecture requires a thorough understanding of current techniques, issues, and concerns. To this end, the NBD-PWG collected use cases to gain an understanding of current applications of Big Data, conducted a survey of reference architectures to understand commonalities within Big Data architectures in use, developed a taxonomy to understand and organize the information collected, and reviewed existing technologies and trends relevant to Big Data. The results of these NBD-PWG activities were used in the development of the NBDRA (Figure 1) and the interfaces presented herein. Detailed descriptions of these activities can be found in the other volumes of the NBDIF.

Figure 1: NIST Big Data Reference Architecture (NBDRA)

This vendor-neutral, technology- and infrastructure-agnostic conceptual model, the NBDRA, is shown in Figure 1 and represents a Big Data system composed of five logical functional components connected by interoperability interfaces (i.e., services). Two fabrics envelop the components, representing the interwoven nature of management and security and privacy with all five of the components. These two fabrics provide services and functionality to the five main roles in the areas specific to Big Data and are crucial to any Big Data solution. Note: None of the terminology or diagrams in these documents is intended to be normative or to imply any business or deployment model. The terms provider and consumer as used are descriptive of general roles and are meant to be informative in nature.

5

439

440441442443444445446447

448449

450451452453454455456457


The NBDRA is organized around five major roles and multiple sub-roles aligned along two axes representing the two Big Data value chains: the Information Value (horizontal axis) and the Information Technology (IT; vertical axis). Along the Information Value axis, the value is created by data collection, integration, analysis, and applying the results following the value chain. Along the IT axis, the value is created by providing networking, infrastructure, platforms, application tools, and other IT services for hosting of and operating the Big Data in support of required data applications. At the intersection of both axes is the Big Data Application Provider role, indicating that data analytics and its implementation provide the value to Big Data stakeholders in both value chains. The term provider as part of the Big Data Application Provider and Big Data Framework Provider is there to indicate that those roles provide or implement specific activities and functions within the system. It does not designate a service model or business entity.

The DATA arrows in Figure 1 show the flow of data between the system’s main roles. Data flows between the roles either physically (i.e., by value) or by providing its location and the means to access it (i.e., by reference). The SW arrows show transfer of software tools for processing of Big Data in situ. The Service Use arrows represent software programmable interfaces. While the main focus of the NBDRA is to represent the run-time environment, all three types of communications or transactions can happen in the configuration phase as well. Manual agreements (e.g., service-level agreements) and human interactions that may exist throughout the system are not shown in the NBDRA.

Detailed information on the NBDRA conceptual model is presented in the NBDIF: Volume 6, Reference Architecture document.

Prior to outlining the specific interfaces, general requirements are introduced and the interfaces are defined.

2.1 HIGH-LEVEL REQUIREMENTS OF THE INTERFACE APPROACH

This section focuses on the high-level requirements of the interface approach that are needed to implement the reference architecture depicted in Figure 1.

2.1.1 TECHNOLOGY- AND VENDOR-AGNOSTICDue to the many different tools, services, and infrastructures available in the general area of Big Data, an interface ought to be as vendor-independent as possible, while, at the same time, be able to leverage best practices. Hence, a methodology is needed that allows extension of interfaces to adapt and leverage existing approaches, but also allows the interfaces to provide merit in easy specifications that assist the formulation and definition of the NBDRA.

2.1.2 SUPPORT OF PLUG-IN COMPUTE INFRASTRUCTUREAs Big Data is not just about hosting data, but about analyzing data, the interfaces provided herein must encapsulate a rich infrastructure environment that is used by data scientists. This includes the ability to integrate (or plug-in) various compute resources and services to provide the necessary compute power to analyze the data. These resources and services include the following:

Access to hierarchy of compute resources from the laptop/desktop, servers, data clusters, and clouds;

The ability to integrate special-purpose hardware such as graphics processing units (GPUs) and field-programmable gate arrays (FPGAs) that are used in accelerated analysis of data; and

The integration of services including microservices that allow the analysis of the data by delegating them to hosted or dynamically deployed services on the infrastructure of choice.

6

458459460461462463464465466467468

469470471472473474475

476477

478479

480

481

482483

484

485486487488489

490

491492493494

495496497498499500


2.1.3 ORCHESTRATION OF INFRASTRUCTURE AND SERVICESFrom review of the use case collection, presented in the NBDIF: Volume 3, Use Cases and General Requirements document [4], the need arose to address the mechanism of preparing suitable infrastructures for various use cases. As not every infrastructure is suited for every use case, a custom infrastructure may be needed. As such, this document is not attempting to deliver a single deployed NBDRA, but allow the setup of an infrastructure that satisfies the particular use case. To achieve this task, it is necessary to provision software stacks and services while orchestrating their deployment and leveraging infrastructures. It is not the focus of this document to replace existing orchestration software and services, but provide an interface to them to leverage them as part of defining and creating the infrastructure. Various orchestration frameworks and services could therefore be leveraged, even as part of the same framework, and work in orchestrated fashion to achieve the goal of preparing an infrastructure suitable for one or more applications.

2.1.4 ORCHESTRATION OF BIG DATA APPLICATIONS AND EXPERIMENTSThe creation of the infrastructure suitable for Big Data applications provides the basic computing environment. However, Big Data applications may require the creation of sophisticated applications as part of interactive experiments to analyze and probe the data. For this purpose, the applications must be able to orchestrate and interact with experiments conducted on the data while assuring reproducibility and correctness of the data. For this purpose, a System Orchestrator (either the data scientists or a service acting on behalf of the data scientist) is used as the command center to interact on behalf of the Big Data Application Provider to orchestrate dataflow from Data Provider, carry out the Big Data application life cycle with the help of the Big Data Framework Provider, and enable the Data Consumer to consume Big Data processing results. An interface is needed to describe these interactions and to allow leveraging of experiment management frameworks in scripted fashion. A customization of parameters is needed on several levels. On the highest level, application-motivated parameters are needed to drive the orchestration of the experiment. On lower levels, these high-level parameters may drive and create service-level agreements, augmented specifications, and parameters that could even lead to the orchestration of infrastructure and services to satisfy experiment needs.

2.1.5 REUSABILITYThe interfaces provided must encourage reusability of the infrastructure, services, and experiments described by them. This includes (1) reusability of available analytics packages and services for adoption; (2) deployment of customizable analytics tools and services; and (3) operational adjustments that allow the services and infrastructure to be adapted while at the same time allowing for reproducible experiment execution.

2.1.6 EXECUTION WORKLOADSOne of the important aspects of distributed Big Data services can be that the data served is simply too big to be moved to a different location. Instead, an interface could allow the description and packaging of analytics algorithms, and potentially also tools, as a payload to a data service. This can be best achieved, not by sending the detailed execution, but by sending an interface description that describes how such an algorithm or tool can be created on the server and be executed under security considerations (integrated with authentication and authorization in mind).

2.1.7 SECURITY AND PRIVACY FABRIC REQUIREMENTSAlthough the focus of this document is not security and privacy, which are documented in the NBDIF: Volume 4, Security and Privacy [8], the interfaces defined herein must be capable of integration into a secure reference architecture that supports secure execution, secure data transfer, and privacy.

7

501

502503504505506507508509510511512

513

514515516517518519520521522523524525526527

528

529530531532533

534

535536537538539540

541

542543544


Consequently, the interfaces defined herein can be augmented with frameworks and solutions that provide such mechanisms. Thus, diverse requirement needs stemming from different use cases addressing security need to be distinguished. To contrast that the security requirements between applications can vary drastically, the following example is provided. Although many of the interfaces and their objects to support Big Data applications in physics are similar to those in healthcare, they differ in the integration of security interfaces and policies. While in physics the protection of data is less of an issue, it is a stringent requirement in healthcare. Thus, deriving architectural frameworks for both may use largely similar components, but addressing security issues will be very different. The security of interfaces may be addressed in other documents. In this document, they are considered an advanced use case showcasing that the validity of the specifications introduced here is preserved, even if security and privacy requirements differ vastly among application use cases.

2.2 COMPONENT-SPECIFIC INTERFACE REQUIREMENTSThis section summarizes the requirements for the interfaces of the NBDRA components. The five components are listed in Figure 1 and addressed in Section 2.2.1 (System Orchestrator Interface Requirements) and Section 2.2.4 (Big Data Application Provider to Big Data Framework Provider Interface) of this document. The five main functional components of the NBDRA represent the different technical roles within a Big Data system and are the following:

System Orchestrator: Defines and integrates the required data application activities into an operational vertical system (see Section 2.2.1);

Data Provider: Introduces new data or information feeds into the Big Data system (see Section 2.2.2);

Data Consumer: Includes end users or other systems that use the results of the Big Data Application Provider (see Section 2.2.3);

Big Data Application Provider: Executes a data life cycle to meet security and privacy requirements as well as System Orchestrator-defined requirements (see Section 2.2.4);

Big Data Framework Provider: Establishes a computing framework in which to execute certain transformation applications while protecting the privacy and integrity of data (see Section 2.2.5); and

Big Data Application Provider to Framework Provider Interface: Defines an interface between the application specification and the provider (see Section 2.2.6).

2.2.1 SYSTEM ORCHESTRATOR INTERFACE REQUIREMENTSThe System Orchestrator role includes defining and integrating the required data application activities into an operational vertical system. Typically, the System Orchestrator involves a collection of more specific roles, performed by one or more actors, which manage and orchestrate the operation of the Big Data system. These actors may be human components, software components, or some combination of the two. The function of the System Orchestrator is to configure and manage the other components of the Big Data architecture to implement one or more workloads that the architecture is designed to execute. The workloads managed by the System Orchestrator may be assigning/provisioning framework components to individual physical or virtual nodes at the lower level, or providing a graphical user interface that supports the specification of workflows linking together multiple applications and components at the higher level. The System Orchestrator may also, through the Management Fabric, monitor the workloads and system to confirm that specific quality of service requirements is met for each workload, and may elastically assign and provision additional physical or virtual resources to meet workload requirements resulting from changes/surges in the data or number of users/transactions. The interface to the System Orchestrator must be capable of specifying the task of orchestration the deployment, configuration, and the execution of applications within the NBDRA. A simple vendor-neutral specification to coordinate the various parts

8

545546547548549550551552553554555

556

557558559560561

562563564565566567568569570571572573574

575

576577578579580581582583584585586587588589590


either as simple parallel language tasks or as a workflow specification is needed to facilitate the overall coordination. Integration of existing tools and services into the System Orchestrator as extensible interfaces is desirable.

2.2.2 DATA PROVIDER INTERFACE REQUIREMENTSThe Data Provider role introduces new data or information feeds into the Big Data system for discovery, access, and transformation by the Big Data system. New data feeds are distinct from the data already in use by the system and residing in the various system repositories. Similar technologies can be used to access both new data feeds and existing data. The Data Provider actors can be anything from a sensor, to a human inputting data manually, to another Big Data system. Interfaces for data providers must be able to specify a data provider so it can be located by a data consumer. It also must include enough details to identify the services offered so they can be pragmatically reused by consumers. Interfaces to describe pipes and filters must be addressed.

2.2.3 DATA CONSUMER INTERFACE REQUIREMENTSLike the Data Provider, the role of Data Consumer within the NBDRA can be an actual end user or another system. In many ways, this role is the mirror image of the Data Provider, with the entire Big Data framework appearing like a Data Provider to the Data Consumer. The activities associated with the Data Consumer role include the following:

Search and Retrieve, Download, Analyze Locally, Reporting, Visualization, and Data to Use for Their Own Processes.

The interface for the data consumer must be able to describe the consuming services and how they retrieve information or leverage data consumers.

2.2.4 BIG DATA APPLICATION INTERFACE PROVIDER REQUIREMENTSThe Big Data Application Provider role executes a specific set of operations along the data life cycle to meet the requirements established by the System Orchestrator, as well as meeting security and privacy requirements. The Big Data Application Provider is the architecture component that encapsulates the business logic and functionality to be executed by the architecture. The interfaces to describe Big Data applications include interfaces for the various subcomponents including collections, preparation/curation, analytics, visualization, and access. Some of the interfaces used in these subcomponents can be reused from other interfaces, which are introduced in other sections of this document. Where appropriate, application-specific interfaces will be identified and examples provided with a focus on use cases as identified in the NBDIF: Volume 3, Use Cases and General Requirements.

2.2.4.1 CollectionIn general, the collection activity of the Big Data Application Provider handles the interface with the Data Provider. This may be a general service, such as a file server or web server configured by the System Orchestrator to accept or perform specific collections of data, or it may be an application-specific service designed to pull data or receive pushes of data from the Data Provider. Since this activity is receiving data at a minimum, it must store/buffer the received data until it is persisted through the Big Data Framework Provider. This persistence need not be to physical media but may simply be to an in-memory queue or other service provided by the processing frameworks of the Big Data Framework Provider. The collection activity is likely where the extraction portion of the Extract, Transform, Load (ETL)/Extract, Load,

9

591592593

594

595596597598599600601602

603

604605606607

608609610611612613

614615

616

617618619620621622623624625

626

627628629630631632633634


Transform (ELT) cycle is performed. At the initial collection stage, sets of data (e.g., data records) of similar structure are collected (and combined), resulting in uniform security, policy, and other considerations. Initial metadata is created (e.g., subjects with keys are identified) to facilitate subsequent aggregation or look-up methods.

2.2.4.2 PreparationThe preparation activity is where the transformation portion of the ETL/ELT cycle is likely performed, although analytics activity will also likely perform advanced parts of the transformation. Tasks performed by this activity could include data validation (e.g., checksums/hashes, format checks), cleansing (e.g., eliminating bad records/fields), outlier removal, standardization, reformatting, or encapsulating. This activity is also where source data will frequently be persisted to archive storage in the Big Data Framework Provider and provenance data will be verified or attached/associated. Verification or attachment may include optimization of data through manipulations (e.g., deduplication) and indexing to optimize the analytics process. This activity may also aggregate data from different Data Providers, leveraging metadata keys to create an expanded and enhanced data set.

2.2.4.3 AnalyticsThe analytics activity of the Big Data Application Provider includes the encoding of the low-level business logic of the Big Data system (with higher-level business process logic being encoded by the System Orchestrator). The activity implements the techniques to extract knowledge from the data based on the requirements of the vertical application. The requirements specify the data processing algorithms to produce new insights that will address the technical goal. The analytics activity will leverage the processing frameworks to implement the associated logic. This typically involves the activity providing software that implements the analytic logic to the batch and/or streaming elements of the processing framework for execution. The messaging/communication framework of the Big Data Framework Provider may be used to pass data or control functions to the application logic running in the processing frameworks. The analytic logic may be broken up into multiple modules to be executed by the processing frameworks which communicate, through the messaging/communication framework, with each other and other functions instantiated by the Big Data Application Provider.

2.2.4.4 VisualizationThe visualization activity of the Big Data Application Provider prepares elements of the processed data and the output of the analytic activity for presentation to the Data Consumer. The objective of this activity is to format and present data in such a way as to optimally communicate meaning and knowledge. The visualization preparation may involve producing a text-based report or rendering the analytic results as some form of graphic. The resulting output may be a static visualization and may simply be stored through the Big Data Framework Provider for later access. However, the visualization activity frequently interacts with the access activity, the analytics activity, and the Big Data Framework Provider (processing and platform) to provide interactive visualization of the data to the Data Consumer based on parameters provided to the access activity by the Data Consumer. The visualization activity may be completely application-implemented, leverage one or more application libraries, or may use specialized visualization processing frameworks within the Big Data Framework Provider.

2.2.4.5 AccessThe access activity within the Big Data Application Provider is focused on the communication/interaction with the Data Consumer. Like the collection activity, the access activity may be a generic service such as a web server or application server that is configured by the System Orchestrator to handle specific requests from the Data Consumer. This activity would interface with the visualization and analytic activities to respond to requests from the Data Consumer (who may be a person) and uses the processing

10

635636637638

639

640641642643644645646647648

649

650651652653654655656657658659660661

662

663664665666667668669670671672673

674

675676677678679


and platform frameworks to retrieve data to respond to Data Consumer requests. In addition, the access activity confirms that descriptive and administrative metadata and metadata schemes are captured and maintained for access by the Data Consumer and as data is transferred to the Data Consumer. The interface with the Data Consumer may be synchronous or asynchronous in nature and may use a pull or push paradigm for data transfer.

2.2.5 BIG DATA PROVIDER FRAMEWORK INTERFACE REQUIREMENTSData for Big Data applications are delivered through data providers. They can be either local providers, data contributed by a user, or distributed data providers, data on the Internet. This interface must be able to provide the following functionality:

Interfaces to files, Interfaces to virtual data directories, Interfaces to data streams, and Interfaces to data filters.

2.2.5.1 Infrastructures Interface RequirementsThis Big Data Framework Provider element provides all the resources necessary to host/run the activities of the other components of the Big Data system. Typically, these resources consist of some combination of physical resources, which may host/support similar virtual resources. The NBDRA needs interfaces that can be used to deal with the underlying infrastructure to address networking, computing, and storage.

2.2.5.2 Platforms Interface RequirementsAs part of the NBDRA platforms, interfaces are needed that can address platform needs and services for data organization, data distribution, indexed storage, and file systems.

2.2.5.3 Processing Interface RequirementsThe processing frameworks for Big Data provide the necessary infrastructure software to support implementation of applications that can deal with the volume, velocity, variety, and variability of data. Processing frameworks define how the computation and processing of the data is organized. Big Data applications rely on various platforms and technologies to meet the challenges of scalable data analytics and operation. A requirement is the ability to interface easily with computing services that offer specific analytics services, batch processing capabilities, interactive analysis, and data streaming.

2.2.5.4 Crosscutting Interface RequirementsSeveral crosscutting interface requirements within the Big Data Framework Provider include messaging, communication, and resource management. Often these services may be hidden from explicit interface use as they are part of larger systems that expose higher-level functionality through their interfaces. However, such interfaces may also be exposed on a lower level in case finer-grained control is needed. The need for such crosscutting interface requirements will be extracted from the NBDIF: Volume 3, Use Cases and General Requirements document.

2.2.5.5 Messaging/Communications FrameworksMessaging and communications frameworks have their roots in the High Performance Computing environments long popular in the scientific and research communities. Messaging/Communications Frameworks were developed to provide application programming interfaces (APIs) for the reliable queuing, transmission, and receipt of data.

11

680681682683684

685

686687688

689690691692

693

694695696697

698

699700

701

702703704705706707

708

709710711712713714

715

716717718719


2.2.5.6 Resource Management FrameworkAs Big Data systems have evolved and become more complex, and as businesses work to leverage limited computation and storage resources to address a broader range of applications and business challenges, the requirement to effectively manage those resources has grown significantly. While tools for resource management and elastic computing have expanded and matured in response to the needs of cloud providers and virtualization technologies, Big Data introduces unique requirements for these tools. However, Big Data frameworks tend to fall more into a distributed computing paradigm, which presents additional challenges.

2.2.6 BIG DATA APPLICATION PROVIDER TO BIG DATA FRAMEWORK PROVIDER INTERFACE

The Big Data Framework Provider typically consists of one or more hierarchically organized instances of the components in the NBDRA IT value chain (Figure 1). There is no requirement that all instances at a given level in the hierarchy be of the same technology. In fact, most Big Data implementations are hybrids that combine multiple technology approaches to provide flexibility or meet the complete range of requirements, which are driven from the Big Data Application Provider.

12

720

721722723724725726727

728

729

730731732733734


3 SPECIFICATION PARADIGMThis section summarizes the elementary services that are important to the NBDRA.

3.1 HYBRID AND MULTIPLE FRAMEWORKSTo avoid vendor lock-in, Big Data systems must be able to deal with hybrid and multiple frameworks. This is not only true for Clouds, containers, DevOps, but also for components of the NBDRA.

3.2 DESIGN BY RESOURCE-ORIENTED ARCHITECTUREA resource-oriented architecture represents a software architecture and programming paradigm for designing and developing software in the form of resources. It is often associated with REpresentational State Transfer (REST) interfaces. The resources are software components which can be reused in concrete reference implementations. The service specification is conducted with OpenAPI, allowing use to provide it in a very general form that is independent of the framework or computer language in which the services can be specified. Note that OpenAPI defines services in REST The previous version only specified the resource objects.

3.3 DESIGN BY EXAMPLETo accelerate discussion among the NBD-PWG members, contributors to this document are encouraged to also provide the NBD-PWG with examples that can be included in an appendix.

3.4 VERSION MANAGEMENTDuring the design phase and development period of each version of this document, enhancements are managed through GitHub and community contributions are managed via GitHub issues. This allows preservation of the history of this document. When a new version is ready, the version will be tagged in GitHub. Older versions will, through this process, also be available as historical documents. Discussions about objects in written form are communicated as GitHub issues.

3.5 INTERFACE COMPLIANCYDue to the easy extensibility of the objects in this document and their implicit interfaces, it is important to introduce a terminology that allows the definition of interface compliancy. We define three levels of interface compliance as follows:

Full Compliance: These are reference implementations that provide full compliance to the objects defined in this document. A version number will be added to assure that the snapshot in time of the objects is associated with the version. This reference implementation will implement all objects.

Partial Compliance: These are reference implementations that provide partial compliance to the objects defined in this document. A version number will be added to assure that the snapshot in time of the objects is associated with the version. This reference implementation will implement a partial list of the objects. A document will be generated during the reference implementation that

13

735

736

737

738739

740

741742743744745746747

748

749750

751

752753754755756

757

758759760

761762763764765766767768


lists all objects defined, but also lists the objects that are not defined by the reference architecture. The document will outline which objects and interfaces have been implemented.

Full and Extended Compliance: These are interfaces that in addition to the full compliance also introduce additional interfaces and extend them. A document will be generated during the reference implementation that lists the differences to the document defined here.

The documents generated during the reference implementation can then be forwarded to the Reference Architecture Subgroup for further discussion and for possible future modifications based on additional practical user feedback.

14

769770771772773

774775776


4 EXAMPLE SPECIFICATIONThe specifications to this document are provided through an automated document creation process so that the actual OpenAPI specifications are the source for the document. Thus we will have all OpenAPI specifications located in the following directory in GitHub:

https://github.com/cloudmesh-community/nist/tree/master/spec

Limitations of the current implementation are as follows. It is a demonstration that showcases the generation of a fully functioning REST service based on the specifications provided in this document. However, it is expected that scalability, distribution of services, and other advanced options need to be addressed based on application requirements.

4.1 LIST OF SPECIFICATIONSThe following table lists the current set of resource objects that we are defining in this draft. Additional objects are also available at:


Table 1: List of Specifications

Service Version Date

Alias 3.1.1 29-01-2019

Batch Job 3.0.2 10-30-2018

Containers 3.1.1 02-15-2019

Database 3.1.1 02-15-2019

Default 3.1.1 02-15-2019

Deployment 3.0.2 10-30-2018

File 3.1.1 02-15-2019

Filter 3.0.2 10-30-2018

Flavor 3.1.1 02-15-2019

Hadoop 3.1.1 02-15-2019

Image 3.1.1 02-15-2019

Keyvaluestore 3.1.1 02-15-2019

Microservice 3.0.1

Nic 3.0.1

Organization 3.1.1 02-15-2019

15

777

778779780

781

782783784785

786

787788

789

790

791




Service Version Date

Key 3.1.1 02-15-2019

Replica 3.1.1 02-15-2019

Reservation 3.0.2 10-30-2018

Scheduler 3.1.1 02-15-2019

Scheduler 3.1.1 02-15-2019

Secgroup 3.1.1 02-12-2019

Batch Job 3.0.0 02-08-2019

Stream 3.0.2 10-30-2018

Timestamp 3.1.1 02-15-2019

User 3.1.1 02-15-2019

Variables 3.1.1 02-15-2019

Variables 3.1.1 02-15-2019

VCluster 3.1.1 02-11-2019

Virtual Cluster 3.1.1 02-11-2019

Virtualdirectory 3.1.1 02-15-2019

Virtual Machine 3.1.1 02-15-2019

Figure 2 shows the provider view of the specification resources.

Figure 2: Provider view

Figure 3 shows the resource view of the specification resources.

16

792

793

794795

796

797


Figure 3: Resource view

4.2 IDENTITYAs part of services an identity often needs to be specified. In addition, such persons are often part of groups and have roles within these groups. Thus, four important terms related to the identity are distinguished as follows:

User: The information identifying the profile of a person Group: A group that a person may belong to that is important to define access to services Role: A role given to a person as part of the group that can refine access rights. Organization: The information representing an Organization that manages a Big Data Service

4.2.1 AUTHENTICATIONAt this time, mechanisms are not yet included to manage authentication to external services such as clouds that can stage virtual machines. However, the group has shown multiple solutions to this in cloudmesh.

Local configuration file: A configuration file is managed locally to allow access to the clouds. It is the designer’s responsibility not to expose such credentials.

Session based authentication: No passwords are stored in the configuration file and access is granted on a per session basis where the password needs to be entered.

Service based authentication: The authentication is delegated to an external process. One example here is Auth.

The service that acts in behalf of the user needs to have access to the appropriate cloud provider credentials

An example for a configuration file is provided at the following location:

https://github.com/cloudmesh-community/cm/blob/master/cm4/etc/cloudmesh4.yaml

17

798799

800

801802803

804805806807

808

809810811

812813814815816817818819

820

821

https://github.com/cloudmesh-community/cm/blob/master/cm4/etc/cloudmesh4.yaml


4.2.2 ORGANIZATIONAn important concept in many services is the management of a group of users in an organization that includes Big Data services. User management within an organization can be achieved through several concepts. First, it can be achieved by defining the organization. Second, this organization can contain a number of users. Third, users within the organization can have a variety of roles and be separated in various groups. These roles and groups are used, for example, to distinguish different access rights to services.

4.2.2.1 Properties OrganizationProperty Type Description

name string Name of the organization

users array[#/definitions/User] list of users

timestamp

timestamps associated with the organization

4.2.2.2 Paths

4.2.2.2.1 /cloudmesh/organization

4.2.2.2.1.1 GET /cloudmesh/organizationReturns all organizations

Responses

Code Description Schema

200 organization info

4.2.2.2.1.2 PUT /cloudmesh/organizationCreates a new organization

Responses

Code Description

Schema

201 Created

Parameters

Name Located in Description Required Schema

organization

body The new organization to create False Organization

4.2.2.2.2 /cloudmesh/organization/{name}

4.2.2.2.2.1 GET /cloudmesh/organization/{name}Returns the organization

Responses

18

822

823824825826827828

829

830

831

832

833

834

835

836

837

838

839

840

841

842




Organization

Parameters

Name

Located in Description Required Schema

name path The name of the organization

True

4.2.2.2.3 /cloudmesh/organization/users

4.2.2.2.3.1 GET /cloudmesh/organization/usersReturns all users of the organization

Responses



Organization

4.2.2.2.4 /cloudmesh/organization/users/{username}

4.2.2.2.4.1 GET /cloudmesh/organization/users/{username}Returns the specific user of that organization

Responses



Organization

Parameters


username

path The username

True

4.2.2.3 organization.yamlswagger: "2.0"info: version: "3.1.1" x-status: defined x-date: 02-15-2019 title: Organization description: |- An important concept in many services is the management of a group of users in an organization that includes Big Data services. User management within an organization can be achieved through several concepts. First, it can be achieved by defining the organization. Second, this organization can contain a number of users. Third, users within the organization can have a variety of

19

843

844

845

846

847

848

849

850

851

852

853854855856857858859860861862863864865866867


roles and be separated in various groups. These roles and groups are used, for example, to distinguish different access rights to services.

termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/organization: get: tags: - Organization summary: Returns all organizations description: Returns all organizations operationId: cloudmesh.organization.get produces: - application/json responses: '200': description: organization info schema: type: array items: $ref: '#/definitions/Organization' put: tags: - Organization summary: Create a new organization description: Create a new organization operationId: cloudmesh.organization.put parameters: - in: body name: organization description: The new organization to create schema: $ref: '#/definitions/Organization' responses: '201': description: Created '/cloudmesh/organization/{name}': get: tags: - Organization summary: Returns the organization description: Returns the organization operationId: cloudmesh.organization.get_by_name parameters: - name: name description: The name of the organization in: path required: true type: string produces: - application/json responses: '200': description: organization info schema: $ref: '#/definitions/Organization'

20

868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936


/cloudmesh/organization/users: get: tags: - Organization summary: Returns all users of the organization description: Returns all users of the organization operationId: cloudmesh.organization.get_user produces: - application/json responses: '200': description: organization info schema: $ref: '#/definitions/Organization' /cloudmesh/organization/users/{username}: get: tags: - Organization summary: Returns the spcific user of that organization description: Returns the spcific user of that organization operationId: cloudmesh.organization.get_user_by_name parameters: - name: username description: The username in: path required: true type: string produces: - application/json responses: '200': description: organization info schema: $ref: '#/definitions/Organization'/cloudmesh/organization/users/{username}: put: tags: - Organization summary: Create a new user in the organization description: Create a new user in the organization operationId: cloudmesh.organization.put_user parameters: - name: username description: The username in: path required: true type: string produces: - application/json responses: '200': description: organization info schema: $ref: '#/definitions/Organization'definitions: Organization: type: object properties: name: description: Name of the organization type: string users: description: list of users type: array items: $ref: '#/definitions/User' timestamp: description: timestamps associated with the organization

21

937938939940941942943944945946947948949950951952953954955956957958959960961962963964965966967968969970971972973974975976977978979980981982983984985986987988989990991992993994995996997998999

10001001100210031004


$ref: '#/definitions/Timestamp'

4.2.3 USERServices need to specify which users have access to them. User information can be reused in other services. Users are useful to create a virtual organization that depends on user data. Users can be added, removed, and listed. A group associated with the user can be used to augment users to be part of one or more groups. A number of roles can identify a specific role of a user.

Terminology

Group: A user can be part of a Group Role: A user can have a role within that Group

4.2.3.1 Properties UserProperty Type Description

uuid string A unique id for the user

username string The unique username associated with the user

group array[string] A list of groups that are associated to the user

role array[string] A list of groups that are associated to the user

resource array[string] A list of resources the user has access to

description

string A description for this user

firstname string The firstanme of the user

lastname string The lastname of the user

publickey string The lastname of the user

email string The email of the user

timestamp timestamps associated with the resource

4.2.3.2 Paths

4.2.3.2.1 /cloudmesh/user

4.2.3.2.1.1 GET /cloudmesh/userReturns all users

Responses


200 user information

4.2.3.2.1.2 PUT /cloudmesh/userCreates a new user

Responses

Code Descriptio Schema

22

1005

1007

1008100910101011

1012

10131014

1015

1016

1017

1018

1019

1020

1021

1022

1023


n

201 Created

Parameters

Name Located in

Description Required Schema

user body The new user to create False User

4.2.3.2.2 /cloudmesh/user/{name}

4.2.3.2.2.1 GET /cloudmesh/user/{name}Returns the user of a user while looking it up with the username

Responses


200 user information User

Parameters

Name Located in


name path name of the user

True

4.2.3.3 user.yaml---swagger: "2.0"info: version: 3.1.1 x-status: defined x-date: 02-15-2019 title: User description: |- Services need to specify which users have access to them. User information can be reused in other services. Users are useful to create a virtual organization that depends on user data. Users can be added, removed, and listed. A group associated with the user can be used to augment users to be part of one or more groups. A number of roles can identify a specific role of a user. Terminology

* *Group*: A user can be part of a Group * *Role*: A user can have a role within that Group

termsOfService: https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt contact: name: Cloudmesh User url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080schemes: - httpconsumes:

23

1024

1025

1026

1027

1028

1029

103010311032103310341035103610371038103910401041104210431044104510461047104810491050105110521053105410551056105710581059106010611062


- application/jsonproduces: - application/jsonpaths: /cloudmesh/user: get: tags: - User summary: Returns all users description: Returns all users operationId: cloudmesh.user.get produces: - application/json responses: 200: description: user information schema: type: array items: $ref: '#/definitions/User' put: tags: - User summary: Create a new user description: Create a new user operationId: cloudmesh.user.put parameters: - in: body name: user description: The new user to create schema: $ref: '#/definitions/User' responses: 201: description: Created /cloudmesh/user/{name}: get: tags: - User summary: Returns the user of a user while looking it up with the username description: Returns the user of a user while looking it up with the username operationId: cloudmesh.user.get_by_name parameters: - name: name description: name of the user in: path required: true type: string produces: - application/json responses: 200: description: user information schema: $ref: '#/definitions/User'definitions: User: type: object properties: uuid: type: string description: A unique id for the user username: type: string description: The unique username associated with the user group: type: array description: A list of groups that are associated to the user items:

24

106310641065106610671068106910701071107210731074107510761077107810791080108110821083108410851086108710881089109010911092109310941095109610971098109911001101110211031104110511061107110811091110111111121113111411151116111711181119112011211122112311241125112611271128112911301131


type: string role: type: array description: A list of groups that are associated to the user items: type: string resource: type: array description: A list of resources the user has access to items: type: string description: type: string description: A description for this user firstname: type: string description: The firstanme of the user lastname: type: string description: The lastname of the user publickey: type: string description: The lastname of the user email: type: string description: The email of the user timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.2.4 PUBLICKEYSTOREMany services and frameworks use Secure Shell (SSH) keys to authenticate. This service allows the convenient storage of the public keys.

4.2.4.1 Properties PublicKeyProperty Type Description

name string The name of the public key

value string The value of the public key

kind string The key kind such as rsa, dsa

group string An optional group name allowing to group keys to create custom groups

comment string A comment for the public key

uri string The uri of the public key if any

fingerprint string The fingerprint of the public key


4.2.4.2 Paths

4.2.4.2.1 /cloudmesh/publickeystore

4.2.4.2.1.1 GET /cloudmesh/publickeystoreReturns all public keys

Responses

25

11321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160

1161

11621163

1164

1165

1166

1167

1168

1169


Code

Description Schema

200 public key information

4.2.4.2.1.2 PUT /cloudmesh/publickeystorePut a new public key into the public key store

Responses

Code Description

Schema

201 Created

Parameters


publickey body The public key to be uploaded False PublicKey

4.2.4.2.2 /cloudmesh/publickeystore/{name}

4.2.4.2.2.1 GET /cloudmesh/publickeystore/{name}Returns a public key by name

Responses


200 public key information PublicKey

Parameters

Name


name path The name of the public key True

4.2.4.3 publickeystore.yamlswagger: "2.0"info: version: 3.1.1 x-status: defined x-date: 02-15-2019 title: Key description: |- Many services and frameworks use Secure Shell (SSH) keys to authenticate. This service allows the convenient storage of the public keys.

termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - http

26

1170

1171

1172

1173

1174

1175

1176

1177

1178

1179118011811182118311841185118611871188118911901191119211931194119511961197119811991200


consumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/publickeystore: get: tags: - Publickeystore summary: Returns all public keys description: Returns all public keys operationId: cloudmesh.publickeystore.get produces: - application/json responses: '200': description: public key information schema: type: array items: $ref: '#/definitions/PublicKey' put: tags: - Publickeystore summary: Put a new public key into the public key store description: Put a new public key into the public key store operationId: cloudmesh.publickeystore.put parameters: - in: body name: publickey description: The public key to be uploaded schema: $ref: '#/definitions/PublicKey' responses: '201': description: Created '/cloudmesh/publickeystore/{name}': get: tags: - Publickeystore summary: Returns a public key by name description: Returns a public key by name operationId: cloudmesh.publickeystore.get_by_name parameters: - name: name in: path required: true type: string description: The name of the public key produces: - application/json responses: '200': description: public key information schema: $ref: '#/definitions/PublicKey'definitions: PublicKey: type: object description: the public key properties: name: type: string description: The name of the public key value: type: string description: The value of the public key kind: type: string

27

120112021203120412051206120712081209121012111212121312141215121612171218121912201221122212231224122512261227122812291230123112321233123412351236123712381239124012411242124312441245124612471248124912501251125212531254125512561257125812591260126112621263126412651266126712681269


description: The key kind such as rsa, dsa group: type: string description: An optional group name allowing to group keys to create custom groups comment: type: string description: A comment for the public key uri: type: string description: The uri of the public key if any fingerprint: type: string description: The fingerprint of the public key timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.3 GENERAL RESOURCES4.3.1 TIMESTAMPData often needs to be timestamped to indicate when it has been accessed, created, or modified. All objects defined in this document will have, in their final version, a timestamp. The date-time string is defined in RFC3339.

4.3.1.1 Properties TimestampProperty Type Description

accessed string

The time stamp when the object was last accessed

created string

The time stamp when the object was created

modified string

The time stamp when the object was modified

4.3.1.2 Paths

4.3.1.2.1 /cloudmesh/timestamp

4.3.1.2.1.1 GET /cloudmesh/timestampReturns all timestamps

Responses

Code

Description Schema

200 timestamp info

4.3.1.2.1.2 PUT /cloudmesh/timestampCreate a new timestamp

Responses

28

12701271127212731274127512761277127812791280128112821283128412851286

1287

1288

128912901291

1292

1293

1294

1295

1296

1297

1298

1299

1300

https://xml2rfc.ietf.org/public/rfc/html/rfc3339.html#anchor14


Code Description

Schema

201 Created

Parameters


timestamp body The new timestamp to create False Timestamp

4.3.1.2.2 /cloudmesh/timestamp/{name}

4.3.1.2.2.1 GET /cloudmesh/timestamp/{name}Returns a timestamp

Responses


200 timestamp info

Timestamp

Parameters

Name


name path the name of the time stamp

True

4.3.1.3 timestamp.yamlswagger: "2.0"info: version: 3.1.1 x-status: defined x-date: 02-15-2019 title: Timestamp description: |- Data often needs to be timestamped to indicate when it has been accessed, created, or modified. All objects defined in this document will have, in their final version, a timestamp. The date-time string is defined in [RFC3339](https://xml2rfc.ietf.org/public/rfc/html/rfc3339.html#anchor14).

termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/timestamp: get: tags:

29

1301

1302

1303

1304

1305

1306

13071308130913101311131213131314131513161317131813191320132113221323132413251326132713281329133013311332133313341335133613371338


- Timestamp summary: Returns all timestamps description: Returns all timestamps operationId: cloudmesh.timestamp.get produces: - application/json responses: '200': description: timestamp info schema: type: array items: $ref: '#/definitions/Timestamp' put: tags: - Timestamp summary: Create a new timestamp description: Create a new timestamp operationId: cloudmesh.timestamp.put parameters: - in: body name: timestamp description: The new timestamp to create schema: $ref: '#/definitions/Timestamp' responses: '201': description: Created '/cloudmesh/timestamp/{name}': get: tags: - Timestamp summary: Returns a timestamp description: Returns a timestamp operationId: cloudmesh.timestamp.get_by_name parameters: - name: name in: path required: true type: string description: the name of the time stamp produces: - application/json responses: '200': description: timestamp info schema: $ref: '#/definitions/Timestamp'definitions: Timestamp: type: object description: the timestamp properties: accessed: type: string format: date-time description: The time stamp when the object was last accessed created: type: string format: date-time description: The time stamp when the object was created modified: type: string format: date-time description: The time stamp when the object was modified

30

13391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403


4.3.2 ALIASA user may desire to create an alias for a resource. An alias may be used to more easily remember a resource. A User can deploy a service in which such aliases are stored. The aliases could naturally be shared with other users. A resource could have one or more aliases. The reason for an alias is that a resource may have a complex name but a user may want to refer to the resource using a name that is suitable for the user’s application.

4.3.2.1 Properties AliasProperty Type Description

name string The name of the alias

origin string The original object name


4.3.2.2 Paths

4.3.2.2.1 /cloudmesh/alias

4.3.2.2.1.1 GET /cloudmesh/aliasReturns all aliases

Responses

Code

Description Schema

200 alias information

4.3.2.2.1.2 PUT /cloudmesh/aliasCreates a new alias

Responses

Code Description

Schema

201 Created

Parameters


alias body The new alias to create

True Alias

4.3.2.2.2 /cloudmesh/alias/{name}

4.3.2.2.2.1 GET /cloudmesh/alias/{name}Returns an alias by name

Responses

31

1404

14051406140714081409

1410

1411

1412

1413

1414

1415

1416

1417

1418

1419

1420

1421

1422

1423


Code Description

Schema

200 alias info Alias

Parameters


name path The name of the alias

True

4.3.2.3 alias.yamlswagger: "2.0"info: version: 3.1.1 x-status: defined x-date: 29-01-2019 title: Alias description: |-

A user may desire to create an alias for a resource. An alias may be used to more eaily remember a resource. A User can deploy a service in which such aliasses are stored. The aliasses could naturally be shared with other users. A resource could have one or more aliasses. The reason for an alias is that a resource may have a complex name but a user may want to refer to the resource using a name that is suitable for the user's application.

termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/alias: get: tags: - Alias summary: Returns all aliases description: Returns all aliases operationId: cloudmesh.alias.get produces: - application/json responses: '200': description: alias information schema: type: array items: $ref: '#/definitions/Alias' put: tags: - Alias summary: Create a new alias description: Create a new alias operationId: cloudmesh.alias.put parameters: - in: body

32

1424

142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478


name: alias required: true description: The new alias to create schema: $ref: '#/definitions/Alias' responses: '201': description: Created '/cloudmesh/alias/{name}': get: tags: - Alias summary: Returns an alias by name description: Returns an alias by name operationId: cloudmesh.alias.get_by_name parameters: - name: name in: path required: true type: string description: The name of the alias produces: - application/json responses: '200': description: alias info schema: $ref: '#/definitions/Alias'definitions: Alias: type: object description: the alias properties: name: type: string description: The name of the alias origin: type: string description: The original object name timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.3.3 VARIABLESVariables are used to store simple values. Each variable can have a type, which is also provided as demonstrated in the object below. The variable value format is defined as string to allow maximal probability.

4.3.3.1 Properties VariablesProperty Type Description

name string name of the variable

value string type of the variable

kind string value of the variable


33

147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520

1521

152215231524

1525


4.3.3.2 Paths

4.3.3.2.1 /cloudmesh/variables

4.3.3.2.1.1 GET /cloudmesh/variablesReturns all variabless

Responses


200 variables info

4.3.3.2.1.2 PUT /cloudmesh/variablesCreate a new variables

Responses

Code Description

Schema

201 Created

Parameters

Name Located in Description Required

Schema

variables

body The new variables to create

False Variables

4.3.3.2.2 /cloudmesh/variables/{name}

4.3.3.2.2.1 GET /cloudmesh/variables/{name}Returns a variables

Responses

Code

Description Schema

200 variables info Variables

Parameters

Name


name path Name of the variable

True

4.3.3.3 variables.yamlswagger: "2.0"info: version: 3.1.1 x-status: defined

34

1526

1527

1528

1529

1530

1531

1532

1533

1534

1535

1536

1537

1538

1539

15401541154215431544


x-date: 02-15-2019 title: Variables description: |- Variables are used to store simple values. Each variable can have a type, which is also provided as demonstrated in the object below. The variable value format is defined as string to allow maximal probability. termsOfService: https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/variables: get: tags: - Variables summary: Returns all variabless description: Returns all variabless operationId: cloudmesh.variables.get produces: - application/json responses: 200: description: variables info schema: type: array items: $ref: '#/definitions/Variables' put: tags: - Variables summary: Create a new variables description: Create a new variables operationId: cloudmesh.variables.add parameters: - in: body name: variables description: The new variables to create schema: $ref: '#/definitions/Variables' responses: 201: description: Created '/cloudmesh/variables/{name}': get: tags: - Variables summary: Returns a variables description: Returns a variables operationId: cloudmesh.variables.get_by_name parameters: - name: name description: Name of the variable in: path required: true type: string produces: - application/json

35

154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613


responses: 200: description: variables info schema: $ref: '#/definitions/Variables'definitions: Variables: type: object description: the variables properties: name: type: string description: name of the variable value: type: string description: type of the variable kind: type: string description: value of the variable timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.3.4 KEYVALUESTOREKeyvaluestore is a service to store key, value, and type information, all of which are stored as Strings. However, the type could be used to transform it into a non-string type.

4.3.4.1 Properties KeyProperty Type Description

uuid string The uuid of the key, tha uuid must be unique

name string The name of the key. The name must be unique

description string A description of the key

value string The value, or content of the key

kind string The type of the key


4.3.4.2 Paths

4.3.4.2.1 /cloudmesh/keyvaluestore/key

4.3.4.2.1.1 GET /cloudmesh/keyvaluestore/keyReturns all keys with their values and types included

Responses

Code Description

Schema

200 key info

4.3.4.2.1.2 PUT /cloudmesh/keyvaluestore/keyCreates a new key

Responses

36

1614161516161617161816191620162116221623162416251626162716281629163016311632163316341635

1636

16371638

1639

1640

1641

1642

1643

1644

1645

1646

1647


Code Description

Schema

201 Created

Parameters

Name Located in


key body The new key to create True Key

4.3.4.2.2 /cloudmesh/keyvaluestore/key/{name}

4.3.4.2.2.1 GET /cloudmesh/keyvaluestore/key/{name}Returns the key with its detailed information by name of the key

Responses

Code Description

Schema

200 key info Key

Parameters

Name

Located in Description Required

Schema

name path name of the key True

4.3.4.3 keyvaluestore.yaml---swagger: "2.0"info: version: 3.1.1 x-status: compare to variable and merge x-date: 02-15-2019 title: Keyvaluestore description: |-

Keyvaluestore is a service to store key, value, and type information, all of which are stored as Strings. However the type could be use to transform it into a non strin type.

termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/keyvaluestore/key: get: tags: - Keyvaluestore

37

1648

1649

1650

1651

1652

1653

16541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685


summary: Returns all keys with their values and types included description: Returns all keys with their values and types included operationId: cloudmesh.keyvaluestore.get produces: - application/json responses: '200': description: key info schema: type: array items: $ref: '#/definitions/Key' put: tags: - Keyvaluestore summary: Create a new key description: Create a new key operationId: cloudmesh.keyvaluestore.put parameters: - in: body name: key description: The new key to create required: true schema: $ref: '#/definitions/Key' responses: '201': description: Created /cloudmesh/keyvaluestore/key/{name}: get: tags: - Keyvaluestore summary: Returns the key with its detailed information by name of the key description: Returns the key with its detailed information by name of the key operationId: cloudmesh.keyvaluestore.get_by_name parameters: - name: name description: name of the key in: path required: true type: string produces: - application/json responses: '200': description: key info schema: $ref: '#/definitions/Key'definitions: Key: type: object properties: uuid: type: string description: The uuid of the key, tha uuid must be unique name: type: string description: The name of the key. The name must be unique description: type: string description: A description of the key value: type: string description: The value, or content of the key kind: type: string description: The type of the key timestamp:

38

16861687168816891690169116921693169416951696169716981699170017011702170317041705170617071708170917101711171217131714171517161717171817191720172117221723172417251726172717281729173017311732173317341735173617371738173917401741174217431744174517461747174817491750175117521753


description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.3.5 DEFAULTA default is a special variable that has a context associated with it. This allows one to define values that can be easily retrieved based on the associated context. For example, a default could be the image name for a cloud where the context is defined by the cloud name. In addition to the context, the service name is also specified since a service could have multiple contexts. To be able to define the kind of service there is also a kind attribute.

4.3.5.1 Properties DefaultProperty Type Description

name string The name of the default

value string The type of the default

kind string The value of the default

service string The name of the service

context string The context of the service


4.3.5.2 Paths

4.3.5.2.1 /cloudmesh/default

4.3.5.2.1.1 GET /cloudmesh/defaultReturns all defaults

Responses

Code Description

Schema

200 default info

4.3.5.2.1.2 PUT /cloudmesh/defaultCreates a new default

Responses

Code Description

Schema

201 Created

Parameters

Name Located in


default body The new default to create True Default

39

17541755

1756

17571758175917601761

1762

1763

1764

1765

1766

1767

1768

1769

1770

1771


4.3.5.2.2 /cloudmesh/default/{name}

4.3.5.2.2.1 GET /cloudmesh/default/{name}Returns a default

Responses

Code Description

Schema

200 default info Default

Parameters


name path The name of the default

True

4.3.5.3 default.yamlswagger: "2.0"info: version: 3.1.1 x-status: defined x-date: 02-15-2019 title: Default description: |- A default is a special variable that has a context associated with it. This allows one to define values that can be easily retrieved based on the associated context. For example, a default could be the image name for a cloud where the context is defined by the cloud name. In addition to the context, the service name is also specified since a service could have multiple contexts. To be able to define the kind of service there is also a kind attribute. termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/default: get: tags: - Default summary: Returns all defaults description: Returns all defaults operationId: cloudmesh.default.get produces: - application/json responses: '200': description: default info schema: type: array items:

40

1772

1773

1774

1775

1776

1777177817791780178117821783178417851786178717881789179017911792179317941795179617971798179918001801180218031804180518061807180818091810181118121813181418151816181718181819182018211822


$ref: '#/definitions/Default' put: tags: - Default summary: Create a new default description: Create a new default operationId: cloudmesh.default.add parameters: - in: body name: default required: true description: The new default to create schema: $ref: '#/definitions/Default' responses: '201': description: Created '/cloudmesh/default/{name}': get: tags: - Default summary: Returns a default description: Returns a default operationId: cloudmesh.default.get_by_name parameters: - name: name in: path required: true type: string description: The name of the default produces: - application/json responses: '200': description: default info schema: $ref: '#/definitions/Default'definitions: Default: type: object description: the defaults properties: name: type: string description: The name of the default value: type: string description: The type of the default kind: type: string description: The value of the default service: type: string description: The name of the service context: type: string description: The context of the service timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.4 DATA MANAGEMENT4.4.1 FILEExample:

41

182318241825182618271828182918301831183218331834183518361837183818391840184118421843184418451846184718481849185018511852185318541855185618571858185918601861186218631864186518661867186818691870187118721873187418751876187718781879188018811882

1883

1884

1885


"name": "report.dat","endpoint": "file://[email protected]:/data/report.dat","checksum": {"sha256":"c01b39c7a35ccc ....... ebfeb45c69f08e17dfe3ef375a7b"},"accessed": "1.1.2017:05:00:00:EST","created": "1.1.2017:05:00:00:EST","modified": "1.1.2017:05:00:00:EST",size": ["GB", "Byte"]

A file is a resource allowing storage of data as a traditional file being processed. The interface to a file provides the mechanism to appropriately locate a file in a distributed system. File identification includes the name, endpoint, checksum, and size. Additional parameters, such as a timestamp can also be stored. The interface only describes the location of the file. The file object has name, endpoint (location), size (in gigabytes [GB], megabytes [MB], and Bytes), and checksum for integrity check.

4.4.1.1 Properties FileProperty Type Description

name string The name of the file

endpoint string The location of the file

checksum string The checksum of the file

size integer

The size of the file in byte

timestamp timestamps associated with the file

4.4.1.2 Paths

4.4.1.2.1 /cloudmesh/file

4.4.1.2.1.1 GET /cloudmesh/fileReturns all files

Responses

Code Description

Schema

200 file info

4.4.1.2.1.2 PUT /cloudmesh/fileCreates a new file

Responses

Code Description

Schema

201 Created

42

1886188718881889189018911892

18931894189518961897

1898

1899

1900

1901

1902

1903

1904

1905

1906


Parameters

Name


Schema

file body The new file record to create True File

4.4.1.2.2 /cloudmesh/file/{name}

4.4.1.2.2.1 GET /cloudmesh/file/{name}Returns a file by name

Responses

Code Description

Schema

200 file info File

Parameters

Name


name path The name of the file

True

4.4.1.3 file.yamlswagger: "2.0"info: version: 3.1.1 x-status: defined x-date: 02-15-2019 title: File description: |- A file is a resource allowing storage of data as a traditional file being processed. The interface to a file provides the mechanism to appropriately locate a file in a distributed system. File identification includes the name, endpoint, checksum, and size. Additional parameters, such as a timestamp can also be stored. The interface only describes the location of the file. The file object has name, endpoint (location), size (in gigabytes [GB], megabytes [MB], and Bytes), and checksum for integrity check. termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/file: get: tags: - File

43

1907

1908

1909

1910

1911

1912

191319141915191619171918191919201921192219231924192519261927192819291930193119321933193419351936193719381939194019411942194319441945194619471948


summary: Returns all files description: Returns all files operationId: cloudmesh.file.get produces: - application/json responses: '200': description: file info schema: type: array items: $ref: '#/definitions/File' put: tags: - File summary: Create a new file description: Create a new file operationId: cloudmesh.file.put parameters: - in: body name: file required: true description: The new file record to create schema: $ref: '#/definitions/File' responses: '201': description: Created '/cloudmesh/file/{name}': get: tags: - File summary: Returns a file by name description: Returns a file by name operationId: cloudmesh.file.get_by_name parameters: - name: name in: path required: true type: string description: The name of the file produces: - application/json responses: '200': description: file info schema: $ref: '#/definitions/File'definitions: File: type: object description: an object representing a file properties: name: type: string description: The name of the file endpoint: type: string description: The location of the file checksum: type: string description: The checksum of the file size: type: integer description: The size of the file in byte timestamp: description: timestamps associated with the file $ref: '#/definitions/Timestamp'

44

19491950195119521953195419551956195719581959196019611962196319641965196619671968196919701971197219731974197519761977197819791980198119821983198419851986198719881989199019911992199319941995199619971998199920002001200220032004200520062007200820092010201120122013201420152016


4.4.2 REPLICAIn many distributed systems, it is important that a file can be replicated among different systems to provide faster access. It is important to provide a mechanism to trace the pedigree of the file while pointing to its original source.

4.4.2.1 Properties ReplicaProperty Type Description

name string the name of the replica

filename string the original filename

endpoint string The location of the file

checksum string The checksum of the file

size integer

The size of the file in byte

timestamp

The timestamp of the replica. The timestamp refers to when this entry has been accessed or changed, not the object this replica is pointing to.

4.4.2.2 Paths

4.4.2.2.1 /cloudmesh/replica

4.4.2.2.1.1 GET /cloudmesh/replicaReturns all replicas records

Responses

Code Description

Schema

200 replica info

4.4.2.2.1.2 PUT /cloudmesh/replicaCreates a new replica

Responses

Code Description

Schema

201 Created

Parameters

Name Located in


replica body The new replica entry to create True Replica

45

2017

2019

202020212022

2023

2024

2025

2026

2027

2028

2029

2030

2031

2032


4.4.2.2.2 /cloudmesh/replica/{name}

4.4.2.2.2.1 GET /cloudmesh/replica/{name}Returns a replica by name

Responses

Code Description

Schema

200 replica info Replica

Parameters

Name


name path name of the replica entry

True

4.4.2.3 replica.yamlswagger: "2.0"info: version: 3.1.1 x-status: defined x-date: 02-15-2019 title: Replica description: |- In many distributed systems, it is important that a file can be replicated among different systems to provide faster access. It is important to provide a mechanism to trace the pedigree of the file while pointing to its original source.

termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/replica: get: tags: - Replica summary: Returns all replicas records description: Returns all replicas records operationId: cloudmesh.replica.get produces: - application/json responses: 200: description: replica info schema: type: array items: $ref: '#/definitions/Replica'

46

2033

2034

2035

2036

2037

20382039204020412042204320442045204620472048204920502051205220532054205520562057205820592060206120622063206420652066206720682069207020712072207320742075207620772078207920802081


put: tags: - Replica summary: Create a new replica description: Create a new replica operationId: cloudmesh.replica.add parameters: - in: body name: replica required: true description: The new replica entry to create schema: $ref: '#/definitions/Replica' responses: 201: description: Created '/cloudmesh/replica/{name}': get: tags: - Replica summary: Returns a replica by name description: Returns a replica by name operationId: cloudmesh.replica.get_by_name parameters: - name: name description: name of the replica entry in: path required: true type: string produces: - application/json responses: 200: description: replica info schema: $ref: '#/definitions/Replica'definitions: Replica: type: object description: An entry representing a file replica record properties: name: type: string description: the name of the replica filename: type: string description: the original filename endpoint: type: string description: The location of the file checksum: type: string description: The checksum of the file size: type: integer description: The size of the file in byte timestamp: $ref: '#/definitions/Timestamp' description: The timestamp of the replica. The timestamp refers to when this entry has been accessed or changed, not the object this replica is pointing to.

4.4.3 DATABASEA database could have a name, an endpoint (e.g., host, port), and a protocol used (e.g., SQL, MongoDB).

47

208220832084208520862087208820892090209120922093209420952096209720982099210021012102210321042105210621072108210921102111211221132114211521162117211821192120212121222123212421252126212721282129213021312132213321342135213621372138213921402141

2143

2144


4.4.3.1 Properties DatabaseProperty Type Description

name string

name of the database

description string

description of the database

endpoint string

endpoint of the database

kind string

the kind of the database


4.4.3.2 Paths

4.4.3.2.1 /cloudmesh/database

4.4.3.2.1.1 GET /cloudmesh/databaseReturns all databases

Responses


200 database info

4.4.3.2.1.2 PUT /cloudmesh/databaseCreates a new database

Responses

Code Description

Schema

201 Created

Parameters


database

body The new database record to create

True Database

4.4.3.2.2 /cloudmesh/database/{name}

4.4.3.2.2.1 GET /cloudmesh/database/{name}Returns a database by name

Responses

Code

Description Schema

48

2145

2146

2147

2148

2149

2150

2151

2152

2153

2154

2155

2156

2157

2158


200 database info Database

Parameters

Name


name path name of the database

True

4.4.3.3 database.yamlswagger: "2.0"info: version: 3.1.1 x-status: defined x-date: 02-15-2019 title: Database description: |- A database could have a name, an endpoint (e.g., host, port), and a protocol used (e.g., SQL, MongoDB).

termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/database: get: tags: - Database summary: Returns all databases description: Returns all databases operationId: cloudmesh.datbase.get produces: - application/json responses: 200: description: database info schema: type: array items: $ref: '#/definitions/Database' put: tags: - Database summary: Create a new database description: Create a new database operationId: cloudmesh.database.put parameters: - in: body name: database required: true description: The new database record to create schema: $ref: '#/definitions/Database' responses:

49

2159

21602161216221632164216521662167216821692170217121722173217421752176217721782179218021812182218321842185218621872188218921902191219221932194219521962197219821992200220122022203220422052206220722082209221022112212221322142215


201: description: Created '/cloudmesh/database/{name}': get: tags: - Database summary: Returns a database by name description: Returns a database by name operationId: cloudmesh.database.get_by_name parameters: - name: name description: name of the database in: path required: true type: string produces: - application/json responses: 200: description: database info schema: $ref: '#/definitions/Database'definitions: Database: type: object description: This defines a database object as an entry properties: name: type: string description: name of the database description: type: string description: description of the database endpoint: type: string description: endpoint of the database kind: type: string description: the kind of the database timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.4.4 VIRTUAL DIRECTORYA virtual directory is a collection of files or replicas of the files. A virtual directory can contain a number of entities including files, streams, and other virtual directories as part of a collection. The element in the collection can either be defined by uuid or by name.

4.4.4.1 Properties UnauthorizedErrorProperty Type Description

code string Code form of the error

message string Human readable form of the error

4.4.4.2 Properties VirtualdirectoryProperty Type Description

name string The name of the virtual directory

description string description of the virtual directory

50

221622172218221922202221222222232224222522262227222822292230223122322233223422352236223722382239224022412242224322442245224622472248224922502251225222532254225522562257

2258

225922602261

2262

2263


host string remote host of the virtual directory

location string remote location, e.g., a directory with full path on a host

protocol string access protocol, e.g., HTTP, FTP, SSH, etc.

credential object credential to access, e.g., username, password


4.4.4.3 Paths

4.4.4.3.1 /cloudmesh/virtualdirectory

4.4.4.3.1.1 GET /cloudmesh/virtualdirectoryReturns all virtualdirectorys

Responses

Code

Description Schema

200 virtualdirectory info


401 unauthorized error UnauthorizedError

4.4.4.3.1.2 PUT /cloudmesh/virtualdirectoryCreates a new virtualdirectory

Responses

Code Description

Schema

201 Created

Parameters


virtualdirectory body The new virtualdirectory to create True Virtualdirectory

4.4.4.3.2 /cloudmesh/virtualdirectory/{name}

4.4.4.3.2.1 GET /cloudmesh/virtualdirectory/{name}Returns a virtualdirectory by name

Responses

Code

Description Schema

200 virtualdirectory info Virtualdirectory

Parameters

Nam Located in Description Required Schema

51

2264

2265

2266

2267

2268

2269

2270

2271

2272

2273

2274

2275

2276

2277


e

name path name of the virtual directory

True

4.4.4.4 virtualdirectory.yaml---swagger: "2.0"info: version: 3.1.1 x-date: 02-15-2019 x-status: defined title: Virtualdirectory description: |- A virtual directory is a collection of files or replicas of the files. A virtual directory can contain a number of entities including files, streams, and other virtual directories as part of a collection. The element in the collection can either be defined by uuid or by name.

termsOfService: https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt contact: name: NIST BDRA Interface Subgroup Service Example url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonsecurityDefinitions: httpbasic: type: basic apikey: type: apiKey in: query name: api_key apisecret: type: apiKey in: query name: api_secretpaths: /cloudmesh/virtualdirectory: get: tags: - Virtualdirectory summary: Returns all virtualdirectorys description: Returns all virtualdirectorys operationId: cloudmesh.virtualdirectory.get produces: - application/json security: - httpbasic: [] responses: '200': description: virtualdirectory info schema: type: array items: $ref: '#/definitions/Virtualdirectory' '401': description: unauthorized error schema:

52

22782279228022812282228322842285228622872288228922902291229222932294229522962297229822992300230123022303230423052306230723082309231023112312231323142315231623172318231923202321232223232324232523262327232823292330233123322333233423352336233723382339


$ref: '#/definitions/UnauthorizedError' put: tags: - Virtualdirectory summary: Create a new virtualdirectory description: Create a new virtualdirectory operationId: cloudmesh.virtualdirectory.add parameters: - in: body name: virtualdirectory required: true description: The new virtualdirectory to create schema: $ref: '#/definitions/Virtualdirectory' security: - apikey: [] apisecret: [] responses: '201': description: Created '/cloudmesh/virtualdirectory/{name}': get: tags: - Virtualdirectory summary: Returns a virtualdirectory by name description: Returns a virtualdirectory by name operationId: cloudmesh.virtualdirectory.get_by_name parameters: - name: name description: name of the virtual directory in: path required: true type: string produces: - application/json security: - httpbasic: [] responses: '200': description: virtualdirectory info schema: $ref: '#/definitions/Virtualdirectory'definitions: UnauthorizedError: type: object description: unauthorized error properties: code: type: string description: Code form of the error message: type: string description: Human readable form of the error Virtualdirectory: type: object description: the virtualdirectory properties: name: description: The name of the virtual directory type: string description: description: description of the virtual directory type: string host: description: remote host of the virtual directory type: string location: description: remote location, e.g., a directory with full path on a host type: string

53

234023412342234323442345234623472348234923502351235223532354235523562357235823592360236123622363236423652366236723682369237023712372237323742375237623772378237923802381238223832384238523862387238823892390239123922393239423952396239723982399240024012402240324042405240624072408


protocol: description: access protocol, e.g., HTTP, FTP, SSH, etc. type: string credential: description: credential to access, e.g., username, password type: object timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.5 COMPUTE MANAGEMENT - VIRTUAL CLUSTERS4.5.1 VIRTUAL CLUSTERA Virtual Cluster is modeled as one Front-end, or manager node, and one or multiple compute nodes. The manager node usually served as a login node and can be accessed from outside via a public IP. The compute nodes are connected to the manager node via a private, usually high performance (high throughput and low latency), network and thus accessible only from the manager node. To use the virtual cluster, you login to the manager node, and from there you can login to any compute node, or submit a job to run on the compute nodes.

4.5.1.1 Properties VirtualClusterProperty Type Description

name string The name of the virtual cluster

description string A description of the virtual cluster

nnodes integer number of nodes of the virtual cluster

owner string owner of the virtual cluster

manager Manager node of the virtual cluster

nodes array[#/definitions/Node] List of nodes of the virtual cluster


4.5.1.2 Properties NodeProperty Type Description

name string name of the node

state string power state of the node

ncpu integer number of virtual CPUs of the node

ram string RAM size of the node

disk string Disk size of the node

nics array[#/definitions/NIC]

List of network interfaces of the node

4.5.1.3 Properties NICPropert Type Description

54

240924102411241224132414241524162417

2419

2420

242124222423242424252426

2427

2428

2429


y

mac string MAC address of the node

ip string IP address of the node

4.5.1.4 Paths

4.5.1.4.1 /cloudmesh/virtualcluster/virtualcluster

4.5.1.4.1.1 GET /cloudmesh/virtualcluster/virtualclusterReturns all virtual clusters

Responses

Code Description

Schema

200 profile info

4.5.1.4.1.2 PUT /cloudmesh/virtualcluster/virtualclusterCreates a new virtual cluster

Responses

Code Description

Schema

201 Created

Parameters


Schema

virtualcluster

body The new virtual cluster to create True VirtualCluster

4.5.1.4.2 /cloudmesh/virtualcluster/virtualcluster/{virtualclustername}

4.5.1.4.2.1 GET /cloudmesh/virtualcluster/virtualcluster/{virtualclustername}Returns a virtual cluster by its name

Responses

Code

Description Schema

200 virtualcluster info VirtualCluster

Parameters


Schema

virtualclustername

path name of the virtual cluster True

55

2430

2431

2432

2433

2434

2435

2436

2437

2438

2439

2440

2441

2442

2443


4.5.1.4.3 /cloudmesh/virtualcluster/virtualcluster/{virtualclustername}/manager

4.5.1.4.3.1 GET /cloudmesh/virtualcluster/virtualcluster/{virtualclustername}/managerReturns the manager node info of the specified virtual cluster

Responses

Code

Description Schema

200 virtual cluster manager node info Node

Parameters

Name Located in


virtualclustername path name of the virtua cluster True

4.5.1.4.4 /cloudmesh/virtualcluster/virtualcluster/{virtualclustername}/{nodename}

4.5.1.4.4.1 GET /cloudmesh/virtualcluster/virtualcluster/{virtualclustername}/{nodename}Returns the specified node info of the specified virtualcluster

Responses


200 virtualcluster node info

Node

Parameters

Name Located in


virtualclustername path name of the virtua cluster True

nodename path name of the name True

4.5.1.5 vc.yaml---swagger: "2.0"info: version: 3.1.1 x-date: 02-11-2019 x-status: defined title: "Virtual Cluster" description: |-

A Virtual Cluster is modeled as one Front-end, or manager node, and one or multiple compute nodes. The manager node usually served as a login node and can be accessed from outside via a public IP. The compute nodes are connected to the manager node via a private, usually high performance (high throughput and low latency), network and thus accessible only from the manager node. To use the virtual cluster, you login to the manager node, and from there you can login to any compute node, or submit a job to run on the compute nodes. termsOfService: https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt contact:

56

2444

2445

2446

2447

2448

2449

2450

2451

2452

2453

245424552456245724582459246024612462246324642465246624672468246924702471247224732474


name: NIST BDRA Interface Subgroup Service Example url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/virtualcluster/virtualcluster: get: tags: - Virtualcluster summary: Returns all virtual clusters description: Returns all virtual clusters operationId: cloudmesh.virtualcluster.get produces: - application/json responses: "200": description: profile info schema: type: array items: $ref: "#/definitions/VirtualCluster" put: tags: - Virtualcluster summary: Create a new virtual cluster description: Create a new virtual cluster operationId: cloudmesh.virtualcluster.add parameters: - in: body name: virtualcluster description: The new virtual cluster to create required: true schema: $ref: "#/definitions/VirtualCluster" responses: "201": description: Created /cloudmesh/virtualcluster/virtualcluster/{virtualclustername}: get: tags: - Virtualcluster summary: Returns a virtual cluster by its name description: Returns a virtual cluster by its name operationId: cloudmesh.virtualcluster.getByName parameters: - name: virtualclustername description: name of the virtual cluster in: path required: true type: string produces: - application/json responses: "200": description: virtualcluster info schema: $ref: "#/definitions/VirtualCluster" /cloudmesh/virtualcluster/virtualcluster/{virtualclustername}/manager: get: tags: - Virtualcluster summary: Returns the manager node info of the specified virtual cluster

57

247524762477247824792480248124822483248424852486248724882489249024912492249324942495249624972498249925002501250225032504250525062507250825092510251125122513251425152516251725182519252025212522252325242525252625272528252925302531253225332534253525362537253825392540254125422543


description: Returns the manager node info of the specified virtual cluster operationId: cloudmesh.virtualcluster.getManagerByName parameters: - name: virtualclustername description: name of the virtua cluster in: path required: true type: string produces: - application/json responses: "200": description: virtual cluster manager node info schema: $ref: "#/definitions/Node" /cloudmesh/virtualcluster/virtualcluster/{virtualclustername}/{nodename}: get: tags: - Virtualcluster summary: Returns the specified node info of the specified virtualcluster description: Returns the specified node info of the specified virtualcluster operationId: cloudmesh.virtualcluster.getNodeByName parameters: - name: virtualclustername description: name of the virtua cluster in: path required: true type: string - name: nodename description: name of the name in: path required: true type: string produces: - application/json responses: "200": description: virtualcluster node info schema: $ref: "#/definitions/Node"definitions: VirtualCluster: type: object properties: name: description: The name of the virtual cluster type: string description: type: string description: A description of the virtual cluster nnodes: type: integer description: number of nodes of the virtual cluster owner: type: string description: owner of the virtual cluster manager: description: Manager node of the virtual cluster $ref: "#/definitions/Node" nodes: description: List of nodes of the virtual cluster type: array items: $ref: "#/definitions/Node" timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

Node:

58

254425452546254725482549255025512552255325542555255625572558255925602561256225632564256525662567256825692570257125722573257425752576257725782579258025812582258325842585258625872588258925902591259225932594259525962597259825992600260126022603260426052606260726082609261026112612


type: object properties: name: type: string description: name of the node state: type: string description: power state of the node ncpu: type: integer description: number of virtual CPUs of the node ram: type: string description: RAM size of the node disk: type: string description: Disk size of the node nics: type: array description: List of network interfaces of the node items: $ref: "#/definitions/NIC" NIC: type: object properties: mac: type: string description: MAC address of the node ip: type: string description: IP address of the node

4.5.2 SCHEDULERThe scheduler is a service to store scheduler, value, and type information, all of which are stored as strings.

4.5.2.1 Properties SchedulerProperty Type Description

name string name of the scheduler

value string information of the scheduler

kind string the scheduler kind or type


4.5.2.2 Paths

4.5.2.2.1 /cloudmesh/schedulers

4.5.2.2.1.1 GET /cloudmesh/schedulersReturns all schedulers

Responses


200 scheduler info

59

2613261426152616261726182619262026212622262326242625262626272628262926302631263226332634263526362637263826392640264126422643

2644

26452646

2647

2648

2649

2650

2651

2652


4.5.2.2.1.2 PUT /cloudmesh/schedulersCreate a new scheduler

Responses

Code Description

Schema

201 Created

Parameters


scheduler

body The new scheduler to create

False Scheduler

4.5.2.2.2 /cloudmesh/scheduler/{name}

4.5.2.2.2.1 GET /cloudmesh/scheduler/{name}Returns a scheduler

Responses


200 scheduler info

Scheduler

Parameters

Name Located in


name path name of the scheduler True

4.5.2.3 scheduler.yamlswagger: "2.0"info: version: 3.1.1 x-date: 02-15-2019 x-status: defined title: Scheduler description: |- The scheduler is a service to store scheduler, value, and type information, all of which are stored as strings. termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths:

60

2653

2654

2655

2656

2657

2658

2659

2660

2661

26622663266426652666266726682669267026712672267326742675267626772678267926802681268226832684268526862687


/cloudmesh/schedulers: get: tags: - Scheduler summary: Returns all schedulers description: Returns all schedulers operationId: cloudmesh.scheduler.get produces: - application/json responses: '200': description: scheduler info schema: type: array items: $ref: '#/definitions/Scheduler' put: tags: - Scheduler summary: Create a new scheduler description: Create a new scheduler operationId: cloudmesh.scheduler.add parameters: - in: body name: scheduler description: The new scheduler to create schema: $ref: '#/definitions/Scheduler' responses: '201': description: Created '/cloudmesh/scheduler/{name}': get: tags: - Scheduler summary: Returns a scheduler description: Returns a scheduler operationId: cloudmesh.scheduler.get_by_name parameters: - name: name description: name of the scheduler in: path required: true type: string produces: - application/json responses: '200': description: scheduler info schema: $ref: '#/definitions/Scheduler'definitions: Scheduler: type: object description: the scheduler properties: name: type: string description: name of the scheduler value: type: string description: information of the scheduler kind: type: string description: the scheduler kind or type timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

61

26882689269026912692269326942695269626972698269927002701270227032704270527062707270827092710271127122713271427152716271727182719272027212722272327242725272627272728272927302731273227332734273527362737273827392740274127422743274427452746274727482749275027512752275327542755


4.6 COMPUTE MANAGEMENT - VIRTUAL MACHINESThis section summarizes a basic interface specification of virtual machines.

4.6.1 IMAGEImage defines an object representing a system image in a cloud that can be used to start virtual machines.

4.6.1.1 Properties ImageProperty Type Description

id string A unique id for the image

name string Name of the image

label string A label that can be defined by the user for the image

description string A description for the image

collection string The collection in which the image fits

cloud string A cloud provider for which the image is designed. If multiple are using the image, they are passed along as space seperated strings

os_type string The OS of the image

osVersion string The OS version of the image

min_requirement

minimum requirement to run the image

status string The status of the image

progress integer

The loading progress percentage of the image

visibility string The visibility of the image

timestamp timestamps associated with the file

4.6.1.2 Properties Minimum_RequirementsProperty Type Description

disk_space

integer

Minimum disk space in bytes required for the image

ram integer

Minimum ram size in bytes to run the image

cpu string Minimum cpu required to run the image including number of cores and clock speed

4.6.1.3 Paths

4.6.1.3.1 /cloudmesh/image

4.6.1.3.1.1 GET /cloudmesh/imageReturns all general description images

62

2756

2757

2758

2759

2760

2761

2762

2763

2764

2765


Responses


200 default image info

4.6.1.3.1.2 PUT /cloudmesh/imageCreates a new image

Responses

Code Description

Schema

201 Created

Parameters

Name


image body The new image record to create True Image

4.6.1.3.2 /cloudmesh/image/{name}

4.6.1.3.2.1 GET /cloudmesh/image/{name}Returns general description of an image

Responses

Code Description

Schema

200 image info Image

Parameters

Name


Schema

name path name of the image True

4.6.1.4 image.yamlswagger: "2.0"info: version: 3.1.1 x-date: 02-15-2019 x-status: defined title: Image description: |- Image defines an object representing a system image in a cloud that can be used to start virtual machines. termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist/spec/ license:

63

2766

2767

2768

2769

2770

2771

2772

2773

2774

2775

27762777277827792780278127822783278427852786278727882789279027912792


name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/image: get: tags: - Image summary: Returns all general description images description: Returns all general description images operationId: cloudmesh.image.get produces: - application/json responses: '200': description: default image info schema: type: array items: $ref: '#/definitions/Image' put: tags: - Image summary: Create a new image description: Create a new image operationId: cloudmesh.image.put parameters: - in: body name: image required: true description: The new image record to create schema: $ref: '#/definitions/Image' responses: '201': description: Created '/cloudmesh/image/{name}': get: tags: - Image summary: Returns general description of an image description: Returns general description of an image operationId: cloudmesh.image.get_by_name parameters: - name: name description: name of the image in: path required: true type: string produces: - application/json responses: '200': description: image info schema: $ref: '#/definitions/Image'definitions: Image: type: object properties: id: type: string description: A unique id for the image name:

64

279327942795279627972798279928002801280228032804280528062807280828092810281128122813281428152816281728182819282028212822282328242825282628272828282928302831283228332834283528362837283828392840284128422843284428452846284728482849285028512852285328542855285628572858285928602861


type: string description: Name of the image label: type: string description: A label that can be defined by the user for the image description: type: string description: A description for the image collection: type: string description: The collection in which the image fits cloud: type: string description: A cloud provider for which the image is designed. If multiple are using the image, they are passed along as space seperated strings os_type: type: string description: The OS of the image osVersion: type: string description: The OS version of the image min_requirement: $ref: '#/definitions/Minimum_Requirements' description: minimum requirement to run the image status: type: string description: The status of the image progress: type: integer description: The loading progress percentage of the image visibility: description: The visibility of the image type: string timestamp: description: timestamps associated with the file $ref: '#/definitions/Timestamp' Minimum_Requirements: type: object properties: disk_space: type: integer description: Minimum disk space in bytes required for the image ram: type: integer description: Minimum ram size in bytes to run the image cpu: type: string description: Minimum cpu required to run the image including number of cores and clock speed

4.6.2 FLAVORThe flavor specifies elementary information about a compute node. This information includes name, id, label, ram size, swap size, disk space, availability of ephemeral disk, available bandwidth, price value, cloud name, and date and time of last update. Flavors and the corresponding information are essential to size a virtual cluster appropriately.

65

2862286328642865286628672868286928702871287228732874287528762877287828792880288128822883288428852886288728882889289028912892289328942895289628972898289929002901290229032904290529062907290829092910

2911

2912291329142915


4.6.2.1 Properties FlavorProperty Type Description

name string name of the flavor

id string the id of the flavor

label string a label that a user can set for this flavor

description string A description for the commpute node

ram integer number of bytes used for the image in RAM

swap integer number of bytes used for the image in SWAP

disk integer number of bytes used for the disk

ephemeral_disk boolean specifies wether the flavor features an ephemeral disk

bandwidth integer bandwidth of the node

price integer price for the flavor

cloud string name of the cloud this flavor is used


4.6.2.2 Paths

4.6.2.2.1 /cloudmesh/flavors

4.6.2.2.1.1 GET /cloudmesh/flavorsReturns all flavors

Responses

Code Description

Schema

200 flavor info

4.6.2.2.1.2 PUT /cloudmesh/flavorsCreates a new flavor

Responses

Code Description

Schema

201 Created

66

2916

2917

2918

2919

2920

2921

2922

2923

2924


Parameters


flavor body The new flavor to create

True Flavor

4.6.2.2.2 /cloudmesh/flavor/{name}

4.6.2.2.2.1 GET /cloudmesh/flavor/{name}Returns a flavor

Responses

Code Description

Schema

200 flavor info Flavor

Parameters

Name


name path name of the flavor True

4.6.2.3 flavor.yamlswagger: "2.0"info: version: 3.1.1 x-date: 02-15-2019 x-status: defined title: Flavor description: |- The flavor specifies elementary information about a compute node. This information includes name, id, label, ram size, swap size, disk space, availability of ephemeral disk, available bandwidth, price value, cloud name, and date and time of last update. Flavors and the correponding information are essential to size a virtual cluster appropriately.

termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apache url: https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txthost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/flavors: get: tags: - Flavor summary: Returns all flavors description: Returns all flavors

67

2925

2926

2927

2928

2929

2930

2931293229332934293529362937293829392940294129422943294429452946294729482949295029512952295329542955295629572958295929602961296229632964296529662967


operationId: cloudmesh.flavor.get produces: - application/json responses: '200': description: flavor info schema: type: array items: $ref: '#/definitions/Flavor' put: tags: - Flavor summary: Create a new flavor description: Create a new flavor operationId: cloudmesh.flavor.add parameters: - in: body name: flavor required: true description: The new flavor to create schema: $ref: '#/definitions/Flavor' responses: '201': description: Created '/cloudmesh/flavor/{name}': get: tags: - Flavor summary: Returns a flavor description: Returns a flavor operationId: cloudmesh.flavor.get_by_name parameters: - name: name description: name of the flavor in: path required: true type: string produces: - application/json responses: '200': description: flavor info schema: $ref: '#/definitions/Flavor'definitions: Flavor: type: object description: the flavor properties: name: type: string description: name of the flavor id: type: string description: the id of the flavor label: type: string description: a label that a user can set for this flavor description: type: string description: A description for the commpute node ram: type: integer description: number of bytes used for the image in RAM swap: type: integer description: number of bytes used for the image in SWAP

68

296829692970297129722973297429752976297729782979298029812982298329842985298629872988298929902991299229932994299529962997299829993000300130023003300430053006300730083009301030113012301330143015301630173018301930203021302230233024302530263027302830293030303130323033303430353036


disk: type: integer description: number of bytes used for the disk ephemeral_disk: type: boolean description: specifies wether the flavor features an ephemeral disk bandwidth: type: integer description: bandwidth of the node price: type: integer description: price for the flavor cloud: type: string description: name of the cloud this flavor is used timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.6.3 VMVM is a service to manage virtual machines.

4.6.3.1 Properties VMProperty Type Description

provider string

Name of the provider

id string

a unique id for the vm

name string

the name of the vm

image string

the image for the vm

region string

an optional region

size string

The size of the vm

state string

The state of the vm

private_ips string

The private IPs

public_ips string

The public IPS

metadata string

The meta data passed along to the VM


69

303730383039304030413042304330443045304630473048304930503051305230533054

3055

3056

3057


4.6.3.2 Paths

4.6.3.2.1 /vm

4.6.3.2.1.1 GET /vmReturns the list of the vms

Responses

Code

Description Schema

200 Listing the VMs

Parameters

Name


Schema

cloud query specify the cloud from which we list, if ommitted all clouds are returned.

False

4.6.3.3 vm.yamlswagger: "2.0"info: description: |-

VM is a service to manage virtual machines. version: 3.1.1 x-date: 02-15-2019 x-status: defined title: Virtual Machine termsOfService: https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt contact: name: NIST BDRA Interface Subgroup Service Example url: https://cloudmesh-community.github.io/nist/spec/ license: name: Apachehost: localhost:8080basePath: /cloudmesh/v3schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /vm: get: tags: - VM summary: Returns the list of the vms description: Returns the list of the vms operationId: cloudmesh.vm.get produces: - application/json parameters: - name: cloud in: query type: string description: 'specify the cloud from which we list, if ommitted all clouds are returned.' required: false

70

3058

3059

3060

3061

3062

3063

306430653066306730683069307030713072307330743075307630773078307930803081308230833084308530863087308830893090309130923093309430953096309730983099310031013102310331043105


responses: '200': description: Listing the VMs schema: type: array items: $ref: '#/definitions/VM'definitions: VM: type: object properties: provider: type: string description: Name of the provider id: type: string description: a unique id for the vm name: type: string description: the name of the vm image: type: string description: the image for the vm region: type: string description: an optional region size: type: string description: The size of the vm state: type: string description: The state of the vm private_ips: type: string description: The private IPs public_ips: type: string description: The public IPS metadata: type: string description: The meta data passed along to the VM timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp' example: image: image metadata: metadata size: size provider: provider name: name private_ips: private_ips id: id state: state region: region public_ips: public_ips

4.6.4 SECGROUPA security group defines the incoming and outgoing security rules which can then be assigned to a node when a node is being created. Once the node is up, the connection to and from the node will be decided by the security group rules, in addition to any other possible rules applied on network devices or from the instance’s firewall settings. A security group may have one or multiple rules and a node may be associated with one or more security groups.

71

3106310731083109311031113112311331143115311631173118311931203121312231233124312531263127312831293130313131323133313431353136313731383139314031413142314331443145314631473148314931503151315231533154315531563157315831593160

3161

31623163316431653166


4.6.4.1 Properties SecgroupProperty Type Description

uuid string Unique identifier of the security group

name string name of the secgroup

description string describes what the secgroup is for

rules array[#/definitions/SecGroupRule]

List of Secgroup rules

4.6.4.2 Properties SecGroupRuleProperty Type Description

uuid string Unique identifier of the rule

ingress boolean The defined security group rule is for ingress if True

egress boolean The defined security group rule is for egress if True

remote_group string Name of the group if the rule is defined by group instead of IP range

protocol string The protocol used such as TCP, UDP, ICMP

from_port integer Port range starting port

to_port integer Port range ending port

cidr string The source or destination network in CIDR notation, e.g., 129.79.0.0/16


4.6.4.3 Paths

4.6.4.3.1 /cloudmesh/secgroup

4.6.4.3.1.1 GET /cloudmesh/secgroupReturns all secgroups

Responses


200 secgroup info

4.6.4.3.1.2 POST /cloudmesh/secgroupCreates a new secgroup

Responses

Code Description

Schema

201 Created

Parameters

Name Located Description Required Schema

72

3167

3168

3169

3170

3171

3172

3173

3174

3175

3176

3177


in

secgroup body The new secgroup to create

False Secgroup

4.6.4.3.2 /cloudmesh/secgroup/{secgroup}

4.6.4.3.2.1 GET /cloudmesh/secgroup/{secgroup}Find a secgroup by name

Responses


200 secgroup info Secgroup

Parameters

Name Located in

Description Required

Schema

secgroup path name of the security group True

4.6.4.3.3 /cloudmesh/secgroup/{secgroup}/rule

4.6.4.3.3.1 GET /cloudmesh/secgroup/{secgroup}/ruleGet the defined security group rules for a specified secgroup

Responses

Code

Description Schema

200 security group rules info

Parameters

Name Located in


Schema

secgroup path name of the security group True

4.6.4.3.3.2 POST /cloudmesh/secgroup/{secgroup}/ruleCreates a new rule in secgroup

Responses

Code Description

Schema

201 Created

Parameters


secgroup path The name of the new secgroup to create True

secgroupob body The secgroup object to create True SecGroupRule

73

3178

3179

3180

3181

3182

3183

3184

3185

3186

3187

3188

3189

3190

3191


j

4.6.4.3.4 /cloudmesh/secgroup/{secgroup}/rule/{ruleid}

4.6.4.3.4.1 GET /cloudmesh/secgroup/{secgroup}/rule/{ruleid}Creates a new rule in secgroup

Responses

Code

Description Schema

200 The security group rule definition info

SecGroupRule

Parameters


Schema

secgroup

path The named of the secgroup from which the rule will be deleted

True

ruleid path The uuid of the rule to be deleted True

4.6.4.3.4.2 DELETE /cloudmesh/secgroup/{secgroup}/rule/{ruleid}Creates a new rule in secgroup

Responses


202 Deletion request accepted and will be performed

Parameters


Schema

secgroup

path The named of the secgroup from which the rule will be deleted

True

ruleid path The uuid of the rule to be deleted True

4.6.4.4 secgroup.yamlswagger: "2.0"info: version: 3.1.1 x-date: 02-12-2019 x-status: defined title: Secgroup description: |- A security group defines the incoming and outgoing security rules which can then be assigned to a node when a node is being created. Once the node is up the connection to and from the node will be decided by the security group rules, in addition to any other possible rules applied on network devices or from the instance"s firewall settings.

74

3192

3193

3194

3195

3196

3197

3198

3199

3200

320132023203320432053206320732083209321032113212321332143215


A security group may have one or multiple rules and a node may be associated with one or more security groups. termsOfService: "https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt" contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/secgroup: get: tags: - Secgroup summary: Returns all secgroups description: Returns all secgroups operationId: cloudmesh.secgroup.get produces: - application/json responses: "200": description: secgroup info schema: type: array items: $ref: "#/definitions/Secgroup" post: tags: - Secgroup summary: Create a new secgroup description: Create a new secgroup operationId: cloudmesh.secgroup.add parameters: - in: body name: secgroup description: The new secgroup to create schema: $ref: "#/definitions/Secgroup" responses: "201": description: Created "/cloudmesh/secgroup/{secgroup}": get: tags: - Secgroup summary: Find a secgroup by name description: Find a secgroup by name operationId: cloudmesh.secgroup.get_by_name parameters: - name: secgroup description: name of the security group in: path required: true type: string produces: - application/json responses: "200": description: secgroup info schema: $ref: "#/definitions/Secgroup" "/cloudmesh/secgroup/{secgroup}/rule":

75

321632173218321932203221322232233224322532263227322832293230323132323233323432353236323732383239324032413242324332443245324632473248324932503251325232533254325532563257325832593260326132623263326432653266326732683269327032713272327332743275327632773278327932803281328232833284


get: tags: - Secgroup summary: Get the defiend security group rules for a specified secgroup description: Get the defiend security group rules for a specified secgroup operationId: cloudmesh.secgroup.get_rules_in_secgroup parameters: - name: secgroup description: name of the security group in: path required: true type: string produces: - application/json responses: "200": description: security group rules info schema: type: array items: $ref: "#/definitions/SecGroupRule" post: tags: - Secgroup summary: Create a new rule in the specified security group description: Create a new rule in secgroup operationId: cloudmesh.secgroup.add_rule_in_secgroup parameters: - in: path name: secgroup required: true description: The name of the new secgroup to create type: string - in: body name: secgroupobj required: true description: The secgroup object to create schema: $ref: "#/definitions/SecGroupRule" responses: "201": description: Created "/cloudmesh/secgroup/{secgroup}/rule/{ruleid}": get: tags: - Secgroup summary: Get an existing rule from the specified security group description: Create a new rule in secgroup operationId: cloudmesh.secgroup.get_rule_by_id_in_secgroup parameters: - in: path name: secgroup required: true description: The named of the secgroup from which the rule will be deleted type: string - in: path name: ruleid required: true description: The uuid of the rule to be deleted type: string responses: "200": description: The security group rule definition info schema: $ref: "#/definitions/SecGroupRule" delete: tags: - Secgroup summary: Delete an existing rule from the specified security group

76

328532863287328832893290329132923293329432953296329732983299330033013302330333043305330633073308330933103311331233133314331533163317331833193320332133223323332433253326332733283329333033313332333333343335333633373338333933403341334233433344334533463347334833493350335133523353


description: Create a new rule in secgroup operationId: cloudmesh.secgroup.delete_rule_by_id_in_secgroup parameters: - in: path name: secgroup required: true description: The named of the secgroup from which the rule will be deleted type: string - in: path name: ruleid required: true description: The uuid of the rule to be deleted type: string responses: "202": description: Deletion request accepted and will be performeddefinitions: Secgroup: type: object description: the security group object properties: uuid: type: string description: Unique identifier of the security group name: type: string description: name of the secgroup description: type: string description: describes what the secgroup is for rules: type: array description: List of Secgroup rules items: $ref: "#/definitions/SecGroupRule" SecGroupRule: type: object description: security group rule properties: uuid: type: string description: Unique identifier of the rule ingress: type: boolean description: The defined security group rule is for ingress if True egress: type: boolean description: The defined security group rule is for egress if True remote_group: type: string description: Name of the group if the rule is defined by group instead of IP range protocol: type: string description: The protocol used such as TCP, UDP, ICMP from_port: type: integer description: Port range starting port to_port: type: integer description: Port range ending port cidr: type: string description: The source or destination network in CIDR notation, e.g., 129.79.0.0/16 timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

77

335433553356335733583359336033613362336333643365336633673368336933703371337233733374337533763377337833793380338133823383338433853386338733883389339033913392339333943395339633973398339934003401340234033404340534063407340834093410341134123413341434153416341734183419


4.6.5 NICA service to store Network Interface Controller (NIC) information. All of them are stored as Strings.

4.6.5.1 Properties NicProperty Type Description

name string name of the nic

kind string kind of the nic, such as wireless

mac string the mac address

ip string the ip address

mask string the network mask

broadcast string the broadcast address

gateway string the gateway address

mtu integer the mtu

bandwidth integer the bandwidth in bps


4.6.5.2 Paths

4.6.5.2.1 /cloudmesh/nics

4.6.5.2.1.1 GET /cloudmesh/nicsReturns all nics

Responses

Code Description

Schema

200 nic info

4.6.5.2.1.2 PUT /cloudmesh/nicsCreates a new nic

Responses

Code Description

Schema

201 Created

Parameters


nic body The new nic to create

False Nic

78

3420

3421

3422

3423

3424

3425

3426

3427

3428

3429

3430

3431


4.6.5.2.2 /cloudmesh/nic/{name}

4.6.5.2.2.1 GET /cloudmesh/nic/{name}Returns a nic

Responses

Code Description

Schema

200 nic info Nic

Parameters


name path Get nic info by name

True

4.6.5.3 nic.yamlswagger: "2.0"info: version: 3.0.1 title: Nic description: |- A service to store Network Interface Controller (NIC) information. All of them are stored as Strings.

* TODO: assign and improve termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/nics: get: tags: - NIC summary: Returns all nics description: Returns all nics operationId: cloudmesh.nic.get produces: - application/json responses: '200': description: nic info schema: type: array items: $ref: '#/definitions/Nic' put: tags: - NIC summary: Create a new nic

79

3432

3433

3434

3435

3436

3437343834393440344134423443344434453446344734483449345034513452345334543455345634573458345934603461346234633464346534663467346834693470347134723473347434753476347734783479348034813482


description: Create a new nic operationId: cloudmesh.nic.put parameters: - in: body name: nic description: The new nic to create schema: $ref: '#/definitions/Nic' responses: '201': description: Created '/cloudmesh/nic/{name}': get: tags: - NIC summary: Returns a nic description: Returns a nic operationId: cloudmesh.nic.get_by_name parameters: - name: name description: Get nic info by name in: path required: true type: string produces: - application/json responses: '200': description: nic info schema: $ref: '#/definitions/Nic'definitions: Nic: type: object description: the nic properties: name: type: string description: name of the nic kind: type: string description: kind of the nic, such as wireless mac: type: string description: the mac address ip: type: string description: the ip address mask: type: string description: the network mask broadcast: type: string description: the broadcast address gateway: type: string description: the gateway address mtu: type: integer description: the mtu bandwidth: type: integer description: the bandwidth in bps timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

80

348334843485348634873488348934903491349234933494349534963497349834993500350135023503350435053506350735083509351035113512351335143515351635173518351935203521352235233524352535263527352835293530353135323533353435353536353735383539354035413542354335443545354635473548


4.7 COMPUTE MANAGEMENT - CONTAINERS4.7.1 CONTAINERSA service to store containers.

4.7.1.1 Properties ContainerProperty Type Description

name string name of the container

version string version of the container

label string label of the container

type string type of the container

definition string definition or manifest of the container

imgURI string URI of the container

tags array[string] tags of the container

timestamp

timestamps associated with the resource

4.7.1.2 Paths

4.7.1.2.1 /cloudmesh/container

4.7.1.2.1.1 GET /cloudmesh/containerReturns all containers

Responses


200 containers info

4.7.1.2.1.2 PUT /cloudmesh/containerCreates a new container

Responses

Code Description

Schema

201 Created

Parameters


containers

body The new containers to create

False Container

81

3550

3551

3552

3553

3554

3555

3556

3557

3558

3559

3560

3561

3562


4.7.1.2.2 /cloudmesh/container/{name}

4.7.1.2.2.1 GET /cloudmesh/container/{name}Returns a container

Responses

Code

Description Schema

200 containers info Container

Parameters

Name Located in


Schema

name path name of the container True

4.7.1.3 containers.yamlswagger: "2.0"info: version: 3.1.1 x-date: 02-15-2019 x-status: defined title: Containers description: |-

A service to store containers.

termsOfService: https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/container: get: tags: - Container summary: Returns all containers description: Returns all containers operationId: cloudmesh.conteiners.get produces: - application/json responses: 200: description: containers info schema: type: array items: $ref: '#/definitions/Container' put: tags: - Container summary: Create a new containers description: Create a new containers

82

3563

3564

3565

3566

3567

3568356935703571357235733574357535763577357835793580358135823583358435853586358735883589359035913592359335943595359635973598359936003601360236033604360536063607360836093610361136123613


operationId: cloudmesh.containers.put parameters: - in: body name: containers description: The new containers to create schema: $ref: '#/definitions/Container' responses: 201: description: Created '/cloudmesh/container/{name}': get: tags: - Container summary: Returns a containers description: Returns a containers operationId: cloudmesh.containers.get_by_name parameters: - name: name description: name of the container in: path required: true type: string produces: - application/json responses: 200: description: containers info schema: $ref: '#/definitions/Container'definitions: Container: type: object description: A record representing a container properties: name: type: string description: name of the container version: type: string description: version of the container label: type: string description: label of the container type: type: string description: type of the container definition: type: string description: definition or manifest of the container imgURI: type: string description: URI of the container tags: type: array description: tags of the container items: type: string timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

83

3614361536163617361836193620362136223623362436253626362736283629363036313632363336343635363636373638363936403641364236433644364536463647364836493650365136523653365436553656365736583659366036613662366336643665366636673668366936703671367236733674


4.8 COMPUTE MANAGEMENT - FUNCTIONS4.8.1 MICROSERVICEAs part of microservices, a function with parameters that can be invoked has been defined.

4.8.1.1 Properties MicroserviceProperty Type Description

name string name of the microservice

endpoint string the end point of the microservice

function string the function the microservice represents


4.8.1.2 Paths

4.8.1.2.1 /cloudmesh/microservices

4.8.1.2.1.1 GET /cloudmesh/microservicesReturns all microservices

Responses

Code

Description Schema

200 microservice info

4.8.1.2.1.2 PUT /cloudmesh/microservicesCreates a new microservice

Responses

Code Description

Schema

201 Created

Parameters

Name Located in


Schema

microservice body The new microservice to create False Microservice

4.8.1.2.2 /cloudmesh/microservice/{name}

4.8.1.2.2.1 GET /cloudmesh/microservice/{name}Returns a microservice

Responses

Cod Description Schema

84

3675

3676

3677

3678

3679

3680

3681

3682

3683

3684

3685

3686

3687

3688

3689

3690

3691


e

200 microservice info Microservice

Parameters

Name


name path Get microservie info by name True

4.8.1.3 microservice.yamlswagger: "2.0"info: version: 3.0.1 title: Microservice description: |- As part of microservices, a function with parameters that can be invoked has been defined.

* TODO: assign and improve termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/microservices: get: tags: - Microservice summary: Returns all microservices description: Returns all microservices operationId: cloudmesh.microservice.get produces: - application/json responses: '200': description: microservice info schema: type: array items: $ref: '#/definitions/Microservice' put: tags: - Microservice summary: Create a new microservice description: Create a new microservice operationId: cloudmesh.microservice.put parameters: - in: body name: microservice description: The new microservice to create schema: $ref: '#/definitions/Microservice' responses:

85

3692

3693369436953696369736983699370037013702370337043705370637073708370937103711371237133714371537163717371837193720372137223723372437253726372737283729373037313732373337343735373637373738373937403741374237433744374537463747


'201': description: Created '/cloudmesh/microservice/{name}': get: tags: - Microservice summary: Returns a microservice description: Returns a microservice operationId: cloudmesh.microservice.get_by_name parameters: - name: name in: path description: Get microservie info by name required: true type: string produces: - application/json responses: '200': description: microservice info schema: $ref: '#/definitions/Microservice'definitions: Microservice: type: object description: the microservice properties: name: type: string description: name of the microservice endpoint: type: string description: the end point of the microservice function: type: string description: the function the microservice represents timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.9 BATCH PROCESSING4.9.1 BATCHJOBVM is a service to manage virtual machines.

86

374837493750375137523753375437553756375737583759376037613762376337643765376637673768376937703771377237733774377537763777377837793780378137823783378437853786

3787

3788

3789



provider string


id string


name string

the name of the vm

image string


region string

an optional region

size string

The size of the vm

state string

The state of the vm

private_ips string

The private IPs

public_ips string

The public IPS

metadata string



4.9.1.2 Paths

4.9.1.2.1 /vm


Responses

Code

Description Schema

200 Listing the VMs

Parameters

Name


Schema

cloud query specify the cloud from which we list, if omitted all clouds are returned.

False

87

3790

3791

3792

3793

3794

3795

3796


4.9.1.3 batchjob.yamlswagger: "2.0"info: version: 3.0.2 x-date: 10-30-2018 title: Batch Job description: |- Computing jobs that can run without end user interaction, and are scheduled through queuing systems, are called batch jobs. Batch jobs are used to minimize human interaction and allow the submission and scheduling of many jobs in parallel while attempting to utilize the resources through a resource scheduler more efficiently or simply in sequential order. Batch processing scarce resources managed by batch queues are highly optimized and in many cases, provide significant performance advantages over other services. Disadvantages include the limited and preinstalled software stacks that, in some cases, do not allow the latest applications to run.

termsOfService: https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/batch/job: get: tags: - Batchjob summary: Returns all batchjobs description: Returns all batchjobs operationId: cloudmesh.batchjob.get produces: - application/json responses: 200: description: batchjob info schema: type: array items: $ref: '#/definitions/Batchjob' put: tags: - Batchjob summary: Create a new batchjob description: Create a new batchjob operationId: cloudmesh.batchjob.add parameters: - in: body name: batchjob description: The new batchjob to create schema: $ref: '#/definitions/Batchjob' responses: 201: description: Created '/cloudmesh/batch/job/{name}': get: tags:

88

37973798379938003801380238033804380538063807380838093810381138123813381438153816381738183819382038213822382338243825382638273828382938303831383238333834383538363837383838393840384138423843384438453846384738483849385038513852385338543855385638573858385938603861386238633864


- Batchjob summary: Returns a batchjob description: Returns a batchjob operationId: cloudmesh.batchjob.get_by_name parameters: - name: name description: Get batchjob info by name in: path required: true type: string produces: - application/json responses: 200: description: batchjob info schema: $ref: '#/definitions/Batchjob'definitions: Batchjob: type: object description: the batchjob properties: name: type: string description: name of the batchjob output: type: string description: name of the output file(s) TBD array script: type: string description: the script of the job cmd: type: string description: a command can be scpecified alternatively to the script queue: type: string description: the queue name id: type: string description: after submissin the job gets an id cluster: type: string description: the name of the cluster th job is submitted to time: type: string description: the time the job is to be started duration: type: string description: the duration of the job script_path: type: string description: the path to a script nodes: type: string description: the nodes used to execute the job dir: type: string description: the directory in which to execute the job timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.9.2 SLURMJOBVM is a service to manage virtual machines.

89

3865386638673868386938703871387238733874387538763877387838793880388138823883388438853886388738883889389038913892389338943895389638973898389939003901390239033904390539063907390839093910391139123913391439153916391739183919392039213922392339243925

3926

3927



provider string


id string


name string

the name of the vm

image string


region string

an optional region

size string

The size of the vm

state string

The state of the vm

private_ips string

The private IPs

public_ips string

The public IPS

metadata string



4.9.2.2 Paths

4.9.2.2.1 /vm


Responses

Code

Description Schema

200 Listing the VMs

Parameters

Name


Schema

cloud query specify the cloud from which we list, if ommitted all clouds are returned.

False

90

3928

3929

3930

3931

3932

3933

3934


4.9.2.3 slurmjob.yamlswagger: "2.0"info: version: 3.0.0 x-date: 02-08-2019 title: Batch Job description: |-

Computing jobs based on a slurm script that can be run on a remote slurm cluster without end user interaction are called Slurm jobs. Slurm jobs are helpful to minimize the user's interaction with the slurm workload manager and allows seamless communication to the remote server. The script as well as the accompanied files are automatically transfered to the remote slurm cluster and results are automatically collected when ready.

termsOfService: https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/slurmjob/job: get: tags: - Slurmjob summary: Returns all slurmjobs description: Returns all slurmjobs operationId: cloudmesh.slurmjob.get produces: - application/json responses: 200: description: slurmjob info schema: type: array items: $ref: '#/definitions/Slurmjob' put: tags: - Slurmjob summary: Create a new slurmjob description: Create a new slurmjob operationId: cloudmesh.slurmjob.add parameters: - name: slurmjob in: body description: The new slurmjob to create schema: $ref: '#/definitions/Slurmjob' responses: 200: description: Created '/cloudmesh/slurmjob/job/{name}': get: tags: - Slurmjob summary: Returns a slurmjob description: Returns a slurmjob operationId: cloudmesh.slurmjob.get_by_name

91

39353936393739383939394039413942394339443945394639473948394939503951395239533954395539563957395839593960396139623963396439653966396739683969397039713972397339743975397639773978397939803981398239833984398539863987398839893990399139923993399439953996399739983999400040014002


parameters: - name: name description: name of the slurmjob in: path required: true type: string produces: - application/json responses: 200: description: slurmjob info schema: $ref: '#/definitions/Slurmjob'definitions: Slurmjob: type: object description: the batch job properties: name: type: string description: name of the batch job suffix: type: string description: the suffix of the filename for a job clustername: type: string description: slurm cluster on which the job is gonna run input_type: type: string description: type of the input for the script that is going to be run on remote cluster, possible values are params OR params+file remote_path: type: string description: path in the remotes on which the scripts is gonna be copied to and ran from items: $ref: '#/definitions/Path' slurm_script: type: string description: path of the slurm script items: $ref: '#/definitions/Path' job_script: type: string description: path of the file that is going to be run on the cluster via slurm script items: $ref: '#/definitions/Path' argfile: type: string description: path of the file that has to be passed to the file as an argument (if any) items: $ref: '#/definitions/Path' local_folder: type: string description: local path to which the results are gonna be copied items: $ref: '#/definitions/Path' timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

Path: type: object description: path information properties: folder_path: type: string description: absolute path to the folder file_path: type: string

92

400340044005400640074008400940104011401240134014401540164017401840194020402140224023402440254026402740284029403040314032403340344035403640374038403940404041404240434044404540464047404840494050405140524053405440554056405740584059406040614062406340644065406640674068406940704071


description: absolute path to the file file_name: type: string description: the name of the script/file

4.10 RESERVATION4.10.1 RESERVATIONSome services may consume a considerable amount of resources, necessitating the reservation of resources.

4.10.1.1 Properties ReservationProperty Type Description

name string name of the reservation

service string the name of the service on which we reserve

description

string the description of the reservation

start string the start time and date

end string the end time and date


4.10.1.2 Paths

4.10.1.2.1 /cloudmesh/reservations

4.10.1.2.1.1 GET /cloudmesh/reservationsReturns all reservations

Responses


200 reservation info

4.10.1.2.1.2 PUT /cloudmesh/reservationsCreates a new reservation

Responses

Code Description

Schema

201 Created

Parameters

Name Located in


Schema

93

4072407340744075

4076

4077

40784079

4080

4081

4082

4083

4084

4085

4086

4087

4088

4089


reservation body The new reservation to create False Reservation

4.10.1.2.2 /cloudmesh/reservation/{name}

4.10.1.2.2.1 GET /cloudmesh/reservation/{name}Returns a reservation

Responses

Code

Description Schema

200 reservation info Reservation

Parameters

Name


name path Get reservation info by name

True

4.10.1.3 reservation.yamlswagger: "2.0"info: version: 3.0.2 x-date: 10-30-2018 title: Reservation description: |- Some services may consume a considerable amount of resources, necessitating the reservation of resources.

* TODO: assign and improve termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/reservations: get: tags: - Reservation summary: Returns all reservations description: Returns all reservations operationId: cloudmesh.reservation.get produces: - application/json responses: '200': description: reservation info schema: type: array

94

4090

4091

4092

4093

4094

40954096409740984099410041014102410341044105410641074108410941104111411241134114411541164117411841194120412141224123412441254126412741284129413041314132413341344135


items: $ref: '#/definitions/Reservation' put: tags: - Reservation summary: Create a new reservation description: Create a new reservation operationId: cloudmesh.reservation.add parameters: - in: body name: reservation description: The new reservation to create schema: $ref: '#/definitions/Reservation' responses: '201': description: Created '/cloudmesh/reservation/{name}': get: tags: - Reservation summary: Returns a reservation description: Returns a reservation operationId: cloudmesh.reservation.get_by_name parameters: - name: name description: Get reservation info by name in: path required: true type: string produces: - application/json responses: '200': description: reservation info schema: $ref: '#/definitions/Reservation'definitions: Reservation: type: object description: the reservation properties: name: type: string description: name of the reservation service: type: string description: the name of the service on which we reserve description: type: string description: the description of the reservation start: type: string format: date description: the start time and date end: type: string format: date description: the end time and date timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

95

41364137413841394140414141424143414441454146414741484149415041514152415341544155415641574158415941604161416241634164416541664167416841694170417141724173417441754176417741784179418041814182418341844185418641874188418941904191419241934194419541964197


4.11 DATA STREAMS4.11.1 STREAMThe stream object describes a data flow, providing information about the rate and number of items exchanged while issuing requests to the stream. A stream may return data items in a specific format that is defined by the stream.

4.11.1.1 Properties StreamProperty Type Description

name string name of the stream

format string format of the stream

rate integer the rate of messages

limit integer the limit of items send


4.11.1.2 Paths

4.11.1.2.1 /cloudmesh/streams

4.11.1.2.1.1 GET /cloudmesh/streamsReturns all streams

Responses

Code Description

Schema

200 stream info

4.11.1.2.1.2 PUT /cloudmesh/streamsCreates a new stream

Responses

Code Description

Schema

201 Created

Parameters


stream

body The new stream to create False Stream

96

4198

4199

420042014202

4203

4204

4205

4206

4207

4208

4209

4210

4211

4212


4.11.1.2.2 /cloudmesh/stream/{name}

4.11.1.2.2.1 GET /cloudmesh/stream/{name}Returns a stream

Responses

Code Description

Schema

200 stream info Stream

Parameters


name path Get stream info by name

True

4.11.1.3 stream.yamlswagger: "2.0"info: version: 3.0.2 x-date: 10-30-2018 title: Stream description: |- The stream object describes a data flow, providing information about the rate and number of items exchanged while issuing requests to the stream. A stream may return data items in a specific format that is defined by the stream.

* TODO: assign and improve termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/streams: get: tags: - Stream summary: Returns all streams description: Returns all streams operationId: cloudmesh.stream.get produces: - application/json responses: '200': description: stream info schema: type: array items: $ref: '#/definitions/Stream' put:

97

4213

4214

4215

4216

4217

4218421942204221422242234224422542264227422842294230423142324233423442354236423742384239424042414242424342444245424642474248424942504251425242534254425542564257425842594260426142624263


tags: - Stream summary: Create a new stream description: Create a new stream operationId: cloudmesh.stream.add parameters: - in: body name: stream description: The new stream to create schema: $ref: '#/definitions/Stream' responses: '201': description: Created '/cloudmesh/stream/{name}': get: tags: - Stream summary: Returns a stream description: Returns a stream operationId: cloudmesh.stream.get_by_name parameters: - name: name description: Get stream info by name in: path required: true type: string produces: - application/json responses: '200': description: stream info schema: $ref: '#/definitions/Stream'definitions: Stream: type: object description: the stream properties: name: type: string description: name of the stream format: type: string description: format of the stream rate: type: integer description: the rate of messages limit: type: integer description: the limit of items send timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.11.2 FILTERFilters can operate on a variety of objects and reduce the information received based on a search criterion.

4.11.2.1 Properties FilterProperty Type Description

name string

name of the filter

function strin the function of the data exchanged in the stream

98

426442654266426742684269427042714272427342744275427642774278427942804281428242834284428542864287428842894290429142924293429442954296429742984299430043014302430343044305430643074308430943104311431243134314431543164317

4318

4319

4320


g

kind string

the filter kind or type


4.11.2.2 Paths

4.11.2.2.1 /cloudmesh/filters

4.11.2.2.1.1 GET /cloudmesh/filtersReturns all filters

Responses

Code Description

Schema

200 filter info

4.11.2.2.1.2 PUT /cloudmesh/filtersCreates a new filter

Responses

Code Description

Schema

201 Created

Parameters

Name


filter body The new filter to create False Filter

4.11.2.2.2 /cloudmesh/filter/{name}

4.11.2.2.2.1 GET /cloudmesh/filter/{name}Returns a filter

Responses

Code Description

Schema

200 filter info Filter

Parameters

Name


name path Get filter info by name

True

99

4321

4322

4323

4324

4325

4326

4327

4328

4329

4330

4331

4332

4333

4334


4.11.2.3 filter.yamlswagger: "2.0"info: version: 3.0.2 x-date: 10-30-2018 title: Filter description: |- Filters can operate on a variety of objects and reduce the information received based on a search criterion.

* TODO: assign for review and improvement termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/filters: get: tags: - Filter summary: Returns all filters description: Returns all filters operationId: cloudmesh.filter.get produces: - application/json responses: '200': description: filter info schema: type: array items: $ref: '#/definitions/Filter' put: tags: - Filter summary: Create a new filter description: Create a new filter operationId: cloudmesh.filter.put parameters: - in: body name: filter description: The new filter to create schema: $ref: '#/definitions/Filter' responses: '201': description: Created '/cloudmesh/filter/{name}': get: tags: - Filter summary: Returns a filter description: Returns a filter operationId: cloudmesh.filter.get_by_name parameters: - name: name description: Get filter info by name

100

43354336433743384339434043414342434343444345434643474348434943504351435243534354435543564357435843594360436143624363436443654366436743684369437043714372437343744375437643774378437943804381438243834384438543864387438843894390439143924393439443954396439743984399440044014402


in: path required: true type: string produces: - application/json responses: '200': description: filter info schema: $ref: '#/definitions/Filter'definitions: Filter: type: object description: the filter properties: name: type: string description: name of the filter function: type: string description: the function of the data exchanged in the stream kind: type: string description: the filter kind or type timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.12 MAPREDUCE4.12.1 HADOOPA service to store the information of a hadoop deployment definition. All of the attributes are stored as Strings.

101

440344044405440644074408440944104411441244134414441544164417441844194420442144224423442444254426442744284429

4430

4431

44324433


4.12.1.1 Properties HadoopProperty Type Description

name string name of the hadoop cluster

deployment_type string the type of the deployer such as ansible

deployment_git string the url where to find the deployment scripts

resource_managers integer number of resource managers

namenodes integer number of name nodes

datanodes integer number of data nodes

historynodes integer number of history nodes

journalnodes integer number of journal nodes

yarn boolean

true if yarn is used

hdfs boolean

true if hdfs is used


4.12.1.2 Paths

4.12.1.2.1 /cloudmesh/hadoop

4.12.1.2.1.1 GET /cloudmesh/hadoopReturns all hadoops

Responses

Code Description

Schema

200 hadoop info

4.12.1.2.1.2 PUT /cloudmesh/hadoopCreates a new hadoop

Responses

Code Description

Schema

201 Created

Parameters

Name Located in


Schema

hadoop body The new hadoop to create False Hadoop

102

4434

4435

4436

4437

4438

4439

4440

4441

4442

4443


4.12.1.2.2 /cloudmesh/hadoop/{name}

4.12.1.2.2.1 GET /cloudmesh/hadoop/{name}Returns a hadoop

Responses

Code Description

Schema

200 hadoop info Hadoop

Parameters

Name


name path name of the hadoop

True

4.12.1.3 hadoop.yamlswagger: "2.0"info: version: 3.1.1 x-date: 02-15-2019 x-status: defined title: Hadoop description: |- A service to store the information of a hadoop deployment definition. All of the attributes are stored as Strings.

* TODO: assign for review and imporvement termsOfService: 'https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt' contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/hadoop: get: tags: - Hadoop summary: Returns all hadoops description: Returns all hadoops operationId: cloudmesh.hadoop.get produces: - application/json responses: '200': description: hadoop info schema: type: array items: $ref: '#/definitions/Hadoop'

103

4444

4445

4446

4447

4448

44494450445144524453445444554456445744584459446044614462446344644465446644674468446944704471447244734474447544764477447844794480448144824483448444854486448744884489449044914492


put: tags: - Hadoop summary: Create a new hadoop description: Create a new hadoop operationId: cloudmesh.hadoop.put parameters: - in: body name: hadoop description: The new hadoop to create schema: $ref: '#/definitions/Hadoop' responses: '201': description: Created '/cloudmesh/hadoop/{name}': get: tags: - Hadoop summary: Returns a hadoop description: Returns a hadoop operationId: cloudmesh.hadoop.get_by_name parameters: - name: name in: path description: name of the hadoop required: true type: string produces: - application/json responses: '200': description: hadoop info schema: $ref: '#/definitions/Hadoop'definitions: Hadoop: type: object description: An object representing a Hadoop deployment definition properties: name: type: string description: name of the hadoop cluster deployment_type: type: string description: the type of the deployer such as ansible deployment_git: type: string description: the url where to find the deployment scripts resource_managers: type: integer description: number of resource managers namenodes: type: integer description: number of name nodes datanodes: type: integer description: number of data nodes historynodes: type: integer description: number of history nodes journalnodes: type: integer description: number of journal nodes yarn: type: boolean description: true if yarn is used hdfs: type: boolean

104

449344944495449644974498449945004501450245034504450545064507450845094510451145124513451445154516451745184519452045214522452345244525452645274528452945304531453245334534453545364537453845394540454145424543454445454546454745484549455045514552455345544555455645574558455945604561


description: true if hdfs is used timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

4.13 DEPLOYMENT4.13.1 DEPLOYMENTA service to store deployment, value, type information. All of them are stored as Strings.

4.13.1.1 Properties DeploymentProperty Type Description

cluster string the name of the cluster

stack array[object]

ERROR: description missing


4.13.1.2 Paths

4.13.1.2.1 /cloudmesh/deployments

4.13.1.2.1.1 GET /cloudmesh/deploymentsReturns all deployments

Responses


200 deployment info

4.13.1.2.1.2 PUT /cloudmesh/deploymentsCreates a new deployment

Responses

Code Description

Schema

201 Created

Parameters

Name Located in


deployment body The new deployment to create False Deployment

105

4562456345644565

4566

4567

4568

4569

4570

4571

4572

4573

4574

4575

4576

4577

4578


4.13.1.2.2 /cloudmesh/deployment/{name}

4.13.1.2.2.1 GET /cloudmesh/deployment/{name}Returns a deployment

Responses


200 deployment info Deployment

Parameters

Name


name path Get deployment info by name True

4.13.1.3 deployment.yamlswagger: "2.0"info: version: 3.0.2 x-date: 10-30-2018 title: Deployment description: |- A service to store deployment, value, type information. All of them are stored as Strings.

* TODO: assign for review and improvement termsOfService: https://github.com/cloudmesh-community/nist/blob/master/LICENSE.txt contact: name: NIST BDRA Interface Subgroup url: https://cloudmesh-community.github.io/nist license: name: Apachehost: localhost:8080schemes: - httpconsumes: - application/jsonproduces: - application/jsonpaths: /cloudmesh/deployments: get: tags: - Deployment summary: Returns all deployments description: Returns all deployments operationId: cloudmesh.deployment.get produces: - application/json responses: 200: description: deployment info schema: type: array items: $ref: '#/definitions/Deployment' put: tags: - Deployment summary: Create a new deployment

106

4579

4580

4581

4582

4583

45844585458645874588458945904591459245934594459545964597459845994600460146024603460446054606460746084609461046114612461346144615461646174618461946204621462246234624462546264627462846294630


description: Create a new deployment operationId: cloudmesh.deployment.add parameters: - in: body name: deployment description: The new deployment to create schema: $ref: '#/definitions/Deployment' responses: '201': description: Created '/cloudmesh/deployment/{name}': get: tags: - Deployment summary: Returns a deployment description: Returns a deployment operationId: cloudmesh.deployment.get_by_name parameters: - name: name description: Get deployment info by name in: path required: true type: string produces: - application/json responses: '200': description: deployment info schema: $ref: '#/definitions/Deployment'definitions: Deployment: type: object description: the deployment properties: cluster: type: string description: the name of the cluster stack: type: array items: type: object #layers: string timestamp: description: timestamps associated with the resource $ref: '#/definitions/Timestamp'

107

46314632463346344635463646374638463946404641464246434644464546464647464846494650465146524653465446554656465746584659466046614662466346644665466646674668466946704671467246734674467546764677


5 STATUS CODES AND ERROR RESPONSES

In case of an error or a successful response, the response header contains a HTTP code (see https://tools.ietf.org/html/rfc7231). The response body usually contains the following:

The HTTP response code; An accompanying message for the HTTP response code; and A field or object where the error occurred.

Table 2: HTTP Response Codes

HTTP Response

Description Code

200 OK success code, for GET or HEAD request.

201 Created success code, for POST request.

204 No Content success code, for DELETE request.

300 The value returned when an external ID exists in more than one record.

304 The request content has not changed since a specified date and time.

400 The request could not be understood.

401 The session ID or OAuth token used has expired or is invalid.

403 The request has been refused.

404 The requested resource could not be found.

405 The method specified in the Request-Line isn’t allowed for the resource specified in the URI.

415 The entity in the request is in a format that’s not supported by the specified method.

108

4678

4679

46804681

468246834684

4685

https://tools.ietf.org/html/rfc7231


Appendix A: Acronyms and TermsThe following acronyms and terms are used in this volume.

ACID Atomicity, Consistency, Isolation, Durability

API Application Programming Interface

ASCII American Standard Code for Information Interchange

BASE Basically Available, Soft state, Eventual consistency

Container See http://csrc.nist.gov/publications/drafts/800-180/sp800-180_draft.pdf

Cloud Computing The practice of using a network of remote servers hosted on the Internet to store, manage, and process data, rather than a local server or a personal computer. See http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-145.pdf.

DevOps A clipped compound of software DEVelopment and information technology OPerationS

Deployment The action of installing software on resources

HTTP HyperText Transfer Protocol HTTPS HTTP Secure

Hybrid Cloud See http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-145.pdf.

IaaS Infrastructure as a Service SaaS Software as a Service

ITL Information Technology Laboratory

Microservice Architecture Is an approach to build applications based on many smaller modular services. Each module supports a specific goal and uses a simple, well-defined interface to communicate with other sets of services.

NBD-PWG NIST Big Data Public Working Group

NBDRA NIST Big Data Reference Architecture

NBDRAI NIST Big Data Reference Architecture Interface

NIST National Institute of Standards and Technology

OS Operating System

REST REpresentational State Transfer

Replica A duplicate of a file on another resource to avoid costly transfer costs in case of frequent access.

Serverless Computing Serverless computing specifies the paradigm of function as a service (FaaS). It is a cloud computing code execution model in which a cloud provider manages the function deployment and utilization while clients can utilize them. The charge model is based on execution of the function rather than the cost to manage and host the VM or container.

Software Stack A set of programs and services that are installed on a resource to support applications.

109

4686

4687

4688

4689

4690

4691

4692

469346944695

46964697

4698

4699

47004701

4702

4703

470447054706

4707

4708

4709

4710

4711

4712

47134714

47154716471747184719

47204721

http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-145.pdf



http://csrc.nist.gov/publications/drafts/800-180/sp800-180_draft.pdf


Virtual Filesysyem An abstraction layer on top of a distributed physical file system to allow easy access to the files by the user or application.

Virtual Machine A virtual machine (VM) is a software computer that, like a physical computer, runs an operating system and applications. The VM is composed of a set of specification and configuration files and is backed by the physical resources of a host.

Virtual Cluster A virtual cluster is a software cluster that integrate either VMs, containers, or physical resources into an agglomeration of compute resources. A virtual cluster allows users to authenticate and authorize to the virtual compute nodes to utilize them for calculations. Optional high-level services that can be deployed on a virtual cluster may simplify interaction with the virtual cluster or provide higher-level services.

Workflow The sequence of processes or tasks

WWW World Wide Web

110

47224723

4724472547264727

472847294730473147324733

4734

4735

4736


Appendix B: Bibliography[1] W. L. Chang (Co-Chair), N. Grady (Subgroup Co-chair), and NIST Big Data Public

Working Group, “NIST Big Data Interoperability Framework: Volume 1, Big Data Definitions (NIST SP 1500-1r1 VERSION 2),” Jun. 2018.

[2] W. L. Chang (Co-Chair), N. Grady (Subgroup Co-chair), and NIST Big Data Public Working Group, “NIST Big Data Interoperability Framework: Volume 2, Big Data Taxonomies (NIST SP 1500-2r1 VERSION 2),” Jun. 2018.

[3] W. L. Chang (Co-Chair) and G. Fox (Subgroup Co-chair), “NIST Big Data Interoperability Framework: Volume 3, Big Data Use Cases and General Requirements (NIST SP 1500-3r1 VERSION 2),” Jun. 2018.

[4] W. L. Chang (Co-Chair), A. Roy (Subgroup Co-chair), M. Underwood (Subgroup Co-chair), and NIST Big Data Public Working Group, “NIST Big Data Interoperability Framework: Volume 4, Big Data Security and Privacy (NIST SP 1500-4r1 VERSION 2),” Jun. 2018.

[5] W. L. Chang (Co-Chair), S. Mishra (Editor), and NIST Big Data Public Working Group, “NIST Big Data Interoperability Framework: Volume 5, Big Data Architectures White Paper Survey (NIST SP 1500-5 VERSION 1),” Sep. 2015.

[6] W. L. Chang (Co-Chair), D. Boyd (Subgroup Co-chair), and NIST Big Data Public Working Group, “NIST Big Data Interoperability Framework: Volume 6, Big Data Reference Architecture (NIST SP 1500-6r1 VERSION 2),” Jun. 2018.

[7] W. L. Chang (Co-Chair), R. Reinsch (Subgroup Co-chair), and NIST Big Data Public Working Group, “NIST Big Data Interoperability Framework: Volume 7, Big Data Standards Roadmap (NIST SP 1500-7r1 VERSION 2),” Jun. 2018.

[8] W. L. Chang (Co-Chair), R. Reinsch (Subgroup Co-chair), and NIST Big Data Public Working Group, “NIST Big Data Interoperability Framework: Volume 9, Adoption and Modernization (NIST SP 1500-10 VERSION 1),” Jun. 2018.

[9] T. White House Office of Science and Technology Policy, “Big Data is a Big Deal,” OSTP Blog, 2012. [Online]. Available: http://www.whitehouse.gov/blog/2012/03/29/big-data-big-deal. [Accessed: 21-Feb-2014].

[10] Department of Defense, “The DoDAF Architecture Framework Version 2.02,” Apr. 2010.

111

4737

473847394740

474147424743

474447454746

4747474847494750

475147524753

475447554756

475747584759

476047614762

476347644765

4766

4767

Date post:	03-Jul-2019
Category:	Documents
Upload:	buique
View:	212 times
Download:	0 times

Introduction · Web viewPhilips Gregor von Laszewski Indiana University Badi Abdhul Wahid Indiana...

Documents