+ All Categories
Home > Documents > Norstore dataseminar - overview - Sigma2 · Computational Chemistry (CTCC) t 1IZTJDT PG (FPMPHJDBM...

Norstore dataseminar - overview - Sigma2 · Computational Chemistry (CTCC) t 1IZTJDT PG (FPMPHJDBM...

Date post: 07-Sep-2018
Category:
Upload: lykhanh
View: 214 times
Download: 0 times
Share this document with a friend
19
Norstore Overview Andreas O Jaunsen UNINETT Sigma 2
Transcript

NJORDnjord.hpc.ntnu.no is an IBM p575 at NTNU with 186 compute nodes each with eight dual-core CPUs (power5+ 1.9 Ghz). Total 2976 cores. The nodes have 16 GB memory (4 nodes have 128 GB memory). The nodes are connected with IBM HPS switch technology. Operating system: AIX.

TROMSØSTALLO

TRONDHEIM NJORD

BERGENHEXAGON

OSLOTITAN

TITANtitan.uio.no is the compute cluster at UiO with Sun X2200 dual-CPU quad-core nodes (AMD 2.3 Ghz). The nodes have 16 GB memory. Titan also includes three Sun X4600 32-core nodes (two with 128 GB memory, one with 256 GB memory). Total 2528 cores. The Notur share is part of a larger configuration that consists of ca. 5000 cores. InfiniBand node-interconnect. Operating system: Rocks Linux.

and Technology (NTNU)

Life Sciences (UMB)

(HiMOLDE)

and Business Administration (NHH)

Remote Sensing Center (NERSC)

Stavanger (IRIS)

Research (NILU)

Research (Nofima)

Cancer Research

Centers of Excellence

Research (BCCR)

(CBM)

Evolutionary Synthesis (CEES)

Neuroscience (CMBN)

Structures (CESOS)

Computational Chemistry (CTCC)

(PGP)

STALLOstallo.uit.no is the compute cluster at UiT with 704 HP BL 460c dual-CPU quad-core nodes (Xeon 2.66 Ghz). Total 5632 cores. The nodes have 16 GB memory (50 nodes have 32 GB memory). Approximately 60% of the compute nodes is interconnected with InfiniBand. Operating system: Rocks Linux

HEXAGONnodes (AMD 2.3 Ghz). Total 5552 cores. Each node has 4 GB memory (128 nodes have 8 GB memory). The nodes are interconnected with Cray’s SeaStar2 3D torus. Operating system: Linux.

High Performance Computing for New Discoveries

The Ultimate Answer to the Ultimate Question of Life, The Universe, and Everything. Made easy.*Hitchhiker’s Guide To The Galaxy

*

Researchers from the following organizations are using the Notur infrastructure:

Scientific areas using the Notur infrastructure:

The National Infrastructure for Computational Science

www.notur.no

Norstore OverviewAndreas O Jaunsen

UNINETT Sigma2

UNINETT Sigma2• Funded (5+5 yrs) by RCN, UiT, NTNU, UiB and UiO

• Coordinate e-infrastructures for science and participation in relevant activities internationally

• Procure and operate systems for high-performance computing and data storage

• Develop non-trivial end-user computing and data services

• Resource Allocation Committee allocates resources

10G10G

10G

10G

Forskningsnettet

Publish data

PreservationData access

Publish scientific results

Analysis

Data collection

Project proposal / plan

Metadata

Processing

Data creation

Prepare results

project area

data archive

data archive+

Publish data

Preservation Data access

Publish scientific results

Analysis

Data collection

Project proposal / plan

Metadata

Processing

Data creation

Prepare results

project area

data archive

data archive+

Resource Allocation• Two calls yearly for applications of resource

allocation (starting Apr 1 and Oct 1), computing and storage

• https://www.metacenter.no/mas/application/project

• RFK is appointed by Sigma2 and allocates the resources according to set guidelines

• Extra resources can be applied for at any time, albeit with allocation limits

Project area

• Provide users with storage resources (1TB - 1PB) on disk/tape, no automated backup

• High-speed data staging in development to facilitate efficient data transfer between Notur and Norstore (1TB/hour)

• Self-service administration: user initiated password reset and project managers administrate project members using the project management interface

Project administration interface

$ ssh [email protected]############################################################### ## _ __ ## _ _ ___ _ _ __| |_ ___ _ _ ___ ___ / \ ## | ' \/ _ \ '_(_-< _/ _ \ '_/ -_)___| () | ## |_||_\___/_| /__/\__\___/_| \___| \__/ ## ## Welcome to Norstore. ## ## projects are in /projects ## tape : see https://www.norstore.no/services/tape-storage ## ## News: for heavy data processing please use our dedicated ## server "cruncher.norstore" which is accessible ## from the login nodes. It is a server with 8 cpus ## and 12GB of RAM. ## ###############################################################-bash$ dusage -p ns9999k==========================================================================================Project Account Resource Type Usage Soft Limit Hard Limit NS9999K $PROJECT norstore_osl Disk 0.282TB 1TB 1TB NS9999K $PROJECT tapestore Tape 0.006TB NS9999K $PROJECT replica Tape 0.005TB NS9999K $PROJECT tape total Tape 0.012TB 1TB 1TB NS9999K $PROJECT norstore_osl Files 327 None None NS9999K $PROJECT tapestore Files 10 NS9999K $PROJECT replica Files 5 NS9999K $PROJECT tape total Files 15 100 100 ------------------------------------------------------------------------------------------

Service for sensitive data (TSD)

• Service developed and operated by UiO/USIT

• Supported & funded by NorStore since 2012

• Secure storage and handling of (person-identifiable) sensitive data – offered from July 1, 2013

• USIT offers dedicated computing cluster for sensitive data (colossus) in “secure” environment

Research Data Archive

• providing access to public-funded research data

• long-term (10 yrs to permanent) archive service

• federated authentication (Feide)

• web-based user interface

• requires that users document their data according to the Norstore meta-data schema within a grace-period of 3 months from date of upload

• DOI identifiers (using dedicated Norstore prefix doi:10.11582)

• offer NLOD + CCv4 license

• support web-based (limited) uploads and staging from project area

• link data to scientific publications

• data can not be modified or deleted

Generic and Community Services

Generic ServicesN

SDN

NO

RM

AP

Basic tools

Technology

Resources

ESC

G

elix

ir.no

Cla

rino

Nor

brai

n

•Data Archive•Project space

•DOI (PID)•Accounting/usage•iRODS

•HNAS, QStar•OpenStack SWIFT

•T-finity 950•HUS VM, Hitatchi

4 PB tape

/projects

Norstore Research Data Archive

0.5PB disk

3 PB disk

iRODS

iROD

S

UiO

UiT

datamigrering

Reporting• Reporting primarily consist of registering project ID

and infrastructure (Notur or Norstore) as funding source in CRISTin (https://www.norstore.no/access/report)

• A user survey is conducted yearly for all active users (project managers)

• It is essential that these requirements are obeyed as they are required to document the legitimate need for the infrastructure


Recommended