Post on 03-Jan-2016
transcript
Katie AntypasUser Services GroupLawrence Berkeley National Lab
17 February 2012
JGI Training Series
Until all users are migrated to NERSC we plan to hold weekly Friday sessions
More on file and data management
Open Office Hours
Review of batch system policiesCrius
RheaTheia
Kronos?
Hyperion
Oceanus
Iapetus
Themis
Introduction to NIM
On NIM you can change your password, change your shell and set security questions
Login to nim.nersc.gov
Look under the actions menu to do the above tasks
File systems best practices
• Unfortunately disk is still expensive
• All of the JGI’s data can not be stored on disk within the current budget
• Archive and delete data you no longer need
• Disk usage will be controlled through quotas in some cases and purging in others
Only the “house” file system will be available on both JGI and NERSC systems initially
JGI Space NERSC SpaceCompute clusterSome submit hosts
Most web servers
NetappsNetapps “projectb”“projectb”
househouse
•If your data needs access to both servers in JGI space and the compute cluster, it MUST go into “house”•In other words – move data out of Netapps
But “house” is 90% full……
House 90%House 90%
File systems above 90% are lower performing and at higher risk of failure
We need your help deleting data from “house” and moving data from the netapps to “house”
NERSC has set up 2 fast “data transfer nodes”just for JGI users
Login to dtn03.nersc.gov or dtn04.nersc.govType >df to see all the mounted file systems Back up data to HPSS (you authenticated at last week’s training don’t remember? Type hsi and then enter your NIM password)
> cd /house/path/to/your/data> hsi put <filename>
Or archive an entire directory> htar –cvf tarname.tar directory/
There are two areas of storage within the “project” layout of the “projectb” file system
/projectb/
projectdirs/ scratch/
PI/ RD/ fungal/ metagenome/ micro/ plant/ comparative/ user/
• Group directories• Not purged• Subject to quota
• User directories• cd $SCRATCH •Purged, 12 weeks•1 TB, 500,000 inode quota
Request a projectb directory for your group through the Jira ticket system
Request a larger /scratch quota through the Jira ticket system
ssh phoebe.nersc.gov
Use the fast data transfer nodes to move data between file systems
Login to dtn03.nersc.gov or dtn04.nersc.govType >df to see all the mounted file systems You can move data to 3 file systems $HOME “project” “scratch”
> mv /old/path/filename /new/path/filename
It is important for every group to come up with a data retention policy
How long should we keep the raw data?
Can the data be deleted or should it be archived? Can we set up an
automated way to archive and delete data?
The JGI compute clusters have been consolidated into Crius with the following shares
Crius
RheaTheia
Kronos?
Hyperion
Oceanus
Iapetus
Themis
Users should submit jobs to the normal queue
Jobs running longer than 12 hours or requesting large amounts of memory could see longer wait times
Useful commands