Introduction to Unix/Linux
Ken-ichi Nomura, Ph.D.
Center for High-Performance Computing and Communications
Outline
1. HPCC Overview
2. Basic Commands and Utilities
3. Text Editor
4. Directory, Software and Account Issues
HPCC clusters: - HP SL230 (128GB memory), HP SL250
(64 GB memory, dual NVIDIA K20m GPUs), Dual Xeon 8-core 2.4 GHz, on 56.6-Gbit FDR Infiniband network
- 2,225-node, 4-core, 6-core, and 12-core dual-processor cluster contains Dell, Oracle Sun, HP, and IBM compute nodes on a 10-Gbit Myrinet network
- 4 large-memory nodes with 1 TB of RAM and 4x10-core Intel Xeon processors.
- 2 head nodes, hpc-login1.usc.edu and hpc-login2.usc.edu
HPCC Overview
Infiniband Myrinet
hpc-‐login1 hpc-‐login2
HPCC
Internet
HPCC website: http://hpcc.usc.edu/
HPCC Overview
- About - Research - User Support - New User
Guide - Frequently
Asked Questions
- Contact Us
HPCC Website: HPCC Policies
5
http://hpcc.usc.edu/support/accounts/hpcc-policies/
HPCC Website: Account Application
6
http://hpcc.usc.edu/support/accounts/applying-for-a-hpcc-account/
Machine Room
Outline
1. HPCC Overview
2. Basic Commands and Utilities
3. Text Editor
4. Directory, Software and Account Issues
What is Linux?
Command Line Interface
Graphical User Interface
Operating System (OS)
ls List file and/or directory names pwd Print working (current) directory mkdir/rmdir Create/remove directory cd Change directory chmod Change file or directory permission cp/mv/rm Copy/move/remove file or directory cat/more/less Display file contents chmod/chown change permission/ownership man Display online manual
Basic Commands
[~]$ ls docs hpc-23 Makefile R util workshop [~]$ ls –a … [~]$ pwd /home/rcf-12/knomura [~]$ mkdir tmp [~]$ cd tmp/ [tmp]$ cp ~/.bash_profile dot_bash_profile [tmp]$ cat dot_bash_profile [tmp]$ ls -l dot_bash_profile [tmp]$ chmod u-r dot_bash_profile [tmp]$ cat dot_bash_profile cat: dot_bash_profile: Permission denied [tmp]$ chmod u+r dot_bash_profile
Basic Commands: Example
[tmp]$ cp dot_bash_profile DOT_bash_profile [tmp]$ ls dot_bash_profile DOT_bash_profile [tmp]$ ls -l *_bash_profile -rw-r--r-- 1 knomura m-cacs 208 Oct 3 11:43 dot_bash_profile -rw-r--r-- 1 knomura m-cacs 208 Oct 3 11:47 DOT_bash_profile [tmp]$ mv dot_bash_profile Dot_Bash_Profile [tmp]$ ls –la … [~]$ cd .. [~]$ ls [~]$ pwd [~]$ rm -rv tmp/ removed `tmp/DOT_bash_profile' removed `tmp/Dot_Bash_Profile' removed directory: `tmp'
Basic Commands: Example
Shell (bash and tcsh) • Shell is a command processor by which users tell what to do to
system. • Command-line Interface (CLI) or Character user interface (CUI) • To check which shell you are using, type echo $0 • Auto logout after 20 minutes of inactivity
Bash Config Files • Configuration files to setup user environment, for
examples command prompt, path, alias so on. Sometimes called “dot file”
• .bash_profile and .bashrc are stored in each user’s home directory
• When bash is invoked as a login shell, it first reads /etc/profile, if that file exists, then looks for .bash_profile and .profile
• When bash is invoked as a non-login shell, only read .profile
Csh & Tcsh Config Files
• .login & .cshrc are in each user’s home directory
• When tcsh is invoked as a login shell, it reads first .tcshrc or, if .tcshrc is not found, .cshrc, then .history, then .login, and finally .cshdirs
• When tcsh is invoked as a non-login shell, only read /etc/csh.cshrc and .cshrc
- File and directory have ownership and permission - Three types of permission, readable, writeable and
executable - Each permission is given to three groups, owner, group and
others [~]$ ls -l .bash_profile -rw-r--r-- 1 knomura m-cacs 208 Sep 26 14:11 .bash_profile
r readable, w writable, x executable u user (owner), g group, o others, a all
Permission & Ownership
chmod Change file/directory permission chown Change file/directory owner
Example) chmod a+w file Add W permission to all users chmod o-x file Remove E permission from others chmod 750 file Add R/W/E permission to myself, R/E
permission to group no permission to others chown usr:grp file Change ownership of a file/directory
r(4) readable, w(2) writable, x(1) executable u(I) user (owner), g(II) group, o(III) others, a all
Permission & Ownership
Task 1: Change Permission and Run clock.sh
#!/bin/bash while true ; do echo -n " " date +"%r" sleep 1 done
1. Add executable permission to clock.sh
2. Type ./clock.sh 3. Terminate the script by Ctrl+c [~]$ ./clock.sh 02:21:02 PM 02:21:03 PM 02:21:04 PM
…
Useful Commands env display environmental variables echo/export display/set an environmental variable grep print lines matching a pattern head/tail show first/last several lines sort sort text ps/top display currently running jobs kill/killall terminate a process (not for PBS job) tar archive/unarchive files bc calculator
[~]$ env [~]$ echo $PATH [~]$ export PATH=$PATH:~/bin [~]$ env | grep PATH [~]$ env > env.txt [~]$ cat env.txt [~]$ head (or tail) env.txt [~]$ grep -n knomura env.txt [~]$ sort env.txt | grep -n knomura > env2.txt [~]$ ps -u knomura [~]$ ps -ef | grep knomura [~]$ top [~]$ kill 123456
Useful Commands: Example Archive multiple files/directories into one file, then compress it. [~]$ tar cvfz out.tar.gz file1 file2 dir1 Unzip and extract from an archived file. [~]$ tar xvfz output.tar.gz Display archived file contents [~]$ tar tvfz output.tar.gz
Special Characters/Commands ~ home directory . current directory .. parent directory * wild card > redirect output >> redirect output (append) <<, <<< here document/string | pipe `command` back quotes, replace `…` with
command output
tab Complete name of command, file or directory
clear/ctrl+l Clear terminal ctrl+a Go to the beginning of line ctrl+e Go to the end of line history Display command history éê Display command history ! Rerun a command in history
Special Characters/Commands
Outline
1. HPCC Overview
2. Basic Commands and Utilities
3. Text Editor
4. Directory, Software and Account Issues
Text Editor: GNU nano
http://www.nano-editor.org/
Arrow-keys Move cursor Enter Change line CTRL+a Move to the beginning of line CTRL+e Move to the end of line CTRL+v Move forward one page CTRL+y Move backward one page
Text Editor: GNU nano
CTRL+o Save file CTRL+w Search text CTRL+d Delete a character CTRL+k Remove a line CTRL+u Paste buffer CTRL+x Save data and exit
Text Editor: GNU nano
Task 2: bash scripting 1 #!/bin/bash 2 3 # function definition 4 function pi { 5 6 # max number of RNG. use this for normalization. 7 RANDMAX=32767 8 9 # reset the number dot within a circle 10 in=0 11 for ((n=0; n<${1}; n++)); do 12 x=${RANDOM} 13 y=${RANDOM} 14 z=`echo "(${x}^2 + ${y}^2)/(${RANDMAX}*${RANDMAX})" | bc` 15 16 [ $z -lt 1 ] && let in+=1; 17 done 18 }
pi.sh
Task 2: pi.sh 20 # read total number of trial, number of procs 21 echo -n "type total number of trial: " 22 read total 23 24 # call the PI estimator 25 pi ${total} 26 27 echo "total trial=${total} PI = `echo ${in}/${total}*4 | bc -l`"
1. Fix pi.sh with text editor 2. Try 10, 50, 100, 1000 for the total number of trail
to see how the result improves
Outline
1. HPCC Overview
2. Basic Commands and Utilities
3. Text Editor
4. Directory, Software and Account Issues
System Directories
/bin, /sbin executable files (commands) /etc system config files /home users home directories /lib, /lib64, /usr/lib64 library files /proc process information /tmp temporary space /usr/usc software repository
Directory and Quota
• Home directory /home/rcf-xx/username Each user has 1 GB of disk quota and 100,000 files of file quota • Project directory /home/rcf-proj/projectid Quota on project directory varies depending on each project If you need more space, request from project page. https://www-rcf.usc.edu/rcfdocs/hpcc/allocations/
Monitoring Your Quota: myquota
----------------------------------------- Disk Quota for knomura ID 55302 Used Soft Hard Files 8146 100000 101000 Bytes 839.56M 4.30G 4.30G ----------------------------------------- Disk Quota for /home/rcf-proj/hpcc ID 419 Used Soft Hard Files 400792 1000000 1100000 Bytes 236.94G 400.00G 402.00G -----------------------------------------
myquota shows the quota on your home & project directories.
• Program may crash due to the quota, being unable to write files in either home directory or project directory
• Pay attention to the file quota (max number of files) also.
Account, Balance & core-hour • To be able to submit your jobs to the HPCC cluster,
your account needs some balance (core-hour) • Whenever your job finishes (successfully or
unsuccessfully), your account is charged by the number of processors used times the wallclock time the job spent
Example) If you request 2 nodes with 4 processors per nodes for 2 hours (-l nodes=2:ppn=4,walltime=2:00:00), the total charge is 2x4x2 = 16 core-hour.
Monitoring Your Balance: mybalance mybalance shows the current balance of your account [~]$ mybalance -h Balance Name ----------- ------ 11106267.27 lc_an 20000.00 lc_kn1
- Job will not start if your balance is not enough. - Submit a request from HPCC website for additional
core-hour. - Make sure the ppn value matches with the number of
processors actually used. (Don’t use ppn=12 for single-core job!)
Software Repository: /usr/usc Many applications are available. Compilers: gnu, intel, pgi Numerical Libraries: mpich, openmpi, cuda, fftw, petsc Molecular Simulation: NAMD, gromacs, amber Quantum Chemistry: gaussian, schrodinger Numerical Environment: matlab, R, python
hpc-login1.usc.edu for 32-bit applications hpc-login2.usc.edu for 64-bit applications
Software Repository: /usr/usc How to use the installed software on HPCC: - Find application and version you want to use - Source setup script. There are two setup scripts, one for
bash the other for tcsh
Example) Matlab R2013a
[~]$ ls /usr/usc/matlab/ 2009a 2009b 2010b 2011a 2013a 2013b default Scripts [~]$ source /usr/usc/matlab/2013a/setup.sh [~]$ matlab -nodisplay
< M A T L A B (R) > Copyright 1984-2013 The MathWorks, Inc.
R2013a (8.1.0.604) 64-bit (glnxa64) February 15, 2013
>>
System Software vs Repo Software • Some software and libraries (e.g. gcc, python, fftw) come
with OS
• Although command name is the same, the system software and repo software are often different (versions, libraries, developers). Make sure that you use what you want to use
• which command shows the absolute path of a command
[~]$ which python /usr/bin/python [~]$ source /usr/usc/python/enthought/default/setup.sh [~]$ which python /usr/usc/python/enthought/default/bin/python