CCPR WorkshopIntroduction to the Cluster

July 13, 2006

Outline

What is a cluster? How can CCPR’s cluster help you? Using the cluster

• Connecting from Windows

• Unix commands

• Submitting jobs

• Managing jobs

What is a cluster?

Think of a cluster as having two parts:• Hardware: a collection of computers

• Cluster management system: software for coordinating hardware with a collection of computational tasks (batch jobs)

CCPR’s Cluster: Management System

Jobs are submitted to the cluster via the cluster management system• An advanced batch system

• Efficiently allocates resources to jobs

• Remembers job requirements

• Remembers node specs and availability

• Schedules jobs according to requested resources and priority

• Users can easily view jobs

How does this help your research?

Easy access to software and data Faster processors More RAM Easy to share data, programs, etc. with

colleagues via the cluster More advanced users: parallel processing,

compilers, etc. Your pc is available for other work when you

submit a job to the cluster Submit a job and forget about it

Using the Cluster

Connecting from Windows Unix Commands Submitting jobs Managing jobs

Secure Shell

Secure Shell software (2 pieces)1. SSH Secure Shell Client (Telnet)

2. SSH Secure File Transfer Client (FTP).

• Download software from

www.ssc.ucla.edu Computing Security

Secure Shell: Connecting to lexis from Windows

Connect to lexis via SSH Secure Shell Client

• Startup SSH Secure Shell/File Transfer

• Click Quick Connect

• Enter:• Host Name: lexis.ccpr.ucla.edu

• User Name: your lexis account name

• Port Number: 22

• Click connect and enter password Can add “profile” for repeated use Same process for SSH Secure File Transfer Client

Samba: Transferring files from pc to lexis

Analagous to “mapping a drive” Allows user to view lexis files via

Windows Explorer interface Must use vpn when you’re off campus.

Download from• http://www.bol.ucla.edu/services/vpn/

Step-by-step instructions:• http://lexis.ccpr.ucla.edu/using/samba/

Samba: Transferring files from pc to lexis Open Windows Explorer on the local machine. Select Tools->Map Network Drive In the "Folder" text box enter:

\\lexis.ccpr.ucla.edu\your-user-name Click "Reconnect at login" if you want the drive to be

mounted each time the computer starts up. Click the "different user name" link to enter your user

name and password for the connection. Click "OK" then click the "Finish" button on the Map

Network Drive window.

Unix commands

passwd Change lexis password

smbpasswd Change samba password

pwd Print working directory

cd Change directory

mv filename1 filename2 Rename/move filename1 to filename2

cp filename1 filename2 Copy filename1 to filename2

More Unix Commands

cat file Display contents of file to screen

cat file1 >> file2 Concatenate file1 and file2

more file Display contents of file to screen

ls List files

ls –lah List files in current directory (options l=LONG, a=ALL, h = HUMAN-READABLE

More Unix commands

du –h Disk usage (option h=HUMAN-READABLE)

.forward forward mail to another account

gzip Compress file

gunzip Expand compressed file

ln –s actual link Create symbolic link

which name search for name

Getting help in Unix

man command list help for command

man –k command

keyword search for command

whatis command brief description of command

apropos keyword

list commands with keyword in their description

Unix Commands: Using Aliases

Alias• Create an alias for commonly used

commands

• Example: alias ls=“ls –lah”• Anytime you type ls, you will see the ls command

with options lah (l=long, a=all, h=human-readable)

• Edit .aliases file to create and remove aliases

Unix commands: Pipes and grep

history displays a “history” of previously entered commands

history | more history, but one page at a time

history | grep cat sends output of history to commnd “grep cat”, which displays only lines containing “cat”

history | wc Counts number of newlines, words, and bytes contained in output of history

Editing Files on Lexis

Pico – easiest editor. Open and type. Commands listed at bottom of editor.

Vi – more advanced. You can move around quickly, do copy/paste, search, etc, but keystrokes required for everything.

Emacs – most advanced

Submitting Stata and SAS jobs Submit jobs by typing qjob, qstata, qsas,

qmatlab at Unix prompt You will then be prompted for:

• filename

• memory (number in megs, ex. 1050)

• if memory > 1 gig -> Over 2 hours (y/n),

• job type (for qjob only)

• submit (y/n) Alternatively, submit all options without being

prompted:• qstata sample.do 500 y

Files produced by qstata

filename.log - This is the log file from Stata. The output of your entire Stata job is caputured in this file.

filename.do.cmd - This is the Command file that is submitted to the cluster.

filename.do.joblog - This file will capture the output of your CMD file and any errors that may occur (Stata output is not captured by this log.)

Job Management

RAM Compute time requested

Job sent to…

<1 GB n/a Any node, no time constraints

>2 GB n/a >4-GB node

1-2 GB <2 hours “2-hour” node, terminates after 2 hours

1-2 GB >2 hours >4-GB node

Job Management

qstat or qstatus to view overall cluster usage

qstat –j jobnumber for information particular to your job

qdel jobnumber to kill your job

Working interactively - qinteractive

To work interactively, type “qinteractive” at prompt

Default RAM allotted is 256 • Request x megs by typing “qinteractive x”

Please use interactive nodes responsibly• Limited number of spaces

• Testing only

• Logoff promptly

Questions/Comments