Command Line Variations Selected Readings in Chapters 3, 5, 6

Project 1 kickoff Streams & Redirection: Sections 5.2, 5.3 Pipes and Tees: Sections 5.4, 5.5 Filenames & Wildcards: Section 3.1 Job Control: Section 5.10 Filter Commands: details in Chapter 6

cat, tr, head, tail, cut, paste, sort,

wc (word count)

Figure 5-4

Standard Streams

Standard I/O Standard Output (stdout) labelled (1) above

– default place to which programs write

Standard Input (stdin) labelled (0) above– default place from which programs read

Standard Error (stderr) labelled (2) above– default place where errors are reported

To demonstrate -- cat– Echoes everything you typed in with an <enter>– Quits when you press Ctrl-d at a new line -- (EOF)

Figure 5-6

Redirecting Standard Output

Redirecting Standard Output cat file1 file2 > file3

– concatenates file1 and file2 into file3– file3 is created if not there– Any pre-existing file3 is clobbered if there

cat file1 file2 >> file3– file3 is created if not there– file3 is appended to if it is there

cat > file3– file3 is created from whatever user provides from

standard input

Figure 5-7

Print who Output

Redirecting Standard Error

In bash (for shell scripts), standard error is redirected using 2> – cat myfile > yourfile 2> yourerrorfile

A better example that generates errors:ls –R / recursively list all files from rootyou will get many screens of output

Redirect output into a file:ls –R / > allfiles save output from aboveyou will get many “permission denied errors”more allfiles but you can look at listing in file now

Redirect errors to a second file:ls –R / > allfiles 2> errors errors go into file errors

If you don’t care about errorsls –R / > allfiles 2> /dev/null /dev/null is a ‘bit bucket’

where bits go to die!

Figure 5-5

Redirecting Standard Input

Redirecting Standard Input

terra[1] $ cat < oldfile > newfile A more useful example:

– terra[2] $ tr string1 string2 Read from standard input. Character n of string1 translated to

character n of string2. Results written to standard output.

– Example of use:terra[3] $ tr aeoiu eoiuaterra[4] $ tr a-z A-Z < file1 > file2

/dev/null

/dev/null– A virtual file that is always empty.– Copy things to here and they disappear.

cp myfile /dev/null mv myfile /dev/null

– Copy from here and get an empty file. cp /dev/null myfile

– Redirect error messages to this file cat myfile > yourfile 2> /dev/null Basically, all error messages are discarded.

Figure 6-1

Concept Of A Filter

Basic Filters Filters are programs that:

– Read stdin.– Modify it.– Write the results to stdout.

Filters typically do not need user input. Example:

– tr (translate): Read stdin Echo to stdout, translating some specified

characters Many filters can also take file names as

operands for input, instead of using stdin.

Filtering using tr (translate command) Using tr to translate what you type into upper case

tr [a-z] [A-Z] (enter, type, enter etc. ctrl-D to quit)

Using tr to translate a file using input redirection

tr [a-z] [A-Z] < foods (now foods will appear UC)

tr [a-z] [A-Z] foods (also OK, tr accepts filename args)

Using tr to translate a file using input redirection and store results using output redirection

tr [a-z] [A-Z] < foods > foodsCAP

tr [a-z] [A-Z] foods > foodsCAP (also OK, see above)

(now foodsCAP will hold UC output from foods)

More filters: grep, wc, sort (see Ch6) grep patternstr

– Read stdin and write lines with patternstr to stdout

terra[1] $ grep "unix is easy" < myfile1 > myfile2

terra[1] $ grep "unix is easy" myfile1 > myfile2 (also OK)– Write all lines of myfile1 containing phrase unix is easy

to myfile2 wc

– Count the number of chars/words/lines on stdin– Write the resulting statistics to stdout – Example: (wc < mars.txt OR wc mars.txt )

sort– Sort all the input lines in alphabetical order and write to

the standard output.

Figure 5-8

Piping Output to Next Command

Pipes The pipe:

– Connects stdout of one program with stdin of another

– General form:command1 | command2

– stdout of command1 used as stdin for command2– Example:

terra[1] $ cat foods | grep apples | wc -l A pipe works the same as (but more efficient than)

terra[2] $ cat foods > tmp1

terra[3] $ grep apples < tmp1 > tmp2

terra[4] $ wc -l < tmp2 Actually cat is redundant: grep apples foods | wc -l

Redirecting and Pipes (2) Note: The name of a command always comes

first on the line. There may be a tendency to say:

terra[1] $ foods > grep apples | wc -l – This is WRONG!!!– Your shell will go looking for a program named

foods To do it correctly, many alternatives!

terra[1] $ cat foods | grep apples | wc -l

terra[2] $ grep apples < foods | wc -l

terra[3] $ grep apples foods | wc -l

terra[4] $ grep -c apples foods

The tee Command (don’t worry about this) tee - replicate the standard output

– cat readme.txt | tee myfile

tee

stdin

stdout myfile

Unix Filename Rules Almost any character is valid in a file name

– all the punctuation and digits– the one exception is the / (slash) character– the following are not encouraged

? * [ ] “ ” ’ ( ) & : ; !– the following are not encouraged as the first

character - ~

– control characters are also allowed, but are not encouraged

UPPER and lower case letters are different– A.txt and a.txt are different files

Unix Filename Extensions No enforced extensions

– The following are all legal Unix file names a a. .a … a.b.c

Remember files beginning with dot are hidden– ls cannot see them, use ls -a

. and .. are reserved for current and parent directories

Unix Filename Extensions Even though Unix doesn't enforce extensions,

– “.” and an extension are still used for clarity .jpg for JPEG images .tex for LaTeX files .sh for shell scripts .txt for text files .mp3 for MP3’s

– some applications may enforce their own extensions Compilers look for these extensions by default

– .c means a C program file– .C or .cpp or .cc for C++ program files– .h for C or C++ header files– .o means an object file

Unix Executable Files Executable files usually have no extensions

– cannot execute file a.exe by just typing a– telling executable files from data files can be difficult

“file” commandUse: file filenameResult: print the type of the fileExample: terra[1] $ file ~/.bashrc

.bashrc: executable bash-shell script

Filenames and pathnames have limits on lengths– 1024 characters typically– these are pretty long (much better than MS-DOS

days and the 8.3 filenames)

Fixing Filename Mistakes It is very easy to get the wrong stuff into filenames

– Say you accidentally typed

terra[3] $ cp myfile -i– What if you type

terra[4] $ rm -i The shell thinks -i is an option, not a file Getting rid of these files can be painful

There is an easy way to fix this... – You simply type

terra[5] $ rm -- -i– Many commands use “--” to say there are no more options

Creates a filewith name -i

Filename Wildcarding (See section 3.1) Wildcarding is the use of “special” characters to represent

or match a sequence of other characters– a short sequence of characters can match a long one– a sequence may also match a large number of sequences

Often use wildcard characters to match filenames– filename substitution – generally known as “globbing”

Wildcard characters* matches a sequence of zero or more characters

– Example: a*.c* matches abc.c, abra.cpp,

? matches any single character– Example: a?.c matches ab.c, ax.c, but not abc.c

[...] matches any one character between the braces– Example: b[aei]tmatches bat, bet, or bit, not baet

Filename Wildcarding (2) Wildcard sequences can be combined

terra[6] $ mv a*.[ch] cfiles/ mv all files beginning with a and ending with .c or .h into

the directory cfilesterra[7] $ ls [abc]*.?

list files whose name begins with a, b, or c and ends with . (dot) followed by a single character

Wildcards do not cross "/" boundaries– Example: csnow*c does not match csnow/codec

Wildcards are expanded by the shell, and not by the program– Programmers of commands do not worry about searching

the directory tree for matching file names– The program just sees the list of files matched

Filename Wildcarding (3) Matching the dot

– A dot ( . ) at the beginning of a filename, or immediately following a /

must be matched explicitly.– Similar to the character /– Example:

terra[8] $ cat .c* As mentioned earlier, [....] matches any one of the

characters enclosed– Within “[...]”, a pair of characters separated by “-” matches

any character lexically between the two Example:

terra[9] $ ls [a-z]*

cat all files whose namesbegin with .c

lists all files beginningwith a character betweenASCII ‘a’ and ASCII ‘z’

Processes and Job Control

Figure 5-14

Job States

Foreground and Background (1)

Unix is a multi-tasking operating system– some of these tasks are being done by other

users logged in– some are being done by you in the

background e.g. watching for incoming mail

When you run a task (a Unix command, like ls or vi) it executes in the foreground of your shell– it has the “control" of your screen and

keyboard

Foreground and Background (2) If you still want to use your shell command line

terra[1] $ a_heavy_task & [1] 13607 terra[2] $

When you put a task in background– task keeps running, but you continue to work at the

shell in the foreground– if any output is done, it appears on your screen

immediately (can be confusing)– if input is required, process prints a message and

stops– when it is done, a message will be printed

Example ls –R / (again) Kick off a long command ls –R / > listall Suspend, Ctrl-Z Send to background bg

[1] ls –R / >listall& Oops, still get errors! Now you have to kill a background

job: kill %1 (or kill -9 %1)

[1] terminated Restart it the right way, in background, with err redirect:

ls –R / > listall 2> /dev/null &

Foreground and Background (3) Explicit background processes are needed

less often with windowing systems– Just go to another window and run the

command But explicit background processes are

used often ...– A command needs a long time, you do not

want to close that window by accident

– Run a job at the background and logout

– netscape& will open a new window, but leave the current shell window still available to use

A Simple Script We use the following shell script to illustrate job control Edit a file make_noise

terra[1] $ cat > make_noise#!/bin/shwhile truedo date sleep 1done^Dterra[2] $ chmod u+x make_noise

make_noise then is a shell script repeats to print the time for every second, until you terminate it using Ctrl-c.

Job Control – Suspending Jobs

csh, tcsh, and bash allow you to manage the running of different processes

Suspending jobs– the Ctrl-z special character stops the job

terra[1] $ make_noiseFri May 16 14:14:43 EDT 2003……^ZSuspendedterra[2] $ vi readme^Z

Job Control - Monitoring Jobs The "jobs" command shows which of your

jobs are running and/or stopped.

terra[3] $ jobs

[1] + Suspended make_noise

[2] + Suspended vi readme

Here there are two suspended processes, the make_noise and a vi process.

Job Control – Resuming Jobs Putting jobs back into the foreground:

– Use the "fg" command to move a job into the foreground.

terra[4] $ fg %2

– Puts job number 2 into the foreground.

– Works with either a background or stopped job.

Putting jobs into the background:terra[5] $ bg %1

Job Control – Killing Jobs Jobs can also be killed

– Use the Unix "kill" command

terra[6] $ kill %1

or if it won't die ...

terra[7] $ kill –9 %1

Jobs can be stopped and continuedterra[8] $ a_heavy_task &

terra[9] $ stop %1

terra[10] $ bg %1

Using ps (1) Jobs are really just a special case of Unix

processes ps can list your current processes

terra[11] $ ps

PID TT S TIME COMMAND

2312 pts/0 T 0:00 vi

2296 pts/0 R 0:00 tcsh

2313 pts/0 R 0:00 ps

Using ps (2) The ps command takes a number of

options -l gives you a long listing of what is going

on -u loginid tells you about loginid's

processes use man ps to see more options

kill pid kills the process pid– TERM signal will be sent to the process pid– kill -9 or kill -KILL will send the KILL signal– Use man –s 5 kill to find out more signals