Tool Building 1: Shell
CS 360
tool1
Page 2 tools1 CS 360, WSU Vancouver
What is Systems Programming? (again)
hardware
driversOS services
system programsapplications
interfaces
users
impl
emen
tatio
n la
yers
• implementing• building• Testing• Installing• deploying
System programs typically create environments
for applications
Page 3 tools1 CS 360, WSU Vancouver
Some Tools That System Programmers Create For a software engineering team:
organize versions of product sources package designs & sources for peer review build the product test product ....
For a web site operator: update site per new look and prices ensure file security in place ...
For a LAN administrator add a new hire to all address books install new software in all workstations ...
Such tools manipulatefiles and directories
Page 4 tools1 CS 360, WSU Vancouver
What Does the Shell Provide? Interactive interface to all system capabilities
navigate & manipulate file system invoke system commands & programs customize your environment
Programming environment for system programmers automate above interactions glue together smaller programs prototype solutions
Page 5 tools1 CS 360, WSU Vancouver
Software Structure of Unix, Again
Programs:executable files numerous & specific
Shell:command line environment few & powerful
users
shell
programs
unixlibraries
unixkernel
Page 6 tools1 CS 360, WSU Vancouver
Shell Examples Find words in a file
C program: 50+ lines of code Shell: 4 lines
Find most popular word C program: 75+ lines of code Shell: 2 lines
Outline a directory structure C program alone: 100+ lines of code Shell plus C program: 30 lines
Clone a directory structure C program: 150+ lines of code Shell: 1 line
Shell can reduce code bulk dramatically
Page 7 tools1 CS 360, WSU Vancouver
Use C and Shell Together
C programming = data structures + algorithms
Shell programming = file system + utilities
Page 8 tools1 CS 360, WSU Vancouver
The Shell is a Scripting Language
A professional uses both language types
Page 9 tools1 CS 360, WSU Vancouver
Which Shell?
Unix architecture allows arbitrary shells The most popular have evolved and become quite powerful Always avoid religious wars and become productive!
Bourne shell "SH"
C shell "CSH"
Korn shell "KSH"
GNU Born Again shell "BASH"
editors "EMACS" & "VI"
our choice
Page 10 tools1 CS 360, WSU Vancouver
Basic Shell Concepts
Command Lines Pipes Useful Utilities
Page 11 tools1 CS 360, WSU Vancouver
Canonical Shell Command Format
command [ -switch ] … [ argument ] ...
• file name: ../bin/doit•built-in verb: cd ..
• simple tokens: grep -e abc ../src/foo.c ../src/bar.c
•wildcards: grep -e abc ../src/*.c•quoted text: grep -e "abc xyz” ../src/*.c•shell variables grep -e “$text” ../src/*.c•quoted text: grep -e ‘$text’ ../src/*.c
your program doesn'tsee the quotes
or wildcardsor variables
• single letters: ls -a -1•combined single letters: ls -a1•values: cc -o foobar foobar.c•strange and wonderful: find . -print
Page 12 tools1 CS 360, WSU Vancouver
Wildcards Expand into a List of Pathnamesls ../src/*.c ls ../src/abc.c ../src/xyz.c
These are the most common pattern characters: * matches any string of characters, including none ? matches any single character [a-z] matches any character in range ~ your home directory ~bob Bob's home directory
Notes: invoked programs don't see the patterns
but, if no matches, pattern is passed to program * doesn't match names beginning with period when first in a pattern tilde interpreted only if first character in token
Page 13 tools1 CS 360, WSU Vancouver
Echo Command Writes To Stdout% echo hello therehello there% echo
% echo "this is one argument"this is one argument% echo "*"*% echo *abc def xyz% echo $PWD/home/langd% echo "$PWD"/home/langd% echo '$PWD'$PWD% x="some value"% echo "$x" $x '$x'some value some value $x
$"'
note wellthe rules for:
Page 14 tools1 CS 360, WSU Vancouver
Many Utilities are Filters (Text In/Text Out)
stdin stdoutprogram
Shell always provides your program with three open file descriptors: 0: "standard input" 1: "standard output" 2: "standard error" (= stdout usually)
Examples: translate lower to upper case: tr "a-z" "A-Z" find lines with "abc": grep -e "abc" delete lines with "abc": sed -e "/abc/d" extract lines starting with "xyz": sed -n -e "/^xyz/p" sort into ascending order: sort copy input to output: cat write to output: echo "hello, world!"
Page 15 tools1 CS 360, WSU Vancouver
Use Redirection to Control Stdout and Stdin Stdout:
Your terminal unless redirected:
Stdin: Your terminal unless redirected:
Stderr: Your terminal unless redirected:
% grep -e "abc" > my-output-file% grep -e "abc" >> my-output-file
% grep -e "abc" < my-input-file
% grep -e "abc" 2> my-error-file% grep -e "abc" 2> /dev/null% grep -e "abc" 2>&1% echo "error!" >&2
Page 16 tools1 CS 360, WSU Vancouver
Connect Filters into Pipelines
% tr " " "\n"
% tr " " "\n" | sort
% tr " " "\n" | sort | uniq
words
sorted words
unique words
tr " " "\n"stdin sort uniq stdout
Notes: thus stdout of one program is the stdin for the following program programs in pipe execute concurrently (no temp files used) can combine with redirection
% tr " " "\n" | sort | uniq < gettysburg > result
|
Page 17 tools1 CS 360, WSU Vancouver
Use Grep & Sed to Find or Change Lines Grep finds lines:
grep [ -e pattern ] … display lines in stdin that contain any of the patterns
Sed rewrites lines: sed [ -e action ] … copy stdin to stdout, doing all actions on each line
Some Sed actions: /pattern/d delete line if contains pattern /pattern/r file read (copy to stdout) file after the line the pattern
matches s/pattern/replace/g substitute replace for all occurrences of pattern
Grep & Sed patterns: . match any character [a-z] match any character from "a" to "z" [^a-z] match any character except from "a" to "z" ^ match beginning of line $ match end of line (how match a dollar sign?) pattern* match zero or more occurrences of pattern
(match is greedy -- try some experiments)
Page 18 tools1 CS 360, WSU Vancouver
Some Sed Examples Replace each occurrence of MYNAME with “Dick":
Delete all lines exactly 1 character long:
% sed -e "s/MYNAME/Dick/g"
% sed -e '/^.$/d'
why the single quotes?
Page 19 tools1 CS 360, WSU Vancouver
Use Scripts to Package Commands for Reuse Count number of files in directory "foo" that have "doug" in name:
A script named "hunt" to do the same thing:
A new version that accepts two parameters:
Consider these invocations of the script: new-hunt foo doug betty new-hunt foo new-hunt
% ls -1 foo | grep -e "doug" | wc -l3
ls -1 foo | grep -e "doug" | wc -l
ls -1 $1 | grep -e $2 | wc -l
% new-hunt foo "doug"3
(notice the quietness!)% hunt3
new-hunt
hunt
Page 20 tools1 CS 360, WSU Vancouver
Scripts & Paths Shell finds programs by looking for executable files in the path:
File is executable if has "x" mode:
A script is a text file that is executable:
The script is just a sequence of shell commands in sequence can have parameters supports if, while, and other programming constructs
% echo $PATH.:/home/langd/bin:/net/class/cs360/bin:/usr/bin
% ls -l my-program-rwxrwxr-x ~~~~~~~~~~ my-program
% vi my-script% chmod +x my-script
("whence foo"= where foowill be found)
Page 21 tools1 CS 360, WSU Vancouver
Use the Shell Utility Toolkit Basic line oriented processing:
grep find patterns sed edit lines cut extract columns sort sort lines into specified order uniq display unique lines in a sorted file tail, head display last lines of a file, or first lines wc count words, lines, and characters
File system manipulation: cp, mv, rm, ls copy, move, delete, list files chmod, chown change file access modes, file owner find navigate directory structure testing files for attributes touch change last-modified time on a file
Miscellaneous: echo echo arguments on stdout cat copy a file or files to stdout tr translate characters date echo current date and time man see manual page on a command
Scripting Languages awk process files based on rich pattern matching & text manipulation perl do the same with more C-like facilities
documentation = man pages
Page 22 tools1 CS 360, WSU Vancouver
Good Things to Know Editing last command:
Type ESC Then can edit recent commands using VI keystrokes (e.g.: k = go up, i = insert)
Using special keys: End terminal stdin with control-D Interrupt execution of a program with control-C
Continuing beyond one line: End with backslash to continue line Unbalanced quotes etc. also continue Prompt will change to indicate
% hunt foo "doug>
Page 23 tools1 CS 360, WSU Vancouver
Whiteboard Exercises
Examples of scripts and pipelinesI love Unix
Page 24 tools1 CS 360, WSU Vancouver
Some Word Scripts
This is a example docment, with some words.This is on line two (which follows line 1).This is on line three (which follows line 2.0).John siad, "Hello, world!".
test file "document1":
Note: following scripts are stored in the directory "/class/cs360/lab/tools1/words"
• extract words• find misspelled words• find most popular word
Page 25 tools1 CS 360, WSU Vancouver
Words0:
tr " " "\n"
% words0 < document1Thisisaexampledocment,withsomewords.Thisisonlinetwo(whichfollowsline1).Thisis... more follows ...
script:
operation:
goal:• display words
preparation:
% vi words1% chmod +x words0
words0
Page 26 tools1 CS 360, WSU Vancouver
Words1:
tr " " "\n" | sort | uniq
% words1 < document1"Hello,(which1).2.0).JohnThisadocment,examplefollowsislineonsiad,somethreetwowithwords.world!".
script:
operation:
goal:• display unique words
preparation:
% vi words1% chmod +x words1
words1
Page 27 tools1 CS 360, WSU Vancouver
Words2:
tr " " "\n" | \tr "A-Z" "a-z" | \sort | \uniq
% words2 < document1"hello,(which1).2.0).adocment,examplefollowsisjohnlineonsiad,somethisthreetwowithwords.world!".
script:
operation:
goal:• display unique words• treat upper case as lower case
preparation:
% vi words2% chmod +x words2
words2
Page 28 tools1 CS 360, WSU Vancouver
Words3:
tr "\$,;./!()'\"" " " | \tr -s " " "\n" | \tr "A-Z" "a-z" | \sort | \uniq
% words3 < document1
012adocmentexamplefollowshelloisjohnlineonsiadsomethisthreetwowhichwithwordsworld
script:
operation:
goal:• display unique words• treat upper case as lower case• treat punctuation as space• collapse spans of spaces
preparation:
% vi words3% chmod +x words3
words3
Page 29 tools1 CS 360, WSU Vancouver
Spell:
words3 > /tmp/$$
comm -23 /tmp/$$ /usr/share/dict/words
rm /tmp/$$
% spell < document1012docmentfollowsjohnsiadwords
script:
operation:
goal:• display words that are not in dictionary
preparation:
% vi spell% chmod +x spell
spell
Page 30 tools1 CS 360, WSU Vancouver
Popular
tr "\$,;./!()'\"" " " | \tr -s " " "\n" | \tr "A-Z" "a-z" | \sort | \uniq -c | \sort -nr | \head -1
script:
goal:• find words using words3 logic• sort words by number of occurrences• print most popular word
% popular < document14 lineoperation:
popular
Page 31 tools1 CS 360, WSU Vancouver
Clone a Directory Structure
cp -mr $1 $2
script:
goal:• copy file or directory recursively• preserve ownership, modes, and times
% clone /home/langd /backup
operation:
preparation:
% vi clone% chmod +x clone
clone
Page 32 tools1 CS 360, WSU Vancouver
Outline a Directory Structure
% find foo -type dfoofoo/binfoo/labfoo/lab/introfoo/lab/intro/my-directoryfoo/lab/dev1foo/lab/iofoo/demofoo/demo/afoo/demo/bfoo/hellofoo/examplefoo/example/onefoo/example/twofoo/example/two/three
"find" walks directories:
(lots of wonderful switches!)
% outline foofoo /bin /lab /intro /my-directory /dev1 /io /demo /a /b /hello /example /one /two /three
What we want:
We will create this using a small program that replaces text to left of slashes with blanks
Page 33 tools1 CS 360, WSU Vancouver
Outline a Directory Structure
find $1 -type d | unslash
script:% outline foofoo /bin /lab /intro /my-directory /dev1 /io /demo /a /b /hello /example /one /two /three
goal:• outline a directory structure
preparation:
% vi outline% chmod +x outline% vi unslash.c% cc -o unslash unslash.c
operation:
outline
Page 34 tools1 CS 360, WSU Vancouver
Implementation of Unslash Outline the logic:
read stdin lines– if not the first line:
overwrite with blanks all text to left of rightmost slash– write the line
return
Refine the logic: key variables:
– let "line" be an array that holds the line just read– let i index each character in line– let s index the rightmost slash or -1 if no slash
to do the overwriting:– set s by examining each character (left-to-right)– replace each line[0..s-1] with a blank
Translate logic into C next slide ...
0 1 2 3 4 5 6 7 8 9 10line before / a b / c / x y . z \0
s 5line after / x y . z \0
Page 35 tools1 CS 360, WSU Vancouver
Code for Unslash is Simple#include "lineio.h"#define LINEMAX 1000
int main (int argc, char *argv[]) {
char line[LINEMAX]; /* input line & work area */int n; /* number of line 0, 1, ... */int i; /* examine line[i] next */int s; /* line[s] is a slash or s == -1 */
n = 0;while (getline (line)) {
/* overwrite text, if not first line */if (n > 0) {
/* set s to index of rightmost slash or to -1 */i = 0; s = -1;while (line[i] != '\0') {
if (line[i] == '/') s = i;++i;
}
/* replace line[0..s-1] with blanks */i = 0;while (i < s) {
line[i] = ' ';++i;
}}
/* write line and increment n */putline (line);++n;
}}
note how we find rightmost
slash
unslash.c
Page 36 tools1 CS 360, WSU Vancouver
Discussion ... In this situation:
by using a Shell utility, we had an "almost" solution immediately by reformulating the problem as a filter, we could write a simple C program to
complete the solution
This is a general approach for tool building with the shell:
1. Use files and directories2. Use utilities whenever possible3. Fill gaps with small C programs4. Connect components with pipes5. Format data as text