Korn Shell (ksh)ProgrammingTutorialPhilip Brown

This material is copyrighted by Philip Brown, © January 2002-2009

1

Contents1. Introduction......................................................................................................................................... 4

1.1. Why scripting?.............................................................................................................................. 4

1.2. What can you do with ksh? .......................................................................................................... 4

1.3. Why ksh, not XYZsh for programming?........................................................................................ 5

2. Ksh preparedness ................................................................................................................................ 5

2.1. Step 0. Learn how to type. ........................................................................................................... 5

2.2. Step 1. Learn how to use a text editor. ........................................................................................ 5

2.3. Step 2. Understand variables. ...................................................................................................... 6

2.4. Step 3. Put everything in appropriate variables........................................................................... 6

2.5. Step 4. Know your quotes ............................................................................................................ 7

3. Ksh basics............................................................................................................................................. 8

3.1. Conditionals.................................................................................................................................. 8

3.1.1. IF ............................................................................................................................................ 8

3.1.2. CASE....................................................................................................................................... 8

3.2. Loops ............................................................................................................................................ 9

3.2.1. WHILE .................................................................................................................................... 9

3.2.2. UNTIL ..................................................................................................................................... 9

3.2.3. FOR ........................................................................................................................................ 9

4. Advanced variable usage................................................................................................................... 10

4.1. Braces ......................................................................................................................................... 10

4.2. Arrays.......................................................................................................................................... 10

4.3. Special variables ......................................................................................................................... 11

4.4. Tweaks with variables ................................................................................................................ 11

5. Ksh and POSIX utilities....................................................................................................................... 12

5.1. cut............................................................................................................................................... 12

5.2. join.............................................................................................................................................. 12

5.3. comm.......................................................................................................................................... 12

5.4. fmt .............................................................................................................................................. 13

5.5. grep and egrep ........................................................................................................................... 13

5.6. sed .............................................................................................................................................. 13

5.7. awk ............................................................................................................................................. 14

6. Ksh Functions..................................................................................................................................... 14

6.1. Why are functions critical?......................................................................................................... 14

2

6.2. A trivial function ......................................................................................................................... 15

6.3. Debugging your functions .......................................................................................................... 15

6.4. CRITICAL ISSUE: exit vs return .................................................................................................... 15

6.5. CRITICAL ISSUE: "scope" for function variables! ........................................................................ 16

6.6. Write Comments! ....................................................................................................................... 17

7. Ksh built-in functions......................................................................................................................... 17

7.1. Read and Set............................................................................................................................... 18

7.2. The test function ........................................................................................................................ 18

7.3. Built-in math............................................................................................................................... 18

8. Redirection and Pipes........................................................................................................................ 19

8.1. Redirection ................................................................................................................................. 19

8.2. Inline redirection ........................................................................................................................ 20

9. Pipes .................................................................................................................................................. 21

9.1. Combining pipes and redirection ............................................................................................... 21

9.2. Indirect redirection (Inline files)................................................................................................. 21

10. Other Stuff....................................................................................................................................... 22

10.1. eval ........................................................................................................................................... 22

10.2. Backticks ................................................................................................................................... 22

10.3. Text positioning/color games................................................................................................... 23

10.4. Number-based menus.............................................................................................................. 23

10.5. Raw TCP access......................................................................................................................... 24

10.6. Graphics and ksh ...................................................................................................................... 24

11. Paranoia, and good programming practices ................................................................................... 24

11.1. Comment your code................................................................................................................. 25

11.2. INDENT! .................................................................................................................................... 25

11.3. Error checking........................................................................................................................... 26

11.4. cron job paranoia ..................................................................................................................... 27

12. Example of script development....................................................................................................... 28

12.1. The good, the bad, and the ugly............................................................................................... 28

12.2. The newbie progammer version .............................................................................................. 28

12.3. The sysadmin-in-training version ............................................................................................. 29

12.4. The Senior Admin version ........................................................................................................ 29

12.5. The Senior Systems Programmer version ................................................................................ 30

13. Summary of positive features ......................................................................................................... 33

3

4

1. IntroductionThis is the top level of my "Intro to Korn shell programming" tree. Korn shell is a 'shell-scripting' language, as well as a user-level login shell. It is also a superset of a POSIX.1compliant shell, which is great for ensuring portability.

1.1. Why scripting?

Scripting, when done right, is a fast, easy way to "get the job done", without the usual"code,compile,test,debug" overhead of writing in C or some other compiled language. It iseasier than C for multiple reasons:

1. Scripting commands tend to be more readable than low-level code. (with the exceptionof perl)

2. Scriping languages tend to come with powerful tools attached3. There is no "compile" phase, so "tweaking" can be done rapidly.

UNIX tends to take #2 to extremes, since it comes standard with "powerful tools" that can bestrung together with pipes or other mechanisms, to get the result you want, with a shortdevelopment time. It may not be as efficient as a fully compiled solution, but quite often it can"get the job done" in a few seconds of run time, compared to 1 second or less for a compiledprogram.

A quick scripting solution can be used as a prototype. Then, when you have been using theprototype happily for a while, and you have evolved the behaviour your end users are happiestwith, you can go back and code a faster, more robust solution in a lower-level language.

That is not to say that scripts cannot be robust! It is possible to do a great deal of errorchecking in scripts. Unfortunately, it is not common practice to do so.

1.2. What can you do with ksh?

A heck of a lot!

You have access to the full range of UNIX utilities, plus some nifty built-in resources.

Generally speaking, UNIX scripting is a matter of using the various command line utilities asappropriate, with the shell as a means of facilitating communication between each step.

Unfortunately, running all these separate 'external' programs can sometimes result in thingsworking rather slowly. Which is why ksh has a few more things "built in" to it than the older'sh'.

5

1.3. Why ksh, not XYZsh for programming?

Bourne shell has been THE "standard" UNIX shellscript language for many years. However,it does lack some things that would be very useful, at a basic level. Some of those things wereadded to C-shell (csh) later on. However, csh is undesirable to use in programming, forvarious reasons.

Happily, ksh adds most of the things that people say csh has, but sh does not. So much so, thatksh became the basis for the "POSIX shell". Which means that all properly POSIX-compliantsystems MUST HAVE something compatible, even though it is now named "sh" like the oldBourne shell. (For example, /usr/xpg4/bin/sh, is a LINK to /bin/ksh, on solaris!) Moreprecisely, the behaviour of a POSIX-compliant shell is specified in "IEEE POSIX 1003.2"

BTW: "Korn shell" was written by "David Korn", around 1982 at AT&T labs. You can nowfreely download the full source from AT&T if you're not lucky enough to use an OS thatcomes with ksh already. It's "open source", even!

2. Ksh preparednessHowdy, and welcome to the intro to Korn shell programming, AKA POSIX.1 shell.

This is the first part of the larger tutorial. It assumes you want to learn how to really be aprogrammer in ksh, as opposed to someone who just quickly throws something together in afile and stops as soon as it works.

This particular chapter assumes you have done minimal or no sh programming before, so hasa lot more general stuff. Here are the most important things to know and do, before reallygetting serious about shellscripting.

2.1. Step 0. Learn how to type.

No, I mean type PROPERLY, and unconciously. You should be able to type faster than youcan write, and not be sweating while doing so. The reason for this is that good programmingpractices involve extra typing. You won't follow the best practices, if you are too fussed upabout "How can I use as few characters as possible to get all this typing done?"

2.2. Step 1. Learn how to use a text editor.

No, not microsoft word, or word perfect. Those are "word processors". I'm talking aboutsomething small, fast, and quick, that makes shoving pure text around easy. [Note that emacsdoes not fit either 'small', 'fast', OR 'quick' :-)]

You will have to be very comfortable with your choice of text editor, because thats how youmake shellscripts. All examples given should be put into some file. You can then run it with"ksh file".

6

Or, do the more official way; Put the directions below, exactly as-is, into a file, and follow thedirections in it.

#!/bin/ksh# the above must always be the first line. But generally, lines# starting with '#' are comments. They dont do anything.# This is the only time I will put in the '#!/bin/ksh' bit. But# EVERY EXAMPLE NEEDS IT, unless you want to run the examples with# 'ksh filename' every time.## If for some odd reason, you dont have ksh in /bin/ksh, change# the path above, as appropriate.## Then do 'chmod 0755 name-of-this-file'. After that,# you will be able to use the filename directly like a command

echo Yeup, you got the script to work!

2.3. Step 2. Understand variables.

Hopefully, you already understand the concept of a variable. It is a place you can store a valueto, and then do operations on "whatever is in this place",vs the value directly.

In shellscripts, a variable can contain a collection of letters and/or numbers [aka a 'string'] , aswell as pure numbers.

You set a variable by using

variablename="some string here"OR

variablename=1234You access what is IN a variable, by putting a dollar-sign in front of it.echo $variablename

ORecho ${variablename}If you have JUST a number in a variable, you can do math operations on it. But that comeslater on in this tutorial.

2.4. Step 3. Put everything in appropriate variables

Well, okay, not EVERYTHING :-) But properly named variables make the script more easilyreadable. There isn't really a 'simple' example for this, since it is only "obvious" in largescripts. So either just take my word for it, or stop reading and go somewhere else now!

An example of "proper" variable naming practice:

#Okay, this script doesnt do anything useful, it is just for demo purposes.# and normally, I would put in more safety checks, but this is a quickie.INPUTFILE="$1"USERLIST="$2"OUTPUTFILE="$3"

7

count=0

while read username ; dogrep $username $USERLIST >>$OUTPUTFILEcount=$(($count+1))

done < $INPUTFILEecho user count is $count

While the script may not be totally readable to you yet, I think you'll agree it is a LOT clearerthan the following;

i=0while read line ; do

grep $line $2 >> $3i=$(($i+1))

done <$1echo $i

Note that '$1' means the first argument to your script.'$*' means "all the arguments together'$#' means "how many arguments are there?"

2.5. Step 4. Know your quotes

It is very important to know when, and what type, of quotes to use.Quotes are generally used to group things together into a single entity.

Single-quotes are literal quotes.Double-quotes can have their contents expanded

echo "$PWD"prints out your current directoryecho '$PWD'

prints out the string $PWDecho $PWDplusthis

prints out NOTHING. There is no such variable "PWDplusthisecho "$PWD"plusthis

prints out your current directory, and the string "plusthis" immediately following it. Youcould also accomplish this with the alternate form of using variables,echo ${PWD}plusthis

There is also what is sometimes called the `back quote`, or ` backtick`: This is not used toquote things, but actually to evaluate and run commands.

8

3. Ksh basicsThis is a quickie page to run through basic "program flow control" commands, if you arecompletely new to shell programming. The basic ways to shape a program, are loops, andconditionals. Conditionals say "run this command, IF some condition is true". Loops say"repeat these commands" (usually, until some condition is met, and then you stop repeating).

3.1. Conditionals3.1.1. IF

The basic type of condition is "if".if [ $? -eq 0 ] ; then

print we are okayelse

print something failedfiIF the variable $? is equal to 0, THEN print out a message. Otherwise (else), print out adifferent message. FYI, "$?" checks the exit status of the last command run.

The final 'fi' is required. This is to allow you to group multiple things together. You can havemultiple things between if and else, or between else and fi, or both.You can even skip the 'else' altogether, if you dont need an alternate case.

if [ $? -eq 0 ] ; thenprint we are okayprint We can do as much as we like here

fi

3.1.2. CASE

The case statement functions like 'switch' in some other languages. Given a particular variable,jump to a particular set of commands, based on the value of that variable.

While the syntax is similar to C on the surface, there are some major differences;

The variable being checked can be a string, not just a number There is no "fall through" with ;;. You hit only one set of commands.. UNLESS you

use ";&" instead of ";;' To make up for no 'fall through', you can 'share' variable states You can use WILDCARDS to match strings

echo input yes or noread answercase $answer in

yes|Yes|y)echo got a positive answer# the following ';;' is mandatory for every set# of comparative xxx) that you do;;

no)

9

echo got a 'no';;

q*|Q*)#assume the user wants to quitexit;;

*)echo This is the default clause. we are not sure why orecho what someone would be typing, but we could takeecho action on it here;;

esac

3.2. Loops3.2.1. WHILE

The basic loop is the 'while' loop; "while" something is true, keep looping.

There are two ways to stop the loop. The obvious way is when the 'something' is no longertrue. The other way is with a 'break' command.

keeplooping=1;while [[ $keeplooping -eq 1 ]] ; do

read quitnowif [[ "$quitnow" = "yes" ]] ; then

keeplooping=0fiif [[ "$quitnow" = "q" ]] ; then

break;fi

done

3.2.2. UNTIL

The other kind of loop in ksh, is 'until'. The difference between them is that 'while' implieslooping while something remains true.'until', implies looping until something false, becomes trueuntil [[ $stopnow -eq 1 ]] ; do

echo just run this oncestopnow=1;echo we should not be here again.

done

3.2.3. FOR

A "for loop", is a "limited loop". It loops a specific number of times, to match a specificnumber of items. Once you start the loop, the number of times you will repeat is fixed.

The basic syntax is

10

for var in one two three ; doecho $var

doneWhatever name you put in place of 'var', will be updated by each value following "in". So theabove loop will print outonetwothreeBut you can also have variables defining the item list. They will be checked ONLY ONCE,when you start the loop.list="one two three"for var in $list ; do

echo $var# Note: Changing this does NOT affect the loop itemslist="nolist"

doneThe two things to note are:

1. It stills prints out "one" "two" "three"2. Do NOT quote "$list", if you want the 'for' command to use multiple items

If you used "$list" in the 'for' line, it would print out a SINGLE LINE, "one two three"

4. Advanced variable usage4.1. Braces

Sometimes, you want to immediately follow a variable with a string. This can cause issues ifyou use the typical grammar of "echo $a" to use a variable, since ksh by default, attempts todo "intelligent" parsing of variable names, but it cannot read your mind.Compare the difference in output in the following lines:two=2print one$twothreeprint one${two}threeThere is no variable named "twothree", so ksh defaults it to an empty value, for the first printline. However, when you use braces to explicitly show ksh {this is the variable name}, itunderstands that you want the variable named "two", to be expanded in the middle of thoseother letters.

4.2. Arrays

Yes, you CAN have arrays in ksh, unlike old bourne shell. The syntax is as follows:

# This is an OPTIONAL way to quickly null out prior valuesset -A array#array[1]="one"array[2]="two"array[3]="three"three=3

print ${array[1]}

11

print ${array[2]}print ${array[3]}print ${array[three]}

4.3. Special variables

There are some "special" variables that ksh itself gives values to. Here are the ones I findinteresting

PWD - always the current directory RANDOM - a different number every time you access it $$ - the current process id (of the script, not the user's shell) PPID - the "parent process"s ID. (BUT NOT ALWAYS, FOR FUNCTIONS) $? - exit status of last command run by the script PS1 - your "prompt". "PS1='$PWD:> '" is interesting. $1 to $9 - arguments 1 to 9 passed to your script or function (you can actually have

higher, but you need to use braces for those)

4.4. Tweaks with variables

Both bourne shell and KSH have lots of strange little tweaks you can do with the ${} operator.The ones I like are below.

To give a default value if and ONLY if a variable is not already set, use this construct:

APP_DIR=${APP_DIR:-/usr/local/bin}

(KSH only)You can also get funky, by running an actual command to generate the value. For example

DATESTRING=${DATESTRING:-$(date)}

(KSH only)To count the number of characters contained in a variable string, use ${#varname}.

echo num of chars in stringvar is ${#stringvar}

12

5. Ksh and POSIX utilitiesPOSIX.1 (or is it POSIX.2?) compliant systems (eg: most current versions of UNIX) comewith certain incredibly useful utilities. The short list is:

cut, join, comm, fmt, grep, egrep, sed, awk

Any of these commands (and many others) can be used within your shellscripts to manupulatedata.

Some of these are programming languages themselves. Sed is fairly complex, and AWK isactually its own mini-programming language. So I'll just skim over some basic hints andtricks.

5.1. cut

"cut" is a small, lean version of what most people use awk for. It will "cut" a file up intocolumns, and particularly, only the columns you specify. Its drawbacks are:

1. It is picky about argument order. You MUST use the -d argument before the -fargument

2. It defaults to a tab, SPECIFICALLY, as its delimiter of columns.

The first one is just irritating. The second one is a major drawback, if you want to be flexibleabout files. This is the reason why AWK is used more, even for this trivial type of operator:Awk defaults to letting ANY whitespace define columns.

5.2. join

join is similar to a "database join" command, except it works with files. If you have two files,both with information sorted by username, you can "join" them in one file, IF and ONLY IFthey are also sorted by that same field. For examplejohn_s John Smithin one file, andjohn_s 1234 marlebone rdwill be joined to make a single line,john_s John Smith 1234 marlebone rd

If the files do not already have a common field, you could either use the paste utility to jointhe two files, or give each file line numbers before joining them, with

cat -n file1 >file1.numbered

5.3. comm

I think of "comm" as being short for "compare", in a way. But technically, it stands for"common lines". First, run any two files through "sort". Then you can run 'comm file1 file2' totell you which lines are ONLY in file1, or ONLY in file2, or both. Or any combination.

13

For example

comm -1 file1 file2means "Do not show me lines ONLY in file1." Which is the same thing as saying "Show melines that are ONLY in file2", and also "Show me lines that are in BOTH file1 and file2".

5.4. fmt

fmt is a simple command that takes some informational text file, and word-wraps it nicely tofit within the confines of a fixed-width terminal. Okay, it isn't so useful in shellscripts, but itscool enough I just wanted to mention it :-)

pr is similarly useful. But where fmt was more oriented towards paragaphs, pr is morespecifically toward page-by-page formatting.

5.5. grep and egrep

These are two commands that have more depth to them than is presented here. But generallyspeaking, you can use them to find specific lines in a file (or files) that have information youcare about.One of the more obvious uses of them is to find out user information. For example,grep joeuser /etc/passwdwill give you the line in the passwd file that is about account 'joeuser'. If you are suitableparanoid, you would actually usegrep '^joeuser:' /etc/passwdto make sure it did not accidentally pick up information about 'joeuser2' as well.

(Note: this is just an example: often, awk is more suitable than grep, for /etc/passwd fiddling)

5.6. sed

Sed actually has multiple uses, but its simplest use is "substitute this string, where you see thatstring". The syntax for this is

sed 's/oldstring/newstring/'

This will look at every line of input, and change the FIRST instance of "oldstring" to"newstring".

If you want it to change EVERY instance on a line, you must use the 'global' modifier at theend:

sed 's/oldstring/newstring/g'

If you want to substitute either an oldstring or a newstring that has slashes in it, you can use adifferent separator character:

sed 's:/old/path:/new/path:'

14

5.7. awk

Awk really deserves its own tutorial, since it is its own mini-language. And, it has one!

But if you dont have time to look through it, the most common use for AWK is to print outspecific columns of a file. You can specify what character separates columns. The default is'whitespace' (space, or TAB). But the cannonical example, is "How do I print out the first andfifth columns/fields of the password file?"

awk -F: '{print $1,$5}' /etc/passwd

"-F:" defines the "field separator" to be ':'

The bit between single-quotes is a mini-program that awk interprets. You can tell awkfilename(s), after you tell it what program to run. OR you can use it in a pipe.

You must use single-quotes for the mini-program, to avoid $1 being expanded by the shellitself. In this case, you want awk to literally see '$1'

"$x" means the 'x'th columnThe comma is a quick way to say "put a space here".If you instead did

awk -F: '{print $1 $5}' /etc/passwd

awk would not put any space between the columns!

If you are interested in learning more about AWK, read my AWK tutorial

6. Ksh FunctionsFunctions are the key to writing just about ANY program that is longer than a page or so oftext. Other languages may call functions something else. But essentially, its all a matter ofbreaking up a large program, into smaller, managable chunks. Ideally, functions are sort oflike 'objects' for program flow. You pick a part of your program that is pretty much self-contained, and make it into its own 'function'

6.1. Why are functions critical?

Properly written functions can exist by themselves, and affect only small things external tothemselves. You should DOCUMENT what things it changes external to itself. Then you canlook very carefully just at the function, and determine whether it actually does what you thinkit does :-)

15

When your program isn't working properly(WHEN, not if), you can then put in little debugnotes to yourself in the approximate section you think is broken. If you suspect a function isnot working, then all you have to verify is

Is the INPUT to the function correct? Is the OUTPUT from the function correct?

Once you have done that, you then know the entire function is correct, for that particular setof input(s), and you can look for errors elsewhere.

6.2. A trivial function

printmessage() {echo "Hello, this is the printmessage function"

}

printmessage

The first part, from the first "printmessage()" all the way through the final '}', is the functiondefinition. It only defines what the function does, when you decide to call it. It does not DOanything, until you actually say "I want to call this function now".

You call a function in ksh, by pretending it is a regular command, as shown above. Just havethe function name as the first part of your line. Or any other place commands go. For example,

echo The message is: `printmessage`

Remember: Just like its own separate shellscript. Which means if you access "$1" in afunction, it is the first argument passed in to the function, not the shellscript.

6.3. Debugging your functions

If you are really really having difficulties, it should be easy to copy the entire function intoanother file, and test it separately from the main program.

This same type of modularity can be achived by making separate script files, instead offunctions. In some ways, that is almost preferable, because it is then easier to test each part byitself. But functions run much faster than separate shellscripts.

A nice way to start a large project is to start with multiple, separate shellscripts, but thenencapsulate them into functions in your main script, once you are happy with how they work.

6.4. CRITICAL ISSUE: exit vs return

THE main difference when changing between shellscripts and functions, is the use of "exit".

16

'exit' will exit the entire script, whether it is in a function or not.'return' will just quit the function. Like 'exit', however, it can return the default "sucess" valueof 0, or any number from 1-255 that you specify. You can then check the return value of afunction, just in the same way you can check the return value of an external program, with the$? variable.

# This is just a dummy script. It does not DO anything

fatal(){echo FATAL ERROR# This will quit the 'fatal' function, and the entire script that# it is in!exit

}

lessthanfour(){if [[ "$1" = "" ]] ; then echo "hey, give me an argument" ; return

1; fi

# we should use 'else' here, but this is just a demonstrationif [[ $1 -lt 4 ]] ; then

echo Argument is less than 4# We are DONE with this function. Dont do anything else in# here. But the shellscript will continue at the callerreturn

fi

echo Argument is equal to or GREATER than 4echo We could do other stuff if we wanted to now

}

echo note that the above functions are not even called. They are justecho defined

A bare "return" in a shellscript is an error. It can only be used inside a function.

6.5. CRITICAL ISSUE: "scope" for function variables!

Be warned: Functions act almost just like external scripts... except that by default, allvariables are SHARED between the same ksh process! If you change a variable name inside afunction.... that variable's value will still be changed after you have left the function!! Run thisscript to see what I mean.#!/bin/sh# Acts the same with /bin/sh, or /bin/ksh, or /bin/bashsubfunc(){

echo sub: var starts as $varvar=2echo sub: var is now $var

}var=1echo var starts as $var, before calling function '"subfunc"'subfunc # calls the functionecho var after function is now $varTo avoid this behaviour, and give what is known as "local scope" to a variable, you can usethe typeset command, to define a variable as local to the function.#!/bin/ksh

17

# You must use a modern sh like /bin/ksh, or /bin/bash for thissubfunc(){

typeset varecho sub: var starts as $var '(empty)'var=2echo sub: var is now $var

}var=1echo var starts as $var, before calling function '"subfunc"'subfunc # calls the functionecho var after function is now $varAnother exception to this is if you call a function in the 'background', or as part of a pipe (likeecho val | function )This makes the function be called in a separate ksh process, which cannot dynamically sharevariables back to the parent shell. Another way that this happens, is if you use backticks tocall the function. This treats the function like an external call, and forks a new shell. Thismeans the variable from the parent will not be updated. Eg:

func() {newval=$(($1 + 1))echo $newvalecho in func: newval ends as $newval

}newval=1echo newval in main is $newvaloutput=`func $newval`func $newvalecho output is : $outputecho newval finishes in main as $newval

6.6. Write Comments!

Lastly, as mentioned in the good practices chapter, dont forget to comment your functions!While shellscripts are generally easier to read than most programming languages, you reallycan't beat actual human language to explain what a function is doing!

7. Ksh built-in functionsCalling the many UNIX programs in /bin and elsewhere can be very useful. The onedrawback is speed. Every time you call a separate program, there is a lot of overhead instarting it up. So the conciencious programmer always tries to use built-in functions overexternal ones. In particular, ksh programmers should always try use '[[ ]]' over '[ ]', exceptwhere [] is neccessary

The more useful functions in ksh I find are:

'read' and 'set' functions built-in 'test': [[ ]]

(But this is apparently NOT a part of POSIX!!) built-in math : $(( )) built-in 'typeset'

18

See the manpages for 'test' and 'typeset', if you want full info on those beasties.

7.1. Read and Setread varname

will set the variable varname to have the value of the next line read in from standard input.

What often comes next, is

set $varnameThis sets the argument variables $1, $2, etc to be set as if the program were called with$varname as the argument string to the shellscript. So, if the value of varname is "first secondthird", then $1="first", $2="second", and $3="third".

Note that if you want to access "double-digit" arguments, you cannot use "$10". it will getinterpreted as "$1,""0". To access argument #10 and higher you must explicitly define thelimits of the variable string, with braces:

echo ${10}This is also good to know, if you wish to follow a variable immediately followed by a string.Compare the output from the following lines:a="A "echo $astringecho ${a}string

7.2. The test function

In brief, 'test' can let you check the status of files OR string values.Here are the most common uses for it.if [[ $? -ne 0 ]] ; then echo status is bad ; fiif [[ "$var" != "good" ]] ; then echo var is not good ; fiif [[ ! -f /file/name ]] ; then echo /file/name is not there ; fiif [[ ! -d /dir/name ]] ; then echo /dir/name is not a directory ; fi

Please note that [[]] is a special built-in version of test, that is almost, but not 100%, like thestandard []. The main difference being that wildcard expansion does not work within [[]].

7.3. Built-in math

The math evaluator is very useful. Everything inside the double-parens gets evaluated withbasic math functions. For example;

four=$((2 + 2))eight=$(($four + 4))print $(($four * $eight))

Warning: Some versions of ksh allow you to use floating point with $(()). Most do NOT.

Also, be wary of assumptions. Being "built in" is not always faster than an external progam.For example, it is trivial to write a shell-only equivalent of the trivial awk usage, "awk '{print$2}'", to print the second column. However, compare them on a long file:

19

# function to emulate awk '{print $2}'sh_awk(){

while read one two three ; doprint $twodone

}

# and now, compare the speed of the two methods

time sh_awk </usr/dict/words >/dev/nulltime awk '{print $2}' </usr/dict/words >/dev/null

The awk version will be much much faster. This is because ksh scripts are interpreted, eachand every time it executes a line. AWK, however, loads up its programming in one go, andfigures out what it is doing ONE TIME. Once that overhead has been put aside, it then canrepeat its instructions very fast.

8. Redirection and PipesThere are lots of strange and interesting ways to connect utilities together. Most of these youhave probably already seen.

The standard redirect to file;

ls > /tmp/listing

and piping output from one command to another

ls | wc -l

But bourne-shell derivatives give you even more power than that.

Most properly written programs output in one of two ways.

1. Progress messages go to stdout, error messages go to stderr2. Data goes to stdout, error AND progress messsages go to stderr

If you know which of the categories your utilities fall into, you can do interesting things.

8.1. Redirection

An uncommon program to use for this example is the "fuser" program under solaris. it givesyou a long listing of what processes are using a particular file. For example:

$ fuser /bin/sh/bin/sh: 13067tm 21262tm

20

If you wanted to see just the processes using that file, you might initially groan and wonderhow best to parse it with awk or something. However, fuser actually splits up the data for youalready. It puts the stuff you may not care about on stderr, and the meaty 'data' on stdout. So ifyou throw away stderr, with the '2>' special redirect, you get

$ fuser /bin/sh 2>/dev/null13067 21262

which is then trivially usable.

Unfortunately, not all programs are that straightforward :-) However, it is good to be aware ofthese things, and also of status returns. The 'grep' command actually returns a status based onwhether it found a line. The status of the last command is stored in the '$?' variable. So if allyou care about is, "is 'biggles' in /etc/hosts?" you can do the following:

grep biggles /etc/hosts >/dev/nullif [[ $? -eq 0 ]] ; then

echo YESelse

echo NOfiAs usual, there are lots of other ways to accomplish this task, even using the same 'grep'command. However, this method has the advantage that it does not waste OS cycles with atemp file, nor does it waste memory with a potentially very long variable.(If you were looking for something that could potentially match hundreds of lines, thenvar=`grep something /file/name` could get very long)

8.2. Inline redirection

You have seen redirection TO a file. But you can also redirect input, from a file. For programsthat can take data in stdin, this is useful. The 'wc' can take a filename as an argument, or usestdin. So all the following are roughly equivalent in result, although internally, differentthings happen:wc -l /etc/hostswc -l < /etc/hostscat /etc/hosts | wc -l

Additionally, if there are a some fixed lines you want to use, and you do not want to bothermaking a temporary file, you can pretend part of your script is a separate file!. This is donewith the special '<<' redirect operator.

command << EOFmeans, "run 'command', but make its stdin come from this file right here, until you see thestring 'EOF'"

EOF is the traditional string. But you can actually use any unique string you want.Additionally, you can use variable expansion in this section!

DATE=`date`HOST=`uname -n`mailx -s 'long warning' root << EOF

21

Something went horribly wrong with system $HOSTat $DATEEOF

9. PipesIn case you missed it before, pipes take the output of one command, and put it on the input ofanother command. You can actually string these together, as seen here;grep hostspec /etc/hosts| awk '{print $1}' | fgrep '^10.1.' | wc -l

This is a fairly easy way to find what entries in /etc/hosts both match a particular pattern intheir name, AND have a particular IP address ranage.

The "disadvantage" to this, is that it is very wasteful. Whenever you use more than one pipe ata time, you should wonder if there is a better way to do it. And indeed for this case, there mostcertainly IS a better way:

grep '^10\.1\..*hostspec' /etc/hosts | wc -l

There is actually a way to do this with a single awk command. But this is not a lesson on howto use AWK!

9.1. Combining pipes and redirection

An interesting example of pipes with stdin/err and redirection is the "tar" command. If youuse "tar cvf file.tar dirname", it will create a tar file, and print out all the names of the files indirname it is putting in the tarfile. It is also possible to take the same 'tar' data, and dump it tostdout. This is useful if you want to compress at the same time you are archiving:tar cf - dirname | compress > file.tar.ZBut it is important to note that pipes by default only take the data on stdout! So it is possibleto get an interactive view of the process, by usingtar cvf - dirname | compress > file.tar.Zstdout has been redirected to the pipe, but stderr is still being displayed to your terminal, soyou will get a file-by-file progress report. Or of course, you could redirect it somewhere else,withtar cvf - dirname 2>/tmp/tarfile.list | compress > file.tar.Z

9.2. Indirect redirection (Inline files)

Additionally, there is a special type of pipes+redirection. This only works on systems with/dev/fd/X support. You can automatically generate a "fake" file as the result of a commandthat does not normally generate a file. The name of the fake files will be/dev/fd/{somenumberhere}

Here's an example that doesnt do anything useful

wc -l <(echo one line) <(echo another line)

22

wc will report that it saw two files, "/dev/fd/4", and "/dev/fd/5", and each "file" had 1 lineeach. From its own perspective, wc was called simply aswc -l /dev/fd/4 /dev/fd/5

There are two useful components to this:

1. You can handle MULTIPLE commands' output at once2. It's a quick-n-dirty way to create a pipeline out of a command that "requires" a

filename (as long as it only processes its input in a single continuous stream).

10. Other StuffAnd here's stuff that I cant fit anywhere else :-)

eval Backticks Text positioning/color/curses stuff Number-based menus Raw TCP access Graphics and ksh

10.1. eval

The eval command is a way to pretend you type something directly. This is a very dangerouscommand. Think carefully before using it.

One way of using eval, is to use an external command to set variables that you do not knowthe name of beforehand. Or a GROUP of variables. A common use of this, is to set terminal-size variables on login:

eval `resize`

10.2. Backticks

There are ways to put the output of one command as the command line of another one. Thereare two methods of doing this that are basically equivalent:echo This is the uptime: `uptime`echo This is the uptime: $(uptime)Technically, the second one is the POSIX-preferred one.

In addition to creating dynamic output, this is also very useful for setting variables:

datestring=`date`echo "The current date is: $datestring"

23

10.3. Text positioning/color games

This is actually a huge topic, and almost deserves its own tutorial. But I'm just going tomention it briefly.

Some people may be familiar with the "curses" library. It is a way to manipulate and movearound text on a screen, reguardless of what kind of "terminal" the user is using.

As mentioned, this is a potentially huge topic. So, I'm just going to give you a trivial example,and say "Go read the man-page on tput". Well, okay, actually, you have to read the "tput"manpage, AND either the "terminfo" or "termcap" manpage to figure out what magical 3-5letter name to use. For example, it should tell you that "cup" is short for the "cursor_address"command. But you must use "cup", NOT "cursor_address", with tput.

tput inittput cleartput cup 3 2print -n "Here is a clean screen, with these words near the top"endline=`tput cols`tput cup $(($endline - 2))print "and now, back to you"sleep 2

The above example clear the screen, prints the given line at a SPECIFIC place on the screen,then puts the cursor back down near the bottom of the screen for you.

PS: If you've been doing a lot of funky things with the screen, you might want to do a

tput resetas the last thing before your shellscript exits.

10.4. Number-based menus

You dont have to build your own "choose a number" function: ksh has one already! But notethat it returns the value of the line, not the number of the line.select word in one two three exit; do

echo word is $wordecho reply is $REPLYif [[ "$word" = "exit" ]] ; then

break;fi

done

This will print out a mini-menu like the following:1) one2) two3) three4) exit#?

24

Note that this will loop between "do ... done" until you trigger a break somehow! (or until theuser control-c's or whatever). So dont forget an exit condition!

10.5. Raw TCP access

Ksh88 has a built in virtual filesystem that looks like it is under /dev/tcp. You can use it tocreate connections to specific ports, if you know the IP address.

Here is a trivial example that just opens up a connection to an SMTP server. Note that theconnection is half-duplex: You do NOT see data that you send to the other side.

#!/bin/ksh -p

MAILHOST=127.0.0.1exec 3<>/dev/tcp/$MAILHOST/25 || exit 1

read -r BANNER <&3echo BANNER is $BANNERprint -u3 HELO myhost.comread -r REPLY <&3echo REPLY is $REPLY

The output will look something like the following:

BANNER is 220 yourhost.domain.com ESMTP Sendmail 8.11.6+Sun/8.11.6; Tue, 3Dec 2002 17:30:01 -0800 (PST)REPLY is 250 yourhost.domain.com Hello localhost [127.0.0.1], pleased tomeet you

Note that we use the "-r" flag to read. In this particular example, it is not neccessary. But inthe general case, it will give you the data "raw". Be warned that if the shell cannot open theport, it will kill your entire script, with status 1, automatically

You can also dump the rest of the data waiting on the socket, to whereever you like, by doing

cat <&3 >somefile

10.6. Graphics and ksh

Not many people are aware of this, but there is actually a graphical version of ksh, called"dtksh". It was created as part of "CDE". Any of the modern UNIX(tm)es should come with it,in /usr/dt/bin/dtksh. If you are interested, take a look at some dtksh demos that someone elsehas written. And/or, you might see if you have a /usr/dt/share/examples/dtksh/ directorypresent on your machine.

11. Paranoia, and good programming practicesFirst, a bit of good programming practice:

25

11.1. Comment your code.

You really should at MINIMUM have some comment about every page (that's every 24 lines).Ideally, you should always comment all your functions. One-line functions can probably standby themselves, but otherwise, a quick single line comment is a good thing to have for smallfunctions.

# This function cleans up the file before printingpcleanup(){

....}

For longer functions, you should really use formal comment spec. Something like

# Function xyz# Usage: xyz arg1 arg2 arg3 arg4# arg1 is the device# arg2 is a file# arg3 is how badly you want to mangle it# arg4 is an optional output file# Result: the file will be transferred to the device, with the# appropriate ioctls. Any err messages will be saved in the output# file, or to stderr otherwisexyz(){

...}Note that shellscripts are themselves one large "function". So dont forget basic comments onyour shellscript's functionality at the top of the file!

11.2. INDENT!

Every time you start a function, indent.

Every time you are in a while loop, or if clause, indent.

This makes it easier to see at a glance what level you are at.

# top levelprint this is the top

somefunction(){# indent in functionsprint we are in the function now

if [[ "$1" != "" ] ; then# yes, ANOTHER indent!print "Hey, we have an argument: $1"print full args are $*

else# indent again!!print "oh well. no arguments to play with"

fi

26

print leaving somefunction now}

# And now we can clearly see that all this stuff is outside any function.# This makes it easier to find the "main line" of the scriptprint original shellscript args are $0print lets try somefunctionsomefunction heres some args

11.3. Error checking

If you are using scripts to check on important systems, or perform crucial functions, it is veryimportant to provide feedback to the calling process when and if something goes wrong.

The simplest method of doing this, is a simple

exit 1# (but it would be nice to print out SOME message before exiting!)

Nice programs will notice that your script exited with a non-zero status. [Remember, thestatus of the last command is in '$?']Ideally, they will complain.On the other hand, sometimes your own scripts are the ones that are doing the calling!

In that type of situation, it may be suitable to have a top-level script that keeps an eye onthings. A simple example is:

fatal(){# Something went horribly wrong.# print out an errormessage if provided, then exit with an# "error" status

if [[ "$1" != "" ]] ; thenprint Something went horribly wrong with "$1"

elseprint "something went horribly wrong (Somewhere?)"

fi

# You mail also want to SEND EMAIL, or trigger a PAGER or# something here:# mailx -s "Arrrg! we failed checks!" admin@yourhost.com <

/dev/null#exit 1

}

check_on_modemsif [[ $? -ne 0 ]] ; then fatal modems ; fi

check_on_networkif [[ $? -ne 0 ]] ; then fatal network ; fi

check_on_serversif [[ $? -ne 0 ]] ; then fatal servers ; fi

27

Note that even my paranoid 'fatal' function, IS PARANOID!Normally, it is assumed you will call it with "fatal what-failed". But if you somehow dont, itnotices, and provides a default.

Sometimes, making the assumption that $1 contains valid data can completely screw up therest of your function or script. So if it is an important function, assume nothing!

This is particularly true of CGI scripts. [Gasp]. Yes, Virginia, it IS possible to write CGI insomething other than perl.

11.4. cron job paranoia

If you are coding a script that specifically is in a cronjob, there are two things to be aware of.The number one most important thing is;

Set your PATH variable!!!People always forget this one. It doesnt matter what is in your .profile, cron will reset thePATH variable to something really short and stupid, like just /usr/bin. So set yourPATH=/whatever:/usr/bin explicitly in your cron scripts.

The second tip is more an advisory note.

cron by default saves anything that gets send to 'stderr', and MAILS IT to the owner of thecron job. So, sometimes, if you just want a minor error logged somewhere, it is sufficient tojust do

print "Hey, this is kinda wierd" >/dev/fd/2which will send the output of the print to stderr, which will then go to email. Another way ofdoing it, if your system does not have /dev/fd, could beprint "Wow, this is tortured" 1>&2

Contrariwise, if you want to throw away ALL output, use

command >/dev/null 2>&1

If you do not regularly read email for the user in question, you can either set up an alias forthat user, to forward all its email to you, or do

export MAILTO=my@address.here

The MAILTO trick does not work on all cron demons, however.

28

12. Example of script development12.1. The good, the bad, and the ugly

Hopefully, you have read through all the other chapters by this point. This page will show youthe "big picture" of shellscript writing.

Here are four versions of essentially the same program: a wrapper to edit a file under SCCSversion control.The basic task is to use the sccs command to "check out" a file under version control, and thenautomatically edit the file. The script will then be used by "users", aka coders, who may notbe particularly advanced UNIX users. Hence, the need for a wrapper script.

While the basic functionality is the same across all versions , the differences in safety andusability between the first version and the last version are staggering.

The first one is extremely bad: it would be written by someone who has just picked up a bookon shellscripting, and has decided, "I'm a programmer now".

The second one is an improvement, showing some consideration to potential users by havingsafety checks.

The third one is a good, solid version. It's a positive role model for your scripts.

The final one is a full-blown, paranoid, commented program unto itself, with whiz-bangfeatures. This is the way a professional programmer would write it. Don't let that put you off:it's the way you can and should write it too! You might start with a version like the initialdumb one, as the initial step in your code development, just to make sure you have the basicfunctionality for the task. But after it is shown to work, you should upgrade it to a morereasonable one immediately.

Note: there is a summary of good practices show in this page, at the bottom.

12.2. The newbie progammer version

#!/bin/ksh

sccs edit $1if [ "$EDITOR" = "" ] ; then

EDITOR=vifi$EDITOR $1

This version makes somewhat of an attempt to be user friendly, by having a check for a user-specified EDITOR setting, and using it if available. However, there are no comments, no errorchecking, and no help for the user whatsoever!

29

12.3. The sysadmin-in-training version

#!/bin/ksh

# Author: Joe Newguy

if [ $# -lt 1 ] ; thenprint "This program will check out a file, or files,

with sccs"exit 1

fi

# set EDITOR var to "vi" if not already setEDITOR=${EDITOR:-vi}

sccs edit $@$EDITOR $@

This is somewhat of a step above the prior version. It accepts multiple files as potentialarguments. It's always nice to be flexible about the number of files your scripts can handle. Italso has a usage message, if the script is called without arguments. Plus, it's always a goodidea to put your name in in, unless you're working for the company of "Me, Myself and I,Inc."

Unfortunately, there is still quite a bit lacking, as you can tell by comparing it to the nextversion.

12.4. The Senior Admin version

#!/bin/ksh

# SCCS editing wrapper, version 0.3# Author - Sys Admin# Usage: see usage() function, below

usage(){print sedit - a wrapper to edit files under SCCSprint "usage: sedit file {file2 ...}"

}

# Set EDITOR var to "vi" if not already set to somethingEDITOR=${EDITOR:-vi}# Could already be in path, but it doesnt hurt to add itagain.# Sorry, I assume solaris machines everywhere: adjust asneeded,# if your sccs lives somewheres elseSCCSBIN=/usr/ccs/binif [ ! -x $SCCSBIN/sccs ] ; then

print ERROR: sccs not installed on this machine.Cannot continue.

usageexit 1

fi

30

PATH=$SCCSBIN:$PATH

if [ $# -lt 1 ] ; thenusageprint ERROR: no files specifiedexit 1

fi

# Yes, I could use "sccs edit $@" and check for a singleerror, but this# approach allows for finer error reportingfor f in $@ ; do

sccs edit $fif [ $? -ne 0 ] ; then

print ERROR checking out file $fif [ "$filelist" != "" ] ; then

print "Have checked out $filelist"fiexit 1

fifilelist="$filelist $f"

done

$EDITOR $filelistif [ $? -eq 0 ] ; then

print ERROR: $EDITOR returned error statusexit 1

fi

This guy has been around the block a few times. He's a responsible sysadmin, who likes to bedisaster-prepared. In this case, the most likely "disaster" is 100 calls from developers asking"Why doesnt it work for me?" So when things break, it's a good idea to provide as muchinformation as possible to the user.

Nice things to note:

Sets special variables at the top of the script, in a unified place Paranoid checks about EVERYTHING Returns an error status from the script, on non-clean exit condition ("exit 1", vs "exit") Use of comments. Not only does he specify what he is doing, he clarifies what he is

NOT doing, and why.

Compare and contrast the first version of the program, to this one. Then try to make your ownscripts be more like this!

12.5. The Senior Systems Programmer version

#!/bin/ksh

# SCCS editing wrapper, version 1.3# Author - Phil Brown

31

# Usage: see usage() function, below

usage(){print sedit - a wrapper to edit files under SCCSprint "Usage: sedit [-c|-C] [-f] file {file2 ...}"print " -c check in file(s) after edit is

complete"print " -C check in all files with single

revision message"print " -f ignore errors in checkout"

}

# Set EDITOR var to "vi" if not already set to somethingEDITOR=${EDITOR:-vi}# Could already be in path, but it doesnt hurt to add itagain.# Sorry, I assume solaris machines everywhere: adjust asneeded.PATH=$PATH:/usr/ccs/binif [ ! -x /usr/ccs/bin/sccs ] ; then

print ERROR: sccs not installed on this machine.Cannot continue.

usageexit 1

fi

while getopts "cCfh" argdo

case $arg inc)

checkin="yes";;

C)checkinall="yes";;

f)force="yes";;

h|*)usageexit 1;;

esacdone

shift $(($OPTIND - 1))

if [ $# -lt 1 ] ; thenusageprint ERROR: no files specifiedexit 1

fi

if [ "$checkinall" != "" ] && [ "$checkin" != "" ] ; thenprint WARNING: -c and -C used. Will use -C.

fi

# Yes, I could use "sccs edit $@" and check for a singleerror, but this# approach allows for finer error reporting.

32

# "$@" is a special construct that catches spaces infilenames.# Note that "$*" is NOT 100% the same thing.for f in "$@" ; do

sccs edit "$f"if [ $? -ne 0 ] ; then

print ERROR checking out file $fif [ "$force" = "" ] ; then

if [ "$filelist" != "" ] ; thenprint "Have checked out

$filelist"fiexit 1

fi# else, -f is in effect. Keep going

fifilelist="$filelist $f"

done

# I would like to use "$filelist", but that does notpreserve spaces# in file names$EDITOR "$@"if [ $? -eq 0 ] ; then

print ERROR: $EDITOR returned error statusexit 1

fi

if [ "$checkinall" != "" ] ; then# -C option used. re-check in all files at once.sccs delget $filelistif [ $? -ne 0 ] ; then

print "ERROR checking in files?"exit 1

fiexit 0

fiif [ "$checkin" != "" ] ; then

# -c option used. re-check in each file.for file in $filelist ; do

sccs delget $fileif [ $? -ne 0 ] ; then

print "WARNING: failed to check in$file"

fi# do NOT stop after error. Keep trying to

check# in any other files

donefi

This guy has been around the block a few times. Heck, he helped BUILD the block ;-)

This was originally my third and final version. It's the way I would really write the script. ButI decided it might be a bit daunting to new scripting folks, so I made a new intermediate thirdstep, above.

Additional things beyond the previous version:

33

Provides optional extra functionality, where it makes sense. (added -c, -C, and -foption flags.) This shows understanding of writing scripts, AND understanding of thearea of the task (SCCS version control)

Use of the 'getopts' standard util, rather than hand-coding a custom argument parser

13. Summary of positive featuresHere is a summary of all the positives added through the different versions.

COMMENT YOUR SCRIPTS!!! Check for user-environment variables, where appropriate When setting/overriding special variables, do it at the top of the script, in a unified

place Accept one OR MORE files as arguments, when it makes sense to do so. Have a "usage" message Be Paranoid: check return statuses Return correct statuses yourself: ("exit 1", vs "exit", on error) Take advantage of 'getopts' when adding optional feature flags