Basic Bash Scripting

transcript

BASIC BASH SCRIPTINGKatherine HolcombUVACSE

Uses for Scripting• Automate repetitive tasks

• Could also use languages like Perl or Python but it’s still handy to know some bash

• Write job scripts for the PBS queueing system• For most users these must be a shell script (bash, ksh,

or tcsh)• Anything you can type on the command line can be put into a script

Example#!/bin/bash#This is a bash scriptecho “Hello world!”echo “I am done now!” #my comment

• The first line tells the system to start the shell and execute the commands. It is sometimes called a “shebang.”

Executing Your Script• Suppose the previous script was in a file hello.sh

• By default it is just text. You must explicitly make it executable

chmod a+x hello.sh• Then you can just invoke it by name at the command line

./hello.sh

Comments and Continuations• All text from the # to the end of the line is ignored • To continue a statement on to the next line, type \<enter>

#This is a commentecho “This line is too long I \ want it to be on two lines”

• Multiple statements can be placed on the same line when separated by the ;

a=20; b=3

Sourcing• If you execute the script as a standalone script, it starts a new shell -- that is what the shebang does -- if you do not use that you can type

bash myscript.sh• Sometimes you just want to bundle up some commands that you repeat regularly and execute them within the current shell. To do this you should not include a shebang, and you must source the script

. <script> or source <script>

Variables• Like all programming languages, the shell permits you to define variables.

• Variables and environment variables are essentially the same thing. Environment variables are conventionally written with all capital letters and are used to set properties. Shell variables are used for assignments and such.

• Arrays are supported • Scalar variables are referenced with $• When assigning variables do not put spaces around the equals sign!

String Expressions and Variables• Bash isn’t very good at math so much of what we do with it involves strings.

• String expressions• Single quotes (‘hard quotes’): the expression is

evaluated literally (no substitutions)• Double quotes (“soft quotes”): the $, \ (backslash),

and ` (backtick) are taken to be shell characters$ indicates a variable name follows\ escapes the following character` command` command is executed and its output streams are returned --In newer versions $(command)is synonymous

Exampleecho `date`echo ‘This is worth $2’var=2echo “This is worth $2”echo “This is worth $var”echo “This is worth \$$var”

Conditionals• The shell has an if else construct:if [[ <condition> ]]then commandselif #optional

commandselse #optional more commandsfi

Conditions• The conditions take different forms depending on what you want to compare.

• String comparison operators:• = != < > -z -n• equal, not equal, lexically before, lexically after (both these may need to be escaped), zero length, not zero length.

• Can use == but behaves differently in [] versus [[]]

Example: String Comparisonsif [[ $name == “Tom” ]]; then

echo Tom, you are assigned to Room 12elif [[ $name == “Harry” ]]; then

echo Harry, please go to Room 3elif [[ -z $name ]]; then

echo You did not tell me your nameelse

echo Your name is $namefi

Numeric Comparisons• Numeric comparisons are possible:

-eq -ne -gt -ge -lt -le

if [[ $a -eq 2 ]]; then #note ; stuff fi

Testing File Properties• This is a very common occurrence in bash scripts.

• There are many operators for this. These are the most common:-e <file> : file exists-f <file> : file exists and is a regular file-s <file> : file exists and has length > 0-d <dir> : exists and is a directory

Exampleif [[ -d mydir ]]; thenif [[ -f the_file ]]; thencp the_file mydir

fielsemkdir mydirecho “Created mydir”

Other Conditional Operators! : not ; negates what follows-a : and ; for compound conditionals

can use && with [[]]-o : or ; for compound conditionals

can use || with [[]]

Case Statement

case expression inpattern1 )statements ;;

pattern2 )statements ;;

pattern3 )statements ;;

Examplecase $filename in *.f) echo “Fortran 77 source file” ;; *.c) echo “C source file” ;; *.py) echo “Python script file” ;; *) #optional, indicates default echo “I don’t know what this file is” ;;esac

Loops• The bash for loop is a little different from the equivalent in C/C++/Fortran (but is similar to Perl or Python)

for variable in iteratordo

commandsdone

Examplesfor i in 1 2 3 4 5 ; do

echo “I am on step $i”donefor i in {1..5} #bash 3.0 and updo

echo “I am on step $i”donefor i in 0{1..9} {10..100}; do echo “File extension will be $i”done

Three-Expression Forfor (( EXPR1; EXPR2; EXPR3 ))do statementsdone• Example:for (( i=0; i<$IMAX;i++ )); do echo $name”.”$idone

while loopwhile [ condition ] docommandcommandcommand

One of the commands in the while loop must update the condition so that it eventually becomes false.

break• If you need to exit a loop before its termination condition

is reached you can use the break statement.

while [ condition ]do

if [ disaster ]; thenbreak

ficommandcommand

continue• To skip the commands for an iteration use continuefor i in iterator; do

if [[ something ]]; thencontinue

ficommand command

Bash Arithmetic• We said earlier that bash is bad at arithmetic, but some basic operations can be performed.

• Expressions to be evaluated numerically must be enclosed in double parentheses. It works only with integers.

x=$((4+20))i=$(($i+1))

If you need more advanced math (even just fractions!) you must use bc.

bc• If you really, really, really must do math in a bash script, most of the time you must use bc

• The syntax is very peculiarx=$(echo “3*8+$z” | bc)

Command-Line Arguments• Many bash scripts need to read arguments from the command line. The arguments are indicated by special variables $0, $1, $2, etc.

• $0 is the name of the command itself• The subsequent ones are the options• To have a variable number of options, use the shift built-in.

• The special variable $# is the number of arguments (not counting the command name)

Example

#!/bin/bash USAGE="Usage:$0 dir1 dir2 dir3 ...dirN" if [ "$#" == "0" ]; then

echo "$USAGE" exit 1

fi while [ $# -gt 0 ]; do

echo “$1”shift

More Useful Example• while plus casewhile [ $# -gt 0 ]; do case “$1” in

-v) verbose=“on”;;-*)

echo >&2 “USAGE: $0 [-v] [file]”exit 1;;

*) break;; # defaultesacshift

Functionsfunction name() {

statementstatementstatement VALUE=integerreturn $VALUE}

• the keyword function is optional in newer versions of bash. The parentheses are always left empty.

• Function definitions must precede any invocations.

Function Arguments• Function arguments are passed in the caller. In the function they are treated like command-line options.#!/bin/bashfunction writeout() { echo $1}writeout “Hello World”

Variables• Variables set in the function are global to the script!

#!/bin/bashmyvar="hello"

myfunc() {

myvar="one two three" for x in $myvar do echo $x done}

myfunc

echo $myvar $x

Making Local Variables• We can use the keyword local to avoid clobbering our global variables.

#!bin/bashmyvar="hello" myfunc() { local x local myvar="one two three" for x in $myvar ; do echo $x done } myfunc echo $myvar $x

Return Values• Strictly speaking, a function returns only its exit status.

• The returned value must be an integer• You can get the value with $?• e.g.myfunc $1 $2result=$?

String Operations• Bash has a number of built-in string operations.• Concatenation

• Just write them together (literals should be quoted)newstring=$oldstring”.ext”

• String length ${#string}

• Extract substring• Strings count from 0—first character is numbered 0• Extract from pos to the end${string:pos}• Extract len characters starting at pos${string:pos:len}

Clipping Strings• It is very common in bash scripts to clip off part of a string so it can be remodeled.• Delete shortest match from front of string ${string#substring}• Delete longest match from front of string ${string##substring}• Delete shortest match from back of string ${string%substring}• Delete longest match from back of string ${string%%substring}

Arrays• Array variables exist but have a somewhat unusual syntax.

• Arrays are zero based so the first index is 0• Initialize arrays with a list enclosed in parentheses• Obtaining the value of an item in an array requires use of ${}val=${arr[$i]}

${arr[@]} # All of the items in the array @ or * work in this and next example

${#arr[@]} # Number of items in the array ${#arr[0]} # Length of item zero

Example

my_arr=(1 2 3 4 5 6)for num in ${my_arr[@]}; do echo $numdone

Realistic Examplejpg_files=(`ls *jpg`)for file in ${jpg_files[*]}; do if [[ -n $file ]]; then convert $file ${file%%”.jpg”}.png fidone

Herefiles• A herefile or here document is a block of text that is dynamically generated when the script is run CMD << DelimiterlinelineDelimiter

Example#!/bin/bash # 'echo' is fine for printing single line messages, # but somewhat problematic for for message blocks. # A 'cat' here document overcomes this limitation. cat <<End-of-message------------------------------------- This is line 1 of the message. This is line 2 of the message. This is line 3 of the message. This is line 4 of the message.This is the last line of the message. ------------------------------------- End-of-message

Regular Expressions• Regular expressions are generalizations of the wildcards often used for simple file commands.

• A regular expression consists of a pattern that attempts to match text.

• It contains one or more of:• A character set• An anchor (to the line position)• Modifiers

• Without an anchor or repeat it will find the leftmost match and stop.

Regex: Character sets and modifiers• The character set is the set of characters that must be matched literally.

• Modifiers expand the potential matches. • * matches any number of repeats of the pattern before it (note that this is different from its use in the shell) including 0 matches.

• ? Matches 0 or 1 character (also different from the shell wildcard).

• + matches one or more, but not 0, occurrences.• . matches any single character, except newline

More Modifiers• \ escapes the preceding character, meaning it is to be used literally and not as a regex symbol.

• \ can also indicate nonprinting characters, e.g. \t for tab.

• () group the pattern into a subexpression• | pipe is or

• [gray|grey] equivalent to [gr(a|e)y]

Regex: Ranges and Repetition• [] enclose a set of characters to be matched.• - indicates a range of characters (must be a subsequence of the ASCII sequence)

• {n} where n is a digit, indicates exactly n repetitions of the preceding character or range.

• {n,m} matches n to m occurrences.• {n,} matches n or more occurrences.

Regex: Anchors and Negation• ^ inside square brackets negates what follows it• ^ outside square brackets means “beginning of target string”

• $ means “end of target string”• . Matches “any character in the indicated position”

• Note: the “target string” is usually but not always a line in a file.

Examples• AABB* matches• AAB• AABB• AABBBB• But not• AB• ABB• ABBBB

Examples (Cont.)• [a-zA-Z] matches any letter• [^a-z] matches anything except lower-case letters

• .all matches all, call, ball, mall, and so forth. Also matches shall (since it contains hall).

• Regex patterns are said to be greedy since they find a match with the most generous interpretation of the pattern.

Extensions and Shortcuts• Most shells and languages support some shortcuts:• \w : [A-Za-z0-9_]• \s : [ \t\r\n] some flavors have a few more

rare whitespace characters• \d : [0-9]• \D : ^\d• ^\W: ^\w• ^\S: ^\s• NB \D\S is not the same as ^\d\s; in fact it

matches anything. ^\d\s matches a but not 1

Grep, Sed and Awk• grep or egrep can be used with regular expressions.

• sed is the stream editor. It is used to script editing of files.

• awk is a programming language to extract data and print reports.

grep examples• grep “regex” filename

• The quotes are often needed; they make sure it’s interpreted as a regex

• We assume Gnu grep on Linux (it is also called egrep)• egrep matches anywhere in the string

• grep ^root /etc/passwd• grep :$ /etc/passwd

sed examples• Sed operates on standard input and outputs to stdout unless told otherwise. Thus you must redirect.

• An option will tell it to overwrite the old file. This is often a mistake (if you do this, be sure to debug the sed command carefully).

• sed ‘command’ < old > new• Note hard quotes—best practice is to use them

• sed ‘s/day/night/’ < old > new• Remember, expressions are greedy; day/night here

changes Sunday to Sunnight

awk examples• awk ‘pattern {actions}’ file• Similar to C, awk “action” lines end with ;• For each record (line) awk splits on whitespace. The results are stored as fields and can be referenced as $1, $2, etc. The entire line is $0 and $NF indicates the number of fields in the line.

• awk ‘pattern1 {actions} pattern2 {actions}’ file

• awk ‘Smith’ employees.txt• awk ‘{print $2, $NF;}’ employees.txt

Some Resources• Linux Shell Scripting Tutorial:

http://bash.cyberciti.biz/guide/Main_Page

• How-To:http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html

• Advanced Scripting Tutorialhttp://tldp.org/LDP/abs/html/

More Resources• Regex Tutorials

http://www.regular-expressions.info/http://www.zytrax.com/tech/web/regex.htm

• sed tutorialhttp://www.grymoire.com/Unix/Sed.html

• awk tutorialhttp://www.grymoire.com/Unix/Awk.html(but note that we have gawk on Linux)

Basic Bash Scripting

Documents