Unix Training

<Insert Picture Here>

Introduction to Unix

Table of Contents

• Introduction• Basic Commands• Process Check• CPU/Memory Utilization• Network Commands• Korn Shell and C-Shell• .Profile Features and Password files• Crontab and Cron jobs• Shell Scripts• SED and AWK• Interface to SQL

© 2009 Oracle Corporation – Proprietary and Confidential

History of UNIX

• Ken Thompson of AT&T Bell Laboratories designed UNIX in late 1960s

• Two versions of UNIX that emerged are AT&T Unix and BSD Unix

• In 1989, AT&T and Sun Microsystems joined together and developed system V release 4 (SVR4)

• Two of the main standards mainly in use are POSIX (Portable Operating System Interface) and X/open standard.

• In 1988, MIT formed Xconsortium developed vendor-neutral Xwindow.


Features of UNIX

• Multi-user, multitasking, timesharing

• Portability

• Modularity

• File structure

• Security

• Strong networking support & advanced graphics


Layered Architecture

Following is the architecture of Unix system. Viewing the system as a set of layers.


UNIX System Architecture

• Unix system follows a layered approach. It has four layers

• The innermost layer is the hardware layer

• In the second layer, the kernel is placed

• The utilities and other application programs form the third layer

• Fourth layer is the one with which the user actually interacts.


The Unix file system is characterized by• A hierarchical structure• Consistent treatment of the data• The ability to create and delete files• Dynamic growth of files• The protection of file data• The treatment of peripheral devices


The File System

KernelKernel is the part of the OS which directly makes interface with the hardware system.

• Three major tasks of kernel:– Process Management– Device Management– File Management

• Three additional services of kernel– Virtual Memory– Networking– Network File Systems

• Other Features– Multiprocessor support– Thread support


Kernel continued…Block diagram of System Kernel


The Shell• Shell is a command-line interpreter and script host that provides a traditional user interface for the Unix

operating system.

• Features of Shell are:– Interactive Processing– Background Processing– I/O Redirection– Pipes– Shell Scripts– Shell Variables– Programming Constructs


Common UNIX Flavours

• BSD : Berkeley, BSD

• Solaris : Sun Microsystems, Sys 5/BSD

• Ultrix : Digital Equipment Corporation, BSD

• OSF 1 : Digital Equipment Corporation, BSD/sys 5

• HPUX : Hewlett-Packard, Sys 5

• AIX : IBM, Sys 5 / BSD

• IRIX : Silicon Graphics, Sys 5

• GNU/Linux : GNU, BSD/Posix


Basic Commands

• pwd– Displays the current working directory.

• date– Displays the current date and time

• who– Displays the names of all the users who have currently logged in

• who am i– Displays the name of the current user.


Basic Commands

• mkdirThe mkdir command creates one or more new directories specified by the Directory parameter.

Syntax: mkdir [ -m Mode ] [ -p ] Directory ...

options : – -m Mode Sets the permission bits for the newly-created directories to the value specified by the

Mode variable. – -p Creates missing intermediate path name directories. If the -p flag is not specified, the parent

directory of each-newly created directory must already exist.

Notes:– The mkdir command ignores any Directory parameter that names an existing directory. No error is

issued.


Basic Commands

• rm– The rm command removes the entries for the specified File parameter from a directory.

Syntax: rm [ -f ] [ -r ] [ -R ] [ -i ] [ -e ] File ...

• options : – -e Displays a message after each file is deleted.– -f Does not prompt before removing a write-protected file. – -i Prompts you before deleting each file. – -r Permits recursive removal of directories and their contents when the File parameter is a directory.

This flag is equivalent to the -R flag.– -R Permits recursive removal of directories and their contents when the File parameter is a directory.

This flag is equivalent to the -r flag.


Basic Commands

• cd The cd command sets the current working directory of a process.

Syntax: cd [directory]

Examples

1 To change the current working directory to the login (home) directory, type: cd

2 To change to an arbitrary directory, type:

cd /usr/include (This changes the current directory to /usr/include.)

3 To go down one level of the directory tree, type:

cd <directory name>


Basic Commands

• lslisting directory contents

Syntax :ls [options] [file….]

ls command has rich options. Some of them are listed below

options: -l list in long format

-a list all files including those beginning with a dot

-i list inode no of file in first column

-s reports disk blocks occupied by file

-R recursively list all sub directories

-F mark type of each file

-C display files in columns


Basic Commands


Basic Commands

• findFinds files with a matching expression.

The find command recursively searches the directory tree for each specified Path parameter, seeking files that match a Boolean expression written

Syntax: find [options] Path ... [ Expression ]

options : -H Cause the file information and file type evaluated for each symbolic link encountered on the command line to be those of the file referenced by the link, and not the link itself. If the referenced file does not exist, the file information and type shall be for the link itself. File information for all symbolic links not on the command line shall be that of the link itself.

-L Cause the file information and file type evaluated for each symbolic link to be those of the file referenced by the link, and not the link


Basic Commands

• grepThe grep command searches for the pattern specified by the Pattern parameter and writes each matching line to standard output.

Syntax: grep [ -E | -F ] [ -i ] [ -h ] [ -H ] [ -L ] [ -r | -R ] [ -s ] [ -v ] [ -w ] [ -x ] [ -y ] [ [ [ -b ] [ -n ] ] | [ -c | -l | -q ] ] [ -p [Separator ] ] { [ -e PatternList ... ] [ -f PatternFile ... ] | PatternList ... } [ File ... ]

Notes:– Do not run the grep command on a special file because it produces unpredictable results.– Input lines should not contain the NULL character.– Input files should end with the newline character.– The newline character will not be matched by the regular expressions.– Although some flags can be specified simultaneously, some flags override others. For example, the -l

option takes precedence over all other flags. And if you specify both the -E and -F flags, the last one specified takes priority.


Basic Commands

• cutThe cut command cuts bytes, characters, or fields from each line of a file and writes these bytes, characters, or fields to standard output. If you do not specify the File parameter, the cut command reads standard input.

Syntax: cut { -b List [ -n ] | -c List | -f List [ -s ] [ -d Character ] } [ File ... ]

options : – -b List Specifies byte positions. – -c List Specifies character positions. – -d Character Uses the character specified by the Character variable as the field delimiter when you

specify the -f flag. – -f List Specifies a list of fields assumed to be separated in the file by a delimiter character, which is by

default the tab character. – -n Suppresses splitting of multibyte characters. Use only with the -b flag. If the last byte of a character

falls within the range denoted by the List variable of the -b flag, the character is written; otherwise, the character is excluded.

– -s Suppresses lines that do not contain delimiter characters. Use only with the -f flag.



Basic Commands

• chmodThe chmod command (abbreviated from change mode) is a shell command and C language function in Unix and Unix-like environments. The command is based on the function. When executed, it can change file system modes of files and directories. The modes include permissions and special modes.

Syntax: chmod [options] mode[,mode] file1 [[file2 ...]]

Reference Class Description

u user the owner of the file

g group users who are members of the file's group

o others users who are not the owner of the file or members of the group

a all all three of the above, is the same as ugo

options : – R: Changes the modes of directories and files recursively – v: Verbose mode; lists all files as they are being processed – r,4 read read a file or list a directory's contents – W,2 write write to a file or directory – X,1 execute execute a file or recurse a directory tree

Example:

chmod ug+rw sample or chmod 660 sample

Getting Help on Commands

• The Unix manual, usually called man pages, is available online to explain the usage of the Unix system and commands.

Syntax:

man [options] command_name

– Common Options• -k keyword list command synopsis line for all keyword matches• -M path to man pages• -a show all matching man pages (SVR4)

• help - gives command syntax


Common shells

Shell Name Developed by Where Remark

BASH ( Bourne-Again SHell )

Brian Fox and Chet Ramey Free Software Foundation Most common shell in Linux. It's Freeware shell.

CSH (C SHell) Bill JoyUniversity of California (For

BSD)

The C shell's syntax and usage are very similar tothe C programming language.

KSH (Korn SHell) David Korn AT & T Bell Labs --

TCSHSee the man page.Type $ man tcsh

--

TCSH is an enhanced but completely compatible version of the Berkeley UNIX C shell (CSH).

Tips:

To find all available shells in your system type following command: $ cat /etc/shells

To find your current shell type following command $ echo $SHELL


Common Shells

Every shell has one or two files which are used at login to configure the user's environment.

In the C shell these are the .login and .cshrc files.

The .login file is read in when you first login to the machine and is primarily used to establish global parameters.

The .cshrc file establishes the environment settings for each shell process which is executed during the session.

In the Korn shell these roles can both be filled by .profile and .kshrc file.

The most obvious difference between C shell and Bourne shell syntax was how their built-in conditional operations called out to test--the C shell used parentheses, and the Bourne shell used square brackets. (Later versions of Bourne shell would rectify this by using a built-in test operator). Variable assignment was also performed slightly differently with "set a = b" rather than Bourne shell's "a=b".


Process

• A process is the execution of a program and consists of a pattern of bytes that the CPU interprets as machine instructions. In practical terms, a process on a Unix system is the entity that is created by the system call fork

• Every process except process 0 is created when another process executes the fork system call. The process that invoked the fork system call is the parent process.

• Process 0 is a special process that is created when the system boots.• To identify the list of process that is currently running issue the following command

ps

It fetches the pid, tty, time, and the command which has started the process.– -f lists the pid of the parent process also.– -u lists the processes of a given user (eg: ps –u <username>)– -a lists the processes of all the users– -e lists all the processes including the system processes


Background Process

• The & is an important little character in UNIX; it means "run the command in the background"; i.e., detach it from the window it was started from, so it does not block the command line.

• If the command terminates with an ampersand (&), UNIX executes the command in the background

• Shell returns by displaying the process ID (PID) and job id of the process

• For example to run sleep in the background, type

Sleep 10 &


Background Process

• nohup

– Lets processes to continue to run even after logout. It is most often used to run commands in the

background as daemons– The output of the command is sent to nohup.out if not redirected

$ nohup command args

$ nohup sort emp.lst &

[1] 21356

nohup: appending output to `nohup.out‘

• wait– can be used when a process has to wait for the output of a background process– The wait command, can be used to let the shell wait for all background processes terminate.

$ wait– It is possible to wait for completion of one specific process as well.

• Nice– Used to reduce the priority of jobs


Controlling Background Processes

• jobs– When a process is running, backgrounded or suspended, it will be entered onto a list along with a job

number. To examine this list, issue jobs command– An example of a job list could be

• fg % <job id>– To restart (foreground) a suspended processes. For example, to restart sleep 1000, type fg %1

• bg %<job id>– Runs a process in the background


The kill Command

• It is sometimes necessary to kill a process (for example, when an executing program is in an infinite loop)

• To kill a job running in the foreground, type ^C (control c). For example, run

• To kill a suspended or background process, type

• If a process refuses to be killed, uses the -9 option, i.e. type

kill -9 <process id/job number>

– Will terminates the process abruptly


© 2008 Oracle Corporation – Proprietary and Confidential 30

Bellow commands will give the CPU/Memory Information.

sar : System activity reporter mpstat : Report per-processor or per-processor-set statistics ps : Reports information about active processes top : Provides a dynamic real-time view of a running system

Example:

Command to display the current CPU usagesar –u

Command to display per-processor statistics; 12 seconds apart; 5 timesmpstat 12 5

Command to show the information about all processes currently running on the systemps –e

Command to get the cpu and memory utilization of a processtop -p <pid>

CPU/Memory Utilization Commands


Networking Commands

Bellow commands will give the detail information of network connectivity in UNIX.

ping : Sends answer request to other hosts. ftp : Transfer files to another host (insecure) ifconfig : Display Ip address information rout : Show routing information telnet : Connect (TCP) to a specific machine/port netstat : Display network connections, routing tables, and interface statistics scp : Secure copy( copies files between hosts on a network, prompt for password) rcp : Remote file copy( copies files between machines, does not prompt for password)

Example: Command to display the host name or IP address of the computerping <Host-Name>

Command to connect from local machine to the remote machineftp <machine-name>

Command to display the IP address information of the local machineifconfig

Command to Verifying a route to a specific networkroute –n get <IP-Address>

Command to connect to a specific machinetelnet <machine-name>


Networking Commands – Continue…

Command to display routenetstat –rn

Command to copy the entire contents of the somedir directory on [email protected] back to local machine somedir directory.scp -r [email protected]:somedir somedir

Command to copy “document1” from the local machine to the user’s home directory on the computer with URL unix.ofss.edu

rcp document1 unix.ofss.edu:document1

cron is a time-based job scheduler in Unix-like computer operating systems. 'cron' is short for 'chronograph'.

Cron is the name of program that enables unix users to execute commands or scripts (groups of commands) automatically at a specified time/date

cron is driven by a crontab, a configuration file that specifies shell commands to run periodically on a given schedule.

Syntax:

The crontab files are stored where the lists of jobs and other instructions to the cron daemon are kept. Users can have their own individual crontab files and often there is a system wide crontab file (usually in /etc or a subdirectory of /etc) which only system administrators can edit.

Each line of a crontab file represents a job and follows a particular format as a series of fields, separated by spaces and/or tabs. Each field can have a single value or a series of values.

Crontab

Crontab Syntax

A crontab file has five fields for specifying day , date and time followed by the command to be run at that interval

crontab File

* in the value field above means all legal values as in braces for that column. The value column can have a * or a list of elements separated by commas. An element is either a number in the ranges shown above or two numbers in the range separated by a hyphen (meaning an inclusive range).

Note: The specification of days can be made in two fields: month day and weekday. If both are specified in an entry, they are cumulative meaning both of the entries will get executed .

A line in crontab file like below removes the tmp files from /home/someuser/tmp each day at 6:30 PM.

30 18 * * * rm /home/someuser/tmp/*

Changing the parameter values as below will cause this command to run at different time schedule below :

Crontab Example


Password File

/etc/passwd file stores essential information, which is required during login i.e. user account information

etc/passwd is a text file, that contains a list of the system's accounts, giving for each account some useful information like user ID, group ID, home directory, shell, etc. It should have general read permission as many utilities, like ls use it to map user IDs to user names, but write access only for the superuser (root).

Understanding fields in /etc/passwd

The /etc/passwd contains one entry per line for each user (or user account) of the system. All fields are separated by a colon (:) symbol. Total seven fields as follows.

Generally, passwd file entry looks as follows (click to enlarge image):


Password File Username: It is used when user logs in. It should be between 1 and 32 characters in length.

Password: An x character indicates that encrypted password is stored in /etc/shadow file.

User ID (UID): Each user must be assigned a user ID (UID). UID 0 (zero) is reserved for root and UIDs 1-99 are reserved for other predefined accounts. Further UID 100-999 are reserved by system for administrative and system accounts/groups.

Group ID (GID): The primary group ID (stored in /etc/group file)

User ID Info: The comment field. It allow you to add extra information about the users such as user's full name, phone number etc. This field use by finger command

Home directory: The absolute path to the directory the user will be in when they log in. If this directory does not exists then users directory becomes /

Command/shell: The absolute path of a command or shell (/bin/bash). Typically, this is a shell. Please note that it does not have to be a shell.

/etc/passwd is only used for local users only. To see list of all users, enter:

$ cat /etc/passwd

Day 2


Shell Scripts• The Unix operating system provides a flexible set of simple tools to allow you to perform a wide variety of

system-management, text-processing, and general-purpose tasks. These simple tools can be used in very powerful ways by typing them together programmatically , using “Shell Scripts” or “Shell Programs”

• The UNIX "shell" itself is a user-interface program that accepts commands from the user and executes them. It can also accept the same commands written as a list in a file, along with various other statements that the shell can interpret to provide input, output, decision-making, looping, variable storage, option specification, and so on. This file is a shell program

• They are excellent for system-management tasks but not for general-purpose programming of any sophistication. Shell programs, though generally simple to write, are also tricky to debug and slow in operation


How to write shell script

• Following steps are required to write shell script:– (1) Use any editor like vi or mcedit to write shell script.

– (2) After writing shell script set execute permission for your script as followssyntax:

chmod permission <your-script-name>

– 3) Execute your script assyntax:

bash <your-script-name>sh <your-script-name>./<your-script-name>


First shell script

$ vi first## My first shell script#clearecho “Hello World"

$ vi first## My first shell script#clearecho “Hello World"

After saving the above script, you can run the script as follows:

$ chmod 755 first$ ./first $ chmod 755 first$ ./first

Hello World $ Hello World $

Output


Shell Scripts - Variables

A variable is a place where you can store a value, and then do operations on the value directly. In shell scripts, a variable can contain a collection of letters and/or numbers, as well as pure numbers.

You set a variable by using

variablename="some string here"

OR

variablename=1234

The value of the variable can be accessed, by putting a dollar-sign in front of it.

echo $variablename

OR

echo ${variablename}

If you have JUST a number in a variable, you can do math operations on it.


Shell Scripts – variables and quotes

Quotes are generally used to group things together into a single entity. Single-quotes are literal quotes. Double-quotes can have their contents expanded

echo "$PWD“

prints out your current directory

echo '$PWD‘

prints out the string $PWD

echo $PWDplusthis

prints out NOTHING. There is no such variable "PWDplusthis

echo "$PWD"plusthis

prints out your current directory, and the string "plusthis" immediately following it. This can also be accomplished with the alternate form of using variables,

echo ${PWD}plusthis



Shell Scripts – Conditional Statements

IF

The basic type of condition is "if".

if [ $? -eq 0 ] ; then

echo we are okay

else

echo something failed

fi

IF the variable $? is equal to 0, THEN print out a message. Otherwise (else), print out a different message. "$?" checks the exit status of the last command run. The final 'fi' is required. This is to allow you to group multiple things together. Multiple lines can be written between if and else, or between else and fi, or both.You can even skip the 'else' altogether, if an alternate case is not required.


Shell Scripts – Conditional Statements


Shell Scripts – Loops

WHILE

The basic loop is the 'while' loop; "while" something is true, keep looping. There are two ways to stop the loop. The obvious way is when the 'something' is no longer true. The other way is with a 'break' command.

Example:

keeplooping=1;

while [[ $keeplooping -eq 1 ]] ; do

echo type yes or q to quit

read quitnow

if [[ "$quitnow" = "yes" ]] ; then

keeplooping=0

fi

if [[ "$quitnow" = "q" ]] ; then

break;

fi

done



UNTIL

The other kind of loop in ksh, is 'until'. The difference between them is that 'while' implies looping while something remains true.'until', implies looping until something false, becomes true

Example:

stopnow = 0;

until [[ $stopnow -eq 1 ]] ; do

echo just run this once

stopnow=1;

echo we should not be here again.

done



FOR

A "for loop", is a "limited loop". It loops a specific number of times, to match a specific number of items. Once you start the loop, the number of times you will repeat is fixed. The basic syntax is

for var in one two three ; do

echo $var

done

Whatever name you put in place of 'var', will be updated by each value following "in". So the above loop will print out

one

two

three

But you can also have variables defining the item list. They will be checked ONLY ONCE, when you start the loop. The other way for same is

list="one two three"

for var in $list ; do

done


Shell Scripts – Advanced Variable usage

Braces

Sometimes, you want to immediately follow a variable with a string. This can cause issues if you use the typical grammar of "echo $a" to use a variable, since ksh by default, attempts to do "intelligent" parsing of variable names, but it cannot read your mind.Compare the difference in output in the following lines:

two=2

print one$twothree

print one${two}three

There is no variable named "twothree", so ksh defaults it to an empty value, for the first print line. However, when you use braces to explicitly show ksh {this is the variable name}, it understands that you want the variable named "two", to be expanded in the middle of those other letters.


Shell Scripts – Advanced Variable usage

Arrays arrays can be defined in ksh, unlike old bourne shell. The syntax is as follows:

# This is an OPTIONAL way to quickly null out prior values

set -A array

array[1]="one"

array[2]="two"

array[3]="three"

three=3

print ${array[1]}

print ${array[2]}

print ${array[3]}

print ${array[three]}

.profile features

• .profile is a Unix shell script which is executed for a user every time that user logs in.

• A standard .profile will set important environment variables such as $PATH

• The .profile file will be stored in the users home directory.

• .profile is used by the Bourne, Korn, and Bash shells. The C and TCSH shells use startup files called .login and .cshrc instead

• In addition to .profile, the Korn shell also uses a startup file called .kshrc

• Also, the /etc/profile shared startup file will by executed by all Bourne, Korn, and Bash shells


Sample .profile file


• Following is the snapshot of an existing .profile file

SED and AWK

• SED (stream editor) is a Unix utility which (a) parses text files and (b) implements a programming language which can apply textual transformations to such files. It reads input files line by line (sequentially), applying the operation which has been specified via the command line (or a sed script), and then outputs the line. It was developed by Lee E. McMahon of Bell Labs.

• Usage:– The following example shows a typical use of sed, where the -e option indicates that the sed

expression follows:– sed -e 's/oldstuff/newstuff/g' inputFileName > outputFileName– In many versions, the -e is not required to precede the expression. The s stands for substitute. The g

stands for global, which means that all matching occurrences in the line would be replaced. The regular expression (i.e. pattern) to be searched is placed after the first delimiting symbol (slash here) and the replacement follows the second symbol. Slash is the conventional symbol


The ‘sed’ Command/Language

• Under Unix, sed is often used as a filter in a pipeline:cat file-name | sed -e 's/x/y/g‘

• Several substitutions or other commands can be put together in a file called, for example, commands.sed and then be applied using the -f option to read the commands from the file:

sed -f commands.sed inputFileName > outputFileName

• Besides substitution, other forms of simple processing are possible. For example, the following uses the d command to delete lines that are either blank or only contain spaces:

sed -e '/^ *$/d' inputFileName The caret (^) matches the beginning of the line The dollar sign ($) matches the end of the line The asterisk (*) matches zero or more occurrences of the previous character A bracketed expression delimited by [ and ] matches any of the characters inside the brackets .

Examples “ sed- substitution ”

Print line 1 to 3

substitute (find & replace) “Line" with “Fine" on each line

Replaces only 4th instance in a line

Examples “ sed- substitution ”

substitute “Fine" with “Line" ONLY for lines which contain “Fine"

substitute “Line" with “Mine" EXCEPT for lines which contain “Fine"

delete 2nd line of the file

delete first 2 lines of the file

delete all other lines except first 2

Examples “sed deleting records”

Examples “sed multiple commands”

If you have (multiple) large number of sed commands you can create sed command file and execute with –f option

sed is a powerful utility and got many options which cannot be covered in the presentation. Text file sed1line.txt contains sed commands with different parameters and its purpose. Please refer to sed1line.txt


SED and AWK

• AWK is a programming language that is designed for processing text-based data, either in files or data streams. The name AWK is derived from the family names of its authors — Alfred Aho, Peter Weinberger, and Brian Kernighan.

• AWK is a language for processing files of text. A file is treated as a sequence of records, and by default each line is a record. Each line is broken up into a sequence of fields, so we can think of the first word in a line as the first field, the second word as the second field, and so on. An AWK program is of a sequence of pattern-action statements. AWK reads the input a line at a time. A line is scanned for each pattern in the program, and for each pattern that matches, the associated action is executed." - Alfred V. Aho

• AWK is an example of a programming language that extensively uses the string data type, associative arrays (that is, arrays indexed by key strings), and regular expressions

Structure of AWK programs and Commands

An AWK program is a series of pattern action pairs, written as:

pattern { action }

Pattern is typically an expression and action is a series of commands. Each line of input is tested against all the patterns in turn and the action is executed for each expression that is true. Either the pattern or the action may be omitted. The pattern defaults to matching every line of input. The default action is to print the line of input.

In addition to a simple AWK expression, the pattern can be BEGIN or END causing the action to be executed before or after all lines of input have been read, or pattern1, pattern2 which matches the range of lines of input starting with a line that matches pattern1 up to and including the line that matches pattern2 before again trying to match against pattern1 on future lines.

Commands: AWK commands are the statement that is substituted for action in the examples above. AWK commands can include function calls, variable assignments, calculations, or any combination thereof. AWK contains built-in support for many functions; many more are provided by the various flavors of AWK.

Running an AWK Program

There are several ways to run an Awk program

awk ‘program’ input_file(s)

• program and input files are provided as command-line

Arguments

awk ‘program’

• program is a command-line argument; input is taken from standard input.

awk -f program_file_name input_files

• program is read from a file

Detailed Structure of an AWK Program

• An Awk program consists of:

An optional BEGIN segment

• For processing to execute prior to reading input

pattern - action pairs

• Processing for input data For each pattern matched, the

corresponding action is taken

An optional END segment

• Processing after end of input data

BEGIN{action}

pattern {action}

pattern {action}

.

.

.

pattern { action}

END {action}

Special pattern BEGIN matches before the first input line is read

END matches after the last input line has been read

• This allows for initial and wrap-up processing

BEGIN { print “NAME RATE HOURS”; print “” }

{ print }

END { print “total number of employees is”, NR }

AWK Program Structure…

• Awk patterns are good for selecting specific lines from the input for further processing

• Selection by Comparison

$2 >=5 { print }

• Selection by Computation

$2 * $3 > 50 { printf(“%6.2f for %s\n”, $2 * $3, $1) }

• Selection by Text Content

$1 == “Susie”

• Combinations of Patterns

$2 >= 4 || $3 >= 20

AWK Program Structure… Selection

• Validating data is a common operation

• Awk is excellent at data validation

NF != 3 { print $0, “number of fields not equal to 3” }

$2 < 3.35 { print $0, “rate is below minimum wage” }

$2 > 10 { print $0, “rate exceeds $10 per hour” }

$3 < 0 { print $0, “negative hours worked” }

$3 > 60 { print $0, “too many hours worked” }

AWK Program Structure… Data Validation

• Awk uses the same regular expressions we’ve been using

^ $ beginning of/end of field

. any character

[abcd] character class

[^abcd] negated character class

[a-z] range of characters

(regex1|regex2) alternation

* zero or more occurrences of preceding expression

+ one or more occurrences of preceding expression

? zero or one occurrence of preceding expression

Regular Expressions in AWK

• $0, $1, $2, … ,$NF

NR - Number of records read

FNR - Number of records read from current file

NF - Number of fields in current record

FILENAME - name of current input file

FS - Field separator, space or TAB by default

OFS - Output field separator, space by default

ARGC/ARGV - Argument Count, Argument Value array

Used to get arguments from the command line

AWK Variables

• Awk provides arrays for storing groups of related

data values

# reverse.awk - print input in reverse order by line

BEGIN

{

line[NR] = $0 # remember each line

}

END

{

i = NR # print lines in reverse order

while (i > 0)

{

print line[i]

i = i - 1

}

}

Arrays in AWK

= assignment operator; sets a variable equal to a value or string

== equality operator; returns TRUE is both sides are equal

!= inverse equality operator

&& logical AND

|| logical OR

! logical NOT

<, >, <=, >= relational operators

+, -, /, *, %, ^

Operators in AWK

• Awk provides several control flow statements for making

decisions and writing loops

• If-Else

if (expression is true or non-zero){

statement1

}

else {

statement2

}

where statement1 and/or statement2 can be multiple statements

enclosed in curly braces { }s

the else and associated statement2 are optional

AWK Control Flow Statements

• While

while (expression is true or non-zero) {

statement1

}

• For

for(expression1; expression2; expression3) {

statement1

}

• Do While

do {

statement1

}

while (expression)

AWK Loop Control

• Accessed via built-ins ARGC and ARGV

• ARGC is set to the number of command line arguments

• ARGV[ ] contains each of the arguments

For the command line

awk ‘script’ filename

• ARGC == 2

• ARGV[0] == “awk”

• ARGV[1] == “filename

• the script is not considered an argument

AWK Command Line Arguments

• How do you get input into your awk script other

than on the command line?

• The getline function provides input capabilities

• getline is used to read input from either the current input or from a file or pipe

• getline returns 1 if a record was present, 0 if an end-of-file was encountered, and –1 if some error occurred

AWK getline function

• Printing Every Line

If an action has no pattern, the action is performed for all input lines

{ print } will print all input lines on stdout

{ print $0 } will do the same thing

Printing Certain Fields

Multiple items can be printed on the same output line with a single

print statement

{ print $1, $3 }

Expressions separated by a comma are, by default, separated by a single space when output

Simple Output From AWK

Starting SQL*Plus from Unix:

$sqlplus “user2/user2@BROKER“

Connected SQL

show user USER is "SYSTEM"

@test.sql

exit

Examples

sqlplus -s /nolog <<-EOF

Connect user2/user2@BROKER

@test.sql # exceute sql scripts

exit

-EOF

Date post:	02-Feb-2016
Category:	Documents
Upload:	sivasankar-nandakumar
View:	8 times
Download:	0 times

Unix Training

Documents