Unix day4 v1.3

Tech Mahindra Limited confidential© Tech Mahindra Limited 2008

Unix

Day 4

CONFIDENTIAL© Copyright 2008 Tech Mahindra Limited 2

Objectives At the end of this session, you will be able to:

Read/Write awk scripts Work with compression and archiving commands Understand File Transfer Protocol Understand process management in UNIX


Agenda: Day 4 AWK Scripting

Advanced Commands

FTP Overview

Unix Process Control


AWK Scripting


AWK It is an advance filter with scripting facility A pattern scanning and processing language Named after its creators: Aho, Weinberger and Kernighan It combines pattern matching, comparison, line

decomposition, numerical operations and C-like programming features into one program


AWK A typical awk command has two components,

address/pattern and an action

awk ‘address/pattern { action }’ file/s

Example:$awk ‘/unix/ { print }’ sample This command looks for all the lines/records which are containing pattern “unix” from a sample file and those lines/records is displayed on to the console


AWK The print statement, when used without any field specifiers,

prints entire line If pattern is omitted, the default pattern is the entire file If action is omitted, the default action is to print


AWK AWK uses the special “variable” $0 to indicate the entire

line. It also identifies fields by $1, $2, $3 etc. AWK identifies fields and record in terms of delimiters

(space/tab and newline by default) AWK uses -F option for indicating the field separator AWK command or script can be directly embedded inside a

shell script


AWK$ awk -F”:” ‘/itp/ { print $1, $3, $2 }’ passwd

This command specifies that the field separator in passwdfile is “:” and we want to print first, third and second fields, if the pattern “itp” is found on any line.


AWK: Operators The AWK can handle numbers, both integer and floating

type, and all the relational tests can be handled by AWK

Arithmetic Operators : +,-,*,/,%,++,-- Assignment Operators : =,+=,-=,*=,/= Relational Operators : <, <=,>,>=,==,!= String Comparison : ~, !~ Logical Operators : &&, ||, !

$awk -F”:” ‘$1 == “itp3”||$1 == “itp5” { print $2, $3}’ passwd


AWK: Formatted Printing The output can be printed in a formatted manner

Specifier Meaning %c ASCII Character %d Integer %e Floating Point

(Exponetial Format) %f Floating Point (Fixed

Format) %s String


Examples Let “emp” is a file containing employee records in the

format empno, emp_name, designation, department, doj and salary. The fields are separated by delimeter ‘:’.

Following examples illustrate AWK commands to be executed on this file:

1. awk '/director/ {print}' emp2. awk '/director/' emp3. awk '/director/ {print $0}’ emp4. awk -F "|" '/director/{print}' emp5. awk -F "|" '/director|manager/ {print}' emp6. awk -F "|" '/director/ {print $2,$3,$4}' emp


AWK: Regular Expressions Pattern matching meta-characters

^,$,.,[list], [^list]* : zero or more occurrences of preceding character or group+ : one or more occurrences of the preceding character or group(exp)(exp1| exp2)

awk –F”:” ‘$2 ~ /^Ram/ {print $2}’ custfile awk –F”:” ‘$2 !~ /Ram$/ {print $2}’ custfile


More Examples1. awk -F "|" 'NR==2,NR==4 {print NR,$2,$3}' emp

2. awk -F "|" '/director/ {printf"%3d %-20s %d",NR,$2,$6}' emp

3. awk -F "|" '$3 == "director" || $3 == "manager" {printf"%3d %-20s %d",NR,$2,$6}' emp

4. awk -F "|" '$6 >= 7500 {printf"%3d %-20s %d",NR,$2,$6}' emp


AWK: User Defined Variables AWK allows us to create variables in the command itself

awk -F "|" '$3=="director" && $6>7000 { knt = knt + 1printf"%3d %-20s",knt,$2}' emp


AWK: BEGIN and END Sections A complete AWK can have 3 sections: BEGIN, awk body and

END sections BEGIN section is executed only once and before execution of

body awk body comprises of expression and action and will be

executed for no. of records present in file parameter END section is executed after the execution of body

BEGIN{ print “Wel-Come to awk”} {

expression and action } END{ print “End of awk”}


AWK: Control Structures if Statement:

if ($5 >20000) interest = 0.10 * $5

else interest = 0.05 * $5


AWK: Control Structures while loop:

while (condition){

Statements}

while (i < 10) print $2 * i


AWK: Control Structures for loop:

for (initialization; condition ; increment/decrement){

statements}

for(i=1; i<10; i++) print $2 * i


getline How do you get input into your awk script other than on the

command line?

The getline function provides input capabilities

getline is used to read input from either the current input or from a file or pipe

getline returns 1 if a record was present, 0 if an end-of-file was encountered, and –1 if some error occurred

In this case, awk sets the variable ERRNO to a string describing the error that occurred


getline Function

Use of getline

Expression Sets

With no arguments

getline $0, NF, NR, FNR

Into a variable

getline var var, NR, FNR

From a file getline <"file" $0, NF

Into a variable from a file

getline var <"file"

var

From a pipe "cmd" | getline $0, NF

Into a variable from a pipe

"cmd" | getline var

var


getline with no Arguments#getline1.awk - demonstrate the getline function

BEGIN{

print "What is your first name and major? “

while (getline > 0) print "Hi", $1 ", your major is", $2 "."

}


getline from a File#getline3.awk - demo getline for reading records# from a file

BEGIN{

while (getline < "emp.data" >0) print $0

}


getline from a Pipe#getline4.awk - show using getline with a pipe

BEGIN{

while ("who" | getline) nr++

print "There are", nr, "people logged on.“

}


getline into a Variable from a Pipe #getline5.awk - show using getline with a pipe

BEGIN {

"date" | getline current_time

print "Report printed on " current_time

}


AWK: Built in Variables

NR cumulative no. of records read

FS Input field separator

OFS Output field separator

NF Number of fields in current record

FILENAME Current input file


AWK: Arrays Awk also permits the use of arrays The naming convention is the same as it is for variables,

and, the array does not have to be declared Unlike arrays in other programming language any number or

string in awk may be used as an array index, not just consecutive integers.

Arrays in awk are associative i.e. each array is a collection of pairs: an index, and its corresponding array element value


Assigning Array Elements Array elements can be assigned values just like awk

variables:

array[subscript] = value e.g.arr[1]=“one”arr[“ten”]=10

arr[1.1]=“fractional value”


Array (Contd.)#Awk script to display the total no. of employees and Total Salary being paid #in each department

BEGIN{ FS=":"}

{ emp_count[$3]++ sal_sum[$3]+=$5}END{ printf("%-15s%-15s%-15s\n", "Depatment","Emp_count","Total Salary")

print("--------------------------------------------------------\n");

for ( dept in emp_count) { printf("%15s\t%3d\t\t%0d\n",dept,emp_count[dept],sal_sum[dept])

}}


AWK: Built-in Functions AWK has several built-in functions for arithmetic and string

operation

int(x) returns integer value of xsqrt(x) returns square root of xlength returns length of complete recordlength(x) returns length of xsubstr(s1,m,n) returns portion of string s1 of length n,

starting from position m in str s1


Storing AWK in a File If AWK command spans more than a single line, it is

advisable to write AWK in a file The extension should be preferably .awk to distinguish it

from other files Quotes are not required To run the AWK script, use the command $awk [-F field_delimeter] -f awkfile.awk filename


AWK: Parameter Passing One can pass parameters to AWK from command line The parameters passed from command line will be collected

in ARGV array The count of parameters will be assigned to ARGC ARGV[] and ARGC can be used in BEGIN section


AWK: Parameter Passing The positional parameters have to be placed in single

quotes. This would enable AWK to distinguish between a positional parameter and a field identifier.

Example: $5 > ‘$1’

checks fifth field with the first positional parameter


AWK: Parameter Passing Parameters can be named parameters

Example

awk –F”:” –f sample.awk “empno=1001” emp

Inside the sample.awk, one can refer the parameter value by name “empno”


Reading input from Single/Multiple Files Using getline

BEGIN { system("clear");

printf "Student Code\tStudent Name\tTotal Percentage\n" } { printf "\n%d\t%s\t", $1, $2

val=$1

if(getline < "studentMarks.dat" > 0 ) { if($1 == val)

printf "%15.2f", $2}

} END { printf "\n\nEnd of the Report\n\n" }

The system function gives a call to clear command

Storing the StudentCode of studentMaster.dat in variable val

The geline function reads each time 1 record from the second file studentMarks.dat

Comparing the first field of studentMarks.dat with val i.e. first field of studentMaster.dat and displaying the percentage marks of the student


Advanced Commands and Utilities


Changing the File Ownership The command used to change the file owner is “chown” chown <new_owne> <file>

$ chown TestUser1 datafile

chown <new_owner>:<new_group> <file> $ chown TestUser1:BT99 datafile

The command used to change the group of a file is “chgrp” chgrp <new_group> <file> $ chown BT99 datafile


File Compression – gzip /gunzip Compressing a File:

$ gzip testfileThe command compresses the testfile to testfile.gz

$ gzip –c testfile > testfile.gzThe command makes a copy of testfile and compresses it to testfile.gz

Decompressing a File:$ gunzip testfile.gzThe command decompresses the testfile.gz to testfile


tar Utility The tar command is used for creating an archive of a directory

hierarchy.

tar archives are a handy way of sending a bunch of files (or a program distribution) across the network or posting them on the internet.

Begin by creating a tar archive of the files. Transmit that tar archive over the network or post it online. Untar the files where you want them.

Syntax: tar [–cvfx] <archive_name>.tar <files>

-c Create a new archive-v verbosely list file processed-f use archive file-t list the contents of an archive-x extract files from an archive


tar utilityUsage: Create a tar archive of your home directory and place it in

your working directory: tar –cvf myhome.tar home/

View the contents of the tar archive: tar –tvf myhome.tar

Extract the tar archive to your current working directory: tar –xvf myhome.tar


FTP Overview


FTP Overview File Transfer Protocol (FTP) is a common method of

transferring files between computer systems The netstat command can be used by all the users to check

the services that are running

The example below shows the expected output, there would be no output at all if FTP is not running

$ netstat -a | grep ftp


Connecting to FTP Connection to FTP Server can be done from Windows

command prompt by two ways as mentioned below:

ftp IPAddresse.g. C:\> ftp 10.11.5.208Connected to 10.11.5.208.220 (vsFTPd 2.0.1)User (10.11.5.208:(none)): user1331 Please specify the password.Password:230 Login successful.ftp>

ftp e.g. c:\> ftp ftp> open 10.11.5.208


Transferring a File: FTP To transfer a file from Windows to Unix use the ftp put/send

command. (The file need to be present in the current dir on Windows)

ftp> put testfileor

ftp> send testfile

To transfer a file from Unix to Windows use the ftp get command. (The file need to be present in the current dir on Unix)

ftp> get testfileor

ftp> recv testfile


Transferring Multiple Files: FTP To transfer multiple files from Windows to Unix use the ftp

mput command. (The files need to be present in the current dir on windows)

ftp> mput testfile1 testfile2 testfile3

To transfer multiple files from Unix to Windows use the ftp mget command. (The files need to be present in the current dir on Unix)

ftp> mget testfile1 testfile2 testfile2

Note: All the files will be sent in text mode by default; To send

binary files give the command:ftp>binary


Miscellaneous FTP Commands Help on ftp command

ftp> help or

ftp> ?

Prompt command is by default on and it interacts with the user for all the command. It used toggle from on to off and vice versa

ftp> prompt

Present working directory ftp>pwd

List files on the ftp server ftp> ls

To change directory ftp>cd dirnamme

To make directory ftp>mkdir dirnamme

To change to the dir on Windows ftp> lcd dirname

To remove directory ftp>rmdir dirnamme

Delete file on the ftp server ftp> delete testfile1

Rename file on the ftp server ftp> rename testfile1

prg.sh

To terminate the ftp session ftp> close or ftp> disconnect

To terminate the ftp session and exit

ftp> byeor

ftp> quit


Database Connectivity Connection to mysql database can be done using the echo

command or using the here << document in the script as follows :

eg: echo "use menagerie;select * from pet;" | mysql

Or mysql << EOF>use menagerie;>select * from pet;>EOF



CONFIDENTIAL© Copyright 2008 Tech Mahindra Limited 4905/01/23 CONFIDENTIAL© Copyright 2008 Tech Mahindra Limited 49

ps Each command running on Unix system is termed as a

process ps command shows process status and displays the attribute of

a process

Usage: $ ps

Options $ ps -f --> full option $ ps -f -u itp5 --> gives processes of user itp5 $ ps -a --> all users processes. $ ps -e --> all processes on the system

including system processes


Process Priority Each process has a priority

It decides: Sense of urgency Which process should get execution time first

Priority is denoted by a number from –20 to 19 -20 is the highest priority and 19 is the lowest

05/01/23 CONFIDENTIAL© Copyright 2008 Tech Mahindra Limited 50


& and jobs& To execute any process in background use & at the end of

the command

Usage: <command name> & Example: $ sh test &

jobs To check out jobs currently running in background use the

command jobs

Usage: $ jobs



fg If we want to switch the background job to foreground we

use fg command fg Brings any background job to foreground

Usage: fg [job number] Example: fg 2

The number specified here is the control number listed by jobs command, NOT the Process ID

fg without any parameters takes the most recent task



kill When we need to forcefully finish some process we use this

command

Kill is used to terminate a process. The command uses one or more PIDs as its arguments.

Usage: kill <process id> Example: Kill -9 105 It will terminate job with PID 105

The option –9 indicates sure kill signal.

$ kill $! The system variable $! Stores the PID of the last background

job



bg Once a job has been suspended or stopped, it will not do any

work

If that job is switched to the background, it can continue on its way

Usage: bg [job number] Example: bg 1 bg without any arguments moves the most recent task into the

background



nohup Sometimes we need a job to be running even if we logout

With the command, nohup you can continue to run programs even after you log out

Usage: nohup <command name> Example: nohup sh a.sh



Job scheduling using crontab cron is a unix, solaris utility that allows tasks to be

automatically run in the background at regular intervals by the cron daemon

These tasks are often termed as cron jobs in unix, solaris Crontab (CRON TABle) is a file which contains the schedule

of cron entries to be run and at specified times


Crontab Syntax crontab file has five fields for specifying day, date and time followed by the

command to be run at that interval

* * * * * command to be executed- - - - -| | | | || | | | +----- day of week (0 - 6) (Sunday=0)| | | +------- month (1 - 12)| | +--------- day of month (1 - 31)| +----------- hour (0 - 23)+------------- min (0 - 59)


Crontab examples remove the tmp files from /home/someuser/tmp each day at

6:30 PM

30 18 * * * rm /home/someuser/tmp/*

30 0 1 1,6,12 * -- 00:30 Hrs on 1st of Jan, June & Dec.

0 20 * 10 1-5 --8.00 PM every weekday (Mon-Fri) only in Oct.

0 0 1,10,15 * * -- midnight on 1st ,10th & 15th of month

5,10 0 10 * 1 -- At 12.05,12.10 every Monday & on 10th of every month


SummaryIn this session, we have covered:

AWK Scripting

Advanced Commands

FTP Overview



Thank You

Date post:	27-Jan-2017
Category:	Software
Upload:	xavier-john
View:	236 times
Download:	1 times

Unix day4 v1.3

Software