Date post: | 26-Jan-2015 |
Category: |
Technology |
Upload: | andres-maneiro |
View: | 104 times |
Download: | 0 times |
Master on Free Software
Systems Integration on Free Software
Enrique Ocaña González
Master on Free Software
Index (1/4)
● Administrative tasks– System startup– Users & groups– Task automating– Auditing– Backups
Master on Free Software
Index (2/4)
● Basic commands– Files & directories– Process control– Documentation– Redirections & pipes
Master on Free Software
Index (3/4)
● Script programming– Environment variables– Input/output– Command grouping– Wildcards– Quoting– Control structures
Master on Free Software
Index (4/4)
● Advanced commands– Regular expressions– Grep– Sed– Awk
Master on Free Software
Administrativetasks
Master on Free Software
Index
● System startup● Users & groups● Task automating● Auditing● Backups
Master on Free Software
Introduction
● Administration: security– Two roles
● User: Not concerned about administrative tasks● Administrator (root): Responsible for the proper
operation of the system
– System designed thinking in program and user data separation
– Advice: Only work as root when needed● Avoids unintended damage of the system
Master on Free Software
Introduction
● Administration: transparency– Config and utilities based on plain text files
and scripts– Looking at the configuration, reading
documentation and following what is done by the scripts, it's possible to understand what the system does
– The root of a problem can be found in a simpler way than in closed systems, where the internal system operation isn't known
Master on Free Software
Introduction
● Administration: support– User community at internet– Specialized companies
Master on Free Software
Introduction
● Steps to solve a problem– Read documentation
● /usr/share/doc/package● Project website● FAQs and howtos
– Upgrade the program to the latest version (maybe the bugs have been solved)
– Get in touch with the developers– Return the favour colaborating in projects ;-)
Master on Free Software
Index
● System startup● Users & groups● Task automating● Auditing● Backups
Master on Free Software
System startup
● System startup sequence:
1.BIOS load, POST tasks, boot sequence selection
2.Boot sector load (MBR)
3.Boot loader: grub, lilo, ntldr
4.Kernel load: /boot/vmlinuz
5. init program execution
6.System tasks: /etc/init.d/rcS
7.Runlevel specific tasks: /etc/rcN.d
Master on Free Software
System shutdown
● System shutdown– Init process receives a state change signal
that lets stopping (runlevel 0) or restarting (runlevel 6) the system
● shutdown● halt● init
– Executes the actions associated to the new runlevel in a stablished order
Master on Free Software
System shutdown
● System shutdown stages:
1. Shutdown warning to users and services
2. Execution of actions associated to the system shutdown (runlevel change)
3. Proper filesystem unmounting
4. Computer power off or reset
Master on Free Software
Init and runlevels
● Init process:– Father or all the processes (PID 1)– Has different states or runlevels with services
and actions associated, to be executed when entering into the runlevel or leaving it
– The most common runlevels are:● 0 System shutdown● 6 System restart● 1 Single user mode● 2-5 Multiuser (2 by default)
Master on Free Software
Init and runlevels
● Configuration files:– Master init configuration: /etc/event.d
● Used to be at: /etc/inittab
– Real init scripts placed at /etc/init.d – System bootstrap init links: /etc/rcS.d– Runlevel links: /etc/rcN.d (N=0...6)
Master on Free Software
Init and runlevels
● Symbolic links format:– S|K + order + service name
● Eg. at rc2.d: S89cron > ../init.d/cron● S Start the service when entering the runlevel● K Kill the service when entering the runlevel
● Manual startup:– /etc/init.d/service {start|stop|restart|reload|forcereload}
– /etc/init.d/service shows help
Master on Free Software
Boot manager
● Common boot managers:– lilo– grub
iAdditional documentation: boot_loader_config.pdf
Master on Free Software
Index
● System startup● Users & groups● Task automating● Auditing● Backups
Master on Free Software
Users & groups
● root: the user with the highest privileges● Groups allow granting and grouping
privileges of a set of users● UID, GID: Unique numerical identifiers
– root has UID 0
● Base directory or “home”– /home/user
– /root
Master on Free Software
Users & groups
● Files involved:– /etc/passwd: Information about the user
accounts● Login● Password
– Crypted, “x” when using shadow passwords, “!” when the account is disabled
● UID● Home directory path● Shell● Name, surname, other identification info
Master on Free Software
Users & groups
● Files involved:– /etc/shadow: Like passwd, but only readable
by root and meant to store the passwords
– /etc/group: Info about groups and which users belong to each one
– /etc/gshadow: Non public info about the groups
Master on Free Software
User management
● User suplantationsu user
● User task suplantation (“run as”)sudo user command
● Suplantation access control– File /etc/sudoers (don't touch!)
– visudo
Master on Free Software
User management
● User creation– adduser user
● Creation process
1. Entry in passwd file
2. Entry in shadow file
3. Creation of /home/user
4. Copy of the files in /etc/skel
5. Password setting
Master on Free Software
User management
● User deletion– deluser user
● Creation process
1. Delete entries in passwd and shadow files
2. Finds all the files outside the home directory and deletes them, making a security copy
Master on Free Software
Group management
● Adding a group– addgroup group
● Deleting a group– delgroup group
● Process: Adds/deletes the entries in the files /etc/group and /etc/gshadow
Master on Free Software
Group management
● Adding a user to an existing group– adduser user group
● Deleting a group– adduser user group
● Process: Adds/deletes the entries in the files /etc/group and /etc/gshadow
Master on Free Software
Password changing
● Change the password of a user– passwd Changes the password of the
current user
– passwd user group Executed as root, changes the password of the user specified
● Executed by a normal user, the command asks for the old password
● Executed by root, it doesn't
Master on Free Software
User monitoring
● Each time a user logs into the system, at some logs are recorded:– /var/log/syslog Generic log for all the
system events. Maybe also logged on auth.
– /var/log/wtmp Specific log for all the user logins in the system.
● Binary file
– /var/log/btmp Specific log for all the failed user logins in the system.
Master on Free Software
User monitoring
● Commands– who (or w) Shows who is currently logged
into the system and what are they doing
– last / lastb All the users that logged into the system recently. Lastb shows info of failed attempts
– slay Kicks a user out of the system and kills all her processes
Master on Free Software
Index
● System startup● Users & groups● Task automating● Auditing● Backups
Master on Free Software
Task automating
● cron System utility that allows launching periodic processes (tasks, actions...) in an unattended way, at a scheduled date
● at System utility that allows to program the execution of a process, but without repetition. It allows to defer the unattended process until a date
Master on Free Software
Repeated tasks: cron
● The cron service consists of 2 programs:– crond Program (daemon) offering the
service to the system– crontab Program that allows an user to
manage her tasks● crontab e Edit the scheduled tasks of the
current user● crontab l List the tasks of the current user● crontab r Remove all the tasks of the current
user
Master on Free Software
Repeated tasks: cron
● Crontab edition:– crontab e Opens an editor predefined by
the variable EDITOR, to edit the user crontab● Tip: export EDITOR=nano
– Each crontab line has the following format, always split by spaces
● Minute: Integer number between 0 and 59● Hour: Integer in the range 0-23● Day: Integer in the range 1-31 (must be a valid
day if the month is also specified)
Master on Free Software
Repeated tasks: cron
● Crontab edition:● Month: Integer in the range 1-12● Week_day: Integer in the range 0-7, where 0 and
7 mean Sunday. The short name of the week day can also be used (mon, tue, wed...)
● Command: Command to be executed. The command can be something like “ls /proc >> /tmp/proc” or a command that executes a customized script written by the user
Master on Free Software
Repeated tasks: cron
● Moreover...– A “*” in any field indicates all valid values for
that field (respecting the restricions of the rest of the fields)
– A “–“ between numbers means an interval between those numbers
– A list of values split by “,” means a value list– A “/” means step values. Eg: “*/15” would
mean that the task is executed each 15 minutes.
Master on Free Software
Repeated tasks: cron
● Moreover...– Lines starting with “#” are comments and
aren't had into account– A task is executed in the moment when all
the requirements of the schedule parameters are met
Master on Free Software
Repeated tasks: cron
● Moreover...– Special parameters (instead of the 5 said
fields)● @reboot One time when the computer boots● @yearly Once a year, eq. to 0 0 1 1 *● @daily Each day, eq. to 0 0 * * *● @weekly Each week, eq. to 0 0 * * 0● @monthly Each month, eq. to 0 0 1 * *● @hourly Each hour, eq. to 0 * * * *
Master on Free Software
Repeated tasks: cron
● Examples (crontab -e) # min hour day month weekday command
# Run 5 min after midnight, every day5 0 * * * /bin/daily.sh >> /tmp/out 2>&1
# Run at 2:15pm on the first of every month15 14 1 * * $HOME/bin/monthly
# Run at 10pm on weekdays0 22 * * 15 /usr/local/bin/script.sh
# Run 23 min after midnight, 2am, 4am... everyday23 023/2 * * * script.sh
Master on Free Software
Delayed tasks: at
● at allows to defer the execution of a task. Usage: at datetaskCTRL+d
● Example: $ at 4:45pmat> echo “Time to go”CTRL+d
Master on Free Software
Delayed tasks: at
● atq Shows pending tasks # atq1 20080101 10:35 b peter2 20080215 00:00 c peter5 20080210 01:00 Z root
● at -c 2 Shows all the execution environment for task 2
● atrm tasknum Deletes the specified task from the task queue
Master on Free Software
Delayed tasks: at
● Considerations– Admin can control which users cat use at: /etc/at.allow, at.deny. She can also see the tasks of all users
– at has a queue system to set the priority of the tasks
– batch is a command derived form at that allows executing a task only when the system load goes below of 0.8
Master on Free Software
Index
● System startup● Users & groups● Task automating● Auditing● Backups
Master on Free Software
Log location and format
● At log files are located at /var/log
● Important files– messages System wide messages
– syslog Messages gathered by the log daemon (syslogd)
– debug System logs useful to solve problems
– auth.log Login and logoff attempts
Master on Free Software
Log location and format
● Log format
● Log continuous viewing
tail f logfile
Oct 26 18:17:01 localhost CRON[3384]: (pam_unix) session opened...
Date and timeComputer
Process name and PID
Log line
Master on Free Software
Log location and format
● Log files with a non-standard format– dmesg Last system startup info
– faillog Info about the last time users attempted to log into the system (fail number, max allowed, last fail).
● Viewed using faillog u user
– lastlog List of users and the last time they logged into the system.
● Viewed using lastlog
Master on Free Software
Log location and format
● Log files with a non-standard format– Xfree86.0.log, Xorg.0.log Info from the
X-Window system.● Viewed with standard viewers (tail, etc.)
ihttp://www.debian.org/doc/manuals/users-guide/ch-rsi.en.html
Master on Free Software
Log analysis and tools
● Log analysis allows to detect intrusions in the computer and hardware/software problems
● logcheck Looks for strange log entries and sends them to the administrator via email
ilogcheck.pdf
Master on Free Software
Log configuration
● Log file configuration: /etc/syslog.conf
● Format:– Selector: service.priority [[, service...];
service.priority...]● Services: auth, authpriv, cron, daemon, ftp, kern,
lpr, news, syslog, user, uucp, local0...7, *, none● Priorities: debug, info, notice, warning, crit, alert,
emerg, *, none– =priority: only that priority, not higher ones– !priority: only that priority, not lower ones– !=priority: exactly that priority
Master on Free Software
Log configuration
● Format:– Action
● /.../file: Log to log file (use complete path)● |pipe: Log to named pipe● /dev/console: Log to the console● @hostname: Log to a remote computer● user1, user2, ...: Send to a list of users● *: Send to all the users
Master on Free Software
Log configuration
● Example:
kern.* /var/adm/kernelkern.crit @earthkern.crit /dev/consolekern.info;kern.!err /var/adm/kernelinfomail.*;mail.!=info /var/adm/mail*.* @mars
iman syslog.conf, man syslogd
Master on Free Software
Log configuration
● Log rotation: logs are rotated periodically. They are compressed and deleted by age order
● Example:– /var/log/syslog– /var/log/syslog.0– /var/log/syslog.1.gz– /var/log/syslog.2.gz
Master on Free Software
Log configuration
● Configuración● /etc/logrotate.conf: Main log rotation
configuration● /etc/logrotate.d/*: Specific
configuration for particular services
iman logrotate
Master on Free Software
Index
● System startup● Users & groups● Task automating● Auditing● Backups
Master on Free Software
Backup policy
● Backup levels– Level 0: Complete backup– Level 1: Incremental, changes from level 0– Levels 2..9: Changes from previous level
● Storage media– Tape– CD, DVD– Hard disk– Network
Master on Free Software
Backup policy
● Backup calendar– Diary complete
M T W T F S S0 0 0 0 0 0 0
– Weekly complete, diary level 1 M T W T F S S
1 1 1 1 1 1 0
– Weekly complete, diary level 1 M T W T F S S
1 2 3 4 5 6 0
Backup level
Master on Free Software
Backup policy
● Backup calendar– Monthly hanoi tower
M T W T F S S3 2 5 4 7 6 13 2 5 4 7 6 13 2 5 4 7 6 13 2 5 4 7 6 0
ihttp://www.backupschedule.net/backupSchedules/towerOfHanoi.html
Master on Free Software
Automating and scripts
● Ad-hoc backup: tar ● Saving
tar zcvf file.tgz files...
● Restoringtar zxvf file.tgz [files...]
gzip create verbose file
extract
Master on Free Software
Automating and scripts
● Ad-hoc backup: tar – Other functions
● A: Concatenate some tar files (not tgz!)● d: Find differences between tar/tgz file and files
on disk● delete: Delete files in tar/tgz file● r: Append files at the end of a tar/tgz file● t: List contents● u: Only append files newer than the ones in the
tar/tgz file
Master on Free Software
Automating and scripts
● Other useful commands– cpio: Input/output copy to file
find . print depth | cpio ov > tree.cpio
cpio idv < tree.cpio
– rsync: Remote directory synchronizationrsync aP e ssh user@host:rem_path loc_path
– dd: Exact copy from one device to another (or to a file)dd if=/dev/hda1 of=bak_hda1.bin
Master on Free Software
Automating and scripts
● Other useful commands– mt: Magnetic tape management
mt f /dev/st0 eject
mt retension
mt status
– Example of an ad-hoc backup on tape#!/bin/sh# SYSTEM CONFIG AND DATA BACKUPtar jcvlf exclude=/var/www /var /etc \ | buffer B s 10k > /dev/st0mt offline
Master on Free Software
Automating and scripts
● Simple automating: cron● Advanced backup utilities: Amanda
– Backup automating– Network backup server, multiple client
platforms (even Windows using Samba)– Mail reminder for administrators– Tape dump– Intelligent estimation of backup leven based
on previous statistics
Master on Free Software
Automating and scripts
● Amanda configuration– Multiple configs. Eg: /etc/amanda/DailySet1
– Authorized clients: /etc/amandahosts
– Main config: amanda.conf
– List of disks to be dumped: disklist
– List of tapes to be used: tapelist (don't touch!)
Master on Free Software
Automating and scripts
● Tape labeling for Amanda useamlabel [f] DailySet1 DMP00
Force, for a previously labeled tape
Configuration tobe used
Tape identifier (syntax defined at amanda.conf)
Master on Free Software
root@cesar:~ # cd /root@cesar:/ # amrecover C DailySet1 s localhost.localdomainAMRECOVER Version 2.4.4p3. Contacting server on localhost ...220 localhost.localdomain AMANDA index server (2.4.4p3) ready.200 Access OKSetting restore date to today (20041103)200 Working date set to 20041103....amrecover> listdisk200 List of disk for host localhost.localdomain201 /home201 /etc201 /var200 List of disk for host localhost.localdomainamrecover> setdisk /var200 Disk set to /var.
Automating and scripts
● Recovering: amrecover
The files are going to be extracted in the current directory. It's important to change to the root directory (“/”) of the computer to be restored, in particular if it's a satellite computer (not the one hosting the backup)
Master on Free Software
Automating and scripts
● Recovering: amrecoverroot@cesar:~ # cd /root@cesar:/ # amrecover C DailySet1 s localhost.localdomainAMRECOVER Version 2.4.4p3. Contacting server on localhost ...220 localhost.localdomain AMANDA index server (2.4.4p3) ready.200 Access OKSetting restore date to today (20041103)200 Working date set to 20041103....amrecover> listdisk200 List of disk for host localhost.localdomain201 /home201 /etc201 /var200 List of disk for host localhost.localdomainamrecover> setdisk /var200 Disk set to /var.
NOTE: check access permissions at /etc/amandahosts
Master on Free Software
Automating and scripts
● Recovering: amrecoveramrecover> setdate 20041020200 Working date set to 20041020.amrecover> ls...20041013 spool/20041013 www/amrecover> add wwwAdded dir /www at date 20041013amrecover> listTAPE DailySet5 LEVEL 0 DATE 20040930 /wwwTAPE /home/amand... LEVEL 1 DATE 20041013 /wwwamrecover> extract...
iman amanda , /usr/share/doc/amanda , http://www.amanda.org
Master on Free Software
Basic commands
Master on Free Software
Index
● Basic commands– Files & directories– Process control– Documentation– Redirections & pipes
Master on Free Software
Index
● Basic commands– Files & directories– Process control– Documentation– Redirections & pipes
Master on Free Software
Files basic commands
● I-node: Data structure in a filesystem. Stores basic data about the file, directory or another object in the filesystem– Size and physical location (disk block)– Owner and group– Mode (permissions)– Creation, modification and last access time– Reference counter, accounting how many
hard links are pointing to the inode
Master on Free Software
Files basic commands
● Example:$ ls lisatotal 16766122 4 drwxrxrx 2 root root 4096 .765537 4 drwxrwxrwx 11 root root 4096 ..766125 4 rwrr 2 root root 5 file1766125 4 rwrr 2 root root 5 file2766126 0 lrwxrwxrwx 1 root root 8 file3 > file1
Inode number
Used blocks
Mode
Reference counter
Owner and group
Used bytes
Master on Free Software
Files basic commands
● Permissions (mode)– Permissions
● Read (r): 4 (octal)● Write (w): 2● Execution (x): 1
– Granularity● User (u)● Group (g)● Others (o)● All (a): ugo
750
User: 4(read)+2(write)+1(execution)
Group: 4(read)+1(execution)
Others: (no access)
Master on Free Software
Files basic commands
● Special permissions (4th nibble)– Setuid bit (o+s, 4): Sets the owner of the file
as the effective user when the file is executed– Setgid bit (g+s, 2): Sets the group of the file
as the effective user when executed– Sticky bit (STIcky, Save Text Image) (o+t, 1)
● For files: Keeps the code in swap, just if it's executed later
● For directories: Only the owner of the files inside and the owner of the directory can delete files in the directory
! UNSAFE!!
Master on Free Software
Files basic commands
● Links– Hard links: The file has two entry points at
the same time– Soft links: Reference to the true path of the
file
a b c d...1234...
a b c d...1234...
Master on Free Software
Files basic commands
● Example$ ls lisatotal 16766122 4 drwxrxrx 2 root root 4096 .765537 4 drwxrwxrwx 11 root root 4096 ..766125 4 rwrr 2 root root 5 file1766125 4 rwrr 2 root root 5 file2766126 0 lrwxrwxrwx 1 root root 8 file3 > file1
a b c d...1234...
a b c d...1234...
Master on Free Software
Files basic commands
● Example$ ls lisatotal 16766122 4 drwxrxrx 2 root root 4096 .765537 4 drwxrwxrwx 11 root root 4096 ..766125 4 rwrr 2 root root 5 file1766125 4 rwrr 2 root root 5 file2766126 0 lrwxrwxrwx 1 root root 5 file3 > file1
File the symbolic link is pointing to
Byte size (note that “file1” string takes actually 5 bytes)
Reference counter. file1 and file2 are actually two hard links to the same content, thus its reference counter is 2
Master on Free Software
Files basic commands
● Example$ ls lisatotal 16766122 4 drwxrxrx 2 root root 4096 .765537 4 drwxrwxrwx 11 root root 4096 ..766125 4 rwrr 2 root root 5 file1766125 4 rwrr 2 root root 5 file2766126 0 lrwxrwxrwx 1 root root 5 file3 > file1
The current directory (.) has 2 references: Entry (.) of that directory and entry mydirname in its parent directory.Parent directory (..) has 11 references: the entry tmp in the root directory and the “..” entries in the other 10 directories it has.
Master on Free Software
Files basic commands
● Special files– Devices (high number, low number)
● Blockbrwrw 3, 0 /dev/hda
● Charactercrwrw 9,96 /dev/st0a
– Named pipes (FIFO, socketpairs)srwxrwxrwx /tmp/.X11unix/X0
Master on Free Software
Files basic commands
● Special directories: “.”, “..”
● Hidden files: Start by “.”. Eg: .bashrc
● Brief of the file types:d Directoryl Linkc Char. deviceb Block device
p Pipess Sockets Ordinary
Master on Free Software
Files basic commands
● pwd Shows the current absolute path
● cp Copy filescp [ir] ( file file | dir dir | filelist dir )
● cd Change directorycd path
● mv Move or rename files and directoriesmv file newfile
Master on Free Software
Files basic commands
● mkdir Create directoriesmkdir [p] directory
● ls Shows the content of a directory or info about a file
ls [laR] [dirname]|[filename]
● rmdir Deletes empty directoriesrmdir directory
Master on Free Software
Files basic commands
● rm Deletes filesrm [ir] file1 file2...
● find Searchs files by name, etc.find [searchdir] name “filename”
● cat Shows the content of a file without being able to edit it and without pauses
cat file
Master on Free Software
Files basic commands
● less/more Shows the content of a file without being able to edit it but page by page
less file
● chmod Permission (mode) changechmod [options] mode file
– Examples:● chmod u+srwx,gw,o+rxt prueba● chmod 5755 prueba
Master on Free Software
Files basic commands
● ln Link creation– Hard link: Not allowed among directories
ln sourcefile destinationlink
– Soft linkln s sourcefile destinationlink
Master on Free Software
Index
● Basic commands– Files & directories– Process control– Documentation– Redirections & pipes
Master on Free Software
Processes
● Linux is multitasked– Each instance of a command or program
being executed is a process
● The kernel controls all the processes– Allocates resources (memory, cpu,...)
– Schedules per priority the processes to be executed
– Controls resource access. Each process has an owner
– Process info at /proc
Master on Free Software
Processes
● Important properties characterizing a process– Process identification number (PID)– Parent id number (PPID)– State: Running, Sleep (waiting for a resource)– Owner: Usually the user running the process– Priority: importance in respect to other
processes
Master on Free Software
Process control commands
● ps Lists running processes (snapshot)ps aux
● top Interactively lists running processes. Useful for monitoring. Allows process management
● kill Sends a termination signalkill KILL pid
Shows user, PID, %CPU, %MEM, STAT, time...
Causes the abrupt finishing of the process with PID pid
Master on Free Software
Process control commands
● nice Starts process with given priority. Range from -20 (higher) to 19 (lower)
nice +10 process● snice, renice
● bg (shell) Sends process to background
– Same as & when launching from shell
● fg (shell) Brings a process to foreground. Opposite to bg
Master on Free Software
Process control commands
● jobs (shell) Show processes in background for the current user
● CTRL+z (shell) Sends a STOP signal (bg)
● CTRL+c (shell) Aborts the execution
– ls /proc Numbers are currently running processes
Master on Free Software
Index
● Basic commands– Files & directories– Process control– Documentation– Redirections & pipes
Master on Free Software
Documentation
● program help Shows parameters and a brief summary of the program. Given by the own program.
● man [command|function] Show online help for the program or command
● apropos word Show commands related to the given word
Master on Free Software
Documentation
● info program Shows extended information about a program– Browseable text format
– Organized in a different way than man
● /usr/doc/package/usr/share/doc/package Specific doc about each package: license, usage examples, etc.
Master on Free Software
Index
● Basic commands– Files & directories– Process control– Documentation– Redirections & pipes
Master on Free Software
Shell: redirections & pipes
● In Unix, the shell was traditionally sh. In GNU/Linux: bash, ksh, csh, dash,...
● Each process has at least three I/O ways:– stdin (0): Standard input, usually bound to
the current terminal (keyboard)– stdout (1): Standard output, usually bound
to the terminal (screen)– stderr (2): Standard error, bound to the
terminal (screen) together with stdout
Master on Free Software
Shell: redirections & pipes
● Output redirection– prog > file Overwrites the output file
– prog >> file Appends to output file
● Input redirection– prog < file Reads input from file
● Error redirection– prog 2> file Overwrites the output file
– prog 2>> file Appends
Master on Free Software
Shell: redirections & pipes
● Examples:
$ echo Hello > out.txt$ echo Hello again >> out.txt$ cat out.txtHelloHello again
$ echo See you > out.txt$ echo See you again > out.txt$ cat out.txtSee you again
$ wc < out.txt1 3 14
$ wc xyz.txtwc: xyz.txt: File not found
$ wc xyz.txt 2> err.txt$ cat err.txtwc: xyz.txt: File not found
Master on Free Software
Shell: redirections & pipes
● Pipes: One program sends data through a pipe and another one reads it from the opposite side
● prog1|prog2
● Examples:$ echo See you later | wc
1 3 14
$ tar zcvf . | nc 192.168.1.2 10000$ nc l p 10000 | tar zxvf
Master on Free Software
Scriptprogramming
Master on Free Software
Index
● Script programming– Environment variables– Input/output– Command grouping– Wildcards– Quoting– Control structures
Master on Free Software
Index
● Script programming– Environment variables– Input/output– Command grouping– Wildcards– Quoting– Control structures
Master on Free Software
Defining env. variables
● Variable declarationVARIABLE=”value”
● Variable usage (substitution for its content)$VARIABLE ${VARIABLE}
● Variable deletionunset VARIABLE
Quotes aren't mandatory, but recommended
Don't type spaces
Master on Free Software
Defining env. variables
● Examples:$ S=”_s_”; SEA=”_sea_”
$ echo $SEA; echo ${S}EA
_sea_
_s_EA
Master on Free Software
Defining env. variables
● Variable substitution● $param, ${param} Simple substitution
(for the value of param)● ${param:value} If param is null, use
value● ${param:=value} If param is null, use
value and assign param=value
Master on Free Software
Defining env. variables
● Variable substitution● ${param:?value} If param is null, write
value to stderr. If value isn't specified, writes “param: param null or not set”
● ${param:+value} If param is defined, use value. If not, don't substitute
iman bash (“EXPANSION, Parameter expansion” section)
Master on Free Software
Defining env. variables
● Examples:$ echo “Name is ${NAME:unknown}”Name is unknown
$ NAME=Peter; echo “Name is ${NAME:unknown}”Name is Peter
$ echo “Before: ${N}, Now: ${N:='Peter'}, After: ${N}”Before: , Now: 'Peter', After: 'Peter'
$ unset NAME; echo “Name is ${NAME:?}”bash: NAME: parameter null or not set
$ NAME='Peter'; echo “${NAME:+Exists}”Exists
Master on Free Software
Defining env. variables
● Data read to be stored in variablesread A B C...
– Input fields are usually split by spaces, line feeds or tabs
– This behavior can be modified changing the IFS environment variable (internal field separator)
– Example:$ export IFS=”:”; read A B C; echo “A/B/C”1:new york;z77c6s1/new york/z77c6s
Master on Free Software
Global variables, Language
● Variables assigned in the current shell (locals) aren't inherited by other child shells. They must be exportedexport DISPLAY='localhost:0.0'
● Read only variables: readonly VARIABLE
● Examples:$ NAME=Peter$ NAME=John$ readonly NAME$ NAME=Peterbash: NAME: readonly variable
Master on Free Software
Global variables, Language
● List of defined variables: set
● Most important global variables– DISPLAY X-Window server which the
current process will send windows to
– COLUMNS Number of console columns
– EDITOR Favourite text editor
– EUID Current effective User ID
– HOME Path to the user home directory
Master on Free Software
Global variables, Language
● Most important global variables– HOSTNAME Computer name
– HOSTTYPE Computer architecture
– IFS Internal field separator, input field separator (parameters, etc.)
– LINES Number of console lines
– OSTYPE Operating system
– PAGER Favourite pager (eg: more, less, most)
Master on Free Software
Global variables, Language
● Most important global variables– PATH Executable files search path. Split by
“:”– Example:
PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin
– PPID Parent Process ID
– PS1 Primary prompt string. Supports substitutions
iman bash (“PROMPTING” section)
Master on Free Software
Global variables, Language
● Most important global variables– PS2 Secondary prompt string (when some
lines are used to type a command). Eg: cutting with “\”
– PS3 Prompt string for select command
– PS4 Nesting indicator in debug mode
– PWD Current work path
– SHELL Current command interpreter
Master on Free Software
Global variables, Language
● Most important global variables– TERM Terminal emulator. Eg: linux, vt100,
xterm
– UID Current User ID
– USER Current user
iman bash (“PARAMETERS, Shell variables” section)
Master on Free Software
Global variables, Language
● Language related variables– LANG Locale (lang. localization) to be used.
● Eg: es_ES, es_ES@euro, es_GL, es_ES.UTF-8
– LC_ALL Has preference over LANG and any other LC_ variable to choose a localization
– LC_COLLATE Preference order for path sorting, range expressions, etc.
– LC_TYPE Character interpretation and character class behavior
Master on Free Software
Global variables, Language
● Language related variables– LC_MESSAGES Translation for system
messages and any other string like $”message”
– LC_NUMERIC Numeric format for a particular localization
– LC_TIME Date and time format
iman locale
Master on Free Software
Global variables, Language
● Shell special variables– $0...$9 Shell executable name and 9 first
arguments that were passed
– $# Number of args. passed to the shell
– $* String with all the args. as one single value
– $@ Like $*, but when wrapped by double quotes, wraps each parameter by double quotes
Master on Free Software
Global variables, Language
● Shell special variables– $$ Current script or session PID
– $! PID of the program last sent to background
– $? Exit status of the last program not executed in background
– $ Current options in effect
– ~ Path to the user home directory
iman bash (“PARAMETERS, Special parameters” section)
Master on Free Software
Index
● Script programming– Environment variables– Input/output– Command grouping– Wildcards– Quoting– Control structures
Master on Free Software
Input/output
● Input/output– >, < Stdin and stdout redirection
– | Pipes, redirection of a command stdout to the stdin of the next one
– << Read stdin from the script itself until a mark is found (here documents)
$ cat << END> Hello, I'm $USER.> Good afternoon.ENDHello, I'm root.Good afternoon.
Master on Free Software
Input/output
● Input/output– >> Stdout redirection in append mode
– 0<file, 0<<file Redirection of the input of a descriptor (0) to a file
– 2>file, 2>>file Redirection of the output of a descriptor (2) to a file
– 0<&5 Redirection of the input of a descriptor (0) to the file used by another descriptor (5)
Master on Free Software
Input/output
● Input/output– 2>&1 Redirection of the output of a
descriptor (2) to the file used by another descriptor
– <& Closes stdin
– >& Closes stdout
– script>>file Redirection to a file of the stdout for all the output of the commands of a script
Master on Free Software
Input/output
● Input/output– for ... done >> file Redirection to a
file of the stdout for all the operation of a loop
● Interesting files for input/output– /dev/null Discards everything sent to it.
Useful to avoid displaying a command output
– /dev/tty Our own terminal. Anything sent to it, is displayed on the screen
Master on Free Software
Index
● Script programming– Environment variables– Input/output– Command grouping– Wildcards– Quoting– Control structures
Master on Free Software
Command grouping
● Command grouping– Semicolon (;) Chains commands in the
same line as if they were a single one
– Parenthesis ( ) Executes the code inside in a subshell. Changes in variables don't stay after returning to the current shell
$ TMP=stays; (TMP=you_cant_see_me; echo $TMP); echo $TMP
Typing spaces and ending in ; isn't mandatory
Master on Free Software
Command grouping
● Command grouping– Curly braces { } Executes the code inside
in the current shell
– Double ampersand (&&) Executes the next command only if the previous one returned a true result (equal to 0)
$ TMP=stays; { TMP=you_cant_see_me; echo $TMP; }; echo $TMP
Typing spaces and ending in ; is mandatory
$ ls mydir && echo 'That was the content of mydir'
Master on Free Software
Command grouping
● Command grouping– Double pipe (||) Executes the next
command only if the previous one returned a false result (different from 0)$ ls mydir || echo 'The directory mydir doesn\'t exist'
Master on Free Software
Index
● Script programming– Environment variables– Input/output– Command grouping– Wildcards– Quoting– Control structures
Master on Free Software
Wildcards
● File name matching in the shell: wildcards– ? Matches with one single character
– * Matches zero or more characters
– [abc...] Matches with any of the listed characters
– [!abc...] Matches with all but the listed characters
Master on Free Software
Wildcards
● File name matching in the shell: wildcards– {str1,str2,...} Makes a list of
expressions using one string each timefile.{txt,pdf,rtf}
file.txt file.pdf file.rtf
– \ (escape) Overrides the meaning of any special character, even end of line
Master on Free Software
Index
● Script programming– Environment variables– Input/output– Command grouping– Wildcards– Quoting– Control structures
Master on Free Software
Quoting
● Double quotes (“”) Overrides the special meaning of all the characters but $, ` and \
● Single quotes ('') Overrides the special meaning of all the characters
$ echo “The user is $USER”The user is root
$ echo 'The user is $USER'The user is $USER
Master on Free Software
Quoting
● Back quotes (``) Command substitution. It's executed and output is put in place of the command text
● Backslash (\) Overrides the special meaning of the next character, even inside quotes
$ echo “Today is ###`date`###”Today is ###Thu Jan 10 11:52:26 CET 2008###
$ echo “Double quotes \”are nice\”, aren't they?”Double quotes “are nice”, aren't they?
Master on Free Software
Index
● Script programming– Environment variables– Input/output– Command grouping– Wildcards– Quoting– Control structures
Master on Free Software
Control structures
● Condition evaluation– test 5 = 5
– [ 5 = 5 ]
● The result is stored in $? (errorlevel)– True: 0– False: different from 0
These two commands are the same (see /usr/bin). It's important to keep the space. 5 = 5 are in fact command arguments
Master on Free Software
Control structures
● Possible test conditions– string string isn't null
– n string string isn't null
– string1 = string2 Are equal
– string1 != string2 Are distinct
– eq Numerically equal to (put between the values to be compared)
– ne Not equal
Master on Free Software
Control structures
● Possible test conditions– gt Greater than
– ge Greater or equal
– lt Lower than
– le Lower or equal
– b File is a special block file
– c File is a special character file
– d Is a directory
Master on Free Software
Control structures
● Possible test conditions– f Is an ordinary file
– g Has the setgid bit active
– k Has the sticky bit active
– p Is a named pipe
– r Can be read by the current process
– s Size larger than 0
– t File descriptor is open and linked to a terminal
Master on Free Software
Control structures
● Possible test conditions– u Has setuid bit active
– w Can be written by current process
– x Can be executed by current process
– a AND of previous and next arguments
– o OR of previous and next arguments
iman test
Master on Free Software
Index
● Script programming– Environment variables– Input/output– Command grouping– Wildcards– Quoting– Control structures
Master on Free Software
Control structures
● If conditional structureif [ $n lt 5 ]
then
echo 'lower than'
elseif [ $n gt 5 ]
then
echo 'greater than'
else
echo 'are...'
echo '...equal'
fi
Master on Free Software
Control structures
● Case conditional structurecase $option in
h) echo 'This script has no help'
;;
n) echo 'The computer name is:'
hostname
;;
*) echo 'Default option'
;;
esac
Master on Free Software
Control structures
● For loop (list)for variable in 1 2 3 /tmp/* `ls` list...
do
echo 'One element:'
echo $variable
done
● If for parameters are skipped, positional parameters $1..$9 are used
List formed by single elements, files in a path (even with wildcards expanded to multiple values), output of command execution, etc.
Master on Free Software
Control structures
● For loop (counter)for ((i=0;$i<5;i++))
do
echo 'Number:'
echo $i
done
Value change
Condition
Initialization
Master on Free Software
Control structures
● While loopi=1
while [ $i le 5 ]
do
echo $i
i=`expr $i + 1`
done
Master on Free Software
Control structures
● Until loopi=1
until [ $i gt 5 ]
do
echo $i
i=`expr $i + 1`
done
● Iteration control in loops– break Exits the loop immediately
– continue Jumps to the next iteration
Master on Free Software
Control structures
● Functions– Declaration
union () {
echo 'Joining the arguments'
return “$1 union $2”
}
anotherFunction () { echo 'hello'; }
● Usageunion “one” “two”
echo $?
The space between the function and () isn't mandatory, nor even the { appearing in the same line. Parameters are not declared.
Parameters appear as positional parameters
Result is declared by return and used with $?
Master on Free Software
Control structures
● Functions– Global scope of the variables: variables
declared or changed inside a function stay when the call ends
● Code inclusion: dot (.)#!/bin/sh
. utilities.sh
myUsefulFunction 1 2 3 4
Master on Free Software
Code examples
● Parameter shifting#!/bin/sh# FILE printargs.shwhile [ $# gt 0 ]do echo “$@” shiftdone
$ printargs.sh fly from “New York” to Moscow fly from New York to Moscow from New York to Moscow New York to Moscow to Moscow Moscow
Shifts the positional parameters deleting the first one ($1)
Master on Free Software
Code examples
● Command line parameter processing (using shift)#!/bin/sh# setether: set an Ethernet interface's IP configwhile [ $# gt 1 ]do case $1 in a) ARP=”arp”; shift;; b) BROADCAST=$2; shift 2;; i) IPADDRESS=$2; shift 2;; m) NETMASK=$2; shift 2;; n) NETWORK=$2; shift 2;; *) echo “setether: illegal option: $1”; exit 1;; esacdoneINTERFACE=$1ifconfig $INTERFACE $IPADDRESS netmask $NETMASK \ broadcast $BROADCAST $ARP; route add net $NETWORK
Shifts two positional parameters in a row
Master on Free Software
Code examples
● Command line parameter processing (using getopts)#!/bin/sh# setether: set an Ethernet interface's IP configwhile getopts ab:e:i:m:n: optiondo case $1 in a) ARP=”arp”;; b) BROADCAST=$OPTARG;; i) IPADDRESS=$OPTARG;; m) NETMASK=$OPTARG;; n) NETWORK=$OPTARG;; *) echo “setether: illegal option: $option”; exit 1;; esacdoneINTERFACE=$1ifconfig $INTERFACE $IPADDRESS netmask $NETMASK \ broadcast $BROADCAST $ARP; route add net $NETWORK
The : indicates that this parameter takes an argument
Master on Free Software
Advancedcommands
Master on Free Software
Index
● Advanced commands– Regular expressions– Grep– Sed– Awk
Master on Free Software
Index
● Advanced commands– Regular expressions– Grep– Sed– Awk
Master on Free Software
Regular expressions
● Regex: Powerful text pattern matching expressions
● Use metacharacters to mean repetition, existence or ranges in character patterns
● Common metacharacters– Dot (.) Matches with any character but line
feed
– Asterisk (*) Matches 0 or more appearings of the char. or regex preceeding it
Master on Free Software
Regular expressions
● Common metacharacters– Brackets [] Match any of the char. classes
enclosed by them● Circumflex (^) after the opening bracket
Inverse match● Minus () Specifies a range● Metacharacters loose their meaning inside the
brackets. Eg: [ at the begining of the list
Master on Free Software
Regular expressions
● Common metacharacters– Circumflex (^) Means line begining
– Dollar ($) Means line ending
– Backslash (\) Escape next metacharacter
● Example# grep '^.* 15:[09][09].*kernel.*$' /var/log/messagesSep 17 15:04:13 localhost kernel: eth1: remaining active...Sep 17 15:04:18 localhost kernel: eth1: DSPCFG accepted after...Sep 17 15:04:18 localhost kernel: eth1: link up....
Master on Free Software
Regular expressions
● Metacharacters exclusive to sed, grep, egrep and awk– Slash-brace (/{m,n/}) Matches a range of
appearings of the single character immediately before the expression
– Variants● /{m/} Exactly m repetitions
● /{m,/} At least m repetitions
● /{m,n/} Betwee m and n repetitions
Master on Free Software
Regular expressions
● Metacharacters exclusive to sed, grep, egrep and awk– Plus (+) Matches one or more appearings of
the character or regex before it
– Question mark (?) 0 or 1 appearings
– Pipe (|) Matches the previous expression or the next one
– Parenthesis () Groups regular expressions
Master on Free Software
Regular expressions
● Tip: Remember that some characters are interpreted by the shell (hello|hi) world
● Regex learning & exploring utilities:– kodos– kregexpeditor– visual-regexp
iman -S7 regex
\(hello\|hi\)\ world'(hello|hi) world'
Master on Free Software
Index
● Advanced commands– Regular expressions– Grep– Sed– Awk
Master on Free Software
Grep
● Grep– Outputs the lines from stdin or file(s)
matching a pattern– Only supports basic regexps– Example: Modified files in a local repository
svnstatus | grep v nbproject | grep ^M
Inverse matching
Master on Free Software
Grep
● Egrep– Supports extended syntax
● Rgrep– Supports recursive search– Use -E for extended syntax
Master on Free Software
Index
● Advanced commands– Regular expressions– Grep– Sed– Awk
Master on Free Software
Sed
● Sed: Stream editor for filtering and transforming textsed e script [file]
sed f scriptfile [file]
● Substitutions/source/destination/modifiers
– Example: for SRC in `find type f`do DEST=`echo $SRC | sed e 's/\.txt/\.bak/'` mv $SRC $DESTdone
Master on Free Software
Sed
● Substitution– Useful modifiers
● g Global (multiple) replacement per linesed e 's/\./,/g'
● w file Write pattern to filesed e 's/kernel/&/w kernel.txt' e 's/gconf/&/w gconf.txt' < /var/log/syslog
● p / !p Print/hide substitued lines (use n to hide default output). ! inverts the restrictionls lisa | sed n e 's/total/total/p'
Master on Free Software
Sed
● Substitution– Using parts of the result
● Whole result (&)● Substrings: \(, \), \1, \2...
sed e 's/\([09]*\)\.\([09]*\)/Integer: \1, decimal: \2/'3452345.46657Integer part: 3452345, decimal part: 46657
Master on Free Software
Sed
● Patterns and ranges to apply commands– Line:
sed n e '12 s/src/dest'
– Range of lines:sed n e '12,25 s/src/dest'
sed n e '1,25 p'
sed n e '12,$ s/src/dest/!p'
Master on Free Software
Sed
● Patterns and ranges to apply commands– Matching a pattern:
sed n e '/pattern/ s/src/dest'
– Matching a range between pattern occurences:sed n e '/begin/,/end/ p'
Master on Free Software
Sed
● Command grouping { }sed n e '
/begin/,/end/ {
s/1/A/gp
s/2/B/gp
s/3/C/gp
}
p
'
When the pattern is matched, the block is executed.When not matched, the general command is executed.
Master on Free Software
Sed
● More advanced commands– a Append lines
– i Insert lines
– c Change lines
– = Print line number
iman sed
http://www.grymoire.com/Unix/Sed.htmlhttp://www.panix.com/~elflord/unix/sed.html
Master on Free Software
Index
● Advanced commands– Regular expressions– Grep– Sed– Awk
Master on Free Software
Awk
● Awk: pattern scanning and processing languageawk program [file]
awk f programfile [file]
● Most common syntax– Line matching
awk /regex/ file
Master on Free Software
Awk
● Most common syntax– Matching and processing
awk '
BEGIN {print “Initialization”}
/regex1/ {print $1, $2}
/regex2/ {print $3, $4}
END {print “Final action”}
' file
–
i-th word (like if it was a parameter) in the matched line. $0 is the whole line.
Master on Free Software
Awk
● Most common syntax– Programming structures inside code blocks
{ if ($1==”1”) print “Winner”;else print “Looser”; }
{ x=1; while (x<100) { x=x*2; } }
{ for (i=0;i<10;i++) { print i; }
Master on Free Software
Awk
● Most common syntax– Predefined variables (most important)
● NR Current count of number of input lines
● NF Count of number of words in current line
● FILENAME Current processed file name
iman awk
http://www.vectorsite.net/tsawk.html