Intro to Linux Systems Administration

Systems Administration• Administering the system?

• Keep the system up in a consistent state

• Monitor performance

• Babysit users, make changes on their behalf

• Install, configure, upgrade, maintain

• Backup, restore, disaster recovery

Sysadmins• System administration handled by various

people– Full time dedicated sysadmins on site– Remote services– Generic ‘IT’ personnel– That user that seems to know what they’re doing

• Can be a skill set central to a career path, or a means to an end

Privilege Hierarchy• Want to divide system privilege by account

• First step is file level permissions– Default permissions limit end users in what

configuration files they can read and which programs they can run

• Next level is within system programs– Limit certain functions to only users with

‘elevated’ privileges

The Superuser• By default, one account has elevated

privileges to issue any command, access any file, and perform every function

• Superuser, a.k.a. root– Technically, can change to anything – but don’t

• User and group number 0

The Superuser, cont• Must limit use of root

– Inexperienced users can cause serious harm– Use of root for non-privileged tasks unnecessary

and can be open to attack– Security and privacy violations – root can look at

anyone’s files

• Limit what root can do remotely

• Ensure a strong password

Superuser Privileges• What usually works best is short periods of

superuser privilege, only when necessary

• Obtain privileges, complete task, relenquish privileges

• Most common ways are su and sudo

• Can also use the setuid/setgid method (Ch. 4), but not recommended

su• Short for substitute or switch user• Syntax: su [options] [username]

– If username is omitted, root is assumed

• After issuing command, prompted for that user’s password

• A new shell opened with the privileges of that user

• Once done issuing commands, must type exit

sudo• Allows you to issue a single command as

another user

• Syntax: sudo [options] [-u user] command

• Again, if no user specified, root assumed

• New shell opened with user’s privileges

• Specified command executed

• Shell exited

sudoers• Must configure a user to run commands as

another user when using sudo• Permissions stored in /etc/sudoers• Use utility visudo to edit this file (run as

root)

• Permissions granted to users or groups, to certain commands or all, and with or without password being required

Other permissions models• Some Linux distributions such as Ubuntu

obscure away the root account altogether• By default the end user doesn’t know the root

password– Can’t login as root– Can’t su

• Must rely on sudo (and the graphical gksudo) to obtain privilege, along with ‘Unlock’ functions in GUI

System Operation• Booting the system

• Runlevels

• Modes

• Shutting down the system

Booting the System• Power on, POST, hardware initialization

• Boot device selected by BIOS/user interaction

• Master boot record of boot device read

• Initializes the bootloader– lilo (LInux LOader)– grub (GRand Unified Bootloader)

Booting, cont• Boot loader selects and loads an OS kernel

• Kernel stored as an compiled image file

• Kernel loads modules for hardware and software functions

• Interrupts, device management, memory management, paging

• Last thing kernel does is call init

init• First non-kernel code loaded

• Process number 1

• Acts as parent to all other processes on system

• Handles starting services and programs

• Based on runlevel, runs the appropriate scripts

Runlevels• A set of defined system states that init can bring

the system into (varies on distro)• 0: Halt/shutdown• 1: Single user mode• 2: Multiuser mode• 3: Multiuser mode with networking• 4: Not used• 5: Multiuser mode with networking and GUI• 6: Reboot

Runlevels, cont• On boot, init checks /etc/inittab to see what

runlevel to bring system to

• To change runlevel after boot– telinit runlevel– shutdown/halt/reboot

• Any time the runlevel changes, init consults a set of scripts to determine what to stop/start

Scripts• Init works with run command (rc) scripts

• Found in /etc/rc.d• All scripts housed in /etc/rc.d/init.d• Each script takes a parameter for changing

operation (start/stop/halt/reboot)

• Each runlevel has it’s own directory– /etc/rc.d/rcN.d

Scripts, cont• In each runlevel directory, there are symbolic

links to scripts in /etc/rc.d/init.d• The name of the link is crucial

– Starting with S means start in this runlevel– Starting with K means kill in this runlevel– After S/K, there is an order number

• Start ascending• Kill descending

Notes• What we’ve described is the traditional Linux

init/boot process

• Different distros do things differently– launchd in Mac OS X– Upstart in Ubuntu Linux– Initng in Debian, Gentoo, others

• The classic init is called System V init

Single User Mode• Runlevel 1

• Console only – no terminals

• Very minimal environment

• Some filesystems might not be mounted

• Maintenance of filesystems

• Fixing configuration errors

• Disaster recovery

Multiuser Mode• Runlevels 2-5

• Runlevel 2 allows terminal logins

• Runlevel 3 allows remote terminal logins

• Runlevel 5 enable X11 graphical environment

• Runlevels 3 and 5 are the most common levels for day-to-day operations

Shutting Down the System• Syntax:shutdown [options] time [message]– Time: XX:XX or +X or NOW– -k: don’t really shutdown, just send message– -r: reboot– -h: halt– -c: cancel a shutdown

• halt: calls shutdown –h• reboot: calls shutdown -r

Scheduling• Linux systems uses the Cron system for time-

based job scheduling• Allows users to schedule jobs to run• Allows sysadmins to run jobs and batch

processes• Different distros implement the structures

differently• Most use /etc/crontab as primary set of

instructions• Sometimes other files are used, like /var/spool/cron/*

crontab• Each line schedules a job

• Syntax:* * * * * command

• First field is minutes (0-59)

• Second field is hours (0-23)

• Third is day of the month (1-31)

• Fourth is month of year (1-12)

• Fifth is day of week (0-6, starting with Sun)

Filesystem Management• A Linux installation can be comprised of

many different filesystems

• Each filesystem (except for swap) is connected to the filesystem hierarchy at a specific point in the tree

• This is referred to as the mount point

• A sysadmin uses mount, umount and /etc/fstab to manage these mounts

mount• Syntax (most commonly):mount –t type device directory

• Associates a device (partition, CD-ROM, etc) formatted with a particular type of filesystem with a specified directory in the hierarchy

• Requires root privileges to mount in most cases

• mount with no arguments displays list of mounted filesystems

umount• Syntax:umount directory | device

• Removes that association

• Cannot umount if device is still being accessed (i.e. open files)

• Again, most likely requires root privileges

fstab• For filesystems that should be mounted on

boot every time, put them in /etc/fstab• Basically a tab delimited file that contains the

command line parameters you’d give to mount– Device– Mount point (directory)– FS type– Options (Readonly, attributes, etc)

Creating New Filesystems• First use fdisk device to create a

partition– Similar in function to old fdisk from DOS– Use ? to display commands, p to display partition

info

• Once partition created, must be formatted– mkfs –t type filesystem

• Once formatted, you can mount it

Filesystem Integrity• Filesystem problems? Corrupt files? Forced

into single user mode to fix errors?• fsck• Syntax:fsck [options] –t type filesystem

• Again, usually need root permissions

• Also, filesystem should NOT be mounted while running fsck – can cause damage

Monitoring Disk Usage• du – disk usage on files and directories

• df – reports filesystem utilization

• lsof – list open file handles

• quota – configure and display user quotas– quotactl– quotacheck– quotaon– edquota

Installing Software• The open source movement has provided an

enormous volume of freely available programs

• Two primary methods of installing programs– By source– By package manager

Installing by Source• Download source code

• Usually comes in a compressed tar archive (.tar.gz or similar)

• Extract source code

• Configure the installation (usually ./configure)

• Then compile (make)

• Then copy into filesystem (make install)

Package Managers• There are a wide variety of package managers

available for different Linux distributions

• In turn, there are several different types of packages available for each of these managers

• Packages are an archived version of the source code

• Often tailored to a specific architecture or distribution

RPM• Red Hat Package Manager

• Package format and manager created by Red Hat developers

• Used widely by Red Hat, Red Hat-based distros, and many others

• System maintains a local RPM database to maintain consistency and track installs

RPM, cont• Many different utilities for managing RPMs• rpm: command line package manager for

installing/removing/configuring packages• up2date: command line package manager

that fetches packages from internet and resolves dependencies

• yum, yast: similar to up2date• Many GUI frontends available to these

utilities

deb• Debian package format

• Used in Debian Linux and it’s derivatives such as Ubuntu and Knoppix

• Contains compressed binary data and metadata

• Again, usually specific to a distro and an architecture

deb cont• dpkg: Debian package manager, for

installing/removing/configuring packages

• apt: Advanced Package Tool, for installing and configuring packages from online sources. Also does dependency resolution

• Again, graphical front ends available for each of these

User Administration• User configuration stored in /etc/passwd• File got it’s name because it originally

contained passwords as well– Security problem – too many processes need to

read passwd– A shadow file used now instead (more in a sec)

• Each line contains info for one user

passwdjsmith:x:1001:1001:Joe Smith,Rm27,(234)555-8910,(234)555-0044,email:/home/jsmith:/bin/bash

• First field is username• Second was password – now a dummy char• Third is userid (uid)• Fourth is groupid (gid)• Fifth is GECOS field

– Full name, contact info– Gen. Elec. Comprehensive OS

• Sixth is user’s home directory• Seventh is user’s default shell

passwd, cont• Originally passwd contained a user’s

password information

• How it works– User picks a password– A random number is generated (called the salt)– The salt and the password is passed into a hash

function (a one-way cryptographic algorithm)– The salt and result are stored in ASCII

passwd, cont• Problem – user-level programs need to read passwd– Get user name, location– Home directory, shell

• So passwd was world readable

• So anyone on system could see a user’s salted hash

• It’s encrypted – what’s the big deal???

passwd, cont• Original salt was 12-bit ... 4096 possibilities

• Many early users used bad passwords– Dictionary words

• Even with 1970’s computing, it wouldn’t take very long to try all combinations of salts and passwords through the hash function

• Just wait for a match

• Brute force crack

shadow• Wasn’t acceptable to have passwd world

readable if it contained hashes

• So salted hashes moved to a new file• /etc/shadow• Format similar to passwd, one user per line

• Readable only by root

shadow, contjsmith:$1$CzzxUSse$bKJL9wAns39vlxQlBZ8wd/:13744:0:99999:7:::

• First field is username

• Second is the salted hash or account status– NP or ! or null for blank password– LK or * for locked/disabled account– !! for account with expired password

• Third is days since last password change– Measured from epoch (midnight UTC 1/1/1970)

shadow, cont• Fourth is days until password is eligible to be

changed

• Fifth is days before change is required

• Sixth is days before expiration to warn

• Seventh is days before account expires

• Eighth is days since epoch when account expires

• Ninth is unused/reserved

Adding Users• If you really wanted to, edit /etc/passwd

by hand

• Some distributions have graphical or simplified ways to add users

• Most widely available however is command line utility useradd

Adding Users, cont• Syntax: useradd [options] [-g group] [-d home] \

[-s shell] username

• -g to define user’s initial group

• -d to define user’s home directory

• -s to define user’s default shell

• Other options for expiration, using defaults, etc

Deleting Users• Again, could just hack /etc/passwd

• More elegant:

• Syntax: userdel [-r] username• -r to delete home directory and it’s contents

Modifying Users• Syntax: usermod [options] username• Options are pretty much identical to those of

useradd

• Also, -l to change the user’s login name

• And –G to list additional groups to add user to

Group Management• Group info housed in /etc/group• Similar to user management

• groupadd

• groupdel

• groupmod

Daemons as Users• For the most part, Linux daemons (services)

each run as a unique user account

• Provides additional security by segregating processes and files

• Running daemons as root usually a bad idea

• Accounts usually created automatically and assigned passwords

• Usually disabled from logging into system

Networking• Linux is a powerful networking operating

system

• Much of it developed in tandem with the Internet

• Ability to work as a client, server, or network device– Proxies, firewalls, routers, bridges, etc

Networking, cont• Overall networking usually governed by /etc/rc.d/init.d/network

• Invoked in runlevels 3 and 5 usually

• Network device/interface configurations in either /etc/sysconfig/networking or in /etc/sysconfig/network-scripts

• Can either edit manually, or use utilities to manage

ifconfig• Displays or alters network device configs

• Syntax: ifconfig interface [options]

• With no options, shows interface’s config

• If interface omitted as well, show all configs

• Options include flags, IP address, subnet mask, etc

route• Display or change routing

• In simple configurations, mostly used to set default gateway

• Syntax: route [options] [add/delete] [target]

• With no arguments, show route table

hostname• Used to set/display the computer’s network

name• Depending on what protocols your network

uses, may also need to look at– domainname– dnsdomainname

• Especially important for Internet-accessible systems

• Can be defined in /etc/sysconfig/network

Interfaces• By default, wired ethernet interfaces are

found as ethX, with X starting at 0

• These are aliases to the actual physical adapter and driver

• To enable an interface:– ifup interface

• To disable an interface:– ifdown interface

Interfaces, cont• Other types of interfaces exist

– ppp, slip, atm, etc

• Management of them work similarly

• Wireless interfaces a bit different– Use iwconfig to manage these and display info– Has the additional options for frequency,

encryption, channel, passphrases, etc

Networking• As with most things, GUI tools available

• Similar to TCP/IP configuration in Windows

• More advanced operations (bridging, NAT/IP Masquerading, advanced routing) take a little more configuration

• Default firewall software is iptables or ipchains

Network Shares• Samba SMB/CIFS

• CUPS

• NFS

Kernel Modification• Vast majority of Linux kernel releases

incredibly stable

• New features/improvements

• Bug fixes

• Modules vs. in kernel

• We need to recompile

Kernel Mods, cont• If we just want to upgrade to a newer kernel

release, there are a couple of options

• Can download and install new kernel packages (RPM, deb, etc)

• Pre-compiled, and most package managers do all the work

• Or the manual way …– Necessary to do any real customization

Kernel Compilation• First, need to get kernel source code

• www.kernel.org

• Current mainline branch is 2.6

• For legacy systems/apps, 2.4 is still available

• Usually a tar.gz or tar.bz2

• Copy to either a temp location, or maybe /usr/src/kernel/

Kernel Compilation, cont• Once you have the compressed archive,

uncompress and extract contents

• Should make a directory named after the kernel release– i.e. linux-2.6.31.6/

• Now go into that directory

• Should see lots of directories for different aspects of the system, and a Makefile

Kernel Compilation, cont• Now we need to configure kernel

– Select options– Choose which items should be modules vs. in

kernel itself

• To import in the previous system config– make oldconfig

• The config is stored in the .config file

Kernel Compilation, cont• Want to configure from scratch? Or further

customize?

• A few different methods– make menuconfig (ncurses)– make xconfig (X11 Qt)– make gconfig (X11 Gtk)

• All basically do the same thing – make selections

Kernel Compilation, cont• Once you’ve done the config and saved it,

time to compile• make• Will take a while

• Lots of info will scroll by

• Don’t worry about warnings, it’s cool

• Errors would be bad though

Kernel Compilation, cont• Once kernel itself is compiled, must compile

the kernel modules• make modules• Once that’s done, we need to install the

modules into the correct location in the filesystem

• make modules_install

Kernel Compilation, cont• Now we need to install the kernel into the

right spot• make install• This moves three things to /boot

– The system map (symbol lookup in memory)– The config– The kernel image (vmlinuz)

• vm = virtual memory support (from UNIX days)• z = compressed

Kernel Compilation• Now we have the kernel in place

• But we need the info necessary to launch init

• We need an initial filesystem loaded so that init has what is necessary to load devices and other filesystems (including /)

• So we use a temporary, memory contained filesystem – a RAM disk

Kernel Compilation, cont• So we need to create an initrd – a RAM Disk

for init to work with before the real filesystems is mounted

• So go to /boot• mkinitrd –o initrd.img-<kern-ver>• Makes an image of the necessary filesystem

components for that version of the kernel

Kernel Compilation, cont• Now all the pieces are in place

• One last step – tell the bootloader about it

• Edit /boot/grub/menu.lst

• Basically just copy the block from the current running kernel, change the version info, and you’re done

• In most cases, you can usually instead just issue update-grub, but should still check

Kernel Compilation, cont• Example grub block

title Red Hat Enterprise Linux ES (2.6.9-5.ELsmp)

root (hd0,0) kernel /boot/vmlinuz root=/dev/hdb1 ro initrd /boot/initrd.img-2.6.25

savedefault

boot

Kernel Compilation, cont• Now you can reboot and try it out

• Check the grub menu for the new kernel you installed and select it

• System should boot fine and everything should work

• Panic? Reboot, select old kernel, boot into it

• Retrace your steps, debug kernel, etc