CIS 191: Linux and Unix

Class 2September 16th, 2015

TIL

• File path on Linux cannot exceed 256 characters

Outline

Review of Last Week

Overview of Basic Networking Tools

Configuration

Bash Operators

Bash Scripting

Standard Input, Output, Error

1. Command Line Arguments2. Standard Input (stdin)3. Standard Output (stdout)4. Standard Error (stderr)

Redirection and Pipes

• command-1 | command-2 pipes command-1’s stdout to command-2’s stdin

• command < file redirects the contents of file to command’s stdin

• command > file redirects command’s stdout to file• command 2> file redirects command’s stderr to file• $(command) treats the standard output of command as

command-line arguments (more on this today)

Command line Arguments and stdout

• echo arg1 arg2 … argn repeats the command-line arguments arg1 arg2 … argn on standard output

• cat file writes the contents of the file to standard output– But don’t overuse this one – remember, there’s usually a better

way!

Quiz

• cp ~cis191/html/resources/summer .• turnin –c cis191 –p quiz1

Outline

Review of Last Week

Overview of Basic Networking Tools

Configuration

Bash Operators

Bash Scripting

The following is a basic overview

• We will have a lecture on networking and the basics of the internet a few lectures down the road.

• I wanted to spend a few minutes this week talking about basic usage for these commands, so that they won’t be a black box

Hostnames

• We’ll talk about this some more later, but a host name is essentially a mapping from a string of characters to a network (IP) address.

http://seas.upenn.edu/~cis191

Root DNS (edu server)

Engineering school

Upenn’s server

ssh

• If you specify a user@hostname, (or an IP address, if you know that instead), you can sign in as a user on a remote machine!– You can authenticate either with a password (what you’ve likely

been doing), or with a your private key (from an RSA public/private key pair)

– We’ll be going over the basics of RSA encryption in a later lecture

• ssh username@hostname

scp – Secure Copy

• You’ve used this one several times in your homework already

• Works just like cp, except that you can specify host names as a way to copy files between machines!– Also works to copy files locally, if you’re into that

• Uses ssh for data transfer, so it offers the same protections as ssh

• scp from to• Typical usage:

– scp hw2.html cis191@eniac.seas.upenn.edu:html/hw/

tar – archive utility

• Use tar to create or extract tar archives• This is useful for compression – files are represented in a

compact way for easy transmission between machines.

• Usage• Extract

– tar –xvf tarball.tar.gz

• Compress– tar –czvf tarball.tar.gz file1 file2 …

wget – retrieve a web page to a file

• The usage on this one is pretty simple!• Just run wget website (make sure to specify the protocol)

– For example: http://google.com says that google.com should be reached using the http protocol (hyper text transfer protocol)

• wget supports recursive downloads (preserves directory structure of the website you are downloading)

• Downloads are placed into a file (index.html, typically) by default– In the directory in which you run wget

• See the man page for more details

curl – retrieve a webpage to stdout

• curl functions pretty much the same as wget– With some expected differences in options (see man)

• With the one major exception that instead of saving to a file, curl dumps the contents of the page to stdout

• This is often more convenient than the extra level of indirection of a file, if you are constructing a pipeline

Outline

Review of Last Week

Overview of Basic Networking Tools

Configuration

Bash Operators

Bash Scripting

Why configuration files?

• Many programs load a file (or files) during initialization– These files tells the loading program how it should behave

• These files are typically prefaced with a period– This indicates to the shell that the file is “hidden”– Hidden files are not displayed by default in GUI navigators or

when invoking ls without the –a option.– Configuration files tend to be hidden by default to reduce clutter

source

• A built-in shell command• Executes a script within the current process

– Normally, executing a program forks off a new process, which runs the program to execution

– Source instead runs commands inside of an already process (useful for configuring an already-running-process!)

• Many *nix implementations source a configuration file after a program has launched!

Editor configuration files

• Vim and emacs both have their own configuration files– .vimrc– .emacs– These allow a user to modify the default behavior of these

editors

• These files are sourced after the editor has launched

.bashrc and .bash_profile

• ~/.profile or ~/.bash_profile– Sourced at login (ssh, opening your virtual machine…)– This is a configuration file that’s used for a login shell– In Ubuntu linux, the only login shell you see if you’re using a GUI

is, well, the actual login screen!– .bash_profile is more specific than .profile

• For bash, it will override .profile if it exists; .profile works with other shells too!

• ~/.bashrc– Configuration that’s applied to a non-login shell– This is what’s going to be read when you open up a terminal

(like xterm or gnome-terminal) in your ubuntu instances– Good place to put configuration for your login shell

.bashrc and .bash_profile

• What you’re putting in here are just bash commands!• A separate line in my .bashrc file is

– fortune | cowsay– That’s how I get the little cow of wisdom whenever I open my

terminal

The OSX trickster strikes again

• Shells you open in OSX are actually all login shells• So, your .bashrc will actually not be read if you have one

on linux– Remember, login shells read the .profile or .bash_profile files…

• If you still want to use your .bashrc, just add the following to your .bash_profile:

if [ -f ~/.bashrc ]; then source ~/.bashrc

fi

alias command

• This is a way to give yourself shortcuts!• For example, I use the following to make “matlab”

launch the interactive terminal version of the program, rather than the heavyweight GUI

alias matlab="/Applications/MATLAB_R2013b.app/bin/matlab -nodesktop -nosplash”

• Can you guys think of how this might be useful?

Outline

Review of Last Week

Overview of Basic Networking Tools

Configuration

Bash Operators

Bash Scripting

$( ) or ` `

• The $(command) construct – Runs a command, then places the output in the command line– Think of it as executing a command in-line– The backtick (`) can be used in place of $()

• This can be useful when you want to run a command, then use the output of that command within another command

$( ) example

$ cat instructionsls

$ $(cat instructions)cautionary.pngcompiler_complaint.pngcommand_line_fu.png real_programmers.pngrtfm.pngsandwich.png

More on $( )

• The usefulness of this construct will become much more apparent once we’ve taken a look at some other bash constructs– Coming right up in a few sections!

• The character $ usually indicates the start of a command-line replacement

Multiple commands in bash

• There are a couple of ways to execute lists of commands• Semicolons

– command-1; command-2– executes command-1 followed by command-2

• &&– command-1 && command-2– executes command-1, then command-2 if command-1 succeeds

• ||– command-1 || command-2– executes command-1, then command-2 if command-1 fails

Success vs failure

• In general, POSIX and the C standard says that if a program is successful, it should return a 0.– And if it fails it should be a non-zero value

• You can use these modes to figure out how a program exited– Though there’s no real standard; you always need to check the

man pages to see what a call might return

Background jobs

• But what about the single &?• command & executes command as a background process

– Background processes can read from standard input, but can’t write to standard output!

• Only one program can occupy the foreground at a time– Well, one process group, but that’s not something we’re going

to cover in this class.– When you type commands into the shell, the shell itself is in the

foreground!

Briefly: Job control

• We’ll hopefully have a more detailed lecture on this stuff down the line

• Notation: ^Z = Ctrl-Z• Recall ^C and ^\ (SIGTERM and SIGQUIT)• ^Z sends a SIGSTOP signal to a job (tells it to stop).

– This effectively pauses a job (and makes it the most recently modified job)

fg, bg, and jobs

• These are tools for interacting with the job queue• jobs lists the current jobs and their states, along with

their [reference id] inside square braces• fg places the most recently modified job in the

foreground• bg places the most recently modified job in the

background• Both fg and bg can take a reference id as an argument,

and they will then operate on that id instead of the most recently modified job

Outline

Review of Last Week

Overview of Basic Networking Tools

Configuration

Bash Operators

Bash Scripting

Notation

• In the following, I’ll preface lines that are being run on the command line with “$”

• So if you see $ echo, that means I’m running echo on the command line (since each line typically starts with a $ in linux/unix)

Nothing new, really

• We’ve been writing shell scripts all along!• Especially in the last homework

– Isn’t it a script if you’re chaining a bunch of commands together via pipes and saving the output via redirects?

• But it can be a real pain to have type out all those commands over and over again…

• Luckily, saving these scripts to a file is not difficult!

Suppose the following

• We have the following in my-script.sh

#!/bin/shecho Hello. I’m a script!

• And to run it

$ sh my-script.shHello. I am a script.

Executable scripts

$ ./my-script.shbash: ./my-script.sh: Permission denied$ chmod u+x my-script.sh$ ./my-script.shHello. I am a script.

Recap

• sh file runs the file parameter as a shell script!– We’re really just calling the ‘sh’ interpreter and passing it our

script as a command line argument!

• If we want to run a script directly, we have to give it the execute permission– Internally , we have to start the file with a shebang

• #!– **The shebang must be the first characters in the file!!**– /bin/sh is the interpreter for the script (if you type #!/bin/sh)– One can also have /bin/bash or /usr/bin/perl…

• Just specifying an interpreter, really

But we can do so much more!

• Shell scripts support variables, foreach constructs, if-else statements, while loops, arithmetic…

• We’ll go over the basics today, and then talk about control structures next week

Shell Variables

• A variable is just a piece of data which is given a name• In the Bourne shell, variables are identified by

alphanumeric names, and they hold string values.• To assign a string to a variable, use the = operator

– str=text– greeting=hello– phrase=“this is a multiword sentence”

• Notes– NO SPACES should be put between the variable name, the

equals sign, or the string it is being assigned– Strings containing spaces need to be escaped (\ ) or put in

quotes

Strings in Shell Scripts

• The bourne shell uses spaces to separate text into tokens• But a space preceded by a backslash is treated as part of

a token!– This is known as escaping a space (\ )– This is similar to how values are escaped in regular expressions,

which we will discuss soon– $ rm the file and $ rm the\ file behave very differently!

• The first version tries to delete two files: the and file• The second version tries to delete a single file named “the file”

Using a shell variable

• To retrieve a value stored in a variable, you need to precede the name with a $. This evaluates the variable.

• Variables that haven’t been assigned (that can be anything!) are treated as holding the empty string.

Shell Variable Examples

$ question=whoami$ echo questionquestion$ echo $questionwhoami$ $questionxiuruo

What’s going on here?

Evaluated Variables are replaced…

• Similar to the $( ) construct that we looked at earlier!• $ $question => $ whoami => xiuruo (that’s me!)• $ echo $question => $ echo whoami => whoami• $ $(echo $question) => ?

Evaluated Variables are replaced…

• Similar to the $( ) construct that we looked at earlier!• $ $question => $ whoami => xiuruo (that’s me!)• $ echo $question => $ echo whoami => whoami• $ $(echo $question) => ?

• So, if you want to print the name of a variable, make sure to use echo! Otherwise the shell will try to evaluate the contents on you.

$ echo $var

Built-in Shell Variables

• These are provided to you by the shell!– $SHELL is the location of the user’s login shell– $HOME is the user’s home directory– $UID is the user’s uid (but only in bash…)

• Can also get this with id -u– $USER is the user’s username– $$ is the shell’s process id

Environment Variables

• There are other ‘built-in’ variables which can be checked by programs running in a shell (the shell’s environment)

• You can view your shell’s environment by running env– Typically one searches through the environment by piping the

output of env into grep

• You can add variables to the environment as well by running the export command– export me = my– env | grep ‘me’

• me = my

• One way to persist environment changes is by adding an export command to your .bashrc or .profile

Feel free to quote me on this

• In bash, there’s a pretty big difference between single and double quotes

• Single quotes will take everything inside as the literal characters, and won’t try any tomfoolery! This of course means they cannot contain a single quote (it would signify the end of the string, and you can’t escape characters within single quotes!)

• Double quotes try to expand certain characters ($, `, \, and !). This means that double quotes will still evaluate your shell variables or other expressions!– So $ echo “$USER” => xiuruo– And $ echo ‘$USER’ => ?