+ All Categories
Home > Documents > Introduction to Linux Command Line

Introduction to Linux Command Line

Date post: 23-Dec-2021
Category:
Upload: others
View: 14 times
Download: 0 times
Share this document with a friend
32
Introduction to Linux Command Line Introduction to Linux Command Line September 13, 2016 Patrick Bills, Research Consultant Institute for Cyber-Enabled Research [email protected]
Transcript
Page 1: Introduction to Linux Command Line

Introduction to Linux Command Line

Introduction to Linux Command Line

September 13, 2016

Patrick Bills,Research Consultant

Institute for Cyber-Enabled Research

[email protected]

Page 2: Introduction to Linux Command Line

Introduction to Linux Command Line

Goal : gain understanding of:● What is Linux & Unix● Know how to get connected to a linux server● The unix 'shell': Using a computer with only

the keyboard● Commands and Utilities● Manipulating Files● Scripting

Getting Started

Page 3: Introduction to Linux Command Line

Introduction to Linux Command Line

How we will work and help you

During exercises we want to know who needs help, and when to move on

Use the green and red sticky notes provided to help me help you.

No sticky => I am workingGreen => I am done and ready to move onRed => I am stuck and need help

After we move on, or start new exercise, remove Green sticky

Page 4: Introduction to Linux Command Line

Introduction to Linux Command Line

Class structure

Get ConnectedTerminal and Shell conceptsWorking with Directories and FilesEnvironment : configuration, settings and variablesEditing files Working with a data fileScripting with shellUsing other scripting environments (Python, Perl, C)Processes (& , fg, bg, ps aux, top)GUI X11 (gedit)

Page 5: Introduction to Linux Command Line

Introduction to Linux Command Line

MSU HPC runs Linux

We'll use the MSU High Performance Computing Cluster for this course.

● consistent experience (Windows, Mac laptop diversity)

● it's a system we know● highly availability● course is pre-requisite

for other HPCC courses

Page 6: Introduction to Linux Command Line

Introduction to Linux Command Line

Get Started

1. Install/identify laptop software to connect2. Connect to a Unix machine with secure shell3. Does your account work? troubleshoot if necessary4. test basic commands

Step 1: Install Software (if you haven't)Windows : install "MobaXterm" free Unix emulation program

When installed, start program and then start New Session

Mac OS X : includes a unix terminal, but install Xquartz for GUIOpen the application /Applications/Utilities/Terminal.app

Page 7: Introduction to Linux Command Line

Introduction to Linux Command Line

Get Connected

Mac or Linux: Terminal

Start Terminal from /Applications/UtilitiesType ssh [email protected]

Enter password - it's hidden and doesn't look like you are typing

Windows:

Start Moba XtermStart new session

Select SSH Enter host = hpcc.msu.eduUser = your netidPort = 22 Click 'connect'

Page 8: Introduction to Linux Command Line

Introduction to Linux Command Line

What are shell commands?

The shell lets you run programs both utilities and programs you write, and capture output of those programs● Shell can be the glue script to bind programs on the system together● Shell language is one tool in your Unix automation toolbox ● Other tools: SED, awk, grep, Perl, Python, Ruby, C, etc● These all make it easy to ask the system to do something.● Unix command can be called from the above scripting languages, and

those scripts can be called from the shell● Unix philosophy : best tool for the job, but allow the tools to work

together

Page 9: Introduction to Linux Command Line

Introduction to Linux Command Line

What is a command?$<command> <argument(s)> <optional file name>

ls = list files ; arguments -l = long format; -a = all files including ‘hidden’ filesA command is a verb. The the thing that takes action. The command is a program that runs.The options are the ‘adverbs’ - qualifiers of those command ‘verbs.’• wc -m dictionary.txt• we count, characterly, the dictionary text• ls -l $HOME• we list, longly, our home’s files

System information: date command$ date$ date -h$ date --help

Using Shell Commands and Utilities to interact with Linux

Page 10: Introduction to Linux Command Line

Introduction to Linux Command Line

Exploring files and folders with 'ls'

Step one, explore your home directory with 'ls' Note when giving command examples, $ at the beginning is the prompt. Please try these commands

$ ls shows files

$ ls -l shows files in long view

$ ls -al shows all files ; what new things do you see?

$ ls /mnt/home lists folders of current users

$ ls /mnt/home/userid enter your own user id here

Now try someone else's folder - what happens?

$ ls /mnt/home/somebody

Page 11: Introduction to Linux Command Line

Introduction to Linux Command Line

About Files and Folders

Linux has single directory 'tree', separated by slash, the top is the 'root' All additional disks are connected on /mnt, 'mounted' No concept of drive letters

Page 12: Introduction to Linux Command Line

Introduction to Linux Command Line

On our system /mnt/home/ are all user home directories folders/mnt/scratch/ are working files/bin and /usr/bin and /opt/software software

1. Use pwd to determine your current point in the tree 2. Use ls -l to see all files in your folder; note it starts with two entries

single dot (. ) short cut representing current folder double dot ( .. ) short cuts representing parent folder

3. cd = change directory. cd .. => go up one4. ~ is shortcut for your home directory5. Files are specified by the "path" through directory tree6. Path can be full ( /mnt/home/billspat/.bashrc ) starting with /7. or relative , from the current location ~/.bashrc

Exploring files and folders

Page 13: Introduction to Linux Command Line

Introduction to Linux Command Line

File Permissions

Linux has method to keep files private and save. permission for user, user groups, and all others. drwxr-xr-x 4 cliff user 1024 Jun 18 09:40 hello.txt

-rw-r--r-- 1 cliff user 767392 Jun 6 14:28 backup.tar.gz

^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^

| | | | | | | | | | |

| | | | | owner group size date time name

| | | | number of links to file or directory contents

| | | permissions for world

| | permissions for members of group

| permissions for owner of file: r=read, w=write, x=execute

type of file: - = normal file, d=directory, l=symbolic link, etc

Compare ls -l ~ and also ls -l /mnt/scratch/billspat/class

Page 14: Introduction to Linux Command Line

Introduction to Linux Command Line

File Permissions continued

User, Group, others have Read, Write or Execute permissions

Read and Write make sense, but execute? Programs must have this to run

Permission are set with the "chmod" command, and ownership set with "chown" See the man pages or

You can only change these for files you are the owner of

Don't grant other/world permission on your home folder. Keep it private.

Page 15: Introduction to Linux Command Line

Introduction to Linux Command Line

Linux includes a built in manual for nearly all commands. Type 'man' followed by the command e.g $man rm

To navigate the man pages use the arrow keys to scroll up and down or use the enter key to advance a line, space bar to advance a page, letter 'u' to go back a page. Use the q key to quit out of the display.The manual pages often include these sections:

● Name--a one line description of what it does.● Synopsis--basic syntax for the command line.● Description--describes the program's functionalities.● Options--lists command line options available for this program.● Examples--examples of some of the options available● See Also--list of related commands.

Note that options can be with single dash "-" or double dash "--"

Learning commands with manual pages

Page 16: Introduction to Linux Command Line

Introduction to Linux Command Line

review the man pages of common file/directory commands

ls mkdir cp pwd cat

● How can you list all files in a folder, including hidden files that start with a dot (.) ?

● Which command would create a folder "class" in your home directory? Can you do that? what are the options for this command?

● What does pwd do?

● The 'cd' command changes directory, but what happens when you examine the manual page with man cd ? Why?

Learning about commands with 'man'

Page 17: Introduction to Linux Command Line

Introduction to Linux Command Line

External Resources to learn Linux and Bash

http://ss64.com/bash/ "An A-Z Index of the Bash command line for Linux." with links to 'man' manual page for each. It includes many commands that you will never use, or are specific to certain versions of linuxhttps://fosswire.com/post/2007/08/unixlinux-command-cheat-sheet/ useful cheatsheet organized by topichttps://cvw.cac.cornell.edu/linux/

Full intro to Linuxhttp://guide.bash.academy

step by step intro with lots of explanatory texthttp://www.grymoire.com/Unix/Awk.html

complex intro to awk

Page 18: Introduction to Linux Command Line

Introduction to Linux Command Line

Specifying Files and Folders Wildcards

Exercise: copy files for this class are in /mnt/scratch/billspat/class to your own class folder in your home directory Which folders are there? How would you copy my 'docs' folder in my class folder to yours?

Method 1: create doc folder in your home, and copy each filemkdir ~/class/doc

cd ~/class/doc

cp /mnt/scratch/billspat/class/doc/bash_cheat_sheet.pdf .

Method 2: use short cutmkdir ~/class/doccp /mnt/scratch/billspat/class/doc/* ~/class/doc/*

Method 3: copy ALL folders all at oncecp -R /mnt/scratch/billspat/class/* ~/class/

Page 19: Introduction to Linux Command Line

Introduction to Linux Command Line

Shell Environment Variables

Shell maintains, and you can set 'variables' that the shell uses for configuration and in your scripts. Variables start with $, and can be seen with echo $VARNAME● Explore common variables with the echo command and list what they

are$HOME $USER $SHELL $PATH

● Combine variables in an echo command to make a greeting. ● Set a variable of your own, then use in the sentenceGREETING=Hello; echo $GREETING, $USER - welcome to $HOME

● Create a shortcut using the "alias" builtin, which has the basic syntaxalias shortcut="command" Note: must surround command in quotes

Page 20: Introduction to Linux Command Line

Introduction to Linux Command Line

$PATH, one of most important variables

BASH looks for programs in each folder listed in $PATH

When you type a program, how does the shell find it? Let's try the which command, it shows the location of programs...

which date # in /bin

which less # in /usr/bin

which matlab # in /opt/software

each of these folders in the path variable.

Page 21: Introduction to Linux Command Line

Introduction to Linux Command Line

Command input/output

Most commands take text as input and output results as textCommon ways to get text in and out

$ ls -l output text to the screen$ ls -l > filelist.txt redirect text into a new file$ ls -l | grep txt pipe text into another command (chaining)

$ ls $HOME data or file as command line option$ wc -l < filelist.txt redirect file as input into command$ cat filelist.txt | wc -l output file, pipe into commandWhy did we have to use 'cat' in the above command ?

Page 22: Introduction to Linux Command Line

Introduction to Linux Command Line

Utilities for Data

wc count words, lines or characters

example: who | wc -l # number of users logged in

grep finds patterns in files or data, returns just matching lines

example: who | grep $USER # ARE YOU LOGGED IN?

awk can extract individual columns from data, among many other things. awk <options> '{ <program> }' filename

example: awk -F ',' '{print $3}' data.csv

using the field separator option, show column 2 of file with commas

uniq reduce list to uniq values

example: who | awk '{print $1}' | uniq

Page 23: Introduction to Linux Command Line

Introduction to Linux Command Line

Utilities for data

sort given a list of items, sort in various waysexample: who | sort

head list only top n lines of filetail list only last n lines of filezip create zip of several filesunzip unzip tar create (tar -c ) or extract (tar -x) "tape archive" filetr transpose characters example: tr ':' ',' < /etc/passwd

Exercise : using the man pages, find out what the default number of lines that head and tail will display, and how to limit those to just one line

Page 24: Introduction to Linux Command Line

Introduction to Linux Command Line

Exercise : find largest file by sorting

Challenge : find the largest file in a folder, for example /mnt/research/common-data/Bio/genomes/hg19

How do you show all files with sizes? Which column is the size in, each column separated by one or more spaces

sort command has a -k option specifies which column to sort on. You want to sort it numerically, and man sort tells the option needed to "compare according to numerical value" See man sort

Can you combine these commands with the pipe |

Page 25: Introduction to Linux Command Line

Introduction to Linux Command Line

Working with data files

Assure you've copied all file/folders into ~/class folder● Can you unzip the file in class/data folder? see man unzip● Challenge : what is the name of the data file in the zip?● how long is the file (how many rows?)● how many columns● can you show just the column names?● can you show only rows for Michigan?

Commands to use : wc , grep

Bonus: use grep and awk to show poll numbers for Obama

Page 26: Introduction to Linux Command Line

Introduction to Linux Command Line

Data file challenge

Answerscd ~/class/data

unzip data.zip

ls

wc -l polls.csv

head -1 polls.csv

head -1 polls.csv

Bonus

cat polls.csv | awk -F ',' '{print $2}'

Page 27: Introduction to Linux Command Line

Introduction to Linux Command Line

Using Nano ; a simple text editor

When you must edit a file directly on the system, need a terminal-only editor, e.g. nano

Start: nano myfile.txtExit: ^x (control + x), then answer 'y' , then press enter

Example command: ^k cut, ^u uncut (paste)

Page 28: Introduction to Linux Command Line

Introduction to Linux Command Line

Edit a file with nano

Confirm you can open the file greet.sh in class/code folder

method 1 nano ~/class/code/greet.sh

method 2cd ~/class/code

nano greet.sh

This is an example shell script. you can execute it by specifying the path and the file ~/class/code/greet.sh

but first must make it "executable" with chmod u+x ~/class/code/greet.sh

#!/bin/bashGREETING=HelloQ=`df -h $HOME | tail -n 1 | awk '{print $4}'`

echo $GREETING, $USER! Your Home dir is $Q full.echo It\'s currently `date +%c`

Page 29: Introduction to Linux Command Line

Introduction to Linux Command Line

Modify shell script

There is another script in the code folder written in python

Exerise: view this program to see what it doesWhat does it do? How do you think you run it?

python ~/class/code/forecast.py 48824

Exercise: add this command to the greet.sh script and see if it runs. Pick any zipcode

Bonus : use shell arguments ( $1 ) instead of hard coded zip code

Page 30: Introduction to Linux Command Line

Introduction to Linux Command Line

Process

Unix is a series of processes

ps list your processps aux list all processestop list user process using the most CPU

press ctrl+c to exit top

run process in the backgroundpython ~/class/code/forecast.py 48824&

ps

Page 31: Introduction to Linux Command Line

Introduction to Linux Command Line

Using GUI

Windows: Moba XTerm works out of the boxMac: must launch Xquartz, then ssh -X [email protected] in the xterm

demo

Launch GUI editor$ gedit bonus try $ gedit&

Open the shell script and editBonus : Launch matlab

Page 32: Introduction to Linux Command Line

Introduction to Linux Command Line

Discussion

Q/A and optional work

Exercise : can you compile the class folder into a zip file to save it?

How will you use Linux?

Compiler C programs?


Recommended