Colloquium - sedv1.0.1

A. Magee

March 26, 2010

1 / 22

Colloquium - sed, v1.0.1

A. Magee

Outline

1 IntroductionWhat does sed offer?When should I use sed?

2 Learning by exampleSample FileProbing for infoEighty-sixing specific linesThe almighty regular expression

3 Sed’s limitations

2 / 22

Colloquium - sed, v1.0.1

A. Magee

Outline

1 IntroductionWhat does sed offer?When should I use sed?

2 Learning by exampleSample FileProbing for infoEighty-sixing specific linesThe almighty regular expression

3 Sed’s limitations

2 / 22

Colloquium - sed, v1.0.1

A. Magee

Outline

1 IntroductionWhat does sed offer?When should I use sed?

2 Learning by exampleSample FileProbing for infoEighty-sixing specific linesThe almighty regular expression

3 Sed’s limitations

2 / 22

Colloquium - sed, v1.0.1

A. Magee

Introduction What?

What does sed offer?

sed is a stream editor.

It operates on a file or stream of characters where a newline characterterminates a line.

It can operate on specific lines that you describe.

It can make changes immediately without a conventional GUI window.

It can make programatic text manipulation quick and painless.

3 / 22

Colloquium - sed, v1.0.1

A. Magee

Introduction What?

What does sed offer?

sed is a stream editor.

It operates on a file or stream of characters where a newline characterterminates a line.

It can operate on specific lines that you describe.

It can make changes immediately without a conventional GUI window.

It can make programatic text manipulation quick and painless.

3 / 22

Colloquium - sed, v1.0.1

A. Magee

Introduction What?

What does sed offer?

sed is a stream editor.

It operates on a file or stream of characters where a newline characterterminates a line.

It can operate on specific lines that you describe.

It can make changes immediately without a conventional GUI window.

It can make programatic text manipulation quick and painless.

3 / 22

Colloquium - sed, v1.0.1

A. Magee

Introduction What?

What does sed offer?

sed is a stream editor.

It operates on a file or stream of characters where a newline characterterminates a line.

It can operate on specific lines that you describe.

It can make changes immediately without a conventional GUI window.

It can make programatic text manipulation quick and painless.

3 / 22

Colloquium - sed, v1.0.1

A. Magee

Introduction When?

When should I use sed?

For repetitive file edits or probes.

For editing a file at precisely defined places.

When you are too lazy (or smart) to open a WYSIWYG editor.

4 / 22

Colloquium - sed, v1.0.1

A. Magee

Introduction When?

When should I use sed?

For repetitive file edits or probes.

For editing a file at precisely defined places.

When you are too lazy (or smart) to open a WYSIWYG editor.

4 / 22

Colloquium - sed, v1.0.1

A. Magee

Introduction When?

When should I use sed?

For repetitive file edits or probes.

For editing a file at precisely defined places.

When you are too lazy (or smart) to open a WYSIWYG editor.

4 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Sample File

A sample file

Here’s a short file that we can play with, let’s call it sample.txt.

This is some text.

1234 Main St. #9Cambridge MA, 12345gold 3 roadsilver 3 streetbronze 1 avenuediamond 5 lane

5 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Printing

Example 1

> sed -n ’4p’ sample.txtCambridge MA, 12345

-n: Suppress default behavior of echoing each line as it is processed.

4: The line number to process (1addr).

p: Print the lines addressed.

6 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Printing

Example 2 & 3

> sed -n ’3,4p’ sample.txt1234 Main St. #9Cambridge MA, 12345

3,4: The line number range to process (1addr,2addr).

> sed -n ’$p’ sample.txtdiamond 5 lane

$: Special character means last line.

7 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Printing

Example 4 & 5

> sed -n "$p" sample.txt

returns nothing: Where is the error, bash or sed?

> num=5> sed -n "${num}p" sample.txtgold 3 road

${num}: bash interrupts this first because of the double quotes.Appendix:Bash notes

8 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Printing

Example 6

> sed -n ’1∼3p’ sample.txtThis is some text.Cambridge MA, 12345bronze 1 avenue

x∼y: Print every y’th line starting at x.

9 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Deleting

Example 7

> sed -i .bak ’2d’ sample.txt

-i: Edit the file in place, an extension can be provided to make abackup (.bak).

Notice we dropped the -n.

d: Delete the addressed lines.

To restore the file: mv sample.txt{.bak,} Appendix:Bash notes

10 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Deleting

Example 7 - cont.

> cat sample.txtThis is some text.1234 Main St. #9Cambridge MA, 12345gold 3 roadsilver 3 streetbronze 1 avenuediamond 5 lane> cat sample.txt.bakThis is some text.

1234 Main St. #9Cambridge MA, 12345gold 3 roadsilver 3 streetbronze 1 avenuediamond 5 lane

11 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Regex

Example 8

> sed ’s/ˆ$//g’ sample.txt

s: Search and replace command.

s/pattern/replacement/options

ˆ: Special character that means beginning of line.

$: Special character that means end of line.

Remember earlier when it meant end of file?Why is it different now?

12 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Regex

Example 8 - cont.

> sed ’s/ˆ$//g’ sample.txt

What does ˆ$ mean?

What happens when there is no replacement argument?

g: global option - Applies the command to every occurrence.

Other options:

# - Only replaces the #th occurrence.p - Prints the replacements made.

13 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Regex

Example 8 - cont.

> sed ’s/ˆ$//g’ sample.txt

What does ˆ$ mean?

What happens when there is no replacement argument?

g: global option - Applies the command to every occurrence.

Other options:

# - Only replaces the #th occurrence.p - Prints the replacements made.

13 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Regex

Example 8 - cont.

> sed ’s/ˆ$//g’ sample.txt

What does ˆ$ mean?

What happens when there is no replacement argument?

g: global option - Applies the command to every occurrence.

Other options:

# - Only replaces the #th occurrence.p - Prints the replacements made.

13 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Regex

Example 9

> sed -E ’s/[0-9]+/###/g’ sample.txtThis is some text.

### Main St. ####Cambridge MA, ###gold ### roadsilver ### streetbronze ### avenuediamond ### lane

-E: Use extended regex (very common, suggest alias). Appendix:Bash notes

[0-9]+: Match one or more digits.

###: Replacement is three octothorpes.

14 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Regex

Example 10

> sed -E ’s/[0-9]/#/g’ sample.txtThis is some text.

#### Main St. ##Cambridge MA, #####gold # roadsilver # streetbronze # avenuediamond # lane

[0-9]: Match one digit.

#: Replacement is a single octothorpe.

Notice the length of the hidden number is the same as the actualnumber.

15 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Regex

Example 11

> sed -n -E ’s/[0-9]/#/gp’ sample.txt#### Main St. ##Cambridge MA, #####gold # roadsilver # streetbronze # avenuediamond # lane

From earlier, the -n suppresses the default printing behavior.

The p option prints lines that were matched with replacements.

Effect is that non-matched lines are not printed.

16 / 22

Colloquium - sed, v1.0.1

A. Magee

Examples Regex

Example 12

> sed -e ’s/ˆ$//’ -e ’s/ /,/g’ sample.txtThis,is,some,text.

1234,Main,St.,#9Cambridge,MA,,12345gold,3,roadsilver,3,streetbronze,1,avenuediamond,5,lane

s/ˆ$//: We’ve seen this before.

-e: Allows more than one command (repeat for each command).Slightly more efficient.

s/ /,/g: What does this do?

17 / 22

Colloquium - sed, v1.0.1

A. Magee

Sed’s limitations

What basic sed can’t do

Non-greedy matching (easily). Recommend ssed (super-sed).

Multiple line matches, or any operation that needs the newlinecharacter.

Arithmetics, don’t even try. Use awk instead.

18 / 22

Colloquium - sed, v1.0.1

A. Magee

Appendix

4 AppendixQuick notes on bashComplex examples

19 / 22

Colloquium - sed, v1.0.1

A. Magee

Appendix Quick notes on bash

Bashiness

See the bash presentation for much more info.From the bash reference manual:

The braces ${...} are required when a parameter is followed by acharacter that is not to be interpreted as part of its name.

Brace expansion is a mechanism by which arbitrary strings may begenerated. ’sample.txt{.bak,}’ expands literally to ’sample.txt.baksample.txt’ because the second argument is null

Aliases are used to keep shortcuts to commands:alias sed=’sed -E’

20 / 22

Colloquium - sed, v1.0.1

A. Magee

Appendix Complex examples

Some harder stuff

> sed -E -i ’s/[[:space:]]+/,/g’ somefile> sed -i ’s/.$//’ somefile> sed -E -i ’/ˆ#/ s/[0-9]+//’ somefile> sed -i ’\:s:[0-9][0-9]*::g’ somefile

21 / 22

Colloquium - sed, v1.0.1

A. Magee

Appendix Complex examples

Harder stuff explained

1 Replace all groups of whitespace with a comma.

2 Convert Windows line endings to Unix line endings. The .$ matches\r\n and replaces the \r with null.

3 /.../ gives addresses matching a pattern, like awk and perl. Deletesthe first number of any line that starts with a #.

4 Starting with \ redefines the argument separator; a colon in this case.

5 Notice that [0-9]+ requires -E, whereas [0-9][0-9]* does not. Theymatch the same expressions.

22 / 22

Colloquium - sed, v1.0.1

A. Magee