Text Manipulation with sed

Replace text on the fly, without even starting an editor, using this classic tool.

The filter sed can process text from standard input and write its results to standard output. The input can be redirected from a file, and the output also can be redirected to a file using your shell's redirection capabilities. It has hundreds of uses, and once you learn sed, you really would miss it if you lost it.

sed can append lines, remove lines, change lines, rearrange lines, substitute text strings and more. Using sed, you can write simple scripts that can become powerful text manipulating commands.

sed can use regular expressions to define what processing will occur on lines of text and which lines it processes. If you have never seen or used regular expressions before, you may want to become familiar with the basic syntax of regular expressions. In this article, we use a few regular expressions to make sed do some simple text processing.

Ways to Run sed

sed can be run on the command line as follows:

cat sample.txt | sed -e '1,15d'

You can cat the file sample.txt and use the pipe to redirect its output (the lines of text) into the sed command. The -e option to sed tells it to use the next item as the sed command. The d command tells sed to delete lines 1–15 of the input stream, which in this case is the lines read from sample.txt. The rest of the file (if any) appears on standard output, your terminal window, unless redirected elsewhere.

Also, you simply can specify the input file as a command-line argument, so the above sed command also can be written as:

sed -e '1,15d' sample.txt

You also can tell sed to read commands from a script file by using the [-f script-file] option.

sed Command Format

A sed command has this format:

[pattern1][,pattern2][!] command [args]

The pattern1 and pattern2 are optional line ranges. Some commands don't use the patterns, some commands use only one and some can use both to specify a range of lines that the sed command can operate on, as we did in our simple example above.

pattern1 and pattern2 can be numbers, in which case they are treated like line numbers. They can also be a regular expression delimited by slashes (/pattern/). When using regular expression patterns, all lines that match the expression are filtered through the sed command.

If no pattern is specified, the sed command operates on every line of input.

The ! causes sed to operate on every line not included in the pattern range. You can change our example above to be:

cat sample.txt | sed -e '1,15!d'

This command deletes all lines except lines 1–15.

A Few sed Commands

Here are a few basic sed examples. These can all be run right from the command line. Testing and debugging your sed commands individually on the command line before integrating them into a larger script will save you a lot of time that otherwise would be spent debugging the commands from within a running script.

Let's say that you have a file that lists customers called customer.txt. For the following examples, it contains simple lines of text, like this:

Sam Jones
Brenda Jones
Carl Simon
Liz Smith

Let's use some sed commands to manipulate this file. For example, if you want to remove lines containing Carl Simon and update your customer file, you can do the following:


cat customer.txt | \
sed -e '/Carl Simon/d' > customer.txt

The pattern /Carl Simon/ is used by sed as a regular expression and matches every line that has that pattern somewhere on the line. The d command deletes every line that matches the pattern. So, any lines containing Carl Simon are removed from the file.

If you want to perform some type of text substitution on a text file, the s command is probably what you are looking for. It substitutes one text string for another. We tend to use this a lot in our scripts. For example, if Sam Jones calls up and tells you that you should have him listed as Samuel Jones, you can use this command to make the change:


cat customer.txt | \
sed -e 's/Sam Jones/Samuel Jones/' > customer.txt

The s command in sed has three slashes that follow the s. The text between the first and second slash is the pattern you want to match. The text between the second and third slash contains the pattern that you want to substitute for the first pattern. If you wanted all instances of Sam to be Samuel (not just Sam Jones), you could rewrite this example as follows:


cat customer.txt | \
sed -e 's/Sam/Samuel/' > customer.txt

The commands for append (a), replace (c) and insert (i) typically need to have the sed commands specified in a separate script file. For example, say you want to append the line After Brenda right after the line that contains the text Brenda. You can use the a sed command to append the text there. However, you need to put the sed commands in a separate script file, so fire up your favorite editor and create the following sed command file:


#
# sed command file (# are comment lines)
#
# append the line 'After Brenda'
# in this customer file
#
/Brenda/a\
After Brenda

Save this script file as sed1.cmd. Then, to run sed using this script file, use this syntax:


sed -f sed1.cmd customer.txt

You should see the contents of your customer file with the additional line added after the line Brenda Jones. The pattern /Brenda/ (in the sed command file) determine where in the output our appended line appears.

The difference between the append command and the insert command is where the text is added. For the append command, the text is added after the line containing the match. For the insert command, the text is added before the line that contains the match.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

sed deletion help

Waha87's picture

Hello all..
plz help me.

I have a doubt regarding deletion usind sed. We can use following cammand to delete lines between 5 and 10 from filename.txt.

sed '5,10d' filename.txt

I have two variable $startline and $endline. How do use sed command with these variables? when i use
sed '$startline,endlined filename.txt
i am getting errors.
I know this is a basic syntax error, but plz help me to solve this.

how to do this: From a file

Anonymous's picture

how to do this:

From a file containing telephone director, create a new list from this
file that shows surname first, followed by a comma(,) and then the first
name and rest of the line.
ex- gupta, shiv 98797630

unnecessary pipe

the dsc's picture

Often you don't need to pipe a "cat file.txt" to sed, you can sed the file directly.

cs

Anonymous's picture

/home/sphinx/TUTORIAL/53/train/raw/u1078.raw
/home/sphinx/TUTORIAL/53/train/raw/u1079.raw
/home/sphinx/TUTORIAL/53/train/raw/u1080.raw

i have above text in my 777.txt file . what i want is replace /home/sphinx/TUTORIAL/53/raw/ with blank space and .raw also should be replaced with blank space..... pls help me i forget..... i studied long back about sed awk cut ,reg exprs

Assuming by "blank space"

Mitch Frazier's picture

Assuming by "blank space" you mean change them to zero length strings, this should do it:

sed -e 's;/home/sphinx/TUTORIAL/53/train/raw/;;' -e 's/\.raw//' 777.txt

This will output:

u1078
u1079
u1080

Mitch Frazier is an Associate Editor for Linux Journal.

One Flaw

parl's picture

Yes, the problem is as indicated. Evidently he didn't check that his test bed would work. I was wondering if there was a system to generate unique temporary file names so that you would have something like:

#assign temp but unique file name to TEMP$

sed -f work.cmd database.txt > TEMP$

#analyze TEMP$ to ensure it is OK

mv --force TEMP$ databasee.txt

It occurred to me that the date command could be used initially to generate a filename.

For example: the date output of Thu Jun 16 15:45:41 PDT 2005 could be massaged to become 2005Jun16154541PTD.txt, which should be unique.

I imagine someone has done this already, but I haven't looked for it (yet).

parl

Check out the man page on

Anonymous's picture

Check out the man page on mktemp

sed... a cautionary note on re-directions

elhannah's picture

Good introductory article to sed.

One observation though:

I would not recommend users issue command of the form:
$ cat fname.txt |
sed -e s/something/something else/ > fname.txt

In the above example, which is semantically similar to the examples in the article the user is asking the shell to use fname.txt as input and output! Unless the specific commands are designed to handle this (e.g., sort which handles this via the "-o fname" option), asking the semantics of the shell to handle this is very dangerous. Depending on the shell, the version of the shell, etc., the above example may actually give the user an empty result file, a truncated file, or a corrupt file. Instead, I would recommend redirection to some intermediate file, then after inspection and satisfaction with results, copy intermediate file back to original.

sed usage

jignesh's picture

yes this problem exist
when we use the same file as source and destination this problem is seen.
for exp:
sed -e 's/2/3' d3.txt > d3.txt
will return the empty file
this can ba dangerous in live sceanrios
so paly with cautions

:)
jignesh

Use the "-i" option. "Edit in

Anonymous's picture

Use the "-i" option.
"Edit in place".

Isn't for all

Anonymous's picture

-i option is not always present.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix