Bash Quoting

August 26th, 2009 by Mitch Frazier in

Your rating: None Average: 4.6 (14 votes)

Quoting things in bash is quite simple... until it isn't. I've written scripts where I'd swear 90% of the effort was getting bash to quote things correctly.

The basics of quoting are simple, use single or double quotes when a value contains spaces:

a="hello world"
a='hello world'

But single and double quotes are different. Single quotes don't support any kind of variable substitution or escaping of special characters inside the quoted string. Double quotes, on the other hand, support both:

a="hello \"there\" world"
a='hello "there" world'
a='hello \'there\' world'    # causes an error
a="hello 'there' world"

b="there"
a='hello \"$b\" world'       # a is >>hello \"$b\" world
a="hello \"$b\" world"       # a is >>hello "there" world

One way around the problem that single quotes don't support variable substitution is to quote the individual literal parts of the string and then put the variable substitutions between them:

b='"there"'
a='"hello" '$b' "world"'     # a is: >>"hello" "there" "world"<<

Note that "$b" is not actually inside a quoted part of the string. Since there's no space between the two literal parts of the string and the "$b", bash puts the final result into a single string and then assigns it to a.

One place where quoting problems often occur is when you're dealing with files that have spaces in their names. For example, if we have a file named "file with spaces" and we run the following script:

#/bin/bash

for i in $(find .)
do
    echo $i
done

We don't get what we want:

$ bash t1.sh
.
./file
with
spaces
./t2.sh
./t1.sh
...

One solution is to put the file names into a bash array when the Internal Field Separator is set to just a newline. Then we use the value "${array[@]}" as the list for the for loop.

#/bin/bash

newline='
'

OIFS=$IFS
IFS=$newline
files=($(find .))
IFS=$OIFS

for i in "${files[@]}"
do
    echo $i
done

This produces the correct output:

$ bash t2.sh
.
./file with spaces
./t2.sh
./t1.sh
...

The difference between "${array[*]}" and "${array[@]}" is analagous to the difference between "$*" and "$@", namely that in the later case the result is that each word becomes separately quoted rather than the entire thing being quoted as a single string.

Another option, which avoids special quoting is to just set the Internal Field Separator variable before the for loop runs and reset it as the first statement in the loop body:

#/bin/bash

newline='
'

OIFS=$IFS
IFS=$newline

for i in $(find .)
do
    IFS=$OIFS
    echo $i
done

Another quoting problem is when you have a string which you'd like to break up into separate words but you'd also like to honor any quoted substrings in the string. Consider the string:

a="hello \"there big\" world"

and suppose you'd like to break that into its three parts:

  • hello
  • there big
  • world

If you run this:

#!/bin/bash

a="hello \"there big\" world"

for i in $a
do
    echo $i
done

It produces this:

$ bash t4.sh
hello
"there
big"
world

The trick here is to use a special form of the set command to reset $* (and therefore, $1, $2, etc). My first version is always like this:

#!/bin/bash

a="hello \"there big\" world"
set -- $a

for i in "$@"
do
    echo $i
done

And of course it doesn't work:

$ bash t5.sh
hello
"there
big"
world

Then I usually remember you have to eval the set statement:

#!/bin/bash

a="hello \"there big\" world"
eval set -- $a

for i in "$@"
do
    echo $i
done

At which point it works:

$ bash t6.sh
hello
there big
world

at this point I'm usually tired of quoting things.

__________________________

Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.


Special Magazine Offer -- Free Gift with Subscription
Receive a free digital copy of Linux Journal's System Administration Special Edition as well as instant online access to current and past issues. CLICK HERE for offer

Linux Journal: delivering readers the advice and inspiration they need to get the most out of their Linux systems since 1994.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Very interesting! I always forget $@ and eval. Regarding filenames, I prefer while/read to avoid the case of extra long or too many filenames (because * expands to the full list!):


#-- print one filename by line even with spaces
#-- or too long or too many names
find . | while read FILE ; do
   echo "$FILE"
done

#-- rename filenames changing spaces to dashes
find . | while read FILE ; do
   NEWNAME=$( echo "$FILE" | tr ' ' '-')
   mv -v "$FILE" "$NEWNAME"
done

This way I have the full filename in a variable without too much hassle. To process only the current directory, I add -maxdepth 1 to the find command.

There is a problem with this approach: FILE is only valid inside the while loop, because is run as a separate bash process (Eats some memory, but is slightly faster). When the input comes from a file, we can do this to avoid the extra process:


#-- store the file list for posterior processing
find . > filename.list.txt

#-- show all the names with uncommon characters only
# space is common :) and last char in grep regex
# dash must be the first one to be considered by itself
while read $FILE ; do
   echo "$FILE" | grep "[^-a-zA-Z0-9._/ ]"
done < filename.list.txt

Regards,
Fjor

David LeBlanc's picture

This is sweet. I've been

On November 13th, 2009 David LeBlanc (not verified) says:

This is sweet. I've been looking for a over an hour for a way to treat quoted strings with spaces in them as a single token; many articles I have read come close but none hit the nail on the head like yours did. The magic is all in the "eval set..." Thanks so much!

Prasinos's picture

IFS

On September 15th, 2009 Prasinos (not verified) says:

In bash it is possible to specify the field separator using control characters. The "newline" var in the above examples can be written as:


newline=$'\n'

which is more readable.

Mitch Frazier's picture

Yes it is

On September 16th, 2009 Mitch Frazier says:

Thanks, I didn't know that. I see it now in the man page.

__________________________

Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.

AprilCoolsDay's picture

This is madness

On September 3rd, 2009 AprilCoolsDay (not verified) says:

Does anyone know of a shell without all this quoting complications?

Or do I have to stick to Python and Ruby?

Naresh's picture

In the

On September 2nd, 2009 Naresh (not verified) says:

In the program
#/bin/bash

newline='
'

OIFS=$IFS
IFS=$newline

for i in $(find .)
do
IFS=$OIFS
echo $i
done

why we need IFS=$OIFS in the loop?
Can't we write this once after loop?

Mitch Frazier's picture

IFS

On September 2nd, 2009 Mitch Frazier says:

In this simple case, yes you could put IFS=$OIFS outside the loop, but that might not always be the case if other parts of the loop expect a "normal" IFS value. This example is a bit contrived, but suppose the loop looked like this:

for i in $(find .)
do
    for j in $(echo a b c)
    do
        echo $j
    done
    echo $i
done

Here the variable j only takes on one value, "a b c", because IFS contains only newline, and not tab or space.

__________________________

Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.

Nicholas's picture

Quoting is fine but why not

On September 1st, 2009 Nicholas (not verified) says:

Quoting is fine but why not use python instead?

tehmasp's picture

Can you elaborate on the

On September 10th, 2009 tehmasp says:

Can you elaborate on the Python advantage? Not familiar w/ it.
Thanks!

Josh's picture

Ouch, brain hurts. Funny you

On August 27th, 2009 Josh (not verified) says:

Ouch, brain hurts.

Funny you should bring this up - I was struggling with a super-stupid-simple issue this week that didn't work because of quoting. Thanks for the article!

Tyler Wagner's picture

What about files passed on the command line?

On August 27th, 2009 Tyler Wagner (not verified) says:

Thanks very much for the IFS tip. That seems a good way around this problem.

However, what about passing in filenames on the command line? I'd like to call my script as so:

$ script "file 1 with spaces" "file 2 with spaces"

.. and be able to parse them correctly. Suggestions?

Mitch Frazier's picture

Use "$@"

On August 27th, 2009 Mitch Frazier says:

Something like this should work:

for i in "$@"
do
    echo "$i"
done

The secret is with quoting $@, rather than getting all the command line arguments quoted as a single string (which is what "$*" gets you), when you use "$@" you get each individual argument separately quoted, and if you quoted things on the command line they will continue to be quoted. When I say quoted, I really mean it keeps them together as you specified them and doesn't break them at spaces.

__________________________

Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.

Tyler Wagner's picture

Doh! Thanks!

On August 29th, 2009 Tyler Wagner (not verified) says:

Wow! All this time I've been misusing $* and $@. I've tried various combinations of quoting $@, $*, $1, and $i (in a for loop), and I'd never gotten this to work right before now. Thanks so much!

I finally have a way to quickly, quietly, reliably unzip a bunch of zip files with spaces in their names, from the command line.

Thanks.

Vance's picture

Re: Use "$@"

On August 28th, 2009 Vance (not verified) says:

The following is slightly simpler and does the same thing:

for i
do
    echo "$i"
done

With no in, the for just iterates over all the positional parameters.

Anonymous's picture

Very useful article! I have

On August 27th, 2009 Anonymous (not verified) says:

Very useful article! I have run into these issues too and I these tips help a lot.

Post new comment

Please note that comments may not appear immediately, so there is no need to repost your comment.
The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <pre> <ul> <ol> <li> <dl> <dt> <dd> <i> <b>
  • Lines and paragraphs break automatically.

More information about formatting options

Newsletter

Each week Linux Journal editors will tell you what's hot in the world of Linux. You will receive late breaking news, technical tips and tricks, and links to in-depth stories featured on www.linuxjournal.com.
Sign up for our Email Newsletter

Tech Tip Videos

From the Magazine

December 2009, #188

If last month's Infrastrucuture issue was too "big" for you then try on this month's Embedded issue. Find out how to use Player for programming mobile robots, build a humidity controller for your root cellar, find out how to reduce the boot time of your embedded system, and if you're new to embedded systems find out the basics that go into one. You can also read about the Beagle Board, the Mesh Potato and a spate of other interestingly named items. And along with our regular columns don't miss our new monthly column: Economy Size Geek.


Read this issue