Bash Quoting
August 26th, 2009 by Mitch Frazier in
Quoting things in bash is quite simple... until it isn't. I've written scripts where I'd swear 90% of the effort was getting bash to quote things correctly.
The basics of quoting are simple, use single or double quotes when a value contains spaces:
a="hello world"
a='hello world'
But single and double quotes are different. Single quotes don't support any kind of variable substitution or escaping of special characters inside the quoted string. Double quotes, on the other hand, support both:
a="hello \"there\" world"
a='hello "there" world'
a='hello \'there\' world' # causes an error
a="hello 'there' world"
b="there"
a='hello \"$b\" world' # a is >>hello \"$b\" world
a="hello \"$b\" world" # a is >>hello "there" world
One way around the problem that single quotes don't support variable substitution is to quote the individual literal parts of the string and then put the variable substitutions between them:
b='"there"'
a='"hello" '$b' "world"' # a is: >>"hello" "there" "world"<<
Note that "$b" is not actually inside a quoted part of the string. Since there's no space between the two literal parts of the string and the "$b", bash puts the final result into a single string and then assigns it to a.
One place where quoting problems often occur is when you're dealing with files that have spaces in their names. For example, if we have a file named "file with spaces" and we run the following script:
#/bin/bash
for i in $(find .)
do
echo $i
done
We don't get what we want:
$ bash t1.sh
.
./file
with
spaces
./t2.sh
./t1.sh
...
One solution is to put the file names into a bash array when the Internal Field Separator is set to just a newline. Then we use the value "${array[@]}" as the list for the for loop.
#/bin/bash
newline='
'
OIFS=$IFS
IFS=$newline
files=($(find .))
IFS=$OIFS
for i in "${files[@]}"
do
echo $i
done
This produces the correct output:
$ bash t2.sh
.
./file with spaces
./t2.sh
./t1.sh
...
The difference between "${array[*]}" and "${array[@]}" is analagous to the difference between "$*" and "$@", namely that in the later case the result is that each word becomes separately quoted rather than the entire thing being quoted as a single string.
Another option, which avoids special quoting is to just set the Internal Field Separator variable before the for loop runs and reset it as the first statement in the loop body:
#/bin/bash
newline='
'
OIFS=$IFS
IFS=$newline
for i in $(find .)
do
IFS=$OIFS
echo $i
done
Another quoting problem is when you have a string which you'd like to break up into separate words but you'd also like to honor any quoted substrings in the string. Consider the string:
a="hello \"there big\" world"
and suppose you'd like to break that into its three parts:
- hello
- there big
- world
If you run this:
#!/bin/bash
a="hello \"there big\" world"
for i in $a
do
echo $i
done
It produces this:
$ bash t4.sh
hello
"there
big"
world
The trick here is to use a special form of the set command to reset $* (and therefore, $1, $2, etc). My first version is always like this:
#!/bin/bash
a="hello \"there big\" world"
set -- $a
for i in "$@"
do
echo $i
done
And of course it doesn't work:
$ bash t5.sh
hello
"there
big"
world
Then I usually remember you have to eval the set statement:
#!/bin/bash
a="hello \"there big\" world"
eval set -- $a
for i in "$@"
do
echo $i
done
At which point it works:
$ bash t6.sh
hello
there big
world
at this point I'm usually tired of quoting things.
__________________________Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.
Special Magazine Offer -- Free Gift with Subscription
Receive a free digital copy of Linux Journal's System Administration Special Edition as well as instant online access to current and past issues. CLICK HERE for offer
Linux Journal: delivering readers the advice and inspiration they need to get the most out of their Linux systems since 1994.
Subscribe now!
The Latest
Newsletter
Tech Tip Videos
- Nov-19-09
- Nov-04-09
Recently Popular
From the Magazine
December 2009, #188
If last month's Infrastrucuture issue was too "big" for you then try on this month's Embedded issue. Find out how to use Player for programming mobile robots, build a humidity controller for your root cellar, find out how to reduce the boot time of your embedded system, and if you're new to embedded systems find out the basics that go into one. You can also read about the Beagle Board, the Mesh Potato and a spate of other interestingly named items. And along with our regular columns don't miss our new monthly column: Economy Size Geek.
Delicious
Digg
StumbleUpon
Reddit
Facebook








When there are too long or too many filenames
On November 21st, 2009 Fjor says:
Very interesting! I always forget $@ and eval. Regarding filenames, I prefer while/read to avoid the case of extra long or too many filenames (because * expands to the full list!):
This way I have the full filename in a variable without too much hassle. To process only the current directory, I add -maxdepth 1 to the find command.
There is a problem with this approach: FILE is only valid inside the while loop, because is run as a separate bash process (Eats some memory, but is slightly faster). When the input comes from a file, we can do this to avoid the extra process:
Regards,
Fjor
This is sweet. I've been
On November 13th, 2009 David LeBlanc (not verified) says:
This is sweet. I've been looking for a over an hour for a way to treat quoted strings with spaces in them as a single token; many articles I have read come close but none hit the nail on the head like yours did. The magic is all in the "eval set..." Thanks so much!
IFS
On September 15th, 2009 Prasinos (not verified) says:
In bash it is possible to specify the field separator using control characters. The "newline" var in the above examples can be written as:
newline=$'\n'
which is more readable.
Yes it is
On September 16th, 2009 Mitch Frazier says:
Thanks, I didn't know that. I see it now in the man page.
__________________________Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.
This is madness
On September 3rd, 2009 AprilCoolsDay (not verified) says:
Does anyone know of a shell without all this quoting complications?
Or do I have to stick to Python and Ruby?
In the
On September 2nd, 2009 Naresh (not verified) says:
In the program
#/bin/bash
newline='
'
OIFS=$IFS
IFS=$newline
for i in $(find .)
do
IFS=$OIFS
echo $i
done
why we need IFS=$OIFS in the loop?
Can't we write this once after loop?
IFS
On September 2nd, 2009 Mitch Frazier says:
In this simple case, yes you could put IFS=$OIFS outside the loop, but that might not always be the case if other parts of the loop expect a "normal" IFS value. This example is a bit contrived, but suppose the loop looked like this:
for i in $(find .) do for j in $(echo a b c) do echo $j done echo $i doneHere the variable j only takes on one value, "a b c", because IFS contains only newline, and not tab or space.
__________________________Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.
Quoting is fine but why not
On September 1st, 2009 Nicholas (not verified) says:
Quoting is fine but why not use python instead?
Can you elaborate on the
On September 10th, 2009 tehmasp says:
Can you elaborate on the Python advantage? Not familiar w/ it.
Thanks!
Ouch, brain hurts. Funny you
On August 27th, 2009 Josh (not verified) says:
Ouch, brain hurts.
Funny you should bring this up - I was struggling with a super-stupid-simple issue this week that didn't work because of quoting. Thanks for the article!
What about files passed on the command line?
On August 27th, 2009 Tyler Wagner (not verified) says:
Thanks very much for the IFS tip. That seems a good way around this problem.
However, what about passing in filenames on the command line? I'd like to call my script as so:
$ script "file 1 with spaces" "file 2 with spaces"
.. and be able to parse them correctly. Suggestions?
Use "$@"
On August 27th, 2009 Mitch Frazier says:
Something like this should work:
for i in "$@" do echo "$i" doneThe secret is with quoting $@, rather than getting all the command line arguments quoted as a single string (which is what "$*" gets you), when you use "$@" you get each individual argument separately quoted, and if you quoted things on the command line they will continue to be quoted. When I say quoted, I really mean it keeps them together as you specified them and doesn't break them at spaces.
__________________________Mitch Frazier is an Associate Editor for Linux Journal and the Web Editor for linuxjournal.com.
Doh! Thanks!
On August 29th, 2009 Tyler Wagner (not verified) says:
Wow! All this time I've been misusing $* and $@. I've tried various combinations of quoting $@, $*, $1, and $i (in a for loop), and I'd never gotten this to work right before now. Thanks so much!
I finally have a way to quickly, quietly, reliably unzip a bunch of zip files with spaces in their names, from the command line.
Thanks.
Re: Use "$@"
On August 28th, 2009 Vance (not verified) says:
The following is slightly simpler and does the same thing:
for i do echo "$i" doneWith no in, the for just iterates over all the positional parameters.
Very useful article! I have
On August 27th, 2009 Anonymous (not verified) says:
Very useful article! I have run into these issues too and I these tips help a lot.
Post new comment