Bash Quoting

 in

Quoting things in bash is quite simple... until it isn't. I've written scripts where I'd swear 90% of the effort was getting bash to quote things correctly.

The basics of quoting are simple, use single or double quotes when a value contains spaces:

a="hello world"
a='hello world'

But single and double quotes are different. Single quotes don't support any kind of variable substitution or escaping of special characters inside the quoted string. Double quotes, on the other hand, support both:

a="hello \"there\" world"
a='hello "there" world'
a='hello \'there\' world'    # causes an error
a="hello 'there' world"

b="there"
a='hello \"$b\" world'       # a is >>hello \"$b\" world
a="hello \"$b\" world"       # a is >>hello "there" world

One way around the problem that single quotes don't support variable substitution is to quote the individual literal parts of the string and then put the variable substitutions between them:

b='"there"'
a='"hello" '$b' "world"'     # a is: >>"hello" "there" "world"<<

Note that "$b" is not actually inside a quoted part of the string. Since there's no space between the two literal parts of the string and the "$b", bash puts the final result into a single string and then assigns it to a.

One place where quoting problems often occur is when you're dealing with files that have spaces in their names. For example, if we have a file named "file with spaces" and we run the following script:

#/bin/bash

for i in $(find .)
do
    echo $i
done

We don't get what we want:

$ bash t1.sh
.
./file
with
spaces
./t2.sh
./t1.sh
...

One solution is to put the file names into a bash array when the Internal Field Separator is set to just a newline. Then we use the value "${array[@]}" as the list for the for loop.

#/bin/bash

newline='
'

OIFS=$IFS
IFS=$newline
files=($(find .))
IFS=$OIFS

for i in "${files[@]}"
do
    echo $i
done

This produces the correct output:

$ bash t2.sh
.
./file with spaces
./t2.sh
./t1.sh
...

The difference between "${array[*]}" and "${array[@]}" is analagous to the difference between "$*" and "$@", namely that in the later case the result is that each word becomes separately quoted rather than the entire thing being quoted as a single string.

Another option, which avoids special quoting is to just set the Internal Field Separator variable before the for loop runs and reset it as the first statement in the loop body:

#/bin/bash

newline='
'

OIFS=$IFS
IFS=$newline

for i in $(find .)
do
    IFS=$OIFS
    echo $i
done

Another quoting problem is when you have a string which you'd like to break up into separate words but you'd also like to honor any quoted substrings in the string. Consider the string:

a="hello \"there big\" world"

and suppose you'd like to break that into its three parts:

  • hello
  • there big
  • world

If you run this:

#!/bin/bash

a="hello \"there big\" world"

for i in $a
do
    echo $i
done

It produces this:

$ bash t4.sh
hello
"there
big"
world

The trick here is to use a special form of the set command to reset $* (and therefore, $1, $2, etc). My first version is always like this:

#!/bin/bash

a="hello \"there big\" world"
set -- $a

for i in "$@"
do
    echo $i
done

And of course it doesn't work:

$ bash t5.sh
hello
"there
big"
world

Then I usually remember you have to eval the set statement:

#!/bin/bash

a="hello \"there big\" world"
eval set -- $a

for i in "$@"
do
    echo $i
done

At which point it works:

$ bash t6.sh
hello
there big
world

at this point I'm usually tired of quoting things.

______________________

Mitch Frazier is an Associate Editor for Linux Journal.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

When there are too long or too many filenames

Fjor's picture

Very interesting! I always forget $@ and eval. Regarding filenames, I prefer while/read to avoid the case of extra long or too many filenames (because * expands to the full list!):


#-- print one filename by line even with spaces
#-- or too long or too many names
find . | while read FILE ; do
   echo "$FILE"
done

#-- rename filenames changing spaces to dashes
find . | while read FILE ; do
   NEWNAME=$( echo "$FILE" | tr ' ' '-')
   mv -v "$FILE" "$NEWNAME"
done

This way I have the full filename in a variable without too much hassle. To process only the current directory, I add -maxdepth 1 to the find command.

There is a problem with this approach: FILE is only valid inside the while loop, because is run as a separate bash process (Eats some memory, but is slightly faster). When the input comes from a file, we can do this to avoid the extra process:


#-- store the file list for posterior processing
find . > filename.list.txt

#-- show all the names with uncommon characters only
# space is common :) and last char in grep regex
# dash must be the first one to be considered by itself
while read $FILE ; do
   echo "$FILE" | grep "[^-a-zA-Z0-9._/ ]"
done < filename.list.txt

Regards,
Fjor

This is sweet. I've been

David LeBlanc's picture

This is sweet. I've been looking for a over an hour for a way to treat quoted strings with spaces in them as a single token; many articles I have read come close but none hit the nail on the head like yours did. The magic is all in the "eval set..." Thanks so much!

IFS

Prasinos's picture

In bash it is possible to specify the field separator using control characters. The "newline" var in the above examples can be written as:


newline=$'\n'

which is more readable.

Yes it is

Mitch Frazier's picture

Thanks, I didn't know that. I see it now in the man page.

Mitch Frazier is an Associate Editor for Linux Journal.

This is madness

AprilCoolsDay's picture

Does anyone know of a shell without all this quoting complications?

Or do I have to stick to Python and Ruby?

In the

Naresh's picture

In the program
#/bin/bash

newline='
'

OIFS=$IFS
IFS=$newline

for i in $(find .)
do
IFS=$OIFS
echo $i
done

why we need IFS=$OIFS in the loop?
Can't we write this once after loop?

IFS

Mitch Frazier's picture

In this simple case, yes you could put IFS=$OIFS outside the loop, but that might not always be the case if other parts of the loop expect a "normal" IFS value. This example is a bit contrived, but suppose the loop looked like this:

for i in $(find .)
do
    for j in $(echo a b c)
    do
        echo $j
    done
    echo $i
done

Here the variable j only takes on one value, "a b c", because IFS contains only newline, and not tab or space.

Mitch Frazier is an Associate Editor for Linux Journal.

Quoting is fine but why not

Nicholas's picture

Quoting is fine but why not use python instead?

Can you elaborate on the

tehmasp's picture

Can you elaborate on the Python advantage? Not familiar w/ it.
Thanks!

Ouch, brain hurts. Funny you

Josh's picture

Ouch, brain hurts.

Funny you should bring this up - I was struggling with a super-stupid-simple issue this week that didn't work because of quoting. Thanks for the article!

What about files passed on the command line?

Tyler Wagner's picture

Thanks very much for the IFS tip. That seems a good way around this problem.

However, what about passing in filenames on the command line? I'd like to call my script as so:

$ script "file 1 with spaces" "file 2 with spaces"

.. and be able to parse them correctly. Suggestions?

Use "$@"

Mitch Frazier's picture

Something like this should work:

for i in "$@"
do
    echo "$i"
done

The secret is with quoting $@, rather than getting all the command line arguments quoted as a single string (which is what "$*" gets you), when you use "$@" you get each individual argument separately quoted, and if you quoted things on the command line they will continue to be quoted. When I say quoted, I really mean it keeps them together as you specified them and doesn't break them at spaces.

Mitch Frazier is an Associate Editor for Linux Journal.

Doh! Thanks!

Tyler Wagner's picture

Wow! All this time I've been misusing $* and $@. I've tried various combinations of quoting $@, $*, $1, and $i (in a for loop), and I'd never gotten this to work right before now. Thanks so much!

I finally have a way to quickly, quietly, reliably unzip a bunch of zip files with spaces in their names, from the command line.

Thanks.

Re: Use "$@"

Vance's picture

The following is slightly simpler and does the same thing:

for i
do
    echo "$i"
done

With no in, the for just iterates over all the positional parameters.

Very useful article! I have

Anonymous's picture

Very useful article! I have run into these issues too and I these tips help a lot.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState