Bash Quoting

 in

Quoting things in bash is quite simple... until it isn't. I've written scripts where I'd swear 90% of the effort was getting bash to quote things correctly.

The basics of quoting are simple, use single or double quotes when a value contains spaces:

a="hello world"
a='hello world'

But single and double quotes are different. Single quotes don't support any kind of variable substitution or escaping of special characters inside the quoted string. Double quotes, on the other hand, support both:

a="hello \"there\" world"
a='hello "there" world'
a='hello \'there\' world'    # causes an error
a="hello 'there' world"

b="there"
a='hello \"$b\" world'       # a is >>hello \"$b\" world
a="hello \"$b\" world"       # a is >>hello "there" world

One way around the problem that single quotes don't support variable substitution is to quote the individual literal parts of the string and then put the variable substitutions between them:

b='"there"'
a='"hello" '$b' "world"'     # a is: >>"hello" "there" "world"<<

Note that "$b" is not actually inside a quoted part of the string. Since there's no space between the two literal parts of the string and the "$b", bash puts the final result into a single string and then assigns it to a.

One place where quoting problems often occur is when you're dealing with files that have spaces in their names. For example, if we have a file named "file with spaces" and we run the following script:

#/bin/bash

for i in $(find .)
do
    echo $i
done

We don't get what we want:

$ bash t1.sh
.
./file
with
spaces
./t2.sh
./t1.sh
...

One solution is to put the file names into a bash array when the Internal Field Separator is set to just a newline. Then we use the value "${array[@]}" as the list for the for loop.

#/bin/bash

newline='
'

OIFS=$IFS
IFS=$newline
files=($(find .))
IFS=$OIFS

for i in "${files[@]}"
do
    echo $i
done

This produces the correct output:

$ bash t2.sh
.
./file with spaces
./t2.sh
./t1.sh
...

The difference between "${array[*]}" and "${array[@]}" is analagous to the difference between "$*" and "$@", namely that in the later case the result is that each word becomes separately quoted rather than the entire thing being quoted as a single string.

Another option, which avoids special quoting is to just set the Internal Field Separator variable before the for loop runs and reset it as the first statement in the loop body:

#/bin/bash

newline='
'

OIFS=$IFS
IFS=$newline

for i in $(find .)
do
    IFS=$OIFS
    echo $i
done

Another quoting problem is when you have a string which you'd like to break up into separate words but you'd also like to honor any quoted substrings in the string. Consider the string:

a="hello \"there big\" world"

and suppose you'd like to break that into its three parts:

  • hello
  • there big
  • world

If you run this:

#!/bin/bash

a="hello \"there big\" world"

for i in $a
do
    echo $i
done

It produces this:

$ bash t4.sh
hello
"there
big"
world

The trick here is to use a special form of the set command to reset $* (and therefore, $1, $2, etc). My first version is always like this:

#!/bin/bash

a="hello \"there big\" world"
set -- $a

for i in "$@"
do
    echo $i
done

And of course it doesn't work:

$ bash t5.sh
hello
"there
big"
world

Then I usually remember you have to eval the set statement:

#!/bin/bash

a="hello \"there big\" world"
eval set -- $a

for i in "$@"
do
    echo $i
done

At which point it works:

$ bash t6.sh
hello
there big
world

at this point I'm usually tired of quoting things.

______________________

Mitch Frazier is an Associate Editor for Linux Journal.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

When there are too long or too many filenames

Fjor's picture

Very interesting! I always forget $@ and eval. Regarding filenames, I prefer while/read to avoid the case of extra long or too many filenames (because * expands to the full list!):


#-- print one filename by line even with spaces
#-- or too long or too many names
find . | while read FILE ; do
   echo "$FILE"
done

#-- rename filenames changing spaces to dashes
find . | while read FILE ; do
   NEWNAME=$( echo "$FILE" | tr ' ' '-')
   mv -v "$FILE" "$NEWNAME"
done

This way I have the full filename in a variable without too much hassle. To process only the current directory, I add -maxdepth 1 to the find command.

There is a problem with this approach: FILE is only valid inside the while loop, because is run as a separate bash process (Eats some memory, but is slightly faster). When the input comes from a file, we can do this to avoid the extra process:


#-- store the file list for posterior processing
find . > filename.list.txt

#-- show all the names with uncommon characters only
# space is common :) and last char in grep regex
# dash must be the first one to be considered by itself
while read $FILE ; do
   echo "$FILE" | grep "[^-a-zA-Z0-9._/ ]"
done < filename.list.txt

Regards,
Fjor

This is sweet. I've been

David LeBlanc's picture

This is sweet. I've been looking for a over an hour for a way to treat quoted strings with spaces in them as a single token; many articles I have read come close but none hit the nail on the head like yours did. The magic is all in the "eval set..." Thanks so much!

IFS

Prasinos's picture

In bash it is possible to specify the field separator using control characters. The "newline" var in the above examples can be written as:


newline=$'\n'

which is more readable.

Yes it is

Mitch Frazier's picture

Thanks, I didn't know that. I see it now in the man page.

Mitch Frazier is an Associate Editor for Linux Journal.

This is madness

AprilCoolsDay's picture

Does anyone know of a shell without all this quoting complications?

Or do I have to stick to Python and Ruby?

In the

Naresh's picture

In the program
#/bin/bash

newline='
'

OIFS=$IFS
IFS=$newline

for i in $(find .)
do
IFS=$OIFS
echo $i
done

why we need IFS=$OIFS in the loop?
Can't we write this once after loop?

IFS

Mitch Frazier's picture

In this simple case, yes you could put IFS=$OIFS outside the loop, but that might not always be the case if other parts of the loop expect a "normal" IFS value. This example is a bit contrived, but suppose the loop looked like this:

for i in $(find .)
do
    for j in $(echo a b c)
    do
        echo $j
    done
    echo $i
done

Here the variable j only takes on one value, "a b c", because IFS contains only newline, and not tab or space.

Mitch Frazier is an Associate Editor for Linux Journal.

Quoting is fine but why not

Nicholas's picture

Quoting is fine but why not use python instead?

Can you elaborate on the

tehmasp's picture

Can you elaborate on the Python advantage? Not familiar w/ it.
Thanks!

Ouch, brain hurts. Funny you

Josh's picture

Ouch, brain hurts.

Funny you should bring this up - I was struggling with a super-stupid-simple issue this week that didn't work because of quoting. Thanks for the article!

What about files passed on the command line?

Tyler Wagner's picture

Thanks very much for the IFS tip. That seems a good way around this problem.

However, what about passing in filenames on the command line? I'd like to call my script as so:

$ script "file 1 with spaces" "file 2 with spaces"

.. and be able to parse them correctly. Suggestions?

Use "$@"

Mitch Frazier's picture

Something like this should work:

for i in "$@"
do
    echo "$i"
done

The secret is with quoting $@, rather than getting all the command line arguments quoted as a single string (which is what "$*" gets you), when you use "$@" you get each individual argument separately quoted, and if you quoted things on the command line they will continue to be quoted. When I say quoted, I really mean it keeps them together as you specified them and doesn't break them at spaces.

Mitch Frazier is an Associate Editor for Linux Journal.

Doh! Thanks!

Tyler Wagner's picture

Wow! All this time I've been misusing $* and $@. I've tried various combinations of quoting $@, $*, $1, and $i (in a for loop), and I'd never gotten this to work right before now. Thanks so much!

I finally have a way to quickly, quietly, reliably unzip a bunch of zip files with spaces in their names, from the command line.

Thanks.

Re: Use "$@"

Vance's picture

The following is slightly simpler and does the same thing:

for i
do
    echo "$i"
done

With no in, the for just iterates over all the positional parameters.

Very useful article! I have

Anonymous's picture

Very useful article! I have run into these issues too and I these tips help a lot.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix