Work the Shell - Dealing with Spaces in Filenames

 in
How do you deal with this problem in your scripts?

In the good old days when UNIX was young, no one ever would have thought of putting a space in a filename. It simply wasn't done—just as you'd never do that on a DOS or Windows system. Filenames were short, succinct and well-formed, like HW43.DOC.

Most of the Linux command-line utilities and the shells themselves have been designed based on the premise that a space delimits a field value rather than being an acceptable component of a filename. If you've done any scripting, you already know this. Spaces in filenames can cause great trouble in shell scripts! Here's a simple example:

for name in $(ls | grep a)
do
  echo "File #$count = $name"
  count=$(( $count + 1 ))
done

To set the stage, I've created a directory with some tricky filenames:

$ ls
"quoted" beastly filename      sample2.txt
multi-word file name.pdf           test.sh

Yes, to maximize trouble, I have a filename that includes quotes as well as a space. Don't get me started on having an escape character or non-printable character in the name though. It's doable, but I'd rename it as soon as possible.

Not all the filenames above have an “a” in them, so let's see what happens when the fragmentary script is run in this directory:

$ ./test.sh
File # = "quoted"
File #1 = beastly
File #2 = filename
File #3 = multi-word
File #4 = file
File #5 = name.pdf
File #6 = sample2.txt

Oh, is that ugly and wrong!

The shell can deal with these filenames if they're simple enough, and the for loop for name in *a* yields three filenames, not six, but somewhere along your scripting journey, you inevitably will slam into the problem of embedded spaces.

The most common error is to forget to quote filenames when you use them elsewhere in the script, of course. As an example, let's work on a script that replaces spaces in filenames with underscores.

File Renaming with a Bug

The obvious solution to such renaming is something like this:

for name in "* *"
do
  newname="$(echo $name | sed 's/ /_/g')"
  mv $name $newname
done

This doesn't work, however, and in a most fascinating way:

mv "quoted" beastly filename multi-word file 
 ↪name.pdf sample2.txt test.sh "quoted" 
 ↪beastly filename multi-word file 
 ↪name.pdf sample2.txt test.sh 
↪"quoted"_beastly_filename_multi-word_file_
↪name.pdf_sample2.txt_test.sh_"quoted"_beastly_
↪filename_multi-word_file_name.pdf_sample2.txt_test.sh

What's happened is that "* *" simply produces two full filename listings rather than just those filenames that contain a space—oops. Let's try a different pattern:

for name in *\ *

That does the trick, but we've not compensated for the fact that when the shell sees a line like:

mv multi-word file name.pdf multi-word_file_name.pdf

it's going to complain that it's seeing four filename arguments to the mv command, not the required two:

usage: mv [-f | -i | -n] [-v] source target
       mv [-f | -i | -n] [-v] source ... directory

In this case, the solution is to quote the filename variable:

mv "$name" $newname

As a discipline, it's always good to quote filenames you reference in any context to ensure that when the shell passes them to the command as arguments, the filenames with embedded spaces are handled properly.

This isn't a universal solution, however, because if you're using subshells and pipes, it can be pretty darn hard for quotes to survive multiple steps.

One path to travel is to set IFS, the internal field separator, in the shell to something other than a space, as explained in the Bash man page:

IFS: The Internal Field Separator that is used for word splitting after expansion and to split lines into words with the read built-in command. The default value is “<space><tab><new-line>”.

That's useful for “read”, particularly if you're reading lines of text and want to have a different field separator (think flat-file text database files), but it still doesn't really solve our filename problem.

One thing I've used in the past, although it's a sloppy, crude solution, is to start by changing spaces to some unlikely sequence of characters, run through all the processing, and change them back at the last second. For example:

safename="$(echo name | sed 's/ /_-_/g')"

______________________

Dave Taylor has been hacking shell scripts for over thirty years. Really. He's the author of the popular "Wicked Cool Shell Scripts" and can be found on Twitter as @DaveTaylor and more generally at www.DaveTaylorOnline.com.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Why not using the IFS??

Saladin's picture

I've gone through this dilemma in recent project and found using IFS simple and very effective,

Is this possible for all cases and the one mentioned in the article above?

IFS_backup=$IFS

IFS=$(echo -en "\n\b")

#do what ever you want ... then restore the original variable

IFS=$IFSBU

--

Salah Aldeen

Regards,

The problem, internal shell conversions

xae's picture

To manage this problem I had been using code like this,


for filename in *; do aux=$filename/; LS[${#LS[@]}]=${aux%/}; done
...
rm "${LS[24]}", as example

The catch here is to avoid the conversions taken by the shell when you access to the content of a variable, the trick is to append to the variable a character, trying to make the shell guess that it's a string, and then you can gain access without infamous conversions, but at later you ougth to remove the char in order to use it as a filename.

The chars candidates for this job are NUll, $'/0, and "/".

s/find -print/find -print0/ ?

mattcen's picture

Dave, did you mean '-print0' as opposed to '-print'? While both kind of make sense, when paired with 'xargs -0' I think the former makes *more* sense.

With respect to spaces, I usually don't bother dealing with them unless I *know* that they'll be there (e.g. I'm manipulating data that non-geek customers access via a Samba share and put plenty of weird characters in).

One of the other solutions I've considered, but not really had to use yet, is to switch to Perl or Python for these particular use-cases, given they handle escaping of spaces somewhat better.

It's something I'd rather not do given I'm more familiar with Bash than either of the above, but it may be the lease bad solution, even if I do have to dig up my text books to work out how to do something that would be trivial in Bash.

--
Regards,
Matthew Cengia

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix