Work the Shell - Dealing with Spaces in Filenames
In the good old days when UNIX was young, no one ever would have thought of putting a space in a filename. It simply wasn't done—just as you'd never do that on a DOS or Windows system. Filenames were short, succinct and well-formed, like HW43.DOC.
Most of the Linux command-line utilities and the shells themselves have been designed based on the premise that a space delimits a field value rather than being an acceptable component of a filename. If you've done any scripting, you already know this. Spaces in filenames can cause great trouble in shell scripts! Here's a simple example:
for name in $(ls | grep a) do echo "File #$count = $name" count=$(( $count + 1 )) done
To set the stage, I've created a directory with some tricky filenames:
$ ls "quoted" beastly filename sample2.txt multi-word file name.pdf test.sh
Yes, to maximize trouble, I have a filename that includes quotes as well as a space. Don't get me started on having an escape character or non-printable character in the name though. It's doable, but I'd rename it as soon as possible.
Not all the filenames above have an “a” in them, so let's see what happens when the fragmentary script is run in this directory:
$ ./test.sh File # = "quoted" File #1 = beastly File #2 = filename File #3 = multi-word File #4 = file File #5 = name.pdf File #6 = sample2.txt
Oh, is that ugly and wrong!
The shell can deal with these filenames if they're simple enough, and the for loop for name in *a* yields three filenames, not six, but somewhere along your scripting journey, you inevitably will slam into the problem of embedded spaces.
The most common error is to forget to quote filenames when you use them elsewhere in the script, of course. As an example, let's work on a script that replaces spaces in filenames with underscores.
The obvious solution to such renaming is something like this:
for name in "* *" do newname="$(echo $name | sed 's/ /_/g')" mv $name $newname done
This doesn't work, however, and in a most fascinating way:
mv "quoted" beastly filename multi-word file ↪name.pdf sample2.txt test.sh "quoted" ↪beastly filename multi-word file ↪name.pdf sample2.txt test.sh ↪"quoted"_beastly_filename_multi-word_file_ ↪name.pdf_sample2.txt_test.sh_"quoted"_beastly_ ↪filename_multi-word_file_name.pdf_sample2.txt_test.sh
What's happened is that "* *" simply produces two full filename listings rather than just those filenames that contain a space—oops. Let's try a different pattern:
for name in *\ *
That does the trick, but we've not compensated for the fact that when the shell sees a line like:
mv multi-word file name.pdf multi-word_file_name.pdf
it's going to complain that it's seeing four filename arguments to the mv command, not the required two:
usage: mv [-f | -i | -n] [-v] source target mv [-f | -i | -n] [-v] source ... directory
In this case, the solution is to quote the filename variable:
mv "$name" $newname
As a discipline, it's always good to quote filenames you reference in any context to ensure that when the shell passes them to the command as arguments, the filenames with embedded spaces are handled properly.
This isn't a universal solution, however, because if you're using subshells and pipes, it can be pretty darn hard for quotes to survive multiple steps.
One path to travel is to set IFS, the internal field separator, in the shell to something other than a space, as explained in the Bash man page:
IFS: The Internal Field Separator that is used for word splitting after expansion and to split lines into words with the read built-in command. The default value is “<space><tab><new-line>”.
That's useful for “read”, particularly if you're reading lines of text and want to have a different field separator (think flat-file text database files), but it still doesn't really solve our filename problem.
One thing I've used in the past, although it's a sloppy, crude solution, is to start by changing spaces to some unlikely sequence of characters, run through all the processing, and change them back at the last second. For example:
safename="$(echo name | sed 's/ /_-_/g')"
Dave Taylor has been hacking shell scripts for over thirty years. Really. He's the author of the popular "Wicked Cool Shell Scripts" and can be found on Twitter as @DaveTaylor and more generally at www.DaveTaylorOnline.com.
|PostgreSQL, the NoSQL Database||Jan 29, 2015|
|HPC Cluster Grant Accepting Applications!||Jan 28, 2015|
|Sharing Admin Privileges for Many Hosts Securely||Jan 28, 2015|
|Red Hat Enterprise Linux 7.1 beta available on IBM Power Platform||Jan 23, 2015|
|Designing with Linux||Jan 22, 2015|
|Wondershaper—QOS in a Pinch||Jan 21, 2015|
- PostgreSQL, the NoSQL Database
- Sharing Admin Privileges for Many Hosts Securely
- HPC Cluster Grant Accepting Applications!
- Designing with Linux
- Wondershaper—QOS in a Pinch
- Internet of Things Blows Away CES, and it May Be Hunting for YOU Next
- January 2015 Issue of Linux Journal: Security
- Ideal Backups with zbackup
- Slow System? iotop Is Your Friend
- Red Hat Enterprise Linux 7.1 beta available on IBM Power Platform
Editorial Advisory Panel
Thank you to our 2014 Editorial Advisors!
- Jeff Parent
- Brad Baillio
- Nick Baronian
- Steve Case
- Chadalavada Kalyana
- Caleb Cullen
- Keir Davis
- Michael Eager
- Nick Faltys
- Dennis Frey
- Philip Jacob
- Jay Kruizenga
- Steve Marquez
- Dave McAllister
- Craig Oda
- Mike Roberts
- Chris Stark
- Patrick Swartz
- David Lynch
- Alicia Gibb
- Thomas Quinlan
- Carson McDonald
- Kristen Shoemaker
- Charnell Luchich
- James Walker
- Victor Gregorio
- Hari Boukis
- Brian Conner
- David Lane