Applying xargs
There are some commands that turn out to be more useful than first meets the eye. In my opinion, xargs is one of those commands. It takes the standard input and uses it to build a command line. It's nothing fancy, but it's very handy in some situations.
As soon as you have a list of files, you can easily do something to them. A favorite, common enough to have a shell script of its own on my machine, is clean-titles.sh. It simply locates all backup files using the pattern *~, and then passes them on to rm. The result is a nice and clean current working directory and sub-tree.
#!/bin/sh find -iname '*~' | xargs rm
Do not forget your single quotes around the pattern, otherwise bash might expand it for you.
Another place where xargs comes in handy is when you want to find files based on contents and perform some sort of action on them. For instance, let's locate all those pesky TODO comments and open up those files in kate.
grep TODO -r . | sed 's/:.*//' | sort -u | xargs kate -u
The -u argument to kate ensures that xargs reuses an existing session instead of opening a new window. This is just the way that I prefer to have it, and I even have an alias setup for kate, so that I always used kate -u. However, aliases are not used by xargs, so I have to add the flag explicitly.
Something completely different, but somewhat similar, is the xclip command. In a perfect world, I just might want to give all the TODOs to a colleague. Just replacing xargs with xclip puts all the filenames in the clipboard.
grep TODO -r . | sed 's/:.*//' | sort -u | xclip
Now I only need to add the header before I paste it all into a mail. "Hi, I expect you to complete these by tomorrow!"
Johan Thelin is a consultant working with Qt, embedded and free
software. On-line, he is known as e8johan.
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Sponsored by AMD
Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6
Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.
Learn more about catching the bad guy in this free white paper.
Sponsored by DLT Solutions
| Designing Electronics with Linux | May 22, 2013 |
| Dynamic DNS—an Object Lesson in Problem Solving | May 21, 2013 |
| Using Salt Stack and Vagrant for Drupal Development | May 20, 2013 |
| Making Linux and Android Get Along (It's Not as Hard as It Sounds) | May 16, 2013 |
| Drupal Is a Framework: Why Everyone Needs to Understand This | May 15, 2013 |
| Home, My Backup Data Center | May 13, 2013 |
- New Products
- Linux Systems Administrator
- Senior Perl Developer
- Technical Support Rep
- UX Designer
- Web & UI Developer (JavaScript & j Query)
- Designing Electronics with Linux
- Dynamic DNS—an Object Lesson in Problem Solving
- Using Salt Stack and Vagrant for Drupal Development
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Nice article, thanks for the
3 hours 30 min ago - I once had a better way I
9 hours 16 min ago - Not only you I too assumed
9 hours 33 min ago - another very interesting
11 hours 26 min ago - Reply to comment | Linux Journal
13 hours 20 min ago - Reply to comment | Linux Journal
20 hours 14 min ago - Reply to comment | Linux Journal
20 hours 30 min ago - Favorite (and easily brute-forced) pw's
22 hours 21 min ago - Have you tried Boxen? It's a
1 day 4 hours ago - seo services in india
1 day 8 hours ago
Enter to Win an Adafruit Pi Cobbler Breakout Kit for Raspberry Pi

It's Raspberry Pi month at Linux Journal. Each week in May, Adafruit will be giving away a Pi-related prize to a lucky, randomly drawn LJ reader. Winners will be announced weekly.
Fill out the fields below to enter to win this week's prize-- a Pi Cobbler Breakout Kit for Raspberry Pi.
Congratulations to our winners so far:
- 5-8-13, Pi Starter Pack: Jack Davis
- 5-15-13, Pi Model B 512MB RAM: Patrick Dunn
- 5-21-13, Prototyping Pi Plate Kit: Philip Kirby
- Next winner announced on 5-27-13!
Featured Jobs
| Linux Systems Administrator | Houston and Austin, Texas | Host Gator |
| Senior Perl Developer | Austin, Texas | Host Gator |
| Technical Support Rep | Houston and Austin, Texas | Host Gator |
| UX Designer | Austin, Texas | Host Gator |
| Web & UI Developer (JavaScript & j Query) | Austin, Texas | Host Gator |
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?



Comments
Good
Yes, it's a good tool.
I like to use it do grep.
$ find . -name "*.cc" | xgargs grep "xxx"
Find can execute a grep, no need to pipe to xargs
Find can already do this without piping to xargs:
$ find . -name "*.cc" -print -exec grep {} \;
this can be easily done without using xargs!
grep -r xxx *.cc
Non, since the *.cc is going
Non, since the *.cc is going to be expanded, this will /not/ actually recurse.
my favorite is 'ps -aux |
my favorite is 'ps -aux | grep filename | xargs kill -9' to quickly kill a process
pgrep filename | xargs kill
pgrep filename | xargs kill -9
pkill -9 filename
pkill -9 filename
GNU Parallel
If you like xargs you may like GNU Parallel, too. It does not suffer from the separator problem http://en.wikipedia.org/wiki/Xargs#The_separator_problem
Watch the intro videos to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ
Common uses for me: ls
Common uses for me:
ls *.mp4|xargs -i -P4 ./d2.sh {} # convert files for my DMP, and multi-thread it.
ls *.avi|sed s/avi$/mp4/|xargs -i rm {} #remove source files after conversion
ls|grep -v .bz2$|xargs -i -P4 bzip2 {} # multi-thread zipping, but only files that are not already zipped.
Ok now try all them with spaces :-)
These days I had to cope with silly file and directory names full of spaces, just to discover how difficult is trying to work around them by shell commands in a general way. It is not something than can be solved by quoting or using -print0 in find commands or -0 in xargs commands.
I found extremely useful using sed to replace spaces with other chars by send and restore them when required. E.g.
sed -e 's/ /\c@/'
is extremely useful to replace temporarily spaces with NULLs when all other tricks fail.
Hit by the separator problem
It seems you were hit by the separator problem http://en.wikipedia.org/wiki/Xargs#The_separator_problem
You may find GNU Parallel does what you mean without trickery.
Watch the intro videos to learn more http://www.youtube.com/watch?v=OpaiGYxkSuQ
Avoiding multiple command invocation with find
If you use
find -name search_string -exec command {} +
command is invoked just once with a list of names rather on each match individually. This can speed things up when there are a lot of matches.
Thank you all!
Great to hear all your feedback. There are so many ways to perform this task, it is great to learn how you all do it.
Find is a more versatile tool than first meets the eye and can do much of the tasks described here - it might even deserve a blog of its own.
And for spaces in filenames - who uses that? :-)
Johan Thelin is a consultant working with Qt, embedded and free
software. On-line, he is known as e8johan.
Re:
Try working on files created by windows users. Rarely will those have underscores instead of spaces.
Special characters in filenames
But I have yet to see even a Windows user have \n in the file name. That is why \n is the default record separator for GNU Parallel and thus it deals nicely with:
My brother's 12" records
Try it. You might like it. http://www.youtube.com/watch?v=OpaiGYxkSuQ
You can also get the same
You can also get the same result with:
find . -name Thumbs.db -exec rm {} \;
Another use of xargs is to replace a string in many files using sed:
find . \( -name "*.php" -or -name "*.html" \) | xargs grep -l StringA | xargs sed -i -e 's/StringA/StringB/g'
What is the difference
What is the difference between :
$ find -iname '*~' | xargs rm
and
$ rm $(find -iname '*~')
?
I guess the second one would cause problems when there are spaces in the file names, but according to a previous comment the first one would not work either.
It's also going to fail if
It's also going to fail if you have more than a couple of hundred of file. See ARG_MAX in <linux/limits.h>.
What is the difference
What is the difference between :
$ find -iname '*~' | xargs rm
and
$ rm $(find -iname '*~')
?
I guess the second one would cause problems when there are spaces in the file names, but according to a previous comment the first one would not work either.
Chunking
Both have problems with spacey file names.
The difference is in chunking: xargs accepts "a few" of its input arguments and builds and runs the "rm …" command, then accepts "a few more", runs another command, and so on. If the number of files in the find output is small, there will be no difference at all -- one "rm" command will run. But if the list is very large, the second form will fail because the argument list gets too long.
This is also why "find … | xargs rm" is superior to "find … -exec rm {} \;". The first form does "a few" at a time, while the second form runs one rm command for each file. Again, for few results, no important difference, but for many results, the xargs solution can be much faster.
There are switches to xargs to be more explicit about how many is "a few," but it's pretty rare to actually need to bother with that. By default, xargs does "a lot, but not too many."
Here's a general-purpose
Here's a general-purpose wrapper I use when I want to act on every source file in a directory tree (for example "findsrc.sh | xargs -0 etags" to generate a TAGS file).
> cat findsrc.sh
#!/bin/bash
find . -name .svn -prune -o -name "*test*" -prune -o -name "*stub*" -prune -o -name "*.[ch]" -a -name "*$1*" -print0
The main features are the "-name .svn -prune" and "-name "*.[ch] -a -name "*$1*" -print0" clauses, which avoid descending into .svn directories and return matching c source files respectively, these would probably need to be customized for your project, for example if you use different source control or language.
Might want to mention -print0 and -0
If the file names from the find have spaces, then xargs will see it as two or more files. The easy thing to do is add -print0 to find and -0 to xargs.
How about 'grep -l TODO *
How about 'grep -l TODO * |xargs kate -u' ?
About the find command, the
About the find command, the -delete switch is more handy than this. This won't work if filenames have spaces. Special switches needed for that.