Linux: xargs vs. exec {}

By Daniel Miessler on October 29th, 2006: Tagged as Linux | Sysadmin | Unix

There is a bit of a debate in some circles about using xargs vs. the -exec {} option that’s built into find itself. To me, however, it’s not much of a debate; -exec isn’t nearly as good as xargs for what I use find for. I tend to use it to perform tasks involving many files. “Move all these files there”, “copy all those directories there”, “Delete these links.”, etc.

This is where -exec breaks down and xargs shows its superiority. When you use -exec to do the work you run a separate instance of the called program for each element of input. So if find comes up with 10,000 results, you run exec 10,000 times. With xargs, you build up the input into bundles and run them through the command as few times as possible, which is often just once. When dealing with hundreds or thousands of elements this is a big win for xargs.
That’s all nice and stuff, but you probably want to see it in action, right? Let’s run some numbers. Below is a listing of 1,668 .jpg files on my OS X system using both -exec and xargs:

# time find . -name “*.jpg” -exec ls {} \; 

real    0m6.618s
user    0m1.465s
sys     0m4.396s
Hmm, that’s not bad — seven seconds for over around 1,600 files, right? Let’s try it with xargs.

# time find . -name “*.jpg” -print0 | xargs -0 ls 

real    0m1.120s
user    0m0.594s
sys     0m0.527s
That’s one (1) second vs seven (7) seconds. Seriously; xargs is the way to go.

--

10 Comments »

  1. .. and if you are doing a hardcore batch job for 15000 files, are those 6 seconds of shell processing really that important?

    I use xargs but definitely for different reasons than few second difference… I’m not aware if the following example is even possible with -exec:

    ls *jpg|xargs -ifile convert file -resize 90x -crop 90×90 0 0 file

    Comment by mikkom — 10/30/2006 @ 11:50 am

  2. mikkom: I’m not too familiar with xargs, but I think this find expression will do what you want:

    find . -maxdepth 1 -name ‘*jpg’ -exec convert ‘{}’ -resize 90x -crop 90×90 0 0 ‘{}’ ‘;’

    I usually put ” around {} in case of spaces.

    Comment by Rory — 10/30/2006 @ 12:35 pm

  3. What impant did harddrive caching have on your test?

    Comment by Pieter Breed — 10/30/2006 @ 2:59 pm

  4. If you try the commands in the other order, i.e. xargs first then -exec the exec is faster, at least in my little test of finding all jpgs in my home dir. xargs took 2mins exec took 21 secs.

    /me wonders if find caches results or something to speed it up when you do a similar find afterwards.

    Comment by TD — 10/30/2006 @ 3:00 pm

  5. TD: it’s filesystem kernel-level cache. This is simply incorrect method of benchmarking. May the author check inner details better next time. Kudos.

    Comment by admp — 10/30/2006 @ 4:54 pm

  6. mikkom,

    If you’re using ‘bash’ as your shell (you should be), it’s often easier to just say ‘for i in *.jpg; do convert file blah…

    Comment by Mike — 10/30/2006 @ 9:04 pm

  7. To those who mentioned caching, I performed the test at least ten times using both commands to produce the numbers I did. I ran into the caching issue as well on other systems, but on the system in question I think I was getting real numbers. Again, both numbers come from the command being run multiple times. I didn’t run one without the cache and one with it.

    That, I agree, would have been lame…

    Comment by Daniel Miessler — 10/30/2006 @ 11:26 pm

  8. Mike,

    bash loops work great until you hit a filename with a space.

    Comment by tomchuk — 10/30/2006 @ 11:27 pm

  9. When using the -i or -I string replacement for xargs (needed for any complex operation such as the convert example above), then it becomes impossible for xargs to group the arguments together and it has to execute them one at a time as in -exec. If you change the example above to:

    time find . -name “*.jpg” -print0 | xargs -0 -ifile ls file

    you’ll see it has almost exactly the same running time as -exec.

    This dual behavior of xargs can be surprising - a complex command written with -i may work fine since it only receives one argument at a time, but break suddenly if the -i is removed and the command faces batched arguments. It’s probably best to use the -exec option to find(1) wherever you would need to use -i in xargs to avoid this confusion.

    Comment by foof — 11/1/2006 @ 1:45 am

  10. tomchuck,

    bash has no problem looping through a list of files with our without spaces, you just have to make sure and put quotes around the loop variable:

    for png in *.png ; do echo pngcrush “$png”; done

    Comment by Justin McBroom — 11/1/2006 @ 5:08 pm

RSS Feed For This Post...
This Post's TrackBack URI

Leave a Comment...