[Chapter 9] 9.20 Too Many Files for the Command Line

9.20 Too Many Files for the Command Line

A pair of backquotes (``) (9.16) lets you run a command like find (17.1) and put its output onto another command's command line. For example:

pr lpr
% pr -n `find . -type f -mtime -1 -print` | lpr

pr lpr	% `pr` -n `find . -type f -mtime -1 -print` \| lpr

would give a list of all the files you edited today to pr and pipe pr's output to the printer.

One day I was making global substitutions to a lot of files (34.3, 28.9) and got the error Arguments too long when I tried to print the files I had edited. Turned out that find output such a long list of files that it overflowed the command line:

% pr -n ./path/file1 ./path/path/file2 ./path/file3 ... | lpr

(This can happen for any command inside backquotes, not just find.)

I had to split find's standard output into chunks that wouldn't be too long. This was on a UNIX that didn't have the great xargs (9.21), so I decided to use fmt (35.2) instead. fmt reads its standard input and collects enough text to fill an output line. I used fmt -1000, which makes output lines about 1000 characters wide-long enough so I wouldn't need too many pr commands, but not too long... I started a Bourne shell, which lets you pipe to the input of a loop (45.23). The shell prints secondary prompts (9.13) until you finish entering the loop:

read exit
% sh $ find . -type f -mtime -1 -print | > fmt -1000 | > while read files > do pr -n $files > done | lpr $ exit %

read exit	% `sh` $ `find . -type f -mtime -1 -print \|` > `fmt -1000 \|` > `while` `read` `files` > `do pr -n $files` > `done \| lpr` $ `exit` %

The shell put each line of filenames from fmt -1000 into the files shell variable, ran pr with those filenames, and piped the output of all the prs to the standard input of lpr. The lpr command didn't know that it was being fed by lots of pr commands - all it saw was a series of 66 - line pages that the prs output.

If you have xargs on your system, you can do the same thing this way:

% find . -type f -mtime -1 -print | xargs pr -n | lpr

Warning!	xargs reads text from its standard input, collects a reasonable amount, then runs the command line `pr -n` `path/file path/file...`. Then xargs reads more text and runs pr again, over and over, until it's read all the text. The output of the xargs command (which is actually the output of all those prs) is fed to a single lpr command.

Parting shot (by ML): there's really no excuse for xargs or any of these other tricks; they're just a patch for a design error. UNIX should be able to handle arbitrarily long command lines; maybe in some future version, it will.

- JP


9.19 For the Impatient: Type-Ahead		9.21 Handle Too-Long Command Lines with xargs