44.14 Putting awk, sed, etc., Inside Shell Scripts

In SunExpert magazine, in his article on awk (January, 1991), Peter Collinson suggests a stylization similar to this for awk programs in shell scripts (44.1):

#!/bin/sh
awkprog='
/foo/{print $3}
/bar/{print $4}'

awk "$awkprog" $*

He argues that this is more intelligible in long pipelines because it separates the program from the command. For example:

grep foo $input | sed .... | awk "$awkprog" - | ...

Not everyone is thrilled by the "advantages" of writing awk this way, but it's true that there are disadvantages to writing awk in the standard way.

Here's an even more complex variation:

<<\	#!/bin/sh temp=/tmp/awk.prog.$$ cat > $temp <<\END /foo/{print $3} /bar/{print $4} END awk -f $temp $1 rm -f $temp

This version makes it a bit easier to create complex programs dynamically. The final awk command becomes the equivalent of a shell eval (8.10); it executes something that has been built up at runtime. The first strategy (program in shell variable) could also be massaged to work this way.

As another example a program that I used once was really just one long pipeline, about 200 lines long. Huge awk scripts and sed scripts intervened in the middle. As a result, it was almost completely unintelligible. But if you start each program with a comment block and end it with a pipe, the result can be fairly easy to read. It's more direct than using big shell variables or temporary files, especially if there are several scripts.

#
# READ THE FILE AND DO XXX WITH awk:
#
awk '
   ...the indented awk program...
   ...
   ...
' |
#
# SORT BY THE FIRST FIELD, THEN BY XXX:
#
sort +0n -1 +3r |
#
# MASSAGE THE LINES WITH sed AND XXX:
#
sed '
   ...

Multiline pipes like that one are uglier in the C shell because each line has to end with a backslash (\) (8.15). Articles 8.14 and 8.15 have more about quoting.