Jun 082012

When chaining together shell commands with pipe ( | ), it’s easy to just turn it into a stream of consciousness. “This” produces “that” which is the input for the next command. This is fine, but it does often produce bad habits which can lead to sloppiness when these idioms are repeated in scripts.

A good example is when you want to find a string in a file, and then print out a single field of that line. Say, you want to find the userid for a given username in the password file.

The stream of consciousness method leads one to think, “grep will give the line that contains the pattern, and then awk will format the result by stripping out the field I need”. So if we want to find the ID of user “ptolemy”, where the line in our password file looks like this:


we might type this:

  # grep ptolemy /etc/passwd | awk -F: '{print $3}'   
(where -F specifies the field separator, a colon in the /etc/passwd)

But this isn’t really correct. If the string “ptolemy” matches somewhere else as well, this will be ambiguous. Moreover, this is just inelegant.

The reason is, AWK already can do pattern matching, and much better than grep can. So you could do this:

  # awk -F: /ptolemy/'{print $3}' /etc/passwd

Or better yet, AWK can specify its string comparison to an individual field – rather than the whole line. So a much better command would be this:

  # awk -F: '$1=="ptolemy" {print $3}' /etc/passwd

Which will precisely return the third password field if and only if the first field of the password file is “ptolemy”. And it uses only one command, rather than two.

Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.

He lives and works in London.

  2 Responses to “Don’t pipe grep into awk”

  1. Good to know indeed, thanks!

  2. All these years I’ve done the first step with grep and cut

    It will take me sometime to change this ingrained habit 🙂

    Thanks for sharing

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>