This will output any line not containing "frank"

As a full-stack developer, you likely spend a good portion of your day working at the Linux or Unix command line. Searching through files and directories to find specific pieces of information is a common task, and one of the most powerful tools for doing this is the grep command.

In this tutorial, we‘ll dive deep into grep and learn how to harness its full text searching capabilities. Whether you‘re a grep novice or a seasoned veteran, by the end of this article you‘ll have a solid understanding of how to recursively search directories, match complex text patterns using regular expressions, and integrate grep into your command line workflow. Let‘s get started!

What is grep?

grep stands for "globally search for a regular expression and print matching lines". It‘s a command line utility that originated in Unix but is also standard on Linux and macOS. At its most basic, grep allows you to search plain text files for lines matching a specific pattern.

For example, let‘s say we have a file named server.log with these contents:

127.0.0.1 – frank [10/Oct/2020:13:55:36 -0700] "GET /apache.jpg HTTP/1.0" 200 2326
127.0.0.1 – frank [10/Oct/2020:13:56:12 -0700] "GET /favicon.ico HTTP/1.0" 404 1997

To search this file for lines containing "frank", we could use grep like this:

$ grep ‘frank‘ server.log
127.0.0.1 – frank [10/Oct/2020:13:55:36 -0700] "GET /apache.jpg HTTP/1.0" 200 2326
127.0.0.1 – frank [10/Oct/2020:13:56:12 -0700] "GET /favicon.ico HTTP/1.0" 404 1997

This simple example demonstrates the core functionality of grep – it prints out lines from the specified file(s) that contain the search pattern, which in this case is just the literal string "frank".

But grep is capable of much more than simple string matching. Let‘s explore some of its most useful options and features.

Important grep Options

grep supports a large number of command line options that modify its behavior. Here are some of the most important ones to know:

-i (–ignore-case)

By default, grep matches are case-sensitive. Use the -i flag to make the search case-insensitive:

$ grep -i ‘frank‘ server.log
127.0.0.1 – frank [10/Oct/2020:13:55:36 -0700] "GET /apache.jpg HTTP/1.0" 200 2326
127.0.0.1 – Frank [10/Oct/2020:13:56:12 -0700] "GET /favicon.ico HTTP/1.0" 404 1997

-v (–invert-match)

Sometimes you want to find lines that don‘t match the given pattern. The -v option inverts the match logic:

$ grep -v ‘frank‘ server.log

-r (–recursive)

To search all files in the current directory and its subdirectories recursively, use the -r flag:

$ grep -r ‘frank‘ /var/log/
/var/log/system.log:Oct 10 10:34:10 frank-laptop kernel[0]: lo0: link state changed to DOWN
/var/log/wifi.log:Oct 10 08:14:22 :: frank connected to Home WiFi

This will traverse the entire directory tree starting at /var/log/ and search each file for "frank".

-l (–files-with-matches)

If you just need to know which files contain matches without seeing the actual matching lines, use -l:

$ grep -rl ‘frank‘ /var/log/
/var/log/system.log
/var/log/wifi.log

-n (–line-number)

To see line numbers along with the matching lines, use the -n option:

$ grep -n ‘frank‘ server.log
1:127.0.0.1 – frank [10/Oct/2020:13:55:36 -0700] "GET /apache.jpg HTTP/1.0" 200 2326
2:127.0.0.1 – frank [10/Oct/2020:13:56:12 -0700] "GET /favicon.ico HTTP/1.0" 404 1997

There are many more grep options worth exploring. Refer to the grep manual page for the complete list.

Regular Expressions

Beyond simple string matching, the true power of grep lies in its support for regular expressions. Regular expressions (regex for short) allow you to construct elaborate patterns to precisely match only the lines you‘re interested in.

Here are some of the most useful constructs to use in grep regular expressions:

^ and $ Anchors

The ^ anchor matches the start of the line, and $ matches the end of the line. For example, to find lines that start with "127":

$ grep ‘^127‘ server.log
127.0.0.1 – frank [10/Oct/2020:13:55:36 -0700] "GET /apache.jpg HTTP/1.0" 200 2326
127.0.0.1 – frank [10/Oct/2020:13:56:12 -0700] "GET /favicon.ico HTTP/1.0" 404 1997

. Wildcard

The . special character matches any single character except newline. To match "frank" or "Frank" you could use:

$ grep ‘.rank‘ server.log
127.0.0.1 – frank [10/Oct/2020:13:55:36 -0700] "GET /apache.jpg HTTP/1.0" 200 2326
127.0.0.1 – Frank [10/Oct/2020:13:56:12 -0700] "GET /favicon.ico HTTP/1.0" 404 1997

Quantifiers

You can use quantifiers to match a variable number of characters:

• • matches the preceding element 0 or more times
• • matches the preceding element 1 or more times
• ? matches the preceding element 0 or 1 times

For example, to match "jpeg", "jpg", "png", etc:

$ grep ‘jp*g‘ server.log
127.0.0.1 – frank [10/Oct/2020:13:55:36 -0700] "GET /apache.jpg HTTP/1.0" 200 2326

Character Classes

Square brackets let you define character classes to match any single character inside the brackets. For instance, to find lines containing any vowel:

$ grep ‘[aeiou]‘ server.log
127.0.0.1 – frank [10/Oct/2020:13:55:36 -0700] "GET /apache.jpg HTTP/1.0" 200 2326
127.0.0.1 – frank [10/Oct/2020:13:56:12 -0700] "GET /favicon.ico HTTP/1.0" 404 1997

Alternation

The | character lets you match one pattern or another. To find "frank" or "jimmy":

$ grep ‘frank|jimmy‘ server.log
127.0.0.1 – frank [10/Oct/2020:13:55:36 -0700] "GET /apache.jpg HTTP/1.0" 200 2326
127.0.0.1 – frank [10/Oct/2020:13:56:12 -0700] "GET /favicon.ico HTTP/1.0" 404 1997

This only scratches the surface of what‘s possible with grep and regular expressions. Spend some time practicing and refer to a regex tutorial for more advanced concepts.

Using grep with Pipes

Like most Linux commands, grep output can be redirected using a pipe (|) to send it as input to another command. This allows chaining multiple commands together to perform complex operations.

For example, you could use grep to find error messages in a log file, pipe that to another grep to filter out a specific error code, then pipe that to wc -l to count the number of matching lines:

$ grep ‘ERROR‘ app.log | grep ‘500‘ | wc -l
7

Here are a few other common commands that are often used with grep:

grep and cut

cut is useful for extracting certain fields from lines of text. For instance, to get a list of all users in /etc/passwd:

$ grep -v ‘^#‘ /etc/passwd | cut -d: -f1
root
daemon
bin
sys

grep and sort

sort will alphabetically sort the lines. To get a sorted list of users:

$ grep -v ‘^#‘ /etc/passwd | cut -d: -f1 | sort
bin
daemon
root
sys

grep and uniq

uniq removes duplicate adjacent lines. To get a sorted list of unique users:

$ grep -v ‘^#‘ /etc/passwd | cut -d: -f1 | sort | uniq
bin
daemon
root
sys

The possibilities for command pipelines are endless. Whenever you find yourself using grep, think about how you can combine it with other tools to craft a complete solution right there in the terminal.

grep Examples

Let‘s walk through a few real-world examples to solidify our grep knowledge.

Finding Error Messages

Imagine you‘re a sysadmin investigating reports of 500 Internal Server Errors on a web server. The access logs are in the common Apache combined log format.

To find all log entries containing 500 errors:

$ grep -r ‘ 500 ‘ /var/log/apache2/
/var/log/apache2/access.log.1:127.0.0.1 – – [10/Oct/2020:15:27:05 -0700] "GET /reports HTTP/1.0" 500 182
/var/log/apache2/access.log.1:127.0.0.1 – – [10/Oct/2020:15:40:32 -0700] "POST /submit HTTP/1.0" 500 4634

Note the single quotes around the pattern to avoid the shell interpreting the space characters.

Searching Source Code

As a developer debugging an issue, you may need to search a large codebase for references to a specific function. To find all lines containing calls to the "getUserById" function in .js files:

$ grep -r –include="*.js" ‘getUserById‘ ./src
./src/api/users.js: const user = await getUserById(id);
./src/auth/oauth.js: return getUserById(profile.id);

The –include option restricts matches to files with names matching the given pattern.

Analyzing Logs

Web server access logs are a treasure trove of data about usage patterns and traffic. To get a quick count of how many requests used the POST method:

$ grep -c ‘POST‘ access.log
248

Piping a few greps together lets us dissect the data further. For example, to see the top 5 most frequent HTTP response codes:

$ grep -oE ‘\s[0-9]{3}\s‘ access.log | sort | uniq -c | sort -rn | head -5
7814 200
6673 304
4117 301
810 404
221 500

Here grep extracts just the status code using a regex, then sort and uniq -c count the unique codes, then a final sort -rn and head -5 give us the top 5 by descending frequency.

grep Alternatives

While grep is ubiquitous and a solid default choice, there are several alternative tools that offer enhanced features for searching files:

ack

ack is designed for searching source code. It automatically skips non-text files and directories like .git. It also uses Perl-compatible regular expressions which are more expressive than grep‘s default syntax.

ag (The Silver Searcher)

ag is an extremely fast grep replacement optimized for large codebases. It ignores version control metadata and knows which file extensions to search based on the code‘s language.

rg (ripgrep)

rg combines the speed of ag with additional features like multi-threaded search and support for fancy regex engines like PCRE2.

These tools aim to be "sane defaults" greps for source code search. They‘re worth investigating if you find yourself doing a lot of greping in code.

Conclusion

The grep command is a veritable Swiss Army knife for slicing and dicing text at the Linux command line. Its power and flexibility make it an indispensable tool to have in your developer toolbox.

In this article, we‘ve covered:

• Basics of using grep to search files for simple text patterns
• Important grep command line options for controlling output and behavior
• Constructing regex patterns for precise matching
• Combining grep with other commands via pipes
• Real-world usage examples for debugging and data analysis
• Alternative tools suitable for searching source code

I encourage you to practice using grep on your own systems and refer back to this guide as you learn. The time spent mastering grep will pay dividends and make you a more proficient developer and Linux user.

Here are some additional resources for continuing your grep and regex education:

• GNU Grep Manual: https://www.gnu.org/software/grep/manual/grep.html
• Regex Tutorial: https://medium.com/factory-mind/regex-tutorial-a-simple-cheatsheet-by-examples-649dc1c3f285
• Regex 101 (interactive tester): https://regex101.com/

Happy grepping!