Where Can I Use Regular Expressions?

Regular expressions are most commonly used for text filtering. For example, to change every occurrence of the letter "a" in a string to a capital "A", you might echo the string and pipe the result to sed like this:

echo "This is a test, this is only a test" | sed 's/a/A/g'

You can also use regular expressions to search for strings in a file or a block of text by using the grep command. For example, to look for the word "bar" in the file foo.txt, you might do this:

grep "bar" foo.txt

# or

cat foo.txt | grep "bar"

Finally, on occasion, it can be useful to use regular expressions in control statements. This advanced usage is described further in Using Regular Expressions in Control Statements.

Types of Regular Expressions

There are three basic types of regular expressions: basic regular expressions, extended regular expressions, and Perl regular expressions. Throughout this chapter, the sections points out areas in which they diverge. This section is just a summary of the differences. For more detail, see the appropriate section.

Basic regular expressions and extended regular expressions differ in the following areas:

• Basic regular expressions use a backslash prior to grouping/capturing parentheses (and prior to pipe operators within these parentheses). Extended regular expressions do not. These operators are described in Grouping Operators.

• Basic regular expressions use a backslash prior to a plus sign when used to mean “one or more of the previous character or group”. Extended regular expressions do not. This operator is described in Wildcards and Repetition Operators.

• Basic regular expressions use a backslash prior to a question mark when used to mean “zero or one of the previous character or group”. Extended regular expressions do not. This operator is described in Wildcards and Repetition Operators.

Perl regular expressions are equivalent to extended regular expressions with a few additional features:

• Perl can (optionally) use a dollar sign instead of a backslash to represent variables in substitution patterns, as described in Capturing Operators and Variables.

• Perl supports noncapturing parentheses, as described in Noncapturing Parentheses.

• The order of multiple options within parentheses can be important when substrings come into play, as described in Grouping Operators.

• Perl allows you to include a literal square bracket anywhere within a character class by preceding it with a backslash, as described in Quoting Special Characters.

• Perl adds a number of additional switches that are equivalent to certain special characters and character classes. These are described in Character Class Shortcuts.

• Perl supports a broader range of modifiers. These are described in Using Modifiers.

Regular Expression Syntax

The fundamental format for regular expressions is one of the following, depending on what you are trying to do:

/search_pattern/modifiers

command/search_pattern/modifiers

command/search_pattern/replacement/modifiers

The first syntax is a basic search syntax. In the absence of a command prefix, such a regular expression returns the lines matching the search pattern. In some cases, the slash marks may be (or must be) omitted—in the pattern argument to the grep command, for example.

The second syntax is used for most commands. In this form, some operation occurs on lines matching the pattern. This may be a form of matching, or it may involve removing the portions of the line that match the pattern.

The third syntax is used for substitution commands. These can be thought of as a more complex form of search and replace.

For example, the following command searches for the word 'test' within the specified file:

# Expression: /test/

grep 'test' poem.txt

The availability of commands and flags varies somewhat between regular expression variants, and is described in the relevant sections.

Positional Anchors and Flags

A common way to significantly alter regular expression matching is through the use of positional anchors and flags.

Positional anchors allow you to specify the position within a line of text where an expression is allowed to match. There are two positional anchors that are regularly used: caret (^) and dollar ($). When placed at the beginning or end of an expression, these match the beginning and end of a line of text, respectively.

For example:

# Expression: /^Mary/

grep "^Mary" < poem.txt

This matches the word "Mary", but only when it appears at the beginning of a line. Similarly, the following matches the word "fox," but only at the end of a line:

# Expression: /fox$/

grep "fox$" < poem.txt

The other common technique for altering the matching behavior of a regular expression is through the use of flags. These flags, when placed at the end of a regular expression, can change whether a regular expression is allowed to match across multiple lines, whether the matching is case sensitive or insensitive, and various other aspects of matching.

The most commonly used flag is the global flag. By default, only the first occurrence of a search term is matched. This is mainly of concern when performing substitutions. The global flag changes this so that a substitution alters every match in the line instead of just the first one.

For example:

# Expression: s/Mary/Joe/

sed "s/Mary/Joe/" < poem.txt

This replaces only the first occurrence of "Mary" with "Joe." By adding the global flag to the expression, it instead replaces every occurrence, as shown in the following example:

# Expression s/Mary/Joe/g

sed "s/Mary/Joe/g" < poem.txt

Wildcards and Repetition Operators

One of the most common ways to enhance searching through regular expressions is with the use of wildcard matching.

A wildcard is a symbol that takes the place of any other symbol. In regular expressions, a period (.) is considered a wildcard, as it matches any single character. For example:

# Expression: /wa./

grep 'wa.' poem.txt

This matches lines containing both "was" and "want" because the dot can match any character.

Wildcards are typically combined with repetition operators to match lines in which only a portion of the content is known. For example, you might want to search for every line containing "Mary" with the word "lamb" appearing later. You might specify the expression like this:

# Expression: /Mary.*lamb/

grep "Mary.*lamb" poem.txt

This searches for Mary followed by zero or more characters, followed by lamb.

Of course, you probably want at least one character between those to avoid matches for strings containing "Marylamb". The most common way to solve this is with the plus (+) operator. However, you can construct this expression in several ways:

# Expression (Basic): /Mary.\+lamb/

# Expression (Extended): /Mary.+lamb/

# Expression: /Mary..*lamb/

grep "Mary.\+lamb" poem.txt

grep -E "Mary.+lamb" poem.txt    # extended regexp

grep "Mary..*lamb" poem.txt

The first dot in the third expression matches a single character. The dot-asterisk afterwards matches be zero or more additional characters. Thus, these three statements are equivalent.

The final useful repetition operator is the question mark operator (?). This operator matches zero or one repetitions of whatever precedes it.

For example, if you want to match both Mary and Marry, you might use an expression like this:

# Expression (Basic): /Marr\?y/

# Expression (Extended): /Marr?y/

grep "Marr\?y" poem.txt

grep -E "Marr?y" poem.txt

The question mark causes the preceding r to be optional, and thus, this expression matches lines containing either “Mary” or “Marry.”

In summary, the basic wildcard and repetition operators are:

• period (.)—wildcard; matches a single character.

• question mark (\? or ?)—matches 0 or 1 of the previous character, grouping, or wildcard. (This operator differs depending on whether you are using basic or extended regular expressions.)

• asterisk(*)—matches zero or more of the previous character, grouping, or wildcard.

• plus(\+ or +)—matches one or more of the previous character, grouping, or wildcard. (This operator differs depending on whether you are using basic or extended regular expressions.)

Character Classes and Groups

Searching for certain keywords can be useful, but it is often not enough. It is often useful to search for the presence or absence of key characters at a given position in a search string.

For example, assume that you require the words Mary and lamb to be within the same sentence. To do this, you need to only allow certain characters to appear between the two words. This can be achieved through the use of character classes.

There are two basic types of character classes: predefined character classes and custom, or user-defined character classes. These are described in the following sections.

# Expression: /Mary[[:alpha:][:digit:][:blank:]][[:alpha:][:digit:][:blank:]]*lamb/

grep 'Mary[[:alpha:][:digit:][:blank:]][[:alpha:][:digit:][:blank:]]*lamb' poem.txt

# Expression: /Mary[a-z A-Z]*lamb/

grep "Mary[a-z A-Z]*lamb" poem.txt

# /Mary[^.]*lamb/

grep "Mary[^.]*lamb" poem.txt

#Expression (Basic): /Mary \(had \)\?a/

#Expression (Extended): /Mary (had )?a/

grep "Mary \(had \)\?a" poem.txt

grep -E "Mary (had )?a" poem.txt

#Expression (Basic): /\(Mary\|had\|lamb\)/

#Expression (Extended): /(Mary|had|lamb)/

grep '\(Mary\|had\|lamb\)' poem.txt

grep -E '(Mary|had|lamb)' poem.txt

sed -E 's/(lamb|lamb,)/orange/' poem.txt

sed -E 's/(lamb,|lamb)/orange/' poem.txt

perl -pi.bak -e 's/(lamb|lamb,)/orange/' < poem.txt

perl -pi.bak -e 's/(lamb,|lamb)/orange/' < poem.txt

perl -pi.bak -e 's/lamb,?/orange/' < poem.txt

# Expression (Extended): /const(ant|ellation|) (.*)/

# Expression (Extended): /const(ant|ellation)? (.*)/

grep -E 'const(ant|ellation)? (.*)'

# Expression (Extended): /const(ant|ellation|()) (.*)/

grep -E "const(ant|ellation|()) (.*)" poem.txt

Quoting Special Characters

As seen in previous sections, a number of characters have special meaning in regular expressions. For example, character classes are surrounded by square brackets, and the dash and caret characters have special meaning. You might ask how you can search for one of these characters. This is where quoting comes in.

In regular expressions, certain nonletter characters may have some special meaning, depending on context. To treat these characters as an ordinary character, you can prefix them with a backslash character (\). This also means that the backslash character is special in any context, so to match a literal backslash character, you must quote it with a second backslash.

There is one exception, however. To make a close bracket be a member of a character class, you do not quote it. Instead, you make it be the first character in the class.

For example, to search for any string containing a backslash or a close bracket, you might use the following regular expression:

# Expression: /[]\\]/

grep '[]\\]' poem.txt

It looks a bit cryptic, but it is really relatively straightforward. The outer slashes delimit the regular expression. The brackets immediately inside the outer slashes are character class delimiters. The first close bracket immediately follows the open bracket, which makes it match an actual close bracket character instead of ending the character class. The two backslashes afterwards are, in fact, a quoted backslash, which makes this character class match the literal backslash character.

As a general rule, at least in extended regular expressions, any nonalphanumeric character can safely be quoted whether it is necessary to do so or not. If quoting it is not necessary, the extra backslash is simply ignored. However, it is not always safe to quote letters or numbers, as these have special meanings in certain regular expression dialects, as described in Capturing Operators and Variables and Perl and Python Extensions. In addition, quoting parentheses may not do what you might expect in some dialects, as described in Capturing Operators and Variables.

In basic regular expressions the behavior when quoting characters other than parentheses, curly braces, numbers, and characters within a character class is undefined.

Capturing Operators and Variables

In Wildcards and Repetition Operators, this chapter described ways to create more complicated patterns to match for the search portion of a search and replace operation. This section describes more powerful operations for the replacement portion of a search and replace operation.

Capturing operators and variables are used to take pieces of the original input text, capture them while searching, and then substitute those bits into the middle of the replacement text.

The easiest way to explain capturing operators and variables is by example. Suppose you want to swap the words quick and lazy in the string, "The quick brown fox jumped over the lazy dog." You might write an expression like this:

# Expression (Basic): s/The \(.*\) brown \(.*\) the \(.*\) dog/The \3 brown \2 the \1 dog/

# Expression (Extended): s/The (.*) brown (.*) the (.*) dog/The \3 brown \2 the \1 dog/

When you pass these expressions to sed, the last line of poem.txt should become "The lazy brown fox jumped over the quick dog."

# Expression (Basic): s/The (.*) brown (.*) the (.*) dog/The \3 brown \2 the \1 dog/

sed "s/The \(.*\) brown \(.*\) the \(.*\) dog/The \3 brown \2 the \1 dog/" < poem.txt

# Expression (Extended): s/The \(.*\) brown \(.*\) the \(.*\) dog/The \3 brown \2 the \1 dog/

sed -E "s/The (.*) brown (.*) the (.*) dog/The \3 brown \2 the \1 dog/" < poem.txt

# Perl supports extended form, but also supports

# using a dollar sign for the variable name.  (Note

# the use of single quotes to prevent the shell from

# doing variable substitution on $1, $2, and $3.)

perl -pi.bak -e "s/The (.*) brown (.*) the (.*) dog/The \3 brown \2 the \1 dog/" < poem.txt

perl -pi.bak -e 's/The (.*) brown (.*) the (.*) dog/The $3 brown $2 the $1 dog/' < poem.txt

The content between each pair of parentheses (in this case—see note) is captured into its own buffer, numbered consecutively. Thus, in this expression, the content between “the” and “brown” is captured into a buffer. Then, the content between “brown” and “the” is captured. Finally, the content between “the” and “dog” is captured.

In the replacement string, the delimiter words (“The”, “brown”, “the”, and “dog”) are inserted, and the contents of the capture buffers are inserted in the opposite order.

Mixing Capturing and Grouping Operators

Since parentheses serve both as capturing and grouping operators, use of grouping may result in unexpected consequences when capturing text in the same expression. For example, the following expression will behave very differently depending on input:

# Expression /const(ant)? (.*)/

The text you probably intended to capture is in the second buffer, not the first.

Using Modifiers

The overall behavior of a regular expression can be tuned using a number of modifiers. For example:

/foo/i

In this example, the /i modifier makes the regular expression match in a case-insensitive fashion. Thus, this matches both “Foo” and “fOo”.

Not all commands and languages support all modifiers. For example, most versions of the sed command support only the /g modifier.

The basic modifiers are:

• /g—replace globally. Without this flag, a substitution command replaces only the first matching occurrence per line. With this flag, a substitution command also replaces subsequent matches.

• /i—use case insensitive matching (Perl extension; equivalent to grep -i).

• /m—multiline matching (Perl extension). the $ and ^ anchors should match at newline boundaries in addition to matching at the beginning an end of the string as a whole. The dot (.) does not match newline characters.

• /o—compile once (Perl extension). In Perl, if a regular expression includes a variable as part of the pattern, the regular expression engine must recompile the expression every time it is used because the variable contents might have changed.

  If you know that the contents will not change after they are set the first time, the /o flag disables recompilation of the expression. For regular expressions that do not contain variables, this switch has no effect.

• /s—single-line matching (Perl extension). The $ and ^ anchors should not match at newline boundaries. With this modifier, they only match at the very beginning and end of the string as a whole. The dot (.) matches newline characters just like any other character.

• /x—extend readability (Perl extension). This mode causes matching to ignore all whitespace between tokens in the expression unless quoted or wrapped in brackets (in most languages) and to treat a hash mark (#) as the start of a single-line comment.

  The purpose of this mode is to allow you to split complex regular expressions into multiple lines. For example, in Perl, you might detect a date like this:

  if ($foo =~ /(\d\d\d\d) # year

  \s*-\s* # separator

  (\d\d) # month

  \s*-\s* # separator

  (\d\d) # day

  /x) {

  print "Date detected\n";

  }

  The syntactical details vary from language to language.

Perl and Python Extensions

The regular expression dialect used in Perl, Python, and many other languages, are a further extension of extended regular expressions. Some of the major differences include:

• Addition of shortcuts for character classes. See Character Class Shortcuts.

• Addition of quotation operators. In a regular expression, the contents of variables appearing between \Q and \E are automatically quoted, and thus treated as literal text even if the variable contains characters that ordinarily have special meaning in a regular expression. These operators are useful when user input, stored in a Perl variable, is used as part of a regular expression.

• Support for retrieving captured values outside the scope of the expression; the captured values are stored in the variables $1, $2, and so on. (See Capturing Operators and Variables for information about capturing parts of a regular expression.)

• Addition of nongreedy matching. See Nongreedy Wildcard Matching for more information.

• Noncapturing parentheses. See Noncapturing Parentheses for more information.

You can find links to additional resources that describe these extensions in For More Information.

/Mary.*lamb/

/Mary.*?lamb/

# Expression (Perl and Similar ONLY): /Mary (?:had)* a little lamb\./

perl -pi.bak -e "s/Mary (?:had )*a little lamb\./Lovely day, isn't it?/" < poem.txt

Using Regular Expressions in Control Statements

The shell’s test command (described in The test Command and Bracket Notation) does not natively support regular expressions, so in order to use regular expressions in control statements, you must take advantage of the ability to execute arbitrary external commands (more specifically, the grep command) instead of using bracket notation.

As shown throughout this chapter, the grep command takes a stream of text (or a path or list of paths) and prints every line that matches the specified regular expression. What you may not have noticed, however, is that its exit status changes depending on whether the input matches the specified expression.

The grep command exits with a successful exit status (0) if the input matches the specified expression at least once or a failed exit status (generally 1) if the pattern does not match. Thus, you can easily use it to control an if statement.

For example:

if (echo "$MYVAR" | grep "bar" > /dev/null) ; then

    echo "The value of MYVAR ($MYVAR) contains \"bar\"."

fi

In the above example, the rightmost exit status (from grep) is treated as the exit status for the group of commands (assuming that the echo command succeeds, which it always should). The redirect to /dev/null prevents the text output from being printed to the user’s screen.

Regular expressions can also be used in other control statements such as while loops. For example, the following snippet counts the occurrences of the letter ‘x’ in a single-line string:

MYVAR="xxxxxx"

while (echo "$MYVAR" | grep 'x' > /dev/null) ; do

    # Be sure to change MYVAR here!

    echo "got x"

    MYVAR="$(echo "$MYVAR" | sed -E 's/x//')"

done

Of course, this contrived snippet is a good example of when you should avoid regular expressions; testing for an empty string makes this snippet run roughly twice as fast.