A guide to regular expressions on linux

Many tools in unix systems support matching patterns: grep, less, sed, just to name a few. Knowing how to use them greatly increases their ability to filter and transform text on the terminal or in scripts.

Basic and extended regular expression syntax

The POSIX standard defines almost exactly the same set of regular expression features, but with two different syntax sets.

The first is BRE ("Basic Regular Expressions"), which treats almost all characters as literal text and hides almost all syntax behind escape sequences. In contrast, ERE ("Extended Regular Expressions") instead behave more like regular expression syntax familiar from programming languages, where syntax is expressed as normal symbols, and matching the literal symbol requires an escape sequence instead. Additionally, ERE extends the feature set by two common operators missing from BRE.

Most POSIX tools supporting expression (such as grep, less, awk or sed) default to BRE syntax, but can switch to ERE when passing the -E flag (must be uppercase).

Basic text matching

Regular expressions match any characters not defined as part of the syntax literally, meaning this is a valid regular expression:

dog

and will match any line containing the word "dog" anywhere (case sensitive).

As a special case, a dot . is a placeholder for any character:

d.g

This expression would still match "dog" anywhere in the line, but also any other text with a single character between d and g, for example "dig", "dug" or "d4g".

Character classes

Likely one of the most powerful concepts of BRE and ERE are character classes, expressed using square brackets. There are two types, the most widely known being custom classes where the user defines a set of characters or ranges:

[abc]

The example would match one character if it is either a, b or c. Ranges can be expressed using hyphens:

[a-zA-Z0-9]

If you want to match literal square brackets instead of character classes, you need to escape them with backslashes like \[ or \].

Character classes can be inverted to match anything except the defined set by leading with ^:

[^a-z]

This would match anything except a lowercase alphabetical letter.

The second type of character classes are the builtin ones with predefined contents:

[[:digit:]]: A single digit

[[:alpha:]]: Alphabetical character

[[:alnum:]]: Alphanumeric character or digit

[[:blank:]]: Space of horizontal tab (\t)

[[:space:]]: Whitespace character, one of: (space), \t, \n, \v, \f, \r)

[[:cntrl:]]: Invisible control character, the first 32 ASCII characters (0x00 - 0x1F) and the last one (0x7F). Does not include whitespace characters.

[[:graph:]]: Graphical character - any character that produces a visible glyph when printed (meaning no control characters or whitespace)

[[:print:]]: Printable character - any character that can be printed (including whitespace), meaning anything other than control characters

[[:lower:]]: An lowercase character

[[:upper:]]: An uppercase character

[[:punct:]]: A symbol character, includes normal punctuation like dots and commas, but also all symbols like dollar signs and parenthesis

[[:xdigit:]]: Hexadecimal digit, matching 0-9 or a-f (case insensitive)

Both character classes can be combined in a single custom character class, but note that predefined ones use only one set of square brackets when used within a custom class:

[a-zABC[:digit:]]

The above expression would match any lowercase letter, or the uppercase letters A, B or C, or any digit. If you instead want to use a predefined character class alone, you always have to wrap it in two square bracket pairs, because they are only valid within square bracket expressions:

[[:alpha:]]

Character classes depend on system locale, meaning their definitions change depending on the local alphabet. For example, the pattern [a-z] or [[:alpha:]] would match the letter ä using a german or UTF-8 locale, but not for an american or british one.

Grouping and back references

Regular expressions can include groups of subexpressions, which can double as capturing groups in some contexts.

A group is defined by writing an expression between \( and \) in BRE, or( and ) In ERE.

This difference means that the same regular expression will behave differently depending on mode.

For example, the expression:

(dog)

Will match lines containing "dog" in ERE, but only ones containing "(dog") in BRE syntax.

When replacing text using tools like sed, groups can function as capturing groups that extract the matched content and can paste it somewhere else.

Consider this example sed command:

sed -E 's/Usage: ([0-9][0-9])%/\1/g'

Ignoring the sed regex syntax elements, the expression first looks for matches to Usage: ([0-9][0-9])%, for example a string like "Usage: 51%". It then extracts the two digits just before and replaces the entire line with \1, which will paste the first match of the capture group (51) in this case.

Successive capturing group matches can be accessed with backreferences, written as\1, \2, \3 and so on. Backreferences can also be used in the matching pattern directly:

BRE:

\([0-9]\)\1

ERE:

([0-9])\1

This pattern matches any digit that repeats once: the first capturing group matches any digit, and the second backreference \1 matches the previously matched digit a second time.

Repetition

Any matching rule, from literal characters to ranges or entire matching groups can be allowed a specific amount of times.

The most commonly known is * that matches the previous rule zero or more times. The following expression:

ab*

Will match a lowercase a followed by any amount of b characters, meaning all of these would match: "a", "ab", "abbbb", "abbbbbbbbbbbbbbb".

Any preceding rule can be repeated, including character classes and entire groups:

[a-z]*
(ab.d)*

The more specific curly brace syntax instead matches the previous rule a set number of times.

The simplest is to require an exact amount:

BRE:

A\{3\}

ERE:

A{3}

These expressions match A only if it appears exactly three times in a row, meaning only the string "AAA".

When supplying two numbers inside curly brackets, the preceding rule is matched between a minimum and maximum amount of times instead. As with normal groups, BRE requires curly braces to be escaped while ERE does not:

BRE:

A\{2,3\}

ERE:

A{2,3}

Both expressions match the character A, but only if they are at least 2, at most 3 in a row. Note that the upper limit is optional and can be omitted:

ERE:

A{2,}

This expression matches the letter A, but only if it is repeated 2 or more times.

The ERE syntax extends the expressions with the symbols + and ? that are common in most regular expression formats, but missing from BRE entirely.

The + symbol matches the previous rule one or more times, while ? matches it zero or at most one time.

Alternation

Both expression formats include support for alternation to define multiple possible values for the same position with \| in BRE or only| in ERE.

BRE:

cat\|dog

ERE:

cat|dog

The expression matches any line containing either "cat" or "dog". Alternatives apply to the entire preceding expression. In order to restrict what it can match, confine it to a group:

ERE

I like (cat|dog)s

Now the expression matches either "I like cats" or "I like dogs".

Anchors

Expression patterns match any part of a line by default. To instead define where that match is supposed to be, you can use the anchors ^ for the beginning of the line and $ to signal its end. Both anchors do not require special quoting in BRE or ERE.

^foo
foo$
^foo$

The first pattern only matches lines starting with "foo", the second only lines ending with "foo", and the last one exclusively lines that are exactly "foo" (nothing before or after).