Specifying a Regular Expression

Contents of Article


Introduction

Performance Considerations

Compatibility Considerations

Examples

Using the SPFTest Program


Introduction


Using a Regular Expression string in a FIND, CHANGE, JOIN, EXCLUDE or similar command provides you with a powerful tool for searching.  If you have never used regular expressions they may appear quite complex.  However many users will already be familiar with them from their use on other systems like Linux or Unix.


Note: Users familiar with regular expressions should be aware that there are many variants of regular expression syntax implemented in different systems and applications, some with fairly subtle differences.  The description below describes the regular expression engine supported by SPFLite.  The RegEx engine chosen for SPFLite is PCRE (Perl Compatible Regular Expression).  This engine is widely used and supports a robust RegEx syntax.  Because regular expressions can be complex, SPFLite provides a small program to allow you to experiment with Regular Expressions and become familiar with their use.  See Using the SPFTest Program for details.  


There are several sites on the web providing assistance in learning about and testing Regular Expressions.  One such tutorial can be found at www.regexlab.com.


Note: Starting with release 8.3 of SPFLite, it is possible for a Regular Expression to find strings of length zero, because of  the new PCRE Regular Expression engine; this is something that did not occur in prior releases.  This should not affect most users, but if you encounter any issues because of this, be sure to let us know on the SPFLite forum, so we can make sure everything is working properly.


The regular expression must be enclosed in quotes (single, double, or accent) preceded or followed by the character R.  

Examples:

R'abc$'

'abc$'R

SubTopics




PCRE Regular Expression Syntax Summary


This PCRE Syntax documentation courtesy of www.pcre.org - Copyright © 1997-2014 University of Cambridge.  This is a summary only. The full description can be found at http://www.pcre.org/original/doc/html/pcrepattern.html



QUOTING


 \x         where x is non-alphanumeric is a literal x

 \Q...\E    treat enclosed characters as literal


Note that in PCRE, the code \Q differs from prior SPFLite RegEx usage of \q.

See Compatibility Considerations below for more information.


CHARACTERS


 \a         alarm, that is, the BEL character (hex 07)

 \cx        "control-x", where x is any ASCII character

 \e         escape (hex 1B)

 \f         form feed (hex 0C)

 \n         newline (hex 0A)

 \r         carriage return (hex 0D)

 \t         tab (hex 09)

 \0dd       character with octal code 0dd

 \ddd       character with octal code ddd, or backreference

 \o{ddd..}  character with octal code ddd..

 \xhh       character with hex code hh

 \x{hhh..}  character with hex code hhh..


Note that \0dd is always an octal code, and that \8 and \9 are the literal characters "8" and "9".

Note that in PCRE, the code \c differs from prior SPFLite RegEx usage.

See Compatibility Considerations below for more information.



CHARACTER TYPES


These codes represent any character except for newline.  In "dotall mode", these  codes represent any character whatsoever.


 \C         one data unit, even in UTF mode (best avoided)

 \d         a decimal digit

 \D         a character that is not a decimal digit

 \h         a horizontal white space character

 \H         a character that is not a horizontal white space character

 \N         a character that is not a newline

 \p{xx}     a character with the xx property

 \P{xx}     a character without the xx property

 \R         a newline sequence

 \s         a white space character

 \S         a character that is not a white space character

 \v         a vertical white space character

 \V         a character that is not a vertical white space character

 \w         a "word" character

 \W         a "non-word" character

 \X         a Unicode extended grapheme cluster


By default, \d, \s, and \w match only ASCII characters.

Note that in PCRE, the code \s differs from prior SPFLite RegEx usage.

See Compatibility Considerations below for more information.



GENERAL CATEGORY PROPERTIES FOR \p and \P


 C          Other

 Cc         Control

 Cf         Format

 Cn         Unassigned

 Co         Private use

 Cs         Surrogate


 L          Letter

 Ll         Lower case letter

 Lm         Modifier letter

 Lo         Other letter

 Lt         Title case letter

 Lu         Upper case letter

 L&         Ll, Lu, or Lt


 M          Mark

 Mc         Spacing mark

 Me         Enclosing mark

 Mn         Non-spacing mark


 N          Number

 Nd         Decimal number

 Nl         Letter number

 No         Other number


 P          Punctuation

 Pc         Connector punctuation

 Pd         Dash punctuation

 Pe         Close punctuation

 Pf         Final punctuation

 Pi         Initial punctuation

 Po         Other punctuation

 Ps         Open punctuation


 S          Symbol

 Sc         Currency symbol

 Sk         Modifier symbol

 Sm         Mathematical symbol

 So         Other symbol


 Z          Separator

 Zl         Line separator

 Zp         Paragraph separator

 Zs         Space separator



PCRE SPECIAL CATEGORY PROPERTIES FOR \p and \P


 Xan        Alphanumeric: union of properties L and N

 Xps        POSIX space: property Z or tab, NL, VT, FF, CR

 Xsp        Perl space: property Z or tab, NL, VT, FF, CR

 Xuc        Univerally-named character: one that can be

            represented by a Universal Character Name

 Xwd        Perl word: property Xan or underscore



CHARACTER CLASSES


 [...]       positive character class

 [^...]      negative character class

 [x-y]       range (can be used for hex characters)

 [[:xxx:]]   positive POSIX named set; see list below

 [[:^xxx:]]  negative POSIX named set; see list below


    xxx   can be one of the following:


 alnum       alphanumeric

 alpha       alphabetic

 ascii       characters in the range of 0 to 127 (\x00 to \x7F)

 blank       space or tab

 cntrl       control character

 digit       decimal digit

 graph       printing, excluding space

 lower       lower case letter

 print       printing, including space

 punct       printing, excluding alphanumeric

 space       white space

 upper       upper case letter

 word        same as \w

 xdigit      hexadecimal digit



QUANTIFIERS


 ?           0 or 1, greedy

 ?+          0 or 1, possessive

 ??          0 or 1, lazy

 *           0 or more, greedy

 *+          0 or more, possessive

 *?          0 or more, lazy

 +           1 or more, greedy

 ++          1 or more, possessive

 +?          1 or more, lazy

 {n}         exactly n

 {n,m}       at least n, no more than m, greedy

 {n,m}+      at least n, no more than m, possessive

 {n,m}?      at least n, no more than m, lazy

 {n,}        n or more, greedy

 {n,}+       n or more, possessive

 {n,}?       n or more, lazy


Note that in PCRE, the quantifier code ? may be used in some cases like the code \s from prior SPFLite RegEx usage.

See Compatibility Considerations below for more information.



ANCHORS AND SIMPLE ASSERTIONS


 \b          word boundary

 \B          not a word boundary

 ^           start of subject

              also after internal newline in multiline mode

 \A          start of subject

 $           end of subject

              also before newline at end of subject

              also before internal newline in multiline mode

 \Z          end of subject

              also before newline at end of subject

 \z          end of subject

 \G          first matching position in subject



MATCH POINT RESET


 \K          reset start of match

\K is honored in positive assertions, but ignored in negative ones.



ALTERNATION


 expr|expr|expr...



CAPTURING


 (...)           capturing group

 (?<name>...)    named capturing group (Perl)

 (?'name'...)    named capturing group (Perl)

 (?P<name>...)   named capturing group (Python)

 (?:...)         non-capturing group

 (?|...)         non-capturing group; reset group numbers for

                 capturing groups in each alternative


ATOMIC GROUPS


 (?>...)         atomic, non-capturing group



COMMENT


 (?#....)        comment (not nestable)



OPTION SETTING


 (?i)            caseless

 (?J)            allow duplicate names

 (?m)            multiline

 (?s)            single line (dotall)

 (?U)            default ungreedy (lazy)

 (?x)            extended (ignore white space)

 (?-...)         unset option(s)


The following are recognized only at the very start of a pattern or after one of the newline or \R options with similar syntax. More than one of them may appear.


 (*LIMIT_MATCH=d) set the match limit to d (decimal number)

 (*LIMIT_RECURSION=d) set the recursion limit to d (decimal number)

 (*NO_AUTO_POSSESS) no auto-possessification (PCRE_NO_AUTO_POSSESS)

 (*NO_START_OPT) no start-match optimization (PCRE_NO_START_OPTIMIZE)

 (*UTF8)         set UTF-8 mode: 8-bit library (PCRE_UTF8)

 (*UTF16)        set UTF-16 mode: 16-bit library (PCRE_UTF16)

 (*UTF32)        set UTF-32 mode: 32-bit library (PCRE_UTF32)

 (*UTF)          set appropriate UTF mode for the library in use

 (*UCP)          set PCRE_UCP (use Unicode properties for \d etc)


Note that LIMIT_MATCH and LIMIT_RECURSION can only reduce the value of the limits set by the caller of pcre_exec(), not increase them.



NEWLINE CONVENTION


These are recognized only at the very start of the pattern or after option settings with a similar syntax.

 (*CR)           carriage return only

 (*LF)           linefeed only

 (*CRLF)         carriage return followed by linefeed

 (*ANYCRLF)      all three of the above

 (*ANY)          any Unicode newline sequence



WHAT  \R  MATCHES


These are recognized only at the very start of the pattern or after option setting with a similar syntax.


 (*BSR_ANYCRLF)  CR, LF, or CRLF

 (*BSR_UNICODE)  any Unicode newline sequence


Note: Be careful not to confuse \R with \r.  The code \r only means a carriage return character (\x0D), while the meaning of \R depends on the settings above.


LOOKAHEAD AND LOOKBEHIND ASSERTIONS


 (?=...)         positive look ahead

 (?!...)         negative look ahead

 (?<=...)        positive look behind

 (?<!...)        negative look behind


Each top-level branch of a look behind must be of a fixed length.


BACKREFERENCES


 \n              reference by number (can be ambiguous)

 \gn             reference by number

 \g{n}           reference by number

 \g{-n}          relative reference by number

 \k<name>        reference by name (Perl)

 \k'name'        reference by name (Perl)

 \g{name}        reference by name (Perl)

 \k{name}        reference by name (.NET)

 (?P=name)       reference by name (Python)



SUBROUTINE REFERENCES (POSSIBLY RECURSIVE)


 (?R)            recurse whole pattern

 (?n)            call subpattern by absolute number

 (?+n)           call subpattern by relative number

 (?-n)           call subpattern by relative number

 (?&name)        call subpattern by name (Perl)

 (?P>name)       call subpattern by name (Python)

 \g<name>        call subpattern by name (Oniguruma)

 \g'name'        call subpattern by name (Oniguruma)

 \g<n>           call subpattern by absolute number (Oniguruma)

 \g'n'           call subpattern by absolute number (Oniguruma)

 \g<+n>          call subpattern by relative number (PCRE extension)

 \g'+n'          call subpattern by relative number (PCRE extension)

 \g<-n>          call subpattern by relative number (PCRE extension)

 \g'-n'          call subpattern by relative number (PCRE extension)



CONDITIONAL PATTERNS


 (?(condition)yes-pattern)

 (?(condition)yes-pattern|no-pattern)


 (?(n)...        absolute reference condition

 (?(+n)...       relative reference condition

 (?(-n)...       relative reference condition

 (?(<name>)...   named reference condition (Perl)

 (?('name')...   named reference condition (Perl)

 (?(name)...     named reference condition (PCRE)

 (?(R)...        overall recursion condition

 (?(Rn)...       specific group recursion condition

 (?(R&name)...   specific recursion condition

 (?(DEFINE)...   define subpattern for reference

 (?(assert)...   assertion condition



BACKTRACKING CONTROL


The following act immediately they are reached:


 (*ACCEPT)       force successful match

 (*FAIL)         force backtrack; synonym (*F)

 (*MARK:NAME)    set name to be passed back; synonym (*:NAME)


The following act only when a subsequent match failure causes a backtrack to reach them. They all force a match failure, but they differ in what happens afterwards. Those that advance the start-of-match point do so only if the pattern is not anchored.


 (*COMMIT)       overall failure, no advance of starting point

 (*PRUNE)        advance to next starting character

 (*PRUNE:NAME)   equivalent to (*MARK:NAME)(*PRUNE)

 (*SKIP)         advance to current matching position

 (*SKIP:NAME)    advance to position corresponding to an earlier

                 (*MARK:NAME); if not found, the (*SKIP) is ignored

 (*THEN)         local failure, backtrack to next alternation

 (*THEN:NAME)    equivalent to (*MARK:NAME)(*THEN)



CALLOUTS


 (?C)      callout

 (?Cn)     callout with data n



Performance Considerations

Be aware that the *, + and ? meta-characters can cause the regular expression engine to perform backtracking, which can cause its pattern-matching to run a little slower.  As with all regular expressions, performance is dependent on file size and the complexity of the expression.


Compatibility Considerations

The PCRE regular expression syntax is mostly compatible with what SPFLite offered in prior releases.  You will generally not need to be concerned about compatibility if you only issue RegEx expressions on the command line with R strings.  If you have stored regular expressions, either in a macro or a keyboard definition, you should be aware of the following differences.

Pre-PCRE Regular Expression syntax prior to release 8.3

PCRE Regular Expression syntax as of release 8.3



\c


Signifies a case-sensitive search

PRCE uses this code in an incompatible way.  The PCRE code \cx to indicate a "control" character.  For example, if you want to use a Ctrl-A as a literal value, you would specify \cA.


SPFLite honors the CASE C/T setting when executing a PCRE regular expression.  Where you might previously have used \c to mean "case sensitive", you can now issue a CASE C primary command prior to executing a regular expression.


If you want to embed a code in your RegEx string to control case sensitivity, (?i) means case insensitive (like CASE T) and (?-i) means case sensitive (like CASE C).


\q


Signifies a literal double-quote character "

PCRE does not implement the \q code.  It does have a \Q code, but with an unrelated meaning.


The prior RegEx engine allowed this code to make it easier to specify literal double-quote characters in Basic strings.  You will generally not need to do this.  A double-quote character " has no special meaning in a PCRE RegEx, and can be directly specified as a data value, as long it's not also being used to enclose the R string as a whole.


If you are creating a RegEx string within a SPFLite macro script, you can use the Basic string equate $DQ to get a double-quote character.


You can also insert a double quote by using the RegEx hex value for the ASCII double quote, which is \x22.


\s


Signifies a request to match the shortest-possible string

PRCE uses this code in an incompatible way.  The PCRE code \s is used to search for a "white space" character.


PCRE uses the term "greedy" when a string-matching process could match against a variable number of characters, and the longest possible length is taken.  When the shortest-possible length is taken, this is called "lazy" or "non-greedy".  Thus, the old code of \s meant "lazy" or "non-greedy" in PCRE terminology.


To set "lazy" matching in PCRE, you can use a general option setting code of (?U), and greedy mode would be (?-U).  These options affect the entire RegEx expression.  PCRE matching is "greedy" by default, so the code (?-U) is not usually required.


PCRE also lets you control the greedy-vs-lazy matching on a finer level, using a ? in certain types of syntax.  See the PCRE documentation under QUANTIFIERS for more information.



Examples

Given a file with lines:

"Blah blah blah blah"

"Please send email to FredFarkle@mydomain.com"

"Thank you very much."

The line with the email address could be found with:

FIND R"([a-z0-9._/+-]+)(@[a-z0-9.-]+)"

Given the lines:

"Balance due by 15th of the month"

"Amount owed: $42.75 and is overdue!"

The line with the Balance amount could be found with:

FIND R"\$[0-9.,]+"


Created with the Personal Edition of HelpNDoc: Full-featured Documentation generator