Contents of Article


Introduction

Performance Considerations

Compatibility Considerations

Examples

Using the SPFTest Program


Introduction


Using a Regular Expression string in a FIND, CHANGE, JOIN, EXCLUDE or similar command provides you with a powerful tool for searching. If you have never used regular expressions they may appear quite complex. However many users will already be familiar with them from their use on other systems like Linux or Unix.


Note: Users familiar with regular expressions should be aware that there are many variants of regular expression syntax implemented in different systems and applications, some with fairly subtle differences. The description below describes the regular expression engine supported by SPFLite. The RegEx engine chosen for SPFLite is PCRE (Perl Compatible Regular Expression). This engine is widely used and supports a robust RegEx syntax. Because regular expressions can be complex, SPFLite provides a small program to allow you to experiment with Regular Expressions and become familiar with their use. See Using the SPFTest Program for details.


There are several sites on the web providing assistance in learning about and testing Regular Expressions. One such tutorial can be found at www.regexlab.com.


The regular expression must be enclosed in quotes (single, double, or accent) preceded or followed by the character R.

 

Examples:

R'abc$'

'abc$'R

SubTopics




PCRE Regular Expression Syntax Summary


This PCRE Syntax documentation courtesy of www.pcre.org - Copyright © 1997-2014 University of Cambridge. This is a summary only. The full description can be found at http://www.pcre.org/original/doc/html/pcrepattern.html



QUOTING


  \x         where x is non-alphanumeric is a literal x

  \Q...\E    treat enclosed characters as literal


Note that in PCRE, the code \Q differs from prior SPFLite RegEx usage of \q.

See Compatibility Considerations below for more information.


CHARACTERS


  \a         alarm, that is, the BEL character (hex 07)

  \cx        "control-x", where x is any ASCII character

  \e         escape (hex 1B)

  \f         form feed (hex 0C)

  \n         newline (hex 0A)

  \r         carriage return (hex 0D)

  \t         tab (hex 09)

  \0dd       character with octal code 0dd

  \ddd       character with octal code ddd, or backreference

  \o{ddd..}  character with octal code ddd..

  \xhh       character with hex code hh

  \x{hhh..}  character with hex code hhh..


Note that \0dd is always an octal code, and that \8 and \9 are the literal characters "8" and "9".

Note that in PCRE, the code \c differs from prior SPFLite RegEx usage.

See Compatibility Considerations below for more information.



CHARACTER TYPES


These codes represent any character except for newline. In "dotall mode", these  codes represent any character whatsoever.


  \C         one data unit, even in UTF mode (best avoided)

  \d         a decimal digit

  \D         a character that is not a decimal digit

  \h         a horizontal white space character

  \H         a character that is not a horizontal white space character

  \N         a character that is not a newline

  \p{xx}     a character with the xx property

  \P{xx}     a character without the xx property

  \R         a newline sequence

  \s         a white space character

  \S         a character that is not a white space character

  \v         a vertical white space character

  \V         a character that is not a vertical white space character

  \w         a "word" character

  \W         a "non-word" character

  \X         a Unicode extended grapheme cluster


By default, \d, \s, and \w match only ASCII characters.

Note that in PCRE, the code \s differs from prior SPFLite RegEx usage.

See Compatibility Considerations below for more information.



GENERAL CATEGORY PROPERTIES FOR \p and \P


  C          Other

  Cc         Control

  Cf         Format

  Cn         Unassigned

  Co         Private use

  Cs         Surrogate


  L          Letter

  Ll         Lower case letter

  Lm         Modifier letter

  Lo         Other letter

  Lt         Title case letter

  Lu         Upper case letter

  L&         Ll, Lu, or Lt


  M          Mark

  Mc         Spacing mark

  Me         Enclosing mark

  Mn         Non-spacing mark


  N          Number

  Nd         Decimal number

  Nl         Letter number

  No         Other number


  P          Punctuation

  Pc         Connector punctuation

  Pd         Dash punctuation

  Pe         Close punctuation

  Pf         Final punctuation

  Pi         Initial punctuation

  Po         Other punctuation

  Ps         Open punctuation


  S          Symbol

  Sc         Currency symbol

  Sk         Modifier symbol

  Sm         Mathematical symbol

  So         Other symbol


  Z          Separator

  Zl         Line separator

  Zp         Paragraph separator

  Zs         Space separator



PCRE SPECIAL CATEGORY PROPERTIES FOR \p and \P


  Xan        Alphanumeric: union of properties L and N

  Xps        POSIX space: property Z or tab, NL, VT, FF, CR

  Xsp        Perl space: property Z or tab, NL, VT, FF, CR

  Xuc        Univerally-named character: one that can be

             represented by a Universal Character Name

  Xwd        Perl word: property Xan or underscore



CHARACTER CLASSES


  [...]       positive character class

  [^...]      negative character class

  [x-y]       range (can be used for hex characters)

  [[:xxx:]]   positive POSIX named set; see list below

  [[:^xxx:]]  negative POSIX named set; see list below


     xxx   can be one of the following:


  alnum       alphanumeric

  alpha       alphabetic

  ascii       characters in the range of 0 to 127 (\x00 to \x7F)

  blank       space or tab

  cntrl       control character

  digit       decimal digit

  graph       printing, excluding space

  lower       lower case letter

  print       printing, including space

  punct       printing, excluding alphanumeric

  space       white space

  upper       upper case letter

  word        same as \w

  xdigit      hexadecimal digit



QUANTIFIERS


  ?           0 or 1, greedy

  ?+          0 or 1, possessive

  ??          0 or 1, lazy

  *           0 or more, greedy

  *+          0 or more, possessive

  *?          0 or more, lazy

  +           1 or more, greedy

  ++          1 or more, possessive

  +?          1 or more, lazy

  {n}         exactly n

  {n,m}       at least n, no more than m, greedy

  {n,m}+      at least n, no more than m, possessive

  {n,m}?      at least n, no more than m, lazy

  {n,}        n or more, greedy

  {n,}+       n or more, possessive

  {n,}?       n or more, lazy


Note that in PCRE, the quantifier code ? may be used in some cases like the code \s from prior SPFLite RegEx usage.

See Compatibility Considerations below for more information.



ANCHORS AND SIMPLE ASSERTIONS


  \b          word boundary

  \B          not a word boundary

  ^           start of subject

               also after internal newline in multiline mode

  \A          start of subject

  $           end of subject

               also before newline at end of subject

               also before internal newline in multiline mode

  \Z          end of subject

               also before newline at end of subject

  \z          end of subject

  \G          first matching position in subject



MATCH POINT RESET


  \K          reset start of match

\K is honored in positive assertions, but ignored in negative ones.



ALTERNATION


  expr|expr|expr...



CAPTURING


  (...)           capturing group

  (?<name>...)    named capturing group (Perl)

  (?'name'...)    named capturing group (Perl)

  (?P<name>...)   named capturing group (Python)

  (?:...)         non-capturing group

  (?|...)         non-capturing group; reset group numbers for

                  capturing groups in each alternative


ATOMIC GROUPS


  (?>...)         atomic, non-capturing group



COMMENT


  (?#....)        comment (not nestable)



OPTION SETTING


  (?i)            caseless

  (?J)            allow duplicate names

  (?m)            multiline

  (?s)            single line (dotall)

  (?U)            default ungreedy (lazy)

  (?x)            extended (ignore white space)

  (?-...)         unset option(s)


The following are recognized only at the very start of a pattern or after one of the newline or \R options with similar syntax. More than one of them may appear.


  (*LIMIT_MATCH=d) set the match limit to d (decimal number)

  (*LIMIT_RECURSION=d) set the recursion limit to d (decimal number)

  (*NO_AUTO_POSSESS) no auto-possessification (PCRE_NO_AUTO_POSSESS)

  (*NO_START_OPT) no start-match optimization (PCRE_NO_START_OPTIMIZE)

  (*UTF8)         set UTF-8 mode: 8-bit library (PCRE_UTF8)

  (*UTF16)        set UTF-16 mode: 16-bit library (PCRE_UTF16)

  (*UTF32)        set UTF-32 mode: 32-bit library (PCRE_UTF32)

  (*UTF)          set appropriate UTF mode for the library in use

  (*UCP)          set PCRE_UCP (use Unicode properties for \d etc)


Note that LIMIT_MATCH and LIMIT_RECURSION can only reduce the value of the limits set by the caller of pcre_exec(), not increase them.



NEWLINE CONVENTION


These are recognized only at the very start of the pattern or after option settings with a similar syntax.

  (*CR)           carriage return only

  (*LF)           linefeed only

  (*CRLF)         carriage return followed by linefeed

  (*ANYCRLF)      all three of the above

  (*ANY)          any Unicode newline sequence



WHAT  \R  MATCHES


These are recognized only at the very start of the pattern or after option setting with a similar syntax.


  (*BSR_ANYCRLF)  CR, LF, or CRLF

  (*BSR_UNICODE)  any Unicode newline sequence


Note: Be careful not to confuse \R with \r. The code \r only means a carriage return character (\x0D), while the meaning of \R depends on the settings above.


LOOKAHEAD AND LOOKBEHIND ASSERTIONS


  (?=...)         positive look ahead

  (?!...)         negative look ahead

  (?<=...)        positive look behind

  (?<!...)        negative look behind


Each top-level branch of a look behind must be of a fixed length. 


BACKREFERENCES


  \n              reference by number (can be ambiguous)

  \gn             reference by number

  \g{n}           reference by number

  \g{-n}          relative reference by number

  \k<name>        reference by name (Perl)

  \k'name'        reference by name (Perl)

  \g{name}        reference by name (Perl)

  \k{name}        reference by name (.NET)

  (?P=name)       reference by name (Python)



SUBROUTINE REFERENCES (POSSIBLY RECURSIVE)


  (?R)            recurse whole pattern

  (?n)            call subpattern by absolute number

  (?+n)           call subpattern by relative number

  (?-n)           call subpattern by relative number

  (?&name)        call subpattern by name (Perl)

  (?P>name)       call subpattern by name (Python)

  \g<name>        call subpattern by name (Oniguruma)

  \g'name'        call subpattern by name (Oniguruma)

  \g<n>           call subpattern by absolute number (Oniguruma)

  \g'n'           call subpattern by absolute number (Oniguruma)

  \g<+n>          call subpattern by relative number (PCRE extension)

  \g'+n'          call subpattern by relative number (PCRE extension)

  \g<-n>          call subpattern by relative number (PCRE extension)

  \g'-n'          call subpattern by relative number (PCRE extension)



CONDITIONAL PATTERNS


  (?(condition)yes-pattern)

  (?(condition)yes-pattern|no-pattern)


  (?(n)... absolute reference condition

  (?(+n)... relative reference condition

  (?(-n)... relative reference condition

  (?(<name>)... named reference condition (Perl)

  (?('name')... named reference condition (Perl)

  (?(name)... named reference condition (PCRE)

  (?(R)... overall recursion condition

  (?(Rn)... specific group recursion condition

  (?(R&name)... specific recursion condition

  (?(DEFINE)... define subpattern for reference

  (?(assert)... assertion condition



BACKTRACKING CONTROL


The following act immediately they are reached:


  (*ACCEPT)       force successful match

  (*FAIL)         force backtrack; synonym (*F)

  (*MARK:NAME)    set name to be passed back; synonym (*:NAME)


The following act only when a subsequent match failure causes a backtrack to reach them. They all force a match failure, but they differ in what happens afterwards. Those that advance the start-of-match point do so only if the pattern is not anchored.


  (*COMMIT)       overall failure, no advance of starting point

  (*PRUNE)        advance to next starting character

  (*PRUNE:NAME)   equivalent to (*MARK:NAME)(*PRUNE)

  (*SKIP)         advance to current matching position

  (*SKIP:NAME)    advance to position corresponding to an earlier

                  (*MARK:NAME); if not found, the (*SKIP) is ignored

  (*THEN)         local failure, backtrack to next alternation

  (*THEN:NAME)    equivalent to (*MARK:NAME)(*THEN)



CALLOUTS


  (?C)      callout

  (?Cn)     callout with data n



Performance Considerations

Be aware that the *, + and ? meta-characters can cause the regular expression engine to perform backtracking, which can cause its pattern-matching to run a little slower. As with all regular expressions, performance is dependent on file size and the complexity of the expression.


Compatibility Considerations

The PCRE regular expression syntax is mostly compatible with what SPFLite offered in prior releases. You will generally not need to be concerned about compatibility if you only issue RegEx expressions on the command line with R strings. If you have stored regular expressions, either in a macro or a keyboard definition, you should be aware of the following differences.

Pre-PCRE Regular Expression syntax prior to release 8.3

PCRE Regular Expression syntax as of release 8.3



\c


Signifies a case-sensitive search

PRCE uses this code in an incompatible way. The PCRE code \cx to indicate a "control" character. For example, if you want to use a Ctrl-A as a literal value, you would specify \cA.


SPFLite honors the CASE C/T setting when executing a PCRE regular expression. Where you might previously have used \c to mean "case sensitive", you can now issue a CASE C primary command prior to executing a regular expression.


If you want to embed a code in your RegEx string to control case sensitivity, (?i) means case insensitive (like CASE T) and (?-i) means case sensitive (like CASE C).


\q


Signifies a literal double-quote character "

PCRE does not implement the \q code. It does have a \Q code, but with an unrelated meaning.


The prior RegEx engine allowed this code to make it easier to specify literal double-quote characters in Basic strings. You will generally not need to do this. A double-quote character " has no special meaning in a PCRE RegEx, and can be directly specified as a data value, as long it's not also being used to enclose the R string as a whole.


If you are creating a RegEx string within a SPFLite macro script, you can use the Basic string equate $DQ to get a double-quote character.


You can also insert a double quote by using the RegEx hex value for the ASCII double quote, which is \x22.


\s


Signifies a request to match the shortest-possible string

PRCE uses this code in an incompatible way. The PCRE code \s is used to search for a "white space" character.


PCRE uses the term "greedy" when a string-matching process could match against a variable number of characters, and the longest possible length is taken. When the shortest-possible length is taken, this is called "lazy" or "non-greedy". Thus, the old code of \s meant "lazy" or "non-greedy" in PCRE terminology.


To set "lazy" matching in PCRE, you can use a general option setting code of (?U), and greedy mode would be (?-U). These options affect the entire RegEx expression. PCRE matching is "greedy" by default, so the code (?-U) is not usually required.


PCRE also lets you control the greedy-vs-lazy matching on a finer level, using a ? in certain types of syntax. See the PCRE documentation under QUANTIFIERS for more information.



Examples

Given a file with lines:

"Blah blah blah blah"

"Please send email to FredFarkle@mydomain.com"

"Thank you very much."

The line with the email address could be found with:

FIND R"([a-z0-9._/+-]+)(@[a-z0-9.-]+)"

Given the lines:

"Balance due by 15th of the month"

"Amount owed: $42.75 and is overdue!"

The line with the Balance amount could be found with:

FIND R"\$[0-9.,]+"


Created with the Personal Edition of HelpNDoc: Write eBooks for the Kindle