Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
PCRE2PARTIAL(3)		   Library Functions Manual	       PCRE2PARTIAL(3)

NAME
       PCRE2 - Perl-compatible regular expressions (revised API)

PARTIAL	MATCHING IN PCRE2

       In  normal use of PCRE2,	if there is a match up to the end of a subject
       string, but more	characters are needed to  match	 the  entire  pattern,
       PCRE2_ERROR_NOMATCH  is	returned,  just	 like any other	failing	match.
       There are circumstances where it	might be helpful to  distinguish  this
       "partial	match" case.

       One  example  is	 an application	where the subject string is very long,
       and not all available at	once. The requirement here is to be able to do
       the matching segment by segment,	but special action is  needed  when  a
       matched substring spans the boundary between two	segments.

       Another	example	is checking a user input string	as it is typed,	to en-
       sure that it conforms to	a required format. Invalid characters  can  be
       immediately diagnosed and rejected, giving instant feedback.

       Partial	matching  is a PCRE2-specific feature; it is not Perl-compati-
       ble. It is requested  by	 setting  one  of  the	PCRE2_PARTIAL_HARD  or
       PCRE2_PARTIAL_SOFT  options  when calling a matching function. The dif-
       ference between the two options is whether or not a  partial  match  is
       preferred  to  an alternative complete match, though the	details	differ
       between the two types of	matching function. If both  options  are  set,
       PCRE2_PARTIAL_HARD takes	precedence.

       If  you	want to	use partial matching with just-in-time optimized code,
       as well as setting a partial match option for  the  matching  function,
       you  must  also	call pcre2_jit_compile() with one or both of these op-
       tions:

	 PCRE2_JIT_PARTIAL_HARD
	 PCRE2_JIT_PARTIAL_SOFT

       PCRE2_JIT_COMPLETE should also be set if	you are	going to run  non-par-
       tial  matches  on  the same pattern. Separate code is compiled for each
       mode. If	the appropriate	JIT mode has not been  compiled,  interpretive
       matching	code is	used.

       Setting	a partial matching option disables two of PCRE2's standard op-
       timization hints. PCRE2 remembers the last literal code unit in a  pat-
       tern,  and  abandons  matching  immediately if it is not	present	in the
       subject string.	This optimization cannot be used for a subject	string
       that  might match only partially. PCRE2 also remembers a	minimum	length
       of a matching string, and does not bother to run	the matching  function
       on  shorter  strings.  This  optimization  is also disabled for partial
       matching.

REQUIREMENTS FOR A PARTIAL MATCH

       A possible partial match	occurs during matching when  the  end  of  the
       subject	string is reached successfully,	but either more	characters are
       needed to complete the match, or	the addition of	more characters	 might
       change what is matched.

       Example	1: if the pattern is /abc/ and the subject is "ab", more char-
       acters are definitely needed to complete	a match.  In  this  case  both
       hard and	soft matching options yield a partial match.

       Example	2: if the pattern is /ab+/ and the subject is "ab", a complete
       match can be found, but the addition of more  characters	 might	change
       what  is	 matched. In this case,	only PCRE2_PARTIAL_HARD	returns	a par-
       tial match; PCRE2_PARTIAL_SOFT returns the complete match.

       On reaching the end of the subject, when	PCRE2_PARTIAL_HARD is set,  if
       the next	pattern	item is	\z, \Z,	\b, \B,	or $ there is always a partial
       match.	Otherwise, for both options, the next pattern item must	be one
       that inspects a character, and at least one of the  following  must  be
       true:

       (1)  At	least  one  character has already been inspected. An inspected
       character need not form part of the final  matched  string;  lookbehind
       assertions  and the \K escape sequence provide ways of inspecting char-
       acters before the start of a matched string.

       (2) The pattern contains	one or more lookbehind assertions. This	condi-
       tion exists in case there is a lookbehind that inspects characters  be-
       fore the	start of the match.

       (3)  There  is a	special	case when the whole pattern can	match an empty
       string.	When the starting point	is at the  end	of  the	 subject,  the
       empty  string  match is a possibility, and if PCRE2_PARTIAL_SOFT	is set
       and neither of the above	conditions is true, it is  returned.  However,
       because	adding	more  characters  might	 result	 in a non-empty	match,
       PCRE2_PARTIAL_HARD returns a partial match, which in  this  case	 means
       "there  is going	to be a	match at this point, but until some more char-
       acters are added, we do not know	if it will be an empty string or some-
       thing longer".

PARTIAL	MATCHING USING pcre2_match()

       When  a	partial	 matching  option  is  set,  the  result  of   calling
       pcre2_match() can be one	of the following:

       A successful match
	 A complete match has been found, starting and ending within this sub-
	 ject.

       PCRE2_ERROR_NOMATCH
	 No match can start anywhere in	this subject.

       PCRE2_ERROR_PARTIAL
	 Adding	 more  characters may result in	a complete match that uses one
	 or more characters from the end of this subject.

       When a partial match is returned, the first two elements	in the ovector
       point to	the portion of the subject that	was matched, but the values in
       the rest	of the ovector are undefined. The appearance of	\K in the pat-
       tern has	no effect for a	partial	match. Consider	this pattern:

	 /abc\K123/

       If it is	matched	against	"456abc123xyz" the result is a complete	match,
       and the ovector defines the matched string as "123", because \K	resets
       the  "start  of	match" point. However, if a partial match is requested
       and the subject string is "456abc12", a partial match is	found for  the
       string  "abc12",	 because  all these characters are needed for a	subse-
       quent re-match with additional characters.

       If there	is more	than one partial match,	the first one that  was	 found
       provides	the data that is returned. Consider this pattern:

	 /123\w+X|dogY/

       If  this	is matched against the subject string "abc123dog", both	alter-
       natives fail to match, but the end of the  subject  is  reached	during
       matching,  so PCRE2_ERROR_PARTIAL is returned. The offsets are set to 3
       and 9, identifying "123dog" as the first	partial	match. (In this	 exam-
       ple,  there are two partial matches, because "dog" on its own partially
       matches the second alternative.)

   How a partial match is processed by pcre2_match()

       What happens when a partial match is identified depends on which	of the
       two partial matching options is set.

       If PCRE2_PARTIAL_HARD is	set, PCRE2_ERROR_PARTIAL is returned  as  soon
       as  a partial match is found, without continuing	to search for possible
       complete	matches. This option is	"hard" because it prefers  an  earlier
       partial match over a later complete match. For this reason, the assump-
       tion  is	 made  that  the end of	the supplied subject string is not the
       true end	of the available data, which is	why \z,	\Z, \b,	\B, and	$  al-
       ways give a partial match.

       If  PCRE2_PARTIAL_SOFT  is  set,	 the  partial match is remembered, but
       matching	continues as normal, and other alternatives in the pattern are
       tried. If no complete match can be found,  PCRE2_ERROR_PARTIAL  is  re-
       turned instead of PCRE2_ERROR_NOMATCH. This option is "soft" because it
       prefers a complete match	over a partial match. All the various matching
       items  in a pattern behave as if	the subject string is potentially com-
       plete; \z, \Z, and $ match at the end of	the subject,  as  normal,  and
       for \b and \B the end of	the subject is treated as a non-alphanumeric.

       The  difference	between	the two	partial	matching options can be	illus-
       trated by a pattern such	as:

	 /dog(sbody)?/

       This matches either "dog" or "dogsbody",	greedily (that is, it  prefers
       the  longer  string  if	possible). If it is matched against the	string
       "dog" with PCRE2_PARTIAL_SOFT, it yields	a complete  match  for	"dog".
       However,	 if  PCRE2_PARTIAL_HARD	is set,	the result is PCRE2_ERROR_PAR-
       TIAL. On	the other hand,	if the pattern is made ungreedy	the result  is
       different:

	 /dog(sbody)??/

       In  this	 case  the  result  is always a	complete match because that is
       found first, and	matching never	continues  after  finding  a  complete
       match. It might be easier to follow this	explanation by thinking	of the
       two patterns like this:

	 /dog(sbody)?/	  is the same as  /dogsbody|dog/
	 /dog(sbody)??/	  is the same as  /dog|dogsbody/

       The  second pattern will	never match "dogsbody",	because	it will	always
       find the	shorter	match first.

   Example of partial matching using pcre2test

       The pcre2test data modifiers partial_hard (or ph) and partial_soft  (or
       ps)  set	 PCRE2_PARTIAL_HARD and	PCRE2_PARTIAL_SOFT, respectively, when
       calling pcre2_match(). Here is a	run of pcre2test using a pattern  that
       matches the whole subject in the	form of	a date:

	   re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
	 data> 25dec3\=ph
	 Partial match:	23dec3
	 data> 3ju\=ph
	 Partial match:	3ju
	 data> 3juj\=ph
	 No match

       This  example  gives  the  same	results	for both hard and soft partial
       matching	options. Here is an example where there	is a difference:

	   re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
	 data> 25jun04\=ps
	  0: 25jun04
	  1: jun
	 data> 25jun04\=ph
	 Partial match:	25jun04

       With  PCRE2_PARTIAL_SOFT,  the  subject	is  matched  completely.   For
       PCRE2_PARTIAL_HARD, however, the	subject	is assumed not to be complete,
       so there	is only	a partial match.

MULTI-SEGMENT MATCHING WITH pcre2_match()

       PCRE  was  not originally designed with multi-segment matching in mind.
       However,	over time, features (including	partial	 matching)  that  make
       multi-segment matching possible have been added.	A very long string can
       be  searched  segment  by  segment by calling pcre2_match() repeatedly,
       with the	aim of achieving the same results that would happen if the en-
       tire string was available for searching all  the	 time.	Normally,  the
       strings	that  are  being  sought are much shorter than each individual
       segment,	and are	in the middle of very long strings, so the pattern  is
       normally	not anchored.

       Special	logic  must  be	implemented to handle a	matched	substring that
       spans a segment boundary. PCRE2_PARTIAL_HARD should be used, because it
       returns a partial match at the end of a segment whenever	there  is  the
       possibility  of	changing  the  match  by  adding  more characters. The
       PCRE2_NOTBOL option should also be set for all but the first segment.

       When a partial match occurs, the	next segment must be added to the cur-
       rent subject and	the match re-run, using	the  startoffset  argument  of
       pcre2_match()  to  begin	 at the	point where the	partial	match started.
       For example:

	   re> /\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d/
	 data> ...the date is 23ja\=ph
	 Partial match:	23ja
	 data> ...the date is 23jan19 and on that day...\=offset=15
	  0: 23jan19
	  1: jan

       Note the	use of the offset modifier to start the	new  match  where  the
       partial match was found.	In this	example, the next segment was added to
       the  one	 in  which  the	 partial  match	 was  found.  This is the most
       straightforward approach, typically using a memory buffer that is twice
       the size	of each	segment. After a partial match,	the first half of  the
       buffer  is  discarded,  the  second  half  is moved to the start	of the
       buffer, and a new segment is added before repeating the match as	in the
       example above. After a no match,	the entire buffer can be discarded.

       If there	are memory constraints,	you may	want to	discard	text that pre-
       cedes a partial match before adding the	next  segment.	Unfortunately,
       this  is	 not  at  present straightforward. In cases such as the	above,
       where the pattern does not contain any lookbehinds, it is sufficient to
       retain only the partially matched substring. However,  if  the  pattern
       contains	 a  lookbehind assertion, characters that precede the start of
       the partial match may have been inspected during	the matching  process.
       When  pcre2test displays	a partial match, it indicates these characters
       with '<'	if the allusedtext modifier is set:

	   re> "(?<=123)abc"
	 data> xx123ab\=ph,allusedtext
	 Partial match:	123ab
			<<<

       However,	the allusedtext	modifier is not	available  for	JIT  matching,
       because	JIT  matching  does  not  record the first (or last) consulted
       characters.  For	this reason, this information is not available via the
       API. It is therefore not	possible in general to obtain the exact	number
       of characters that must be retained in order to get the right match re-
       sult. If	you cannot retain the  entire  segment,	 you  must  find  some
       heuristic way of	choosing.

       If  you know the	approximate length of the matching substrings, you can
       use that	to decide how much text	to retain. The only lookbehind	infor-
       mation  that  is	 currently  available via the API is the length	of the
       longest individual lookbehind in	a pattern, but this can	be  misleading
       if  there  are  nested  lookbehinds.  The  value	 returned  by  calling
       pcre2_pattern_info() with the PCRE2_INFO_MAXLOOKBEHIND  option  is  the
       maximum number of characters (not code units) that any individual look-
       behind	moves	back   when   it  is  processed.  A  pattern  such  as
       "(?<=(?<!b)a)" has a maximum lookbehind value of	one, but inspects  two
       characters before its starting point.

       In  a  non-UTF or a 32-bit case,	moving back is just a subtraction, but
       in UTF-8	or UTF-16 you have  to	count  characters  while  moving  back
       through the code	units.

PARTIAL	MATCHING USING pcre2_dfa_match()

       The DFA function	moves along the	subject	string character by character,
       without	backtracking,  searching  for  all possible matches simultane-
       ously. If the end of the	subject	is reached before the end of the  pat-
       tern, there is the possibility of a partial match.

       When PCRE2_PARTIAL_SOFT is set, PCRE2_ERROR_PARTIAL is returned only if
       there  have  been  no complete matches. Otherwise, the complete matches
       are returned.  If PCRE2_PARTIAL_HARD is	set,  a	 partial  match	 takes
       precedence  over	 any  complete matches.	The portion of the string that
       was matched when	the longest partial match was  found  is  set  as  the
       first matching string.

       Because	the DFA	function always	searches for all possible matches, and
       there is	no difference between greedy and ungreedy repetition, its  be-
       haviour	is different from the pcre2_match(). Consider the string "dog"
       matched against this ungreedy pattern:

	 /dog(sbody)??/

       Whereas the standard function stops as soon as it  finds	 the  complete
       match  for  "dog",  the	DFA  function also finds the partial match for
       "dogsbody", and so returns that when PCRE2_PARTIAL_HARD is set.

MULTI-SEGMENT MATCHING WITH pcre2_dfa_match()

       When a partial match has	been found using the DFA matching function, it
       is possible to continue the match by providing additional subject  data
       and  calling  the function again	with the same compiled regular expres-
       sion, this time setting the PCRE2_DFA_RESTART option. You must pass the
       same working space as before, because this is where details of the pre-
       vious partial match are stored. You can set the	PCRE2_PARTIAL_SOFT  or
       PCRE2_PARTIAL_HARD  options  with PCRE2_DFA_RESTART to continue partial
       matching	over multiple segments.	Here is	an example using pcre2test:

	   re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
	 data> 23ja\=dfa,ps
	 Partial match:	23ja
	 data> n05\=dfa,dfa_restart
	  0: n05

       The first call has "23ja" as the	subject, and requests  partial	match-
       ing;  the  second  call	has  "n05"  as	the  subject for the continued
       (restarted) match.  Notice that when the	match is  complete,  only  the
       last  part  is  shown;  PCRE2 does not retain the previously partially-
       matched string. It is up	to the calling program to do that if it	 needs
       to.  This  means	 that, for an unanchored pattern, if a continued match
       fails, it is not	possible to try	again at a  new	 starting  point.  All
       this facility is	capable	of doing is continuing with the	previous match
       attempt.	For example, consider this pattern:

	 1234|3789

       If  the	first  part of the subject is "ABC123",	a partial match	of the
       first alternative is found at offset 3. There is	no partial  match  for
       the second alternative, because such a match does not start at the same
       point  in  the  subject	string.	Attempting to continue with the	string
       "7890" does not yield a match  because  only  those  alternatives  that
       match  at one point in the subject are remembered. Depending on the ap-
       plication, this may or may not be what you want.

       If you do want to allow for starting again at the next  character,  one
       way  of	doing it is to retain some or all of the segment and try a new
       complete	match, as described for	pcre2_match() above. Another possibil-
       ity is to work with two buffers.	If a partial match at offset n in  the
       first  buffer  is followed by "no match"	when PCRE2_DFA_RESTART is used
       on the second buffer, you can then try a	new match starting  at	offset
       n+1 in the first	buffer.

AUTHOR

       Philip Hazel
       Retired from University Computing Service
       Cambridge, England.

REVISION

       Last updated: 27	November 2024
       Copyright (c) 1997-2019 University of Cambridge.

PCRE2 10.45		       27 November 2024		       PCRE2PARTIAL(3)

Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=pcre2partial&sektion=3&manpath=FreeBSD+Ports+14.3.quarterly>

home | help