Re: [v2.35.0 regression] some PCRE hangs under UTF-8 locale (was: [PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns)

["SZEDER Gábor" <szeder.dev@gmail.com>]

On Sun, Jan 30, 2022 at 08:55:02AM +0100, René Scharfe wrote:
> Am 29.01.22 um 18:25 schrieb SZEDER Gábor:
> > On Sat, Dec 18, 2021 at 08:50:02PM +0100, René Scharfe wrote:
> >> compile_pcre2_pattern() currently uses the option PCRE2_UTF only for
> >> patterns with non-ASCII characters.  Patterns with ASCII wildcards can
> >> match non-ASCII strings, though.  Without that option PCRE2 mishandles
> >> UTF-8 input, though -- it matches parts of multi-byte characters.  Fix
> >> that by using PCRE2_UTF even for ASCII-only patterns.
> >>
> >> This is a remake of the reverted ae39ba431a (grep/pcre2: fix an edge
> >> case concerning ascii patterns and UTF-8 data, 2021-10-15).  The change
> >> to the condition and the test are simplified and more targeted.
> >>
> >> Original-patch-by: Hamza Mahfooz <someguy@effective-light.com>
> >> Signed-off-by: René Scharfe <l.s.r@web.de>
> >> ---
> >>  grep.c                          | 2 +-
> >>  t/t7812-grep-icase-non-ascii.sh | 6 ++++++
> >>  2 files changed, 7 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/grep.c b/grep.c
> >> index fe847a0111..5badb6d851 100644
> >> --- a/grep.c
> >> +++ b/grep.c
> >> @@ -382,7 +382,7 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt
> >>  		}
> >>  		options |= PCRE2_CASELESS;
> >>  	}
> >> -	if (!opt->ignore_locale && is_utf8_locale() && has_non_ascii(p->pattern) &&
> >> +	if (!opt->ignore_locale && is_utf8_locale() &&
> >>  	    !(!opt->ignore_case && (p->fixed || p->is_fixed)))
> >>  		options |= (PCRE2_UTF | PCRE2_MATCH_INVALID_UTF);
> >>
> >
> > I tried to use 'git grep -P' for the first time ever, and it hung
> > right away, spinning all CPUs at 100%.  I could narrow it down, both
> > the complexity of the pattern and the size of the input, see the test
> > below, and it bisects to this patch.
> >
> >
> >   ---   >8   ---
> >
> > #!/bin/sh
> >
> > test_description='test'
> >
> > . ./test-lib.sh
> >
> > test_expect_success PCRE 'test' '
> > 	# LC_ALL=C works
> > 	LC_ALL=en_US.UTF-8 &&
> > 	cat >ascii <<-\EOF &&
> > 	foo
> > 	 bar
> > 	 baz
> > 	EOF
> > 	cat >utf8 <<-\EOF &&
> > 	foo
> > 	 bar
> > 	 báz
> > 	EOF
> > 	git add ascii utf8 &&
> >
> > 	# These all work as expected:
> > 	git grep --threads=1 -P " " ascii &&
> > 	git grep --threads=1 -P "^ " ascii &&
> > 	git grep --threads=1 -P "\s" ascii &&
> > 	git grep --threads=1 -P "^\s" ascii &&
> > 	git grep --threads=1 -P " " utf8 &&
> > 	git grep --threads=1 -P "^ " utf8 &&
> > 	git grep --threads=1 -P "\s" utf8 &&
> >
> > 	# This hangs (but it does work with basic and extended regexp):
> > 	git grep --threads=1 -P "^\s" utf8
> > '
> >
> > test_done
> 
> I get the following result and no hang with PCRE2 10.39:
> 
>    utf8: bar
>    utf8: báz
> 
> e0c6029 (Fix inifinite loop when a single byte newline is searched in
> JIT., 2020-05-29) [1] sounds like it might have fixed it.  It's part of
> version 10.36.

I saw this hang on two Ubuntu 20.04 based boxes, which predate that
fix you mention only by a month or two, and apparently the almost two
years since then was not enough for this fix to trickle down into
updated 20.04 pcre packages, because:

> Do you still get the error when you disable JIT, i.e. when you use the
> pattern "(*NO_JIT)^\s" instead?

No, with this pattern it works as expected.

So is there a more convenient way to disable PCRE JIT in Git?  FWIW,
(non-git) 'grep -P' works with the same patterns.

> René
> 
> 
> [1] https://github.com/PhilipHazel/pcre2/commit/e0c6029a62db9c2161941ecdf459205382d4d379