libpcre2 breaks COMPILED_SOURCE=1 tags

libpcre2 breaks COMPILED_SOURCE=1 tags

[Carlos Llamas]

Newer versions of libpre2 have dropped default support for \K in
lookarounds. Unfortunately, scripts/tags.sh relies on this option to
collect all the _compiled_ sources. This is the error:

  $ make COMPILED_SOURCE=1 tags
    GEN     tags
  grep: \K is not allowed in lookarounds (but see PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK)

It seems there isn't an official maintainer for this script and I can't
quite understand the regex used for the query. Does anyone have a good
alternative for this? The regex pattern was introduced in commit
4f491bb6ea2a ("scripts/tags.sh: collect compiled source precisely").

The previous form of all_compiled_sources() seems to work for me and I'm
guessing it wasn't efficient enough? I can't find much info about the
switch to find/grep either. I believe the initial thread is this:
https://lore.kernel.org/all/20200423103801.GA3730892@kroah.com/

--
Carlos Llamas

Re: libpcre2 breaks COMPILED_SOURCE=1 tags

[Greg Kroah-Hartman]

On Mon, Feb 13, 2023 at 08:07:16PM +0000, Carlos Llamas wrote:
> Newer versions of libpre2 have dropped default support for \K in
> lookarounds. Unfortunately, scripts/tags.sh relies on this option to
> collect all the _compiled_ sources. This is the error:
> 
>   $ make COMPILED_SOURCE=1 tags
>     GEN     tags
>   grep: \K is not allowed in lookarounds (but see PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK)
> 
> It seems there isn't an official maintainer for this script and I can't
> quite understand the regex used for the query. Does anyone have a good
> alternative for this? The regex pattern was introduced in commit
> 4f491bb6ea2a ("scripts/tags.sh: collect compiled source precisely").
> 
> The previous form of all_compiled_sources() seems to work for me and I'm
> guessing it wasn't efficient enough? I can't find much info about the
> switch to find/grep either. I believe the initial thread is this:
> https://lore.kernel.org/all/20200423103801.GA3730892@kroah.com/

Can you send a patch to revert the offending change?  That would be the
simplest for now, right?

thanks,

greg k-h

[PATCH] scripts/tags.sh: fix incompatibility with PCRE2

[Carlos Llamas]

Starting with release 10.38 PCRE2 drops default support for using \K in
lookaround patterns as described in [1]. Unfortunately, scripts/tags.sh
relies on such functionality to collect all_compiled_soures() leading to
the following error:

  $ make COMPILED_SOURCE=1 tags
    GEN     tags
  grep: \K is not allowed in lookarounds (but see PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK)

The usage of \K for this pattern was introduced in commit 4f491bb6ea2a
("scripts/tags.sh: collect compiled source precisely") which speeds up
the generation of tags significantly.

In order to fix this issue without compromising the performance we can
switch over to an equivalent sed expression. The same matching pattern
is preserved here except \K is replaced with a backreference \1.

[1] https://www.pcre.org/current/doc/html/pcre2syntax.html#SEC11

Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Cristian Ciocaltea <cristian.ciocaltea@collabora.com>
Cc: Masahiro Yamada <masahiroy@kernel.org>
Cc: Jialu Xu <xujialu@vimux.org>
Cc: Vipin Sharma <vipinsh@google.com>
Signed-off-by: Carlos Llamas <cmllamas@google.com>
---
 scripts/tags.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/tags.sh b/scripts/tags.sh
index e137cf15aae9..0d045182c08c 100755
--- a/scripts/tags.sh
+++ b/scripts/tags.sh
@@ -91,7 +91,7 @@ all_compiled_sources()
 	{
 		echo include/generated/autoconf.h
 		find $ignore -name "*.cmd" -exec \
-			grep -Poh '(?(?=^source_.* \K).*|(?=^  \K\S).*(?= \\))' {} \+ |
+			sed -n -E 's/^source_.* (.*)/\1/p; s/^  (\S.*) \\/\1/p' {} \+ |
 		awk '!a[$0]++'
 	} | xargs realpath -esq $([ -z "$KBUILD_ABS_SRCTREE" ] && echo --relative-to=.) |
 	sort -u
-- 
2.39.1.581.gbfd45094c4-goog

Re: [PATCH] scripts/tags.sh: fix incompatibility with PCRE2

[Greg Kroah-Hartman]

On Tue, Feb 14, 2023 at 08:41:42PM +0000, Carlos Llamas wrote:
> Starting with release 10.38 PCRE2 drops default support for using \K in
> lookaround patterns as described in [1]. Unfortunately, scripts/tags.sh
> relies on such functionality to collect all_compiled_soures() leading to
> the following error:
> 
>   $ make COMPILED_SOURCE=1 tags
>     GEN     tags
>   grep: \K is not allowed in lookarounds (but see PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK)
> 
> The usage of \K for this pattern was introduced in commit 4f491bb6ea2a
> ("scripts/tags.sh: collect compiled source precisely") which speeds up
> the generation of tags significantly.
> 
> In order to fix this issue without compromising the performance we can
> switch over to an equivalent sed expression. The same matching pattern
> is preserved here except \K is replaced with a backreference \1.
> 
> [1] https://www.pcre.org/current/doc/html/pcre2syntax.html#SEC11
> 
> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Cc: Cristian Ciocaltea <cristian.ciocaltea@collabora.com>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Cc: Jialu Xu <xujialu@vimux.org>
> Cc: Vipin Sharma <vipinsh@google.com>
> Signed-off-by: Carlos Llamas <cmllamas@google.com>
> ---
>  scripts/tags.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Shouldn't this get a "Fixes:" tag with the commit that caused the
problem?  And if this affects older kernels (as it does), it should also
get a cc: stable?

thanks,

greg k-h

Re: [PATCH] scripts/tags.sh: fix incompatibility with PCRE2

[Carlos Llamas]

On Wed, Feb 15, 2023 at 05:48:00PM +0100, Greg Kroah-Hartman wrote:
> 
> Shouldn't this get a "Fixes:" tag with the commit that caused the
> problem?  And if this affects older kernels (as it does), it should also
> get a cc: stable?

I was hesitant to mark 4f491bb6ea2a as the offending commit as it was
PERL5 and PCRE2 that decided to drop this functionality. The commit was
technically correct before this. However, you are right that this
impacts stable too and if it makes the backport process easier I'll add
the "Fixes:" tag and cc: stable as suggested.

--
Carlos Llamas

[PATCH v2] scripts/tags.sh: fix incompatibility with PCRE2

[Carlos Llamas]

Starting with release 10.38 PCRE2 drops default support for using \K in
lookaround patterns as described in [1]. Unfortunately, scripts/tags.sh
relies on such functionality to collect all_compiled_soures() leading to
the following error:

  $ make COMPILED_SOURCE=1 tags
    GEN     tags
  grep: \K is not allowed in lookarounds (but see PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK)

The usage of \K for this pattern was introduced in commit 4f491bb6ea2a
("scripts/tags.sh: collect compiled source precisely") which speeds up
the generation of tags significantly.

In order to fix this issue without compromising the performance we can
switch over to an equivalent sed expression. The same matching pattern
is preserved here except \K is replaced with a backreference \1.

[1] https://www.pcre.org/current/doc/html/pcre2syntax.html#SEC11

Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Cristian Ciocaltea <cristian.ciocaltea@collabora.com>
Cc: Masahiro Yamada <masahiroy@kernel.org>
Cc: Jialu Xu <xujialu@vimux.org>
Cc: Vipin Sharma <vipinsh@google.com>
Cc: stable@vger.kernel.org
Fixes: 4f491bb6ea2a ("scripts/tags.sh: collect compiled source precisely")
Signed-off-by: Carlos Llamas <cmllamas@google.com>
---
v2: add Fixes tag and cc stable

 scripts/tags.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/tags.sh b/scripts/tags.sh
index e137cf15aae9..0d045182c08c 100755
--- a/scripts/tags.sh
+++ b/scripts/tags.sh
@@ -91,7 +91,7 @@ all_compiled_sources()
 	{
 		echo include/generated/autoconf.h
 		find $ignore -name "*.cmd" -exec \
-			grep -Poh '(?(?=^source_.* \K).*|(?=^  \K\S).*(?= \\))' {} \+ |
+			sed -n -E 's/^source_.* (.*)/\1/p; s/^  (\S.*) \\/\1/p' {} \+ |
 		awk '!a[$0]++'
 	} | xargs realpath -esq $([ -z "$KBUILD_ABS_SRCTREE" ] && echo --relative-to=.) |
 	sort -u
-- 
2.39.1.637.g21b0678d19-goog