[PATCH -rc v1] gcov: Disable gcov build with GCC 10

[PATCH -rc v1] gcov: Disable gcov build with GCC 10

[Leon Romanovsky]

From: Leon Romanovsky <leonro@nvidia.com>

GCOV built with GCC 10 doesn't initialize n_function variable.
This produces different kernel panics as was seen by Colin in
Ubuntu [1] and me in FC 32 [2].

As a workaround, let's disable GCOV build for broken GCC 10 version.

[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1891288
[2] https://lore.kernel.org/lkml/20200827133932.3338519-1-leon@kernel.org
Cc: Colin Ian King <colin.king@canonical.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 As was discussed:
 https://lore.kernel.org/lkml/CAHk-=whbijeSdSvx-Xcr0DPMj0BiwhJ+uiNnDSVZcr_h_kg7UA@mail.gmail.com/
---
 kernel/gcov/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/gcov/Kconfig b/kernel/gcov/Kconfig
index 3110c77230c7..bb4b680e8455 100644
--- a/kernel/gcov/Kconfig
+++ b/kernel/gcov/Kconfig
@@ -4,6 +4,7 @@ menu "GCOV-based kernel profiling"
 config GCOV_KERNEL
 	bool "Enable gcov-based kernel profiling"
 	depends on DEBUG_FS
+	depends on !CC_IS_GCC || GCC_VERSION < 100000
 	select CONSTRUCTORS if !UML
 	default n
 	help
--
2.26.2

Re: [PATCH -rc v1] gcov: Disable gcov build with GCC 10

[Peter Oberparleiter]

On 04.09.2020 17:58, Leon Romanovsky wrote:
> GCOV built with GCC 10 doesn't initialize n_function variable.
> This produces different kernel panics as was seen by Colin in
> Ubuntu [1] and me in FC 32 [2].
> 
> As a workaround, let's disable GCOV build for broken GCC 10 version.
> 
> [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1891288
> [2] https://lore.kernel.org/lkml/20200827133932.3338519-1-leon@kernel.org
> Cc: Colin Ian King <colin.king@canonical.com>
> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>

The following patch should fix the problem with gcov and GCC 10.
I successfully tested it with kernel 5.9-rc4 on s390 using GCC 10.1.1
and also with GCC 9.1.1 to see that it didn't break support for previous
GCC versions. In both cases there were no kernel panics and gcov worked
fine.

Could you try this patch to see if it also fixes the problem in your
environment?

---8<---
From: Peter Oberparleiter <oberpar@linux.ibm.com>
Subject: [PATCH] gcov: add support for GCC 10.1

Using gcov to collect coverage data for kernels compiled with GCC 10.1
causes random malfunctions and kernel crashes. This is the result of a
changed GCOV_COUNTERS value in GCC 10.1 that causes a mismatch between
the layout of the gcov_info structure created by GCC profiling code and
the related structure used by the kernel.

Fix this by updating the in-kernel GCOV_COUNTERS value. Also re-enable
config GCOV_KERNEL for use with GCC 10.

Reported-by: Colin Ian King <colin.king@canonical.com>
Reported-by: Leon Romanovsky <leonro@nvidia.com>
Signed-off-by: Peter Oberparleiter <oberpar@linux.ibm.com>
---
 kernel/gcov/Kconfig   | 1 -
 kernel/gcov/gcc_4_7.c | 4 +++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/gcov/Kconfig b/kernel/gcov/Kconfig
index bb4b680e8455..3110c77230c7 100644
--- a/kernel/gcov/Kconfig
+++ b/kernel/gcov/Kconfig
@@ -4,7 +4,6 @@ menu "GCOV-based kernel profiling"
 config GCOV_KERNEL
 	bool "Enable gcov-based kernel profiling"
 	depends on DEBUG_FS
-	depends on !CC_IS_GCC || GCC_VERSION < 100000
 	select CONSTRUCTORS if !UML
 	default n
 	help
diff --git a/kernel/gcov/gcc_4_7.c b/kernel/gcov/gcc_4_7.c
index 908fdf5098c3..53c67c87f141 100644
--- a/kernel/gcov/gcc_4_7.c
+++ b/kernel/gcov/gcc_4_7.c
@@ -19,7 +19,9 @@
 #include <linux/vmalloc.h>
 #include "gcov.h"

-#if (__GNUC__ >= 7)
+#if (__GNUC__ >= 10)
+#define GCOV_COUNTERS			8
+#elif (__GNUC__ >= 7)
 #define GCOV_COUNTERS			9
 #elif (__GNUC__ > 5) || (__GNUC__ == 5 && __GNUC_MINOR__ >= 1)
 #define GCOV_COUNTERS			10
-- 
2.21.0

Re: [PATCH -rc v1] gcov: Disable gcov build with GCC 10

[Leon Romanovsky]

On Thu, Sep 10, 2020 at 02:52:01PM +0200, Peter Oberparleiter wrote:
> On 04.09.2020 17:58, Leon Romanovsky wrote:
> > GCOV built with GCC 10 doesn't initialize n_function variable.
> > This produces different kernel panics as was seen by Colin in
> > Ubuntu [1] and me in FC 32 [2].
> >
> > As a workaround, let's disable GCOV build for broken GCC 10 version.
> >
> > [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1891288
> > [2] https://lore.kernel.org/lkml/20200827133932.3338519-1-leon@kernel.org
> > Cc: Colin Ian King <colin.king@canonical.com>
> > Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
>
> The following patch should fix the problem with gcov and GCC 10.
> I successfully tested it with kernel 5.9-rc4 on s390 using GCC 10.1.1
> and also with GCC 9.1.1 to see that it didn't break support for previous
> GCC versions. In both cases there were no kernel panics and gcov worked
> fine.
>
> Could you try this patch to see if it also fixes the problem in your
> environment?

It worked for me, Thanks
Tested-by: Leon Romanovsky <leonro@nvidia.com>

>
> ---8<---
> From: Peter Oberparleiter <oberpar@linux.ibm.com>
> Subject: [PATCH] gcov: add support for GCC 10.1
>
> Using gcov to collect coverage data for kernels compiled with GCC 10.1
> causes random malfunctions and kernel crashes. This is the result of a
> changed GCOV_COUNTERS value in GCC 10.1 that causes a mismatch between
> the layout of the gcov_info structure created by GCC profiling code and
> the related structure used by the kernel.
>
> Fix this by updating the in-kernel GCOV_COUNTERS value. Also re-enable
> config GCOV_KERNEL for use with GCC 10.
>
> Reported-by: Colin Ian King <colin.king@canonical.com>
> Reported-by: Leon Romanovsky <leonro@nvidia.com>
> Signed-off-by: Peter Oberparleiter <oberpar@linux.ibm.com>
> ---
>  kernel/gcov/Kconfig   | 1 -
>  kernel/gcov/gcc_4_7.c | 4 +++-
>  2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/kernel/gcov/Kconfig b/kernel/gcov/Kconfig
> index bb4b680e8455..3110c77230c7 100644
> --- a/kernel/gcov/Kconfig
> +++ b/kernel/gcov/Kconfig
> @@ -4,7 +4,6 @@ menu "GCOV-based kernel profiling"
>  config GCOV_KERNEL
>  	bool "Enable gcov-based kernel profiling"
>  	depends on DEBUG_FS
> -	depends on !CC_IS_GCC || GCC_VERSION < 100000
>  	select CONSTRUCTORS if !UML
>  	default n
>  	help
> diff --git a/kernel/gcov/gcc_4_7.c b/kernel/gcov/gcc_4_7.c
> index 908fdf5098c3..53c67c87f141 100644
> --- a/kernel/gcov/gcc_4_7.c
> +++ b/kernel/gcov/gcc_4_7.c
> @@ -19,7 +19,9 @@
>  #include <linux/vmalloc.h>
>  #include "gcov.h"
>
> -#if (__GNUC__ >= 7)
> +#if (__GNUC__ >= 10)
> +#define GCOV_COUNTERS			8
> +#elif (__GNUC__ >= 7)
>  #define GCOV_COUNTERS			9
>  #elif (__GNUC__ > 5) || (__GNUC__ == 5 && __GNUC_MINOR__ >= 1)
>  #define GCOV_COUNTERS			10
> --
> 2.21.0
>

Re: [PATCH -rc v1] gcov: Disable gcov build with GCC 10

[Linus Torvalds]

On Thu, Sep 10, 2020 at 5:52 AM Peter Oberparleiter
<oberpar@linux.ibm.com> wrote:
>
> Fix this by updating the in-kernel GCOV_COUNTERS value. Also re-enable
> config GCOV_KERNEL for use with GCC 10.

Lovely.

Is there some way we could see this value automatically, or at least
have a check for it? Right now it's that _very_ magical number that
depends on a gcc version in odd and undocumented ways..

IOW - I'm assuming user space gcov infrastructure finds this number
some way, and wondering if we couldn't do the same?

Or is the gcov tool itself just doing the same kind of thing, and
having magic numbers?

I get the feeling that somebody who knows gcov would go "You are just
doing this all completely incorrectly, you should do XYZ" when they
see that GCOV_COUNTERS thing.

Maybe just a script that finds the right header file in the gcc
installation and extracts it from there, if only to verify the magic
number that we have?

               Linus

RE: [PATCH -rc v1] gcov: Disable gcov build with GCC 10

[David Laight]

From: Linus Torvalds
> Sent: 10 September 2020 20:19
> 
> On Thu, Sep 10, 2020 at 5:52 AM Peter Oberparleiter
> <oberpar@linux.ibm.com> wrote:
> >
> > Fix this by updating the in-kernel GCOV_COUNTERS value. Also re-enable
> > config GCOV_KERNEL for use with GCC 10.
> 
> Lovely.
> 
> Is there some way we could see this value automatically, or at least
> have a check for it? Right now it's that _very_ magical number that
> depends on a gcc version in odd and undocumented ways..
> 
> IOW - I'm assuming user space gcov infrastructure finds this number
> some way, and wondering if we couldn't do the same?
> 
> Or is the gcov tool itself just doing the same kind of thing, and
> having magic numbers?
> 
> I get the feeling that somebody who knows gcov would go "You are just
> doing this all completely incorrectly, you should do XYZ" when they
> see that GCOV_COUNTERS thing.
> 
> Maybe just a script that finds the right header file in the gcc
> installation and extracts it from there, if only to verify the magic
> number that we have?

I was wondering what happens if files compiled with different
versions of gcc get linked together?

Not too far-fetched for someone to release a .a file containing
'gconv' objects.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: [PATCH -rc v1] gcov: Disable gcov build with GCC 10

[Peter Oberparleiter]

On 10.09.2020 23:49, David Laight wrote:
> I was wondering what happens if files compiled with different
> versions of gcc get linked together?

This is not supported by GCC. At runtime libgcov will reject all data
from object files compiled with a different GCC version.

-- 
Peter Oberparleiter
Linux on Z Development - IBM Germany

Re: [PATCH -rc v1] gcov: Disable gcov build with GCC 10

[Peter Oberparleiter]

(Adding GCC's gcov maintainer Martin Liška)

On 10.09.2020 21:18, Linus Torvalds wrote:
> On Thu, Sep 10, 2020 at 5:52 AM Peter Oberparleiter
> <oberpar@linux.ibm.com> wrote:
>>
>> Fix this by updating the in-kernel GCOV_COUNTERS value. Also re-enable
>> config GCOV_KERNEL for use with GCC 10.
> 
> Lovely.
> 
> Is there some way we could see this value automatically, or at least
> have a check for it? Right now it's that _very_ magical number that
> depends on a gcc version in odd and undocumented ways..

I don't think there is, except for maybe parsing GCC debuginfo files
which I would consider an extreme prereq for compiling a kernel with
gcov support. Also GCOV_COUNTERS is only one part of the problem - any
change to struct gcov_info could have a similar effect.

> IOW - I'm assuming user space gcov infrastructure finds this number
> some way, and wondering if we couldn't do the same?
> 
> Or is the gcov tool itself just doing the same kind of thing, and
> having magic numbers?

Short answer: GCC compiles this number into GCC and the associated GCC
library and requires that only matching versions are used. The gcov_info
definition is not available outside the GCC source tree.

Longer answer:

When GCC is called to compile a program with coverage profiling support
it adds inline profiling code and data to the resulting assembler
output. This profiling code updates in-memory counters during program
execution and calls GCC library functions (libgcov) to - among other
things - register the gcov_info data and to write out the resulting data
file when the program terminates. The gcov tool consumes this data file
format, which is different from the in-memory gcov_info data.

The gcov kernel support emulates portions of libgcov - it receives the
gcov_info struct during initial registration, and creates a gcov data
file in debugfs for consumption by the gcov tool.

> I get the feeling that somebody who knows gcov would go "You are just
> doing this all completely incorrectly, you should do XYZ" when they
> see that GCOV_COUNTERS thing.

@Martin: would you care to comment from a GCC point of view?

> Maybe just a script that finds the right header file in the gcc
> installation and extracts it from there, if only to verify the magic
> number that we have?

The next best thing that comes to my mind would be a build-time script
that checks the size of the gcov_info struct generated by GCC and
compare that to the size of the kernel's gcov_info for that GCC version
(by parsing assembler output). This could catch some common classes of
changes to gcov_info, while not restricting usage of gcov kernel support
too much when gcov_info doesn't change (as was the case e.g. between GCC
7 and 9).

The "next worst thing" would be to disable CONFIG_GCOV_KERNEL for
unknown GCC versions until someone tests it and updates the associated
Kconfig file. This catches all changes, but at the cost of possibly
unnecessarily restricting GCC versions plus requiring regular
gcov-kernel updates.


-- 
Peter Oberparleiter
Linux on Z Development - IBM Germany

Re: [PATCH -rc v1] gcov: Disable gcov build with GCC 10

[Colin Ian King]

On 10/09/2020 13:52, Peter Oberparleiter wrote:
> On 04.09.2020 17:58, Leon Romanovsky wrote:
>> GCOV built with GCC 10 doesn't initialize n_function variable.
>> This produces different kernel panics as was seen by Colin in
>> Ubuntu [1] and me in FC 32 [2].
>>
>> As a workaround, let's disable GCOV build for broken GCC 10 version.
>>
>> [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1891288
>> [2] https://lore.kernel.org/lkml/20200827133932.3338519-1-leon@kernel.org
>> Cc: Colin Ian King <colin.king@canonical.com>
>> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> 
> The following patch should fix the problem with gcov and GCC 10.
> I successfully tested it with kernel 5.9-rc4 on s390 using GCC 10.1.1
> and also with GCC 9.1.1 to see that it didn't break support for previous
> GCC versions. In both cases there were no kernel panics and gcov worked
> fine.
> 
> Could you try this patch to see if it also fixes the problem in your
> environment?
> 

This works perfectly, tested with gcc 10.2 on linux 5.9-rc4 (tip) on an
ARM64 Socionext Synquacer E-Series (Cortex A-53) system.

thanks!

Tested-and-Acked-by: Colin Ian King <colin.king@canonical.com>

> ---8<---
> From: Peter Oberparleiter <oberpar@linux.ibm.com>
> Subject: [PATCH] gcov: add support for GCC 10.1
> 
> Using gcov to collect coverage data for kernels compiled with GCC 10.1
> causes random malfunctions and kernel crashes. This is the result of a
> changed GCOV_COUNTERS value in GCC 10.1 that causes a mismatch between
> the layout of the gcov_info structure created by GCC profiling code and
> the related structure used by the kernel.
> 
> Fix this by updating the in-kernel GCOV_COUNTERS value. Also re-enable
> config GCOV_KERNEL for use with GCC 10.
> 
> Reported-by: Colin Ian King <colin.king@canonical.com>
> Reported-by: Leon Romanovsky <leonro@nvidia.com>
> Signed-off-by: Peter Oberparleiter <oberpar@linux.ibm.com>
> ---
>  kernel/gcov/Kconfig   | 1 -
>  kernel/gcov/gcc_4_7.c | 4 +++-
>  2 files changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/gcov/Kconfig b/kernel/gcov/Kconfig
> index bb4b680e8455..3110c77230c7 100644
> --- a/kernel/gcov/Kconfig
> +++ b/kernel/gcov/Kconfig
> @@ -4,7 +4,6 @@ menu "GCOV-based kernel profiling"
>  config GCOV_KERNEL
>  	bool "Enable gcov-based kernel profiling"
>  	depends on DEBUG_FS
> -	depends on !CC_IS_GCC || GCC_VERSION < 100000
>  	select CONSTRUCTORS if !UML
>  	default n
>  	help
> diff --git a/kernel/gcov/gcc_4_7.c b/kernel/gcov/gcc_4_7.c
> index 908fdf5098c3..53c67c87f141 100644
> --- a/kernel/gcov/gcc_4_7.c
> +++ b/kernel/gcov/gcc_4_7.c
> @@ -19,7 +19,9 @@
>  #include <linux/vmalloc.h>
>  #include "gcov.h"
> 
> -#if (__GNUC__ >= 7)
> +#if (__GNUC__ >= 10)
> +#define GCOV_COUNTERS			8
> +#elif (__GNUC__ >= 7)
>  #define GCOV_COUNTERS			9
>  #elif (__GNUC__ > 5) || (__GNUC__ == 5 && __GNUC_MINOR__ >= 1)
>  #define GCOV_COUNTERS			10
>