Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Omar Sandoval]

Hi, Alexander and Sven,

I just got around to testing drgn on s390x on 6.10-rc4, and it appears
to be broken. I bisected it to commit 56b1069c40c7 ("s390/boot: Rework
deployment of the kernel image") and narrowed it down to an issue with
the KERNELOFFSET value reported in vmcoreinfo.

On my test kernel, the ELF symbol for init_task is 0xc96f00:

  $ eu-readelf -s build/vmtest/s390x/kernel-6.10.0-rc4-vmtest30.1default/build/vmlinux | grep ' init_task$'
  72273: 0000000000c96f00   4352 OBJECT  GLOBAL DEFAULT       18 init_task

And the address in the loaded kernel is 0x3ffffeaaf00:

  # grep ' init_task$' /proc/kallsyms
  000003ffffeaaf00 D init_task

0x3ffffeaaf00 - 0xc96f00 is 0x3ffff214000

However, this doesn't match the value of KERNELOFFSET in vmcoreinfo:

  # eu-readelf -n /proc/kcore | grep KERNELOFFSET
    KERNELOFFSET=3ffff314000

It's off by 0x100000. This causes drgn to compute the wrong addresses
for all global variables.

For context, I'm testing using QEMU emulation on an x86-64 host. Note
that it logs "KASLR disabled: CPU has no PRNG" early during boot. My
exact setup is:

  $ git clone https://github.com/osandov/drgn.git
  $ cd drgn
  $ python3 -m vmtest.rootfsbuild -a s390x --build-drgn
  $ python3 -m vmtest.vm -k 's390x:6.10.*' bash -i
  # python3 -m drgn
  >>> prog['init_task'].comm
  (char [16])""

That should be printing "swapper/0".

Any ideas what's going on here?

Thanks!
Omar

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Alexander Gordeev]

On Thu, Jun 20, 2024 at 04:34:15PM -0700, Omar Sandoval wrote:

Hi Omar!

Thank you for the report!

...
> Any ideas what's going on here?

Yes. I will look into it this week.

> Thanks!
> Omar

Thanks!

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Alexander Gordeev]

On Thu, Jun 20, 2024 at 04:34:15PM -0700, Omar Sandoval wrote:

Hi Omar,

> Hi, Alexander and Sven,
> 
> I just got around to testing drgn on s390x on 6.10-rc4, and it appears
> to be broken. I bisected it to commit 56b1069c40c7 ("s390/boot: Rework
> deployment of the kernel image") and narrowed it down to an issue with
> the KERNELOFFSET value reported in vmcoreinfo.
> 
> On my test kernel, the ELF symbol for init_task is 0xc96f00:
> 
>   $ eu-readelf -s build/vmtest/s390x/kernel-6.10.0-rc4-vmtest30.1default/build/vmlinux | grep ' init_task$'
>   72273: 0000000000c96f00   4352 OBJECT  GLOBAL DEFAULT       18 init_task
> 
> And the address in the loaded kernel is 0x3ffffeaaf00:
> 
>   # grep ' init_task$' /proc/kallsyms
>   000003ffffeaaf00 D init_task
> 
> 0x3ffffeaaf00 - 0xc96f00 is 0x3ffff214000
> 
> However, this doesn't match the value of KERNELOFFSET in vmcoreinfo:
> 
>   # eu-readelf -n /proc/kcore | grep KERNELOFFSET
>     KERNELOFFSET=3ffff314000
> 
> It's off by 0x100000. This causes drgn to compute the wrong addresses
> for all global variables.
> 
> For context, I'm testing using QEMU emulation on an x86-64 host. Note
> that it logs "KASLR disabled: CPU has no PRNG" early during boot. My
> exact setup is:
> 
>   $ git clone https://github.com/osandov/drgn.git
>   $ cd drgn
>   $ python3 -m vmtest.rootfsbuild -a s390x --build-drgn
>   $ python3 -m vmtest.vm -k 's390x:6.10.*' bash -i
>   # python3 -m drgn
>   >>> prog['init_task'].comm
>   (char [16])""
> 
> That should be printing "swapper/0".
> 
> Any ideas what's going on here?

On s390 no kernel symbol exists below 0x100000 offset within the
vmlinux image and thus this part is never mapped into the kernel
memory. That way KERNELOFFSET turns out to be off on value of
0x100000 - and that is what you observe.

That breaks the way drgn finds a kernel symbol, but does not
exactly contradicts to the existing KERNELOFFSET description
(Documentation/admin-guide/kdump/vmcoreinfo.rst):

===
KERNELOFFSET
------------

The kernel randomization offset. Used to compute the page offset. If
KASLR is disabled, this value is zero.
===

I would say to some degree there is also inconsisten with regard
to /proc/ files existence:
/proc/kcore    is enabled by CONFIG_PROC_KCORE option, while
/proc/kallsyms is enabled by CONFIG_KALLSYMS option.
I assume drgn expects both files exist and does not work otherwise.

Nevertheless, it is still possible to refer to only one file for
symbol resolution and use an always-present symbol. E.g _stext
could be leveraged like this:

# grep -w init_task /proc/kallsyms
000003ffe13e9400 D init_task
# grep -w _stext /proc/kallsyms
000003ffe0000000 T _stext

0x3ffe13e9400 - 0x3ffe0000000 == 0x13e9400

# eu-readelf -s vmlinux | grep -w _stext
178112: 0000000000100000      0 NOTYPE  GLOBAL DEFAULT        1 _stext

0x13e9400 + 0x100000 == 0x14e9400

# eu-readelf -s vmlinux | grep -w init_task
  498: 0000000000000000      0 FILE    LOCAL  DEFAULT      ABS init_task.c
182344: 00000000014e9400   8960 OBJECT  GLOBAL DEFAULT       28 init_task

I guess, the above holds true for all architectures.
If so, I would suggest consider using that approach.

Having said that, we will try to turn KERNELOFFSET from a synthetic
value "Used to compute the page offset" to what drgn expects it to be.

> Thanks!
> Omar

Thanks!

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Petr Tesařík]

On Thu, 27 Jun 2024 15:51:09 +0200
Alexander Gordeev <agordeev@linux.ibm.com> wrote:

> On Thu, Jun 20, 2024 at 04:34:15PM -0700, Omar Sandoval wrote:
> 
> Hi Omar,
> 
> > Hi, Alexander and Sven,
> > 
> > I just got around to testing drgn on s390x on 6.10-rc4, and it appears
> > to be broken. I bisected it to commit 56b1069c40c7 ("s390/boot: Rework
> > deployment of the kernel image") and narrowed it down to an issue with
> > the KERNELOFFSET value reported in vmcoreinfo.
> > 
> > On my test kernel, the ELF symbol for init_task is 0xc96f00:
> > 
> >   $ eu-readelf -s build/vmtest/s390x/kernel-6.10.0-rc4-vmtest30.1default/build/vmlinux | grep ' init_task$'
> >   72273: 0000000000c96f00   4352 OBJECT  GLOBAL DEFAULT       18 init_task
> > 
> > And the address in the loaded kernel is 0x3ffffeaaf00:
> > 
> >   # grep ' init_task$' /proc/kallsyms
> >   000003ffffeaaf00 D init_task
> > 
> > 0x3ffffeaaf00 - 0xc96f00 is 0x3ffff214000
> > 
> > However, this doesn't match the value of KERNELOFFSET in vmcoreinfo:
> > 
> >   # eu-readelf -n /proc/kcore | grep KERNELOFFSET
> >     KERNELOFFSET=3ffff314000
> > 
> > It's off by 0x100000. This causes drgn to compute the wrong addresses
> > for all global variables.
> > 
> > For context, I'm testing using QEMU emulation on an x86-64 host. Note
> > that it logs "KASLR disabled: CPU has no PRNG" early during boot. My
> > exact setup is:
> > 
> >   $ git clone https://github.com/osandov/drgn.git
> >   $ cd drgn
> >   $ python3 -m vmtest.rootfsbuild -a s390x --build-drgn
> >   $ python3 -m vmtest.vm -k 's390x:6.10.*' bash -i
> >   # python3 -m drgn  
> >   >>> prog['init_task'].comm  
> >   (char [16])""
> > 
> > That should be printing "swapper/0".
> > 
> > Any ideas what's going on here?  
> 
> On s390 no kernel symbol exists below 0x100000 offset within the
> vmlinux image and thus this part is never mapped into the kernel
> memory. That way KERNELOFFSET turns out to be off on value of
> 0x100000 - and that is what you observe.
> 
> That breaks the way drgn finds a kernel symbol, but does not
> exactly contradicts to the existing KERNELOFFSET description
> (Documentation/admin-guide/kdump/vmcoreinfo.rst):
> 
> ===
> KERNELOFFSET
> ------------
> 
> The kernel randomization offset. Used to compute the page offset. If
> KASLR is disabled, this value is zero.
> ===
> 
> I would say to some degree there is also inconsisten with regard
> to /proc/ files existence:
> /proc/kcore    is enabled by CONFIG_PROC_KCORE option, while
> /proc/kallsyms is enabled by CONFIG_KALLSYMS option.
> I assume drgn expects both files exist and does not work otherwise.
> 
> Nevertheless, it is still possible to refer to only one file for
> symbol resolution and use an always-present symbol. E.g _stext
> could be leveraged like this:
> 
> # grep -w init_task /proc/kallsyms
> 000003ffe13e9400 D init_task
> # grep -w _stext /proc/kallsyms
> 000003ffe0000000 T _stext
> 
> 0x3ffe13e9400 - 0x3ffe0000000 == 0x13e9400
> 
> # eu-readelf -s vmlinux | grep -w _stext
> 178112: 0000000000100000      0 NOTYPE  GLOBAL DEFAULT        1 _stext
> 
> 0x13e9400 + 0x100000 == 0x14e9400
> 
> # eu-readelf -s vmlinux | grep -w init_task
>   498: 0000000000000000      0 FILE    LOCAL  DEFAULT      ABS init_task.c
> 182344: 00000000014e9400   8960 OBJECT  GLOBAL DEFAULT       28 init_task
> 
> I guess, the above holds true for all architectures.
> If so, I would suggest consider using that approach.
> 
> Having said that, we will try to turn KERNELOFFSET from a synthetic
> value "Used to compute the page offset" to what drgn expects it to be.

Thinking about it now, I'm not sure it makes life easier. Because then
we'll have some old kernels with the current (unexpected) definition of
KERNELOFFSET and some new kernels with a more standard definition of
it, but if I read vmcoreinfo, how do I know if the value has the old or
the new meaning?

Petr T

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Omar Sandoval]

On Tue, Jul 09, 2024 at 09:21:41PM +0200, Petr Tesařík wrote:
> On Thu, 27 Jun 2024 15:51:09 +0200
> Alexander Gordeev <agordeev@linux.ibm.com> wrote:
> 
> > On Thu, Jun 20, 2024 at 04:34:15PM -0700, Omar Sandoval wrote:
> > 
> > Hi Omar,
> > 
> > > Hi, Alexander and Sven,
> > > 
> > > I just got around to testing drgn on s390x on 6.10-rc4, and it appears
> > > to be broken. I bisected it to commit 56b1069c40c7 ("s390/boot: Rework
> > > deployment of the kernel image") and narrowed it down to an issue with
> > > the KERNELOFFSET value reported in vmcoreinfo.
> > > 
> > > On my test kernel, the ELF symbol for init_task is 0xc96f00:
> > > 
> > >   $ eu-readelf -s build/vmtest/s390x/kernel-6.10.0-rc4-vmtest30.1default/build/vmlinux | grep ' init_task$'
> > >   72273: 0000000000c96f00   4352 OBJECT  GLOBAL DEFAULT       18 init_task
> > > 
> > > And the address in the loaded kernel is 0x3ffffeaaf00:
> > > 
> > >   # grep ' init_task$' /proc/kallsyms
> > >   000003ffffeaaf00 D init_task
> > > 
> > > 0x3ffffeaaf00 - 0xc96f00 is 0x3ffff214000
> > > 
> > > However, this doesn't match the value of KERNELOFFSET in vmcoreinfo:
> > > 
> > >   # eu-readelf -n /proc/kcore | grep KERNELOFFSET
> > >     KERNELOFFSET=3ffff314000
> > > 
> > > It's off by 0x100000. This causes drgn to compute the wrong addresses
> > > for all global variables.
> > > 
> > > For context, I'm testing using QEMU emulation on an x86-64 host. Note
> > > that it logs "KASLR disabled: CPU has no PRNG" early during boot. My
> > > exact setup is:
> > > 
> > >   $ git clone https://github.com/osandov/drgn.git
> > >   $ cd drgn
> > >   $ python3 -m vmtest.rootfsbuild -a s390x --build-drgn
> > >   $ python3 -m vmtest.vm -k 's390x:6.10.*' bash -i
> > >   # python3 -m drgn  
> > >   >>> prog['init_task'].comm  
> > >   (char [16])""
> > > 
> > > That should be printing "swapper/0".
> > > 
> > > Any ideas what's going on here?  
> > 
> > On s390 no kernel symbol exists below 0x100000 offset within the
> > vmlinux image and thus this part is never mapped into the kernel
> > memory. That way KERNELOFFSET turns out to be off on value of
> > 0x100000 - and that is what you observe.
> > 
> > That breaks the way drgn finds a kernel symbol, but does not
> > exactly contradicts to the existing KERNELOFFSET description
> > (Documentation/admin-guide/kdump/vmcoreinfo.rst):
> > 
> > ===
> > KERNELOFFSET
> > ------------
> > 
> > The kernel randomization offset. Used to compute the page offset. If
> > KASLR is disabled, this value is zero.
> > ===
> > 
> > I would say to some degree there is also inconsisten with regard
> > to /proc/ files existence:
> > /proc/kcore    is enabled by CONFIG_PROC_KCORE option, while
> > /proc/kallsyms is enabled by CONFIG_KALLSYMS option.
> > I assume drgn expects both files exist and does not work otherwise.

drgn doesn't use kallsyms, partially because it's not guaranteed to
exist as you pointed out, and partially because it's slow.

> > Nevertheless, it is still possible to refer to only one file for
> > symbol resolution and use an always-present symbol. E.g _stext
> > could be leveraged like this:
> > 
> > # grep -w init_task /proc/kallsyms
> > 000003ffe13e9400 D init_task
> > # grep -w _stext /proc/kallsyms
> > 000003ffe0000000 T _stext
> > 
> > 0x3ffe13e9400 - 0x3ffe0000000 == 0x13e9400
> > 
> > # eu-readelf -s vmlinux | grep -w _stext
> > 178112: 0000000000100000      0 NOTYPE  GLOBAL DEFAULT        1 _stext
> > 
> > 0x13e9400 + 0x100000 == 0x14e9400
> > 
> > # eu-readelf -s vmlinux | grep -w init_task
> >   498: 0000000000000000      0 FILE    LOCAL  DEFAULT      ABS init_task.c
> > 182344: 00000000014e9400   8960 OBJECT  GLOBAL DEFAULT       28 init_task
> > 
> > I guess, the above holds true for all architectures.
> > If so, I would suggest consider using that approach.
> > 
> > Having said that, we will try to turn KERNELOFFSET from a synthetic
> > value "Used to compute the page offset" to what drgn expects it to be.
> 
> Thinking about it now, I'm not sure it makes life easier. Because then
> we'll have some old kernels with the current (unexpected) definition of
> KERNELOFFSET and some new kernels with a more standard definition of
> it, but if I read vmcoreinfo, how do I know if the value has the old or
> the new meaning?

I wasn't able to get real KASLR working on my s390x VM, but what I found
in testing without KASLR was:

- Before commit c98d2ecae08f ("s390/mm: Uncouple physical vs virtual
  address spaces"), KERNELOFFSET was not set at all (this is expected).
- After commit c98d2ecae08f ("s390/mm: Uncouple physical vs virtual
  address spaces"), but before commit 56b1069c40c7 ("s390/boot: Rework
  deployment of the kernel image"), KERNELOFFSET was set in a way that
  drgn understands even without KASLR (that's a little odd but fine with
  me).
- After commit 56b1069c40c7 ("s390/boot: Rework deployment of the kernel
  image"), KERNELOFFSET was set "incorrectly"

So at least for no KASLR, the breakage has been limited only to the 6.10
rcs, which isn't too late to fix. I'd be curious what the behavior was
with KASLR before 6.10, though.

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Petr Tesařík]

On Tue, 9 Jul 2024 13:55:58 -0700
Omar Sandoval <osandov@osandov.com> wrote:

> On Tue, Jul 09, 2024 at 09:21:41PM +0200, Petr Tesařík wrote:
> > On Thu, 27 Jun 2024 15:51:09 +0200
> > Alexander Gordeev <agordeev@linux.ibm.com> wrote:
> >   
> > > On Thu, Jun 20, 2024 at 04:34:15PM -0700, Omar Sandoval wrote:
> > > 
> > > Hi Omar,
> > >   
> > > > Hi, Alexander and Sven,
> > > > 
> > > > I just got around to testing drgn on s390x on 6.10-rc4, and it appears
> > > > to be broken. I bisected it to commit 56b1069c40c7 ("s390/boot: Rework
> > > > deployment of the kernel image") and narrowed it down to an issue with
> > > > the KERNELOFFSET value reported in vmcoreinfo.
> > > > 
> > > > On my test kernel, the ELF symbol for init_task is 0xc96f00:
> > > > 
> > > >   $ eu-readelf -s build/vmtest/s390x/kernel-6.10.0-rc4-vmtest30.1default/build/vmlinux | grep ' init_task$'
> > > >   72273: 0000000000c96f00   4352 OBJECT  GLOBAL DEFAULT       18 init_task
> > > > 
> > > > And the address in the loaded kernel is 0x3ffffeaaf00:
> > > > 
> > > >   # grep ' init_task$' /proc/kallsyms
> > > >   000003ffffeaaf00 D init_task
> > > > 
> > > > 0x3ffffeaaf00 - 0xc96f00 is 0x3ffff214000
> > > > 
> > > > However, this doesn't match the value of KERNELOFFSET in vmcoreinfo:
> > > > 
> > > >   # eu-readelf -n /proc/kcore | grep KERNELOFFSET
> > > >     KERNELOFFSET=3ffff314000
> > > > 
> > > > It's off by 0x100000. This causes drgn to compute the wrong addresses
> > > > for all global variables.
> > > > 
> > > > For context, I'm testing using QEMU emulation on an x86-64 host. Note
> > > > that it logs "KASLR disabled: CPU has no PRNG" early during boot. My
> > > > exact setup is:
> > > > 
> > > >   $ git clone https://github.com/osandov/drgn.git
> > > >   $ cd drgn
> > > >   $ python3 -m vmtest.rootfsbuild -a s390x --build-drgn
> > > >   $ python3 -m vmtest.vm -k 's390x:6.10.*' bash -i
> > > >   # python3 -m drgn    
> > > >   >>> prog['init_task'].comm    
> > > >   (char [16])""
> > > > 
> > > > That should be printing "swapper/0".
> > > > 
> > > > Any ideas what's going on here?    
> > > 
> > > On s390 no kernel symbol exists below 0x100000 offset within the
> > > vmlinux image and thus this part is never mapped into the kernel
> > > memory. That way KERNELOFFSET turns out to be off on value of
> > > 0x100000 - and that is what you observe.
> > > 
> > > That breaks the way drgn finds a kernel symbol, but does not
> > > exactly contradicts to the existing KERNELOFFSET description
> > > (Documentation/admin-guide/kdump/vmcoreinfo.rst):
> > > 
> > > ===
> > > KERNELOFFSET
> > > ------------
> > > 
> > > The kernel randomization offset. Used to compute the page offset. If
> > > KASLR is disabled, this value is zero.
> > > ===
> > > 
> > > I would say to some degree there is also inconsisten with regard
> > > to /proc/ files existence:
> > > /proc/kcore    is enabled by CONFIG_PROC_KCORE option, while
> > > /proc/kallsyms is enabled by CONFIG_KALLSYMS option.
> > > I assume drgn expects both files exist and does not work otherwise.  
> 
> drgn doesn't use kallsyms, partially because it's not guaranteed to
> exist as you pointed out, and partially because it's slow.
> 
> > > Nevertheless, it is still possible to refer to only one file for
> > > symbol resolution and use an always-present symbol. E.g _stext
> > > could be leveraged like this:
> > > 
> > > # grep -w init_task /proc/kallsyms
> > > 000003ffe13e9400 D init_task
> > > # grep -w _stext /proc/kallsyms
> > > 000003ffe0000000 T _stext
> > > 
> > > 0x3ffe13e9400 - 0x3ffe0000000 == 0x13e9400
> > > 
> > > # eu-readelf -s vmlinux | grep -w _stext
> > > 178112: 0000000000100000      0 NOTYPE  GLOBAL DEFAULT        1 _stext
> > > 
> > > 0x13e9400 + 0x100000 == 0x14e9400
> > > 
> > > # eu-readelf -s vmlinux | grep -w init_task
> > >   498: 0000000000000000      0 FILE    LOCAL  DEFAULT      ABS init_task.c
> > > 182344: 00000000014e9400   8960 OBJECT  GLOBAL DEFAULT       28 init_task
> > > 
> > > I guess, the above holds true for all architectures.
> > > If so, I would suggest consider using that approach.
> > > 
> > > Having said that, we will try to turn KERNELOFFSET from a synthetic
> > > value "Used to compute the page offset" to what drgn expects it to be.  
> > 
> > Thinking about it now, I'm not sure it makes life easier. Because then
> > we'll have some old kernels with the current (unexpected) definition of
> > KERNELOFFSET and some new kernels with a more standard definition of
> > it, but if I read vmcoreinfo, how do I know if the value has the old or
> > the new meaning?  
> 
> I wasn't able to get real KASLR working on my s390x VM, but what I found
> in testing without KASLR was:
> 
> - Before commit c98d2ecae08f ("s390/mm: Uncouple physical vs virtual
>   address spaces"), KERNELOFFSET was not set at all (this is expected).
> - After commit c98d2ecae08f ("s390/mm: Uncouple physical vs virtual
>   address spaces"), but before commit 56b1069c40c7 ("s390/boot: Rework
>   deployment of the kernel image"), KERNELOFFSET was set in a way that
>   drgn understands even without KASLR (that's a little odd but fine with
>   me).
> - After commit 56b1069c40c7 ("s390/boot: Rework deployment of the kernel
>   image"), KERNELOFFSET was set "incorrectly"
> 
> So at least for no KASLR, the breakage has been limited only to the 6.10
> rcs, which isn't too late to fix. I'd be curious what the behavior was
> with KASLR before 6.10, though.

OK, I'll check SLES 15 SP5 (kernel 5.14) and SP6 (kernel 6.4). Both
enable KASLR, but it can be turned off on the command line (or I can
even rebuild the kernel without CONFIG_RANDOMIZE_BASE if that makes a
difference).

Petr T

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Alexander Gordeev]

On Wed, Jul 10, 2024 at 07:02:46AM +0200, Petr Tesařík wrote:
...
> > I wasn't able to get real KASLR working on my s390x VM, but what I found
> > in testing without KASLR was:
> > 
> > - Before commit c98d2ecae08f ("s390/mm: Uncouple physical vs virtual
> >   address spaces"), KERNELOFFSET was not set at all (this is expected).
> > - After commit c98d2ecae08f ("s390/mm: Uncouple physical vs virtual
> >   address spaces"), but before commit 56b1069c40c7 ("s390/boot: Rework
> >   deployment of the kernel image"), KERNELOFFSET was set in a way that
> >   drgn understands even without KASLR (that's a little odd but fine with
> >   me).
> > - After commit 56b1069c40c7 ("s390/boot: Rework deployment of the kernel
> >   image"), KERNELOFFSET was set "incorrectly"
> > 
> > So at least for no KASLR, the breakage has been limited only to the 6.10
> > rcs, which isn't too late to fix. I'd be curious what the behavior was
> > with KASLR before 6.10, though.
> 
> OK, I'll check SLES 15 SP5 (kernel 5.14) and SP6 (kernel 6.4). Both
> enable KASLR, but it can be turned off on the command line (or I can
> even rebuild the kernel without CONFIG_RANDOMIZE_BASE if that makes a
> difference).

'nokaslr' kernel option should be enough.

> Petr T

Thanks!

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Omar Sandoval]

On Wed, Jul 10, 2024 at 07:02:46AM +0200, Petr Tesařík wrote:
> On Tue, 9 Jul 2024 13:55:58 -0700
> Omar Sandoval <osandov@osandov.com> wrote:
> 
> > On Tue, Jul 09, 2024 at 09:21:41PM +0200, Petr Tesařík wrote:
> > > On Thu, 27 Jun 2024 15:51:09 +0200
> > > Alexander Gordeev <agordeev@linux.ibm.com> wrote:
> > >   
> > > > On Thu, Jun 20, 2024 at 04:34:15PM -0700, Omar Sandoval wrote:
> > > > 
> > > > Hi Omar,
> > > >   
> > > > > Hi, Alexander and Sven,
> > > > > 
> > > > > I just got around to testing drgn on s390x on 6.10-rc4, and it appears
> > > > > to be broken. I bisected it to commit 56b1069c40c7 ("s390/boot: Rework
> > > > > deployment of the kernel image") and narrowed it down to an issue with
> > > > > the KERNELOFFSET value reported in vmcoreinfo.
> > > > > 
> > > > > On my test kernel, the ELF symbol for init_task is 0xc96f00:
> > > > > 
> > > > >   $ eu-readelf -s build/vmtest/s390x/kernel-6.10.0-rc4-vmtest30.1default/build/vmlinux | grep ' init_task$'
> > > > >   72273: 0000000000c96f00   4352 OBJECT  GLOBAL DEFAULT       18 init_task
> > > > > 
> > > > > And the address in the loaded kernel is 0x3ffffeaaf00:
> > > > > 
> > > > >   # grep ' init_task$' /proc/kallsyms
> > > > >   000003ffffeaaf00 D init_task
> > > > > 
> > > > > 0x3ffffeaaf00 - 0xc96f00 is 0x3ffff214000
> > > > > 
> > > > > However, this doesn't match the value of KERNELOFFSET in vmcoreinfo:
> > > > > 
> > > > >   # eu-readelf -n /proc/kcore | grep KERNELOFFSET
> > > > >     KERNELOFFSET=3ffff314000
> > > > > 
> > > > > It's off by 0x100000. This causes drgn to compute the wrong addresses
> > > > > for all global variables.
> > > > > 
> > > > > For context, I'm testing using QEMU emulation on an x86-64 host. Note
> > > > > that it logs "KASLR disabled: CPU has no PRNG" early during boot. My
> > > > > exact setup is:
> > > > > 
> > > > >   $ git clone https://github.com/osandov/drgn.git
> > > > >   $ cd drgn
> > > > >   $ python3 -m vmtest.rootfsbuild -a s390x --build-drgn
> > > > >   $ python3 -m vmtest.vm -k 's390x:6.10.*' bash -i
> > > > >   # python3 -m drgn    
> > > > >   >>> prog['init_task'].comm    
> > > > >   (char [16])""
> > > > > 
> > > > > That should be printing "swapper/0".
> > > > > 
> > > > > Any ideas what's going on here?    
> > > > 
> > > > On s390 no kernel symbol exists below 0x100000 offset within the
> > > > vmlinux image and thus this part is never mapped into the kernel
> > > > memory. That way KERNELOFFSET turns out to be off on value of
> > > > 0x100000 - and that is what you observe.
> > > > 
> > > > That breaks the way drgn finds a kernel symbol, but does not
> > > > exactly contradicts to the existing KERNELOFFSET description
> > > > (Documentation/admin-guide/kdump/vmcoreinfo.rst):
> > > > 
> > > > ===
> > > > KERNELOFFSET
> > > > ------------
> > > > 
> > > > The kernel randomization offset. Used to compute the page offset. If
> > > > KASLR is disabled, this value is zero.
> > > > ===
> > > > 
> > > > I would say to some degree there is also inconsisten with regard
> > > > to /proc/ files existence:
> > > > /proc/kcore    is enabled by CONFIG_PROC_KCORE option, while
> > > > /proc/kallsyms is enabled by CONFIG_KALLSYMS option.
> > > > I assume drgn expects both files exist and does not work otherwise.  
> > 
> > drgn doesn't use kallsyms, partially because it's not guaranteed to
> > exist as you pointed out, and partially because it's slow.
> > 
> > > > Nevertheless, it is still possible to refer to only one file for
> > > > symbol resolution and use an always-present symbol. E.g _stext
> > > > could be leveraged like this:
> > > > 
> > > > # grep -w init_task /proc/kallsyms
> > > > 000003ffe13e9400 D init_task
> > > > # grep -w _stext /proc/kallsyms
> > > > 000003ffe0000000 T _stext
> > > > 
> > > > 0x3ffe13e9400 - 0x3ffe0000000 == 0x13e9400
> > > > 
> > > > # eu-readelf -s vmlinux | grep -w _stext
> > > > 178112: 0000000000100000      0 NOTYPE  GLOBAL DEFAULT        1 _stext
> > > > 
> > > > 0x13e9400 + 0x100000 == 0x14e9400
> > > > 
> > > > # eu-readelf -s vmlinux | grep -w init_task
> > > >   498: 0000000000000000      0 FILE    LOCAL  DEFAULT      ABS init_task.c
> > > > 182344: 00000000014e9400   8960 OBJECT  GLOBAL DEFAULT       28 init_task
> > > > 
> > > > I guess, the above holds true for all architectures.
> > > > If so, I would suggest consider using that approach.
> > > > 
> > > > Having said that, we will try to turn KERNELOFFSET from a synthetic
> > > > value "Used to compute the page offset" to what drgn expects it to be.  
> > > 
> > > Thinking about it now, I'm not sure it makes life easier. Because then
> > > we'll have some old kernels with the current (unexpected) definition of
> > > KERNELOFFSET and some new kernels with a more standard definition of
> > > it, but if I read vmcoreinfo, how do I know if the value has the old or
> > > the new meaning?  
> > 
> > I wasn't able to get real KASLR working on my s390x VM, but what I found
> > in testing without KASLR was:
> > 
> > - Before commit c98d2ecae08f ("s390/mm: Uncouple physical vs virtual
> >   address spaces"), KERNELOFFSET was not set at all (this is expected).
> > - After commit c98d2ecae08f ("s390/mm: Uncouple physical vs virtual
> >   address spaces"), but before commit 56b1069c40c7 ("s390/boot: Rework
> >   deployment of the kernel image"), KERNELOFFSET was set in a way that
> >   drgn understands even without KASLR (that's a little odd but fine with
> >   me).
> > - After commit 56b1069c40c7 ("s390/boot: Rework deployment of the kernel
> >   image"), KERNELOFFSET was set "incorrectly"
> > 
> > So at least for no KASLR, the breakage has been limited only to the 6.10
> > rcs, which isn't too late to fix. I'd be curious what the behavior was
> > with KASLR before 6.10, though.
> 
> OK, I'll check SLES 15 SP5 (kernel 5.14) and SP6 (kernel 6.4). Both
> enable KASLR, but it can be turned off on the command line (or I can
> even rebuild the kernel without CONFIG_RANDOMIZE_BASE if that makes a
> difference).

The case I'm interested here is with KASLR enabled. In those kernel
versions, is KERNELOFFSET the difference between the addresses in
vmlinux and the actual addresses in memory?

Thanks,
Omar

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Alexander Gordeev]

On Tue, Jul 09, 2024 at 09:21:41PM +0200, Petr Tesařík wrote:

Hi Petr,
...
> > I would say to some degree there is also inconsisten with regard
> > to /proc/ files existence:
> > /proc/kcore    is enabled by CONFIG_PROC_KCORE option, while
> > /proc/kallsyms is enabled by CONFIG_KALLSYMS option.
> > I assume drgn expects both files exist and does not work otherwise.
> > 
> > Nevertheless, it is still possible to refer to only one file for
> > symbol resolution and use an always-present symbol. E.g _stext
> > could be leveraged like this:
> > 
> > # grep -w init_task /proc/kallsyms
> > 000003ffe13e9400 D init_task
> > # grep -w _stext /proc/kallsyms
> > 000003ffe0000000 T _stext
> > 
> > 0x3ffe13e9400 - 0x3ffe0000000 == 0x13e9400
> > 
> > # eu-readelf -s vmlinux | grep -w _stext
> > 178112: 0000000000100000      0 NOTYPE  GLOBAL DEFAULT        1 _stext
> > 
> > 0x13e9400 + 0x100000 == 0x14e9400
> > 
> > # eu-readelf -s vmlinux | grep -w init_task
> >   498: 0000000000000000      0 FILE    LOCAL  DEFAULT      ABS init_task.c
> > 182344: 00000000014e9400   8960 OBJECT  GLOBAL DEFAULT       28 init_task
> > 
> > I guess, the above holds true for all architectures.
> > If so, I would suggest consider using that approach.
> > 
> > Having said that, we will try to turn KERNELOFFSET from a synthetic
> > value "Used to compute the page offset" to what drgn expects it to be.
> 
> Thinking about it now, I'm not sure it makes life easier. Because then
> we'll have some old kernels with the current (unexpected) definition of
> KERNELOFFSET and some new kernels with a more standard definition of
> it, but if I read vmcoreinfo, how do I know if the value has the old or
> the new meaning?

The approach I suggested to consider would not use KERNELOFFSET at all.

> Petr T

Thanks!

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Omar Sandoval]

On Wed, Jul 10, 2024 at 10:53:11AM +0200, Alexander Gordeev wrote:
> On Tue, Jul 09, 2024 at 09:21:41PM +0200, Petr Tesařík wrote:
> 
> Hi Petr,
> ...
> > > I would say to some degree there is also inconsisten with regard
> > > to /proc/ files existence:
> > > /proc/kcore    is enabled by CONFIG_PROC_KCORE option, while
> > > /proc/kallsyms is enabled by CONFIG_KALLSYMS option.
> > > I assume drgn expects both files exist and does not work otherwise.
> > > 
> > > Nevertheless, it is still possible to refer to only one file for
> > > symbol resolution and use an always-present symbol. E.g _stext
> > > could be leveraged like this:
> > > 
> > > # grep -w init_task /proc/kallsyms
> > > 000003ffe13e9400 D init_task
> > > # grep -w _stext /proc/kallsyms
> > > 000003ffe0000000 T _stext
> > > 
> > > 0x3ffe13e9400 - 0x3ffe0000000 == 0x13e9400
> > > 
> > > # eu-readelf -s vmlinux | grep -w _stext
> > > 178112: 0000000000100000      0 NOTYPE  GLOBAL DEFAULT        1 _stext
> > > 
> > > 0x13e9400 + 0x100000 == 0x14e9400
> > > 
> > > # eu-readelf -s vmlinux | grep -w init_task
> > >   498: 0000000000000000      0 FILE    LOCAL  DEFAULT      ABS init_task.c
> > > 182344: 00000000014e9400   8960 OBJECT  GLOBAL DEFAULT       28 init_task
> > > 
> > > I guess, the above holds true for all architectures.
> > > If so, I would suggest consider using that approach.
> > > 
> > > Having said that, we will try to turn KERNELOFFSET from a synthetic
> > > value "Used to compute the page offset" to what drgn expects it to be.
> > 
> > Thinking about it now, I'm not sure it makes life easier. Because then
> > we'll have some old kernels with the current (unexpected) definition of
> > KERNELOFFSET and some new kernels with a more standard definition of
> > it, but if I read vmcoreinfo, how do I know if the value has the old or
> > the new meaning?
> 
> The approach I suggested to consider would not use KERNELOFFSET at all.

Hi,

Unfortunately, using kallsyms isn't feasible for a few reasons:

1) /proc/kallsyms is not available when debugging a core dump.
2) As you pointed out, /proc/kallsyms is not necessarily enabled
   together with /proc/kcore.
3) Reading /proc/kallsyms is slow. On my system, drgn starts up in about
   250ms, and reading /proc/kallsyms takes about 110 ms.  This would
   slow down startup too much.

drgn intentionally avoids kallsyms for these reasons. (There is one
corner case on old < 4.11 kernels where we need kallsyms, but that will
go away eventually.)

I'd be really grateful if KERNELOFFSET was always the difference between
addresses in the kernel image and in memory like it is on other
architectures, or even if there was another, s390x-specific field in
vmcoreinfo for that.

Thanks,
Omar

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Alexander Gordeev]

On Thu, Jul 11, 2024 at 04:30:22PM -0700, Omar Sandoval wrote:
Hi Omar,
...
> I'd be really grateful if KERNELOFFSET was always the difference between
> addresses in the kernel image and in memory like it is on other
> architectures, or even if there was another, s390x-specific field in
> vmcoreinfo for that.

Do you mean another s390x-specific field that would be the same
as KERNELOFFSET on other architectures?

> Thanks,
> Omar

Thanks!

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Omar Sandoval]

On Fri, Jul 12, 2024 at 03:42:17PM +0200, Alexander Gordeev wrote:
> On Thu, Jul 11, 2024 at 04:30:22PM -0700, Omar Sandoval wrote:
> Hi Omar,
> ...
> > I'd be really grateful if KERNELOFFSET was always the difference between
> > addresses in the kernel image and in memory like it is on other
> > architectures, or even if there was another, s390x-specific field in
> > vmcoreinfo for that.
> 
> Do you mean another s390x-specific field that would be the same
> as KERNELOFFSET on other architectures?

Yeah. I'd still prefer if KERNELOFFSET itself was updated, but if that's
really not an option, another field could be a workaround.

Thanks,
Omar

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Alexander Gordeev]

On Fri, Jul 12, 2024 at 07:17:38AM -0700, Omar Sandoval wrote:

Hi Omar,

> On Fri, Jul 12, 2024 at 03:42:17PM +0200, Alexander Gordeev wrote:
> > On Thu, Jul 11, 2024 at 04:30:22PM -0700, Omar Sandoval wrote:
> > Hi Omar,
> > ...
> > > I'd be really grateful if KERNELOFFSET was always the difference between
> > > addresses in the kernel image and in memory like it is on other
> > > architectures, or even if there was another, s390x-specific field in
> > > vmcoreinfo for that.
> > 
> > Do you mean another s390x-specific field that would be the same
> > as KERNELOFFSET on other architectures?
> 
> Yeah. I'd still prefer if KERNELOFFSET itself was updated, but if that's
> really not an option, another field could be a workaround.

We have different approaches in how to get it done, which are
being investigated.

> Thanks,
> Omar

Thanks!

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Alexander Gordeev]

> On Fri, Jul 12, 2024 at 07:17:38AM -0700, Omar Sandoval wrote:
Hi Omar,
..
> We have different approaches in how to get it done, which are
> being investigated.

Could you please check whether the recent kernel solves the issue.

Thanks!

Re: Incorrect vmcoreinfo KERNELOFFSET after "s390/boot: Rework deployment of the kernel image"

[Omar Sandoval]

On Mon, Aug 26, 2024 at 10:22:29AM +0200, Alexander Gordeev wrote:
> > On Fri, Jul 12, 2024 at 07:17:38AM -0700, Omar Sandoval wrote:
> Hi Omar,
> ..
> > We have different approaches in how to get it done, which are
> > being investigated.
> 
> Could you please check whether the recent kernel solves the issue.
> 
> Thanks!

Yes, this appears to be fixed in 6.11-rc5. Thank you!

(Now that this issue is fixed, I can see that drgn has some legitimate
test failures on s390x caused by the paddr/vaddr split, but that needs
to be fixed in drgn.)

Thanks,
Omar