Re: ACPI NVS range conflicting with Dom0 page tables (or kernel image)

["Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com>]

[-- Attachment #1: Type: text/plain, Size: 9338 bytes --]

On Tue, Aug 06, 2024 at 05:24:22PM +0200, Jürgen Groß wrote:
> On 06.08.24 17:21, Marek Marczykowski-Górecki wrote:
> > On Tue, Aug 06, 2024 at 04:12:32PM +0200, Jürgen Groß wrote:
> > > Marek,
> > > 
> > > On 17.06.24 16:03, Marek Marczykowski-Górecki wrote:
> > > > On Mon, Jun 17, 2024 at 01:22:37PM +0200, Jan Beulich wrote:
> > > > > Hello,
> > > > > 
> > > > > while it feels like we had a similar situation before, I can't seem to be
> > > > > able to find traces thereof, or associated (Linux) commits.
> > > > 
> > > > Is it some AMD Threadripper system by a chance? Previous thread on this
> > > > issue:
> > > > https://lore.kernel.org/xen-devel/CAOCpoWdOH=xGxiQSC1c5Ueb1THxAjH4WiZbCZq-QT+d_KAk3SA@mail.gmail.com/
> > > > 
> > > > > With
> > > > > 
> > > > > (XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x4000000
> > > > > ...
> > > > > (XEN)  Dom0 alloc.:   0000000440000000->0000000448000000 (619175 pages to be allocated)
> > > > > ...
> > > > > (XEN)  Loaded kernel: ffffffff81000000->ffffffff84000000
> > > > > 
> > > > > the kernel occupies the space from 16Mb to 64Mb in the initial allocation.
> > > > > Page tables come (almost) directly above:
> > > > > 
> > > > > (XEN)  Page tables:   ffffffff84001000->ffffffff84026000
> > > > > 
> > > > > I.e. they're just above the 64Mb boundary. Yet sadly in the host E820 map
> > > > > there is
> > > > > 
> > > > > (XEN)  [0000000004000000, 0000000004009fff] (ACPI NVS)
> > > > > 
> > > > > i.e. a non-RAM range starting at 64Mb. The kernel (currently) won't tolerate
> > > > > such an overlap (also if it was overlapping the kernel image, e.g. if on the
> > > > > machine in question s sufficiently much larger kernel was used). Yet with its
> > > > > fundamental goal of making its E820 match the host one I'm also in trouble
> > > > > thinking of possible solutions / workarounds. I certainly do not see Xen
> > > > > trying to cover for this, as the E820 map re-arrangement is purely a kernel
> > > > > side decision (forward ported kernels got away without, and what e.g. the
> > > > > BSDs do is entirely unknown to me).
> > > > 
> > > > In Qubes we have worked around the issue by moving the kernel lower
> > > > (CONFIG_PHYSICAL_START=0x200000):
> > > > https://github.com/QubesOS/qubes-linux-kernel/commit/3e8be4ac1682370977d4d0dc1d782c428d860282
> > > > 
> > > > Far from ideal, but gets it bootable...
> > > > 
> > > 
> > > could you test the attached kernel patches? They should fix the issue without
> > > having to modify CONFIG_PHYSICAL_START.
> > > 
> > > I have tested them to boot up without problem on my test system, but I don't
> > > have access to a system showing the E820 map conflict you are seeing.
> > > 
> > > The patches have been developed against kernel 6.11-rc2, but I think they
> > > should apply to a 6.10 and maybe even an older kernel.
> > 
> > Sure, but tomorrow-ish.
> 
> Thanks.

Seems to work :)

Snippets from Xen log:

    (XEN) EFI RAM map:
    (XEN)  [0000000000000000, 000000000009ffff] (usable)
    (XEN)  [00000000000a0000, 00000000000fffff] (reserved)
    (XEN)  [0000000000100000, 0000000003ffffff] (usable)
    (XEN)  [0000000004000000, 0000000004011fff] (ACPI NVS)
    (XEN)  [0000000004012000, 0000000009df1fff] (usable)
    (XEN)  [0000000009df2000, 0000000009ffffff] (reserved)
    (XEN)  [000000000a000000, 00000000a8840fff] (usable)
    (XEN)  [00000000a8841000, 00000000a9d9ffff] (reserved)
    (XEN)  [00000000a9da0000, 00000000a9dd4fff] (ACPI data)
    (XEN)  [00000000a9dd5000, 00000000a9dd5fff] (reserved)
    (XEN)  [00000000a9dd6000, 00000000a9f20fff] (ACPI data)
    (XEN)  [00000000a9f21000, 00000000aa099fff] (ACPI NVS)
    (XEN)  [00000000aa09a000, 00000000ab1fefff] (reserved)
    (XEN)  [00000000ab1ff000, 00000000abffffff] (usable)
    (XEN)  [00000000ac000000, 00000000afffffff] (reserved)
    (XEN)  [00000000b2500000, 00000000b2580fff] (reserved)
    (XEN)  [00000000b3580000, 00000000b3600fff] (reserved)
    (XEN)  [00000000e2100000, 00000000e2280fff] (reserved)
    (XEN)  [00000000fa180000, 00000000fa200fff] (reserved)
    (XEN)  [00000000fa300000, 00000000fa3fffff] (reserved)
    (XEN)  [00000000fea00000, 00000000feafffff] (reserved)
    (XEN)  [00000000fec00000, 00000000fec00fff] (reserved)
    (XEN)  [00000000fec10000, 00000000fec10fff] (reserved)
    (XEN)  [00000000fed00000, 00000000fed00fff] (reserved)
    (XEN)  [00000000fed40000, 00000000fed44fff] (reserved)
    (XEN)  [00000000fed80000, 00000000fed8ffff] (reserved)
    (XEN)  [00000000fedc2000, 00000000fedcffff] (reserved)
    (XEN)  [00000000fedd4000, 00000000fedd5fff] (reserved)
    (XEN)  [00000000fee00000, 00000000feefffff] (reserved)
    (XEN)  [00000000ff000000, 00000000ffffffff] (reserved)
    (XEN)  [0000000100000000, 000000104f1fffff] (usable)
    (XEN)  [000000104f200000, 000000104fffffff] (reserved)
    (XEN)  [0000010000000000, 00000100103fffff] (reserved)
    (XEN)  [0000018030000000, 00000180403fffff] (reserved)
    (XEN)  [0000018060000000, 00000180703fffff] (reserved)
    (XEN)  [0000020090000000, 00000200a03fffff] (reserved)
    ...
    (XEN) Dom0 has maximum 1400 PIRQs
    (XEN)  Xen  kernel: 64-bit, lsb
    (XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x4800000
    (XEN) PHYSICAL MEMORY ARRANGEMENT:
    (XEN)  Dom0 alloc.:   0000001010000000->0000001018000000 (1000315 pages to be allocated)
    (XEN)  Init. ramdisk: 000000104b57b000->000000104f1ffe72
    (XEN) VIRTUAL MEMORY ARRANGEMENT:
    (XEN)  Loaded kernel: ffffffff81000000->ffffffff84800000
    (XEN)  Phys-Mach map: 0000008000000000->0000008000800000
    (XEN)  Start info:    ffffffff84800000->ffffffff848004b8
    (XEN)  Page tables:   ffffffff84801000->ffffffff8482a000
    (XEN)  Boot stack:    ffffffff8482a000->ffffffff8482b000
    (XEN)  TOTAL:         ffffffff80000000->ffffffff84c00000
    (XEN)  ENTRY ADDRESS: ffffffff838b7640

So, it would indeed conflict with the ACPI NVS region, but the system
started, and later dom0 reports this region remapped:

    [    0.000000] BIOS-provided physical RAM map:
    [    0.000000] Xen: [mem 0x0000000000000000-0x000000000007ffff] usable
    [    0.000000] Xen: [mem 0x0000000000080000-0x00000000000fffff] reserved
    [    0.000000] Xen: [mem 0x0000000000100000-0x0000000009df1fff] usable
    [    0.000000] Xen: [mem 0x0000000009df2000-0x0000000009ffffff] reserved
    [    0.000000] Xen: [mem 0x000000000a000000-0x00000000a8840fff] usable
    [    0.000000] Xen: [mem 0x00000000a8841000-0x00000000a9d9ffff] reserved
    [    0.000000] Xen: [mem 0x00000000a9da0000-0x00000000a9dd4fff] ACPI data
    [    0.000000] Xen: [mem 0x00000000a9dd5000-0x00000000a9dd5fff] reserved
    [    0.000000] Xen: [mem 0x00000000a9dd6000-0x00000000a9f20fff] ACPI data
    [    0.000000] Xen: [mem 0x00000000a9f21000-0x00000000aa099fff] ACPI NVS
    [    0.000000] Xen: [mem 0x00000000aa09a000-0x00000000ab1fefff] reserved
    [    0.000000] Xen: [mem 0x00000000ab1ff000-0x00000000abffffff] usable
    [    0.000000] Xen: [mem 0x00000000ac000000-0x00000000afffffff] reserved
    [    0.000000] Xen: [mem 0x00000000b2500000-0x00000000b2580fff] reserved
    [    0.000000] Xen: [mem 0x00000000b3580000-0x00000000b3600fff] reserved
    [    0.000000] Xen: [mem 0x00000000e2100000-0x00000000e2280fff] reserved
    [    0.000000] Xen: [mem 0x00000000fa180000-0x00000000fa200fff] reserved
    [    0.000000] Xen: [mem 0x00000000fa300000-0x00000000fa3fffff] reserved
    [    0.000000] Xen: [mem 0x00000000fea00000-0x00000000feafffff] reserved
    [    0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
    [    0.000000] Xen: [mem 0x00000000fec10000-0x00000000fec10fff] reserved
    [    0.000000] Xen: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
    [    0.000000] Xen: [mem 0x00000000fed40000-0x00000000fed44fff] reserved
    [    0.000000] Xen: [mem 0x00000000fed80000-0x00000000fed8ffff] reserved
    [    0.000000] Xen: [mem 0x00000000fedc2000-0x00000000fedcffff] reserved
    [    0.000000] Xen: [mem 0x00000000fedd4000-0x00000000fedd5fff] reserved
    [    0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
    [    0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
    [    0.000000] Xen: [mem 0x0000000100000000-0x0000000156c4bfff] usable
    [    0.000000] Xen: [mem 0x000000104f1ee000-0x000000104f1fffff] ACPI NVS
    [    0.000000] Xen: [mem 0x000000104f200000-0x000000104fffffff] reserved
    [    0.000000] Xen: [mem 0x0000010000000000-0x00000100103fffff] reserved
    [    0.000000] Xen: [mem 0x0000018030000000-0x00000180403fffff] reserved
    [    0.000000] Xen: [mem 0x0000018060000000-0x00000180703fffff] reserved
    [    0.000000] Xen: [mem 0x0000020090000000-0x00000200a03fffff] reserved

> > > If possible it would be nice to verify suspend to disk still working, as
> > > the kernel will need to access the ACPI NVS area in this case.
> > 
> > That might be harder, as Qubes OS doesn't support suspend to disk, but
> > I'll see if something can be done.
> 
> Thinking about it - can this ever work with Xen?

Indeed, this is not going to work.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]