Re: grub causing NVDIMMs to be treated as normal memory

[Andrei Borzenkov <arvidjaar@gmail.com>]

27.11.2015 02:24, Elliott, Robert (Persistent Memory) пишет:
> 
>> -----Original Message-----
>> From: Andrei Borzenkov [mailto:arvidjaar@gmail.com]
>> Sent: Thursday, November 26, 2015 10:55 AM
>> To: Elliott, Robert (Persistent Memory) <elliott@hpe.com>; The
>> development of GNU GRUB <grub-devel@gnu.org>;
>> dan.j.williams@intel.com; linux-nvdimm@lists.01.org
>> Subject: Re: grub causing NVDIMMs to be treated as normal memory
>>
>> 26.11.2015 09:15, Elliott, Robert (Persistent Memory) пишет:
>> ...
>>
>>> ...
>>> mmap/efi/mmap.c:66: EFI memory region 0x880000000-0xc80000000: 14
>>> Unknown memory type 14, considering reserved
>>> mmap/efi/mmap.c:66: EFI memory region 0x1480000000-0x1a80000000:
>> 14
>>> Unknown memory type 14, considering reserved
>>>
>>>
>>>
>>> This is booting a new kernel with the EFI boot stub, so I can't
>>> confirm that it fixes the issue of exposing the address range as
>>> normal, but based on the print it's probably working.  I could
>>> add prints in grub-core/loader/i386/linux.c grub_e820_add_region
>>> to confirm the e820 contents at exit.
>>>
>>
>> Thanks. If it results in wrong e820 type we have much larger problem
>> so I do not think it necessary.
>>
>> I pushed it; as I understand this should serve as stopgap. If I get
>> around to implement persistent memory type would you test it? But as
>> I cannot tell when it happens, feel free to submit patch in the
>> meantime :)
> 
> As an experiment, I:
> * compiled a linux 4.4rc2 kernel with CONFIG_EFI_STUB disabled
> * added some prints in the linux kernel setup_memory_map()
>   to print the incoming bios_params e820 entries
> * modified the grub patch to set the range to AVAILABLE rather 
>   than RESERVED (simulating the bug, to see its impacts on
>   the kernel)
> 
> Before (CONFIG_EFI_STUB=y, grub mapping to RESERVED):
> bp: [mem 0x0000000000000000-0x0000000000092fff] usable
> bp: [mem 0x0000000000093000-0x0000000000093fff] reserved
> bp: [mem 0x0000000000094000-0x000000000009ffff] usable
> bp: [mem 0x0000000000100000-0x000000006affbfff] usable
> bp: [mem 0x000000006affc000-0x000000006b5fbfff] reserved
> bp: [mem 0x000000006b5fc000-0x000000006b5fcfff] usable
> bp: [mem 0x000000006b5fd000-0x000000006b67dfff] reserved
> bp: [mem 0x000000006b67e000-0x00000000784fefff] usable
> bp: [mem 0x00000000784ff000-0x00000000791fefff] reserved <--b
> bp: [mem 0x00000000791ff000-0x000000007b5fefff] ACPI NVS
> bp: [mem 0x000000007b5ff000-0x000000007b7fefff] ACPI data
> bp: [mem 0x000000007b7ff000-0x000000007b7fffff] usable
> bp: [mem 0x0000000100000000-0x000000087fffffff] usable <--a
> bp: [mem 0x0000000c80000000-0x000000147fffffff] usable <--a
> bp: [mem 0x0000000080000000-0x000000008fffffff] reserved
> bp: [mem 0x0000000880000000-0x0000000c7fffffff] reserved <--a
> bp: [mem 0x0000001480000000-0x0000001a7fffffff] reserved <--a
> 
> After (CONFIG_EFI_STUB=n, grub mapping to AVAILABLE):
> bp: [mem 0x0000000000000000-0x0000000000092fff] usable
> bp: [mem 0x0000000000093000-0x0000000000093fff] reserved
> bp: [mem 0x0000000000094000-0x000000000009ffff] usable
> bp: [mem 0x0000000000100000-0x000000006affbfff] usable
> bp: [mem 0x000000006affc000-0x000000006b5fbfff] reserved
> bp: [mem 0x000000006b5fc000-0x000000006b5fcfff] usable
> bp: [mem 0x000000006b5fd000-0x000000006b67dfff] reserved
> bp: [mem 0x000000006b67e000-0x00000000784fefff] usable
> bp: [mem 0x00000000784ff000-0x00000000788fefff] reserved <--b
> bp: [mem 0x00000000788ff000-0x00000000790fefff] type 20  <--b
> bp: [mem 0x00000000790ff000-0x00000000791fefff] reserved <--b
> bp: [mem 0x00000000791ff000-0x000000007b5fefff] ACPI NVS
> bp: [mem 0x000000007b5ff000-0x000000007b7fefff] ACPI data
> bp: [mem 0x000000007b7ff000-0x000000007b7fffff] usable
> bp: [mem 0x0000000080000000-0x000000008fffffff] reserved
> bp: [mem 0x0000000100000000-0x0000001a7fffffff] usable <--a
> 
> 
> Note (a): The 088 and 148 ranges were merged into the 
> 010 and 0c8 usable ranges, as expected for this experiment.
> 
> The NVDIMM drivers in the kernel don't load (nmem devices load,
> but namespaces fail and no pmem devices show up):
> [   27.835319] nd_pmem namespace0.0: could not reserve region [0x0x0000000880000000:0x400000000]
> [   27.835323]  ndbus1: nd_pmem.probe(namespace0.0) = -16
> [   27.835355] nd_pmem namespace2.0: could not reserve region [0x0x0000001880000000:0x200000000]
> [   27.835356]  ndbus1: nd_pmem.probe(namespace2.0) = -16
> [   27.835373] nd_pmem namespace1.0: could not reserve region [0x0x0000001480000000:0x400000000]
> [   27.835374]  ndbus1: nd_pmem.probe(namespace1.0) = -16
> 
> That shows up as normal memory in /proc/iomem and elsewhere:
> 100000000-1a7fffffff : System RAM
> 

May be it is too early in the morning, but could you explain whether the
test outcome confirms the patch worked or not? :)

> $ numactl --hardware
> available: 1 nodes (0)
> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
> node 0 size: 104614 MB
> node 0 free: 103303 MB
> node distances:
> node   0
>   0:  10
> 
> 
> 
> Note (b): The internal GRUB_MEMORY_CODE (20) value is 
> leaking through to the E820 table.
> 
> That appears to be from this patch on 2013-10-14:
> 	6de9ee86 Pass-through unknown E820 types

If we are discussing ACPI 6.0 systems here, it explicitly says that
values above 12 should be treated as reserved. Does it cause problems?

> which deleted a switch statement mapping the internal codes 
> to E820 values and included this:
> -    case GRUB_MEMORY_CODE:
> -    case GRUB_MEMORY_RESERVED:
> -      ctx->cur.type = GRUB_E820_RESERVED;
> 
> based on this hope:
> +/* GRUB types conveniently match E820 types.  */
> 
> That's only true for types 1 through 5:
> typedef enum grub_memory_type
>   {
>     GRUB_MEMORY_AVAILABLE = 1,
>     GRUB_MEMORY_RESERVED = 2,
>     GRUB_MEMORY_ACPI = 3,
>     GRUB_MEMORY_NVS = 4,
>     GRUB_MEMORY_BADRAM = 5,
>     GRUB_MEMORY_COREBOOT_TABLES = 16,
>     GRUB_MEMORY_CODE = 20,
>     /* This one is special: it's used internally but is never reported
>        by firmware. Don't use -1 as it's used internally for other purposes. */
>     GRUB_MEMORY_HOLE = -2,
>     GRUB_MEMORY_MAX = 0x10000
>   } grub_memory_type_t;
> 
> 
> ---
> Robert Elliott, HPE Persistent Memory
> 
> 
> 
>