Legacy NVDIMM-F over EFI

Legacy NVDIMM-F over EFI

[Yigal Korman]

Hi all,
I've got a couple of legacy 'type 12' NVDIMM-F modules in our lab I've
been successfully testing.
Recently I moved them to a new system where EFI bootloader is used
(CentOS 7 installed with EFI) and  was surprised to see that they're
not identified as legacy 'type 12' persistent memory, are not caught
by the 'nd_pmem' driver and I don't get /dev/pmemX devices.
They show as 'type 7' in the E820 map and 'Persistent memory' in /proc/iomem.
Is this the intended behavior? shouldn't they be identified as legacy
if they don't have NFIT?

Thanks,
Yigal
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: Legacy NVDIMM-F over EFI

[Dan Williams]

On Tue, May 17, 2016 at 8:36 AM, Yigal Korman <yigal@plexistor.com> wrote:
> Hi all,
> I've got a couple of legacy 'type 12' NVDIMM-F modules in our lab I've
> been successfully testing.
> Recently I moved them to a new system where EFI bootloader is used
> (CentOS 7 installed with EFI) and  was surprised to see that they're
> not identified as legacy 'type 12' persistent memory, are not caught
> by the 'nd_pmem' driver and I don't get /dev/pmemX devices.
> They show as 'type 7' in the E820 map and 'Persistent memory' in /proc/iomem.
> Is this the intended behavior? shouldn't they be identified as legacy
> if they don't have NFIT?

If the BIOS identifies it as type-7 it is broken, as far as Linux is
concerned, if does not also export an NFIT with an ACPI0012 device.
One would hope there are no new BIOS images being produced that use
type-12...
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: Legacy NVDIMM-F over EFI

[Jeff Moyer]

Dan Williams <dan.j.williams@intel.com> writes:

> On Tue, May 17, 2016 at 8:36 AM, Yigal Korman <yigal@plexistor.com> wrote:
>> Hi all,
>> I've got a couple of legacy 'type 12' NVDIMM-F modules in our lab I've
>> been successfully testing.

Just for clarification, NVDIMM-F is not byte-addressable persistent
memory.  I think you mean NVDIMM-N.

Cheers,
Jeff
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: Legacy NVDIMM-F over EFI

[Yigal Korman]

On May 17, 2016 6:57 PM, "Dan Williams" <dan.j.williams@intel.com> wrote:
>
> On Tue, May 17, 2016 at 8:36 AM, Yigal Korman <yigal@plexistor.com> wrote:
> > Hi all,
> > I've got a couple of legacy 'type 12' NVDIMM-F modules in our lab I've
> > been successfully testing.
> > Recently I moved them to a new system where EFI bootloader is used
> > (CentOS 7 installed with EFI) and  was surprised to see that they're
> > not identified as legacy 'type 12' persistent memory, are not caught
> > by the 'nd_pmem' driver and I don't get /dev/pmemX devices.
> > They show as 'type 7' in the E820 map and 'Persistent memory' in
/proc/iomem.
> > Is this the intended behavior? shouldn't they be identified as legacy
> > if they don't have NFIT?
>
> If the BIOS identifies it as type-7 it is broken, as far as Linux is
> concerned, if does not also export an NFIT with an ACPI0012 device.
> One would hope there are no new BIOS images being produced that use
> type-12...

I'm not sure I understand, what was supposed to the right thing for the
BIOS to do? AFAIK there's no type-12 in EFI...
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: Legacy NVDIMM-F over EFI

[Dan Williams]

On Tue, May 17, 2016 at 9:30 AM, Yigal Korman <yigal@plexistor.com> wrote:
>
> On May 17, 2016 6:57 PM, "Dan Williams" <dan.j.williams@intel.com> wrote:
>>
>> On Tue, May 17, 2016 at 8:36 AM, Yigal Korman <yigal@plexistor.com> wrote:
>> > Hi all,
>> > I've got a couple of legacy 'type 12' NVDIMM-F modules in our lab I've
>> > been successfully testing.
>> > Recently I moved them to a new system where EFI bootloader is used
>> > (CentOS 7 installed with EFI) and  was surprised to see that they're
>> > not identified as legacy 'type 12' persistent memory, are not caught
>> > by the 'nd_pmem' driver and I don't get /dev/pmemX devices.
>> > They show as 'type 7' in the E820 map and 'Persistent memory' in
>> > /proc/iomem.
>> > Is this the intended behavior? shouldn't they be identified as legacy
>> > if they don't have NFIT?
>>
>> If the BIOS identifies it as type-7 it is broken, as far as Linux is
>> concerned, if does not also export an NFIT with an ACPI0012 device.
>> One would hope there are no new BIOS images being produced that use
>> type-12...
>
> I'm not sure I understand, what was supposed to the right thing for the BIOS
> to do? AFAIK there's no type-12 in EFI...

The BIOS should mark the region as type-7, define an ACPI0012 device,
and publish an NFIT table.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: Legacy NVDIMM-F over EFI

[Toshi Kani]

On Tue, 2016-05-17 at 08:57 -0700, Dan Williams wrote:
> On Tue, May 17, 2016 at 8:36 AM, Yigal Korman <yigal@plexistor.com>
> wrote:
> > 
> > Hi all,
> > I've got a couple of legacy 'type 12' NVDIMM-F modules in our lab I've
> > been successfully testing.
> > Recently I moved them to a new system where EFI bootloader is used
> > (CentOS 7 installed with EFI) and  was surprised to see that they're
> > not identified as legacy 'type 12' persistent memory, are not caught
> > by the 'nd_pmem' driver and I don't get /dev/pmemX devices.
> > They show as 'type 7' in the E820 map and 'Persistent memory' in
> > /proc/iomem.
> > Is this the intended behavior? shouldn't they be identified as legacy
> > if they don't have NFIT?
>
> If the BIOS identifies it as type-7 it is broken, as far as Linux is
> concerned, if does not also export an NFIT with an ACPI0012 device.
> One would hope there are no new BIOS images being produced that use
> type-12...

On an EFI system and a kernel with CONFIG_EFI_STUB set, the kernel uses EFI
memory table, instead of e820.  So, e820 type-12 cannot be conveyed to the
kernel.

Since this EFI BIOS is broken as Dan pointed out, you might want to try
booting from a kernel with CONFIG_EFI_STUB unset to see if the kernel gets
e820 type-12.  (Note, if e820 is set to type-7, this would lead the kernel
to overwrite data in NVDIMM due to a bug in grub.)

-Toshi
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: Legacy NVDIMM-F over EFI

[Toshi Kani]

On Tue, 2016-05-17 at 10:47 -0600, Toshi Kani wrote:
> On Tue, 2016-05-17 at 08:57 -0700, Dan Williams wrote:
> > 
> > On Tue, May 17, 2016 at 8:36 AM, Yigal Korman <yigal@plexistor.com>
> > wrote:
> > > 
> > > 
> > > Hi all,
> > > I've got a couple of legacy 'type 12' NVDIMM-F modules in our lab
> > > I've been successfully testing.
> > > Recently I moved them to a new system where EFI bootloader is used
> > > (CentOS 7 installed with EFI) and  was surprised to see that they're
> > > not identified as legacy 'type 12' persistent memory, are not caught
> > > by the 'nd_pmem' driver and I don't get /dev/pmemX devices.
> > > They show as 'type 7' in the E820 map and 'Persistent memory' in
> > > /proc/iomem.
> > > Is this the intended behavior? shouldn't they be identified as legacy
> > > if they don't have NFIT?
> >
> > If the BIOS identifies it as type-7 it is broken, as far as Linux is
> > concerned, if does not also export an NFIT with an ACPI0012 device.
> > One would hope there are no new BIOS images being produced that use
> > type-12...
> On an EFI system and a kernel with CONFIG_EFI_STUB set, the kernel uses
> EFI memory table, instead of e820.  So, e820 type-12 cannot be conveyed
> to the kernel.
> 
> Since this EFI BIOS is broken as Dan pointed out, you might want to try
> booting from a kernel with CONFIG_EFI_STUB unset to see if the kernel
> gets e820 type-12.  (Note, if e820 is set to type-7, this would lead the
> kernel to overwrite data in NVDIMM due to a bug in grub.)

Thinking further, I do not think it will work.  You will need to switch to
legacy BIOS mode (assuming it has such a switch) in order to use e820.

-Toshi
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: Legacy NVDIMM-F over EFI

[Yigal Korman]

On Tue, May 17, 2016 at 7:33 PM, Dan Williams <dan.j.williams@intel.com> wrote:
>
> On Tue, May 17, 2016 at 9:30 AM, Yigal Korman <yigal@plexistor.com> wrote:
> >
> > On May 17, 2016 6:57 PM, "Dan Williams" <dan.j.williams@intel.com> wrote:
> >>
> >> On Tue, May 17, 2016 at 8:36 AM, Yigal Korman <yigal@plexistor.com> wrote:
> >> > Hi all,
> >> > I've got a couple of legacy 'type 12' NVDIMM-F modules in our lab I've
> >> > been successfully testing.
> >> > Recently I moved them to a new system where EFI bootloader is used
> >> > (CentOS 7 installed with EFI) and  was surprised to see that they're
> >> > not identified as legacy 'type 12' persistent memory, are not caught
> >> > by the 'nd_pmem' driver and I don't get /dev/pmemX devices.
> >> > They show as 'type 7' in the E820 map and 'Persistent memory' in
> >> > /proc/iomem.
> >> > Is this the intended behavior? shouldn't they be identified as legacy
> >> > if they don't have NFIT?
> >>
> >> If the BIOS identifies it as type-7 it is broken, as far as Linux is
> >> concerned, if does not also export an NFIT with an ACPI0012 device.
> >> One would hope there are no new BIOS images being produced that use
> >> type-12...
> >
> > I'm not sure I understand, what was supposed to the right thing for the BIOS
> > to do? AFAIK there's no type-12 in EFI...
>
> The BIOS should mark the region as type-7, define an ACPI0012 device,
> and publish an NFIT table.

You mean the EFI BIOS should identify the legacy NVDIMMs and create a
fake/default NFIT table to define them?

Most of the latest servers I know of don't have support for NFIT in
them yet, and people still want to use NVDIMMs and EFI on them,
perhaps it would be better to have the kernel construct a default NFIT
table if one is not given?

Thanks,
Y
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: Legacy NVDIMM-F over EFI

[Dan Williams]

On Thu, May 19, 2016 at 12:28 AM, Yigal Korman <yigal@plexistor.com> wrote:
> On Tue, May 17, 2016 at 7:33 PM, Dan Williams <dan.j.williams@intel.com> wrote:
>>
>> On Tue, May 17, 2016 at 9:30 AM, Yigal Korman <yigal@plexistor.com> wrote:
>> >
>> > On May 17, 2016 6:57 PM, "Dan Williams" <dan.j.williams@intel.com> wrote:
>> >>
>> >> On Tue, May 17, 2016 at 8:36 AM, Yigal Korman <yigal@plexistor.com> wrote:
>> >> > Hi all,
>> >> > I've got a couple of legacy 'type 12' NVDIMM-F modules in our lab I've
>> >> > been successfully testing.
>> >> > Recently I moved them to a new system where EFI bootloader is used
>> >> > (CentOS 7 installed with EFI) and  was surprised to see that they're
>> >> > not identified as legacy 'type 12' persistent memory, are not caught
>> >> > by the 'nd_pmem' driver and I don't get /dev/pmemX devices.
>> >> > They show as 'type 7' in the E820 map and 'Persistent memory' in
>> >> > /proc/iomem.
>> >> > Is this the intended behavior? shouldn't they be identified as legacy
>> >> > if they don't have NFIT?
>> >>
>> >> If the BIOS identifies it as type-7 it is broken, as far as Linux is
>> >> concerned, if does not also export an NFIT with an ACPI0012 device.
>> >> One would hope there are no new BIOS images being produced that use
>> >> type-12...
>> >
>> > I'm not sure I understand, what was supposed to the right thing for the BIOS
>> > to do? AFAIK there's no type-12 in EFI...
>>
>> The BIOS should mark the region as type-7, define an ACPI0012 device,
>> and publish an NFIT table.
>
> You mean the EFI BIOS should identify the legacy NVDIMMs and create a
> fake/default NFIT table to define them?

Yes, there's really no such thing as legacy DIMMs, only pre-ACPI 6.x
platforms.  If the ACPI implementation version is 6.x it should
implement an NFIT for an NVDIMM.

> Most of the latest servers I know of don't have support for NFIT in
> them yet, and people still want to use NVDIMMs and EFI on them,
> perhaps it would be better to have the kernel construct a default NFIT
> table if one is not given?

We have the "memmap=nn[KMG]!ss[KMG]" command line option, but I
otherwise don't see a safe way for the kernel to automatically
construct an NFIT.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: Legacy NVDIMM-F over EFI

[Dan Williams]

On Thu, May 19, 2016 at 6:54 AM, Dan Williams <dan.j.williams@intel.com> wrote:
> On Thu, May 19, 2016 at 12:28 AM, Yigal Korman <yigal@plexistor.com> wrote:
>> On Tue, May 17, 2016 at 7:33 PM, Dan Williams <dan.j.williams@intel.com> wrote:
>>>
>>> On Tue, May 17, 2016 at 9:30 AM, Yigal Korman <yigal@plexistor.com> wrote:
>>> >
>>> > On May 17, 2016 6:57 PM, "Dan Williams" <dan.j.williams@intel.com> wrote:
>>> >>
>>> >> On Tue, May 17, 2016 at 8:36 AM, Yigal Korman <yigal@plexistor.com> wrote:
>>> >> > Hi all,
>>> >> > I've got a couple of legacy 'type 12' NVDIMM-F modules in our lab I've
>>> >> > been successfully testing.
>>> >> > Recently I moved them to a new system where EFI bootloader is used
>>> >> > (CentOS 7 installed with EFI) and  was surprised to see that they're
>>> >> > not identified as legacy 'type 12' persistent memory, are not caught
>>> >> > by the 'nd_pmem' driver and I don't get /dev/pmemX devices.
>>> >> > They show as 'type 7' in the E820 map and 'Persistent memory' in
>>> >> > /proc/iomem.
>>> >> > Is this the intended behavior? shouldn't they be identified as legacy
>>> >> > if they don't have NFIT?
>>> >>
>>> >> If the BIOS identifies it as type-7 it is broken, as far as Linux is
>>> >> concerned, if does not also export an NFIT with an ACPI0012 device.
>>> >> One would hope there are no new BIOS images being produced that use
>>> >> type-12...
>>> >
>>> > I'm not sure I understand, what was supposed to the right thing for the BIOS
>>> > to do? AFAIK there's no type-12 in EFI...
>>>
>>> The BIOS should mark the region as type-7, define an ACPI0012 device,
>>> and publish an NFIT table.
>>
>> You mean the EFI BIOS should identify the legacy NVDIMMs and create a
>> fake/default NFIT table to define them?
>
> Yes, there's really no such thing as legacy DIMMs, only pre-ACPI 6.x
> platforms.  If the ACPI implementation version is 6.x it should
> implement an NFIT for an NVDIMM.
>
>> Most of the latest servers I know of don't have support for NFIT in
>> them yet, and people still want to use NVDIMMs and EFI on them,
>> perhaps it would be better to have the kernel construct a default NFIT
>> table if one is not given?
>
> We have the "memmap=nn[KMG]!ss[KMG]" command line option, but I
> otherwise don't see a safe way for the kernel to automatically
> construct an NFIT.

As far as I can see it is a small amount of BIOS enabling, we need:

1/ an ACPI0012 device defined in the DSDT to trigger the NFIT driver to load

Device (NVDR) {
    Name (_HID, "ACPI0012" /* NVDIMM Root Device */)  // _HID: Hardware ID
}

2/ an NFIT with a single "System Physical Address Range" entry.  Here
is an example program that can generate such an NFIT:

https://github.com/pmem/ndctl/blob/master/builtin-create-nfit.c
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm