* Re: [Qemu-devel] QEMU NVDIMM as type 7 in e820 table
2017-07-28 19:45 ` Ross Zwisler
@ 2017-07-28 20:19 ` Dan Williams
2017-07-28 20:19 ` Kani, Toshimitsu
2017-07-29 10:49 ` Haozhong Zhang
2 siblings, 0 replies; 8+ messages in thread
From: Dan Williams @ 2017-07-28 20:19 UTC (permalink / raw)
To: Ross Zwisler
Cc: Qemu Developers, linux-nvdimm@lists.01.org, Haozhong Zhang,
Michael S. Tsirkin, Stefan Hajnoczi, Xiao Guangrong,
Igor Mammedov, Linda Knippers, Kani, Toshimitsu
On Fri, Jul 28, 2017 at 12:45 PM, Ross Zwisler
<ross.zwisler@linux.intel.com> wrote:
> On Fri, Jul 28, 2017 at 11:11:10AM -0700, Dan Williams wrote:
>> On Fri, Jul 28, 2017 at 11:04 AM, Ross Zwisler
[..]
> [ Adding Linda & Toshi to see if they have an opinion. ]
>
> I guess maybe we don't need it. Yep, /proc/iomem looks good:
>
> # cat /proc/iomem
> 00000000-00000fff : Reserved
> 00001000-0009fbff : System RAM
> ...
> 100000000-23fffffff : System RAM
> 240000000-a3fffffff : Persistent Memory
> 240000000-a3fffffff : namespace0.0
>
> I was just worried that this was an inconsistency between the way that virtual
> NVDIMMs are presented vs the way that they will be presented on bare metal. I
> at least look at the e820 table to get my bearings of how memory is laid out -
> maybe I just need to look at /proc/iomem instead?
I don't think e820 matters as long as /proc/iomem ends up correct, and
the effective type is "reserved" . There's nothing in the
specification that requires the OS to validate e820 ranges vs NFIT.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Qemu-devel] QEMU NVDIMM as type 7 in e820 table
2017-07-28 19:45 ` Ross Zwisler
2017-07-28 20:19 ` Dan Williams
@ 2017-07-28 20:19 ` Kani, Toshimitsu
2017-07-29 10:49 ` Haozhong Zhang
2 siblings, 0 replies; 8+ messages in thread
From: Kani, Toshimitsu @ 2017-07-28 20:19 UTC (permalink / raw)
To: dan.j.williams@intel.com, ross.zwisler@linux.intel.com
Cc: guangrong.xiao@gmail.com, haozhong.zhang@intel.com,
mst@redhat.com, stefanha@gmail.com, linux-nvdimm@lists.01.org,
Knippers, Linda, qemu-devel@nongnu.org, imammedo@redhat.com
On Fri, 2017-07-28 at 13:45 -0600, Ross Zwisler wrote:
> On Fri, Jul 28, 2017 at 11:11:10AM -0700, Dan Williams wrote:
> > On Fri, Jul 28, 2017 at 11:04 AM, Ross Zwisler
> > <ross.zwisler@linux.intel.com> wrote:
:
> > Do you need that informationin e820? Linux effectively ignores
> > type-7. As long as the range is treated as reserved it's not clear
> > that you need the e820 entry. We also infect the persistent type
> > back into the memory map when the NFIT driver loads. /proc/iomem
> > should show the right data.
>
> [ Adding Linda & Toshi to see if they have an opinion. ]
>
> I guess maybe we don't need it. Yep, /proc/iomem looks good:
>
> # cat /proc/iomem
> 00000000-00000fff : Reserved
> 00001000-0009fbff : System RAM
> ...
> 100000000-23fffffff : System RAM
> 240000000-a3fffffff : Persistent Memory
> 240000000-a3fffffff : namespace0.0
>
> I was just worried that this was an inconsistency between the way
> that virtual NVDIMMs are presented vs the way that they will be
> presented on bare metal. I at least look at the e820 table to get my
> bearings of how memory is laid out - maybe I just need to look at
> /proc/iomem instead?
FW should present a persistent memory range in e820 or UEFI memory
descriptor table. So, it's a good practice for QEMU to do it as well.
That said, the NFIT driver inserts a persistent memory range to the
kernel IO resource table from NFIT, so we are OK without this info.
Yes, /proc/iomem shows how the resources are managed by the kernel.
Thanks,
-Toshi
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Qemu-devel] QEMU NVDIMM as type 7 in e820 table
2017-07-28 19:45 ` Ross Zwisler
2017-07-28 20:19 ` Dan Williams
2017-07-28 20:19 ` Kani, Toshimitsu
@ 2017-07-29 10:49 ` Haozhong Zhang
2017-07-31 15:48 ` Ross Zwisler
2 siblings, 1 reply; 8+ messages in thread
From: Haozhong Zhang @ 2017-07-29 10:49 UTC (permalink / raw)
To: Ross Zwisler
Cc: Dan Williams, Qemu Developers, linux-nvdimm@lists.01.org,
Michael S. Tsirkin, Stefan Hajnoczi, Xiao Guangrong,
Igor Mammedov, Linda Knippers, Kani, Toshimitsu
On 07/28/17 13:45 -0600, Ross Zwisler wrote:
> On Fri, Jul 28, 2017 at 11:11:10AM -0700, Dan Williams wrote:
> > On Fri, Jul 28, 2017 at 11:04 AM, Ross Zwisler
> > <ross.zwisler@linux.intel.com> wrote:
> > > I've been using the virtualized NVDIMM support in QEMU for testing, and I
> > > noticed that the physical addresses used by the virtual NVDIMMs aren't present
> > > in the guest's e820 table.
> > >
> > > Here is the e820 table on my QEMU instance where I have one 32 GiB virtual
> > > NVDIMM:
> > >
> > > [ 0.000000] e820: BIOS-provided physical RAM map:
> > > [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
> > > [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
> > > [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
> > > [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
> > > [ 0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
> > > [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
> > > [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
> > > [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
> > >
> > > The physical addresses used by the virtual NVDIMM are 0x240000000-0xA40000000.
> > > You can see this by looking at ndctl and the values we get from the NFIT:
> > >
> > > # ndctl list -R
> > > {
> > > "dev":"region0",
> > > "size":34359738368,
> > > "available_size":0,
> > > "type":"pmem"
> > > }
> > >
> > > # grep . /sys/bus/nd/devices/region0/{resource,size}
> > > region0/resource:0x240000000
> > > region0/size:34359738368
> > >
> > > Or you can see the same info by using iasl to dump
> > > /sys/firmware/acpi/tables/NFIT:
> > >
> > > [028h 0040 2] Subtable Type : 0000 [System Physical Address Range]
> > > [02Ah 0042 2] Length : 0038
> > >
> > > [02Ch 0044 2] Range Index : 0002
> > > [02Eh 0046 2] Flags (decoded below) : 0003
> > > Add/Online Operation Only : 1
> > > Proximity Domain Valid : 1
> > > [030h 0048 4] Reserved : 00000000
> > > [034h 0052 4] Proximity Domain : 00000000
> > > [038h 0056 16] Address Range GUID : 66F0D379-B4F3-4074-AC43-0D3318B78CDB
> > > [048h 0072 8] Address Range Base : 0000000240000000
> > > [050h 0080 8] Address Range Length : 0000000800000000
> > > [058h 0088 8] Memory Map Attribute : 0000000000008008
> > >
> > > I expected to see a type 7 region for the NVDIMM physical address range in the
> > > e820 table, so something like:
> > >
> > > [ 0.000000] e820: BIOS-provided physical RAM map:
> > > [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
> > > [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
> > > [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
> > > [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
> > > [ 0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
> > > [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
> > > [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
> > > [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
> > > [ 0.000000] BIOS-e820: [mem 0x0000000240000000-0x0000000A40000000] persistent (type 7)
> > >
> >
> > Do you need that informationin e820? Linux effectively ignores type-7.
> > As long as the range is treated as reserved it's not clear that you
> > need the e820 entry. We also infect the persistent type back into the
> > memory map when the NFIT driver loads. /proc/iomem should show the
> > right data.
>
> [ Adding Linda & Toshi to see if they have an opinion. ]
>
> I guess maybe we don't need it. Yep, /proc/iomem looks good:
>
> # cat /proc/iomem
> 00000000-00000fff : Reserved
> 00001000-0009fbff : System RAM
> ...
> 100000000-23fffffff : System RAM
> 240000000-a3fffffff : Persistent Memory
> 240000000-a3fffffff : namespace0.0
>
> I was just worried that this was an inconsistency between the way that virtual
> NVDIMMs are presented vs the way that they will be presented on bare metal. I
> at least look at the e820 table to get my bearings of how memory is laid out -
> maybe I just need to look at /proc/iomem instead?
Do any OS or applications rely on the E820 information or the
consistency between E820 and NFIT to properly work? If any, I can make
a QEMU patch to build type-7 e820 entries.
Haozhong
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [Qemu-devel] QEMU NVDIMM as type 7 in e820 table
2017-07-29 10:49 ` Haozhong Zhang
@ 2017-07-31 15:48 ` Ross Zwisler
2017-07-31 16:03 ` Igor Mammedov
0 siblings, 1 reply; 8+ messages in thread
From: Ross Zwisler @ 2017-07-31 15:48 UTC (permalink / raw)
To: Ross Zwisler, Dan Williams, Qemu Developers,
linux-nvdimm@lists.01.org, Michael S. Tsirkin, Stefan Hajnoczi,
Xiao Guangrong, Igor Mammedov, Linda Knippers, Kani, Toshimitsu
On Sat, Jul 29, 2017 at 06:49:33PM +0800, Haozhong Zhang wrote:
> On 07/28/17 13:45 -0600, Ross Zwisler wrote:
> > On Fri, Jul 28, 2017 at 11:11:10AM -0700, Dan Williams wrote:
> > > On Fri, Jul 28, 2017 at 11:04 AM, Ross Zwisler
> > > <ross.zwisler@linux.intel.com> wrote:
> > > > I've been using the virtualized NVDIMM support in QEMU for testing, and I
> > > > noticed that the physical addresses used by the virtual NVDIMMs aren't present
> > > > in the guest's e820 table.
> > > >
> > > > Here is the e820 table on my QEMU instance where I have one 32 GiB virtual
> > > > NVDIMM:
> > > >
> > > > [ 0.000000] e820: BIOS-provided physical RAM map:
> > > > [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
> > > > [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
> > > > [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
> > > > [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
> > > > [ 0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
> > > > [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
> > > > [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
> > > > [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
> > > >
> > > > The physical addresses used by the virtual NVDIMM are 0x240000000-0xA40000000.
> > > > You can see this by looking at ndctl and the values we get from the NFIT:
> > > >
> > > > # ndctl list -R
> > > > {
> > > > "dev":"region0",
> > > > "size":34359738368,
> > > > "available_size":0,
> > > > "type":"pmem"
> > > > }
> > > >
> > > > # grep . /sys/bus/nd/devices/region0/{resource,size}
> > > > region0/resource:0x240000000
> > > > region0/size:34359738368
> > > >
> > > > Or you can see the same info by using iasl to dump
> > > > /sys/firmware/acpi/tables/NFIT:
> > > >
> > > > [028h 0040 2] Subtable Type : 0000 [System Physical Address Range]
> > > > [02Ah 0042 2] Length : 0038
> > > >
> > > > [02Ch 0044 2] Range Index : 0002
> > > > [02Eh 0046 2] Flags (decoded below) : 0003
> > > > Add/Online Operation Only : 1
> > > > Proximity Domain Valid : 1
> > > > [030h 0048 4] Reserved : 00000000
> > > > [034h 0052 4] Proximity Domain : 00000000
> > > > [038h 0056 16] Address Range GUID : 66F0D379-B4F3-4074-AC43-0D3318B78CDB
> > > > [048h 0072 8] Address Range Base : 0000000240000000
> > > > [050h 0080 8] Address Range Length : 0000000800000000
> > > > [058h 0088 8] Memory Map Attribute : 0000000000008008
> > > >
> > > > I expected to see a type 7 region for the NVDIMM physical address range in the
> > > > e820 table, so something like:
> > > >
> > > > [ 0.000000] e820: BIOS-provided physical RAM map:
> > > > [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
> > > > [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
> > > > [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
> > > > [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
> > > > [ 0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
> > > > [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
> > > > [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
> > > > [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
> > > > [ 0.000000] BIOS-e820: [mem 0x0000000240000000-0x0000000A40000000] persistent (type 7)
> > > >
> > >
> > > Do you need that informationin e820? Linux effectively ignores type-7.
> > > As long as the range is treated as reserved it's not clear that you
> > > need the e820 entry. We also infect the persistent type back into the
> > > memory map when the NFIT driver loads. /proc/iomem should show the
> > > right data.
> >
> > [ Adding Linda & Toshi to see if they have an opinion. ]
> >
> > I guess maybe we don't need it. Yep, /proc/iomem looks good:
> >
> > # cat /proc/iomem
> > 00000000-00000fff : Reserved
> > 00001000-0009fbff : System RAM
> > ...
> > 100000000-23fffffff : System RAM
> > 240000000-a3fffffff : Persistent Memory
> > 240000000-a3fffffff : namespace0.0
> >
> > I was just worried that this was an inconsistency between the way that virtual
> > NVDIMMs are presented vs the way that they will be presented on bare metal. I
> > at least look at the e820 table to get my bearings of how memory is laid out -
> > maybe I just need to look at /proc/iomem instead?
>
> Do any OS or applications rely on the E820 information or the
> consistency between E820 and NFIT to properly work? If any, I can make
> a QEMU patch to build type-7 e820 entries.
I don't know of any off hand, but IMO it would be good to have this
consistency.
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [Qemu-devel] QEMU NVDIMM as type 7 in e820 table
2017-07-31 15:48 ` Ross Zwisler
@ 2017-07-31 16:03 ` Igor Mammedov
0 siblings, 0 replies; 8+ messages in thread
From: Igor Mammedov @ 2017-07-31 16:03 UTC (permalink / raw)
To: Ross Zwisler
Cc: Dan Williams, Qemu Developers, linux-nvdimm@lists.01.org,
Michael S. Tsirkin, Stefan Hajnoczi, Xiao Guangrong,
Linda Knippers, Kani, Toshimitsu
On Mon, 31 Jul 2017 09:48:08 -0600
Ross Zwisler <ross.zwisler@linux.intel.com> wrote:
> On Sat, Jul 29, 2017 at 06:49:33PM +0800, Haozhong Zhang wrote:
> > On 07/28/17 13:45 -0600, Ross Zwisler wrote:
> > > On Fri, Jul 28, 2017 at 11:11:10AM -0700, Dan Williams wrote:
> > > > On Fri, Jul 28, 2017 at 11:04 AM, Ross Zwisler
> > > > <ross.zwisler@linux.intel.com> wrote:
> > > > > I've been using the virtualized NVDIMM support in QEMU for testing, and I
> > > > > noticed that the physical addresses used by the virtual NVDIMMs aren't present
> > > > > in the guest's e820 table.
> > > > >
> > > > > Here is the e820 table on my QEMU instance where I have one 32 GiB virtual
> > > > > NVDIMM:
> > > > >
> > > > > [ 0.000000] e820: BIOS-provided physical RAM map:
> > > > > [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
> > > > > [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
> > > > > [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
> > > > > [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
> > > > > [ 0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
> > > > > [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
> > > > > [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
> > > > > [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
> > > > >
> > > > > The physical addresses used by the virtual NVDIMM are 0x240000000-0xA40000000.
> > > > > You can see this by looking at ndctl and the values we get from the NFIT:
> > > > >
> > > > > # ndctl list -R
> > > > > {
> > > > > "dev":"region0",
> > > > > "size":34359738368,
> > > > > "available_size":0,
> > > > > "type":"pmem"
> > > > > }
> > > > >
> > > > > # grep . /sys/bus/nd/devices/region0/{resource,size}
> > > > > region0/resource:0x240000000
> > > > > region0/size:34359738368
> > > > >
> > > > > Or you can see the same info by using iasl to dump
> > > > > /sys/firmware/acpi/tables/NFIT:
> > > > >
> > > > > [028h 0040 2] Subtable Type : 0000 [System Physical Address Range]
> > > > > [02Ah 0042 2] Length : 0038
> > > > >
> > > > > [02Ch 0044 2] Range Index : 0002
> > > > > [02Eh 0046 2] Flags (decoded below) : 0003
> > > > > Add/Online Operation Only : 1
> > > > > Proximity Domain Valid : 1
> > > > > [030h 0048 4] Reserved : 00000000
> > > > > [034h 0052 4] Proximity Domain : 00000000
> > > > > [038h 0056 16] Address Range GUID : 66F0D379-B4F3-4074-AC43-0D3318B78CDB
> > > > > [048h 0072 8] Address Range Base : 0000000240000000
> > > > > [050h 0080 8] Address Range Length : 0000000800000000
> > > > > [058h 0088 8] Memory Map Attribute : 0000000000008008
> > > > >
> > > > > I expected to see a type 7 region for the NVDIMM physical address range in the
> > > > > e820 table, so something like:
> > > > >
> > > > > [ 0.000000] e820: BIOS-provided physical RAM map:
> > > > > [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
> > > > > [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
> > > > > [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
> > > > > [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable
> > > > > [ 0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved
> > > > > [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
> > > > > [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
> > > > > [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
> > > > > [ 0.000000] BIOS-e820: [mem 0x0000000240000000-0x0000000A40000000] persistent (type 7)
> > > > >
> > > >
> > > > Do you need that informationin e820? Linux effectively ignores type-7.
> > > > As long as the range is treated as reserved it's not clear that you
> > > > need the e820 entry. We also infect the persistent type back into the
> > > > memory map when the NFIT driver loads. /proc/iomem should show the
> > > > right data.
> > >
> > > [ Adding Linda & Toshi to see if they have an opinion. ]
> > >
> > > I guess maybe we don't need it. Yep, /proc/iomem looks good:
> > >
> > > # cat /proc/iomem
> > > 00000000-00000fff : Reserved
> > > 00001000-0009fbff : System RAM
> > > ...
> > > 100000000-23fffffff : System RAM
> > > 240000000-a3fffffff : Persistent Memory
> > > 240000000-a3fffffff : namespace0.0
> > >
> > > I was just worried that this was an inconsistency between the way that virtual
> > > NVDIMMs are presented vs the way that they will be presented on bare metal. I
> > > at least look at the e820 table to get my bearings of how memory is laid out -
> > > maybe I just need to look at /proc/iomem instead?
> >
> > Do any OS or applications rely on the E820 information or the
> > consistency between E820 and NFIT to properly work? If any, I can make
> > a QEMU patch to build type-7 e820 entries.
>
> I don't know of any off hand, but IMO it would be good to have this
> consistency.
Maybe we should wait till there is an actual hardware that does it and OS
which uses it.
^ permalink raw reply [flat|nested] 8+ messages in thread