[Qemu-devel] Devdax backend for vnvdimm not working

[Qemu-devel] Devdax backend for vnvdimm not working

[Kamal Kakri]

Hi,

I have a namespace (backed by real nvdimm h/w) configured in devdax mode:
# ndctl list -N -n namespace0.0
[
  {
    "dev":"namespace0.0",
    "mode":"devdax",
    "map":"dev",
    "size":16909336576,
    "uuid":"e5265c0a-d902-41ce-b1d1-87a78c358aa4",
    "chardev":"dax0.0"
  }
]


When I try to configure my vm using "virsh" with something like this:
    <memory model='nvdimm' access='shared'>
      <source>
        <path>/dev/dax0.0</path>
        <align unit='KiB'>2048</align>
        <pmem/>
      </source>
      <target>
        <size unit='KiB'>2097152</size>
        <node>0</node>
      </target>
      <alias name='nvdimm0'/>
      <address type='dimm' slot='0'/>
    </memory>


I get an error while starting my vm:
# virsh start fox22vm
start: domain(optdata): fox22vm
start: found option <domain>: fox22vm
start: <domain> trying as domain NAME
error: Failed to start domain fox22vm
error: internal error: qemu unexpectedly closed the monitor: ftruncate: Invalid argument
2019-01-15T18:00:36.587467Z qemu-kvm: -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/dev/dax0.0,share=yes,size=2147483648: unable to map backing store for guest RAM: Invalid argument

I have tried to add "align unit" to my virsh xml but it does not seem to pass it onto qemu (as much as I can see from the qemu cmd line in the logs at /var/log/libvirt/qemu)


Can anyone pls hint as to what possibly could be the issue ? Is this alignment related or something else ?

Here are my versions:
# virsh --version
4.5.0
[root@fox22hv libexec]# /usr/libexec/qemu-kvm --version
QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-18.el7_6.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
# uname -a
Linux fox22hv 4.19.0-1.el7.elrepo.x86_64 #1 SMP Mon Oct 22 10:40:32 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

Re: [Qemu-devel] Devdax backend for vnvdimm not working

[Kamal Kakri]

 My devdax device is 2MB aligned so I dont think this is alignment related.# ndctl list -X -n namespace0.0
[
  {
    "dev":"namespace0.0",
    "mode":"devdax",
    "map":"dev",
    "size":16909336576,
    "uuid":"e5265c0a-d902-41ce-b1d1-87a78c358aa4",
    "daxregion":{
      "id":0,
      "size":16909336576,
      "align":2097152,  ------------------------->
      "devices":[
        {
          "chardev":"dax0.0",
          "size":16909336576
        }
      ]
    }
  }
]


I even tried to create vnvdimm of exact same size as my devdax namespace, thinking that it would not need to ftruncate in that case but I still get the same error:
# virsh start fox22vm
start: domain(optdata): fox22vm
start: found option <domain>: fox22vm
start: <domain> trying as domain NAME
error: Failed to start domain fox22vm
error: internal error: qemu unexpectedly closed the monitor: ftruncate: Invalid argument
2019-01-15T21:58:52.647221Z qemu-kvm: -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/dev/dax0.0,share=yes,size=16909336576: unable to map backing store for guest RAM: Invalid argument


Has anyone ever got devdax backed vnvdimm configured and working? 
If yes then pls let me know what libvirt/qemu/kernel version you have used to get it working.

Thanks.
    On Tuesday, January 15, 2019, 3:13:27 PM EST, Kamal Kakri <kamalkakri2017@yahoo.com> wrote:  
 
 Hi,

I have a namespace (backed by real nvdimm h/w) configured in devdax mode:
# ndctl list -N -n namespace0.0
[
  {
    "dev":"namespace0.0",
    "mode":"devdax",
    "map":"dev",
    "size":16909336576,
    "uuid":"e5265c0a-d902-41ce-b1d1-87a78c358aa4",
    "chardev":"dax0.0"
  }
]


When I try to configure my vm using "virsh" with something like this:
    <memory model='nvdimm' access='shared'>
      <source>
        <path>/dev/dax0.0</path>
        <align unit='KiB'>2048</align>
        <pmem/>
      </source>
      <target>
        <size unit='KiB'>2097152</size>
        <node>0</node>
      </target>
      <alias name='nvdimm0'/>
      <address type='dimm' slot='0'/>
    </memory>


I get an error while starting my vm:
# virsh start fox22vm
start: domain(optdata): fox22vm
start: found option <domain>: fox22vm
start: <domain> trying as domain NAME
error: Failed to start domain fox22vm
error: internal error: qemu unexpectedly closed the monitor: ftruncate: Invalid argument
2019-01-15T18:00:36.587467Z qemu-kvm: -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/dev/dax0.0,share=yes,size=2147483648: unable to map backing store for guest RAM: Invalid argument

I have tried to add "align unit" to my virsh xml but it does not seem to pass it onto qemu (as much as I can see from the qemu cmd line in the logs at /var/log/libvirt/qemu)


Can anyone pls hint as to what possibly could be the issue ? Is this alignment related or something else ?

Here are my versions:
# virsh --version
4.5.0
[root@fox22hv libexec]# /usr/libexec/qemu-kvm --version
QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-18.el7_6.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
# uname -a
Linux fox22hv 4.19.0-1.el7.elrepo.x86_64 #1 SMP Mon Oct 22 10:40:32 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux




  

Re: [Qemu-devel] Devdax backend for vnvdimm not working

[Yi Zhang]

On 2019-01-15 at 22:51:09 +0000, Kamal Kakri via Qemu-devel wrote:
>  My devdax device is 2MB aligned so I dont think this is alignment related.# ndctl list -X -n namespace0.0
> [
>   {
>     "dev":"namespace0.0",
>     "mode":"devdax",
>     "map":"dev",
>     "size":16909336576,
>     "uuid":"e5265c0a-d902-41ce-b1d1-87a78c358aa4",
>     "daxregion":{
>       "id":0,
>       "size":16909336576,
>       "align":2097152,  ------------------------->
>       "devices":[
>         {
>           "chardev":"dax0.0",
>           "size":16909336576
>         }
>       ]
>     }
>   }
> ]
> 
> 
> I even tried to create vnvdimm of exact same size as my devdax namespace, thinking that it would not need to ftruncate in that case but I still get the same error:
> # virsh start fox22vm
> start: domain(optdata): fox22vm
> start: found option <domain>: fox22vm
> start: <domain> trying as domain NAME
> error: Failed to start domain fox22vm
> error: internal error: qemu unexpectedly closed the monitor: ftruncate: Invalid argument
> 2019-01-15T21:58:52.647221Z qemu-kvm: -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/dev/dax0.0,share=yes,size=16909336576: unable to map backing store for guest RAM: Invalid argument
> 
> 
> Has anyone ever got devdax backed vnvdimm configured and working? 
> If yes then pls let me know what libvirt/qemu/kernel version you have used to get it working.
> 
> Thanks.
>     On Tuesday, January 15, 2019, 3:13:27 PM EST, Kamal Kakri <kamalkakri2017@yahoo.com> wrote:  
>  
>  Hi,
> 
> I have a namespace (backed by real nvdimm h/w) configured in devdax mode:
> # ndctl list -N -n namespace0.0
> [
>   {
>     "dev":"namespace0.0",
>     "mode":"devdax",
>     "map":"dev",
>     "size":16909336576,
>     "uuid":"e5265c0a-d902-41ce-b1d1-87a78c358aa4",
>     "chardev":"dax0.0"
>   }
> ]
> 
> 
> When I try to configure my vm using "virsh" with something like this:
>     <memory model='nvdimm' access='shared'>
>       <source>
>         <path>/dev/dax0.0</path>
>         <align unit='KiB'>2048</align>
this should be alignsize, Add Luyao.
>         <pmem/>
>       </source>
>       <target>
>         <size unit='KiB'>2097152</size>
>         <node>0</node>
>       </target>
>       <alias name='nvdimm0'/>
>       <address type='dimm' slot='0'/>
>     </memory>
> 
> 
> I get an error while starting my vm:
> # virsh start fox22vm
> start: domain(optdata): fox22vm
> start: found option <domain>: fox22vm
> start: <domain> trying as domain NAME
> error: Failed to start domain fox22vm
> error: internal error: qemu unexpectedly closed the monitor: ftruncate: Invalid argument
> 2019-01-15T18:00:36.587467Z qemu-kvm: -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/dev/dax0.0,share=yes,size=2147483648: unable to map backing store for guest RAM: Invalid argument
> 
> I have tried to add "align unit" to my virsh xml but it does not seem to pass it onto qemu (as much as I can see from the qemu cmd line in the logs at /var/log/libvirt/qemu)
> 
> 
> Can anyone pls hint as to what possibly could be the issue ? Is this alignment related or something else ?
> 
> Here are my versions:
> # virsh --version
> 4.5.0
> [root@fox22hv libexec]# /usr/libexec/qemu-kvm --version
> QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-18.el7_6.1)
> Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
> # uname -a
> Linux fox22hv 4.19.0-1.el7.elrepo.x86_64 #1 SMP Mon Oct 22 10:40:32 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> 
> 
>   

Re: [Qemu-devel] Devdax backend for vnvdimm not working

[Wei Yang]

On Tue, Jan 15, 2019 at 08:13:27PM +0000, Kamal Kakri via Qemu-devel wrote:
>Hi,
>
>I have a namespace (backed by real nvdimm h/w) configured in devdax mode:
># ndctl list -N -n namespace0.0
>[
>  {
>    "dev":"namespace0.0",
>    "mode":"devdax",
>    "map":"dev",
>    "size":16909336576,
>    "uuid":"e5265c0a-d902-41ce-b1d1-87a78c358aa4",
>    "chardev":"dax0.0"
>  }
>]
>

Above is your host configuration, right?

>
>When I try to configure my vm using "virsh" with something like this:
>    <memory model='nvdimm' access='shared'>
>      <source>
>        <path>/dev/dax0.0</path>
>        <align unit='KiB'>2048</align>
>        <pmem/>
>      </source>
>      <target>
>        <size unit='KiB'>2097152</size>
>        <node>0</node>
>      </target>
>      <alias name='nvdimm0'/>
>      <address type='dimm' slot='0'/>
>    </memory>
>

I am not familiar with virsh, while I guess align here is the parameter passed
to qemu.

Hmm...  as I being told, the alignment should be 128M.

>
>I get an error while starting my vm:
># virsh start fox22vm
>start: domain(optdata): fox22vm
>start: found option <domain>: fox22vm
>start: <domain> trying as domain NAME
>error: Failed to start domain fox22vm
>error: internal error: qemu unexpectedly closed the monitor: ftruncate: Invalid argument
>2019-01-15T18:00:36.587467Z qemu-kvm: -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/dev/dax0.0,share=yes,size=2147483648: unable to map backing store for guest RAM: Invalid argument

The error message here shows the error is caused by mmap for an invalid
argument. Hmm... but I am not sure which one is incorrect.


Usually I run qemu directly, here is a qemu command line which works at my
side:

qemu/x86_64-softmmu/qemu-system-x86_64 -machine pc,nvdimm \
	-m 6G,slots=4,maxmem=128G -smp 8 --enable-kvm \
	-drive file=/home/richard/guest/fedora29.img,format=raw -nographic \
	-drive file=/home/richard/guest/project.img,format=raw \
	-object memory-backend-file,id=mem1,share=on,mem-path=/dev/dax0.0,size=10G,align=128M \
	-device nvdimm,id=nvdimm1,memdev=mem1,label-size=128k

Maybe you can compare the difference between your virsh result to see the
difference.

>
>I have tried to add "align unit" to my virsh xml but it does not seem to pass it onto qemu (as much as I can see from the qemu cmd line in the logs at /var/log/libvirt/qemu)
>
>
>Can anyone pls hint as to what possibly could be the issue ? Is this alignment related or something else ?
>
>Here are my versions:
># virsh --version
>4.5.0
>[root@fox22hv libexec]# /usr/libexec/qemu-kvm --version
>QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-18.el7_6.1)
>Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
># uname -a
>Linux fox22hv 4.19.0-1.el7.elrepo.x86_64 #1 SMP Mon Oct 22 10:40:32 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
>
>
>

-- 
Wei Yang
Help you, Help me

Re: [Qemu-devel] Devdax backend for vnvdimm not working

[Yi Zhang]

On 2019-01-16 at 09:12:33 +0800, Wei Yang wrote:
> On Tue, Jan 15, 2019 at 08:13:27PM +0000, Kamal Kakri via Qemu-devel wrote:
> >Hi,
> >
> >I have a namespace (backed by real nvdimm h/w) configured in devdax mode:
> ># ndctl list -N -n namespace0.0
> >[
> >  {
> >    "dev":"namespace0.0",
> >    "mode":"devdax",
> >    "map":"dev",
> >    "size":16909336576,
> >    "uuid":"e5265c0a-d902-41ce-b1d1-87a78c358aa4",
> >    "chardev":"dax0.0"
> >  }
> >]
> >
> 
> Above is your host configuration, right?
> 
> >
> >When I try to configure my vm using "virsh" with something like this:
> >    <memory model='nvdimm' access='shared'>
> >      <source>
> >        <path>/dev/dax0.0</path>
> >        <align unit='KiB'>2048</align>
so that will apply the align=2048K for memory-backend-file?
> >        <pmem/>
> >      </source>
> >      <target>
> >        <size unit='KiB'>2097152</size>
> >        <node>0</node>
> >      </target>
> >      <alias name='nvdimm0'/>
> >      <address type='dimm' slot='0'/>
> >    </memory>
> >
> 
> I am not familiar with virsh, while I guess align here is the parameter passed
> to qemu.
> 
> Hmm...  as I being told, the alignment should be 128M.
2MB align is OK for this case.
> 
> >
> >I get an error while starting my vm:
> ># virsh start fox22vm
> >start: domain(optdata): fox22vm
> >start: found option <domain>: fox22vm
> >start: <domain> trying as domain NAME
> >error: Failed to start domain fox22vm
> >error: internal error: qemu unexpectedly closed the monitor: ftruncate: Invalid argument
> >2019-01-15T18:00:36.587467Z qemu-kvm: -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/dev/dax0.0,share=yes,size=2147483648: unable to map backing store for guest RAM: Invalid argument
I didn't see the algin=2M option is applied here, which is must and minimal
granularity of devdax backend, is anything I missed?
> 
> The error message here shows the error is caused by mmap for an invalid
> argument. Hmm... but I am not sure which one is incorrect.
> 
> 
> Usually I run qemu directly, here is a qemu command line which works at my
> side:
> 
> qemu/x86_64-softmmu/qemu-system-x86_64 -machine pc,nvdimm \
> 	-m 6G,slots=4,maxmem=128G -smp 8 --enable-kvm \
> 	-drive file=/home/richard/guest/fedora29.img,format=raw -nographic \
> 	-drive file=/home/richard/guest/project.img,format=raw \
> 	-object memory-backend-file,id=mem1,share=on,mem-path=/dev/dax0.0,size=10G,align=128M \
> 	-device nvdimm,id=nvdimm1,memdev=mem1,label-size=128k
> 
> Maybe you can compare the difference between your virsh result to see the
> difference.
> 
> >
> >I have tried to add "align unit" to my virsh xml but it does not seem to pass it onto qemu (as much as I can see from the qemu cmd line in the logs at /var/log/libvirt/qemu)
> >
> >
> >Can anyone pls hint as to what possibly could be the issue ? Is this alignment related or something else ?
yes the alignment.
> >
> >Here are my versions:
> ># virsh --version
> >4.5.0
> >[root@fox22hv libexec]# /usr/libexec/qemu-kvm --version
> >QEMU emulator version 2.12.0 (qemu-kvm-rhev-2.12.0-18.el7_6.1)
> >Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
> ># uname -a
> >Linux fox22hv 4.19.0-1.el7.elrepo.x86_64 #1 SMP Mon Oct 22 10:40:32 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
> >
> >
> >
> 
> -- 
> Wei Yang
> Help you, Help me
> 

Re: [Qemu-devel] Devdax backend for vnvdimm not working

[Wei Yang]

On Wed, Jan 16, 2019 at 10:18:05AM +0800, Yi Zhang wrote:
>On 2019-01-16 at 09:12:33 +0800, Wei Yang wrote:
>> On Tue, Jan 15, 2019 at 08:13:27PM +0000, Kamal Kakri via Qemu-devel wrote:
>> >Hi,
>> >
>> >I have a namespace (backed by real nvdimm h/w) configured in devdax mode:
>> ># ndctl list -N -n namespace0.0
>> >[
>> >  {
>> >    "dev":"namespace0.0",
>> >    "mode":"devdax",
>> >    "map":"dev",
>> >    "size":16909336576,
>> >    "uuid":"e5265c0a-d902-41ce-b1d1-87a78c358aa4",
>> >    "chardev":"dax0.0"
>> >  }
>> >]
>> >
>> 
>> Above is your host configuration, right?
>> 
>> >
>> >When I try to configure my vm using "virsh" with something like this:
>> >    <memory model='nvdimm' access='shared'>
>> >      <source>
>> >        <path>/dev/dax0.0</path>
>> >        <align unit='KiB'>2048</align>
>so that will apply the align=2048K for memory-backend-file?
>> >        <pmem/>
>> >      </source>
>> >      <target>
>> >        <size unit='KiB'>2097152</size>
>> >        <node>0</node>
>> >      </target>
>> >      <alias name='nvdimm0'/>
>> >      <address type='dimm' slot='0'/>
>> >    </memory>
>> >
>> 
>> I am not familiar with virsh, while I guess align here is the parameter passed
>> to qemu.
>> 
>> Hmm...  as I being told, the alignment should be 128M.
>2MB align is OK for this case.

Ok, so would you mind putting more word on the alignment?

Which case requires 128M and which requires 2M?

I got lost here.

-- 
Wei Yang
Help you, Help me