[Qemu-devel] Does PCI hotplug work with IVSHMEM?

[Qemu-devel] Does PCI hotplug work with IVSHMEM?

[Ivano Cerrato]

Dear all,

we're adding dynamically an IVSHMEM device on a VM that is already 
running, but apparently this is not correctly recognized by the Guest OS.
Instead, everything works if we reboot the VM after adding the new 
IVSHMEM device.

This is the list of steps we execute:

1) Launch a new Guest VM with Qemu

2) Create a new IVSHMEM metadata file in the Host

3) Map that file as a new IVSHMEM device in the Guest
    For this step, we use the "device_add" command from Qemu:

      (qemu) device_add ivshmem,size=2048M,shm=fd:/dev/hugepages
           /rtemap_0:0x0:0x40000000:/dev/zero:0x0:0x3fffc000:/var/run
           /.dpdk_ivshmem_metadata_vm_1:0x0:0x4000

4) List the available PCI devices in the Guest with "lshw":

      $ sudo lshw
      ....
      *-memory UNCLAIMED
          description: RAM memory
          product: Virtio Inter-VM shared memory
          vendor: Red Hat, Inc
          physical id: 4
          bus info: pci@0000:00:04.0
          version: 00
          width: 64 bits
          clock: 33MHz (30.3ns)
          configuration: latency=0
          resources: memory:e0000000-e00000ff

5) Reboot the Guest VM and re-do the 'lshw' command:

      $ sudo lshw
      ...
      *-memory UNCLAIMED
         description: RAM memory
         product: Virtio Inter-VM shared memory
         vendor: Red Hat, Inc
         physical id: 4
         bus info: pci@0000:00:04.0
         version: 00
         width: 64 bits
         clock: 33MHz (30.3ns)
         configuration: latency=0
         resources: iomemory:10-f memory:febd0000-febd00ff
               memory:180000000-1ffffffff


It seems to us that, after the reboot, the IVSHMEM is mapped in a 
different way than immediately after plugging it in a running VM 
(compare the output of the line 'resources').

This has the side effect that we can use a DPDK application based on the 
IVSHMEM only in the second case; the DPDK doens't see the shared memory 
in the first case.

Is there any way to force the Guest OS to recognize the new device 
without rebooting? Such as rmmod/insmod or equivalent?

Thank you for your answer,

     Ivano

Re: [Qemu-devel] Does PCI hotplug work with IVSHMEM?

[Marc-André Lureau]

Hi

On Tue, Jul 21, 2015 at 12:13 PM, Ivano Cerrato <ivano.cerrato@polito.it> wrote:
> Is there any way to force the Guest OS to recognize the new device without
> rebooting? Such as rmmod/insmod or equivalent?

This can be observed in qemu monitor too, info qtree:

      dev: ivshmem, id ""
        chardev = ""
        size = "2048M"
        vectors = 1 (0x1)
        ioeventfd = false
        msi = true
        shm = "foo"
        role = ""
        use64 = 1 (0x1)
        addr = 05.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class RAM controller, addr 00:05.0, pci id 1af4:1110 (sub 1af4:1100)
        bar 0: mem at 0x40000000 [0x400000ff]
        bar 2: mem at 0xffffffffffffffff [0x7ffffffe]

Check dmesg, it fails to allocate for me, but with 1G it works (f22
x86_64). I don't know what's the reason behind it.

Note that it's not safe to do hotplug with ivshmem (it may assert in
various ways), I proposed a patch series a few weeks ago, I am going
to send an updated version soon.

-- 
Marc-André Lureau

Re: [Qemu-devel] Does PCI hotplug work with IVSHMEM?

[Marc-André Lureau]

Hi

On Tue, Jul 21, 2015 at 6:37 PM, Marc-André Lureau
<marcandre.lureau@gmail.com> wrote:
>
> Check dmesg, it fails to allocate for me, but with 1G it works (f22
> x86_64). I don't know what's the reason behind it.

I should have said "it failed to assign":

[   69.781673] pci 0000:00:05.0: BAR 2: no space for [mem size
0x80000000 64bit pref]
[   69.781675] pci 0000:00:05.0: BAR 2: failed to assign [mem size
0x80000000 64bit pref]


-- 
Marc-André Lureau

Re: [Qemu-devel] Does PCI hotplug work with IVSHMEM?

[Ivano Cerrato]

Thank you for the answer!
Why do you say that it is not safe to do hotplug with ivshmem?

     Ivano

Il 21/07/2015 18:37, Marc-André Lureau ha scritto:
> Hi
>
> On Tue, Jul 21, 2015 at 12:13 PM, Ivano Cerrato <ivano.cerrato@polito.it> wrote:
>> Is there any way to force the Guest OS to recognize the new device without
>> rebooting? Such as rmmod/insmod or equivalent?
> This can be observed in qemu monitor too, info qtree:
>
>        dev: ivshmem, id ""
>          chardev = ""
>          size = "2048M"
>          vectors = 1 (0x1)
>          ioeventfd = false
>          msi = true
>          shm = "foo"
>          role = ""
>          use64 = 1 (0x1)
>          addr = 05.0
>          romfile = ""
>          rombar = 1 (0x1)
>          multifunction = false
>          command_serr_enable = true
>          class RAM controller, addr 00:05.0, pci id 1af4:1110 (sub 1af4:1100)
>          bar 0: mem at 0x40000000 [0x400000ff]
>          bar 2: mem at 0xffffffffffffffff [0x7ffffffe]
>
> Check dmesg, it fails to allocate for me, but with 1G it works (f22
> x86_64). I don't know what's the reason behind it.
>
> Note that it's not safe to do hotplug with ivshmem (it may assert in
> various ways), I proposed a patch series a few weeks ago, I am going
> to send an updated version soon.
>

Re: [Qemu-devel] Does PCI hotplug work with IVSHMEM?

[Ivano Cerrato]

Hi Marc-André, all,
even with 1G hugepages, we are not able to hotplug an ivshmem device.  
The outputs we get are the following:

* In qemu monitor:

       $info qtree

       ......
       dev: ivshmem, id ""
         chardev = <null>
         size = "2048M"
         vectors = 1
         ioeventfd = off
         msi = on
         shm = "fd"
         role = <null>
         use64 = 1
         addr = 04.0
         romfile = <null>
         rombar = 1
         multifunction = off
         command_serr_enable = on
         class RAM controller, addr 00:04.0, pci id 1af4:1110 (sub 
1af4:1100)
         bar 0: mem at 0xe0000000 [0xe00000ff]
         bar 2: mem at 0xffffffffffffffff [0x7ffffffe]

* In the guest:

         $sudo lshw
         ....
         *-memory UNCLAIMED
              description: RAM memory
              product: Virtio Inter-VM shared memory
              vendor: Red Hat, Inc
              physical id: 4
              bus info: pci@0000:00:04.0
              version: 00
              width: 64 bits
              clock: 33MHz (30.3ns)
              configuration: latency=0
              resources: memory:e0000000-e00000ff


         $dmesg
         ....
         [   18.365599] pci 0000:00:04.0: [1af4:1110] type 00 class 0x050000
         [   18.365657] pci 0000:00:04.0: reg 0x10: [mem 
0x00000000-0x000000ff]
         [   18.365746] pci 0000:00:04.0: reg 0x18: [mem 
0x00000000-0x7fffffff 64bit pref]
         [   18.366002] pci 0000:00:04.0: BAR 2: can't assign mem pref 
(size 0x80000000)
         [   18.366005] pci 0000:00:04.0: BAR 0: assigned [mem 
0xe0000000-0xe00000ff]
         [   18.366739] ACPI: Error installing CMOS-RTC region handler
         [   18.366876] pci 0000:00:00.0: no hotplug settings from platform
         [   18.366877] pci 0000:00:00.0: using default PCI settings
         [   18.366900] pci 0000:00:01.0: no hotplug settings from platform
         [   18.366901] pci 0000:00:01.0: using default PCI settings
         [   18.366923] ata_piix 0000:00:01.1: no hotplug settings from 
platform
         [   18.366924] ata_piix 0000:00:01.1: using default PCI settings
         [   18.366945] piix4_smbus 0000:00:01.3: no hotplug settings 
from platform
         [   18.366946] piix4_smbus 0000:00:01.3: using default PCI settings
         [   18.366967] cirrus 0000:00:02.0: no hotplug settings from 
platform
         [   18.366968] cirrus 0000:00:02.0: using default PCI settings
         [   18.366989] 8139cp 0000:00:03.0: no hotplug settings from 
platform
         [   18.366990] 8139cp 0000:00:03.0: using default PCI settings
         [   18.367011] pci 0000:00:04.0: no hotplug settings from platform
         [   18.367012] pci 0000:00:04.0: using default PCI settings


The host is running the Linux kernel 1.13.0-43-generic, while the guest 
runs the kernel 3.12.0-rc1 .
Instead, the QEMU version is the 1.6.2 with the patch for DPDK.

Any help would be really appreciated.

Thank you in advance,

     Ivano


Il 21/07/2015 18:37, Marc-André Lureau ha scritto:
> Hi
>
> On Tue, Jul 21, 2015 at 12:13 PM, Ivano Cerrato <ivano.cerrato@polito.it> wrote:
>> Is there any way to force the Guest OS to recognize the new device without
>> rebooting? Such as rmmod/insmod or equivalent?
> This can be observed in qemu monitor too, info qtree:
>
>        dev: ivshmem, id ""
>          chardev = ""
>          size = "2048M"
>          vectors = 1 (0x1)
>          ioeventfd = false
>          msi = true
>          shm = "foo"
>          role = ""
>          use64 = 1 (0x1)
>          addr = 05.0
>          romfile = ""
>          rombar = 1 (0x1)
>          multifunction = false
>          command_serr_enable = true
>          class RAM controller, addr 00:05.0, pci id 1af4:1110 (sub 1af4:1100)
>          bar 0: mem at 0x40000000 [0x400000ff]
>          bar 2: mem at 0xffffffffffffffff [0x7ffffffe]
>
> Check dmesg, it fails to allocate for me, but with 1G it works (f22
> x86_64). I don't know what's the reason behind it.
>
> Note that it's not safe to do hotplug with ivshmem (it may assert in
> various ways), I proposed a patch series a few weeks ago, I am going
> to send an updated version soon.
>

Re: [Qemu-devel] Does PCI hotplug work with IVSHMEM?

[Marc-André Lureau]

Hi

On Tue, Aug 4, 2015 at 4:14 PM, Ivano Cerrato <ivano.cerrato@polito.it> wrote:
> even with 1G hugepages, we are not able to hotplug an ivshmem device.  The
> outputs we get are the following:

It's not about the size of hugepages, it's the size of the ivshmem
memory. It seems 1G is the limit, otherwise the guest fails to assign
the PCI BAR. I don't know the reason, I can only make guesses:
anybody?


-- 
Marc-André Lureau

Re: [Qemu-devel] Does PCI hotplug work with IVSHMEM?

[Ivano Cerrato]

We tried with 2MB hugepages, and it works fine..

     Ivano

Il 04/08/2015 16:21, Marc-André Lureau ha scritto:
> Hi
>
> On Tue, Aug 4, 2015 at 4:14 PM, Ivano Cerrato <ivano.cerrato@polito.it> wrote:
>> even with 1G hugepages, we are not able to hotplug an ivshmem device.  The
>> outputs we get are the following:
> It's not about the size of hugepages, it's the size of the ivshmem
> memory. It seems 1G is the limit, otherwise the guest fails to assign
> the PCI BAR. I don't know the reason, I can only make guesses:
> anybody?
>
>