[Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Gonglei (Arei)]

Hi,

I noticed that the qemu-2.1 release change log says
" PCIe: Basic hot-plug/hot-unplug support for Q35 machine."

And then I made a testing for the hotplugging function of Q35.
But I'm failed, and I got the dmesg log in guest os as below:

[ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on Slot (0 - 4)
[ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot (0 - 4)
[ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 - powering on due to button press.
[ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link status
[ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on Slot (0 - 4)
[ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on Slot (0 - 4)

Steps of testing:

#1. QEMU version:

	The lateset master tree source.

#2. Command line:

./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device ide-drive,bus=ide.2,drive=MacHDD \ 
 -drive id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
QEMU 2.0.93 monitor - type 'help' for more information
(qemu) device_add virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0
(qemu) info network
hub 0
 \ user.0: index=0,type=user,net=10.0.2.0,restrict=off
 \ e1000.0: index=0,type=nic,model=e1000,macaddr=52:54:00:12:34:56

#3. Guest OS:

	Both rhel-server-7.0-x86_64 and SLES-11-SP3-DVD-x86_64-GM.

#4. Content of ../docs/q35-chipset.cfg:

[snip]

##
# Example PCIe switch with two downstream ports
#
[device "pcie-switch-upstream-port-1"]
  driver = "x3130-upstream"
  bus = "ich9-pcie-port-4"
  addr = "00.0"

[device "pcie-switch-downstream-port-1-1"]
  driver = "xio3130-downstream"
  multifunction = "on"
  bus = "pcie-switch-upstream-port-1"
  addr = "00.0"
  port = "1"
  chassis = "5"

[device "pcie-switch-downstream-port-1-2"]
  driver = "xio3130-downstream"
  multifunction = "on"
  bus = "pcie-switch-upstream-port-1"
  addr = "00.1"
  port = "1"
  chassis = "6"

[device "ich9-ehci-1"]
  driver = "ich9-usb-ehci1"
  multifunction = "on"
  bus = "pcie.0"
  addr = "1d.7"

[snip]

BTW, except testing hotplug of downstream ports, I also test the root port
with "device_add virtio-net-pci,id=nic2,bus=ich9-pcie-port-1,addr=1.0", but
I got the same result.

Am I wrong in some places? Or a QEMU bug?  Thanks for any help.

Best regards,
-Gonglei

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Michael S. Tsirkin]

On Fri, Aug 15, 2014 at 07:33:29AM +0000, Gonglei (Arei) wrote:
> Hi,
> 
> I noticed that the qemu-2.1 release change log says
> " PCIe: Basic hot-plug/hot-unplug support for Q35 machine."
> And then I made a testing for the hotplugging function of Q35.
> But I'm failed, and I got the dmesg log in guest os as below:
> 
> [ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on Slot (0 - 4)
> [ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot (0 - 4)
> [ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 - powering on due to button press.
> [ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link status
> [ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on Slot (0 - 4)
> [ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on Slot (0 - 4)
> 
> Steps of testing:
> 
> #1. QEMU version:
> 
> 	The lateset master tree source.
> 
> #2. Command line:
> 
> ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device ide-drive,bus=ide.2,drive=MacHDD \ 
>  -drive id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
> QEMU 2.0.93 monitor - type 'help' for more information
> (qemu) device_add virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0

I don't think you can use any slot except slot 0 for pci express.
> (qemu) info network
> hub 0
>  \ user.0: index=0,type=user,net=10.0.2.0,restrict=off
>  \ e1000.0: index=0,type=nic,model=e1000,macaddr=52:54:00:12:34:56
> 
> #3. Guest OS:
> 
> 	Both rhel-server-7.0-x86_64 and SLES-11-SP3-DVD-x86_64-GM.
> 
> #4. Content of ../docs/q35-chipset.cfg:
> 
> [snip]
> 
> ##
> # Example PCIe switch with two downstream ports
> #
> [device "pcie-switch-upstream-port-1"]
>   driver = "x3130-upstream"
>   bus = "ich9-pcie-port-4"
>   addr = "00.0"
> 
> [device "pcie-switch-downstream-port-1-1"]
>   driver = "xio3130-downstream"
>   multifunction = "on"
>   bus = "pcie-switch-upstream-port-1"
>   addr = "00.0"
>   port = "1"
>   chassis = "5"
> 
> [device "pcie-switch-downstream-port-1-2"]
>   driver = "xio3130-downstream"
>   multifunction = "on"
>   bus = "pcie-switch-upstream-port-1"
>   addr = "00.1"
>   port = "1"
>   chassis = "6"
> 
> [device "ich9-ehci-1"]
>   driver = "ich9-usb-ehci1"
>   multifunction = "on"
>   bus = "pcie.0"
>   addr = "1d.7"
> 
> [snip]
> 
> BTW, except testing hotplug of downstream ports, I also test the root port
> with "device_add virtio-net-pci,id=nic2,bus=ich9-pcie-port-1,addr=1.0", but
> I got the same result.

Same thing.

> Am I wrong in some places? Or a QEMU bug?  Thanks for any help.
> 
> Best regards,
> -Gonglei
> 

We might want to add a check for this, to produce a
better error message.

-- 
MST

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Marcel Apfelbaum]

On Sun, 2014-08-17 at 13:00 +0200, Michael S. Tsirkin wrote:
> On Fri, Aug 15, 2014 at 07:33:29AM +0000, Gonglei (Arei) wrote:
> > Hi,
> > 
> > I noticed that the qemu-2.1 release change log says
> > " PCIe: Basic hot-plug/hot-unplug support for Q35 machine."
> > And then I made a testing for the hotplugging function of Q35.
> > But I'm failed, and I got the dmesg log in guest os as below:
> > 
> > [ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on Slot (0 - 4)
> > [ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot (0 - 4)
> > [ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 - powering on due to button press.
> > [ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link status
> > [ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on Slot (0 - 4)
> > [ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on Slot (0 - 4)
> > 
> > Steps of testing:
> > 
> > #1. QEMU version:
> > 
> > 	The lateset master tree source.
> > 
> > #2. Command line:
> > 
> > ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device ide-drive,bus=ide.2,drive=MacHDD \ 
> >  -drive id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
> > QEMU 2.0.93 monitor - type 'help' for more information
> > (qemu) device_add virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0
> 
> I don't think you can use any slot except slot 0 for pci express.
Indeed, only slot 0 can be used, try the same without specifying the addr property.

> > (qemu) info network
> > hub 0
> >  \ user.0: index=0,type=user,net=10.0.2.0,restrict=off
> >  \ e1000.0: index=0,type=nic,model=e1000,macaddr=52:54:00:12:34:56
> > 
> > #3. Guest OS:
> > 
> > 	Both rhel-server-7.0-x86_64 and SLES-11-SP3-DVD-x86_64-GM.
> > 
> > #4. Content of ../docs/q35-chipset.cfg:
> > 
> > [snip]
> > 
> > ##
> > # Example PCIe switch with two downstream ports
> > #
> > [device "pcie-switch-upstream-port-1"]
> >   driver = "x3130-upstream"
> >   bus = "ich9-pcie-port-4"
> >   addr = "00.0"
> > 
> > [device "pcie-switch-downstream-port-1-1"]
> >   driver = "xio3130-downstream"
> >   multifunction = "on"
> >   bus = "pcie-switch-upstream-port-1"
> >   addr = "00.0"
> >   port = "1"
> >   chassis = "5"
> > 
> > [device "pcie-switch-downstream-port-1-2"]
> >   driver = "xio3130-downstream"
> >   multifunction = "on"
> >   bus = "pcie-switch-upstream-port-1"
> >   addr = "00.1"
> >   port = "1"
> >   chassis = "6"
> > 
> > [device "ich9-ehci-1"]
> >   driver = "ich9-usb-ehci1"
> >   multifunction = "on"
> >   bus = "pcie.0"
> >   addr = "1d.7"
> > 
> > [snip]
> > 
> > BTW, except testing hotplug of downstream ports, I also test the root port
> > with "device_add virtio-net-pci,id=nic2,bus=ich9-pcie-port-1,addr=1.0", but
> > I got the same result.
> 
> Same thing.
Agreed, and also make sure that ich9-pcie-port-1 is a root port's bus and not
an upstream port's bus.

Thanks,
Marcel

> 
> > Am I wrong in some places? Or a QEMU bug?  Thanks for any help.
> > 
> > Best regards,
> > -Gonglei
> > 
> 
> We might want to add a check for this, to produce a
> better error message.
> 

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Gonglei (Arei)]

Hi,

> Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> 
> On Sun, 2014-08-17 at 13:00 +0200, Michael S. Tsirkin wrote:
> > On Fri, Aug 15, 2014 at 07:33:29AM +0000, Gonglei (Arei) wrote:
> > > Hi,
> > >
> > > I noticed that the qemu-2.1 release change log says
> > > " PCIe: Basic hot-plug/hot-unplug support for Q35 machine."
> > > And then I made a testing for the hotplugging function of Q35.
> > > But I'm failed, and I got the dmesg log in guest os as below:
> > >
> > > [ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on Slot (0 - 4)
> > > [ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot (0 - 4)
> > > [ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 - powering on due
> to button press.
> > > [ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link status
> > > [ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on Slot (0 - 4)
> > > [ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on Slot (0 - 4)
> > >
> > > Steps of testing:
> > >
> > > #1. QEMU version:
> > >
> > > 	The lateset master tree source.
> > >
> > > #2. Command line:
> > >
> > > ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device
> ide-drive,bus=ide.2,drive=MacHDD \
> > >  -drive id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img
> -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
> > > QEMU 2.0.93 monitor - type 'help' for more information
> > > (qemu) device_add
> virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0
> >
> > I don't think you can use any slot except slot 0 for pci express.

OK. Does the PCIe specification say that?
I appreciate very much that you explain more.

> Indeed, only slot 0 can be used, try the same without specifying the addr
> property.
> 

OK, got it. Thank you so much.

> > > (qemu) info network
> > > hub 0
> > >  \ user.0: index=0,type=user,net=10.0.2.0,restrict=off
> > >  \ e1000.0: index=0,type=nic,model=e1000,macaddr=52:54:00:12:34:56
> > >
> > > #3. Guest OS:
> > >
> > > 	Both rhel-server-7.0-x86_64 and SLES-11-SP3-DVD-x86_64-GM.
> > >
> > > #4. Content of ../docs/q35-chipset.cfg:
> > >
> > > [snip]
> > >
> > > ##
> > > # Example PCIe switch with two downstream ports
> > > #
> > > [device "pcie-switch-upstream-port-1"]
> > >   driver = "x3130-upstream"
> > >   bus = "ich9-pcie-port-4"
> > >   addr = "00.0"
> > >
> > > [device "pcie-switch-downstream-port-1-1"]
> > >   driver = "xio3130-downstream"
> > >   multifunction = "on"
> > >   bus = "pcie-switch-upstream-port-1"
> > >   addr = "00.0"
> > >   port = "1"
> > >   chassis = "5"
> > >
> > > [device "pcie-switch-downstream-port-1-2"]
> > >   driver = "xio3130-downstream"
> > >   multifunction = "on"
> > >   bus = "pcie-switch-upstream-port-1"
> > >   addr = "00.1"
> > >   port = "1"
> > >   chassis = "6"
> > >
> > > [device "ich9-ehci-1"]
> > >   driver = "ich9-usb-ehci1"
> > >   multifunction = "on"
> > >   bus = "pcie.0"
> > >   addr = "1d.7"
> > >
> > > [snip]
> > >
> > > BTW, except testing hotplug of downstream ports, I also test the root port
> > > with "device_add virtio-net-pci,id=nic2,bus=ich9-pcie-port-1,addr=1.0", but
> > > I got the same result.
> >
> > Same thing.
> Agreed, and also make sure that ich9-pcie-port-1 is a root port's bus and not
> an upstream port's bus.
> 
Yes, it is.

> Thanks,
> Marcel
> 
> >
> > > Am I wrong in some places? Or a QEMU bug?  Thanks for any help.
> > >
> > > Best regards,
> > > -Gonglei
> > >
> >
> > We might want to add a check for this, to produce a
> > better error message.
> >
> 
Work in progress. Thanks!

I also find other questions. Will post in a patch series.

Best regards,
-Gonglei

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Paolo Bonzini]

Il 18/08/2014 11:12, Gonglei (Arei) ha scritto:
> Hi,
> 
>> Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
>>
>> On Sun, 2014-08-17 at 13:00 +0200, Michael S. Tsirkin wrote:
>>> On Fri, Aug 15, 2014 at 07:33:29AM +0000, Gonglei (Arei) wrote:
>>>> Hi,
>>>>
>>>> I noticed that the qemu-2.1 release change log says
>>>> " PCIe: Basic hot-plug/hot-unplug support for Q35 machine."
>>>> And then I made a testing for the hotplugging function of Q35.
>>>> But I'm failed, and I got the dmesg log in guest os as below:
>>>>
>>>> [ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on Slot (0 - 4)
>>>> [ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot (0 - 4)
>>>> [ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 - powering on due
>> to button press.
>>>> [ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link status
>>>> [ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on Slot (0 - 4)
>>>> [ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on Slot (0 - 4)
>>>>
>>>> Steps of testing:
>>>>
>>>> #1. QEMU version:
>>>>
>>>> 	The lateset master tree source.
>>>>
>>>> #2. Command line:
>>>>
>>>> ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device
>> ide-drive,bus=ide.2,drive=MacHDD \
>>>>  -drive id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img
>> -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
>>>> QEMU 2.0.93 monitor - type 'help' for more information
>>>> (qemu) device_add
>> virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0
>>>
>>> I don't think you can use any slot except slot 0 for pci express.
> 
> OK. Does the PCIe specification say that?
> I appreciate very much that you explain more.

The closest I could find is in "7.3. Configuration Transaction
Rules"/"7.3.1. Device Number":

With non-ARI Devices, PCI Express components are restricted to
implementing a single Device Number on their primary interface (Upstream
Port) [...] Downstream Ports that do not have ARI Forwarding enabled
must associate only Device 0 with the device attached to the Logical Bus
representing the Link from the Port. Configuration Requests
targeting the Bus Number associated with a Link specifying Device Number
0 are delivered to the device attached to the Link; Configuration
Requests specifying all other Device Numbers (1-31)
must be terminated by the Switch Downstream Port or the Root Port with
an Unsupported Request Completion Status (equivalent to Master Abort in
PCI).

Paolo

>> Indeed, only slot 0 can be used, try the same without specifying the addr
>> property.
>>
> 
> OK, got it. Thank you so much.
> 
>>>> (qemu) info network
>>>> hub 0
>>>>  \ user.0: index=0,type=user,net=10.0.2.0,restrict=off
>>>>  \ e1000.0: index=0,type=nic,model=e1000,macaddr=52:54:00:12:34:56
>>>>
>>>> #3. Guest OS:
>>>>
>>>> 	Both rhel-server-7.0-x86_64 and SLES-11-SP3-DVD-x86_64-GM.
>>>>
>>>> #4. Content of ../docs/q35-chipset.cfg:
>>>>
>>>> [snip]
>>>>
>>>> ##
>>>> # Example PCIe switch with two downstream ports
>>>> #
>>>> [device "pcie-switch-upstream-port-1"]
>>>>   driver = "x3130-upstream"
>>>>   bus = "ich9-pcie-port-4"
>>>>   addr = "00.0"
>>>>
>>>> [device "pcie-switch-downstream-port-1-1"]
>>>>   driver = "xio3130-downstream"
>>>>   multifunction = "on"
>>>>   bus = "pcie-switch-upstream-port-1"
>>>>   addr = "00.0"
>>>>   port = "1"
>>>>   chassis = "5"
>>>>
>>>> [device "pcie-switch-downstream-port-1-2"]
>>>>   driver = "xio3130-downstream"
>>>>   multifunction = "on"
>>>>   bus = "pcie-switch-upstream-port-1"
>>>>   addr = "00.1"
>>>>   port = "1"
>>>>   chassis = "6"
>>>>
>>>> [device "ich9-ehci-1"]
>>>>   driver = "ich9-usb-ehci1"
>>>>   multifunction = "on"
>>>>   bus = "pcie.0"
>>>>   addr = "1d.7"
>>>>
>>>> [snip]
>>>>
>>>> BTW, except testing hotplug of downstream ports, I also test the root port
>>>> with "device_add virtio-net-pci,id=nic2,bus=ich9-pcie-port-1,addr=1.0", but
>>>> I got the same result.
>>>
>>> Same thing.
>> Agreed, and also make sure that ich9-pcie-port-1 is a root port's bus and not
>> an upstream port's bus.
>>
> Yes, it is.
> 
>> Thanks,
>> Marcel
>>
>>>
>>>> Am I wrong in some places? Or a QEMU bug?  Thanks for any help.
>>>>
>>>> Best regards,
>>>> -Gonglei
>>>>
>>>
>>> We might want to add a check for this, to produce a
>>> better error message.
>>>
>>
> Work in progress. Thanks!
> 
> I also find other questions. Will post in a patch series.
> 
> Best regards,
> -Gonglei
> 

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Gonglei (Arei)]

> >> Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> >>
> >> On Sun, 2014-08-17 at 13:00 +0200, Michael S. Tsirkin wrote:
> >>> On Fri, Aug 15, 2014 at 07:33:29AM +0000, Gonglei (Arei) wrote:
> >>>> Hi,
> >>>>
> >>>> I noticed that the qemu-2.1 release change log says
> >>>> " PCIe: Basic hot-plug/hot-unplug support for Q35 machine."
> >>>> And then I made a testing for the hotplugging function of Q35.
> >>>> But I'm failed, and I got the dmesg log in guest os as below:
> >>>>
> >>>> [ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on Slot (0 - 4)
> >>>> [ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot (0 - 4)
> >>>> [ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 - powering on
> due
> >> to button press.
> >>>> [ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link status
> >>>> [ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on Slot (0 - 4)
> >>>> [ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on Slot (0 - 4)
> >>>>
> >>>> Steps of testing:
> >>>>
> >>>> #1. QEMU version:
> >>>>
> >>>> 	The lateset master tree source.
> >>>>
> >>>> #2. Command line:
> >>>>
> >>>> ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device
> >> ide-drive,bus=ide.2,drive=MacHDD \
> >>>>  -drive
> id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img
> >> -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
> >>>> QEMU 2.0.93 monitor - type 'help' for more information
> >>>> (qemu) device_add
> >> virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0
> >>>
> >>> I don't think you can use any slot except slot 0 for pci express.
> >
> > OK. Does the PCIe specification say that?
> > I appreciate very much that you explain more.
> 
> The closest I could find is in "7.3. Configuration Transaction
> Rules"/"7.3.1. Device Number":
> 
> With non-ARI Devices, PCI Express components are restricted to
> implementing a single Device Number on their primary interface (Upstream
> Port) [...] Downstream Ports that do not have ARI Forwarding enabled
> must associate only Device 0 with the device attached to the Logical Bus
> representing the Link from the Port. Configuration Requests
> targeting the Bus Number associated with a Link specifying Device Number
> 0 are delivered to the device attached to the Link; Configuration
> Requests specifying all other Device Numbers (1-31)
> must be terminated by the Switch Downstream Port or the Root Port with
> an Unsupported Request Completion Status (equivalent to Master Abort in
> PCI).
> 
Thanks a lot, Paolo.

And I found another issue when cold-plugging don't using slot 0, the PCIe device also can't
be searched in guest os. 

So, I have some questions and ideas:

1. Does qemu support ARI Forwarding for PCIe at present? If yes, how to enable it ?
2. If not, we should add some check for PCIe root ports and downstream ports,
 meanwhile add explaining document.
3. Those check should add in general code level, both hotplug and coldplug.

Am I right?  Thanks.

Best regards,
-Gonglei

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Paolo Bonzini]

Il 19/08/2014 08:25, Gonglei (Arei) ha scritto:
> 
> 1. Does qemu support ARI Forwarding for PCIe at present? If yes, how to enable it ?

No, not yet.

> 2. If not, we should add some check for PCIe root ports and downstream ports,
>  meanwhile add explaining document.
> 3. Those check should add in general code level, both hotplug and coldplug.

That would be possible, yes.

Paolo

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Gonglei (Arei)]

Hi,

> -----Original Message-----
> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> Sent: Tuesday, August 19, 2014 4:06 PM
> To: Gonglei (Arei); Marcel Apfelbaum; Michael S. Tsirkin
> Cc: qemu-devel@nongnu.org; imammedo@redhat.com; armbru@redhat.com;
> Huangweidong (C)
> Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> 
> Il 19/08/2014 08:25, Gonglei (Arei) ha scritto:
> >
> > 1. Does qemu support ARI Forwarding for PCIe at present? If yes, how to
> enable it ?
> 
> No, not yet.
> 
> > 2. If not, we should add some check for PCIe root ports and downstream
> ports,
> >  meanwhile add explaining document.
> > 3. Those check should add in general code level, both hotplug and coldplug.
> 
> That would be possible, yes.
> 
OK, I got it. Some work in progress for this. 

Thanks again, Paolo.

Best regards,
-Gonglei

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Michael S. Tsirkin]

On Tue, Aug 19, 2014 at 06:25:56AM +0000, Gonglei (Arei) wrote:
> > >> Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> > >>
> > >> On Sun, 2014-08-17 at 13:00 +0200, Michael S. Tsirkin wrote:
> > >>> On Fri, Aug 15, 2014 at 07:33:29AM +0000, Gonglei (Arei) wrote:
> > >>>> Hi,
> > >>>>
> > >>>> I noticed that the qemu-2.1 release change log says
> > >>>> " PCIe: Basic hot-plug/hot-unplug support for Q35 machine."
> > >>>> And then I made a testing for the hotplugging function of Q35.
> > >>>> But I'm failed, and I got the dmesg log in guest os as below:
> > >>>>
> > >>>> [ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on Slot (0 - 4)
> > >>>> [ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot (0 - 4)
> > >>>> [ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 - powering on
> > due
> > >> to button press.
> > >>>> [ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link status
> > >>>> [ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on Slot (0 - 4)
> > >>>> [ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on Slot (0 - 4)
> > >>>>
> > >>>> Steps of testing:
> > >>>>
> > >>>> #1. QEMU version:
> > >>>>
> > >>>> 	The lateset master tree source.
> > >>>>
> > >>>> #2. Command line:
> > >>>>
> > >>>> ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device
> > >> ide-drive,bus=ide.2,drive=MacHDD \
> > >>>>  -drive
> > id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img
> > >> -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
> > >>>> QEMU 2.0.93 monitor - type 'help' for more information
> > >>>> (qemu) device_add
> > >> virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0
> > >>>
> > >>> I don't think you can use any slot except slot 0 for pci express.
> > >
> > > OK. Does the PCIe specification say that?
> > > I appreciate very much that you explain more.
> > 
> > The closest I could find is in "7.3. Configuration Transaction
> > Rules"/"7.3.1. Device Number":
> > 
> > With non-ARI Devices, PCI Express components are restricted to
> > implementing a single Device Number on their primary interface (Upstream
> > Port) [...] Downstream Ports that do not have ARI Forwarding enabled
> > must associate only Device 0 with the device attached to the Logical Bus
> > representing the Link from the Port. Configuration Requests
> > targeting the Bus Number associated with a Link specifying Device Number
> > 0 are delivered to the device attached to the Link; Configuration
> > Requests specifying all other Device Numbers (1-31)
> > must be terminated by the Switch Downstream Port or the Root Port with
> > an Unsupported Request Completion Status (equivalent to Master Abort in
> > PCI).
> > 
> Thanks a lot, Paolo.
> 
> And I found another issue when cold-plugging don't using slot 0, the PCIe device also can't
> be searched in guest os. 
> 
> So, I have some questions and ideas:
> 
> 1. Does qemu support ARI Forwarding for PCIe at present? If yes, how to enable it ?

What do you mean by forwarding?
What would you like to do?
We do have code to emulate ARI, I don't think many people
tested it so it's likely incomplete.

> 2. If not, we should add some check for PCIe root ports and downstream ports,
>  meanwhile add explaining document.

You want an attempt to add a device at slot !=0 to report an error?
We can do this if device at slot 0 does not have ARI support.
Seems like a low priority issue I think.

> 3. Those check should add in general code level, both hotplug and coldplug.

Generally it's not clear how we want to support hotplug for
multifunction devices. One way it to add functions != 0 then
add function 0 and notify guest.
If so, then of course we can't check things ...

> Am I right?  Thanks.
> 
> Best regards,
> -Gonglei

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Gonglei (Arei)]

> -----Original Message-----
> From: Michael S. Tsirkin [mailto:mst@redhat.com]
> Sent: Wednesday, August 20, 2014 5:19 AM
> To: Gonglei (Arei)
> Cc: Paolo Bonzini; Marcel Apfelbaum; qemu-devel@nongnu.org;
> imammedo@redhat.com; armbru@redhat.com; Huangweidong (C)
> Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> 
> On Tue, Aug 19, 2014 at 06:25:56AM +0000, Gonglei (Arei) wrote:
> > > >> Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> > > >>
> > > >> On Sun, 2014-08-17 at 13:00 +0200, Michael S. Tsirkin wrote:
> > > >>> On Fri, Aug 15, 2014 at 07:33:29AM +0000, Gonglei (Arei) wrote:
> > > >>>> Hi,
> > > >>>>
> > > >>>> I noticed that the qemu-2.1 release change log says
> > > >>>> " PCIe: Basic hot-plug/hot-unplug support for Q35 machine."
> > > >>>> And then I made a testing for the hotplugging function of Q35.
> > > >>>> But I'm failed, and I got the dmesg log in guest os as below:
> > > >>>>
> > > >>>> [ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on Slot (0 -
> 4)
> > > >>>> [ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot (0 - 4)
> > > >>>> [ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 - powering
> on
> > > due
> > > >> to button press.
> > > >>>> [ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link status
> > > >>>> [ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on Slot (0
> - 4)
> > > >>>> [ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on Slot (0
> - 4)
> > > >>>>
> > > >>>> Steps of testing:
> > > >>>>
> > > >>>> #1. QEMU version:
> > > >>>>
> > > >>>> 	The lateset master tree source.
> > > >>>>
> > > >>>> #2. Command line:
> > > >>>>
> > > >>>> ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device
> > > >> ide-drive,bus=ide.2,drive=MacHDD \
> > > >>>>  -drive
> > > id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img
> > > >> -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
> > > >>>> QEMU 2.0.93 monitor - type 'help' for more information
> > > >>>> (qemu) device_add
> > > >> virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0
> > > >>>
> > > >>> I don't think you can use any slot except slot 0 for pci express.
> > > >
> > > > OK. Does the PCIe specification say that?
> > > > I appreciate very much that you explain more.
> > >
> > > The closest I could find is in "7.3. Configuration Transaction
> > > Rules"/"7.3.1. Device Number":
> > >
> > > With non-ARI Devices, PCI Express components are restricted to
> > > implementing a single Device Number on their primary interface (Upstream
> > > Port) [...] Downstream Ports that do not have ARI Forwarding enabled
> > > must associate only Device 0 with the device attached to the Logical Bus
> > > representing the Link from the Port. Configuration Requests
> > > targeting the Bus Number associated with a Link specifying Device Number
> > > 0 are delivered to the device attached to the Link; Configuration
> > > Requests specifying all other Device Numbers (1-31)
> > > must be terminated by the Switch Downstream Port or the Root Port with
> > > an Unsupported Request Completion Status (equivalent to Master Abort in
> > > PCI).
> > >
> > Thanks a lot, Paolo.
> >
> > And I found another issue when cold-plugging don't using slot 0, the PCIe
> device also can't
> > be searched in guest os.
> >
> > So, I have some questions and ideas:
> >
> > 1. Does qemu support ARI Forwarding for PCIe at present? If yes, how to
> enable it ?
> 
> What do you mean by forwarding?

Just ARI supporting, came from the PCIe spec.

> What would you like to do?

I just want to add some check. Because it made me confused, 
and other people IMHO.

> We do have code to emulate ARI, I don't think many people
> tested it so it's likely incomplete.
> 
Yes, you have said (pcie_ari_init).

> > 2. If not, we should add some check for PCIe root ports and downstream
> ports,
> >  meanwhile add explaining document.
> 
> You want an attempt to add a device at slot !=0 to report an error?
> We can do this if device at slot 0 does not have ARI support.
> Seems like a low priority issue I think.
> 
Hmm. I have post a patch you have seen. :)

> > 3. Those check should add in general code level, both hotplug and coldplug.
> 
> Generally it's not clear how we want to support hotplug for
> multifunction devices. One way it to add functions != 0 then
> add function 0 and notify guest.
> If so, then of course we can't check things ...
> 
Actually, I just check the device number is 0 or not, don't include functions.

BTW, I have tested a scenario: 

Command line:

./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device ide-drive,bus=ide.2,drive=MacHDD -drive id=MacHDD,if=none, \ 
file=/mnt/sdb/gonglei/image/redhat_q35.img -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg \
-device virtio-net-pci,id=nic1,bus=pcie-switch-downstream-port-1-1,addr=0.1             #coldplug slot 0 function 1
QEMU 2.1.50 monitor - type 'help' for more information
(qemu) device_add virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=0.0   #hotplug slot 0 function 0
(qemu) info network
hub 0
 \ user.0: index=0,type=user,net=10.0.2.0,restrict=off
 \ e1000.0: index=0,type=nic,model=e1000,macaddr=52:54:00:12:34:56
nic2: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:12:34:58
(qemu)

And finally, the guest os only recognized nic2.

Best regards,
-Gonglei

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Knut Omang]

On Wed, 2014-08-20 at 02:16 +0000, Gonglei (Arei) wrote:
> > -----Original Message-----
> > From: Michael S. Tsirkin [mailto:mst@redhat.com]
> > Sent: Wednesday, August 20, 2014 5:19 AM
> > To: Gonglei (Arei)
> > Cc: Paolo Bonzini; Marcel Apfelbaum; qemu-devel@nongnu.org;
> > imammedo@redhat.com; armbru@redhat.com; Huangweidong (C)
> > Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> > 
> > On Tue, Aug 19, 2014 at 06:25:56AM +0000, Gonglei (Arei) wrote:
> > > > >> Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> > > > >>
> > > > >> On Sun, 2014-08-17 at 13:00 +0200, Michael S. Tsirkin wrote:
> > > > >>> On Fri, Aug 15, 2014 at 07:33:29AM +0000, Gonglei (Arei) wrote:
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> I noticed that the qemu-2.1 release change log says
> > > > >>>> " PCIe: Basic hot-plug/hot-unplug support for Q35 machine."
> > > > >>>> And then I made a testing for the hotplugging function of Q35.
> > > > >>>> But I'm failed, and I got the dmesg log in guest os as below:
> > > > >>>>
> > > > >>>> [ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on Slot (0 -
> > 4)
> > > > >>>> [ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot (0 - 4)
> > > > >>>> [ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 - powering
> > on
> > > > due
> > > > >> to button press.
> > > > >>>> [ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link status
> > > > >>>> [ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on Slot (0
> > - 4)
> > > > >>>> [ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on Slot (0
> > - 4)
> > > > >>>>
> > > > >>>> Steps of testing:
> > > > >>>>
> > > > >>>> #1. QEMU version:
> > > > >>>>
> > > > >>>> 	The lateset master tree source.
> > > > >>>>
> > > > >>>> #2. Command line:
> > > > >>>>
> > > > >>>> ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device
> > > > >> ide-drive,bus=ide.2,drive=MacHDD \
> > > > >>>>  -drive
> > > > id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img
> > > > >> -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
> > > > >>>> QEMU 2.0.93 monitor - type 'help' for more information
> > > > >>>> (qemu) device_add
> > > > >> virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0
> > > > >>>
> > > > >>> I don't think you can use any slot except slot 0 for pci express.
> > > > >
> > > > > OK. Does the PCIe specification say that?
> > > > > I appreciate very much that you explain more.
> > > >
> > > > The closest I could find is in "7.3. Configuration Transaction
> > > > Rules"/"7.3.1. Device Number":
> > > >
> > > > With non-ARI Devices, PCI Express components are restricted to
> > > > implementing a single Device Number on their primary interface (Upstream
> > > > Port) [...] Downstream Ports that do not have ARI Forwarding enabled
> > > > must associate only Device 0 with the device attached to the Logical Bus
> > > > representing the Link from the Port. Configuration Requests
> > > > targeting the Bus Number associated with a Link specifying Device Number
> > > > 0 are delivered to the device attached to the Link; Configuration
> > > > Requests specifying all other Device Numbers (1-31)
> > > > must be terminated by the Switch Downstream Port or the Root Port with
> > > > an Unsupported Request Completion Status (equivalent to Master Abort in
> > > > PCI).
> > > >
> > > Thanks a lot, Paolo.
> > > And I found another issue when cold-plugging don't using slot 0, the PCIe
> > device also can't
> > > be searched in guest os.
> > >
> > > So, I have some questions and ideas:
> > >
> > > 1. Does qemu support ARI Forwarding for PCIe at present? If yes, how to
> > enable it ?
> > 
> > What do you mean by forwarding?
> 
> Just ARI supporting, came from the PCIe spec.
> 
> > What would you like to do?
> 
> I just want to add some check. Because it made me confused, 
> and other people IMHO.
> 
> > We do have code to emulate ARI, I don't think many people
> > tested it so it's likely incomplete.
> > 
> Yes, you have said (pcie_ari_init).

I have actually been playing around with this a bit in the context of
SR/IOV, and have some patches waiting to be posted. I haven't had any
success with the downstream switch - as you indicate, pieces may be
missing, but in my case, using the ioh3420 root port worked fine after
setting the ARI forwarding capability and a few other minor fixes that
may even be considered trivial patches. I will put my acts together and
rebase these and post in a separate mail.

Basically I am able to run and hotplug an ARI capable device on any root
port. So instead of using the downstream switch, you could just add
another ioh3420 for your next device, if that suits your needs.

Note that there are two places to add "ARI support", one is the
forwarding capability of bridges/switches, the other is the device's
capability to be an ARI device, represented by the ARI PCIe capability.

> > > 2. If not, we should add some check for PCIe root ports and downstream
> > ports,
> > >  meanwhile add explaining document.
> > 
> > You want an attempt to add a device at slot !=0 to report an error?
> > We can do this if device at slot 0 does not have ARI support.
> > Seems like a low priority issue I think.

If I understand the spec right it is devices in slots > 0 that is
limited, it is that each slot may only expose one device, even if that
device has more than 8 functions and supports ARI, in which case we
should create a master abort. But the better solution in my view would
be just to implement ARI forwarding..

Knut

> Hmm. I have post a patch you have seen. :)
> 
> > > 3. Those check should add in general code level, both hotplug and coldplug.
> > 
> > Generally it's not clear how we want to support hotplug for
> > multifunction devices. One way it to add functions != 0 then
> > add function 0 and notify guest.
> > If so, then of course we can't check things ...
> > 
> Actually, I just check the device number is 0 or not, don't include functions.
> 
> BTW, I have tested a scenario: 
> 
> Command line:
> 
> ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device ide-drive,bus=ide.2,drive=MacHDD -drive id=MacHDD,if=none, \ 
> file=/mnt/sdb/gonglei/image/redhat_q35.img -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg \
> -device virtio-net-pci,id=nic1,bus=pcie-switch-downstream-port-1-1,addr=0.1             #coldplug slot 0 function 1
> QEMU 2.1.50 monitor - type 'help' for more information
> (qemu) device_add virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=0.0   #hotplug slot 0 function 0
> (qemu) info network
> hub 0
>  \ user.0: index=0,type=user,net=10.0.2.0,restrict=off
>  \ e1000.0: index=0,type=nic,model=e1000,macaddr=52:54:00:12:34:56
> nic2: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:12:34:58
> (qemu)
> 
> And finally, the guest os only recognized nic2.
> 
> Best regards,
> -Gonglei
> 

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Knut Omang]

On Wed, 2014-08-20 at 06:39 +0200, Knut Omang wrote:
> On Wed, 2014-08-20 at 02:16 +0000, Gonglei (Arei) wrote:
> > > -----Original Message-----
> > > From: Michael S. Tsirkin [mailto:mst@redhat.com]
> > > Sent: Wednesday, August 20, 2014 5:19 AM
> > > To: Gonglei (Arei)
> > > Cc: Paolo Bonzini; Marcel Apfelbaum; qemu-devel@nongnu.org;
> > > imammedo@redhat.com; armbru@redhat.com; Huangweidong (C)
> > > Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> > > 
> > > On Tue, Aug 19, 2014 at 06:25:56AM +0000, Gonglei (Arei) wrote:
> > > > > >> Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> > > > > >>
> > > > > >> On Sun, 2014-08-17 at 13:00 +0200, Michael S. Tsirkin wrote:
> > > > > >>> On Fri, Aug 15, 2014 at 07:33:29AM +0000, Gonglei (Arei) wrote:
> > > > > >>>> Hi,
> > > > > >>>>
> > > > > >>>> I noticed that the qemu-2.1 release change log says
> > > > > >>>> " PCIe: Basic hot-plug/hot-unplug support for Q35 machine."
> > > > > >>>> And then I made a testing for the hotplugging function of Q35.
> > > > > >>>> But I'm failed, and I got the dmesg log in guest os as below:
> > > > > >>>>
> > > > > >>>> [ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on Slot (0 -
> > > 4)
> > > > > >>>> [ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot (0 - 4)
> > > > > >>>> [ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 - powering
> > > on
> > > > > due
> > > > > >> to button press.
> > > > > >>>> [ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link status
> > > > > >>>> [ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on Slot (0
> > > - 4)
> > > > > >>>> [ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on Slot (0
> > > - 4)
> > > > > >>>>
> > > > > >>>> Steps of testing:
> > > > > >>>>
> > > > > >>>> #1. QEMU version:
> > > > > >>>>
> > > > > >>>> 	The lateset master tree source.
> > > > > >>>>
> > > > > >>>> #2. Command line:
> > > > > >>>>
> > > > > >>>> ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device
> > > > > >> ide-drive,bus=ide.2,drive=MacHDD \
> > > > > >>>>  -drive
> > > > > id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img
> > > > > >> -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
> > > > > >>>> QEMU 2.0.93 monitor - type 'help' for more information
> > > > > >>>> (qemu) device_add
> > > > > >> virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0
> > > > > >>>
> > > > > >>> I don't think you can use any slot except slot 0 for pci express.
> > > > > >
> > > > > > OK. Does the PCIe specification say that?
> > > > > > I appreciate very much that you explain more.
> > > > >
> > > > > The closest I could find is in "7.3. Configuration Transaction
> > > > > Rules"/"7.3.1. Device Number":
> > > > >
> > > > > With non-ARI Devices, PCI Express components are restricted to
> > > > > implementing a single Device Number on their primary interface (Upstream
> > > > > Port) [...] Downstream Ports that do not have ARI Forwarding enabled
> > > > > must associate only Device 0 with the device attached to the Logical Bus
> > > > > representing the Link from the Port. Configuration Requests
> > > > > targeting the Bus Number associated with a Link specifying Device Number
> > > > > 0 are delivered to the device attached to the Link; Configuration
> > > > > Requests specifying all other Device Numbers (1-31)
> > > > > must be terminated by the Switch Downstream Port or the Root Port with
> > > > > an Unsupported Request Completion Status (equivalent to Master Abort in
> > > > > PCI).
> > > > >
> > > > Thanks a lot, Paolo.
> > > > And I found another issue when cold-plugging don't using slot 0, the PCIe
> > > device also can't
> > > > be searched in guest os.
> > > >
> > > > So, I have some questions and ideas:
> > > >
> > > > 1. Does qemu support ARI Forwarding for PCIe at present? If yes, how to
> > > enable it ?
> > > 
> > > What do you mean by forwarding?
> > 
> > Just ARI supporting, came from the PCIe spec.
> > 
> > > What would you like to do?
> > 
> > I just want to add some check. Because it made me confused, 
> > and other people IMHO.
> > 
> > > We do have code to emulate ARI, I don't think many people
> > > tested it so it's likely incomplete.
> > > 
> > Yes, you have said (pcie_ari_init).
> 
> I have actually been playing around with this a bit in the context of
> SR/IOV, and have some patches waiting to be posted. I haven't had any
> success with the downstream switch - as you indicate, pieces may be
> missing, but in my case, using the ioh3420 root port worked fine after
> setting the ARI forwarding capability and a few other minor fixes that
> may even be considered trivial patches. I will put my acts together and
> rebase these and post in a separate mail.
> 
> Basically I am able to run and hotplug an ARI capable device on any root
> port. So instead of using the downstream switch, you could just add
> another ioh3420 for your next device, if that suits your needs.
> 
> Note that there are two places to add "ARI support", one is the
> forwarding capability of bridges/switches, the other is the device's
> capability to be an ARI device, represented by the ARI PCIe capability.
> 
> > > > 2. If not, we should add some check for PCIe root ports and downstream
> > > ports,
> > > >  meanwhile add explaining document.
> > > 
> > > You want an attempt to add a device at slot !=0 to report an error?
> > > We can do this if device at slot 0 does not have ARI support.
> > > Seems like a low priority issue I think.
> 
> If I understand the spec right it is devices in slots > 0 that is

Sorry - just noticed that a confusing "not" was lost here - I meant:

"it is not devices in slots > 0 that are"

> limited, it is that each slot may only expose one device, even if that
> device has more than 8 functions and supports ARI, in which case we
> should create a master abort. But the better solution in my view would
> be just to implement ARI forwarding..

Knut

> > Hmm. I have post a patch you have seen. :)
> > 
> > > > 3. Those check should add in general code level, both hotplug and coldplug.
> > > 
> > > Generally it's not clear how we want to support hotplug for
> > > multifunction devices. One way it to add functions != 0 then
> > > add function 0 and notify guest.
> > > If so, then of course we can't check things ...
> > > 
> > Actually, I just check the device number is 0 or not, don't include functions.
> > 
> > BTW, I have tested a scenario: 
> > 
> > Command line:
> > 
> > ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35 -device ide-drive,bus=ide.2,drive=MacHDD -drive id=MacHDD,if=none, \ 
> > file=/mnt/sdb/gonglei/image/redhat_q35.img -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg \
> > -device virtio-net-pci,id=nic1,bus=pcie-switch-downstream-port-1-1,addr=0.1             #coldplug slot 0 function 1
> > QEMU 2.1.50 monitor - type 'help' for more information
> > (qemu) device_add virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=0.0   #hotplug slot 0 function 0
> > (qemu) info network
> > hub 0
> >  \ user.0: index=0,type=user,net=10.0.2.0,restrict=off
> >  \ e1000.0: index=0,type=nic,model=e1000,macaddr=52:54:00:12:34:56
> > nic2: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:12:34:58
> > (qemu)
> > 
> > And finally, the guest os only recognized nic2.
> > 
> > Best regards,
> > -Gonglei
> > 
> 
> 
> 

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Gonglei (Arei)]

> On Wed, 2014-08-20 at 06:39 +0200, Knut Omang wrote:
> > On Wed, 2014-08-20 at 02:16 +0000, Gonglei (Arei) wrote:
> > > > -----Original Message-----
> > > > From: Michael S. Tsirkin [mailto:mst@redhat.com]
> > > > Sent: Wednesday, August 20, 2014 5:19 AM
> > > > To: Gonglei (Arei)
> > > > Cc: Paolo Bonzini; Marcel Apfelbaum; qemu-devel@nongnu.org;
> > > > imammedo@redhat.com; armbru@redhat.com; Huangweidong (C)
> > > > Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> > > >
> > > > On Tue, Aug 19, 2014 at 06:25:56AM +0000, Gonglei (Arei) wrote:
> > > > > > >> Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35
> machine?
> > > > > > >>
> > > > > > >> On Sun, 2014-08-17 at 13:00 +0200, Michael S. Tsirkin wrote:
> > > > > > >>> On Fri, Aug 15, 2014 at 07:33:29AM +0000, Gonglei (Arei) wrote:
> > > > > > >>>> Hi,
> > > > > > >>>>
> > > > > > >>>> I noticed that the qemu-2.1 release change log says
> > > > > > >>>> " PCIe: Basic hot-plug/hot-unplug support for Q35 machine."
> > > > > > >>>> And then I made a testing for the hotplugging function of Q35.
> > > > > > >>>> But I'm failed, and I got the dmesg log in guest os as below:
> > > > > > >>>>
> > > > > > >>>> [ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on
> Slot (0 -
> > > > 4)
> > > > > > >>>> [ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot
> (0 - 4)
> > > > > > >>>> [ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 -
> powering
> > > > on
> > > > > > due
> > > > > > >> to button press.
> > > > > > >>>> [ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link
> status
> > > > > > >>>> [ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on
> Slot (0
> > > > - 4)
> > > > > > >>>> [ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on
> Slot (0
> > > > - 4)
> > > > > > >>>>
> > > > > > >>>> Steps of testing:
> > > > > > >>>>
> > > > > > >>>> #1. QEMU version:
> > > > > > >>>>
> > > > > > >>>> 	The lateset master tree source.
> > > > > > >>>>
> > > > > > >>>> #2. Command line:
> > > > > > >>>>
> > > > > > >>>> ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35
> -device
> > > > > > >> ide-drive,bus=ide.2,drive=MacHDD \
> > > > > > >>>>  -drive
> > > > > > id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img
> > > > > > >> -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
> > > > > > >>>> QEMU 2.0.93 monitor - type 'help' for more information
> > > > > > >>>> (qemu) device_add
> > > > > > >>
> virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0
> > > > > > >>>
> > > > > > >>> I don't think you can use any slot except slot 0 for pci express.
> > > > > > >
> > > > > > > OK. Does the PCIe specification say that?
> > > > > > > I appreciate very much that you explain more.
> > > > > >
> > > > > > The closest I could find is in "7.3. Configuration Transaction
> > > > > > Rules"/"7.3.1. Device Number":
> > > > > >
> > > > > > With non-ARI Devices, PCI Express components are restricted to
> > > > > > implementing a single Device Number on their primary interface
> (Upstream
> > > > > > Port) [...] Downstream Ports that do not have ARI Forwarding enabled
> > > > > > must associate only Device 0 with the device attached to the Logical
> Bus
> > > > > > representing the Link from the Port. Configuration Requests
> > > > > > targeting the Bus Number associated with a Link specifying Device
> Number
> > > > > > 0 are delivered to the device attached to the Link; Configuration
> > > > > > Requests specifying all other Device Numbers (1-31)
> > > > > > must be terminated by the Switch Downstream Port or the Root Port
> with
> > > > > > an Unsupported Request Completion Status (equivalent to Master
> Abort in
> > > > > > PCI).
> > > > > >
> > > > > Thanks a lot, Paolo.
> > > > > And I found another issue when cold-plugging don't using slot 0, the
> PCIe
> > > > device also can't
> > > > > be searched in guest os.
> > > > >
> > > > > So, I have some questions and ideas:
> > > > >
> > > > > 1. Does qemu support ARI Forwarding for PCIe at present? If yes, how
> to
> > > > enable it ?
> > > >
> > > > What do you mean by forwarding?
> > >
> > > Just ARI supporting, came from the PCIe spec.
> > >
> > > > What would you like to do?
> > >
> > > I just want to add some check. Because it made me confused,
> > > and other people IMHO.
> > >
> > > > We do have code to emulate ARI, I don't think many people
> > > > tested it so it's likely incomplete.
> > > >
> > > Yes, you have said (pcie_ari_init).
> >
> > I have actually been playing around with this a bit in the context of
> > SR/IOV, and have some patches waiting to be posted. I haven't had any
> > success with the downstream switch - as you indicate, pieces may be
> > missing, but in my case, using the ioh3420 root port worked fine after
> > setting the ARI forwarding capability and a few other minor fixes that
> > may even be considered trivial patches. I will put my acts together and
> > rebase these and post in a separate mail.
> >
> > Basically I am able to run and hotplug an ARI capable device on any root
> > port. So instead of using the downstream switch, you could just add
> > another ioh3420 for your next device, if that suits your needs.
> >
> > Note that there are two places to add "ARI support", one is the
> > forwarding capability of bridges/switches, the other is the device's
> > capability to be an ARI device, represented by the ARI PCIe capability.
> >
Hi, you are welcome to post your patches.
At present, both root roots and downstream ports 
are not support a device with slot > 0.

> > > > > 2. If not, we should add some check for PCIe root ports and downstream
> > > > ports,
> > > > >  meanwhile add explaining document.
> > > >
> > > > You want an attempt to add a device at slot !=0 to report an error?
> > > > We can do this if device at slot 0 does not have ARI support.
> > > > Seems like a low priority issue I think.
> >
> > If I understand the spec right it is devices in slots > 0 that is
> 
> Sorry - just noticed that a confusing "not" was lost here - I meant:
> 
> "it is not devices in slots > 0 that are"
> 
> > limited, it is that each slot may only expose one device, even if that
> > device has more than 8 functions and supports ARI, in which case we
> > should create a master abort.

PCIe spec says that Downstream Ports that do not have ARI Forwarding 
enabled must associate only Device 0 with the device attached to the Logical
Bus representing the Link from the Port.	

> > But the better solution in my view would
> > be just to implement ARI forwarding..
> 
Yes.

Best regards,
-Gonglei

Re: [Qemu-devel] [Question] Why doesn't PCIe hotplug work for Q35 machine?

[Knut Omang]

On Wed, 2014-08-20 at 06:04 +0000, Gonglei (Arei) wrote:
> > On Wed, 2014-08-20 at 06:39 +0200, Knut Omang wrote:
> > > On Wed, 2014-08-20 at 02:16 +0000, Gonglei (Arei) wrote:
> > > > > -----Original Message-----
> > > > > From: Michael S. Tsirkin [mailto:mst@redhat.com]
> > > > > Sent: Wednesday, August 20, 2014 5:19 AM
> > > > > To: Gonglei (Arei)
> > > > > Cc: Paolo Bonzini; Marcel Apfelbaum; qemu-devel@nongnu.org;
> > > > > imammedo@redhat.com; armbru@redhat.com; Huangweidong (C)
> > > > > Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35 machine?
> > > > >
> > > > > On Tue, Aug 19, 2014 at 06:25:56AM +0000, Gonglei (Arei) wrote:
> > > > > > > >> Subject: Re: [Question] Why doesn't PCIe hotplug work for Q35
> > machine?
> > > > > > > >>
> > > > > > > >> On Sun, 2014-08-17 at 13:00 +0200, Michael S. Tsirkin wrote:
> > > > > > > >>> On Fri, Aug 15, 2014 at 07:33:29AM +0000, Gonglei (Arei) wrote:
> > > > > > > >>>> Hi,
> > > > > > > >>>>
> > > > > > > >>>> I noticed that the qemu-2.1 release change log says
> > > > > > > >>>> " PCIe: Basic hot-plug/hot-unplug support for Q35 machine."
> > > > > > > >>>> And then I made a testing for the hotplugging function of Q35.
> > > > > > > >>>> But I'm failed, and I got the dmesg log in guest os as below:
> > > > > > > >>>>
> > > > > > > >>>> [ 159.035250] Pciehp 0000:05:00.0:pcie24: Button pressed on
> > Slot (0 -
> > > > > 4)
> > > > > > > >>>> [ 159.035274] Pciehp 0000:05:00.0:pcie24: Card present on Slot
> > (0 - 4)
> > > > > > > >>>> [ 159.036517] Pciehp 0000:05:00.0:pcie24: PCI slot #0 - 4 -
> > powering
> > > > > on
> > > > > > > due
> > > > > > > >> to button press.
> > > > > > > >>>> [ 159.188049] Pciehp 0000:05:00.0:pcie24: Failed to check link
> > status
> > > > > > > >>>> [ 159.201968] Pciehp 0000:05:00.0:pcie24: Card not present on
> > Slot (0
> > > > > - 4)
> > > > > > > >>>> [ 159.202529] Pciehp 0000:05:00.0:pcie24: Already disabled on
> > Slot (0
> > > > > - 4)
> > > > > > > >>>>
> > > > > > > >>>> Steps of testing:
> > > > > > > >>>>
> > > > > > > >>>> #1. QEMU version:
> > > > > > > >>>>
> > > > > > > >>>> 	The lateset master tree source.
> > > > > > > >>>>
> > > > > > > >>>> #2. Command line:
> > > > > > > >>>>
> > > > > > > >>>> ./qemu-system-x86_64 -enable-kvm -m 2048 -machine q35
> > -device
> > > > > > > >> ide-drive,bus=ide.2,drive=MacHDD \
> > > > > > > >>>>  -drive
> > > > > > > id=MacHDD,if=none,file=/mnt/sdb/gonglei/image/redhat_q35.img
> > > > > > > >> -monitor stdio -vnc :10 -readconfig ../docs/q35-chipset.cfg
> > > > > > > >>>> QEMU 2.0.93 monitor - type 'help' for more information
> > > > > > > >>>> (qemu) device_add
> > > > > > > >>
> > virtio-net-pci,id=nic2,bus=pcie-switch-downstream-port-1-1,addr=1.0
> > > > > > > >>>
> > > > > > > >>> I don't think you can use any slot except slot 0 for pci express.
> > > > > > > >
> > > > > > > > OK. Does the PCIe specification say that?
> > > > > > > > I appreciate very much that you explain more.
> > > > > > >
> > > > > > > The closest I could find is in "7.3. Configuration Transaction
> > > > > > > Rules"/"7.3.1. Device Number":
> > > > > > >
> > > > > > > With non-ARI Devices, PCI Express components are restricted to
> > > > > > > implementing a single Device Number on their primary interface
> > (Upstream
> > > > > > > Port) [...] Downstream Ports that do not have ARI Forwarding enabled
> > > > > > > must associate only Device 0 with the device attached to the Logical
> > Bus
> > > > > > > representing the Link from the Port. Configuration Requests
> > > > > > > targeting the Bus Number associated with a Link specifying Device
> > Number
> > > > > > > 0 are delivered to the device attached to the Link; Configuration
> > > > > > > Requests specifying all other Device Numbers (1-31)
> > > > > > > must be terminated by the Switch Downstream Port or the Root Port
> > with
> > > > > > > an Unsupported Request Completion Status (equivalent to Master
> > Abort in
> > > > > > > PCI).
> > > > > > >
> > > > > > Thanks a lot, Paolo.
> > > > > > And I found another issue when cold-plugging don't using slot 0, the
> > PCIe
> > > > > device also can't
> > > > > > be searched in guest os.
> > > > > >
> > > > > > So, I have some questions and ideas:
> > > > > >
> > > > > > 1. Does qemu support ARI Forwarding for PCIe at present? If yes, how
> > to
> > > > > enable it ?
> > > > >
> > > > > What do you mean by forwarding?
> > > >
> > > > Just ARI supporting, came from the PCIe spec.
> > > >
> > > > > What would you like to do?
> > > >
> > > > I just want to add some check. Because it made me confused,
> > > > and other people IMHO.
> > > >
> > > > > We do have code to emulate ARI, I don't think many people
> > > > > tested it so it's likely incomplete.
> > > > >
> > > > Yes, you have said (pcie_ari_init).
> > >
> > > I have actually been playing around with this a bit in the context of
> > > SR/IOV, and have some patches waiting to be posted. I haven't had any
> > > success with the downstream switch - as you indicate, pieces may be
> > > missing, but in my case, using the ioh3420 root port worked fine after
> > > setting the ARI forwarding capability and a few other minor fixes that
> > > may even be considered trivial patches. I will put my acts together and
> > > rebase these and post in a separate mail.
> > >
> > > Basically I am able to run and hotplug an ARI capable device on any root
> > > port. So instead of using the downstream switch, you could just add
> > > another ioh3420 for your next device, if that suits your needs.
> > >
> > > Note that there are two places to add "ARI support", one is the
> > > forwarding capability of bridges/switches, the other is the device's
> > > capability to be an ARI device, represented by the ARI PCIe capability.
> > >
> Hi, you are welcome to post your patches.
> At present, both root roots and downstream ports 
> are not support a device with slot > 0.

Working on it right now ;-)

> > > > > > 2. If not, we should add some check for PCIe root ports and downstream
> > > > > ports,
> > > > > >  meanwhile add explaining document.
> > > > >
> > > > > You want an attempt to add a device at slot !=0 to report an error?
> > > > > We can do this if device at slot 0 does not have ARI support.
> > > > > Seems like a low priority issue I think.
> > >
> > > If I understand the spec right it is devices in slots > 0 that is
> > 
> > Sorry - just noticed that a confusing "not" was lost here - I meant:
> > 
> > "it is not devices in slots > 0 that are"
> > 
> > > limited, it is that each slot may only expose one device, even if that
> > > device has more than 8 functions and supports ARI, in which case we
> > > should create a master abort.
> 
> PCIe spec says that Downstream Ports that do not have ARI Forwarding 
> enabled must associate only Device 0 with the device attached to the Logical
> Bus representing the Link from the Port.	

Yes, so the way I understand it is that lets say you have a device with
16 logical functions, then only functions 0-7 represented with device ID
0.0-0.7 may be exposed for that port, as function #9 and above would get
device ID 1.0-1.7 which would require ARI forwarding.

Knut

> > > But the better solution in my view would
> > > be just to implement ARI forwarding..
> > 
> Yes.
> 
> Best regards,
> -Gonglei