[Qemu-devel] [Issues] PCI hotplug does not work well on pc platform?

[Qemu-devel] [Issues] PCI hotplug does not work well on pc platform?

[Liu, Jing2]

Hi everyone,

I have two questions.
1. PCI hotplug on pci.0 must manually rescan in guest. The ACPI hotplug 
handler sends the GPE event to guest but it seems guest doesn't receive 
it? I tried to open ACPI debug level/layer to 0xffffffff, in order to 
see if there is any message after device_add in monitor, but no message 
comes out until I manually rescan. Also tried printk in 
acpi_ev_gpe_xrupt_handler() and acpi_ev_sci_xrupt_handler(). No output 
in dmesg.
(I'm sure that CONFIG_HOTPLUG_PCI_PCIE=y, CONFIG_HOTPLUG_PCI_CPCI=y, 
CONFIG_HOTPLUG_PCI=y, CONFIG_HOTPLUG_PCI_ACPI=y)

Whether this is a kind of design or a known issue? Does guest receive 
the request, where can I find the

2. I want to try hotplugging on pci-bridge on pc platform, using shpc. I 
set shpc=on, but when I do device_add, qemu still calls 
acpi_pcihp_device_plug_cb? Why it does not call pci_bridge_dev_hotplug_cb?
(CONFIG_HOTPLUG_PCI_SHPC=y)

Thanks very much!
Jing

Re: [Qemu-devel] [Issues] PCI hotplug does not work well on pc platform?

[Igor Mammedov]

On Wed, 30 Jan 2019 21:02:10 +0800
"Liu, Jing2" <jing2.liu@linux.intel.com> wrote:

> Hi everyone,
> 
> I have two questions.
> 1. PCI hotplug on pci.0 must manually rescan in guest. The ACPI hotplug 
> handler sends the GPE event to guest but it seems guest doesn't receive 
> it? I tried to open ACPI debug level/layer to 0xffffffff, in order to 
> see if there is any message after device_add in monitor, but no message 
> comes out until I manually rescan. Also tried printk in 
> acpi_ev_gpe_xrupt_handler() and acpi_ev_sci_xrupt_handler(). No output 
> in dmesg.
> (I'm sure that CONFIG_HOTPLUG_PCI_PCIE=y, CONFIG_HOTPLUG_PCI_CPCI=y, 
> CONFIG_HOTPLUG_PCI=y, CONFIG_HOTPLUG_PCI_ACPI=y)
> 
> Whether this is a kind of design or a known issue? Does guest receive 
> the request, where can I find the
does it work with known to work kernels (RHEL7)?

Also sharing used QEMU version and command line could help.

> 
> 2. I want to try hotplugging on pci-bridge on pc platform, using shpc. I 
> set shpc=on, but when I do device_add, qemu still calls 
> acpi_pcihp_device_plug_cb? Why it does not call pci_bridge_dev_hotplug_cb?
> (CONFIG_HOTPLUG_PCI_SHPC=y)

try to disable ACPI hotplug for bridges
 -global PIIX4_PM.acpi-pci-hotplug-with-bridge-support=off

> Thanks very much!
> Jing
> 

Re: [Qemu-devel] [Issues] PCI hotplug does not work well on pc platform?

[Liu, Jing2]

Hi Igor,

Thanks for your reply!

On 2/5/2019 11:47 PM, Igor Mammedov wrote:
> On Wed, 30 Jan 2019 21:02:10 +0800
> "Liu, Jing2" <jing2.liu@linux.intel.com> wrote:
> 
>> Hi everyone,
>>
>> I have two questions.
>> 1. PCI hotplug on pci.0 must manually rescan in guest. The ACPI hotplug
>> handler sends the GPE event to guest but it seems guest doesn't receive
>> it? I tried to open ACPI debug level/layer to 0xffffffff, in order to
>> see if there is any message after device_add in monitor, but no message
>> comes out until I manually rescan. Also tried printk in
>> acpi_ev_gpe_xrupt_handler() and acpi_ev_sci_xrupt_handler(). No output
>> in dmesg.
>> (I'm sure that CONFIG_HOTPLUG_PCI_PCIE=y, CONFIG_HOTPLUG_PCI_CPCI=y,
>> CONFIG_HOTPLUG_PCI=y, CONFIG_HOTPLUG_PCI_ACPI=y)
>>
>> Whether this is a kind of design or a known issue? Does guest receive
>> the request, where can I find the
> does it work with known to work kernels (RHEL7)?
> 
> Also sharing used QEMU version and command line could help.
> 
Is there any key config of kernel in guest, besides those I listed above?

I used guest kernel v4.18 and qemu upstream v3.1.0
Command line:
sudo /home/xxx/qemu/x86_64-softmmu/qemu-system-x86_64  \
-machine pc,accel=kvm,kernel_irqchip -cpu host -m 1G,slots=2,maxmem=10G \
-nographic -no-user-config -nodefaults -vga none \
-drive file=/home/xxx/image/clear-24690-kvm.img,if=virtio,format=raw \
  -smp sockets=1,cpus=4,cores=2,maxcpus=8 \
-device virtio-serial-pci,id=virtio-serial0,disable-modern,addr=0x5 \
-monitor tcp:0.0.0.0:5000,nowait,server \
-chardev stdio,id=charconsole0 -device 
virtconsole,chardev=charconsole0,id=console0  \
-kernel /home/xxx/linux-stable/arch/x86/boot/bzImage \
-append 'root=/dev/vda3 rw rootfstype=ext4 data=ordered 
rcupdate.rcu_expedited=1 pci=lastbus=0 pci=realloc=on tsc=reliable 
no_timer_check reboot=t noapictimer console=hvc0 iommu=off  panic=1 
initcall_debug acpi.debug_layer=0x40000 acpi.debug_level=4 ' \
-device pci-bridge,bus=pci.0,id=br1,chassis_nr=1,shpc=on,addr=6  \

>>
>> 2. I want to try hotplugging on pci-bridge on pc platform, using shpc. I
>> set shpc=on, but when I do device_add, qemu still calls
>> acpi_pcihp_device_plug_cb? Why it does not call pci_bridge_dev_hotplug_cb?
>> (CONFIG_HOTPLUG_PCI_SHPC=y)
> 
> try to disable ACPI hotplug for bridges
>   -global PIIX4_PM.acpi-pci-hotplug-with-bridge-support=off
> 
I'll try it!

Thanks!
Jing
> 

Re: [Qemu-devel] [Issues] PCI hotplug does not work well on pc platform?

[Igor Mammedov]

On Wed, 13 Feb 2019 15:40:57 +0800
"Liu, Jing2" <jing2.liu@linux.intel.com> wrote:

> Hi Igor,
> 
> Thanks for your reply!
> 
> On 2/5/2019 11:47 PM, Igor Mammedov wrote:
> > On Wed, 30 Jan 2019 21:02:10 +0800
> > "Liu, Jing2" <jing2.liu@linux.intel.com> wrote:
> >   
> >> Hi everyone,
> >>
> >> I have two questions.
> >> 1. PCI hotplug on pci.0 must manually rescan in guest. The ACPI hotplug
> >> handler sends the GPE event to guest but it seems guest doesn't receive
> >> it? I tried to open ACPI debug level/layer to 0xffffffff, in order to
> >> see if there is any message after device_add in monitor, but no message
> >> comes out until I manually rescan. Also tried printk in
> >> acpi_ev_gpe_xrupt_handler() and acpi_ev_sci_xrupt_handler(). No output
> >> in dmesg.
> >> (I'm sure that CONFIG_HOTPLUG_PCI_PCIE=y, CONFIG_HOTPLUG_PCI_CPCI=y,
> >> CONFIG_HOTPLUG_PCI=y, CONFIG_HOTPLUG_PCI_ACPI=y)
> >>
> >> Whether this is a kind of design or a known issue? Does guest receive
> >> the request, where can I find the  
> > does it work with known to work kernels (RHEL7)?
> > 
> > Also sharing used QEMU version and command line could help.
> >   
> Is there any key config of kernel in guest, besides those I listed above?
Maybe Marcel knows something about it
(CCed)

> 
> I used guest kernel v4.18 and qemu upstream v3.1.0
> Command line:
> sudo /home/xxx/qemu/x86_64-softmmu/qemu-system-x86_64  \
> -machine pc,accel=kvm,kernel_irqchip -cpu host -m 1G,slots=2,maxmem=10G \
> -nographic -no-user-config -nodefaults -vga none \
> -drive file=/home/xxx/image/clear-24690-kvm.img,if=virtio,format=raw \
>   -smp sockets=1,cpus=4,cores=2,maxcpus=8 \
> -device virtio-serial-pci,id=virtio-serial0,disable-modern,addr=0x5 \
> -monitor tcp:0.0.0.0:5000,nowait,server \
> -chardev stdio,id=charconsole0 -device 
> virtconsole,chardev=charconsole0,id=console0  \
> -kernel /home/xxx/linux-stable/arch/x86/boot/bzImage \
> -append 'root=/dev/vda3 rw rootfstype=ext4 data=ordered 
> rcupdate.rcu_expedited=1 pci=lastbus=0 pci=realloc=on tsc=reliable 
> no_timer_check reboot=t noapictimer console=hvc0 iommu=off  panic=1 
> initcall_debug acpi.debug_layer=0x40000 acpi.debug_level=4 ' \
> -device pci-bridge,bus=pci.0,id=br1,chassis_nr=1,shpc=on,addr=6  \
> 
> >>
> >> 2. I want to try hotplugging on pci-bridge on pc platform, using shpc. I
> >> set shpc=on, but when I do device_add, qemu still calls
> >> acpi_pcihp_device_plug_cb? Why it does not call pci_bridge_dev_hotplug_cb?
> >> (CONFIG_HOTPLUG_PCI_SHPC=y)  
> > 
> > try to disable ACPI hotplug for bridges
> >   -global PIIX4_PM.acpi-pci-hotplug-with-bridge-support=off
> >   
> I'll try it!
> 
> Thanks!
> Jing
> >   
> 

Re: [Qemu-devel] [Issues] PCI hotplug does not work well on pc platform?

[Marcel Apfelbaum]

Hi Igor, Jing

On 2/14/19 1:31 PM, Igor Mammedov wrote:
> On Wed, 13 Feb 2019 15:40:57 +0800
> "Liu, Jing2" <jing2.liu@linux.intel.com> wrote:
>
>> Hi Igor,
>>
>> Thanks for your reply!
>>
>> On 2/5/2019 11:47 PM, Igor Mammedov wrote:
>>> On Wed, 30 Jan 2019 21:02:10 +0800
>>> "Liu, Jing2" <jing2.liu@linux.intel.com> wrote:
>>>    
>>>> Hi everyone,
>>>>
>>>> I have two questions.
>>>> 1. PCI hotplug on pci.0 must manually rescan in guest. The ACPI hotplug
>>>> handler sends the GPE event to guest but it seems guest doesn't receive
>>>> it? I tried to open ACPI debug level/layer to 0xffffffff, in order to
>>>> see if there is any message after device_add in monitor, but no message
>>>> comes out until I manually rescan. Also tried printk in
>>>> acpi_ev_gpe_xrupt_handler() and acpi_ev_sci_xrupt_handler(). No output
>>>> in dmesg.
>>>> (I'm sure that CONFIG_HOTPLUG_PCI_PCIE=y, CONFIG_HOTPLUG_PCI_CPCI=y,
>>>> CONFIG_HOTPLUG_PCI=y, CONFIG_HOTPLUG_PCI_ACPI=y)

What about |CONFIG_HOTPLUG_PCI_SHPC=y ?

|
>>>> Whether this is a kind of design or a known issue? Does guest receive
>>>> the request, where can I find the
>>> does it work with known to work kernels (RHEL7)?
>>>
>>> Also sharing used QEMU version and command line could help.
>>>    
>> Is there any key config of kernel in guest, besides those I listed above?
> Maybe Marcel knows something about it
> (CCed)

May I ask why do you need SHPC hotplug and not the ACPI based hotplug?

Anyway, please look at dmesg (in guest) to see which hoplug handler is 
registered for
the pci bridge slots. You should have 2 options: acpiphp and shpchp.

If you see something like this:
[    0.014285] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.014314] PCI: Using configuration type 1 for base access
[    0.016011] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.016041] cryptd: max_cpu_qlen set to 1000
[    0.016078] ACPI: Added _OSI(Module Device)
[    0.016079] ACPI: Added _OSI(Processor Device)
[    0.016080] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.016081] ACPI: Added _OSI(Processor Aggregator Device)
[    0.016082] ACPI: Added _OSI(Linux-Dell-Video)
[    0.016578] ACPI: 1 ACPI AML tables successfully acquired and loaded
[    0.018248] ACPI: Interpreter enabled
[    0.018260] ACPI: (supports S0 S5)
[    0.018261] ACPI: Using IOAPIC for interrupt routing
[    0.018276] PCI: Using host bridge windows from ACPI; if necessary, 
use "pci=nocrs" and report a bug
[    0.018375] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.020616] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.020620] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments 
MSI]
[    0.020624] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    0.020630] acpi PNP0A03:00: fail to add MMCONFIG information, can't 
access extended PCI configuration space under this bridge.
[    0.020884] acpiphp: Slot [3] registered
[    0.020915] acpiphp: Slot [4] registered
[    0.020955] acpiphp: Slot [6] registered
[    0.020985] acpiphp: Slot [7] registered
[...]


it means the ACPI subsystem takes charge of PCI hotplug and shpc based 
hotplug will not work.


Bottom line, you just have to ensure the shpchp takes responsibility for 
the pci bridge hotplug.

Thanks,
Marcel

>> I used guest kernel v4.18 and qemu upstream v3.1.0
>> Command line:
>> sudo /home/xxx/qemu/x86_64-softmmu/qemu-system-x86_64  \
>> -machine pc,accel=kvm,kernel_irqchip -cpu host -m 1G,slots=2,maxmem=10G \
>> -nographic -no-user-config -nodefaults -vga none \
>> -drive file=/home/xxx/image/clear-24690-kvm.img,if=virtio,format=raw \
>>    -smp sockets=1,cpus=4,cores=2,maxcpus=8 \
>> -device virtio-serial-pci,id=virtio-serial0,disable-modern,addr=0x5 \
>> -monitor tcp:0.0.0.0:5000,nowait,server \
>> -chardev stdio,id=charconsole0 -device
>> virtconsole,chardev=charconsole0,id=console0  \
>> -kernel /home/xxx/linux-stable/arch/x86/boot/bzImage \
>> -append 'root=/dev/vda3 rw rootfstype=ext4 data=ordered
>> rcupdate.rcu_expedited=1 pci=lastbus=0 pci=realloc=on tsc=reliable
>> no_timer_check reboot=t noapictimer console=hvc0 iommu=off  panic=1
>> initcall_debug acpi.debug_layer=0x40000 acpi.debug_level=4 ' \
>> -device pci-bridge,bus=pci.0,id=br1,chassis_nr=1,shpc=on,addr=6  \
>>
>>>> 2. I want to try hotplugging on pci-bridge on pc platform, using shpc. I
>>>> set shpc=on, but when I do device_add, qemu still calls
>>>> acpi_pcihp_device_plug_cb? Why it does not call pci_bridge_dev_hotplug_cb?
>>>> (CONFIG_HOTPLUG_PCI_SHPC=y)
>>> try to disable ACPI hotplug for bridges
>>>    -global PIIX4_PM.acpi-pci-hotplug-with-bridge-support=off
>>>    
>> I'll try it!
>>
>> Thanks!
>> Jing
>>>    

Re: [Qemu-devel] [Issues] PCI hotplug does not work well on pc platform?

[Igor Mammedov]

On Thu, 14 Feb 2019 15:12:35 +0200
Marcel Apfelbaum <marcel.apfelbaum@gmail.com> wrote:

> Hi Igor, Jing
> 
> On 2/14/19 1:31 PM, Igor Mammedov wrote:
> > On Wed, 13 Feb 2019 15:40:57 +0800
> > "Liu, Jing2" <jing2.liu@linux.intel.com> wrote:
> >  
> >> Hi Igor,
> >>
> >> Thanks for your reply!
> >>
> >> On 2/5/2019 11:47 PM, Igor Mammedov wrote:  
> >>> On Wed, 30 Jan 2019 21:02:10 +0800
> >>> "Liu, Jing2" <jing2.liu@linux.intel.com> wrote:
> >>>      
> >>>> Hi everyone,
> >>>>
> >>>> I have two questions.
> >>>> 1. PCI hotplug on pci.0 must manually rescan in guest. The ACPI hotplug
> >>>> handler sends the GPE event to guest but it seems guest doesn't receive
> >>>> it? I tried to open ACPI debug level/layer to 0xffffffff, in order to
> >>>> see if there is any message after device_add in monitor, but no message
> >>>> comes out until I manually rescan. Also tried printk in
> >>>> acpi_ev_gpe_xrupt_handler() and acpi_ev_sci_xrupt_handler(). No output
> >>>> in dmesg.
> >>>> (I'm sure that CONFIG_HOTPLUG_PCI_PCIE=y, CONFIG_HOTPLUG_PCI_CPCI=y,
> >>>> CONFIG_HOTPLUG_PCI=y, CONFIG_HOTPLUG_PCI_ACPI=y)  
> 
> What about |CONFIG_HOTPLUG_PCI_SHPC=y ?
> 
> |
> >>>> Whether this is a kind of design or a known issue? Does guest receive
> >>>> the request, where can I find the  
> >>> does it work with known to work kernels (RHEL7)?
> >>>
> >>> Also sharing used QEMU version and command line could help.
> >>>      
> >> Is there any key config of kernel in guest, besides those I listed above?  
> > Maybe Marcel knows something about it
> > (CCed)  
> 
> May I ask why do you need SHPC hotplug and not the ACPI based hotplug?
I'd say SHPC one is preferred one, ACPI hotplug is rather complicated
QEMU hack to make legacy Windows guest support hotplug of PCI devices.
 
[...]

Re: [Qemu-devel] [Issues] PCI hotplug does not work well on pc platform?

[Marcel Apfelbaum]

On 2/14/19 4:15 PM, Igor Mammedov wrote:
> On Thu, 14 Feb 2019 15:12:35 +0200
> Marcel Apfelbaum <marcel.apfelbaum@gmail.com> wrote:
>
>> Hi Igor, Jing
>>
>> On 2/14/19 1:31 PM, Igor Mammedov wrote:
>>> On Wed, 13 Feb 2019 15:40:57 +0800
>>> "Liu, Jing2" <jing2.liu@linux.intel.com> wrote:
>>>   
>>>> Hi Igor,
>>>>
>>>> Thanks for your reply!
>>>>
>>>> On 2/5/2019 11:47 PM, Igor Mammedov wrote:
>>>>> On Wed, 30 Jan 2019 21:02:10 +0800
>>>>> "Liu, Jing2" <jing2.liu@linux.intel.com> wrote:
>>>>>       
>>>>>> Hi everyone,
>>>>>>
>>>>>> I have two questions.
>>>>>> 1. PCI hotplug on pci.0 must manually rescan in guest. The ACPI hotplug
>>>>>> handler sends the GPE event to guest but it seems guest doesn't receive
>>>>>> it? I tried to open ACPI debug level/layer to 0xffffffff, in order to
>>>>>> see if there is any message after device_add in monitor, but no message
>>>>>> comes out until I manually rescan. Also tried printk in
>>>>>> acpi_ev_gpe_xrupt_handler() and acpi_ev_sci_xrupt_handler(). No output
>>>>>> in dmesg.
>>>>>> (I'm sure that CONFIG_HOTPLUG_PCI_PCIE=y, CONFIG_HOTPLUG_PCI_CPCI=y,
>>>>>> CONFIG_HOTPLUG_PCI=y, CONFIG_HOTPLUG_PCI_ACPI=y)
>> What about |CONFIG_HOTPLUG_PCI_SHPC=y ?
>>
>> |
>>>>>> Whether this is a kind of design or a known issue? Does guest receive
>>>>>> the request, where can I find the
>>>>> does it work with known to work kernels (RHEL7)?
>>>>>
>>>>> Also sharing used QEMU version and command line could help.
>>>>>       
>>>> Is there any key config of kernel in guest, besides those I listed above?
>>> Maybe Marcel knows something about it
>>> (CCed)
>> May I ask why do you need SHPC hotplug and not the ACPI based hotplug?
> I'd say SHPC one is preferred one, ACPI hotplug is rather complicated
> QEMU hack to make legacy Windows guest support hotplug of PCI devices.

I can understand and agree, but is much less tested; anyway I distinctly
remember I fix all the problems with PCI SHPC hotplug, it should work.

Thanks,
Marcel

>   
> [...]

Re: [Qemu-devel] [Issues] PCI hotplug does not work well on pc platform?

[Liu, Jing2]

Hi Marcel and Igor,

Thanks very much for your help!

On 2/14/2019 10:42 PM, Marcel Apfelbaum wrote:
> 
> 
[...]
>>>>>>> I have two questions.
>>>>>>> 1. PCI hotplug on pci.0 must manually rescan in guest. The ACPI 
>>>>>>> hotplug
>>>>>>> handler sends the GPE event to guest but it seems guest doesn't 
>>>>>>> receive
>>>>>>> it? I tried to open ACPI debug level/layer to 0xffffffff, in 
>>>>>>> order to
>>>>>>> see if there is any message after device_add in monitor, but no 
>>>>>>> message
>>>>>>> comes out until I manually rescan. Also tried printk in
>>>>>>> acpi_ev_gpe_xrupt_handler() and acpi_ev_sci_xrupt_handler(). No 
>>>>>>> output
>>>>>>> in dmesg.
>>>>>>> (I'm sure that CONFIG_HOTPLUG_PCI_PCIE=y, CONFIG_HOTPLUG_PCI_CPCI=y,
>>>>>>> CONFIG_HOTPLUG_PCI=y, CONFIG_HOTPLUG_PCI_ACPI=y)
>>> What about |CONFIG_HOTPLUG_PCI_SHPC=y ?
>>>

Thanks and I finally found it is because I set hardware reduced acpi 
before and now it is correct.

>>> |
>>>>>>> Whether this is a kind of design or a known issue? Does guest 
>>>>>>> receive
>>>>>>> the request, where can I find the
>>>>>> does it work with known to work kernels (RHEL7)?
>>>>>>
>>>>>> Also sharing used QEMU version and command line could help.
>>>>> Is there any key config of kernel in guest, besides those I listed 
>>>>> above?
>>>> Maybe Marcel knows something about it
>>>> (CCed)
>>> May I ask why do you need SHPC hotplug and not the ACPI based hotplug?
Oh, it is some customer's requests. And the method to set PIIX4_PM 
property works.


Thanks!
Jing