Re: pci-express hotplug

[Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>]

Jens Axboe wrote:
> On Tue, Oct 13 2009, Kenji Kaneshige wrote:
>> Jens Axboe wrote:
>>> On Tue, Oct 13 2009, Kenji Kaneshige wrote:
>>>> Jens Axboe wrote:
>>>>> Hi,
>>>>>
>>>>> I'm trying to get pci-express hotplug working in a box here. I don't
>>>>> really care about the hotplug aspect, I just want the darn pci-e slots
>>>>> that are designated hotplug slots to actually WORK. When I load pciehp,
>>>>> I get:
>>>>>
>>>>> Firmware did not grant requested _OSC control
>>>>> Firmware did not grant requested _OSC control
>>>>> Firmware did not grant requested _OSC control
>>>>> Firmware did not grant requested _OSC control
>>>>> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
>>>>> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
>>>>> Firmware did not grant requested _OSC control
>>>>> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>>> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
>>>>> Firmware did not grant requested _OSC control
>>>>> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>>> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
>>>>> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
>>>>> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
>>>>> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>>>>
>>>>> and the devices in the hotplug slots stay off. Is this an ACPI/bios
>>>>> issue? How can I debug this?
>>>>>
>>>> Could you give me the result of "ls -lR /sys/bus/pci/slots/"
>>>> after loading pciehp?
>>> I have attached the result of that ls prior to loading pciehp/acpiphp
>>> (pre-load), after loading pciehp (pciehp-load), and with acpiphp loaded
>>> only as well (acpiphp-load).
>>>
>> Thank you for the info. From the information, I confirmed that hotplug
>> slots are detected by pciehp even though _OSC evaluation failed. There
>> are two ways to take control from the firmware through ACPI control
>> method. One is _OSC control method, and the other is OSHP control method.
>> I guess your ACPI fimware has both _OSC and OSHP on DSDT (ACPI Namespace),
>> and pciehp assumes that it took control through OSHP after the _OSC
>> evaluation failure. I think this pciehp's behavior is wrong because of
>> the following reasons and I think pciehp driver mis-detected the hotplug
>> slots on your environment because of this.
>>
>> - According to the PCI firmware specification, pciehp driver must use the
>>  result of _OSC, if the platform implements both _OSC and OSHP.
>> - OSHP control method seems only for SHPC, not for PCI Express native hot-
>>  plug. So pciehp must not evaluate OSHP to take control from firmware.
>>
>> To confirm this, could you send me the dmesg output after loading pciehp
>> with 'debug_acpi' of pci_hotplug (PCI hotplug core driver) enabled?
>> For example,
>>
>>    $ su
>>    # echo Y > /sys/module/pci_hotplug/parameters/debug_acpi
>>    # modprobe pciehp
>>    # dmesg
> 
> See below.
> 
>> And if it is possible, could you send me DSDT of your platform?
> 
> Not sure I can do that, I'll check.
> 
>> Anyway, my recommendation is using acpiphp on your environment because
>> your firmware didn't grant control over hotplug control through _OSC.
>> From the information, acpiphp also detects the hotplug slots successfully.
>> Please try "echo 1 > /sys/bus/pci/slots/<slot#>/power". It would turn on
>> the slot and initialize adapter card on the slot.
> 
> It does find the 4 slots correctly. But if I try to turn on the power,
> nothing happens and 'power' stays at 0. If I do the same with pciehp, I
> get the same hang as described when using pciehp with pciehp_force=1.
> But apparently this machine is getting a board replacement very soon, so
> it may solve itself. Unless you think it should work and there's
> something I can try to check, then lets just leave this issue until I
> get it upgraded and return from kernel summit / JLS.
> 

Could you try pciehp with "pciehp_debug" option enabled(*), and give me
the following information?

  - "cat /sys/bus/pci/slots/*/*" output
  - dmesg output after "echo 1 > /sys/bus/pci/slots/<slot#>/power"

(*) you can enable "pciehp_debug" option as follows

  # modprobe pciehp pciehp_debug

I'm not sure, but I think one of the possibility is that your hot-plug
controller doesn't support power controller. On such slot, the slot is
usually turned on automatically at adapter card insertion. So card
re-insertion might make some difference.

By the way, I would like to know if your system has SHPC based hotplug
slots. Could you load shpchp driver and send me the
"ls -lR /sys/bus/pci/slots/" output? And I would like to see the dmesg
output with "debug_acpi" of pci_hotplug enabled. For example,

    $ su
    # echo Y > /sys/module/pci_hotplug/parameters/debug_acpi
    # modprobe shpchp
    # ls -lR /sys/bus/pci/slots
    # dmesg

> 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
> Firmware did not grant requested _OSC control
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP5 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:05.0 (\_SB_.IOH0.MRP5)
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
> Firmware did not grant requested _OSC control
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP7 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:07.0 (\_SB_.IOH0.MRP7)
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
> Firmware did not grant requested _OSC control
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.PEX0 
> acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0.PEX0 OSHP not found
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0 
> acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0 OSHP not found
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Cannot get control of hotplug hardware for pci 0000:00:1c.0
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPI 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:07.0 (\_SB_.IOH1.MRPI)
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPK 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:09.0 (\_SB_.IOH1.MRPK)
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
> Firmware did not grant requested _OSC control
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP5 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:05.0 (\_SB_.IOH0.MRP5)
> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
> Firmware did not grant requested _OSC control
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP7 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:07.0 (\_SB_.IOH0.MRP7)
> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
> Firmware did not grant requested _OSC control
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.PEX0 
> acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0.PEX0 OSHP not found
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0 
> acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0 OSHP not found
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Cannot get control of hotplug hardware for pci 0000:00:1c.0
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPI 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:07.0 (\_SB_.IOH1.MRPI)
> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 _id 340e ss_vid 0 ss_did 0
> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPK 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:09.0 (\_SB_.IOH1.MRPK)
> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
> 

It seems your environment has two PCI root bridges and one of them
(IOH0) provides both _OSC and OSHP, and the other (IOH1) provides
only OSHP. The _OSC under IOH0 doesn't grant hot-plug control, so
I think your system expects that at least the slots under IOH0 is
handled by acpiphp. But I don't have any idea about the slots
under IOH1 for now.

Thanks,
Kenji Kaneshige