pci-express hotplug

pci-express hotplug

[Jens Axboe]

Hi,

I'm trying to get pci-express hotplug working in a box here. I don't
really care about the hotplug aspect, I just want the darn pci-e slots
that are designated hotplug slots to actually WORK. When I load pciehp,
I get:

Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
Firmware did not grant requested _OSC control
pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
Firmware did not grant requested _OSC control
pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
pciehp: PCI Express Hot Plug Controller Driver version: 0.4

and the devices in the hotplug slots stay off. Is this an ACPI/bios
issue? How can I debug this?

-- 
Jens Axboe

Re: pci-express hotplug

[Greg KH]

On Mon, Oct 12, 2009 at 02:06:20PM +0200, Jens Axboe wrote:
> Hi,
> 
> I'm trying to get pci-express hotplug working in a box here. I don't
> really care about the hotplug aspect, I just want the darn pci-e slots
> that are designated hotplug slots to actually WORK. When I load pciehp,
> I get:
> 
> Firmware did not grant requested _OSC control
> Firmware did not grant requested _OSC control
> Firmware did not grant requested _OSC control
> Firmware did not grant requested _OSC control
> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
> Firmware did not grant requested _OSC control
> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
> Firmware did not grant requested _OSC control
> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
> 
> and the devices in the hotplug slots stay off. Is this an ACPI/bios
> issue? How can I debug this?

Can you try the acpiphp driver instead?  That's usually the driver you
want to use for "modern" systems (i.e. anything made in the past 5
years.)

thanks,

greg k-h

Re: pci-express hotplug

[Jens Axboe]

On Mon, Oct 12 2009, Greg KH wrote:
> On Mon, Oct 12, 2009 at 02:06:20PM +0200, Jens Axboe wrote:
> > Hi,
> > 
> > I'm trying to get pci-express hotplug working in a box here. I don't
> > really care about the hotplug aspect, I just want the darn pci-e slots
> > that are designated hotplug slots to actually WORK. When I load pciehp,
> > I get:
> > 
> > Firmware did not grant requested _OSC control
> > Firmware did not grant requested _OSC control
> > Firmware did not grant requested _OSC control
> > Firmware did not grant requested _OSC control
> > pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
> > pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
> > Firmware did not grant requested _OSC control
> > pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
> > pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
> > Firmware did not grant requested _OSC control
> > pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
> > pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
> > pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
> > pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
> > pciehp: PCI Express Hot Plug Controller Driver version: 0.4
> > 
> > and the devices in the hotplug slots stay off. Is this an ACPI/bios
> > issue? How can I debug this?
> 
> Can you try the acpiphp driver instead?  That's usually the driver you
> want to use for "modern" systems (i.e. anything made in the past 5
> years.)

I should have mentioned that I tried that too. It doesn't complain, but
I don't see my cards anywhere afterwards. I'm a hotplug newbie, do I
need to do anything else?

-- 
Jens Axboe

Re: pci-express hotplug

[Mark Lord]

Jens Axboe wrote:
> On Mon, Oct 12 2009, Greg KH wrote:
>> On Mon, Oct 12, 2009 at 02:06:20PM +0200, Jens Axboe wrote:
>>> Hi,
>>>
>>> I'm trying to get pci-express hotplug working in a box here. I don't
>>> really care about the hotplug aspect, I just want the darn pci-e slots
>>> that are designated hotplug slots to actually WORK. When I load pciehp,
>>> I get:
>>>
>>> Firmware did not grant requested _OSC control
>>> Firmware did not grant requested _OSC control
>>> Firmware did not grant requested _OSC control
>>> Firmware did not grant requested _OSC control
>>> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
>>> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
>>> Firmware did not grant requested _OSC control
>>> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
>>> Firmware did not grant requested _OSC control
>>> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
>>> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
>>> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
>>> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>>
>>> and the devices in the hotplug slots stay off. Is this an ACPI/bios
>>> issue? How can I debug this?
>> Can you try the acpiphp driver instead?  That's usually the driver you
>> want to use for "modern" systems (i.e. anything made in the past 5
>> years.)
> 
> I should have mentioned that I tried that too. It doesn't complain, but
> I don't see my cards anywhere afterwards. I'm a hotplug newbie, do I
> need to do anything else?
 
Tried this yet:

   options pciehp pciehp_force=1

??

Re: pci-express hotplug

[Jens Axboe]

On Mon, Oct 12 2009, Mark Lord wrote:
> Jens Axboe wrote:
>> On Mon, Oct 12 2009, Greg KH wrote:
>>> On Mon, Oct 12, 2009 at 02:06:20PM +0200, Jens Axboe wrote:
>>>> Hi,
>>>>
>>>> I'm trying to get pci-express hotplug working in a box here. I don't
>>>> really care about the hotplug aspect, I just want the darn pci-e slots
>>>> that are designated hotplug slots to actually WORK. When I load pciehp,
>>>> I get:
>>>>
>>>> Firmware did not grant requested _OSC control
>>>> Firmware did not grant requested _OSC control
>>>> Firmware did not grant requested _OSC control
>>>> Firmware did not grant requested _OSC control
>>>> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
>>>> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
>>>> Firmware did not grant requested _OSC control
>>>> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
>>>> Firmware did not grant requested _OSC control
>>>> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
>>>> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
>>>> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
>>>> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>>>
>>>> and the devices in the hotplug slots stay off. Is this an ACPI/bios
>>>> issue? How can I debug this?
>>> Can you try the acpiphp driver instead?  That's usually the driver you
>>> want to use for "modern" systems (i.e. anything made in the past 5
>>> years.)
>>
>> I should have mentioned that I tried that too. It doesn't complain, but
>> I don't see my cards anywhere afterwards. I'm a hotplug newbie, do I
>> need to do anything else?
>
> Tried this yet:
>
>   options pciehp pciehp_force=1

Nope, but it does find and register the hotplug slots, so I didn't think
it would make a difference. The _OSC is there.

I'll try tonight, just in case.

-- 
Jens Axboe

Re: pci-express hotplug

[Alex Chiang]

> >>> On Mon, Oct 12, 2009 at 02:06:20PM +0200, Jens Axboe wrote:
> >>>> I'm trying to get pci-express hotplug working in a box here. I don't
> >>>> really care about the hotplug aspect, I just want the darn pci-e slots
> >>>> that are designated hotplug slots to actually WORK. When I load pciehp,
> >>>> I get:
> >>>>
> >>>> Firmware did not grant requested _OSC control
> >>>> Firmware did not grant requested _OSC control
> >>>> Firmware did not grant requested _OSC control
> >>>> Firmware did not grant requested _OSC control

This isn't just a benign message. It means the OS asked to take
over control of the slots and firmware really did say, "nope,
sorry".

Which means that this:

> On Mon, Oct 12 2009, Mark Lord wrote:
> > Tried this yet:
> >
> >   options pciehp pciehp_force=1

Is generally a bad idea.

Don't do it unless you really know your platform well.

> >> On Mon, Oct 12 2009, Greg KH wrote:
> >>> Can you try the acpiphp driver instead?  That's usually the
> >>> driver you want to use for "modern" systems (i.e. anything
> >>> made in the past 5 years.)
> >>
> >> I should have mentioned that I tried that too. It doesn't
> >> complain, but I don't see my cards anywhere afterwards. I'm
> >> a hotplug newbie, do I need to do anything else?

Can you modprobe acpiphp with debug=1? And send the output?

Thanks.

/ac

Re: pci-express hotplug

[Jens Axboe]

On Mon, Oct 12 2009, Alex Chiang wrote:
> > >>> On Mon, Oct 12, 2009 at 02:06:20PM +0200, Jens Axboe wrote:
> > >>>> I'm trying to get pci-express hotplug working in a box here. I don't
> > >>>> really care about the hotplug aspect, I just want the darn pci-e slots
> > >>>> that are designated hotplug slots to actually WORK. When I load pciehp,
> > >>>> I get:
> > >>>>
> > >>>> Firmware did not grant requested _OSC control
> > >>>> Firmware did not grant requested _OSC control
> > >>>> Firmware did not grant requested _OSC control
> > >>>> Firmware did not grant requested _OSC control
> 
> This isn't just a benign message. It means the OS asked to take
> over control of the slots and firmware really did say, "nope,
> sorry".
> 
> Which means that this:
> 
> > On Mon, Oct 12 2009, Mark Lord wrote:
> > > Tried this yet:
> > >
> > >   options pciehp pciehp_force=1
> 
> Is generally a bad idea.
> 
> Don't do it unless you really know your platform well.

Since I had nothing to lose, I tried it. This is what it prints:

Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
pciehp 0000:00:05.0:pcie04: Bypassing BIOS check for pciehp use on 0000:00:05.0
pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
pciehp 0000:00:05.0:pcie04: Power fault on Slot(1)
pciehp 0000:00:05.0:pcie04: Power fault bit 0 set

and modprobe is continually in msleep() afterwards.

> > >> On Mon, Oct 12 2009, Greg KH wrote:
> > >>> Can you try the acpiphp driver instead?  That's usually the
> > >>> driver you want to use for "modern" systems (i.e. anything
> > >>> made in the past 5 years.)
> > >>
> > >> I should have mentioned that I tried that too. It doesn't
> > >> complain, but I don't see my cards anywhere afterwards. I'm
> > >> a hotplug newbie, do I need to do anything else?
> 
> Can you modprobe acpiphp with debug=1? And send the output?

acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
acpiphp: Slot [1] registered
acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
acpiphp: Slot [2] registered
acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
acpiphp: Slot [6] registered
acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
acpiphp: Slot [7] registered
acpiphp_glue: Bus 0000:87 has 1 slot
acpiphp_glue: Bus 0000:84 has 1 slot
acpiphp_glue: Bus 0000:0b has 1 slot
acpiphp_glue: Bus 0000:08 has 1 slot
acpiphp_glue: Total 4 slots

-- 
Jens Axboe

Re: pci-express hotplug

[Alex Chiang]

* Jens Axboe <jens.axboe@oracle.com>:
> > > >> On Mon, Oct 12 2009, Greg KH wrote:
> > > >>> Can you try the acpiphp driver instead?  That's usually the
> > > >>> driver you want to use for "modern" systems (i.e. anything
> > > >>> made in the past 5 years.)
> > > >>
> > > >> I should have mentioned that I tried that too. It doesn't
> > > >> complain, but I don't see my cards anywhere afterwards. I'm
> > > >> a hotplug newbie, do I need to do anything else?
> > 
> > Can you modprobe acpiphp with debug=1? And send the output?
> 
> acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
> acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
> acpiphp: Slot [1] registered
> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
> acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
> acpiphp: Slot [2] registered
> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
> acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
> acpiphp: Slot [6] registered
> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
> acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
> acpiphp: Slot [7] registered
> acpiphp_glue: Bus 0000:87 has 1 slot
> acpiphp_glue: Bus 0000:84 has 1 slot
> acpiphp_glue: Bus 0000:0b has 1 slot
> acpiphp_glue: Bus 0000:08 has 1 slot
> acpiphp_glue: Total 4 slots

You mentioned in another mail that you echoed 1 into the various
slots' power files.

Did you do that after modprobing acpiphp with debug=1?

If so, there should be debug output when you try and turn them
on.

Also, quick dummy check, you are trying to power on populated
slots, right? :)

Can you send the output of lspci -vv? And I like the output of
lspci -vt as well... Both before and after loading acpiphp
please.

Thanks.

/ac

Re: pci-express hotplug

[Jens Axboe]

On Tue, Oct 13 2009, Alex Chiang wrote:
> * Jens Axboe <jens.axboe@oracle.com>:
> > > > >> On Mon, Oct 12 2009, Greg KH wrote:
> > > > >>> Can you try the acpiphp driver instead?  That's usually the
> > > > >>> driver you want to use for "modern" systems (i.e. anything
> > > > >>> made in the past 5 years.)
> > > > >>
> > > > >> I should have mentioned that I tried that too. It doesn't
> > > > >> complain, but I don't see my cards anywhere afterwards. I'm
> > > > >> a hotplug newbie, do I need to do anything else?
> > > 
> > > Can you modprobe acpiphp with debug=1? And send the output?
> > 
> > acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
> > acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
> > acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
> > acpiphp: Slot [1] registered
> > acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
> > acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
> > acpiphp: Slot [2] registered
> > acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
> > acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
> > acpiphp: Slot [6] registered
> > acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
> > acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
> > acpiphp: Slot [7] registered
> > acpiphp_glue: Bus 0000:87 has 1 slot
> > acpiphp_glue: Bus 0000:84 has 1 slot
> > acpiphp_glue: Bus 0000:0b has 1 slot
> > acpiphp_glue: Bus 0000:08 has 1 slot
> > acpiphp_glue: Total 4 slots
> 
> You mentioned in another mail that you echoed 1 into the various
> slots' power files.
> 
> Did you do that after modprobing acpiphp with debug=1?
> 
> If so, there should be debug output when you try and turn them
> on.

It produces:

acpiphp: enable_slot - physical_slot = 1
acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
acpiphp: enable_slot - physical_slot = 2
acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
acpiphp: enable_slot - physical_slot = 6
acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
acpiphp: enable_slot - physical_slot = 7
acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL

I have a card in one of the slots only this time.

> Also, quick dummy check, you are trying to power on populated
> slots, right? :)

Yes :-)

> Can you send the output of lspci -vv? And I like the output of
> lspci -vt as well... Both before and after loading acpiphp
> please.

Send privately.

-- 
Jens Axboe

Re: pci-express hotplug

[Alex Chiang]

* Jens Axboe <jens.axboe@oracle.com>:
> On Tue, Oct 13 2009, Alex Chiang wrote:
> > > > Can you modprobe acpiphp with debug=1? And send the output?
> > > 
> > > acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
> > > acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
> > > acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
> > > acpiphp: Slot [1] registered
> > > acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
> > > acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
> > > acpiphp: Slot [2] registered
> > > acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
> > > acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
> > > acpiphp: Slot [6] registered
> > > acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
> > > acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
> > > acpiphp: Slot [7] registered
> > > acpiphp_glue: Bus 0000:87 has 1 slot
> > > acpiphp_glue: Bus 0000:84 has 1 slot
> > > acpiphp_glue: Bus 0000:0b has 1 slot
> > > acpiphp_glue: Bus 0000:08 has 1 slot
> > > acpiphp_glue: Total 4 slots
> > 
> > You mentioned in another mail that you echoed 1 into the various
> > slots' power files.
> > 
> > Did you do that after modprobing acpiphp with debug=1?
> > 
> > If so, there should be debug output when you try and turn them
> > on.
> 
> It produces:
> 
> acpiphp: enable_slot - physical_slot = 1
> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> acpiphp: enable_slot - physical_slot = 2
> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> acpiphp: enable_slot - physical_slot = 6
> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> acpiphp: enable_slot - physical_slot = 7
> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL

Hm, so for some reason, firmware on your machine is telling us
that it doesn't think cards are present and/or enabled.

Unfortunately, I don't know why your firmware would be saying
that. We could add some more debug printks to see what firmware
thinks about your system... Or we could just wait and see what
happens after you get your hardware replaced.

> I have a card in one of the slots only this time.
> 
> > Also, quick dummy check, you are trying to power on populated
> > slots, right? :)
> 
> Yes :-)
> 
> > Can you send the output of lspci -vv? And I like the output of
> > lspci -vt as well... Both before and after loading acpiphp
> > please.
> 
> Send privately.

No difference in before and after. Odd.

If you want to poke us again after your hardware swap, please do
so. Sorry for being not so helpful. :-/

/ac

Re: pci-express hotplug

[Jens Axboe]

On Tue, Oct 20 2009, Alex Chiang wrote:
> * Jens Axboe <jens.axboe@oracle.com>:
> > On Tue, Oct 13 2009, Alex Chiang wrote:
> > > > > Can you modprobe acpiphp with debug=1? And send the output?
> > > > 
> > > > acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
> > > > acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
> > > > acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
> > > > acpiphp: Slot [1] registered
> > > > acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
> > > > acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
> > > > acpiphp: Slot [2] registered
> > > > acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
> > > > acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
> > > > acpiphp: Slot [6] registered
> > > > acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
> > > > acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
> > > > acpiphp: Slot [7] registered
> > > > acpiphp_glue: Bus 0000:87 has 1 slot
> > > > acpiphp_glue: Bus 0000:84 has 1 slot
> > > > acpiphp_glue: Bus 0000:0b has 1 slot
> > > > acpiphp_glue: Bus 0000:08 has 1 slot
> > > > acpiphp_glue: Total 4 slots
> > > 
> > > You mentioned in another mail that you echoed 1 into the various
> > > slots' power files.
> > > 
> > > Did you do that after modprobing acpiphp with debug=1?
> > > 
> > > If so, there should be debug output when you try and turn them
> > > on.
> > 
> > It produces:
> > 
> > acpiphp: enable_slot - physical_slot = 1
> > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > acpiphp: enable_slot - physical_slot = 2
> > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > acpiphp: enable_slot - physical_slot = 6
> > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > acpiphp: enable_slot - physical_slot = 7
> > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> 
> Hm, so for some reason, firmware on your machine is telling us
> that it doesn't think cards are present and/or enabled.
> 
> Unfortunately, I don't know why your firmware would be saying
> that. We could add some more debug printks to see what firmware
> thinks about your system... Or we could just wait and see what
> happens after you get your hardware replaced.

New board, the exact same thing happens.

> > I have a card in one of the slots only this time.
> > 
> > > Also, quick dummy check, you are trying to power on populated
> > > slots, right? :)
> > 
> > Yes :-)
> > 
> > > Can you send the output of lspci -vv? And I like the output of
> > > lspci -vt as well... Both before and after loading acpiphp
> > > please.
> > 
> > Send privately.
> 
> No difference in before and after. Odd.
> 
> If you want to poke us again after your hardware swap, please do
> so. Sorry for being not so helpful. :-/

Poke :-)

One more thing I tried was pushing the power button on the slot
manually. With acpiphp, I get the same messages as above. Using pciehp,
I get the same power fault bit interrupt storm. So no difference from
using the sysfs interface or doing it on the box side, doesn't work
either way.

-- 
Jens Axboe

Re: pci-express hotplug

[Alex Chiang]

* Jens Axboe <jens.axboe@oracle.com>:
> > > acpiphp: enable_slot - physical_slot = 1
> > > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > > acpiphp: enable_slot - physical_slot = 2
> > > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > > acpiphp: enable_slot - physical_slot = 6
> > > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > > acpiphp: enable_slot - physical_slot = 7
> > > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > 
> > Hm, so for some reason, firmware on your machine is telling us
> > that it doesn't think cards are present and/or enabled.
> > 
> > Unfortunately, I don't know why your firmware would be saying
> > that. We could add some more debug printks to see what firmware
> > thinks about your system... Or we could just wait and see what
> > happens after you get your hardware replaced.

Let's try and find out why firmware is telling us that we didn't
get ACPI_STA_ALL.

Can you please apply this debug patch and send the output? Again,
please modprobe with debug=1.

Thanks,
/ac

---
diff --git a/drivers/pci/hotplug/acpiphp_glue.c b/drivers/pci/hotplug/acpiphp_glue.c
index 58d25a1..2caa447 100644
--- a/drivers/pci/hotplug/acpiphp_glue.c
+++ b/drivers/pci/hotplug/acpiphp_glue.c
@@ -797,9 +797,13 @@ static int power_on_slot(struct acpiphp_slot *slot)
 	struct list_head *l;
 	int retval = 0;
 
+	printk("%s\n", __func__);
+
 	/* if already enabled, just skip */
-	if (slot->flags & SLOT_POWEREDON)
+	if (slot->flags & SLOT_POWEREDON) {
+		printk("  slot %ld already powered on\n", slot->sun);
 		goto err_exit;
+	}
 
 	list_for_each (l, &slot->funcs) {
 		func = list_entry(l, struct acpiphp_func, sibling);
@@ -813,6 +817,8 @@ static int power_on_slot(struct acpiphp_slot *slot)
 				goto err_exit;
 			} else
 				break;
+		} else {
+			printk("  no _PS0\n");
 		}
 	}
 
@@ -1122,11 +1128,14 @@ static unsigned int get_slot_status(struct acpiphp_slot *slot)
 	struct list_head *l;
 	struct acpiphp_func *func;
 
+	printk("%s\n", __func__);
+
 	list_for_each (l, &slot->funcs) {
 		func = list_entry(l, struct acpiphp_func, sibling);
 
 		if (func->flags & FUNC_HAS_STA) {
 			status = acpi_evaluate_integer(func->handle, "_STA", NULL, &sta);
+			printk("  FUNC_HAS_STA status %d _STA %#lx\n", status, sta);
 			if (ACPI_SUCCESS(status) && sta)
 				break;
 		} else {
@@ -1134,6 +1143,7 @@ static unsigned int get_slot_status(struct acpiphp_slot *slot)
 						  PCI_DEVFN(slot->device,
 							    func->function),
 						  PCI_VENDOR_ID, &dvid);
+			printk("  reading config space dvid %#lx\n", dvid);
 			if (dvid != 0xffffffff) {
 				sta = ACPI_STA_ALL;
 				break;

Re: pci-express hotplug

[Jens Axboe]

On Mon, Oct 26 2009, Alex Chiang wrote:
> * Jens Axboe <jens.axboe@oracle.com>:
> > > > acpiphp: enable_slot - physical_slot = 1
> > > > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > > > acpiphp: enable_slot - physical_slot = 2
> > > > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > > > acpiphp: enable_slot - physical_slot = 6
> > > > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > > > acpiphp: enable_slot - physical_slot = 7
> > > > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > > 
> > > Hm, so for some reason, firmware on your machine is telling us
> > > that it doesn't think cards are present and/or enabled.
> > > 
> > > Unfortunately, I don't know why your firmware would be saying
> > > that. We could add some more debug printks to see what firmware
> > > thinks about your system... Or we could just wait and see what
> > > happens after you get your hardware replaced.
> 
> Let's try and find out why firmware is telling us that we didn't
> get ACPI_STA_ALL.
> 
> Can you please apply this debug patch and send the output? Again,
> please modprobe with debug=1.

acpiphp: enable_slot - physical_slot = 1
power_on_slot
  no _PS0
  no _PS0
  no _PS0
  no _PS0
  no _PS0
  no _PS0
  no _PS0
  no _PS0
get_slot_status
  reading config space dvid 0xffffffff
  reading config space dvid 0xffffffff
  reading config space dvid 0xffffffff
  reading config space dvid 0xffffffff
  reading config space dvid 0xffffffff
  reading config space dvid 0xffffffff
  reading config space dvid 0xffffffff
  reading config space dvid 0xffffffff
acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL

-- 
Jens Axboe

Re: pci-express hotplug

[Jens Axboe]

On Tue, Oct 27 2009, Jens Axboe wrote:
> On Mon, Oct 26 2009, Alex Chiang wrote:
> > * Jens Axboe <jens.axboe@oracle.com>:
> > > > > acpiphp: enable_slot - physical_slot = 1
> > > > > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > > > > acpiphp: enable_slot - physical_slot = 2
> > > > > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > > > > acpiphp: enable_slot - physical_slot = 6
> > > > > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > > > > acpiphp: enable_slot - physical_slot = 7
> > > > > acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> > > > 
> > > > Hm, so for some reason, firmware on your machine is telling us
> > > > that it doesn't think cards are present and/or enabled.
> > > > 
> > > > Unfortunately, I don't know why your firmware would be saying
> > > > that. We could add some more debug printks to see what firmware
> > > > thinks about your system... Or we could just wait and see what
> > > > happens after you get your hardware replaced.
> > 
> > Let's try and find out why firmware is telling us that we didn't
> > get ACPI_STA_ALL.
> > 
> > Can you please apply this debug patch and send the output? Again,
> > please modprobe with debug=1.
> 
> acpiphp: enable_slot - physical_slot = 1
> power_on_slot
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
> get_slot_status
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL

Since this is a new board and BIOS, below is the info from loading
acpiphp with debug enabled and acpi debug enabled.

acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
get_slot_status
get_slot_status
acpiphp: Slot [1] registered
acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
get_slot_status
get_slot_status
acpiphp: Slot [2] registered
acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
get_slot_status
get_slot_status
acpiphp: Slot [6] registered
acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
get_slot_status
get_slot_status
acpiphp: Slot [7] registered
acpiphp_glue: Bus 0000:87 has 1 slot
acpiphp_glue: Bus 0000:84 has 1 slot
acpiphp_glue: Bus 0000:0b has 1 slot
acpiphp_glue: Bus 0000:08 has 1 slot
acpiphp_glue: Total 4 slots
acpiphp: Slot [1] unregistered
acpiphp: release_slot - physical_slot = 1
acpiphp: Slot [2] unregistered
acpiphp: release_slot - physical_slot = 2
acpiphp: Slot [6] unregistered
acpiphp: release_slot - physical_slot = 6
acpiphp: Slot [7] unregistered
acpiphp: release_slot - physical_slot = 7

-- 
Jens Axboe

Re: pci-express hotplug

[Alex Chiang]

* Jens Axboe <jens.axboe@oracle.com>:
> 
> Since this is a new board and BIOS, below is the info from loading
> acpiphp with debug enabled and acpi debug enabled.

Thanks. Can you please send your DSDT as well please?

You can obtain that with the acpidump tools. If they're not part
of your distro, you can find them on lesswatts.org.

Thanks.

/ac

Re: pci-express hotplug

[Jens Axboe]

On Tue, Oct 27 2009, Alex Chiang wrote:
> * Jens Axboe <jens.axboe@oracle.com>:
> > 
> > Since this is a new board and BIOS, below is the info from loading
> > acpiphp with debug enabled and acpi debug enabled.
> 
> Thanks. Can you please send your DSDT as well please?
> 
> You can obtain that with the acpidump tools. If they're not part
> of your distro, you can find them on lesswatts.org.

Sent privately.

-- 
Jens Axboe

Re: pci-express hotplug

[Alex Chiang]

* Jens Axboe <jens.axboe@oracle.com>:
> On Tue, Oct 27 2009, Alex Chiang wrote:
> > * Jens Axboe <jens.axboe@oracle.com>:
> > > 
> > > Since this is a new board and BIOS, below is the info from loading
> > > acpiphp with debug enabled and acpi debug enabled.
> > 
> > Thanks. Can you please send your DSDT as well please?
> > 
> > You can obtain that with the acpidump tools. If they're not part
> > of your distro, you can find them on lesswatts.org.
> 
> Sent privately.

Sorry, one more RTT of debug info needed. :-/

Can you send dmesg output and contents of /proc/iomem?

Thanks.

Re: pci-express hotplug

[Jens Axboe]

Just a note for the archives - after chatting with Alex on irc about
this issue and trying other cards, the likely suspect seems to be the
specific card used and/or the firmware on that card. Hotplug works
otherwise, just not with that card at least.

-- 
Jens Axboe

Re: pci-express hotplug

[Alex Chiang]

* Jens Axboe <jens.axboe@oracle.com>:
> 
> acpiphp: enable_slot - physical_slot = 1
> power_on_slot
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
> get_slot_status
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
>   reading config space dvid 0xffffffff
> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL

Hm, as Kenji-san writes in an earlier email:

	The direct cause of the problem that your slot was not
	turned on is power fault. I guess acpiphp is suffering
	the same problem.  Unfortunately, it's difficult for me
	to analyze the root cause of this power fault. Please ask
	the hardware vendor about it. I hope board replacement
	will fix the problem.

In get_slot_status(), we're trying to read the card's vendor ID,
which is a mandatory PCI config space register. The fact that we
can't even read that suggests something is going wrong way
earlier before we get to this point.

Bjorn wondered on irc if your slots are physically working. Do
you know if they work under Windows? If they do, then it would be
good to find out how your bridges are being programmed, which I
believe you can discover with the Device Manager.

/ac

Re: pci-express hotplug

[Alex Chiang]

* Jens Axboe <jens.axboe@oracle.com>:
> 
> acpiphp: enable_slot - physical_slot = 1
> power_on_slot
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0
>   no _PS0

One final thought -- your DSDT doesn't provide any power methods
such as _PS[0-3] (I grepped your DSDT so basing my statement on
more than just the output above), and without those, I'm pretty
sure that there's no way for the OS to communicate to the BIOS
that we want to power those slots on.

So, something funky is going on with your BIOS. This isn't some
weird proto board or something, is it? ;)

/ac

Re: pci-express hotplug

[Jens Axboe]

On Wed, Oct 28 2009, Alex Chiang wrote:
> * Jens Axboe <jens.axboe@oracle.com>:
> > 
> > acpiphp: enable_slot - physical_slot = 1
> > power_on_slot
> >   no _PS0
> >   no _PS0
> >   no _PS0
> >   no _PS0
> >   no _PS0
> >   no _PS0
> >   no _PS0
> >   no _PS0
> 
> One final thought -- your DSDT doesn't provide any power methods
> such as _PS[0-3] (I grepped your DSDT so basing my statement on
> more than just the output above), and without those, I'm pretty
> sure that there's no way for the OS to communicate to the BIOS
> that we want to power those slots on.
> 
> So, something funky is going on with your BIOS. This isn't some
> weird proto board or something, is it? ;)

It's pre-production, but not a prototype. I'll take it up with the
vendor.

-- 
Jens Axboe

Re: pci-express hotplug

[Kenji Kaneshige]

Jens Axboe wrote:
> On Tue, Oct 20 2009, Alex Chiang wrote:
>> * Jens Axboe <jens.axboe@oracle.com>:
>>> On Tue, Oct 13 2009, Alex Chiang wrote:
>>>>>> Can you modprobe acpiphp with debug=1? And send the output?
>>>>> acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
>>>>> acpiphp: Slot [1] registered
>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
>>>>> acpiphp: Slot [2] registered
>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
>>>>> acpiphp: Slot [6] registered
>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
>>>>> acpiphp: Slot [7] registered
>>>>> acpiphp_glue: Bus 0000:87 has 1 slot
>>>>> acpiphp_glue: Bus 0000:84 has 1 slot
>>>>> acpiphp_glue: Bus 0000:0b has 1 slot
>>>>> acpiphp_glue: Bus 0000:08 has 1 slot
>>>>> acpiphp_glue: Total 4 slots
>>>> You mentioned in another mail that you echoed 1 into the various
>>>> slots' power files.
>>>>
>>>> Did you do that after modprobing acpiphp with debug=1?
>>>>
>>>> If so, there should be debug output when you try and turn them
>>>> on.
>>> It produces:
>>>
>>> acpiphp: enable_slot - physical_slot = 1
>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>> acpiphp: enable_slot - physical_slot = 2
>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>> acpiphp: enable_slot - physical_slot = 6
>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>> acpiphp: enable_slot - physical_slot = 7
>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>> Hm, so for some reason, firmware on your machine is telling us
>> that it doesn't think cards are present and/or enabled.
>>
>> Unfortunately, I don't know why your firmware would be saying
>> that. We could add some more debug printks to see what firmware
>> thinks about your system... Or we could just wait and see what
>> happens after you get your hardware replaced.
> 
> New board, the exact same thing happens.
> 
>>> I have a card in one of the slots only this time.
>>>
>>>> Also, quick dummy check, you are trying to power on populated
>>>> slots, right? :)
>>> Yes :-)
>>>
>>>> Can you send the output of lspci -vv? And I like the output of
>>>> lspci -vt as well... Both before and after loading acpiphp
>>>> please.
>>> Send privately.
>> No difference in before and after. Odd.
>>
>> If you want to poke us again after your hardware swap, please do
>> so. Sorry for being not so helpful. :-/
> 
> Poke :-)
> 
> One more thing I tried was pushing the power button on the slot
> manually. With acpiphp, I get the same messages as above. Using pciehp,
> I get the same power fault bit interrupt storm. So no difference from
> using the sysfs interface or doing it on the box side, doesn't work
> either way.
> 

I'd like to confirm power fault interrupt storm, just in case.
Could you get /proc/interrupts information after power fault
problem happens and send it to me?

Thanks,
Kenji Kaneshige

Re: pci-express hotplug

[Jens Axboe]

On Tue, Oct 27 2009, Kenji Kaneshige wrote:
> Jens Axboe wrote:
>> On Tue, Oct 20 2009, Alex Chiang wrote:
>>> * Jens Axboe <jens.axboe@oracle.com>:
>>>> On Tue, Oct 13 2009, Alex Chiang wrote:
>>>>>>> Can you modprobe acpiphp with debug=1? And send the output?
>>>>>> acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
>>>>>> acpiphp: Slot [1] registered
>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
>>>>>> acpiphp: Slot [2] registered
>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
>>>>>> acpiphp: Slot [6] registered
>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
>>>>>> acpiphp: Slot [7] registered
>>>>>> acpiphp_glue: Bus 0000:87 has 1 slot
>>>>>> acpiphp_glue: Bus 0000:84 has 1 slot
>>>>>> acpiphp_glue: Bus 0000:0b has 1 slot
>>>>>> acpiphp_glue: Bus 0000:08 has 1 slot
>>>>>> acpiphp_glue: Total 4 slots
>>>>> You mentioned in another mail that you echoed 1 into the various
>>>>> slots' power files.
>>>>>
>>>>> Did you do that after modprobing acpiphp with debug=1?
>>>>>
>>>>> If so, there should be debug output when you try and turn them
>>>>> on.
>>>> It produces:
>>>>
>>>> acpiphp: enable_slot - physical_slot = 1
>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>> acpiphp: enable_slot - physical_slot = 2
>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>> acpiphp: enable_slot - physical_slot = 6
>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>> acpiphp: enable_slot - physical_slot = 7
>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>> Hm, so for some reason, firmware on your machine is telling us
>>> that it doesn't think cards are present and/or enabled.
>>>
>>> Unfortunately, I don't know why your firmware would be saying
>>> that. We could add some more debug printks to see what firmware
>>> thinks about your system... Or we could just wait and see what
>>> happens after you get your hardware replaced.
>>
>> New board, the exact same thing happens.
>>
>>>> I have a card in one of the slots only this time.
>>>>
>>>>> Also, quick dummy check, you are trying to power on populated
>>>>> slots, right? :)
>>>> Yes :-)
>>>>
>>>>> Can you send the output of lspci -vv? And I like the output of
>>>>> lspci -vt as well... Both before and after loading acpiphp
>>>>> please.
>>>> Send privately.
>>> No difference in before and after. Odd.
>>>
>>> If you want to poke us again after your hardware swap, please do
>>> so. Sorry for being not so helpful. :-/
>>
>> Poke :-)
>>
>> One more thing I tried was pushing the power button on the slot
>> manually. With acpiphp, I get the same messages as above. Using pciehp,
>> I get the same power fault bit interrupt storm. So no difference from
>> using the sysfs interface or doing it on the box side, doesn't work
>> either way.
>>
>
> I'd like to confirm power fault interrupt storm, just in case.
> Could you get /proc/interrupts information after power fault
> problem happens and send it to me?

The box pretty much hangs when I try to power on a slot with pciehp, so
it's not easy to do... It doesn't hang with acpiphp, but doesn't work
either (see previous reply to Alex).

-- 
Jens Axboe

Re: pci-express hotplug

[Jens Axboe]

On Tue, Oct 27 2009, Jens Axboe wrote:
> On Tue, Oct 27 2009, Kenji Kaneshige wrote:
> > Jens Axboe wrote:
> >> On Tue, Oct 20 2009, Alex Chiang wrote:
> >>> * Jens Axboe <jens.axboe@oracle.com>:
> >>>> On Tue, Oct 13 2009, Alex Chiang wrote:
> >>>>>>> Can you modprobe acpiphp with debug=1? And send the output?
> >>>>>> acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
> >>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
> >>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
> >>>>>> acpiphp: Slot [1] registered
> >>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
> >>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
> >>>>>> acpiphp: Slot [2] registered
> >>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
> >>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
> >>>>>> acpiphp: Slot [6] registered
> >>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
> >>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
> >>>>>> acpiphp: Slot [7] registered
> >>>>>> acpiphp_glue: Bus 0000:87 has 1 slot
> >>>>>> acpiphp_glue: Bus 0000:84 has 1 slot
> >>>>>> acpiphp_glue: Bus 0000:0b has 1 slot
> >>>>>> acpiphp_glue: Bus 0000:08 has 1 slot
> >>>>>> acpiphp_glue: Total 4 slots
> >>>>> You mentioned in another mail that you echoed 1 into the various
> >>>>> slots' power files.
> >>>>>
> >>>>> Did you do that after modprobing acpiphp with debug=1?
> >>>>>
> >>>>> If so, there should be debug output when you try and turn them
> >>>>> on.
> >>>> It produces:
> >>>>
> >>>> acpiphp: enable_slot - physical_slot = 1
> >>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> >>>> acpiphp: enable_slot - physical_slot = 2
> >>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> >>>> acpiphp: enable_slot - physical_slot = 6
> >>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> >>>> acpiphp: enable_slot - physical_slot = 7
> >>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
> >>> Hm, so for some reason, firmware on your machine is telling us
> >>> that it doesn't think cards are present and/or enabled.
> >>>
> >>> Unfortunately, I don't know why your firmware would be saying
> >>> that. We could add some more debug printks to see what firmware
> >>> thinks about your system... Or we could just wait and see what
> >>> happens after you get your hardware replaced.
> >>
> >> New board, the exact same thing happens.
> >>
> >>>> I have a card in one of the slots only this time.
> >>>>
> >>>>> Also, quick dummy check, you are trying to power on populated
> >>>>> slots, right? :)
> >>>> Yes :-)
> >>>>
> >>>>> Can you send the output of lspci -vv? And I like the output of
> >>>>> lspci -vt as well... Both before and after loading acpiphp
> >>>>> please.
> >>>> Send privately.
> >>> No difference in before and after. Odd.
> >>>
> >>> If you want to poke us again after your hardware swap, please do
> >>> so. Sorry for being not so helpful. :-/
> >>
> >> Poke :-)
> >>
> >> One more thing I tried was pushing the power button on the slot
> >> manually. With acpiphp, I get the same messages as above. Using pciehp,
> >> I get the same power fault bit interrupt storm. So no difference from
> >> using the sysfs interface or doing it on the box side, doesn't work
> >> either way.
> >>
> >
> > I'd like to confirm power fault interrupt storm, just in case.
> > Could you get /proc/interrupts information after power fault
> > problem happens and send it to me?
> 
> The box pretty much hangs when I try to power on a slot with pciehp, so
> it's not easy to do... It doesn't hang with acpiphp, but doesn't work
> either (see previous reply to Alex).

Ditto new debug info, in case it is of any assistance. Loading pciehp
with pciehp_debug=1.

acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
Firmware did not grant requested _OSC control
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP5 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:05.0 (\_SB_.IOH0.MRP5)
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
Firmware did not grant requested _OSC control
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP7 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:07.0 (\_SB_.IOH0.MRP7)
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
Firmware did not grant requested _OSC control
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.PEX0 
acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0.PEX0 OSHP not found
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0 
acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0 OSHP not found
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Cannot get control of hotplug hardware for pci 0000:00:1c.0
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPI 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:07.0 (\_SB_.IOH1.MRPI)
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPK 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:09.0 (\_SB_.IOH1.MRPK)
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
Firmware did not grant requested _OSC control
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP5 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:05.0 (\_SB_.IOH0.MRP5)
pciehp 0000:00:05.0:pcie04: Hotplug Controller:
pciehp 0000:00:05.0:pcie04:   Seg/Bus/Dev/Func/IRQ : 0000:00:05.0 IRQ 75
pciehp 0000:00:05.0:pcie04:   Vendor ID            : 0x8086
pciehp 0000:00:05.0:pcie04:   Device ID            : 0x340c
pciehp 0000:00:05.0:pcie04:   Subsystem ID         : 0x0000
pciehp 0000:00:05.0:pcie04:   Subsystem Vendor ID  : 0x0000
pciehp 0000:00:05.0:pcie04:   PCIe Cap offset      : 0x90
pciehp 0000:00:05.0:pcie04:   PCI resource [7]     : 0x1000@0x1000
pciehp 0000:00:05.0:pcie04:   PCI resource [8]     : 0x200000@0x91a00000
pciehp 0000:00:05.0:pcie04:   PCI resource [9]     : 0x200000@0x91c00000
pciehp 0000:00:05.0:pcie04: Slot Capabilities      : 0x0008005b
pciehp 0000:00:05.0:pcie04:   Physical Slot Number : 1
pciehp 0000:00:05.0:pcie04:   Attention Button     : yes
pciehp 0000:00:05.0:pcie04:   Power Controller     : yes
pciehp 0000:00:05.0:pcie04:   MRL Sensor           :  no
pciehp 0000:00:05.0:pcie04:   Attention Indicator  : yes
pciehp 0000:00:05.0:pcie04:   Power Indicator      : yes
pciehp 0000:00:05.0:pcie04:   Hot-Plug Surprise    :  no
pciehp 0000:00:05.0:pcie04:   EMI Present          :  no
pciehp 0000:00:05.0:pcie04:   Command Completed    : yes
pciehp 0000:00:05.0:pcie04: Slot Status            : 0x0040
pciehp 0000:00:05.0:pcie04: Slot Control           : 0x07c0
pciehp 0000:00:05.0:pcie04: Link Active Reporting supported
pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
pciehp 0000:00:05.0:pcie04: Registering domain:bus:dev=0000:08:00 sun=1
pciehp 0000:00:05.0:pcie04: get_power_status: physical_slot = 1
pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 7c0
pciehp 0000:00:05.0:pcie04: get_attention_status: physical_slot = 1
pciehp 0000:00:05.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 7c0
pciehp 0000:00:05.0:pcie04: get_latch_status: physical_slot = 1
pciehp 0000:00:05.0:pcie04: get_adapter_status: physical_slot = 1
pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
Firmware did not grant requested _OSC control
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP7 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:07.0 (\_SB_.IOH0.MRP7)
pciehp 0000:00:07.0:pcie04: Hotplug Controller:
pciehp 0000:00:07.0:pcie04:   Seg/Bus/Dev/Func/IRQ : 0000:00:07.0 IRQ 76
pciehp 0000:00:07.0:pcie04:   Vendor ID            : 0x8086
pciehp 0000:00:07.0:pcie04:   Device ID            : 0x340e
pciehp 0000:00:07.0:pcie04:   Subsystem ID         : 0x0000
pciehp 0000:00:07.0:pcie04:   Subsystem Vendor ID  : 0x0000
pciehp 0000:00:07.0:pcie04:   PCIe Cap offset      : 0x90
pciehp 0000:00:07.0:pcie04:   PCI resource [7]     : 0x1000@0x2000
pciehp 0000:00:07.0:pcie04:   PCI resource [8]     : 0x200000@0x91e00000
pciehp 0000:00:07.0:pcie04:   PCI resource [9]     : 0x200000@0x92000000
pciehp 0000:00:07.0:pcie04: Slot Capabilities      : 0x0010005b
pciehp 0000:00:07.0:pcie04:   Physical Slot Number : 2
pciehp 0000:00:07.0:pcie04:   Attention Button     : yes
pciehp 0000:00:07.0:pcie04:   Power Controller     : yes
pciehp 0000:00:07.0:pcie04:   MRL Sensor           :  no
pciehp 0000:00:07.0:pcie04:   Attention Indicator  : yes
pciehp 0000:00:07.0:pcie04:   Power Indicator      : yes
pciehp 0000:00:07.0:pcie04:   Hot-Plug Surprise    :  no
pciehp 0000:00:07.0:pcie04:   EMI Present          :  no
pciehp 0000:00:07.0:pcie04:   Command Completed    : yes
pciehp 0000:00:07.0:pcie04: Slot Status            : 0x0000
pciehp 0000:00:07.0:pcie04: Slot Control           : 0x07c0
pciehp 0000:00:07.0:pcie04: Link Active Reporting supported
pciehp 0000:00:07.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
pciehp 0000:00:07.0:pcie04: Registering domain:bus:dev=0000:0b:00 sun=2
pciehp 0000:00:07.0:pcie04: get_power_status: physical_slot = 2
pciehp 0000:00:07.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 7c0
pciehp 0000:00:07.0:pcie04: get_attention_status: physical_slot = 2
pciehp 0000:00:07.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 7c0
pciehp 0000:00:07.0:pcie04: get_latch_status: physical_slot = 2
pciehp 0000:00:07.0:pcie04: get_adapter_status: physical_slot = 2
pciehp 0000:00:07.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:07.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:07.0:pcie04: pciehp_power_off_slot: SLOTCTRL a8 write cmd 400
pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
Firmware did not grant requested _OSC control
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.PEX0 
acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0.PEX0 OSHP not found
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0 
acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0 OSHP not found
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Cannot get control of hotplug hardware for pci 0000:00:1c.0
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPI 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:07.0 (\_SB_.IOH1.MRPI)
pciehp 0000:80:07.0:pcie04: Hotplug Controller:
pciehp 0000:80:07.0:pcie04:   Seg/Bus/Dev/Func/IRQ : 0000:80:07.0 IRQ 84
pciehp 0000:80:07.0:pcie04:   Vendor ID            : 0x8086
pciehp 0000:80:07.0:pcie04:   Device ID            : 0x340e
pciehp 0000:80:07.0:pcie04:   Subsystem ID         : 0x0000
pciehp 0000:80:07.0:pcie04:   Subsystem Vendor ID  : 0x0000
pciehp 0000:80:07.0:pcie04:   PCIe Cap offset      : 0x90
pciehp 0000:80:07.0:pcie04:   PCI resource [7]     : 0x1000@0x8000
pciehp 0000:80:07.0:pcie04:   PCI resource [8]     : 0x200000@0x92700000
pciehp 0000:80:07.0:pcie04:   PCI resource [9]     : 0x200000@0x92900000
pciehp 0000:80:07.0:pcie04: Slot Capabilities      : 0x0030005b
pciehp 0000:80:07.0:pcie04:   Physical Slot Number : 6
pciehp 0000:80:07.0:pcie04:   Attention Button     : yes
pciehp 0000:80:07.0:pcie04:   Power Controller     : yes
pciehp 0000:80:07.0:pcie04:   MRL Sensor           :  no
pciehp 0000:80:07.0:pcie04:   Attention Indicator  : yes
pciehp 0000:80:07.0:pcie04:   Power Indicator      : yes
pciehp 0000:80:07.0:pcie04:   Hot-Plug Surprise    :  no
pciehp 0000:80:07.0:pcie04:   EMI Present          :  no
pciehp 0000:80:07.0:pcie04:   Command Completed    : yes
pciehp 0000:80:07.0:pcie04: Slot Status            : 0x0040
pciehp 0000:80:07.0:pcie04: Slot Control           : 0x0740
pciehp 0000:80:07.0:pcie04: Link Active Reporting supported
pciehp 0000:80:07.0:pcie04: Command not completed in 1000 msec
pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
pciehp 0000:80:07.0:pcie04: Registering domain:bus:dev=0000:84:00 sun=6
pciehp 0000:80:07.0:pcie04: get_power_status: physical_slot = 6
pciehp 0000:80:07.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 740
pciehp 0000:80:07.0:pcie04: get_attention_status: physical_slot = 6
pciehp 0000:80:07.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 740
pciehp 0000:80:07.0:pcie04: get_latch_status: physical_slot = 6
pciehp 0000:80:07.0:pcie04: get_adapter_status: physical_slot = 6
pciehp 0000:80:07.0:pcie04: Command not completed in 1000 msec
pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPK 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:09.0 (\_SB_.IOH1.MRPK)
pciehp 0000:80:09.0:pcie04: Hotplug Controller:
pciehp 0000:80:09.0:pcie04:   Seg/Bus/Dev/Func/IRQ : 0000:80:09.0 IRQ 85
pciehp 0000:80:09.0:pcie04:   Vendor ID            : 0x8086
pciehp 0000:80:09.0:pcie04:   Device ID            : 0x3410
pciehp 0000:80:09.0:pcie04:   Subsystem ID         : 0x0000
pciehp 0000:80:09.0:pcie04:   Subsystem Vendor ID  : 0x0000
pciehp 0000:80:09.0:pcie04:   PCIe Cap offset      : 0x90
pciehp 0000:80:09.0:pcie04:   PCI resource [7]     : 0x1000@0x9000
pciehp 0000:80:09.0:pcie04:   PCI resource [8]     : 0x200000@0x92b00000
pciehp 0000:80:09.0:pcie04:   PCI resource [9]     : 0x200000@0x92d00000
pciehp 0000:80:09.0:pcie04: Slot Capabilities      : 0x0038005b
pciehp 0000:80:09.0:pcie04:   Physical Slot Number : 7
pciehp 0000:80:09.0:pcie04:   Attention Button     : yes
pciehp 0000:80:09.0:pcie04:   Power Controller     : yes
pciehp 0000:80:09.0:pcie04:   MRL Sensor           :  no
pciehp 0000:80:09.0:pcie04:   Attention Indicator  : yes
pciehp 0000:80:09.0:pcie04:   Power Indicator      : yes
pciehp 0000:80:09.0:pcie04:   Hot-Plug Surprise    :  no
pciehp 0000:80:09.0:pcie04:   EMI Present          :  no
pciehp 0000:80:09.0:pcie04:   Command Completed    : yes
pciehp 0000:80:09.0:pcie04: Slot Status            : 0x0000
pciehp 0000:80:09.0:pcie04: Slot Control           : 0x07c0
pciehp 0000:80:09.0:pcie04: Link Active Reporting supported
pciehp 0000:80:09.0:pcie04: Command not completed in 1000 msec
pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
pciehp 0000:80:09.0:pcie04: Registering domain:bus:dev=0000:87:00 sun=7
pciehp 0000:80:09.0:pcie04: get_power_status: physical_slot = 7
pciehp 0000:80:09.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 7c0
pciehp 0000:80:09.0:pcie04: get_attention_status: physical_slot = 7
pciehp 0000:80:09.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 7c0
pciehp 0000:80:09.0:pcie04: get_latch_status: physical_slot = 7
pciehp 0000:80:09.0:pcie04: get_adapter_status: physical_slot = 7
pciehp 0000:80:09.0:pcie04: Command not completed in 1000 msec
pciehp 0000:80:09.0:pcie04: Command not completed in 1000 msec
pciehp 0000:80:09.0:pcie04: pciehp_power_off_slot: SLOTCTRL a8 write cmd 400
pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
pciehp: pcie_port_service_register = 0
pciehp: PCI Express Hot Plug Controller Driver version: 0.4

-- 
Jens Axboe

Re: pci-express hotplug

[Kenji Kaneshige]

Jens Axboe wrote:
> On Tue, Oct 27 2009, Kenji Kaneshige wrote:
>> Jens Axboe wrote:
>>> On Tue, Oct 20 2009, Alex Chiang wrote:
>>>> * Jens Axboe <jens.axboe@oracle.com>:
>>>>> On Tue, Oct 13 2009, Alex Chiang wrote:
>>>>>>>> Can you modprobe acpiphp with debug=1? And send the output?
>>>>>>> acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
>>>>>>> acpiphp: Slot [1] registered
>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
>>>>>>> acpiphp: Slot [2] registered
>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
>>>>>>> acpiphp: Slot [6] registered
>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
>>>>>>> acpiphp: Slot [7] registered
>>>>>>> acpiphp_glue: Bus 0000:87 has 1 slot
>>>>>>> acpiphp_glue: Bus 0000:84 has 1 slot
>>>>>>> acpiphp_glue: Bus 0000:0b has 1 slot
>>>>>>> acpiphp_glue: Bus 0000:08 has 1 slot
>>>>>>> acpiphp_glue: Total 4 slots
>>>>>> You mentioned in another mail that you echoed 1 into the various
>>>>>> slots' power files.
>>>>>>
>>>>>> Did you do that after modprobing acpiphp with debug=1?
>>>>>>
>>>>>> If so, there should be debug output when you try and turn them
>>>>>> on.
>>>>> It produces:
>>>>>
>>>>> acpiphp: enable_slot - physical_slot = 1
>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>> acpiphp: enable_slot - physical_slot = 2
>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>> acpiphp: enable_slot - physical_slot = 6
>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>> acpiphp: enable_slot - physical_slot = 7
>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>> Hm, so for some reason, firmware on your machine is telling us
>>>> that it doesn't think cards are present and/or enabled.
>>>>
>>>> Unfortunately, I don't know why your firmware would be saying
>>>> that. We could add some more debug printks to see what firmware
>>>> thinks about your system... Or we could just wait and see what
>>>> happens after you get your hardware replaced.
>>> New board, the exact same thing happens.
>>>
>>>>> I have a card in one of the slots only this time.
>>>>>
>>>>>> Also, quick dummy check, you are trying to power on populated
>>>>>> slots, right? :)
>>>>> Yes :-)
>>>>>
>>>>>> Can you send the output of lspci -vv? And I like the output of
>>>>>> lspci -vt as well... Both before and after loading acpiphp
>>>>>> please.
>>>>> Send privately.
>>>> No difference in before and after. Odd.
>>>>
>>>> If you want to poke us again after your hardware swap, please do
>>>> so. Sorry for being not so helpful. :-/
>>> Poke :-)
>>>
>>> One more thing I tried was pushing the power button on the slot
>>> manually. With acpiphp, I get the same messages as above. Using pciehp,
>>> I get the same power fault bit interrupt storm. So no difference from
>>> using the sysfs interface or doing it on the box side, doesn't work
>>> either way.
>>>
>> I'd like to confirm power fault interrupt storm, just in case.
>> Could you get /proc/interrupts information after power fault
>> problem happens and send it to me?
> 
> The box pretty much hangs when I try to power on a slot with pciehp, so
> it's not easy to do... It doesn't hang with acpiphp, but doesn't work
> either (see previous reply to Alex).
> 

Ok, I'll try to make a debug patch.
Maybe I can send it tomorrow.

Thanks,
Kenji Kaneshige

Re: pci-express hotplug

[Kenji Kaneshige]

[-- Attachment #1: Type: text/plain, Size: 3874 bytes --]

Jens Axboe wrote:
> On Tue, Oct 27 2009, Kenji Kaneshige wrote:
>> Jens Axboe wrote:
>>> On Tue, Oct 20 2009, Alex Chiang wrote:
>>>> * Jens Axboe <jens.axboe@oracle.com>:
>>>>> On Tue, Oct 13 2009, Alex Chiang wrote:
>>>>>>>> Can you modprobe acpiphp with debug=1? And send the output?
>>>>>>> acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
>>>>>>> acpiphp: Slot [1] registered
>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
>>>>>>> acpiphp: Slot [2] registered
>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
>>>>>>> acpiphp: Slot [6] registered
>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
>>>>>>> acpiphp: Slot [7] registered
>>>>>>> acpiphp_glue: Bus 0000:87 has 1 slot
>>>>>>> acpiphp_glue: Bus 0000:84 has 1 slot
>>>>>>> acpiphp_glue: Bus 0000:0b has 1 slot
>>>>>>> acpiphp_glue: Bus 0000:08 has 1 slot
>>>>>>> acpiphp_glue: Total 4 slots
>>>>>> You mentioned in another mail that you echoed 1 into the various
>>>>>> slots' power files.
>>>>>>
>>>>>> Did you do that after modprobing acpiphp with debug=1?
>>>>>>
>>>>>> If so, there should be debug output when you try and turn them
>>>>>> on.
>>>>> It produces:
>>>>>
>>>>> acpiphp: enable_slot - physical_slot = 1
>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>> acpiphp: enable_slot - physical_slot = 2
>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>> acpiphp: enable_slot - physical_slot = 6
>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>> acpiphp: enable_slot - physical_slot = 7
>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>> Hm, so for some reason, firmware on your machine is telling us
>>>> that it doesn't think cards are present and/or enabled.
>>>>
>>>> Unfortunately, I don't know why your firmware would be saying
>>>> that. We could add some more debug printks to see what firmware
>>>> thinks about your system... Or we could just wait and see what
>>>> happens after you get your hardware replaced.
>>> New board, the exact same thing happens.
>>>
>>>>> I have a card in one of the slots only this time.
>>>>>
>>>>>> Also, quick dummy check, you are trying to power on populated
>>>>>> slots, right? :)
>>>>> Yes :-)
>>>>>
>>>>>> Can you send the output of lspci -vv? And I like the output of
>>>>>> lspci -vt as well... Both before and after loading acpiphp
>>>>>> please.
>>>>> Send privately.
>>>> No difference in before and after. Odd.
>>>>
>>>> If you want to poke us again after your hardware swap, please do
>>>> so. Sorry for being not so helpful. :-/
>>> Poke :-)
>>>
>>> One more thing I tried was pushing the power button on the slot
>>> manually. With acpiphp, I get the same messages as above. Using pciehp,
>>> I get the same power fault bit interrupt storm. So no difference from
>>> using the sysfs interface or doing it on the box side, doesn't work
>>> either way.
>>>
>> I'd like to confirm power fault interrupt storm, just in case.
>> Could you get /proc/interrupts information after power fault
>> problem happens and send it to me?
> 
> The box pretty much hangs when I try to power on a slot with pciehp, so
> it's not easy to do... It doesn't hang with acpiphp, but doesn't work
> either (see previous reply to Alex).
> 

Could you try the attached debugging patch? With this patch, power
fault interrupt would be disabled after 100 power fault detected (
I hope so). You can get /proc/interrupts after that.

Thanks,
Kenji Kaneshige



[-- Attachment #2: pciehp-power-fault-debug.patch --]
[-- Type: text/plain, Size: 1079 bytes --]

---
 drivers/pci/hotplug/pciehp_hpc.c |    8 ++++++++
 1 file changed, 8 insertions(+)

Index: 20091026/drivers/pci/hotplug/pciehp_hpc.c
===================================================================
--- 20091026.orig/drivers/pci/hotplug/pciehp_hpc.c
+++ 20091026/drivers/pci/hotplug/pciehp_hpc.c
@@ -612,6 +612,7 @@ static irqreturn_t pcie_isr(int irq, voi
 	struct controller *ctrl = (struct controller *)dev_id;
 	struct slot *slot = ctrl->slot;
 	u16 detected, intr_loc;
+	static int nr_power_faults = 0;
 
 	/*
 	 * In order to guarantee that all interrupt events are
@@ -664,6 +665,13 @@ static irqreturn_t pcie_isr(int irq, voi
 	if (intr_loc & PCI_EXP_SLTSTA_PDC)
 		pciehp_handle_presence_change(slot);
 
+	if ((intr_loc & PCI_EXP_SLTSTA_PFD) && (++nr_power_faults > 100)) {
+		u16 reg16;
+		pciehp_readw(ctrl, PCI_EXP_SLTCTL, &reg16);
+		reg16 &= ~PCI_EXP_SLTCTL_PFDE;
+		pciehp_writew(ctrl, PCI_EXP_SLTCTL, reg16);
+	}
+
 	/* Check Power Fault Detected */
 	if ((intr_loc & PCI_EXP_SLTSTA_PFD) && !ctrl->power_fault_detected) {
 		ctrl->power_fault_detected = 1;

Re: pci-express hotplug

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 10720 bytes --]

On Wed, Oct 28 2009, Kenji Kaneshige wrote:
> Jens Axboe wrote:
>> On Tue, Oct 27 2009, Kenji Kaneshige wrote:
>>> Jens Axboe wrote:
>>>> On Tue, Oct 20 2009, Alex Chiang wrote:
>>>>> * Jens Axboe <jens.axboe@oracle.com>:
>>>>>> On Tue, Oct 13 2009, Alex Chiang wrote:
>>>>>>>>> Can you modprobe acpiphp with debug=1? And send the output?
>>>>>>>> acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
>>>>>>>> acpiphp: Slot [1] registered
>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
>>>>>>>> acpiphp: Slot [2] registered
>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
>>>>>>>> acpiphp: Slot [6] registered
>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
>>>>>>>> acpiphp: Slot [7] registered
>>>>>>>> acpiphp_glue: Bus 0000:87 has 1 slot
>>>>>>>> acpiphp_glue: Bus 0000:84 has 1 slot
>>>>>>>> acpiphp_glue: Bus 0000:0b has 1 slot
>>>>>>>> acpiphp_glue: Bus 0000:08 has 1 slot
>>>>>>>> acpiphp_glue: Total 4 slots
>>>>>>> You mentioned in another mail that you echoed 1 into the various
>>>>>>> slots' power files.
>>>>>>>
>>>>>>> Did you do that after modprobing acpiphp with debug=1?
>>>>>>>
>>>>>>> If so, there should be debug output when you try and turn them
>>>>>>> on.
>>>>>> It produces:
>>>>>>
>>>>>> acpiphp: enable_slot - physical_slot = 1
>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>> acpiphp: enable_slot - physical_slot = 2
>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>> acpiphp: enable_slot - physical_slot = 6
>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>> acpiphp: enable_slot - physical_slot = 7
>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>> Hm, so for some reason, firmware on your machine is telling us
>>>>> that it doesn't think cards are present and/or enabled.
>>>>>
>>>>> Unfortunately, I don't know why your firmware would be saying
>>>>> that. We could add some more debug printks to see what firmware
>>>>> thinks about your system... Or we could just wait and see what
>>>>> happens after you get your hardware replaced.
>>>> New board, the exact same thing happens.
>>>>
>>>>>> I have a card in one of the slots only this time.
>>>>>>
>>>>>>> Also, quick dummy check, you are trying to power on populated
>>>>>>> slots, right? :)
>>>>>> Yes :-)
>>>>>>
>>>>>>> Can you send the output of lspci -vv? And I like the output of
>>>>>>> lspci -vt as well... Both before and after loading acpiphp
>>>>>>> please.
>>>>>> Send privately.
>>>>> No difference in before and after. Odd.
>>>>>
>>>>> If you want to poke us again after your hardware swap, please do
>>>>> so. Sorry for being not so helpful. :-/
>>>> Poke :-)
>>>>
>>>> One more thing I tried was pushing the power button on the slot
>>>> manually. With acpiphp, I get the same messages as above. Using pciehp,
>>>> I get the same power fault bit interrupt storm. So no difference from
>>>> using the sysfs interface or doing it on the box side, doesn't work
>>>> either way.
>>>>
>>> I'd like to confirm power fault interrupt storm, just in case.
>>> Could you get /proc/interrupts information after power fault
>>> problem happens and send it to me?
>>
>> The box pretty much hangs when I try to power on a slot with pciehp, so
>> it's not easy to do... It doesn't hang with acpiphp, but doesn't work
>> either (see previous reply to Alex).
>>
>
> Could you try the attached debugging patch? With this patch, power
> fault interrupt would be disabled after 100 power fault detected (
> I hope so). You can get /proc/interrupts after that.

Here is the output of doing the power on with that patch applied.

pciehp 0000:00:05.0:pcie04: enable_slot: physical_slot = 1
pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 77b
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 10
pciehp 0000:00:05.0:pcie04: pciehp_power_on_slot: SLOTCTRL a8 write cmd 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 10
pciehp 0000:00:05.0:pcie04: pciehp_green_led_blink: SLOTCTRL a8 write cmd 200
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: Power fault interrupt received
pciehp 0000:00:05.0:pcie04: Power fault on Slot(1)
pciehp 0000:00:05.0:pcie04: Power fault bit 0 set
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
pciehp 0000:00:05.0:pcie04: Data Link Layer Link Active not set in 1000 msec
pciehp 0000:00:05.0:pcie04: pciehp_check_link_status: lnk_status = 1001
pciehp 0000:00:05.0:pcie04: Link Training Error occurs 
pciehp 0000:00:05.0:pcie04: Failed to check link status
pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:05.0:pcie04: pciehp_set_attention_status: SLOTCTRL a8 write cmd 40
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 12
pciehp 0000:00:05.0:pcie04: pciehp_green_led_off: SLOTCTRL a8 write cmd 300
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 12
pciehp 0000:00:05.0:pcie04: pciehp_power_off_slot: SLOTCTRL a8 write cmd 400
pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:05.0:pcie04: pciehp_green_led_off: SLOTCTRL a8 write cmd 300
pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:05.0:pcie04: pciehp_set_attention_status: SLOTCTRL a8 write cmd 40
pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 779
pciehp 0000:00:05.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 779

-- 
Jens Axboe


[-- Attachment #2: interrupts --]
[-- Type: text/plain, Size: 25893 bytes --]

            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      CPU16      CPU17      CPU18      CPU19      CPU20      CPU21      CPU22      CPU23      CPU24      CPU25      CPU26      CPU27      CPU28      CPU29      CPU30      CPU31      CPU32      CPU33      CPU34      CPU35      CPU36      CPU37      CPU38      CPU39      CPU40      CPU41      CPU42      CPU43      CPU44      CPU45      CPU46      CPU47      CPU48      CPU49      CPU50      CPU51      CPU52      CPU53      CPU54      CPU55      CPU56      CPU57      CPU58      CPU59      CPU60      CPU61      CPU62      CPU63      
   0:        398          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      timer
   9:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
  16:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb1
  18:        294          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb4, uhci_hcd:usb8
  19:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3, uhci_hcd:usb7
  21:         48          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb2
  23:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb5, ehci_hcd:usb6
  24:       2678      12250          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   megasas
  75:          5          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
  76:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
  84:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
  85:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
  86:         60          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ahci
  87:         32          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-tx-0
  88:         29          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0        531          0       1348          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-tx-1
  89:         29          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-tx-2
  90:         67          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-tx-3
  91:        181          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          5         33          0          0         27          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-rx-0
  92:       1038          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0        799          0       1552          0        148          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-rx-1
  93:        169          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0        148          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-rx-2
  94:         25          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         62          0          0          0        110          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0-rx-3
  95:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth0
 NMI:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Non-maskable interrupts
 LOC:      41427      31601      18530      20254      13942      14884       9672       9172      12495       9142       8580       8844      11468       8168       8284       7810      10967       7621       7423       7300       9554       7097       6929       6997       8874       6663       6457       6487       9016       6396       6155       6532      18224       8574       7290      10847       6309       5434       5204       4898       6007       4701       4340       4204       4739       4241       3805       3698       3929       3496       3261       3078       3194       2847       2747       3162       3692       2296       2189       2317       4592       2367       1740       3375   Local timer interrupts
 SPU:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Performance monitoring interrupts
 PND:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Performance pending work
 RES:        471         43         94         67         40          4          3          2         40          3          1          5          9          2          1          4          1          1          0          1          2          0          5          5          2          0          1          0          1          5          0         23       2489         16          9         60         29          1          5         23          9          5          1          1          4          1          2          1         12          4          8          2          1          1          2          4        118          1          1          1         64         32          0        187   Rescheduling interrupts
 CAL:       5162        434       1897       3862        133        133        133        133        132        133        133        133        133        133        133        133        133        133        133        133        133        133        133        133        131        133        133        133        133        133        133         93        102        133        133        133        132        133        133        133        133        133        133        133        133        133        133        133        133        133        133        133        133        133        133        133        132        133        133        133        132        133        133        111   Function call interrupts
 TLB:        218         79         78        135         56         29         19         17         40         32         10          6         11         10          6          4         11         15          5          5          9          5          7          6         12          2          6          4         12         14          2         49        789         98         48        152         49         20         18         24         40         32          5          7         48          8          4          1         34         10          3          1         31          8          1          9         19          3          6          9         28          6          4         14   TLB shootdowns
 TRM:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Machine check exceptions
 MCP:          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3          3   Machine check polls
 ERR:          0
 MIS:          0

Re: pci-express hotplug

[Kenji Kaneshige]

Jens Axboe wrote:
> On Wed, Oct 28 2009, Kenji Kaneshige wrote:
>> Jens Axboe wrote:
>>> On Tue, Oct 27 2009, Kenji Kaneshige wrote:
>>>> Jens Axboe wrote:
>>>>> On Tue, Oct 20 2009, Alex Chiang wrote:
>>>>>> * Jens Axboe <jens.axboe@oracle.com>:
>>>>>>> On Tue, Oct 13 2009, Alex Chiang wrote:
>>>>>>>>>> Can you modprobe acpiphp with debug=1? And send the output?
>>>>>>>>> acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
>>>>>>>>> acpiphp: Slot [1] registered
>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
>>>>>>>>> acpiphp: Slot [2] registered
>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
>>>>>>>>> acpiphp: Slot [6] registered
>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
>>>>>>>>> acpiphp: Slot [7] registered
>>>>>>>>> acpiphp_glue: Bus 0000:87 has 1 slot
>>>>>>>>> acpiphp_glue: Bus 0000:84 has 1 slot
>>>>>>>>> acpiphp_glue: Bus 0000:0b has 1 slot
>>>>>>>>> acpiphp_glue: Bus 0000:08 has 1 slot
>>>>>>>>> acpiphp_glue: Total 4 slots
>>>>>>>> You mentioned in another mail that you echoed 1 into the various
>>>>>>>> slots' power files.
>>>>>>>>
>>>>>>>> Did you do that after modprobing acpiphp with debug=1?
>>>>>>>>
>>>>>>>> If so, there should be debug output when you try and turn them
>>>>>>>> on.
>>>>>>> It produces:
>>>>>>>
>>>>>>> acpiphp: enable_slot - physical_slot = 1
>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>> acpiphp: enable_slot - physical_slot = 2
>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>> acpiphp: enable_slot - physical_slot = 6
>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>> acpiphp: enable_slot - physical_slot = 7
>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>> Hm, so for some reason, firmware on your machine is telling us
>>>>>> that it doesn't think cards are present and/or enabled.
>>>>>>
>>>>>> Unfortunately, I don't know why your firmware would be saying
>>>>>> that. We could add some more debug printks to see what firmware
>>>>>> thinks about your system... Or we could just wait and see what
>>>>>> happens after you get your hardware replaced.
>>>>> New board, the exact same thing happens.
>>>>>
>>>>>>> I have a card in one of the slots only this time.
>>>>>>>
>>>>>>>> Also, quick dummy check, you are trying to power on populated
>>>>>>>> slots, right? :)
>>>>>>> Yes :-)
>>>>>>>
>>>>>>>> Can you send the output of lspci -vv? And I like the output of
>>>>>>>> lspci -vt as well... Both before and after loading acpiphp
>>>>>>>> please.
>>>>>>> Send privately.
>>>>>> No difference in before and after. Odd.
>>>>>>
>>>>>> If you want to poke us again after your hardware swap, please do
>>>>>> so. Sorry for being not so helpful. :-/
>>>>> Poke :-)
>>>>>
>>>>> One more thing I tried was pushing the power button on the slot
>>>>> manually. With acpiphp, I get the same messages as above. Using pciehp,
>>>>> I get the same power fault bit interrupt storm. So no difference from
>>>>> using the sysfs interface or doing it on the box side, doesn't work
>>>>> either way.
>>>>>
>>>> I'd like to confirm power fault interrupt storm, just in case.
>>>> Could you get /proc/interrupts information after power fault
>>>> problem happens and send it to me?
>>> The box pretty much hangs when I try to power on a slot with pciehp, so
>>> it's not easy to do... It doesn't hang with acpiphp, but doesn't work
>>> either (see previous reply to Alex).
>>>
>> Could you try the attached debugging patch? With this patch, power
>> fault interrupt would be disabled after 100 power fault detected (
>> I hope so). You can get /proc/interrupts after that.
> 
> Here is the output of doing the power on with that patch applied.
> 
> pciehp 0000:00:05.0:pcie04: enable_slot: physical_slot = 1
> pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 77b
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 10
> pciehp 0000:00:05.0:pcie04: pciehp_power_on_slot: SLOTCTRL a8 write cmd 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 10
> pciehp 0000:00:05.0:pcie04: pciehp_green_led_blink: SLOTCTRL a8 write cmd 200
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: Power fault interrupt received
> pciehp 0000:00:05.0:pcie04: Power fault on Slot(1)
> pciehp 0000:00:05.0:pcie04: Power fault bit 0 set
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> pciehp 0000:00:05.0:pcie04: Data Link Layer Link Active not set in 1000 msec
> pciehp 0000:00:05.0:pcie04: pciehp_check_link_status: lnk_status = 1001
> pciehp 0000:00:05.0:pcie04: Link Training Error occurs 
> pciehp 0000:00:05.0:pcie04: Failed to check link status
> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
> pciehp 0000:00:05.0:pcie04: pciehp_set_attention_status: SLOTCTRL a8 write cmd 40
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 12
> pciehp 0000:00:05.0:pcie04: pciehp_green_led_off: SLOTCTRL a8 write cmd 300
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 12
> pciehp 0000:00:05.0:pcie04: pciehp_power_off_slot: SLOTCTRL a8 write cmd 400
> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
> pciehp 0000:00:05.0:pcie04: pciehp_green_led_off: SLOTCTRL a8 write cmd 300
> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
> pciehp 0000:00:05.0:pcie04: pciehp_set_attention_status: SLOTCTRL a8 write cmd 40
> pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 779
> pciehp 0000:00:05.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 779
> 

>From the console log, it seems that my debug patch worked as I expected
(power fault event interrupts ware disabled after 100 power fault event).
But for some reasons, /proc/interrupts indicates only 5 interrupts of
pciehp. Just in case, did you get /proc/interrupts after doing power on?

Thanks,
Kenji Kaneshige

Re: pci-express hotplug

[Jens Axboe]

On Thu, Oct 29 2009, Kenji Kaneshige wrote:
> Jens Axboe wrote:
>> On Wed, Oct 28 2009, Kenji Kaneshige wrote:
>>> Jens Axboe wrote:
>>>> On Tue, Oct 27 2009, Kenji Kaneshige wrote:
>>>>> Jens Axboe wrote:
>>>>>> On Tue, Oct 20 2009, Alex Chiang wrote:
>>>>>>> * Jens Axboe <jens.axboe@oracle.com>:
>>>>>>>> On Tue, Oct 13 2009, Alex Chiang wrote:
>>>>>>>>>>> Can you modprobe acpiphp with debug=1? And send the output?
>>>>>>>>>> acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
>>>>>>>>>> acpiphp: Slot [1] registered
>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
>>>>>>>>>> acpiphp: Slot [2] registered
>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
>>>>>>>>>> acpiphp: Slot [6] registered
>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
>>>>>>>>>> acpiphp: Slot [7] registered
>>>>>>>>>> acpiphp_glue: Bus 0000:87 has 1 slot
>>>>>>>>>> acpiphp_glue: Bus 0000:84 has 1 slot
>>>>>>>>>> acpiphp_glue: Bus 0000:0b has 1 slot
>>>>>>>>>> acpiphp_glue: Bus 0000:08 has 1 slot
>>>>>>>>>> acpiphp_glue: Total 4 slots
>>>>>>>>> You mentioned in another mail that you echoed 1 into the various
>>>>>>>>> slots' power files.
>>>>>>>>>
>>>>>>>>> Did you do that after modprobing acpiphp with debug=1?
>>>>>>>>>
>>>>>>>>> If so, there should be debug output when you try and turn them
>>>>>>>>> on.
>>>>>>>> It produces:
>>>>>>>>
>>>>>>>> acpiphp: enable_slot - physical_slot = 1
>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>> acpiphp: enable_slot - physical_slot = 2
>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>> acpiphp: enable_slot - physical_slot = 6
>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>> acpiphp: enable_slot - physical_slot = 7
>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>> Hm, so for some reason, firmware on your machine is telling us
>>>>>>> that it doesn't think cards are present and/or enabled.
>>>>>>>
>>>>>>> Unfortunately, I don't know why your firmware would be saying
>>>>>>> that. We could add some more debug printks to see what firmware
>>>>>>> thinks about your system... Or we could just wait and see what
>>>>>>> happens after you get your hardware replaced.
>>>>>> New board, the exact same thing happens.
>>>>>>
>>>>>>>> I have a card in one of the slots only this time.
>>>>>>>>
>>>>>>>>> Also, quick dummy check, you are trying to power on populated
>>>>>>>>> slots, right? :)
>>>>>>>> Yes :-)
>>>>>>>>
>>>>>>>>> Can you send the output of lspci -vv? And I like the output of
>>>>>>>>> lspci -vt as well... Both before and after loading acpiphp
>>>>>>>>> please.
>>>>>>>> Send privately.
>>>>>>> No difference in before and after. Odd.
>>>>>>>
>>>>>>> If you want to poke us again after your hardware swap, please do
>>>>>>> so. Sorry for being not so helpful. :-/
>>>>>> Poke :-)
>>>>>>
>>>>>> One more thing I tried was pushing the power button on the slot
>>>>>> manually. With acpiphp, I get the same messages as above. Using pciehp,
>>>>>> I get the same power fault bit interrupt storm. So no difference from
>>>>>> using the sysfs interface or doing it on the box side, doesn't work
>>>>>> either way.
>>>>>>
>>>>> I'd like to confirm power fault interrupt storm, just in case.
>>>>> Could you get /proc/interrupts information after power fault
>>>>> problem happens and send it to me?
>>>> The box pretty much hangs when I try to power on a slot with pciehp, so
>>>> it's not easy to do... It doesn't hang with acpiphp, but doesn't work
>>>> either (see previous reply to Alex).
>>>>
>>> Could you try the attached debugging patch? With this patch, power
>>> fault interrupt would be disabled after 100 power fault detected (
>>> I hope so). You can get /proc/interrupts after that.
>>
>> Here is the output of doing the power on with that patch applied.
>>
>> pciehp 0000:00:05.0:pcie04: enable_slot: physical_slot = 1
>> pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 77b
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 10
>> pciehp 0000:00:05.0:pcie04: pciehp_power_on_slot: SLOTCTRL a8 write cmd 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 10
>> pciehp 0000:00:05.0:pcie04: pciehp_green_led_blink: SLOTCTRL a8 write cmd 200
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: Power fault interrupt received
>> pciehp 0000:00:05.0:pcie04: Power fault on Slot(1)
>> pciehp 0000:00:05.0:pcie04: Power fault bit 0 set
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> pciehp 0000:00:05.0:pcie04: Data Link Layer Link Active not set in 1000 msec
>> pciehp 0000:00:05.0:pcie04: pciehp_check_link_status: lnk_status = 1001
>> pciehp 0000:00:05.0:pcie04: Link Training Error occurs pciehp 
>> 0000:00:05.0:pcie04: Failed to check link status
>> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
>> pciehp 0000:00:05.0:pcie04: pciehp_set_attention_status: SLOTCTRL a8 write cmd 40
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 12
>> pciehp 0000:00:05.0:pcie04: pciehp_green_led_off: SLOTCTRL a8 write cmd 300
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 12
>> pciehp 0000:00:05.0:pcie04: pciehp_power_off_slot: SLOTCTRL a8 write cmd 400
>> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
>> pciehp 0000:00:05.0:pcie04: pciehp_green_led_off: SLOTCTRL a8 write cmd 300
>> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
>> pciehp 0000:00:05.0:pcie04: pciehp_set_attention_status: SLOTCTRL a8 write cmd 40
>> pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 779
>> pciehp 0000:00:05.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 779
>>
>
> From the console log, it seems that my debug patch worked as I expected
> (power fault event interrupts ware disabled after 100 power fault event).
> But for some reasons, /proc/interrupts indicates only 5 interrupts of
> pciehp. Just in case, did you get /proc/interrupts after doing power on?

Nope, it was captured post the power on attempt and the above log dump.

-- 
Jens Axboe

Re: pci-express hotplug

[Kenji Kaneshige]

Jens Axboe wrote:
> On Thu, Oct 29 2009, Kenji Kaneshige wrote:
>> Jens Axboe wrote:
>>> On Wed, Oct 28 2009, Kenji Kaneshige wrote:
>>>> Jens Axboe wrote:
>>>>> On Tue, Oct 27 2009, Kenji Kaneshige wrote:
>>>>>> Jens Axboe wrote:
>>>>>>> On Tue, Oct 20 2009, Alex Chiang wrote:
>>>>>>>> * Jens Axboe <jens.axboe@oracle.com>:
>>>>>>>>> On Tue, Oct 13 2009, Alex Chiang wrote:
>>>>>>>>>>>> Can you modprobe acpiphp with debug=1? And send the output?
>>>>>>>>>>> acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
>>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
>>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
>>>>>>>>>>> acpiphp: Slot [1] registered
>>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
>>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
>>>>>>>>>>> acpiphp: Slot [2] registered
>>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
>>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
>>>>>>>>>>> acpiphp: Slot [6] registered
>>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
>>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
>>>>>>>>>>> acpiphp: Slot [7] registered
>>>>>>>>>>> acpiphp_glue: Bus 0000:87 has 1 slot
>>>>>>>>>>> acpiphp_glue: Bus 0000:84 has 1 slot
>>>>>>>>>>> acpiphp_glue: Bus 0000:0b has 1 slot
>>>>>>>>>>> acpiphp_glue: Bus 0000:08 has 1 slot
>>>>>>>>>>> acpiphp_glue: Total 4 slots
>>>>>>>>>> You mentioned in another mail that you echoed 1 into the various
>>>>>>>>>> slots' power files.
>>>>>>>>>>
>>>>>>>>>> Did you do that after modprobing acpiphp with debug=1?
>>>>>>>>>>
>>>>>>>>>> If so, there should be debug output when you try and turn them
>>>>>>>>>> on.
>>>>>>>>> It produces:
>>>>>>>>>
>>>>>>>>> acpiphp: enable_slot - physical_slot = 1
>>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>>> acpiphp: enable_slot - physical_slot = 2
>>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>>> acpiphp: enable_slot - physical_slot = 6
>>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>>> acpiphp: enable_slot - physical_slot = 7
>>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>> Hm, so for some reason, firmware on your machine is telling us
>>>>>>>> that it doesn't think cards are present and/or enabled.
>>>>>>>>
>>>>>>>> Unfortunately, I don't know why your firmware would be saying
>>>>>>>> that. We could add some more debug printks to see what firmware
>>>>>>>> thinks about your system... Or we could just wait and see what
>>>>>>>> happens after you get your hardware replaced.
>>>>>>> New board, the exact same thing happens.
>>>>>>>
>>>>>>>>> I have a card in one of the slots only this time.
>>>>>>>>>
>>>>>>>>>> Also, quick dummy check, you are trying to power on populated
>>>>>>>>>> slots, right? :)
>>>>>>>>> Yes :-)
>>>>>>>>>
>>>>>>>>>> Can you send the output of lspci -vv? And I like the output of
>>>>>>>>>> lspci -vt as well... Both before and after loading acpiphp
>>>>>>>>>> please.
>>>>>>>>> Send privately.
>>>>>>>> No difference in before and after. Odd.
>>>>>>>>
>>>>>>>> If you want to poke us again after your hardware swap, please do
>>>>>>>> so. Sorry for being not so helpful. :-/
>>>>>>> Poke :-)
>>>>>>>
>>>>>>> One more thing I tried was pushing the power button on the slot
>>>>>>> manually. With acpiphp, I get the same messages as above. Using pciehp,
>>>>>>> I get the same power fault bit interrupt storm. So no difference from
>>>>>>> using the sysfs interface or doing it on the box side, doesn't work
>>>>>>> either way.
>>>>>>>
>>>>>> I'd like to confirm power fault interrupt storm, just in case.
>>>>>> Could you get /proc/interrupts information after power fault
>>>>>> problem happens and send it to me?
>>>>> The box pretty much hangs when I try to power on a slot with pciehp, so
>>>>> it's not easy to do... It doesn't hang with acpiphp, but doesn't work
>>>>> either (see previous reply to Alex).
>>>>>
>>>> Could you try the attached debugging patch? With this patch, power
>>>> fault interrupt would be disabled after 100 power fault detected (
>>>> I hope so). You can get /proc/interrupts after that.
>>> Here is the output of doing the power on with that patch applied.
>>>
>>> pciehp 0000:00:05.0:pcie04: enable_slot: physical_slot = 1
>>> pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 77b
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 10
>>> pciehp 0000:00:05.0:pcie04: pciehp_power_on_slot: SLOTCTRL a8 write cmd 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 10
>>> pciehp 0000:00:05.0:pcie04: pciehp_green_led_blink: SLOTCTRL a8 write cmd 200
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: Power fault interrupt received
>>> pciehp 0000:00:05.0:pcie04: Power fault on Slot(1)
>>> pciehp 0000:00:05.0:pcie04: Power fault bit 0 set
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>> pciehp 0000:00:05.0:pcie04: Data Link Layer Link Active not set in 1000 msec
>>> pciehp 0000:00:05.0:pcie04: pciehp_check_link_status: lnk_status = 1001
>>> pciehp 0000:00:05.0:pcie04: Link Training Error occurs pciehp 
>>> 0000:00:05.0:pcie04: Failed to check link status
>>> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
>>> pciehp 0000:00:05.0:pcie04: pciehp_set_attention_status: SLOTCTRL a8 write cmd 40
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 12
>>> pciehp 0000:00:05.0:pcie04: pciehp_green_led_off: SLOTCTRL a8 write cmd 300
>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 12
>>> pciehp 0000:00:05.0:pcie04: pciehp_power_off_slot: SLOTCTRL a8 write cmd 400
>>> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
>>> pciehp 0000:00:05.0:pcie04: pciehp_green_led_off: SLOTCTRL a8 write cmd 300
>>> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
>>> pciehp 0000:00:05.0:pcie04: pciehp_set_attention_status: SLOTCTRL a8 write cmd 40
>>> pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 779
>>> pciehp 0000:00:05.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 779
>>>
>> From the console log, it seems that my debug patch worked as I expected
>> (power fault event interrupts ware disabled after 100 power fault event).
>> But for some reasons, /proc/interrupts indicates only 5 interrupts of
>> pciehp. Just in case, did you get /proc/interrupts after doing power on?
> 
> Nope, it was captured post the power on attempt and the above log dump.
> 

Can I confirm that? (sorry for my poor English skill)

The /proc/interrupt was captured *before* the power on attempt and the log.
Correct?

Thanks,
Kenji Kaneshige

Re: pci-express hotplug

[Jens Axboe]

On Thu, Oct 29 2009, Kenji Kaneshige wrote:
> Jens Axboe wrote:
>> On Thu, Oct 29 2009, Kenji Kaneshige wrote:
>>> Jens Axboe wrote:
>>>> On Wed, Oct 28 2009, Kenji Kaneshige wrote:
>>>>> Jens Axboe wrote:
>>>>>> On Tue, Oct 27 2009, Kenji Kaneshige wrote:
>>>>>>> Jens Axboe wrote:
>>>>>>>> On Tue, Oct 20 2009, Alex Chiang wrote:
>>>>>>>>> * Jens Axboe <jens.axboe@oracle.com>:
>>>>>>>>>> On Tue, Oct 13 2009, Alex Chiang wrote:
>>>>>>>>>>>>> Can you modprobe acpiphp with debug=1? And send the output?
>>>>>>>>>>>> acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
>>>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
>>>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
>>>>>>>>>>>> acpiphp: Slot [1] registered
>>>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
>>>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
>>>>>>>>>>>> acpiphp: Slot [2] registered
>>>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
>>>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
>>>>>>>>>>>> acpiphp: Slot [6] registered
>>>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
>>>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
>>>>>>>>>>>> acpiphp: Slot [7] registered
>>>>>>>>>>>> acpiphp_glue: Bus 0000:87 has 1 slot
>>>>>>>>>>>> acpiphp_glue: Bus 0000:84 has 1 slot
>>>>>>>>>>>> acpiphp_glue: Bus 0000:0b has 1 slot
>>>>>>>>>>>> acpiphp_glue: Bus 0000:08 has 1 slot
>>>>>>>>>>>> acpiphp_glue: Total 4 slots
>>>>>>>>>>> You mentioned in another mail that you echoed 1 into the various
>>>>>>>>>>> slots' power files.
>>>>>>>>>>>
>>>>>>>>>>> Did you do that after modprobing acpiphp with debug=1?
>>>>>>>>>>>
>>>>>>>>>>> If so, there should be debug output when you try and turn them
>>>>>>>>>>> on.
>>>>>>>>>> It produces:
>>>>>>>>>>
>>>>>>>>>> acpiphp: enable_slot - physical_slot = 1
>>>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>>>> acpiphp: enable_slot - physical_slot = 2
>>>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>>>> acpiphp: enable_slot - physical_slot = 6
>>>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>>>> acpiphp: enable_slot - physical_slot = 7
>>>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>>> Hm, so for some reason, firmware on your machine is telling us
>>>>>>>>> that it doesn't think cards are present and/or enabled.
>>>>>>>>>
>>>>>>>>> Unfortunately, I don't know why your firmware would be saying
>>>>>>>>> that. We could add some more debug printks to see what firmware
>>>>>>>>> thinks about your system... Or we could just wait and see what
>>>>>>>>> happens after you get your hardware replaced.
>>>>>>>> New board, the exact same thing happens.
>>>>>>>>
>>>>>>>>>> I have a card in one of the slots only this time.
>>>>>>>>>>
>>>>>>>>>>> Also, quick dummy check, you are trying to power on populated
>>>>>>>>>>> slots, right? :)
>>>>>>>>>> Yes :-)
>>>>>>>>>>
>>>>>>>>>>> Can you send the output of lspci -vv? And I like the output of
>>>>>>>>>>> lspci -vt as well... Both before and after loading acpiphp
>>>>>>>>>>> please.
>>>>>>>>>> Send privately.
>>>>>>>>> No difference in before and after. Odd.
>>>>>>>>>
>>>>>>>>> If you want to poke us again after your hardware swap, please do
>>>>>>>>> so. Sorry for being not so helpful. :-/
>>>>>>>> Poke :-)
>>>>>>>>
>>>>>>>> One more thing I tried was pushing the power button on the slot
>>>>>>>> manually. With acpiphp, I get the same messages as above. Using pciehp,
>>>>>>>> I get the same power fault bit interrupt storm. So no difference from
>>>>>>>> using the sysfs interface or doing it on the box side, doesn't work
>>>>>>>> either way.
>>>>>>>>
>>>>>>> I'd like to confirm power fault interrupt storm, just in case.
>>>>>>> Could you get /proc/interrupts information after power fault
>>>>>>> problem happens and send it to me?
>>>>>> The box pretty much hangs when I try to power on a slot with pciehp, so
>>>>>> it's not easy to do... It doesn't hang with acpiphp, but doesn't work
>>>>>> either (see previous reply to Alex).
>>>>>>
>>>>> Could you try the attached debugging patch? With this patch, power
>>>>> fault interrupt would be disabled after 100 power fault detected (
>>>>> I hope so). You can get /proc/interrupts after that.
>>>> Here is the output of doing the power on with that patch applied.
>>>>
>>>> pciehp 0000:00:05.0:pcie04: enable_slot: physical_slot = 1
>>>> pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 77b
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 10
>>>> pciehp 0000:00:05.0:pcie04: pciehp_power_on_slot: SLOTCTRL a8 write cmd 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 10
>>>> pciehp 0000:00:05.0:pcie04: pciehp_green_led_blink: SLOTCTRL a8 write cmd 200
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: Power fault interrupt received
>>>> pciehp 0000:00:05.0:pcie04: Power fault on Slot(1)
>>>> pciehp 0000:00:05.0:pcie04: Power fault bit 0 set
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>> pciehp 0000:00:05.0:pcie04: Data Link Layer Link Active not set in 1000 msec
>>>> pciehp 0000:00:05.0:pcie04: pciehp_check_link_status: lnk_status = 1001
>>>> pciehp 0000:00:05.0:pcie04: Link Training Error occurs pciehp  
>>>> 0000:00:05.0:pcie04: Failed to check link status
>>>> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
>>>> pciehp 0000:00:05.0:pcie04: pciehp_set_attention_status: SLOTCTRL a8 write cmd 40
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 12
>>>> pciehp 0000:00:05.0:pcie04: pciehp_green_led_off: SLOTCTRL a8 write cmd 300
>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 12
>>>> pciehp 0000:00:05.0:pcie04: pciehp_power_off_slot: SLOTCTRL a8 write cmd 400
>>>> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
>>>> pciehp 0000:00:05.0:pcie04: pciehp_green_led_off: SLOTCTRL a8 write cmd 300
>>>> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
>>>> pciehp 0000:00:05.0:pcie04: pciehp_set_attention_status: SLOTCTRL a8 write cmd 40
>>>> pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 779
>>>> pciehp 0000:00:05.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 779
>>>>
>>> From the console log, it seems that my debug patch worked as I expected
>>> (power fault event interrupts ware disabled after 100 power fault event).
>>> But for some reasons, /proc/interrupts indicates only 5 interrupts of
>>> pciehp. Just in case, did you get /proc/interrupts after doing power on?
>>
>> Nope, it was captured post the power on attempt and the above log dump.
>>
>
> Can I confirm that? (sorry for my poor English skill)
>
> The /proc/interrupt was captured *before* the power on attempt and the log.
> Correct?

No, the /proc/interrupt output was captured AFTER the power on attempt
and the log capture shown above.

-- 
Jens Axboe

Re: pci-express hotplug

[Kenji Kaneshige]

[-- Attachment #1: Type: text/plain, Size: 14240 bytes --]

Jens Axboe wrote:
> On Thu, Oct 29 2009, Kenji Kaneshige wrote:
>> Jens Axboe wrote:
>>> On Thu, Oct 29 2009, Kenji Kaneshige wrote:
>>>> Jens Axboe wrote:
>>>>> On Wed, Oct 28 2009, Kenji Kaneshige wrote:
>>>>>> Jens Axboe wrote:
>>>>>>> On Tue, Oct 27 2009, Kenji Kaneshige wrote:
>>>>>>>> Jens Axboe wrote:
>>>>>>>>> On Tue, Oct 20 2009, Alex Chiang wrote:
>>>>>>>>>> * Jens Axboe <jens.axboe@oracle.com>:
>>>>>>>>>>> On Tue, Oct 13 2009, Alex Chiang wrote:
>>>>>>>>>>>>>> Can you modprobe acpiphp with debug=1? And send the output?
>>>>>>>>>>>>> acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
>>>>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:05.0
>>>>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 1 at PCI 0000:08:00
>>>>>>>>>>>>> acpiphp: Slot [1] registered
>>>>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:00:07.0
>>>>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 2 at PCI 0000:0b:00
>>>>>>>>>>>>> acpiphp: Slot [2] registered
>>>>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:07.0
>>>>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 6 at PCI 0000:84:00
>>>>>>>>>>>>> acpiphp: Slot [6] registered
>>>>>>>>>>>>> acpiphp_glue: found PCI-to-PCI bridge at PCI 0000:80:09.0
>>>>>>>>>>>>> acpiphp_glue: found ACPI PCI Hotplug slot 7 at PCI 0000:87:00
>>>>>>>>>>>>> acpiphp: Slot [7] registered
>>>>>>>>>>>>> acpiphp_glue: Bus 0000:87 has 1 slot
>>>>>>>>>>>>> acpiphp_glue: Bus 0000:84 has 1 slot
>>>>>>>>>>>>> acpiphp_glue: Bus 0000:0b has 1 slot
>>>>>>>>>>>>> acpiphp_glue: Bus 0000:08 has 1 slot
>>>>>>>>>>>>> acpiphp_glue: Total 4 slots
>>>>>>>>>>>> You mentioned in another mail that you echoed 1 into the various
>>>>>>>>>>>> slots' power files.
>>>>>>>>>>>>
>>>>>>>>>>>> Did you do that after modprobing acpiphp with debug=1?
>>>>>>>>>>>>
>>>>>>>>>>>> If so, there should be debug output when you try and turn them
>>>>>>>>>>>> on.
>>>>>>>>>>> It produces:
>>>>>>>>>>>
>>>>>>>>>>> acpiphp: enable_slot - physical_slot = 1
>>>>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>>>>> acpiphp: enable_slot - physical_slot = 2
>>>>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>>>>> acpiphp: enable_slot - physical_slot = 6
>>>>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>>>>> acpiphp: enable_slot - physical_slot = 7
>>>>>>>>>>> acpiphp_glue: acpiphp_enable_slot: Slot status is not ACPI_STA_ALL
>>>>>>>>>> Hm, so for some reason, firmware on your machine is telling us
>>>>>>>>>> that it doesn't think cards are present and/or enabled.
>>>>>>>>>>
>>>>>>>>>> Unfortunately, I don't know why your firmware would be saying
>>>>>>>>>> that. We could add some more debug printks to see what firmware
>>>>>>>>>> thinks about your system... Or we could just wait and see what
>>>>>>>>>> happens after you get your hardware replaced.
>>>>>>>>> New board, the exact same thing happens.
>>>>>>>>>
>>>>>>>>>>> I have a card in one of the slots only this time.
>>>>>>>>>>>
>>>>>>>>>>>> Also, quick dummy check, you are trying to power on populated
>>>>>>>>>>>> slots, right? :)
>>>>>>>>>>> Yes :-)
>>>>>>>>>>>
>>>>>>>>>>>> Can you send the output of lspci -vv? And I like the output of
>>>>>>>>>>>> lspci -vt as well... Both before and after loading acpiphp
>>>>>>>>>>>> please.
>>>>>>>>>>> Send privately.
>>>>>>>>>> No difference in before and after. Odd.
>>>>>>>>>>
>>>>>>>>>> If you want to poke us again after your hardware swap, please do
>>>>>>>>>> so. Sorry for being not so helpful. :-/
>>>>>>>>> Poke :-)
>>>>>>>>>
>>>>>>>>> One more thing I tried was pushing the power button on the slot
>>>>>>>>> manually. With acpiphp, I get the same messages as above. Using pciehp,
>>>>>>>>> I get the same power fault bit interrupt storm. So no difference from
>>>>>>>>> using the sysfs interface or doing it on the box side, doesn't work
>>>>>>>>> either way.
>>>>>>>>>
>>>>>>>> I'd like to confirm power fault interrupt storm, just in case.
>>>>>>>> Could you get /proc/interrupts information after power fault
>>>>>>>> problem happens and send it to me?
>>>>>>> The box pretty much hangs when I try to power on a slot with pciehp, so
>>>>>>> it's not easy to do... It doesn't hang with acpiphp, but doesn't work
>>>>>>> either (see previous reply to Alex).
>>>>>>>
>>>>>> Could you try the attached debugging patch? With this patch, power
>>>>>> fault interrupt would be disabled after 100 power fault detected (
>>>>>> I hope so). You can get /proc/interrupts after that.
>>>>> Here is the output of doing the power on with that patch applied.
>>>>>
>>>>> pciehp 0000:00:05.0:pcie04: enable_slot: physical_slot = 1
>>>>> pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 77b
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 10
>>>>> pciehp 0000:00:05.0:pcie04: pciehp_power_on_slot: SLOTCTRL a8 write cmd 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 10
>>>>> pciehp 0000:00:05.0:pcie04: pciehp_green_led_blink: SLOTCTRL a8 write cmd 200
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: Power fault interrupt received
>>>>> pciehp 0000:00:05.0:pcie04: Power fault on Slot(1)
>>>>> pciehp 0000:00:05.0:pcie04: Power fault bit 0 set
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>>>>> pciehp 0000:00:05.0:pcie04: Data Link Layer Link Active not set in 1000 msec
>>>>> pciehp 0000:00:05.0:pcie04: pciehp_check_link_status: lnk_status = 1001
>>>>> pciehp 0000:00:05.0:pcie04: Link Training Error occurs pciehp  
>>>>> 0000:00:05.0:pcie04: Failed to check link status
>>>>> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
>>>>> pciehp 0000:00:05.0:pcie04: pciehp_set_attention_status: SLOTCTRL a8 write cmd 40
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 12
>>>>> pciehp 0000:00:05.0:pcie04: pciehp_green_led_off: SLOTCTRL a8 write cmd 300
>>>>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 12
>>>>> pciehp 0000:00:05.0:pcie04: pciehp_power_off_slot: SLOTCTRL a8 write cmd 400
>>>>> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
>>>>> pciehp 0000:00:05.0:pcie04: pciehp_green_led_off: SLOTCTRL a8 write cmd 300
>>>>> pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
>>>>> pciehp 0000:00:05.0:pcie04: pciehp_set_attention_status: SLOTCTRL a8 write cmd 40
>>>>> pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 779
>>>>> pciehp 0000:00:05.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 779
>>>>>
>>>> From the console log, it seems that my debug patch worked as I expected
>>>> (power fault event interrupts ware disabled after 100 power fault event).
>>>> But for some reasons, /proc/interrupts indicates only 5 interrupts of
>>>> pciehp. Just in case, did you get /proc/interrupts after doing power on?
>>> Nope, it was captured post the power on attempt and the above log dump.
>>>
>> Can I confirm that? (sorry for my poor English skill)
>>
>> The /proc/interrupt was captured *before* the power on attempt and the log.
>> Correct?
> 
> No, the /proc/interrupt output was captured AFTER the power on attempt
> and the log capture shown above.
> 

Thank you very much for confirmation.

The pciehp driver has a code that calls interrupt service routine
internally. So I would like to confirm if "pcie_isr: intr_loc 2"
message storm is caused by interrupts, not by internal loop.
Unfortunately, I could not confirm it from /proc/interrupts.

According the current handle_edge_irq() implementation, there seems
some cases interrupts are not counted up in /proc/interrups (I think
it could happen when next interrupts comes while first interrupt is
being handled). In addition, what I did in the debug patch was just
disabling power fault interrupts by touching hardware register, and
it actually worked just fine. So "pcie_isr: intr_loc 2" storm should
be caused by power fault interrupt storm.

I made a two patches to prevent pciehp problems happining on your
machine. Could you try those patches one by one? 

- pciehp-fix-power-fault-interrupt-storm-problem.patch
  This patch is against 2.6.32-rc5. This is for fixing the power
  fault interrupt storm problem. Could you check if interrupt
  storm doesn't happen with this patch?

- pci-hotplug-fix-oshp-evaluation.patch
  This patch is against 2.6.32-rc5. This is for fixing the slot
  mis-detection problem. According to the debug information you
  sent me before, your system seems to expect that at least slot
  1 and slot 2 are NOT handled by pciehp. With this patch, those
  slot will not be detected by pciehp.

In both testing, could you use pciehp_debug option, and send me
the console log?

Thanks,
Kenji Kaneshige


[-- Attachment #2: pciehp-fix-power-fault-interrupt-storm-problem.patch --]
[-- Type: text/plain, Size: 4547 bytes --]

Enabling power fault detected event notification in current pciehp
might cause power fault interrupt storm on some machines. On those
machines. On those machines, power fault detected bit in the slot
status register was set again immediately when it is cleared in the
interrupt service routine, and next power fault detected interrupt was
notified again. Therefore, disable power fault detected event
notification for now.

This patch also removes unnecessary handling for power fault cleared
event because this event is not supported by PCIe spec.

Signed-off-by: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>

---
 drivers/pci/hotplug/pciehp_ctrl.c |   35 ++++++++++-------------------------
 drivers/pci/hotplug/pciehp_hpc.c  |   26 +++++++++++---------------
 2 files changed, 21 insertions(+), 40 deletions(-)

Index: 20091026/drivers/pci/hotplug/pciehp_ctrl.c
===================================================================
--- 20091026.orig/drivers/pci/hotplug/pciehp_ctrl.c
+++ 20091026/drivers/pci/hotplug/pciehp_ctrl.c
@@ -142,23 +142,9 @@ u8 pciehp_handle_power_fault(struct slot
 
 	/* power fault */
 	ctrl_dbg(ctrl, "Power fault interrupt received\n");
-
-	if (!pciehp_query_power_fault(p_slot)) {
-		/*
-		 * power fault Cleared
-		 */
-		ctrl_info(ctrl, "Power fault cleared on Slot(%s)\n",
-			  slot_name(p_slot));
-		event_type = INT_POWER_FAULT_CLEAR;
-	} else {
-		/*
-		 *   power fault
-		 */
-		ctrl_info(ctrl, "Power fault on Slot(%s)\n", slot_name(p_slot));
-		event_type = INT_POWER_FAULT;
-		ctrl_info(ctrl, "Power fault bit %x set\n", 0);
-	}
-
+	ctrl_err(ctrl, "Power fault on slot %s\n", slot_name(p_slot));
+	event_type = INT_POWER_FAULT;
+	ctrl_info(ctrl, "Power fault bit %x set\n", 0);
 	queue_interrupt_event(p_slot, event_type);
 
 	return 1;
@@ -220,18 +206,17 @@ static int board_added(struct slot *p_sl
 	if (PWR_LED(ctrl))
 		pciehp_green_led_blink(p_slot);
 
+	/* Check for a power fault */
+	if (ctrl->power_fault_detected || pciehp_query_power_fault(p_slot)) {
+		ctrl_err(ctrl, "Power fault on slot %s\n", slot_name(p_slot));
+		retval = -EIO;
+		goto err_exit;
+	}
+
 	/* Check link training status */
 	retval = pciehp_check_link_status(ctrl);
 	if (retval) {
 		ctrl_err(ctrl, "Failed to check link status\n");
-		set_slot_off(ctrl, p_slot);
-		return retval;
-	}
-
-	/* Check for a power fault */
-	if (pciehp_query_power_fault(p_slot)) {
-		ctrl_dbg(ctrl, "Power fault detected\n");
-		retval = -EIO;
 		goto err_exit;
 	}
 
Index: 20091026/drivers/pci/hotplug/pciehp_hpc.c
===================================================================
--- 20091026.orig/drivers/pci/hotplug/pciehp_hpc.c
+++ 20091026/drivers/pci/hotplug/pciehp_hpc.c
@@ -514,15 +514,10 @@ int pciehp_power_on_slot(struct slot * s
 			return retval;
 		}
 	}
+	ctrl->power_fault_detected = 0;
 
 	slot_cmd = POWER_ON;
 	cmd_mask = PCI_EXP_SLTCTL_PCC;
-	if (!pciehp_poll_mode) {
-		/* Enable power fault detection turned off at power off time */
-		slot_cmd |= PCI_EXP_SLTCTL_PFDE;
-		cmd_mask |= PCI_EXP_SLTCTL_PFDE;
-	}
-
 	retval = pcie_write_cmd(ctrl, slot_cmd, cmd_mask);
 	if (retval) {
 		ctrl_err(ctrl, "Write %x command failed!\n", slot_cmd);
@@ -531,7 +526,6 @@ int pciehp_power_on_slot(struct slot * s
 	ctrl_dbg(ctrl, "%s: SLOTCTRL %x write cmd %x\n",
 		 __func__, ctrl->cap_base + PCI_EXP_SLTCTL, slot_cmd);
 
-	ctrl->power_fault_detected = 0;
 	return retval;
 }
 
@@ -586,12 +580,6 @@ int pciehp_power_off_slot(struct slot * 
 
 	slot_cmd = POWER_OFF;
 	cmd_mask = PCI_EXP_SLTCTL_PCC;
-	if (!pciehp_poll_mode) {
-		/* Disable power fault detection */
-		slot_cmd &= ~PCI_EXP_SLTCTL_PFDE;
-		cmd_mask |= PCI_EXP_SLTCTL_PFDE;
-	}
-
 	retval = pcie_write_cmd(ctrl, slot_cmd, cmd_mask);
 	if (retval) {
 		ctrl_err(ctrl, "Write command failed!\n");
@@ -840,11 +828,19 @@ int pcie_enable_notification(struct cont
 {
 	u16 cmd, mask;
 
+	/*
+	 * TBD: Power fault detected software notification support.
+	 *
+	 * Power fault detected software notification is not enabled
+	 * now, because it caused power fault detected interrupt storm
+	 * on some machines. On those machines, power fault detected
+	 * bit in the slot status register was set again immediately
+	 * when it is cleared in the interrupt service routine, and
+	 * next power fault detected interrupt was notified again.
+	 */
 	cmd = PCI_EXP_SLTCTL_PDCE;
 	if (ATTN_BUTTN(ctrl))
 		cmd |= PCI_EXP_SLTCTL_ABPE;
-	if (POWER_CTRL(ctrl))
-		cmd |= PCI_EXP_SLTCTL_PFDE;
 	if (MRL_SENS(ctrl))
 		cmd |= PCI_EXP_SLTCTL_MRLSCE;
 	if (!pciehp_poll_mode)

[-- Attachment #3: pci-hotplug-fix-oshp-evaluation.patch --]
[-- Type: text/plain, Size: 1053 bytes --]

If firmware doesn't grant over native hotplug control through ACPI
_OSC method, we must not evaluate OSHP.

Signed-off-by: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>

---
 drivers/pci/hotplug/acpi_pcihp.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Index: 20091026/drivers/pci/hotplug/acpi_pcihp.c
===================================================================
--- 20091026.orig/drivers/pci/hotplug/acpi_pcihp.c
+++ 20091026/drivers/pci/hotplug/acpi_pcihp.c
@@ -362,6 +362,8 @@ int acpi_get_hp_hw_control_from_firmware
 		status = acpi_pci_osc_control_set(handle, flags);
 		if (ACPI_SUCCESS(status))
 			goto got_one;
+		if (status == AE_SUPPORT)
+			goto no_control;
 		kfree(string.pointer);
 		string = (struct acpi_buffer){ ACPI_ALLOCATE_BUFFER, NULL };
 	}
@@ -394,10 +396,9 @@ int acpi_get_hp_hw_control_from_firmware
 		if (ACPI_FAILURE(status))
 			break;
 	}
-
+no_control:
 	dbg("Cannot get control of hotplug hardware for pci %s\n",
 	    pci_name(pdev));
-
 	kfree(string.pointer);
 	return -ENODEV;
 got_one:

Re: pci-express hotplug

[Kenji Kaneshige]

Jens Axboe wrote:
> Hi,
> 
> I'm trying to get pci-express hotplug working in a box here. I don't
> really care about the hotplug aspect, I just want the darn pci-e slots
> that are designated hotplug slots to actually WORK. When I load pciehp,
> I get:
> 
> Firmware did not grant requested _OSC control
> Firmware did not grant requested _OSC control
> Firmware did not grant requested _OSC control
> Firmware did not grant requested _OSC control
> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
> Firmware did not grant requested _OSC control
> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
> Firmware did not grant requested _OSC control
> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
> 
> and the devices in the hotplug slots stay off. Is this an ACPI/bios
> issue? How can I debug this?
> 

Could you give me the result of "ls -lR /sys/bus/pci/slots/"
after loading pciehp?

Thanks,
Kenji Kaneshige

Re: pci-express hotplug

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

On Tue, Oct 13 2009, Kenji Kaneshige wrote:
> Jens Axboe wrote:
>> Hi,
>>
>> I'm trying to get pci-express hotplug working in a box here. I don't
>> really care about the hotplug aspect, I just want the darn pci-e slots
>> that are designated hotplug slots to actually WORK. When I load pciehp,
>> I get:
>>
>> Firmware did not grant requested _OSC control
>> Firmware did not grant requested _OSC control
>> Firmware did not grant requested _OSC control
>> Firmware did not grant requested _OSC control
>> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
>> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
>> Firmware did not grant requested _OSC control
>> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
>> Firmware did not grant requested _OSC control
>> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
>> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
>> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
>> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>
>> and the devices in the hotplug slots stay off. Is this an ACPI/bios
>> issue? How can I debug this?
>>
>
> Could you give me the result of "ls -lR /sys/bus/pci/slots/"
> after loading pciehp?

I have attached the result of that ls prior to loading pciehp/acpiphp
(pre-load), after loading pciehp (pciehp-load), and with acpiphp loaded
only as well (acpiphp-load).

-- 
Jens Axboe


[-- Attachment #2: acpiphp-load.bz2 --]
[-- Type: application/octet-stream, Size: 2062 bytes --]

[-- Attachment #3: pre-load.bz2 --]
[-- Type: application/octet-stream, Size: 1983 bytes --]

[-- Attachment #4: pciehp-load.bz2 --]
[-- Type: application/octet-stream, Size: 2120 bytes --]

Re: pci-express hotplug

[Kenji Kaneshige]

Jens Axboe wrote:
> On Tue, Oct 13 2009, Kenji Kaneshige wrote:
>> Jens Axboe wrote:
>>> Hi,
>>>
>>> I'm trying to get pci-express hotplug working in a box here. I don't
>>> really care about the hotplug aspect, I just want the darn pci-e slots
>>> that are designated hotplug slots to actually WORK. When I load pciehp,
>>> I get:
>>>
>>> Firmware did not grant requested _OSC control
>>> Firmware did not grant requested _OSC control
>>> Firmware did not grant requested _OSC control
>>> Firmware did not grant requested _OSC control
>>> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
>>> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
>>> Firmware did not grant requested _OSC control
>>> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
>>> Firmware did not grant requested _OSC control
>>> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
>>> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
>>> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
>>> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>>
>>> and the devices in the hotplug slots stay off. Is this an ACPI/bios
>>> issue? How can I debug this?
>>>
>> Could you give me the result of "ls -lR /sys/bus/pci/slots/"
>> after loading pciehp?
> 
> I have attached the result of that ls prior to loading pciehp/acpiphp
> (pre-load), after loading pciehp (pciehp-load), and with acpiphp loaded
> only as well (acpiphp-load).
> 

Thank you for the info. From the information, I confirmed that hotplug
slots are detected by pciehp even though _OSC evaluation failed. There
are two ways to take control from the firmware through ACPI control
method. One is _OSC control method, and the other is OSHP control method.
I guess your ACPI fimware has both _OSC and OSHP on DSDT (ACPI Namespace),
and pciehp assumes that it took control through OSHP after the _OSC
evaluation failure. I think this pciehp's behavior is wrong because of
the following reasons and I think pciehp driver mis-detected the hotplug
slots on your environment because of this.

- According to the PCI firmware specification, pciehp driver must use the
  result of _OSC, if the platform implements both _OSC and OSHP.
- OSHP control method seems only for SHPC, not for PCI Express native hot-
  plug. So pciehp must not evaluate OSHP to take control from firmware.

To confirm this, could you send me the dmesg output after loading pciehp
with 'debug_acpi' of pci_hotplug (PCI hotplug core driver) enabled?
For example,

    $ su
    # echo Y > /sys/module/pci_hotplug/parameters/debug_acpi
    # modprobe pciehp
    # dmesg

And if it is possible, could you send me DSDT of your platform?

Anyway, my recommendation is using acpiphp on your environment because
your firmware didn't grant control over hotplug control through _OSC.
>From the information, acpiphp also detects the hotplug slots successfully.
Please try "echo 1 > /sys/bus/pci/slots/<slot#>/power". It would turn on
the slot and initialize adapter card on the slot.

Thanks,
Kenji Kaneshige

Re: pci-express hotplug

[Jens Axboe]

On Tue, Oct 13 2009, Kenji Kaneshige wrote:
> Jens Axboe wrote:
>> On Tue, Oct 13 2009, Kenji Kaneshige wrote:
>>> Jens Axboe wrote:
>>>> Hi,
>>>>
>>>> I'm trying to get pci-express hotplug working in a box here. I don't
>>>> really care about the hotplug aspect, I just want the darn pci-e slots
>>>> that are designated hotplug slots to actually WORK. When I load pciehp,
>>>> I get:
>>>>
>>>> Firmware did not grant requested _OSC control
>>>> Firmware did not grant requested _OSC control
>>>> Firmware did not grant requested _OSC control
>>>> Firmware did not grant requested _OSC control
>>>> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
>>>> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
>>>> Firmware did not grant requested _OSC control
>>>> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
>>>> Firmware did not grant requested _OSC control
>>>> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
>>>> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
>>>> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
>>>> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>>>
>>>> and the devices in the hotplug slots stay off. Is this an ACPI/bios
>>>> issue? How can I debug this?
>>>>
>>> Could you give me the result of "ls -lR /sys/bus/pci/slots/"
>>> after loading pciehp?
>>
>> I have attached the result of that ls prior to loading pciehp/acpiphp
>> (pre-load), after loading pciehp (pciehp-load), and with acpiphp loaded
>> only as well (acpiphp-load).
>>
>
> Thank you for the info. From the information, I confirmed that hotplug
> slots are detected by pciehp even though _OSC evaluation failed. There
> are two ways to take control from the firmware through ACPI control
> method. One is _OSC control method, and the other is OSHP control method.
> I guess your ACPI fimware has both _OSC and OSHP on DSDT (ACPI Namespace),
> and pciehp assumes that it took control through OSHP after the _OSC
> evaluation failure. I think this pciehp's behavior is wrong because of
> the following reasons and I think pciehp driver mis-detected the hotplug
> slots on your environment because of this.
>
> - According to the PCI firmware specification, pciehp driver must use the
>  result of _OSC, if the platform implements both _OSC and OSHP.
> - OSHP control method seems only for SHPC, not for PCI Express native hot-
>  plug. So pciehp must not evaluate OSHP to take control from firmware.
>
> To confirm this, could you send me the dmesg output after loading pciehp
> with 'debug_acpi' of pci_hotplug (PCI hotplug core driver) enabled?
> For example,
>
>    $ su
>    # echo Y > /sys/module/pci_hotplug/parameters/debug_acpi
>    # modprobe pciehp
>    # dmesg

See below.

> And if it is possible, could you send me DSDT of your platform?

Not sure I can do that, I'll check.

> Anyway, my recommendation is using acpiphp on your environment because
> your firmware didn't grant control over hotplug control through _OSC.
> From the information, acpiphp also detects the hotplug slots successfully.
> Please try "echo 1 > /sys/bus/pci/slots/<slot#>/power". It would turn on
> the slot and initialize adapter card on the slot.

It does find the 4 slots correctly. But if I try to turn on the power,
nothing happens and 'power' stays at 0. If I do the same with pciehp, I
get the same hang as described when using pciehp with pciehp_force=1.
But apparently this machine is getting a board replacement very soon, so
it may solve itself. Unless you think it should work and there's
something I can try to check, then lets just leave this issue until I
get it upgraded and return from kernel summit / JLS.


acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
Firmware did not grant requested _OSC control
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP5 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:05.0 (\_SB_.IOH0.MRP5)
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
Firmware did not grant requested _OSC control
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP7 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:07.0 (\_SB_.IOH0.MRP7)
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
Firmware did not grant requested _OSC control
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.PEX0 
acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0.PEX0 OSHP not found
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0 
acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0 OSHP not found
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Cannot get control of hotplug hardware for pci 0000:00:1c.0
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPI 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:07.0 (\_SB_.IOH1.MRPI)
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPK 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:09.0 (\_SB_.IOH1.MRPK)
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
Firmware did not grant requested _OSC control
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP5 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:05.0 (\_SB_.IOH0.MRP5)
pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
Firmware did not grant requested _OSC control
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP7 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:07.0 (\_SB_.IOH0.MRP7)
pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
Firmware did not grant requested _OSC control
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.PEX0 
acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0.PEX0 OSHP not found
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0 
acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0 OSHP not found
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Cannot get control of hotplug hardware for pci 0000:00:1c.0
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPI 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:07.0 (\_SB_.IOH1.MRPI)
pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPK 
acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:09.0 (\_SB_.IOH1.MRPK)
pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
pciehp: PCI Express Hot Plug Controller Driver version: 0.4

-- 
Jens Axboe

Re: pci-express hotplug

[Kenji Kaneshige]

Jens Axboe wrote:
> On Tue, Oct 13 2009, Kenji Kaneshige wrote:
>> Jens Axboe wrote:
>>> On Tue, Oct 13 2009, Kenji Kaneshige wrote:
>>>> Jens Axboe wrote:
>>>>> Hi,
>>>>>
>>>>> I'm trying to get pci-express hotplug working in a box here. I don't
>>>>> really care about the hotplug aspect, I just want the darn pci-e slots
>>>>> that are designated hotplug slots to actually WORK. When I load pciehp,
>>>>> I get:
>>>>>
>>>>> Firmware did not grant requested _OSC control
>>>>> Firmware did not grant requested _OSC control
>>>>> Firmware did not grant requested _OSC control
>>>>> Firmware did not grant requested _OSC control
>>>>> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
>>>>> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
>>>>> Firmware did not grant requested _OSC control
>>>>> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>>> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
>>>>> Firmware did not grant requested _OSC control
>>>>> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>>> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
>>>>> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
>>>>> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
>>>>> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>>>>
>>>>> and the devices in the hotplug slots stay off. Is this an ACPI/bios
>>>>> issue? How can I debug this?
>>>>>
>>>> Could you give me the result of "ls -lR /sys/bus/pci/slots/"
>>>> after loading pciehp?
>>> I have attached the result of that ls prior to loading pciehp/acpiphp
>>> (pre-load), after loading pciehp (pciehp-load), and with acpiphp loaded
>>> only as well (acpiphp-load).
>>>
>> Thank you for the info. From the information, I confirmed that hotplug
>> slots are detected by pciehp even though _OSC evaluation failed. There
>> are two ways to take control from the firmware through ACPI control
>> method. One is _OSC control method, and the other is OSHP control method.
>> I guess your ACPI fimware has both _OSC and OSHP on DSDT (ACPI Namespace),
>> and pciehp assumes that it took control through OSHP after the _OSC
>> evaluation failure. I think this pciehp's behavior is wrong because of
>> the following reasons and I think pciehp driver mis-detected the hotplug
>> slots on your environment because of this.
>>
>> - According to the PCI firmware specification, pciehp driver must use the
>>  result of _OSC, if the platform implements both _OSC and OSHP.
>> - OSHP control method seems only for SHPC, not for PCI Express native hot-
>>  plug. So pciehp must not evaluate OSHP to take control from firmware.
>>
>> To confirm this, could you send me the dmesg output after loading pciehp
>> with 'debug_acpi' of pci_hotplug (PCI hotplug core driver) enabled?
>> For example,
>>
>>    $ su
>>    # echo Y > /sys/module/pci_hotplug/parameters/debug_acpi
>>    # modprobe pciehp
>>    # dmesg
> 
> See below.
> 
>> And if it is possible, could you send me DSDT of your platform?
> 
> Not sure I can do that, I'll check.
> 
>> Anyway, my recommendation is using acpiphp on your environment because
>> your firmware didn't grant control over hotplug control through _OSC.
>> From the information, acpiphp also detects the hotplug slots successfully.
>> Please try "echo 1 > /sys/bus/pci/slots/<slot#>/power". It would turn on
>> the slot and initialize adapter card on the slot.
> 
> It does find the 4 slots correctly. But if I try to turn on the power,
> nothing happens and 'power' stays at 0. If I do the same with pciehp, I
> get the same hang as described when using pciehp with pciehp_force=1.
> But apparently this machine is getting a board replacement very soon, so
> it may solve itself. Unless you think it should work and there's
> something I can try to check, then lets just leave this issue until I
> get it upgraded and return from kernel summit / JLS.
> 

Could you try pciehp with "pciehp_debug" option enabled(*), and give me
the following information?

  - "cat /sys/bus/pci/slots/*/*" output
  - dmesg output after "echo 1 > /sys/bus/pci/slots/<slot#>/power"

(*) you can enable "pciehp_debug" option as follows

  # modprobe pciehp pciehp_debug

I'm not sure, but I think one of the possibility is that your hot-plug
controller doesn't support power controller. On such slot, the slot is
usually turned on automatically at adapter card insertion. So card
re-insertion might make some difference.

By the way, I would like to know if your system has SHPC based hotplug
slots. Could you load shpchp driver and send me the
"ls -lR /sys/bus/pci/slots/" output? And I would like to see the dmesg
output with "debug_acpi" of pci_hotplug enabled. For example,

    $ su
    # echo Y > /sys/module/pci_hotplug/parameters/debug_acpi
    # modprobe shpchp
    # ls -lR /sys/bus/pci/slots
    # dmesg

> 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
> Firmware did not grant requested _OSC control
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP5 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:05.0 (\_SB_.IOH0.MRP5)
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
> Firmware did not grant requested _OSC control
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP7 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:07.0 (\_SB_.IOH0.MRP7)
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
> Firmware did not grant requested _OSC control
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.PEX0 
> acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0.PEX0 OSHP not found
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0 
> acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0 OSHP not found
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Cannot get control of hotplug hardware for pci 0000:00:1c.0
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPI 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:07.0 (\_SB_.IOH1.MRPI)
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPK 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:09.0 (\_SB_.IOH1.MRPK)
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
> Firmware did not grant requested _OSC control
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP5 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:05.0 (\_SB_.IOH0.MRP5)
> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
> Firmware did not grant requested _OSC control
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.MRP7 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:00:07.0 (\_SB_.IOH0.MRP7)
> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0
> Firmware did not grant requested _OSC control
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0.PEX0 
> acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0.PEX0 OSHP not found
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH0 
> acpi_pcihp: acpi_run_oshp: acpi_run_oshp:\_SB_.IOH0 OSHP not found
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Cannot get control of hotplug hardware for pci 0000:00:1c.0
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPI 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:07.0 (\_SB_.IOH1.MRPI)
> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 _id 340e ss_vid 0 ss_did 0
> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Trying to get hotplug control for \_SB_.IOH1.MRPK 
> acpi_pcihp: acpi_get_hp_hw_control_from_firmware: Gained control for hotplug HW for pci 0000:80:09.0 (\_SB_.IOH1.MRPK)
> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
> 

It seems your environment has two PCI root bridges and one of them
(IOH0) provides both _OSC and OSHP, and the other (IOH1) provides
only OSHP. The _OSC under IOH0 doesn't grant hot-plug control, so
I think your system expects that at least the slots under IOH0 is
handled by acpiphp. But I don't have any idea about the slots
under IOH1 for now.

Thanks,
Kenji Kaneshige

Re: pci-express hotplug

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 6019 bytes --]

On Wed, Oct 14 2009, Kenji Kaneshige wrote:
> Jens Axboe wrote:
>> On Tue, Oct 13 2009, Kenji Kaneshige wrote:
>>> Jens Axboe wrote:
>>>> On Tue, Oct 13 2009, Kenji Kaneshige wrote:
>>>>> Jens Axboe wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm trying to get pci-express hotplug working in a box here. I don't
>>>>>> really care about the hotplug aspect, I just want the darn pci-e slots
>>>>>> that are designated hotplug slots to actually WORK. When I load pciehp,
>>>>>> I get:
>>>>>>
>>>>>> Firmware did not grant requested _OSC control
>>>>>> Firmware did not grant requested _OSC control
>>>>>> Firmware did not grant requested _OSC control
>>>>>> Firmware did not grant requested _OSC control
>>>>>> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
>>>>>> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
>>>>>> Firmware did not grant requested _OSC control
>>>>>> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>>>> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
>>>>>> Firmware did not grant requested _OSC control
>>>>>> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>>>> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
>>>>>> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
>>>>>> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
>>>>>> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>>>>>
>>>>>> and the devices in the hotplug slots stay off. Is this an ACPI/bios
>>>>>> issue? How can I debug this?
>>>>>>
>>>>> Could you give me the result of "ls -lR /sys/bus/pci/slots/"
>>>>> after loading pciehp?
>>>> I have attached the result of that ls prior to loading pciehp/acpiphp
>>>> (pre-load), after loading pciehp (pciehp-load), and with acpiphp loaded
>>>> only as well (acpiphp-load).
>>>>
>>> Thank you for the info. From the information, I confirmed that hotplug
>>> slots are detected by pciehp even though _OSC evaluation failed. There
>>> are two ways to take control from the firmware through ACPI control
>>> method. One is _OSC control method, and the other is OSHP control method.
>>> I guess your ACPI fimware has both _OSC and OSHP on DSDT (ACPI Namespace),
>>> and pciehp assumes that it took control through OSHP after the _OSC
>>> evaluation failure. I think this pciehp's behavior is wrong because of
>>> the following reasons and I think pciehp driver mis-detected the hotplug
>>> slots on your environment because of this.
>>>
>>> - According to the PCI firmware specification, pciehp driver must use the
>>>  result of _OSC, if the platform implements both _OSC and OSHP.
>>> - OSHP control method seems only for SHPC, not for PCI Express native hot-
>>>  plug. So pciehp must not evaluate OSHP to take control from firmware.
>>>
>>> To confirm this, could you send me the dmesg output after loading pciehp
>>> with 'debug_acpi' of pci_hotplug (PCI hotplug core driver) enabled?
>>> For example,
>>>
>>>    $ su
>>>    # echo Y > /sys/module/pci_hotplug/parameters/debug_acpi
>>>    # modprobe pciehp
>>>    # dmesg
>>
>> See below.
>>
>>> And if it is possible, could you send me DSDT of your platform?
>>
>> Not sure I can do that, I'll check.
>>
>>> Anyway, my recommendation is using acpiphp on your environment because
>>> your firmware didn't grant control over hotplug control through _OSC.
>>> From the information, acpiphp also detects the hotplug slots successfully.
>>> Please try "echo 1 > /sys/bus/pci/slots/<slot#>/power". It would turn on
>>> the slot and initialize adapter card on the slot.
>>
>> It does find the 4 slots correctly. But if I try to turn on the power,
>> nothing happens and 'power' stays at 0. If I do the same with pciehp, I
>> get the same hang as described when using pciehp with pciehp_force=1.
>> But apparently this machine is getting a board replacement very soon, so
>> it may solve itself. Unless you think it should work and there's
>> something I can try to check, then lets just leave this issue until I
>> get it upgraded and return from kernel summit / JLS.
>>
>
> Could you try pciehp with "pciehp_debug" option enabled(*), and give me
> the following information?

I've attached the output of loading pciehp with the debug option
enabled.

>  - "cat /sys/bus/pci/slots/*/*" output

Attached as slots

>  - dmesg output after "echo 1 > /sys/bus/pci/slots/<slot#>/power"

# echo 1 > /sys/bus/pci/slots/1/power
pciehp 0000:00:05.0:pcie04: Power fault on Slot(1)
pciehp 0000:00:05.0:pcie04: Power fault bit 0 set
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
[...]

That last line repeats infinitely.

> (*) you can enable "pciehp_debug" option as follows
>
>  # modprobe pciehp pciehp_debug
>
> I'm not sure, but I think one of the possibility is that your hot-plug
> controller doesn't support power controller. On such slot, the slot is
> usually turned on automatically at adapter card insertion. So card
> re-insertion might make some difference.

The dmesg seems to suggest it has a power controller. I also tried
removing and replugging the card, which is detected:

pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 8
pciehp 0000:00:05.0:pcie04: Presence/Notify input change
pciehp 0000:00:05.0:pcie04: Card not present on Slot(1)
pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 8
pciehp 0000:00:05.0:pcie04: Presence/Notify input change
pciehp 0000:00:05.0:pcie04: Card present on Slot(1)

but the slot isn't powered on still.

> By the way, I would like to know if your system has SHPC based hotplug
> slots. Could you load shpchp driver and send me the
> "ls -lR /sys/bus/pci/slots/" output? And I would like to see the dmesg
> output with "debug_acpi" of pci_hotplug enabled. For example,
>
>    $ su
>    # echo Y > /sys/module/pci_hotplug/parameters/debug_acpi
>    # modprobe shpchp

This doesn't dump anything but:

shpchp: Standard Hot Plug PCI Controller Driver version: 0.4

>    # ls -lR /sys/bus/pci/slots
>    # dmesg

Attached.

-- 
Jens Axboe


[-- Attachment #2: ls-lr-slots --]
[-- Type: text/plain, Size: 2610 bytes --]

^[[0m/sys/bus/pci/slots/:
total 0
drwxr-xr-x 2 root root 0 2009-10-14 10:44 ^[[01;34m1^[[0m/
drwxr-xr-x 2 root root 0 2009-10-14 10:44 ^[[01;34m2^[[0m/
drwxr-xr-x 2 root root 0 2009-10-14 10:44 ^[[01;34m6^[[0m/
drwxr-xr-x 2 root root 0 2009-10-14 10:44 ^[[01;34m7^[[0m/

/sys/bus/pci/slots/1:
total 0
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0madapter^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0maddress^[[0m
-rw-r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mattention^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mcur_bus_speed^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mlatch^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mmax_bus_speed^[[0m
lrwxrwxrwx 1 root root    0 2009-10-14 10:44 ^[[01;36mmodule^[[0m -> ^[[01;34m../../../../module/pciehp^[[0m/
-rw-r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mpower^[[0m

/sys/bus/pci/slots/2:
total 0
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0madapter^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0maddress^[[0m
-rw-r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mattention^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mcur_bus_speed^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mlatch^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mmax_bus_speed^[[0m
lrwxrwxrwx 1 root root    0 2009-10-14 10:44 ^[[01;36mmodule^[[0m -> ^[[01;34m../../../../module/pciehp^[[0m/
-rw-r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mpower^[[0m

/sys/bus/pci/slots/6:
total 0
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0madapter^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0maddress^[[0m
-rw-r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mattention^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mcur_bus_speed^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mlatch^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mmax_bus_speed^[[0m
lrwxrwxrwx 1 root root    0 2009-10-14 10:44 ^[[01;36mmodule^[[0m -> ^[[01;34m../../../../module/pciehp^[[0m/
-rw-r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mpower^[[0m

/sys/bus/pci/slots/7:
total 0
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0madapter^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0maddress^[[0m
-rw-r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mattention^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mcur_bus_speed^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mlatch^[[0m
-r--r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mmax_bus_speed^[[0m
lrwxrwxrwx 1 root root    0 2009-10-14 10:44 ^[[01;36mmodule^[[0m -> ^[[01;34m../../../../module/pciehp^[[0m/
-rw-r--r-- 1 root root 4096 2009-10-14 10:44 ^[[0mpower^[[0m
^[[m

[-- Attachment #3: slots --]
[-- Type: text/plain, Size: 196 bytes --]

Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
pciehp 0000:

[-- Attachment #4: pciehp-load-debug --]
[-- Type: text/plain, Size: 9400 bytes --]

Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
pciehp 0000:00:05.0:pcie04: Hotplug Controller:
pciehp 0000:00:05.0:pcie04:   Seg/Bus/Dev/Func/IRQ : 0000:00:05.0 IRQ 75
pciehp 0000:00:05.0:pcie04:   Vendor ID            : 0x8086
pciehp 0000:00:05.0:pcie04:   Device ID            : 0x340c
pciehp 0000:00:05.0:pcie04:   Subsystem ID         : 0x0000
pciehp 0000:00:05.0:pcie04:   Subsystem Vendor ID  : 0x0000
pciehp 0000:00:05.0:pcie04:   PCIe Cap offset      : 0x90
pciehp 0000:00:05.0:pcie04:   PCI resource [7]     : 0x1000@0x2000
pciehp 0000:00:05.0:pcie04:   PCI resource [8]     : 0x1000000@0x92900000
pciehp 0000:00:05.0:pcie04:   PCI resource [9]     : 0x4000000@0xc7f00000
pciehp 0000:00:05.0:pcie04: Slot Capabilities      : 0x0008005b
pciehp 0000:00:05.0:pcie04:   Physical Slot Number : 1
pciehp 0000:00:05.0:pcie04:   Attention Button     : yes
pciehp 0000:00:05.0:pcie04:   Power Controller     : yes
pciehp 0000:00:05.0:pcie04:   MRL Sensor           :  no
pciehp 0000:00:05.0:pcie04:   Attention Indicator  : yes
pciehp 0000:00:05.0:pcie04:   Power Indicator      : yes
pciehp 0000:00:05.0:pcie04:   Hot-Plug Surprise    :  no
pciehp 0000:00:05.0:pcie04:   EMI Present          :  no
pciehp 0000:00:05.0:pcie04:   Command Completed    : yes
pciehp 0000:00:05.0:pcie04: Slot Status            : 0x0040
pciehp 0000:00:05.0:pcie04: Slot Control           : 0x07cf
pciehp 0000:00:05.0:pcie04: Link Active Reporting supported
pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
pciehp 0000:00:05.0:pcie04: Registering domain:bus:dev=0000:08:00 sun=1
pciehp 0000:00:05.0:pcie04: get_power_status: physical_slot = 1
pciehp 0000:00:05.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 7c0
pciehp 0000:00:05.0:pcie04: get_attention_status: physical_slot = 1
pciehp 0000:00:05.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 7c0
pciehp 0000:00:05.0:pcie04: get_latch_status: physical_slot = 1
pciehp 0000:00:05.0:pcie04: get_adapter_status: physical_slot = 1
pciehp 0000:00:05.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
Firmware did not grant requested _OSC control
pciehp 0000:00:07.0:pcie04: Hotplug Controller:
pciehp 0000:00:07.0:pcie04:   Seg/Bus/Dev/Func/IRQ : 0000:00:07.0 IRQ 76
pciehp 0000:00:07.0:pcie04:   Vendor ID            : 0x8086
pciehp 0000:00:07.0:pcie04:   Device ID            : 0x340e
pciehp 0000:00:07.0:pcie04:   Subsystem ID         : 0x0000
pciehp 0000:00:07.0:pcie04:   Subsystem Vendor ID  : 0x0000
pciehp 0000:00:07.0:pcie04:   PCIe Cap offset      : 0x90
pciehp 0000:00:07.0:pcie04:   PCI resource [7]     : 0x1000@0x1000
pciehp 0000:00:07.0:pcie04:   PCI resource [8]     : 0x1000000@0x91900000
pciehp 0000:00:07.0:pcie04:   PCI resource [9]     : 0x4000000@0xcbf00000
pciehp 0000:00:07.0:pcie04: Slot Capabilities      : 0x0010005b
pciehp 0000:00:07.0:pcie04:   Physical Slot Number : 2
pciehp 0000:00:07.0:pcie04:   Attention Button     : yes
pciehp 0000:00:07.0:pcie04:   Power Controller     : yes
pciehp 0000:00:07.0:pcie04:   MRL Sensor           :  no
pciehp 0000:00:07.0:pcie04:   Attention Indicator  : yes
pciehp 0000:00:07.0:pcie04:   Power Indicator      : yes
pciehp 0000:00:07.0:pcie04:   Hot-Plug Surprise    :  no
pciehp 0000:00:07.0:pcie04:   EMI Present          :  no
pciehp 0000:00:07.0:pcie04:   Command Completed    : yes
pciehp 0000:00:07.0:pcie04: Slot Status            : 0x0000
pciehp 0000:00:07.0:pcie04: Slot Control           : 0x07cf
pciehp 0000:00:07.0:pcie04: Link Active Reporting supported
pciehp 0000:00:07.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
pciehp 0000:00:07.0:pcie04: Registering domain:bus:dev=0000:0b:00 sun=2
pciehp 0000:00:07.0:pcie04: get_power_status: physical_slot = 2
pciehp 0000:00:07.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 7c0
pciehp 0000:00:07.0:pcie04: get_attention_status: physical_slot = 2
pciehp 0000:00:07.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 7c0
pciehp 0000:00:07.0:pcie04: get_latch_status: physical_slot = 2
pciehp 0000:00:07.0:pcie04: get_adapter_status: physical_slot = 2
pciehp 0000:00:07.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:07.0:pcie04: Command not completed in 1000 msec
pciehp 0000:00:07.0:pcie04: pciehp_power_off_slot: SLOTCTRL a8 write cmd 400
pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
Firmware did not grant requested _OSC control
pciehp 0000:80:07.0:pcie04: Hotplug Controller:
pciehp 0000:80:07.0:pcie04:   Seg/Bus/Dev/Func/IRQ : 0000:80:07.0 IRQ 84
pciehp 0000:80:07.0:pcie04:   Vendor ID            : 0x8086
pciehp 0000:80:07.0:pcie04:   Device ID            : 0x340e
pciehp 0000:80:07.0:pcie04:   Subsystem ID         : 0x0000
pciehp 0000:80:07.0:pcie04:   Subsystem Vendor ID  : 0x0000
pciehp 0000:80:07.0:pcie04:   PCIe Cap offset      : 0x90
pciehp 0000:80:07.0:pcie04:   PCI resource [7]     : 0x1000@0x9000
pciehp 0000:80:07.0:pcie04:   PCI resource [8]     : 0x1000000@0xd1000000
pciehp 0000:80:07.0:pcie04:   PCI resource [9]     : 0x4000000@0xf3f00000
pciehp 0000:80:07.0:pcie04: Slot Capabilities      : 0x0030005b
pciehp 0000:80:07.0:pcie04:   Physical Slot Number : 6
pciehp 0000:80:07.0:pcie04:   Attention Button     : yes
pciehp 0000:80:07.0:pcie04:   Power Controller     : yes
pciehp 0000:80:07.0:pcie04:   MRL Sensor           :  no
pciehp 0000:80:07.0:pcie04:   Attention Indicator  : yes
pciehp 0000:80:07.0:pcie04:   Power Indicator      : yes
pciehp 0000:80:07.0:pcie04:   Hot-Plug Surprise    :  no
pciehp 0000:80:07.0:pcie04:   EMI Present          :  no
pciehp 0000:80:07.0:pcie04:   Command Completed    : yes
pciehp 0000:80:07.0:pcie04: Slot Status            : 0x0000
pciehp 0000:80:07.0:pcie04: Slot Control           : 0x07cf
pciehp 0000:80:07.0:pcie04: Link Active Reporting supported
pciehp 0000:80:07.0:pcie04: Command not completed in 1000 msec
pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
pciehp 0000:80:07.0:pcie04: Registering domain:bus:dev=0000:84:00 sun=6
pciehp 0000:80:07.0:pcie04: get_power_status: physical_slot = 6
pciehp 0000:80:07.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 7c0
pciehp 0000:80:07.0:pcie04: get_attention_status: physical_slot = 6
pciehp 0000:80:07.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 7c0
pciehp 0000:80:07.0:pcie04: get_latch_status: physical_slot = 6
pciehp 0000:80:07.0:pcie04: get_adapter_status: physical_slot = 6
pciehp 0000:80:07.0:pcie04: Command not completed in 1000 msec
pciehp 0000:80:07.0:pcie04: Command not completed in 1000 msec
pciehp 0000:80:07.0:pcie04: pciehp_power_off_slot: SLOTCTRL a8 write cmd 400
pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
pciehp 0000:80:09.0:pcie04: Hotplug Controller:
pciehp 0000:80:09.0:pcie04:   Seg/Bus/Dev/Func/IRQ : 0000:80:09.0 IRQ 85
pciehp 0000:80:09.0:pcie04:   Vendor ID            : 0x8086
pciehp 0000:80:09.0:pcie04:   Device ID            : 0x3410
pciehp 0000:80:09.0:pcie04:   Subsystem ID         : 0x0000
pciehp 0000:80:09.0:pcie04:   Subsystem Vendor ID  : 0x0000
pciehp 0000:80:09.0:pcie04:   PCIe Cap offset      : 0x90
pciehp 0000:80:09.0:pcie04:   PCI resource [7]     : 0x1000@0x8000
pciehp 0000:80:09.0:pcie04:   PCI resource [8]     : 0x1000000@0xd0000000
pciehp 0000:80:09.0:pcie04:   PCI resource [9]     : 0x4000000@0xf7f00000
pciehp 0000:80:09.0:pcie04: Slot Capabilities      : 0x0038005b
pciehp 0000:80:09.0:pcie04:   Physical Slot Number : 7
pciehp 0000:80:09.0:pcie04:   Attention Button     : yes
pciehp 0000:80:09.0:pcie04:   Power Controller     : yes
pciehp 0000:80:09.0:pcie04:   MRL Sensor           :  no
pciehp 0000:80:09.0:pcie04:   Attention Indicator  : yes
pciehp 0000:80:09.0:pcie04:   Power Indicator      : yes
pciehp 0000:80:09.0:pcie04:   Hot-Plug Surprise    :  no
pciehp 0000:80:09.0:pcie04:   EMI Present          :  no
pciehp 0000:80:09.0:pcie04:   Command Completed    : yes
pciehp 0000:80:09.0:pcie04: Slot Status            : 0x0000
pciehp 0000:80:09.0:pcie04: Slot Control           : 0x07cf
pciehp 0000:80:09.0:pcie04: Link Active Reporting supported
pciehp 0000:80:09.0:pcie04: Command not completed in 1000 msec
pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
pciehp 0000:80:09.0:pcie04: Registering domain:bus:dev=0000:87:00 sun=7
pciehp 0000:80:09.0:pcie04: get_power_status: physical_slot = 7
pciehp 0000:80:09.0:pcie04: pciehp_get_power_status: SLOTCTRL a8 value read 7c0
pciehp 0000:80:09.0:pcie04: get_attention_status: physical_slot = 7
pciehp 0000:80:09.0:pcie04: pciehp_get_attention_status: SLOTCTRL a8, value read 7c0
pciehp 0000:80:09.0:pcie04: get_latch_status: physical_slot = 7
pciehp 0000:80:09.0:pcie04: get_adapter_status: physical_slot = 7
pciehp 0000:80:09.0:pcie04: Command not completed in 1000 msec
pciehp 0000:80:09.0:pcie04: Command not completed in 1000 msec
pciehp 0000:80:09.0:pcie04: pciehp_power_off_slot: SLOTCTRL a8 write cmd 400
pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
pciehp: pcie_port_service_register = 0
pciehp: PCI Express Hot Plug Controller Driver version: 0.4

Re: pci-express hotplug

[Kenji Kaneshige]

Jens Axboe wrote:
> On Wed, Oct 14 2009, Kenji Kaneshige wrote:
>> Jens Axboe wrote:
>>> On Tue, Oct 13 2009, Kenji Kaneshige wrote:
>>>> Jens Axboe wrote:
>>>>> On Tue, Oct 13 2009, Kenji Kaneshige wrote:
>>>>>> Jens Axboe wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm trying to get pci-express hotplug working in a box here. I don't
>>>>>>> really care about the hotplug aspect, I just want the darn pci-e slots
>>>>>>> that are designated hotplug slots to actually WORK. When I load pciehp,
>>>>>>> I get:
>>>>>>>
>>>>>>> Firmware did not grant requested _OSC control
>>>>>>> Firmware did not grant requested _OSC control
>>>>>>> Firmware did not grant requested _OSC control
>>>>>>> Firmware did not grant requested _OSC control
>>>>>>> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
>>>>>>> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
>>>>>>> Firmware did not grant requested _OSC control
>>>>>>> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>>>>> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
>>>>>>> Firmware did not grant requested _OSC control
>>>>>>> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>>>>> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
>>>>>>> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
>>>>>>> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
>>>>>>> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>>>>>>
>>>>>>> and the devices in the hotplug slots stay off. Is this an ACPI/bios
>>>>>>> issue? How can I debug this?
>>>>>>>
>>>>>> Could you give me the result of "ls -lR /sys/bus/pci/slots/"
>>>>>> after loading pciehp?
>>>>> I have attached the result of that ls prior to loading pciehp/acpiphp
>>>>> (pre-load), after loading pciehp (pciehp-load), and with acpiphp loaded
>>>>> only as well (acpiphp-load).
>>>>>
>>>> Thank you for the info. From the information, I confirmed that hotplug
>>>> slots are detected by pciehp even though _OSC evaluation failed. There
>>>> are two ways to take control from the firmware through ACPI control
>>>> method. One is _OSC control method, and the other is OSHP control method.
>>>> I guess your ACPI fimware has both _OSC and OSHP on DSDT (ACPI Namespace),
>>>> and pciehp assumes that it took control through OSHP after the _OSC
>>>> evaluation failure. I think this pciehp's behavior is wrong because of
>>>> the following reasons and I think pciehp driver mis-detected the hotplug
>>>> slots on your environment because of this.
>>>>
>>>> - According to the PCI firmware specification, pciehp driver must use the
>>>>  result of _OSC, if the platform implements both _OSC and OSHP.
>>>> - OSHP control method seems only for SHPC, not for PCI Express native hot-
>>>>  plug. So pciehp must not evaluate OSHP to take control from firmware.
>>>>
>>>> To confirm this, could you send me the dmesg output after loading pciehp
>>>> with 'debug_acpi' of pci_hotplug (PCI hotplug core driver) enabled?
>>>> For example,
>>>>
>>>>    $ su
>>>>    # echo Y > /sys/module/pci_hotplug/parameters/debug_acpi
>>>>    # modprobe pciehp
>>>>    # dmesg
>>> See below.
>>>
>>>> And if it is possible, could you send me DSDT of your platform?
>>> Not sure I can do that, I'll check.
>>>
>>>> Anyway, my recommendation is using acpiphp on your environment because
>>>> your firmware didn't grant control over hotplug control through _OSC.
>>>> From the information, acpiphp also detects the hotplug slots successfully.
>>>> Please try "echo 1 > /sys/bus/pci/slots/<slot#>/power". It would turn on
>>>> the slot and initialize adapter card on the slot.
>>> It does find the 4 slots correctly. But if I try to turn on the power,
>>> nothing happens and 'power' stays at 0. If I do the same with pciehp, I
>>> get the same hang as described when using pciehp with pciehp_force=1.
>>> But apparently this machine is getting a board replacement very soon, so
>>> it may solve itself. Unless you think it should work and there's
>>> something I can try to check, then lets just leave this issue until I
>>> get it upgraded and return from kernel summit / JLS.
>>>
>> Could you try pciehp with "pciehp_debug" option enabled(*), and give me
>> the following information?
> 
> I've attached the output of loading pciehp with the debug option
> enabled.
> 
>>  - "cat /sys/bus/pci/slots/*/*" output
> 
> Attached as slots
> 
>>  - dmesg output after "echo 1 > /sys/bus/pci/slots/<slot#>/power"
> 
> # echo 1 > /sys/bus/pci/slots/1/power
> pciehp 0000:00:05.0:pcie04: Power fault on Slot(1)
> pciehp 0000:00:05.0:pcie04: Power fault bit 0 set
> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
> [...]
> 
> That last line repeats infinitely.

Thank you very much for information.

The direct cause of the problem that your slot was not turned on
is power fault. I guess acpiphp is suffering the same problem.
Unfortunately, it's difficult for me to analyze the root cause
of this power fault. Please ask the hardware vendor about it. I
hope board replacement will fix the problem.

By the way, thanks to your report, I noticed the several points
that might need to be fixed as follows. I'll try to improve that.

- The message "Firmware did not grant requested _OSC control" is
  confusing and similar message is already displayed by the caller
  of acpi_pci_osc_control_set(). Therefore, it should be removed.

- If the platform has _OSC control method, OSHP should not be
  evaluated.

- (maybe) pciehp must not evaluate OSHP (But your platform seems
  to provide OSHP for several PCIe hotplug slots because your
  platform provides OSHP even though it doesn't have any SHPC
  based PCI/PCI-X hot-plug slots. I need to check PCI firmware
  spec again).

- pciehp needs something to prevent power fault interrupt storm.

Thanks,
Kenji Kaneshige

Re: pci-express hotplug

[Jens Axboe]

On Thu, Oct 15 2009, Kenji Kaneshige wrote:
> Jens Axboe wrote:
>> On Wed, Oct 14 2009, Kenji Kaneshige wrote:
>>> Jens Axboe wrote:
>>>> On Tue, Oct 13 2009, Kenji Kaneshige wrote:
>>>>> Jens Axboe wrote:
>>>>>> On Tue, Oct 13 2009, Kenji Kaneshige wrote:
>>>>>>> Jens Axboe wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm trying to get pci-express hotplug working in a box here. I don't
>>>>>>>> really care about the hotplug aspect, I just want the darn pci-e slots
>>>>>>>> that are designated hotplug slots to actually WORK. When I load pciehp,
>>>>>>>> I get:
>>>>>>>>
>>>>>>>> Firmware did not grant requested _OSC control
>>>>>>>> Firmware did not grant requested _OSC control
>>>>>>>> Firmware did not grant requested _OSC control
>>>>>>>> Firmware did not grant requested _OSC control
>>>>>>>> pciehp 0000:00:05.0:pcie04: HPC vendor_id 8086 device_id 340c ss_vid 0 ss_did 0
>>>>>>>> pciehp 0000:00:05.0:pcie04: service driver pciehp loaded
>>>>>>>> Firmware did not grant requested _OSC control
>>>>>>>> pciehp 0000:00:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>>>>>> pciehp 0000:00:07.0:pcie04: service driver pciehp loaded
>>>>>>>> Firmware did not grant requested _OSC control
>>>>>>>> pciehp 0000:80:07.0:pcie04: HPC vendor_id 8086 device_id 340e ss_vid 0 ss_did 0
>>>>>>>> pciehp 0000:80:07.0:pcie04: service driver pciehp loaded
>>>>>>>> pciehp 0000:80:09.0:pcie04: HPC vendor_id 8086 device_id 3410 ss_vid 0 ss_did 0
>>>>>>>> pciehp 0000:80:09.0:pcie04: service driver pciehp loaded
>>>>>>>> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>>>>>>>
>>>>>>>> and the devices in the hotplug slots stay off. Is this an ACPI/bios
>>>>>>>> issue? How can I debug this?
>>>>>>>>
>>>>>>> Could you give me the result of "ls -lR /sys/bus/pci/slots/"
>>>>>>> after loading pciehp?
>>>>>> I have attached the result of that ls prior to loading pciehp/acpiphp
>>>>>> (pre-load), after loading pciehp (pciehp-load), and with acpiphp loaded
>>>>>> only as well (acpiphp-load).
>>>>>>
>>>>> Thank you for the info. From the information, I confirmed that hotplug
>>>>> slots are detected by pciehp even though _OSC evaluation failed. There
>>>>> are two ways to take control from the firmware through ACPI control
>>>>> method. One is _OSC control method, and the other is OSHP control method.
>>>>> I guess your ACPI fimware has both _OSC and OSHP on DSDT (ACPI Namespace),
>>>>> and pciehp assumes that it took control through OSHP after the _OSC
>>>>> evaluation failure. I think this pciehp's behavior is wrong because of
>>>>> the following reasons and I think pciehp driver mis-detected the hotplug
>>>>> slots on your environment because of this.
>>>>>
>>>>> - According to the PCI firmware specification, pciehp driver must use the
>>>>>  result of _OSC, if the platform implements both _OSC and OSHP.
>>>>> - OSHP control method seems only for SHPC, not for PCI Express native hot-
>>>>>  plug. So pciehp must not evaluate OSHP to take control from firmware.
>>>>>
>>>>> To confirm this, could you send me the dmesg output after loading pciehp
>>>>> with 'debug_acpi' of pci_hotplug (PCI hotplug core driver) enabled?
>>>>> For example,
>>>>>
>>>>>    $ su
>>>>>    # echo Y > /sys/module/pci_hotplug/parameters/debug_acpi
>>>>>    # modprobe pciehp
>>>>>    # dmesg
>>>> See below.
>>>>
>>>>> And if it is possible, could you send me DSDT of your platform?
>>>> Not sure I can do that, I'll check.
>>>>
>>>>> Anyway, my recommendation is using acpiphp on your environment because
>>>>> your firmware didn't grant control over hotplug control through _OSC.
>>>>> From the information, acpiphp also detects the hotplug slots successfully.
>>>>> Please try "echo 1 > /sys/bus/pci/slots/<slot#>/power". It would turn on
>>>>> the slot and initialize adapter card on the slot.
>>>> It does find the 4 slots correctly. But if I try to turn on the power,
>>>> nothing happens and 'power' stays at 0. If I do the same with pciehp, I
>>>> get the same hang as described when using pciehp with pciehp_force=1.
>>>> But apparently this machine is getting a board replacement very soon, so
>>>> it may solve itself. Unless you think it should work and there's
>>>> something I can try to check, then lets just leave this issue until I
>>>> get it upgraded and return from kernel summit / JLS.
>>>>
>>> Could you try pciehp with "pciehp_debug" option enabled(*), and give me
>>> the following information?
>>
>> I've attached the output of loading pciehp with the debug option
>> enabled.
>>
>>>  - "cat /sys/bus/pci/slots/*/*" output
>>
>> Attached as slots
>>
>>>  - dmesg output after "echo 1 > /sys/bus/pci/slots/<slot#>/power"
>>
>> # echo 1 > /sys/bus/pci/slots/1/power
>> pciehp 0000:00:05.0:pcie04: Power fault on Slot(1)
>> pciehp 0000:00:05.0:pcie04: Power fault bit 0 set
>> pciehp 0000:00:05.0:pcie04: pcie_isr: intr_loc 2
>> [...]
>>
>> That last line repeats infinitely.
>
> Thank you very much for information.
>
> The direct cause of the problem that your slot was not turned on
> is power fault. I guess acpiphp is suffering the same problem.
> Unfortunately, it's difficult for me to analyze the root cause
> of this power fault. Please ask the hardware vendor about it. I
> hope board replacement will fix the problem.

OK, I'll try with the new board when back and see what happens. If the
power fault persists, I'll poke the vendor about it.

> By the way, thanks to your report, I noticed the several points
> that might need to be fixed as follows. I'll try to improve that.
>
> - The message "Firmware did not grant requested _OSC control" is
>  confusing and similar message is already displayed by the caller
>  of acpi_pci_osc_control_set(). Therefore, it should be removed.

It's one of those messages that mean very little to you, unless you have
an understanding of how hotplug is supposed to work. So removing it
sounds god.

> - If the platform has _OSC control method, OSHP should not be
>  evaluated.
>
> - (maybe) pciehp must not evaluate OSHP (But your platform seems
>  to provide OSHP for several PCIe hotplug slots because your
>  platform provides OSHP even though it doesn't have any SHPC
>  based PCI/PCI-X hot-plug slots. I need to check PCI firmware
>  spec again).
>
> - pciehp needs something to prevent power fault interrupt storm.

Definitely, it essentially hangs the box and requires a reboot.

Thanks a lot for looking into these issues, I'll be back with a status
message when I've tried the new board.

-- 
Jens Axboe