Cabled PCIE device hotplug/hot-removal

Cabled PCIE device hotplug/hot-removal

[Shlomo Pongratz]

Hi,

I have a  PCIe device connected with cables to a host (kernel 3.5.0-rc7+).
What are the handler I need to add to a driver in order to catch cable
pull-out / push-in?

The pcisig presentation
"http://www.pcisig.com/developers/main/training_materials/get_document?doc_idôca40ec1721aa7fadd05e2903b0172c36656294"
talks about PERST# detection for cabled PCIe hotplug.

Does the kernel detect it?

None of the pci_driver or pci_error_handlers callbacks were called
when using powered down/up the device (electrically).

Thanks.

Shlomo Pongratz

Cabled PCIE device hotplug/hot-removal

[Shlomo Pongratz]

Hi,

I have a  PCIe device connected with cables to a host (kernel 3.5.0-rc7+).
What are the handler I need to add to a driver in order to catch cable
pull-out / push-in?

The pcisig presentation
"http://www.pcisig.com/developers/main/training_materials/get_document?doc_id=f4ca40ec1721aa7fadd05e2903b0172c36656294"
talks about PERST# detection for cabled PCIe hotplug.

Does the kernel detect it?

None of the pci_driver or pci_error_handlers callbacks were called
when using powered down/up the device (electrically).

Thanks.

Shlomo Pongratz

Re: Cabled PCIE device hotplug/hot-removal

[Greg KH]

On Thu, Jul 26, 2012 at 08:03:09PM +0300, Shlomo Pongratz wrote:
> Hi,
> 
> I have a  PCIe device connected with cables to a host (kernel 3.5.0-rc7+).
> What are the handler I need to add to a driver in order to catch cable
> pull-out / push-in?

You shouldn't need anything special, aren't you already getting probe
and disconnect callbacks when the device is discovered and removed?

> The pcisig presentation
> "http://www.pcisig.com/developers/main/training_materials/get_document?doc_idôca40ec1721aa7fadd05e2903b0172c36656294"
> talks about PERST# detection for cabled PCIe hotplug.
> 
> Does the kernel detect it?
> 
> None of the pci_driver or pci_error_handlers callbacks were called
> when using powered down/up the device (electrically).

Do you have the pciehp driver loaded properly when this happens?

thanks,

greg k-h

Re: Cabled PCIE device hotplug/hot-removal

[Greg KH]

On Thu, Jul 26, 2012 at 08:03:09PM +0300, Shlomo Pongratz wrote:
> Hi,
> 
> I have a  PCIe device connected with cables to a host (kernel 3.5.0-rc7+).
> What are the handler I need to add to a driver in order to catch cable
> pull-out / push-in?

You shouldn't need anything special, aren't you already getting probe
and disconnect callbacks when the device is discovered and removed?

> The pcisig presentation
> "http://www.pcisig.com/developers/main/training_materials/get_document?doc_id=f4ca40ec1721aa7fadd05e2903b0172c36656294"
> talks about PERST# detection for cabled PCIe hotplug.
> 
> Does the kernel detect it?
> 
> None of the pci_driver or pci_error_handlers callbacks were called
> when using powered down/up the device (electrically).

Do you have the pciehp driver loaded properly when this happens?

thanks,

greg k-h

Re: Cabled PCIE device hotplug/hot-removal

[Shlomo Pongratz]

On Thu, Jul 26, 2012 at 8:09 PM, Greg KH <greg@kroah.com> wrote:
> On Thu, Jul 26, 2012 at 08:03:09PM +0300, Shlomo Pongratz wrote:
>> Hi,
>>
>> I have a  PCIe device connected with cables to a host (kernel 3.5.0-rc7+).
>> What are the handler I need to add to a driver in order to catch cable
>> pull-out / push-in?
>
> You shouldn't need anything special, aren't you already getting probe
> and disconnect callbacks when the device is discovered and removed?

When the device is loaded the probe is called
When I "echo 1" to the remove sysfs entry the remove callback is called.
But when I power down the card no callback is called.
>
>> The pcisig presentation
>> "http://www.pcisig.com/developers/main/training_materials/get_document?doc_idôca40ec1721aa7fadd05e2903b0172c36656294"
>> talks about PERST# detection for cabled PCIe hotplug.
>>
>> Does the kernel detect it?
>>
>> None of the pci_driver or pci_error_handlers callbacks were called
>> when using powered down/up the device (electrically).
>
> Do you have the pciehp driver loaded properly when this happens?

My {dot}config includes it into the kernel, i.e.
CONFIG_HOTPLUG_PCI_PCIE=y

>
> thanks,
>
> greg k-h

Thanks,

Re: Cabled PCIE device hotplug/hot-removal

[Shlomo Pongratz]

On Thu, Jul 26, 2012 at 8:09 PM, Greg KH <greg@kroah.com> wrote:
> On Thu, Jul 26, 2012 at 08:03:09PM +0300, Shlomo Pongratz wrote:
>> Hi,
>>
>> I have a  PCIe device connected with cables to a host (kernel 3.5.0-rc7+).
>> What are the handler I need to add to a driver in order to catch cable
>> pull-out / push-in?
>
> You shouldn't need anything special, aren't you already getting probe
> and disconnect callbacks when the device is discovered and removed?

When the device is loaded the probe is called
When I "echo 1" to the remove sysfs entry the remove callback is called.
But when I power down the card no callback is called.
>
>> The pcisig presentation
>> "http://www.pcisig.com/developers/main/training_materials/get_document?doc_id=f4ca40ec1721aa7fadd05e2903b0172c36656294"
>> talks about PERST# detection for cabled PCIe hotplug.
>>
>> Does the kernel detect it?
>>
>> None of the pci_driver or pci_error_handlers callbacks were called
>> when using powered down/up the device (electrically).
>
> Do you have the pciehp driver loaded properly when this happens?

My {dot}config includes it into the kernel, i.e.
CONFIG_HOTPLUG_PCI_PCIE=y

>
> thanks,
>
> greg k-h

Thanks,

Re: Cabled PCIE device hotplug/hot-removal

[Greg KH]

On Sun, Jul 29, 2012 at 03:36:09PM +0300, Shlomo Pongratz wrote:
> On Thu, Jul 26, 2012 at 8:09 PM, Greg KH <greg@kroah.com> wrote:
> > On Thu, Jul 26, 2012 at 08:03:09PM +0300, Shlomo Pongratz wrote:
> >> Hi,
> >>
> >> I have a  PCIe device connected with cables to a host (kernel 3.5.0-rc7+).
> >> What are the handler I need to add to a driver in order to catch cable
> >> pull-out / push-in?
> >
> > You shouldn't need anything special, aren't you already getting probe
> > and disconnect callbacks when the device is discovered and removed?
> 
> When the device is loaded the probe is called
> When I "echo 1" to the remove sysfs entry the remove callback is called.
> But when I power down the card no callback is called.

Then I would blame your pci hotplug controller, that is its
responsibility.

> >> The pcisig presentation
> >> "http://www.pcisig.com/developers/main/training_materials/get_document?doc_idôca40ec1721aa7fadd05e2903b0172c36656294"
> >> talks about PERST# detection for cabled PCIe hotplug.
> >>
> >> Does the kernel detect it?
> >>
> >> None of the pci_driver or pci_error_handlers callbacks were called
> >> when using powered down/up the device (electrically).
> >
> > Do you have the pciehp driver loaded properly when this happens?
> 
> My {dot}config includes it into the kernel, i.e.
> CONFIG_HOTPLUG_PCI_PCIE=y

And is it working properly?  Does your hardware support this (i.e. your
BIOS?)

greg k-h

Re: Cabled PCIE device hotplug/hot-removal

[Greg KH]

On Sun, Jul 29, 2012 at 03:36:09PM +0300, Shlomo Pongratz wrote:
> On Thu, Jul 26, 2012 at 8:09 PM, Greg KH <greg@kroah.com> wrote:
> > On Thu, Jul 26, 2012 at 08:03:09PM +0300, Shlomo Pongratz wrote:
> >> Hi,
> >>
> >> I have a  PCIe device connected with cables to a host (kernel 3.5.0-rc7+).
> >> What are the handler I need to add to a driver in order to catch cable
> >> pull-out / push-in?
> >
> > You shouldn't need anything special, aren't you already getting probe
> > and disconnect callbacks when the device is discovered and removed?
> 
> When the device is loaded the probe is called
> When I "echo 1" to the remove sysfs entry the remove callback is called.
> But when I power down the card no callback is called.

Then I would blame your pci hotplug controller, that is its
responsibility.

> >> The pcisig presentation
> >> "http://www.pcisig.com/developers/main/training_materials/get_document?doc_id=f4ca40ec1721aa7fadd05e2903b0172c36656294"
> >> talks about PERST# detection for cabled PCIe hotplug.
> >>
> >> Does the kernel detect it?
> >>
> >> None of the pci_driver or pci_error_handlers callbacks were called
> >> when using powered down/up the device (electrically).
> >
> > Do you have the pciehp driver loaded properly when this happens?
> 
> My {dot}config includes it into the kernel, i.e.
> CONFIG_HOTPLUG_PCI_PCIE=y

And is it working properly?  Does your hardware support this (i.e. your
BIOS?)

greg k-h

RE: Cabled PCIE device hotplug/hot-removal

[Kaneshige, Kenji]

> -----Original Message-----
> From: linux-pci-owner@vger.kernel.org
> [mailto:linux-pci-owner@vger.kernel.org] On Behalf Of Shlomo Pongratz
> Sent: Sunday, July 29, 2012 9:36 PM
> To: Greg KH
> Cc: linux-pci@vger.kernel.org; linux-hotplug@vger.kernel.org
> Subject: Re: Cabled PCIE device hotplug/hot-removal
> 
> On Thu, Jul 26, 2012 at 8:09 PM, Greg KH <greg@kroah.com> wrote:
> > On Thu, Jul 26, 2012 at 08:03:09PM +0300, Shlomo Pongratz wrote:
> >> Hi,
> >>
> >> I have a  PCIe device connected with cables to a host (kernel 3.5.0-rc7+).
> >> What are the handler I need to add to a driver in order to catch cable
> >> pull-out / push-in?
> >
> > You shouldn't need anything special, aren't you already getting probe
> > and disconnect callbacks when the device is discovered and removed?
> 

Note that if the controller doesn't support "surprise removal", pciehp does
nothing (just print a message) when plugging/unplugging the device. You need
to power on the slot by "echo 1 > /sys/bus/pci/slots/XXX/power" after plugging
the device. By this, pciehp will power on the slot and driver's .prove
callback will be called. And for removal, you need to
"echo 0 > /sys/bus/pci/slots/XXX/power" before unplugging the device. By doing
this, driver's .remove callback will be called and then pciehp will power
down the slot.

If the controller supports "surprise removal" driver's .probe/.remove callbacks
are called automatically by the pciehp's event handler when plugging/unplugging
event (presence change detect event) happens.


> When the device is loaded the probe is called
> When I "echo 1" to the remove sysfs entry the remove callback is called.
> But when I power down the card no callback is called.

You need to "echo 0 > /sys/bus/pci/slots/XXX/power" to remove the device
if you are using pciehp.


> >
> >> The pcisig presentation
> >>
> "http://www.pcisig.com/developers/main/training_materials/get_document
> ?doc_idôca40ec1721aa7fadd05e2903b0172c36656294"
> >> talks about PERST# detection for cabled PCIe hotplug.
> >>
> >> Does the kernel detect it?
> >>
> >> None of the pci_driver or pci_error_handlers callbacks were called
> >> when using powered down/up the device (electrically).

How did you power down/up the device (electrically)?

> >
> > Do you have the pciehp driver loaded properly when this happens?
> 
> My {dot}config includes it into the kernel, i.e.
> CONFIG_HOTPLUG_PCI_PCIE=y
> 

You can see what kind of hot-plug driver manages the slot by checking
/sys/bus/pci/slots/XXX/module symbolic link. On my system which has some
PCIe hot-plug slots, I can see the following

$ ls /sys/bus/pci/slots/* -l
/sys/bus/pci/slots/21:
total 0
-r--r--r--. 1 root root 4096 Aug  1 02:55 adapter
-r--r--r--. 1 root root 4096 Aug  1 02:55 address
-rw-r--r--. 1 root root 4096 Aug  1 02:55 attention
-r--r--r--. 1 root root 4096 Aug  1 02:55 cur_bus_speed
-r--r--r--. 1 root root 4096 Aug  1 02:55 latch
-r--r--r--. 1 root root 4096 Aug  1 02:55 max_bus_speed
lrwxrwxrwx. 1 root root    0 Aug  1 02:55 module -> ../../../../module/pciehp
-rw-r--r--. 1 root root 4096 Aug  1 02:55 power

/sys/bus/pci/slots/22:
<snip.>

And pciehp_debug option might help you. With this option, you can get the
controller infomation like below.

pciehp 0000:0d:00.0:pcie24: Hotplug Controller:
pciehp 0000:0d:00.0:pcie24:   Seg/Bus/Dev/Func/IRQ : 0000:0d:00.0 IRQ 51
pciehp 0000:0d:00.0:pcie24:   Vendor ID            : 0x111d
pciehp 0000:0d:00.0:pcie24:   Device ID            : 0x807f
pciehp 0000:0d:00.0:pcie24:   Subsystem ID         : 0x159c
pciehp 0000:0d:00.0:pcie24:   Subsystem Vendor ID  : 0x10cf
pciehp 0000:0d:00.0:pcie24:   PCIe Cap offset      : 0x40
pciehp 0000:0d:00.0:pcie24:   PCI resource [14]     : [mem 0x92a00000-0x92bfffff]
pciehp 0000:0d:00.0:pcie24:   PCI resource [15]     : [mem 0x92000000-0x921fffff 64bit pref]
pciehp 0000:0d:00.0:pcie24: Slot Capabilities      : 0x00a8005e
pciehp 0000:0d:00.0:pcie24:   Physical Slot Number : 21
pciehp 0000:0d:00.0:pcie24:   Attention Button     :  no
pciehp 0000:0d:00.0:pcie24:   Power Controller     : yes
pciehp 0000:0d:00.0:pcie24:   MRL Sensor           : yes
pciehp 0000:0d:00.0:pcie24:   Attention Indicator  : yes
pciehp 0000:0d:00.0:pcie24:   Power Indicator      : yes
pciehp 0000:0d:00.0:pcie24:   Hot-Plug Surprise    :  no
pciehp 0000:0d:00.0:pcie24:   EMI Present          :  no
pciehp 0000:0d:00.0:pcie24:   Command Completed    : yes
pciehp 0000:0d:00.0:pcie24: Slot Status            : 0x0000
pciehp 0000:0d:00.0:pcie24: Slot Control           : 0x07c0
pciehp 0000:0d:00.0:pcie24: Link Active Reporting supported

Regards,
Kenji Kaneshige

RE: Cabled PCIE device hotplug/hot-removal

[Kaneshige, Kenji]

> -----Original Message-----
> From: linux-pci-owner@vger.kernel.org
> [mailto:linux-pci-owner@vger.kernel.org] On Behalf Of Shlomo Pongratz
> Sent: Sunday, July 29, 2012 9:36 PM
> To: Greg KH
> Cc: linux-pci@vger.kernel.org; linux-hotplug@vger.kernel.org
> Subject: Re: Cabled PCIE device hotplug/hot-removal
> 
> On Thu, Jul 26, 2012 at 8:09 PM, Greg KH <greg@kroah.com> wrote:
> > On Thu, Jul 26, 2012 at 08:03:09PM +0300, Shlomo Pongratz wrote:
> >> Hi,
> >>
> >> I have a  PCIe device connected with cables to a host (kernel 3.5.0-rc7+).
> >> What are the handler I need to add to a driver in order to catch cable
> >> pull-out / push-in?
> >
> > You shouldn't need anything special, aren't you already getting probe
> > and disconnect callbacks when the device is discovered and removed?
> 

Note that if the controller doesn't support "surprise removal", pciehp does
nothing (just print a message) when plugging/unplugging the device. You need
to power on the slot by "echo 1 > /sys/bus/pci/slots/XXX/power" after plugging
the device. By this, pciehp will power on the slot and driver's .prove
callback will be called. And for removal, you need to
"echo 0 > /sys/bus/pci/slots/XXX/power" before unplugging the device. By doing
this, driver's .remove callback will be called and then pciehp will power
down the slot.

If the controller supports "surprise removal" driver's .probe/.remove callbacks
are called automatically by the pciehp's event handler when plugging/unplugging
event (presence change detect event) happens.


> When the device is loaded the probe is called
> When I "echo 1" to the remove sysfs entry the remove callback is called.
> But when I power down the card no callback is called.

You need to "echo 0 > /sys/bus/pci/slots/XXX/power" to remove the device
if you are using pciehp.


> >
> >> The pcisig presentation
> >>
> "http://www.pcisig.com/developers/main/training_materials/get_document
> ?doc_id=f4ca40ec1721aa7fadd05e2903b0172c36656294"
> >> talks about PERST# detection for cabled PCIe hotplug.
> >>
> >> Does the kernel detect it?
> >>
> >> None of the pci_driver or pci_error_handlers callbacks were called
> >> when using powered down/up the device (electrically).

How did you power down/up the device (electrically)?

> >
> > Do you have the pciehp driver loaded properly when this happens?
> 
> My {dot}config includes it into the kernel, i.e.
> CONFIG_HOTPLUG_PCI_PCIE=y
> 

You can see what kind of hot-plug driver manages the slot by checking
/sys/bus/pci/slots/XXX/module symbolic link. On my system which has some
PCIe hot-plug slots, I can see the following

$ ls /sys/bus/pci/slots/* -l
/sys/bus/pci/slots/21:
total 0
-r--r--r--. 1 root root 4096 Aug  1 02:55 adapter
-r--r--r--. 1 root root 4096 Aug  1 02:55 address
-rw-r--r--. 1 root root 4096 Aug  1 02:55 attention
-r--r--r--. 1 root root 4096 Aug  1 02:55 cur_bus_speed
-r--r--r--. 1 root root 4096 Aug  1 02:55 latch
-r--r--r--. 1 root root 4096 Aug  1 02:55 max_bus_speed
lrwxrwxrwx. 1 root root    0 Aug  1 02:55 module -> ../../../../module/pciehp
-rw-r--r--. 1 root root 4096 Aug  1 02:55 power

/sys/bus/pci/slots/22:
<snip.>

And pciehp_debug option might help you. With this option, you can get the
controller infomation like below.

pciehp 0000:0d:00.0:pcie24: Hotplug Controller:
pciehp 0000:0d:00.0:pcie24:   Seg/Bus/Dev/Func/IRQ : 0000:0d:00.0 IRQ 51
pciehp 0000:0d:00.0:pcie24:   Vendor ID            : 0x111d
pciehp 0000:0d:00.0:pcie24:   Device ID            : 0x807f
pciehp 0000:0d:00.0:pcie24:   Subsystem ID         : 0x159c
pciehp 0000:0d:00.0:pcie24:   Subsystem Vendor ID  : 0x10cf
pciehp 0000:0d:00.0:pcie24:   PCIe Cap offset      : 0x40
pciehp 0000:0d:00.0:pcie24:   PCI resource [14]     : [mem 0x92a00000-0x92bfffff]
pciehp 0000:0d:00.0:pcie24:   PCI resource [15]     : [mem 0x92000000-0x921fffff 64bit pref]
pciehp 0000:0d:00.0:pcie24: Slot Capabilities      : 0x00a8005e
pciehp 0000:0d:00.0:pcie24:   Physical Slot Number : 21
pciehp 0000:0d:00.0:pcie24:   Attention Button     :  no
pciehp 0000:0d:00.0:pcie24:   Power Controller     : yes
pciehp 0000:0d:00.0:pcie24:   MRL Sensor           : yes
pciehp 0000:0d:00.0:pcie24:   Attention Indicator  : yes
pciehp 0000:0d:00.0:pcie24:   Power Indicator      : yes
pciehp 0000:0d:00.0:pcie24:   Hot-Plug Surprise    :  no
pciehp 0000:0d:00.0:pcie24:   EMI Present          :  no
pciehp 0000:0d:00.0:pcie24:   Command Completed    : yes
pciehp 0000:0d:00.0:pcie24: Slot Status            : 0x0000
pciehp 0000:0d:00.0:pcie24: Slot Control           : 0x07c0
pciehp 0000:0d:00.0:pcie24: Link Active Reporting supported

Regards,
Kenji Kaneshige

Re: Cabled PCIE device hotplug/hot-removal

[Shlomo Pongratz]

On Tue, Jul 31, 2012 at 2:02 PM, Kaneshige, Kenji
<kaneshige.kenji@jp.fujitsu.com> wrote:
>> -----Original Message-----
>> From: linux-pci-owner@vger.kernel.org
>> [mailto:linux-pci-owner@vger.kernel.org] On Behalf Of Shlomo Pongratz
>> Sent: Sunday, July 29, 2012 9:36 PM
>> To: Greg KH
>> Cc: linux-pci@vger.kernel.org; linux-hotplug@vger.kernel.org
>> Subject: Re: Cabled PCIE device hotplug/hot-removal
>>
>> On Thu, Jul 26, 2012 at 8:09 PM, Greg KH <greg@kroah.com> wrote:
>> > On Thu, Jul 26, 2012 at 08:03:09PM +0300, Shlomo Pongratz wrote:
>> >> Hi,
>> >>
>> >> I have a  PCIe device connected with cables to a host (kernel 3.5.0-rc7+).
>> >> What are the handler I need to add to a driver in order to catch cable
>> >> pull-out / push-in?
>> >
>> > You shouldn't need anything special, aren't you already getting probe
>> > and disconnect callbacks when the device is discovered and removed?
>>
>
> Note that if the controller doesn't support "surprise removal", pciehp does
> nothing (just print a message) when plugging/unplugging the device. You need
> to power on the slot by "echo 1 > /sys/bus/pci/slots/XXX/power" after plugging
> the device. By this, pciehp will power on the slot and driver's .prove
> callback will be called. And for removal, you need to
> "echo 0 > /sys/bus/pci/slots/XXX/power" before unplugging the device. By doing
> this, driver's .remove callback will be called and then pciehp will power
> down the slot.
>
> If the controller supports "surprise removal" driver's .probe/.remove callbacks
> are called automatically by the pciehp's event handler when plugging/unplugging
> event (presence change detect event) happens.
>
>
>> When the device is loaded the probe is called
>> When I "echo 1" to the remove sysfs entry the remove callback is called.
>> But when I power down the card no callback is called.
>
> You need to "echo 0 > /sys/bus/pci/slots/XXX/power" to remove the device
> if you are using pciehp.
>
>
>> >
>> >> The pcisig presentation
>> >>
>> "http://www.pcisig.com/developers/main/training_materials/get_document
>> ?doc_idôca40ec1721aa7fadd05e2903b0172c36656294"
>> >> talks about PERST# detection for cabled PCIe hotplug.
>> >>
>> >> Does the kernel detect it?
>> >>
>> >> None of the pci_driver or pci_error_handlers callbacks were called
>> >> when using powered down/up the device (electrically).
>
> How did you power down/up the device (electrically)?
>
>> >
>> > Do you have the pciehp driver loaded properly when this happens?
>>
>> My {dot}config includes it into the kernel, i.e.
>> CONFIG_HOTPLUG_PCI_PCIE=y
>>
>
> You can see what kind of hot-plug driver manages the slot by checking
> /sys/bus/pci/slots/XXX/module symbolic link. On my system which has some
> PCIe hot-plug slots, I can see the following
>
> $ ls /sys/bus/pci/slots/* -l
> /sys/bus/pci/slots/21:
> total 0
> -r--r--r--. 1 root root 4096 Aug  1 02:55 adapter
> -r--r--r--. 1 root root 4096 Aug  1 02:55 address
> -rw-r--r--. 1 root root 4096 Aug  1 02:55 attention
> -r--r--r--. 1 root root 4096 Aug  1 02:55 cur_bus_speed
> -r--r--r--. 1 root root 4096 Aug  1 02:55 latch
> -r--r--r--. 1 root root 4096 Aug  1 02:55 max_bus_speed
> lrwxrwxrwx. 1 root root    0 Aug  1 02:55 module -> ../../../../module/pciehp
> -rw-r--r--. 1 root root 4096 Aug  1 02:55 power
>
> /sys/bus/pci/slots/22:
> <snip.>
>
> And pciehp_debug option might help you. With this option, you can get the
> controller infomation like below.
>
> pciehp 0000:0d:00.0:pcie24: Hotplug Controller:
> pciehp 0000:0d:00.0:pcie24:   Seg/Bus/Dev/Func/IRQ : 0000:0d:00.0 IRQ 51
> pciehp 0000:0d:00.0:pcie24:   Vendor ID            : 0x111d
> pciehp 0000:0d:00.0:pcie24:   Device ID            : 0x807f
> pciehp 0000:0d:00.0:pcie24:   Subsystem ID         : 0x159c
> pciehp 0000:0d:00.0:pcie24:   Subsystem Vendor ID  : 0x10cf
> pciehp 0000:0d:00.0:pcie24:   PCIe Cap offset      : 0x40
> pciehp 0000:0d:00.0:pcie24:   PCI resource [14]     : [mem 0x92a00000-0x92bfffff]
> pciehp 0000:0d:00.0:pcie24:   PCI resource [15]     : [mem 0x92000000-0x921fffff 64bit pref]
> pciehp 0000:0d:00.0:pcie24: Slot Capabilities      : 0x00a8005e
> pciehp 0000:0d:00.0:pcie24:   Physical Slot Number : 21
> pciehp 0000:0d:00.0:pcie24:   Attention Button     :  no
> pciehp 0000:0d:00.0:pcie24:   Power Controller     : yes
> pciehp 0000:0d:00.0:pcie24:   MRL Sensor           : yes
> pciehp 0000:0d:00.0:pcie24:   Attention Indicator  : yes
> pciehp 0000:0d:00.0:pcie24:   Power Indicator      : yes
> pciehp 0000:0d:00.0:pcie24:   Hot-Plug Surprise    :  no
> pciehp 0000:0d:00.0:pcie24:   EMI Present          :  no
> pciehp 0000:0d:00.0:pcie24:   Command Completed    : yes
> pciehp 0000:0d:00.0:pcie24: Slot Status            : 0x0000
> pciehp 0000:0d:00.0:pcie24: Slot Control           : 0x07c0
> pciehp 0000:0d:00.0:pcie24: Link Active Reporting supported
>
> Regards,
> Kenji Kaneshige
>

O.K. so if using lspci -vvv is see that with the PCI-bridge the slot
capability is marked with minus in the support surprise mode I need to
get another system.

That is

Capabilities: [6c] Express (v2) Root Port (Slot+), MSI 00
.....
       SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise-
                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-

Re: Cabled PCIE device hotplug/hot-removal

[Shlomo Pongratz]

On Tue, Jul 31, 2012 at 2:02 PM, Kaneshige, Kenji
<kaneshige.kenji@jp.fujitsu.com> wrote:
>> -----Original Message-----
>> From: linux-pci-owner@vger.kernel.org
>> [mailto:linux-pci-owner@vger.kernel.org] On Behalf Of Shlomo Pongratz
>> Sent: Sunday, July 29, 2012 9:36 PM
>> To: Greg KH
>> Cc: linux-pci@vger.kernel.org; linux-hotplug@vger.kernel.org
>> Subject: Re: Cabled PCIE device hotplug/hot-removal
>>
>> On Thu, Jul 26, 2012 at 8:09 PM, Greg KH <greg@kroah.com> wrote:
>> > On Thu, Jul 26, 2012 at 08:03:09PM +0300, Shlomo Pongratz wrote:
>> >> Hi,
>> >>
>> >> I have a  PCIe device connected with cables to a host (kernel 3.5.0-rc7+).
>> >> What are the handler I need to add to a driver in order to catch cable
>> >> pull-out / push-in?
>> >
>> > You shouldn't need anything special, aren't you already getting probe
>> > and disconnect callbacks when the device is discovered and removed?
>>
>
> Note that if the controller doesn't support "surprise removal", pciehp does
> nothing (just print a message) when plugging/unplugging the device. You need
> to power on the slot by "echo 1 > /sys/bus/pci/slots/XXX/power" after plugging
> the device. By this, pciehp will power on the slot and driver's .prove
> callback will be called. And for removal, you need to
> "echo 0 > /sys/bus/pci/slots/XXX/power" before unplugging the device. By doing
> this, driver's .remove callback will be called and then pciehp will power
> down the slot.
>
> If the controller supports "surprise removal" driver's .probe/.remove callbacks
> are called automatically by the pciehp's event handler when plugging/unplugging
> event (presence change detect event) happens.
>
>
>> When the device is loaded the probe is called
>> When I "echo 1" to the remove sysfs entry the remove callback is called.
>> But when I power down the card no callback is called.
>
> You need to "echo 0 > /sys/bus/pci/slots/XXX/power" to remove the device
> if you are using pciehp.
>
>
>> >
>> >> The pcisig presentation
>> >>
>> "http://www.pcisig.com/developers/main/training_materials/get_document
>> ?doc_id=f4ca40ec1721aa7fadd05e2903b0172c36656294"
>> >> talks about PERST# detection for cabled PCIe hotplug.
>> >>
>> >> Does the kernel detect it?
>> >>
>> >> None of the pci_driver or pci_error_handlers callbacks were called
>> >> when using powered down/up the device (electrically).
>
> How did you power down/up the device (electrically)?
>
>> >
>> > Do you have the pciehp driver loaded properly when this happens?
>>
>> My {dot}config includes it into the kernel, i.e.
>> CONFIG_HOTPLUG_PCI_PCIE=y
>>
>
> You can see what kind of hot-plug driver manages the slot by checking
> /sys/bus/pci/slots/XXX/module symbolic link. On my system which has some
> PCIe hot-plug slots, I can see the following
>
> $ ls /sys/bus/pci/slots/* -l
> /sys/bus/pci/slots/21:
> total 0
> -r--r--r--. 1 root root 4096 Aug  1 02:55 adapter
> -r--r--r--. 1 root root 4096 Aug  1 02:55 address
> -rw-r--r--. 1 root root 4096 Aug  1 02:55 attention
> -r--r--r--. 1 root root 4096 Aug  1 02:55 cur_bus_speed
> -r--r--r--. 1 root root 4096 Aug  1 02:55 latch
> -r--r--r--. 1 root root 4096 Aug  1 02:55 max_bus_speed
> lrwxrwxrwx. 1 root root    0 Aug  1 02:55 module -> ../../../../module/pciehp
> -rw-r--r--. 1 root root 4096 Aug  1 02:55 power
>
> /sys/bus/pci/slots/22:
> <snip.>
>
> And pciehp_debug option might help you. With this option, you can get the
> controller infomation like below.
>
> pciehp 0000:0d:00.0:pcie24: Hotplug Controller:
> pciehp 0000:0d:00.0:pcie24:   Seg/Bus/Dev/Func/IRQ : 0000:0d:00.0 IRQ 51
> pciehp 0000:0d:00.0:pcie24:   Vendor ID            : 0x111d
> pciehp 0000:0d:00.0:pcie24:   Device ID            : 0x807f
> pciehp 0000:0d:00.0:pcie24:   Subsystem ID         : 0x159c
> pciehp 0000:0d:00.0:pcie24:   Subsystem Vendor ID  : 0x10cf
> pciehp 0000:0d:00.0:pcie24:   PCIe Cap offset      : 0x40
> pciehp 0000:0d:00.0:pcie24:   PCI resource [14]     : [mem 0x92a00000-0x92bfffff]
> pciehp 0000:0d:00.0:pcie24:   PCI resource [15]     : [mem 0x92000000-0x921fffff 64bit pref]
> pciehp 0000:0d:00.0:pcie24: Slot Capabilities      : 0x00a8005e
> pciehp 0000:0d:00.0:pcie24:   Physical Slot Number : 21
> pciehp 0000:0d:00.0:pcie24:   Attention Button     :  no
> pciehp 0000:0d:00.0:pcie24:   Power Controller     : yes
> pciehp 0000:0d:00.0:pcie24:   MRL Sensor           : yes
> pciehp 0000:0d:00.0:pcie24:   Attention Indicator  : yes
> pciehp 0000:0d:00.0:pcie24:   Power Indicator      : yes
> pciehp 0000:0d:00.0:pcie24:   Hot-Plug Surprise    :  no
> pciehp 0000:0d:00.0:pcie24:   EMI Present          :  no
> pciehp 0000:0d:00.0:pcie24:   Command Completed    : yes
> pciehp 0000:0d:00.0:pcie24: Slot Status            : 0x0000
> pciehp 0000:0d:00.0:pcie24: Slot Control           : 0x07c0
> pciehp 0000:0d:00.0:pcie24: Link Active Reporting supported
>
> Regards,
> Kenji Kaneshige
>

O.K. so if using lspci -vvv is see that with the PCI-bridge the slot
capability is marked with minus in the support surprise mode I need to
get another system.

That is

Capabilities: [6c] Express (v2) Root Port (Slot+), MSI 00
.....
       SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise-
                        Slot #  0, PowerLimit 0.000000; Interlock- NoCompl-