PV driver domains and S3 sleep

PV driver domains and S3 sleep

[Rafal Wojtczuk]

Hello,

The topic is self-explanatory: how to ensure that a PV driver domain correctly 
prepares its PCI devices for S3 sleep?
If I do "pm-suspend" in dom0, and the driver domain has active network interfaces, 
suspend hangs the system. Yes, in case of this particular machine, suspend works
fine when there is no driver domain. 
It is possible to manually invoke scripts from /usr/lib64/pm-utils/sleep.d/ in driver 
domain. In the test case, "ifconfig down wlan0" in the driver domain allows
the suspend to go smoothly. But generally, is it enough ? The kernel device driver should 
prepare the PCI device properly for S3, shouldn't it ?  
Would it be more proper to [somehow] notify a driver domain _kernel_ that we are 
going to S3 (just like dom0 kernel is notified), and let it execute all necessary actions 
(including, but not only, launching of usermode pm-utils scripts), just like dom0 kernel 
does ? Would it work at all, considering that driver domain kernel has no access to 
ACPI tables ? 
Currently, how are these issues taken care of in the mainstream Xen? 

Thanks in advance,
Rafal Wojtczuk

Re: PV driver domains and S3 sleep

[Keir Fraser]

On 16/09/2010 12:44, "Rafal Wojtczuk" <rafal@invisiblethingslab.com> wrote:

> The topic is self-explanatory: how to ensure that a PV driver domain correctly
> prepares its PCI devices for S3 sleep?
> If I do "pm-suspend" in dom0, and the driver domain has active network
> interfaces, 
> suspend hangs the system. Yes, in case of this particular machine, suspend
> works
> fine when there is no driver domain.
> It is possible to manually invoke scripts from /usr/lib64/pm-utils/sleep.d/ in
> driver 
> domain. In the test case, "ifconfig down wlan0" in the driver domain allows
> the suspend to go smoothly. But generally, is it enough ? The kernel device
> driver should 
> prepare the PCI device properly for S3, shouldn't it ?
> Would it be more proper to [somehow] notify a driver domain _kernel_ that we
> are 
> going to S3 (just like dom0 kernel is notified), and let it execute all
> necessary actions
> (including, but not only, launching of usermode pm-utils scripts), just like
> dom0 kernel 
> does ? Would it work at all, considering that driver domain kernel has no
> access to 
> ACPI tables ? 
> Currently, how are these issues taken care of in the mainstream Xen?

I don't think it currently is handled. HVM driver domains (using VT-d or
equivalent) can be put into virtual S3. We would need an equivalent concept
for PV driver domains. Or for devices to be hot-unplugged from the driver
domain, and re-plugged on resume?

 -- Keir

Re: PV driver domains and S3 sleep

[Joanna Rutkowska]

[-- Attachment #1.1: Type: text/plain, Size: 1736 bytes --]

On 09/16/10 13:52, Keir Fraser wrote:
> On 16/09/2010 12:44, "Rafal Wojtczuk" <rafal@invisiblethingslab.com> wrote:
> 
>> The topic is self-explanatory: how to ensure that a PV driver domain correctly
>> prepares its PCI devices for S3 sleep?
>> If I do "pm-suspend" in dom0, and the driver domain has active network
>> interfaces, 
>> suspend hangs the system. Yes, in case of this particular machine, suspend
>> works
>> fine when there is no driver domain.
>> It is possible to manually invoke scripts from /usr/lib64/pm-utils/sleep.d/ in
>> driver 
>> domain. In the test case, "ifconfig down wlan0" in the driver domain allows
>> the suspend to go smoothly. But generally, is it enough ? The kernel device
>> driver should 
>> prepare the PCI device properly for S3, shouldn't it ?
>> Would it be more proper to [somehow] notify a driver domain _kernel_ that we
>> are 
>> going to S3 (just like dom0 kernel is notified), and let it execute all
>> necessary actions
>> (including, but not only, launching of usermode pm-utils scripts), just like
>> dom0 kernel 
>> does ? Would it work at all, considering that driver domain kernel has no
>> access to 
>> ACPI tables ? 
>> Currently, how are these issues taken care of in the mainstream Xen?
> 
> I don't think it currently is handled. HVM driver domains (using VT-d or
> equivalent) can be put into virtual S3. We would need an equivalent concept
> for PV driver domains. Or for devices to be hot-unplugged from the driver
> domain, and re-plugged on resume?
> 

But, can you explain how Xen notifies Dom0 when the system enters S3,
and if the same mechanism could be (easily) used to do the same for a
driver PV domain?

Thanks,
joanna.


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 518 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PV driver domains and S3 sleep

[Jeremy Fitzhardinge]

 On 09/16/2010 12:04 PM, Joanna Rutkowska wrote:
> On 09/16/10 13:52, Keir Fraser wrote:
>> On 16/09/2010 12:44, "Rafal Wojtczuk" <rafal@invisiblethingslab.com> wrote:
>>
>>> The topic is self-explanatory: how to ensure that a PV driver domain correctly
>>> prepares its PCI devices for S3 sleep?
>>> If I do "pm-suspend" in dom0, and the driver domain has active network
>>> interfaces, 
>>> suspend hangs the system. Yes, in case of this particular machine, suspend
>>> works
>>> fine when there is no driver domain.
>>> It is possible to manually invoke scripts from /usr/lib64/pm-utils/sleep.d/ in
>>> driver 
>>> domain. In the test case, "ifconfig down wlan0" in the driver domain allows
>>> the suspend to go smoothly. But generally, is it enough ? The kernel device
>>> driver should 
>>> prepare the PCI device properly for S3, shouldn't it ?
>>> Would it be more proper to [somehow] notify a driver domain _kernel_ that we
>>> are 
>>> going to S3 (just like dom0 kernel is notified), and let it execute all
>>> necessary actions
>>> (including, but not only, launching of usermode pm-utils scripts), just like
>>> dom0 kernel 
>>> does ? Would it work at all, considering that driver domain kernel has no
>>> access to 
>>> ACPI tables ? 
>>> Currently, how are these issues taken care of in the mainstream Xen?
>> I don't think it currently is handled. HVM driver domains (using VT-d or
>> equivalent) can be put into virtual S3. We would need an equivalent concept
>> for PV driver domains. Or for devices to be hot-unplugged from the driver
>> domain, and re-plugged on resume?
>>
> But, can you explain how Xen notifies Dom0 when the system enters S3,
> and if the same mechanism could be (easily) used to do the same for a
> driver PV domain?

The dom0 kernel initiates S3 itself (possibly in response to a
lid-switch or the like), so it knows it is going into S3.  As part of
that it can do something to notify all the other domains that they in
turn need to do something.

I think the simplest thing to do is just do a regular PV save/restore on
the domains, but without needing to save their pages to disk.  That way
their regular device model suspend/resume will do the right thing to the
hardware devices.  The only change that may be needed is to make sure
that the normal PCI suspend/resume bus calls are done on the Xen
save/restore path.

Or perhaps the domains will need to know whether its a normal
save/restore, vs S3, vs S5.  In that case we could add a xenstore key
which they could watch to know they need to do something.  But it would
need a bit of thought to handshake the whole process.

    J

Re: PV driver domains and S3 sleep

[Rafal Wojtczuk]

On Thu, Sep 16, 2010 at 05:22:50PM -0700, Jeremy Fitzhardinge wrote:
> >>> The topic is self-explanatory: how to ensure that a PV driver domain correctly
> >>> prepares its PCI devices for S3 sleep?
[cut]
> I think the simplest thing to do is just do a regular PV save/restore on
> the domains, but without needing to save their pages to disk.  That way

I suspect suspend/resume of the driver domain will kill established net backend/frontend 
connections ? So we also would have to network-detach all VMs interfaces, and
reattach. It does not look pretty. 

Regards,
Rafal Wojtczuk

Re: PV driver domains and S3 sleep

[Jeremy Fitzhardinge]

 On 09/24/2010 07:30 AM, Rafal Wojtczuk wrote:
> On Thu, Sep 16, 2010 at 05:22:50PM -0700, Jeremy Fitzhardinge wrote:
>>>>> The topic is self-explanatory: how to ensure that a PV driver domain correctly
>>>>> prepares its PCI devices for S3 sleep?
> [cut]
>> I think the simplest thing to do is just do a regular PV save/restore on
>> the domains, but without needing to save their pages to disk.  That way
> I suspect suspend/resume of the driver domain will kill established net backend/frontend 
> connections ? So we also would have to network-detach all VMs interfaces, and
> reattach. It does not look pretty. 
>

Not generally.  The blkfront and netfront drivers don't really do
anything on a save; they certainly don't change the xenbus connection
state.  The normal mode of operation for save/restore or migration is
that after resuming the frontends suddenly find their backends are no
longer connected and will quietly attempt to reconnect before doing on,
resulting in just a little IO hiccup.

In this case, the backends will still be there and will remain in a
connected state, so the frontends won't even notice after resuming.

However, the pcifront driver would implement the suspend method and make
sure the pci bus does its normal suspend operation.

My main concern is that I'm not sure how the handshake with dom0 would
work, so that it knows the suspend is finished - oh I guess the normal
way, waiting for the domain to suspend itself (or timeout mysteriously).

But it might just be better to add a pciback xenstore key to tell
pcifront to do whatever's required for an ACPI suspend (S3, S5 or
whatever), and have a corresponding pcifront xenstore key to handshake
that the suspend has completed (or failed).  At least that way you could
get some useful diagnostics on failure.

    J

PCI hotplug problem [was: PV driver domains and S3 sleep]

[Rafal Wojtczuk]

On Thu, Sep 16, 2010 at 12:52:02PM +0100, Keir Fraser wrote:
> > The topic is self-explanatory: how to ensure that a PV driver domain correctly
> > prepares its PCI devices for S3 sleep?
[cut]
> > Currently, how are these issues taken care of in the mainstream Xen?

> I don't think it currently is handled. HVM driver domains (using VT-d or
> equivalent) can be put into virtual S3. We would need an equivalent concept
> for PV driver domains. Or for devices to be hot-unplugged from the driver
> domain, and re-plugged on resume?

The idea of using PCI hotplug is nice, however, PCI hotplug does not seem to
work with the used setup (xen-3.4.3, all 64bit). Hot-unplug works, however the 
following hotplug makes the driver domain kernel spit out the following:

Sep 24 09:46:01 localhost kernel: [  113.045927] pcifront pci-0: Rescanning
PCI Frontend Bus 0000:00
Sep 24 09:46:15 localhost kernel: [  126.843990] pcifront pci-0: Rescanning
PCI Frontend Bus 0000:00
Sep 24 09:46:15 localhost kernel: [  126.846217] pcifront pci-0: New device
on 0000:00:01.00 found.
Sep 24 09:46:15 localhost kernel: [  126.846523] iwlagn 0000:00:01.0: device
not available (can't reserve [mem 0xf8000000-0xf8001fff 64bit])

^C
[root@localhost ~]# cat /proc/iomem 
f6000000-f600ffff : 0000:00:00.0
  f6000000-f600ffff : tg3
[root@localhost ~]# lspci
00:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5787M Gigabit
Ethernet PCI Express (rev 02)
00:01.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN
[Kedron] Network Connection (rev 61)

Nothing suspicious in xend, Xen and dom0 logs.

The domU and dom0 kernels are the same, 2.6.34.1-10.xenlinux (SUSE patches
for 2.6.34.1).

With old pvops (2.6.31.9-1.pvops0) in domU, the message on the hot-plug is similar:
Sep 24 09:50:40 localhost kernel: pcifront pci-0: Rescanning PCI Frontend
Bus 0000:00
Sep 24 09:50:51 localhost kernel: pcifront pci-0: Rescanning PCI Frontend
Bus 0000:00
Sep 24 09:50:51 localhost kernel: pcifront pci-0: New device on
0000:00:01.00 found.
Sep 24 09:50:51 localhost kernel: iwlagn 0000:00:01.0: device not available
because of BAR 0 [0xf8000000-0xf8001fff] collisions

Others seem to experience similar problems (e.g.
http://permalink.gmane.org/gmane.comp.emulators.xen.devel/80766). Does
anyone know the solution ?

Regards,
Rafal Wojtczuk

Re: PCI hotplug problem [was: PV driver domains and S3 sleep]

[Konrad Rzeszutek Wilk]

On Fri, Sep 24, 2010 at 04:24:58PM +0200, Rafal Wojtczuk wrote:
> On Thu, Sep 16, 2010 at 12:52:02PM +0100, Keir Fraser wrote:
> > > The topic is self-explanatory: how to ensure that a PV driver domain correctly
> > > prepares its PCI devices for S3 sleep?
> [cut]
> > > Currently, how are these issues taken care of in the mainstream Xen?
> 
> > I don't think it currently is handled. HVM driver domains (using VT-d or
> > equivalent) can be put into virtual S3. We would need an equivalent concept
> > for PV driver domains. Or for devices to be hot-unplugged from the driver
> > domain, and re-plugged on resume?
> 
> The idea of using PCI hotplug is nice, however, PCI hotplug does not seem to
> work with the used setup (xen-3.4.3, all 64bit). Hot-unplug works, however the 
> following hotplug makes the driver domain kernel spit out the following:
> 
> Sep 24 09:46:01 localhost kernel: [  113.045927] pcifront pci-0: Rescanning
> PCI Frontend Bus 0000:00
> Sep 24 09:46:15 localhost kernel: [  126.843990] pcifront pci-0: Rescanning
> PCI Frontend Bus 0000:00
> Sep 24 09:46:15 localhost kernel: [  126.846217] pcifront pci-0: New device
> on 0000:00:01.00 found.
> Sep 24 09:46:15 localhost kernel: [  126.846523] iwlagn 0000:00:01.0: device
> not available (can't reserve [mem 0xf8000000-0xf8001fff 64bit])
> 
> ^C
> [root@localhost ~]# cat /proc/iomem 
> f6000000-f600ffff : 0000:00:00.0
>   f6000000-f600ffff : tg3
> [root@localhost ~]# lspci
> 00:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5787M Gigabit
> Ethernet PCI Express (rev 02)
> 00:01.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN
> [Kedron] Network Connection (rev 61)
> 
> Nothing suspicious in xend, Xen and dom0 logs.
> 
> The domU and dom0 kernels are the same, 2.6.34.1-10.xenlinux (SUSE patches
> for 2.6.34.1).
> 
> With old pvops (2.6.31.9-1.pvops0) in domU, the message on the hot-plug is similar:
> Sep 24 09:50:40 localhost kernel: pcifront pci-0: Rescanning PCI Frontend
> Bus 0000:00
> Sep 24 09:50:51 localhost kernel: pcifront pci-0: Rescanning PCI Frontend
> Bus 0000:00
> Sep 24 09:50:51 localhost kernel: pcifront pci-0: New device on
> 0000:00:01.00 found.
> Sep 24 09:50:51 localhost kernel: iwlagn 0000:00:01.0: device not available
> because of BAR 0 [0xf8000000-0xf8001fff] collisions
> 
> Others seem to experience similar problems (e.g.
> http://permalink.gmane.org/gmane.comp.emulators.xen.devel/80766). Does
> anyone know the solution ?

I had an off-mailing list conversation with that fellow and I spun out
a bunch of patches to fix his issue.

You need these patches:
Konrad Rzeszutek Wilk (3):
      xen-pcifront: Enforce scanning of device functions on initial execution.
      xen-pcifront: Claim PCI resources before going live.
      xen-pcifront: Don't race with udev when discovering new devices.

I think they are in Jeremy's upstream tree.. ah, right you guys aren't using
Jeremy's tree.

Get them from: git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git

pv/pcifront-2.6.34

you also might want to update your pciback driver too (pv/pciback-2.6.32)
> 
> Regards,
> Rafal Wojtczuk
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: PCI hotplug problem

[Rafal Wojtczuk]

On Mon, Sep 27, 2010 at 01:07:05PM -0400, Konrad Rzeszutek Wilk wrote:

> > The idea of using PCI hotplug is nice, however, PCI hotplug does not seem to
> > work with the used setup (xen-3.4.3, all 64bit). Hot-unplug works, however the 
> > following hotplug makes the driver domain kernel spit out the following:
[cut]
> > Sep 24 09:46:15 localhost kernel: [  126.846523] iwlagn 0000:00:01.0: device
> > not available (can't reserve [mem 0xf8000000-0xf8001fff 64bit])
[cut]
> > Others seem to experience similar problems (e.g.
> > http://permalink.gmane.org/gmane.comp.emulators.xen.devel/80766). Does
> > anyone know the solution ?
> 
> I had an off-mailing list conversation with that fellow and I spun out
> a bunch of patches to fix his issue.
> 
> You need these patches:
> Konrad Rzeszutek Wilk (3):
>       xen-pcifront: Enforce scanning of device functions on initial execution.
>       xen-pcifront: Claim PCI resources before going live.
>       xen-pcifront: Don't race with udev when discovering new devices.
> 
> I think they are in Jeremy's upstream tree.. ah, right you guys aren't using
> Jeremy's tree.
> 
> Get them from: git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> 
> pv/pcifront-2.6.34

Indeed these patches help, thank you.
There is one more problem with the linux-2.6.18-xen.hg pcifront (that affect
derived code, e.g. OpenSUSE kernel, too). unbind_from_irqhandler() is
mistakenly passed evtchn, instead of irq. Compare line 68 of
http://xenbits.xensource.com/linux-2.6.18-xen.hg?file/a66a7c64b1d0/drivers/xen/pcifront/xenbus.c
with pvops equivalent
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/pci/xen-pcifront.c;h=10868aeae818d69980b8519f8a77b38d6ab58a4c;hb=HEAD#l758

The following patch helps.
Regards,
Rafal Wojtczuk


unbind_from_irqhandler takes irq, not evtchn, as its first argument.

Signed-off-by: Rafal Wojtczuk <rafal@invisiblethingslab.com>
--- linux-2.6.34.1/drivers/xen/pcifront/xenbus.c.orig   2010-09-29 16:47:39.961674359 +0200
+++ linux-2.6.34.1/drivers/xen/pcifront/xenbus.c        2010-09-29 16:47:49.458675391 +0200
@@ -61,7 +61,7 @@ static void free_pdev(struct pcifront_de

        /*For PCIE_AER error handling job*/
        flush_scheduled_work();
-       unbind_from_irqhandler(pdev->evtchn, pdev);
+       unbind_from_irqhandler(irq_from_evtchn(pdev->evtchn), pdev);

        if (pdev->evtchn != INVALID_EVTCHN)
                xenbus_free_evtchn(pdev->xdev, pdev->evtchn);

Re: PCI hotplug problem

[Jan Beulich]

>>> On 01.10.10 at 16:24, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
> There is one more problem with the linux-2.6.18-xen.hg pcifront (that affect
> derived code, e.g. OpenSUSE kernel, too). unbind_from_irqhandler() is
> mistakenly passed evtchn, instead of irq. Compare line 68 of
> http://xenbits.xensource.com/linux-2.6.18-xen.hg?file/a66a7c64b1d0/drivers/xen 
> /pcifront/xenbus.c
> with pvops equivalent
> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=drivers/pc 
> i/xen-pcifront.c;h=10868aeae818d69980b8519f8a77b38d6ab58a4c;hb=HEAD#l758
> 
> The following patch helps.

Except there is no irq_from_evtchn() in the original tree. I'll post a
better, more complete patch later.

Jan

> Regards,
> Rafal Wojtczuk
> 
> 
> unbind_from_irqhandler takes irq, not evtchn, as its first argument.
> 
> Signed-off-by: Rafal Wojtczuk <rafal@invisiblethingslab.com>
> --- linux-2.6.34.1/drivers/xen/pcifront/xenbus.c.orig   2010-09-29 
> 16:47:39.961674359 +0200
> +++ linux-2.6.34.1/drivers/xen/pcifront/xenbus.c        2010-09-29 
> 16:47:49.458675391 +0200
> @@ -61,7 +61,7 @@ static void free_pdev(struct pcifront_de
> 
>         /*For PCIE_AER error handling job*/
>         flush_scheduled_work();
> -       unbind_from_irqhandler(pdev->evtchn, pdev);
> +       unbind_from_irqhandler(irq_from_evtchn(pdev->evtchn), pdev);
> 
>         if (pdev->evtchn != INVALID_EVTCHN)
>                 xenbus_free_evtchn(pdev->xdev, pdev->evtchn);

Re: PV driver domains and S3 sleep

[Konrad Rzeszutek Wilk]

On Thu, Sep 16, 2010 at 01:44:24PM +0200, Rafal Wojtczuk wrote:
> Hello,
> 
> The topic is self-explanatory: how to ensure that a PV driver domain correctly 
> prepares its PCI devices for S3 sleep?
> If I do "pm-suspend" in dom0, and the driver domain has active network interfaces, 
> suspend hangs the system. Yes, in case of this particular machine, suspend works
> fine when there is no driver domain. 
> It is possible to manually invoke scripts from /usr/lib64/pm-utils/sleep.d/ in driver 
> domain. In the test case, "ifconfig down wlan0" in the driver domain allows
> the suspend to go smoothly. But generally, is it enough ? The kernel device driver should 

The pci_disable calls that are made do put the devices in the D3 (or is it D0 state).
However those calls are not made when you do 'ifconfig X down' (I think). You need
to do 'rmmod ipw2100' to trigger those calls, or trigger the drivers' suspend call
invocation.

The drivers' suspend call invocation is a twisty maze of dependency (ie, must first
suspend the driver, and only after that you can suspend the PCI bus).

The S3 suspend on Linux also freezez the user space, cgroups, and whole bunch
of other stuff. But you don't care about that.

What I think you care about is to put the device in the appropiate D state.

> prepare the PCI device properly for S3, shouldn't it ?  
> Would it be more proper to [somehow] notify a driver domain _kernel_ that we are 
> going to S3 (just like dom0 kernel is notified), and let it execute all necessary actions 
> (including, but not only, launching of usermode pm-utils scripts), just like dom0 kernel 
> does ? Would it work at all, considering that driver domain kernel has no access to 
> ACPI tables ? 

I think that depends on the PCI device. In laptop world, the wireless card can
do some weird stuff when you press Ctrl-F5 for example - it would invoke some 
ACPI code (well, the Linux kernel AML driver would invoke it), which then
disables/unloads the driver as appropiate. With DomU having no ACPI support, it means
that the Dom0 would yank the PCI device away from the DomU - which actually considering
that we are using pciback as the owner, would mean you could pass a request to the
DomU saying: "Hey, reconfigure now. Device going away." And I think that might
work today actually.

But back to putting the device in the appropiate D state. You could
pass in DomU a call akin to doing "suspend" > /sys/power/state
which should do the appropiate PCI move.

> Currently, how are these issues taken care of in the mainstream Xen? 

Never explored I fear.
> 
> Thanks in advance,
> Rafal Wojtczuk
> 
> 
>  
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: pci hotplug problem

[shivprashant]

 Hi Kenji,

I too presumed that. The board (based on MPC8315erdb from Freescale) has
two PCIe controllers to which two slots are connected. The controllers
are compatible with PCI Express Base Specification Revision 1.0a and the
PCI Interface is compatible with PCI Local Bus Specification Rev. 2.3.
The driver makes use of PCI Hotplug controller core to
register/deregister hotplug slots.

Did I answer your question?

Regards,
Shivaprashanth

On Tuesday 14 June 2011 07:42 AM, Kenji Kaneshige wrote:
> Shivaprashanth,
>
> Sorry for my delayed responding.
> (I'm on vacation this week)
>
> I could not find any hotplug capable slot defined by PCIe spec in your
> lspci output. It should be the reason why pciehp detects no slot on
> your system. What kind of hotplug controller does your hotplug driver
> handle?
>
> Regards,
> Kenji Kaneshige
>
>
> (2011/06/09 18:30), shivprashant wrote:
>>   below is the dmesg log after loading pciehp.ko with debug enabled
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------
>> -sh-2.05b# insmod /pciehp.ko pciehp_debug=1
>> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>> -sh-2.05b# dmesg
>> Using MPC831x RDB machine description
>> Linux version 2.6.36.1 (root@shiva-desktop) (gcc version 4.1.2) #5 Thu
>> Jun 9 14:40:17 IST 2011
>> Found legacy serial port 0 for /immr@e0000000/serial@4500
>> memà004500, taddrà004500, irq=0, clk\x133333333, speed=0
>> Found legacy serial port 1 for /immr@e0000000/serial@4600
>> memà004600, taddrà004600, irq=0, clk\x133333333, speed=0
>> Found FSL PCI host bridge at 0x00000000e0008500. Firmware bus number: 0->0
>> PCI host bridge /pci@e0008500 (primary) ranges:
>> MEM 0x0000000090000000..0x000000009fffffff ->  0x0000000090000000
>> MEM 0x0000000080000000..0x000000008fffffff ->  0x0000000080000000 Prefetch
>> IO 0x00000000e0300000..0x00000000e03fffff ->  0x0000000000000000
>> Found FSL PCI host bridge at 0x00000000e0009000. Firmware bus number: 0->255
>> PCI host bridge /pcie@e0009000 ranges:
>> MEM 0x0000000040000000..0x000000004fffffff ->  0x0000000040000000
>> IO 0x0000000051000000..0x00000000517fffff ->  0x0000000000000000
>> Found FSL PCI host bridge at 0x00000000e000a000. Firmware bus number: 0->255
>> PCI host bridge /pcie@e000a000 ranges:
>> MEM 0x0000000060000000..0x000000006fffffff ->  0x0000000060000000
>> IO 0x0000000071000000..0x00000000717fffff ->  0x0000000000000000
>> Top of RAM: 0x10000000, Total RAM: 0x10000000
>> Memory hole size: 0MB
>> Zone PFN ranges:
>> DMA 0x00000000 ->  0x00010000
>> Normal empty
>> Movable zone start PFN for each node
>> early_node_map[1] active PFN ranges
>> 0: 0x00000000 ->  0x00010000
>> On node 0 totalpages: 65536
>> free_area_init_node: node 0, pgdat c03fde3c, node_mem_map c0441000
>> DMA zone: 512 pages used for memmap
>> DMA zone: 0 pages reserved
>> DMA zone: 65024 pages, LIFO batch:15
>> Built 1 zonelists in Zone order, mobility grouping on. Total pages: 65024
>> Kernel command line: console=ttyS0,115200 root=/dev/nfs rw
>> nfsroot\x172.16.9.85:/home/shiva/Projects/AJA/pciehp-v2/aja-ltib/ro
>> otfs ip\x172.16.9.100:172.16.9.85:172.16.9.1:255.255.255.0::eth0:off
>> PID hash table entries: 1024 (order: 0, 4096 bytes)
>> Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
>> Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
>> Memory: 255496k/262144k available (3972k kernel code, 6648k reserved,
>> 144k data, 204k bss, 176k init)
>> Kernel virtual memory layout:
>> * 0xfffdf000..0xfffff000 : fixmap
>> * 0xfcef8000..0xfe000000 : early ioremap
>> * 0xd1000000..0xfcef8000 : vmalloc&  ioremap
>> Hierarchical RCU implementation.
>> RCU-based detection of stalled CPUs is disabled.
>> Verbose stalled-CPUs detection is disabled.
>> NR_IRQS:512 nr_irqs:512
>> IPIC (128 IRQ sources) at d1000700
>> time_init: decrementer frequency = 33.333334 MHz
>> time_init: processor frequency = 400.000008 MHz
>> clocksource: timebase mult[77ffffd] shift[22] registered
>> clockevent: decrementer mult[888888b] shift[32] cpu[0]
>> Console: colour dummy device 80x25
>> pid_max: default: 32768 minimum: 301
>> Mount-cache hash table entries: 512
>> NET: Registered protocol family 16
>> alloc irq_desc for 38 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 38 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 38
>> PCI: Probing PCI hardware
>> pci 0000:00:00.0: reg 10: [mem 0x00000000-0x000fffff]
>> pci 0000:00:00.0: reg 18: [mem 0x00000000-0x0fffffff 64bit pref]
>> pci 0000:00:00.0: supports D1 D2
>> pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
>> pci 0000:00:00.0: PME# disabled
>> pci 0000:00:10.0: reg 10: [mem 0x90000000-0x900007ff]
>> pci 0000:00:10.0: reg 14: [mem 0x90004000-0x90007fff]
>> pci 0000:00:10.0: supports D1 D2
>> pci 0000:00:10.0: PME# supported from D0 D1 D2 D3hot
>> pci 0000:00:10.0: PME# disabled
>> alloc irq_desc for 48 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 48 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 48
>> pci_bus 0000:00: resource 0 [io 0x0000-0xfffff]
>> pci_bus 0000:00: resource 1 [mem 0x90000000-0x9fffffff]
>> pci_bus 0000:00: resource 2 [mem 0x80000000-0x8fffffff pref]
>> pci_bus 0001:01: resource 0 [io 0xff7fe000-0xffffdfff]
>> pci_bus 0001:01: resource 1 [mem 0x40000000-0x4fffffff]
>> pci_bus 0002:02: resource 0 [io 0xfeffc000-0xff7fbfff]
>> pci_bus 0002:02: resource 1 [mem 0x60000000-0x6fffffff]
>> Registering ipic with sysfs...
>> bio: create slab<bio-0>  at 0
>> vgaarb: loaded
>> SCSI subsystem initialized
>> libata version 3.00 loaded.
>> Freescale Elo / Elo Plus DMA driver
>> Switching to clocksource timebase
>> NET: Registered protocol family 2
>> IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
>> TCP established hash table entries: 8192 (order: 4, 65536 bytes)
>> TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
>> TCP: Hash tables configured (established 8192 bind 8192)
>> TCP reno registered
>> UDP hash table entries: 256 (order: 0, 4096 bytes)
>> UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
>> NET: Registered protocol family 1
>> RPC: Registered udp transport module.
>> RPC: Registered tcp transport module.
>> RPC: Registered tcp NFSv4.1 backchannel transport module.
>> PCI: CLS 32 bytes, default 32
>> alloc irq_desc for 16 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 9 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 16
>> alloc irq_desc for 17 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 10 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 17
>> alloc irq_desc for 77 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 77 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 77
>> alloc irq_desc for 18 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 16 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 18
>> alloc irq_desc for 71 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 71 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 71
>> of:fsl-elo-dma e00082a8.dma: #0 (fsl,elo-dma-channel), irq 71
>> of:fsl-elo-dma e00082a8.dma: #1 (fsl,elo-dma-channel), irq 71
>> of:fsl-elo-dma e00082a8.dma: #2 (fsl,elo-dma-channel), irq 71
>> of:fsl-elo-dma e00082a8.dma: #3 (fsl,elo-dma-channel), irq 71
>> alloc irq_desc for 32 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 32 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 32
>> alloc irq_desc for 33 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 33 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 33
>> alloc irq_desc for 34 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 34 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 34
>> alloc irq_desc for 44 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 44 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 44
>> alloc irq_desc for 45 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 45 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 45
>> alloc irq_desc for 67 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 67 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 67
>> alloc irq_desc for 19 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 4 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 19
>> alloc irq_desc for 81 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 81 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 81
>> alloc irq_desc for 82 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 82 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 82
>> alloc irq_desc for 86 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 86 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 86
>> alloc irq_desc for 87 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 87 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 87
>> alloc irq_desc for 88 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 88 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 88
>> alloc irq_desc for 89 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 89 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 89
>> Setting up Freescale MSI support
>> alloc irq_desc for 80 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 80 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 80
>> alloc irq_desc for 90 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 90 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 90
>> alloc irq_desc for 78 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 78 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 78
>> alloc irq_desc for 84 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 84 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 84
>> alloc irq_desc for 72 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 72 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 72
>> alloc irq_desc for 91 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 91 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 91
>> alloc irq_desc for 79 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 79 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 79
>> alloc irq_desc for 85 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 85 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 85
>> alloc irq_desc for 73 on node 0
>> alloc kstat_irqs on node 0
>> irq: irq 73 on host /immr@e0000000/interrupt-controller@700 mapped to
>> virtual irq 73
>> msgmni has been set to 499
>> io scheduler noop registered
>> io scheduler deadline registered
>> io scheduler cfq registered (default)
>> pci_hotplug: PCI Hot Plug PCI Core version: 0.5
>> Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
>> serial8250.0: ttyS0 at MMIO 0xe0004500 (irq = 16) is a 16550A
>> console [ttyS0] enabled
>> serial8250.0: ttyS1 at MMIO 0xe0004600 (irq = 17) is a 16550A
>> e0004500.serial: ttyS0 at MMIO 0xe0004500 (irq = 16) is a 16550
>> e0004600.serial: ttyS1 at MMIO 0xe0004600 (irq = 17) is a 16550
>> brd: module loaded
>> loop: module loaded
>> f8000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank. Manufacturer
>> ID 0x000001 Chip ID 0x002301
>> f8000000.flash: Found 1 x16 devices at 0x4000000 in 16-bit bank
>> Amd/Fujitsu Extended Query Table at 0x0040
>> Amd/Fujitsu Extended Query version 1.3.
>> number of CFI chips: 2
>> RedBoot partition parsing not available
>> Creating 8 MTD partitions on "f8000000.flash":
>> 0x000000000000-0x000002000000 : "AppA"
>> 0x000002000000-0x000004000000 : "AppB"
>> 0x000004000000-0x000006000000 : "SafeBoot"
>> 0x000007ea0000-0x000007ec0000 : "ConfigB"
>> 0x000007f60000-0x000007f80000 : "UserRegsA"
>> 0x000007f80000-0x000007fa0000 : "UserRegsB"
>> 0x000007fa0000-0x000007fc0000 : "ConfigA"
>> 0x000006400000-0x000006420000 : "UBootShared"
>> of:mpc8xxx_spi e0007000.spi: at 0xd103c000 (irq = 18), CPU mode
>> Fixed MDIO Bus: probed
>> eth0: Gianfar Ethernet Controller Version 1.2, 00:0c:17:88:02:35
>> eth0: Running with NAPI enabled
>> eth0: RX BD ring size for Q[0]: 256
>> eth0: TX BD ring size for Q[0]: 256
>> Freescale PowerQUICC MII Bus: probed
>> i2c /dev entries driver
>> of:mpc-i2c e0003000.i2c: timeout 1000000 us
>> rtc-ds1307 0-0068: rtc core: registered ds1339 as rtc0
>> TCP cubic registered
>> NET: Registered protocol family 17
>> rtc-ds1307 0-0068: setting system clock to 2011-06-09 02:04:50 UTC
>> (1307585090)
>> IP-Config: Complete:
>> device=eth0, addr\x172.16.9.100, mask%5.255.255.0, gw\x172.16.9.1,
>> host\x172.16.9.100, domain=, nis-domain=(none),
>> bootserver\x172.16.9.85, rootserver\x172.16.9.85, rootpath>> Looking up port of RPC 100003/2 on 172.16.9.85
>> PHY: mdio@e0024520:01 - Link is Up - 100/Full
>> Looking up port of RPC 100005/1 on 172.16.9.85
>> VFS: Mounted root (nfs filesystem) on device 0:12.
>> Freeing unused kernel memory: 176k init
>> pciehp: pcie_port_service_register = 0
>> pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>> -sh-2.05b#
>> -sh-2.05b#
>> -sh-2.05b# lspci -vv
>> 00:00.0 Power PC: Freescale Semiconductor Inc Unknown device 00b5 (rev 12)
>> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR+ FastB2B-
>> Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSELúst>TAbort-<TAbort-
>> <MAbort+>SERR-<PERR-
>> Latency: 128, Cache Line Size: 32 bytes
>> Interrupt: pin A routed to IRQ 0
>> Region 2: Memory at<unassigned>  (64-bit, prefetchable)
>> Region 4: Memory at<unassigned>  (64-bit, non-prefetchable)
>> Capabilities: [48] #06 [0000]
>> Capabilities: [80] Power Management version 3
>> Flags: PMEClk+ DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
>> Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>
>> 00:10.0 FireWire (IEEE 1394): Texas Instruments TSB82AA2 IEEE-1394b Link
>> Layer Controller (rev 02) (prog-if 10 [OHCI])
>> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
>> Stepping- SERR- FastB2B-
>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium>TAbort-
>> <TAbort-<MAbort->SERR-<PERR-
>> Latency: 128 (500ns min, 1000ns max), Cache Line Size: 32 bytes
>> Interrupt: pin A routed to IRQ 48
>> Region 0: Memory at 90000000 (32-bit, non-prefetchable) [size=2K]
>> Region 1: Memory at 90004000 (32-bit, non-prefetchable) [size\x16K]
>> Capabilities: [44] Power Management version 2
>> Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
>> Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>> On Thursday 09 June 2011 02:01 PM, Kenji Kaneshige wrote:
>>> (2011/06/09 16:54), shivprashant wrote:
>>>>    i built pciehp statically. the driver gets loaded but probe is not
>>>> called. do i need to modify anything in pciehp driver to make it work
>>>> with my platform?
>>>>
>>> I have no idea so far.
>>>
>>> Can you send me the following information?
>>>
>>> - dmesg output after loading pciehp with pciehp_debug module option
>>>    (if you built pciehp as builtin driver, please add pciehp.pciehp_debug
>>>     to kernel parameter and boot)
>>>
>>> - "lspci -vv" output (as root user).
>>>
>>> Regards,
>>> Kenji Kaneshige
>>>
>>>
>>>
>>>> On Thursday 09 June 2011 01:09 PM, Kenji Kaneshige wrote:
>>>>> Why not use pciehp?
>>>>>
>>>>> Thanks,
>>>>> Kenji Kaneshige
>>>>>
>>>>>
>>>>> (2011/06/09 14:31), shivprashant wrote:
>>>>>> hi,
>>>>>> i am writing pci express hotplug driver for mpc8315erdb(powerpc). the driver uses PCI hotplug subsystem not pcie port bus driver. i am able to do insertion and removal of pci express cards and confirm using 'lspci' command. driver is working with SATA and PATA memory cards with some minor modifications in the kernel. the problem i am facing now is when i try to repeatedly insert and remove the kernel is crashing, although non-fatal. normal insertion/removal works fine but repeated insertion/removal fails. pls guide me where am i wrong or what changes i should make to recover from pci errors.
>>>>>>
>>>>>> Following is the crash log when repeatedly inserting and removing the PATA cards (i am using delkin 32GB cards),
>>>>>> scsi20 : pata_jmicron
>>>>>> scsi21 : pata_jmicron
>>>>>> ata21: PATA max UDMA/100 cmd 0xff7fa010 ctl 0xff7fa020 bmdma 0xff7fa000 irq 21
>>>>>> ata22: PATA max UDMA/100 cmd 0xff7fa018 ctl 0xff7fa024 bmdma 0xff7fa008 irq 21
>>>>>> PCIE Card Removed from Slot 0
>>>>>> Oops: Machine check, sig: 7 [#1]
>>>>>> MPC831x RDB
>>>>>> last sysfs file: /sys/devices/pci0002:02/0002:02:00.0/0002:03:00.0/class
>>>>>> Modules linked in:
>>>>>> NIP: c00124c4 LR: c00124b0 CTR: c0017998
>>>>>> REGS: cfad3cb0 TRAP: 0200 Not tainted (2.6.36.1)
>>>>>> MSR: 00049030<EE,ME,IR,DR>   CR: 22204084 XER: 20000000
>>>>>> TASK = cfa9c030[1282] 'scsi_eh_20' THREAD: cfad2000
>>>>>> GPR00: 00000000 cfad3d60 cfa9c030 00000000 cfad3d08 00e00000 a91e4fb9 00000001
>>>>>> GPR08: c03fd688 fdef7000 fd6f7000 c042a0e0 00005a07 00020d78 c02fbd0c ffffffff
>>>>>> GPR16: 00000000 00000000 00000000 00000004 c01ff498 00000001 cfac94c8 cfac8000
>>>>>> GPR24: c01fa87c 00000003 ffff2a2d cfac801c 00000001 00000000 cfac9430 fd6f8012
>>>>>> NIP [c00124c4] ioread8+0x2c/0x328
>>>>>> LR [c00124b0] ioread8+0x18/0x328
>>>>>> Call Trace:
>>>>>> [cfad3d60] [c002bccc] msleep+0x1c/0x34 (unreliable)
>>>>>> [cfad3d70] [c01fa77c] ata_sff_wait_after_reset+0x6c/0x16c
>>>>>> [cfad3da0] [c01fa934] ata_sff_softreset+0xb8/0x204
>>>>>> [cfad3de0] [c01f2604] ata_do_reset+0xa0/0xb4
>>>>>> [cfad3e00] [c01f33ac] ata_eh_reset+0x3e4/0xcf4
>>>>>> [cfad3e80] [c01f59e8] ata_eh_recover+0x284/0x10e0
>>>>>> [cfad3f00] [c01f6a0c] ata_do_eh+0x4c/0xb0
>>>>>> [cfad3f20] [c01f926c] ata_sff_error_handler+0x110/0x140
>>>>>> [cfad3f40] [c01f7680] ata_scsi_error+0x2ec/0x4e0
>>>>>> [cfad3f70] [c01d2664] scsi_error_handler+0xc4/0x39c
>>>>>> [cfad3fb0] [c0039d4c] kthread+0x7c/0x80
>>>>>> [cfad3ff0] [c000f200] kernel_thread+0x4c/0x68
>>>>>> Instruction dump:
>>>>>> 4e800020 9421fff0 7c0802a6 93e1000c 7c7f1b78 90010014 481835f5 2f830000
>>>>>> 38600000 419e0018 7c0004ac 881f0000<0c000000>   4c00012c 5403063e 80010014
>>>>>> ---[ end trace fb9766f858e66ef2 ]---
>>>>>>
>>>>>>
>>>>>>
>>
>


-- 
With Regards,
Shivaprashanth H