PCI hotplug problem [was: PV driver domains and S3 sleep]

[Rafal Wojtczuk <rafal@invisiblethingslab.com>]

On Thu, Sep 16, 2010 at 12:52:02PM +0100, Keir Fraser wrote:
> > The topic is self-explanatory: how to ensure that a PV driver domain correctly
> > prepares its PCI devices for S3 sleep?
[cut]
> > Currently, how are these issues taken care of in the mainstream Xen?

> I don't think it currently is handled. HVM driver domains (using VT-d or
> equivalent) can be put into virtual S3. We would need an equivalent concept
> for PV driver domains. Or for devices to be hot-unplugged from the driver
> domain, and re-plugged on resume?

The idea of using PCI hotplug is nice, however, PCI hotplug does not seem to
work with the used setup (xen-3.4.3, all 64bit). Hot-unplug works, however the 
following hotplug makes the driver domain kernel spit out the following:

Sep 24 09:46:01 localhost kernel: [  113.045927] pcifront pci-0: Rescanning
PCI Frontend Bus 0000:00
Sep 24 09:46:15 localhost kernel: [  126.843990] pcifront pci-0: Rescanning
PCI Frontend Bus 0000:00
Sep 24 09:46:15 localhost kernel: [  126.846217] pcifront pci-0: New device
on 0000:00:01.00 found.
Sep 24 09:46:15 localhost kernel: [  126.846523] iwlagn 0000:00:01.0: device
not available (can't reserve [mem 0xf8000000-0xf8001fff 64bit])

^C
[root@localhost ~]# cat /proc/iomem 
f6000000-f600ffff : 0000:00:00.0
  f6000000-f600ffff : tg3
[root@localhost ~]# lspci
00:00.0 Ethernet controller: Broadcom Corporation NetLink BCM5787M Gigabit
Ethernet PCI Express (rev 02)
00:01.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN
[Kedron] Network Connection (rev 61)

Nothing suspicious in xend, Xen and dom0 logs.

The domU and dom0 kernels are the same, 2.6.34.1-10.xenlinux (SUSE patches
for 2.6.34.1).

With old pvops (2.6.31.9-1.pvops0) in domU, the message on the hot-plug is similar:
Sep 24 09:50:40 localhost kernel: pcifront pci-0: Rescanning PCI Frontend
Bus 0000:00
Sep 24 09:50:51 localhost kernel: pcifront pci-0: Rescanning PCI Frontend
Bus 0000:00
Sep 24 09:50:51 localhost kernel: pcifront pci-0: New device on
0000:00:01.00 found.
Sep 24 09:50:51 localhost kernel: iwlagn 0000:00:01.0: device not available
because of BAR 0 [0xf8000000-0xf8001fff] collisions

Others seem to experience similar problems (e.g.
http://permalink.gmane.org/gmane.comp.emulators.xen.devel/80766). Does
anyone know the solution ?

Regards,
Rafal Wojtczuk