[RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Zhang Rui]

device parallel resume support for PCI devices

Signed-off-by: Zhang Rui <rui.zhang@intel.com>
---
 drivers/pci/bus.c    |    4 ++++
 drivers/pci/remove.c |    2 ++
 2 files changed, 6 insertions(+)

Index: linux-2.6/drivers/pci/bus.c
===================================================================
--- linux-2.6.orig/drivers/pci/bus.c
+++ linux-2.6/drivers/pci/bus.c
@@ -85,6 +85,10 @@ int pci_bus_add_device(struct pci_dev *d
 		return retval;
 
 	dev->is_added = 1;
+
+	/* parallel resume for pci devices on bus 0 */
+	if (dev->bus->number == 0)
+		device_parallel_resume_register(&dev->dev, 1);
 	pci_proc_attach_device(dev);
 	pci_create_sysfs_dev_files(dev);
 	return 0;
Index: linux-2.6/drivers/pci/remove.c
===================================================================
--- linux-2.6.orig/drivers/pci/remove.c
+++ linux-2.6/drivers/pci/remove.c
@@ -92,6 +92,8 @@ EXPORT_SYMBOL(pci_remove_bus);
  */
 void pci_remove_bus_device(struct pci_dev *dev)
 {
+	if (dev->bus->number == 0 && dev->dev.resumed)
+		device_parallel_resume_unregister(&dev->dev);
 	if (dev->subordinate) {
 		struct pci_bus *b = dev->subordinate;
 

Re: [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Matthew Wilcox]

On Fri, Dec 19, 2008 at 02:34:13PM +0800, Zhang Rui wrote:
> +	/* parallel resume for pci devices on bus 0 */
> +	if (dev->bus->number == 0)
> +		device_parallel_resume_register(&dev->dev, 1);

"bus 0" is presumably supposed to mean "devices on a root bus".  I think
you should be checking for dev->bus->parent == NULL instead.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Alan Stern]

On Fri, 19 Dec 2008, Zhang Rui wrote:

> device parallel resume support for PCI devices

There are some PCI devices which won't work right if they are resumed
in parallel.  The example I have in mind is a USB host controller
device with both a high-speed EHCI controller function and some
companion low/full-speed UHCI or OHCI controller functions.  The EHCI
driver relies on the fact that the EHCI controller has a higher
function number and therefore is resumed _after_ the companion
controllers.

We need a way to express dependencies like this between devices.  Or at 
least, there has to be a way to prevent certain devices from resuming 
in parallel.

Alan Stern

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Len Brown]

On Fri, 19 Dec 2008, Alan Stern wrote:

> On Fri, 19 Dec 2008, Zhang Rui wrote:
> 
> > device parallel resume support for PCI devices
> 
> There are some PCI devices which won't work right if they are resumed
> in parallel.  The example I have in mind is a USB host controller
> device with both a high-speed EHCI controller function and some
> companion low/full-speed UHCI or OHCI controller functions.  The EHCI
> driver relies on the fact that the EHCI controller has a higher
> function number and therefore is resumed _after_ the companion
> controllers.
> 
> We need a way to express dependencies like this between devices.  Or at 
> least, there has to be a way to prevent certain devices from resuming 
> in parallel.

Perhaps the most pragmatic approach would be to identify which devices
benefit the most from parallel resume, and use some hooks so that
those device drivers can be optimized to resume in parallel
while everything else takes the existing serialized path?

-- Len Brown, Intel Open Source Technology Center

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Zhang Rui]

On Fri, 2008-12-19 at 23:09 +0800, Alan Stern wrote:
> On Fri, 19 Dec 2008, Zhang Rui wrote:
> 
> > device parallel resume support for PCI devices
> 
> There are some PCI devices which won't work right if they are resumed
> in parallel.  The example I have in mind is a USB host controller
> device with both a high-speed EHCI controller function and some
> companion low/full-speed UHCI or OHCI controller functions.  The EHCI
> driver relies on the fact that the EHCI controller has a higher
> function number and therefore is resumed _after_ the companion
> controllers.
> 
well, I never got this problem before.
is this normal? does this exist on the Intel hardware?
I'd like to do some test if I have chance.

> We need a way to express dependencies like this between devices.  Or at 
> least, there has to be a way to prevent certain devices from resuming 
> in parallel.
> 
agree.
Just like the case above, can you give me a lspci output and show me
what exactly the dependency is?

thanks,
rui

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Alan Stern]

On Mon, 22 Dec 2008, Zhang Rui wrote:

> On Fri, 2008-12-19 at 23:09 +0800, Alan Stern wrote:
> > On Fri, 19 Dec 2008, Zhang Rui wrote:
> > 
> > > device parallel resume support for PCI devices
> > 
> > There are some PCI devices which won't work right if they are resumed
> > in parallel.  The example I have in mind is a USB host controller
> > device with both a high-speed EHCI controller function and some
> > companion low/full-speed UHCI or OHCI controller functions.  The EHCI
> > driver relies on the fact that the EHCI controller has a higher
> > function number and therefore is resumed _after_ the companion
> > controllers.
> > 
> well, I never got this problem before.
> is this normal? does this exist on the Intel hardware?
> I'd like to do some test if I have chance.

Yes, it is normal, and it does exist on Intel hardware.  It isn't a 
_problem_ (or at least, it hasn't been a problem up until now) because 
the order of resuming devices is always the same as the order of 
registration, which in turn is the same as the order of discovery.

You can see this on any Intel ICHx chipset.  I don't have an example 
right here, but take a look at the machines you have lying around.  
There are PCI UHCI controllers at 1d.0, 1d.1, ... and a PCI EHCI 
controller at 1d.7.

The issue arises when you have a low-speed or full-speed device plugged 
in.  The device is handed over from the EHCI controller to one of the 
companion UHCI controllers, because EHCI manages only high-speed 
devices.

Now consider what happens when the system wakes up from hibernation.  
The controllers are re-initialized, and when ehci-hcd sees the
low/full-speed device, it tries to hand the device over to the
companion controller again.  But if the EHCI controller's resume method
runs first then the handover will fail, because the companion
controller has not yet been initialized.  As a result, the device will
be lost and will have to go through rediscovery and re-enumeration.  
This could be very annoying if, for example, the device contains a
mounted filesystem.

> Just like the case above, can you give me a lspci output and show me
> what exactly the dependency is?

The dependency is simple: The 1d.7 device must be resumed _after_ 1d.0, 
1d.1, ..., 1d.6.

Alan Stern

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Zhang Rui]

On Mon, 2008-12-22 at 23:52 +0800, Alan Stern wrote:
> On Mon, 22 Dec 2008, Zhang Rui wrote:
> 
> > On Fri, 2008-12-19 at 23:09 +0800, Alan Stern wrote:
> > > On Fri, 19 Dec 2008, Zhang Rui wrote:
> > > 
> > > > device parallel resume support for PCI devices
> > > 
> > > There are some PCI devices which won't work right if they are resumed
> > > in parallel.  The example I have in mind is a USB host controller
> > > device with both a high-speed EHCI controller function and some
> > > companion low/full-speed UHCI or OHCI controller functions.  The EHCI
> > > driver relies on the fact that the EHCI controller has a higher
> > > function number and therefore is resumed _after_ the companion
> > > controllers.
> > > 
> > well, I never got this problem before.
> > is this normal? does this exist on the Intel hardware?
> > I'd like to do some test if I have chance.
> 
> Yes, it is normal, and it does exist on Intel hardware.  It isn't a 
> _problem_ (or at least, it hasn't been a problem up until now) because 
> the order of resuming devices is always the same as the order of 
> registration, which in turn is the same as the order of discovery.
> 
> You can see this on any Intel ICHx chipset.  I don't have an example 
> right here, but take a look at the machines you have lying around.  
> There are PCI UHCI controllers at 1d.0, 1d.1, ... and a PCI EHCI 
> controller at 1d.7.
> 
> The issue arises when you have a low-speed or full-speed device plugged 
> in.  The device is handed over from the EHCI controller to one of the 
> companion UHCI controllers, because EHCI manages only high-speed 
> devices.
> 
> Now consider what happens when the system wakes up from hibernation.  
> The controllers are re-initialized, and when ehci-hcd sees the
> low/full-speed device, it tries to hand the device over to the
> companion controller again.  But if the EHCI controller's resume method
> runs first then the handover will fail, because the companion
> controller has not yet been initialized.  As a result, the device will
> be lost and will have to go through rediscovery and re-enumeration.  
> This could be very annoying if, for example, the device contains a
> mounted filesystem.
> 
> > Just like the case above, can you give me a lspci output and show me
> > what exactly the dependency is?
> 
> The dependency is simple: The 1d.7 device must be resumed _after_ 1d.0, 
> 1d.1, ..., 1d.6.
Then I have a question, what if the uchi_hcd is not loaded?
the ehci host controller doesn't work in this case?

thanks,
rui

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Alan Stern]

On Wed, 24 Dec 2008, Zhang Rui wrote:

> Then I have a question, what if the uchi_hcd is not loaded?
> the ehci host controller doesn't work in this case?

It does work.  But it handles only high-speed devices; low-speed and 
full-speed devices will be ignored (unless they are connected through 
a high-speed hub).

Alan Stern

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Zhao Yakui]

On Mon, 2008-12-22 at 23:52 +0800, Alan Stern wrote:
> On Mon, 22 Dec 2008, Zhang Rui wrote:
> 
> > On Fri, 2008-12-19 at 23:09 +0800, Alan Stern wrote:
> > > On Fri, 19 Dec 2008, Zhang Rui wrote:
> > > 
> > > > device parallel resume support for PCI devices
> > > 
> > > There are some PCI devices which won't work right if they are resumed
> > > in parallel.  The example I have in mind is a USB host controller
> > > device with both a high-speed EHCI controller function and some
> > > companion low/full-speed UHCI or OHCI controller functions.  The EHCI
> > > driver relies on the fact that the EHCI controller has a higher
> > > function number and therefore is resumed _after_ the companion
> > > controllers.
> > > 
> > well, I never got this problem before.
> > is this normal? does this exist on the Intel hardware?
> > I'd like to do some test if I have chance.
> 
> Yes, it is normal, and it does exist on Intel hardware.  It isn't a 
> _problem_ (or at least, it hasn't been a problem up until now) because 
> the order of resuming devices is always the same as the order of 
> registration, which in turn is the same as the order of discovery.
> 
> You can see this on any Intel ICHx chipset.  I don't have an example 
> right here, but take a look at the machines you have lying around.  
> There are PCI UHCI controllers at 1d.0, 1d.1, ... and a PCI EHCI 
> controller at 1d.7.
> 
> The issue arises when you have a low-speed or full-speed device plugged 
> in.  The device is handed over from the EHCI controller to one of the 
> companion UHCI controllers, because EHCI manages only high-speed 
> devices.

> Now consider what happens when the system wakes up from hibernation.  
> The controllers are re-initialized, and when ehci-hcd sees the
> low/full-speed device, it tries to hand the device over to the
> companion controller again.  But if the EHCI controller's resume method
> runs first then the handover will fail, because the companion
> controller has not yet been initialized.  As a result, the device will
> be lost and will have to go through rediscovery and re-enumeration.  
> This could be very annoying if, for example, the device contains a
> mounted filesystem.
Hi, Alan
   Thanks for the detailed explanation. Understand the dependency
between UHCI and EHCI. This should be considered in course of resuming.
   I still have another question. 
   There exists a ROOT HUB device for every UHCI/EHCI controllers.
Should the dependency between UHCI root hub and EHCI root HUB be
considered? 
   Should the UHCI root hub be resumed before EHCI root hub? (Of course
the EHCI host controller will be resumed after UHCI host controllers).

thanks    
> 
> > Just like the case above, can you give me a lspci output and show me
> > what exactly the dependency is?
> 
> The dependency is simple: The 1d.7 device must be resumed _after_ 1d.0, 
> 1d.1, ..., 1d.6.

> 
> Alan Stern
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Zhao Yakui]

On Wed, 2008-12-24 at 10:47 +0800, Alan Stern wrote:
> On Wed, 24 Dec 2008, Zhang Rui wrote:
> 
> > Then I have a question, what if the uchi_hcd is not loaded?
> > the ehci host controller doesn't work in this case?
> 
> It does work.  But it handles only high-speed devices; low-speed and 
> full-speed devices will be ignored (unless they are connected through 
> a high-speed hub).
Hi, Alan
    We do a test on the laptop based on intel ICH9 chipset. In the test
the UHCI HCD driver is not loaded in the boot phase. A low-speed USB
mouse is plugged. In such case the USB mouse is not detected. After the
boot is finished, the uhci hcd driver is loaded manually. Then USB mouse
is loaded and can work well.
    Is this different with that EHCI driver is resumed before UHCI
driver?
    Do there exist the rediscovery and re-enumberation for the low-speed
device if the EHCI driver resume is finished before the UHCI driver?
    
thanks.
     
> 
> Alan Stern
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Zhao Yakui]

On Wed, 2008-12-24 at 10:47 +0800, Alan Stern wrote:
> On Wed, 24 Dec 2008, Zhang Rui wrote:
> 
> > Then I have a question, what if the uchi_hcd is not loaded?
> > the ehci host controller doesn't work in this case?
> 
> It does work.  But it handles only high-speed devices; low-speed and 
> full-speed devices will be ignored (unless they are connected through 
> a high-speed hub).
Hi, Alan
    We do a test on the laptop based on intel ICH9 chipset. In the test
the UHCI HCD driver is not loaded in the boot phase. A low-speed USB
mouse is plugged. In such case the USB mouse is not detected. After the
boot is finished, the uhci hcd driver is loaded manually. Then USB mouse
is loaded and can work well.
    Is this different with that EHCI driver is resumed before UHCI
driver?
    Do there exist the rediscovery and re-enumberation for the low-speed
device if the EHCI driver resume is finished before the UHCI driver?
    
thanks.
     
> 
> Alan Stern
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Alan Stern]

On Fri, 16 Jan 2009, Zhao Yakui wrote:

> Hi, Alan
>    Thanks for the detailed explanation. Understand the dependency
> between UHCI and EHCI. This should be considered in course of resuming.
>    I still have another question. 
>    There exists a ROOT HUB device for every UHCI/EHCI controllers.
> Should the dependency between UHCI root hub and EHCI root HUB be
> considered? 

No, there is no dependency between the root hubs.  Only between the PCI 
host controller devices.

>    Should the UHCI root hub be resumed before EHCI root hub? (Of course
> the EHCI host controller will be resumed after UHCI host controllers).

It doesn't matter.  We are already guaranteed that the EHCI root hub 
will be resumed after the EHCI host controller (likewise for UHCI)
and nothing more is needed.

Alan Stern

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Alan Stern]

On Fri, 16 Jan 2009, Zhao Yakui wrote:

> Hi, Alan
>     We do a test on the laptop based on intel ICH9 chipset. In the test
> the UHCI HCD driver is not loaded in the boot phase. A low-speed USB
> mouse is plugged. In such case the USB mouse is not detected. After the
> boot is finished, the uhci hcd driver is loaded manually. Then USB mouse
> is loaded and can work well.
>     Is this different with that EHCI driver is resumed before UHCI
> driver?

I don't understand your question.  Can you rephrase it more explicitly?

>     Do there exist the rediscovery and re-enumberation for the low-speed
> device if the EHCI driver resume is finished before the UHCI driver?

I still don't understand.  You said your test involved booting without
uhci-hcd, then plugging in a mouse, then loading uhci-hcd manually.  
The test did not involve suspending or resuming.  So why ask a question
about EHCI driver resume?  The question has nothing to do with the
test.

Alan Stern

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Zhao Yakui]

On Fri, 2009-01-16 at 23:10 +0800, Alan Stern wrote:
> On Fri, 16 Jan 2009, Zhao Yakui wrote:
> 
> > Hi, Alan
> >     We do a test on the laptop based on intel ICH9 chipset. In the test
> > the UHCI HCD driver is not loaded in the boot phase. A low-speed USB
> > mouse is plugged. In such case the USB mouse is not detected. After the
> > boot is finished, the uhci hcd driver is loaded manually. Then USB mouse
> > is loaded and can work well.
> >     Is this different with that EHCI driver is resumed before UHCI
> > driver?
> 
> I don't understand your question.  Can you rephrase it more explicitly?
> 
> >     Do there exist the rediscovery and re-enumberation for the low-speed
> > device if the EHCI driver resume is finished before the UHCI driver?
> 
> I still don't understand.  You said your test involved booting without
> uhci-hcd, then plugging in a mouse, then loading uhci-hcd manually.  
> The test did not involve suspending or resuming.  So why ask a question
> about EHCI driver resume?  The question has nothing to do with the
> test.
The low-speed USB device is ignored by EHCI HCD, And when the uhci
driver is loaded after EHCI driver, the low-speed USB device can be
handler correctly.
So we hope to resume EHCI before UHCI and see whether the exists the
rediscovery and re-enumeration.
In fact there exists quite a lot of delay in UHCI/EHCI driver. We want
to use the parallel resume to reduce the resume time.

I also do another test, in which the EHCI hcd driver is resumed before
UHCI. This is based on Intel ICH9 chipset.
     >1a.0/1a.1/1a.2 is UHCI
     >1a.7 is EHCI 
     EHCI will be resumed before the corresponding UHCI.

    In my test there is no rediscovery and re-enumeration about
low-speed USB device. And after it is resumed, the two low-speed device
can still work well.
    
So my question is whether the EHCI can be resumed before UHCI. If we
can't, we want to do partial parallel. This means that the UHCI devices
will be resumed parallelly . After the UHCI resume is finished, the ECHI
will be resumed.  But we don't know whether the resume order of root HUB
should be considered.
If we can, it will be very simple.

Thanks for your comments and advice.
Best regards
Yakui
> 
> Alan Stern
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Alan Stern]

On Sat, 17 Jan 2009, Zhao Yakui wrote:

> The low-speed USB device is ignored by EHCI HCD, And when the uhci
> driver is loaded after EHCI driver, the low-speed USB device can be
> handler correctly.
> So we hope to resume EHCI before UHCI and see whether the exists the
> rediscovery and re-enumeration.

Okay; go ahead.  The order in which the drivers are loaded doesn't make 
any difference.

> In fact there exists quite a lot of delay in UHCI/EHCI driver. We want
> to use the parallel resume to reduce the resume time.

Yes, that would be nice.

> I also do another test, in which the EHCI hcd driver is resumed before
> UHCI. This is based on Intel ICH9 chipset.
>      >1a.0/1a.1/1a.2 is UHCI
>      >1a.7 is EHCI 
>      EHCI will be resumed before the corresponding UHCI.
> 
>     In my test there is no rediscovery and re-enumeration about
> low-speed USB device. And after it is resumed, the two low-speed device
> can still work well.

First, are you testing suspend-to-ram or are you testing hibernation?  
A good test would be hibernation and unplugging/replugging the mouse
while the system is asleep.  An even better test would be 
unplugging/replugging the computer's power cord while it is asleep.

Second, can you provide the dmesg log showing what happens when you run
your test using a kernel with CONFIG_USB_DEBUG enabled?

> So my question is whether the EHCI can be resumed before UHCI.

I'll have to see the detailed log of your test.  I know that on my 
ICH4 system, similar tests in the past have caused rediscovery.

>  If we
> can't, we want to do partial parallel. This means that the UHCI devices
> will be resumed parallelly . After the UHCI resume is finished, the ECHI
> will be resumed.  But we don't know whether the resume order of root HUB
> should be considered.
> If we can, it will be very simple.

That would be okay.  If each UHCI and OHCI companion controller is
resumed before its partner EHCI controller, then the order of resuming
the root hubs doesn't matter.

Alan Stern

Re: [linux-pm] [RFC PATCH 2/3] PCI: parallel resume for pci devices on bus 0

[Zhao Yakui]

On Sun, 2009-01-18 at 01:15 +0800, Alan Stern wrote:
> On Sat, 17 Jan 2009, Zhao Yakui wrote:
> 
> > The low-speed USB device is ignored by EHCI HCD, And when the uhci
> > driver is loaded after EHCI driver, the low-speed USB device can be
> > handler correctly.
> > So we hope to resume EHCI before UHCI and see whether the exists the
> > rediscovery and re-enumeration.
> 
> Okay; go ahead.  The order in which the drivers are loaded doesn't make 
> any difference.
> 
> > In fact there exists quite a lot of delay in UHCI/EHCI driver. We want
> > to use the parallel resume to reduce the resume time.
> 
> Yes, that would be nice.
> 
> > I also do another test, in which the EHCI hcd driver is resumed before
> > UHCI. This is based on Intel ICH9 chipset.
> >      >1a.0/1a.1/1a.2 is UHCI
> >      >1a.7 is EHCI 
> >      EHCI will be resumed before the corresponding UHCI.
> > 
> >     In my test there is no rediscovery and re-enumeration about
> > low-speed USB device. And after it is resumed, the two low-speed device
> > can still work well.
> 
> First, are you testing suspend-to-ram or are you testing hibernation?  
> A good test would be hibernation and unplugging/replugging the mouse
> while the system is asleep.  An even better test would be 
> unplugging/replugging the computer's power cord while it is asleep.
> 
> Second, can you provide the dmesg log showing what happens when you run
> your test using a kernel with CONFIG_USB_DEBUG enabled?
Thanks for the detailed explanation.
I test it by using suspend to ram. In such case there is no rediscovery.
Of course I will do the hibernation test and then send the detailed log
to you.
Thanks.
> 
> > So my question is whether the EHCI can be resumed before UHCI.
> 
> I'll have to see the detailed log of your test.  I know that on my 
> ICH4 system, similar tests in the past have caused rediscovery.
> 
> >  If we
> > can't, we want to do partial parallel. This means that the UHCI devices
> > will be resumed parallelly . After the UHCI resume is finished, the ECHI
> > will be resumed.  But we don't know whether the resume order of root HUB
> > should be considered.
> > If we can, it will be very simple.
> 
> That would be okay.  If each UHCI and OHCI companion controller is
> resumed before its partner EHCI controller, then the order of resuming
> the root hubs doesn't matter.
> 
> Alan Stern
>