[Proposal]Another way to save/restore PCI config space for suspend/resume

[Proposal]Another way to save/restore PCI config space for suspend/resume

[Li Shaohua]

Hi,
We suffer from PCI config space issue for a long time, which causes many
system can't correctly resume. Current Linux mechanism isn't sufficient.
Here is a another idea: 
Record all PCI writes in Linux kernel, and redo all the write after
resume in order. The idea assumes Firmware will restore all PCI config
space to the boot time state, which is true at least for IA32.

Reason:
1. Current PCI save/restore routines only cover first 64 bytes
2. No PCI bridge driver currently.
3. Some special devices can't or are difficult to save/restore config
space with current model. Such as PCI link device, it's a sysdev, but
its resume code can't be invoked with irq disabled.
4. ACPI possibly changes special devices' config space, such as host
bridge or LPC bridge. The special devices generally are vender specific,
and possibly will not have a driver forever.

Possibly we must consider other factors:
1.tracking writes alone will not be enough. Some PCI devices may have
restrictions such as something has to be written after it is read and
the like. Still we should be able to do this if we can trace all pci
reads and writes and repeat it at restore.
2. For support hotplug, add a callback for hotplug PCI remove. When a
device is removed, all records about it are removed.
What's your opinions?

Thanks,
Shaohua

Re: [ACPI] [Proposal]Another way to save/restore PCI config space for suspend/resume

[Andi Kleen]

On Tue, Oct 26, 2004 at 12:50:57PM +0800, Li Shaohua wrote:
> Hi,
> We suffer from PCI config space issue for a long time, which causes many
> system can't correctly resume. Current Linux mechanism isn't sufficient.
> Here is a another idea: 
> Record all PCI writes in Linux kernel, and redo all the write after
> resume in order. The idea assumes Firmware will restore all PCI config

This won't work very well for some cases. e.g. on AMD x86-64 the 
IOMMU is flushed by setting/clearing a bit in PCI config space.
AGP implementations work similar. You really don't want to track
all these flushes, it would be far too costly.

> space to the boot time state, which is true at least for IA32.
> 
> Reason:
> 1. Current PCI save/restore routines only cover first 64 bytes

The driver could set a flag if it wants more.

> 2. No PCI bridge driver currently.

That could be fixed I guess? 

> 3. Some special devices can't or are difficult to save/restore config
> space with current model. Such as PCI link device, it's a sysdev, but
> its resume code can't be invoked with irq disabled.

In this case it would be IMHO better to have specialized suspend/resume
functions in the drivers for these oddball devices.

Most likely they will require some special handling anyways
(like special delays etc.) that can't be done by the generic code 

> 4. ACPI possibly changes special devices' config space, such as host
> bridge or LPC bridge. The special devices generally are vender specific,
> and possibly will not have a driver forever.

I didn't get that one.

-Andi

Re: [ACPI] [Proposal]Another way to save/restore PCI config space for suspend/resume

[Len Brown]

What this comes down to is that extended config space is device-specific.
Generic solutions will fail.  Only device drivers will work.

If there are no drivers for PCI bridges to properly save/restore
their config space, then should create them, even if this is all the 
drivers do.

-Len

Re: [ACPI] [Proposal]Another way to save/restore PCI config space for suspend/resume

[Arjan van de Ven]

On Tue, 2004-10-26 at 02:11 -0400, Len Brown wrote:
> What this comes down to is that extended config space is device-specific.
> Generic solutions will fail.  Only device drivers will work.
> 
> If there are no drivers for PCI bridges to properly save/restore
> their config space, then should create them, even if this is all the 
> drivers do.

note that by default, if there is no driver, the first 64 bytes of
config space are saved/restored.
-- 

Re: [ACPI] [Proposal]Another way to save/restore PCI config space for suspend/resume

[Karol Kozimor]

Thus wrote Arjan van de Ven:
> > What this comes down to is that extended config space is device-specific.
> > Generic solutions will fail.  Only device drivers will work.
> > 
> > If there are no drivers for PCI bridges to properly save/restore
> > their config space, then should create them, even if this is all the 
> > drivers do.
> note that by default, if there is no driver, the first 64 bytes of
> config space are saved/restored.

That's not enough -- some devices with no drivers (think LPC bridges) might
need more (see http://bugme.osdl.org/show_bug.cgi?id=3609).
Best regards,

-- 
Karol 'sziwan' Kozimor
sziwan@hell.org.pl

Re: [ACPI] [Proposal]Another way to save/restore PCI config space for suspend/resume

[Matthew Garrett]

On Tue, 2004-10-26 at 10:42 +0200, Arjan van de Ven wrote:
> On Tue, 2004-10-26 at 02:11 -0400, Len Brown wrote:
> > What this comes down to is that extended config space is device-specific.
> > Generic solutions will fail.  Only device drivers will work.
> > 
> > If there are no drivers for PCI bridges to properly save/restore
> > their config space, then should create them, even if this is all the 
> > drivers do.
> 
> note that by default, if there is no driver, the first 64 bytes of
> config space are saved/restored.

On one of my machines, doing this causes the cardbus bridge to explode
on resume (every other character of printks suddenly starts getting left
out, and then the machine hangs). This happens even if I've never loaded
the yenta driver. The naive approach certainly doesn't seem to be safe
on all hardware.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [ACPI] [Proposal]Another way to save/restore PCI config space for suspend/resume

[Li Shaohua]

On Tue, 2004-10-26 at 13:11, Andi Kleen wrote:
> On Tue, Oct 26, 2004 at 12:50:57PM +0800, Li Shaohua wrote:
> > Hi,
> > We suffer from PCI config space issue for a long time, which causes many
> > system can't correctly resume. Current Linux mechanism isn't sufficient.
> > Here is a another idea: 
> > Record all PCI writes in Linux kernel, and redo all the write after
> > resume in order. The idea assumes Firmware will restore all PCI config
> 
> This won't work very well for some cases. e.g. on AMD x86-64 the 
> IOMMU is flushed by setting/clearing a bit in PCI config space.
> AGP implementations work similar. You really don't want to track
> all these flushes, it would be far too costly.
Possibly we can consider some optimizations, such as a driver can
explicitly disable the 'pci record'. The main problem we encountered is
the correctness of suspend/resume.
> 
> > space to the boot time state, which is true at least for IA32.
> > 
> > Reason:
> > 1. Current PCI save/restore routines only cover first 64 bytes
> 
> The driver could set a flag if it wants more.
Extend PCI config space is device specific, general code can't do it.

> > 2. No PCI bridge driver currently.
> 
> That could be fixed I guess? 
if all PCI devices and bridges have drivers, this could be fixed. But I
think it's far away. Another issue here is the hierarchy of devices. A
device in the below of device tree doesn't means it must be resumed
later. Such as a PCI IRQ router, it must resume before all PCI devices.

> 
> > 3. Some special devices can't or are difficult to save/restore config
> > space with current model. Such as PCI link device, it's a sysdev, but
> > its resume code can't be invoked with irq disabled.
> 
> In this case it would be IMHO better to have specialized suspend/resume
> functions in the drivers for these oddball devices.
> 
> Most likely they will require some special handling anyways
> (like special delays etc.) that can't be done by the generic code 
> 
> > 4. ACPI possibly changes special devices' config space, such as host
> > bridge or LPC bridge. The special devices generally are vender specific,
> > and possibly will not have a driver forever.
> 
> I didn't get that one.
One case here is the ASL code will disable/enable EHCI per current OS
(such as disable EHCI if OS isn't win) in some systems. The
disable/enable bit is in ICH.

Thanks,
Shaohua

Re: [Proposal]Another way to save/restore PCI config space for suspend/resume

[Pavel Machek]

Hi!

> We suffer from PCI config space issue for a long time, which causes many
> system can't correctly resume. Current Linux mechanism isn't sufficient.
> Here is a another idea: 
> Record all PCI writes in Linux kernel, and redo all the write after
> resume in order. The idea assumes Firmware will restore all PCI config
> space to the boot time state, which is true at least for IA32.

That looks extremely ugly to me. If you want to do something special
in resume function, just do it there. It will probably share a lot of
code with your init function, anyway.
								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: [ACPI] Re: [Proposal]Another way to save/restore PCI config space for suspend/resume

[Li Shaohua]

On Tue, 2004-10-26 at 17:21, Pavel Machek wrote:
> Hi!
> 
> > We suffer from PCI config space issue for a long time, which causes many
> > system can't correctly resume. Current Linux mechanism isn't sufficient.
> > Here is a another idea: 
> > Record all PCI writes in Linux kernel, and redo all the write after
> > resume in order. The idea assumes Firmware will restore all PCI config
> > space to the boot time state, which is true at least for IA32.
> 
> That looks extremely ugly to me. If you want to do something special
> in resume function, just do it there. It will probably share a lot of
> code with your init function, anyway.
How can you handle devices without driver? And how to save/restore
config space for special devices, such as LPC bridge and host bridge?

-Shaohua

Re: [ACPI] [Proposal]Another way to save/restore PCI config space for suspend/resume

[Hiroshi 2 Itoh]

Hi,

acpi-devel-admin@lists.sourceforge.net wrote on 2004/10/26 13:50:57:

> Hi,
> We suffer from PCI config space issue for a long time, which causes many
> system can't correctly resume. Current Linux mechanism isn't sufficient.
> Here is a another idea:
> Record all PCI writes in Linux kernel, and redo all the write after
> resume in order. The idea assumes Firmware will restore all PCI config
> space to the boot time state, which is true at least for IA32.
>

I think a basic problem of current Linux device model is that there is no
effective message path from sibling devices to their root device.
Although the message direction from a root device to sibling devices is
natural from the viewpoint of device enumeration, the direction from
sibling devices to a root device is required for effective arbitration for
device configuration and power management.

The Windows driver model uses the direction from sibling drivers to a root
bus driver mainly, i.e. sibling drivers are layered on a root bus driver.
While we need a kind of callback mechanism from PCI (sibling) devices to
PCI bus (root) device instead because their normal call interface is from a
root device to sibling devices.

- Hiro.

Re: [ACPI] [Proposal]Another way to save/restore PCI config space for suspend/resume

[Li Shaohua]

On Wed, 2004-10-27 at 09:32, Hiroshi 2 Itoh wrote:
> 
> 
> Hi,
> 
> acpi-devel-admin@lists.sourceforge.net wrote on 2004/10/26 13:50:57:
> 
> > Hi,
> > We suffer from PCI config space issue for a long time, which causes many
> > system can't correctly resume. Current Linux mechanism isn't sufficient.
> > Here is a another idea:
> > Record all PCI writes in Linux kernel, and redo all the write after
> > resume in order. The idea assumes Firmware will restore all PCI config
> > space to the boot time state, which is true at least for IA32.
> >
> 
> I think a basic problem of current Linux device model is that there is no
> effective message path from sibling devices to their root device.
> Although the message direction from a root device to sibling devices is
> natural from the viewpoint of device enumeration, the direction from
> sibling devices to a root device is required for effective arbitration for
> device configuration and power management.
> 
> The Windows driver model uses the direction from sibling drivers to a root
> bus driver mainly, i.e. sibling drivers are layered on a root bus driver.
> While we need a kind of callback mechanism from PCI (sibling) devices to
> PCI bus (root) device instead because their normal call interface is from a
> root device to sibling devices.
Hiro-san,
I don't really understand why this is related with suspend/resume. Could
you please explain it more clearly?

Thanks,
Shaohua

Re: [ACPI] [Proposal]Another way to save/restore PCI config space for suspend/resume

[Hiroshi 2 Itoh]

> >
> > I think a basic problem of current Linux device model is that there is
no
> > effective message path from sibling devices to their root device.
> > Although the message direction from a root device to sibling devices is
> > natural from the viewpoint of device enumeration, the direction from
> > sibling devices to a root device is required for effective arbitration
for
> > device configuration and power management.
> >
> > The Windows driver model uses the direction from sibling drivers to a
root
> > bus driver mainly, i.e. sibling drivers are layered on a root bus
driver.
> > While we need a kind of callback mechanism from PCI (sibling) devices
to
> > PCI bus (root) device instead because their normal call interface is
from a
> > root device to sibling devices.
> Hiro-san,
> I don't really understand why this is related with suspend/resume. Could
> you please explain it more clearly?
>
Hi,

What I mean is that:

There are some bridge devices to be supported by a driver for various
device types and vendors. In the long run PCI drivers will have power
dependency one another in the long run. It is natual that PCI bridge driver
reports its child devices and their dependency to PCI core because PCI core
driver needs to know power up/down sequence at suspend/resume time. So I
think callbacks from bridge-to-core is useful. Especially it is more useful
for the core to get exact timing to power up the next driver if some
devices have certain latency to power up.

- Hiro.

Re: [Proposal]Another way to save/restore PCI config space for suspend/resume

[Rajesh Shah]

On Tue, Oct 26, 2004 at 12:50:57PM +0800, Li Shaohua wrote:
> Hi,
> We suffer from PCI config space issue for a long time, which causes many
> system can't correctly resume. Current Linux mechanism isn't sufficient.
> Here is a another idea: 
> Record all PCI writes in Linux kernel, and redo all the write after
> resume in order. The idea assumes Firmware will restore all PCI config
> space to the boot time state, which is true at least for IA32.
> 
A large percentage of them may be irrelevant with respect to 
suspend/resume (e.g. pci device tree and resource scan...).  Apart 
from the performance problems, generic code doing device specific 
config accesses may have correctness problems. For example, you 
will not be able to capture/replay config cycles or other device 
specific initialization (e.g. using memory mapped IO) that BIOS may 
have done before boot. This may be required for correct access to
device specific areas. The same thing is true about device specific 
config accesses that may have been done by the corresponding 
driver after boot. Without device specific knowledge, we may see 
unpredictable behavior and non-repeatable and hard to debug problems.

I don't see how generic code can do the right thing for device
specific accesses. Devices like p2p bridges that have standard
definitions can be handled separately.

Rajesh