Re: Real-life pci errors (Was: Re: PCI Error Recovery API Proposal. (WAS:: [PATCH/RFC]PCIErrorRecovery)

Benjamin Herrenschmidt
2005-03-19T01:25:49Z

On Fri, 2005-03-18 at 18:35 -0600, Linas Vepstas wrote:
> On Sat, Mar 19, 2005 at 10:13:02AM +1100, Benjamin Herrenschmidt was heard to remark:
> >
> > Additionally, in "real life", very few errors are cause by known errata.
> > If the drivers know about the errata, they usually already work around
> > them. Afaik, most of the errors are caused by transcient conditions on
> > the bus or the device, like a bit beeing flipped, or thermal
> > conditions...
>
>
> Heh. Let me describe "real life" a bit more accurately.
>
> We've been running with pci error detection enabled here for the last
> two years. Based on this experience, the ballpark figures are:
>
> 90% of all detected errors were device driver bugs coupled to
> pci card hardware errata

Well, this have been in-lab testing to fight driver bugs/errata on early
rlease kernels, I'm talking about the context of a released solution
with stable drivers/hw.

> 9% poorly seated pci cards (remove/reseat will make problem go away)
>
> 1% transient/other.

Ok.

> We've seen *EVERY* and I mean *EVERY* device driver that we've put
> under stress tests (e.g. peak i/o rates for > 72 hours, e.g.
> massive tcp/nfs traffic, massive disk i/o traffic, etc), *EVERY*
> driver tripped on an EEH error detect that was traced back to
> a device driver bug. Not to blame the drivers, a lot of these
> were related to pci card hardware/foirmware bugs. For example,
> I think grepping for "split completion" and "NAPI" in the
> patches/errata for e100 and e1000 for the last year will reveal
> some of the stuff that was found. As far as I know,
> for every bug found, a patch made it into mainline.

Yah, those are a pain. But then, it isn't the context described by
Nguyen where the driver "knows" about the errata and how to recover.
It's the context of a bug where the driver does not know what's going on
and/or doesn't have the proper workaround. My point was more that there
are very few cases where a driver will have to do recovery of PCI error
in known cases where it actually expect an error to happen.

> As a rule, it seems that finding these device driver bugs was
> very hard; we had some people work on these for months, and in
> the case of the e1000, we managed to get Intel engineers to fly
> out here and stare at PCI bus traces for a few days. (Thanks Intel!)
> Ditto for Emulex. For ipr, we had inhouse people.
>
> So overall, PCI error detection did have the expected effect
> (protecting the kernel from corruption, e.g. due to DMA's going
> to wild addresses), but I don't think anybody expected that the
> vast majority would be software/hardware bugs, instead of transient
> effects.
>
> What's ironic in all of this is that by adding error recovery,
> device driver bugs will be able to hide more effectively ...
> if there's a pci bus error due to a driver bug, the pci card
> will get rebooted, the kernel will burp for 3 seconds, and
> things will keep going, and most sysadmins won't notice or
> won't care.

Yes, but it will be logged at least, so we'll spot a lot of these during
our tests.

Ben.