Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Marvin Justice]

The error :

Curious - OSB4 thinks the DMA is still running.
OSB4 wait exit

appears to occur during the "rewrite" phase of Bonnie++ every time we run it.
The setup is a Tyan 2515 with a Seagate ST310211A drive. Bonnie ran for ~48 
hours on 2.4.10 without slowdown (I assume this is because the extra check in 
-ac isn't present) but there was some file system corruption after the 1st 24 
hours:

EXT2-fs error (device ide0(3,6)): ext2_free_blocks: Freeing blocks not in 
datazone - block = 2138996092, count = 1

Here's the lspci:

00:00.0 Host bridge: ServerWorks CNB20LE (rev 06)
	Flags: bus master, medium devsel, latency 32

00:00.1 Host bridge: ServerWorks CNB20LE (rev 06)
	Flags: bus master, medium devsel, latency 16

[snip]

00:0f.0 ISA bridge: ServerWorks OSB4 (rev 50)
	Subsystem: ServerWorks OSB4
	Flags: bus master, medium devsel, latency 0

00:0f.1 IDE interface: ServerWorks: Unknown device 0211 (prog-if 8a [Master 
SecP PriP])
	Flags: bus master, medium devsel, latency 64
	I/O ports at ffa0 [size=16]


-- 
Marvin Justice
Software Developer
BOXX Technologies, Inc.
www.boxxtech.com
512-235-6318 (V)
412-835-0434 (F)

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Alan Cox]

> Curious - OSB4 thinks the DMA is still running.
> OSB4 wait exit

This is an OSB4 trap for a problem I've seen with seagate drivers on some
boxes. Under very high load with UDMA seagate drives the OSB4 returns to
us with the chip reporting DMA in progress. The next I/O after this will
cause disk corruption as it DMA's the last 4 bytes of the previous write,
then the data shifted up by 4 bytes until the end of that I/O. I need
to confirm nobody else is seeing this with other working workloads, then
I'll swap it for a panic - better to die than kill the disc contents.

Serverworks have been looking at the problem but have yet to duplicate it.
As far as we can tell (and Red Hat have been working with a customer on this
directly) your choices are

1.	Use multiword DMA not UDMA
2.	Use non seagate disks with that controller

I am hopeful that serverworks will figure out what is up, but not every box
sees it - and indeed they've yet to be able to reproduce it.


Alan

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Marvin Justice]

Alright. Until the situation clarifies you might tell RH turn off OSB4 by 
default in future releases. It caused us no end of grief with 2.4.3-12.

-- 
Marvin Justice
Software Developer
BOXX Technologies, Inc.
www.boxxtech.com
512-235-6318 (V)
512-835-0434 (F)


On Sunday 30 September 2001 03:50 pm, Alan Cox wrote:
> > Curious - OSB4 thinks the DMA is still running.
> > OSB4 wait exit
>
> This is an OSB4 trap for a problem I've seen with seagate drivers on some
> boxes. Under very high load with UDMA seagate drives the OSB4 returns to
> us with the chip reporting DMA in progress. The next I/O after this will
> cause disk corruption as it DMA's the last 4 bytes of the previous write,
> then the data shifted up by 4 bytes until the end of that I/O. I need
> to confirm nobody else is seeing this with other working workloads, then
> I'll swap it for a panic - better to die than kill the disc contents.
>
> Serverworks have been looking at the problem but have yet to duplicate it.
> As far as we can tell (and Red Hat have been working with a customer on
> this directly) your choices are
>
> 1.	Use multiword DMA not UDMA
> 2.	Use non seagate disks with that controller
>
> I am hopeful that serverworks will figure out what is up, but not every box
> sees it - and indeed they've yet to be able to reproduce it.
>
>
> Alan

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Alan Cox]

> Alright. Until the situation clarifies you might tell RH turn off OSB4 by 
> default in future releases. It caused us no end of grief with 2.4.3-12.

Blacklisting seagate drives on the OSB4 is already on the todo list - its
taken a long time to figure out what was going on - it was only because
someone had > 100 boxes and knew they all did this that we got anywhere

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Steven Timm]

As mentioned earlier on this list, we saw similar corruption,
detectable by the fact that if you do a ls -R on the file system
there will be input/output errors due to inodes that are no longer
there, eventually.
We began with a system with a Seagate system drive (master, ide0), same
model as in Marvin's post..   IDE cd rom
as slave on the ide0 bus, and two IBM data drives on the ide1 bus.

The vendor eventually swapped out all the Seagate drives for
Western Digital ones, which does not make the problem go away,
but makes it much less frequent and harder to reproduce, but
at least there is something in /var/log/messages to show for it
whenever it does happen.  It is wrong to think this problem happens
only with Seagate drives.

Steve

------------------------------------------------------------------
Steven C. Timm (630) 840-8525  timm@fnal.gov  http://home.fnal.gov/~timm/
Fermilab Computing Division/Operating Systems Support
Scientific Computing Support Group--Computing Farms Operations

On Sun, 30 Sep 2001, Alan Cox wrote:
>
> 1.	Use multiword DMA not UDMA
> 2.	Use non seagate disks with that controller
>
> I am hopeful that serverworks will figure out what is up, but not every box
> sees it - and indeed they've yet to be able to reproduce it.
>
>
> Alan
>

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Mark Hahn]

> We began with a system with a Seagate system drive (master, ide0), same
> model as in Marvin's post..   IDE cd rom
> as slave on the ide0 bus, and two IBM data drives on the ide1 bus.
...
> whenever it does happen.  It is wrong to think this problem happens
> only with Seagate drives.

indeed, it seems that the most common trigger for this problem
is simply putting two devices from different vendors on the same channel.
Seagate drives are fairly notorious for not getting along with others.

it's best to think of IDE as a point-to-point link.

regards, mark hahn.

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Steven Timm]

------------------------------------------------------------------
Steven C. Timm (630) 840-8525  timm@fnal.gov  http://home.fnal.gov/~timm/
Fermilab Computing Division/Operating Systems Support
Scientific Computing Support Group--Computing Farms Operations

On Mon, 1 Oct 2001, Mark Hahn wrote:

> > We began with a system with a Seagate system drive (master, ide0), same
> > model as in Marvin's post..   IDE cd rom
> > as slave on the ide0 bus, and two IBM data drives on the ide1 bus.
> ...
> > whenever it does happen.  It is wrong to think this problem happens
> > only with Seagate drives.
>
> indeed, it seems that the most common trigger for this problem
> is simply putting two devices from different vendors on the same channel.
> Seagate drives are fairly notorious for not getting along with others.
>
> it's best to think of IDE as a point-to-point link.

True... however, we were still able to reproduce these problems
with only one drive on the bus.

Steve



>
> regards, mark hahn.
>
>

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Steven Timm]

On Sun, 30 Sep 2001, Marvin Justice wrote:

> The error :
>
> Curious - OSB4 thinks the DMA is still running.
> OSB4 wait exit
>
> appears to occur during the "rewrite" phase of Bonnie++ every time we run it.
> The setup is a Tyan 2515 with a Seagate ST310211A drive. Bonnie ran for ~48
> hours on 2.4.10 without slowdown (I assume this is because the extra check in
> -ac isn't present) but there was some file system corruption after the 1st 24
> hours:
>
> EXT2-fs error (device ide0(3,6)): ext2_free_blocks: Freeing blocks not in
> datazone - block = 2138996092, count = 1
>
> Here's the lspci:
>
> 00:00.0 Host bridge: ServerWorks CNB20LE (rev 06)
> 	Flags: bus master, medium devsel, latency 32
>
> 00:00.1 Host bridge: ServerWorks CNB20LE (rev 06)
> 	Flags: bus master, medium devsel, latency 16
>
> [snip]
>
> 00:0f.0 ISA bridge: ServerWorks OSB4 (rev 50)
> 	Subsystem: ServerWorks OSB4
> 	Flags: bus master, medium devsel, latency 0
>
> 00:0f.1 IDE interface: ServerWorks: Unknown device 0211 (prog-if 8a [Master
> SecP PriP])
> 	Flags: bus master, medium devsel, latency 64
> 	I/O ports at ffa0 [size=16]
>
>
> --
> Marvin Justice
> Software Developer
> BOXX Technologies, Inc.
> www.boxxtech.com
> 512-235-6318 (V)
> 412-835-0434 (F)
>

re: DMA problem (?) w/2.4.6-xfs and Serverworks OSB4 Chipset

[Steven Timm]

I had previously reported DMA problems with ultraDMA using
Serverworks OSB4 (LE) chipset talking to IDE drives including
Western Digital and Seagate.

I am now running 2.4.9-0.18 kernel as found at Redhat's Rawhide
which includes Alan's latest diagnostic patches, and am getting
the message like this:

end_request: I/O error, dev 16:01 (hdc), sector 2487128
Curious - OSB4 thinks the DMA is still running.
OSB4 wait exit.
hdc: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hdc: dma_intr: error=0x40 { UncorrectableError }, LBAsect=2487198,
sector=2487128

------------------------

with the "Curious..." patch being the new diagnostics added by Alan.
All the various failure modes which we see include this error
message now.

What does this imply...is there any hope for a fix?

Steve Timm


------------------------------------------------------------------
Steven C. Timm (630) 840-8525  timm@fnal.gov  http://home.fnal.gov/~timm/
Fermilab Computing Division/Operating Systems Support
Scientific Computing Support Group--Computing Farms Operations

Re: DMA problem (?) w/2.4.6-xfs and Serverworks OSB4 Chipset

[Alan Cox]

> I am now running 2.4.9-0.18 kernel as found at Redhat's Rawhide
> which includes Alan's latest diagnostic patches, and am getting
> the message like this:
> 
> end_request: I/O error, dev 16:01 (hdc), sector 2487128
> Curious - OSB4 thinks the DMA is still running.
> OSB4 wait exit.
> hdc: dma_intr: status=0x51 { DriveReady SeekComplete Error }
> hdc: dma_intr: error=0x40 { UncorrectableError }, LBAsect=2487198,
> sector=2487128
> 
> with the "Curious..." patch being the new diagnostics added by Alan.
> All the various failure modes which we see include this error
> message now.
> 
> What does this imply...is there any hope for a fix?

At the moment nope. Probably we should simply default to MWDMA not UDMA
on the OSB4.

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Steven Timm]

The problem, as described earlier in this thread by Adam McKenna,
describes errors such as

hda: timeout waiting for DMA
hda: irq timeout: status 0x58 { DriveReady SeekComplete DataRequest}

I have been seeing these errors on a number of systems, although
only intermittently, and they have been similar in nature
on all kernels between 2.2.16 and 2.4.7.

A couple months ago it was reported that someone was testing a
2.4.7-ac2 patch level and not having any lock-ups with ultraDMA.
My question--there was a new file "serverworks.c" inserted in the
2.4.6 ac patches.  Does anyone know if that made it into the kernel
and supposedly fixed the problem?  (My guess is that it has *not*
made it in, although there is a drivers/ide/serverworks.c file
in my 2.4.7-2.9 version of the kernel, it is not the same length
and there are some diffs).  If it didn't make it in,
is there any other patch that is known to work to fix this problem
and make ultradma work properly?  (and if I should be able to RTFM
and figure this out for myself, someone please give me a pointer.)


Some have reported
that using multi-word DMA (hdparm -X34) works but this is
not fast enough for us.

We have seen quite a difference on systems that are otherwise
the same (Supermicro 370DLE w/serverworks OSB4 LE chipset) by swapping
different models of hard disk drives.  With some types of drive
(Seagate) we
observe massive corruption of the file system but nothing reported
in /var/log/messages or on the console.  Currently we see hda
timeouts (but only on about 10 systems over the course of 2 months)
which hang the machine but after a reboot things are fine (Western
Digital).

There have been some rumours on other lists that this bug is tied
to simultaneous access of both IDE busses.  Any truth to that?

If anyone can provide help in tracking the existence of the patch,
or has successfully got UltraDMA working on one of these systems by
any means necessary, please let me know.

Thanks

Steve Timm


------------------------------------------------------------------
Steven C. Timm (630) 840-8525  timm@fnal.gov  http://home.fnal.gov/~timm/
Fermilab Computing Division/Operating Systems Support
Scientific Computing Support Group--Computing Farms Operations

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Alan Cox]

> 2.4.7-ac2 patch level and not having any lock-ups with ultraDMA.
> My question--there was a new file "serverworks.c" inserted in the
> 2.4.6 ac patches.  Does anyone know if that made it into the kernel
> and supposedly fixed the problem?  (My guess is that it has *not*

Short answer is "it should"

Long answer "I have been chasing a specific problem with OSB4, seagate
drives and UDMA corruption. We can reliably reproduce it and see it on one
set of machines. Serverworks cannot reproduce it elsewhere"

So unless your box when running current -ac starts spewing messages about
DMA completions seeming broken - it should work. I only mention this
because you write:

> We have seen quite a difference on systems that are otherwise
> the same (Supermicro 370DLE w/serverworks OSB4 LE chipset) by swapping
> different models of hard disk drives.  With some types of drive
> (Seagate) we
> observe massive corruption of the file system but nothing reported
> in /var/log/messages or on the console.  Currently we see hda
> timeouts (but only on about 10 systems over the course of 2 months)
> which hang the machine but after a reboot things are fine (Western
> Digital).

I would thus be very interested if the current -ac "hardware just did
something impossibly stupid" trap is hit.

Alan

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Steven Timm]

Thanks, Alan for your reply to this,...just a couple clarifications
needed...


On Thu, 27 Sep 2001, Alan Cox wrote:

> > 2.4.7-ac2 patch level and not having any lock-ups with ultraDMA.
> > My question--there was a new file "serverworks.c" inserted in the
> > 2.4.6 ac patches.  Does anyone know if that made it into the kernel
> > and supposedly fixed the problem?  (My guess is that it has *not*
>
> Short answer is "it should"
>
OK--"it should" fix the bug--my question is, how far has the
serverworks.c which was in the 2.4.6-ac series now propagated
into Linus' tree (and eventually into the RedHat tree?)  Also,
are the later 2.4.7 and higher -ac patches significantly
different from that which was found in the 2.4.6-ac patches?

> Long answer "I have been chasing a specific problem with OSB4, seagate
> drives and UDMA corruption. We can reliably reproduce it and see it on one
> set of machines. Serverworks cannot reproduce it elsewhere"
>
Too bad we didn't know this a month ago... we had 136 of these machines
and were seeing it all the time, could have given you a perfect test
bed...we found the quickest way
to reproduce it was just to do ls -R /usr | grep Input/output error
and managed to make 80 of the 136 machines misbehave within a couple
of days time.  As a result of that the vendor swapped all the drives
to western digital and left us with a bug that has only happened on
10 machines over the course of 2 months--where the DMA timeouts
will hang the machine but not corrupt the data.

> So unless your box when running current -ac starts spewing messages about
> DMA completions seeming broken - it should work. I only mention this
> because you write:
>
> > We have seen quite a difference on systems that are otherwise
> > the same (Supermicro 370DLE w/serverworks OSB4 LE chipset) by swapping
> > different models of hard disk drives.  With some types of drive
> > (Seagate) we
> > observe massive corruption of the file system but nothing reported
> > in /var/log/messages or on the console.  Currently we see hda
> > timeouts (but only on about 10 systems over the course of 2 months)
> > which hang the machine but after a reboot things are fine (Western
> > Digital).
>
> I would thus be very interested if the current -ac "hardware just did
> something impossibly stupid" trap is hit.
>
> Alan
>
Just tell me what you think your latest and greatest patch is that
you would want to see tested and I'll be glad to give it a test on this
cluster.

Thanks

Steve Timm

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Alan Cox]

> serverworks.c which was in the 2.4.6-ac series now propagated
> into Linus' tree (and eventually into the RedHat tree?)  Also,
> are the later 2.4.7 and higher -ac patches significantly
> different from that which was found in the 2.4.6-ac patches?

Most vendor are -ac based. The IDE changes in Linus tree match the 
Andre code with my fixes for the Dell cable detect. The -ac tree contains
an additional trap for the seagate problem we are chasing but is otherwise
the same

> Just tell me what you think your latest and greatest patch is that
> you would want to see tested and I'll be glad to give it a test on this
> cluster.

The extra trap in the -ac tree may be useful as it should trigger on the
seagate problem. If you can pull a box out of service I have a test set from
an RH customer with this problem that replicates the UDMA problem remarkably
reliably and rapidly for them (but not on serverworks own tests so its
not a simple problem). It would be useful to know if the seagate problem is
replicable and shows up the same way on your boxen

Info on the precise seagate drives, and pci data on the machines would be
very useful too so I can compare it to the other case and also pass it on
to serverworks.

Alan

Re: DMA problem (?) w/2.4.6-xfs and ServerWorks OSB4 Chipset

[Benjamin S. Scarlet]

On Thursday 27 September 2001 06:55 pm, Alan Cox wrote:
> Long answer "I have been chasing a specific problem with OSB4, seagate
> drives and UDMA corruption. We can reliably reproduce it and see it on one
> set of machines. Serverworks cannot reproduce it elsewhere"

> I would thus be very interested if the current -ac "hardware just did
> something impossibly stupid" trap is hit.

Are you otherwise interested in machines seeing this problem? I have a 
candidate. I'm running a SuperMicro 370DL3 with a Seagate ST320423A. For the 
sake of simplicity I turned off DMA and filled the whole drive full of 
zeroes. I see data corruption (non-zero data) on an otherwise inactive 
machine just reading the drive back with "od", if I have DMA on with some 
settings (udma2 in particular fails). Currently I'm using just kernel 2.4.9, 
but if it would be useful I could run tests with kernels of your choosing.

	Ben Scarlet