[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=14579





--- Comment #8 from lkolbe@techfak.uni-bielefeld.de  2010-04-13 09:25:22 ---
fyi: Supermicro's X7DWN+ mainboard needs a BIOS-Update (at least version 1.2b,
no changelog available though) to cope with multiple SAS-controllers under
linux. In the end, Kashyap Desai was right when he suspected it had something
to do with IRQ-routing.

Seagates' firmware problems (timeouts) with Adaptecs RAID-Controller didn't
help with this either, as every few days more than two disks at a time were
thrown out of the array resulting in loss of said array. Supposedly, the newest
firmware 'DN06' for the Barracuda.ES2 drives fix these problems, but our
distributor was so kind to replace all our Seagate drives with Hitachis.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=14579


Alan <alan@lxorguk.ukuu.org.uk> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID




-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=14579


Alan <alan@lxorguk.ukuu.org.uk> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED




-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] New: Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=14579

           Summary: Devices disappear; on bus reset machine hangs; on I/O
                    machine hangs
           Product: IO/Storage
           Version: 2.5
    Kernel Version: Linux version 2.6.30-1-amd64 (Debian
                    2.6.30-5+techfaklenny1)
                    (sfrey@TechFak.Uni-Bielefeld.DE) (gcc version 4.3.2
                    (Debian 4.3.2-1.1) ) #1 SMP Sat Aug 15 11:20:08 UTC
                    2009
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: blocking
          Priority: P1
         Component: SCSI
        AssignedTo: linux-scsi@vger.kernel.org
        ReportedBy: sfrey@techfak.uni-bielefeld.de
        Regression: No


Created an attachment (id=23729)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=23729)
lspci -vvv output

On our soon-to-be primary backup machine with an Adaptec 52445 (for the disk
pool) and an LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (for
the Tape Library), we see random failures:
On Tape-IO (using btape from bacula), a few minutes into the writes, the
changer device and both tape devices disappear from the system. Rescanning the
bus sometimes helps, but sometimes results in a hardy freeze.

On Tape-IO, a few minutes into the writes, the system freezes.

lspci -vvv, lsscsi and dmesg and /proc/modules are attached, this might also be
related to #14577 as it's the same machine. Attached is also a screenshot of
the oops we saw on the last crash that happened durign a btape run.

The system uses stock Debian/Lenny with a custom built 2.6.30.5 kernel.

-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=14579





--- Comment #1 from Sascha Frey <sfrey@techfak.uni-bielefeld.de>  2009-11-10 15:29:20 ---
Created an attachment (id=23730)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=23730)
cat /proc/modules output

-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=14579





--- Comment #2 from Sascha Frey <sfrey@techfak.uni-bielefeld.de>  2009-11-10 15:30:02 ---
Created an attachment (id=23731)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=23731)
lsscsi output

-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=14579





--- Comment #3 from Sascha Frey <sfrey@techfak.uni-bielefeld.de>  2009-11-10 15:30:46 ---
Created an attachment (id=23732)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=23732)
screenshot of last oops

-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=14579





--- Comment #4 from Sascha Frey <sfrey@techfak.uni-bielefeld.de>  2009-11-10 15:35:41 ---
Created an attachment (id=23733)
 --> (http://bugzilla.kernel.org/attachment.cgi?id=23733)
dmesg output

-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=14579


lkolbe@techfak.uni-bielefeld.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Depends on|                            |14577




-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=14579


lkolbe@techfak.uni-bielefeld.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |lkolbe@techfak.uni-bielefel
                   |                            |d.de




--- Comment #5 from lkolbe@techfak.uni-bielefeld.de  2009-11-18 14:05:34 ---
fyi

From: "Desai, Kashyap" <Kashyap.Desai@lsi.com>
To: "Support@techfak.uni-bielefeld.de" <Support@techfak.uni-bielefeld.de>,
        "linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>
CC: Lukas Kolbe <lkolbe@techfak.uni-bielefeld.de>
Date: Fri, 13 Nov 2009 17:29:43 +0530
Subject: RE: Bug 14579 -  Devices disappear... and Bug 14577 - Data
    corruption with Adaptec
Message-ID: <0D1E8821739E724A86F4D16902CE275C1C93C04462@inbmail01.lsi.com>
References: <20091111160220.GC5705@TechFak.Uni-Bielefeld.DE>
 <20091112225825.GA20808@TechFak.Uni-Bielefeld.DE>
In-Reply-To: <20091112225825.GA20808@TechFak.Uni-Bielefeld.DE>

Subject line is related to *Adaptec* and there are some places LSI related =
issue is pointed out. Little confusing to me. Is it possible to rewrite wha=
t is an issue related to LSI card?

>From dmesg log I can figure out 3.04.07 is mpt fusion driver version.
Please update LSI driver using latest upstream driver version 3.04.13. And =
see what a result is.

- Kashyap

-----Original Message-----
From: linux-scsi-owner@vger.kernel.org [mailto:linux-scsi-owner@vger.kernel=
.org] On Behalf Of Sascha Frey
Sent: Friday, November 13, 2009 4:28 AM
To: linux-scsi@vger.kernel.org
Cc: Lukas Kolbe
Subject: Re: Bug 14579 - Devices disappear... and Bug 14577 - Data corrupti=
on with Adaptec

Hi,

Lukas Kolbe wrote:
>we'd really appreciate any hints and help we can get for the following
>bugs:
>http://bugzilla.kernel.org/show_bug.cgi?id=3D14579

We've done some further testing:
it's very hard to trigger this bug. Sometimes the machine freezes after
a few minutes into tape access and sometimes it works days - or even
weeks - without any problem.

The bug only appears during tape I/O (regardless of which tape program is
used: btape, dd or tar).
In most cases the tape write ends with an input/output error. After this
error occurred, any access to the tape library robot (connected through
the SAS interface of the first drive) fails:

# mtx unload 1 1
Unloading drive 1 into Storage Element 1...mtx: Request Sense: Long Report=
=3Dyes
mtx: Request Sense: Valid Residual=3Dno
mtx: Request Sense: Error Code=3D70 (Current)
mtx: Request Sense: Sense Key=3DIllegal Request
mtx: Request Sense: FileMark=3Dno
mtx: Request Sense: EOM=3Dno
mtx: Request Sense: ILI=3Dno
mtx: Request Sense: Additional Sense Code =3D 53
mtx: Request Sense: Additional Sense Qualifier =3D 01
mtx: Request Sense: BPV=3Dno
mtx: Request Sense: Error in CDB=3Dno
mtx: Request Sense: SKSV=3Dno
MOVE MEDIUM from Element Address 257 to 4096 Failed

After resetting the scsi bus (echo "- - -" >
/sys/class/scsi_host/host5/scan) the tape drives are revitalized, but
the changer device disappears. Even after a cold restart of the whole
library the device keeps missing.

Yet another problem: restting the SCSI bus of the LSI SAS HBA sometimes
results in a hardy freeze (console stuck; no log messages).

> [...]
>
>I do not believe it's a hardware fault at the moment as the machine
>ran OK under Solaris for a few weeks (including successful btape runs).
>

The very same piece of hardware worked fine using Solaris 10 with heavy
disk and tape I/O at the same time for two months.

We really prefer using Linux instead, but we're in pressure of time.


We appreciate any help resolving this bug!




Regards,
Sascha Frey


From: "Desai, Kashyap" <Kashyap.Desai@lsi.com>
To: "support@TechFak.Uni-Bielefeld.DE" <support@TechFak.Uni-Bielefeld.DE>
CC: "linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>
Date: Wed, 18 Nov 2009 10:24:38 +0530
Subject: RE: Bug 14579 -  Devices disappear... and Bug 14577 - Data
    corruption with Adaptec
Message-ID: <0D1E8821739E724A86F4D16902CE275C1C93C74A49@inbmail01.lsi.com>
References: <20091111160220.GC5705@TechFak.Uni-Bielefeld.DE>
 <20091112225825.GA20808@TechFak.Uni-Bielefeld.DE>
 <0D1E8821739E724A86F4D16902CE275C1C93C04462@inbmail01.lsi.com>
 <20091117142242.GA15638@TechFak.Uni-Bielefeld.DE>
In-Reply-To: <20091117142242.GA15638@TechFak.Uni-Bielefeld.DE>

Hello Lukas,


> -----Original Message-----
> From: Lukas Kolbe [mailto:lkolbe@TechFak.Uni-Bielefeld.DE]
> Sent: Tuesday, November 17, 2009 7:53 PM
> To: Desai, Kashyap
> Cc: linux-scsi@vger.kernel.org
> Subject: Re: Bug 14579 - Devices disappear... and Bug 14577 - Data
> corruption with Adaptec
>=20
> Desai, Kashyap wrote:
>=20
> >Subject line is related to *Adaptec* and there are some places LSI
> >related issue is pointed out. Little confusing to me. Is it possible to
> >rewrite what is an issue related to LSI card?
>=20
> Sorry for that one. This system has an Adaptec Controller for its
> Storage array and an LSI controller for the tape library. Bug 14577 is
> about a possible data corruption on 2.6.32-rc6 that seems to be either a
> hardware error (currently trying to find that out) or a regression in
> 2.6.32-rc6, as 2.6.30 is very happy with its storage.
OK. In data corruption condition only LSI driver and controller are involve=
d? I mean can I nullify Adaptec controller's roll in your test?
>=20
> Finally, the real problem here is Bug 14579 that is about the systems
> problems when using the tape library.
>=20
> >From dmesg log I can figure out 3.04.07 is mpt fusion driver version.
> >Please update LSI driver using latest upstream driver version 3.04.13.
> And see what a result is.
>=20
> Thanks for the pointer. Linus' current tree contains 3.04.12 - where can
> I find 3.04.13?

It is there in 2.6.32-rc5. Not sure in which exact rc version it is include=
d, but I have 2.6.32-rc5 tree in my setup and for that kernel mptfusion ver=
sion is 3.104.13
>=20
> >- Kashyap
>=20
> Kind regards,
> Lukas Kolbe

-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=14579





--- Comment #6 from lkolbe@techfak.uni-bielefeld.de  2009-11-18 14:07:10 ---
fyi


>From lkolbe@TechFak.Uni-Bielefeld.DE Wed Nov 18 14:39:08 2009
Date: Wed, 18 Nov 2009 14:39:09 +0100
From: Lukas Kolbe <lkolbe@TechFak.Uni-Bielefeld.DE>
To: "Desai, Kashyap" <Kashyap.Desai@lsi.com>
Cc: "linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>
Subject: Re: Bug 14579 -  Devices disappear... and Bug 14577 - Data
    corruption with Adaptec
Message-ID: <20091118133909.GD16440@TechFak.Uni-Bielefeld.DE>
References: <20091111160220.GC5705@TechFak.Uni-Bielefeld.DE>
<20091112225825.GA20808@TechFak.Uni-Bielefeld.DE>
<0D1E8821739E724A86F4D16902CE275C1C93C04462@inbmail01.lsi.com>
<20091117142242.GA15638@TechFak.Uni-Bielefeld.DE>
<0D1E8821739E724A86F4D16902CE275C1C93C74A49@inbmail01.lsi.com>
In-Reply-To: <0D1E8821739E724A86F4D16902CE275C1C93C74A49@inbmail01.lsi.com>

Desai, Kashyap wrote:

>> >Subject line is related to *Adaptec* and there are some places LSI
>> >related issue is pointed out. Little confusing to me. Is it possible to
>> >rewrite what is an issue related to LSI card?
>> 
>> Sorry for that one. This system has an Adaptec Controller for its
>> Storage array and an LSI controller for the tape library. Bug 14577 is
>> about a possible data corruption on 2.6.32-rc6 that seems to be either a
>> hardware error (currently trying to find that out) or a regression in
>> 2.6.32-rc6, as 2.6.30 is very happy with its storage.
>OK. In data corruption condition only LSI driver and controller are
>involved? I mean can I nullify Adaptec controller's roll in your test?

No, it is the other way round. We have 24 1TB Seagate harddisks
connected in a RAID 60 to the adaptec controller, and a Tandberg T80
with two IBM Ultrium-HH4 tape drives connected to the LSI controller.

The system is installed on an LVM volume within the RAID 60.
The data corruption occurs when we try to boot 2.6.32-rc6, we get write
errors and the boot process stops somewhere. So, it seems the data
corruption is related _only_ with the Adaptec Controller, the RAID array
or the harddisks.

>> Finally, the real problem here is Bug 14579 that is about the systems
>> problems when using the tape library.
>> 
>> >From dmesg log I can figure out 3.04.07 is mpt fusion driver version.
>> >Please update LSI driver using latest upstream driver version 3.04.13.
>> And see what a result is.
>> 
>> Thanks for the pointer. Linus' current tree contains 3.04.12 - where can
>> I find 3.04.13?
>
>It is there in 2.6.32-rc5. Not sure in which exact rc version it is
>included, but I have 2.6.32-rc5 tree in my setup and for that kernel
>mptfusion version is 3.104.13

Okay, I grep'ed for 3.04 in the source and only got one reference to the
older version number. But there lies the problem: Unless we can fix the
Adaptec-Bug first (or confirm it is a hardware issue), we can't boot
2.6.32-rc on that machine to test the new LSI driver version. Is it
easily possible to backport/include the mptfusion in 2.6.30?

Thanks for the help and kind regards, 
-- 
Lukas Kolbe

-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=14579


Bug 14579 depends on bug 14577, which changed state.

Bug 14577 Summary: Data Corruption with Adaptec 52445, Firmware 5.2-0 (17380)
http://bugzilla.kernel.org/show_bug.cgi?id=14577

           What    |Old Value                   |New Value
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |CODE_FIX



-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=14579


Alan <alan@lxorguk.ukuu.org.uk> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |alan@lxorguk.ukuu.org.uk
     Kernel Version|Linux version               |2.6.30-1-amd64 Debian
                   |2.6.30-1-amd64 (Debian      |
                   |2.6.30-5+techfaklenny1)     |
                   |(sfrey@TechFak.Uni-Bielefel |
                   |d.DE) (gcc version 4.3.2    |
                   |(Debian 4.3.2-1.1) ) #1 SMP |
                   |Sat Aug 15 11:20:08 UTC     |
                   |2009                        |




-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

[Bug 14579] Devices disappear; on bus reset machine hangs; on I/O machine hangs

[bugzilla-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=14579





--- Comment #7 from lkolbe@techfak.uni-bielefeld.de  2010-02-10 10:10:46 ---
As it turned out this machine was faulty (all 24 disks began dematerializing
under our feet) so we replaced it. We do now face similar issues, though I'll
open a new bug for that one to have a clear separation. Whoever has the
permission to close this bug might do so, please.

Kind regards, 
Lukas

-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.