3ware 9550 using 3w-9xxx driver lockups ?

3ware 9550 using 3w-9xxx driver lockups ?

[Andy Davidson]

Hi,

One of our servers locked up (no kernel panic, but no response to 
console/network) with the following output on console :

Aug  8 14:10:34 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING 
(0x04:0x0023): Sector repair completed:port=1,LBA=0x4383A7F.
Aug  8 14:10:39 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING 
(0x04:0x0023): Sector repair completed:port=1,LBA=0x438C4D9.
Aug  8 14:10:41 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING 
(0x04:0x0023): Sector repair completed:port=1,LBA=0x438C4DC.
Aug  8 14:10:47 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING 
(0x04:0x0023): Sector repair completed:port=1,LBA=0x438FB34.
Aug  8 14:10:49 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING 
(0x04:0x0023): Sector repair completed:port=1,LBA=0x438FB36.
Aug  8 14:10:52 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING 
(0x04:0x0023): Sector repair completed:port=1,LBA=0x439051D.
Aug  8 14:11:08 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING 
(0x04:0x0023): Sector repair completed:port=1,LBA=0x4378D8A.

I'd put this down to bad hardware until an identically spec'd machine 
(with fresh disks) did the same thing.


Anyone else noticed any issues using the newer 3-ware 9550S cards with 
the 3w-9xxx driver ?


Other details :
Linux how-mail-1 2.6.16-2-686-smp #1 SMP Sat Jul 15 22:33:00 UTC 2006 i686

from lspci:
03:03.0 RAID bus controller: 3ware Inc 9550SX SATA-RAID


3ware 9000 Storage Controller device driver for Linux v2.26.02.007.
ACPI: PCI Interrupt 0000:03:03.0[A] -> GSI 24 (level, low) -> IRQ 177
scsi0 : 3ware 9000 Storage Controller
3w-9xxx: scsi0: Found a 3ware 9000 Storage Controller at 0xda300000, 
IRQ: 177.
3w-9xxx: scsi0: Firmware FE9X 3.01.01.028, BIOS BE9X 3.01.00.024, Ports: 
4. Vendor: AMCC      Model: 9550SX-4LP DISK
Rev: 3.01 Type:   Direct-Access                      ANSI SCSI revision: 03f


We'd welcome any help ?

Thanks,
Andy

Re: 3ware 9550 using 3w-9xxx driver lockups ?

[adam radford]

Looks to me like you have a bad drive (or several bad drives, if
other drives on other machines are reallocating sectors as well).

Many reallocated sectors could be a sign that your drive is about to die...
Are you booting off this disk?

Have you run smartmontools on those bad disks?

You fail to mention your raid configuration.

Are you running the latest 3ware firmware for that controller?  There
is a Linux specific userspace firmware update utility.

Also, this topic is better served on the linux-scsi email list.

-Adam

On 8/8/06, Andy Davidson <andy@nosignal.org> wrote:
>
>
> Hi,
>
> One of our servers locked up (no kernel panic, but no response to
> console/network) with the following output on console :
>
> Aug  8 14:10:34 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
> (0x04:0x0023): Sector repair completed:port=1,LBA=0x4383A7F.
> Aug  8 14:10:39 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
> (0x04:0x0023): Sector repair completed:port=1,LBA=0x438C4D9.
> Aug  8 14:10:41 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
> (0x04:0x0023): Sector repair completed:port=1,LBA=0x438C4DC.
> Aug  8 14:10:47 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
> (0x04:0x0023): Sector repair completed:port=1,LBA=0x438FB34.
> Aug  8 14:10:49 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
> (0x04:0x0023): Sector repair completed:port=1,LBA=0x438FB36.
> Aug  8 14:10:52 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
> (0x04:0x0023): Sector repair completed:port=1,LBA=0x439051D.
> Aug  8 14:11:08 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
> (0x04:0x0023): Sector repair completed:port=1,LBA=0x4378D8A.
>
> I'd put this down to bad hardware until an identically spec'd machine
> (with fresh disks) did the same thing.
>
>
> Anyone else noticed any issues using the newer 3-ware 9550S cards with
> the 3w-9xxx driver ?
>
>
> Other details :
> Linux how-mail-1 2.6.16-2-686-smp #1 SMP Sat Jul 15 22:33:00 UTC 2006 i686
>
> from lspci:
> 03:03.0 RAID bus controller: 3ware Inc 9550SX SATA-RAID
>
>
> 3ware 9000 Storage Controller device driver for Linux v2.26.02.007.
> ACPI: PCI Interrupt 0000:03:03.0[A] -> GSI 24 (level, low) -> IRQ 177
> scsi0 : 3ware 9000 Storage Controller
> 3w-9xxx: scsi0: Found a 3ware 9000 Storage Controller at 0xda300000,
> IRQ: 177.
> 3w-9xxx: scsi0: Firmware FE9X 3.01.01.028, BIOS BE9X 3.01.00.024, Ports:
> 4. Vendor: AMCC      Model: 9550SX-4LP DISK
> Rev: 3.01 Type:   Direct-Access                      ANSI SCSI revision: 03f
>
>
> We'd welcome any help ?
>
> Thanks,
> Andy
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: 3ware 9550 using 3w-9xxx driver lockups ?

[Jeff V. Merkey]

adam radford wrote:

> Looks to me like you have a bad drive (or several bad drives, if
> other drives on other machines are reallocating sectors as well).
>
> Many reallocated sectors could be a sign that your drive is about to 
> die...
> Are you booting off this disk?
>
> Have you run smartmontools on those bad disks?
>
> You fail to mention your raid configuration.
>
> Are you running the latest 3ware firmware for that controller?  There
> is a Linux specific userspace firmware update utility.
>
> Also, this topic is better served on the linux-scsi email list.
>
> -Adam
>
> On 8/8/06, Andy Davidson <andy@nosignal.org> wrote:
>
>>
>>
>> Hi,
>>
>> One of our servers locked up (no kernel panic, but no response to
>> console/network) with the following output on console :
>>
>> Aug  8 14:10:34 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
>> (0x04:0x0023): Sector repair completed:port=1,LBA=0x4383A7F.
>> Aug  8 14:10:39 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
>> (0x04:0x0023): Sector repair completed:port=1,LBA=0x438C4D9.
>> Aug  8 14:10:41 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
>> (0x04:0x0023): Sector repair completed:port=1,LBA=0x438C4DC.
>> Aug  8 14:10:47 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
>> (0x04:0x0023): Sector repair completed:port=1,LBA=0x438FB34.
>> Aug  8 14:10:49 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
>> (0x04:0x0023): Sector repair completed:port=1,LBA=0x438FB36.
>> Aug  8 14:10:52 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
>> (0x04:0x0023): Sector repair completed:port=1,LBA=0x439051D.
>> Aug  8 14:11:08 how-mail-1 kernel: 3w-9xxx: scsi0: AEN: WARNING
>> (0x04:0x0023): Sector repair completed:port=1,LBA=0x4378D8A.
>>
>> I'd put this down to bad hardware until an identically spec'd machine
>> (with fresh disks) did the same thing.
>>
>>
>> Anyone else noticed any issues using the newer 3-ware 9550S cards with
>> the 3w-9xxx driver ?
>>
>>
>> Other details :
>> Linux how-mail-1 2.6.16-2-686-smp #1 SMP Sat Jul 15 22:33:00 UTC 2006 
>> i686
>>
>> from lspci:
>> 03:03.0 RAID bus controller: 3ware Inc 9550SX SATA-RAID
>>
>>
>> 3ware 9000 Storage Controller device driver for Linux v2.26.02.007.
>> ACPI: PCI Interrupt 0000:03:03.0[A] -> GSI 24 (level, low) -> IRQ 177
>> scsi0 : 3ware 9000 Storage Controller
>> 3w-9xxx: scsi0: Found a 3ware 9000 Storage Controller at 0xda300000,
>> IRQ: 177.
>> 3w-9xxx: scsi0: Firmware FE9X 3.01.01.028, BIOS BE9X 3.01.00.024, Ports:
>> 4. Vendor: AMCC      Model: 9550SX-4LP DISK
>> Rev: 3.01 Type:   Direct-Access                      ANSI SCSI 
>> revision: 03f
>>
>>
>> We'd welcome any help ?
>>
>> Thanks,
>> Andy
>> -
>> To unsubscribe from this list: send the line "unsubscribe 
>> linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>>
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
Adam,

Have him check the cables as well unless he using the multi-lane 
controller.  I see this a lot with loose ATA cables
on units, but since we moved over to the multi-lane architecture -- 
great work on the multi-lane version of the adapters, BTW.
He should also check is he's getting -2 bio errors.  I see -2 bio errors 
in parallel with this if its a bad cable.

Jeff

Re: 3ware 9550 using 3w-9xxx driver lockups ?

[Florian Weimer]

* Andy Davidson:

> Anyone else noticed any issues using the newer 3-ware 9550S cards with
> the 3w-9xxx driver ?

We've seen them as well, but tracked it down to a faulty drive.  You
should run smartctl against your drives and look for entries in the
error log.  (smartctl has speciall support for 3ware controllers, see
its manpage.)