sata_mv and Highpoint RocketRAID 230x, corruption?

sata_mv and Highpoint RocketRAID 230x, corruption?

[Mathias Burén]

Hi mailing list,

(please cc me as I'm not subscribed)

I'm currently migrating my system from pure LVM to HDD partitions >
mdadm RAID5 > lvm > filesystem(s). I got a bit worried when I saw this
in dmesg:

"sata_mv 0000:05:00.0: version 1.28
sata_mv 0000:05:00.0: PCI INT A -> Link[LN4A] -> GSI 19 (level, low) -> IRQ 19
sata_mv: Highpoint RocketRAID BIOS CORRUPTS DATA on all attached
drives, regardless of if/how they are configured. BEWARE!
sata_mv: For data safety, do not use sectors 8-9 on "Legacy" drives,
and avoid the final two gigabytes on all RocketRAID BIOS initialized
drives.
sata_mv 0000:05:00.0: Gen-IIE 32 slots 4 ports SCSI mode IRQ via INTx
sata_mv 0000:05:00.0: setting latency timer to 64
"

I'm currently not using the BIOS of the raid controller for anything
else then staggered disk spinup. The HDD partitions start at sector
2048 (to get a 1MB alignment since they're 4k sector drives, WD20EARS)
and end at the last sector.
What I'm worried about is the corruption mentioned in dmesg, is this
explained somewhere in more detail? Google didn't reveal much. Am I in
danger?

Thanks,

// Mathias on Archlinux

Re: sata_mv and Highpoint RocketRAID 230x, corruption?

[Mark Lord]

On 10-10-20 04:03 PM, Mathias Burén wrote:
..
> I'm currently not using the BIOS of the raid controller for anything
> else then staggered disk spinup. The HDD partitions start at sector
> 2048 (to get a 1MB alignment since they're 4k sector drives, WD20EARS)
> and end at the last sector.
> What I'm worried about is the corruption mentioned in dmesg, is this
> explained somewhere in more detail? Google didn't reveal much. Am I in
> danger?

Yes.  Just repartition the drives to avoid the final 2GB of each drive,
and then you'll be safe.

The RocketRaid BIOS that I examined here a couple of years ago,
liked to write "metadata" over top of whatever was in certain sectors
near the end of the drive.  EVEN FOR NON-RAID DRIVES.

I think it was the last even (power-of-two) multiple of 1GB or something,
so if you leave the final 2GB untouched, you're guaranteed to avoid it.

Thus the recommendation.

Cheers

RE: sata_mv and Highpoint RocketRAID 230x, corruption?

[Daniel Taylor]

 

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org 
> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Mark Lord
> Sent: Friday, October 22, 2010 7:00 PM
> To: Mathias Burén
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: sata_mv and Highpoint RocketRAID 230x, corruption?
> 
> On 10-10-20 04:03 PM, Mathias Burén wrote:
> ..
> > I'm currently not using the BIOS of the raid controller for anything
> > else then staggered disk spinup. The HDD partitions start at sector
> > 2048 (to get a 1MB alignment since they're 4k sector 
> drives, WD20EARS)
> > and end at the last sector.
> > What I'm worried about is the corruption mentioned in dmesg, is this
> > explained somewhere in more detail? Google didn't reveal 
> much. Am I in
> > danger?
> 
> Yes.  Just repartition the drives to avoid the final 2GB of 
> each drive,
> and then you'll be safe.
> 
> The RocketRaid BIOS that I examined here a couple of years ago,
> liked to write "metadata" over top of whatever was in certain sectors
> near the end of the drive.  EVEN FOR NON-RAID DRIVES.
> 
> I think it was the last even (power-of-two) multiple of 1GB 
> or something,
> so if you leave the final 2GB untouched, you're guaranteed to 
> avoid it.
> 
> Thus the recommendation.

What happens to the backup GPT at the end of the disk?

> 
> Cheers
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: sata_mv and Highpoint RocketRAID 230x, corruption?

[Mathias Burén]

Hi,

Thanks for the clarification. Since I've (stupidly enough) partitioned
the drives from 1MB until the end, I'm most likely affected by this
stupid RAID BIOS.
That might explain why I'm getting this error (full dmesg at
http://pastebin.ca/1970873 ):

ata2.00: status: { DRDY }
ata2: hard resetting link
ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata2.00: configured for UDMA/133
ata2.00: device reported invalid CHS sector 0
sd 1:0:0:0: [sdb]  Result: hostbyte=0x00 driverbyte=0x08
sd 1:0:0:0: [sdb]  Sense Key : 0xb [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
        72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
        00 00 00 00
sd 1:0:0:0: [sdb]  ASC=0x0 ASCQ=0x0
sd 1:0:0:0: [sdb] CDB: cdb[0]=0x28: 28 00 e7 70 c8 e8 00 05 40 00
end_request: I/O error, dev sdb, sector 3882928360
md/raid:md0: read error not correctable (sector 3882926312 on sdb1).
md/raid:md0: Disk failure on sdb1, disabling device.
<1>md/raid:md0: Operation continuing on 2 devices.
md/raid:md0: read error not correctable (sector 3882926320 on sdb1).
md/raid:md0: read error not correctable (sector 3882926328 on sdb1).
md/raid:md0: read error not correctable (sector 3882926336 on sdb1).
md/raid:md0: read error not correctable (sector 3882926344 on sdb1).
md/raid:md0: read error not correctable (sector 3882926352 on sdb1).
md/raid:md0: read error not correctable (sector 3882926360 on sdb1).
md/raid:md0: read error not correctable (sector 3882926368 on sdb1).
md/raid:md0: read error not correctable (sector 3882926376 on sdb1).
md/raid:md0: read error not correctable (sector 3882926384 on sdb1).

And:

[root@ion ~]# fdisk -l /dev/sdb

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x73ad1b41

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  3907029167  1953513560   fd  Linux raid autodetect
[root@ion ~]#

So the sectors are the last ones of the drive, the raid controllers
BIOS corrupted my data... my mdadm RAID5 array is forever broken, I
can't rebuild it!

The only option that is left is to assemble the array with --force and
save all the data, repartition the drives (1MB until (capacity -
2GB)), and build the array from scratch?

Now, where to store 6 terabytes of data while rebuilding everything. :-/

Cheers


On 23 October 2010 02:59, Mark Lord <kernel@teksavvy.com> wrote:
> On 10-10-20 04:03 PM, Mathias Burén wrote:
> ..
>>
>> I'm currently not using the BIOS of the raid controller for anything
>> else then staggered disk spinup. The HDD partitions start at sector
>> 2048 (to get a 1MB alignment since they're 4k sector drives, WD20EARS)
>> and end at the last sector.
>> What I'm worried about is the corruption mentioned in dmesg, is this
>> explained somewhere in more detail? Google didn't reveal much. Am I in
>> danger?
>
> Yes.  Just repartition the drives to avoid the final 2GB of each drive,
> and then you'll be safe.
>
> The RocketRaid BIOS that I examined here a couple of years ago,
> liked to write "metadata" over top of whatever was in certain sectors
> near the end of the drive.  EVEN FOR NON-RAID DRIVES.
>
> I think it was the last even (power-of-two) multiple of 1GB or something,
> so if you leave the final 2GB untouched, you're guaranteed to avoid it.
>
> Thus the recommendation.
>
> Cheers
>

Re: sata_mv and Highpoint RocketRAID 230x, corruption?

[Mark Lord]

On 10-10-23 08:57 AM, Mathias Burén wrote:
> Hi,
>
> Thanks for the clarification. Since I've (stupidly enough) partitioned
> the drives from 1MB until the end, I'm most likely affected by this
> stupid RAID BIOS.
> That might explain why I'm getting this error (full dmesg at
> http://pastebin.ca/1970873 ):
>
> ata2.00: status: { DRDY }
> ata2: hard resetting link
> ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> ata2.00: configured for UDMA/133
> ata2.00: device reported invalid CHS sector 0
> sd 1:0:0:0: [sdb]  Result: hostbyte=0x00 driverbyte=0x08
> sd 1:0:0:0: [sdb]  Sense Key : 0xb [current] [descriptor]
> Descriptor sense data with sense descriptors (in hex):
>          72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
>          00 00 00 00
> sd 1:0:0:0: [sdb]  ASC=0x0 ASCQ=0x0
> sd 1:0:0:0: [sdb] CDB: cdb[0]=0x28: 28 00 e7 70 c8 e8 00 05 40 00
> end_request: I/O error, dev sdb, sector 3882928360
> md/raid:md0: read error not correctable (sector 3882926312 on sdb1).
> md/raid:md0: Disk failure on sdb1, disabling device.


No, that error looks like a real disk media error -- bad sector(s) on the drive.

The BIOS issue merely gives corrupted data, not read errors.

Re: sata_mv and Highpoint RocketRAID 230x, corruption?

[Mathias Burén]

Hi,

Interesting, as the badblocks program doesn't think these sectors are
bad. Can I test them any other way?

Thanks,

// Mathias

On 23 October 2010 16:19, Mark Lord <kernel@teksavvy.com> wrote:
> On 10-10-23 08:57 AM, Mathias Burén wrote:
>>
>> Hi,
>>
>> Thanks for the clarification. Since I've (stupidly enough) partitioned
>> the drives from 1MB until the end, I'm most likely affected by this
>> stupid RAID BIOS.
>> That might explain why I'm getting this error (full dmesg at
>> http://pastebin.ca/1970873 ):
>>
>> ata2.00: status: { DRDY }
>> ata2: hard resetting link
>> ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
>> ata2.00: configured for UDMA/133
>> ata2.00: device reported invalid CHS sector 0
>> sd 1:0:0:0: [sdb]  Result: hostbyte=0x00 driverbyte=0x08
>> sd 1:0:0:0: [sdb]  Sense Key : 0xb [current] [descriptor]
>> Descriptor sense data with sense descriptors (in hex):
>>         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
>>         00 00 00 00
>> sd 1:0:0:0: [sdb]  ASC=0x0 ASCQ=0x0
>> sd 1:0:0:0: [sdb] CDB: cdb[0]=0x28: 28 00 e7 70 c8 e8 00 05 40 00
>> end_request: I/O error, dev sdb, sector 3882928360
>> md/raid:md0: read error not correctable (sector 3882926312 on sdb1).
>> md/raid:md0: Disk failure on sdb1, disabling device.
>
>
> No, that error looks like a real disk media error -- bad sector(s) on the
> drive.
>
> The BIOS issue merely gives corrupted data, not read errors.
>

Re: sata_mv and Highpoint RocketRAID 230x, corruption?

[Mark Lord]

On 10-10-22 10:21 PM, Daniel Taylor wrote:
>Mark Lord wrote:
>>
>> The RocketRaid BIOS that I examined here a couple of years ago,
>> liked to write "metadata" over top of whatever was in certain sectors
>> near the end of the drive.  EVEN FOR NON-RAID DRIVES.
>>
>> I think it was the last even (power-of-two) multiple of 1GB
>> or something,
>> so if you leave the final 2GB untouched, you're guaranteed to
>> avoid it.
>>
>> Thus the recommendation.
>
> What happens to the backup GPT at the end of the disk?

Dunno.

Re: sata_mv and Highpoint RocketRAID 230x, corruption?

[Mark Lord]

On 10-10-23 11:20 AM, Mathias Burén wrote:
> Hi,
>
> Interesting, as the badblocks program doesn't think these sectors are
> bad. Can I test them any other way?
..
> On 23 October 2010 16:19, Mark Lord<kernel@teksavvy.com>  wrote:
>> On 10-10-23 08:57 AM, Mathias Burén wrote:
..
>>> ata2.00: status: { DRDY }
>>> ata2: hard resetting link
>>> ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
>>> ata2.00: configured for UDMA/133
>>> ata2.00: device reported invalid CHS sector 0
>>> sd 1:0:0:0: [sdb]  Result: hostbyte=0x00 driverbyte=0x08
>>> sd 1:0:0:0: [sdb]  Sense Key : 0xb [current] [descriptor]
>>> Descriptor sense data with sense descriptors (in hex):
>>>          72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
>>>          00 00 00 00
>>> sd 1:0:0:0: [sdb]  ASC=0x0 ASCQ=0x0
>>> sd 1:0:0:0: [sdb] CDB: cdb[0]=0x28: 28 00 e7 70 c8 e8 00 05 40 00
>>> end_request: I/O error, dev sdb, sector 3882928360
>>> md/raid:md0: read error not correctable (sector 3882926312 on sdb1).
>>> md/raid:md0: Disk failure on sdb1, disabling device.
>>
>>
>> No, that error looks like a real disk media error -- bad sector(s) on the drive.
>>
>> The BIOS issue merely gives corrupted data, not read errors.

MMm.. you're right.
I just now looked at the full dmesg you posted,
and those are NOT media errors.

It looks like NCQ commands are behaving strangely for some reason
in your 2.6.36 kernel.

Can you retest with, say, 2.6.34 ?
There were a number of sata_mv updates in between,
and I'm wondering if perhaps one of them broke something?

Or if you just want to stabilize things, then turn off NCQ.

Cheers

Re: sata_mv and Highpoint RocketRAID 230x, corruption?

[Mathias Burén]

Good! (that it's not a media error) I've ran extended SMART tests on
the drive as well, and everything seemed fine.

I'm going to try with 2.6.35 series now, see if I can salvage some data.

Thanks,

// Mathias

On 23 October 2010 16:49, Mark Lord <kernel@teksavvy.com> wrote:
> On 10-10-23 11:20 AM, Mathias Burén wrote:
>>
>> Hi,
>>
>> Interesting, as the badblocks program doesn't think these sectors are
>> bad. Can I test them any other way?
>
> ..
>>
>> On 23 October 2010 16:19, Mark Lord<kernel@teksavvy.com>  wrote:
>>>
>>> On 10-10-23 08:57 AM, Mathias Burén wrote:
>
> ..
>>>>
>>>> ata2.00: status: { DRDY }
>>>> ata2: hard resetting link
>>>> ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
>>>> ata2.00: configured for UDMA/133
>>>> ata2.00: device reported invalid CHS sector 0
>>>> sd 1:0:0:0: [sdb]  Result: hostbyte=0x00 driverbyte=0x08
>>>> sd 1:0:0:0: [sdb]  Sense Key : 0xb [current] [descriptor]
>>>> Descriptor sense data with sense descriptors (in hex):
>>>>         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
>>>>         00 00 00 00
>>>> sd 1:0:0:0: [sdb]  ASC=0x0 ASCQ=0x0
>>>> sd 1:0:0:0: [sdb] CDB: cdb[0]=0x28: 28 00 e7 70 c8 e8 00 05 40 00
>>>> end_request: I/O error, dev sdb, sector 3882928360
>>>> md/raid:md0: read error not correctable (sector 3882926312 on sdb1).
>>>> md/raid:md0: Disk failure on sdb1, disabling device.
>>>
>>>
>>> No, that error looks like a real disk media error -- bad sector(s) on the
>>> drive.
>>>
>>> The BIOS issue merely gives corrupted data, not read errors.
>
> MMm.. you're right.
> I just now looked at the full dmesg you posted,
> and those are NOT media errors.
>
> It looks like NCQ commands are behaving strangely for some reason
> in your 2.6.36 kernel.
>
> Can you retest with, say, 2.6.34 ?
> There were a number of sata_mv updates in between,
> and I'm wondering if perhaps one of them broke something?
>
> Or if you just want to stabilize things, then turn off NCQ.
>
> Cheers
>

Re: sata_mv and Highpoint RocketRAID 230x, corruption?

[Mathias Burén]

On 23 October 2010 17:08, Mathias Burén <mathias.buren@gmail.com> wrote:
> Good! (that it's not a media error) I've ran extended SMART tests on
> the drive as well, and everything seemed fine.
>
> I'm going to try with 2.6.35 series now, see if I can salvage some data.
>
> Thanks,
>
> // Mathias
>
> On 23 October 2010 16:49, Mark Lord <kernel@teksavvy.com> wrote:
>> On 10-10-23 11:20 AM, Mathias Burén wrote:
>>>
>>> Hi,
>>>
>>> Interesting, as the badblocks program doesn't think these sectors are
>>> bad. Can I test them any other way?
>>
>> ..
>>>
>>> On 23 October 2010 16:19, Mark Lord<kernel@teksavvy.com>  wrote:
>>>>
>>>> On 10-10-23 08:57 AM, Mathias Burén wrote:
>>
>> ..
>>>>>
>>>>> ata2.00: status: { DRDY }
>>>>> ata2: hard resetting link
>>>>> ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
>>>>> ata2.00: configured for UDMA/133
>>>>> ata2.00: device reported invalid CHS sector 0
>>>>> sd 1:0:0:0: [sdb]  Result: hostbyte=0x00 driverbyte=0x08
>>>>> sd 1:0:0:0: [sdb]  Sense Key : 0xb [current] [descriptor]
>>>>> Descriptor sense data with sense descriptors (in hex):
>>>>>         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
>>>>>         00 00 00 00
>>>>> sd 1:0:0:0: [sdb]  ASC=0x0 ASCQ=0x0
>>>>> sd 1:0:0:0: [sdb] CDB: cdb[0]=0x28: 28 00 e7 70 c8 e8 00 05 40 00
>>>>> end_request: I/O error, dev sdb, sector 3882928360
>>>>> md/raid:md0: read error not correctable (sector 3882926312 on sdb1).
>>>>> md/raid:md0: Disk failure on sdb1, disabling device.
>>>>
>>>>
>>>> No, that error looks like a real disk media error -- bad sector(s) on the
>>>> drive.
>>>>
>>>> The BIOS issue merely gives corrupted data, not read errors.
>>
>> MMm.. you're right.
>> I just now looked at the full dmesg you posted,
>> and those are NOT media errors.
>>
>> It looks like NCQ commands are behaving strangely for some reason
>> in your 2.6.36 kernel.
>>
>> Can you retest with, say, 2.6.34 ?
>> There were a number of sata_mv updates in between,
>> and I'm wondering if perhaps one of them broke something?
>>
>> Or if you just want to stabilize things, then turn off NCQ.
>>
>> Cheers
>>
>

Hey again,

Wow, somehow it looks like it's actually OK now. I don't know why to
be honest. Details:

[root@ion raid-MBR-backup]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdg1[0] sdc1[3] sdd1[4] sdb1[1]
      5851054080 blocks super 1.2 level 5, 128k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>

So it successfully grew to 4 devices. Yay! It's online and happy. The
~3.7TB ext4 fs under the LVM beneath md0 is fine.

What I need to do now, is shrink each partition of the 4 drives making
the RAID, to avoid the last 2 GB.

What I've done is, I shrinked md0 with mdadm --grow, so now it looks
like this on one of the drives:

[root@ion raid-MBR-backup]# mdadm -E /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e6595c64:b3ae90b3:f01133ac:3f402d20
           Name : ion:0  (local to host ion)
  Creation Time : Tue Oct 19 08:58:41 2010
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
     Array Size : 11702108160 (5580.00 GiB 5991.48 GB)
  Used Dev Size : 3900702720 (1860.00 GiB 1997.16 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 634f3893:7af5fdd3:7ff344c7:8e3c4cff

    Update Time : Sun Oct 24 14:31:00 2010
       Checksum : 1a7657ec - correct
         Events : 30786

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing)

My question is, is it safe for me to stop md0, delete all 4 partitions
that make up md0, recreate them at the same starting sector, but
ending 2GB from the last sector? Is this safe, will I lose any data?
Just in case I've backuped the MBR (first 512 bytes) of each HDD that
has the partition.

(sorry for top posting..)

Kind regards,
// Mathias

Re: sata_mv and Highpoint RocketRAID 230x, corruption?

[Mark Lord]

On 10-10-24 08:52 AM, Mathias Burén wrote:
>
> My question is, is it safe for me to stop md0, delete all 4 partitions
> that make up md0, recreate them at the same starting sector, but
> ending 2GB from the last sector? Is this safe, will I lose any data?
> Just in case I've backuped the MBR (first 512 bytes) of each HDD that
> has the partition.
..

Dunno.  I'm no RAID expert.  :)

Re: sata_mv and Highpoint RocketRAID 230x, corruption?

[Mathias Burén]

Sorry for the noise,

I already tried that and it broke the RAID. Instead I shrunk the "Used
Dev Size" of each RAID member, like so (mdadm -E dev):

Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
Used Dev Size : 3900702720 (1860.00 GiB 1997.16 GB)

I hope that should be OK. Thanks for all the info.

// Mathias


On 25 October 2010 22:26, Mark Lord <kernel@teksavvy.com> wrote:
> On 10-10-24 08:52 AM, Mathias Burén wrote:
>>
>> My question is, is it safe for me to stop md0, delete all 4 partitions
>> that make up md0, recreate them at the same starting sector, but
>> ending 2GB from the last sector? Is this safe, will I lose any data?
>> Just in case I've backuped the MBR (first 512 bytes) of each HDD that
>> has the partition.
>
> ..
>
> Dunno.  I'm no RAID expert.  :)
>