Spontaneous rebuild

Spontaneous rebuild

[Oliver Martin]

[Please CC me on replies as I'm not subscribed]

Hello!

I've been experimenting with software RAID a bit lately, using two
external 500GB drives. One is connected via USB, one via Firewire. It is
set up as a RAID5 with LVM on top so that I can easily add more drives
when I run out of space.
About a day after the initial setup, things went belly up. First, EXT3
reported strange errors:
EXT3-fs error (device dm-0): ext3_new_block: Allocating block in system
zone - blocks from 106561536, length 1
EXT3-fs error (device dm-0): ext3_new_block: Allocating block in system
zone - blocks from 106561537, length 1
...

There were literally hundreds of these, and they came back immediately
when I reformatted the array. So I tried ReiserFS, which worked fine for
about a day. Then I got errors like these:
ReiserFS: warning: is_tree_node: node level 0 does not match to the
expected one 2
ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in
block 69839092. Fsck?
ReiserFS: dm-0: warning: vs-13070: reiserfs_read_locked_inode: i/o
failure occurred trying to find stat data of [6 10 0x0 SD]

Again, hundreds. So I ran badblocks on the LVM volume, and it reported
some bad blocks near the end. Running badblocks on the md array worked,
so I recreated the LVM stuff and attributed the failures to undervolting
experiments I had been doing (this is my old laptop running as a server).

Anyway, the problems are back: To test my theory that everything is
alright with the CPU running within its specs, I removed one of the
drives while copying some large files yesterday. Initially, everything
seemed to work out nicely, and by the morning, the rebuild had finished.
Again, I unmounted the filesystem and ran badblocks -svn on the LVM. It
ran without gripes for some hours, but just now I saw md had started to
rebuild the array again out of the blue:

Dec  1 20:04:49 quassel kernel: usb 4-5.2: reset high speed USB device
using ehci_hcd and address 4
Dec  2 01:06:02 quassel kernel: md: data-check of RAID array md0
Dec  2 01:06:02 quassel kernel: md: minimum _guaranteed_  speed: 1000
KB/sec/disk.
Dec  2 01:06:02 quassel kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for data-check.
Dec  2 01:06:02 quassel kernel: md: using 128k window, over a total of
488383936 blocks.
Dec  2 03:57:24 quassel kernel: usb 4-5.2: reset high speed USB device
using ehci_hcd and address 4

I'm not sure the USB resets are related to the problem - device 4-5.2 is
part of the array, but I get these sometimes at random intervals and
they don't seem to hurt normally. Besides, the first one was long before
the rebuild started, and the second one long afterwards.

Any ideas why md is rebuilding the array? And could this be related to
the bad blocks problem I had first? badblocks is still running, I'll
post an update when it is finished.
In the meantime, mdadm --detail /dev/md0 and mdadm --examine
/dev/sd[bc]1 don't give me any clues as to what went wrong, both disks
are marked as "active sync", and the whole array is "active, recovering".

Before I forget, I'm running 2.6.23.1 with this config:
http://stud4.tuwien.ac.at/~e0626486/config-2.6.23.1-hrt3-fw

Thanks,
Oliver

Re: Spontaneous rebuild

[Justin Piszcz]

On Sun, 2 Dec 2007, Oliver Martin wrote:

> [Please CC me on replies as I'm not subscribed]
>
> Hello!
>
> I've been experimenting with software RAID a bit lately, using two
> external 500GB drives. One is connected via USB, one via Firewire. It is
> set up as a RAID5 with LVM on top so that I can easily add more drives
> when I run out of space.
> About a day after the initial setup, things went belly up. First, EXT3
> reported strange errors:
> EXT3-fs error (device dm-0): ext3_new_block: Allocating block in system
> zone - blocks from 106561536, length 1
> EXT3-fs error (device dm-0): ext3_new_block: Allocating block in system
> zone - blocks from 106561537, length 1
> ...
>
> There were literally hundreds of these, and they came back immediately
> when I reformatted the array. So I tried ReiserFS, which worked fine for
> about a day. Then I got errors like these:
> ReiserFS: warning: is_tree_node: node level 0 does not match to the
> expected one 2
> ReiserFS: dm-0: warning: vs-5150: search_by_key: invalid format found in
> block 69839092. Fsck?
> ReiserFS: dm-0: warning: vs-13070: reiserfs_read_locked_inode: i/o
> failure occurred trying to find stat data of [6 10 0x0 SD]
>
> Again, hundreds. So I ran badblocks on the LVM volume, and it reported
> some bad blocks near the end. Running badblocks on the md array worked,
> so I recreated the LVM stuff and attributed the failures to undervolting
> experiments I had been doing (this is my old laptop running as a server).
>
> Anyway, the problems are back: To test my theory that everything is
> alright with the CPU running within its specs, I removed one of the
> drives while copying some large files yesterday. Initially, everything
> seemed to work out nicely, and by the morning, the rebuild had finished.
> Again, I unmounted the filesystem and ran badblocks -svn on the LVM. It
> ran without gripes for some hours, but just now I saw md had started to
> rebuild the array again out of the blue:
>
> Dec  1 20:04:49 quassel kernel: usb 4-5.2: reset high speed USB device
> using ehci_hcd and address 4
> Dec  2 01:06:02 quassel kernel: md: data-check of RAID array md0
> Dec  2 01:06:02 quassel kernel: md: minimum _guaranteed_  speed: 1000
> KB/sec/disk.
> Dec  2 01:06:02 quassel kernel: md: using maximum available idle IO
> bandwidth (but not more than 200000 KB/sec) for data-check.
> Dec  2 01:06:02 quassel kernel: md: using 128k window, over a total of
> 488383936 blocks.
> Dec  2 03:57:24 quassel kernel: usb 4-5.2: reset high speed USB device
> using ehci_hcd and address 4
>
> I'm not sure the USB resets are related to the problem - device 4-5.2 is
> part of the array, but I get these sometimes at random intervals and
> they don't seem to hurt normally. Besides, the first one was long before
> the rebuild started, and the second one long afterwards.
>
> Any ideas why md is rebuilding the array? And could this be related to
> the bad blocks problem I had first? badblocks is still running, I'll
> post an update when it is finished.
> In the meantime, mdadm --detail /dev/md0 and mdadm --examine
> /dev/sd[bc]1 don't give me any clues as to what went wrong, both disks
> are marked as "active sync", and the whole array is "active, recovering".
>
> Before I forget, I'm running 2.6.23.1 with this config:
> http://stud4.tuwien.ac.at/~e0626486/config-2.6.23.1-hrt3-fw
>
> Thanks,
> Oliver
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

It rebuilds the array because 'something' is causing device 
resets/timeouts on your USB device:

Dec  1 20:04:49 quassel kernel: usb 4-5.2: reset high speed USB device
using ehci_hcd and address 4

Naturally, when it is reset, the device is disconnected and then 
re-appears, when MD see's this it rebuilds the array.

Why it is timing out/resetting the device, that is what you need to find 
out.

Justin.

Re: Spontaneous rebuild

[Oliver Martin]

Justin Piszcz schrieb:
> 
> It rebuilds the array because 'something' is causing device
> resets/timeouts on your USB device:
> 
> Dec  1 20:04:49 quassel kernel: usb 4-5.2: reset high speed USB device
> using ehci_hcd and address 4
> 
> Naturally, when it is reset, the device is disconnected and then
> re-appears, when MD see's this it rebuilds the array.
> 
> Why it is timing out/resetting the device, that is what you need to find
> out.
> 
> Justin.
> 

Thanks for your answer, I'll investigate the USB resets. Still, it seems
strange that the rebuild only started five hours after the reset. Is
this normal?
The reason I said the resets don't seem to hurt is that I also get them
for a second disk (not in a raid), and file transfers aren't
interrupted, I haven't (yet?) seen any data corruption, and other than
the message, the kernel doesn't seem to mind at all.

BTW, this time, badblocks ran through without any errors. The only
strange thing remaining is the rebuild.


Oliver

Re: Spontaneous rebuild

[Janek Kozicki]

> Justin Piszcz schrieb:
> >
> > Naturally, when it is reset, the device is disconnected and then
> > re-appears, when MD see's this it rebuilds the array.

Least you can do is to add an internal bitmap to your raid, this will
make rebuilds faster.... :-/

-- 
Janek Kozicki                                                         |

Re: Spontaneous rebuild

[Neil Brown]

On Sunday December 2, oliver.martin@student.tuwien.ac.at wrote:
> 
> Anyway, the problems are back: To test my theory that everything is
> alright with the CPU running within its specs, I removed one of the
> drives while copying some large files yesterday. Initially, everything
> seemed to work out nicely, and by the morning, the rebuild had finished.
> Again, I unmounted the filesystem and ran badblocks -svn on the LVM. It
> ran without gripes for some hours, but just now I saw md had started to
> rebuild the array again out of the blue:
> 
> Dec  1 20:04:49 quassel kernel: usb 4-5.2: reset high speed USB device
> using ehci_hcd and address 4
> Dec  2 01:06:02 quassel kernel: md: data-check of RAID array md0
                                      ^^^^^^^^^^
> Dec  2 01:06:02 quassel kernel: md: minimum _guaranteed_  speed: 1000
> KB/sec/disk.
> Dec  2 01:06:02 quassel kernel: md: using maximum available idle IO
> bandwidth (but not more than 200000 KB/sec) for data-check.
                                                  ^^^^^^^^^^
> Dec  2 01:06:02 quassel kernel: md: using 128k window, over a total of
> 488383936 blocks.
> Dec  2 03:57:24 quassel kernel: usb 4-5.2: reset high speed USB device
> using ehci_hcd and address 4
> 

This isn't a resync, it is a data check.  "Dec  2" is the first Sunday
of the month.  You probably have a crontab entries that does
   echo check > /sys/block/mdX/md/sync_action

early on the first Sunday of the month.  I know that Debian does this.

It is good to do this occasionally to catch sleeping bad blocks.

NeilBrown

Re: Spontaneous rebuild

[Justin Piszcz]

On Mon, 3 Dec 2007, Neil Brown wrote:

> On Sunday December 2, oliver.martin@student.tuwien.ac.at wrote:
>>
>> Anyway, the problems are back: To test my theory that everything is
>> alright with the CPU running within its specs, I removed one of the
>> drives while copying some large files yesterday. Initially, everything
>> seemed to work out nicely, and by the morning, the rebuild had finished.
>> Again, I unmounted the filesystem and ran badblocks -svn on the LVM. It
>> ran without gripes for some hours, but just now I saw md had started to
>> rebuild the array again out of the blue:
>>
>> Dec  1 20:04:49 quassel kernel: usb 4-5.2: reset high speed USB device
>> using ehci_hcd and address 4
>> Dec  2 01:06:02 quassel kernel: md: data-check of RAID array md0
>                                      ^^^^^^^^^^
>> Dec  2 01:06:02 quassel kernel: md: minimum _guaranteed_  speed: 1000
>> KB/sec/disk.
>> Dec  2 01:06:02 quassel kernel: md: using maximum available idle IO
>> bandwidth (but not more than 200000 KB/sec) for data-check.
>                                                  ^^^^^^^^^^
>> Dec  2 01:06:02 quassel kernel: md: using 128k window, over a total of
>> 488383936 blocks.
>> Dec  2 03:57:24 quassel kernel: usb 4-5.2: reset high speed USB device
>> using ehci_hcd and address 4
>>
>
> This isn't a resync, it is a data check.  "Dec  2" is the first Sunday
> of the month.  You probably have a crontab entries that does
>   echo check > /sys/block/mdX/md/sync_action
>
> early on the first Sunday of the month.  I know that Debian does this.
>
> It is good to do this occasionally to catch sleeping bad blocks.

While we are on the subject of bad blocks, is it possible to do what 3ware 
raid controllers do without an external card?

They know when a block is bad and they remap it to another part of the 
array etc, where as with software raid you never know this is happening 
until the disk is dead.

For example with 3dm2 it notifies you if you have e-mail alerts set to 2 
(warn) it will e-mail you every time there is a sector re-allocation, is 
this possible with software raid or does it *require* HW raid/external 
controller?

Justin.

Re: Spontaneous rebuild

[Richard Scobie]

Justin Piszcz wrote:

> While we are on the subject of bad blocks, is it possible to do what 
> 3ware raid controllers do without an external card?
> 
> They know when a block is bad and they remap it to another part of the 
> array etc, where as with software raid you never know this is happening 
> until the disk is dead.

Are you sure the 3ware software is remapping the bad blocks, or is it 
just reporting the bad blocks were remapped?

As I understand it, bad block remapping (reallocated sectors), are done 
internally at the drive level.

Perhaps all 3ware are doing is running the SMART command for reallocated 
sectors on all drives on a periodic basis and reporting any changes?

Regards,

Richard

Re: Spontaneous rebuild

[Oliver Martin]

Neil Brown schrieb:
> 
> This isn't a resync, it is a data check.  "Dec  2" is the first Sunday
> of the month.  You probably have a crontab entries that does
>    echo check > /sys/block/mdX/md/sync_action
> 
> early on the first Sunday of the month.  I know that Debian does this.
> 
> It is good to do this occasionally to catch sleeping bad blocks.
> 
Duh, thanks for clearing this up. I guess what set the alarm off was
getting what looked like a rebuild to me while stress testing. Yes, I'm
running Debian and I have exactly this entry in my crontab... Perhaps
they should add a short log entry like "starting periodic RAID check" so
that people know there is nothing to worry about.

Or maybe I should just RTFC (read the fine crontab) ;-)

Oliver