RAID1 always resyncs at boot???

RAID1 always resyncs at boot???

[drtebi]

I have browsed numerous threads and got my RAID 1 to work just fine. However,
there is one strange problem I have that I couldn't get an answer to:

After booting, my /proc/mdstat looked like this:

Personalities : [raid1]
read_ahead 1024 sectors
md0 : active raid1 ide/host0/bus0/target0/lun0/part1[0]
ide/host0/bus1/target0/lun0/part1[1]
      120053632 blocks [2/2] [UU]
      [>....................]  resync =  1.3% (1601708/120053632)
finish=164.9min speed=11969K/sec
unused devices: <none>

OK, so I figured the RAID is being built (synced), and waited until it was
done. Then the same command showed everything was fine and running:

Personalities : [raid1]
read_ahead 1024 sectors
md0 : active raid1 ide/host0/bus0/target0/lun0/part1[0]
ide/host0/bus1/target0/lun0/part1[1]
      120053632 blocks [2/2] [UU]

unused devices: <none>

However, the problem is if I reboot, it starts all over again with the resync,
(from 0), every time! Here is what I get from dmesg:
--- snip ---
md: raid1 personality registered as nr 3
md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: Autodetecting RAID arrays.
 [events: 00000010]
 [events: 00000010]
md: autorun ...
md: considering ide/host0/bus1/target0/lun0/part1 ...
md:  adding ide/host0/bus1/target0/lun0/part1 ...
md:  adding ide/host0/bus0/target0/lun0/part1 ...
md: created md0
md: bind<ide/host0/bus0/target0/lun0/part1,1>
md: bind<ide/host0/bus1/target0/lun0/part1,2>
md: running:
<ide/host0/bus1/target0/lun0/part1><ide/host0/bus0/target0/lun0/part1>
md: ide/host0/bus1/target0/lun0/part1's event counter: 00000010
md: ide/host0/bus0/target0/lun0/part1's event counter: 00000010
md: md0: raid array is not clean -- starting background reconstruction
md: RAID level 1 does not need chunksize! Continuing anyway.
md0: max total readahead window set to 124k
md0: 1 data-disks, max readahead per data-disk: 124k
raid1: device ide/host0/bus1/target0/lun0/part1 operational as mirror 1
raid1: device ide/host0/bus0/target0/lun0/part1 operational as mirror 0
raid1: raid set md0 not clean; reconstructing mirrors
raid1: raid set md0 active with 2 out of 2 mirrors
md: updating md0 RAID superblock on device
md: ide/host0/bus1/target0/lun0/part1 [events: 00000011]<6>(write)
ide/host0/bus1/target0/lun0/part1's sb offset: 120053632
md: syncing RAID array md0
md: minimum _guaranteed_ reconstruction speed: 100 KB/sec/disc.
md: using maximum available idle IO bandwith (but not more than 100000 KB/sec)
for reconstruction.
md: using 124k window, over a total of 120053632 blocks.
md: ide/host0/bus0/target0/lun0/part1 [events: 00000011]<6>(write)
ide/host0/bus0/target0/lun0/part1's sb offset: 120053632
md: ... autorun DONE.
--- snip ---

Everything like reading and writing to the md0 works just fine, and still does
now, except the resync starts again at every boot! 
What is wrong, or what is it that I don't understand? Is it supposed to resync
at every boot?
I checked my kernel messages, there was nothing indicating that any of the
drives are bad. I am not using the RAID as a boot drive, simply as a storage.


----------------- Details about my install --------------
My system:
400 MgHz Pentium III
SuperMicro P6SBS
256MB SDRAM (Crucial)
Quantum Viking II 4.5 GB SCSI Disk (holds the Gentoo OS)
2 x Maxtor Diamond 9 120 GB disk (for the RAID1)
3COM NIC

I used Gentoo's LiveCD "x86-basic-1.4-20030911.iso", which is using Kernel
2.4.20, and installed everything from scratch, with RAID support:

[*] Multiple devices driver support (RAID and LVM)
<*>  RAID support
< >   Linear (append) mode
< >   RAID-0 (striping) mode
<*>   RAID-1 (mirroring) mode
< >   RAID-4/RAID-5 mode
< >   Multipath I/O support
< >  Logical volume manager (LVM) support

To create the RAID, I used cfdisk to create one primary partion, so about
114GB, on each drive. I set the partion type to FD.
I rebooted to see if the system read the partions correctly. Then I created
the RAID 1 with mdadm:

mdadm --create /dev/md0 --chunk=128 --level=1 raid-devices=2 /dev/hd[ac]1

This command also starts the RAID. So all that was left to do is create a file
system on the disks and start using them. I chose XFS:

mkfs.xfs -d agcount=64 -l size=32m /dev/md0

This is my /etc/mdadm.conf:

DEVICE /dev/hda1 /dev/hdc1
ARRAY /dev/md0 devices=/dev/hda1,/dev/hdc1

and this my /etc/fstab:

# <fs>               <mountpoint>   <type>      <opts>            <dump/pass>
/dev/sda1            /boot          ext2        noauto,noatime    1 1
/dev/sda5            /              xfs         noatime           0 0
/dev/dsa2            none           swap        sw                0 0
/dev/md0             /raid          xfs         noatime           0 0
/dev/cdroms/cdrom0   /mnt/cdrom     iso9660     noauto,ro         0 0
proc                 /proc          proc        defaults          0 0
tmpfs                /dev/shm       tmpfs       defaults          0 0

...please help/explain what the problem is.

Re: RAID1 always resyncs at boot???

[Neil Brown]

On Thursday November 27, drtebi@drtebi.com wrote:
> 
> Everything like reading and writing to the md0 works just fine, and still does
> now, except the resync starts again at every boot! 
> What is wrong, or what is it that I don't understand? Is it supposed to resync
> at every boot?

No, it is not suppose to resync at every boot, providing that it shuts
down properly.
How do you shut down your machine?  If you just turn the power off,
you would get a resync at every boot.  If you shutdown gracefully with
"/sbin/halt" or similar, you shouldn't.

So: how do you shut down, and do you get any interesting kernel
messages on the console during shutdown?

NeilBrown

Re: RAID1 always resyncs at boot???

[drtebi]

Well,
no, I do a regular shutdown with shutdown -h now, or reboot for rebooting.
I have it working now however.
I think what was wrong is that I had my /etc/fstab like this:
# <fs>               <mountpoint>   <type>      <opts>            <dump/pass>
/dev/sda1            /boot          ext2        noauto,noatime    1 1
/dev/sda5            /              xfs         noatime           0 0
/dev/dsa2            none           swap        sw                0 0
/dev/md0             /raid          xfs         noatime           0 0
/dev/cdroms/cdrom0   /mnt/cdrom     iso9660     noauto,ro         0 0
proc                 /proc          proc        defaults          0 0
tmpfs                /dev/shm       tmpfs       defaults          0 0

I did not have the fschk option (the last one) set to 1 on /dev/md0. At least
someone advised me to do so, so I set it to 1, tried to reboot and it works
just fine now ;)
(here is the thread of this help:
http://forums.devshed.com/t101414/sd5d6d3f616a474180753071f3ce364ef.html )

I still would like to understand WHY this helped, could you explain?

Another question: How do I restart "from scratch", reformatting the drives
does not earase the persistent-superblock etc....

And last one: Now that the RAID is setup (which is not holding the OS, it's
purely used for storage), I would like to use another disk for the OS. So
after I install Linux on another disk, can I just mount the RAID afterwards?

Thanks!

Re: RAID1 always resyncs at boot???

[Neil Brown]

On Thursday November 27, drtebi@drtebi.com wrote:
> Well,
> no, I do a regular shutdown with shutdown -h now, or reboot for rebooting.
> I have it working now however.
> I think what was wrong is that I had my /etc/fstab like this:
> # <fs>               <mountpoint>   <type>      <opts>            <dump/pass>
> /dev/sda1            /boot          ext2        noauto,noatime    1 1
> /dev/sda5            /              xfs         noatime           0 0
> /dev/dsa2            none           swap        sw                0 0
> /dev/md0             /raid          xfs         noatime           0 0
> /dev/cdroms/cdrom0   /mnt/cdrom     iso9660     noauto,ro         0 0
> proc                 /proc          proc        defaults          0 0
> tmpfs                /dev/shm       tmpfs       defaults          0 0
> 
> I did not have the fschk option (the last one) set to 1 on /dev/md0. At least
> someone advised me to do so, so I set it to 1, tried to reboot and it works
> just fine now ;)
> (here is the thread of this help:
> http://forums.devshed.com/t101414/sd5d6d3f616a474180753071f3ce364ef.html )
> 
> I still would like to understand WHY this helped, could you explain?

There is no reason that this change would help.  I am quite sure it is
pure coincidence that is seems to make a difference.

> 
> Another question: How do I restart "from scratch", reformatting the drives
> does not earase the persistent-superblock etc....

Try the "--zero-superblock" option to mdadm (check the man page for
correct spelling).

> 
> And last one: Now that the RAID is setup (which is not holding the OS, it's
> purely used for storage), I would like to use another disk for the OS. So
> after I install Linux on another disk, can I just mount the RAID
> afterwards?

I suspect so, though without precise details I cannot be sure of
exactly what you are asking.

NeilBrown

Re: RAID1 always resyncs at boot???

[drtebi]

Well,
I thought it was fixed, but I suppose I was wrong... 

OK, I admit I was a bit brutal, but anyway.
The problem is that I have power outages here once in a while, it's something
with the wiring in this appartment (yes, I am running this server at home,
have an UPS, but it only lasts 40 min.).
 
So I pulled the plug of the server to give it a test, to see if it comes back
up fine and will reconstruct the array right.

Everything comes back up fine, the RAID array starts reconstructing. But after
an hour or so I find this:

/proc/mdstat:
Nov 28 04:41:56 [kernel] md: md_do_sync() got signal ... exiting
Nov 28 04:41:56 [kernel] raid1: mirror resync was not fully finished,
restarting next time.

.
/bus1/target0/lun0/part1


in my log:
Nov 28 04:41:44 [kernel] hda: dma_intr: status=0x51 { DriveReady SeekComplete
Error }
Nov 28 04:41:44 [kernel] md: (skipping faulty
ide/host0/bus0/target0/lun0/part1 )
Nov 28 04:41:45 [kernel] hda: dma_intr: status=0x51 { DriveReady SeekComplete
Error }
                - Last output repeated 8 times -
Nov 28 04:41:56 [kernel] md: md_do_sync() got signal ... exiting
Nov 28 04:41:56 [kernel] raid1: mirror resync was not fully finished,
restarting next time.

Does this mean one of the drives is bad?

Here is some more info from mdadm:
bully@drtebi: mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.00
  Creation Time : Thu Nov 27 01:46:27 2003
     Raid Level : raid1
     Array Size : 120053632 (114.49 GiB 122.93 GB)
    Device Size : 120053632 (114.49 GiB 122.93 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Nov 28 04:41:44 2003
          State : dirty, no-errors
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0


    Number   Major   Minor   RaidDevice State
       0       3        1        0      faulty  
/dev/ide/host0/bus0/target0/lun0/part1
       1      22        1        1      active sync  
/dev/ide/host0/bus1/target0/lun0/part1
           UUID : 96c32ab9:7f492b81:623580da:29d8a727
         Events : 0.16

------------
bully@drtebi: mdadm --examine /dev/hda1
/dev/hda1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 96c32ab9:7f492b81:623580da:29d8a727
  Creation Time : Thu Nov 27 01:46:27 2003
     Raid Level : raid1
    Device Size : 120053632 (114.49 GiB 122.93 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Fri Nov 28 04:32:17 2003
          State : dirty, no-errors
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : d1fbb8ef - correct
         Events : 0.15


      Number   Major   Minor   RaidDevice State
this     0       3        1        0      active sync  
/dev/ide/host0/bus0/target0/lun0/part1
   0     0       3        1        0      active sync  
/dev/ide/host0/bus0/target0/lun0/part1
   1     1      22        1        1      active sync  
/dev/ide/host0/bus1/target0/lun0/part1

bully@drtebi: mdadm --examine /dev/hda1
/dev/hda1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 96c32ab9:7f492b81:623580da:29d8a727
  Creation Time : Thu Nov 27 01:46:27 2003
     Raid Level : raid1
    Device Size : 120053632 (114.49 GiB 122.93 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Fri Nov 28 04:32:17 2003
          State : dirty, no-errors
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : d1fbb8ef - correct
         Events : 0.15


      Number   Major   Minor   RaidDevice State
this     0       3        1        0      active sync  
/dev/ide/host0/bus0/target0/lun0/part1
   0     0       3        1        0      active sync  
/dev/ide/host0/bus0/target0/lun0/part1
   1     1      22        1        1      active sync  
/dev/ide/host0/bus1/target0/lun0/part1

-------------
bully@drtebi: mdadm --query /dev/md0
/dev/md0: 114.49GiB raid1 2 devices, 0 spares. Use mdadm --detail for more
detail.
/dev/md0: No md super block found, not an md component.

The last message is weird... no md super block?

Thanks,
DrTebi

P.S.: I don't really like IDE drives :(

Re: RAID1 always resyncs at boot???

[Neil Brown]

On Thursday November 27, drtebi@drtebi.com wrote:
> 
> in my log:
> Nov 28 04:41:44 [kernel] hda: dma_intr: status=0x51 { DriveReady SeekComplete
> Error }
> Nov 28 04:41:44 [kernel] md: (skipping faulty
> ide/host0/bus0/target0/lun0/part1 )
> Nov 28 04:41:45 [kernel] hda: dma_intr: status=0x51 { DriveReady SeekComplete
> Error }
>                 - Last output repeated 8 times -
> Nov 28 04:41:56 [kernel] md: md_do_sync() got signal ... exiting
> Nov 28 04:41:56 [kernel] raid1: mirror resync was not fully finished,
> restarting next time.
> 
> Does this mean one of the drives is bad?

Well, it means one of your drives return an error, so it was kicked
out of the array.
Whether it is the drive that is bad, or the controller, or the
power-supply, I cannot know.
You could try removing it from the array and putting it back, and let
it rebuild, and see if the problem happens again.

> 
> -------------
> bully@drtebi: mdadm --query /dev/md0
> /dev/md0: 114.49GiB raid1 2 devices, 0 spares. Use mdadm --detail for more
> detail.
> /dev/md0: No md super block found, not an md component.
> 
> The last message is weird... no md super block?

That is proper.  /dev/md0 is not a component of another md, so it
doesn't have a superblock on it (the individual drives have
superblocks, not the array).

> 
> Thanks,
> DrTebi
> 
> P.S.: I don't really like IDE drives :(
> 
Me neither.

NeilBrown

Re: RAID1 always resyncs at boot???

[drtebi]

I am rebuilding the array now, it will take about 2 hours.

The way to do it is to unmount /dev/md0, and then issue
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/hd[ac]1
correct?

Trying to mark the drive faulty and add it back in (as described in the man
page) does not work. But I suppose that couldn't, since it's already faulty.

You said it might be the power supply? I have a 350 Watt power supply in
there, kind of a cheap one I think. I thought too that connecting 4 drives to
it might be a bit much.

However, if I do the "unplugged concert" again, and the RAID fails to
reconstruct again, what do you suggest? Try another power supply? Give it up,
spend $200 more on SCSI drives? I am a bit clueless, I thought I would sleep
better when having a RAID1 for my data, so far it's the opposite :(

DrTebi


--- Neil Brown <neilb@cse.unsw.edu.au> wrote:

> On Thursday November 27, drtebi@drtebi.com wrote:
> > 
> > in my log:
> > Nov 28 04:41:44 [kernel] hda: dma_intr: status=0x51 { DriveReady
> SeekComplete
> > Error }
> > Nov 28 04:41:44 [kernel] md: (skipping faulty
> > ide/host0/bus0/target0/lun0/part1 )
> > Nov 28 04:41:45 [kernel] hda: dma_intr: status=0x51 { DriveReady
> SeekComplete
> > Error }
> >                 - Last output repeated 8 times -
> > Nov 28 04:41:56 [kernel] md: md_do_sync() got signal ... exiting
> > Nov 28 04:41:56 [kernel] raid1: mirror resync was not fully finished,
> > restarting next time.
> > 
> > Does this mean one of the drives is bad?
> 
> Well, it means one of your drives return an error, so it was kicked
> out of the array.
> Whether it is the drive that is bad, or the controller, or the
> power-supply, I cannot know.
> You could try removing it from the array and putting it back, and let
> it rebuild, and see if the problem happens again.
> 
> > 
> > -------------
> > bully@drtebi: mdadm --query /dev/md0
> > /dev/md0: 114.49GiB raid1 2 devices, 0 spares. Use mdadm --detail for more
> > detail.
> > /dev/md0: No md super block found, not an md component.
> > 
> > The last message is weird... no md super block?
> 
> That is proper.  /dev/md0 is not a component of another md, so it
> doesn't have a superblock on it (the individual drives have
> superblocks, not the array).
> 
> > 
> > Thanks,
> > DrTebi
> > 
> > P.S.: I don't really like IDE drives :(
> > 
> Me neither.
> 
> NeilBrown
> 

Re: RAID1 always resyncs at boot???

[Ricky Beam]

On Thu, 27 Nov 2003 drtebi@drtebi.com wrote:
>Trying to mark the drive faulty and add it back in (as described in the man
>page) does not work. But I suppose that couldn't, since it's already faulty.

Of course, you could remove it completely (RAID doesn't even know it exists)
and proceed.  I would suggest this followed by a zero'ing (dd if=/dev/zero)
of the suspect drive.  If you still see errors, check it's warantee status.

>However, if I do the "unplugged concert" again, and the RAID fails to
>reconstruct again, what do you suggest? Try another power supply? Give it up,
>spend $200 more on SCSI drives? ...

No, buy more cheap IDE drives.  One chooses IDE because they are cheap
with the understanding they will need to be replaced about once every one
to two years.  Your mileage may vary... I've been around computers for
20 years; I've never seen a SCSI drive bad right out of the box.  I've
seen dozens of still-born IDE drives. (and at least one that should never
have been boxed. the low-level format of that drive failed, why they
put a controller on it and sold it is beyond me.)

Personally, when I care about the bits, I'll gladly pay more for drives
that have actually been tested and certified to last more than a week.
However, most people only see $$$ and don't see the headaches that
come from lost data.

--Ricky

Re: RAID1 always resyncs at boot???

[drtebi]

I ended up using Maxtor's "Powermax" program to check the drive, and it turned
out to be bad.
So I will return it and install another one. I am glad I was doing all this as
a "testing phase", before I actually deleted the data I was going to put on
the RAID.

Although I do have more trust in SCSI drives, only one of about 8 failed in my
"career", I will go ahead and get another IDE. 

I bought it and it's "powermax" test failed--quite a bad reputation it should
be, however... they're cheap. So I will try again ;)

Thanks for everybody helping me! I really appreciate it.

DrTebi


--- Ricky Beam <jfbeam@bluetronic.net> wrote:

> On Thu, 27 Nov 2003 drtebi@drtebi.com wrote:
> >Trying to mark the drive faulty and add it back in (as described in the man
> >page) does not work. But I suppose that couldn't, since it's already
faulty.
> 
> Of course, you could remove it completely (RAID doesn't even know it exists)
> and proceed.  I would suggest this followed by a zero'ing (dd if=/dev/zero)
> of the suspect drive.  If you still see errors, check it's warantee status.
> 
> >However, if I do the "unplugged concert" again, and the RAID fails to
> >reconstruct again, what do you suggest? Try another power supply? Give it
up,
> >spend $200 more on SCSI drives? ...
> 
> No, buy more cheap IDE drives.  One chooses IDE because they are cheap
> with the understanding they will need to be replaced about once every one
> to two years.  Your mileage may vary... I've been around computers for
> 20 years; I've never seen a SCSI drive bad right out of the box.  I've
> seen dozens of still-born IDE drives. (and at least one that should never
> have been boxed. the low-level format of that drive failed, why they
> put a controller on it and sold it is beyond me.)
> 
> Personally, when I care about the bits, I'll gladly pay more for drives
> that have actually been tested and certified to last more than a week.
> However, most people only see $$$ and don't see the headaches that
> come from lost data.
> 
> --Ricky
> 
> 
>