Re: md questions [forwarded from already sent mail]

Re: md questions [forwarded from already sent mail]

[Gene Heskett]

On Thursday 22 January 2004 19:28, Gene Heskett wrote:
>Hello Mike;
>
>I've taken the liberty of CC: ing this to Jim Hines, the IT guy at
>wdtv.com, where I was the CE for almost 19 years, now semi-retired.
>He is the one with this problem machine, but isn't on this (lkml)
>mailing list.  If I screw this up, he'll probably reply and correct
>me.
>
>You seem to be the person who answers most of the 'md' questions, so
>here goes one.  Factor in that the list is tired of me so I'd rather
>not make noise in the mailboxes of those who aren't germain to this
>'md' discussion.  If there is a mailing list we should be useing,
>please advise as to signup proceedures.
>
>We built, 2 years ago, a raid server to use as a backup deposit,
> with it doing an rsync of the various clients each night to a
> subdir on the 320Gb raid5 according to the day of the week.
>
>Last week, something went in the toilet on its boot drive running
> ext3 and in the aftermath of an e2fsck everything was there, but in
> the lost+found directory, so the machine was replaced with a
> slightly newer one, the promise cards (20269's I think, but my hand
> isn't anywhere near a bible) were moved in from the older box, and
> the drives enhanced so there are now 4 each 180Gb drives in the
> raid5 setup on those two promise cards, and a couple of the older,
> still good 160Gbs for boot etc on the mobo's own controller.
>
>And since the old box was running rh7.2, Jim figured it probably
>needed to be updated to rh8.0 which was installed on the newer
>system.
>
>Recompileing the 2.4.20 kernel that came with rh8.0 to add the md
>support, then enabling that and formatting the md with the last
>reiserfs 3.x version (from rh8.0's disks I believe) all went well.
>This is what we had been using for 2 years, the only diffs being the
>newer 180Gb drives instead of the former 160's, and the switch from
> a 2.4.18 to a 2.4.20 kernel on a newer mobo.
>
>But, on the restart, with nothing other than the filesystem
> installed on the 'md' drives, it gave us a resync time of about
> 29,000 seconds. We cannot even see why a resync should be running
> since the array was at that point empty.  This was 2 days ago, and
> I've been informed that allthough the recovered crontab scripts
> seem to be working, the write speeds are atrocious, something like
> 16kb/second.  hdparm OTOH, reports the read times to be quite
> respectable and in the 160Mb/sec area.
>
>Also, and not sure if there is any connection, adding a 3rd promise
>card seems to do a fine job of fscking up the drive scanning during
>post.  Jim, haveing those 180's laying around, wanted to setup a
>second md array of 2 of them running in mirror mode in that machine,
>but thats apparently not possible.  It (post) seems to find several
>more drives than actually exist, but none appear to be accessable
>after post.
>
>Recommendations?  Things to check?  We're idiots?

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty: soap,
ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.22% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attornies please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Re: md questions [forward of Jims corrections]

[Gene Heskett]

On Friday 23 January 2004 11:08, Jim Hines wrote:
>On Thursday 22 January 2004 07:28 pm, Gene Heskett wrote:
>> Last week, something went in the toilet on its boot drive running
>> ext3 and in the aftermath of an e2fsck everything was there, but
>> in the lost+found directory, so the machine was replaced with a
>> slightly newer one, the promise cards (20269's I think, but my
>> hand isn't anywhere near a bible) were moved in from the older
>> box, and the drives enhanced so there are now 4 each 180Gb drives
>> in the raid5 setup on those two promise cards, and a couple of the
>> older, still good 160Gbs for boot etc on the mobo's own
>> controller.
>
>Actually, the drives are in the following configuration: Four Maxtor
> 160mb drives, two Western Digital 180MB drives. The Four Maxtors
> are connected to the two Promise 20269 cards, while the WDs are
> connected to the motherboards Highpoint technologies HP370 RAID
> controller, but NOT in hardware RAID mode.
>
>> But, on the restart, with nothing other than the filesystem
>> installed on the 'md' drives, it gave us a resync time of about
>> 29,000 seconds. We cannot even see why a resync should be running
>> since the array was at that point empty.  This was 2 days ago, and
>> I've been informed that allthough the recovered crontab scripts
>> seem to be working, the write speeds are atrocious, something like
>> 16kb/second.  hdparm OTOH, reports the read times to be quite
>> respectable and in the 160Mb/sec area.
>
>The really strange thing is, it starts doing a resync immediatly
> after doing a mkraid /dev/md0.....before the drives have even been
> formatted (to reiserfs for ext3). I then format the drives, and the
> resync continues (I have also tried the newer mdadm tools, with the
> same results).
>
>It finished the resync yesterday, and I have been doing backups, but
> the write times are incredibly slow. One server has been running
> the backup now for over 12hrs and has only gathered about 7 of 66gb
> on the RAID server.
>
>> Also, and not sure if there is any connection, adding a 3rd
>> promise card seems to do a fine job of fscking up the drive
>> scanning during post.  Jim, haveing those 180's laying around,
>> wanted to setup a second md array of 2 of them running in mirror
>> mode in that machine, but thats apparently not possible.  It
>> (post) seems to find several more drives than actually exist, but
>> none appear to be accessable after post.
>
>I gave up on the third Promise card, it is either causing problems
> with the bus, or with the BIOS, so it's gone in favor of the HP370
> motherboard RAID for the two 160s.
>
>> Recommendations?  Things to check?  We're idiots?
>
>Idiots!? Hey! Speak for yourself   ;)
>
>
>Thanks,

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty: soap,
ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.22% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attornies please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.

Re: md questions [forwarded from already sent mail]

[Maarten van den Berg]

On Saturday 24 January 2004 01:31, Gene Heskett wrote:
> On Thursday 22 January 2004 19:28, Gene Heskett wrote:

> >Recompileing the 2.4.20 kernel that came with rh8.0 to add the md
> >support, then enabling that and formatting the md with the last
> >reiserfs 3.x version (from rh8.0's disks I believe) all went well.
> >This is what we had been using for 2 years, the only diffs being the
> >newer 180Gb drives instead of the former 160's, and the switch from
> > a 2.4.18 to a 2.4.20 kernel on a newer mobo.

I haven't used redhat in ages, but isn't md raid part of the 2.4 kernel ?? 
Why are you messing with 'adding' the md support if it is already there ?  
Or does redhat indeed do something that is inexplicable ?  

> >But, on the restart, with nothing other than the filesystem
> > installed on the 'md' drives, it gave us a resync time of about
> > 29,000 seconds. We cannot even see why a resync should be running
> > since the array was at that point empty.  This was 2 days ago, and
> > I've been informed that allthough the recovered crontab scripts
> > seem to be working, the write speeds are atrocious, something like
> > 16kb/second.  hdparm OTOH, reports the read times to be quite
> > respectable and in the 160Mb/sec area.

A resync will always start, the minute you create a raid device. That is 
transparent so any actions like mkfs can be performed nonetheless.
And this is well documented, if I'm not mistaken. (AFAIK)

The speeds are something to worry about.  I myself had the experience of a 
high reconstruction speed at the start of a resync, but then slowing to a 
crawl during the process. The further it got, the slower. Be advised that 
this was due to a -not so obvious- bad disk, so if you observe the same 
results you know where to look.

> >Also, and not sure if there is any connection, adding a 3rd promise
> >card seems to do a fine job of fscking up the drive scanning during
> >post.  Jim, haveing those 180's laying around, wanted to setup a
> >second md array of 2 of them running in mirror mode in that machine,
> >but thats apparently not possible.  It (post) seems to find several
> >more drives than actually exist, but none appear to be accessable
> >after post.

your hardware is malfunctioning

> >Recommendations?  Things to check?  We're idiots?

All of the above ;-) No seriously, it sounds like a problem with the hardware 
somewhere along the line. Can you test the array on the OLD motherboard, by 
just plugging everything in ?  Also, if you're using persistent superblocks 
and type=0xFD, messing with the order in which the drives are attached / 
recognized should not matter. It is confusing, but the array should 
nonetheless assemble itself perfectly. At least in my experience.

Maarten

Re: md questions [forwarded from already sent mail]

[Maarten van den Berg]

On Saturday 24 January 2004 11:31, you wrote:
> On Saturday 24 January 2004 06:58, Maarten van den Berg wrote:
> >On Saturday 24 January 2004 01:31, Gene Heskett wrote:
> >> On Thursday 22 January 2004 19:28, Gene Heskett wrote:

> >All of the above ;-) No seriously, it sounds like a problem with the
> > hardware somewhere along the line. Can you test the array on the
> > OLD motherboard, by just plugging everything in ?  Also, if you're
> > using persistent superblocks and type=0xFD, messing with the order
> > in which the drives are attached / recognized should not matter. It
> > is confusing, but the array should nonetheless assemble itself
> > perfectly. At least in my experience.
>
> How do we make sure the persistent superblocks are in use?  And the
> 0xFD is I assume what fdisk would show us as the disk type?

It's all in the manual. Yes, the partition type in fdisk.

> Jim has been beating his head on this thing for a week.  Trying the
> old mobo might be a possibility, but its boot drive would need
> formatted and 7.2 re-installed and recompiled so md is builtin to dup
> what we had before, or at least make a reasonable copy of it.  What
> I'm trying to say is that we probably cannot recreate it exactly.
> It was the boot drive, not the raid, that got trashed on the other,
> older mobo.  I'm now confused as to whether the 4 160Gb's in the raid
> array are new, or the same ones from 2 years ago when we built the
> first one which had a 400mhz K6-II on it IIRC, with 256 megs of
> dimms.  This one has a bit more horsepower than that.

No, no, NO.  I don't mean recreate the old setup with the old software. I mean 
use the exact same bootdrive as you're using now and the exact same setup !
It's linux, you know, not windows. You can, in most all cases, just change 
hardware without any changes. The exception being if you have a kernel that 
is optimized for a certain cpu. But even then, a new kernel is compiled 
easily, seen as you have already done that anyway.
So the question is: what happens if you ONLY swap the motherboard but leave 
all else in place (controllers, disks and linux bootdrive)

P.S. I'd appreciate it if you reply to the list instead of to me directly.

Maarten

RE: md questions [forwarded from already sent mail]

[Guy]

>All of the above ;-) No seriously, it sounds like a problem with the
>hardware 
>somewhere along the line. Can you test the array on the OLD motherboard, by

>just plugging everything in ?  Also, if you're using persistent superblocks

>and type=0xFD, messing with the order in which the drives are attached / 
>recognized should not matter. It is confusing, but the array should 
>nonetheless assemble itself perfectly. At least in my experience.
>
>Maarten

RedHat used mkraid and raidstart.  I have problems starting my arrays.  Even
by hand using raidstart, but mdadm has no problems.  The problem is related
to the drive order.  It seems raidstart "knows" which disks are in the
array.  If their names change, game over.  I upgraded the firmware on a SCSI
card, now the order that the system "sees" the SCSI cards has changed, so
the disk names are different.  Someone tell RedHat to use mdadm!! Please!!
Oh, once I started the arrays with mdadm, it seems to have corrected the
problem with raidstart.  I guess it re-wrote the disk names, or something.
I wasted about a day on this issue, maybe it was something else I did.  I
did not want to customize any of the standard startup scripts.  Once you do,
it gets harder to support with updates and such.

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Maarten van den Berg
Sent: Saturday, January 24, 2004 6:59 AM
To: linux-raid@vger.kernel.org
Subject: Re: md questions [forwarded from already sent mail]

On Saturday 24 January 2004 01:31, Gene Heskett wrote:
> On Thursday 22 January 2004 19:28, Gene Heskett wrote:

> >Recompileing the 2.4.20 kernel that came with rh8.0 to add the md
> >support, then enabling that and formatting the md with the last
> >reiserfs 3.x version (from rh8.0's disks I believe) all went well.
> >This is what we had been using for 2 years, the only diffs being the
> >newer 180Gb drives instead of the former 160's, and the switch from
> > a 2.4.18 to a 2.4.20 kernel on a newer mobo.

I haven't used redhat in ages, but isn't md raid part of the 2.4 kernel ?? 
Why are you messing with 'adding' the md support if it is already there ?  
Or does redhat indeed do something that is inexplicable ?  

> >But, on the restart, with nothing other than the filesystem
> > installed on the 'md' drives, it gave us a resync time of about
> > 29,000 seconds. We cannot even see why a resync should be running
> > since the array was at that point empty.  This was 2 days ago, and
> > I've been informed that allthough the recovered crontab scripts
> > seem to be working, the write speeds are atrocious, something like
> > 16kb/second.  hdparm OTOH, reports the read times to be quite
> > respectable and in the 160Mb/sec area.

A resync will always start, the minute you create a raid device. That is 
transparent so any actions like mkfs can be performed nonetheless.
And this is well documented, if I'm not mistaken. (AFAIK)

The speeds are something to worry about.  I myself had the experience of a 
high reconstruction speed at the start of a resync, but then slowing to a 
crawl during the process. The further it got, the slower. Be advised that 
this was due to a -not so obvious- bad disk, so if you observe the same 
results you know where to look.

> >Also, and not sure if there is any connection, adding a 3rd promise
> >card seems to do a fine job of fscking up the drive scanning during
> >post.  Jim, haveing those 180's laying around, wanted to setup a
> >second md array of 2 of them running in mirror mode in that machine,
> >but thats apparently not possible.  It (post) seems to find several
> >more drives than actually exist, but none appear to be accessable
> >after post.

your hardware is malfunctioning

> >Recommendations?  Things to check?  We're idiots?

All of the above ;-) No seriously, it sounds like a problem with the
hardware 
somewhere along the line. Can you test the array on the OLD motherboard, by 
just plugging everything in ?  Also, if you're using persistent superblocks 
and type=0xFD, messing with the order in which the drives are attached / 
recognized should not matter. It is confusing, but the array should 
nonetheless assemble itself perfectly. At least in my experience.

Maarten

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: md questions [forwarded from already sent mail]

[Guy]

From "mdadm --detail /dev/md2":
Persistence : Superblock is persistent

From "sfdisk -d /dev/sdc":
/dev/sdc1 : start=       63, size= 35535717, Id=fd, bootable

Maybe it's because RedHat uses modules for the RAID5?
Don't know, but once I caused the drive names to change the arrays would not
start.  The error messages stated the stuff like "sdc is now sdd".  I don't
recall the exact message.  But I do know mdadm worked fine once I configured
the file mdadm.conf.

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Maarten van den Berg
Sent: Saturday, January 24, 2004 2:24 PM
To: linux-raid@vger.kernel.org
Subject: Re: md questions [forwarded from already sent mail]

On Saturday 24 January 2004 17:27, Guy wrote:
> >All of the above ;-) No seriously, it sounds like a problem with the
> >hardware
> >somewhere along the line. Can you test the array on the OLD motherboard,
> > by
> >
> >just plugging everything in ?  Also, if you're using persistent
> > superblocks
> >
> >and type=0xFD, messing with the order in which the drives are attached /
> >recognized should not matter. It is confusing, but the array should
> >nonetheless assemble itself perfectly. At least in my experience.
> >
> >Maarten
>
> RedHat used mkraid and raidstart.  I have problems starting my arrays. 
> Even by hand using raidstart, but mdadm has no problems.  The problem is

Read my lines above about "persistent superblock" and type=FD again.
If you had used that you would _not_have_ to start the array since the
kernel 
will do this automatically for you. I'm sure this is really extensively 
documented so I wonder how you could have missed that.
I any case, if you leave it up to the kernel you won't have to worry about 
drive order, it is automatically recognized.  Whether you use mkraid or
mdadm 
has nothing to do with it, albeit the latter is better.

> related to the drive order.  It seems raidstart "knows" which disks are in
> the array.  If their names change, game over.  I upgraded the firmware on
a
> SCSI card, now the order that the system "sees" the SCSI cards has
changed,
> so the disk names are different.  Someone tell RedHat to use mdadm!!

Again, this has nothing to do with it. Read up on what I said about type=FD 
and persistent superblocks.

Maarten

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: md questions [forwarded from already sent mail]

[Maarten van den Berg]

On Saturday 24 January 2004 17:27, Guy wrote:
> >All of the above ;-) No seriously, it sounds like a problem with the
> >hardware
> >somewhere along the line. Can you test the array on the OLD motherboard,
> > by
> >
> >just plugging everything in ?  Also, if you're using persistent
> > superblocks
> >
> >and type=0xFD, messing with the order in which the drives are attached /
> >recognized should not matter. It is confusing, but the array should
> >nonetheless assemble itself perfectly. At least in my experience.
> >
> >Maarten
>
> RedHat used mkraid and raidstart.  I have problems starting my arrays. 
> Even by hand using raidstart, but mdadm has no problems.  The problem is

Read my lines above about "persistent superblock" and type=FD again.
If you had used that you would _not_have_ to start the array since the kernel 
will do this automatically for you. I'm sure this is really extensively 
documented so I wonder how you could have missed that.
I any case, if you leave it up to the kernel you won't have to worry about 
drive order, it is automatically recognized.  Whether you use mkraid or mdadm 
has nothing to do with it, albeit the latter is better.

> related to the drive order.  It seems raidstart "knows" which disks are in
> the array.  If their names change, game over.  I upgraded the firmware on a
> SCSI card, now the order that the system "sees" the SCSI cards has changed,
> so the disk names are different.  Someone tell RedHat to use mdadm!!

Again, this has nothing to do with it. Read up on what I said about type=FD 
and persistent superblocks.

Maarten