Properly setting up partitions and verbose boot

Properly setting up partitions and verbose boot

[GeneralNMX]

Currently I have a very stupid setup on my home server (not a production
machine). I have four hard drives with three different types of RAID
(1,5,10) on them setup through mdadm. I've been using this for a while and,
as you can guess, I/O Wait is a big issue for me, especially when moving
from different RAID types. I ordered four new hard drives to setup a proper
RAID10 by itself and I'm scrapping the RAID1, instead just consolidating /
into the RAID10. /boot gets its own tiny IDE HDD in a hotswap bay. The RAID5
will consume the 4 old hard drives. 

With my stupid setup, each partition gets its own /dev/mdX device. This is
the only way I know how to do it. On the RAID10, I will need at least two
partitions: / and swap. This means it cannot simply partition the entire
disk Would this cause sub-optimal performance? Is there a way to make an
underlying single RAID10 partition and place the file partitions on top?

I also have a second question. When the disks need to be synced on boot,
mdadm just sits there and does the sync without outputting anything to the
boot log. If I didn't notice the activity lights going off on the SATA
controllers, I would think it's in some infinite loop. Is there a way to
make mdadm more verbose? I'm not running my kernel on "quiet" or "quietboot"
of course. I had to use SysRq just to make sure it was really doing
something and not in some infinite I/O loop.

Matt

Re: Properly setting up partitions and verbose boot

[Keld Jørn Simonsen]

On Sun, Jan 25, 2009 at 11:18:31AM -0500, GeneralNMX wrote:
> 
> Currently I have a very stupid setup on my home server (not a production
> machine). I have four hard drives with three different types of RAID
> (1,5,10) on them setup through mdadm. I've been using this for a while and,
> as you can guess, I/O Wait is a big issue for me, especially when moving
> from different RAID types. I ordered four new hard drives to setup a proper
> RAID10 by itself and I'm scrapping the RAID1, instead just consolidating /
> into the RAID10. /boot gets its own tiny IDE HDD in a hotswap bay. The RAID5
> will consume the 4 old hard drives. 

Why do you think it is stupid?

I have a similar setup described in a howto at:

http://linux-raid.osdl.org/index.php/Preventing_against_a_failing_disk

How big is the iowait issue for you?

What are the performance of your raids?

> With my stupid setup, each partition gets its own /dev/mdX device. This is
> the only way I know how to do it. On the RAID10, I will need at least two
> partitions: / and swap. This means it cannot simply partition the entire
> disk Would this cause sub-optimal performance? Is there a way to make an
> underlying single RAID10 partition and place the file partitions on top?

I dont think it would be suboptimal. But try it out, both solutions and
see for yourself and report to the list your findings!).

I don't think having two raid10.f2 partitions, one for / and one for
swap, will be suboptimal, given the number of drives involved. Each 
raid will do its best for the IO, and the difference between haveing it
all on one raid, vs having it on two raids, would to be be
insignificant. Where  should the extra overhead come from? I even think
the elevator algorithm would be the same, as the elevator is per drive
(As far as I understand it).

do you think that the different raid  types make performance problems?

I think you can partition a MD device into more partitions. 
I have not tried it out for production, though and I dont know how it
performs.

I am in the process of setting up two quad-core machines with 8 GB ram 
for use as virtual servers, and intend to have rai10,f2 in the buttom of
the dom0, and then let the different virtual machines have partitions on
the raid10 array. Is this recommendable? I was thinkin of problems of
both dom0 and domu doing IO, and thus copying io buffers twice.

Best regards
keld

RE: Properly setting up partitions and verbose boot

[GeneralNMX]

From my understanding, there is fault tolerance and then there is the chance
of a disk dying. Obviously, the more disks you have, the greater chance you
have of a disk dying. If we assume all disks start out at some base chance
to fail and degrade, putting multiple RAID types on the same disks can
dramatically increase the wear & tear as the number of disks increase,
especially when you have both a raid5 (which doesn't need to write to all
disks, but will read from all disks) and a raid10 (which probably will write
and read to all disks) on the same physical array of disks. Since fault
tolerance is there to decrease the problems with disks dying, my setup is
obviously sub-optimal. Whenever I access my RAID10, I'm also ever so
slightly degrading my RAID5 and RAID1, and visa-versa.

Now, as for the I/O Wait, this happens when I try to access both the RAID10
and RAID5 at the same time, especially if I'm moving a lot of data from the
RAID10 to the RAID5. While my server was rather old before I upgraded it
just two days ago using spare parts (was a 1997 Supermicro Dual P3 550MHz
1GB SDRAM, now a 2.8GHz P4 w/ HT 1.5GB DDR), I think the I/O Wait was caused
by trying to negotiate the three different RAID arrays at once, which
encompass all four disks, while still allowing access to those arrays. Along
with critical data, I also use the RAID10 as a staging area for large
downloads from other servers due to its speed and reliability. Once I
determine the worth of the data, I usually transfer it to the RAID5, which
does not house critical data (more like it would be annoying if it failed).
Backups of / (sans large downloads) also go on the RAID5 in case the file
systems become corrupted. Again, it would be better to have the backups on
separate physical disks, as corrupt MFTs and seriously corrupt partitions
could hose the entire setup.

But I just do this all for my own enjoyment and education. I don't implement
this stuff in production environments. As much as I'd like to convert my
workplace's Windows XP "Server" using fake raid1 which holds all our
super-critical data...we literally have to reboot that thing every 2-4 hours
due to problems with other software that's on it.


-----Original Message-----
From: Keld Jørn Simonsen [mailto:keld@dkuug.dk] 
Sent: Sunday, January 25, 2009 8:20 PM
To: GeneralNMX
Cc: linux-raid@vger.kernel.org
Subject: Re: Properly setting up partitions and verbose boot

On Sun, Jan 25, 2009 at 11:18:31AM -0500, GeneralNMX wrote:
> 
> Currently I have a very stupid setup on my home server (not a production
> machine). I have four hard drives with three different types of RAID
> (1,5,10) on them setup through mdadm. I've been using this for a while
and,
> as you can guess, I/O Wait is a big issue for me, especially when moving
> from different RAID types. I ordered four new hard drives to setup a
proper
> RAID10 by itself and I'm scrapping the RAID1, instead just consolidating /
> into the RAID10. /boot gets its own tiny IDE HDD in a hotswap bay. The
RAID5
> will consume the 4 old hard drives. 

Why do you think it is stupid?

I have a similar setup described in a howto at:

http://linux-raid.osdl.org/index.php/Preventing_against_a_failing_disk

How big is the iowait issue for you?

What are the performance of your raids?

> With my stupid setup, each partition gets its own /dev/mdX device. This is
> the only way I know how to do it. On the RAID10, I will need at least two
> partitions: / and swap. This means it cannot simply partition the entire
> disk Would this cause sub-optimal performance? Is there a way to make an
> underlying single RAID10 partition and place the file partitions on top?

I dont think it would be suboptimal. But try it out, both solutions and
see for yourself and report to the list your findings!).

I don't think having two raid10.f2 partitions, one for / and one for
swap, will be suboptimal, given the number of drives involved. Each 
raid will do its best for the IO, and the difference between haveing it
all on one raid, vs having it on two raids, would to be be
insignificant. Where  should the extra overhead come from? I even think
the elevator algorithm would be the same, as the elevator is per drive
(As far as I understand it).

do you think that the different raid  types make performance problems?

I think you can partition a MD device into more partitions. 
I have not tried it out for production, though and I dont know how it
performs.

I am in the process of setting up two quad-core machines with 8 GB ram 
for use as virtual servers, and intend to have rai10,f2 in the buttom of
the dom0, and then let the different virtual machines have partitions on
the raid10 array. Is this recommendable? I was thinkin of problems of
both dom0 and domu doing IO, and thus copying io buffers twice.

Best regards
keld

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Properly setting up partitions and verbose boot

['Keld Jørn Simonsen']

On Mon, Jan 26, 2009 at 11:06:32AM -0500, GeneralNMX wrote:
> 
> >From my understanding, there is fault tolerance and then there is the chance
> of a disk dying. Obviously, the more disks you have, the greater chance you
> have of a disk dying. If we assume all disks start out at some base chance
> to fail and degrade, putting multiple RAID types on the same disks can
> dramatically increase the wear & tear as the number of disks increase,
> especially when you have both a raid5 (which doesn't need to write to all
> disks, but will read from all disks) and a raid10 (which probably will write
> and read to all disks) on the same physical array of disks. Since fault
> tolerance is there to decrease the problems with disks dying, my setup is
> obviously sub-optimal. Whenever I access my RAID10, I'm also ever so
> slightly degrading my RAID5 and RAID1, and visa-versa.

Your arrangement does not increase the wear and tear, as far as I can
tell. This compared to a solution where you only have one big raid10,f2
raid. Actually your wear and  tear would be lower, because raid5 does
not write so much if you mainly deal with bigger files, and not database
like operations.

It is of cause true that when you use the raid10, then you impede the
raid5 and raid1, but you would not get more out of a single raid10,
given that you have to have the same number of operations, with the
given number of drives.

comparing your setup to a mono-raid setup, you would get about the same
wear & tear out of it. Basically you have the same number of IO
operations, and they will take the same wear and tear. With a setup like
yours, you most likely will concentrate IO on the / partition which will
be on the faster parts of the disks, which are probably also more
reliable, and you will reduce arm movement, which is good for speed and
wear and tear. In essence you will get the benefits of your raid types
- speed for raid10,f2, storage size for raid5, and wear and rear will be
as usage determines it from your raid types, I see no dramatic
difference there. I see that you think differently, and I would like us
to get some kind of conclusion on this issue. If you are right we should
add info on this in the wiki.

> Now, as for the I/O Wait, this happens when I try to access both the RAID10
> and RAID5 at the same time, especially if I'm moving a lot of data from the
> RAID10 to the RAID5.

I think this would be the same if you moved the data (copying it) within
the RAID10, or within the RAID5. Please try it out, and I would be
interested also to hear your results.

> While my server was rather old before I upgraded it
> just two days ago using spare parts (was a 1997 Supermicro Dual P3 550MHz
> 1GB SDRAM, now a 2.8GHz P4 w/ HT 1.5GB DDR), I think the I/O Wait was caused
> by trying to negotiate the three different RAID arrays at once, which
> encompass all four disks, while still allowing access to those arrays. Along
> with critical data, I also use the RAID10 as a staging area for large
> downloads from other servers due to its speed and reliability. Once I
> determine the worth of the data, I usually transfer it to the RAID5, which
> does not house critical data (more like it would be annoying if it failed).

I don't think RAID10 is much more reliable than RAID5. They are both
guaranteed to survive 1 disk crash, and then RAID10 has a further 50 %
of surviving another disk crash. RAID10,f2 would have faster sequential
read, but raid5 is faster for writing, while also having quite fast
sequential read for big files.


> Backups of / (sans large downloads) also go on the RAID5 in case the file
> systems become corrupted. Again, it would be better to have the backups on
> separate physical disks, as corrupt MFTs and seriously corrupt partitions
> could hose the entire setup.

yes, more disks in separate raids would always be better. The question
is how to best use the available number of disks.

> But I just do this all for my own enjoyment and education.

Good! I hope you enjoy it, and you can also help to educate others, by
reporting your findings here.

> I don't implement
> this stuff in production environments. As much as I'd like to convert my
> workplace's Windows XP "Server" using fake raid1 which holds all our
> super-critical data...we literally have to reboot that thing every 2-4 hours
> due to problems with other software that's on it.

Sad to hear that. Maybe, for my amusement, you could report the IO of
your windows partititions here. 

best regards
keld

> 
> 
> -----Original Message-----
> From: Keld Jørn Simonsen [mailto:keld@dkuug.dk] 
> Sent: Sunday, January 25, 2009 8:20 PM
> To: GeneralNMX
> Cc: linux-raid@vger.kernel.org
> Subject: Re: Properly setting up partitions and verbose boot
> 
> On Sun, Jan 25, 2009 at 11:18:31AM -0500, GeneralNMX wrote:
> > 
> > Currently I have a very stupid setup on my home server (not a production
> > machine). I have four hard drives with three different types of RAID
> > (1,5,10) on them setup through mdadm. I've been using this for a while
> and,
> > as you can guess, I/O Wait is a big issue for me, especially when moving
> > from different RAID types. I ordered four new hard drives to setup a
> proper
> > RAID10 by itself and I'm scrapping the RAID1, instead just consolidating /
> > into the RAID10. /boot gets its own tiny IDE HDD in a hotswap bay. The
> RAID5
> > will consume the 4 old hard drives. 
> 
> Why do you think it is stupid?
> 
> I have a similar setup described in a howto at:
> 
> http://linux-raid.osdl.org/index.php/Preventing_against_a_failing_disk
> 
> How big is the iowait issue for you?
> 
> What are the performance of your raids?
> 
> > With my stupid setup, each partition gets its own /dev/mdX device. This is
> > the only way I know how to do it. On the RAID10, I will need at least two
> > partitions: / and swap. This means it cannot simply partition the entire
> > disk Would this cause sub-optimal performance? Is there a way to make an
> > underlying single RAID10 partition and place the file partitions on top?
> 
> I dont think it would be suboptimal. But try it out, both solutions and
> see for yourself and report to the list your findings!).
> 
> I don't think having two raid10.f2 partitions, one for / and one for
> swap, will be suboptimal, given the number of drives involved. Each 
> raid will do its best for the IO, and the difference between haveing it
> all on one raid, vs having it on two raids, would to be be
> insignificant. Where  should the extra overhead come from? I even think
> the elevator algorithm would be the same, as the elevator is per drive
> (As far as I understand it).
> 
> do you think that the different raid  types make performance problems?
> 
> I think you can partition a MD device into more partitions. 
> I have not tried it out for production, though and I dont know how it
> performs.
> 
> I am in the process of setting up two quad-core machines with 8 GB ram 
> for use as virtual servers, and intend to have rai10,f2 in the buttom of
> the dom0, and then let the different virtual machines have partitions on
> the raid10 array. Is this recommendable? I was thinkin of problems of
> both dom0 and domu doing IO, and thus copying io buffers twice.
> 
> Best regards
> keld
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Properly setting up partitions and verbose boot

['Keld Jørn Simonsen']

On Tue, Jan 27, 2009 at 05:26:08AM +0100, 'Keld Jørn Simonsen' wrote:
> On Mon, Jan 26, 2009 at 11:06:32AM -0500, GeneralNMX wrote:
> > 
> > >From my understanding, there is fault tolerance and then there is the chance
> > of a disk dying. Obviously, the more disks you have, the greater chance you
> > have of a disk dying. If we assume all disks start out at some base chance
> > to fail and degrade, putting multiple RAID types on the same disks can
> > dramatically increase the wear & tear as the number of disks increase,
> > especially when you have both a raid5 (which doesn't need to write to all
> > disks, but will read from all disks) and a raid10 (which probably will write
> > and read to all disks) on the same physical array of disks. Since fault
> > tolerance is there to decrease the problems with disks dying, my setup is
> > obviously sub-optimal. Whenever I access my RAID10, I'm also ever so
> > slightly degrading my RAID5 and RAID1, and visa-versa.
> 
> Your arrangement does not increase the wear and tear, as far as I can
> tell. This compared to a solution where you only have one big raid10,f2
> raid. Actually your wear and  tear would be lower, because raid5 does
> not write so much if you mainly deal with bigger files, and not database
> like operations.
> 

Compared to raid10,f2, raid5 only writes 1/3 of the data for redundancyi
in a 4-drive setup, and it does it in a striping manner, so raid5
is quite fast for sequential writing.

> > Now, as for the I/O Wait, this happens when I try to access both the RAID10
> > and RAID5 at the same time, especially if I'm moving a lot of data from the
> > RAID10 to the RAID5.
> 
> I think this would be the same if you moved the data (copying it) within
> the RAID10, or within the RAID5. Please try it out, and I would be
> interested also to hear your results.

Of cause moving around big files is IO bound.  I think the theoretical 
best performance is sequential read time for the one raid, plus
theoretical write time for the other raid, hoping that random read/write
can be minimized. The theoretical read performance for raid10,f2 is
almost 4 times nominal read speed, and theoretical write time for the
raid5 is almost 3 times nominal speed, in your 4-drive setup. 
I tried some of it out with "cp", just on a single normal partititon,
and it looks like "cp" minimizes the random read/write.

I would be interested in hearing some performance fugures from you.

Best regards
keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html