block level vs. file level

block level vs. file level

[it]

Greetings,

There seem to be two ways of doing software RAID, and I have had trouble 
finding information on this.

First, the raid is done with partitions, for example /dev/sda1 and 
/dev/sdb1 are partitioned as Linux type and the mirroring is done then 
between the two partitions. The writing takes place on a filesystem 
level, and the partition table is not actually mirrored, because it's 
not on /dev/sda1 or /dev/sdb1.

The hardware raid does the mirroring on the block level, so it's 
actually /dev/sda mirroring /dev/sdb - the whole drive, and not 
partitions. There is a way to set this up on software raid. It takes 
more configuration tweaking, but the mirroring then includes the 
partition table as well. This way, if a drive fails, one can replace it 
without pre-partitioning it.

This also raises another point, which is relevant for both cases - same 
exact models of hard disks have different number of cylinders, so if a 
RAID partition is created on a larger drive it cannot be mirrored to a 
smaller drive.

Does anyone have any experience with this?

A.

Re: block level vs. file level

[Andy Smith]

[-- Attachment #1: Type: text/plain, Size: 1505 bytes --]

On Sun, Feb 12, 2006 at 11:31:07AM -0800, it wrote:
> The hardware raid does the mirroring on the block level, so it's 
> actually /dev/sda mirroring /dev/sdb - the whole drive, and not 
> partitions. There is a way to set this up on software raid. It takes 
> more configuration tweaking, but the mirroring then includes the 
> partition table as well. This way, if a drive fails, one can replace it 
> without pre-partitioning it.

That can be less flexible though.  If I have say 4 drives then I
quite often want small /boot, / and swap under RAID-1 then the rest
as a single large partition in RAID-5, -6 or -10 as an LVM PV.

> This also raises another point, which is relevant for both cases - same 
> exact models of hard disks have different number of cylinders, so if a 
> RAID partition is created on a larger drive it cannot be mirrored to a 
> smaller drive.

Same exact models don't usually have different block counts, but
certainly if you replace a dead drive with a different one of the
same advertised capacity you can end up getting one slightly
smaller.

> Does anyone have any experience with this?

Yes, and it's a pain, but if you have to deal with it I think the
wealth of options in md leaves you better able to handle it than
with hardware RAID.  Here's something that happened to me:

http://strugglers.net/wiki/becks.strugglers.net

-- 
http://strugglers.net/wiki/Xen_hosting -- A Xen VPS hosting hobby
Encrypted mail welcome - keyid 0x604DE5DB

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: block level vs. file level

[it]

>
>
>Yes, and it's a pain, but if you have to deal with it I think the
>wealth of options in md leaves you better able to handle it than
>with hardware RAID.  Here's something that happened to me:
>
>http://strugglers.net/wiki/becks.strugglers.net
>

Ouch.

How does hardware raid deal with this? Does it?

Thanks,
A.


Andy Smith wrote:

>On Sun, Feb 12, 2006 at 11:31:07AM -0800, it wrote:
>  
>
>>The hardware raid does the mirroring on the block level, so it's 
>>actually /dev/sda mirroring /dev/sdb - the whole drive, and not 
>>partitions. There is a way to set this up on software raid. It takes 
>>more configuration tweaking, but the mirroring then includes the 
>>partition table as well. This way, if a drive fails, one can replace it 
>>without pre-partitioning it.
>>    
>>
>
>That can be less flexible though.  If I have say 4 drives then I
>quite often want small /boot, / and swap under RAID-1 then the rest
>as a single large partition in RAID-5, -6 or -10 as an LVM PV.
>
>  
>
>>This also raises another point, which is relevant for both cases - same 
>>exact models of hard disks have different number of cylinders, so if a 
>>RAID partition is created on a larger drive it cannot be mirrored to a 
>>smaller drive.
>>    
>>
>
>Same exact models don't usually have different block counts, but
>certainly if you replace a dead drive with a different one of the
>same advertised capacity you can end up getting one slightly
>smaller.
>
>  
>
>>Does anyone have any experience with this?
>>    
>>
>
>Yes, and it's a pain, but if you have to deal with it I think the
>wealth of options in md leaves you better able to handle it than
>with hardware RAID.  Here's something that happened to me:
>
>http://strugglers.net/wiki/becks.strugglers.net
>
>  
>

Re: block level vs. file level

[Molle Bestefich]

it wrote:
> Ouch.
>
> How does hardware raid deal with this? Does it?

Hardware RAID controllers deal with this by rounding the size of
participant devices down to nearest GB, on the assumption that no
drive manufacturers would have the guts to actually sell eg. a 250 GB
drive with less than exactly 250.000.000.000 bytes of space on it.

(It would be nice if the various flavors of Linux fdisk had an option
to do this. It would be very nice if anaconda had an option to do
this.)

Re: block level vs. file level

[Bill Davidsen]

Molle Bestefich wrote:

>it wrote:
>  
>
>>Ouch.
>>
>>How does hardware raid deal with this? Does it?
>>    
>>
>
>Hardware RAID controllers deal with this by rounding the size of
>participant devices down to nearest GB, on the assumption that no
>drive manufacturers would have the guts to actually sell eg. a 250 GB
>drive with less than exactly 250.000.000.000 bytes of space on it.
>
>(It would be nice if the various flavors of Linux fdisk had an option
>to do this. It would be very nice if anaconda had an option to do
>this.)
>
I guess if you care you specify the size of the partition instead of 
"use it all." I use fdisk usually, cfdisk when installing, both let me 
set size, fdisk let's me set starting track and even play with the 
partition table's idea of geometry. What kind of an option did you have 
in mind?

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: block level vs. file level

[Molle Bestefich]

Bill Davidsen wrote:
> Molle Bestefich wrote:
> >it wrote:
> > > Ouch.
> > >
> > > How does hardware raid deal with this? Does it?
> >
> > Hardware RAID controllers deal with this by rounding the size of
> > participant devices down to nearest GB, on the assumption that no
> > drive manufacturers would have the guts to actually sell eg. a 250 GB
> > drive with less than exactly 250.000.000.000 bytes of space on it.
> >
> > (It would be nice if the various flavors of Linux fdisk had an option
> > to do this. It would be very nice if anaconda had an option to do
> > this.)
>
> I guess if you care you specify the size of the partition instead of
> "use it all." I use fdisk usually, cfdisk when installing, both let me
> set size, fdisk let's me set starting track and even play with the
> partition table's idea of geometry. What kind of an option did you have
> in mind?

I don't know.  Examples good enough?

a.) "Do not use space beyond highest GB"
b.) "Do not use last cylinder"

Help texts could be:

a.) "Helps ensure that you can replace fx. a 300GB drive from one
manufacturer with a 300GB from another."
b.) "Leave an area that Windows might use for disk metadata alone."

Re: block level vs. file level

[Bill Davidsen]

Andy Smith wrote:

>On Sun, Feb 12, 2006 at 11:31:07AM -0800, it wrote:
>  
>
>>The hardware raid does the mirroring on the block level, so it's 
>>actually /dev/sda mirroring /dev/sdb - the whole drive, and not 
>>partitions. There is a way to set this up on software raid. It takes 
>>more configuration tweaking, but the mirroring then includes the 
>>partition table as well. This way, if a drive fails, one can replace it 
>>without pre-partitioning it.
>>    
>>
>
>That can be less flexible though.  If I have say 4 drives then I
>quite often want small /boot, / and swap under RAID-1 then the rest
>as a single large partition in RAID-5, -6 or -10 as an LVM PV.
>
>  
>
>>This also raises another point, which is relevant for both cases - same 
>>exact models of hard disks have different number of cylinders, so if a 
>>RAID partition is created on a larger drive it cannot be mirrored to a 
>>smaller drive.
>>    
>>
>
>Same exact models don't usually have different block counts, but
>certainly if you replace a dead drive with a different one of the
>same advertised capacity you can end up getting one slightly
>smaller.
>
>  
>
>>Does anyone have any experience with this?
>>    
>>
>
>Yes, and it's a pain, but if you have to deal with it I think the
>wealth of options in md leaves you better able to handle it than
>with hardware RAID.  Here's something that happened to me:
>
>http://strugglers.net/wiki/becks.strugglers.net
>
>  
>
I don't quite understand your problem with fdisk, you can specify 
starting sector and then either ending sector or size, and call it any 
partition number you want. So I don't see why any other tool is required 
and what "at the end" meant in you tale.

Glad you got it working, but you may have used more solution than you 
have problem.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: block level vs. file level

[PFC]

> This also raises another point, which is relevant for both cases - same  
> exact models of hard disks have different number of cylinders, so if a  
> RAID partition is created on a larger drive it cannot be mirrored to a  
> smaller drive.

	I have a RAID5 with 5 250G drives, but some are 251 GiB (maxtors), some  
are 250.059 GiB (seagate)... say, if I started with 5 Seagates, I could  
later replace one of them with a Maxtor, but not the other way around, as  
the Seagate are just a tiny bit smaller.

	cfdisk says :

     sdb1      250994,42
     sdc1      250056,74

	I suggest, when using software raid, to create partitions that are, say,  
100 megabytes or even a gigabyte smaller than the size of the drive. You  
lose a bit of space, but if you ever need to change one, you won't feel  
stupid with a brand new drive that you can't use because it's a few  
sectors too short.

Re: block level vs. file level

[Andy Smith]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]

On Mon, Feb 13, 2006 at 09:48:49AM +0100, PFC wrote:
> 	I suggest, when using software raid, to create partitions that are, 
> 	say,  100 megabytes or even a gigabyte smaller than the size of the drive. 
> You  lose a bit of space, but if you ever need to change one, you won't 
> feel  stupid with a brand new drive that you can't use because it's a few  
> sectors too short.

After my previous experience what I tend to do now is set aside
about 2GB on each disk to use as components of a RAID-0 that I use
for scratch space (/tmp or whatever, anything that I don't care
about losing) while the machine is running.

That way if I end up by bad luck getting a slightly smaller
replacement drive then I can just do away with or shrink its RAID-0
component while keeping the other partitions the same, yet the space
is not *totally* wasted.

-- 
http://strugglers.net/wiki/Xen_hosting -- A Xen VPS hosting hobby
Encrypted mail welcome - keyid 0x604DE5DB

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]