RAID5 support ?

RAID5 support ?

[Alan Hourihane]

I read on September 23rd that NVIDIA spoke about the need to support
RAID5 in device-mapper and that RedHat was going to speak to them about
supporting it.

How far has this got ? as I can't see anything in CVS yet.

Thanks,

Alan.

Re: RAID5 support ?

[Neil Brown]

On Saturday October 22, alanh@fairlite.demon.co.uk wrote:
> I read on September 23rd that NVIDIA spoke about the need to support
> RAID5 in device-mapper and that RedHat was going to speak to them about
> supporting it.

I'm curious to know why md/raid5 is not sufficient...

More usefully though, I'd be very happy to talk about how md/raid5 can
be made to be sufficient.  I'd be happy for it to integrate more
closely with dm, if that was seen to be of value.

NeilBrown

Re: RAID5 support ?

[Alan Hourihane]

On Sat, 2005-10-22 at 23:49 +1000, Neil Brown wrote:
> On Saturday October 22, alanh@fairlite.demon.co.uk wrote:
> > I read on September 23rd that NVIDIA spoke about the need to support
> > RAID5 in device-mapper and that RedHat was going to speak to them about
> > supporting it.
> 
> I'm curious to know why md/raid5 is not sufficient...
> 
> More usefully though, I'd be very happy to talk about how md/raid5 can
> be made to be sufficient.  I'd be happy for it to integrate more
> closely with dm, if that was seen to be of value.

That'd be useful Neil.

I'll explain the problem.

I've got a SIL3114 controller with 4 x 200GB drives attached. Now that
SIL controller supports RAID5. Given that I set the RAID support up in
the BIOS I can now boot from the array.

If one of those disks die, I understand that the BIOS will still allow
me to boot from the array, even though the primary disk may have died.

In the md/raid5 setup, I'm not sure that's the case and if you lose the
primary you have to muck about with your bootloader to fix things up.

Your insights into this would be good.

Thanks,

Alan.

Re: RAID5 support ?

[Neil Brown]

On Saturday October 22, alanh@fairlite.demon.co.uk wrote:
> > More usefully though, I'd be very happy to talk about how md/raid5 can
> > be made to be sufficient.  I'd be happy for it to integrate more
> > closely with dm, if that was seen to be of value.
> 
> That'd be useful Neil.
> 
> I'll explain the problem.
> 
> I've got a SIL3114 controller with 4 x 200GB drives attached. Now that
> SIL controller supports RAID5. Given that I set the RAID support up in
> the BIOS I can now boot from the array.
> 
> If one of those disks die, I understand that the BIOS will still allow
> me to boot from the array, even though the primary disk may have died.
> 
> In the md/raid5 setup, I'm not sure that's the case and if you lose the
> primary you have to muck about with your bootloader to fix things up.

It seems the core problem here is that you need soft-raid5 in Linux
which can work with the metadata that is stored by the BIOS on the SIL
controller. 
This shouldn't be too hard to do, providing it is reasonably
documented.
'md' has all the meta-data operations reasonably well factored out, so
working with new formats shouldn't be difficult.

I suspect that it would be best to have the code for understanding the
metadata run in user-space rather than in the kernel - I gather that
is what dmraid does.

For raid5, we really need synchronous metadata updates when a device
fails, as it is not really safe to write anything after the decision
to fail a device, and before the metadata has been updated.

I am currently working on adding sysfs support to md and raid5 and
would prefer to use this as the interface between md and a user-space
metadata handler (though I could probably be convinced to work under
the dm ioctls as well if that was important).

So the enhancements that seem to be needed to md/raid5 would include:

1/ Introduce a new metadata type which the kernel doesn't read or
   write at all.  When a write is required, it signals userspace
   somehow, and blocks writes until it is told to continue.

2/ Allow all config information to be provided by userspace.  The
   current SET_ARRAY_INFO is not quite up to the task.  e.g. you 
   cannot give a device offset through that interface.


I plan to do (2) anyway, probably through sysfs, but maybe configfs -
I'm not sure yet.

(1) probably needs a bit more thought and some understanding on what
the userspace metadata tool would require.
I imagine having an event counter which is updated whenever a
metadata update is required.
The userspace tool would
  - read a number from the event-counter file
  - extract all the metadata information needed from sysfs
  - write it to the devices
  - write the original event-count to some other sysfs file.

The kernel would not allow further writes until the number written
to the second file matches the most current event counter, thus if
multiple events happened while the metadata was being updated, we
still wouldn't get out of sync.

Of course, we wouldn't want to have to poll the event-counter
file.  We would need some more direct notification of change.  As
I am using sysfs, maybe some sort of hot-plug event... but I'll
have to learn more about hot plug events first.


Does any of this sound useful?
Any other suggestions?

NeilBrown

Re: RAID5 support ?

[Alan Hourihane]

On Mon, 2005-10-24 at 09:54 +1000, Neil Brown wrote:
> On Saturday October 22, alanh@fairlite.demon.co.uk wrote:
> > > More usefully though, I'd be very happy to talk about how md/raid5 can
> > > be made to be sufficient.  I'd be happy for it to integrate more
> > > closely with dm, if that was seen to be of value.
> > 
> > That'd be useful Neil.
> > 
> > I'll explain the problem.
> > 
> > I've got a SIL3114 controller with 4 x 200GB drives attached. Now that
> > SIL controller supports RAID5. Given that I set the RAID support up in
> > the BIOS I can now boot from the array.
> > 
> > If one of those disks die, I understand that the BIOS will still allow
> > me to boot from the array, even though the primary disk may have died.
> > 
> > In the md/raid5 setup, I'm not sure that's the case and if you lose the
> > primary you have to muck about with your bootloader to fix things up.
> 
> It seems the core problem here is that you need soft-raid5 in Linux
> which can work with the metadata that is stored by the BIOS on the SIL
> controller. 
> This shouldn't be too hard to do, providing it is reasonably
> documented.
> 'md' has all the meta-data operations reasonably well factored out, so
> working with new formats shouldn't be difficult.

That to me still doesn't resolve the bootloader problems.

Regardless of the md understanding the meta-data operations, it deals
with partitions and therefore the boot sector of each drive in an md
setup is outside the scope of it's operation.

Unless I'm missing something, this is where device-mapper really is
useful, and presents a single drive to linux.

Alan.

Re: RAID5 support ?

[Neil Brown]

On Monday October 24, alanh@fairlite.demon.co.uk wrote:
> > 'md' has all the meta-data operations reasonably well factored out, so
> > working with new formats shouldn't be difficult.
> 
> That to me still doesn't resolve the bootloader problems.
> 
> Regardless of the md understanding the meta-data operations, it deals
> with partitions and therefore the boot sector of each drive in an md
> setup is outside the scope of it's operation.

md doesn't work with partitions.  It works with devices which may be
partitions or may be the whole device.
I often make raid5 arrays out of a set of devices, none of them
partitioned. 

NeilBrown

Re: RAID5 support ?

[Alan Hourihane]

On Thu, 2005-11-03 at 14:17 +1100, Neil Brown wrote:
> On Monday October 24, alanh@fairlite.demon.co.uk wrote:
> > > 'md' has all the meta-data operations reasonably well factored out, so
> > > working with new formats shouldn't be difficult.
> > 
> > That to me still doesn't resolve the bootloader problems.
> > 
> > Regardless of the md understanding the meta-data operations, it deals
> > with partitions and therefore the boot sector of each drive in an md
> > setup is outside the scope of it's operation.
> 
> md doesn't work with partitions.  It works with devices which may be
> partitions or may be the whole device.
> I often make raid5 arrays out of a set of devices, none of them
> partitioned. 

O.k. Can you explain to me how to do that and still boot from the md
device ?

Alan.

Re: RAID5 support ?

[Neil Brown]

On Thursday November 3, alanh@fairlite.demon.co.uk wrote:
> > md doesn't work with partitions.  It works with devices which may be
> > partitions or may be the whole device.
> > I often make raid5 arrays out of a set of devices, none of them
> > partitioned. 
> 
> O.k. Can you explain to me how to do that and still boot from the md
> device ?

It depends on exactly what you mean by 'still boot from'...

If you mean, "Have to root filesystem on an md device", there are two
ways. 

1/ use a kernel parameter like:
     md=d0,/dev/hda,/dev/hdc root=/dev/md_d0p1

 This tells the kernel to assemble an md array from /dev/hda and
 /dev/hdc.  It will be /dev/md/d0 which is a partitionable array,
 so you can have normal partitioning. /dev/md/d0p1 /dev/md0/d0p2.
 Most of my servers I configure like this, with a raid1 pair on sda
 and sdb, partitioned into root, swap, and the rest.

 This works quite well for ide drives as the device names are stable.
 It is not so good for SCSI as device names can change if you pull a
 device out.  However using sda and sdb for the root device works ok.

2/ use an initrd or initfs image which contains code to find and
  assemble md/raid arrays. mdadm comes with a striped down
  'mdassemble' which does just this.
  It currently needs a config file in the initfs, but I'm working
  towards being able to have the important configuration information
  reliably in the raid superblock.


If you mean "Load the kernel image from an md device" (which is what
"boot from" really means to me), then it depends on your boot loader.
I have lilo configured to boot from a mirrored pair as above.  Booting
from a raid5 almost certainly requires bios support, but once the
kernel and initfs are loaded, md can take it from there (once it can
understand the metadata format).

Hope that helps,

NeilBrown

Re: RAID5 support ?

[Alan Hourihane]

On Thu, 2005-11-03 at 21:07 +1100, Neil Brown wrote:
> On Thursday November 3, alanh@fairlite.demon.co.uk wrote:
> > > md doesn't work with partitions.  It works with devices which may be
> > > partitions or may be the whole device.
> > > I often make raid5 arrays out of a set of devices, none of them
> > > partitioned. 
> > 
> > O.k. Can you explain to me how to do that and still boot from the md
> > device ?
> 
> It depends on exactly what you mean by 'still boot from'...
> 
> If you mean, "Have to root filesystem on an md device", there are two
> ways. 
> 
> 1/ use a kernel parameter like:
>      md=d0,/dev/hda,/dev/hdc root=/dev/md_d0p1
> 
>  This tells the kernel to assemble an md array from /dev/hda and
>  /dev/hdc.  It will be /dev/md/d0 which is a partitionable array,
>  so you can have normal partitioning. /dev/md/d0p1 /dev/md0/d0p2.
>  Most of my servers I configure like this, with a raid1 pair on sda
>  and sdb, partitioned into root, swap, and the rest.
> 
>  This works quite well for ide drives as the device names are stable.
>  It is not so good for SCSI as device names can change if you pull a
>  device out.  However using sda and sdb for the root device works ok.
> 
> 2/ use an initrd or initfs image which contains code to find and
>   assemble md/raid arrays. mdadm comes with a striped down
>   'mdassemble' which does just this.
>   It currently needs a config file in the initfs, but I'm working
>   towards being able to have the important configuration information
>   reliably in the raid superblock.
> 
> 
> If you mean "Load the kernel image from an md device" (which is what
> "boot from" really means to me), then it depends on your boot loader.
> I have lilo configured to boot from a mirrored pair as above.  Booting
> from a raid5 almost certainly requires bios support, but once the
> kernel and initfs are loaded, md can take it from there (once it can
> understand the metadata format).

I mean "Load the kernel image from an md device" which is what it means
to me too.

So, you go on to say that booting from raid5 almost certainly requires
bios support - so I'm back to needing the onboard Sil 3114 BIOS to
handle that and then get md/dmraid to sort the rest out - which gets
back to my original question on when device mapper would get RAID5
support so that dmraid can handle the SIL RAID5 metadata.

Alan.

Re: RAID5 support ?

[Neil Brown]

On Thursday November 3, alanh@fairlite.demon.co.uk wrote:
> 
> I mean "Load the kernel image from an md device" which is what it means
> to me too.
> 
> So, you go on to say that booting from raid5 almost certainly requires
> bios support - so I'm back to needing the onboard Sil 3114 BIOS to
> handle that and then get md/dmraid to sort the rest out - which gets
> back to my original question on when device mapper would get RAID5
> support so that dmraid can handle the SIL RAID5 metadata.

"device mapper" and "dmraid" are of course different things.  
One is a kernel subsystem, the other is a configuration tool.
For dmraid to be able to handle SIL RAID5, it is not necessary for
"device mapper" to get RAID5 support, only for dmraid to have
appropriate access to some sort of raid5 support in the Linux kernel.

I would like md/raid5 (which is quite separate from device mapper) to
be able to provide the required level of support so that if the dmraid
developers so chose, they could use it rather than needing a second
raid5 implementation in the kernel.  Hence my original question, asking
 "how md/raid5 can be made to be sufficient" for the needs for dmraid?

I suspect all it needs is to be able to interface with a user-space
metadata handler, and possibly some extra configuration options to
match the expected functionality of the SiL.
The former I hope be possible by 2.6.16.  The later I'm interested in
advice on.

NeilBrown

Re: RAID5 support ?

[Alan Hourihane]

On Thu, 2005-11-03 at 22:20 +1100, Neil Brown wrote:
> On Thursday November 3, alanh@fairlite.demon.co.uk wrote:
> > 
> > I mean "Load the kernel image from an md device" which is what it means
> > to me too.
> > 
> > So, you go on to say that booting from raid5 almost certainly requires
> > bios support - so I'm back to needing the onboard Sil 3114 BIOS to
> > handle that and then get md/dmraid to sort the rest out - which gets
> > back to my original question on when device mapper would get RAID5
> > support so that dmraid can handle the SIL RAID5 metadata.
> 
> "device mapper" and "dmraid" are of course different things.  
> One is a kernel subsystem, the other is a configuration tool.
> For dmraid to be able to handle SIL RAID5, it is not necessary for
> "device mapper" to get RAID5 support, only for dmraid to have
> appropriate access to some sort of raid5 support in the Linux kernel.
> 
> I would like md/raid5 (which is quite separate from device mapper) to
> be able to provide the required level of support so that if the dmraid
> developers so chose, they could use it rather than needing a second
> raid5 implementation in the kernel.  Hence my original question, asking
>  "how md/raid5 can be made to be sufficient" for the needs for dmraid?

So what exactly is Heinz working on if it isn't support for RAID5 in
device-mapper ?

> I suspect all it needs is to be able to interface with a user-space
> metadata handler, and possibly some extra configuration options to
> match the expected functionality of the SiL.
> The former I hope be possible by 2.6.16.  The later I'm interested in
> advice on.

I'll defer to Heinz to make that choice - he's a lot closer to the code.

Alan.

Re: RAID5 support ?

[Heinz Mauelshagen]

On Thu, Nov 03, 2005 at 11:39:43AM +0000, Alan Hourihane wrote:
> On Thu, 2005-11-03 at 22:20 +1100, Neil Brown wrote:
> > On Thursday November 3, alanh@fairlite.demon.co.uk wrote:
> > > 
> > > I mean "Load the kernel image from an md device" which is what it means
> > > to me too.
> > > 
> > > So, you go on to say that booting from raid5 almost certainly requires
> > > bios support - so I'm back to needing the onboard Sil 3114 BIOS to
> > > handle that and then get md/dmraid to sort the rest out - which gets
> > > back to my original question on when device mapper would get RAID5
> > > support so that dmraid can handle the SIL RAID5 metadata.
> > 
> > "device mapper" and "dmraid" are of course different things.  
> > One is a kernel subsystem, the other is a configuration tool.
> > For dmraid to be able to handle SIL RAID5, it is not necessary for
> > "device mapper" to get RAID5 support, only for dmraid to have
> > appropriate access to some sort of raid5 support in the Linux kernel.
> > 
> > I would like md/raid5 (which is quite separate from device mapper) to
> > be able to provide the required level of support so that if the dmraid
> > developers so chose, they could use it rather than needing a second
> > raid5 implementation in the kernel.  Hence my original question, asking
> >  "how md/raid5 can be made to be sufficient" for the needs for dmraid?
> 
> So what exactly is Heinz working on if it isn't support for RAID5 in
> device-mapper ?

It is a RAID4 and RAID5 device-mapper target.

> 
> > I suspect all it needs is to be able to interface with a user-space
> > metadata handler, and possibly some extra configuration options to
> > match the expected functionality of the SiL.
> > The former I hope be possible by 2.6.16.  The later I'm interested in
> > advice on.
> 
> I'll defer to Heinz to make that choice - he's a lot closer to the code.

The device-mapper kernel subsystem aims to avoid multiple drivers for
block device virtualization being implemented seperately in the Linux
kernel by adding device-mapper targets as we go.

Implementing a target for RAID4 and RAID5 is just the long term approach
to get more mappings supported by device-mapper in order to minimize code
duplication.

> 
> Alan.
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

-- 

Regards,
Heinz    -- The LVM Guy --

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Red Hat GmbH
Consulting Development Engineer                   Am Sonnenhang 11
Cluster and Storage Development                   56242 Marienrachdorf
                                                  Germany
Mauelshagen@RedHat.com                            +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: RAID5 support ?

[Heinz Mauelshagen]

On Mon, Oct 24, 2005 at 09:54:12AM +1000, Neil Brown wrote:
> On Saturday October 22, alanh@fairlite.demon.co.uk wrote:
> > > More usefully though, I'd be very happy to talk about how md/raid5 can
> > > be made to be sufficient.  I'd be happy for it to integrate more
> > > closely with dm, if that was seen to be of value.
> > 
> > That'd be useful Neil.
> > 
> > I'll explain the problem.
> > 
> > I've got a SIL3114 controller with 4 x 200GB drives attached. Now that
> > SIL controller supports RAID5. Given that I set the RAID support up in
> > the BIOS I can now boot from the array.
> > 
> > If one of those disks die, I understand that the BIOS will still allow
> > me to boot from the array, even though the primary disk may have died.
> > 
> > In the md/raid5 setup, I'm not sure that's the case and if you lose the
> > primary you have to muck about with your bootloader to fix things up.
> 
> It seems the core problem here is that you need soft-raid5 in Linux
> which can work with the metadata that is stored by the BIOS on the SIL
> controller. 
> This shouldn't be too hard to do, providing it is reasonably
> documented.
> 'md' has all the meta-data operations reasonably well factored out, so
> working with new formats shouldn't be difficult.
> 
> I suspect that it would be best to have the code for understanding the
> metadata run in user-space rather than in the kernel - I gather that
> is what dmraid does.

Correct. It uses device-mapper, which lags RAID4 + 5 mappings so far, but I'm
working on this. Having those, we can cover the RAID5 ATARAID case for
many different ATARAID solutions in the given device-mapper/dmraid framwork.

Once I have first presentable code for a device-mapper RAID4 + 5 target
(hopefully next week after my return from te US), I'ld appreciate your
help on it.


> 
> For raid5, we really need synchronous metadata updates when a device
> fails, as it is not really safe to write anything after the decision
> to fail a device, and before the metadata has been updated.

Yes, we need to store the information, which device failed, persistently
in order to identify it after a crash. In device-mapper, we have
IO suspend support to make that happen.

FYI: we keep information about which regions (arbitrary sized segments
     of the address space) of the set are dirty with the the
     device-mapper dirty-log so that we can resynchonize those at set startup.

> 
> I am currently working on adding sysfs support to md and raid5 and
> would prefer to use this as the interface between md and a user-space
> metadata handler (though I could probably be convinced to work under
> the dm ioctls as well if that was important).
> 
> So the enhancements that seem to be needed to md/raid5 would include:
> 
> 1/ Introduce a new metadata type which the kernel doesn't read or
>    write at all.  When a write is required, it signals userspace
>    somehow, and blocks writes until it is told to continue.

That's the default with device-mapper, which doesn't read/write any metadata
but keeps it to userspace.

> 
> 2/ Allow all config information to be provided by userspace.  The
>    current SET_ARRAY_INFO is not quite up to the task.  e.g. you 
>    cannot give a device offset through that interface.
> 
> 
> I plan to do (2) anyway, probably through sysfs, but maybe configfs -
> I'm not sure yet.
> 
> (1) probably needs a bit more thought and some understanding on what
> the userspace metadata tool would require.
> I imagine having an event counter which is updated whenever a
> metadata update is required.
> The userspace tool would
>   - read a number from the event-counter file
>   - extract all the metadata information needed from sysfs
>   - write it to the devices
>   - write the original event-count to some other sysfs file.

We do have a dmeventd in libdevmapper already, which can be used to
cover this. Applications can register any mapped device with dmeventd
to be monitored. dmeventd will call into a shared library on any device
event (eg, failure). The library can carry out arbitrary scenarious
such as yours above.

> 
> The kernel would not allow further writes until the number written
> to the second file matches the most current event counter, thus if
> multiple events happened while the metadata was being updated, we
> still wouldn't get out of sync.
> 
> Of course, we wouldn't want to have to poll the event-counter
> file.  We would need some more direct notification of change.  As
> I am using sysfs, maybe some sort of hot-plug event... but I'll
> have to learn more about hot plug events first.
> 
> 
> Does any of this sound useful?
> Any other suggestions?
> 
> NeilBrown
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

-- 

Regards,
Heinz    -- The LVM Guy --

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Red Hat GmbH
Consulting Development Engineer                   Am Sonnenhang 11
Cluster and Storage Development                   56242 Marienrachdorf
                                                  Germany
Mauelshagen@RedHat.com                            +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: RAID5 support ?

[Molle Bestefich]

Heinz Mauelshagen wrote:
> FYI: we keep information about which regions (arbitrary sized segments
>      of the address space) of the set are dirty with the the
>      device-mapper dirty-log so that we can resynchonize those at set startup.

Sounds sort of nasty.
What happens if the user boots to Windows in the mean time?
Windows will see corrupted data?

(I mean, that's the whole point of using a hardware soft-RAID
solution, isn't it?  BIOS and multiple-OS support for the metadata
format / RAID arrays?)

RE: RAID5 support ?

[Jane Liu]

> -----Original Message-----
> From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of Alan Hourihane
> Sent: Saturday, October 22, 2005 4:41 AM
> To: dm-devel@redhat.com
> Subject: [dm-devel] RAID5 support ?
> 
> I read on September 23rd that NVIDIA spoke about the need to support
> RAID5 in device-mapper and that RedHat was going to speak to them
about
> supporting it.
> 
> How far has this got ? as I can't see anything in CVS yet.

We are working closely with Heinz Mauelshagen of RedHat on RAID5
support. Heinz is working hard to get a RAID5 target ready for testing.
We don't know the exact schedule yet, but do hope to have something
reasonably stable within a couple months. Sorry our efforts won't help
with your immediate needs. Please stay tuned...

Regards,
Jane Liu
--------
NVIDIA Corporation