ANNOUNCE: mdadm 2.0-devel-3 DEVELOPMENT RELEASE - A tool for managing Soft RAID under Linux

ANNOUNCE: mdadm 2.0-devel-3 DEVELOPMENT RELEASE - A tool for managing Soft RAID under Linux

[Neil Brown]

I am pleased to announce the availability of 
   mdadm version 2.0-devel-3
It is available at
   http://www.cse.unsw.edu.au/~neilb/source/mdadm/
and
   http://www.{countrycode}.kernel.org/pub/linux/utils/raid/mdadm/

This is just another devel release with a bunch of bug-fixes.
It has one interesting feature in that I have started a test-suite.

If you find a bug and report it, that would be great.
If you provide a test as well, that would be fantastic!


Development of mdadm is now sponsored by Suse Labs, Novell Inc.

Re: RAID5 question.

[David Greaves]

And notice you can apply different readahead to:
The raw devices (/dev/sda)
The md device (/dev/mdX)
Any lvm device (/dev/lvm_name/lvm_device)

David

Raz Ben Jehuda wrote:

>read the blockdev man page
>
>On Thu, 2005-08-04 at 16:06 +0200, djani22@dynamicweb.hu wrote:
>  
>
>>Hi list, Neil!
>>
>>I have a little question because I'v got some performance problem...
>>Is the RAID5 do any type of headahead with the default config?
>>It is possible to disable the readahead? (if it is.)
>>Or just the io_sched do that on the disks?
>>
>>How can I disable the whole headahead on top of the RAID5 array?
>>
>>The RAID5 can do multiple small (<4kb) reads of disks for the multithreaded
>>read requests?
>>(chunk size is 32k)
>>
>>Thanks for helping!
>>
>>Janos
>>
>>-
>>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>    
>>


-- 

Re: RAID5 question.

[djani22]

Thanks a lot for you, and Raz!

The raw devices readahead I already set with the hdparm, and
/sys/block/*/queue/read_ahead_kb, but the md device is only changeable with
blockdev. :-)
I dont use lvm, because it is too slow for me....

But unfortunately the problem is still here for me. :(
I continue searching the bottleneck...

Thanks!

Janos

----- Original Message -----
From: "David Greaves" <david@dgreaves.com>
To: "Raz Ben Jehuda" <razb@bitband.com>
Cc: <djani22@dynamicweb.hu>; <linux-raid@vger.kernel.org>
Sent: Thursday, August 04, 2005 4:56 PM
Subject: Re: RAID5 question.


> And notice you can apply different readahead to:
> The raw devices (/dev/sda)
> The md device (/dev/mdX)
> Any lvm device (/dev/lvm_name/lvm_device)
>
> David
>
> Raz Ben Jehuda wrote:
>
> >read the blockdev man page
> >
> >On Thu, 2005-08-04 at 16:06 +0200, djani22@dynamicweb.hu wrote:
> >
> >
> >>Hi list, Neil!
> >>
> >>I have a little question because I'v got some performance problem...
> >>Is the RAID5 do any type of headahead with the default config?
> >>It is possible to disable the readahead? (if it is.)
> >>Or just the io_sched do that on the disks?
> >>
> >>How can I disable the whole headahead on top of the RAID5 array?
> >>
> >>The RAID5 can do multiple small (<4kb) reads of disks for the
multithreaded
> >>read requests?
> >>(chunk size is 32k)
> >>
> >>Thanks for helping!
> >>
> >>Janos
> >>
> >>-
> >>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >>the body of a message to majordomo@vger.kernel.org
> >>More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >>
>
>
> --
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 question.

[Raz Ben Jehuda]

take a look at /proc/sys/dev/raid/speed_limit_max . it is kilobytes if i
recall correctly.

On Thu, 2005-08-04 at 17:45 +0200, djani22@dynamicweb.hu wrote:
> Thanks a lot for you, and Raz!
> 
> The raw devices readahead I already set with the hdparm, and
> /sys/block/*/queue/read_ahead_kb, but the md device is only changeable with
> blockdev. :-)
> I dont use lvm, because it is too slow for me....
> 
> But unfortunately the problem is still here for me. :(
> I continue searching the bottleneck...
> 
> Thanks!
> 
> Janos
> 
> ----- Original Message -----
> From: "David Greaves" <david@dgreaves.com>
> To: "Raz Ben Jehuda" <razb@bitband.com>
> Cc: <djani22@dynamicweb.hu>; <linux-raid@vger.kernel.org>
> Sent: Thursday, August 04, 2005 4:56 PM
> Subject: Re: RAID5 question.
> 
> 
> > And notice you can apply different readahead to:
> > The raw devices (/dev/sda)
> > The md device (/dev/mdX)
> > Any lvm device (/dev/lvm_name/lvm_device)
> >
> > David
> >
> > Raz Ben Jehuda wrote:
> >
> > >read the blockdev man page
> > >
> > >On Thu, 2005-08-04 at 16:06 +0200, djani22@dynamicweb.hu wrote:
> > >
> > >
> > >>Hi list, Neil!
> > >>
> > >>I have a little question because I'v got some performance problem...
> > >>Is the RAID5 do any type of headahead with the default config?
> > >>It is possible to disable the readahead? (if it is.)
> > >>Or just the io_sched do that on the disks?
> > >>
> > >>How can I disable the whole headahead on top of the RAID5 array?
> > >>
> > >>The RAID5 can do multiple small (<4kb) reads of disks for the
> multithreaded
> > >>read requests?
> > >>(chunk size is 32k)
> > >>
> > >>Thanks for helping!
> > >>
> > >>Janos
> > >>
> > >>-
> > >>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > >>the body of a message to majordomo@vger.kernel.org
> > >>More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >>
> > >>
> >
> >
> > --
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Raz
Long live the penguin

Re: RAID5 question.

[djani22]

Yes, it is there, I know it.
But its only for resync or not? :-)

----- Original Message -----
From: "Raz Ben Jehuda" <razb@bitband.com>
To: <djani22@dynamicweb.hu>
Cc: "David Greaves" <david@dgreaves.com>; <linux-raid@vger.kernel.org>
Sent: Thursday, August 04, 2005 5:56 PM
Subject: Re: RAID5 question.


> take a look at /proc/sys/dev/raid/speed_limit_max . it is kilobytes if i
> recall correctly.
>
> On Thu, 2005-08-04 at 17:45 +0200, djani22@dynamicweb.hu wrote:
> > Thanks a lot for you, and Raz!
> >
> > The raw devices readahead I already set with the hdparm, and
> > /sys/block/*/queue/read_ahead_kb, but the md device is only changeable
with
> > blockdev. :-)
> > I dont use lvm, because it is too slow for me....
> >
> > But unfortunately the problem is still here for me. :(
> > I continue searching the bottleneck...
> >
> > Thanks!
> >
> > Janos
> >
> > ----- Original Message -----
> > From: "David Greaves" <david@dgreaves.com>
> > To: "Raz Ben Jehuda" <razb@bitband.com>
> > Cc: <djani22@dynamicweb.hu>; <linux-raid@vger.kernel.org>
> > Sent: Thursday, August 04, 2005 4:56 PM
> > Subject: Re: RAID5 question.
> >
> >
> > > And notice you can apply different readahead to:
> > > The raw devices (/dev/sda)
> > > The md device (/dev/mdX)
> > > Any lvm device (/dev/lvm_name/lvm_device)
> > >
> > > David
> > >
> > > Raz Ben Jehuda wrote:
> > >
> > > >read the blockdev man page
> > > >
> > > >On Thu, 2005-08-04 at 16:06 +0200, djani22@dynamicweb.hu wrote:
> > > >
> > > >
> > > >>Hi list, Neil!
> > > >>
> > > >>I have a little question because I'v got some performance problem...
> > > >>Is the RAID5 do any type of headahead with the default config?
> > > >>It is possible to disable the readahead? (if it is.)
> > > >>Or just the io_sched do that on the disks?
> > > >>
> > > >>How can I disable the whole headahead on top of the RAID5 array?
> > > >>
> > > >>The RAID5 can do multiple small (<4kb) reads of disks for the
> > multithreaded
> > > >>read requests?
> > > >>(chunk size is 32k)
> > > >>
> > > >>Thanks for helping!
> > > >>
> > > >>Janos
> > > >>
> > > >>-
> > > >>To unsubscribe from this list: send the line "unsubscribe
linux-raid" in
> > > >>the body of a message to majordomo@vger.kernel.org
> > > >>More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > >>
> > > >>
> > >
> > >
> > > --
> > >
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> Raz
> Long live the penguin
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 question.

[Raz Ben Jehuda]

how is the disks performance ? is it OK ?

On Thu, 2005-08-04 at 18:13 +0200, djani22@dynamicweb.hu wrote:
> Yes, it is there, I know it.
> But its only for resync or not? :-)
> 
> ----- Original Message -----
> From: "Raz Ben Jehuda" <razb@bitband.com>
> To: <djani22@dynamicweb.hu>
> Cc: "David Greaves" <david@dgreaves.com>; <linux-raid@vger.kernel.org>
> Sent: Thursday, August 04, 2005 5:56 PM
> Subject: Re: RAID5 question.
> 
> 
> > take a look at /proc/sys/dev/raid/speed_limit_max . it is kilobytes if i
> > recall correctly.
> >
> > On Thu, 2005-08-04 at 17:45 +0200, djani22@dynamicweb.hu wrote:
> > > Thanks a lot for you, and Raz!
> > >
> > > The raw devices readahead I already set with the hdparm, and
> > > /sys/block/*/queue/read_ahead_kb, but the md device is only changeable
> with
> > > blockdev. :-)
> > > I dont use lvm, because it is too slow for me....
> > >
> > > But unfortunately the problem is still here for me. :(
> > > I continue searching the bottleneck...
> > >
> > > Thanks!
> > >
> > > Janos
> > >
> > > ----- Original Message -----
> > > From: "David Greaves" <david@dgreaves.com>
> > > To: "Raz Ben Jehuda" <razb@bitband.com>
> > > Cc: <djani22@dynamicweb.hu>; <linux-raid@vger.kernel.org>
> > > Sent: Thursday, August 04, 2005 4:56 PM
> > > Subject: Re: RAID5 question.
> > >
> > >
> > > > And notice you can apply different readahead to:
> > > > The raw devices (/dev/sda)
> > > > The md device (/dev/mdX)
> > > > Any lvm device (/dev/lvm_name/lvm_device)
> > > >
> > > > David
> > > >
> > > > Raz Ben Jehuda wrote:
> > > >
> > > > >read the blockdev man page
> > > > >
> > > > >On Thu, 2005-08-04 at 16:06 +0200, djani22@dynamicweb.hu wrote:
> > > > >
> > > > >
> > > > >>Hi list, Neil!
> > > > >>
> > > > >>I have a little question because I'v got some performance problem...
> > > > >>Is the RAID5 do any type of headahead with the default config?
> > > > >>It is possible to disable the readahead? (if it is.)
> > > > >>Or just the io_sched do that on the disks?
> > > > >>
> > > > >>How can I disable the whole headahead on top of the RAID5 array?
> > > > >>
> > > > >>The RAID5 can do multiple small (<4kb) reads of disks for the
> > > multithreaded
> > > > >>read requests?
> > > > >>(chunk size is 32k)
> > > > >>
> > > > >>Thanks for helping!
> > > > >>
> > > > >>Janos
> > > > >>
> > > > >>-
> > > > >>To unsubscribe from this list: send the line "unsubscribe
> linux-raid" in
> > > > >>the body of a message to majordomo@vger.kernel.org
> > > > >>More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > > >>
> > > > >>
> > > >
> > > >
> > > > --
> > > >
> > > > -
> > > > To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > >
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > Raz
> > Long live the penguin
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
-- 
Raz
Long live the penguin

Re: RAID5 question.

[djani22]

The answer is more difficult!

I use RAID-s in 3 layers!

Layer 1 : (RAID5 in 11+1 raw disk in 1 pc) x4 (4 disk nodes)
Layer 2 : RAID1 4x 1/2 nodes (only for ability to backup complete nodes)
Layer 3 : RAID0 to 4 nodes -> 1 big 8TB disk. :-)

I use gnbd for this.
The gnbd sends only small packets, and I think, the too much readahead is
the problem, because the whole 8TB array is phisically stripped to 44 raw
disks, but NOT!
When I disable the readahead, the performace is more worse.
Now  I grow the readahead in the raw disk to 1MB, the raid5 to 10MB, the
raid1 to 1MB, and the raid0 to 8MB, and the performance is great now! :-)

(web+ftp  server for ~200-300 people at one time, downloading a lot of
small(1-60KB) and big (~15-50MB) files)

Thansk for you all!

----- Original Message -----
From: "Raz Ben Jehuda" <razb@bitband.com>
To: <djani22@dynamicweb.hu>
Cc: <linux-raid@vger.kernel.org>
Sent: Thursday, August 04, 2005 6:25 PM
Subject: Re: RAID5 question.


> how is the disks performance ? is it OK ?
>
> On Thu, 2005-08-04 at 18:13 +0200, djani22@dynamicweb.hu wrote:
> > Yes, it is there, I know it.
> > But its only for resync or not? :-)
> >
> > ----- Original Message -----
> > From: "Raz Ben Jehuda" <razb@bitband.com>
> > To: <djani22@dynamicweb.hu>
> > Cc: "David Greaves" <david@dgreaves.com>; <linux-raid@vger.kernel.org>
> > Sent: Thursday, August 04, 2005 5:56 PM
> > Subject: Re: RAID5 question.
> >
> >
> > > take a look at /proc/sys/dev/raid/speed_limit_max . it is kilobytes if
i
> > > recall correctly.
> > >
> > > On Thu, 2005-08-04 at 17:45 +0200, djani22@dynamicweb.hu wrote:
> > > > Thanks a lot for you, and Raz!
> > > >
> > > > The raw devices readahead I already set with the hdparm, and
> > > > /sys/block/*/queue/read_ahead_kb, but the md device is only
changeable
> > with
> > > > blockdev. :-)
> > > > I dont use lvm, because it is too slow for me....
> > > >
> > > > But unfortunately the problem is still here for me. :(
> > > > I continue searching the bottleneck...
> > > >
> > > > Thanks!
> > > >
> > > > Janos
> > > >
> > > > ----- Original Message -----
> > > > From: "David Greaves" <david@dgreaves.com>
> > > > To: "Raz Ben Jehuda" <razb@bitband.com>
> > > > Cc: <djani22@dynamicweb.hu>; <linux-raid@vger.kernel.org>
> > > > Sent: Thursday, August 04, 2005 4:56 PM
> > > > Subject: Re: RAID5 question.
> > > >
> > > >
> > > > > And notice you can apply different readahead to:
> > > > > The raw devices (/dev/sda)
> > > > > The md device (/dev/mdX)
> > > > > Any lvm device (/dev/lvm_name/lvm_device)
> > > > >
> > > > > David
> > > > >
> > > > > Raz Ben Jehuda wrote:
> > > > >
> > > > > >read the blockdev man page
> > > > > >
> > > > > >On Thu, 2005-08-04 at 16:06 +0200, djani22@dynamicweb.hu wrote:
> > > > > >
> > > > > >
> > > > > >>Hi list, Neil!
> > > > > >>
> > > > > >>I have a little question because I'v got some performance
problem...
> > > > > >>Is the RAID5 do any type of headahead with the default config?
> > > > > >>It is possible to disable the readahead? (if it is.)
> > > > > >>Or just the io_sched do that on the disks?
> > > > > >>
> > > > > >>How can I disable the whole headahead on top of the RAID5 array?
> > > > > >>
> > > > > >>The RAID5 can do multiple small (<4kb) reads of disks for the
> > > > multithreaded
> > > > > >>read requests?
> > > > > >>(chunk size is 32k)
> > > > > >>
> > > > > >>Thanks for helping!
> > > > > >>
> > > > > >>Janos
> > > > > >>
> > > > > >>-
> > > > > >>To unsubscribe from this list: send the line "unsubscribe
> > linux-raid" in
> > > > > >>the body of a message to majordomo@vger.kernel.org
> > > > > >>More majordomo info at
http://vger.kernel.org/majordomo-info.html
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > -
> > > > > To unsubscribe from this list: send the line "unsubscribe
linux-raid"
> > in
> > > > > the body of a message to majordomo@vger.kernel.org
> > > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > >
> > > > -
> > > > To unsubscribe from this list: send the line "unsubscribe
linux-raid" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > --
> > > Raz
> > > Long live the penguin
> > >
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> --
> Raz
> Long live the penguin
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 question.

[Neil Brown]

On Thursday August 4, djani22@dynamicweb.hu wrote:
> Hi list, Neil!
> 
> I have a little question because I'v got some performance problem...

What performance problem?

> Is the RAID5 do any type of headahead with the default config?

Technically: no.
raid5 itself doesn't do readahead.
The filesystem or VM layer does read-ahead.
The block device can suggest the sort of read-ahead size which might
be appropriate.

> It is possible to disable the readahead? (if it is.)

This has been answered - man blockdev
But you almost certainly don't want to. 

> Or just the io_sched do that on the disks?
> 
> How can I disable the whole headahead on top of the RAID5 array?
> 
> The RAID5 can do multiple small (<4kb) reads of disks for the multithreaded
> read requests?
> (chunk size is 32k)

Multiple small random read requests should evenly distribute over all
drives and give reasonable performance.

But again, you haven't said what your problem is....

NeilBrown

> 
> Thanks for helping!
> 
> Janos

Re: RAID5 question.

[David Greaves]

Neil Brown wrote:

>Technically: no.
>raid5 itself doesn't do readahead.
>  
>
Could you clarify this line please:

>The filesystem or VM layer does read-ahead.
>  
>
 suggests that either: performance should only be affected by the
readahead setting of the block layer immediately below the filesystem
or: lvm and md use the VM layer to access the lower block levels and so
performance is affected by readahead settings for all involved
layers/devices.
I strongly suspect the latter as I run xfs over lvm over md over sata
and I found (I posted the test script aeons ago) that varying the
readhead on different layers affects performance. The best performance
(for my usage) was obtained by setting readahead to zero on all layers
except the topmost (lvm for me).

Comments?

David