Re: LVM on raid10,f2 performance issues

["Keld Jørn Simonsen" <keld@dkuug.dk>]

Hmm, 

Why is the command

 blockdev --setra 65536 /dev/md0

really needed? I think the kernel should set a reasonable default here.

What is the logic? I am in the following trying to discuss what would be
reasonable for a kernel patch to achieve.

The command sets readahead to 32 MiB . Is that really wanted?
I understand that it really is important for our benchmarks to give good
results. But is it useful in real operation? Or can a smaller value
solve the problem?

reading 32 MB takes about 300 - 500 ms - and this needs to be done for
every read, even for small reads. That is a lot. For database operations
this would limit operations to 2 to 3 transactions per second. A normal
7200 rpm drive is capable of say about 100 tps, so this would slow such
transactions down with a factor of 30 to 50...

maybe a parameter to blockdev of 16384 - or 8 MiB - would be sufficient?
This would limit the time spent on each transaction to about 100 ms.

And this could be dependent on the relevant parameters, say drive
numbers and chunk size. Maybe the trick is to read a full stripe set,
that is number of drives times chunk size. For a 4 drive array with
chunk size 256 KiB this would be 1 MiB or a --setra paramaeter of 2048.

Maybe the trick is to read more stripe sets at the same time.
For raid5 and raid6 reads the parity chunks need not be read so it
would be a waiste to read the full stripe set.
I am not fully sure what is going on. Maybe somebody can enlighten me.

Or maybe the readahead is not the real parameter that needs to be set
correctly - but maybe something else needs to be fixed, maybe some
logic.

best regards
keld

On Sun, Jan 18, 2009 at 08:24:42PM -0500, thomas62186218@aol.com wrote:
> Hi everyone,
> 
> I too was seeing miserable read-performance with LVM2 volumes on top of 
> md RAID 10's on my Ubuntu 8.04 64-bit machine. My RAID 10 has 12 x 
> 300GB 15K SAS drives on a 4-port LSI PCIe SAS controller.
> 
> I use:
> blockdev --setra 65536 /dev/md0
> 
> And this dramatically increased my RAID 10 read performance.
> 
> You MUST do the same for your LVM2 volumes for them to see a comparable 
> performance boost.
> 
> blockdev --setra 65536 /dev/mapper/raid10-testvol
> 
> Otherwise, your LVM will default to 256 read-ahead value, which stinks. 
> I increased my read performance by 3.5x with this one change! See below:
> 
> root@b410:~# dd if=/dev/raid10twelve256k/testvol of=/dev/null bs=1M 
> count=10000
> 10000+0 records in
> 10000+0 records out
> 10485760000 bytes (10 GB) copied, 50.8923 s, 206 MB/s
> 
> root@b410:~# blockdev --setra 65536 /dev/mapper/raid10twelve256k-testvol
> 
> root@b410:~# dd if=/dev/raid10twelve256k/testvol of=/dev/null bs=1M 
> count=10000
> 10000+0 records in
> 10000+0 records out
> 10485760000 bytes (10 GB) copied, 14.4057 s, 728 MB/s
> 
> 
> Enjoy!
> -Thomas
> 
> -----Original Message-----
> From: Michal Soltys <soltys@ziu.info>
> To: Holger Mauermann <mauermann@gmail.com>
> Cc: Keld Jørn Simonsen <keld@dkuug.dk>; linux-raid@vger.kernel.org
> Sent: Wed, 3 Dec 2008 1:43 am
> Subject: Re: LVM on raid10,f2 performance issues
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Holger Mauermann wrote: 
> 
> >Keld Jørn Simonsen schrieb: 
> 
> >>How is it if you use t
> he raid10,f2 without lvm? 
> 
> >>What are the numbers? 
> 
> >
> >After a fresh installation LVM performance is now somewhat better. I 
> 
> >don't know what was wrong before. However, it is still not as fast as 
> 
> >the raid10... 
> 
> >
> >dd on raw devices 
> 
> >----------------- 
> 
> >
> >raid10,f2: 
> 
> >  read : 409 MB/s 
> 
> >  write: 212 MB/s 
> 
> >
> >raid10,f2 + lvm: 
> 
> >  read : 249 MB/s 
> 
> >  write: 158 MB/s 
> 
> >
> >
> >sda:  sdb:  sdc:  sdd: 
> 
> >---------------------- 
> 
> >YYYY  ....  ....  XXXX 
> 
> >....  ....  ....  .... 
> 
> >XXXX  YYYY  ....  .... 
> 
> >....  ....  ....  .... 
>  
> 
>  
> 
> Regarding the layout from your first mail - this is how it's supposed 
> to
> be. LVM's header took 3*64KB (you can control that with --metadatasize,
> and verify with e.g. pvs -o+pe_start), and then the first 4MB extent
> (controlled with --physicalextentsize) of the first logical volume
> started - on sdd and continued on sda. Mirrored data was set "far" from
> that, and shifted one disk to the right - as expected from raid10,f2. 
>  
> 
> As for performance, hmmm. Overally - there're few things to consider
> when doing lvm on top of the raid: 
>  
> 
> - stripe vs. extent alignment 
> 
> - stride vs. stripe vs. extent size 
> 
> - filesystem's awareness 
> that there's also raid a layer below 
> 
> - lvm's readahead (iirc, only uppermost layer matters - functioning as 
> a
> hint for the filesystem) 
>  
> 
> But this is particulary important for raid with parities. Here
> everything is aligned already, and parity doesn't exist. 
>  
> 
> But the last point can be relevant - and you did test with filesystem
> after all. Try setting readahead with blockdev or lvchange (the latter
> will be permananet across lv activations). E.g. 
>  
> 
> #lvchange -r 2048 /dev/mapper... 
>  
> 
> and compare to raw raid10: 
>  
> 
> #blockedv --setra 2048 /dev/md... 
>  
> 
> If you did your tests with ext2/3, also try to create it with -E 
> stride=
> stripe-width= option in both cases. Similary to sunit/swidth if you 
> used
> xfs. 
>  
> 
> You might also create volume group with larger extent - such as 512MB
> (as 4MB granularity is often an overkill). Performance wise it 
> shouldn't
> matter in this case though. 
>  
> 
> -- 
> 
> To unsubscribe from this list: send the line "unsubscribe linux-raid" 
> in 
> 
> the body of a message to majordomo@vger.kernel.org 
> 
> More majordomo info at  http://vger.kernel.org/majordomo-info.html 
> 
> 
> 
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html