[linux-lvm] LVM2 seems to chop performance by 33%

[linux-lvm] LVM2 seems to chop performance by 33%

[David Greaves]

Hi

I didn't think lvm2 had such a big performance hit?

65Mb/s on the raid5 device
44Mb/s on the lv

Is this expected?

Kernel 2.6.6

representative dd's:

cu:/huge/editing/tmp# time dd if=/dev/video_vg/video_lv of=/dev/null 
bs=1024k count=4k
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 97.363340 seconds (44112777 bytes/sec)

real    1m37.437s
user    0m0.027s
sys     0m27.126s

yet:
cu:/huge/editing/tmp# time dd if=/dev/md0 of=/dev/null bs=1024k count=4k
4096+0 records in
4096+0 records out
4294967296 bytes transferred in 65.715222 seconds (65357267 bytes/sec)

real    1m5.770s
user    0m0.026s
sys     0m27.524s


cu:/huge/editing/tmp# lvdisplay /dev/video_vg/video_lv
  --- Logical volume ---
  LV Name                /dev/video_vg/video_lv
  VG Name                video_vg
  LV UUID                hyCwfR-d7ZG-lptP-XJFK-vMUb-JWE8-LgQSbb
  LV Write Access        read/write
  LV Status              available
  # open                 2
  LV Size                935.00 GB
  Current LE             14960
  Segments               1
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:0

Changing Read ahead makes no difference (actually it slows it down a 
touch with values of 10 + 100)

cu:/huge/editing/tmp# vgdisplay /dev/video_vg
  --- Volume group ---
  VG Name               video_vg
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               935.00 GB
  PE Size               64.00 MB
  Total PE              14960
  Alloc PE / Size       14960 / 935.00 GB
  Free  PE / Size       0 / 0
  VG UUID               kFVbcB-UgLa-XQjy-mF7R-BT2D-sUaA-uGUmeK

cu:/huge/editing/tmp# pvdisplay /dev/md0
  --- Physical volume ---
  PV Name               /dev/md0
  VG Name               video_vg
  PV Size               935.00 GB / not usable 0
  Allocatable           yes (but full)
  PE Size (KByte)       65536
  Total PE              14960
  Free PE               0
  Allocated PE          14960
  PV UUID               pEWAKP-tbaH-DSXP-W2jF-Pg6H-2z35-k1adQj



David

Re: [linux-lvm] LVM2 seems to chop performance by 33%

[Stuart Harper]

While doing backups of my LVM2 drive, I've found that setting the DD block 
size to 4096 greatly improved my performance. Formerly, DDs on my 240G LVM 
were taking 23hrs to complete with a block size of 512 or 1024. The same 
volume now takes less than 2 hours with a block size of 4096. Large numbers 
did not seem to increase DD speed.

On Thursday 10 June 2004 10:25 am, David Greaves wrote:
> 65Mb/s on the raid5 device
> 44Mb/s on the lv
>
> Is this expected?
>
> Kernel 2.6.6
>
> representative dd's:
>
> cu:/huge/editing/tmp# time dd if=/dev/video_vg/video_lv of=/dev/null
> bs=1024k count=4k
> David

Re: [linux-lvm] LVM2 seems to chop performance by 33%

[Chris Croswhite]

try bs=64k, even better...

On Thu, 2004-06-10 at 08:05, Stuart Harper wrote:
> While doing backups of my LVM2 drive, I've found that setting the DD block 
> size to 4096 greatly improved my performance. Formerly, DDs on my 240G LVM 
> were taking 23hrs to complete with a block size of 512 or 1024. The same 
> volume now takes less than 2 hours with a block size of 4096. Large numbers 
> did not seem to increase DD speed.
> 
> On Thursday 10 June 2004 10:25 am, David Greaves wrote:
> > 65Mb/s on the raid5 device
> > 44Mb/s on the lv
> >
> > Is this expected?
> >
> > Kernel 2.6.6
> >
> > representative dd's:
> >
> > cu:/huge/editing/tmp# time dd if=/dev/video_vg/video_lv of=/dev/null
> > bs=1024k count=4k
> > David
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] LVM2 seems to chop performance by 33%

[David Greaves]

Thanks

I'd tried that, but no real change. I started 1t 128k and also tried 
64k, 256k :) (oh, and 1k)

dd if=/dev/video_vg/video_lv of=/dev/null bs=4k count=256k
262144+0 records in
262144+0 records out
1073741824 bytes transferred in 24.130318 seconds (44497624 bytes/sec)

dd if=/dev/md0 of=/dev/null bs=4k count=256k
262144+0 records in
262144+0 records out
1073741824 bytes transferred in 15.404947 seconds (69701105 bytes/sec)

David

PS wait 'til you see what I'm getting through Rieserfs on top of it!
<sigh>
cu:/huge/editing/tmp# time dd if=dummy.deleteme of=/dev/null bs=4k 
count=256k
262144+0 records in
262144+0 records out
1073741824 bytes transferred in 31.627904 seconds (33949193 bytes/sec)
so throughput down by a factor of 2...
but 1 step at a time....

Stuart Harper wrote:

>While doing backups of my LVM2 drive, I've found that setting the DD block 
>size to 4096 greatly improved my performance. Formerly, DDs on my 240G LVM 
>were taking 23hrs to complete with a block size of 512 or 1024. The same 
>volume now takes less than 2 hours with a block size of 4096. Large numbers 
>did not seem to increase DD speed.
>
>On Thursday 10 June 2004 10:25 am, David Greaves wrote:
>  
>
>>65Mb/s on the raid5 device
>>44Mb/s on the lv
>>
>>Is this expected?
>>
>>Kernel 2.6.6
>>
>>representative dd's:
>>
>>cu:/huge/editing/tmp# time dd if=/dev/video_vg/video_lv of=/dev/null
>>bs=1024k count=4k
>>David
>>    
>>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@redhat.com
>https://www.redhat.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
>  
>

Re: [linux-lvm] LVM2 seems to chop performance by 33%

[Clint Byrum]

On Thu, 2004-06-10 at 08:34, David Greaves wrote:
> Thanks
> 
> I'd tried that, but no real change. I started 1t 128k and also tried 
> 64k, 256k :) (oh, and 1k)
> 

I did some tests a few months ago with bonnie++.. might offer some
encouragement (please don't post this to slashdot.. ;)

http://spamaps.org/raidtests.php

There's a lot of data there, but if you look at the LVM stuff, you might
notice that the concurrent performance (having 3 processes hammering the
disks in different places instead of just one) was quite good when
compared to flat out RAID5. I'll pay 5% performance for manageability
any day. :-D

Re: [linux-lvm] LVM2 seems to chop performance by 33%

[David Greaves]

Clint Byrum wrote:

>On Thu, 2004-06-10 at 08:34, David Greaves wrote:
>  
>
>>Thanks
>>
>>I'd tried that, but no real change. I started 1t 128k and also tried 
>>64k, 256k :) (oh, and 1k)
>>
>>    
>>
>
>I did some tests a few months ago with bonnie++.. might offer some
>encouragement (please don't post this to slashdot.. ;)
>
>http://spamaps.org/raidtests.php
>
>There's a lot of data there, but if you look at the LVM stuff, you might
>notice that the concurrent performance (having 3 processes hammering the
>disks in different places instead of just one) was quite good when
>compared to flat out RAID5. I'll pay 5% performance for manageability
>any day. :-D
>  
>
Thanks to those that made suggestions.

In the end I used
blockdev --setra 4096 on all my devices (/dev/sda,b,c,d and the /dev/md0 
and the /dev/video_vg/video_lv) and this doubled throughput.

I am reading multi-gigabyte video files so these parameters are not for 
everyone.

No-one ever replied as to why blockdev --setra / --getra is not the same 
as that displayed in lvdisplay
And it's not documented that I can find. There's a comment: "Not used by 
device-mapper." And that means.....?
It's ignored? not implemented yet? Good luck?

# lvdisplay /dev/video_vg/huge_lv
  --- Logical volume ---
  LV Name                /dev/video_vg/huge_lv
  VG Name                video_vg
  LV UUID                3kz7n9-97Rg-2LJw-J9ml-1BBS-jGs0-Onh4NI
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                312.50 GB
  Current LE             5000
  Segments               1
  Allocation             inherit
  Read ahead sectors     120
  Block device           253:1

# blockdev --getra /dev/video_vg/huge_lv
4096

<sigh> Let this post be there for Google - the modern man-page for 
linux. (if you've got your fingers crossed!)

David

Re: [linux-lvm] LVM2 seems to chop performance by 33%

[Alasdair G Kergon]

On Sun, Jun 20, 2004 at 07:17:58PM +0100, David Greaves wrote:
> No-one ever replied as to why blockdev --setra / --getra is not the same 
> as that displayed in lvdisplay
> And it's not documented that I can find. There's a comment: "Not used by 
> device-mapper." And that means.....?

Any LVM read ahead setting is ignored by LVM2 i.e. not passed to the kernel.

If it's demonstrated to be useful it could still be implemented, but
the current development priorities are still correctness and new features 
rather than performance.

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] LVM2 seems to chop performance by 33%

[David Greaves]

Alasdair G Kergon wrote:

>On Sun, Jun 20, 2004 at 07:17:58PM +0100, David Greaves wrote:
>  
>
>>No-one ever replied as to why blockdev --setra / --getra is not the same 
>>as that displayed in lvdisplay
>>And it's not documented that I can find. There's a comment: "Not used by 
>>device-mapper." And that means.....?
>>    
>>
>
>Any LVM read ahead setting is ignored by LVM2 i.e. not passed to the kernel.
>
>If it's demonstrated to be useful it could still be implemented, but
>the current development priorities are still correctness and new features 
>rather than performance.
>
>Alasdair
>  
>
Thanks Alasdair

It would be good to modify the docs to be clearer and maybe say "use 
blockdev --setra"

David

Re: [linux-lvm] LVM2 seems to chop performance by 33%

[Miguel Cabeça]

Hi,

> I didn't think lvm2 had such a big performance hit?
>
> 65Mb/s on the raid5 device
> 44Mb/s on the lv
>
Did you try to ajust the read ahead with blockdev on the LV itself?

# blockdev --setra XXXX /dev/sda  (raid5 device, just for comparisson)
# blockdev --setra XXXX /dev/vg/lv (lv)

Try some values for the readahead. On my system I use 2048.

Best regards

Miguel Cabe�a

Re: [linux-lvm] LVM2 seems to chop performance by 33%

[David Greaves]

Miguel Cabe�a wrote:

> Hi,
>
>> I didn't think lvm2 had such a big performance hit?
>>
>> 65Mb/s on the raid5 device
>> 44Mb/s on the lv
>>
> Did you try to ajust the read ahead with blockdev on the LV itself?
>
> # blockdev --setra XXXX /dev/sda  (raid5 device, just for comparisson)
> # blockdev --setra XXXX /dev/vg/lv (lv)
>
> Try some values for the readahead. On my system I use 2048.

No, I didn't
hmm... man blockdev
useful - new to me :) Not seen it on any googling to do with lvm.

I'll try this later when it's idle - thanks for the suggestion.

I do already have
  Read ahead sectors     100
in the lvdisplay

yet blockdev --getra /dev/video_vg/video_lv
reports 256 (and, to be sure --getss says 512byte sectors)

Why is this different?

David