[linux-lvm] Performance of striping

[linux-lvm] Performance of striping

[Hans-Joachim Baader]

[-- Attachment #1: Type: text/plain, Size: 1958 bytes --]

Hi,

last year I built a server with 3 SATA disks. I used a physical volume
for each disk, a volume group on each, and then created identical LVs
in each VG. Then I used dmesetup to create a RAID 0 over all 3 disks.
A single disk has an I/O performance of 70 MB/s. After creating a
filesystem on the RAID and measuring its performance I got the expected
result: 210 MB/s.

But this setup was too complicated for me, so I tried a different
approach. Still a physical volume for each disk (actually partiton 2
which is nearly the complete disk), but only one volume group vg0.
The commands used were:

pvcreate /dev/sda2
pvcreate /dev/sdb2
pvcreate /dev/sdc2
vgcreate vg0 -s 128 /dev/sda2 /dev/sdb2 /dev/sdc2

On vg0, I created a LV which should span all 3 disks and therefore be
equivalent to a RAID 0:

lvcreate --name=test -L 10000M -i 3 vg0

Resulting LV:

  --- Logical volume ---                                                          LV Name                /dev/vg0/ubuntu                                          VG Name                vg0
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                10,12 GB
  Current LE             81
  Segments               1
  Allocation             inherit
  Read ahead sectors     0
  Block device           254:10

However, I get only half the expected performance. I created an XFS
filesystem and mounted it on /mnt/tmp:

# dd if=/dev/zero of=/mnt/tmp/bla bs=1024k count=10000
10485760000 Bytes (10 GB) kopiert, 112,233 Sekunden, 93,4 MB/s
# dd if=/mnt/tmp/bla of=/dev/null bs=1024k count=10000
10485760000 Bytes (10 GB) kopiert, 88,8886 Sekunden, 118 MB/s

This was measured with kernel 2.6.23.12 and 2.6.25.1 on AMD64 X2.
Is it possible to get 210 MB/s?

Thanks,
hjb
-- 
Pro-Linux - Germany's largest volunteer Linux support site
http://www.pro-linux.de/          Public Key ID 0x3DDBDDEA

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Performance of striping

[Heinz Mauelshagen]

On Sat, May 17, 2008 at 11:11:51AM +0200, Hans-Joachim Baader wrote:
> Hi,
> 
> last year I built a server with 3 SATA disks. I used a physical volume
> for each disk, a volume group on each, and then created identical LVs
> in each VG. Then I used dmesetup to create a RAID 0 over all 3 disks.
> A single disk has an I/O performance of 70 MB/s. After creating a
> filesystem on the RAID and measuring its performance I got the expected
> result: 210 MB/s.

Strange config, good result ;)

> 
> But this setup was too complicated for me, so I tried a different
> approach. Still a physical volume for each disk (actually partiton 2
> which is nearly the complete disk), but only one volume group vg0.
> The commands used were:
> 
> pvcreate /dev/sda2
> pvcreate /dev/sdb2
> pvcreate /dev/sdc2
> vgcreate vg0 -s 128 /dev/sda2 /dev/sdb2 /dev/sdc2
> 
> On vg0, I created a LV which should span all 3 disks and therefore be
> equivalent to a RAID 0:
> 
> lvcreate --name=test -L 10000M -i 3 vg0
> 
> Resulting LV:
> 
>   --- Logical volume ---                                                          LV Name                /dev/vg0/ubuntu                                          VG Name                vg0
>   LV Write Access        read/write
>   LV Status              available
>   # open                 1
>   LV Size                10,12 GB
>   Current LE             81
>   Segments               1
>   Allocation             inherit
>   Read ahead sectors     0
>   Block device           254:10
> 
> However, I get only half the expected performance. I created an XFS
> filesystem and mounted it on /mnt/tmp:
> 
> # dd if=/dev/zero of=/mnt/tmp/bla bs=1024k count=10000
> 10485760000 Bytes (10 GB) kopiert, 112,233 Sekunden, 93,4 MB/s
> # dd if=/mnt/tmp/bla of=/dev/null bs=1024k count=10000
> 10485760000 Bytes (10 GB) kopiert, 88,8886 Sekunden, 118 MB/s
> 
> This was measured with kernel 2.6.23.12 and 2.6.25.1 on AMD64 X2.
> Is it possible to get 210 MB/s?

Have you tried fiddling with read ahead ?
"blockdev --setra ..." on the striped LV.

> 
> Thanks,
> hjb
> -- 
> Pro-Linux - Germany's largest volunteer Linux support site
> http://www.pro-linux.de/          Public Key ID 0x3DDBDDEA



> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Red Hat GmbH
Consulting Development Engineer                   Am Sonnenhang 11
Storage Development                               56242 Marienrachdorf
                                                  Germany
Mauelshagen@RedHat.com                            PHONE +49  171 7803392
                                                  FAX   +49 2626 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] Performance of striping

[Hans-Joachim Baader]

[-- Attachment #1: Type: text/plain, Size: 1960 bytes --]

Hi,


> > But this setup was too complicated for me, so I tried a different
> > approach. Still a physical volume for each disk (actually partiton 2
> > which is nearly the complete disk), but only one volume group vg0.
> > The commands used were:
> > 
> > pvcreate /dev/sda2
> > pvcreate /dev/sdb2
> > pvcreate /dev/sdc2
> > vgcreate vg0 -s 128 /dev/sda2 /dev/sdb2 /dev/sdc2
> > 
> > On vg0, I created a LV which should span all 3 disks and therefore be
> > equivalent to a RAID 0:
> > 
> > lvcreate --name=test -L 10000M -i 3 vg0
> > 
> > Resulting LV:
> > 
> >   --- Logical volume ---                                                          LV Name                /dev/vg0/ubuntu                                          VG Name                vg0
> >   LV Write Access        read/write
> >   LV Status              available
> >   # open                 1
> >   LV Size                10,12 GB
> >   Current LE             81
> >   Segments               1
> >   Allocation             inherit
> >   Read ahead sectors     0
> >   Block device           254:10
> > 
> > However, I get only half the expected performance. I created an XFS
> > filesystem and mounted it on /mnt/tmp:
> > 
> > # dd if=/dev/zero of=/mnt/tmp/bla bs=1024k count=10000
> > 10485760000 Bytes (10 GB) kopiert, 112,233 Sekunden, 93,4 MB/s
> > # dd if=/mnt/tmp/bla of=/dev/null bs=1024k count=10000
> > 10485760000 Bytes (10 GB) kopiert, 88,8886 Sekunden, 118 MB/s
> > 
> > This was measured with kernel 2.6.23.12 and 2.6.25.1 on AMD64 X2.
> > Is it possible to get 210 MB/s?
> 
> Have you tried fiddling with read ahead ?
> "blockdev --setra ..." on the striped LV.

read ahead is currently 256 on all devices. Thanks for the hint.
I will try some other values. Any Idea what is recommended?

Thanks,
hjb
-- 
Pro-Linux - Germany's largest volunteer Linux support site
http://www.pro-linux.de/          Public Key ID 0x3DDBDDEA

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Performance of striping

[tkb]

Hans-Joachim Baader wrote:
> Hi,
>
>
>   
>>>       

snip

>> Have you tried fiddling with read ahead ?
>> "blockdev --setra ..." on the striped LV.
>>     
>
> read ahead is currently 256 on all devices. Thanks for the hint.
> I will try some other values. Any Idea what is recommended?
>   

I originally found this script to test read ahead settings on some email 
list - I forget where or who.

#!/bin/bash

RAW_DEVS="/dev/sdb /dev/sdc /dev/sdd /dev/sde"
MD_DEVS=/dev/md17
LV_DEVS=/dev/mapper/vg17-home

LV_RAS="128 256 1024 4096 8192"
MD_RAS="128 256 1024 4096 8192"
RAW_RAS="128 256 1024 4096 8192"

log=/tmp/ra_test.log
rm -f $log

function show_ra()
{
for i in $RAW_DEVS $MD_DEVS $LV_DEVS
do echo -n "$i `blockdev --getra $i`  ::  " | tee $log
echo ""  | tee -a $log
done
}

function set_ra()
{
RA=$1
shift
for dev in $@
do
  blockdev --setra $RA $dev
done
}

function show_performance()
{
COUNT=4000000
/usr/bin/time dd if=$LV_DEVS of=/dev/null count=$COUNT 2>&1 | tee -a $log
echo "<><><>" | tee -a $log
}

for RAW_RA in $RAW_RAS
do
set_ra $RAW_RA $RAW_DEVS
for MD_RA in $MD_RAS
  do
  set_ra $MD_RA $MD_DEVS
  for LV_RA in $LV_RAS
    do
    set_ra $LV_RA $LV_DEVS
    show_ra
    show_performance
    done
  done
done