[linux-lvm] performance comparison soft-hardware RAID + LVM: bad

[linux-lvm] performance comparison soft-hardware RAID + LVM: bad

[Ron Arts]

[-- Attachment #1: Type: text/plain, Size: 2962 bytes --]

Hello,

I am interested in performance for hardware/software RAID
in combination with LVM

So I took a server (Dual Xeon 2.4GHz, 1Gb RAM), a RAID adapter,
some identical SCSI disks and configured it with several of these
options (using RH 8.0) and ran a few bonnie++ benchmarks.

Results are below. Anyone care to comment? Especially LVM performance
disappointed here.

LVM machine setup:

2 18Gb disks. I created 3 partitions on both disks, 128Mb, 512Mb and 17Gb
Equal partitions were combined into RAID-1 devices (md driver).
First md device mounted on /boot, second for swapfile, and third
as basis for LVM

Out of the volume group four LV were created and mounted as follows:

[root@nbs-126 root]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/vg0/root          4225092   1293064   2717400  33% /
/dev/md0                124323     11517    106387  10% /boot
/dev/vg0/home          4225092     32828   3977636   1% /home
none                    514996         0    514996   0% /dev/shm
/dev/vg0/var           4225092     51720   3958744   2% /var
/dev/vg0/mysql        16513960     32828  15642272   1% /var/lib/mysql

Is there a reason for the performance degradation I saw with LVM?

Regards,
Ron Arts


Version  1.02c      ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
nbs-126.offic 2008M 24005  99 66015  49 16148   7 27430  98 86915  15 369.0   1  single disk
nbs-126.offic    2G 23422  99 70919  58 23800  11 25289  86 101485 17 433.4   1  s/w RAID-1
nbs-126.offic    2G  8152  99 49897  94 23092  27  9122  92 78056  38 331.1   2  s/w RAID-1 + LVM
nbs-126.offic 4032M 19695  99 44056  42 14179   9 21526  94 86450  16 344.3   1  h/w RAID-1
nbs-126.offic 4032M 19916  99 24033  22 13343   9 22794  99 111662 30 388.5   1  h/w RAID-5
                     ------Sequential Create------ --------Random Create--------
                     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
nbs-126.office.n 16  2481  99 +++++ +++ +++++ +++  2424  99 +++++ +++  5508  98  single disk
nbs-126.office.n 16  2530  99 +++++ +++ +++++ +++  2437  99 +++++ +++  6062 100  s/w RAID-1 soft
nbs-126.office.n 16   800  99 +++++ +++ 12848  98   807  99 +++++ +++  2591  99  s/w RAID-1 + LVM
nbs-126.office.n 16  2138  98 +++++ +++ 31126  98  2200  99 +++++ +++  5322  98  h/w RAID-1
nbs-126.office.n 16  2182  99 +++++ +++ 27238  86  2172  98 +++++ +++  5261  97  h/w RAID-5


Notes:

The last two are with hardware RAID (GDT 4513RZ), and 2Gb RAM instead of 1, but I adjusted
the -s parameter for bonnie++ accordingly.

commandline:
# ./bonnie++ -d /tmp -s 2048 -x2 -uroot
Results shown are for the second run. Machines were otherwise inactive, and carried a
minimum install.

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3291 bytes --]

Re: [linux-lvm] performance comparison soft-hardware RAID + LVM: bad

[Heinz J . Mauelshagen]

On Wed, Oct 16, 2002 at 12:17:49AM +0200, Ron Arts wrote:
> Hello,
> 
> I am interested in performance for hardware/software RAID
> in combination with LVM
> 
> So I took a server (Dual Xeon 2.4GHz, 1Gb RAM), a RAID adapter,
> some identical SCSI disks and configured it with several of these
> options (using RH 8.0) and ran a few bonnie++ benchmarks.
> 
> Results are below. Anyone care to comment? Especially LVM performance
> disappointed here.
> 
> LVM machine setup:
> 
> 2 18Gb disks. I created 3 partitions on both disks, 128Mb, 512Mb and 17Gb
> Equal partitions were combined into RAID-1 devices (md driver).
> First md device mounted on /boot, second for swapfile, and third
> as basis for LVM
> 
> Out of the volume group four LV were created and mounted as follows:
> 
> [root@nbs-126 root]# df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/vg0/root          4225092   1293064   2717400  33% /
> /dev/md0                124323     11517    106387  10% /boot
> /dev/vg0/home          4225092     32828   3977636   1% /home
> none                    514996         0    514996   0% /dev/shm
> /dev/vg0/var           4225092     51720   3958744   2% /var
> /dev/vg0/mysql        16513960     32828  15642272   1% /var/lib/mysql

Ron, I don't get your configuration by this df output :(

You've got 3 mirrored MDs on 2 disks (1st: 128Mb, 2nd: 512Mb, 3rd: 17Gb)

3rd is used as the one and only physical volume (say /dev/md2)
for volume group "vg0", right?

How comes that the grand total of /dev/vg0/{root, home, var, mysql}
is ~27.84Gb when the volume group only holds the 17Gb of /dev/md2?

> 
> Is there a reason for the performance degradation I saw with LVM?

If my above thoughts are correct, the logical volume(s) used could well be
suffering from bad mapping onto disk(s). You can check the mapping with
either "pvdisplay -v /dev/md2" or for eg. "lvdisplay -v /dev/vg00/root".
I don't feel sure about your MD configuration either.

Hope that spots some light on those numbers which differ from mine
(typically LVM gives a performamce enhancement for sequential input).
The mapping overhead of the LVM1 driver is in the sub percent order.

Regards,
Heinz    -- The LVM Guy --

> 
> Regards,
> Ron Arts
> 
> 
> Version  1.02c      ------Sequential Output------ --Sequential Input- --Random-
>                      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> nbs-126.offic 2008M 24005  99 66015  49 16148   7 27430  98 86915  15 369.0   1  single disk
> nbs-126.offic    2G 23422  99 70919  58 23800  11 25289  86 101485 17 433.4   1  s/w RAID-1
> nbs-126.offic    2G  8152  99 49897  94 23092  27  9122  92 78056  38 331.1   2  s/w RAID-1 + LVM
> nbs-126.offic 4032M 19695  99 44056  42 14179   9 21526  94 86450  16 344.3   1  h/w RAID-1
> nbs-126.offic 4032M 19916  99 24033  22 13343   9 22794  99 111662 30 388.5   1  h/w RAID-5
>                      ------Sequential Create------ --------Random Create--------
>                      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> nbs-126.office.n 16  2481  99 +++++ +++ +++++ +++  2424  99 +++++ +++  5508  98  single disk
> nbs-126.office.n 16  2530  99 +++++ +++ +++++ +++  2437  99 +++++ +++  6062 100  s/w RAID-1 soft
> nbs-126.office.n 16   800  99 +++++ +++ 12848  98   807  99 +++++ +++  2591  99  s/w RAID-1 + LVM
> nbs-126.office.n 16  2138  98 +++++ +++ 31126  98  2200  99 +++++ +++  5322  98  h/w RAID-1
> nbs-126.office.n 16  2182  99 +++++ +++ 27238  86  2172  98 +++++ +++  5261  97  h/w RAID-5
> 
> 
> Notes:
> 
> The last two are with hardware RAID (GDT 4513RZ), and 2Gb RAM instead of 1, but I adjusted
> the -s parameter for bonnie++ accordingly.
> 
> commandline:
> # ./bonnie++ -d /tmp -s 2048 -x2 -uroot
> Results shown are for the second run. Machines were otherwise inactive, and carried a
> minimum install.

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] performance comparison soft-hardware RAID + LVM: bad

[Jon Bendtsen]

Ron Arts wrote:
> 
> Hello,
> 
> I am interested in performance for hardware/software RAID
> in combination with LVM

i've resently tested hardware vs. software on 3 different
3ware cards, and some IDE disks. My results showed that
software raid was somewhat faster than hardware, but i'd
still prefer hardware because of the handling of one
failing disk. (notice that if you dont get a warning when 
a disk fails, you might as well run without raid, or run
raid0)
I've got a repport in .pdf (and .LyX) in english if someone
wants it.

 
> So I took a server (Dual Xeon 2.4GHz, 1Gb RAM), a RAID adapter,
> some identical SCSI disks and configured it with several of these
> options (using RH 8.0) and ran a few bonnie++ benchmarks.

Mine was a single p3 800, with 512MB memory, and i used tiobench

 
> Results are below. Anyone care to comment? Especially LVM performance
> disappointed here.

I cant clearly see what is LVM setup and what isnt. Remember that LVM 
doesnt allocate blocks sequeltial, but by default the first one free.
So, when you create 3 lv's, and then you mkfs them, then you allocate
at least the first block. Then when you fill the rest of the
filesystem...
you allocate the next blocks. Results are one block in the beginning,
a wide gap, and then the rest of the blocks.


> LVM machine setup:
> 
> 2 18Gb disks. I created 3 partitions on both disks, 128Mb, 512Mb and 17Gb
> Equal partitions were combined into RAID-1 devices (md driver).
> First md device mounted on /boot, second for swapfile, and third
> as basis for LVM
> 
> Out of the volume group four LV were created and mounted as follows:
> 
> [root@nbs-126 root]# df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/vg0/root          4225092   1293064   2717400  33% /
> /dev/md0                124323     11517    106387  10% /boot
> /dev/vg0/home          4225092     32828   3977636   1% /home
> none                    514996         0    514996   0% /dev/shm
> /dev/vg0/var           4225092     51720   3958744   2% /var
> /dev/vg0/mysql        16513960     32828  15642272   1% /var/lib/mysql
> 
> Is there a reason for the performance degradation I saw with LVM?

I've done 3 (or 0.5 + 0.5 + 1) benchmarks. The first 2 times i didnt do
it well enough. I dont believe you have done it well enough, you clearly
dont have enough numbers. I found that using tiobench i had to variate
the number of threads (concurrent read/write) and the blocksize, before
i
got the best performance. And it variates alot. (See my .pdf, which i
will
mail to you). I've got lots of numbers. I used gnuplot to create graphs,
but consider using iometer to run your benchmarks, i think it creates
the
graphs for you.
You dont have enough disks, 2 disks might be a widely used common setup,
but
other people use more disks, like 4, 8, ... especialy when using scsi,
which
for some reason doesnt seem to contain as much as IDE disks.
I saw a BIG performance drop when i tried to run tiobench on a smp
system, 
but using another benchmark tool, i saw that it was only tiobench, not
the
performance that suffered.
I also saw a _GIGANTIC_ performance drop when i ran software raid0 ontop
of
2 software raid1. (or was it the other way ? (request my repport and see
for
yourself, the included one is regular performance)





JonB

Re: [linux-lvm] performance comparison soft-hardware RAID + LVM: bad

[Ron Arts]

[-- Attachment #1: Type: text/plain, Size: 3405 bytes --]

Jon Bendtsen wrote:
> Ron Arts wrote:
> 

[snip]

(I received your report, thanks)


>  
> 
>>Results are below. Anyone care to comment? Especially LVM performance
>>disappointed here.
> 
> 
> I cant clearly see what is LVM setup and what isnt. Remember that LVM 
> doesnt allocate blocks sequeltial, but by default the first one free.
> So, when you create 3 lv's, and then you mkfs them, then you allocate
> at least the first block. Then when you fill the rest of the
> filesystem...
> you allocate the next blocks. Results are one block in the beginning,
> a wide gap, and then the rest of the blocks.
> 

Sorry, I don't understand. Why the gap?
Omn the other hand, the underlying devices are RAID-1 in software, the
allocation shouldn't matter should it?

> 
> 
>>LVM machine setup:
>>
>>2 18Gb disks. I created 3 partitions on both disks, 128Mb, 512Mb and 17Gb
>>Equal partitions were combined into RAID-1 devices (md driver).
>>First md device mounted on /boot, second for swapfile, and third
>>as basis for LVM
>>
>>Out of the volume group four LV were created and mounted as follows:
>>
>>[root@nbs-126 root]# df
>>Filesystem           1K-blocks      Used Available Use% Mounted on
>>/dev/vg0/root          4225092   1293064   2717400  33% /
>>/dev/md0                124323     11517    106387  10% /boot
>>/dev/vg0/home          4225092     32828   3977636   1% /home
>>none                    514996         0    514996   0% /dev/shm
>>/dev/vg0/var           4225092     51720   3958744   2% /var
>>/dev/vg0/mysql        16513960     32828  15642272   1% /var/lib/mysql
>>
>>Is there a reason for the performance degradation I saw with LVM?
> 
> 
> I've done 3 (or 0.5 + 0.5 + 1) benchmarks. The first 2 times i didnt do
> it well enough. I dont believe you have done it well enough, you clearly
> dont have enough numbers. I found that using tiobench i had to variate
> the number of threads (concurrent read/write) and the blocksize, before
> i
> got the best performance. And it variates alot. (See my .pdf, which i
> will
> mail to you). I've got lots of numbers. I used gnuplot to create graphs,

Okay, but lots of numbers still don't explain why in this particular case
performance was so slow. If I understand why, I can begin to make
optimizations.

To give some background:
I do this because I need such a setup for a particular application
(MySQL high volume logging server). If I understand the issues involved
I can make more informed choices implementing the application.
Should it log using multiple threads or one? Will readers from the
datbase hinder the writing process a lot? What is the best way
to add disks using LVM, without taking a large performance hit?

This server must be up 24x7. I found something called scsirastools
that can deal with hotswapping SCSI disks under software RAID.

I thought I'd first try some benchmarks with bonnie to get a feel
for the issues involved, and seeing the performance (and CPU) hit
for my LVM setup (and having never used LVM before) I decided
to ask you guys about this.


And thanks for your report, at least it confirmed what I
had seen: software raid is faster then hardware.

Regards,
Ron Arts


-- 
Netland Internet Services
bedrijfsmatige internetoplossingen

http://www.netland.nl   Kruislaan 419              1098 VA Amsterdam
info: 020-5628282       servicedesk: 020-5628280   fax: 020-5628281

Does old mail ever arrive?

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3291 bytes --]

Re: [linux-lvm] performance comparison soft-hardware RAID + LVM: bad

[Arie Bant@mail.com]

 -----Original Message-----
Date:	Wed, 16 Oct 2002 10:34:41 +0200
From:	Jon Bendtsen <jon+lvm@silicide.dk>
	Organization: Silicide A/S
To:	linux-lvm@sistina.com
Subject:	Re: [linux-lvm] performance comparison soft-hardware RAID + LVM:
bad
	Reply-To: linux-lvm@sistina.com

You dont have enough disks, 2 disks might be a widely used common setup, but
other people use more disks, like 4, 8, ... especialy when using scsi, which
for some reason doesnt seem to contain as much as IDE disks.
Jon,

Availability of large SCSI hard disks has nothing to do with it.
Using more spindles is better for performance as a general rule (more
spindles can lead to less head movement, if properly configured, head
movement/rotational delay counts for most of the access time on hard disks.
Also, via SCSI, seeks can be done simultaneously and in overlap with I/O on
other hard disks and/or processor activity), hence you will find that in
SCSI configurations more disks are used for performance reasons.
For this reason only, I keep a stack of, now old, 1/2/4 GB drives to use as
dedicated swap devices and /tmp file systems.
Given the intrinsic limitations, all this is not possible when using IDE, so
the capacity than has to come from large disks, the performance will suffer
accordingly, especially in total system throughput.  One of a number of
reasons why you should always go with SCSI when you want performance.

Regards,

Arie Bant.

Re: [linux-lvm] performance comparison soft-hardware RAID + LVM:bad

[Jon Bendtsen]

Ron Arts wrote:
> 
> Jon Bendtsen wrote:
> > Ron Arts wrote:

[snip]
 

> > I cant clearly see what is LVM setup and what isnt. Remember that LVM
> > doesnt allocate blocks sequeltial, but by default the first one free.
> > So, when you create 3 lv's, and then you mkfs them, then you allocate
> > at least the first block. Then when you fill the rest of the
> > filesystem...
> > you allocate the next blocks. Results are one block in the beginning,
> > a wide gap, and then the rest of the blocks.
> >
> 
> Sorry, I don't understand. Why the gap?
> Omn the other hand, the underlying devices are RAID-1 in software, the
> allocation shouldn't matter should it?

Well, let me try a state of the art (tm) ascii drawing ;-D

 disk1	 disk2
|-----| |-----| 

those you make into a mirror
    |-----|


Lets zoom in on that and see how LVM works	(unallocated vg)
|---------------------------------------------------------------------------------|
So, you make a lv in this vg
|#--------------------------------------------------------------------------------|
Then you make another
|#¤-------------------------------------------------------------------------------|
You allocate some data for the first lv
|#¤#####--------------------------------------------------------------------------|
You allocate some data for the 2. lv
|#¤#####¤¤¤¤¤¤--------------------------------------------------------------------|

Can you see the trend now ??
when you need a "block" you just get the next free. So, unless you are
carefull, or
make it allocate all on creation, then you get this
|#¤#####¤¤¤¤¤¤##¤#¤#¤#¤#¤¤¤¤####¤¤##¤#¤#¤#¤###¤¤¤¤##¤¤##¤¤##¤¤####¤¤¤¤¤#####¤¤##¤#|
And how do you think this changes your performance ? if you are reading
one BIG file.
that covers several # or ¤


> > I've done 3 (or 0.5 + 0.5 + 1) benchmarks. The first 2 times i didnt do
> > it well enough. I dont believe you have done it well enough, you clearly
> > dont have enough numbers. I found that using tiobench i had to variate
> > the number of threads (concurrent read/write) and the blocksize, before
> > i
> > got the best performance. And it variates alot. (See my .pdf, which i
> > will
> > mail to you). I've got lots of numbers. I used gnuplot to create graphs,
> 
> Okay, but lots of numbers still don't explain why in this particular case
> performance was so slow. If I understand why, I can begin to make
> optimizations.

I think it is the default allocation of "blocks" for a lv in a vg.
The point with lots of numbers was that just running one benchmark, with
one concurrent write, aka one "thread" and trying to write one big file,
rather than 10 medium files, doesnt give an adequette view of the
performance.


> To give some background:
> I do this because I need such a setup for a particular application
> (MySQL high volume logging server). If I understand the issues involved
> I can make more informed choices implementing the application.
> Should it log using multiple threads or one? Will readers from the
> datbase hinder the writing process a lot? What is the best way
> to add disks using LVM, without taking a large performance hit?

I see your problem, but since you have many readers/writes, you
NEED to test it with concurrent access, maybe 10-20, or more

 
> This server must be up 24x7. I found something called scsirastools
> that can deal with hotswapping SCSI disks under software RAID.

good. Think of adding many small scsi disks to your raid setup, and
then make it raid10, or raid1 ontop of raid 0 (or the other way arround,
see my repport for which one kills performance)

the proxy server squid can handle more than one area to store the
files. You'll get greater performance by giving it more small areas,
rather than putting the disks into a larger raid array. MAYBE your
mysql logging server can do the same thing ?


> I thought I'd first try some benchmarks with bonnie to get a feel
> for the issues involved, and seeing the performance (and CPU) hit
> for my LVM setup (and having never used LVM before) I decided
> to ask you guys about this.

understandable, but i think you need to do more benchmarking, with
more than one concurrent access.

 
> And thanks for your report, at least it confirmed what I
> had seen: software raid is faster then hardware.

Well, it can be. I havent tried with _MANY_ disks, and i havent
tried with cpu load arround 100%. (per cpu). Actualy, as i wrote 
somewhere in the repport, i would like if someone parallized the
raid, so you could get linear to the number of harddisks performance,
or fill the pci bus. So, someone who reads this... just parallize the
raid, it's not like the first block needs the next block, or the one
before that, so just create a simple asic, it doesnt need to be fast
just many of them, so the pci bus can be filled. 64bit 66MHz of course
;-D




JonB