[linux-lvm] LVM & high system load

[linux-lvm] LVM & high system load

[Dominique LARCHEY-WENDLING]

Hi, 

I am newbie to LVM. I am experiencing high cpu load (system time > 90%)
and
relatively poor IO perf. (my cpu is only PII 333) on LVM volumes. I have 
tested reiserfs and ext2fs and I only get about 2.5Mb/s with 100% CPU
load,
mostly system time.

Due to some failures of vgscan (no volume found) and vgchange (core
dumps,)
I had to upgrade my 2.4.3 kernel to LVM beta7 and also the system tools.
Now
it seems that theses tools work but I still have very high loads, much
higher than what I would have on a fs lying on a partition.

My logical volumes are not stripped.

Is this a known issue ? 

Thanks in advance,

DL

-- 
Dominique Larchey
LORIA, Nancy, France
Tel. (Work) +33 (0) 3 83 59 20 13

----------------------------

[root@charlie temp]# /usr/sbin/vgdisplay mmedia                       
--- Volume group ---
VG Name               mmedia
VG Access             read/write
VG Status             available/resizable
VG #                  0
MAX LV                256
Cur LV                3
Open LV               3
MAX LV Size           255.99 GB
Max PV                256
Cur PV                3
Act PV                3
VG Size               28.49 GB
PE Size               4 MB
Total PE              7294
Alloc PE / Size       5066 / 19.79 GB
Free  PE / Size       2228 / 8.7 GB
VG UUID               dF3isK-xdlk-YF7u-FwxQ-KnPQ-jgG6-qjt73X

[root@charlie temp]# /usr/sbin/pvdisplay /dev/hda1 /dev/hda5 /dev/hda6
--- Physical volume ---
PV Name               /dev/hda1
VG Name               mmedia
PV Size               9.77 GB / NOT usable 1.37 MB [LVM: 130 KB]
PV#                   1
PV Status             available
Allocatable           yes (but full)
Cur LV                1
PE Size (KByte)       4096
Total PE              2500
Free PE               0
Allocated PE          2500
PV UUID               3aeAAl-88en-VZxJ-84gE-Uaxu-AWZm-zoA3jC

--- Physical volume ---
PV Name               /dev/hda5
VG Name               mmedia
PV Size               9.77 GB / NOT usable 1.37 MB [LVM: 130 KB]
PV#                   2
PV Status             available
Allocatable           yes (but full)
Cur LV                3
PE Size (KByte)       4096
Total PE              2500
Free PE               0
Allocated PE          2500
PV UUID               lUiRFu-JEhC-d5JS-Jycy-VFa8-2o2y-6I9V9S

--- Physical volume ---
PV Name               /dev/hda6
VG Name               mmedia
PV Size               8.96 GB / NOT usable 1.73 MB [LVM: 129 KB]
PV#                   3
PV Status             available
Allocatable           yes
Cur LV                0
PE Size (KByte)       4096
Total PE              2294
Free PE               2228
Allocated PE          66
PV UUID               GOf2OV-khJd-w1cJ-AEk3-RvyA-Wdly-p5OuWd

[root@charlie temp]# /usr/sbin/lvdisplay /dev/mmedia/music
/dev/mmedia/temp /dev/mmedia/test 
--- Logical volume ---
LV Name                /dev/mmedia/music
VG Name                mmedia
LV Write Access        read/write
LV Status              available
LV #                   2
# open                 1
LV Size                9 GB
Current LE             2304
Allocated LE           2304
Allocation             next free
Read ahead sectors     120
Block device           58:1

--- Logical volume ---
LV Name                /dev/mmedia/temp
VG Name                mmedia
LV Write Access        read/write
LV Status              available
LV #                   1
# open                 1
LV Size                9.79 GB
Current LE             2506
Allocated LE           2506
Allocation             next free
Read ahead sectors     120
Block device           58:0

--- Logical volume ---
LV Name                /dev/mmedia/test
VG Name                mmedia
LV Write Access        read/write
LV Status              available
LV #                   3
# open                 1
LV Size                1 GB
Current LE             256
Allocated LE           256
Allocation             next free
Read ahead sectors     120
Block device           58:2

Re: [linux-lvm] LVM & high system load

[Dominique LARCHEY-WENDLING]

Dominique LARCHEY-WENDLING wrote:


> relatively poor IO perf. (my cpu is only PII 333) on LVM volumes. I have
> tested reiserfs and ext2fs and I only get about 2.5Mb/s with 100% CPU
> load,
> mostly system time.

More over, doing a CPU intensive task (like kernel comp.) (not involving
LVM, the
sources are not on a LV) cut the IO perf. of LVM fs by half.

So it seems that the LVM is eating a lot of CPU on my system and this is
what is limiting fs perf.

DL

-- 
Dominique Larchey
LORIA, Nancy, France
Tel. (Work) +33 (0) 3 83 59 20 13

Re: [linux-lvm] LVM & high system load

[Andreas Dilger]

Dominique Larchey writes:
> I am newbie to LVM. I am experiencing high cpu load (system time > 90%)
> and relatively poor IO perf. (my cpu is only PII 333) on LVM volumes.

Note that I am running LVM on Pentium 90 w 16MB RAM and no problems...

> I have tested reiserfs and ext2fs and I only get about 2.5Mb/s with 100% CPU
> load, mostly system time.

There is one other person that has reported very slow LVM performance.
However, there is no obvious reason why they have performance issues,
and others are unable to reproduce this problem.

Can you please try the following and report results:

time dd if=/dev/mmedia/music of=/dev/null bs=512
time dd if=/dev/mmedia/music of=/dev/null bs=65536
time dd if=/dev/hda1 of=/dev/null bs=512
time dd if=/dev/hda1 of=/dev/null bs=65536

Cheers, Andreas
-- 
Andreas Dilger                               Turbolinux filesystem development
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] LVM & high system load

[Dominique LARCHEY-WENDLING]

> I doubt it, I've got LVM0.9.1b7 on a 486 without any real load... even when
> compiling the kernel.  I also have reiserfs on 2.2.19.  I noticed in your
> previous post that you had 3 pv's on the same physical disk... that may be
> contributing, depending on your PE layout on the PVs.  

The layout is linear ...

-----------------------------------------------------------------

[root@charlie larchey]# /usr/sbin/lvdisplay -v /dev/mmedia/music

--- Logical volume ---
LV Name                /dev/mmedia/music
VG Name                mmedia
LV Write Access        read/write
LV Status              available
LV #                   2
# open                 2
LV Size                9 GB
Current LE             2304
Allocated LE           2304
Allocation             next free
Read ahead sectors     120
Block device           58:1

   --- Distribution of logical volume on 1 physical volume  ---
   PV Name                  PE on PV     reads      writes
   /dev/hda5                2304         2129836    343926   

   --- logical volume i/o statistic ---
   2129836 reads  343926 writes

   --- Logical extents ---
   LE    PV                        PE     reads      writes
   00000 /dev/hda5                 00006  1025       1875     
   00001 /dev/hda5                 00007  1024       1024     
   00002 /dev/hda5                 00008  1024       1024     
   00003 /dev/hda5                 00009  1024       961      
   00004 /dev/hda5                 00010  1024       0        
.....   
   02298 /dev/hda5                 02304  2048       1024     
   02299 /dev/hda5                 02305  2048       1024     
   02300 /dev/hda5                 02306  2048       1024     
   02301 /dev/hda5                 02307  2048       1024     
   02302 /dev/hda5                 02308  2048       1024     
   02303 /dev/hda5                 02309  2048       1024     

--------------------------------------------------------

> If not, it's definitely silly :)

It is NOT SILLY ... see next post for performance issues

DL

-- 
Dominique Larchey
LORIA, Nancy, France
Tel. (Work) +33 (0) 3 83 59 20 13

Re: [linux-lvm] LVM & high system load

[Dominique LARCHEY-WENDLING]

Andreas Dilger wrote:

> There is one other person that has reported very slow LVM performance.
> However, there is no obvious reason why they have performance issues,
> and others are unable to reproduce this problem.
> 
> Can you please try the following and report results:
> 
> time dd if=/dev/mmedia/music of=/dev/null bs=512
> time dd if=/dev/mmedia/music of=/dev/null bs=65536
> time dd if=/dev/hda1 of=/dev/null bs=512
> time dd if=/dev/hda1 of=/dev/null bs=65536

Here are the tests I have done ...  I use the command line

dd if=/dev/sda of=/dev/null bs=1k count=500k

to fill up the buffer cache before any timing measure.
I point out that /dev/mmedia/music resides on /dev/hda5

time dd if=/dev/mmedia/music of=/dev/null bs=512
count=200000               0m37.113s
time dd if=/dev/hda5 of=/dev/null bs=512
count=200000                       0m35.437s

Ok I am ridiculous ... this has nothing to do with LVM.
It has to do with DMA set off by default by linux 2.4.
This is the reason of poor IO perf. and CPU eating.
(as you might have understood, I usually do not use IDE drives)

After DMA is set on,

time dd if=/dev/mmedia/music of=/dev/null bs=512
count=200000               0m12.934s
time dd if=/dev/hda5 of=/dev/null bs=512
count=200000                       0m12.110s

Sorry for anoying you with my stupidity ...

Perhaps poor perf. for the other person is also DMA related ?-)

Thanks anyway

DL

-- 
Dominique Larchey
LORIA, Nancy, France
Tel. (Work) +33 (0) 3 83 59 20 13

Re: [linux-lvm] LVM & high system load

[Andreas Dilger]

Dominique Larchey writes:
> I point out that /dev/mmedia/music resides on /dev/hda5
> 
> time dd if=/dev/mmedia/music of=/dev/null bs=512
> count=200000               0m37.113s
> time dd if=/dev/hda5 of=/dev/null bs=512
> count=200000                       0m35.437s

LVM shows 4.7% overhead.  This is not so great.

> Ok I am ridiculous ... this has nothing to do with LVM.
> It has to do with DMA set off by default by linux 2.4.
> This is the reason of poor IO perf. and CPU eating.
> 
> After DMA is set on,
> 
> time dd if=/dev/mmedia/music of=/dev/null bs=512
> count=200000               0m12.934s
> time dd if=/dev/hda5 of=/dev/null bs=512
> count=200000                       0m12.110s

Good news - not an LVM problem.  Bad news - LVM shows 6.8% overhead.

> Perhaps poor perf. for the other person is also DMA related ?-)

No, the other person has drastically different performance with LVM for
512-byte I/O compared to raw disk.  For 64k-byte I/O performance on LVM
and raw disk are the same.  I _suppose_ that if I/O is done in small
chunks that PIO mode will have more overhead.  Maybe worth asking if
CPU is high when he is doing 512-byte I/Os.

Cheers, Andreas
-- 
Andreas Dilger                               Turbolinux filesystem development
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] LVM & high system load

[Heinz J. Mauelshagen]

On Tue, May 01, 2001 at 05:25:36PM -0600, Andreas Dilger wrote:
> Dominique Larchey writes:
> > I point out that /dev/mmedia/music resides on /dev/hda5
> > 
> > time dd if=/dev/mmedia/music of=/dev/null bs=512
> > count=200000               0m37.113s
> > time dd if=/dev/hda5 of=/dev/null bs=512
> > count=200000                       0m35.437s
> 
> LVM shows 4.7% overhead.  This is not so great.
> 
> > Ok I am ridiculous ... this has nothing to do with LVM.
> > It has to do with DMA set off by default by linux 2.4.
> > This is the reason of poor IO perf. and CPU eating.
> > 
> > After DMA is set on,
> > 
> > time dd if=/dev/mmedia/music of=/dev/null bs=512
> > count=200000               0m12.934s
> > time dd if=/dev/hda5 of=/dev/null bs=512
> > count=200000                       0m12.110s
> 
> Good news - not an LVM problem.  Bad news - LVM shows 6.8% overhead.

Did nothing else produce io load while the test on the LV was running?

Did you repeat tests in order to get statistic certainty?


> 
> > Perhaps poor perf. for the other person is also DMA related ?-)
> 
> No, the other person has drastically different performance with LVM for
> 512-byte I/O compared to raw disk.  For 64k-byte I/O performance on LVM
> and raw disk are the same.  I _suppose_ that if I/O is done in small
> chunks that PIO mode will have more overhead.  Maybe worth asking if
> CPU is high when he is doing 512-byte I/Os.

Yep.

> 
> Cheers, Andreas
> -- 
> Andreas Dilger                               Turbolinux filesystem development
> http://sourceforge.net/projects/ext2resize/
> http://www-mddsp.enel.ucalgary.ca/People/adilger/
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-