* [linux-lvm] LVM and *bad* performance (no striping)
@ 2001-02-26 15:44 Urs Thuermann
2001-02-26 20:39 ` Joe Thornber
2001-02-26 21:41 ` Rik van Riel
0 siblings, 2 replies; 22+ messages in thread
From: Urs Thuermann @ 2001-02-26 15:44 UTC (permalink / raw)
To: linux-lvm
I tried to use LVM today using Linux 2.4.2 and it mostly works like I
expected. But the performance is unacceptably bad:
isnogud:/root# fdisk -l /dev/sda|grep sda5
/dev/sda5 80 399 2570368+ 8e Linux LVM
isnogud:/root# pvcreate /dev/sda5
pvcreate -- physical volume "/dev/sda5" successfully created
isnogud:/root# vgcreate vg0 /dev/sda5
vgcreate -- INFO: using default physical extent size 4 MB
vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte
vgcreate -- doing automatic backup of volume group "vg0"
vgcreate -- volume group "vg0" successfully created and activated
isnogud:/root# lvcreate -n test /dev/vg0 -L 300M
lvcreate -- doing automatic backup of "vg0"
lvcreate -- logical volume "/dev/vg0/test" successfully created
isnogud:/root# time dd if=/dev/sda5 of=/dev/null count=100000
100000+0 records in
100000+0 records out
real 0m6.720s
user 0m0.300s
sys 0m0.970s
isnogud:/root# time dd if=/dev/vg0/test of=/dev/null count=100000
100000+0 records in
100000+0 records out
real 0m41.851s
user 0m0.350s
sys 0m2.250s
When making an ext2 filesystem on /dev/sda5 or /dev/vg0/test
respectivley, the performance of cp'ing, grep'ing all files and rm'ing
all files, seems to be comparable. But why is the performance of
reading the block device with dd so bad?
urs
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-02-26 15:44 [linux-lvm] LVM and *bad* performance (no striping) Urs Thuermann
@ 2001-02-26 20:39 ` Joe Thornber
2001-02-26 21:03 ` Jorg de Jong
` (2 more replies)
2001-02-26 21:41 ` Rik van Riel
1 sibling, 3 replies; 22+ messages in thread
From: Joe Thornber @ 2001-02-26 20:39 UTC (permalink / raw)
To: linux-lvm
On Mon, Feb 26, 2001 at 04:44:34PM +0100, Urs Thuermann wrote:
> I tried to use LVM today using Linux 2.4.2 and it mostly works like I
> expected. But the performance is unacceptably bad:
Every now and then someone posts to the list claiming LVM performance
is really bad. We go off and try it and find the overhead is
typically less than 1%.
Normally people are trying to stripe to two partitions on the same
device, but you aren't.
Is any one else on the list seeing a similar performance hit ?
- Joe
>
> isnogud:/root# fdisk -l /dev/sda|grep sda5
> /dev/sda5 80 399 2570368+ 8e Linux LVM
> isnogud:/root# pvcreate /dev/sda5
> pvcreate -- physical volume "/dev/sda5" successfully created
>
> isnogud:/root# vgcreate vg0 /dev/sda5
> vgcreate -- INFO: using default physical extent size 4 MB
> vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte
> vgcreate -- doing automatic backup of volume group "vg0"
> vgcreate -- volume group "vg0" successfully created and activated
>
> isnogud:/root# lvcreate -n test /dev/vg0 -L 300M
> lvcreate -- doing automatic backup of "vg0"
> lvcreate -- logical volume "/dev/vg0/test" successfully created
>
> isnogud:/root# time dd if=/dev/sda5 of=/dev/null count=100000
> 100000+0 records in
> 100000+0 records out
>
> real 0m6.720s
> user 0m0.300s
> sys 0m0.970s
> isnogud:/root# time dd if=/dev/vg0/test of=/dev/null count=100000
> 100000+0 records in
> 100000+0 records out
>
> real 0m41.851s
> user 0m0.350s
> sys 0m2.250s
>
> When making an ext2 filesystem on /dev/sda5 or /dev/vg0/test
> respectivley, the performance of cp'ing, grep'ing all files and rm'ing
> all files, seems to be comparable. But why is the performance of
> reading the block device with dd so bad?
>
>
> urs
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-02-26 20:39 ` Joe Thornber
@ 2001-02-26 21:03 ` Jorg de Jong
2001-02-26 21:38 ` Rik van Riel
2001-02-27 8:35 ` Patrick Caulfield
2001-02-27 12:21 ` Urs Thuermann
2 siblings, 1 reply; 22+ messages in thread
From: Jorg de Jong @ 2001-02-26 21:03 UTC (permalink / raw)
To: linux-lvm
Joe Thornber wrote:
>
> On Mon, Feb 26, 2001 at 04:44:34PM +0100, Urs Thuermann wrote:
> > I tried to use LVM today using Linux 2.4.2 and it mostly works like I
> > expected. But the performance is unacceptably bad:
>
> Every now and then someone posts to the list claiming LVM performance
> is really bad. We go off and try it and find the overhead is
> typically less than 1%.
>
> Normally people are trying to stripe to two partitions on the same
> device, but you aren't.
>
> Is any one else on the list seeing a similar performance hit ?
>
> - Joe
here are my stats... (be your owen judge....)
Jorg
[root] /root > time dd if=/dev/hde of=/dev/null count=100000
100000+0 records in
100000+0 records out
0.200u 1.050s 0:03.95 31.6% 0+0k 0+0io 116pf+0w
[root] /root > time dd if=/dev/vg00/vm of=/dev/null count=100000
100000+0 records in
100000+0 records out
0.160u 0.860s 0:02.92 34.9% 0+0k 0+0io 116pf+0w
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-02-26 21:03 ` Jorg de Jong
@ 2001-02-26 21:38 ` Rik van Riel
0 siblings, 0 replies; 22+ messages in thread
From: Rik van Riel @ 2001-02-26 21:38 UTC (permalink / raw)
To: linux-lvm
On Mon, 26 Feb 2001, Jorg de Jong wrote:
> here are my stats... (be your owen judge....)
> [root] /root > time dd if=/dev/hde of=/dev/null count=100000
> 100000+0 records in
> 100000+0 records out
> 0.200u 1.050s 0:03.95 31.6% 0+0k 0+0io 116pf+0w
>
> [root] /root > time dd if=/dev/vg00/vm of=/dev/null count=100000
> 100000+0 records in
> 100000+0 records out
> 0.160u 0.860s 0:02.92 34.9% 0+0k 0+0io 116pf+0w
Interesting, so LVM actually takes a full second LESS to
read in the same data ...
Maybe LVM does some device-level readahead to speed things
up ?
regards,
Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...
http://www.surriel.com/
http://www.conectiva.com/ http://distro.conectiva.com/
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-02-26 15:44 [linux-lvm] LVM and *bad* performance (no striping) Urs Thuermann
2001-02-26 20:39 ` Joe Thornber
@ 2001-02-26 21:41 ` Rik van Riel
2001-02-27 10:33 ` Urs Thuermann
1 sibling, 1 reply; 22+ messages in thread
From: Rik van Riel @ 2001-02-26 21:41 UTC (permalink / raw)
To: linux-lvm
On 26 Feb 2001, Urs Thuermann wrote:
> isnogud:/root# time dd if=/dev/sda5 of=/dev/null count=100000
> 100000+0 records in
> real 0m6.720s
> isnogud:/root# time dd if=/dev/vg0/test of=/dev/null count=100000
> 100000+0 records in
> real 0m41.851s
> When making an ext2 filesystem on /dev/sda5 or /dev/vg0/test
> respectivley, the performance of cp'ing, grep'ing all files and rm'ing
> all files, seems to be comparable. But why is the performance of
> reading the block device with dd so bad?
Maybe dd is reading in 512 byte blocks on /dev/sda5 and
4096 bytes on /dev/vg0/test ?
(I'm not sure if it is, but these blocksizes are the
default blocksize on each device and you only told dd
to "read 100000 blocks" ... this ratio also explains
the time difference of 8 times)
regards,
Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...
http://www.surriel.com/
http://www.conectiva.com/ http://distro.conectiva.com/
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-02-26 20:39 ` Joe Thornber
2001-02-26 21:03 ` Jorg de Jong
@ 2001-02-27 8:35 ` Patrick Caulfield
2001-02-27 12:21 ` Urs Thuermann
2 siblings, 0 replies; 22+ messages in thread
From: Patrick Caulfield @ 2001-02-27 8:35 UTC (permalink / raw)
To: linux-lvm
On Mon, Feb 26, 2001 at 08:39:33PM +0000, Joe Thornber wrote:
> On Mon, Feb 26, 2001 at 04:44:34PM +0100, Urs Thuermann wrote:
> > I tried to use LVM today using Linux 2.4.2 and it mostly works like I
> > expected. But the performance is unacceptably bad:
>
> Every now and then someone posts to the list claiming LVM performance
> is really bad. We go off and try it and find the overhead is
> typically less than 1%.
>
> Normally people are trying to stripe to two partitions on the same
> device, but you aren't.
I've always seen bad performance doing dd on LVM volumes but very good
performance in "real world" performance or bonnie++ tests.
Stop worrying about it, how many times do you *need* to dd the start of
a disk at high speed eh? :-)
patrick
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-02-26 21:41 ` Rik van Riel
@ 2001-02-27 10:33 ` Urs Thuermann
0 siblings, 0 replies; 22+ messages in thread
From: Urs Thuermann @ 2001-02-27 10:33 UTC (permalink / raw)
To: linux-lvm
Rik van Riel <riel@conectiva.com.br> writes:
> Maybe dd is reading in 512 byte blocks on /dev/sda5 and
> 4096 bytes on /dev/vg0/test ?
I've checked it. The man page says if no bs is given, dd defaults to
512 bytes on any device. I've also checked it with dd|wc.
The performance numbers I've observed seem to dependant on caching.
urs
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-02-26 20:39 ` Joe Thornber
2001-02-26 21:03 ` Jorg de Jong
2001-02-27 8:35 ` Patrick Caulfield
@ 2001-02-27 12:21 ` Urs Thuermann
2001-02-28 11:05 ` Heinz J. Mauelshagen
2 siblings, 1 reply; 22+ messages in thread
From: Urs Thuermann @ 2001-02-27 12:21 UTC (permalink / raw)
To: linux-lvm
Joe Thornber <joe@66bassett.freeserve.co.uk> writes:
> Every now and then someone posts to the list claiming LVM performance
> is really bad. We go off and try it and find the overhead is
> typically less than 1%.
>
> Normally people are trying to stripe to two partitions on the same
> device, but you aren't.
>
> Is any one else on the list seeing a similar performance hit ?
I did some more tests to ensure that this is not a caching issue.
Also, I found the performance of dd on LVM depends much on the block
size.
I still have the effect that reading from LVM with small block size
(smaller than 8k on my system) is *much* slower than reading the
physical disk directly. IMHO it'd be worth looking into it and
possibly fixing this performance hit. Could it be possible, that the
read syscall on the LVM block device has so much overhead that reading
small blocks can't cope with SCSI disk's transfer rate, so that
additional disk revolutions are needed to read all the physical
blocks? I find it noticable that when doubling the block size from
1k to 8k the time for reading quite exactly halves until it reaches
the performance of reading /dev/sda5 directly.
In case it matters, the machine I did the test on is a
ASUS P2L97-S mainboard
PII 333MHz
128 MB RAM
Adaptec AIC-7880U onboard UW-SCSI controller
/dev/sda is a IBM DCAS-34330W
kernel: Linux 2.4.2
I've put the commands into a script that uses "set -x" to show what is
exec'ed. Here's the result:
Initialisation:
+ pvcreate /dev/sda5
pvcreate -- physical volume "/dev/sda5" successfully created
+ vgcreate vg0 /dev/sda5
vgcreate -- INFO: using default physical extent size 4 MB
vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte
vgcreate -- doing automatic backup of volume group "vg0"
vgcreate -- volume group "vg0" successfully created and activated
+ lvcreate -n test /dev/vg0 -L 500M
lvcreate -- doing automatic backup of "vg0"
lvcreate -- logical volume "/dev/vg0/test" successfully created
The first test shows, that, with small block sizes, LVM over sda5 is
much slower than reading sda5 directly. I do the reading three times
to see if any caching affects the numbers. This is obviously not the
case:
+ dd if=/dev/sda5 of=/dev/null count=128k bs=512
131072+0 records in
131072+0 records out
real 0m8.862s
user 0m0.120s
sys 0m1.690s
+ dd if=/dev/sda5 of=/dev/null count=128k bs=512
131072+0 records in
131072+0 records out
real 0m8.860s
user 0m0.200s
sys 0m1.670s
+ dd if=/dev/sda5 of=/dev/null count=128k bs=512
131072+0 records in
131072+0 records out
real 0m8.860s
user 0m0.110s
sys 0m1.800s
+ dd if=/dev/vg0/test of=/dev/null count=128k bs=512
131072+0 records in
131072+0 records out
real 0m54.860s
user 0m0.150s
sys 0m2.780s
+ dd if=/dev/vg0/test of=/dev/null count=128k bs=512
131072+0 records in
131072+0 records out
real 0m54.810s
user 0m0.170s
sys 0m2.660s
+ dd if=/dev/vg0/test of=/dev/null count=128k bs=512
131072+0 records in
131072+0 records out
real 0m54.936s
user 0m0.200s
sys 0m3.010s
The next test shows that the performance of LVM is very dependend on
the blocksize but that of sda5 is not, which I find surprising since
the same physical device is involved.
+ dd if=/dev/sda5 of=/dev/null count=512 bs=128k
512+0 records in
512+0 records out
real 0m8.857s
user 0m0.010s
sys 0m1.440s
+ dd if=/dev/vg0/test of=/dev/null count=128k bs=512
131072+0 records in
131072+0 records out
real 0m55.120s
user 0m0.190s
sys 0m2.760s
+ dd if=/dev/vg0/test of=/dev/null count=64k bs=1k
65536+0 records in
65536+0 records out
real 0m55.195s
user 0m0.170s
sys 0m2.970s
+ dd if=/dev/vg0/test of=/dev/null count=32k bs=2k
32768+0 records in
32768+0 records out
real 0m28.470s
user 0m0.090s
sys 0m1.830s
+ dd if=/dev/vg0/test of=/dev/null count=16k bs=4k
16384+0 records in
16384+0 records out
real 0m15.209s
user 0m0.010s
sys 0m1.860s
+ dd if=/dev/vg0/test of=/dev/null count=8k bs=8k
8192+0 records in
8192+0 records out
real 0m8.859s
user 0m0.010s
sys 0m1.670s
+ dd if=/dev/vg0/test of=/dev/null count=4k bs=16k
4096+0 records in
4096+0 records out
real 0m8.860s
user 0m0.010s
sys 0m1.570s
+ dd if=/dev/vg0/test of=/dev/null count=2k bs=32k
2048+0 records in
2048+0 records out
real 0m8.873s
user 0m0.010s
sys 0m1.390s
+ dd if=/dev/vg0/test of=/dev/null count=1k bs=64k
1024+0 records in
1024+0 records out
real 0m8.883s
user 0m0.000s
sys 0m1.380s
+ dd if=/dev/vg0/test of=/dev/null count=512 bs=128k
512+0 records in
512+0 records out
real 0m8.862s
user 0m0.000s
sys 0m1.370s
Performance on filessystems on LVM is better, since the buffer cache
can be used. I thought that the buffer cache works on block devices.
But this seems to suggest that it only works with files systems
mounted on the block device. Is this really so and why?
+ mke2fs -q /dev/vg0/test
mke2fs 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09
+ mount /dev/vg0/test /mnt
+ dd if=/dev/vg0/test of=/dev/null count=512 bs=128k
512+0 records in
512+0 records out
real 0m8.792s
user 0m0.000s
sys 0m1.280s
+ dd if=/dev/vg0/test of=/dev/null count=512 bs=128k
512+0 records in
512+0 records out
real 0m1.044s
user 0m0.000s
sys 0m1.050s
For tests like this it would be useful to be able to completely flush
the kernel's cache. Is there a tool to do this?
urs
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-02-27 12:21 ` Urs Thuermann
@ 2001-02-28 11:05 ` Heinz J. Mauelshagen
2001-04-10 19:30 ` Urs Thuermann
0 siblings, 1 reply; 22+ messages in thread
From: Heinz J. Mauelshagen @ 2001-02-28 11:05 UTC (permalink / raw)
To: linux-lvm
Hi Urs,
I can't reproduce at all what you are seeing with your configuration.
My test configuration is:
- Dual Pentium II/350
- 256MB RAM
- Adaptec 2940UW
- HDD Vendor: QUANTUM Model: XP34550W Rev: LXY4 (/dev/sdb)
- first 300M of the LV involved allocated contiguous at the beginning
of /dev/sdb1
- LVM 0.9.1 Beta 5
- Linux 2.4.2
- dd (GNU fileutils) 4.0
This table records test runs which where run one after the other
*without* any additional load on the machine using dd to copy 300m
into /dev/null. Each run performed twice with little difference + build average.
1k 2k 4k 8k 16k
-----------------------------------------------------------
sdb1 32.63 32.8 32,7 32.55 32.9
vg00/u1 32.58 33.1 32.63 32.4 32.4
This gives *no* hint to me that there's any basical overhead invloved
with LVM as oposed to the native device beeing used at all.
Therefore I guess that the allocation policy of LVM might have an impact here.
LVM calculates the offset of the first PE from the end of the PV.
There could be drives used which suffer from that placement with
regard to drive internal caching, latency or other model specific issues.
Could everybody who posts with regard to this performance subject please provide
- "lvdisplay -v" of the test LV
- the drive vendor and model
- the host adaptor used to access it
- the motherboard and CPU(s)
- the RAM size
in order to get a better understanding.
BTW: Rik van Riel's assumption in another mail of this thread, that LVM
does some sort of read ahead is correct (as you can see with the
above numbers).
Even though we still only have a per block major read ahead in the kernel
*and* multithreading will make the actual read ahead implementation fail
with regard parallel access LVs (as Andrea Arcangeli pointed out a while
ago), it hits quite good on not too heavy loaded systems.
Taking into account that a lot of users most likely don't change the
read ahead default of LVs, it won't have an impact at all.
Regards,
Heinz -- The LVM Guy --
On Tue, Feb 27, 2001 at 01:21:14PM +0100, Urs Thuermann wrote:
> Joe Thornber <joe@66bassett.freeserve.co.uk> writes:
>
> > Every now and then someone posts to the list claiming LVM performance
> > is really bad. We go off and try it and find the overhead is
> > typically less than 1%.
> >
> > Normally people are trying to stripe to two partitions on the same
> > device, but you aren't.
> >
> > Is any one else on the list seeing a similar performance hit ?
>
> I did some more tests to ensure that this is not a caching issue.
> Also, I found the performance of dd on LVM depends much on the block
> size.
>
> I still have the effect that reading from LVM with small block size
> (smaller than 8k on my system) is *much* slower than reading the
> physical disk directly. IMHO it'd be worth looking into it and
> possibly fixing this performance hit. Could it be possible, that the
> read syscall on the LVM block device has so much overhead that reading
> small blocks can't cope with SCSI disk's transfer rate, so that
> additional disk revolutions are needed to read all the physical
> blocks? I find it noticable that when doubling the block size from
> 1k to 8k the time for reading quite exactly halves until it reaches
> the performance of reading /dev/sda5 directly.
>
> In case it matters, the machine I did the test on is a
> ASUS P2L97-S mainboard
> PII 333MHz
> 128 MB RAM
> Adaptec AIC-7880U onboard UW-SCSI controller
> /dev/sda is a IBM DCAS-34330W
> kernel: Linux 2.4.2
>
> I've put the commands into a script that uses "set -x" to show what is
> exec'ed. Here's the result:
>
> Initialisation:
>
> + pvcreate /dev/sda5
> pvcreate -- physical volume "/dev/sda5" successfully created
>
> + vgcreate vg0 /dev/sda5
> vgcreate -- INFO: using default physical extent size 4 MB
> vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte
> vgcreate -- doing automatic backup of volume group "vg0"
> vgcreate -- volume group "vg0" successfully created and activated
>
> + lvcreate -n test /dev/vg0 -L 500M
> lvcreate -- doing automatic backup of "vg0"
> lvcreate -- logical volume "/dev/vg0/test" successfully created
>
>
> The first test shows, that, with small block sizes, LVM over sda5 is
> much slower than reading sda5 directly. I do the reading three times
> to see if any caching affects the numbers. This is obviously not the
> case:
>
> + dd if=/dev/sda5 of=/dev/null count=128k bs=512
> 131072+0 records in
> 131072+0 records out
>
> real 0m8.862s
> user 0m0.120s
> sys 0m1.690s
> + dd if=/dev/sda5 of=/dev/null count=128k bs=512
> 131072+0 records in
> 131072+0 records out
>
> real 0m8.860s
> user 0m0.200s
> sys 0m1.670s
> + dd if=/dev/sda5 of=/dev/null count=128k bs=512
> 131072+0 records in
> 131072+0 records out
>
> real 0m8.860s
> user 0m0.110s
> sys 0m1.800s
> + dd if=/dev/vg0/test of=/dev/null count=128k bs=512
> 131072+0 records in
> 131072+0 records out
>
> real 0m54.860s
> user 0m0.150s
> sys 0m2.780s
> + dd if=/dev/vg0/test of=/dev/null count=128k bs=512
> 131072+0 records in
> 131072+0 records out
>
> real 0m54.810s
> user 0m0.170s
> sys 0m2.660s
> + dd if=/dev/vg0/test of=/dev/null count=128k bs=512
> 131072+0 records in
> 131072+0 records out
>
> real 0m54.936s
> user 0m0.200s
> sys 0m3.010s
>
>
> The next test shows that the performance of LVM is very dependend on
> the blocksize but that of sda5 is not, which I find surprising since
> the same physical device is involved.
>
> + dd if=/dev/sda5 of=/dev/null count=512 bs=128k
> 512+0 records in
> 512+0 records out
>
> real 0m8.857s
> user 0m0.010s
> sys 0m1.440s
> + dd if=/dev/vg0/test of=/dev/null count=128k bs=512
> 131072+0 records in
> 131072+0 records out
>
> real 0m55.120s
> user 0m0.190s
> sys 0m2.760s
> + dd if=/dev/vg0/test of=/dev/null count=64k bs=1k
> 65536+0 records in
> 65536+0 records out
>
> real 0m55.195s
> user 0m0.170s
> sys 0m2.970s
> + dd if=/dev/vg0/test of=/dev/null count=32k bs=2k
> 32768+0 records in
> 32768+0 records out
>
> real 0m28.470s
> user 0m0.090s
> sys 0m1.830s
> + dd if=/dev/vg0/test of=/dev/null count=16k bs=4k
> 16384+0 records in
> 16384+0 records out
>
> real 0m15.209s
> user 0m0.010s
> sys 0m1.860s
> + dd if=/dev/vg0/test of=/dev/null count=8k bs=8k
> 8192+0 records in
> 8192+0 records out
>
> real 0m8.859s
> user 0m0.010s
> sys 0m1.670s
> + dd if=/dev/vg0/test of=/dev/null count=4k bs=16k
> 4096+0 records in
> 4096+0 records out
>
> real 0m8.860s
> user 0m0.010s
> sys 0m1.570s
> + dd if=/dev/vg0/test of=/dev/null count=2k bs=32k
> 2048+0 records in
> 2048+0 records out
>
> real 0m8.873s
> user 0m0.010s
> sys 0m1.390s
> + dd if=/dev/vg0/test of=/dev/null count=1k bs=64k
> 1024+0 records in
> 1024+0 records out
>
> real 0m8.883s
> user 0m0.000s
> sys 0m1.380s
> + dd if=/dev/vg0/test of=/dev/null count=512 bs=128k
> 512+0 records in
> 512+0 records out
>
> real 0m8.862s
> user 0m0.000s
> sys 0m1.370s
>
>
> Performance on filessystems on LVM is better, since the buffer cache
> can be used. I thought that the buffer cache works on block devices.
> But this seems to suggest that it only works with files systems
> mounted on the block device. Is this really so and why?
>
> + mke2fs -q /dev/vg0/test
> mke2fs 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09
> + mount /dev/vg0/test /mnt
> + dd if=/dev/vg0/test of=/dev/null count=512 bs=128k
> 512+0 records in
> 512+0 records out
>
> real 0m8.792s
> user 0m0.000s
> sys 0m1.280s
> + dd if=/dev/vg0/test of=/dev/null count=512 bs=128k
> 512+0 records in
> 512+0 records out
>
> real 0m1.044s
> user 0m0.000s
> sys 0m1.050s
>
>
> For tests like this it would be useful to be able to completely flush
> the kernel's cache. Is there a tool to do this?
>
>
> urs
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
Mauelshagen@Sistina.com +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-02-28 11:05 ` Heinz J. Mauelshagen
@ 2001-04-10 19:30 ` Urs Thuermann
2001-04-10 21:37 ` Andreas Dilger
0 siblings, 1 reply; 22+ messages in thread
From: Urs Thuermann @ 2001-04-10 19:30 UTC (permalink / raw)
To: linux-lvm
Today, I've found some time to make some further performance tests on
my system.
"Heinz J. Mauelshagen" <Mauelshagen@sistina.com> writes:
> Hi Urs,
>
> I can't reproduce at all what you are seeing with your configuration.
>
>
> My test configuration is:
>
> - Dual Pentium II/350
> - 256MB RAM
> - Adaptec 2940UW
> - HDD Vendor: QUANTUM Model: XP34550W Rev: LXY4 (/dev/sdb)
> - first 300M of the LV involved allocated contiguous at the beginning
> of /dev/sdb1
> - LVM 0.9.1 Beta 5
> - Linux 2.4.2
> - dd (GNU fileutils) 4.0
I find surprinsing that we get so different performance results since
my hardware is very similar to yours. Here is it, again:
Mainboard: ASUS P2L97-S with UW-SCSI controller onboard
RAM: 128 MB
CPU: from /proc/cpuinfo:
model name : Pentium II (Deschutes)
stepping : 1
cpu MHz : 334.095
/dev/sda: from /proc/scsi/scsi:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: DCAS-34330W Rev: S65A
Type: Direct-Access ANSI SCSI revision: 02
SCSI controller: from /proc/scsi/aic7xxx/0:
Adaptec AIC7xxx driver version: 6.1.5
aic7880: Wide Channel A, SCSI Id=7, 16/255 SCBs
Channel A Target 0 Negotiation Settings
User: 40.000MB/s transfers (20.000MHz, offset 255, 16bit)
Goal: 40.000MB/s transfers (20.000MHz, offset 8, 16bit)
Curr: 40.000MB/s transfers (20.000MHz, offset 8, 16bit)
Channel A Target 0 Lun 0 Settings
Commands Queued 13557298
Commands Active 0
Command Openings 64
Max Tagged Openings 64
Device Queue Frozen Count 0
kernel: Linux-2.4.3
The size of /dev/sda5 is
# fdisk -l /dev/sda | egrep '^[^/]|sda5'
Disk /dev/sda: 255 heads, 63 sectors, 527 cylinders
Units = cylinders of 16065 * 512 bytes
Device Boot Start End Blocks Id System
/dev/sda5 80 399 2570368+ 8e Linux LVM
i.e. 627 PEs of 4MB each. After
pvcreate /dev/sda5; vgcreate vg0 /dev/sda5; lvcreate -n test vg0 -l 627
I have
# vgdisplay -v
--- Volume group ---
VG Name vg0
VG Access read/write
VG Status available/resizable
VG # 0
MAX LV 256
Cur LV 1
Open LV 0
MAX LV Size 255.99 GB
Max PV 256
Cur PV 1
Act PV 1
VG Size 2.45 GB
PE Size 4 MB
Total PE 627
Alloc PE / Size 627 / 2.45 GB
Free PE / Size 0 / 0
VG UUID zmdKrC-eis5-rgWV-C5rz-TcJE-8tTI-iUUeBj
--- Logical volume ---
LV Name /dev/vg0/test
VG Name vg0
LV Write Access read/write
LV Status available
LV # 1
# open 0
LV Size 2.45 GB
Current LE 627
Allocated LE 627
Allocation next free
Read ahead sectors 120
Block device 58:0
--- Physical volumes ---
PV Name (#) /dev/sda5 (1)
PV Status available / allocatable
Total PE / Free PE 627 / 0
and lvdisplay -v /dev/vg0/test shows a mapping of
--- Logical extents ---
LE PV PE reads writes
00000 /dev/sda5 00000 278528 2
00001 /dev/sda5 00001 266240 0
00002 /dev/sda5 00002 266240 0
00003 /dev/sda5 00003 266240 0
...
00625 /dev/sda5 00625 266240 0
00626 /dev/sda5 00626 266240 0
> This table records test runs which where run one after the other
> *without* any additional load on the machine using dd to copy 300m
> into /dev/null. Each run performed twice with little difference +
> build average.
>
> 1k 2k 4k 8k 16k
> -----------------------------------------------------------
> sdb1 32.63 32.8 32,7 32.55 32.9
> vg00/u1 32.58 33.1 32.63 32.4 32.4
For me, numbers look quite different. I have then run the following
test, which reads all the 627 PEs (2.45 GB) from /dev/vg0/test and the
same amount from /dev/sda5 at different block sizes. The system was
otherwise almost idle when I ran this test.
#!/bin/sh
function doit() {
time dd if=/dev/sda5 of=/dev/null count=$1 bs=$2
time dd if=/dev/sda5 of=/dev/null count=$1 bs=$2
time dd if=/dev/sda5 of=/dev/null count=$1 bs=$2
time dd if=/dev/sda5 of=/dev/null count=$1 bs=$2
time dd if=/dev/vg0/test of=/dev/null count=$1 bs=$2
time dd if=/dev/vg0/test of=/dev/null count=$1 bs=$2
time dd if=/dev/vg0/test of=/dev/null count=$1 bs=$2
time dd if=/dev/vg0/test of=/dev/null count=$1 bs=$2
}
set -x
doit 40128 64k
doit 80256 32k
doit 160512 16k
doit 321024 8k
doit 642048 4k
doit 1284096 2k
doit 2568192 1k
doit 5136384 512
The elapsed times of the 4 runs of each test are averaged, where for
each test the variations among the 4 times were less than 1% of the
average time (times in seconds):
64k 32k 16k 8k 4k 2k 1k 512
------------------------------------------------------------------------------
/dev/sda5 389.4 389.4 388.9 388.7 388.5 388.6 389.1 388.9
/dev/vg0/test 389.3 389.6 390.9 393.1 607.5 1147.2 2194.9 2195.8
> Therefore I guess that the allocation policy of LVM might have an
> impact here.
>
> LVM calculates the offset of the first PE from the end of the PV.
> There could be drives used which suffer from that placement with
> regard to drive internal caching, latency or other model specific
> issues.
I think you mean that when I read a number of blocks from /dev/sda5 I
read them from the beginning of /dev/sda5 while when reading the same
number of blocks from /dev/vg0/test I read from the end of the PV
/dev/sda5 since the PE are allocated from the end, right.
But in my test, I read the whole /dev/sda5 and the whole LV which
has allocated all PEs of /dev/sda5 to it. So about the same sectors
are read, just the order is different. Do you think that can reduce
the performance from 2.45GB/390s = 6.43MB/s to 2.45GB/2195s = 1.14MB/s?
I find this very unlikely.
Is there any other explanation for this performance degradation?
urs
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-04-10 19:30 ` Urs Thuermann
@ 2001-04-10 21:37 ` Andreas Dilger
2001-04-11 11:23 ` Urs Thuermann
0 siblings, 1 reply; 22+ messages in thread
From: Andreas Dilger @ 2001-04-10 21:37 UTC (permalink / raw)
To: linux-lvm
Urs, you write:
> Today, I've found some time to make some further performance tests on
> my system.
>
> The elapsed times of the 4 runs of each test are averaged, where for
> each test the variations among the 4 times were less than 1% of the
> average time (times in seconds):
>
> 64k 32k 16k 8k 4k 2k 1k 512
> ------------------------------------------------------------------------------
> /dev/sda5 389.4 389.4 388.9 388.7 388.5 388.6 389.1 388.9
> /dev/vg0/test 389.3 389.6 390.9 393.1 607.5 1147.2 2194.9 2195.8
>
> > There could be drives used which suffer from that placement with
> > regard to drive internal caching, latency or other model specific
> > issues.
>
> I think you mean that when I read a number of blocks from /dev/sda5 I
> read them from the beginning of /dev/sda5 while when reading the same
> number of blocks from /dev/vg0/test I read from the end of the PV
> /dev/sda5 since the PE are allocated from the end, right.
No, PEs are allocated from the start, but are only aligned with the end
of the disk/partition (to give maximum room to LVM for metadata). If you
have contiguous LE allocation (which you appear to have), the read order
for LVM should basically be exactly the same as for a raw disk. Effectively
all that LVM is doing is adding a small offset to the disk blocks to leave
room for the VGDA, nothing more.
> But in my test, I read the whole /dev/sda5 and the whole LV which
> has allocated all PEs of /dev/sda5 to it. So about the same sectors
> are read, just the order is different. Do you think that can reduce
> the performance from 2.45GB/390s = 6.43MB/s to 2.45GB/2195s = 1.14MB/s?
> I find this very unlikely.
No, shouldn't be a problem with placement, because you are reading 4MB
contiguous chunks (with default PE size). This should be enough to
minimize seek impact.
Do you notice the sound of frantic head seeking when you are running with
small block sizes?
> Is there any other explanation for this performance degradation?
Other than some strange seek pattern, I can't think of anything. If you
look at the (non-snapshot and non-blocking because of pv_move) path of
lvm_map, it boils down to:
pe_num = lvm_block / pe_size;
new_dev = pe[pe_num].dev;
new_block = (old_block % pe_size) + pe[pe_num].start;
So basically a few memory accesses and a few math operations. Nothing
_should_ affect disk performance at all. With < 2k I/O you are forcing
read-modify-write on pages, but this shouldn't be an issue, because you
are also doing the same for the raw disk.
One thing that now comes to mind is if the LVM I/O is not aligned, because
LVM PEs are aligned with the end of the PV and not the start, this _may_
cause performance problems because of misaligned read-modify-write causing
extra seeking (purely speculation).
Try the following patch, which will align the PE structs to 64kB boundaries
from the start of the disk, as well as making the other large VGDA structs
start on 4k boundaries. I have tested the new alignment, and it works
fine with existing LVM code. However, the patch is extracted from 8000+
lines of diff (yuk), so there may be a few cases where it doesn't apply
cleanly or I missed something in a non-critical file. Basically, if it
compiles it will work OK.
Cheers, Andreas
======================= lvm-0.9.1b7-align.diff ===========================
diff -u -u -r1.5.2.38 lvm.h
--- kernel/lvm.h 2001/04/10 17:04:18 1.5.2.38
+++ kernel/lvm.h 2001/04/10 20:44:36
@@ -246,39 +246,34 @@
*/
/* DONT TOUCH THESE !!! */
+#define LVM_VGDA_ALIGN (4096UL - 1)
+#define LVM_VGDA_ROUND(size) (((size) + LVM_VGDA_ALIGN) & ~LVM_VGDA_ALIGN)
+
/* base of PV structure in disk partition */
-#define LVM_PV_DISK_BASE 0L
+#define LVM_PV_DISK_BASE 0L
/* size reserved for PV structure on disk */
-#define LVM_PV_DISK_SIZE 1024L
+#define LVM_PV_DISK_SIZE 1024L
/* base of VG structure in disk partition */
-#define LVM_VG_DISK_BASE LVM_PV_DISK_SIZE
+#define LVM_VG_DISK_BASE LVM_VGDA_ROUND ( LVM_PV_DISK_BASE + \
+ LVM_PV_DISK_SIZE)
/* size reserved for VG structure */
-#define LVM_VG_DISK_SIZE ( 9 * 512L)
-
-/* size reserved for timekeeping */
-#define LVM_TIMESTAMP_DISK_BASE ( LVM_VG_DISK_BASE + LVM_VG_DISK_SIZE)
-#define LVM_TIMESTAMP_DISK_SIZE 512L /* reserved for timekeeping */
+#define LVM_VG_DISK_SIZE sizeof ( vg_disk_t)
/* name list of physical volumes on disk */
-#define LVM_PV_UUIDLIST_DISK_BASE ( LVM_TIMESTAMP_DISK_BASE + \
- LVM_TIMESTAMP_DISK_SIZE)
+#define LVM_PV_UUIDLIST_DISK_BASE LVM_VGDA_ROUND ( LVM_VG_DISK_BASE + \
+ LVM_VG_DISK_SIZE)
/* now for the dynamically calculated parts of the VGDA */
#define LVM_LV_DISK_OFFSET(a, b) ( (a)->lv_on_disk.base + \
sizeof ( lv_disk_t) * b)
-#define LVM_DISK_SIZE(pv) ( (pv)->pe_on_disk.base + \
+#define LVM_VGDA_SIZE(pv) ( (pv)->pe_on_disk.base + \
(pv)->pe_on_disk.size)
#define LVM_PE_DISK_OFFSET(pe, pv) ( pe * pv->pe_size + \
- ( LVM_DISK_SIZE ( pv) / SECTOR_SIZE))
-#define LVM_PE_ON_DISK_BASE(pv) \
- { int rest; \
- pv->pe_on_disk.base = pv->lv_on_disk.base + pv->lv_on_disk.size; \
- if ( ( rest = pv->pe_on_disk.base % SECTOR_SIZE) != 0) \
- pv->pe_on_disk.base += ( SECTOR_SIZE - rest); \
- }
+ ( LVM_VGDA_SIZE ( pv) / SECTOR_SIZE))
+#define LVM_PE_ALIGN (65536UL / SECTOR_SIZE - 1)
/* END default disk spaces and offsets for PVs */
diff -u -u -r1.2.2.8 vgcreate.c
--- tools/vgcreate.c 2001/04/09 12:53:40 1.2.2.8
+++ tools/vgcreate.c 2001/04/10 20:44:43
@@ -101,8 +101,8 @@
int opt_v = 0;
int p = 0;
int p1 = 0;
- int max_lv = MAX_LV;
- int max_pv = MAX_PV;
+ int max_lv = MAX_LV - 1;
+ int max_pv = MAX_PV - 1;
int min_pv_index = 0;
ulong max_pv_size = 0;
ulong min_pv_size = -1;
diff -u -u -r1.5.2.17 liblvm.h
--- tools/lib/liblvm.h 2001/04/09 14:18:25 1.5.2.17
+++ tools/lib/liblvm.h 2001/04/10 20:44:46
@@ -99,13 +99,9 @@
extern struct config_file *config_file;
-#define SECTOR_SIZE 512
#define BLKGETSIZE _IO(0x12,96) /* return device size */
#define BLKRASET _IO(0x12,98) /* Set read ahead */
#define EXT2_SUPER_MAGIC 0xEF53
-#ifndef BLOCK_SIZE
-#define BLOCK_SIZE 1024
-#endif
#define LVM_ID "HM" /* Identifier PV (id in pv_t) */
#define EXPORTED "PV_EXP" /* Identifier exported PV (system_id in pv_t) */
diff -u -u -r1.2.2.2 pv_read_pe.c
--- tools/lib/pv_read_pe.c 2001/04/09 07:49:07 1.2.2.2
+++ tools/lib/pv_read_pe.c 2001/04/10 20:44:52
@@ -58,8 +57,7 @@
*pe = NULL;
size = pv->pe_total * sizeof ( pe_disk_t);
- if ( size + pv->pe_on_disk.base >
- LVM_DISK_SIZE ( pv)) {
+ if ( size + pv->pe_on_disk.base > LVM_VGDA_SIZE ( pv)) {
ret = -LVM_EPV_READ_PE_SIZE;
goto pv_read_pe_end;
}
diff -u -u -r1.1.2.1 pv_release_pe.c
--- tools/lib/pv_release_pe.c 2001/02/19 10:04:34 1.1.2.1
+++ tools/lib/pv_release_pe.c 2001/04/10 20:44:52
@@ -82,8 +82,7 @@
goto pv_release_pe_end;
}
pe_index = ( vg->lv[l]->lv_current_pe[p].pe -
- LVM_DISK_SIZE ( vg->pv[pv_num]) /
- SECTOR_SIZE) /
+ LVM_VGDA_SIZE ( vg->pv[pv_num]) / SECTOR_SIZE) /
vg->pe_size;
debug ( "pv_release_pe -- pv_name: %s pe: %lu sector: %lu\n",
vg->pv[pv_num]->pv_name,
diff -u -u -r1.2.2.1 pv_write_pe.c
--- tools/lib/pv_write_pe.c 2001/02/19 10:04:34 1.2.2.1
+++ tools/lib/pv_write_pe.c 2001/04/10 20:44:52
@@ -50,7 +50,7 @@
pv == NULL) ret = -LVM_EPARAM;
else {
size = pv->pe_total * sizeof ( pe_disk_t);
- if ( size + pv->pe_on_disk.base > LVM_DISK_SIZE ( pv))
+ if ( size + pv->pe_on_disk.base > LVM_VGDA_SIZE ( pv))
ret = -LVM_EPV_WRITE_PE_SIZE;;
if ( ( pv_handle = open ( pv_name, O_WRONLY)) == -1)
ret = -LVM_EPV_WRITE_PE_OPEN;
diff -u -u -r1.4.2.1 vg_setup_for_create.c
--- tools/lib/vg_setup_for_create.c 2001/02/19 10:04:34 1.4.2.1
+++ tools/lib/vg_setup_for_create.c 2001/04/10 20:44:54
@@ -41,12 +41,9 @@
int vg_setup_for_create ( char *vg_name, vg_t *vg, pv_t **pv,
int pe_size, ulong max_pv, ulong max_lv) {
- int l = 0;
int p = 0;
int ret = 0;
uint pe_total = 0;
- ulong rest = 0;
- ulong size = 0;
debug_enter ( "vg_setup_for_create -- CALLED\n");
@@ -76,67 +73,66 @@
vg->pvg_total = 0;
vg->proc = NULL;
memset ( vg->vg_uuid, 0, sizeof ( vg->vg_uuid));
- memcpy ( vg->vg_uuid, lvm_create_uuid ( UUID_LEN), sizeof ( vg->vg_uuid)-1);
+ memcpy ( vg->vg_uuid, lvm_create_uuid ( UUID_LEN), sizeof ( vg->vg_uuid));
+ memset ( vg->pv, 0, sizeof ( pv_t *) * vg->pv_max);
+ memset ( vg->lv, 0, sizeof ( lv_t *) * vg->lv_max);
/* Walk through physical volumes */
pe_total = 0;
for ( p = 0; pv[p] != NULL; p++) {
- if ( pv_get_size ( pv[p]->pv_name, NULL) /
- pe_size < LVM_PE_SIZE_PV_SIZE_REL) {
+ pv_t *pp = pv[p];
+ if ( ( pp->pv_size = pv_get_size ( pp->pv_name, NULL)) / pe_size <
+ LVM_PE_SIZE_PV_SIZE_REL) {
ret = -LVM_EVG_SETUP_FOR_CREATE_PV_SIZE;
goto vg_setup_for_create_end;
}
- vg->pv[p] = pv[p];
- vg->pv[p]->pv_on_disk.base = LVM_PV_DISK_BASE;
- vg->pv[p]->pv_on_disk.size = LVM_PV_DISK_SIZE;
- vg->pv[p]->vg_on_disk.base = LVM_VG_DISK_BASE;
- vg->pv[p]->vg_on_disk.size = LVM_VG_DISK_SIZE;
- vg->pv[p]->pv_uuidlist_on_disk.base = LVM_PV_UUIDLIST_DISK_BASE;
- vg->pv[p]->pv_uuidlist_on_disk.size = ( max_pv + 1) * NAME_LEN;
-
- size = vg->pv[p]->pv_uuidlist_on_disk.base +
- vg->pv[p]->pv_uuidlist_on_disk.size;
- vg->pv[p]->lv_on_disk.base = (size / SECTOR_SIZE) * SECTOR_SIZE;
- if (size % SECTOR_SIZE)
- vg->pv[p]->lv_on_disk.base += SECTOR_SIZE;
- vg->pv[p]->lv_on_disk.size = ( max_lv + 1) * sizeof ( lv_disk_t);
- LVM_PE_ON_DISK_BASE ( vg->pv[p]);
- strcpy ( vg->pv[p]->vg_name, vg->vg_name);
- vg->pv[p]->pv_number = p + 1;
- vg->pv[p]->pv_status = 0; /* bitfield */
- vg->pv[p]->pv_allocatable = PV_ALLOCATABLE; /* bitfield */
- vg->pv[p]->pe_size = vg->pe_size;
-
- rest = vg->pv[p]->pv_size - vg->pv[p]->pe_on_disk.base / SECTOR_SIZE;
- vg->pv[p]->pe_total = rest / vg->pe_size;
- rest -= ( vg->pv[p]->pe_total * vg->pe_size);
- while ( rest * SECTOR_SIZE / sizeof ( pe_disk_t) < vg->pv[p]->pe_total) {
- rest += vg->pe_size;
- vg->pv[p]->pe_total--;
+
+ vg->pv[p] = pp;
+ strcpy ( pp->vg_name, vg->vg_name);
+ pp->pv_number = p + 1;
+ pp->pv_status = 0; /* bitfield */
+ pp->pv_allocatable = PV_ALLOCATABLE; /* bitfield */
+ pp->pe_size = vg->pe_size;
+
+ pp->pv_on_disk.base = LVM_PV_DISK_BASE;
+ pp->pv_on_disk.size = LVM_PV_DISK_SIZE;
+ pp->vg_on_disk.base = LVM_VG_DISK_BASE;
+ pp->vg_on_disk.size = LVM_VG_DISK_SIZE;
+ pp->pv_uuidlist_on_disk.base = LVM_PV_UUIDLIST_DISK_BASE;
+ pp->pv_uuidlist_on_disk.size = ( max_pv + 1) * NAME_LEN;
+
+ pp->lv_on_disk.base = LVM_VGDA_ROUND ( pp->pv_uuidlist_on_disk.base +
+ pp->pv_uuidlist_on_disk.size);
+ pp->lv_on_disk.size = (max_lv + 1) * sizeof ( lv_disk_t);
+ pp->pe_on_disk.base = LVM_VGDA_ROUND ( pp->lv_on_disk.base +
+ pp->lv_on_disk.size);
+
+ /* This may be one too many PEs if we don't have enough space for VGDA */
+ pp->pe_total = (pp->pv_size & ~LVM_PE_ALIGN) / pp->pe_size;
+ pp->pe_on_disk.size = ((pp->pv_size & ~LVM_PE_ALIGN) -
+ pp->pe_total * vg->pe_size) * SECTOR_SIZE -
+ pp->pe_on_disk.base;
+ if (pp->pe_on_disk.size < pp->pe_total * sizeof ( pe_disk_t)) {
+ --pp->pe_total;
+ pp->pe_on_disk.size += vg->pe_size * SECTOR_SIZE;
}
- vg->pv[p]->pe_on_disk.size = ( vg->pv[p]->pv_size -
- vg->pv[p]->pe_total * vg->pe_size -
- vg->pv[p]->pe_on_disk.base / SECTOR_SIZE)
- * SECTOR_SIZE;
- if ( LVM_DISK_SIZE ( vg->pv[p]) % BLOCK_SIZE > 0)
- vg->pv[p]->pe_on_disk.size -= SECTOR_SIZE;
- if ( ( vg->pv[p]->pe = malloc ( vg->pv[p]->pe_total *
- sizeof ( pe_disk_t))) == NULL) {
+ if ( pp->pe_total > LVM_PE_T_MAX) {
+ ret = -LVM_EVG_SETUP_FOR_CREATE_PV_SIZE;
+ goto vg_setup_for_create_end;
+ }
+ if ( ( pp->pe = malloc ( pp->pe_total * sizeof ( pe_disk_t))) == NULL) {
fprintf ( stderr, "malloc error in %s [line %d]\n",
__FILE__, __LINE__);
ret = -LVM_EVG_SETUP_FOR_CREATE_MALLOC;
goto vg_setup_for_create_end;
}
- memset ( vg->pv[p]->pe, 0, vg->pv[p]->pe_total * sizeof ( pe_disk_t));
- vg->pv[p]->pe_allocated = 0;
- vg->pv[p]->pe_stale = 0;
- pe_total += vg->pv[p]->pe_total;
+ memset ( pp->pe, 0, pp->pe_total * sizeof ( pe_disk_t));
+ pp->pe_allocated = 0;
+ pp->pe_stale = 0;
+ pe_total += pp->pe_total;
}
vg->pv_cur = vg->pv_act = p;
vg->pe_total = pe_total;
-
- for ( ; p < vg->pv_max; p++) vg->pv[p] = NULL; /* Rest should be NULL */
- for ( l = 0; l < vg->lv_max; l++) vg->lv[l] = NULL; /* No LVs in NEW VG */
vg_setup_for_create_end:
diff -u -u -r1.2.2.1 vg_setup_for_extend.c
--- tools/lib/vg_setup_for_extend.c 2001/02/19 10:04:34 1.2.2.1
+++ tools/lib/vg_setup_for_extend.c 2001/04/10 20:44:54
@@ -44,7 +44,6 @@
int p1 = 0;
int ret = 0;
uint size = 0;
- ulong rest = 0;
debug_enter ( "vg_setup_for_extend -- CALLED\n");
@@ -66,7 +65,8 @@
}
for ( p = 0; all_pv[p] != NULL; p++) {
- if ( strcmp ( extend_pv[ep], all_pv[p]->pv_name) == 0) {
+ pv_t *pp = all_pv[p];
+ if ( strcmp ( extend_pv[ep], pp->pv_name) == 0) {
debug ( "%s -- checking for maximum physical "
"volume count of %s\n", cmd, vg->vg_name);
if ( np >= vg->pv_max) {
@@ -74,9 +74,9 @@
goto vg_setup_for_extend_end;
}
debug ( "%s -- getting size of physical "
- "volume %s\n", cmd, all_pv[p]->pv_name);
- if ( ( size = pv_get_size ( all_pv[p]->pv_name, NULL)) <= 0) {
- *error_pv_name = all_pv[p]->pv_name;
+ "volume %s\n", cmd, pp->pv_name);
+ if ( ( size = pv_get_size ( pp->pv_name, NULL)) <= 0) {
+ *error_pv_name = pp->pv_name;
ret = -LVM_EVG_SETUP_FOR_EXTEND_PV_GET_SIZE;
goto vg_setup_for_extend_end;
}
@@ -91,72 +92,61 @@
goto vg_setup_for_extend_end;
}
}
- debug ( "%s -- checking for new physical "
- "volume %s\n", cmd, all_pv[p]->pv_name);
- if ( pv_check_new ( all_pv[p]) == FALSE) {
- *error_pv_name = all_pv[p]->pv_name;
+ debug ( "%s -- checking for new physical volume %s\n",
+ cmd, pp->pv_name);
+ if ( pv_check_new ( pp) == FALSE) {
+ *error_pv_name = pp->pv_name;
ret = -LVM_EVG_SETUP_FOR_EXTEND_PV_CHECK_NEW;
goto vg_setup_for_extend_end;
}
- if ( pv_get_size ( all_pv[p]->pv_name, NULL) /
- vg->pv[0]->pe_size < LVM_PE_SIZE_PV_SIZE_REL) {
- *error_pv_name = all_pv[p]->pv_name;
+ if ( size / vg->pv[0]->pe_size < LVM_PE_SIZE_PV_SIZE_REL) {
+ *error_pv_name = pp->pv_name;
ret = -LVM_EVG_SETUP_FOR_EXTEND_PV_SIZE_REL;
goto vg_setup_for_extend_end;
}
/* setup PV and correct VG */
- all_pv[p]->vg_on_disk.base = vg->pv[0]->vg_on_disk.base;
- all_pv[p]->vg_on_disk.size = vg->pv[0]->vg_on_disk.size;
- all_pv[p]->pv_uuidlist_on_disk.base =
- vg->pv[0]->pv_uuidlist_on_disk.base;
- all_pv[p]->pv_uuidlist_on_disk.size =
- vg->pv[0]->pv_uuidlist_on_disk.size;
- all_pv[p]->lv_on_disk.base = vg->pv[0]->lv_on_disk.base;
- all_pv[p]->lv_on_disk.size = vg->pv[0]->lv_on_disk.size;
- all_pv[p]->pe_on_disk.base = vg->pv[0]->pe_on_disk.base;
- rest = all_pv[p]->pv_size - all_pv[p]->pe_on_disk.base /
- SECTOR_SIZE;
- all_pv[p]->pe_total = rest / vg->pe_size;
- rest -= ( all_pv[p]->pe_total * vg->pe_size);
- while ( rest * SECTOR_SIZE / sizeof ( pe_disk_t) <
- all_pv[p]->pe_total) {
- rest += vg->pe_size;
- all_pv[p]->pe_total--;
- }
- all_pv[p]->pe_on_disk.size =
- ( all_pv[p]->pv_size - all_pv[p]->pe_total * vg->pe_size -
- all_pv[p]->pe_on_disk.base / SECTOR_SIZE) * SECTOR_SIZE;
- if ( LVM_DISK_SIZE ( all_pv[p]) % BLOCK_SIZE > 0)
- all_pv[p]->pe_on_disk.size -= SECTOR_SIZE;
- strcpy ( all_pv[p]->vg_name, vg->vg_name);
- all_pv[p]->pv_dev = pv_create_kdev_t ( all_pv[p]->pv_name);
- all_pv[p]->pv_status = 0; /* bitfield */
- all_pv[p]->pv_allocatable = PV_ALLOCATABLE;
- all_pv[p]->pv_size = size;
- all_pv[p]->lv_cur = 0;
- all_pv[p]->pe_size = vg->pe_size;
- all_pv[p]->pe_total = ( all_pv[p]->pv_size -
- ( all_pv[p]->pe_on_disk.base +
- all_pv[p]->pe_on_disk.size) / SECTOR_SIZE ) /
- vg->pe_size;
- all_pv[p]->pe_allocated = 0;
- all_pv[p]->pe_stale = 0;
- if ( ( all_pv[p]->pe = malloc ( all_pv[p]->pe_total *
+ strcpy ( pp->vg_name, vg->vg_name);
+ pp->pv_number = np + 1;
+ pp->pv_dev = pv_create_kdev_t ( pp->pv_name);
+ pp->pv_status = 0; /* bitfield */
+ pp->pv_allocatable = PV_ALLOCATABLE;
+ pp->pv_size = size;
+ pp->lv_cur = 0;
+ pp->pe_size = vg->pe_size;
+
+ pp->vg_on_disk.base = vg->pv[0]->vg_on_disk.base;
+ pp->vg_on_disk.size = vg->pv[0]->vg_on_disk.size;
+ pp->pv_uuidlist_on_disk.base = vg->pv[0]->pv_uuidlist_on_disk.base;
+ pp->pv_uuidlist_on_disk.size = vg->pv[0]->pv_uuidlist_on_disk.size;
+ pp->lv_on_disk.base = vg->pv[0]->lv_on_disk.base;
+ pp->lv_on_disk.size = vg->pv[0]->lv_on_disk.size;
+ pp->pe_on_disk.base = vg->pv[0]->pe_on_disk.base;
+ /* This may be one too many PEs if not enough space for VGDA */
+ pp->pe_total = (pp->pv_size & ~LVM_PE_ALIGN) / pp->pe_size;
+ pp->pe_on_disk.size = ((pp->pv_size & ~LVM_PE_ALIGN) -
+ pp->pe_total * vg->pe_size) * SECTOR_SIZE -
+ pp->pe_on_disk.base;
+ if (pp->pe_on_disk.size < pp->pe_total * sizeof ( pe_disk_t)) {
+ --pp->pe_total;
+ pp->pe_on_disk.size += vg->pe_size * SECTOR_SIZE;
+ }
+ if ( ( pp->pe = malloc ( pp->pe_total *
sizeof ( pe_disk_t))) == NULL) {
fprintf ( stderr, "%s -- malloc error in %s at line %d\n\n",
cmd, __FILE__, __LINE__);
ret = -LVM_EVG_SETUP_FOR_EXTEND_MALLOC;
goto vg_setup_for_extend_end;
}
- memset ( all_pv[p]->pe,
- 0,
- all_pv[p]->pe_total * sizeof ( pe_disk_t));
- vg->pv[np] = all_pv[p];
+ memset ( pp->pe, 0, pp->pe_total * sizeof ( pe_disk_t));
+ pp->pe_allocated = 0;
+ pp->pe_stale = 0;
+
+ vg->pv[np] = pp;
vg->pv_act++;
vg->pv_cur++;
- vg->pe_total += vg->pv[np]->pe_total;
+ vg->pe_total += pp->pe_total;
np++;
new_pv++;
--
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-04-10 21:37 ` Andreas Dilger
@ 2001-04-11 11:23 ` Urs Thuermann
2001-04-11 19:49 ` Andreas Dilger
0 siblings, 1 reply; 22+ messages in thread
From: Urs Thuermann @ 2001-04-11 11:23 UTC (permalink / raw)
To: linux-lvm
Andreas Dilger <adilger@turbolinux.com> writes:
> No, PEs are allocated from the start, but are only aligned with the
> end of the disk/partition (to give maximum room to LVM for
> metadata). If you have contiguous LE allocation (which you appear
> to have), the read order for LVM should basically be exactly the
> same as for a raw disk. Effectively all that LVM is doing is adding
> a small offset to the disk blocks to leave room for the VGDA,
> nothing more.
Yes, I found that out after my posting by writing a 32 Mb ( = 8 PEs)
pattern to the beginning of /dev/vg0/test and searching for it in
/dev/sda5. In my case the offset is of the first PE is 4352 sectors,
since /dev/sda5 has 627 * 4 MB + 4352 * 512 bytes. This small offset
shouldn't have an impact on the performance when reading 2.45 GB.
> Do you notice the sound of frantic head seeking when you are running
> with small block sizes?
No, sounds very similar and only very moderate head seeking (probably
from syslogd).
> So basically a few memory accesses and a few math operations.
> Nothing _should_ affect disk performance at all. With < 2k I/O you
> are forcing read-modify-write on pages, but this shouldn't be an
> issue, because you are also doing the same for the raw disk.
What has that all to do with pages? Isn't it the hardsect_size that
is relevant here? And why is < 2k the relevant limit? If the page
size was relevant this is 4k. If the hardsect_size is relevant, this
is 1k in 2.4.3 (after initialization, but can grow when mounting a FS
with larger block size, due to the bug I reported on this list) and if
I read the src in 0.9.1b7 correctly Andrea's patch makes the
hardsect_size equal to the maximum sector size of all PVs, i.e. 512
bytes with most disks.
> One thing that now comes to mind is if the LVM I/O is not aligned,
> because LVM PEs are aligned with the end of the PV and not the
> start, this _may_ cause performance problems because of misaligned
> read-modify-write causing extra seeking (purely speculation).
I don't understand, what you're saying here (and above). I was
*reading* with dd if=/dev/vg0/test of=/dev/null. So why is there an
RMW involved?
What does happen, when I do a write(2) of a 512-byte block to
/dev/vg0/test, when the hardsect_size of the PV /dev/sda5 is 512 bytes
and the hardsect_size of the LV /dev/vg0/test is 1k? Does the kernel
read 1k from LVM, modify 512 bytes and (later, when syncing) write 1k
back to LVM? If so, the fact that hardsect_size of the LV is larger
than the physical sector size could explain the performance impact.
The kernel would have to modify only parts of a LV block, so (I
assume) it would RMW. With
hardsect_size == sector size == size of the write(2) call
the kernel could probably just write to LVM without prior reading.
But this would still not explain the performance degradation in my
tests where I was *reading* from the LV.
Could you explain the interaction of the kernel, LVM and the physical
device a little bit or give a pointer to some text?
> Try the following patch, which will align the PE structs to 64kB
> boundaries from the start of the disk, as well as making the other
> large VGDA structs start on 4k boundaries.
Does it align to the start of the disk (as you write) or to the start
of the PV? And is this a patch that will go into further releases
anyway or is it just to find my reported performance problem? Since
alignment is changed, I think I will have to recreate the PV, VG and
LV with pvcreate, vgcreate and lvcreate, right? The patched tools and
kernel would not work with my existing LV, would they?
I don't expect this change in alignment to cause a change in
performance. For testing, I have written my own very simple version
of dd which can seek (not only skip) on the input file (I called it di
since it understands only if= but not of=, it always writes to
stdout). With this I have read from /dev/sda5 exactly the same
sectors that /dev/vg0/test allocates (the first PE in vg0 starts at
sector 4352 and /dev/vg0/test allocates PEs from vg0 contigously
starting at PE no 0, see my last mail):
# time ~urs/src/di bs=512 count=131072 if=/dev/sda5 >/dev/null
131072+0
real 0m9.345s
user 0m0.180s
sys 0m1.250s
# time ~urs/src/di bs=512 count=131072 if=/dev/sda5 seek=4352 >/dev/null
131072+0
real 0m8.955s
user 0m0.130s
sys 0m1.570s
# time ~urs/src/di bs=512 count=131072 if=/dev/vg0/test >/dev/null
131072+0
real 0m54.611s
user 0m0.110s
sys 0m2.480s
Anyway, I will try your patch, probably today. I have not applied
patches from the lvm-0.9... to the kernel before. AFAICS, I have to
apply your patch to the lvm-0.9.1beta7 package, then ./configure; cd
PATCHES; make and then apply lvm-0.9.1_beta7-2.4.3.patch to my kernel
src, right?
Also, does lvm-0.9.1_beta7-2.4.3.patch affect files other than
lvm-mod.o, i.e. do I have to restart my kernel or only reload the LVM
module? I have already made the lvm-0.9.1_beta7-2.4.3.patch file and
see lvm.h is changed, but haven't already checked where it is
included.
urs
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-04-11 11:23 ` Urs Thuermann
@ 2001-04-11 19:49 ` Andreas Dilger
2001-04-11 22:00 ` Terje Kvernes
2001-05-15 11:46 ` Urs Thuermann
0 siblings, 2 replies; 22+ messages in thread
From: Andreas Dilger @ 2001-04-11 19:49 UTC (permalink / raw)
To: linux-lvm
Urs, you write:
> > One thing that now comes to mind is if the LVM I/O is not aligned,
> > because LVM PEs are aligned with the end of the PV and not the
> > start, this _may_ cause performance problems because of misaligned
> > read-modify-write causing extra seeking (purely speculation).
>
> I don't understand, what you're saying here (and above). I was
> *reading* with dd if=/dev/vg0/test of=/dev/null. So why is there an
> RMW involved?
Sorry, too sleepy I guess. Obviously don't need read-modify-write
when only reading pages.
> Could you explain the interaction of the kernel, LVM and the physical
> device a little bit or give a pointer to some text?
Ok, the chain of events is as follows (hopefully correct):
Non-LVM read LVM read
------------ --------
sys_read sys_read
block_read block_read
ll_rw_block ll_rw_block
submit_bh submit_bh
generic_make_request generic_make_request
lvm_make_request_fn
lvm_map
__make_request __make_request
The rest should be the same because you have the same IDE driver below.
> > Try the following patch, which will align the PE structs to 64kB
> > boundaries from the start of the disk, as well as making the other
> > large VGDA structs start on 4k boundaries.
>
> Does it align to the start of the disk (as you write) or to the start
> of the PV?
You are correct, it aligns to the start of the PV, and not the disk.
> And is this a patch that will go into further releases anyway or is it
> just to find my reported performance problem? Since alignment is changed,
> I think I will have to recreate the PV, VG and LV with pvcreate, vgcreate
> and lvcreate, right? The patched tools and kernel would not work with
> my existing LV, would they?
I'm hoping this patch will make it into the stock LVM because it aligns
all of the large VGDA structs to PAGE_SIZE (at least 4k pages), and the
PE alignment matches MD RAID device alignment.
Yes, you will need to re-create the PV and VG and LV for this to take
effect. However, the kernel currently knows nothing at all about on-disk
layout of VGDA data. Also, the tools all use the VGDA locations stored
in the PV header (they are written at pvcreate and vgcreate time), so even
if you create your VG/LV with this patch and it doesn't make it into the
stock kernel, it is 100% compatible with unpatched tools.
> I don't expect this change in alignment to cause a change in performance.
Neither do I. Just something that I was thinking about when I had writing
on my mind, not reading.
> For testing, I have written my own very simple version of dd which can
> seek (not only skip) on the input file (I called it di since it understands
> only if= but not of=, it always writes to stdout). With this I have read
> from /dev/sda5 exactly the same sectors that /dev/vg0/test allocates
> (the first PE in vg0 starts at sector 4352 and /dev/vg0/test allocates
> PEs from vg0 contigously starting at PE no 0, see my last mail):
>
> # time ~urs/src/di bs=512 count=131072 if=/dev/sda5 >/dev/null
> 131072+0
>
> real 0m9.345s
> user 0m0.180s
> sys 0m1.250s
> # time ~urs/src/di bs=512 count=131072 if=/dev/sda5 seek=4352 >/dev/null
> 131072+0
>
> real 0m8.955s
> user 0m0.130s
> sys 0m1.570s
> # time ~urs/src/di bs=512 count=131072 if=/dev/vg0/test >/dev/null
> 131072+0
>
> real 0m54.611s
> user 0m0.110s
> sys 0m2.480s
Very strange. I just tested the same on my system, and LVM ends up
_faster_ than raw disk access in each of the 3 tests I did:
dd if=<dev> of=/dev/null bs=512 count=131072
LVM /dev/lvtest/vgtest: 11.099s, 11.393s, 10.785s
IDE /dev/hda9: 12.073s, 11.822s, 11.843s
This may have something to do with how my LV is set up. The first 5
PEs are actually closer to the end of the PV, and then the remaining
PEs are contiguous from the start of the PV.
> Anyway, I will try your patch, probably today. I have not applied
> patches from the lvm-0.9... to the kernel before. AFAICS, I have to
> apply your patch to the lvm-0.9.1beta7 package, then ./configure; cd
> PATCHES; make and then apply lvm-0.9.1_beta7-2.4.3.patch to my kernel
> src, right?
Yes, but you really don't need to change the kernel code to use my
patch. The VGDA alignment is purely a user-space issue. The kernel
never does I/O to the VGDA - it is all read in user-space and passed
to the kernel via ioctl.
> Also, does lvm-0.9.1_beta7-2.4.3.patch affect files other than
> lvm-mod.o, i.e. do I have to restart my kernel or only reload the LVM
> module?
Actually, I think you should be OK with only rebuilding the LVM module.
However, you should be able to see from the patch itself if it affects
anything outside lvm*. AFAIK, noting includes lvm.h in the kernel that
would be affected.
Cheers, Andreas
--
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-04-11 19:49 ` Andreas Dilger
@ 2001-04-11 22:00 ` Terje Kvernes
2001-05-15 11:46 ` Urs Thuermann
1 sibling, 0 replies; 22+ messages in thread
From: Terje Kvernes @ 2001-04-11 22:00 UTC (permalink / raw)
To: linux-lvm
Andreas Dilger <adilger@turbolinux.com> writes:
> Ok, the chain of events is as follows (hopefully correct):
>
> Non-LVM read LVM read
> ------------ --------
> sys_read sys_read
> block_read block_read
> ll_rw_block ll_rw_block
> submit_bh submit_bh
> generic_make_request generic_make_request
> lvm_make_request_fn
> lvm_map
> __make_request __make_request
>
> The rest should be the same because you have the same IDE driver
> below.
I'm going to stick my head in here and say thanks for that info. I
really need to _read_ the code. *sigh*
> Very strange. I just tested the same on my system, and LVM ends up
> _faster_ than raw disk access in each of the 3 tests I did:
okay, I only have the option of testing this on an old IDE drive,
and the times where within 7% of eachother (LVM being the slower of
the two on all runs). it's a bigger difference then I'd like to see,
but it varies with runs, and the box is low on memory and being used
at the same time. the only thing they're really saying is that LVM
isn't a bottleneck.
not that it'll help much in Urs' case though. :/
--
Terje
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-04-11 19:49 ` Andreas Dilger
2001-04-11 22:00 ` Terje Kvernes
@ 2001-05-15 11:46 ` Urs Thuermann
2001-05-15 12:13 ` Patrick Caulfield
2001-05-15 17:08 ` [linux-lvm] LVM and *bad* performance (no striping) Andreas Dilger
1 sibling, 2 replies; 22+ messages in thread
From: Urs Thuermann @ 2001-05-15 11:46 UTC (permalink / raw)
To: linux-lvm
Andreas Dilger <adilger@turbolinux.com> writes:
> I'm hoping this patch will make it into the stock LVM because it aligns
> all of the large VGDA structs to PAGE_SIZE (at least 4k pages), and the
> PE alignment matches MD RAID device alignment.
Sorry, I hadn't time to test your patch earlier. I wanted to try it
yesterday but lvm-0.9.1-beta7 with your patch applied doesn't compile.
There are undefined symbols
$ make
...
gcc -L../tools/lib -llvm-10 -o vgchange vgchange.o
vgchange.o: In function `main':
/home/urs/tmp/LVM/0.9.1_beta7-dilger/tools/vgchange.c:635: undefined reference to `LVM_PE_ON_DISK_BASE'
/home/urs/tmp/LVM/0.9.1_beta7-dilger/tools/vgchange.c:638: undefined reference to `LVM_DISK_SIZE'
When I have done my performance tests I had a stock linux-2.4.3 and
user space tools from lvm-0.9 running. I now repeated the tests with
lvm-0.9.1-beta7, both user space tools and kernel (recreated all PVs,
VGs and LVs). However, the results with these versions are the same.
I also did some tests with another hard disk drive. My previous tests
were with a IBM DCAS-34330W SCSI-U2W. I now did some tests with a NEC
DSE2100S SCSI2 drive. Again, LVM is much slower with the standard
512 byte block size:
# pvcreate /dev/sdc4
pvcreate -- physical volume "/dev/sdc4" successfully created
# vgcreate vg1 /dev/sdc4
vgcreate -- INFO: using default physical extent size 4 MB
vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte
vgcreate -- doing automatic backup of volume group "vg1"
vgcreate -- volume group "vg1" successfully created and activated
# lvcreate -n test vg1 -l31
lvcreate -- doing automatic backup of "vg1"
lvcreate -- logical volume "/dev/vg1/test" successfully created
# time dd if=/dev/sdc4 of=/dev/null
257040+0 records in
257040+0 records out
real 0m37.205s
user 0m0.840s
sys 0m3.780s
# time dd if=/dev/vg1/test of=/dev/null
253952+0 records in
253952+0 records out
real 2m52.368s
user 0m1.330s
sys 0m12.390s
# time dd if=/dev/vg1/test of=/dev/null bs=8k
15872+0 records in
15872+0 records out
real 0m43.501s
user 0m0.050s
sys 0m4.600s
# time dd if=/dev/vg1/test of=/dev/null bs=16k
7936+0 records in
7936+0 records out
real 0m40.806s
user 0m0.070s
sys 0m4.290s
# time dd if=/dev/vg1/test of=/dev/null bs=32k
3968+0 records in
3968+0 records out
real 0m38.777s
user 0m0.030s
sys 0m4.290s
What I find surprising is that the system time increases to 12.390s
with LVM compared to 3.780s on /dev/sdc4. That is approx. 34ms of
overhead per 512-block which I find a lot for the simple mapping of
block numbers. But this still does not explain the much longer real
elapsed time.
Is there another patch (maybe to produce some debugging output) I
could try to find the reason for this performnace hit?
And am I really the only one who sees this? I would really be
interested to hear from some more people about performance for their
LVM.
urs
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-05-15 11:46 ` Urs Thuermann
@ 2001-05-15 12:13 ` Patrick Caulfield
2001-05-26 11:17 ` Urs Thuermann
2001-05-15 17:08 ` [linux-lvm] LVM and *bad* performance (no striping) Andreas Dilger
1 sibling, 1 reply; 22+ messages in thread
From: Patrick Caulfield @ 2001-05-15 12:13 UTC (permalink / raw)
To: linux-lvm
On Tue, May 15, 2001 at 01:46:15PM +0200, Urs Thuermann wrote:
>
> And am I really the only one who sees this? I would really be
> interested to hear from some more people about performance for their
> LVM.
I think one problem is that your "benchmarks" are not very realistic. In
real world use the overhead of LVM is very small. I would respectfully suggest
that you try something like bonnie++ before getting too worried about the
performance of LVM on your machine - unless you spend most of your day doing
"dd" to your disks :-)
Here are some bonnie++ benchmarks of a disk partition and and LV (same disk)
using reiserfs* on my machine:
Disk Partition:
Version 1.00g ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
tyke 496M 10462 98 20675 53 7699 20 11707 96 17808 16 257.0 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 8426 100 +++++ +++ 12622 100 8031 100 +++++ +++ 9635 100
Logical Volume:
Version 1.00g ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
tyke 496M 10577 100 20631 52 7739 20 11676 96 17622 17 260.7 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 8521 100 +++++ +++ 12661 99 8007 100 +++++ +++ 9555 99
patrick
* ext2 is comparable, just the numbers are different.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-05-15 11:46 ` Urs Thuermann
2001-05-15 12:13 ` Patrick Caulfield
@ 2001-05-15 17:08 ` Andreas Dilger
2001-05-26 11:23 ` Urs Thuermann
1 sibling, 1 reply; 22+ messages in thread
From: Andreas Dilger @ 2001-05-15 17:08 UTC (permalink / raw)
To: linux-lvm
Urs writes:
> Sorry, I hadn't time to test your patch earlier. I wanted to try it
> yesterday but lvm-0.9.1-beta7 with your patch applied doesn't compile.
Sorry, I must have missed some parts of the patch for the user-space
tools. Even so, I don't think the patch will help.
> I also did some tests with another hard disk drive. My previous tests
> were with a IBM DCAS-34330W SCSI-U2W. I now did some tests with a NEC
> DSE2100S SCSI2 drive. Again, LVM is much slower with the standard
> 512 byte block size:
One other person had a similar problem. It was because they were
using PIO mode instead of DMA mode on their (IDE) drive. However,
I see you are using a SCSI disk, so I don't know what to suggest...
What SCSI controller are you using?
> And am I really the only one who sees this? I would really be
> interested to hear from some more people about performance for their
> LVM.
I have never seen more than a few percent difference between LVM and
non-LVM performance.
Cheers, Andreas
--
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-05-15 12:13 ` Patrick Caulfield
@ 2001-05-26 11:17 ` Urs Thuermann
2001-05-27 7:47 ` Andreas Dilger
0 siblings, 1 reply; 22+ messages in thread
From: Urs Thuermann @ 2001-05-26 11:17 UTC (permalink / raw)
To: linux-lvm
Patrick Caulfield <caulfield@sistina.com> writes:
> I think one problem is that your "benchmarks" are not very realistic. In
> real world use the overhead of LVM is very small. I would respectfully suggest
> that you try something like bonnie++ before getting too worried about the
> performance of LVM on your machine - unless you spend most of your day doing
> "dd" to your disks :-)
The reason I tested dd on the partition and on the LV is that I
thought when the performance of the block device is bad (like it is on
my LV) this will have an impact on a mounted file system on that LV.
If also thought, the performance of dd if=/dev/vg0/test will be
approximately equal to dd if=/mnt/foo od md5sum foo where foo is large
file on the LV.
I've done some tests with a file system on /dev/sda5 and the on
/dev/vg0/test (which is a LV on /dev/sda5) and I am really surprised:
First, I repeated my previous tests:
$dev
/dev/sda5 /dev/vg0/test
---------------------------------------------+-------------+----------------
dd if=$dev of=/dev/null bs=512 count=1048576 | 71.2s | 122.7s
dd if=$dev of=/dev/null bs=64k count=8192 | 71.5s | 71.7s
Then I made a ext2 filesystem with 1K blocksize on /dev/sda5, mounted
it to /mnt, copied a 512MB random file to it, md5sum'med that fie and
dd'ed that file to /dev/null. Then I recreated /dev/vg0/test and did
the same tests and that block device:
---------------------------------------------+-------------+----------------
cp -a /usr/local/foo /mnt | 96.6s | 123.4s
md5sum /mnt/foo | 89.8s | 92.4s
dd if=/mnt/bar of=/dev/null bs=512 | 71.8s | 72.1s
dd if=/mnt/bar of=/dev/null bs=64k | 71.7s | 72.5s
When I use a 4K blocksize on the ext2 filesystem the times for
/dev/sda5 and /dev/vg0/test are even closer.
Result: Although the block device /dev/vg0/test is significantly
slower than /dev/sda5 (where /dev/vg0/test is located on /dev/sda5)
the *read* performance of the file system on top the block device is
roughly the same. This is probably not some effect of buffer caching
since I did a blockdev --flushbufs $dev before each test, and
/usr/local/bar is 512MB in size which is twice as large as the main
memory size on that machine.
However, writing to an LV is still much slower than to a normal
partition.
urs
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-05-15 17:08 ` [linux-lvm] LVM and *bad* performance (no striping) Andreas Dilger
@ 2001-05-26 11:23 ` Urs Thuermann
0 siblings, 0 replies; 22+ messages in thread
From: Urs Thuermann @ 2001-05-26 11:23 UTC (permalink / raw)
To: linux-lvm
Andreas Dilger <adilger@turbolinux.com> writes:
> What SCSI controller are you using?
It is an Adaptec AIC 7880 which is onboard on my
ASUS P2L97-S mainboard (see some previous posting where I did describe
my hardware in more detail).
Do you have some further suggestion what I could try to find the
reason for this performance hit?
urs
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-05-26 11:17 ` Urs Thuermann
@ 2001-05-27 7:47 ` Andreas Dilger
2001-05-27 9:47 ` Urs Thuermann
2001-05-27 17:01 ` [linux-lvm] LVM and *bad* performance (counterexample) idsfa
0 siblings, 2 replies; 22+ messages in thread
From: Andreas Dilger @ 2001-05-27 7:47 UTC (permalink / raw)
To: linux-lvm
Urs writes:
> First, I repeated my previous tests:
>
> $dev
>
> /dev/sda5 /dev/vg0/test
> ---------------------------------------------+-------------+----------------
> dd if=$dev of=/dev/null bs=512 count=1048576 | 71.2s | 122.7s
> dd if=$dev of=/dev/null bs=64k count=8192 | 71.5s | 71.7s
OK, I have now reproduced this problem on my SMP test system. This is on a
2xP5-200 + 128MB RAM, Advansys ABP940U SCSI, and (old) IBM 0662-S12 and
IBM 0663-L12 1GB SCSI drives. I do NOT have this problem on my laptop
AFAICS (P2-500 + 128MB RAM, Intel PIIX4 IDE, IBM-DARA-218000).
Doing similar tests to the above (dd if=$dev of=/dev/null, bs=512 count=100k)
I get the following results (50MB I/O, 100MB LV, Linux 2.2.18 + LVM 0.9):
/dev/sda = real 12.290s user 0.110s sys 2.890s
/dev/sda = real 12.269s user 0.100s sys 2.900s
/dev/sda1 = real 12.317s user 0.140s sys 2.980s
/dev/sda1 = real 12.251s user 0.130s sys 2.900s
/dev/vgtest/lvtest = real 66.909s user 0.440s sys 10.420 *** 5x as slow
/dev/vgtest/lvtest = real 70.135s user 0.520s sys 11.390 *** 5x as slow
/dev/sdb = real 11.614s user 0.060s sys 3.090s
/dev/sdb = real 11.597s user 0.120s sys 2.910s
/dev/sdb1 = real 11.857s user 0.150s sys 4.020s
/dev/sdb1 = real 11.749s user 0.110s sys 2.800s
/dev/vgtest3/lvtest = real 63.552s user 0.510s sys 10.290s *** 6x as slow
/dev/vgtest3/lvtest = real 62.354s user 0.610s sys 10.020s *** 6x as slow
/dev/hda6 = real 10.848s user 0.180s sys 4.380s
/dev/hda6 = real 10.803s user 0.210s sys 4.300s
/dev/vgtest2/lvtest = real 15.258s user 0.480s sys 9.500s *** 50% slower
/dev/vgtest2/lvtest = real 15.827s user 0.360s sys 9.830s *** 50% slower
I'm not sure if it is an SMP or SCSI issue, or what. Urs, do you also
have an SMP system? Maybe there are locking slowdowns under SMP and
not under UP?
The problem is definitely more noticable on the SCSI disks, but at least
on the IDE disk the increased wall-clock time is wholly accounted for
by increased system time. On the SCSI disks, it would guess that "dd"
is waiting for I/O completions under LVM, and not on the raw disk???
Cheers, Andreas
--
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping)
2001-05-27 7:47 ` Andreas Dilger
@ 2001-05-27 9:47 ` Urs Thuermann
2001-05-27 17:01 ` [linux-lvm] LVM and *bad* performance (counterexample) idsfa
1 sibling, 0 replies; 22+ messages in thread
From: Urs Thuermann @ 2001-05-27 9:47 UTC (permalink / raw)
To: linux-lvm
Andreas Dilger <adilger@turbolinux.com> writes:
> /dev/vgtest/lvtest = real 66.909s user 0.440s sys 10.420 *** 5x as slow
> /dev/vgtest/lvtest = real 70.135s user 0.520s sys 11.390 *** 5x as slow
>
> /dev/vgtest3/lvtest = real 63.552s user 0.510s sys 10.290s *** 6x as slow
> /dev/vgtest3/lvtest = real 62.354s user 0.610s sys 10.020s *** 6x as slow
This is similar to the worst case numbers I have measured.
> I'm not sure if it is an SMP or SCSI issue, or what. Urs, do you also
> have an SMP system? Maybe there are locking slowdowns under SMP and
> not under UP?
No. It's a single Pentium II, 333MHz, kernel 2.4.3 with lvm-0.9.1_beta7
compiled without SMP support:
isnogud:urs$ grep SMP /usr/src/linux-2.4.3-ut/.config
# CONFIG_SMP is not set
The SCSI controller is an Adaptec AIC 7880 on an ASUS P2L97-S
mainboard (onboard). The SCSI disks are
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: DCAS-34330W Rev: S65A
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: IBM Model: DCAS-34330W Rev: S65A
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 02 Lun: 00
Vendor: NEC Model: DSE2100S Rev: 0306
Type: Direct-Access ANSI SCSI revision: 02
i.e. /dev/sda and /dev/sdb are UW-SCSI and /dev/sdc is a SCSI-2 disks.
> The problem is definitely more noticable on the SCSI disks, but at least
> on the IDE disk the increased wall-clock time is wholly accounted for
> by increased system time. On the SCSI disks, it would guess that "dd"
> is waiting for I/O completions under LVM, and not on the raw disk???
How could I check? And do you have an idea, why a file system on top
of such a slow LV is, if at all, only somewhat slower?
urs
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (counterexample)
2001-05-27 7:47 ` Andreas Dilger
2001-05-27 9:47 ` Urs Thuermann
@ 2001-05-27 17:01 ` idsfa
1 sibling, 0 replies; 22+ messages in thread
From: idsfa @ 2001-05-27 17:01 UTC (permalink / raw)
To: linux-lvm
On Sun, May 27, 2001 at 01:47:18AM -0600, Andreas Dilger wrote:
> I'm not sure if it is an SMP or SCSI issue, or what. Urs, do you also
> have an SMP system? Maybe there are locking slowdowns under SMP and
> not under UP?
I haven't got a UP kernel to test right now to confirm, but my SMP
purely IDE system (specs follow) shows rough parity at 512 byte
reads and 2x FASTER on the LV at 4096 byte blocks. When I switch
to a striped LV I see parity at both sizes, with the LV being
slightly faster. For giggles, I also checked out a RAID-1 (mirror)
set as well, which smoked everything, as it should.
Device Block Real User Sys
/dev/md0 512 2.080s 0.110s 0.660s
/dev/hda1 512 5.079s 0.140s 2.040s
/dev/mitethe/nostripe 512 4.795s 0.300s 2.380s
/dev/mitethe/stripe 512 4.350s 0.090s 2.530s
/dev/md0 4096 16.482s 0.060s 6.090s
/dev/mitethe/nostripe 4096 26.278s 0.010s 7.270s
/dev/mitethe/stripe 4096 32.803s 0.170s 12.020s
/dev/hda1 4096 40.947s 0.180s 13.590s
Now comes the part where I admit we're comparing apples to oranges:
Linux 2.4.3
LVM 0.9.1_beta7
2x Pentium III 600MHz
Tyan Tiger 133 Mobo (running at 133 FSB)
Maxtor 53073U6 30GB ATA/66 hda
WDC WD300AB-00BVA0 30GB ATA/66 hdc
/dev/hda:
multcount = 16 (on)
I/O support = 1 (32-bit)
unmaskirq = 1 (on)
using_dma = 1 (on)
keepsettings = 0 (off)
nowerr = 0 (off)
readonly = 0 (off)
readahead = 8 (on)
geometry = 59554/16/63, sectors = 60030432, start = 0
/dev/hdc:
multcount = 16 (on)
I/O support = 1 (32-bit)
unmaskirq = 1 (on)
using_dma = 1 (on)
keepsettings = 0 (off)
nowerr = 0 (off)
readonly = 0 (off)
readahead = 8 (on)
geometry = 58168/16/63, sectors = 58633344, start = 0
--
$ fortune -m Kellen
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2001-05-27 17:01 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2001-02-26 15:44 [linux-lvm] LVM and *bad* performance (no striping) Urs Thuermann
2001-02-26 20:39 ` Joe Thornber
2001-02-26 21:03 ` Jorg de Jong
2001-02-26 21:38 ` Rik van Riel
2001-02-27 8:35 ` Patrick Caulfield
2001-02-27 12:21 ` Urs Thuermann
2001-02-28 11:05 ` Heinz J. Mauelshagen
2001-04-10 19:30 ` Urs Thuermann
2001-04-10 21:37 ` Andreas Dilger
2001-04-11 11:23 ` Urs Thuermann
2001-04-11 19:49 ` Andreas Dilger
2001-04-11 22:00 ` Terje Kvernes
2001-05-15 11:46 ` Urs Thuermann
2001-05-15 12:13 ` Patrick Caulfield
2001-05-26 11:17 ` Urs Thuermann
2001-05-27 7:47 ` Andreas Dilger
2001-05-27 9:47 ` Urs Thuermann
2001-05-27 17:01 ` [linux-lvm] LVM and *bad* performance (counterexample) idsfa
2001-05-15 17:08 ` [linux-lvm] LVM and *bad* performance (no striping) Andreas Dilger
2001-05-26 11:23 ` Urs Thuermann
2001-02-26 21:41 ` Rik van Riel
2001-02-27 10:33 ` Urs Thuermann
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.