* [linux-lvm] LVM and *bad* performance (no striping)
@ 2001-02-26 15:44 Urs Thuermann
2001-02-26 20:39 ` Joe Thornber
2001-02-26 21:41 ` Rik van Riel
0 siblings, 2 replies; 22+ messages in thread
From: Urs Thuermann @ 2001-02-26 15:44 UTC (permalink / raw)
To: linux-lvm
I tried to use LVM today using Linux 2.4.2 and it mostly works like I
expected. But the performance is unacceptably bad:
isnogud:/root# fdisk -l /dev/sda|grep sda5
/dev/sda5 80 399 2570368+ 8e Linux LVM
isnogud:/root# pvcreate /dev/sda5
pvcreate -- physical volume "/dev/sda5" successfully created
isnogud:/root# vgcreate vg0 /dev/sda5
vgcreate -- INFO: using default physical extent size 4 MB
vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte
vgcreate -- doing automatic backup of volume group "vg0"
vgcreate -- volume group "vg0" successfully created and activated
isnogud:/root# lvcreate -n test /dev/vg0 -L 300M
lvcreate -- doing automatic backup of "vg0"
lvcreate -- logical volume "/dev/vg0/test" successfully created
isnogud:/root# time dd if=/dev/sda5 of=/dev/null count=100000
100000+0 records in
100000+0 records out
real 0m6.720s
user 0m0.300s
sys 0m0.970s
isnogud:/root# time dd if=/dev/vg0/test of=/dev/null count=100000
100000+0 records in
100000+0 records out
real 0m41.851s
user 0m0.350s
sys 0m2.250s
When making an ext2 filesystem on /dev/sda5 or /dev/vg0/test
respectivley, the performance of cp'ing, grep'ing all files and rm'ing
all files, seems to be comparable. But why is the performance of
reading the block device with dd so bad?
urs
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-02-26 15:44 [linux-lvm] LVM and *bad* performance (no striping) Urs Thuermann @ 2001-02-26 20:39 ` Joe Thornber 2001-02-26 21:03 ` Jorg de Jong ` (2 more replies) 2001-02-26 21:41 ` Rik van Riel 1 sibling, 3 replies; 22+ messages in thread From: Joe Thornber @ 2001-02-26 20:39 UTC (permalink / raw) To: linux-lvm On Mon, Feb 26, 2001 at 04:44:34PM +0100, Urs Thuermann wrote: > I tried to use LVM today using Linux 2.4.2 and it mostly works like I > expected. But the performance is unacceptably bad: Every now and then someone posts to the list claiming LVM performance is really bad. We go off and try it and find the overhead is typically less than 1%. Normally people are trying to stripe to two partitions on the same device, but you aren't. Is any one else on the list seeing a similar performance hit ? - Joe > > isnogud:/root# fdisk -l /dev/sda|grep sda5 > /dev/sda5 80 399 2570368+ 8e Linux LVM > isnogud:/root# pvcreate /dev/sda5 > pvcreate -- physical volume "/dev/sda5" successfully created > > isnogud:/root# vgcreate vg0 /dev/sda5 > vgcreate -- INFO: using default physical extent size 4 MB > vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte > vgcreate -- doing automatic backup of volume group "vg0" > vgcreate -- volume group "vg0" successfully created and activated > > isnogud:/root# lvcreate -n test /dev/vg0 -L 300M > lvcreate -- doing automatic backup of "vg0" > lvcreate -- logical volume "/dev/vg0/test" successfully created > > isnogud:/root# time dd if=/dev/sda5 of=/dev/null count=100000 > 100000+0 records in > 100000+0 records out > > real 0m6.720s > user 0m0.300s > sys 0m0.970s > isnogud:/root# time dd if=/dev/vg0/test of=/dev/null count=100000 > 100000+0 records in > 100000+0 records out > > real 0m41.851s > user 0m0.350s > sys 0m2.250s > > When making an ext2 filesystem on /dev/sda5 or /dev/vg0/test > respectivley, the performance of cp'ing, grep'ing all files and rm'ing > all files, seems to be comparable. But why is the performance of > reading the block device with dd so bad? > > > urs > _______________________________________________ > linux-lvm mailing list > linux-lvm@sistina.com > http://lists.sistina.com/mailman/listinfo/linux-lvm > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-02-26 20:39 ` Joe Thornber @ 2001-02-26 21:03 ` Jorg de Jong 2001-02-26 21:38 ` Rik van Riel 2001-02-27 8:35 ` Patrick Caulfield 2001-02-27 12:21 ` Urs Thuermann 2 siblings, 1 reply; 22+ messages in thread From: Jorg de Jong @ 2001-02-26 21:03 UTC (permalink / raw) To: linux-lvm Joe Thornber wrote: > > On Mon, Feb 26, 2001 at 04:44:34PM +0100, Urs Thuermann wrote: > > I tried to use LVM today using Linux 2.4.2 and it mostly works like I > > expected. But the performance is unacceptably bad: > > Every now and then someone posts to the list claiming LVM performance > is really bad. We go off and try it and find the overhead is > typically less than 1%. > > Normally people are trying to stripe to two partitions on the same > device, but you aren't. > > Is any one else on the list seeing a similar performance hit ? > > - Joe here are my stats... (be your owen judge....) Jorg [root] /root > time dd if=/dev/hde of=/dev/null count=100000 100000+0 records in 100000+0 records out 0.200u 1.050s 0:03.95 31.6% 0+0k 0+0io 116pf+0w [root] /root > time dd if=/dev/vg00/vm of=/dev/null count=100000 100000+0 records in 100000+0 records out 0.160u 0.860s 0:02.92 34.9% 0+0k 0+0io 116pf+0w ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-02-26 21:03 ` Jorg de Jong @ 2001-02-26 21:38 ` Rik van Riel 0 siblings, 0 replies; 22+ messages in thread From: Rik van Riel @ 2001-02-26 21:38 UTC (permalink / raw) To: linux-lvm On Mon, 26 Feb 2001, Jorg de Jong wrote: > here are my stats... (be your owen judge....) > [root] /root > time dd if=/dev/hde of=/dev/null count=100000 > 100000+0 records in > 100000+0 records out > 0.200u 1.050s 0:03.95 31.6% 0+0k 0+0io 116pf+0w > > [root] /root > time dd if=/dev/vg00/vm of=/dev/null count=100000 > 100000+0 records in > 100000+0 records out > 0.160u 0.860s 0:02.92 34.9% 0+0k 0+0io 116pf+0w Interesting, so LVM actually takes a full second LESS to read in the same data ... Maybe LVM does some device-level readahead to speed things up ? regards, Rik -- Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-02-26 20:39 ` Joe Thornber 2001-02-26 21:03 ` Jorg de Jong @ 2001-02-27 8:35 ` Patrick Caulfield 2001-02-27 12:21 ` Urs Thuermann 2 siblings, 0 replies; 22+ messages in thread From: Patrick Caulfield @ 2001-02-27 8:35 UTC (permalink / raw) To: linux-lvm On Mon, Feb 26, 2001 at 08:39:33PM +0000, Joe Thornber wrote: > On Mon, Feb 26, 2001 at 04:44:34PM +0100, Urs Thuermann wrote: > > I tried to use LVM today using Linux 2.4.2 and it mostly works like I > > expected. But the performance is unacceptably bad: > > Every now and then someone posts to the list claiming LVM performance > is really bad. We go off and try it and find the overhead is > typically less than 1%. > > Normally people are trying to stripe to two partitions on the same > device, but you aren't. I've always seen bad performance doing dd on LVM volumes but very good performance in "real world" performance or bonnie++ tests. Stop worrying about it, how many times do you *need* to dd the start of a disk at high speed eh? :-) patrick ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-02-26 20:39 ` Joe Thornber 2001-02-26 21:03 ` Jorg de Jong 2001-02-27 8:35 ` Patrick Caulfield @ 2001-02-27 12:21 ` Urs Thuermann 2001-02-28 11:05 ` Heinz J. Mauelshagen 2 siblings, 1 reply; 22+ messages in thread From: Urs Thuermann @ 2001-02-27 12:21 UTC (permalink / raw) To: linux-lvm Joe Thornber <joe@66bassett.freeserve.co.uk> writes: > Every now and then someone posts to the list claiming LVM performance > is really bad. We go off and try it and find the overhead is > typically less than 1%. > > Normally people are trying to stripe to two partitions on the same > device, but you aren't. > > Is any one else on the list seeing a similar performance hit ? I did some more tests to ensure that this is not a caching issue. Also, I found the performance of dd on LVM depends much on the block size. I still have the effect that reading from LVM with small block size (smaller than 8k on my system) is *much* slower than reading the physical disk directly. IMHO it'd be worth looking into it and possibly fixing this performance hit. Could it be possible, that the read syscall on the LVM block device has so much overhead that reading small blocks can't cope with SCSI disk's transfer rate, so that additional disk revolutions are needed to read all the physical blocks? I find it noticable that when doubling the block size from 1k to 8k the time for reading quite exactly halves until it reaches the performance of reading /dev/sda5 directly. In case it matters, the machine I did the test on is a ASUS P2L97-S mainboard PII 333MHz 128 MB RAM Adaptec AIC-7880U onboard UW-SCSI controller /dev/sda is a IBM DCAS-34330W kernel: Linux 2.4.2 I've put the commands into a script that uses "set -x" to show what is exec'ed. Here's the result: Initialisation: + pvcreate /dev/sda5 pvcreate -- physical volume "/dev/sda5" successfully created + vgcreate vg0 /dev/sda5 vgcreate -- INFO: using default physical extent size 4 MB vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte vgcreate -- doing automatic backup of volume group "vg0" vgcreate -- volume group "vg0" successfully created and activated + lvcreate -n test /dev/vg0 -L 500M lvcreate -- doing automatic backup of "vg0" lvcreate -- logical volume "/dev/vg0/test" successfully created The first test shows, that, with small block sizes, LVM over sda5 is much slower than reading sda5 directly. I do the reading three times to see if any caching affects the numbers. This is obviously not the case: + dd if=/dev/sda5 of=/dev/null count=128k bs=512 131072+0 records in 131072+0 records out real 0m8.862s user 0m0.120s sys 0m1.690s + dd if=/dev/sda5 of=/dev/null count=128k bs=512 131072+0 records in 131072+0 records out real 0m8.860s user 0m0.200s sys 0m1.670s + dd if=/dev/sda5 of=/dev/null count=128k bs=512 131072+0 records in 131072+0 records out real 0m8.860s user 0m0.110s sys 0m1.800s + dd if=/dev/vg0/test of=/dev/null count=128k bs=512 131072+0 records in 131072+0 records out real 0m54.860s user 0m0.150s sys 0m2.780s + dd if=/dev/vg0/test of=/dev/null count=128k bs=512 131072+0 records in 131072+0 records out real 0m54.810s user 0m0.170s sys 0m2.660s + dd if=/dev/vg0/test of=/dev/null count=128k bs=512 131072+0 records in 131072+0 records out real 0m54.936s user 0m0.200s sys 0m3.010s The next test shows that the performance of LVM is very dependend on the blocksize but that of sda5 is not, which I find surprising since the same physical device is involved. + dd if=/dev/sda5 of=/dev/null count=512 bs=128k 512+0 records in 512+0 records out real 0m8.857s user 0m0.010s sys 0m1.440s + dd if=/dev/vg0/test of=/dev/null count=128k bs=512 131072+0 records in 131072+0 records out real 0m55.120s user 0m0.190s sys 0m2.760s + dd if=/dev/vg0/test of=/dev/null count=64k bs=1k 65536+0 records in 65536+0 records out real 0m55.195s user 0m0.170s sys 0m2.970s + dd if=/dev/vg0/test of=/dev/null count=32k bs=2k 32768+0 records in 32768+0 records out real 0m28.470s user 0m0.090s sys 0m1.830s + dd if=/dev/vg0/test of=/dev/null count=16k bs=4k 16384+0 records in 16384+0 records out real 0m15.209s user 0m0.010s sys 0m1.860s + dd if=/dev/vg0/test of=/dev/null count=8k bs=8k 8192+0 records in 8192+0 records out real 0m8.859s user 0m0.010s sys 0m1.670s + dd if=/dev/vg0/test of=/dev/null count=4k bs=16k 4096+0 records in 4096+0 records out real 0m8.860s user 0m0.010s sys 0m1.570s + dd if=/dev/vg0/test of=/dev/null count=2k bs=32k 2048+0 records in 2048+0 records out real 0m8.873s user 0m0.010s sys 0m1.390s + dd if=/dev/vg0/test of=/dev/null count=1k bs=64k 1024+0 records in 1024+0 records out real 0m8.883s user 0m0.000s sys 0m1.380s + dd if=/dev/vg0/test of=/dev/null count=512 bs=128k 512+0 records in 512+0 records out real 0m8.862s user 0m0.000s sys 0m1.370s Performance on filessystems on LVM is better, since the buffer cache can be used. I thought that the buffer cache works on block devices. But this seems to suggest that it only works with files systems mounted on the block device. Is this really so and why? + mke2fs -q /dev/vg0/test mke2fs 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09 + mount /dev/vg0/test /mnt + dd if=/dev/vg0/test of=/dev/null count=512 bs=128k 512+0 records in 512+0 records out real 0m8.792s user 0m0.000s sys 0m1.280s + dd if=/dev/vg0/test of=/dev/null count=512 bs=128k 512+0 records in 512+0 records out real 0m1.044s user 0m0.000s sys 0m1.050s For tests like this it would be useful to be able to completely flush the kernel's cache. Is there a tool to do this? urs ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-02-27 12:21 ` Urs Thuermann @ 2001-02-28 11:05 ` Heinz J. Mauelshagen 2001-04-10 19:30 ` Urs Thuermann 0 siblings, 1 reply; 22+ messages in thread From: Heinz J. Mauelshagen @ 2001-02-28 11:05 UTC (permalink / raw) To: linux-lvm Hi Urs, I can't reproduce at all what you are seeing with your configuration. My test configuration is: - Dual Pentium II/350 - 256MB RAM - Adaptec 2940UW - HDD Vendor: QUANTUM Model: XP34550W Rev: LXY4 (/dev/sdb) - first 300M of the LV involved allocated contiguous at the beginning of /dev/sdb1 - LVM 0.9.1 Beta 5 - Linux 2.4.2 - dd (GNU fileutils) 4.0 This table records test runs which where run one after the other *without* any additional load on the machine using dd to copy 300m into /dev/null. Each run performed twice with little difference + build average. 1k 2k 4k 8k 16k ----------------------------------------------------------- sdb1 32.63 32.8 32,7 32.55 32.9 vg00/u1 32.58 33.1 32.63 32.4 32.4 This gives *no* hint to me that there's any basical overhead invloved with LVM as oposed to the native device beeing used at all. Therefore I guess that the allocation policy of LVM might have an impact here. LVM calculates the offset of the first PE from the end of the PV. There could be drives used which suffer from that placement with regard to drive internal caching, latency or other model specific issues. Could everybody who posts with regard to this performance subject please provide - "lvdisplay -v" of the test LV - the drive vendor and model - the host adaptor used to access it - the motherboard and CPU(s) - the RAM size in order to get a better understanding. BTW: Rik van Riel's assumption in another mail of this thread, that LVM does some sort of read ahead is correct (as you can see with the above numbers). Even though we still only have a per block major read ahead in the kernel *and* multithreading will make the actual read ahead implementation fail with regard parallel access LVs (as Andrea Arcangeli pointed out a while ago), it hits quite good on not too heavy loaded systems. Taking into account that a lot of users most likely don't change the read ahead default of LVs, it won't have an impact at all. Regards, Heinz -- The LVM Guy -- On Tue, Feb 27, 2001 at 01:21:14PM +0100, Urs Thuermann wrote: > Joe Thornber <joe@66bassett.freeserve.co.uk> writes: > > > Every now and then someone posts to the list claiming LVM performance > > is really bad. We go off and try it and find the overhead is > > typically less than 1%. > > > > Normally people are trying to stripe to two partitions on the same > > device, but you aren't. > > > > Is any one else on the list seeing a similar performance hit ? > > I did some more tests to ensure that this is not a caching issue. > Also, I found the performance of dd on LVM depends much on the block > size. > > I still have the effect that reading from LVM with small block size > (smaller than 8k on my system) is *much* slower than reading the > physical disk directly. IMHO it'd be worth looking into it and > possibly fixing this performance hit. Could it be possible, that the > read syscall on the LVM block device has so much overhead that reading > small blocks can't cope with SCSI disk's transfer rate, so that > additional disk revolutions are needed to read all the physical > blocks? I find it noticable that when doubling the block size from > 1k to 8k the time for reading quite exactly halves until it reaches > the performance of reading /dev/sda5 directly. > > In case it matters, the machine I did the test on is a > ASUS P2L97-S mainboard > PII 333MHz > 128 MB RAM > Adaptec AIC-7880U onboard UW-SCSI controller > /dev/sda is a IBM DCAS-34330W > kernel: Linux 2.4.2 > > I've put the commands into a script that uses "set -x" to show what is > exec'ed. Here's the result: > > Initialisation: > > + pvcreate /dev/sda5 > pvcreate -- physical volume "/dev/sda5" successfully created > > + vgcreate vg0 /dev/sda5 > vgcreate -- INFO: using default physical extent size 4 MB > vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte > vgcreate -- doing automatic backup of volume group "vg0" > vgcreate -- volume group "vg0" successfully created and activated > > + lvcreate -n test /dev/vg0 -L 500M > lvcreate -- doing automatic backup of "vg0" > lvcreate -- logical volume "/dev/vg0/test" successfully created > > > The first test shows, that, with small block sizes, LVM over sda5 is > much slower than reading sda5 directly. I do the reading three times > to see if any caching affects the numbers. This is obviously not the > case: > > + dd if=/dev/sda5 of=/dev/null count=128k bs=512 > 131072+0 records in > 131072+0 records out > > real 0m8.862s > user 0m0.120s > sys 0m1.690s > + dd if=/dev/sda5 of=/dev/null count=128k bs=512 > 131072+0 records in > 131072+0 records out > > real 0m8.860s > user 0m0.200s > sys 0m1.670s > + dd if=/dev/sda5 of=/dev/null count=128k bs=512 > 131072+0 records in > 131072+0 records out > > real 0m8.860s > user 0m0.110s > sys 0m1.800s > + dd if=/dev/vg0/test of=/dev/null count=128k bs=512 > 131072+0 records in > 131072+0 records out > > real 0m54.860s > user 0m0.150s > sys 0m2.780s > + dd if=/dev/vg0/test of=/dev/null count=128k bs=512 > 131072+0 records in > 131072+0 records out > > real 0m54.810s > user 0m0.170s > sys 0m2.660s > + dd if=/dev/vg0/test of=/dev/null count=128k bs=512 > 131072+0 records in > 131072+0 records out > > real 0m54.936s > user 0m0.200s > sys 0m3.010s > > > The next test shows that the performance of LVM is very dependend on > the blocksize but that of sda5 is not, which I find surprising since > the same physical device is involved. > > + dd if=/dev/sda5 of=/dev/null count=512 bs=128k > 512+0 records in > 512+0 records out > > real 0m8.857s > user 0m0.010s > sys 0m1.440s > + dd if=/dev/vg0/test of=/dev/null count=128k bs=512 > 131072+0 records in > 131072+0 records out > > real 0m55.120s > user 0m0.190s > sys 0m2.760s > + dd if=/dev/vg0/test of=/dev/null count=64k bs=1k > 65536+0 records in > 65536+0 records out > > real 0m55.195s > user 0m0.170s > sys 0m2.970s > + dd if=/dev/vg0/test of=/dev/null count=32k bs=2k > 32768+0 records in > 32768+0 records out > > real 0m28.470s > user 0m0.090s > sys 0m1.830s > + dd if=/dev/vg0/test of=/dev/null count=16k bs=4k > 16384+0 records in > 16384+0 records out > > real 0m15.209s > user 0m0.010s > sys 0m1.860s > + dd if=/dev/vg0/test of=/dev/null count=8k bs=8k > 8192+0 records in > 8192+0 records out > > real 0m8.859s > user 0m0.010s > sys 0m1.670s > + dd if=/dev/vg0/test of=/dev/null count=4k bs=16k > 4096+0 records in > 4096+0 records out > > real 0m8.860s > user 0m0.010s > sys 0m1.570s > + dd if=/dev/vg0/test of=/dev/null count=2k bs=32k > 2048+0 records in > 2048+0 records out > > real 0m8.873s > user 0m0.010s > sys 0m1.390s > + dd if=/dev/vg0/test of=/dev/null count=1k bs=64k > 1024+0 records in > 1024+0 records out > > real 0m8.883s > user 0m0.000s > sys 0m1.380s > + dd if=/dev/vg0/test of=/dev/null count=512 bs=128k > 512+0 records in > 512+0 records out > > real 0m8.862s > user 0m0.000s > sys 0m1.370s > > > Performance on filessystems on LVM is better, since the buffer cache > can be used. I thought that the buffer cache works on block devices. > But this seems to suggest that it only works with files systems > mounted on the block device. Is this really so and why? > > + mke2fs -q /dev/vg0/test > mke2fs 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09 > + mount /dev/vg0/test /mnt > + dd if=/dev/vg0/test of=/dev/null count=512 bs=128k > 512+0 records in > 512+0 records out > > real 0m8.792s > user 0m0.000s > sys 0m1.280s > + dd if=/dev/vg0/test of=/dev/null count=512 bs=128k > 512+0 records in > 512+0 records out > > real 0m1.044s > user 0m0.000s > sys 0m1.050s > > > For tests like this it would be useful to be able to completely flush > the kernel's cache. Is there a tool to do this? > > > urs > _______________________________________________ > linux-lvm mailing list > linux-lvm@sistina.com > http://lists.sistina.com/mailman/listinfo/linux-lvm *** Software bugs are stupid. Nevertheless it needs not so stupid people to solve them *** =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Heinz Mauelshagen Sistina Software Inc. Senior Consultant/Developer Am Sonnenhang 11 56242 Marienrachdorf Germany Mauelshagen@Sistina.com +49 2626 141200 FAX 924446 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-02-28 11:05 ` Heinz J. Mauelshagen @ 2001-04-10 19:30 ` Urs Thuermann 2001-04-10 21:37 ` Andreas Dilger 0 siblings, 1 reply; 22+ messages in thread From: Urs Thuermann @ 2001-04-10 19:30 UTC (permalink / raw) To: linux-lvm Today, I've found some time to make some further performance tests on my system. "Heinz J. Mauelshagen" <Mauelshagen@sistina.com> writes: > Hi Urs, > > I can't reproduce at all what you are seeing with your configuration. > > > My test configuration is: > > - Dual Pentium II/350 > - 256MB RAM > - Adaptec 2940UW > - HDD Vendor: QUANTUM Model: XP34550W Rev: LXY4 (/dev/sdb) > - first 300M of the LV involved allocated contiguous at the beginning > of /dev/sdb1 > - LVM 0.9.1 Beta 5 > - Linux 2.4.2 > - dd (GNU fileutils) 4.0 I find surprinsing that we get so different performance results since my hardware is very similar to yours. Here is it, again: Mainboard: ASUS P2L97-S with UW-SCSI controller onboard RAM: 128 MB CPU: from /proc/cpuinfo: model name : Pentium II (Deschutes) stepping : 1 cpu MHz : 334.095 /dev/sda: from /proc/scsi/scsi: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: IBM Model: DCAS-34330W Rev: S65A Type: Direct-Access ANSI SCSI revision: 02 SCSI controller: from /proc/scsi/aic7xxx/0: Adaptec AIC7xxx driver version: 6.1.5 aic7880: Wide Channel A, SCSI Id=7, 16/255 SCBs Channel A Target 0 Negotiation Settings User: 40.000MB/s transfers (20.000MHz, offset 255, 16bit) Goal: 40.000MB/s transfers (20.000MHz, offset 8, 16bit) Curr: 40.000MB/s transfers (20.000MHz, offset 8, 16bit) Channel A Target 0 Lun 0 Settings Commands Queued 13557298 Commands Active 0 Command Openings 64 Max Tagged Openings 64 Device Queue Frozen Count 0 kernel: Linux-2.4.3 The size of /dev/sda5 is # fdisk -l /dev/sda | egrep '^[^/]|sda5' Disk /dev/sda: 255 heads, 63 sectors, 527 cylinders Units = cylinders of 16065 * 512 bytes Device Boot Start End Blocks Id System /dev/sda5 80 399 2570368+ 8e Linux LVM i.e. 627 PEs of 4MB each. After pvcreate /dev/sda5; vgcreate vg0 /dev/sda5; lvcreate -n test vg0 -l 627 I have # vgdisplay -v --- Volume group --- VG Name vg0 VG Access read/write VG Status available/resizable VG # 0 MAX LV 256 Cur LV 1 Open LV 0 MAX LV Size 255.99 GB Max PV 256 Cur PV 1 Act PV 1 VG Size 2.45 GB PE Size 4 MB Total PE 627 Alloc PE / Size 627 / 2.45 GB Free PE / Size 0 / 0 VG UUID zmdKrC-eis5-rgWV-C5rz-TcJE-8tTI-iUUeBj --- Logical volume --- LV Name /dev/vg0/test VG Name vg0 LV Write Access read/write LV Status available LV # 1 # open 0 LV Size 2.45 GB Current LE 627 Allocated LE 627 Allocation next free Read ahead sectors 120 Block device 58:0 --- Physical volumes --- PV Name (#) /dev/sda5 (1) PV Status available / allocatable Total PE / Free PE 627 / 0 and lvdisplay -v /dev/vg0/test shows a mapping of --- Logical extents --- LE PV PE reads writes 00000 /dev/sda5 00000 278528 2 00001 /dev/sda5 00001 266240 0 00002 /dev/sda5 00002 266240 0 00003 /dev/sda5 00003 266240 0 ... 00625 /dev/sda5 00625 266240 0 00626 /dev/sda5 00626 266240 0 > This table records test runs which where run one after the other > *without* any additional load on the machine using dd to copy 300m > into /dev/null. Each run performed twice with little difference + > build average. > > 1k 2k 4k 8k 16k > ----------------------------------------------------------- > sdb1 32.63 32.8 32,7 32.55 32.9 > vg00/u1 32.58 33.1 32.63 32.4 32.4 For me, numbers look quite different. I have then run the following test, which reads all the 627 PEs (2.45 GB) from /dev/vg0/test and the same amount from /dev/sda5 at different block sizes. The system was otherwise almost idle when I ran this test. #!/bin/sh function doit() { time dd if=/dev/sda5 of=/dev/null count=$1 bs=$2 time dd if=/dev/sda5 of=/dev/null count=$1 bs=$2 time dd if=/dev/sda5 of=/dev/null count=$1 bs=$2 time dd if=/dev/sda5 of=/dev/null count=$1 bs=$2 time dd if=/dev/vg0/test of=/dev/null count=$1 bs=$2 time dd if=/dev/vg0/test of=/dev/null count=$1 bs=$2 time dd if=/dev/vg0/test of=/dev/null count=$1 bs=$2 time dd if=/dev/vg0/test of=/dev/null count=$1 bs=$2 } set -x doit 40128 64k doit 80256 32k doit 160512 16k doit 321024 8k doit 642048 4k doit 1284096 2k doit 2568192 1k doit 5136384 512 The elapsed times of the 4 runs of each test are averaged, where for each test the variations among the 4 times were less than 1% of the average time (times in seconds): 64k 32k 16k 8k 4k 2k 1k 512 ------------------------------------------------------------------------------ /dev/sda5 389.4 389.4 388.9 388.7 388.5 388.6 389.1 388.9 /dev/vg0/test 389.3 389.6 390.9 393.1 607.5 1147.2 2194.9 2195.8 > Therefore I guess that the allocation policy of LVM might have an > impact here. > > LVM calculates the offset of the first PE from the end of the PV. > There could be drives used which suffer from that placement with > regard to drive internal caching, latency or other model specific > issues. I think you mean that when I read a number of blocks from /dev/sda5 I read them from the beginning of /dev/sda5 while when reading the same number of blocks from /dev/vg0/test I read from the end of the PV /dev/sda5 since the PE are allocated from the end, right. But in my test, I read the whole /dev/sda5 and the whole LV which has allocated all PEs of /dev/sda5 to it. So about the same sectors are read, just the order is different. Do you think that can reduce the performance from 2.45GB/390s = 6.43MB/s to 2.45GB/2195s = 1.14MB/s? I find this very unlikely. Is there any other explanation for this performance degradation? urs ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-04-10 19:30 ` Urs Thuermann @ 2001-04-10 21:37 ` Andreas Dilger 2001-04-11 11:23 ` Urs Thuermann 0 siblings, 1 reply; 22+ messages in thread From: Andreas Dilger @ 2001-04-10 21:37 UTC (permalink / raw) To: linux-lvm Urs, you write: > Today, I've found some time to make some further performance tests on > my system. > > The elapsed times of the 4 runs of each test are averaged, where for > each test the variations among the 4 times were less than 1% of the > average time (times in seconds): > > 64k 32k 16k 8k 4k 2k 1k 512 > ------------------------------------------------------------------------------ > /dev/sda5 389.4 389.4 388.9 388.7 388.5 388.6 389.1 388.9 > /dev/vg0/test 389.3 389.6 390.9 393.1 607.5 1147.2 2194.9 2195.8 > > > There could be drives used which suffer from that placement with > > regard to drive internal caching, latency or other model specific > > issues. > > I think you mean that when I read a number of blocks from /dev/sda5 I > read them from the beginning of /dev/sda5 while when reading the same > number of blocks from /dev/vg0/test I read from the end of the PV > /dev/sda5 since the PE are allocated from the end, right. No, PEs are allocated from the start, but are only aligned with the end of the disk/partition (to give maximum room to LVM for metadata). If you have contiguous LE allocation (which you appear to have), the read order for LVM should basically be exactly the same as for a raw disk. Effectively all that LVM is doing is adding a small offset to the disk blocks to leave room for the VGDA, nothing more. > But in my test, I read the whole /dev/sda5 and the whole LV which > has allocated all PEs of /dev/sda5 to it. So about the same sectors > are read, just the order is different. Do you think that can reduce > the performance from 2.45GB/390s = 6.43MB/s to 2.45GB/2195s = 1.14MB/s? > I find this very unlikely. No, shouldn't be a problem with placement, because you are reading 4MB contiguous chunks (with default PE size). This should be enough to minimize seek impact. Do you notice the sound of frantic head seeking when you are running with small block sizes? > Is there any other explanation for this performance degradation? Other than some strange seek pattern, I can't think of anything. If you look at the (non-snapshot and non-blocking because of pv_move) path of lvm_map, it boils down to: pe_num = lvm_block / pe_size; new_dev = pe[pe_num].dev; new_block = (old_block % pe_size) + pe[pe_num].start; So basically a few memory accesses and a few math operations. Nothing _should_ affect disk performance at all. With < 2k I/O you are forcing read-modify-write on pages, but this shouldn't be an issue, because you are also doing the same for the raw disk. One thing that now comes to mind is if the LVM I/O is not aligned, because LVM PEs are aligned with the end of the PV and not the start, this _may_ cause performance problems because of misaligned read-modify-write causing extra seeking (purely speculation). Try the following patch, which will align the PE structs to 64kB boundaries from the start of the disk, as well as making the other large VGDA structs start on 4k boundaries. I have tested the new alignment, and it works fine with existing LVM code. However, the patch is extracted from 8000+ lines of diff (yuk), so there may be a few cases where it doesn't apply cleanly or I missed something in a non-critical file. Basically, if it compiles it will work OK. Cheers, Andreas ======================= lvm-0.9.1b7-align.diff =========================== diff -u -u -r1.5.2.38 lvm.h --- kernel/lvm.h 2001/04/10 17:04:18 1.5.2.38 +++ kernel/lvm.h 2001/04/10 20:44:36 @@ -246,39 +246,34 @@ */ /* DONT TOUCH THESE !!! */ +#define LVM_VGDA_ALIGN (4096UL - 1) +#define LVM_VGDA_ROUND(size) (((size) + LVM_VGDA_ALIGN) & ~LVM_VGDA_ALIGN) + /* base of PV structure in disk partition */ -#define LVM_PV_DISK_BASE 0L +#define LVM_PV_DISK_BASE 0L /* size reserved for PV structure on disk */ -#define LVM_PV_DISK_SIZE 1024L +#define LVM_PV_DISK_SIZE 1024L /* base of VG structure in disk partition */ -#define LVM_VG_DISK_BASE LVM_PV_DISK_SIZE +#define LVM_VG_DISK_BASE LVM_VGDA_ROUND ( LVM_PV_DISK_BASE + \ + LVM_PV_DISK_SIZE) /* size reserved for VG structure */ -#define LVM_VG_DISK_SIZE ( 9 * 512L) - -/* size reserved for timekeeping */ -#define LVM_TIMESTAMP_DISK_BASE ( LVM_VG_DISK_BASE + LVM_VG_DISK_SIZE) -#define LVM_TIMESTAMP_DISK_SIZE 512L /* reserved for timekeeping */ +#define LVM_VG_DISK_SIZE sizeof ( vg_disk_t) /* name list of physical volumes on disk */ -#define LVM_PV_UUIDLIST_DISK_BASE ( LVM_TIMESTAMP_DISK_BASE + \ - LVM_TIMESTAMP_DISK_SIZE) +#define LVM_PV_UUIDLIST_DISK_BASE LVM_VGDA_ROUND ( LVM_VG_DISK_BASE + \ + LVM_VG_DISK_SIZE) /* now for the dynamically calculated parts of the VGDA */ #define LVM_LV_DISK_OFFSET(a, b) ( (a)->lv_on_disk.base + \ sizeof ( lv_disk_t) * b) -#define LVM_DISK_SIZE(pv) ( (pv)->pe_on_disk.base + \ +#define LVM_VGDA_SIZE(pv) ( (pv)->pe_on_disk.base + \ (pv)->pe_on_disk.size) #define LVM_PE_DISK_OFFSET(pe, pv) ( pe * pv->pe_size + \ - ( LVM_DISK_SIZE ( pv) / SECTOR_SIZE)) -#define LVM_PE_ON_DISK_BASE(pv) \ - { int rest; \ - pv->pe_on_disk.base = pv->lv_on_disk.base + pv->lv_on_disk.size; \ - if ( ( rest = pv->pe_on_disk.base % SECTOR_SIZE) != 0) \ - pv->pe_on_disk.base += ( SECTOR_SIZE - rest); \ - } + ( LVM_VGDA_SIZE ( pv) / SECTOR_SIZE)) +#define LVM_PE_ALIGN (65536UL / SECTOR_SIZE - 1) /* END default disk spaces and offsets for PVs */ diff -u -u -r1.2.2.8 vgcreate.c --- tools/vgcreate.c 2001/04/09 12:53:40 1.2.2.8 +++ tools/vgcreate.c 2001/04/10 20:44:43 @@ -101,8 +101,8 @@ int opt_v = 0; int p = 0; int p1 = 0; - int max_lv = MAX_LV; - int max_pv = MAX_PV; + int max_lv = MAX_LV - 1; + int max_pv = MAX_PV - 1; int min_pv_index = 0; ulong max_pv_size = 0; ulong min_pv_size = -1; diff -u -u -r1.5.2.17 liblvm.h --- tools/lib/liblvm.h 2001/04/09 14:18:25 1.5.2.17 +++ tools/lib/liblvm.h 2001/04/10 20:44:46 @@ -99,13 +99,9 @@ extern struct config_file *config_file; -#define SECTOR_SIZE 512 #define BLKGETSIZE _IO(0x12,96) /* return device size */ #define BLKRASET _IO(0x12,98) /* Set read ahead */ #define EXT2_SUPER_MAGIC 0xEF53 -#ifndef BLOCK_SIZE -#define BLOCK_SIZE 1024 -#endif #define LVM_ID "HM" /* Identifier PV (id in pv_t) */ #define EXPORTED "PV_EXP" /* Identifier exported PV (system_id in pv_t) */ diff -u -u -r1.2.2.2 pv_read_pe.c --- tools/lib/pv_read_pe.c 2001/04/09 07:49:07 1.2.2.2 +++ tools/lib/pv_read_pe.c 2001/04/10 20:44:52 @@ -58,8 +57,7 @@ *pe = NULL; size = pv->pe_total * sizeof ( pe_disk_t); - if ( size + pv->pe_on_disk.base > - LVM_DISK_SIZE ( pv)) { + if ( size + pv->pe_on_disk.base > LVM_VGDA_SIZE ( pv)) { ret = -LVM_EPV_READ_PE_SIZE; goto pv_read_pe_end; } diff -u -u -r1.1.2.1 pv_release_pe.c --- tools/lib/pv_release_pe.c 2001/02/19 10:04:34 1.1.2.1 +++ tools/lib/pv_release_pe.c 2001/04/10 20:44:52 @@ -82,8 +82,7 @@ goto pv_release_pe_end; } pe_index = ( vg->lv[l]->lv_current_pe[p].pe - - LVM_DISK_SIZE ( vg->pv[pv_num]) / - SECTOR_SIZE) / + LVM_VGDA_SIZE ( vg->pv[pv_num]) / SECTOR_SIZE) / vg->pe_size; debug ( "pv_release_pe -- pv_name: %s pe: %lu sector: %lu\n", vg->pv[pv_num]->pv_name, diff -u -u -r1.2.2.1 pv_write_pe.c --- tools/lib/pv_write_pe.c 2001/02/19 10:04:34 1.2.2.1 +++ tools/lib/pv_write_pe.c 2001/04/10 20:44:52 @@ -50,7 +50,7 @@ pv == NULL) ret = -LVM_EPARAM; else { size = pv->pe_total * sizeof ( pe_disk_t); - if ( size + pv->pe_on_disk.base > LVM_DISK_SIZE ( pv)) + if ( size + pv->pe_on_disk.base > LVM_VGDA_SIZE ( pv)) ret = -LVM_EPV_WRITE_PE_SIZE;; if ( ( pv_handle = open ( pv_name, O_WRONLY)) == -1) ret = -LVM_EPV_WRITE_PE_OPEN; diff -u -u -r1.4.2.1 vg_setup_for_create.c --- tools/lib/vg_setup_for_create.c 2001/02/19 10:04:34 1.4.2.1 +++ tools/lib/vg_setup_for_create.c 2001/04/10 20:44:54 @@ -41,12 +41,9 @@ int vg_setup_for_create ( char *vg_name, vg_t *vg, pv_t **pv, int pe_size, ulong max_pv, ulong max_lv) { - int l = 0; int p = 0; int ret = 0; uint pe_total = 0; - ulong rest = 0; - ulong size = 0; debug_enter ( "vg_setup_for_create -- CALLED\n"); @@ -76,67 +73,66 @@ vg->pvg_total = 0; vg->proc = NULL; memset ( vg->vg_uuid, 0, sizeof ( vg->vg_uuid)); - memcpy ( vg->vg_uuid, lvm_create_uuid ( UUID_LEN), sizeof ( vg->vg_uuid)-1); + memcpy ( vg->vg_uuid, lvm_create_uuid ( UUID_LEN), sizeof ( vg->vg_uuid)); + memset ( vg->pv, 0, sizeof ( pv_t *) * vg->pv_max); + memset ( vg->lv, 0, sizeof ( lv_t *) * vg->lv_max); /* Walk through physical volumes */ pe_total = 0; for ( p = 0; pv[p] != NULL; p++) { - if ( pv_get_size ( pv[p]->pv_name, NULL) / - pe_size < LVM_PE_SIZE_PV_SIZE_REL) { + pv_t *pp = pv[p]; + if ( ( pp->pv_size = pv_get_size ( pp->pv_name, NULL)) / pe_size < + LVM_PE_SIZE_PV_SIZE_REL) { ret = -LVM_EVG_SETUP_FOR_CREATE_PV_SIZE; goto vg_setup_for_create_end; } - vg->pv[p] = pv[p]; - vg->pv[p]->pv_on_disk.base = LVM_PV_DISK_BASE; - vg->pv[p]->pv_on_disk.size = LVM_PV_DISK_SIZE; - vg->pv[p]->vg_on_disk.base = LVM_VG_DISK_BASE; - vg->pv[p]->vg_on_disk.size = LVM_VG_DISK_SIZE; - vg->pv[p]->pv_uuidlist_on_disk.base = LVM_PV_UUIDLIST_DISK_BASE; - vg->pv[p]->pv_uuidlist_on_disk.size = ( max_pv + 1) * NAME_LEN; - - size = vg->pv[p]->pv_uuidlist_on_disk.base + - vg->pv[p]->pv_uuidlist_on_disk.size; - vg->pv[p]->lv_on_disk.base = (size / SECTOR_SIZE) * SECTOR_SIZE; - if (size % SECTOR_SIZE) - vg->pv[p]->lv_on_disk.base += SECTOR_SIZE; - vg->pv[p]->lv_on_disk.size = ( max_lv + 1) * sizeof ( lv_disk_t); - LVM_PE_ON_DISK_BASE ( vg->pv[p]); - strcpy ( vg->pv[p]->vg_name, vg->vg_name); - vg->pv[p]->pv_number = p + 1; - vg->pv[p]->pv_status = 0; /* bitfield */ - vg->pv[p]->pv_allocatable = PV_ALLOCATABLE; /* bitfield */ - vg->pv[p]->pe_size = vg->pe_size; - - rest = vg->pv[p]->pv_size - vg->pv[p]->pe_on_disk.base / SECTOR_SIZE; - vg->pv[p]->pe_total = rest / vg->pe_size; - rest -= ( vg->pv[p]->pe_total * vg->pe_size); - while ( rest * SECTOR_SIZE / sizeof ( pe_disk_t) < vg->pv[p]->pe_total) { - rest += vg->pe_size; - vg->pv[p]->pe_total--; + + vg->pv[p] = pp; + strcpy ( pp->vg_name, vg->vg_name); + pp->pv_number = p + 1; + pp->pv_status = 0; /* bitfield */ + pp->pv_allocatable = PV_ALLOCATABLE; /* bitfield */ + pp->pe_size = vg->pe_size; + + pp->pv_on_disk.base = LVM_PV_DISK_BASE; + pp->pv_on_disk.size = LVM_PV_DISK_SIZE; + pp->vg_on_disk.base = LVM_VG_DISK_BASE; + pp->vg_on_disk.size = LVM_VG_DISK_SIZE; + pp->pv_uuidlist_on_disk.base = LVM_PV_UUIDLIST_DISK_BASE; + pp->pv_uuidlist_on_disk.size = ( max_pv + 1) * NAME_LEN; + + pp->lv_on_disk.base = LVM_VGDA_ROUND ( pp->pv_uuidlist_on_disk.base + + pp->pv_uuidlist_on_disk.size); + pp->lv_on_disk.size = (max_lv + 1) * sizeof ( lv_disk_t); + pp->pe_on_disk.base = LVM_VGDA_ROUND ( pp->lv_on_disk.base + + pp->lv_on_disk.size); + + /* This may be one too many PEs if we don't have enough space for VGDA */ + pp->pe_total = (pp->pv_size & ~LVM_PE_ALIGN) / pp->pe_size; + pp->pe_on_disk.size = ((pp->pv_size & ~LVM_PE_ALIGN) - + pp->pe_total * vg->pe_size) * SECTOR_SIZE - + pp->pe_on_disk.base; + if (pp->pe_on_disk.size < pp->pe_total * sizeof ( pe_disk_t)) { + --pp->pe_total; + pp->pe_on_disk.size += vg->pe_size * SECTOR_SIZE; } - vg->pv[p]->pe_on_disk.size = ( vg->pv[p]->pv_size - - vg->pv[p]->pe_total * vg->pe_size - - vg->pv[p]->pe_on_disk.base / SECTOR_SIZE) - * SECTOR_SIZE; - if ( LVM_DISK_SIZE ( vg->pv[p]) % BLOCK_SIZE > 0) - vg->pv[p]->pe_on_disk.size -= SECTOR_SIZE; - if ( ( vg->pv[p]->pe = malloc ( vg->pv[p]->pe_total * - sizeof ( pe_disk_t))) == NULL) { + if ( pp->pe_total > LVM_PE_T_MAX) { + ret = -LVM_EVG_SETUP_FOR_CREATE_PV_SIZE; + goto vg_setup_for_create_end; + } + if ( ( pp->pe = malloc ( pp->pe_total * sizeof ( pe_disk_t))) == NULL) { fprintf ( stderr, "malloc error in %s [line %d]\n", __FILE__, __LINE__); ret = -LVM_EVG_SETUP_FOR_CREATE_MALLOC; goto vg_setup_for_create_end; } - memset ( vg->pv[p]->pe, 0, vg->pv[p]->pe_total * sizeof ( pe_disk_t)); - vg->pv[p]->pe_allocated = 0; - vg->pv[p]->pe_stale = 0; - pe_total += vg->pv[p]->pe_total; + memset ( pp->pe, 0, pp->pe_total * sizeof ( pe_disk_t)); + pp->pe_allocated = 0; + pp->pe_stale = 0; + pe_total += pp->pe_total; } vg->pv_cur = vg->pv_act = p; vg->pe_total = pe_total; - - for ( ; p < vg->pv_max; p++) vg->pv[p] = NULL; /* Rest should be NULL */ - for ( l = 0; l < vg->lv_max; l++) vg->lv[l] = NULL; /* No LVs in NEW VG */ vg_setup_for_create_end: diff -u -u -r1.2.2.1 vg_setup_for_extend.c --- tools/lib/vg_setup_for_extend.c 2001/02/19 10:04:34 1.2.2.1 +++ tools/lib/vg_setup_for_extend.c 2001/04/10 20:44:54 @@ -44,7 +44,6 @@ int p1 = 0; int ret = 0; uint size = 0; - ulong rest = 0; debug_enter ( "vg_setup_for_extend -- CALLED\n"); @@ -66,7 +65,8 @@ } for ( p = 0; all_pv[p] != NULL; p++) { - if ( strcmp ( extend_pv[ep], all_pv[p]->pv_name) == 0) { + pv_t *pp = all_pv[p]; + if ( strcmp ( extend_pv[ep], pp->pv_name) == 0) { debug ( "%s -- checking for maximum physical " "volume count of %s\n", cmd, vg->vg_name); if ( np >= vg->pv_max) { @@ -74,9 +74,9 @@ goto vg_setup_for_extend_end; } debug ( "%s -- getting size of physical " - "volume %s\n", cmd, all_pv[p]->pv_name); - if ( ( size = pv_get_size ( all_pv[p]->pv_name, NULL)) <= 0) { - *error_pv_name = all_pv[p]->pv_name; + "volume %s\n", cmd, pp->pv_name); + if ( ( size = pv_get_size ( pp->pv_name, NULL)) <= 0) { + *error_pv_name = pp->pv_name; ret = -LVM_EVG_SETUP_FOR_EXTEND_PV_GET_SIZE; goto vg_setup_for_extend_end; } @@ -91,72 +92,61 @@ goto vg_setup_for_extend_end; } } - debug ( "%s -- checking for new physical " - "volume %s\n", cmd, all_pv[p]->pv_name); - if ( pv_check_new ( all_pv[p]) == FALSE) { - *error_pv_name = all_pv[p]->pv_name; + debug ( "%s -- checking for new physical volume %s\n", + cmd, pp->pv_name); + if ( pv_check_new ( pp) == FALSE) { + *error_pv_name = pp->pv_name; ret = -LVM_EVG_SETUP_FOR_EXTEND_PV_CHECK_NEW; goto vg_setup_for_extend_end; } - if ( pv_get_size ( all_pv[p]->pv_name, NULL) / - vg->pv[0]->pe_size < LVM_PE_SIZE_PV_SIZE_REL) { - *error_pv_name = all_pv[p]->pv_name; + if ( size / vg->pv[0]->pe_size < LVM_PE_SIZE_PV_SIZE_REL) { + *error_pv_name = pp->pv_name; ret = -LVM_EVG_SETUP_FOR_EXTEND_PV_SIZE_REL; goto vg_setup_for_extend_end; } /* setup PV and correct VG */ - all_pv[p]->vg_on_disk.base = vg->pv[0]->vg_on_disk.base; - all_pv[p]->vg_on_disk.size = vg->pv[0]->vg_on_disk.size; - all_pv[p]->pv_uuidlist_on_disk.base = - vg->pv[0]->pv_uuidlist_on_disk.base; - all_pv[p]->pv_uuidlist_on_disk.size = - vg->pv[0]->pv_uuidlist_on_disk.size; - all_pv[p]->lv_on_disk.base = vg->pv[0]->lv_on_disk.base; - all_pv[p]->lv_on_disk.size = vg->pv[0]->lv_on_disk.size; - all_pv[p]->pe_on_disk.base = vg->pv[0]->pe_on_disk.base; - rest = all_pv[p]->pv_size - all_pv[p]->pe_on_disk.base / - SECTOR_SIZE; - all_pv[p]->pe_total = rest / vg->pe_size; - rest -= ( all_pv[p]->pe_total * vg->pe_size); - while ( rest * SECTOR_SIZE / sizeof ( pe_disk_t) < - all_pv[p]->pe_total) { - rest += vg->pe_size; - all_pv[p]->pe_total--; - } - all_pv[p]->pe_on_disk.size = - ( all_pv[p]->pv_size - all_pv[p]->pe_total * vg->pe_size - - all_pv[p]->pe_on_disk.base / SECTOR_SIZE) * SECTOR_SIZE; - if ( LVM_DISK_SIZE ( all_pv[p]) % BLOCK_SIZE > 0) - all_pv[p]->pe_on_disk.size -= SECTOR_SIZE; - strcpy ( all_pv[p]->vg_name, vg->vg_name); - all_pv[p]->pv_dev = pv_create_kdev_t ( all_pv[p]->pv_name); - all_pv[p]->pv_status = 0; /* bitfield */ - all_pv[p]->pv_allocatable = PV_ALLOCATABLE; - all_pv[p]->pv_size = size; - all_pv[p]->lv_cur = 0; - all_pv[p]->pe_size = vg->pe_size; - all_pv[p]->pe_total = ( all_pv[p]->pv_size - - ( all_pv[p]->pe_on_disk.base + - all_pv[p]->pe_on_disk.size) / SECTOR_SIZE ) / - vg->pe_size; - all_pv[p]->pe_allocated = 0; - all_pv[p]->pe_stale = 0; - if ( ( all_pv[p]->pe = malloc ( all_pv[p]->pe_total * + strcpy ( pp->vg_name, vg->vg_name); + pp->pv_number = np + 1; + pp->pv_dev = pv_create_kdev_t ( pp->pv_name); + pp->pv_status = 0; /* bitfield */ + pp->pv_allocatable = PV_ALLOCATABLE; + pp->pv_size = size; + pp->lv_cur = 0; + pp->pe_size = vg->pe_size; + + pp->vg_on_disk.base = vg->pv[0]->vg_on_disk.base; + pp->vg_on_disk.size = vg->pv[0]->vg_on_disk.size; + pp->pv_uuidlist_on_disk.base = vg->pv[0]->pv_uuidlist_on_disk.base; + pp->pv_uuidlist_on_disk.size = vg->pv[0]->pv_uuidlist_on_disk.size; + pp->lv_on_disk.base = vg->pv[0]->lv_on_disk.base; + pp->lv_on_disk.size = vg->pv[0]->lv_on_disk.size; + pp->pe_on_disk.base = vg->pv[0]->pe_on_disk.base; + /* This may be one too many PEs if not enough space for VGDA */ + pp->pe_total = (pp->pv_size & ~LVM_PE_ALIGN) / pp->pe_size; + pp->pe_on_disk.size = ((pp->pv_size & ~LVM_PE_ALIGN) - + pp->pe_total * vg->pe_size) * SECTOR_SIZE - + pp->pe_on_disk.base; + if (pp->pe_on_disk.size < pp->pe_total * sizeof ( pe_disk_t)) { + --pp->pe_total; + pp->pe_on_disk.size += vg->pe_size * SECTOR_SIZE; + } + if ( ( pp->pe = malloc ( pp->pe_total * sizeof ( pe_disk_t))) == NULL) { fprintf ( stderr, "%s -- malloc error in %s at line %d\n\n", cmd, __FILE__, __LINE__); ret = -LVM_EVG_SETUP_FOR_EXTEND_MALLOC; goto vg_setup_for_extend_end; } - memset ( all_pv[p]->pe, - 0, - all_pv[p]->pe_total * sizeof ( pe_disk_t)); - vg->pv[np] = all_pv[p]; + memset ( pp->pe, 0, pp->pe_total * sizeof ( pe_disk_t)); + pp->pe_allocated = 0; + pp->pe_stale = 0; + + vg->pv[np] = pp; vg->pv_act++; vg->pv_cur++; - vg->pe_total += vg->pv[np]->pe_total; + vg->pe_total += pp->pe_total; np++; new_pv++; -- Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto, \ would they cancel out, leaving him still hungry?" http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-04-10 21:37 ` Andreas Dilger @ 2001-04-11 11:23 ` Urs Thuermann 2001-04-11 19:49 ` Andreas Dilger 0 siblings, 1 reply; 22+ messages in thread From: Urs Thuermann @ 2001-04-11 11:23 UTC (permalink / raw) To: linux-lvm Andreas Dilger <adilger@turbolinux.com> writes: > No, PEs are allocated from the start, but are only aligned with the > end of the disk/partition (to give maximum room to LVM for > metadata). If you have contiguous LE allocation (which you appear > to have), the read order for LVM should basically be exactly the > same as for a raw disk. Effectively all that LVM is doing is adding > a small offset to the disk blocks to leave room for the VGDA, > nothing more. Yes, I found that out after my posting by writing a 32 Mb ( = 8 PEs) pattern to the beginning of /dev/vg0/test and searching for it in /dev/sda5. In my case the offset is of the first PE is 4352 sectors, since /dev/sda5 has 627 * 4 MB + 4352 * 512 bytes. This small offset shouldn't have an impact on the performance when reading 2.45 GB. > Do you notice the sound of frantic head seeking when you are running > with small block sizes? No, sounds very similar and only very moderate head seeking (probably from syslogd). > So basically a few memory accesses and a few math operations. > Nothing _should_ affect disk performance at all. With < 2k I/O you > are forcing read-modify-write on pages, but this shouldn't be an > issue, because you are also doing the same for the raw disk. What has that all to do with pages? Isn't it the hardsect_size that is relevant here? And why is < 2k the relevant limit? If the page size was relevant this is 4k. If the hardsect_size is relevant, this is 1k in 2.4.3 (after initialization, but can grow when mounting a FS with larger block size, due to the bug I reported on this list) and if I read the src in 0.9.1b7 correctly Andrea's patch makes the hardsect_size equal to the maximum sector size of all PVs, i.e. 512 bytes with most disks. > One thing that now comes to mind is if the LVM I/O is not aligned, > because LVM PEs are aligned with the end of the PV and not the > start, this _may_ cause performance problems because of misaligned > read-modify-write causing extra seeking (purely speculation). I don't understand, what you're saying here (and above). I was *reading* with dd if=/dev/vg0/test of=/dev/null. So why is there an RMW involved? What does happen, when I do a write(2) of a 512-byte block to /dev/vg0/test, when the hardsect_size of the PV /dev/sda5 is 512 bytes and the hardsect_size of the LV /dev/vg0/test is 1k? Does the kernel read 1k from LVM, modify 512 bytes and (later, when syncing) write 1k back to LVM? If so, the fact that hardsect_size of the LV is larger than the physical sector size could explain the performance impact. The kernel would have to modify only parts of a LV block, so (I assume) it would RMW. With hardsect_size == sector size == size of the write(2) call the kernel could probably just write to LVM without prior reading. But this would still not explain the performance degradation in my tests where I was *reading* from the LV. Could you explain the interaction of the kernel, LVM and the physical device a little bit or give a pointer to some text? > Try the following patch, which will align the PE structs to 64kB > boundaries from the start of the disk, as well as making the other > large VGDA structs start on 4k boundaries. Does it align to the start of the disk (as you write) or to the start of the PV? And is this a patch that will go into further releases anyway or is it just to find my reported performance problem? Since alignment is changed, I think I will have to recreate the PV, VG and LV with pvcreate, vgcreate and lvcreate, right? The patched tools and kernel would not work with my existing LV, would they? I don't expect this change in alignment to cause a change in performance. For testing, I have written my own very simple version of dd which can seek (not only skip) on the input file (I called it di since it understands only if= but not of=, it always writes to stdout). With this I have read from /dev/sda5 exactly the same sectors that /dev/vg0/test allocates (the first PE in vg0 starts at sector 4352 and /dev/vg0/test allocates PEs from vg0 contigously starting at PE no 0, see my last mail): # time ~urs/src/di bs=512 count=131072 if=/dev/sda5 >/dev/null 131072+0 real 0m9.345s user 0m0.180s sys 0m1.250s # time ~urs/src/di bs=512 count=131072 if=/dev/sda5 seek=4352 >/dev/null 131072+0 real 0m8.955s user 0m0.130s sys 0m1.570s # time ~urs/src/di bs=512 count=131072 if=/dev/vg0/test >/dev/null 131072+0 real 0m54.611s user 0m0.110s sys 0m2.480s Anyway, I will try your patch, probably today. I have not applied patches from the lvm-0.9... to the kernel before. AFAICS, I have to apply your patch to the lvm-0.9.1beta7 package, then ./configure; cd PATCHES; make and then apply lvm-0.9.1_beta7-2.4.3.patch to my kernel src, right? Also, does lvm-0.9.1_beta7-2.4.3.patch affect files other than lvm-mod.o, i.e. do I have to restart my kernel or only reload the LVM module? I have already made the lvm-0.9.1_beta7-2.4.3.patch file and see lvm.h is changed, but haven't already checked where it is included. urs ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-04-11 11:23 ` Urs Thuermann @ 2001-04-11 19:49 ` Andreas Dilger 2001-04-11 22:00 ` Terje Kvernes 2001-05-15 11:46 ` Urs Thuermann 0 siblings, 2 replies; 22+ messages in thread From: Andreas Dilger @ 2001-04-11 19:49 UTC (permalink / raw) To: linux-lvm Urs, you write: > > One thing that now comes to mind is if the LVM I/O is not aligned, > > because LVM PEs are aligned with the end of the PV and not the > > start, this _may_ cause performance problems because of misaligned > > read-modify-write causing extra seeking (purely speculation). > > I don't understand, what you're saying here (and above). I was > *reading* with dd if=/dev/vg0/test of=/dev/null. So why is there an > RMW involved? Sorry, too sleepy I guess. Obviously don't need read-modify-write when only reading pages. > Could you explain the interaction of the kernel, LVM and the physical > device a little bit or give a pointer to some text? Ok, the chain of events is as follows (hopefully correct): Non-LVM read LVM read ------------ -------- sys_read sys_read block_read block_read ll_rw_block ll_rw_block submit_bh submit_bh generic_make_request generic_make_request lvm_make_request_fn lvm_map __make_request __make_request The rest should be the same because you have the same IDE driver below. > > Try the following patch, which will align the PE structs to 64kB > > boundaries from the start of the disk, as well as making the other > > large VGDA structs start on 4k boundaries. > > Does it align to the start of the disk (as you write) or to the start > of the PV? You are correct, it aligns to the start of the PV, and not the disk. > And is this a patch that will go into further releases anyway or is it > just to find my reported performance problem? Since alignment is changed, > I think I will have to recreate the PV, VG and LV with pvcreate, vgcreate > and lvcreate, right? The patched tools and kernel would not work with > my existing LV, would they? I'm hoping this patch will make it into the stock LVM because it aligns all of the large VGDA structs to PAGE_SIZE (at least 4k pages), and the PE alignment matches MD RAID device alignment. Yes, you will need to re-create the PV and VG and LV for this to take effect. However, the kernel currently knows nothing at all about on-disk layout of VGDA data. Also, the tools all use the VGDA locations stored in the PV header (they are written at pvcreate and vgcreate time), so even if you create your VG/LV with this patch and it doesn't make it into the stock kernel, it is 100% compatible with unpatched tools. > I don't expect this change in alignment to cause a change in performance. Neither do I. Just something that I was thinking about when I had writing on my mind, not reading. > For testing, I have written my own very simple version of dd which can > seek (not only skip) on the input file (I called it di since it understands > only if= but not of=, it always writes to stdout). With this I have read > from /dev/sda5 exactly the same sectors that /dev/vg0/test allocates > (the first PE in vg0 starts at sector 4352 and /dev/vg0/test allocates > PEs from vg0 contigously starting at PE no 0, see my last mail): > > # time ~urs/src/di bs=512 count=131072 if=/dev/sda5 >/dev/null > 131072+0 > > real 0m9.345s > user 0m0.180s > sys 0m1.250s > # time ~urs/src/di bs=512 count=131072 if=/dev/sda5 seek=4352 >/dev/null > 131072+0 > > real 0m8.955s > user 0m0.130s > sys 0m1.570s > # time ~urs/src/di bs=512 count=131072 if=/dev/vg0/test >/dev/null > 131072+0 > > real 0m54.611s > user 0m0.110s > sys 0m2.480s Very strange. I just tested the same on my system, and LVM ends up _faster_ than raw disk access in each of the 3 tests I did: dd if=<dev> of=/dev/null bs=512 count=131072 LVM /dev/lvtest/vgtest: 11.099s, 11.393s, 10.785s IDE /dev/hda9: 12.073s, 11.822s, 11.843s This may have something to do with how my LV is set up. The first 5 PEs are actually closer to the end of the PV, and then the remaining PEs are contiguous from the start of the PV. > Anyway, I will try your patch, probably today. I have not applied > patches from the lvm-0.9... to the kernel before. AFAICS, I have to > apply your patch to the lvm-0.9.1beta7 package, then ./configure; cd > PATCHES; make and then apply lvm-0.9.1_beta7-2.4.3.patch to my kernel > src, right? Yes, but you really don't need to change the kernel code to use my patch. The VGDA alignment is purely a user-space issue. The kernel never does I/O to the VGDA - it is all read in user-space and passed to the kernel via ioctl. > Also, does lvm-0.9.1_beta7-2.4.3.patch affect files other than > lvm-mod.o, i.e. do I have to restart my kernel or only reload the LVM > module? Actually, I think you should be OK with only rebuilding the LVM module. However, you should be able to see from the patch itself if it affects anything outside lvm*. AFAIK, noting includes lvm.h in the kernel that would be affected. Cheers, Andreas -- Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto, \ would they cancel out, leaving him still hungry?" http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-04-11 19:49 ` Andreas Dilger @ 2001-04-11 22:00 ` Terje Kvernes 2001-05-15 11:46 ` Urs Thuermann 1 sibling, 0 replies; 22+ messages in thread From: Terje Kvernes @ 2001-04-11 22:00 UTC (permalink / raw) To: linux-lvm Andreas Dilger <adilger@turbolinux.com> writes: > Ok, the chain of events is as follows (hopefully correct): > > Non-LVM read LVM read > ------------ -------- > sys_read sys_read > block_read block_read > ll_rw_block ll_rw_block > submit_bh submit_bh > generic_make_request generic_make_request > lvm_make_request_fn > lvm_map > __make_request __make_request > > The rest should be the same because you have the same IDE driver > below. I'm going to stick my head in here and say thanks for that info. I really need to _read_ the code. *sigh* > Very strange. I just tested the same on my system, and LVM ends up > _faster_ than raw disk access in each of the 3 tests I did: okay, I only have the option of testing this on an old IDE drive, and the times where within 7% of eachother (LVM being the slower of the two on all runs). it's a bigger difference then I'd like to see, but it varies with runs, and the box is low on memory and being used at the same time. the only thing they're really saying is that LVM isn't a bottleneck. not that it'll help much in Urs' case though. :/ -- Terje ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-04-11 19:49 ` Andreas Dilger 2001-04-11 22:00 ` Terje Kvernes @ 2001-05-15 11:46 ` Urs Thuermann 2001-05-15 12:13 ` Patrick Caulfield 2001-05-15 17:08 ` [linux-lvm] LVM and *bad* performance (no striping) Andreas Dilger 1 sibling, 2 replies; 22+ messages in thread From: Urs Thuermann @ 2001-05-15 11:46 UTC (permalink / raw) To: linux-lvm Andreas Dilger <adilger@turbolinux.com> writes: > I'm hoping this patch will make it into the stock LVM because it aligns > all of the large VGDA structs to PAGE_SIZE (at least 4k pages), and the > PE alignment matches MD RAID device alignment. Sorry, I hadn't time to test your patch earlier. I wanted to try it yesterday but lvm-0.9.1-beta7 with your patch applied doesn't compile. There are undefined symbols $ make ... gcc -L../tools/lib -llvm-10 -o vgchange vgchange.o vgchange.o: In function `main': /home/urs/tmp/LVM/0.9.1_beta7-dilger/tools/vgchange.c:635: undefined reference to `LVM_PE_ON_DISK_BASE' /home/urs/tmp/LVM/0.9.1_beta7-dilger/tools/vgchange.c:638: undefined reference to `LVM_DISK_SIZE' When I have done my performance tests I had a stock linux-2.4.3 and user space tools from lvm-0.9 running. I now repeated the tests with lvm-0.9.1-beta7, both user space tools and kernel (recreated all PVs, VGs and LVs). However, the results with these versions are the same. I also did some tests with another hard disk drive. My previous tests were with a IBM DCAS-34330W SCSI-U2W. I now did some tests with a NEC DSE2100S SCSI2 drive. Again, LVM is much slower with the standard 512 byte block size: # pvcreate /dev/sdc4 pvcreate -- physical volume "/dev/sdc4" successfully created # vgcreate vg1 /dev/sdc4 vgcreate -- INFO: using default physical extent size 4 MB vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte vgcreate -- doing automatic backup of volume group "vg1" vgcreate -- volume group "vg1" successfully created and activated # lvcreate -n test vg1 -l31 lvcreate -- doing automatic backup of "vg1" lvcreate -- logical volume "/dev/vg1/test" successfully created # time dd if=/dev/sdc4 of=/dev/null 257040+0 records in 257040+0 records out real 0m37.205s user 0m0.840s sys 0m3.780s # time dd if=/dev/vg1/test of=/dev/null 253952+0 records in 253952+0 records out real 2m52.368s user 0m1.330s sys 0m12.390s # time dd if=/dev/vg1/test of=/dev/null bs=8k 15872+0 records in 15872+0 records out real 0m43.501s user 0m0.050s sys 0m4.600s # time dd if=/dev/vg1/test of=/dev/null bs=16k 7936+0 records in 7936+0 records out real 0m40.806s user 0m0.070s sys 0m4.290s # time dd if=/dev/vg1/test of=/dev/null bs=32k 3968+0 records in 3968+0 records out real 0m38.777s user 0m0.030s sys 0m4.290s What I find surprising is that the system time increases to 12.390s with LVM compared to 3.780s on /dev/sdc4. That is approx. 34ms of overhead per 512-block which I find a lot for the simple mapping of block numbers. But this still does not explain the much longer real elapsed time. Is there another patch (maybe to produce some debugging output) I could try to find the reason for this performnace hit? And am I really the only one who sees this? I would really be interested to hear from some more people about performance for their LVM. urs ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-05-15 11:46 ` Urs Thuermann @ 2001-05-15 12:13 ` Patrick Caulfield 2001-05-26 11:17 ` Urs Thuermann 2001-05-15 17:08 ` [linux-lvm] LVM and *bad* performance (no striping) Andreas Dilger 1 sibling, 1 reply; 22+ messages in thread From: Patrick Caulfield @ 2001-05-15 12:13 UTC (permalink / raw) To: linux-lvm On Tue, May 15, 2001 at 01:46:15PM +0200, Urs Thuermann wrote: > > And am I really the only one who sees this? I would really be > interested to hear from some more people about performance for their > LVM. I think one problem is that your "benchmarks" are not very realistic. In real world use the overhead of LVM is very small. I would respectfully suggest that you try something like bonnie++ before getting too worried about the performance of LVM on your machine - unless you spend most of your day doing "dd" to your disks :-) Here are some bonnie++ benchmarks of a disk partition and and LV (same disk) using reiserfs* on my machine: Disk Partition: Version 1.00g ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP tyke 496M 10462 98 20675 53 7699 20 11707 96 17808 16 257.0 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8426 100 +++++ +++ 12622 100 8031 100 +++++ +++ 9635 100 Logical Volume: Version 1.00g ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP tyke 496M 10577 100 20631 52 7739 20 11676 96 17622 17 260.7 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8521 100 +++++ +++ 12661 99 8007 100 +++++ +++ 9555 99 patrick * ext2 is comparable, just the numbers are different. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-05-15 12:13 ` Patrick Caulfield @ 2001-05-26 11:17 ` Urs Thuermann 2001-05-27 7:47 ` Andreas Dilger 0 siblings, 1 reply; 22+ messages in thread From: Urs Thuermann @ 2001-05-26 11:17 UTC (permalink / raw) To: linux-lvm Patrick Caulfield <caulfield@sistina.com> writes: > I think one problem is that your "benchmarks" are not very realistic. In > real world use the overhead of LVM is very small. I would respectfully suggest > that you try something like bonnie++ before getting too worried about the > performance of LVM on your machine - unless you spend most of your day doing > "dd" to your disks :-) The reason I tested dd on the partition and on the LV is that I thought when the performance of the block device is bad (like it is on my LV) this will have an impact on a mounted file system on that LV. If also thought, the performance of dd if=/dev/vg0/test will be approximately equal to dd if=/mnt/foo od md5sum foo where foo is large file on the LV. I've done some tests with a file system on /dev/sda5 and the on /dev/vg0/test (which is a LV on /dev/sda5) and I am really surprised: First, I repeated my previous tests: $dev /dev/sda5 /dev/vg0/test ---------------------------------------------+-------------+---------------- dd if=$dev of=/dev/null bs=512 count=1048576 | 71.2s | 122.7s dd if=$dev of=/dev/null bs=64k count=8192 | 71.5s | 71.7s Then I made a ext2 filesystem with 1K blocksize on /dev/sda5, mounted it to /mnt, copied a 512MB random file to it, md5sum'med that fie and dd'ed that file to /dev/null. Then I recreated /dev/vg0/test and did the same tests and that block device: ---------------------------------------------+-------------+---------------- cp -a /usr/local/foo /mnt | 96.6s | 123.4s md5sum /mnt/foo | 89.8s | 92.4s dd if=/mnt/bar of=/dev/null bs=512 | 71.8s | 72.1s dd if=/mnt/bar of=/dev/null bs=64k | 71.7s | 72.5s When I use a 4K blocksize on the ext2 filesystem the times for /dev/sda5 and /dev/vg0/test are even closer. Result: Although the block device /dev/vg0/test is significantly slower than /dev/sda5 (where /dev/vg0/test is located on /dev/sda5) the *read* performance of the file system on top the block device is roughly the same. This is probably not some effect of buffer caching since I did a blockdev --flushbufs $dev before each test, and /usr/local/bar is 512MB in size which is twice as large as the main memory size on that machine. However, writing to an LV is still much slower than to a normal partition. urs ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-05-26 11:17 ` Urs Thuermann @ 2001-05-27 7:47 ` Andreas Dilger 2001-05-27 9:47 ` Urs Thuermann 2001-05-27 17:01 ` [linux-lvm] LVM and *bad* performance (counterexample) idsfa 0 siblings, 2 replies; 22+ messages in thread From: Andreas Dilger @ 2001-05-27 7:47 UTC (permalink / raw) To: linux-lvm Urs writes: > First, I repeated my previous tests: > > $dev > > /dev/sda5 /dev/vg0/test > ---------------------------------------------+-------------+---------------- > dd if=$dev of=/dev/null bs=512 count=1048576 | 71.2s | 122.7s > dd if=$dev of=/dev/null bs=64k count=8192 | 71.5s | 71.7s OK, I have now reproduced this problem on my SMP test system. This is on a 2xP5-200 + 128MB RAM, Advansys ABP940U SCSI, and (old) IBM 0662-S12 and IBM 0663-L12 1GB SCSI drives. I do NOT have this problem on my laptop AFAICS (P2-500 + 128MB RAM, Intel PIIX4 IDE, IBM-DARA-218000). Doing similar tests to the above (dd if=$dev of=/dev/null, bs=512 count=100k) I get the following results (50MB I/O, 100MB LV, Linux 2.2.18 + LVM 0.9): /dev/sda = real 12.290s user 0.110s sys 2.890s /dev/sda = real 12.269s user 0.100s sys 2.900s /dev/sda1 = real 12.317s user 0.140s sys 2.980s /dev/sda1 = real 12.251s user 0.130s sys 2.900s /dev/vgtest/lvtest = real 66.909s user 0.440s sys 10.420 *** 5x as slow /dev/vgtest/lvtest = real 70.135s user 0.520s sys 11.390 *** 5x as slow /dev/sdb = real 11.614s user 0.060s sys 3.090s /dev/sdb = real 11.597s user 0.120s sys 2.910s /dev/sdb1 = real 11.857s user 0.150s sys 4.020s /dev/sdb1 = real 11.749s user 0.110s sys 2.800s /dev/vgtest3/lvtest = real 63.552s user 0.510s sys 10.290s *** 6x as slow /dev/vgtest3/lvtest = real 62.354s user 0.610s sys 10.020s *** 6x as slow /dev/hda6 = real 10.848s user 0.180s sys 4.380s /dev/hda6 = real 10.803s user 0.210s sys 4.300s /dev/vgtest2/lvtest = real 15.258s user 0.480s sys 9.500s *** 50% slower /dev/vgtest2/lvtest = real 15.827s user 0.360s sys 9.830s *** 50% slower I'm not sure if it is an SMP or SCSI issue, or what. Urs, do you also have an SMP system? Maybe there are locking slowdowns under SMP and not under UP? The problem is definitely more noticable on the SCSI disks, but at least on the IDE disk the increased wall-clock time is wholly accounted for by increased system time. On the SCSI disks, it would guess that "dd" is waiting for I/O completions under LVM, and not on the raw disk??? Cheers, Andreas -- Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto, \ would they cancel out, leaving him still hungry?" http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-05-27 7:47 ` Andreas Dilger @ 2001-05-27 9:47 ` Urs Thuermann 2001-05-27 17:01 ` [linux-lvm] LVM and *bad* performance (counterexample) idsfa 1 sibling, 0 replies; 22+ messages in thread From: Urs Thuermann @ 2001-05-27 9:47 UTC (permalink / raw) To: linux-lvm Andreas Dilger <adilger@turbolinux.com> writes: > /dev/vgtest/lvtest = real 66.909s user 0.440s sys 10.420 *** 5x as slow > /dev/vgtest/lvtest = real 70.135s user 0.520s sys 11.390 *** 5x as slow > > /dev/vgtest3/lvtest = real 63.552s user 0.510s sys 10.290s *** 6x as slow > /dev/vgtest3/lvtest = real 62.354s user 0.610s sys 10.020s *** 6x as slow This is similar to the worst case numbers I have measured. > I'm not sure if it is an SMP or SCSI issue, or what. Urs, do you also > have an SMP system? Maybe there are locking slowdowns under SMP and > not under UP? No. It's a single Pentium II, 333MHz, kernel 2.4.3 with lvm-0.9.1_beta7 compiled without SMP support: isnogud:urs$ grep SMP /usr/src/linux-2.4.3-ut/.config # CONFIG_SMP is not set The SCSI controller is an Adaptec AIC 7880 on an ASUS P2L97-S mainboard (onboard). The SCSI disks are Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: IBM Model: DCAS-34330W Rev: S65A Type: Direct-Access ANSI SCSI revision: 02 Host: scsi0 Channel: 00 Id: 01 Lun: 00 Vendor: IBM Model: DCAS-34330W Rev: S65A Type: Direct-Access ANSI SCSI revision: 02 Host: scsi0 Channel: 00 Id: 02 Lun: 00 Vendor: NEC Model: DSE2100S Rev: 0306 Type: Direct-Access ANSI SCSI revision: 02 i.e. /dev/sda and /dev/sdb are UW-SCSI and /dev/sdc is a SCSI-2 disks. > The problem is definitely more noticable on the SCSI disks, but at least > on the IDE disk the increased wall-clock time is wholly accounted for > by increased system time. On the SCSI disks, it would guess that "dd" > is waiting for I/O completions under LVM, and not on the raw disk??? How could I check? And do you have an idea, why a file system on top of such a slow LV is, if at all, only somewhat slower? urs ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (counterexample) 2001-05-27 7:47 ` Andreas Dilger 2001-05-27 9:47 ` Urs Thuermann @ 2001-05-27 17:01 ` idsfa 1 sibling, 0 replies; 22+ messages in thread From: idsfa @ 2001-05-27 17:01 UTC (permalink / raw) To: linux-lvm On Sun, May 27, 2001 at 01:47:18AM -0600, Andreas Dilger wrote: > I'm not sure if it is an SMP or SCSI issue, or what. Urs, do you also > have an SMP system? Maybe there are locking slowdowns under SMP and > not under UP? I haven't got a UP kernel to test right now to confirm, but my SMP purely IDE system (specs follow) shows rough parity at 512 byte reads and 2x FASTER on the LV at 4096 byte blocks. When I switch to a striped LV I see parity at both sizes, with the LV being slightly faster. For giggles, I also checked out a RAID-1 (mirror) set as well, which smoked everything, as it should. Device Block Real User Sys /dev/md0 512 2.080s 0.110s 0.660s /dev/hda1 512 5.079s 0.140s 2.040s /dev/mitethe/nostripe 512 4.795s 0.300s 2.380s /dev/mitethe/stripe 512 4.350s 0.090s 2.530s /dev/md0 4096 16.482s 0.060s 6.090s /dev/mitethe/nostripe 4096 26.278s 0.010s 7.270s /dev/mitethe/stripe 4096 32.803s 0.170s 12.020s /dev/hda1 4096 40.947s 0.180s 13.590s Now comes the part where I admit we're comparing apples to oranges: Linux 2.4.3 LVM 0.9.1_beta7 2x Pentium III 600MHz Tyan Tiger 133 Mobo (running at 133 FSB) Maxtor 53073U6 30GB ATA/66 hda WDC WD300AB-00BVA0 30GB ATA/66 hdc /dev/hda: multcount = 16 (on) I/O support = 1 (32-bit) unmaskirq = 1 (on) using_dma = 1 (on) keepsettings = 0 (off) nowerr = 0 (off) readonly = 0 (off) readahead = 8 (on) geometry = 59554/16/63, sectors = 60030432, start = 0 /dev/hdc: multcount = 16 (on) I/O support = 1 (32-bit) unmaskirq = 1 (on) using_dma = 1 (on) keepsettings = 0 (off) nowerr = 0 (off) readonly = 0 (off) readahead = 8 (on) geometry = 58168/16/63, sectors = 58633344, start = 0 -- $ fortune -m Kellen ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-05-15 11:46 ` Urs Thuermann 2001-05-15 12:13 ` Patrick Caulfield @ 2001-05-15 17:08 ` Andreas Dilger 2001-05-26 11:23 ` Urs Thuermann 1 sibling, 1 reply; 22+ messages in thread From: Andreas Dilger @ 2001-05-15 17:08 UTC (permalink / raw) To: linux-lvm Urs writes: > Sorry, I hadn't time to test your patch earlier. I wanted to try it > yesterday but lvm-0.9.1-beta7 with your patch applied doesn't compile. Sorry, I must have missed some parts of the patch for the user-space tools. Even so, I don't think the patch will help. > I also did some tests with another hard disk drive. My previous tests > were with a IBM DCAS-34330W SCSI-U2W. I now did some tests with a NEC > DSE2100S SCSI2 drive. Again, LVM is much slower with the standard > 512 byte block size: One other person had a similar problem. It was because they were using PIO mode instead of DMA mode on their (IDE) drive. However, I see you are using a SCSI disk, so I don't know what to suggest... What SCSI controller are you using? > And am I really the only one who sees this? I would really be > interested to hear from some more people about performance for their > LVM. I have never seen more than a few percent difference between LVM and non-LVM performance. Cheers, Andreas -- Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto, \ would they cancel out, leaving him still hungry?" http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-05-15 17:08 ` [linux-lvm] LVM and *bad* performance (no striping) Andreas Dilger @ 2001-05-26 11:23 ` Urs Thuermann 0 siblings, 0 replies; 22+ messages in thread From: Urs Thuermann @ 2001-05-26 11:23 UTC (permalink / raw) To: linux-lvm Andreas Dilger <adilger@turbolinux.com> writes: > What SCSI controller are you using? It is an Adaptec AIC 7880 which is onboard on my ASUS P2L97-S mainboard (see some previous posting where I did describe my hardware in more detail). Do you have some further suggestion what I could try to find the reason for this performance hit? urs ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-02-26 15:44 [linux-lvm] LVM and *bad* performance (no striping) Urs Thuermann 2001-02-26 20:39 ` Joe Thornber @ 2001-02-26 21:41 ` Rik van Riel 2001-02-27 10:33 ` Urs Thuermann 1 sibling, 1 reply; 22+ messages in thread From: Rik van Riel @ 2001-02-26 21:41 UTC (permalink / raw) To: linux-lvm On 26 Feb 2001, Urs Thuermann wrote: > isnogud:/root# time dd if=/dev/sda5 of=/dev/null count=100000 > 100000+0 records in > real 0m6.720s > isnogud:/root# time dd if=/dev/vg0/test of=/dev/null count=100000 > 100000+0 records in > real 0m41.851s > When making an ext2 filesystem on /dev/sda5 or /dev/vg0/test > respectivley, the performance of cp'ing, grep'ing all files and rm'ing > all files, seems to be comparable. But why is the performance of > reading the block device with dd so bad? Maybe dd is reading in 512 byte blocks on /dev/sda5 and 4096 bytes on /dev/vg0/test ? (I'm not sure if it is, but these blocksizes are the default blocksize on each device and you only told dd to "read 100000 blocks" ... this ratio also explains the time difference of 8 times) regards, Rik -- Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [linux-lvm] LVM and *bad* performance (no striping) 2001-02-26 21:41 ` Rik van Riel @ 2001-02-27 10:33 ` Urs Thuermann 0 siblings, 0 replies; 22+ messages in thread From: Urs Thuermann @ 2001-02-27 10:33 UTC (permalink / raw) To: linux-lvm Rik van Riel <riel@conectiva.com.br> writes: > Maybe dd is reading in 512 byte blocks on /dev/sda5 and > 4096 bytes on /dev/vg0/test ? I've checked it. The man page says if no bs is given, dd defaults to 512 bytes on any device. I've also checked it with dd|wc. The performance numbers I've observed seem to dependant on caching. urs ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2001-05-27 17:01 UTC | newest] Thread overview: 22+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2001-02-26 15:44 [linux-lvm] LVM and *bad* performance (no striping) Urs Thuermann 2001-02-26 20:39 ` Joe Thornber 2001-02-26 21:03 ` Jorg de Jong 2001-02-26 21:38 ` Rik van Riel 2001-02-27 8:35 ` Patrick Caulfield 2001-02-27 12:21 ` Urs Thuermann 2001-02-28 11:05 ` Heinz J. Mauelshagen 2001-04-10 19:30 ` Urs Thuermann 2001-04-10 21:37 ` Andreas Dilger 2001-04-11 11:23 ` Urs Thuermann 2001-04-11 19:49 ` Andreas Dilger 2001-04-11 22:00 ` Terje Kvernes 2001-05-15 11:46 ` Urs Thuermann 2001-05-15 12:13 ` Patrick Caulfield 2001-05-26 11:17 ` Urs Thuermann 2001-05-27 7:47 ` Andreas Dilger 2001-05-27 9:47 ` Urs Thuermann 2001-05-27 17:01 ` [linux-lvm] LVM and *bad* performance (counterexample) idsfa 2001-05-15 17:08 ` [linux-lvm] LVM and *bad* performance (no striping) Andreas Dilger 2001-05-26 11:23 ` Urs Thuermann 2001-02-26 21:41 ` Rik van Riel 2001-02-27 10:33 ` Urs Thuermann
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.