[linux-lvm] Poor performance with LVM 1.0.7

[linux-lvm] Poor performance with LVM 1.0.7

[Henric Andersson]

Hi all!

I've recently upgraded my poor little fileserver from an old P120 Classic to
a brandspanking new Celeron 2Ghz with 512MB DDR memory. By doing this, I
expected the performance of my fileserver to increase, giving me a total of
9,5MB/s when using samba.

And it did. But not when LVM is concerned :(

I have a VG setup with 3 drives (120GB each, not striped) which contains one
LV running an ext3 filesystem.
At first, I blamed my promise controllers (PDC 20269, ATA/133) but after
some serious testing, I found that whenever I didn't access files that was
stored on the LV, my speed was about 9.5MB/s, but when I transfered the same
file from the LV, I only got about half of it (4.2MB/s). The file I
transfered was roughly about 700-800MB big (same file used in all tests).

After scouring the net for suggestions on how to improve the speed, I came
up empty, and thus, I'm writing this email :)

Some more facts (more available if needed ;)) about the server:

All drives are tuned with 32bit transfers and DMA enabled. I'm running
kernel 2.4.22pre2 (gentoo).
CPU usage is low (around 10-25%) when transfering files, so it does not seem
to be the bottleneck.

Any suggestions and/or ideas are welcomed...

/Henric

Re: [linux-lvm] Poor performance with LVM 1.0.7

[Herbert Pötzl]

On Tue, Aug 26, 2003 at 07:45:54PM +0200, Henric Andersson wrote:
> Hi all!
> 
> I've recently upgraded my poor little fileserver from an old P120 Classic to
> a brandspanking new Celeron 2Ghz with 512MB DDR memory. By doing this, I
> expected the performance of my fileserver to increase, giving me a total of
> 9,5MB/s when using samba.
> 
> And it did. But not when LVM is concerned :(
> 
> I have a VG setup with 3 drives (120GB each, not striped) which contains one
> LV running an ext3 filesystem.
> At first, I blamed my promise controllers (PDC 20269, ATA/133) but after
> some serious testing, I found that whenever I didn't access files that was
> stored on the LV, my speed was about 9.5MB/s, but when I transfered the same
> file from the LV, I only got about half of it (4.2MB/s). The file I
> transfered was roughly about 700-800MB big (same file used in all tests).

check the raw performance of one drive involved
with hdparm -tT ...

> After scouring the net for suggestions on how to improve the speed, I came
> up empty, and thus, I'm writing this email :)
> 
> Some more facts (more available if needed ;)) about the server:
> 
> All drives are tuned with 32bit transfers and DMA enabled. I'm running
> kernel 2.4.22pre2 (gentoo).

just for the fun of doing it, compile vanilla
2.4.22 and give it a try ... you could also add
the following lvm patch ...
http://www.13thfloor.at/VServer/patches-2.4.22-p10c17/07_lvm-1.0.7-2.4.21pre5.patch.bz2

> CPU usage is low (around 10-25%) when transfering files, so it does not seem
> to be the bottleneck.
> 
> Any suggestions and/or ideas are welcomed...

was the performance you had on your old box higher?
or did you just expect an increase in speed, which didn't
happen?

best,
Herbert


> /Henric
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Poor performance with LVM 1.0.7

[Henric Andersson]

> check the raw performance of one drive involved
> with hdparm -tT ...

Did that, it reported between 27-45MB/s on all drives (ranging from the system harddrive to the LVM
harddrives)

> > After scouring the net for suggestions on how to improve the speed, I came
> > up empty, and thus, I'm writing this email :)
> >
> > Some more facts (more available if needed ;)) about the server:
> >
> > All drives are tuned with 32bit transfers and DMA enabled. I'm running
> > kernel 2.4.22pre2 (gentoo).
>
> just for the fun of doing it, compile vanilla
> 2.4.22 and give it a try ... you could also add

Oki, will try it as soon as I get home.

> the following lvm patch ...
> http://www.13thfloor.at/VServer/patches-2.4.22-p10c17/07_lvm-1.0.7-2.4.21pre5.patch.bz2

Hmm? What does this patch do?

> was the performance you had on your old box higher?
> or did you just expect an increase in speed, which didn't
> happen?

Performance on my old box was lower, so I expected it to increase with my new system. And it did, as
long as I don't serve any files from the volumeset, everything runs just fine (9.5MB/s) but whenever I
try to access files on the LV, I get 4-5MB/s ...

Even tried doing:

time cat "somefile on the LV set" > /dev/null

Which performed equally bad (ie, about 4-5MB/s). Doing the same on the /dev/hda1 drive gives ALOT
better performance (more in the line of what hdparm -Tt reports)

/Henric

Re: [linux-lvm] Poor performance with LVM 1.0.7

[Herbert Pötzl]

On Wed, Aug 27, 2003 at 02:05:55PM +0200, Henric Andersson wrote:
> > check the raw performance of one drive involved
> > with hdparm -tT ...
> 
> Did that, it reported between 27-45MB/s on all drives (ranging from the system harddrive to the LVM
> harddrives)

I asked this, because I reported several times that
lvm does have some I/O bottleneck/issue ...

I had a similar problem with RAID5 where I could
get 16Mb/s from each disk and about 30Mb/s from
the RAID5 (4 disks) but only 18Mb/s from the lvm
ontop of that raid ...

> > > After scouring the net for suggestions on how to improve the speed, I came
> > > up empty, and thus, I'm writing this email :)
> > >
> > > Some more facts (more available if needed ;)) about the server:
> > >
> > > All drives are tuned with 32bit transfers and DMA enabled. I'm running
> > > kernel 2.4.22pre2 (gentoo).
> >
> > just for the fun of doing it, compile vanilla
> > 2.4.22 and give it a try ... you could also add
> 
> Oki, will try it as soon as I get home.
> 
> > the following lvm patch ...
> > http://www.13thfloor.at/VServer/patches-2.4.22-p10c17/07_lvm-1.0.7-2.4.21pre5.patch.bz2
> 
> Hmm? What does this patch do?

get lvm up to version 1.0.7 ...

> > was the performance you had on your old box higher?
> > or did you just expect an increase in speed, which didn't
> > happen?
> 
> Performance on my old box was lower, so I expected it to 
> increase with my new system. And it did, as
> long as I don't serve any files from the volumeset, 
> everything runs just fine (9.5MB/s) but whenever I
> try to access files on the LV, I get 4-5MB/s ...
> 
> Even tried doing:
> 
> time cat "somefile on the LV set" > /dev/null
> 
> Which performed equally bad (ie, about 4-5MB/s). 
> Doing the same on the /dev/hda1 drive gives ALOT
> better performance (more in the line of what hdparm -Tt reports)

this was, on one of my systems, the reason why
I removed the lvm and used the plain disk ...

Heinz suggested to use the new LVM2 anyway, and maybe
this is an option for you, so you could try this too 
(if you need patches for LVM2, hopefully they will be 
available for 2.4.22 this evening ...)

HTH,
Herbert

> /Henric

Re: [linux-lvm] Poor performance with LVM 1.0.7

[Henric Andersson]

> I asked this, because I reported several times that
> lvm does have some I/O bottleneck/issue ...
>
> I had a similar problem with RAID5 where I could
> get 16Mb/s from each disk and about 30Mb/s from
> the RAID5 (4 disks) but only 18Mb/s from the lvm
> ontop of that raid ...

I've seen some evidence of that too on the net, but noone seems to have any answer.

Did a test now with bonnie on the LV setup with a 512MB file, and this is the result:

-------Sequential Output--------
Char: 17159K/s (97% CPU)
Block: 21966K/s (7% CPU)
Rewrite: 21052K/s (8% CPU)

---Sequential Input--
Char: 10879K/s (54% CPU)
Block: 18166K/s (2% CPU)

So, clearly, this should be enough to feed samba the 9.5MB/s that all the other non-lvm disks can do.
But apparently not. Is there perhaps some strange configuration of samba with LVM that actually makes
it slower? (Due to buffers, and read sizes?)

> > Even tried doing:
> >
> > time cat "somefile on the LV set" > /dev/null
> >
> > Which performed equally bad (ie, about 4-5MB/s).
> > Doing the same on the /dev/hda1 drive gives ALOT
> > better performance (more in the line of what hdparm -Tt reports)
>
> this was, on one of my systems, the reason why
> I removed the lvm and used the plain disk ...

No good in my case, I need to be able to extend the filesystem and thus, I need LVM.

> Heinz suggested to use the new LVM2 anyway, and maybe
> this is an option for you, so you could try this too
> (if you need patches for LVM2, hopefully they will be
> available for 2.4.22 this evening ...)

Isn't LVM2 just a new bunch of userland tools (+ a devicemapper) ? Or did I miss something?

/Henric