[linux-lvm] Performance problem.

[linux-lvm] Performance problem.

[neuron]

I've had lvm running on my server for quite a while now, not sure if this problem has always been there, or if it just occured.

If I through smb listen to music, and then on the server rm a file about 500mb in size, the music will stop.  Now this is just an example, it's like the entire filesystem stops while removing the file.  Running reiserfs ontop of lvm.  Also, when copying through ftp/smb from the server, I max on about 6mb/sec, which is far from the 12.1mb/sec avrange I've gotten out of that network card before (on a slower computer).

Any suggestions on how to improve performance?

(I realize more information is probably preferable, so just tell me what info you would need, as I'm not really skilled with lvm (= I also don't have anything fancy like a fake raid running)).

neuron

Re: [linux-lvm] Performance problem.

[Henric Andersson]

> If I through smb listen to music, and then on the server rm a file about
500mb in size, the music will stop.  Now this is just an example, it's like
the entire filesystem stops while removing the file.  Running reiserfs ontop
of lvm.  Also, when copying through ftp/smb from the server, I max on about
6mb/sec, which is far from the 12.1mb/sec avrange I've gotten out of that
network card before (on a slower computer).
>
> Any suggestions on how to improve performance?

I have the same problem and have not been able to improve the performance.
Even talked to someone on the list that had a raidsystem and he saw a 50%
loss of performance with LVM1 ... I'm going to try out LVM2 later when I get
the time to do some comparison, if you want, I could email you the results
when I've done it.

/Henric

Re: [linux-lvm] Performance problem.

[neuron]

<snip>
> I have the same problem and have not been able to improve the performance.
> Even talked to someone on the list that had a raidsystem and he saw a 50%
> loss of performance with LVM1 ... I'm going to try out LVM2 later when I get
> the time to do some comparison, if you want, I could email you the results
> when I've done it.

that'd be great.  I've been thinking about trying another kernel to see if it's a bug or something myself.  Haven't gotten around to it yet though.

neuron

Re: [linux-lvm] Performance problem.

[neuron]

nobody else has this problem?  Or any ideas how to debug it/fix it?

Re: [linux-lvm] Performance problem.

[Jean-louis MATTON]

what is your hardware config, mainly for disks ?
in the if I use LVM on top of ide disks some big file transfert, like 
ftp tar ..., the system hang  during the file manipulation.
when moving the same config on SCSI disk I never see this situation.

My config is soft raid1 + lvm + ext3 fs

JL

neuron wrote:

>I've had lvm running on my server for quite a while now, not sure if this problem has always been there, or if it just occured.
>
>If I through smb listen to music, and then on the server rm a file about 500mb in size, the music will stop.  Now this is just an example, it's like the entire filesystem stops while removing the file.  Running reiserfs ontop of lvm.  Also, when copying through ftp/smb from the server, I max on about 6mb/sec, which is far from the 12.1mb/sec avrange I've gotten out of that network card before (on a slower computer).
>
>Any suggestions on how to improve performance?
>
>(I realize more information is probably preferable, so just tell me what info you would need, as I'm not really skilled with lvm (= I also don't have anything fancy like a fake raid running)).
>
>neuron
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@sistina.com
>http://lists.sistina.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>  
>

-- 
Enthusiasm continues to make the difference !

*********************************************************************
Jean-Louis Matton                     Cliniques universitaires St LUC
tel        +32.2.764.37.13                     av. Hippocrate 10/ SIH
fax        +32.2.764.89.22                     B-1200 Bruxelles
Jean-Louis.Matton@clin.ucl.ac.be               BELGIUM
www.saintluc.be
*********************************************************************
                                          

Re: [linux-lvm] Performance problem.

[neuron]

2 120gb western digital 8mb cache, and one 80gb.

also got another 120gb here I'm gonna add.

500mhz athlon, 512mb memory.

gonna test 2.6 kernel now, will have to be compatibility with my old disks though, can't recreate the entire thing.

> what is your hardware config, mainly for disks ?
> in the if I use LVM on top of ide disks some big file transfert, like 
> ftp tar ..., the system hang  during the file manipulation.
> when moving the same config on SCSI disk I never see this situation.
> 
> My config is soft raid1 + lvm + ext3 fs
> 
> JL
> 
> neuron wrote:
> 
> >I've had lvm running on my server for quite a while now, not sure if this problem has always been there, or if it just occured.
> >
> >If I through smb listen to music, and then on the server rm a file about 500mb in size, the music will stop.  Now this is just an example, it's like the entire filesystem stops while removing the file.  Running reiserfs ontop of lvm.  Also, when copying through ftp/smb from the server, I max on about 6mb/sec, which is far from the 12.1mb/sec avrange I've gotten out of that network card before (on a slower computer).
> >
> >Any suggestions on how to improve performance?
> >
> >(I realize more information is probably preferable, so just tell me what info you would need, as I'm not really skilled with lvm (= I also don't have anything fancy like a fake raid running)).
> >
> >neuron
> >
> >_______________________________________________
> >linux-lvm mailing list
> >linux-lvm@sistina.com
> >http://lists.sistina.com/mailman/listinfo/linux-lvm
> >read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> >  
> >
> 
> -- 
> Enthusiasm continues to make the difference !
> 
> *********************************************************************
> Jean-Louis Matton                     Cliniques universitaires St LUC
> tel        +32.2.764.37.13                     av. Hippocrate 10/ SIH
> fax        +32.2.764.89.22                     B-1200 Bruxelles
> Jean-Louis.Matton@clin.ucl.ac.be               BELGIUM
> www.saintluc.be
> *********************************************************************
>                                           
> 
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

Re: [linux-lvm] Performance problem.

[Austin Gonyou]

Thought about trying TCQ for the IDE disks?

On Mon, 2003-11-17 at 10:34, Jean-louis MATTON wrote:
> what is your hardware config, mainly for disks ?
> in the if I use LVM on top of ide disks some big file transfert, like 
> ftp tar ..., the system hang  during the file manipulation.
> when moving the same config on SCSI disk I never see this situation.
> 
> My config is soft raid1 + lvm + ext3 fs
> 
> JL
> 
> neuron wrote:
> 
> >I've had lvm running on my server for quite a while now, not sure if
> this problem has always been there, or if it just occured.
> 
> >
> >If I through smb listen to music, and then on the server rm a file
> about 500mb in size, the music will stop.  Now this is just an
> example, it's like the entire filesystem stops while removing the
> file.  Running reiserfs ontop of lvm.  Also, when copying through
> ftp/smb from the server, I max on about 6mb/sec, which is far from the
> 12.1mb/sec avrange I've gotten out of that network card before (on a
> slower computer).
> 
> >
> >Any suggestions on how to improve performance?
> >
> >(I realize more information is probably preferable, so just tell me
> what info you would need, as I'm not really skilled with lvm (= I also
> don't have anything fancy like a fake raid running)).
> 
> >
> >neuron
> >
> >_______________________________________________
> >linux-lvm mailing list
> >linux-lvm@sistina.com
> >http://lists.sistina.com/mailman/listinfo/linux-lvm
> >read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> >  
> >
> 
> -- 
> Enthusiasm continues to make the difference !
> 
> *********************************************************************
> Jean-Louis Matton                     Cliniques universitaires St LUC
> tel        +32.2.764.37.13                     av. Hippocrate 10/ SIH
> fax        +32.2.764.89.22                     B-1200 Bruxelles
> Jean-Louis.Matton@clin.ucl.ac.be               BELGIUM
> www.saintluc.be
> *********************************************************************
>                                           
> 
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
-- 
Austin Gonyou <austin@coremetrics.com>
Coremetrics, Inc.

Re: [linux-lvm] Performance problem.

[neuron]

That's actually a very good idea, /me 's gonna test that before 2.6.

> Thought about trying TCQ for the IDE disks?
> 
> On Mon, 2003-11-17 at 10:34, Jean-louis MATTON wrote:
> > what is your hardware config, mainly for disks ?
> > in the if I use LVM on top of ide disks some big file transfert, like 
> > ftp tar ..., the system hang  during the file manipulation.
> > when moving the same config on SCSI disk I never see this situation.
> > 
> > My config is soft raid1 + lvm + ext3 fs
> > 
> > JL
> > 
> > neuron wrote:
> > 
> > >I've had lvm running on my server for quite a while now, not sure if
> > this problem has always been there, or if it just occured.
> > 
> > >
> > >If I through smb listen to music, and then on the server rm a file
> > about 500mb in size, the music will stop.  Now this is just an
> > example, it's like the entire filesystem stops while removing the
> > file.  Running reiserfs ontop of lvm.  Also, when copying through
> > ftp/smb from the server, I max on about 6mb/sec, which is far from the
> > 12.1mb/sec avrange I've gotten out of that network card before (on a
> > slower computer).
> > 
> > >
> > >Any suggestions on how to improve performance?
> > >
> > >(I realize more information is probably preferable, so just tell me
> > what info you would need, as I'm not really skilled with lvm (= I also
> > don't have anything fancy like a fake raid running)).
> > 
> > >
> > >neuron
> > >
> > >_______________________________________________
> > >linux-lvm mailing list
> > >linux-lvm@sistina.com
> > >http://lists.sistina.com/mailman/listinfo/linux-lvm
> > >read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> > >  
> > >
> > 
> > -- 
> > Enthusiasm continues to make the difference !
> > 
> > *********************************************************************
> > Jean-Louis Matton                     Cliniques universitaires St LUC
> > tel        +32.2.764.37.13                     av. Hippocrate 10/ SIH
> > fax        +32.2.764.89.22                     B-1200 Bruxelles
> > Jean-Louis.Matton@clin.ucl.ac.be               BELGIUM
> > www.saintluc.be
> > *********************************************************************
> >                                           
> > 
> > 
> > 
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> -- 
> Austin Gonyou <austin@coremetrics.com>
> Coremetrics, Inc.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

Re: [linux-lvm] Performance problem.

[Michael Paesold]

TCQ = tagged command queuing - for ide?

However, please report about success.

Also there is this tool called elvtune. It allowes you to "tune the I/O
elevator per blockdevice queue basis". See the man pages.

You could try to lower the read latency for the affected devices (/dev/hda,
/dev/hdb, etc.)

Regards,
Michael Paesold

neuron wrote:

> That's actually a very good idea, /me 's gonna test that before 2.6.
>
> > Thought about trying TCQ for the IDE disks?
> >
> > On Mon, 2003-11-17 at 10:34, Jean-louis MATTON wrote:
> > > what is your hardware config, mainly for disks ?
> > > in the if I use LVM on top of ide disks some big file transfert, like
> > > ftp tar ..., the system hang  during the file manipulation.
> > > when moving the same config on SCSI disk I never see this situation.
> > >
> > > My config is soft raid1 + lvm + ext3 fs
> > >
> > > JL

Re: [linux-lvm] Performance problem.

[neuron]

seems I was a bit quick on that reply, I remember hearing about TCQ a while ago.  Is it considered stable atm?  Noticed it wasn't even in my make menuconfig, which makes me very sceptical (it is unset in .config though, so the patches are there), running mm-sources, well, considering to run it.

> Thought about trying TCQ for the IDE disks?
> 
> On Mon, 2003-11-17 at 10:34, Jean-louis MATTON wrote:
> > what is your hardware config, mainly for disks ?
> > in the if I use LVM on top of ide disks some big file transfert, like 
> > ftp tar ..., the system hang  during the file manipulation.
> > when moving the same config on SCSI disk I never see this situation.
> > 
> > My config is soft raid1 + lvm + ext3 fs
> > 
> > JL
> > 
> > neuron wrote:
> > 
> > >I've had lvm running on my server for quite a while now, not sure if
> > this problem has always been there, or if it just occured.
> > 
> > >
> > >If I through smb listen to music, and then on the server rm a file
> > about 500mb in size, the music will stop.  Now this is just an
> > example, it's like the entire filesystem stops while removing the
> > file.  Running reiserfs ontop of lvm.  Also, when copying through
> > ftp/smb from the server, I max on about 6mb/sec, which is far from the
> > 12.1mb/sec avrange I've gotten out of that network card before (on a
> > slower computer).
> > 
> > >
> > >Any suggestions on how to improve performance?
> > >
> > >(I realize more information is probably preferable, so just tell me
> > what info you would need, as I'm not really skilled with lvm (= I also
> > don't have anything fancy like a fake raid running)).
> > 
> > >
> > >neuron
> > >
> > >_______________________________________________
> > >linux-lvm mailing list
> > >linux-lvm@sistina.com
> > >http://lists.sistina.com/mailman/listinfo/linux-lvm
> > >read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> > >  
> > >
> > 
> > -- 
> > Enthusiasm continues to make the difference !
> > 
> > *********************************************************************
> > Jean-Louis Matton                     Cliniques universitaires St LUC
> > tel        +32.2.764.37.13                     av. Hippocrate 10/ SIH
> > fax        +32.2.764.89.22                     B-1200 Bruxelles
> > Jean-Louis.Matton@clin.ucl.ac.be               BELGIUM
> > www.saintluc.be
> > *********************************************************************
> >                                           
> > 
> > 
> > 
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> -- 
> Austin Gonyou <austin@coremetrics.com>
> Coremetrics, Inc.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

Re: [linux-lvm] Performance problem.

[Spam]

 TCQ  is  in  2.6  kernels  though.  But not all IDE drives support it
 (Mainly  IBM/Hitachi  does).  Kernel  will  find out and enable it if
 drive  support  it.  I have not had any problems with it enabled, nor
 had it solved any harddisk related problems I have had.

 Howerver  unmask  IRQ  or  to  high  UDMA  transfer  mode have caused
 interruptions.  Usually if there is much electrical noise or you have
 bad  cabling  it  is enough to lower UDMA transfer mode to UDMA 66 or
 UDMA 33.

 (more comments below)
 

> seems I was a bit quick on that reply, I remember hearing about
> TCQ a while ago.  Is it considered stable atm?  Noticed it wasn't
> even in my make menuconfig, which makes me very sceptical (it is
> unset in .config though, so the patches are there), running
> mm-sources, well, considering to run it.

>> Thought about trying TCQ for the IDE disks?
>> 
>> On Mon, 2003-11-17 at 10:34, Jean-louis MATTON wrote:
>> > what is your hardware config, mainly for disks ?
>> > in the if I use LVM on top of ide disks some big file transfert, like
>> > ftp tar ..., the system hang  during the file manipulation.
>> > when moving the same config on SCSI disk I never see this situation.
>> > 
>> > My config is soft raid1 + lvm + ext3 fs
>> > 
>> > JL
>> > 
>> > neuron wrote:
>> > 
>> > >I've had lvm running on my server for quite a while now, not sure if
>> > this problem has always been there, or if it just occured.
>> > 
>> > >
>> > >If I through smb listen to music, and then on the server rm a file
>> > about 500mb in size, the music will stop.  Now this is just an
>> > example, it's like the entire filesystem stops while removing the
>> > file.  Running reiserfs ontop of lvm.  Also, when copying through
>> > ftp/smb from the server, I max on about 6mb/sec, which is far from the
>> > 12.1mb/sec avrange I've gotten out of that network card before (on a
>> > slower computer).

     Actually.  SMB  never gives optimal transfer speeds. On a HUB you
     can  expect  not  much more than 6-7MB/S on a 100mbit connection.
     With  a  Switch  you can get higher, up to 12.5MB/s over FTP (but
     never with SMB due to protocol overhead etc).

     Secondly,   SMB/Samba   is  not  very  nice  when  it  comes  to
     multitasking  from the same client. Doing several operations from
     same client can often cause temp interruptions to streaming audio
     etc...

>> > 
>> > >
>> > >Any suggestions on how to improve performance?
>> > >
>> > >(I realize more information is probably preferable, so just tell me
>> > what info you would need, as I'm not really skilled with lvm (= I also
>> > don't have anything fancy like a fake raid running)).
>> > 
>> > >
>> > >neuron
>> > >
>> > >_______________________________________________

Re: [linux-lvm] Performance problem.

[neuron]

>      Actually.  SMB  never gives optimal transfer speeds. On a HUB you
>      can  expect  not  much more than 6-7MB/S on a 100mbit connection.
>      With  a  Switch  you can get higher, up to 12.5MB/s over FTP (but
>      never with SMB due to protocol overhead etc).

yeah, I thought it was smb at first aswell, but I get the exact same speed over ftp.  And rm'ing a file (on ssh to the server) and playing a mp3 at the same time shouldn't make the mp3 skip 1-3 seconds should it?

And I've had a stable 11mb/sec out of smb before (tweaked).


> Also there is this tool called elvtune. It allowes you to "tune the I/O
> elevator per blockdevice queue basis". See the man pages.
> 
> You could try to lower the read latency for the affected devices (/dev/hda,
> /dev/hdb, etc.)

elvtune shows read_latency of 2048 and write_latency of 8192, that's already quite low isn't it?  If not, have any example values I can test with?

cat .config | grep -i tcq shows 
# CONFIG_BLK_DEV_IDE_TCQ is not set
should I manually set it and compile?  It's not in make_config.

also does anyone have a lvmcreate_initrd script for lvm2?

just did a bonnie++ test on the lvm volume.  Seems VERY slow doesen't it.

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
hollowtube       1G  3787  93 26948  62 10841  14  5885  90 21274  17 175.3   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  4991  98 +++++ +++  4135  90  4613  91 +++++ +++  3912  89
hollowtube,1G,3787,93,26948,62,10841,14,5885,90,21274,17,175.3,1,16,4991,98,+++++,+++,4135,90,4613,91,+++++,+++,3912,89

Re: [linux-lvm] Performance problem.

[Spam]

>>      Actually.  SMB  never gives optimal transfer speeds. On a HUB you
>>      can  expect  not  much more than 6-7MB/S on a 100mbit connection.
>>      With  a  Switch  you can get higher, up to 12.5MB/s over FTP (but
>>      never with SMB due to protocol overhead etc).

> yeah, I thought it was smb at first aswell, but I get the exact
> same speed over ftp.  And rm'ing a file (on ssh to the server) and
> playing a mp3 at the same time shouldn't make the mp3 skip 1-3
> seconds should it?

  No,  that  is  true.  Your low read speed does indicate that you are
  not  using  DMA.  Give  us  the  output of "hdparm /dev/hda". Also a
  simple  benchmark is to do "dd_rescue /dev/hda /dev/null". What does
  that give you in numbers?

> And I've had a stable 11mb/sec out of smb before (tweaked).


>> Also there is this tool called elvtune. It allowes you to "tune the I/O
>> elevator per blockdevice queue basis". See the man pages.
>> 
>> You could try to lower the read latency for the affected devices (/dev/hda,
>> /dev/hdb, etc.)

> elvtune shows read_latency of 2048 and write_latency of 8192,
> that's already quite low isn't it?  If not, have any example values
> I can test with?

> cat .config | grep -i tcq shows 
> # CONFIG_BLK_DEV_IDE_TCQ is not set
> should I manually set it and compile?  It's not in make_config.

  You can try. Though I suspect you will not gain much speed.

> also does anyone have a lvmcreate_initrd script for lvm2?

> just did a bonnie++ test on the lvm volume.  Seems VERY slow doesen't it.

> Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> hollowtube       1G  3787  93 26948  62 10841  14  5885  90 21274 17 175.3   1
>                     ------Sequential Create------ --------Random Create--------
>                     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>                  16  4991  98 +++++ +++  4135  90  4613  91 +++++ +++  3912  89
> hollowtube,1G,3787,93,26948,62,10841,14,5885,90,21274,17,175.3,1,16,4991,98,+++++,+++,4135,90,4613,91,+++++,+++,3912,89

Re: [linux-lvm] Performance problem.

[neuron]

>   No,  that  is  true.  Your low read speed does indicate that you are
>   not  using  DMA.  Give  us  the  output of "hdparm /dev/hda". Also a
>   simple  benchmark is to do "dd_rescue /dev/hda /dev/null". What does
>   that give you in numbers?

/dev/hdd:
 multcount    = 16 (on)
 IO_support   =  1 (32-bit)
 unmaskirq    =  0 (off)
 using_dma    =  1 (on)
 keepsettings =  0 (off)
 readonly     =  0 (off)
 readahead    =  8 (on)
 geometry     = 14589/255/63, sectors = 234375000, start = 0

the cables are new (I thought that they may be a problem), I've gotten far better speeds on the same system and a worse system before I used lvm on the same drives.  I REALLY doubt there's something wrong with the drives (I replaced one of them because of an error, and looked for trouble after replacing that one, but didn't find anything else).

And that dd_rescue will read from /dev/hda and write to /dev/null right?  Meaning it's a read test, and not a write test (= I won't loose any data on it ;) ).

Re: [linux-lvm] Performance problem.

[Spam]

>>   No,  that  is  true.  Your low read speed does indicate that you are
>>   not  using  DMA.  Give  us  the  output of "hdparm /dev/hda". Also a
>>   simple  benchmark is to do "dd_rescue /dev/hda /dev/null". What does
>>   that give you in numbers?

> /dev/hdd:
>  multcount    = 16 (on)
>  IO_support   =  1 (32-bit)
>  unmaskirq    =  0 (off)
>  using_dma    =  1 (on)
>  keepsettings =  0 (off)
>  readonly     =  0 (off)
>  readahead    =  8 (on)
>  geometry     = 14589/255/63, sectors = 234375000, start = 0

  This  looks good. You could try -c3 to enable 32-bit w/sync and also
  enable  Write  cache  -W1 (this may and many not be wise in case you
  have power failures often).

> the cables are new (I thought that they may be a problem), I've
> gotten far better speeds on the same system and a worse system
> before I used lvm on the same drives.  I REALLY doubt there's
> something wrong with the drives (I replaced one of them because of
> an error, and looked for trouble after replacing that one, but
> didn't find anything else).

> And that dd_rescue will read from /dev/hda and write to /dev/null
> right?  Meaning it's a read test, and not a write test (= I won't
> loose any data on it ;) ).

  Yes,  it  is a read-only test! Read the manpage to get all flags you
  can use with it (such as how big blocks it is to read etc).


> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Performance problem.

[neuron]

> dd_rescue /dev/hda /dev/null

Summary for /dev/hda -> /dev/null:rrxfer:         0.0k, succxfer:    230912.0k
dd_rescue: (info): ipos:    231168.0k, opos:    231168.0k, xferd:    231168.0k
                   errs:      0, errxfer:         0.0k, succxfer:    231168.0k
             +curr.rate:    26353kB/s, avg.rate:    25406kB/s, avg.load: 22.1%

Re: [linux-lvm] Performance problem.

[Spam]

>> dd_rescue /dev/hda /dev/null

> Summary for /dev/hda -> /dev/null:rrxfer:         0.0k, succxfer: 230912.0k
> dd_rescue: (info): ipos:    231168.0k, opos:    231168.0k, xferd: 231168.0k
>                    errs:      0, errxfer:         0.0k, succxfer: 231168.0k
>              +curr.rate:    26353kB/s, avg.rate:    25406kB/s, avg.load: 22.1%


  Then you know the read speed with your drives are not a problem. You
  could  try  the  same with /dev/vg/lv instead of your hda and see if
  LVM is slowing down much.

Re: [linux-lvm] Performance problem.

[neuron]

dd_rescue /dev/vg_main/storagelv /dev/null

dd_rescue: (info): ipos:    138384.0k, opos:    138384.0k, xferd:    138384.0k
                   errs:      0, errxfer:         0.0k, succxfer:    138384.0k
             +curr.rate:    27966kB/s, avg.rate:    27553kB/s, avg.load: 20.9%

> >> dd_rescue /dev/hda /dev/null
> 
> > Summary for /dev/hda -> /dev/null:rrxfer:         0.0k, succxfer: 230912.0k
> > dd_rescue: (info): ipos:    231168.0k, opos:    231168.0k, xferd: 231168.0k
> >                    errs:      0, errxfer:         0.0k, succxfer: 231168.0k
> >              +curr.rate:    26353kB/s, avg.rate:    25406kB/s, avg.load: 22.1%
> 
> 

Re: [linux-lvm] Performance problem.

[Henric Andersson]

> dd_rescue /dev/vg_main/storagelv /dev/null
>
> dd_rescue: (info): ipos:    138384.0k, opos:    138384.0k, xferd:
138384.0k
>                    errs:      0, errxfer:         0.0k, succxfer:
138384.0k
>              +curr.rate:    27966kB/s, avg.rate:    27553kB/s, avg.load:
20.9%

This proves nothing I'm afraid. I've done similar tests myself and even
gotten bonnie to produce results that "look good", but these are all
synthetic benchmarks. What counts is realworld performance. I tested bonnie
(locally on lvm and regular partitions) and regular filecopy with time
(using a 512MB file) between LVM to the system drive. Also tried NFS
compared to Samba and it made no difference. FTP proved the same. At the
beginning of all the test where I could see the actual speed, it would be
right on spot (ie, maxing out my 100mbit switched network) but after 10
seconds or so, the speed would drop dramatically to 2-4mb/s (from ~9mb/s)
and then even halt completely for a short while and then resuming at 2-4mb/s
(and this with no other network/disk access on the server). Oh, and to
eliminate the destination drive completely during tests, I also tried "time
cat big512mbfile > /dev/null", which, ofcourse, resulted in the same speed
as every other test I did.

This is starting to become a pain. For example, I'm unable to burn my backup
tarfiles to DVD over the network without empting the buffer (thank good for
powerburn) which is ridiculous since all it requires is a consistant 4mb/s
or so. I'm going to try the devicemapper in 2.6 (LVM2), but I haven't had
the time to do it yet (have to create a similar setup since I dont want to
risk the data for a little test).

Oh, and before you ask, the server is a Celeron@2Ghz, with 512MB memory, all
filesystems are ext3, all drives are master and use the correct ATA/133
cable. All drives use dma and 32bit with sync transfers. The server's sole
purpose in life is to provide storage for my network. It uses an nice Intel
Pro/100 card to ensure stability and performance. The network is 100mbit
full duplex switched to allow max speed. When transfering files from a
normal partition (ie, /dev/hdax) I get about 9-9.5mb/s with samba and
Win2kPro (measured with the Kerio Firewall on the Win2k and IpTraf locally
on the fileserver).

(The serverhardware was purchased since I figured that my previous 120Mhz
Pentium Classic was the bottleneck  :-/ )

/Henric

Re: [linux-lvm] Performance problem.

[Spam]

  Hm.  You  seem  to  have  quite the problem :( Could it be that some
  logging  is  turned  on  and spewing data as soon as you copy larger
  files  etc? Have you looked on CPU usage while copying, or looked on
  IRQ assignments (NIC might share with IDE controller)?

  Could  also  be NIC that is not using a good driver? I have had that
  before  both  with  Intel  and  RealTek  cards.  Actually,  when you
  mention it there are two Intel Pro 100 drivers in the kernel (Donald
  Becker and Intel original I think). Try switching between the two.

  Did you try Samba or FTP locally? If that works good then it is most
  likely something to do with the networking parts of your system.


>> dd_rescue /dev/vg_main/storagelv /dev/null
>>
>> dd_rescue: (info): ipos:    138384.0k, opos:    138384.0k, xferd:
> 138384.0k
>>                    errs:      0, errxfer:         0.0k, succxfer:
> 138384.0k
>>              +curr.rate:    27966kB/s, avg.rate:    27553kB/s, avg.load:
> 20.9%

> This proves nothing I'm afraid. I've done similar tests myself and even
> gotten bonnie to produce results that "look good", but these are all
> synthetic benchmarks. What counts is realworld performance. I tested bonnie
> (locally on lvm and regular partitions) and regular filecopy with time
> (using a 512MB file) between LVM to the system drive. Also tried NFS
> compared to Samba and it made no difference. FTP proved the same. At the
> beginning of all the test where I could see the actual speed, it would be
> right on spot (ie, maxing out my 100mbit switched network) but after 10
> seconds or so, the speed would drop dramatically to 2-4mb/s (from ~9mb/s)
> and then even halt completely for a short while and then resuming at 2-4mb/s
> (and this with no other network/disk access on the server). Oh, and to
> eliminate the destination drive completely during tests, I also tried "time
cat big512mbfile >> /dev/null", which, ofcourse,
cat big512mbfile >> resulted in the same speed
> as every other test I did.

> This is starting to become a pain. For example, I'm unable to burn my backup
> tarfiles to DVD over the network without empting the buffer (thank good for
> powerburn) which is ridiculous since all it requires is a consistant 4mb/s
> or so. I'm going to try the devicemapper in 2.6 (LVM2), but I haven't had
> the time to do it yet (have to create a similar setup since I dont want to
> risk the data for a little test).

> Oh, and before you ask, the server is a Celeron@2Ghz, with 512MB memory, all
> filesystems are ext3, all drives are master and use the correct ATA/133
> cable. All drives use dma and 32bit with sync transfers. The server's sole
> purpose in life is to provide storage for my network. It uses an nice Intel
> Pro/100 card to ensure stability and performance. The network is 100mbit
> full duplex switched to allow max speed. When transfering files from a
> normal partition (ie, /dev/hdax) I get about 9-9.5mb/s with samba and
> Win2kPro (measured with the Kerio Firewall on the Win2k and IpTraf locally
> on the fileserver).

> (The serverhardware was purchased since I figured that my previous 120Mhz
> Pentium Classic was the bottleneck  :-/ )

> /Henric


> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Performance problem.

[neuron]

ftp to localhost
starts on 10.2mb/sec, goes stable for a short while, then stops for a few seconds (4-5), then continues in about 5-6 until the transfer is complete.

all my stuff are on seperate irq's.

when I copy, "system" load skyrockets, 60-85%.
lftp and glftpd is the processes using resources.  I could try smb aswell, but I'm guessing I'll find that eating my system resources aswell.

> 
>   Hm.  You  seem  to  have  quite the problem :( Could it be that some
>   logging  is  turned  on  and spewing data as soon as you copy larger
>   files  etc? Have you looked on CPU usage while copying, or looked on
>   IRQ assignments (NIC might share with IDE controller)?
> 
>   Could  also  be NIC that is not using a good driver? I have had that
>   before  both  with  Intel  and  RealTek  cards.  Actually,  when you
>   mention it there are two Intel Pro 100 drivers in the kernel (Donald
>   Becker and Intel original I think). Try switching between the two.
> 
>   Did you try Samba or FTP locally? If that works good then it is most
>   likely something to do with the networking parts of your system.
>

Re: [linux-lvm] Performance problem.

[Spam]

  I am afraid I do not have much else to suggest other than trying the
  new  2.6.0-test9  kernel  with device mapper. I am running it and so
  far it is very stable (and quick).


> ftp to localhost
> starts on 10.2mb/sec, goes stable for a short while, then stops
> for a few seconds (4-5), then continues in about 5-6 until the
> transfer is complete.

> all my stuff are on seperate irq's.

> when I copy, "system" load skyrockets, 60-85%.
> lftp and glftpd is the processes using resources.  I could try
> smb aswell, but I'm guessing I'll find that eating my system
> resources aswell.

>> 
>>   Hm.  You  seem  to  have  quite the problem :( Could it be that some
>>   logging  is  turned  on  and spewing data as soon as you copy larger
>>   files  etc? Have you looked on CPU usage while copying, or looked on
>>   IRQ assignments (NIC might share with IDE controller)?
>> 
>>   Could  also  be NIC that is not using a good driver? I have had that
>>   before  both  with  Intel  and  RealTek  cards.  Actually,  when you
>>   mention it there are two Intel Pro 100 drivers in the kernel (Donald
>>   Becker and Intel original I think). Try switching between the two.
>> 
>>   Did you try Samba or FTP locally? If that works good then it is most
>>   likely something to do with the networking parts of your system.
>>

> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Performance problem.

[neuron]

> 
>   I am afraid I do not have much else to suggest other than trying the
>   new  2.6.0-test9  kernel  with device mapper. I am running it and so
>   far it is very stable (and quick).
> 

yeah, I run that on my desktop box, and I'm very happy with it.  I was gonna try it earlier, but didn't really have time.
Henric Andersson, if you have the same problems and have the time, let me know the results if you test.

Re: [linux-lvm] Performance problem.

[Henric Andersson]

neuron wrote:
> >   I am afraid I do not have much else to suggest other than trying the
> >   new  2.6.0-test9  kernel  with device mapper. I am running it and so
> >   far it is very stable (and quick).
> >
>
> yeah, I run that on my desktop box, and I'm very happy with it.  I was
gonna try it earlier, but didn't really have time.
> Henric Andersson, if you have the same problems and have the time, let me
know the results if you test.

Was able to borrow a disk from work, so I'm going to make a duplicate of my
system disk and setup a new LVM on two other disks. Then I'm going to
upgrade it to 2.6.0-test9 with devicemapper.

This way, I can learn from my mistakes without loosing my current setup, it
should also give a better idea of how my new setup will perform once I do it
on the real system. Going to do some benchmarking before and after I update
to see what kind of performance difference to expect (if any).

Any tips, ideas or experience anyone want to share before I jump into the
upgrade frenzymode later this evening?

/Henric

Re: [linux-lvm] Performance problem.

[neuron]

great, dunno about testing.  Copying large amounts of data atleast.  Also fragmentation is an issue to keep in mind, so duplicate the disk, don't cp the data over.

If you don't get any performance increase, and got the time, you could also create a lvm2 setup, not using compatibility, and check if that's a considerable performance increase.  Though it would be quite a lot of work for me to move the data to lvm2, I would if I knew it would fix my problems.
 
> Was able to borrow a disk from work, so I'm going to make a duplicate of my
> system disk and setup a new LVM on two other disks. Then I'm going to
> upgrade it to 2.6.0-test9 with devicemapper.
> 
> This way, I can learn from my mistakes without loosing my current setup, it
> should also give a better idea of how my new setup will perform once I do it
> on the real system. Going to do some benchmarking before and after I update
> to see what kind of performance difference to expect (if any).
> 
> Any tips, ideas or experience anyone want to share before I jump into the
> upgrade frenzymode later this evening?
> 
> /Henric
> 

Re: [linux-lvm] Performance problem.

[Henric Andersson]

> great, dunno about testing.
> Copying large amounts of data atleast.
> Also fragmentation is an issue to keep in mind, so duplicate the disk,
don't cp the data over.

The idea was to use dd to duplicate the system disk, then create a LV (LVM1
metadata) outof two new disks which are then filled with data. After that,
I'll begin migrating to devicemapper but with lvm1 metadata.

All tests will in the beginning be read tests (since that's more important
to me). These tests will be "time cat > /dev/null" aswell as accessing the
data remotely via NFS/SMB/FTP. After that test, I'll migrate to LVM2
metadata, since thats what I want to use for real later, and then redo the
tests using the same data.

I assume that it will be possible to migrate the metadata to LVM2?

It's going to be a long evening/night :)

/Henric

Re: [linux-lvm] Performance problem.

[neuron]

no idea if you can convert the data, but as you can resize, you can probably move stuff to lvm2 that way.

> > great, dunno about testing.
> > Copying large amounts of data atleast.
> > Also fragmentation is an issue to keep in mind, so duplicate the disk,
> don't cp the data over.
> 
> The idea was to use dd to duplicate the system disk, then create a LV (LVM1
> metadata) outof two new disks which are then filled with data. After that,
> I'll begin migrating to devicemapper but with lvm1 metadata.
> 
> All tests will in the beginning be read tests (since that's more important
> to me). These tests will be "time cat > /dev/null" aswell as accessing the
> data remotely via NFS/SMB/FTP. After that test, I'll migrate to LVM2
> metadata, since thats what I want to use for real later, and then redo the
> tests using the same data.
> 
> I assume that it will be possible to migrate the metadata to LVM2?
> 
> It's going to be a long evening/night :)
> 
> /Henric
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

Re: [linux-lvm] Performance problem.

[Henric Andersson]

Managed to do some tests, but it's somewhat bewildering. I'm quiting for the
night, but feel free to look at my webpage for how it went.

Go to http://www.sensenet.nu/~ha/lvmtest/

/Henric

Re: [linux-lvm] Performance problem.

[neuron]

hmm... could it be the reboot solved it?  Hasn't for me, you could try running the box for a while, then testing, to see if it needs a reboot every now and then or something like that.

"(unless the moving and jerking of cables somehow fixed a problem I didn't know of)"
could be noise on the cables I suppose, though I doubt that, especially considering I'm experiencing the same thing, and I've replaced the cables fairly recently.

> Managed to do some tests, but it's somewhat bewildering. I'm quiting for the
> night, but feel free to look at my webpage for how it went.
> 
> Go to http://www.sensenet.nu/~ha/lvmtest/
> 
> /Henric
> 

Re: [linux-lvm] Performance problem.

[Henric Andersson]

On Wednesday 19 November 2003 01.01, neuron wrote:
> hmm... could it be the reboot solved it?  

Don't think so, I've rebooted before (poweroutage, etc) and that didn't do 
anything for performance.

> Hasn't for me, you could try
> running the box for a while, then testing, to see if it needs a reboot
> every now and then or something like that.

I'm going to do just that. It really bothers me that this happened. Sure, it's 
nice to see that its fast, but I really want to know WHY :)

> "(unless the moving and jerking of cables somehow fixed a problem I didn't
> know of)" could be noise on the cables I suppose, though I doubt that,
> especially considering I'm experiencing the same thing, and I've replaced
> the cables fairly recently.

Exactly ... so I think we can rule thatone out too. One thing that might be 
the reason behind this improvement on performance is what version of the 
software one is using. But I doubt that I would see a 10x improvement because 
of any updates. Ohwell, I'll try and run one test per week and see if the 
figures change.

/Henric

Re: [linux-lvm] Performance problem.

[Michael Paesold]

neuron wrote:

> elvtune shows read_latency of 2048 and write_latency of 8192,
> that's already quite low isn't it?  If not, have any example
> values I can test with?

Those are the default values for RedHat 8.0, and probably some other
systems. For RedHat 9.0 the defaults are:

/dev/cciss/c0d0 elevator ID             2
        read_latency:           64
        write_latency:          8192
        max_bomb_segments:      6

You could try 64 or even 32 and do your "real-world test" (listening to
MP3...).

Regards,
Michael Paesold

Re: [linux-lvm] Performance problem.

[Steven Lembark]

-- Jean-louis MATTON <Jean-Louis.Matton@clin.ucl.ac.be>

> what is your hardware config, mainly for disks ?
> in the if I use LVM on top of ide disks some big file transfert, like ftp
> tar ..., the system hang  during the file manipulation.
> when moving the same config on SCSI disk I never see this situation.
>
> My config is soft raid1 + lvm + ext3 fs

Have you had the same problem with ext3 on non-LVM/RAID
systems?


--
Steven Lembark                               2930 W. Palmer
Workhorse Computing                       Chicago, IL 60647
                                            +1 888 359 3508