[linux-lvm] cLVM commands often taking a long time to execute

[linux-lvm] cLVM commands often taking a long time to execute

[Riches Jr, Robert M]

[-- Attachment #1: Type: text/plain, Size: 844 bytes --]

I'm running into some odd behavior from clustered LVM commands on a 3-node pacemaker-based cluster.  When other processes on the nodes are doing somewhat heavy sequential I/O traffic, LVM commands such as lvs, pvs, vgs, lvdisplay, pvdisplay, and vgdisplay have widely variable, often very long execution times.  The same LVM command done repetitively will have execution times varying from less than a second to 10-15 seconds, more rarely up to around 60 seconds.  Other read-heavy commands such as find, wc, and sum have much less variability and much shorter tails in the distribution of their execution times.

Is it known what causes the cLVM commands to be so slow at times?

Perchance, are there known ways to get cLVM commands to be less slow?

Or, is this just par for the course for clustered LVM?

Thanks,

Robert Riches


[-- Attachment #2: Type: text/html, Size: 2873 bytes --]

Re: [linux-lvm] cLVM commands often taking a long time to execute

[Vladislav Bogdanov]

09.11.2011 03:06, Riches Jr, Robert M wrote:
> I�m running into some odd behavior from clustered LVM commands on a
> 3-node pacemaker-based cluster.  When other processes on the nodes are
> doing somewhat heavy sequential I/O traffic, LVM commands such as lvs,
> pvs, vgs, lvdisplay, pvdisplay, and vgdisplay have widely variable,
> often very long execution times.  The same LVM command done repetitively
> will have execution times varying from less than a second to 10-15
> seconds, more rarely up to around 60 seconds.  Other read-heavy commands
> such as find, wc, and sum have much less variability and much shorter
> tails in the distribution of their execution times.
> 
> Is it known what causes the cLVM commands to be so slow at times?
> 
> Perchance, are there known ways to get cLVM commands to be less slow?

I use chrt -r 99 for them. And I also run clvmd the same way. But this
is much more about CPU load then IO load.
Also, please make sure that you correctly filter unneeded block devices
from being PV-suspects (filter clause in lvm.conf).

Best,
Vladislav

Re: [linux-lvm] cLVM commands often taking a long time to execute

[Riches Jr, Robert M]

Thank you, Vladislav, for the suggestion.  That could potentially help in cases where the nodes are CPU-bound, likely independent of whether they are I/O-bound.

Thanks,

Robert Riches

-----Original Message-----
From: linux-lvm-bounces@redhat.com [mailto:linux-lvm-bounces@redhat.com] On Behalf Of Vladislav Bogdanov
Sent: Thursday, November 10, 2011 1:06 AM
To: linux-lvm@redhat.com
Subject: Re: [linux-lvm] cLVM commands often taking a long time to execute

09.11.2011 03:06, Riches Jr, Robert M wrote:
> I'm running into some odd behavior from clustered LVM commands on a 
> 3-node pacemaker-based cluster.  When other processes on the nodes are 
> doing somewhat heavy sequential I/O traffic, LVM commands such as lvs, 
> pvs, vgs, lvdisplay, pvdisplay, and vgdisplay have widely variable, 
> often very long execution times.  The same LVM command done 
> repetitively will have execution times varying from less than a second 
> to 10-15 seconds, more rarely up to around 60 seconds.  Other 
> read-heavy commands such as find, wc, and sum have much less 
> variability and much shorter tails in the distribution of their execution times.
> 
> Is it known what causes the cLVM commands to be so slow at times?
> 
> Perchance, are there known ways to get cLVM commands to be less slow?

I use chrt -r 99 for them. And I also run clvmd the same way. But this is much more about CPU load then IO load.
Also, please make sure that you correctly filter unneeded block devices from being PV-suspects (filter clause in lvm.conf).

Best,
Vladislav

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] cLVM commands often taking a long time to execute

[Vladislav Bogdanov]

10.11.2011 19:16, Riches Jr, Robert M wrote:
> Thank you, Vladislav, for the suggestion.  That could potentially
> help in cases where the nodes are CPU-bound, likely independent of
> whether they are I/O-bound.

Except filtering unneeded block devices. This greatly helps in IO-bound
setups.

> 
> Thanks,
> 
> Robert Riches
> 
> -----Original Message----- From: linux-lvm-bounces@redhat.com
> [mailto:linux-lvm-bounces@redhat.com] On Behalf Of Vladislav
> Bogdanov Sent: Thursday, November 10, 2011 1:06 AM To:
> linux-lvm@redhat.com Subject: Re: [linux-lvm] cLVM commands often
> taking a long time to execute
> 
> 09.11.2011 03:06, Riches Jr, Robert M wrote:
>> I'm running into some odd behavior from clustered LVM commands on a
>>  3-node pacemaker-based cluster.  When other processes on the nodes
>> are doing somewhat heavy sequential I/O traffic, LVM commands such
>> as lvs, pvs, vgs, lvdisplay, pvdisplay, and vgdisplay have widely
>> variable, often very long execution times.  The same LVM command
>> done repetitively will have execution times varying from less than
>> a second to 10-15 seconds, more rarely up to around 60 seconds.
>> Other read-heavy commands such as find, wc, and sum have much less
>>  variability and much shorter tails in the distribution of their
>> execution times.
>> 
>> Is it known what causes the cLVM commands to be so slow at times?
>> 
>> Perchance, are there known ways to get cLVM commands to be less
>> slow?
> 
> I use chrt -r 99 for them. And I also run clvmd the same way. But
> this is much more about CPU load then IO load. Also, please make sure
> that you correctly filter unneeded block devices from being
> PV-suspects (filter clause in lvm.conf).
> 
> Best, Vladislav
> 
> _______________________________________________ linux-lvm mailing
> list linux-lvm@redhat.com 
> https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO
> at http://tldp.org/HOWTO/LVM-HOWTO/
> 
> _______________________________________________ linux-lvm mailing
> list linux-lvm@redhat.com 
> https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO
> at http://tldp.org/HOWTO/LVM-HOWTO/