[linux-lvm] Why does every lvm command insist on touching every pv?

[linux-lvm] Why does every lvm command insist on touching every pv?

[Phillip Susi]

Every time I run any lvm command, it goes out and touches every pv,
causing all the disks to wake up from standby.  Why is this?  Shouldn't
a simple lvs command be able to look in the /etc/lvm/cache for what it
needs instead of touching every pv, even ones with no lvs on them?

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Alasdair G Kergon]

On Tue, Jun 15, 2010 at 11:23:47AM -0400, Phillip Susi wrote:
> Every time I run any lvm command, it goes out and touches every pv,
> causing all the disks to wake up from standby.  Why is this?  Shouldn't
> a simple lvs command be able to look in the /etc/lvm/cache for what it
> needs instead of touching every pv, even ones with no lvs on them?
 
It shouldn't normally scan unless it detects something might have changed, but
if you know the volume group name(s) already, put them on the command line: 
  lvs vg1 vg2

(As always -vvvv may give you clues as to what is triggering the scan.)

Alasdair

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Phillip Susi]

On 6/15/2010 12:47 PM, Alasdair G Kergon wrote:
> It shouldn't normally scan unless it detects something might have changed, but
> if you know the volume group name(s) already, put them on the command line: 
>   lvs vg1 vg2
> 
> (As always -vvvv may give you clues as to what is triggering the scan.)

I can see from the output with -vvvv that it is indeed trying to find a
pv label on every block device in /dev.  I thought that I had figured
out this was because lvm.conf was setting scan = [ "/dev" ] in the
devices section, but I tried removing /dev, and just setting it to "",
as well as "none" and lvs -vvvv STILL is opening every block device in /dev.

Shouldn't this not happen if you leave the value of scan empty, and
shouldn't it be left empty when you have udev running lvscan
automatically on block devices that contain physical volumes?

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Takahiro Yasui]

On 06/15/10 13:46, Phillip Susi wrote:
> On 6/15/2010 12:47 PM, Alasdair G Kergon wrote:
>> It shouldn't normally scan unless it detects something might have changed, but
>> if you know the volume group name(s) already, put them on the command line: 
>>   lvs vg1 vg2
>>
>> (As always -vvvv may give you clues as to what is triggering the scan.)
> 
> I can see from the output with -vvvv that it is indeed trying to find a
> pv label on every block device in /dev.  I thought that I had figured
> out this was because lvm.conf was setting scan = [ "/dev" ] in the
> devices section, but I tried removing /dev, and just setting it to "",
> as well as "none" and lvs -vvvv STILL is opening every block device in /dev.

Everytime lvm command is executed, all PVs are accessed.

This is a known issue and what we should fix. If there are a lot of
PVs, execution of lvm command could take a long time. The scan also
affects error recovery time of lvm mirror.

This is an old reference.

Introduce metadata cache feature
https://www.redhat.com/archives/lvm-devel/2009-April/msg00014.html

Thanks,
Taka

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Phillip Susi]

On 6/15/2010 3:17 PM, Takahiro Yasui wrote:
> Everytime lvm command is executed, all PVs are accessed.
> 
> This is a known issue and what we should fix. If there are a lot of
> PVs, execution of lvm command could take a long time. The scan also
> affects error recovery time of lvm mirror.
> 
> This is an old reference.
> 
> Introduce metadata cache feature
> https://www.redhat.com/archives/lvm-devel/2009-April/msg00014.html

This seems to limit the number of metadata copies on pvs that are
accessed to one, but I am seeing even block devices that are not pvs
accessed, like it is scanning for new pvs.  This patch seems like it
would help some though, but it is over a year old.  Did it never get
applied?

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Takahiro Yasui]

On 06/15/10 16:24, Phillip Susi wrote:
> On 6/15/2010 3:17 PM, Takahiro Yasui wrote:
>> Everytime lvm command is executed, all PVs are accessed.
>>
>> This is a known issue and what we should fix. If there are a lot of
>> PVs, execution of lvm command could take a long time. The scan also
>> affects error recovery time of lvm mirror.
>>
>> This is an old reference.
>>
>> Introduce metadata cache feature
>> https://www.redhat.com/archives/lvm-devel/2009-April/msg00014.html
> 
> This seems to limit the number of metadata copies on pvs that are
> accessed to one, but I am seeing even block devices that are not pvs
> accessed, like it is scanning for new pvs.  This patch seems like it
> would help some though, but it is over a year old.  Did it never get
> applied?

As for accessing block devices, you can limit accesses to those
devices by filter option in lvm.conf. This is an example which
allows accesses only to /dev/sd*

    filter = [ "a|/dev/sd|", "r|.*|" ]

You can specify more strict rule like

    filter = [ "a|/dev/sd[a-z]|", "r|.*|" ]

For my patch, I posted it one year ago but I need to have more
discussion how to solve this issue on lvm-devel.

Thanks,
Taka

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Phillip Susi]

On 06/15/2010 04:41 PM, Takahiro Yasui wrote:
> As for accessing block devices, you can limit accesses to those
> devices by filter option in lvm.conf. This is an example which
> allows accesses only to /dev/sd*
> 
>      filter = [ "a|/dev/sd|", "r|.*|" ]
> 
> You can specify more strict rule like
> 
>      filter = [ "a|/dev/sd[a-z]|", "r|.*|" ]

What if I don't want ANY devices to be scanned every time an lvm command
is run?  Shouldn't they be scanned once when udev first detects they
have been attached, and no more?  I thought removing /dev from the scan=
line would do that, but it didn't.

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Zdenek Kabelac]

Dne 16.6.2010 02:34, Phillip Susi napsal(a):
> On 06/15/2010 04:41 PM, Takahiro Yasui wrote:
>> As for accessing block devices, you can limit accesses to those
>> devices by filter option in lvm.conf. This is an example which
>> allows accesses only to /dev/sd*
>>
>>       filter = [ "a|/dev/sd|", "r|.*|" ]
>>
>> You can specify more strict rule like
>>
>>       filter = [ "a|/dev/sd[a-z]|", "r|.*|" ]
>
> What if I don't want ANY devices to be scanned every time an lvm command
> is run?  Shouldn't they be scanned once when udev first detects they
> have been attached, and no more?  I thought removing /dev from the scan=
> line would do that, but it didn't.
>

I'd have say - this is work-in-progress for now - yes, udev should handle all 
this for lvm - but we are not yet there...

Meanwhile - you may write few scripts which will pass exact filters for your 
lvm commands if you know you do not need to touch those devices.

Zdenek

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Takahiro Yasui]

On 06/16/10 05:30, Zdenek Kabelac wrote:
> Dne 16.6.2010 02:34, Phillip Susi napsal(a):
>> On 06/15/2010 04:41 PM, Takahiro Yasui wrote:
...
>> What if I don't want ANY devices to be scanned every time an lvm command
>> is run?  Shouldn't they be scanned once when udev first detects they
>> have been attached, and no more?  I thought removing /dev from the scan=
>> line would do that, but it didn't.
>>
> 
> I'd have say - this is work-in-progress for now - yes, udev should handle all 
> this for lvm - but we are not yet there...

Hi Zdenek,

Sorry I passed over your post.

May I ask you how you are going to handle this issue by udev? The problem
is *not* a scan of devices to update dev-cache but a scan of devices to
update lvm metadata cache in memory by lvm command. Currently lvm does
not have a kind of metadata cache on a device (e.g. /etc/lvm) and lvm
command has to get metadata information stored on each PV whenever it is
executed.

It is helpful if udev can handle this issue, but I'm wondering how it can
do it.

Thanks,
Taka

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Zdenek Kabelac]

Dne 16.6.2010 21:27, Takahiro Yasui napsal(a):
> On 06/16/10 05:30, Zdenek Kabelac wrote:
>> Dne 16.6.2010 02:34, Phillip Susi napsal(a):
>>> On 06/15/2010 04:41 PM, Takahiro Yasui wrote:
> ...
>>> What if I don't want ANY devices to be scanned every time an lvm command
>>> is run?  Shouldn't they be scanned once when udev first detects they
>>> have been attached, and no more?  I thought removing /dev from the scan=
>>> line would do that, but it didn't.
>>>
>>
>> I'd have say - this is work-in-progress for now - yes, udev should handle all
>> this for lvm - but we are not yet there...
> 
> Hi Zdenek,
> 
> Sorry I passed over your post.
> 
> May I ask you how you are going to handle this issue by udev? The problem
> is *not* a scan of devices to update dev-cache but a scan of devices to
> update lvm metadata cache in memory by lvm command. Currently lvm does
> not have a kind of metadata cache on a device (e.g. /etc/lvm) and lvm
> command has to get metadata information stored on each PV whenever it is
> executed.
> 
> It is helpful if udev can handle this issue, but I'm wondering how it can
> do it.

I'm not working on this part, but AFAIK, once we could start 'trust' udev, we
can keep persistent cache aware of any changes that might have happened to
devices listed in metadata. Implementation details are still 'moving topic'.

Obviously you can not skip write/update access to metadata areas, but it
should be possible to avoid scanning for 'read-only' data access.

Also there is another thing in progress - metadata-balance code - where you
essentially do not need to read/write metadata from/to every PV in VG - but
just on reasonable safe amount of them - i.e. 5 from 100 of PVs - the rest of
them is marked invisible (different from pvcreate --metadatasize 0)

Zdenek

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Takahiro Yasui]

On 06/17/10 04:23, Zdenek Kabelac wrote:
> Dne 16.6.2010 21:27, Takahiro Yasui napsal(a):
>> On 06/16/10 05:30, Zdenek Kabelac wrote:
>>> Dne 16.6.2010 02:34, Phillip Susi napsal(a):
>>>> On 06/15/2010 04:41 PM, Takahiro Yasui wrote:
>> ...
>>>> What if I don't want ANY devices to be scanned every time an lvm command
>>>> is run?  Shouldn't they be scanned once when udev first detects they
>>>> have been attached, and no more?  I thought removing /dev from the scan=
>>>> line would do that, but it didn't.
>>>>
...
>> It is helpful if udev can handle this issue, but I'm wondering how it can
>> do it.
> 
> I'm not working on this part, but AFAIK, once we could start 'trust' udev, we
> can keep persistent cache aware of any changes that might have happened to
> devices listed in metadata. Implementation details are still 'moving topic'.
> 
> Obviously you can not skip write/update access to metadata areas, but it
> should be possible to avoid scanning for 'read-only' data access.

Thank you for your explanation. Yes, I agree that it is possible to avoid
scanning for 'read-only' data access, but I also believe it is possible for
'write' adata access.
 
> Also there is another thing in progress - metadata-balance code - where you
> essentially do not need to read/write metadata from/to every PV in VG - but
> just on reasonable safe amount of them - i.e. 5 from 100 of PVs - the rest of
> them is marked invisible (different from pvcreate --metadatasize 0)

AFAIK, metadata-balance feature would reduce the number of disk accesses,
but I believe that the goal is to access PVs related to the VG which lvm
command is going to manipulate. Introducing metadata cache feature on disk
or a kind of daemon managing all metadatas, or using /etc/lvm/backup could
be solution.

I hope we could continue discussing this topic on lvm-devel?

Thanks,
Taka

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Zdenek Kabelac]

Dne 17.6.2010 15:53, Takahiro Yasui napsal(a):
> On 06/17/10 04:23, Zdenek Kabelac wrote:
>> Dne 16.6.2010 21:27, Takahiro Yasui napsal(a):
>>> On 06/16/10 05:30, Zdenek Kabelac wrote:
>>>> Dne 16.6.2010 02:34, Phillip Susi napsal(a):
>>>>> On 06/15/2010 04:41 PM, Takahiro Yasui wrote:
>>> ...
>>>>> What if I don't want ANY devices to be scanned every time an lvm command
>>>>> is run?  Shouldn't they be scanned once when udev first detects they
>>>>> have been attached, and no more?  I thought removing /dev from the scan=
>>>>> line would do that, but it didn't.
>>>>>
> ...
>>> It is helpful if udev can handle this issue, but I'm wondering how it can
>>> do it.
>>
>> I'm not working on this part, but AFAIK, once we could start 'trust' udev, we
>> can keep persistent cache aware of any changes that might have happened to
>> devices listed in metadata. Implementation details are still 'moving topic'.
>>
>> Obviously you can not skip write/update access to metadata areas, but it
>> should be possible to avoid scanning for 'read-only' data access.
> 
> Thank you for your explanation. Yes, I agree that it is possible to avoid
> scanning for 'read-only' data access, but I also believe it is possible for
> 'write' adata access.


With current LVM logic - you can't proceed with usage of LVM metadata unless
they are properly committed to PVs.  i.e. there is no chance you could use
partially stored metadata to just some cached devices. Either you update all
metadata or you fail - there is nothing between these 2 states.

> 
>> Also there is another thing in progress - metadata-balance code - where you
>> essentially do not need to read/write metadata from/to every PV in VG - but
>> just on reasonable safe amount of them - i.e. 5 from 100 of PVs - the rest of
>> them is marked invisible (different from pvcreate --metadatasize 0)
> 
> AFAIK, metadata-balance feature would reduce the number of disk accesses,
> but I believe that the goal is to access PVs related to the VG which lvm
> command is going to manipulate. Introducing metadata cache feature on disk
> or a kind of daemon managing all metadatas, or using /etc/lvm/backup could
> be solution.
> 
> I hope we could continue discussing this topic on lvm-devel?

Sure. Daemon is also planned, but for reduction of write access metadata
balancing should greatly help.  Another step here could be to parallelize all
disk operations on different devices. Also udev handling has some perfomance
optimalization still.

With properly working udev we shouldn't need to do any device scanning as we
will have all 'interesting' devices stored in some cache storage - it could be
file, daemon, udev DB entry....

Zdenek

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Takahiro Yasui]

On 06/18/10 10:27, Zdenek Kabelac wrote:
> Dne 17.6.2010 15:53, Takahiro Yasui napsal(a):
>> On 06/17/10 04:23, Zdenek Kabelac wrote:
>>> Dne 16.6.2010 21:27, Takahiro Yasui napsal(a):
>>>> On 06/16/10 05:30, Zdenek Kabelac wrote:
>>>>> Dne 16.6.2010 02:34, Phillip Susi napsal(a):
>>>>>> On 06/15/2010 04:41 PM, Takahiro Yasui wrote:
>>>> ...
>>>>>> What if I don't want ANY devices to be scanned every time an lvm command
>>>>>> is run?  Shouldn't they be scanned once when udev first detects they
>>>>>> have been attached, and no more?  I thought removing /dev from the scan=
>>>>>> line would do that, but it didn't.
>>>>>>
>> ...
>>>> It is helpful if udev can handle this issue, but I'm wondering how it can
>>>> do it.
>>>
>>> I'm not working on this part, but AFAIK, once we could start 'trust' udev, we
>>> can keep persistent cache aware of any changes that might have happened to
>>> devices listed in metadata. Implementation details are still 'moving topic'.
>>>
>>> Obviously you can not skip write/update access to metadata areas, but it
>>> should be possible to avoid scanning for 'read-only' data access.
>>
>> Thank you for your explanation. Yes, I agree that it is possible to avoid
>> scanning for 'read-only' data access, but I also believe it is possible for
>> 'write' adata access.
> 
> 
> With current LVM logic - you can't proceed with usage of LVM metadata unless
> they are properly committed to PVs.  i.e. there is no chance you could use
> partially stored metadata to just some cached devices. Either you update all
> metadata or you fail - there is nothing between these 2 states.

I agree with you comment in terms that every metadata stored on PVs in the
same VG which are being manipulated should be committed, but I don't think
metadata stored on PVs which belong to *different* VGs need to be committed.

For example, there are six PVs and two VGs as below:

  VG1: PV1, PV2, PV3
  VG2: PV4, PV5, PV6

If we create a new LV or delete VG1, then metadatas on only PV1, PV2, PV3
need to be updated but not for PV4, PV5, PV6.

>>> Also there is another thing in progress - metadata-balance code - where you
>>> essentially do not need to read/write metadata from/to every PV in VG - but
>>> just on reasonable safe amount of them - i.e. 5 from 100 of PVs - the rest of
>>> them is marked invisible (different from pvcreate --metadatasize 0)
>>
>> AFAIK, metadata-balance feature would reduce the number of disk accesses,
>> but I believe that the goal is to access PVs related to the VG which lvm
>> command is going to manipulate. Introducing metadata cache feature on disk
>> or a kind of daemon managing all metadatas, or using /etc/lvm/backup could
>> be solution.
>>
>> I hope we could continue discussing this topic on lvm-devel?
> 
> Sure. Daemon is also planned, but for reduction of write access metadata
> balancing should greatly help.  Another step here could be to parallelize all
> disk operations on different devices. Also udev handling has some perfomance
> optimalization still.
> 
> With properly working udev we shouldn't need to do any device scanning as we
> will have all 'interesting' devices stored in some cache storage - it could be
> file, daemon, udev DB entry....

I agree that these approach improve performance, but I don't think
accesses to devices which don't belong to a VG being handled are not
welcomed.

Thanks,
Taka

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Alasdair G Kergon]

On Fri, Jun 18, 2010 at 10:50:41AM -0400, Takahiro Yasui wrote:
> For example, there are six PVs and two VGs as below:
> 
>   VG1: PV1, PV2, PV3
>   VG2: PV4, PV5, PV6
> 
> If we create a new LV or delete VG1, then metadatas on only PV1, PV2, PV3
> need to be updated but not for PV4, PV5, PV6.
 
Correct - and the tools have always behaved like that!

(I don't think this thread is going anywhere useful.)

Alasdair

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Takahiro Yasui]

On 06/18/10 11:52, Alasdair G Kergon wrote:
> On Fri, Jun 18, 2010 at 10:50:41AM -0400, Takahiro Yasui wrote:
>> For example, there are six PVs and two VGs as below:
>>
>>   VG1: PV1, PV2, PV3
>>   VG2: PV4, PV5, PV6
>>
>> If we create a new LV or delete VG1, then metadatas on only PV1, PV2, PV3
>> need to be updated but not for PV4, PV5, PV6.
>  
> Correct - and the tools have always behaved like that!

Did I miss something? Or can we configure lvm.conf to make lvm
commands behave like that? That's the first topic we were talking
about. As far as I use lvm commands, they don't behave as I expected.

Thanks,
Taka

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Takahiro Yasui]

On 06/18/10 13:10, Takahiro Yasui wrote:
> On 06/18/10 11:52, Alasdair G Kergon wrote:
>> On Fri, Jun 18, 2010 at 10:50:41AM -0400, Takahiro Yasui wrote:
>>> For example, there are six PVs and two VGs as below:
>>>
>>>   VG1: PV1, PV2, PV3
>>>   VG2: PV4, PV5, PV6
>>>
>>> If we create a new LV or delete VG1, then metadatas on only PV1, PV2, PV3
>>> need to be updated but not for PV4, PV5, PV6.
>>  
>> Correct - and the tools have always behaved like that!
> 
> Did I miss something? Or can we configure lvm.conf to make lvm
> commands behave like that? That's the first topic we were talking
> about. As far as I use lvm commands, they don't behave as I expected.

I'm sorry. My explanation was not good. Only PV1-3 are updated and
PV4-6 are not. LVM command have always behaved like this.

I should have added an more sentence like

If we create a new LV or delete VG1, then metadatas on only PV1, PV2, PV3
need to be updated but not for PV4, PV5, PV6. So PV4-6 don't necessarily
need to be accessed.

Sorry for making you confused.

Thanks,
Taka

Re: [linux-lvm] Why does every lvm command insist on touching every pv?

[Takahiro Yasui]

On 06/15/10 20:34, Phillip Susi wrote:
> On 06/15/2010 04:41 PM, Takahiro Yasui wrote:
>> As for accessing block devices, you can limit accesses to those
>> devices by filter option in lvm.conf. This is an example which
>> allows accesses only to /dev/sd*
>>
>>      filter = [ "a|/dev/sd|", "r|.*|" ]
>>
>> You can specify more strict rule like
>>
>>      filter = [ "a|/dev/sd[a-z]|", "r|.*|" ]
> 
> What if I don't want ANY devices to be scanned every time an lvm command
> is run?  Shouldn't they be scanned once when udev first detects they
> have been attached, and no more?  I thought removing /dev from the scan=
> line would do that, but it didn't.

I understand your requests and I have the same opinion as yours.
My suggestion of 'filter' option just reduces the number of
accesses to disks which are not related to PVs. Hopefully I would
like to find a good solution and fix for this issue.

I think that this kind of options are very helpful to give
increased priority to this issue among LVM community.

Thanks,
Taka