[linux-lvm] metadata too large for circular buffer - any way to increase PV metadatasize?

[linux-lvm] metadata too large for circular buffer - any way to increase PV metadatasize?

[John Leach]

Hi,

I'm getting the dreaded "metadata too large for circular buffer" message
trying to create a new LV in a volume group of a few hundred other LVs.

As I understand, this is because the PVs don't have enough space for the
metadata.  This size can be set to a larger value when pvcreating with
--metadatasize but I have obviously missed that boat :)

Is there a way to increase this with live PVs, rather than shifting all
the data off them and recreating and shifting back on?

I have a multinode RedHat Cluster with CLVMD and two PVs.  I can add
additional PVs if that helps (though as I understand *every* PV needs
enough metadata space).

Thanks in advance!

John.

-- 
http://johnleach.co.uk

Re: [linux-lvm] metadata too large for circular buffer - any way to increase PV metadatasize?

[Alasdair G Kergon]

On Thu, Aug 07, 2008 at 07:07:56PM +0100, John Leach wrote:
> As I understand, this is because the PVs don't have enough space for the
> metadata.  This size can be set to a larger value when pvcreating with
> --metadatasize but I have obviously missed that boat :)
> Is there a way to increase this with live PVs, rather than shifting all
> the data off them and recreating and shifting back on?
 
Unfortunately there is no way to increase this retrospectively.
Nor have we written tools to manipulate metadata areas yet.

You might be able to take steps to make the metadata smaller if you have
long names you can shorten or fragmented LVs that can be defragmented with a
sequence of pvmoves.

> I have a multinode RedHat Cluster with CLVMD and two PVs.  I can add
> additional PVs if that helps (though as I understand *every* PV needs
> enough metadata space).

You're right that the best option is to add more PVs.

Roughly:

Create a new PV with a much larger metadata area, add it to the VG,
then stop using the too-small metadata areas on the old PVs.

For extra resiliance it might be wise to add two PVs so you have two
copies of the metadata, or use the pvcreate option to have 2 metadata
copies on the single PV you're adding, one at the front and ane at the end.

To remove the metadata areas you need to:

   get an up-to-date metadata backup (vgcfgbackup)

  pvcreate --restorefile pointing at a copy of that backup file requesting 0 metadata areas
and specifying the same uuid as it had before

  vgcfgrestore from the backup file
 
Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] metadata too large for circular buffer - any way to increase PV metadatasize?

[Alasdair G Kergon]

And if you have problems making space in the metadata to add the PV to it,
a workaround is to temporarily use in-filesystem metadata:

  create a metadata dir in a local filesystem (NOT an LV), and place an up-to-date
  backup copy of the metadata there (from vgcfgbackup)

  update lvm.conf to start using this metadata (metadata/dirs)
  - use verbose display options to check it's doing this

  pvcreate as in my last message to remove the too-small metadata area
  - you're now relying entirely on the in-filesystem metadata

  add the new PVs to the VG as before

  revert the lvm.conf change
  
Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] metadata too large for circular buffer - any way to increase PV metadatasize?

[Alasdair G Kergon]

On Thu, Aug 07, 2008 at 08:52:41PM +0100, Alasdair G Kergon wrote:
> And if you have problems making space in the metadata to add the PV to it,
> a workaround is to temporarily use in-filesystem metadata:
 
I should also point out that although using in-filesystem metadata is a
fully-supported configuration, it does not receive much testing because few
people use it, so if you do need to use that method and hit any problems,
certainly let us know.

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] metadata too large for circular buffer - any way to increase PV metadatasize?

[John Leach]

On Thu, 2008-08-07 at 20:52 +0100, Alasdair G Kergon wrote:
> And if you have problems making space in the metadata to add the PV to it,
> a workaround is to temporarily use in-filesystem metadata:
> 
>   create a metadata dir in a local filesystem (NOT an LV), and place an up-to-date
>   backup copy of the metadata there (from vgcfgbackup)
> 
>   update lvm.conf to start using this metadata (metadata/dirs)
>   - use verbose display options to check it's doing this
> 
>   pvcreate as in my last message to remove the too-small metadata area
>   - you're now relying entirely on the in-filesystem metadata
> 
>   add the new PVs to the VG as before
> 
>   revert the lvm.conf change

Will this work with clustered LVM?  If so, I take it all nodes will have
to be configured to use in-filesystem metadata?

Thanks,

John.
-- 
http://johnleach.co.uk

Re: [linux-lvm] metadata too large for circular buffer - any way to increase PV metadatasize?

[John Leach]

On Thu, 2008-08-07 at 20:41 +0100, Alasdair G Kergon wrote:
> Unfortunately there is no way to increase this retrospectively.
> Nor have we written tools to manipulate metadata areas yet.
> 
> You're right that the best option is to add more PVs.
> 
> Roughly:
> 
> Create a new PV with a much larger metadata area, add it to the VG,
> then stop using the too-small metadata areas on the old PVs.

Hi Alasdair,

thanks for the advice - very much appreciated.  I'm going to test this
out on a test cluster.  It will buy some time whilst we consider what to
do next (probably pvmove things around and recreate PVs with more
metadata space).

Thanks again,

John.

-- 
http://johnleach.co.uk

Re: [linux-lvm] metadata too large for circular buffer - any way to increase PV metadatasize?

[John Leach]

On Thu, 2008-08-07 at 20:41 +0100, Alasdair G Kergon wrote:

> To remove the metadata areas you need to:
> 
>    get an up-to-date metadata backup (vgcfgbackup)
> 
>   pvcreate --restorefile pointing at a copy of that backup file requesting 0 metadata areas
> and specifying the same uuid as it had before
> 
>   vgcfgrestore from the backup file
>  

This doesn't look like something you can do with the volume group active
(with cluster lvm anyway):

[root@testnode0 ~]# pvcreate --restorefile san-metadata -u fTLglk-j1C1-02Z7-8k6l-DTAm-2WNj-9ZGT19 --metadatacopies 0 /dev/hdb -ff
Really INITIALIZE physical volume "/dev/hdb" of volume group "san" [y/n]? y
  Can't open /dev/hdb exclusively.  Mounted filesystem?

Is that right, or am I missing something?

Thanks,

John.

Re: [linux-lvm] metadata too large for circular buffer - any way to increase PV metadatasize?

[Charles Koprowski]

[-- Attachment #1: Type: text/plain, Size: 1288 bytes --]

 On Sun, 10 Aug 2008 18:45:16 +0100, John Leach wrote:

On Thu, 2008-08-07 at 20:41 +0100, Alasdair G Kergon wrote:
> > To remove the metadata areas you need to:
> >
> >    get an up-to-date metadata backup (vgcfgbackup)
> >
> >   pvcreate --restorefile pointing at a copy of that backup file
> requesting 0 metadata areas
> > and specifying the same uuid as it had before
> >
> >   vgcfgrestore from the backup file
> >
> This doesn't look like something you can do with the volume group active
> (with cluster lvm anyway):
> [root testnode0 ~]# pvcreate --restorefile san-metadata -u
> fTLglk-j1C1-02Z7-8k6l-DTAm-2WNj-9ZGT19 --metadatacopies 0 /dev/hdb -ff
> Really INITIALIZE physical volume "/dev/hdb" of volume group "san" [y/n]? y
>   Can't open /dev/hdb exclusively.  Mounted filesystem?
> Is that right, or am I missing something?
>

Hi John,

I know it's been a long time but, Did you found out a solution for this ?

I'm trying to follow the same scenario to increase the metadatasize and got
stuck at the same point.

Thanks

Charles

-- 
Charles Koprowski
Systems & Networks
<http://www.audaxis.com/>
cko@audaxis.com
Tel : +33 3 20 63 88 70 <+33+3+20+63+88+70>
GSM : +33 6 29 32 63 33 <+33+6+29+32+63+33>
<http://twitter.com/audaxis>
<http://www.linkedin.com/company/audaxis>

[-- Attachment #2: Type: text/html, Size: 4514 bytes --]

Re: [linux-lvm] metadata too large for circular buffer - any way to increase PV metadatasize?

[Zdenek Kabelac]

Dne 28.9.2016 v 15:51 Charles Koprowski napsal(a):
>  On Sun, 10 Aug 2008 18:45:16 +0100, John Leach wrote:
>
>     On Thu, 2008-08-07 at 20:41 +0100, Alasdair G Kergon wrote:
>     > To remove the metadata areas you need to:
>     >
>     >    get an up-to-date metadata backup (vgcfgbackup)
>     >
>     >   pvcreate --restorefile pointing at a copy of that backup file
>     requesting 0 metadata areas
>     > and specifying the same uuid as it had before
>     >
>     >   vgcfgrestore from the backup file
>     >
>     This doesn't look like something you can do with the volume group active
>     (with cluster lvm anyway):
>     [root testnode0 ~]# pvcreate --restorefile san-metadata -u
>     fTLglk-j1C1-02Z7-8k6l-DTAm-2WNj-9ZGT19 --metadatacopies 0 /dev/hdb -ff
>     Really INITIALIZE physical volume "/dev/hdb" of volume group "san" [y/n]? y
>       Can't open /dev/hdb exclusively.  Mounted filesystem?
>     Is that right, or am I missing something?
>
>
> Hi John,
>
> I know it's been a long time but, Did you found out a solution for this ?
>
> I'm trying to follow the same scenario to increase the metadatasize and got
> stuck at the same point.




Hi

I've no real idea what's the original problem  - but here is clear error on 
user side. This message:

"Can't open /dev/hdb exclusively.  Mounted filesystem?"

indicates that device is NOT UNUSED and it has to be unused for this operation 
- i.e. you MAY NOT run this operation while there are active LVs from this PV 
(one of possible reasons why  'hdb' cannot be opened exclusively)



To increase 'metadata' size - you ether have to add some NEW PV with
much bigger metadata size space (and such operation still needs some
extra space to proceed with existing size). Then disable existing metadata
areas - so you will end-up of storing bigger MDA only to this new bigger MDA 
on a new PV -  note - it's quite risky plan to leave MDA only on single PV in 
multi-PV VG  - so I'd not advice this for any serious use.

You may PV via pvmove to a new bigger PV.
(adviced operation - though it may take its time...)

There is NOT lvm2  native support for  'online' resize of PV mda size - and 
while it's possible to do this operation manually - it's quite complex task - 
so I'd not advice to do this either - unless you have FULL backup of everything.

So back to your original problem - please describe  EXACTLY what is your 
problem and attach  outputs of  pvs,vgs,lvs  and what you want to achieve.


Regards

Zdenek

Re: [linux-lvm] metadata too large for circular buffer - any way to increase PV metadatasize?

[Charles Koprowski]

[-- Attachment #1: Type: text/plain, Size: 3576 bytes --]

2016-09-29 11:23 GMT+02:00 Zdenek Kabelac <zkabelac@redhat.com>:

> Dne 28.9.2016 v 15:51 Charles Koprowski napsal(a):
>
>>  On Sun, 10 Aug 2008 18:45:16 +0100, John Leach wrote:
>>
>>     On Thu, 2008-08-07 at 20:41 +0100, Alasdair G Kergon wrote:
>>     > To remove the metadata areas you need to:
>>     >
>>     >    get an up-to-date metadata backup (vgcfgbackup)
>>     >
>>     >   pvcreate --restorefile pointing at a copy of that backup file
>>     requesting 0 metadata areas
>>     > and specifying the same uuid as it had before
>>     >
>>     >   vgcfgrestore from the backup file
>>     >
>>     This doesn't look like something you can do with the volume group
>> active
>>     (with cluster lvm anyway):
>>     [root testnode0 ~]# pvcreate --restorefile san-metadata -u
>>     fTLglk-j1C1-02Z7-8k6l-DTAm-2WNj-9ZGT19 --metadatacopies 0 /dev/hdb
>> -ff
>>     Really INITIALIZE physical volume "/dev/hdb" of volume group "san"
>> [y/n]? y
>>       Can't open /dev/hdb exclusively.  Mounted filesystem?
>>     Is that right, or am I missing something?
>>
>>
>> Hi John,
>>
>> I know it's been a long time but, Did you found out a solution for this ?
>>
>> I'm trying to follow the same scenario to increase the metadatasize and
>> got
>> stuck at the same point.
>>
>
>
>
>
> Hi
>
> I've no real idea what's the original problem  - but here is clear error
> on user side. This message:
>
> "Can't open /dev/hdb exclusively.  Mounted filesystem?"
>
> indicates that device is NOT UNUSED and it has to be unused for this
> operation - i.e. you MAY NOT run this operation while there are active LVs
> from this PV (one of possible reasons why  'hdb' cannot be opened
> exclusively)
>
>
>

Hi Zdenek,

I'm running a 5 nodes XEN dom0s cluster which uses a shared VG and few
hundreds LVs to store virtual machines disks.

As we were running out of space in the actual VG, I was about to add a new
PV and take the opportunity to increase the metadata size as you described
below.

I was wondering if I could run the "pvcreate --restorefile --metadatacopies
0" while the VG (and PV) were online.

Now I got the answer :-)

Thank you !

Charles



>
> To increase 'metadata' size - you ether have to add some NEW PV with
> much bigger metadata size space (and such operation still needs some
> extra space to proceed with existing size). Then disable existing metadata
> areas - so you will end-up of storing bigger MDA only to this new bigger
> MDA on a new PV -  note - it's quite risky plan to leave MDA only on single
> PV in multi-PV VG  - so I'd not advice this for any serious use.
>
> You may PV via pvmove to a new bigger PV.
> (adviced operation - though it may take its time...)
>
> There is NOT lvm2  native support for  'online' resize of PV mda size -
> and while it's possible to do this operation manually - it's quite complex
> task - so I'd not advice to do this either - unless you have FULL backup of
> everything.
>
> So back to your original problem - please describe  EXACTLY what is your
> problem and attach  outputs of  pvs,vgs,lvs  and what you want to achieve.
>
>
> Regards
>
> Zdenek
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>



-- 
Charles Koprowski
Systems & Networks
<http://www.audaxis.com/>
cko@audaxis.com
Tel : +33 3 20 63 88 70 <+33+3+20+63+88+70>
GSM : +33 6 29 32 63 33 <+33+6+29+32+63+33>
<http://twitter.com/audaxis>
<http://www.linkedin.com/company/audaxis>

[-- Attachment #2: Type: text/html, Size: 7946 bytes --]