[linux-lvm] No device found for PV but is actually fine

[linux-lvm] No device found for PV but is actually fine

[Michal Svoboda]

Hi all,

I have a weird issue:

# pvs
 No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
 PV         VG    Fmt  Attr PSize   PFree
 /dev/sdc   data2 lvm2 a--  298.09g    0 
 /dev/sdd1  data  lvm2 a--  298.09g    0 

# pvscan 
  No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
  No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
  PV /dev/sdc    VG data2   lvm2 [298.09 GiB / 0    free]
  PV /dev/sdd1   VG data    lvm2 [298.09 GiB / 0    free]
  Total: 2 [596.18 GiB] / in use: 2 [596.18 GiB] / in no VG: 0 [0   ]

# pvscan --cache
  Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sde2 not /dev/md0
  Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sdf2 not /dev/sde2

The VG on the "missing" PV is called "system", but:

# vgdisplay /dev/system
  No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
  --- Volume group ---
  VG Name               system
  System ID             
  Format                lvm2
  Metadata Areas        0
  Metadata Sequence No  28
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                8
  Open LV               8
  Max PV                0
  Cur PV                1
  Act PV                0
  VG Size               232.88 GiB
  PE Size               4.00 MiB
  Total PE              59616
  Alloc PE / Size       33792 / 132.00 GiB
  Free  PE / Size       25824 / 100.88 GiB
  VG UUID               fQoJ8p-BI3D-BcEH-BiQX-JJqf-p6TT-6KIoRt

Actually all LVs on the "system" VG are present and accounted for. The
LVs contain the root file system, and some virtual machine drives, all
of which are running *fine* (and the system in fact boots up from the
"missing" PV :). But I can't resize:

# lvextend -L +16G /dev/system/vm-monitor
  No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
  Cannot change VG system while PVs are missing.
  Consider vgreduce --removemissing.

The "system" VG is situated on a md raid1. I think it all happened after
I replaced all the drives of that md with mdadm --replace with bigger
disks. But I can't be sure, because the system in fact works and except
the inability to resize there are no problems.

I have googled around a bit, but to no avail. Can someone help me
diagnose and repair this?

Version info:

# pvs --version
  LVM version:     2.02.105(2) (2014-01-20)
  Library version: 1.02.84 (2014-01-20)
  Driver version:  4.27.0

# cat /proc/version 
Linux version 3.14.0-4-ARCH (nobody@var-lib-archbuild-extra-x86_64-thomas) (gcc version 4.8.2 20140206 (prerelease) (GCC) ) #1 SMP PREEMPT Wed Apr 9 21:11:25 CEST 2014


Thanks,
Michal Svoboda

Re: [linux-lvm] No device found for PV but is actually fine

[Jack Waterworth]

Hi Michael,

   Check your pvs output to see if any of your pvs have the 'm' 
attribute. This means the device has been marked as missing in the metadata.

You can remove this attribute by running the vgextend command.

     # vgextend --restoremissing /dev/device

I usually see this happen on virtual machines. Is this setup running on 
a virt?

  Jack Waterworth
  Senior Technical Support Engineer
  Red Hat Global Support Services North America
  919.754.4625

On 05/08/2014 06:45 AM, Michal Svoboda wrote:
> Hi all,
>
> I have a weird issue:
>
> # pvs
>   No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
>   PV         VG    Fmt  Attr PSize   PFree
>   /dev/sdc   data2 lvm2 a--  298.09g    0
>   /dev/sdd1  data  lvm2 a--  298.09g    0
>
> # pvscan
>    No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
>    No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
>    PV /dev/sdc    VG data2   lvm2 [298.09 GiB / 0    free]
>    PV /dev/sdd1   VG data    lvm2 [298.09 GiB / 0    free]
>    Total: 2 [596.18 GiB] / in use: 2 [596.18 GiB] / in no VG: 0 [0   ]
>
> # pvscan --cache
>    Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sde2 not /dev/md0
>    Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sdf2 not /dev/sde2
>
> The VG on the "missing" PV is called "system", but:
>
> # vgdisplay /dev/system
>    No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
>    --- Volume group ---
>    VG Name               system
>    System ID
>    Format                lvm2
>    Metadata Areas        0
>    Metadata Sequence No  28
>    VG Access             read/write
>    VG Status             resizable
>    MAX LV                0
>    Cur LV                8
>    Open LV               8
>    Max PV                0
>    Cur PV                1
>    Act PV                0
>    VG Size               232.88 GiB
>    PE Size               4.00 MiB
>    Total PE              59616
>    Alloc PE / Size       33792 / 132.00 GiB
>    Free  PE / Size       25824 / 100.88 GiB
>    VG UUID               fQoJ8p-BI3D-BcEH-BiQX-JJqf-p6TT-6KIoRt
>
> Actually all LVs on the "system" VG are present and accounted for. The
> LVs contain the root file system, and some virtual machine drives, all
> of which are running *fine* (and the system in fact boots up from the
> "missing" PV :). But I can't resize:
>
> # lvextend -L +16G /dev/system/vm-monitor
>    No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
>    Cannot change VG system while PVs are missing.
>    Consider vgreduce --removemissing.
>
> The "system" VG is situated on a md raid1. I think it all happened after
> I replaced all the drives of that md with mdadm --replace with bigger
> disks. But I can't be sure, because the system in fact works and except
> the inability to resize there are no problems.
>
> I have googled around a bit, but to no avail. Can someone help me
> diagnose and repair this?
>
> Version info:
>
> # pvs --version
>    LVM version:     2.02.105(2) (2014-01-20)
>    Library version: 1.02.84 (2014-01-20)
>    Driver version:  4.27.0
>
> # cat /proc/version
> Linux version 3.14.0-4-ARCH (nobody@var-lib-archbuild-extra-x86_64-thomas) (gcc version 4.8.2 20140206 (prerelease) (GCC) ) #1 SMP PREEMPT Wed Apr 9 21:11:25 CEST 2014
>
>
> Thanks,
> Michal Svoboda
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] No device found for PV but is actually fine

[Marian Csontos]

On 05/08/2014 12:45 PM, Michal Svoboda wrote:
> Hi all,
>
> I have a weird issue:
>
> # pvs
>   No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
>   PV         VG    Fmt  Attr PSize   PFree
>   /dev/sdc   data2 lvm2 a--  298.09g    0
>   /dev/sdd1  data  lvm2 a--  298.09g    0
>
> # pvscan
>    No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
>    No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
>    PV /dev/sdc    VG data2   lvm2 [298.09 GiB / 0    free]
>    PV /dev/sdd1   VG data    lvm2 [298.09 GiB / 0    free]
>    Total: 2 [596.18 GiB] / in use: 2 [596.18 GiB] / in no VG: 0 [0   ]
>
> # pvscan --cache
>    Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sde2 not /dev/md0
>    Found duplicate PV JszGc1S16sbsFQweAlaj8Wd6y4BeVqcN: using /dev/sdf2 not /dev/sde2


Peter, is this an error in LVM filtering or is manual filter 
modification required?

Michal(-e), I suggest modifying global_filter to accept only the devices 
you want in the VG.

-- Martian

>
> The VG on the "missing" PV is called "system", but:
>
> # vgdisplay /dev/system
>    No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
>    --- Volume group ---
>    VG Name               system
>    System ID
>    Format                lvm2
>    Metadata Areas        0
>    Metadata Sequence No  28
>    VG Access             read/write
>    VG Status             resizable
>    MAX LV                0
>    Cur LV                8
>    Open LV               8
>    Max PV                0
>    Cur PV                1
>    Act PV                0
>    VG Size               232.88 GiB
>    PE Size               4.00 MiB
>    Total PE              59616
>    Alloc PE / Size       33792 / 132.00 GiB
>    Free  PE / Size       25824 / 100.88 GiB
>    VG UUID               fQoJ8p-BI3D-BcEH-BiQX-JJqf-p6TT-6KIoRt
>
> Actually all LVs on the "system" VG are present and accounted for. The
> LVs contain the root file system, and some virtual machine drives, all
> of which are running *fine* (and the system in fact boots up from the
> "missing" PV :). But I can't resize:
>
> # lvextend -L +16G /dev/system/vm-monitor
>    No device found for PV JszGc1-S16s-bsFQ-weAl-aj8W-d6y4-BeVqcN.
>    Cannot change VG system while PVs are missing.
>    Consider vgreduce --removemissing.
>
> The "system" VG is situated on a md raid1. I think it all happened after
> I replaced all the drives of that md with mdadm --replace with bigger
> disks. But I can't be sure, because the system in fact works and except
> the inability to resize there are no problems.
>
> I have googled around a bit, but to no avail. Can someone help me
> diagnose and repair this?
>
> Version info:
>
> # pvs --version
>    LVM version:     2.02.105(2) (2014-01-20)
>    Library version: 1.02.84 (2014-01-20)
>    Driver version:  4.27.0
>
> # cat /proc/version
> Linux version 3.14.0-4-ARCH (nobody@var-lib-archbuild-extra-x86_64-thomas) (gcc version 4.8.2 20140206 (prerelease) (GCC) ) #1 SMP PREEMPT Wed Apr 9 21:11:25 CEST 2014
>
>
> Thanks,
> Michal Svoboda
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

Re: [linux-lvm] No device found for PV but is actually fine

[Michal Svoboda]

Marian Csontos wrote:
> Michal(-e), I suggest modifying global_filter to accept only the
> devices you want in the VG.

Hi,

I've tried playing with filter and global_filter in lvm.conf, but it did
not help - as if the conf file was ignored. I tried blacklisting and
whitelisting. Does something need to be restarted/rebooted for these
settings to take effect?

I then did some more experiments and eventually solved the issue.

First I tried pvscan --cache /dev/md0, at which point md0 was "accepted"
as existing PV, and the "system" VG would show up in vgdisplay. I was
even able to resize one of the LVs. However, after running pvscan --cache
with no extra agruments, I would receive the duplicate PV errors as in
previous e-mail and the "system" VG would disappear from vgdisplay. :-o

Finally I solved the issue by changing initcpio hook from "mdadm_udev"
to "mdadm". This is arch-linux specific thing that changes stuff that is
exported to the initial ramdisk used for boot. User-wise, that change
would stop using udev auto-magic for assembling md arrays, but use
/etc/mdadm.conf directly instead. I don't know the implementation details.

After that I receive *no more errors* about either missing or duplicate
PV and all VGs/LVs are displayed by the respective display commands. So
for me issue is solved but if anyone wants to get to the bottom I am
open to do some more experiments.


Michal Svoboda

Re: [linux-lvm] No device found for PV but is actually fine

[Marian Csontos]

On 05/09/2014 02:40 PM, Michal Svoboda wrote:
> Marian Csontos wrote:
>> Michal(-e), I suggest modifying global_filter to accept only the
>> devices you want in the VG.
>
> Hi,
>
> I've tried playing with filter and global_filter in lvm.conf, but it did
> not help - as if the conf file was ignored. I tried blacklisting and
> whitelisting. Does something need to be restarted/rebooted for these
> settings to take effect?

Oh yes, you would need to restart lvmetad (on Fedora/RHEL/CentOS 
systemctl restart lvm2-lvmetad.service, not sure it is the same on Arch.)

You may want to scrub the RAID as there may be data not written to both 
legs.

IMO it would be the best to take over udev rules and systemd units from 
Fedora as Peter spends lot of time to keep them up to date and working.

I have seen some threads on Arch forum yesterday related to boot 
problems with LVM on mdadm after upgrading from lvm2-2.02.105 to .106 - 
I think people there will appreciate if you add a comment...

-- Martian

>
> I then did some more experiments and eventually solved the issue.
>
> First I tried pvscan --cache /dev/md0, at which point md0 was "accepted"
> as existing PV, and the "system" VG would show up in vgdisplay. I was
> even able to resize one of the LVs. However, after running pvscan --cache
> with no extra agruments, I would receive the duplicate PV errors as in
> previous e-mail and the "system" VG would disappear from vgdisplay. :-o
>
> Finally I solved the issue by changing initcpio hook from "mdadm_udev"
> to "mdadm". This is arch-linux specific thing that changes stuff that is
> exported to the initial ramdisk used for boot. User-wise, that change
> would stop using udev auto-magic for assembling md arrays, but use
> /etc/mdadm.conf directly instead. I don't know the implementation details.
>
> After that I receive *no more errors* about either missing or duplicate
> PV and all VGs/LVs are displayed by the respective display commands. So
> for me issue is solved but if anyone wants to get to the bottom I am
> open to do some more experiments.
>
>
> Michal Svoboda
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

Re: [linux-lvm] No device found for PV but is actually fine

[Michal Svoboda]

Marian Csontos wrote:
> Oh yes, you would need to restart lvmetad (on Fedora/RHEL/CentOS
> systemctl restart lvm2-lvmetad.service, not sure it is the same on
> Arch.)

Thanks - yeah it's the same.

> You may want to scrub the RAID as there may be data not written to
> both legs.

Thanks for this suggestion too. I did that, found some small amount of
mismatch blocks. Don't know if they are linked to this problem but
better safe than sorry.

Michal Svoboda