[linux-lvm] pvremove won't work

[linux-lvm] pvremove won't work

[Stefan Radovanovici]

Hi All, 

I have a LVM over 3 partitions (3 separate disks) and one of them is failing so I need to remove it from the group. I moved the content to the rest of the disks (pvmove) and now pvdisplay shows:

root@neriak:~# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               storage
  PV Size               186.31 GB / not usable 4.00 MB
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              47694
  Free PE               0
  Allocated PE          47694
  PV UUID               QUmigK-SCow-nMKo-NEcC-cOQY-bAO9-3k6g9K

  --- Physical volume ---
  PV Name               /dev/sdc1
  VG Name               storage
  PV Size               149.05 GB / not usable 1.31 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              38156
  Free PE               19483
  Allocated PE          18673
  PV UUID               idHUfs-rd5i-fVf7-UzE0-YGJT-X1k4-mrp3be

  "/dev/sda4" is a new physical volume of "200.87 GB"
  --- NEW Physical volume ---
  PV Name               /dev/sda4
  VG Name
  PV Size               200.87 GB
  Allocatable           NO
  PE Size (KByte)       0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               TNsOa7-2Ibb-IW9i-C1s5-1NI9-L5e3-WlKnSH


However, I can't remove /dev/sda4 (the partition on the failing disk) from the group: 

root@neriak:~# pvremove /dev/sda4
  Can't open /dev/sda4 exclusively - not removing. Mounted filesystem?

I don't see it mounted anywhere. The LVM is also unmounted. Any ideas of what I might be doing wrong ? I am running LVM 2.02.39 on Ubuntu 9.04 64bit

Thanks in advance,
Stefan

Re: [linux-lvm] pvremove won't work

[Luca Berra]

On Tue, Jun 09, 2009 at 08:39:18AM +0200, Stefan Radovanovici wrote:
>Hi All, 
>
>I have a LVM over 3 partitions (3 separate disks) and one of them is failing so I need to remove it from the group. I moved the content to the rest of the disks (pvmove) and now pvdisplay shows:
....
>However, I can't remove /dev/sda4 (the partition on the failing disk) from the group: 
>
>root@neriak:~# pvremove /dev/sda4
>  Can't open /dev/sda4 exclusively - not removing. Mounted filesystem?
what is the use of doing pvremove on sda4?? you just rip the disk off
and dump it in the wastebin.

to remove a pv from a vg you use vgreduce, but i assume you already did
that, judging from the pvdisplay.

>I don't see it mounted anywhere. The LVM is also unmounted. Any ideas of what I might be doing wrong ? I am running LVM 2.02.39 on Ubuntu 9.04 64bit

check /sys/block/sda/sda4/holders

-- 
Luca Berra -- bluca@comedia.it
         Communication Media & Services S.r.l.
  /"\
  \ /     ASCII RIBBON CAMPAIGN
   X        AGAINST HTML MAIL
  / \

Re: [linux-lvm] pvremove won't work

[Stefan Radovanovici]

On 6/9/2009 8:54 AM, Luca Berra wrote:
>> However, I can't remove /dev/sda4 (the partition on the failing disk)
>> from the group:
>> root@neriak:~# pvremove /dev/sda4
>>  Can't open /dev/sda4 exclusively - not removing. Mounted filesystem?
> what is the use of doing pvremove on sda4?? you just rip the disk off
> and dump it in the wastebin.

Well, yeah, I just wanted to make sure that removing the disk while some of its partitions are part of the LVM won't cause any problems. I can't remove it just yet anyway, the OS is also installed on other partitions on sda. So basically as long as I did a vgreduce, I can take the disk out, is that right? I am assuming /dev/sda4 is still showing in the pvdisplay output because it's still a LVM-type partition. 

> check /sys/block/sda/sda4/holders

I did: 

root@neriak:/sys/block/sda/sda4/holders# l
total 0
drwxr-xr-x 2 root root 0 2009-06-09 08:55 .
drwxr-xr-x 4 root root 0 2009-06-09 08:43 ..
lrwxrwxrwx 1 root root 0 2009-06-09 08:55 dm-1 -> ../../../../../../../../../virtual/block/dm-1
root@neriak:/sys/block/sda/sda4/holders# cd dm-1
root@neriak:/sys/block/sda/sda4/holders/dm-1# l
total 0
drwxr-xr-x  5 root root    0 2009-06-09 08:43 .
drwxr-xr-x 28 root root    0 2009-06-09 08:30 ..
lrwxrwxrwx  1 root root    0 2009-06-09 08:55 bdi -> ../../bdi/252:1
-r--r--r--  1 root root 4096 2009-06-09 08:55 capability
-r--r--r--  1 root root 4096 2009-06-09 08:55 dev
-r--r--r--  1 root root 4096 2009-06-09 08:55 ext_range
drwxr-xr-x  2 root root    0 2009-06-09 08:55 holders
drwxr-xr-x  2 root root    0 2009-06-09 08:55 power
-r--r--r--  1 root root 4096 2009-06-09 08:55 range
-r--r--r--  1 root root 4096 2009-06-09 08:55 removable
-r--r--r--  1 root root 4096 2009-06-09 08:55 ro
-r--r--r--  1 root root 4096 2009-06-09 08:55 size
drwxr-xr-x  2 root root    0 2009-06-09 08:55 slaves
-r--r--r--  1 root root 4096 2009-06-09 08:55 stat
lrwxrwxrwx  1 root root    0 2009-06-09 08:55 subsystem -> ../../../../class/block
-rw-r--r--  1 root root 4096 2009-06-09 08:43 uevent

I have to admit I don't know what I am looking at. What's the dm-1 ?

Regards,
Stefan

Re: [linux-lvm] pvremove won't work

[Bryn M. Reeves]

On Tue, 2009-06-09 at 09:04 +0200, Stefan Radovanovici wrote:
> 
> On 6/9/2009 8:54 AM, Luca Berra wrote:
> >> However, I can't remove /dev/sda4 (the partition on the failing disk)
> >> from the group:
> >> root@neriak:~# pvremove /dev/sda4
> >>  Can't open /dev/sda4 exclusively - not removing. Mounted filesystem?
> > what is the use of doing pvremove on sda4?? you just rip the disk off
> > and dump it in the wastebin.
> 
> Well, yeah, I just wanted to make sure that removing the disk while
>  some of its partitions are part of the LVM won't cause any problems. I
>  can't remove it just yet anyway, the OS is also installed on other
>  partitions on sda. So basically as long as I did a vgreduce, I can
>  take the disk out, is that right? I am assuming /dev/sda4 is still
>  showing in the pvdisplay output because it's still a LVM-type
>  partition. 

It is still labeled. LVM2 PV metadata has two components; a label in the
2nd sector that identifies the device as an LVM2 PV and includes the
UUID and a metadata buffer further into the device that stores details
like the VG the PV belongs to and the LVs that are allocated.

Running vgreduce for a PV edits the metadata on all PVs in the
appropriate VG to remove references to the PV being removed.

The pvremove command just wipes the label from the PV.

> > check /sys/block/sda/sda4/holders
> 
> I did: 
> 
> root@neriak:/sys/block/sda/sda4/holders# l
> total 0
> drwxr-xr-x 2 root root 0 2009-06-09 08:55 .
> drwxr-xr-x 4 root root 0 2009-06-09 08:43 ..
> lrwxrwxrwx 1 root root 0 2009-06-09 08:55 dm-1 -> ../../../../../../../../../virtual/block/dm-1
> root@neriak:/sys/block/sda/sda4/holders# cd dm-1
> root@neriak:/sys/block/sda/sda4/holders/dm-1# l
> total 0

> lrwxrwxrwx  1 root root    0 2009-06-09 08:55 bdi -> ../../bdi/252:1

> I have to admit I don't know what I am looking at. What's the dm-1 ?

Use dmsetup:

$ dmsetup info --maj 252 --min 1

(the output above shows dm is using 252 as its major number for this
device on your system).

At a guess it looks like you somehow removed the PV from the VG while a
logical volume was mapped on it.

Regards,
Bryn.

Re: [linux-lvm] pvremove won't work

[Stefan Radovanovici]

On 6/9/2009 11:46 AM, Bryn M. Reeves wrote:

> Use dmsetup:
> 
> $ dmsetup info --maj 252 --min 1
> 
> (the output above shows dm is using 252 as its major number for this
> device on your system).
> 
> At a guess it looks like you somehow removed the PV from the VG while a
> logical volume was mapped on it.

First of all, thanks for the info. I am not a LVM expert and this is useful information. dmsetup shows this:

root@neriak:/sys/block/sda/sda4/holders/dm-1# dmsetup info --maj 252 --min 1
Name:              storage-pvmove0
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        0
Event number:      1
Major, minor:      252, 1
Number of targets: 1
UUID: LVM-SS2wO5K12BwSo1iceXVSiTaYPQzdgEuUoOPgGYKKRLO109x4837eAHZeiSmzzEqW

I think I somehow messed up while trying to get rid of /dev/sda4. I used a GUI for this not the CLI, first I tried to move the extents to other PVs (which I think is that pvmove does) and then to actually remove it from the group (which is probably what vgreduce does). If I am not mistaken, the first operation gave an error even though it appeared to have worked somehow at the time.

Any way I can fix this ?

Thanks,
Stefan

Re: [linux-lvm] pvremove won't work

[Luca Berra]

On Tue, Jun 09, 2009 at 12:25:10PM +0200, Stefan Radovanovici wrote:
>
>On 6/9/2009 11:46 AM, Bryn M. Reeves wrote:
>
>> Use dmsetup:
>> 
>> $ dmsetup info --maj 252 --min 1
>> 
>> (the output above shows dm is using 252 as its major number for this
>> device on your system).
>> 
>> At a guess it looks like you somehow removed the PV from the VG while a
>> logical volume was mapped on it.
>
>First of all, thanks for the info. I am not a LVM expert and this is useful information. dmsetup shows this:
>
>root@neriak:/sys/block/sda/sda4/holders/dm-1# dmsetup info --maj 252 --min 1
>Name:              storage-pvmove0
>State:             ACTIVE
>Read Ahead:        256
>Tables present:    LIVE
>Open count:        0
>Event number:      1
>Major, minor:      252, 1
>Number of targets: 1
>UUID: LVM-SS2wO5K12BwSo1iceXVSiTaYPQzdgEuUoOPgGYKKRLO109x4837eAHZeiSmzzEqW
>
>I think I somehow messed up while trying to get rid of /dev/sda4. I used a GUI for this not the CLI, first I tried to move the extents to other PVs (which I think is that pvmove does) and then to actually remove it from the group (which is probably what vgreduce does). If I am not mistaken, the first operation gave an error even though it appeared to have worked somehow at the time.
>
it seems to be some leftover from the pvmove operation, unfortunately i
have no idea if some data was not actually moved correctly.
did you record the error message the pvmove operation reported?

Regards,
L.

-- 
Luca Berra -- bluca@comedia.it
         Communication Media & Services S.r.l.
  /"\
  \ /     ASCII RIBBON CAMPAIGN
   X        AGAINST HTML MAIL
  / \

Re: [linux-lvm] pvremove won't work

[Stefan Radovanovici]

On 6/10/2009 7:38 AM, Luca Berra wrote:
>> I think I somehow messed up while trying to get rid of /dev/sda4. I
>> used a GUI for this not the CLI, first I tried to move the extents to
>> other PVs (which I think is that pvmove does) and then to actually
>> remove it from the group (which is probably what vgreduce does). If I
>> am not mistaken, the first operation gave an error even though it
>> appeared to have worked somehow at the time.
>>
> it seems to be some leftover from the pvmove operation, unfortunately i
> have no idea if some data was not actually moved correctly.
> did you record the error message the pvmove operation reported?

No, unfortunately not, it was just a popup in that GUI. The LVM is mounted now, with only the 2 remaining partitions as PVs, and so far the data seems to be in order so maybe pvmove worked. I guess I will live it at that and when the time comes, I will just remove the failing disk from the system.

Regards,
Stefan