[linux-lvm] Device mapper does not really remove the references to removed LV.

[linux-lvm] Device mapper does not really remove the references to removed LV.

[Germain Maurice]

Hi everybody,

I'm facing to some problems in my virtualization cluster and i need some help to prevent it occurring again.

I know that LVM is not cluster aware and we have to use it with lot of caution.
However, i have a shared LVM storage between 6 nodes. I use KVM hosts.

Two days ago i rebooted my whole cluster in order to get the same LVM metadata and the same dmsetup table over the cluster. That's ok.

"vgck SATA6To" is ok.

Right now, I have a problem when deleting an LV.

On the node where the VM lives, i did the lvremove command (lv name : SATA6To/vm-306-disk-3), i have a good state of LVM :

node5:~# lvdisplay -C | grep -- "-306"
  vm-306-disk-1 SATA6To     -wi-ao  10.00G                                      
  vm-306-disk-2 SATA6To     -wi-ao   5.00G     
node5:~# dmsetup table | grep -- "-306"
SATA6To-vm--306--disk--1: 0 20971520 linear 152:16 2231370112
SATA6To-vm--306--disk--2: 0 10485760 linear 152:16 2252341632
node5:~# ls /dev/SATA6To/vm-306-disk-*
/dev/SATA6To/vm-306-disk-1  /dev/SATA6To/vm-306-disk-2
node5:~# ls /dev/mapper/SATA6To-vm--306--disk--*
/dev/mapper/SATA6To-vm--306--disk--1  /dev/mapper/SATA6To-vm--306--disk--2


If i did a "vgscan or lvscan" on the rest of the cluster, LVM well sees that the LV has been removed, however device-mapper still have devices for it :

node3:~# lvdisplay -C | grep -- "-306"
  vm-306-disk-1 SATA6To     -wi-a-  10.00G                                      
  vm-306-disk-2 SATA6To     -wi-a-   5.00G
node3:~# dmsetup table  | grep -- "-306"
SATA6To-vm--306--disk--1: 0 20971520 linear 152:32 2231370112
SATA6To-vm--306--disk--3: 0 8388608 linear 152:32 2262827392
SATA6To-vm--306--disk--2: 0 10485760 linear 152:32 2252341632
node3:~# ls /dev/SATA6To/vm-306-disk-*
/dev/SATA6To/vm-306-disk-1  /dev/SATA6To/vm-306-disk-2	/dev/SATA6To/vm-306-disk-3
node3:~# ls /dev/mapper/SATA6To-vm--306--disk--*
/dev/mapper/SATA6To-vm--306--disk--1  /dev/mapper/SATA6To-vm--306--disk--3
/dev/mapper/SATA6To-vm--306--disk--2


Any idea how to clear all references to the LV previously removed ?

And by the way, the good way to do the best practices when i change something in LVM and to take it in account in the other nodes ?

Is it advised to disable lvm cache (/etc/lvm/cache/.cache) ?

Thank you for your precious help !
Germain

Re: [linux-lvm] Device mapper does not really remove the references to removed LV.

[Zdenek Kabelac]

Dne 14.5.2012 19:37, Germain Maurice napsal(a):
> Hi everybody,
>
> I'm facing to some problems in my virtualization cluster and i need some help to prevent it occurring again.
>
> I know that LVM is not cluster aware and we have to use it with lot of caution.
> However, i have a shared LVM storage between 6 nodes. I use KVM hosts.

Hmm - if you are planing to build your own cluster - assuming you do not want 
to use lvm2 cluster support tools like clmvd - which needs advanced 
configuration) - you will need to write your own daemon, to watch out for 
locks and metadata changes.

>
> Two days ago i rebooted my whole cluster in order to get the same LVM metadata and the same dmsetup table over the cluster. That's ok.
>
> "vgck SATA6To" is ok.
>
> Right now, I have a problem when deleting an LV.
>
> On the node where the VM lives, i did the lvremove command (lv name : SATA6To/vm-306-disk-3), i have a good state of LVM :

You need to propagate such change around your cluster by your daemon - and you 
have to be quite carefully where have you activated such LV before doing lvremove.

lvremove is simple command which does not communicate with other nodes in your 
cluster to let them know something has changed - so you might find some 
inspiration in CLVMD and it's locking technique to distribute locks around the 
cluster.

>
> Any idea how to clear all references to the LV previously removed ?
>

Before you remove LV on any node - you have to make sure all nodes deactivated 
such LV.

IMHO I'd suggest to use clvmd - instead of programming new daemon...

Zdenek

Re: [linux-lvm] Device mapper does not really remove the references to removed LV.

[Germain Maurice]

Thanks for your answer Zdenek,

My cluster is in production and i know the virtualization solution we use is not optimal.
I will not programming a new daemon, but maybe a script that just do the job. The best way
to avoid any problem is to change my solution or deploy clvmd, i know.

For now, I just have to know how to proceed to avoid any problems.

So, before removing an LV, I think I have to do this :
- stopping the VM
- on every nodes : "lvchange -an /dev/VG/LV-to-remove"
- on one node : "lvremove /dev/VG/LV-to-remove"
- "vgscan" on every nodes.

And for creating :
- on the same node :
   - "lvcreate � /dev/VG/new-LV" 
   - "lvchange -ay  /dev/VG/new-LV" or "vgchange -ay"
- on the other nodes : 
   - "vgchange -ay"
- starting the VM on one node

For any LV modification :
- stopping the VM
- on every nodes : "lvchange -an /dev/VG/LV-to-remove"
- on one node : "lvmodify /dev/VG/LV-to-modify" and "lvchange -ay  /dev/VG/new-LV"
- "vgscan" on every nodes.
- starting the VM on one node (doing some stuff inside the VM)

Do you agree with that ?

I can do it manually because i don't have so much deployments to do.


Thank you for your help.



Le 15 mai 2012 � 10:12, Zdenek Kabelac a �crit :

> Dne 14.5.2012 19:37, Germain Maurice napsal(a):
>> Hi everybody,
>> 
>> I'm facing to some problems in my virtualization cluster and i need some help to prevent it occurring again.
>> 
>> I know that LVM is not cluster aware and we have to use it with lot of caution.
>> However, i have a shared LVM storage between 6 nodes. I use KVM hosts.
> 
> Hmm - if you are planing to build your own cluster - assuming you do not want to use lvm2 cluster support tools like clmvd - which needs advanced configuration) - you will need to write your own daemon, to watch out for locks and metadata changes.
> 
>> 
>> Two days ago i rebooted my whole cluster in order to get the same LVM metadata and the same dmsetup table over the cluster. That's ok.
>> 
>> "vgck SATA6To" is ok.
>> 
>> Right now, I have a problem when deleting an LV.
>> 
>> On the node where the VM lives, i did the lvremove command (lv name : SATA6To/vm-306-disk-3), i have a good state of LVM :
> 
> You need to propagate such change around your cluster by your daemon - and you have to be quite carefully where have you activated such LV before doing lvremove.
> 
> lvremove is simple command which does not communicate with other nodes in your cluster to let them know something has changed - so you might find some inspiration in CLVMD and it's locking technique to distribute locks around the cluster.
> 
>> 
>> Any idea how to clear all references to the LV previously removed ?
>> 
> 
> Before you remove LV on any node - you have to make sure all nodes deactivated such LV.
> 
> IMHO I'd suggest to use clvmd - instead of programming new daemon...
> 
> Zdenek
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/