[dm-crypt] Force luksClose

[dm-crypt] Force luksClose

[Dennis Birkholz]

Hello together,

I am using a crypt mapping on a Software-RAID5. As my raid crashed (2 of
5 drives disappeared), I was not able to remove the crypt mapping
because the raid device was not accessible any more. It would be nice to
have a force option so the mapping is removed and I can stop the
degraded raid. The only option was to reboot which is not a very good
solution.

Greets,
Dennis

Re: [dm-crypt] Force luksClose

[Milan Broz]

On 10/23/2012 06:51 AM, Dennis Birkholz wrote:
> I am using a crypt mapping on a Software-RAID5. As my raid crashed (2 of
> 5 drives disappeared), I was not able to remove the crypt mapping
> because the raid device was not accessible any more. It would be nice to
> have a force option so the mapping is removed and I can stop the
> degraded raid. The only option was to reboot which is not a very good
> solution.

The luksClose / remove should remove mapping if underlying device disappeared.

But there was some bug related to this - which cryptsetup version you are using?

Anyway, you can always use "dmsetup remove" and if it fails, "dmsetup remove -f"
(The second will replace mapping with error target, so you can force detach
even used device.)

Milan

Re: [dm-crypt] Force luksClose

[Dennis Birkholz]

Hello,

Am 25.10.2012 12:01, schrieb Milan Broz:
> On 10/23/2012 06:51 AM, Dennis Birkholz wrote:
>> I am using a crypt mapping on a Software-RAID5. As my raid crashed (2 of
>> 5 drives disappeared), I was not able to remove the crypt mapping
>> because the raid device was not accessible any more. It would be nice to
>> have a force option so the mapping is removed and I can stop the
>> degraded raid. The only option was to reboot which is not a very good
>> solution.
> 
> The luksClose / remove should remove mapping if underlying device disappeared.
> 
> But there was some bug related to this - which cryptsetup version you are using?
> 
> Anyway, you can always use "dmsetup remove" and if it fails, "dmsetup remove -f"
> (The second will replace mapping with error target, so you can force detach
> even used device.)

thanks for the information. I am using Debian Stable so it is cryptsetup
1.1.3, maybe I should compile me a more recent version.

Greets,
Dennis

[dm-crypt] force luksClose

[Benjamin Eberhardt]

Dear dm-crypt developers,

since a while I am using dm-crypt with luks on usb-attached external
harddisks. Sometimes the usb connection breaks down and the harddisks
get reconnected under different device names. It is then impossible to
unmount (the "regular" way) and luksClose the "old" device which is
still active. For unmounting i can use the "-l" "lazy" switch to kind
of force unmounting of the luks'ed filesystem even though it is still
"busy". A luksClose of the device although is impossible and the only
way i know so far is to reboot the machine. Is it possible to
implement some kind of lazy luksClose so that "dead" devices of which
the physical layer has dissappeared can be removed?

Thank you,
Benjamin

Re: [dm-crypt] force luksClose

[Matthias Schniedermeyer]

On 29.11.2012 14:02, Benjamin Eberhardt wrote:
> Dear dm-crypt developers,
> 
> since a while I am using dm-crypt with luks on usb-attached external
> harddisks. Sometimes the usb connection breaks down and the harddisks
> get reconnected under different device names. It is then impossible to
> unmount (the "regular" way) and luksClose the "old" device which is
> still active. For unmounting i can use the "-l" "lazy" switch to kind
> of force unmounting of the luks'ed filesystem even though it is still
> "busy". A luksClose of the device although is impossible and the only
> way i know so far is to reboot the machine. Is it possible to
> implement some kind of lazy luksClose so that "dead" devices of which
> the physical layer has dissappeared can be removed?

1: Details
2: Works For Me

When the partition is `umount`able, IOW no program is accessing a file 
on the mount and nothing has a CWD inside the mount. I can umount such a 
mount-point without problems. Linux "recognizes" somehow when the 
backing-store is vanished and just does the umount.
Afterwards the loopaesClose(in my Case) also just works.

Only when you have anything that holds the mountpoint open, there is a 
problem. But that's a case for fuser/lsof and/or making sure that 
nothing like NFS prevents the mountpoint from being destroyed.

IOW. It's a problem of finding the culprit who holds the mountpoint 
open.



-- 

Matthias

Re: [dm-crypt] force luksClose

[Milan Broz]

On 11/29/2012 02:43 PM, Matthias Schniedermeyer wrote:

> IOW. It's a problem of finding the culprit who holds the mountpoint 
> open.

Exactly. You cannot remove open (in-use) device-mapper device.

You can it replace with "error" target though (this will detach
underlying device from mapping.).

Just run "dmsetup remove -f <name>"
instead of cryptsetup remove/luksClose <name>"

But the dead DM device will still be in system. I can easily add such
force option to cryptsetup as well but it will not help much.

The correct way is to force unmout fs (or whatever use this device)
and then remove crypt mapping (see lsblk, lsof etc).

Milan

p.s.
There is a request to add "auto" removal flag for device-mapper devices
(automatic remove after last close, similar to loop auto removal).

Once this option will be in kernel cryptsetup will support it too.
But this will not help with your situation anyway.

Re: [dm-crypt] force luksClose

[Benjamin Eberhardt]

Hi,

thank you very much for your replies. I tried to reproduce the problem
which I had once in order to supply more details, but so far the
procedure you described works very well (identifying the culprit and
luksClose). Ill get back to you if the problem should occur again.

Merry Christmas :)
Benjamin


On 29 November 2012 15:53, Milan Broz <gmazyland@gmail.com> wrote:
> On 11/29/2012 02:43 PM, Matthias Schniedermeyer wrote:
>
>> IOW. It's a problem of finding the culprit who holds the mountpoint
>> open.
>
> Exactly. You cannot remove open (in-use) device-mapper device.
>
> You can it replace with "error" target though (this will detach
> underlying device from mapping.).
>
> Just run "dmsetup remove -f <name>"
> instead of cryptsetup remove/luksClose <name>"
>
> But the dead DM device will still be in system. I can easily add such
> force option to cryptsetup as well but it will not help much.
>
> The correct way is to force unmout fs (or whatever use this device)
> and then remove crypt mapping (see lsblk, lsof etc).
>
> Milan
>
> p.s.
> There is a request to add "auto" removal flag for device-mapper devices
> (automatic remove after last close, similar to loop auto removal).
>
> Once this option will be in kernel cryptsetup will support it too.
> But this will not help with your situation anyway.

Re: [dm-crypt] force luksClose

[Benjamin Eberhardt]

[-- Attachment #1: Type: text/plain, Size: 3158 bytes --]

Hi all,

the aboved mentioned problem happened to me again and this time i made sure
i would try to find every open file and close it first, then unmount the
partition but still luksClose did not work...

The partition is unmounted but still i get this error:

# cryptsetup luksClose sata_p1_2TBa
Device sata_p1_2TBa is busy.

no open files on *sata* davice (both device and mountpoint are called the
same)

# lsof |grep sata
#

blkid doesnt really help me either:

# lsblk
NAME                MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0                 7:0    0   512M  1 loop
└─cryptoswap (dm-0) 254:0    0   512M  0 crypt
mmcblk0             179:0    0  14,9G  0 disk
├─mmcblk0p1         179:1    0    72M  0 part
└─mmcblk0p2         179:2    0  14,9G  0 part  /
sdc                   8:32   0   1,8T  0 disk
sdd                   8:48   0   1,8T  0 disk
└─sdd1                8:49   0   1,8T  0 part

dmsetup force close also does not work:

# dmsetup remove -f sata_p1_2TBa
device-mapper: remove ioctl failed: Device or resource busy
Command failed

here you can see how the mapper device lost its physical device:

/dev/mapper/sata_p1_2TBa is active and is in use.
  type:    LUKS1
  cipher:  -
  keysize: 0 bits
  device:  (null)

How can i identify which process still keeps the device open? Why is still
happening allthough the filesystem is already unmounted? And last, would it
not be better to associate the mapper devices with the uuid of the physical
device? Then it could theoretically work again after the usb disk has
reconnected?

Thank you,
Benjamin



On 25 December 2012 13:17, Benjamin Eberhardt <eberhab@gmail.com> wrote:
>
> Hi,
>
> thank you very much for your replies. I tried to reproduce the problem
> which I had once in order to supply more details, but so far the
> procedure you described works very well (identifying the culprit and
> luksClose). Ill get back to you if the problem should occur again.
>
> Merry Christmas :)
> Benjamin
>
>
> On 29 November 2012 15:53, Milan Broz <gmazyland@gmail.com> wrote:
> > On 11/29/2012 02:43 PM, Matthias Schniedermeyer wrote:
> >
> >> IOW. It's a problem of finding the culprit who holds the mountpoint
> >> open.
> >
> > Exactly. You cannot remove open (in-use) device-mapper device.
> >
> > You can it replace with "error" target though (this will detach
> > underlying device from mapping.).
> >
> > Just run "dmsetup remove -f <name>"
> > instead of cryptsetup remove/luksClose <name>"
> >
> > But the dead DM device will still be in system. I can easily add such
> > force option to cryptsetup as well but it will not help much.
> >
> > The correct way is to force unmout fs (or whatever use this device)
> > and then remove crypt mapping (see lsblk, lsof etc).
> >
> > Milan
> >
> > p.s.
> > There is a request to add "auto" removal flag for device-mapper devices
> > (automatic remove after last close, similar to loop auto removal).
> >
> > Once this option will be in kernel cryptsetup will support it too.
> > But this will not help with your situation anyway.

[-- Attachment #2: Type: text/html, Size: 4052 bytes --]

Re: [dm-crypt] force luksClose

[Matthias Schniedermeyer]

On 21.02.2013 22:16, Benjamin Eberhardt wrote:
> Hi all,
> 
> the aboved mentioned problem happened to me again and this time i made sure
> i would try to find every open file and close it first, then unmount the
> partition but still luksClose did not work...
> 
> The partition is unmounted but still i get this error:
> 
> # cryptsetup luksClose sata_p1_2TBa
> Device sata_p1_2TBa is busy.
> 
> no open files on *sata* davice (both device and mountpoint are called the
> same)

From my personal experience the things that DON'T show up in lsof are:
mountpoints (and bind-mounts)
nfs

So if there is a mountpoint inside, you have to umount that first. If 
you have a NFS-export of the mountpoint or something inside the 
mountpoint you have to remove that export or stop NFS for the moment it 
takes to do the umount.

There may be other things that don't show up in lsof, but i don't know 
about them.



-- 

Matthias

Re: [dm-crypt] force luksClose

[Milan Broz]

On 21.2.2013 22:16, Benjamin Eberhardt wrote:
>
> # cryptsetup luksClose sata_p1_2TBa
> Device sata_p1_2TBa is busy.
>
> no open files on *sata* davice (both device and mountpoint are called the same)
>
> # lsof |grep sata
> #
>
> blkid doesnt really help me either:
>
> # lsblk

There must be something wrong with that device,
it must show in lsblk output (but it is not there).

Seems like it is partially removed (empty table) - can you add
output of

dmsetup table sata_p1_2TBa
dmsetup info -c sata_p1_2TBa
blockdev --getsz /dev/mapper/sata_p1_2TBa ?

(btw another mapping will not show in lsof, but it should
be visible in lsblk. If it is reproducible I would like to at least
fix lsblk so we have some reliable tool to debug it...)

Milan

Re: [dm-crypt] force luksClose

[Benjamin Eberhardt]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]

Thank you all much for your replies. I rebooted then but will perform your
commands and mail the output when it happens again.

Cheers,
Benjamin



On 21 February 2013 23:41, Milan Broz <gmazyland@gmail.com> wrote:

> On 21.2.2013 22:16, Benjamin Eberhardt wrote:
>
>>
>> # cryptsetup luksClose sata_p1_2TBa
>> Device sata_p1_2TBa is busy.
>>
>> no open files on *sata* davice (both device and mountpoint are called the
>> same)
>>
>> # lsof |grep sata
>> #
>>
>> blkid doesnt really help me either:
>>
>> # lsblk
>>
>
> There must be something wrong with that device,
> it must show in lsblk output (but it is not there).
>
> Seems like it is partially removed (empty table) - can you add
> output of
>
> dmsetup table sata_p1_2TBa
> dmsetup info -c sata_p1_2TBa
> blockdev --getsz /dev/mapper/sata_p1_2TBa ?
>
> (btw another mapping will not show in lsof, but it should
> be visible in lsblk. If it is reproducible I would like to at least
> fix lsblk so we have some reliable tool to debug it...)
>
> Milan
>

[-- Attachment #2: Type: text/html, Size: 1652 bytes --]

[dm-crypt] force luksClose

[yac]

Hello

I believe I have the same problem as described here:
http://comments.gmane.org/gmane.linux.kernel.device-mapper.dm-crypt/6250

Here are the last requested commands:

# dmsetup table dlt
0 4096 crypt aes-xts-plain64 0000000000000000000000000000000000000000000000000000000000000000 0 254:8 4096
# dmsetup info -c dlt
Name             Maj Min Stat Open Targ Event  UUID                                            
dlt              254   9 L--w    1    1      0 CRYPT-LUKS1-731df69c78e94882982899390c71bb3b-dlt
# lsblk 
lsblk: dm-0: failed to get device path
lsblk: dm-0: failed to get device path
NAME                MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                   8:0    0  1.8T  0 disk  
└─sda1                8:1    0  1.8T  0 part  
sdb                   8:16   0  2.7T  0 disk  
└─sdb1                8:17   0  2.7T  0 part  
sdc                   8:32   0  2.7T  0 disk  
└─sdc1                8:33   0  2.7T  0 part  
sdd                   8:48   0  2.7T  0 disk  
└─sdd1                8:49   0  2.7T  0 part  
sde                   8:64   0  2.7T  0 disk  
└─sde1                8:65   0  2.7T  0 part  
sdf                   8:80   0  1.8T  0 disk  
sdg                   8:96   1  3.8G  0 disk  
└─sdg1                8:97   1  3.8G  0 part  
loop8                 7:8    0   10M  0 loop  
├─lt-lvol0_rmeta_0  254:4    0    4M  0 lvm   
│ └─lt-lvol0        254:8    0    4M  0 lvm   
│   └─dlt           254:9    0    2M  0 crypt /root/lvm-crypt-test/mnt
└─lt-lvol0_rimage_0 254:5    0   12M  0 lvm   
  └─lt-lvol0        254:8    0    4M  0 lvm   
    └─dlt           254:9    0    2M  0 crypt /root/lvm-crypt-test/mnt
loop9                 7:9    0   10M  0 loop  
├─lt-lvol0_rmeta_1  254:6    0    4M  0 lvm   
│ └─lt-lvol0        254:8    0    4M  0 lvm   
│   └─dlt           254:9    0    2M  0 crypt /root/lvm-crypt-test/mnt
└─lt-lvol0_rimage_1 254:7    0   12M  0 lvm   
  └─lt-lvol0        254:8    0    4M  0 lvm   
    └─dlt           254:9    0    2M  0 crypt /root/lvm-crypt-test/mnt
loop10                7:10   0   10M  0 loop  
└─lt-lvol0_rimage_0 254:5    0   12M  0 lvm   
  └─lt-lvol0        254:8    0    4M  0 lvm   
    └─dlt           254:9    0    2M  0 crypt /root/lvm-crypt-test/mnt
loop11                7:11   0   10M  0 loop  
└─lt-lvol0_rimage_1 254:7    0   12M  0 lvm   
  └─lt-lvol0        254:8    0    4M  0 lvm   
    └─dlt           254:9    0    2M  0 crypt /root/lvm-crypt-test/mnt



This happened to me after I was trying out lvm and then crypt on top
lvm. First, let's have a workspace

# cd /root; mkdir lvm-crypt-test; cd lvm-crypt-test

Fake some block devs

# mknod /dev/lta b 7 8
# mknod /dev/ltb b 7 9
# mknod /dev/ltc b 7 10 
# mknod /dev/ltd b 7 11
# for i in a b c d; do fallocate -l 10M sd$i; losetup /dev/lt$i sd$i; done

Make an LVM raid1

# pvcreate /dev/lt{a,b,c,d}
# vgcreate lt /dev/lt{a,b,c,d}
# lvcreate -m1 -l 100%VG lt /dev/lt{a,b}

Encrypt it

# echo "aaa" > key
# cryptsetup luksFormat /dev/lt/lvol0 -d key

Format and mount

# cryptsetup luksOpen /dev/lt/lvol0 -d key dlt
# mkfs.ext4 /dev/mapper/dlt
# mkdir ./mnt; mount /dev/mapper/dlt ./mnt

Now comes the part where I broke it

# lvextend /dev/lt/lvol0 /dev/lt{c,d} -l 100%VG

I tried to umount mnt but it got stuck in uninterruptible sleep. But
umount -l worked for me.

# cat /proc/mounts  | grep dlt
# cat /proc/mounts  | grep mnt
# 

But I am not able to anything about dlt and the underlying devices. Not
even turn the system off normally as that gets stuck on sync().

# ps auxf | grep D
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     15193  0.0  0.0  27488  4436 ?        D    01:03   0:00 lvs
root     15196  0.0  0.0  27472  4432 ?        D    01:04   0:00 lvdisplay
root     15201  0.0  0.0  12612  2300 ?        D    01:04   0:00 umount mnt/
root     15237  0.0  0.0  12564  1740 ?        D    01:06   0:00 shutdown -h 0 w
root     15372  0.0  0.0   5840   748 ?        D    01:12   0:00 sync
root     17287  0.0  0.0   4016   640 ?        D    01:50   0:00 /sbin/reboot -f
root     17296  0.0  0.0  12576  2512 ?        D    01:51   0:00 dmsetup remove -f dlt
root     25471  0.0  0.0  27480  4532 ?        D    02:05   0:00 lvchange -an /dev/lt/lvol0
root     25529  0.0  0.0  24008  5576 ?        D<L  02:15   0:00 cryptsetup luksClose dlt
root     26114  0.0  0.0  24008  5576 ?        D<L  03:21   0:00 cryptsetup loopaesClose dlt

# lsof /root/lvm-crypt-test/mnt/
# fuser /root/lvm-crypt-test/mnt/
# find /proc/ -maxdepth 2 -type l -name cwd -exec readlink {} \; | sort | uniq
/
/proc/26289
/root
/root/lvm-crypt-test
/sys


Luckily I was able to umount and luksClose all the real block devices
there and now it appears my only option is to go pull the plug.

Best regards
Jan Matejka