How do you force-close a dm device after a disk failure?

How do you force-close a dm device after a disk failure?

[Adam Nielsen]

Hi all,

I'm trying to work out how to force-remove a dm device after the
underlying disks failed.  Because the kernel is still trying to write
blocks to the failed devices, I can't unmount the filesystem, so I
don't seem to be able to just drop the data, get some errors, then have
everything unmounted and clean, ready to try again.

  $ dmsetup remove backup
  device-mapper: remove ioctl on backup failed: Device or resource busy
  Command failed

Using --force says it will replace the device with one that returns I/O
errors which sounds perfect, but that just locks up:

  $ dmsetup remove --force backup
  ^C^C^\^\

Running it through strace shows it getting stuck in the kernel:

  $ strace dmsetup remove --force backup
  [...]
  access("/run/udev/control", F_OK)       = 0
  open("/dev/urandom", O_RDONLY)          = 4
  read(4, "\2\325", 2)                    = 2
  semget(0xd4dd502, 1, IPC_CREAT|IPC_EXCL|0600) = 458754
  semctl(458754, 0, SETVAL, 0x1)          = 0
  semctl(458754, 0, GETVAL, 0x7f254875c47a) = 1
  close(4)                                = 0
  semop(458754, {{0, 1, 0}}, 1)           = 0
  semctl(458754, 0, GETVAL, 0x7f254875c417) = 2
  ioctl(3, DM_DEV_SUSPEND

Is there any way to forcefully remove a dm device, or do you have to
reboot to remove failed devices?

I'm running kernel 4.1.4.

Many thanks,
Adam.

Re: How do you force-close a dm device after a disk failure?

[Zdenek Kabelac]

Dne 14.9.2015 v 02:29 Adam Nielsen napsal(a):
> Hi all,
>
> I'm trying to work out how to force-remove a dm device after the
> underlying disks failed.  Because the kernel is still trying to write
> blocks to the failed devices, I can't unmount the filesystem, so I
> don't seem to be able to just drop the data, get some errors, then have
> everything unmounted and clean, ready to try again.
>
>    $ dmsetup remove backup
>    device-mapper: remove ioctl on backup failed: Device or resource busy
>    Command failed
>
> Using --force says it will replace the device with one that returns I/O
> errors which sounds perfect, but that just locks up:
>
>    $ dmsetup remove --force backup
>    ^C^C^\^\
>
> Running it through strace shows it getting stuck in the kernel:
>
>    $ strace dmsetup remove --force backup
>    [...]
>    access("/run/udev/control", F_OK)       = 0
>    open("/dev/urandom", O_RDONLY)          = 4
>    read(4, "\2\325", 2)                    = 2
>    semget(0xd4dd502, 1, IPC_CREAT|IPC_EXCL|0600) = 458754
>    semctl(458754, 0, SETVAL, 0x1)          = 0
>    semctl(458754, 0, GETVAL, 0x7f254875c47a) = 1
>    close(4)                                = 0
>    semop(458754, {{0, 1, 0}}, 1)           = 0
>    semctl(458754, 0, GETVAL, 0x7f254875c417) = 2
>    ioctl(3, DM_DEV_SUSPEND
>
> Is there any way to forcefully remove a dm device, or do you have to
> reboot to remove failed devices?
>
> I'm running kernel 4.1.4.


You need to show your 'broken' table first.
i.e. you can't replace some targets in some cases (I think suspended thin-pool
with broken metadata cannot be replaced with error target)

However for the 'standard' linear device - you could always replace  opened 
device with error target with '--force'.

What you can't expect is  'device deletion' - as long as device is opened,
it will be present (even if all read & writes will be errored).

As long as kernel structures are in use - device remains present.

Also note - dmsetup remove  supports --deferred removal (see man page).

Zdenek

Re: How do you force-close a dm device after a disk failure?

[Adam Nielsen]

Thanks for your response!

> You need to show your 'broken' table first.

$ dmsetup table
backup: 0 11720531968 crypt aes-xts-plain64
  0000000000000000000000000000000000000000000000000000000000000000 0
  9:10 4096

> However for the 'standard' linear device - you could always replace
> opened device with error target with '--force'.

I'm not sure how to do this, could you please elaborate?  I thought
"dmsetup remove --force" would do this but as that doesn't work, I tried
this also:

$ dmsetup wipe_table backup
<no response, dmsetup frozen, kill -9 doesn't work>

$ dmsetup reload backup --table "0 11720531968 error"
$ dmsetup info backup
Name:              backup
State:             ACTIVE
Read Ahead:        4096
Tables present:    LIVE & INACTIVE
Open count:        1
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: CRYPT-LUKS1-d0b3d38e421545908537dc50f59fb217-backup
$ dmsetup resume backup
<no response, dmsetup frozen, kill -9 doesn't work>

> What you can't expect is  'device deletion' - as long as device is
> opened, it will be present (even if all read & writes will be
> errored).

That's probably ok, what I'm really trying to do is to get the
underlying device released.

> Also note - dmsetup remove  supports --deferred removal (see man
> page).

Oh I didn't notice that.  It doesn't seem to have much of an effect
though:

$ dmsetup remove --deferred backup
$ dmsetup info
Name:              backup
State:             ACTIVE (DEFERRED REMOVE)
Read Ahead:        4096
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: CRYPT-LUKS1-d0b3d38e421545908537dc50f59fb217-backup

The underlying device is still in use by dm-crypt.  I could not unmount
the filesystem stored on the dm device as the kernel said it was still
in use (presumably by the pending writes), so I did a lazy unmount of
the filesystem.  I think this is what's holding the dm device open and
causing dmsetup to stop responding all the time.

Any suggestions how to proceed?

Thanks again,
Adam.

Re: How do you force-close a dm device after a disk failure?

[Zdenek Kabelac]

Dne 14.9.2015 v 10:59 Adam Nielsen napsal(a):
> Thanks for your response!
>
>> You need to show your 'broken' table first.
>
> $ dmsetup table
> backup: 0 11720531968 crypt aes-xts-plain64
>    0000000000000000000000000000000000000000000000000000000000000000 0
>    9:10 4096

It's not really useful to show just 1 device.

Whole dm  table with all deps needs to be known.

'dmsetup table'
'dmsetup status'
'dmsetup ls --tree'
'dmsetup into -f'

>> However for the 'standard' linear device - you could always replace
>> opened device with error target with '--force'.
>
> I'm not sure how to do this, could you please elaborate?  I thought
> "dmsetup remove --force" would do this but as that doesn't work, I tried
> this also:
>
> $ dmsetup reload backup --table "0 11720531968 error"
> $ dmsetup info backup
> Name:              backup
> State:             ACTIVE
> Read Ahead:        4096
> Tables present:    LIVE & INACTIVE
> Open count:        1
> Event number:      0
> Major, minor:      253, 0
> Number of targets: 1
> UUID: CRYPT-LUKS1-d0b3d38e421545908537dc50f59fb217-backup
> $ dmsetup resume backup
> <no response, dmsetup frozen, kill -9 doesn't work>

really state of whole table needs to be known.

>> Also note - dmsetup remove  supports --deferred removal (see man
>> page).
>
> Oh I didn't notice that.  It doesn't seem to have much of an effect
> though:

Sure it will not fix your problem - it's like lazy umount...

>
> The underlying device is still in use by dm-crypt.  I could not unmount
> the filesystem stored on the dm device as the kernel said it was still
> in use (presumably by the pending writes), so I did a lazy unmount of
> the filesystem.  I think this is what's holding the dm device open and
> causing dmsetup to stop responding all the time.
>
> Any suggestions how to proceed?

What is not clear to me is - what is your expectation here ?
Obviously your system is far more broken - so placing 'error' target
for your backup device will not fix it.

You should likely attach also portion of 'dmesg' - there surely will be 
written what is going wrong with your system.

i.e. you cannot expect 'remove --force' will work when your machine start to 
show kernel errors.

Zdenek

Re: How do you force-close a dm device after a disk failure?

[Adam Nielsen]

> Whole dm  table with all deps needs to be known.

$ dmsetup table
backup: 0 11720531968 crypt aes-xts-plain64
  0000000000000000000000000000000000000000000000000000000000000000 0
  9:10 4096

$ dmsetup status
backup: 0 11720531968 crypt

$ dmsetup ls --tree
backup (253:0)
 └─ (9:10)

$ dmsetup info -f
Name:              backup
State:             ACTIVE (DEFERRED REMOVE)
Read Ahead:        4096
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: CRYPT-LUKS1-d0b3d38e421545908537dc50f59fb217-backup

All I'm using it for is to encrypt an mdadm-style RAID array composed
of two external disks, connected temporarily via USB to do a full
system backup with rsync.

> > I'm not sure how to do this, could you please elaborate?  I thought
> > "dmsetup remove --force" would do this but as that doesn't work
> 
> really state of whole table needs to be known.
> 
> >> Also note - dmsetup remove  supports --deferred removal (see man
> >> page).
> >
> > Oh I didn't notice that.  It doesn't seem to have much of an effect
> > though:
> 
> Sure it will not fix your problem - it's like lazy umount...

So replacing the table with the 'error' target won't release the
underlying device, even though that device is not used by the new
target?

> What is not clear to me is - what is your expectation here ?
> Obviously your system is far more broken - so placing 'error' target
> for your backup device will not fix it.
> 
> You should likely attach also portion of 'dmesg' - there surely will
> be written what is going wrong with your system.

What happened was in the middle of the backup, there was some USB
interruption and the disks dropped out, so the writes started failing.
The kernel logs were full of write errors to various sector numbers.  I
think you would have the same result if you set things up with a USB
stick and then unplugged it during a data transfer.

The devices are connected like this:

  dm device "backup"
   |
   +-- mdadm device /dev/md10
        |
        +-- USB/SATA disk A (/dev/sdd)
        |
        +-- USB/SATA disk B (/dev/sde)

The problem is that I can't just reconnect the disks and rerun the
backup.  mdadm refuses to stop the RAID array as it is in use by
the dm device, and it thinks the array is active despite the disks being
unplugged and in a drawer.  If I reconnect the disks they appear as
different devices (sdf and sdg) but I still can't start the "new" array
from these new disk devices, as it tells me the disks are already part
of an active array.

So the only way I can have another go at running this backup is to
close down /dev/md10, and it seems the only way I can do that is to
tell dm to release that device.  It doesn't matter if the dm device
"backup" is unusable, I will just create "backup2" to use for the
second attempt.

But until I can figure out how to get dm to release the underlying
device, I'm stuck!

> i.e. you cannot expect 'remove --force' will work when your machine
> start to show kernel errors.

There were no kernel crashes, just errors related to USB transfers.  I
would assume this is not much different to how a real failed disk might
behave, so I figure it is a situation that should be encountered
relatively often!

Thanks again,
Adam.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: How do you force-close a dm device after a disk failure?

[Zdenek Kabelac]

Dne 14.9.2015 v 11:45 Adam Nielsen napsal(a):
>> Whole dm  table with all deps needs to be known.
>
> $ dmsetup table
> backup: 0 11720531968 crypt aes-xts-plain64
>    0000000000000000000000000000000000000000000000000000000000000000 0
>    9:10 4096
>
> $ dmsetup status
> backup: 0 11720531968 crypt
>
> $ dmsetup ls --tree
> backup (253:0)
>   └─ (9:10)
>
> $ dmsetup info -f
> Name:              backup
> State:             ACTIVE (DEFERRED REMOVE)
> Read Ahead:        4096
> Tables present:    LIVE
> Open count:        1
> Event number:      0
> Major, minor:      253, 0
> Number of targets: 1
> UUID: CRYPT-LUKS1-d0b3d38e421545908537dc50f59fb217-backup
>
> All I'm using it for is to encrypt an mdadm-style RAID array composed
> of two external disks, connected temporarily via USB to do a full
> system backup with rsync.
>
>>> I'm not sure how to do this, could you please elaborate?  I thought
>>> "dmsetup remove --force" would do this but as that doesn't work
>>
>> really state of whole table needs to be known.
>>
>>>> Also note - dmsetup remove  supports --deferred removal (see man
>>>> page).
>>>
>>> Oh I didn't notice that.  It doesn't seem to have much of an effect
>>> though:
>>
>> Sure it will not fix your problem - it's like lazy umount...
>
> So replacing the table with the 'error' target won't release the
> underlying device, even though that device is not used by the new
> target?
>
>> What is not clear to me is - what is your expectation here ?
>> Obviously your system is far more broken - so placing 'error' target
>> for your backup device will not fix it.
>>
>> You should likely attach also portion of 'dmesg' - there surely will
>> be written what is going wrong with your system.
>
> What happened was in the middle of the backup, there was some USB
> interruption and the disks dropped out, so the writes started failing.
> The kernel logs were full of write errors to various sector numbers.  I
> think you would have the same result if you set things up with a USB
> stick and then unplugged it during a data transfer.
>
> The devices are connected like this:
>
>    dm device "backup"
>     |
>     +-- mdadm device /dev/md10
>          |
>          +-- USB/SATA disk A (/dev/sdd)
>          |
>          +-- USB/SATA disk B (/dev/sde)
>
> The problem is that I can't just reconnect the disks and rerun the
> backup.  mdadm refuses to stop the RAID array as it is in use by
> the dm device, and it thinks the array is active despite the disks being
> unplugged and in a drawer.  If I reconnect the disks they appear as
> different devices (sdf and sdg) but I still can't start the "new" array
> from these new disk devices, as it tells me the disks are already part
> of an active array.
>
> So the only way I can have another go at running this backup is to
> close down /dev/md10, and it seems the only way I can do that is to
> tell dm to release that device.  It doesn't matter if the dm device
> "backup" is unusable, I will just create "backup2" to use for the
> second attempt.
>
> But until I can figure out how to get dm to release the underlying
> device, I'm stuck!
>
>> i.e. you cannot expect 'remove --force' will work when your machine
>> start to show kernel errors.
>
> There were no kernel crashes, just errors related to USB transfers.  I
> would assume this is not much different to how a real failed disk might
> behave, so I figure it is a situation that should be encountered
> relatively often!
>



dmsetup reload backup --table "0 11720531968 error"
dmsetup suspend --noflush backup
dmsetup resume backup

Is this working for you ?

Zdenek




--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: How do you force-close a dm device after a disk failure?

[Adam Nielsen]

> > dmsetup reload backup --table "0 11720531968 error"

This one returns without error.

> > dmsetup suspend --noflush backup
> 
> or even better use   --noflush  --nolockfs
> (if there was a mounted filesystem)

This one does not return to the command prompt.  Running it through
strace shows this:

$ strace dmsetup suspend --noflush --nolockfs backup
...
munmap(0x7f9cdea45000, 4096)            = 0
ioctl(3, DM_VERSION, 0x5644ae2ca0d0)    = 0
ioctl(3, DM_DEV_SUSPEND

It always seems to freeze at DM_DEV_SUSPEND.  This ioctl never seems to
return.

> > dmsetup resume backup

Running this anyway also freezes:

$ strace dmsetup resume backup
...
close(4)                                = 0
semop(720903, {{0, 1, 0}}, 1)           = 0
semctl(720903, 0, GETVAL, 0x7fa5b3c28957) = 2
ioctl(3, DM_DEV_SUSPEND

> > Is this working for you ?

Looks like it's a kernel problem with DM_DEV_SUSPEND getting stuck
waiting for something?  There are no messages in dmesg after running
the above commands, not even any 'hung task' warnings.

Thanks,
Adam.

Re: How do you force-close a dm device after a disk failure?

[Zdenek Kabelac]

Dne 16.9.2015 v 02:58 Adam Nielsen napsal(a):
>>> dmsetup reload backup --table "0 11720531968 error"
>
> This one returns without error.
>
>>> dmsetup suspend --noflush backup
>>
>> or even better use   --noflush  --nolockfs
>> (if there was a mounted filesystem)
>
> This one does not return to the command prompt.  Running it through
> strace shows this:
>
> $ strace dmsetup suspend --noflush --nolockfs backup
> ...
> munmap(0x7f9cdea45000, 4096)            = 0
> ioctl(3, DM_VERSION, 0x5644ae2ca0d0)    = 0
> ioctl(3, DM_DEV_SUSPEND
>
> It always seems to freeze at DM_DEV_SUSPEND.  This ioctl never seems to
> return.

As with any other kernel frozen task - try to capture kernel stack trace.
If you properly configured sysrq trigger - easiest is to use:

'echo t >/proc/sysrq-trigger'

(Just make sure you have large enough kernel log buffer so lines are not lost)
Attach compressed trace - this should likely reveal where it blocks.
(I'll try to reproduce myself)


Zdenek

Re: How do you force-close a dm device after a disk failure?

[Adam Nielsen]

> > It always seems to freeze at DM_DEV_SUSPEND.  This ioctl never
> > seems to return.
> 
> As with any other kernel frozen task - try to capture kernel stack
> trace. If you properly configured sysrq trigger - easiest is to use:
> 
> 'echo t >/proc/sysrq-trigger'
> 
> (Just make sure you have large enough kernel log buffer so lines are
> not lost) Attach compressed trace - this should likely reveal where
> it blocks. (I'll try to reproduce myself)

Thanks for the advice.  I'm getting a warning that the buffer is
overflowing.  Is there anything in particular you need?  Here is
something that seems relevant:

dmsetup         D ffff880394467b98     0 24732  24717 0x00000000
 ffff880394467b98 ffff88040d7a1e90 ffff88027b738a30 ffff88040ba67458
 ffff880394468000 ffff8801eaa7b8dc ffff88027b738a30 00000000ffffffff
 ffff8801eaa7b8e0 ffff880394467bb8 ffffffff81588247 ffff8801eaa7b8d8
Call Trace:
 [<ffffffff81588247>] schedule+0x37/0x90
 [<ffffffff81588615>] schedule_preempt_disabled+0x15/0x20
 [<ffffffff81589b55>] __mutex_lock_slowpath+0xd5/0x150
 [<ffffffff81589beb>] mutex_lock+0x1b/0x30
 [<ffffffffa0857968>] dm_suspend+0x38/0xf0 [dm_mod]
 [<ffffffffa085d030>] ? table_load+0x370/0x370 [dm_mod]
 [<ffffffffa085d1c0>] dev_suspend+0x190/0x260 [dm_mod]
 [<ffffffffa085d030>] ? table_load+0x370/0x370 [dm_mod]
 [<ffffffffa085da72>] ctl_ioctl+0x232/0x520 [dm_mod]
 [<ffffffffa085dd73>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
 [<ffffffff811f4606>] do_vfs_ioctl+0x2c6/0x4d0
 [<ffffffff811f4891>] SyS_ioctl+0x81/0xa0
 [<ffffffff8158beae>] system_call_fastpath+0x12/0x71

Assuming 24732 is the PID, that's the "dmsetup suspend --noflush
--nolockfs" one.  There are heaps like the one above (from all my
attempts) with only one like the following, from an unknown command
line:

dmsetup         D ffff88012e2d7a88     0 28744  23911 0x00000004
 ffff88012e2d7a88 ffff88040d74f010 ffff88040398e5e0 ffff88012e2d7b38
 ffff88012e2d8000 ffff8800d9df5080 ffff8800d9df5068 ffffffff00000000
 fffffffe00000001 ffff88012e2d7aa8 ffffffff81588247 0000000000000002
Call Trace:
 [<ffffffff81588247>] schedule+0x37/0x90
 [<ffffffff8158a885>] rwsem_down_write_failed+0x165/0x370
 [<ffffffff810b2ad6>] ? enqueue_entity+0x266/0xd60
 [<ffffffff812d7aa3>] call_rwsem_down_write_failed+0x13/0x20
 [<ffffffff8158a0d4>] ? down_write+0x24/0x40
 [<ffffffff811e3aee>] grab_super+0x2e/0xb0
 [<ffffffff811e4a20>] get_active_super+0x70/0x90
 [<ffffffff8121ab9d>] freeze_bdev+0x6d/0x100
 [<ffffffffa0854f3b>] __dm_suspend+0xeb/0x230 [dm_mod]
 [<ffffffffa08579fa>] dm_suspend+0xca/0xf0 [dm_mod]
 [<ffffffffa085d1db>] dev_suspend+0x1ab/0x260 [dm_mod]
 [<ffffffffa085d030>] ? table_load+0x370/0x370 [dm_mod]
 [<ffffffffa085da72>] ctl_ioctl+0x232/0x520 [dm_mod]
 [<ffffffffa085dd73>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
 [<ffffffff811f4606>] do_vfs_ioctl+0x2c6/0x4d0
 [<ffffffff811f4891>] SyS_ioctl+0x81/0xa0
 [<ffffffff8158beae>] system_call_fastpath+0x12/0x71

The kernel is Arch Linux 4.1.4-1-ARCH.  Let me know if you need anything
else, sorry I don't have much experience with kernel debugging.  Not
sure whether /proc/kcore etc. is useful or not.

Thanks again for your help!

Cheers,
Adam.

Re: How do you force-close a dm device after a disk failure?

[Zdenek Kabelac]

Dne 16.9.2015 v 14:35 Adam Nielsen napsal(a):
>>> It always seems to freeze at DM_DEV_SUSPEND.  This ioctl never
>>> seems to return.
>>
>> As with any other kernel frozen task - try to capture kernel stack
>> trace. If you properly configured sysrq trigger - easiest is to use:
>>
>> 'echo t >/proc/sysrq-trigger'
>>
>> (Just make sure you have large enough kernel log buffer so lines are
>> not lost) Attach compressed trace - this should likely reveal where
>> it blocks. (I'll try to reproduce myself)
>
> Thanks for the advice.  I'm getting a warning that the buffer is
> overflowing.  Is there anything in particular you need?  Here is
> something that seems relevant:
>
> dmsetup         D ffff880394467b98     0 24732  24717 0x00000000
>   ffff880394467b98 ffff88040d7a1e90 ffff88027b738a30 ffff88040ba67458
>   ffff880394468000 ffff8801eaa7b8dc ffff88027b738a30 00000000ffffffff
>   ffff8801eaa7b8e0 ffff880394467bb8 ffffffff81588247 ffff8801eaa7b8d8
> Call Trace:
>   [<ffffffff81588247>] schedule+0x37/0x90
>   [<ffffffff81588615>] schedule_preempt_disabled+0x15/0x20
>   [<ffffffff81589b55>] __mutex_lock_slowpath+0xd5/0x150
>   [<ffffffff81589beb>] mutex_lock+0x1b/0x30
>   [<ffffffffa0857968>] dm_suspend+0x38/0xf0 [dm_mod]
>   [<ffffffffa085d030>] ? table_load+0x370/0x370 [dm_mod]
>   [<ffffffffa085d1c0>] dev_suspend+0x190/0x260 [dm_mod]
>   [<ffffffffa085d030>] ? table_load+0x370/0x370 [dm_mod]
>   [<ffffffffa085da72>] ctl_ioctl+0x232/0x520 [dm_mod]
>   [<ffffffffa085dd73>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
>   [<ffffffff811f4606>] do_vfs_ioctl+0x2c6/0x4d0
>   [<ffffffff811f4891>] SyS_ioctl+0x81/0xa0
>   [<ffffffff8158beae>] system_call_fastpath+0x12/0x71
>
> Assuming 24732 is the PID, that's the "dmsetup suspend --noflush
> --nolockfs" one.  There are heaps like the one above (from all my
> attempts) with only one like the following, from an unknown command
> line:
>
> dmsetup         D ffff88012e2d7a88     0 28744  23911 0x00000004
>   ffff88012e2d7a88 ffff88040d74f010 ffff88040398e5e0 ffff88012e2d7b38
>   ffff88012e2d8000 ffff8800d9df5080 ffff8800d9df5068 ffffffff00000000
>   fffffffe00000001 ffff88012e2d7aa8 ffffffff81588247 0000000000000002
> Call Trace:
>   [<ffffffff81588247>] schedule+0x37/0x90
>   [<ffffffff8158a885>] rwsem_down_write_failed+0x165/0x370
>   [<ffffffff810b2ad6>] ? enqueue_entity+0x266/0xd60
>   [<ffffffff812d7aa3>] call_rwsem_down_write_failed+0x13/0x20
>   [<ffffffff8158a0d4>] ? down_write+0x24/0x40
>   [<ffffffff811e3aee>] grab_super+0x2e/0xb0
>   [<ffffffff811e4a20>] get_active_super+0x70/0x90
>   [<ffffffff8121ab9d>] freeze_bdev+0x6d/0x100
>   [<ffffffffa0854f3b>] __dm_suspend+0xeb/0x230 [dm_mod]
>   [<ffffffffa08579fa>] dm_suspend+0xca/0xf0 [dm_mod]
>   [<ffffffffa085d1db>] dev_suspend+0x1ab/0x260 [dm_mod]
>   [<ffffffffa085d030>] ? table_load+0x370/0x370 [dm_mod]
>   [<ffffffffa085da72>] ctl_ioctl+0x232/0x520 [dm_mod]
>   [<ffffffffa085dd73>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
>   [<ffffffff811f4606>] do_vfs_ioctl+0x2c6/0x4d0
>   [<ffffffff811f4891>] SyS_ioctl+0x81/0xa0
>   [<ffffffff8158beae>] system_call_fastpath+0x12/0x71


Was this the 'ONLY' dmsetup in your listing (i.e. you reproduced case again)?

I mean - your existing reported situation was already hopeless and needed 
reboot - as if  flushing suspend holds some mutexes - no other suspend call 
can fix it ->  you usually have just  1 chance to fix it in right way,
if you go wrong way reboot is unavoidable.

Zdenek

Re: How do you force-close a dm device after a disk failure?

[Zdenek Kabelac]

Dne 16.9.2015 v 14:35 Adam Nielsen napsal(a):
>>> It always seems to freeze at DM_DEV_SUSPEND.  This ioctl never
>>> seems to return.
>>
>> As with any other kernel frozen task - try to capture kernel stack
>> trace. If you properly configured sysrq trigger - easiest is to use:
>>
>> 'echo t >/proc/sysrq-trigger'
>>
>> (Just make sure you have large enough kernel log buffer so lines are
>> not lost) Attach compressed trace - this should likely reveal where
>> it blocks. (I'll try to reproduce myself)
>
> Thanks for the advice.  I'm getting a warning that the buffer is
> overflowing.  Is there anything in particular you need?  Here is
> something that seems relevant:
>
> dmsetup         D ffff880394467b98     0 24732  24717 0x00000000
>   ffff880394467b98 ffff88040d7a1e90 ffff88027b738a30 ffff88040ba67458
>   ffff880394468000 ffff8801eaa7b8dc ffff88027b738a30 00000000ffffffff
>   ffff8801eaa7b8e0 ffff880394467bb8 ffffffff81588247 ffff8801eaa7b8d8
> Call Trace:
>   [<ffffffff81588247>] schedule+0x37/0x90
>   [<ffffffff81588615>] schedule_preempt_disabled+0x15/0x20
>   [<ffffffff81589b55>] __mutex_lock_slowpath+0xd5/0x150
>   [<ffffffff81589beb>] mutex_lock+0x1b/0x30
>   [<ffffffffa0857968>] dm_suspend+0x38/0xf0 [dm_mod]
>   [<ffffffffa085d030>] ? table_load+0x370/0x370 [dm_mod]
>   [<ffffffffa085d1c0>] dev_suspend+0x190/0x260 [dm_mod]
>   [<ffffffffa085d030>] ? table_load+0x370/0x370 [dm_mod]
>   [<ffffffffa085da72>] ctl_ioctl+0x232/0x520 [dm_mod]
>   [<ffffffffa085dd73>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
>   [<ffffffff811f4606>] do_vfs_ioctl+0x2c6/0x4d0
>   [<ffffffff811f4891>] SyS_ioctl+0x81/0xa0
>   [<ffffffff8158beae>] system_call_fastpath+0x12/0x71
>
> Assuming 24732 is the PID, that's the "dmsetup suspend --noflush
> --nolockfs" one.  There are heaps like the one above (from all my
> attempts) with only one like the following, from an unknown command
> line:
>
> dmsetup         D ffff88012e2d7a88     0 28744  23911 0x00000004
>   ffff88012e2d7a88 ffff88040d74f010 ffff88040398e5e0 ffff88012e2d7b38
>   ffff88012e2d8000 ffff8800d9df5080 ffff8800d9df5068 ffffffff00000000
>   fffffffe00000001 ffff88012e2d7aa8 ffffffff81588247 0000000000000002
> Call Trace:
>   [<ffffffff81588247>] schedule+0x37/0x90
>   [<ffffffff8158a885>] rwsem_down_write_failed+0x165/0x370
>   [<ffffffff810b2ad6>] ? enqueue_entity+0x266/0xd60
>   [<ffffffff812d7aa3>] call_rwsem_down_write_failed+0x13/0x20
>   [<ffffffff8158a0d4>] ? down_write+0x24/0x40
>   [<ffffffff811e3aee>] grab_super+0x2e/0xb0
>   [<ffffffff811e4a20>] get_active_super+0x70/0x90
>   [<ffffffff8121ab9d>] freeze_bdev+0x6d/0x100
>   [<ffffffffa0854f3b>] __dm_suspend+0xeb/0x230 [dm_mod]
>   [<ffffffffa08579fa>] dm_suspend+0xca/0xf0 [dm_mod]
>   [<ffffffffa085d1db>] dev_suspend+0x1ab/0x260 [dm_mod]
>   [<ffffffffa085d030>] ? table_load+0x370/0x370 [dm_mod]
>   [<ffffffffa085da72>] ctl_ioctl+0x232/0x520 [dm_mod]
>   [<ffffffffa085dd73>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
>   [<ffffffff811f4606>] do_vfs_ioctl+0x2c6/0x4d0
>   [<ffffffff811f4891>] SyS_ioctl+0x81/0xa0
>   [<ffffffff8158beae>] system_call_fastpath+0x12/0x71
>
> The kernel is Arch Linux 4.1.4-1-ARCH.  Let me know if you need anything
> else, sorry I don't have much experience with kernel debugging.  Not
> sure whether /proc/kcore etc. is useful or not.
>
> Thanks again for your help!
>

Hi

Could you please try to check use 2 extra options for crypto open:

'cryptosetup --perf-submit_from_crypt_cpus --perf-same_cpu_crypt luksOpen'

to check if you will experience same trouble ?

Zdenek

Re: How do you force-close a dm device after a disk failure?

[Lars Ellenberg]

On Mon, Sep 14, 2015 at 07:45:52PM +1000, Adam Nielsen wrote:
> > Whole dm  table with all deps needs to be known.
> 
> $ dmsetup table
> backup: 0 11720531968 crypt aes-xts-plain64
>   0000000000000000000000000000000000000000000000000000000000000000 0
>   9:10 4096
> 
> $ dmsetup status
> backup: 0 11720531968 crypt
> 
> $ dmsetup ls --tree
> backup (253:0)
>  └─ (9:10)
> 
> $ dmsetup info -f
> Name:              backup
> State:             ACTIVE (DEFERRED REMOVE)
> Read Ahead:        4096
> Tables present:    LIVE
> Open count:        1
> Event number:      0
> Major, minor:      253, 0
> Number of targets: 1
> UUID: CRYPT-LUKS1-d0b3d38e421545908537dc50f59fb217-backup
> 
> All I'm using it for is to encrypt an mdadm-style RAID array composed
> of two external disks, connected temporarily via USB to do a full
> system backup with rsync.
> 
> > > I'm not sure how to do this, could you please elaborate?  I thought
> > > "dmsetup remove --force" would do this but as that doesn't work
> > 
> > really state of whole table needs to be known.
> > 
> > >> Also note - dmsetup remove  supports --deferred removal (see man
> > >> page).
> > >
> > > Oh I didn't notice that.  It doesn't seem to have much of an effect
> > > though:
> > 
> > Sure it will not fix your problem - it's like lazy umount...
> 
> So replacing the table with the 'error' target won't release the
> underlying device, even though that device is not used by the new
> target?
> 
> > What is not clear to me is - what is your expectation here ?
> > Obviously your system is far more broken - so placing 'error' target
> > for your backup device will not fix it.
> > 
> > You should likely attach also portion of 'dmesg' - there surely will
> > be written what is going wrong with your system.
> 
> What happened was in the middle of the backup, there was some USB
> interruption and the disks dropped out, so the writes started failing.
> The kernel logs were full of write errors to various sector numbers.  I
> think you would have the same result if you set things up with a USB
> stick and then unplugged it during a data transfer.
> 
> The devices are connected like this:
> 
>   dm device "backup"
>    |
>    +-- mdadm device /dev/md10
>         |
>         +-- USB/SATA disk A (/dev/sdd)
>         |
>         +-- USB/SATA disk B (/dev/sde)

mdadm /dev/md10 --fail /dev/sdd --remove /dev/sdd
mdadm /dev/md10 --fail /dev/sde --remove /dev/sde
(or maybe combine in one command line, if that is supposed to work)

Should kick out both disks from the MD,
should make md10 fail all pending (and new) request,
should even get the stuck dm suspend unstuck.

No?

Cheers,

	Lars Ellenberg

> The problem is that I can't just reconnect the disks and rerun the
> backup.  mdadm refuses to stop the RAID array as it is in use by
> the dm device, and it thinks the array is active despite the disks being
> unplugged and in a drawer.  If I reconnect the disks they appear as
> different devices (sdf and sdg) but I still can't start the "new" array
> from these new disk devices, as it tells me the disks are already part
> of an active array.
> 
> So the only way I can have another go at running this backup is to
> close down /dev/md10, and it seems the only way I can do that is to
> tell dm to release that device.  It doesn't matter if the dm device
> "backup" is unusable, I will just create "backup2" to use for the
> second attempt.
> 
> But until I can figure out how to get dm to release the underlying
> device, I'm stuck!
> 
> > i.e. you cannot expect 'remove --force' will work when your machine
> > start to show kernel errors.
> 
> There were no kernel crashes, just errors related to USB transfers.  I
> would assume this is not much different to how a real failed disk might
> behave, so I figure it is a situation that should be encountered
> relatively often!
> 
> Thanks again,
> Adam.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: How do you force-close a dm device after a disk failure?

[Adam Nielsen]

> Was this the 'ONLY' dmsetup in your listing (i.e. you reproduced case
> again)?

This was the original instance of the problem.  Today I have rebooted
and reproduced the problem on a fresh kernel.

> I mean - your existing reported situation was already hopeless and
> needed reboot - as if  flushing suspend holds some mutexes - no other
> suspend call can fix it ->  you usually have just  1 chance to fix it
> in right way, if you go wrong way reboot is unavoidable.

That sounds like a very unforgiving buggy kernel, if you only have one
chance to fix the problem ;-)

Here is my attempt on the fresh kernel.  I received some write errors
in dmesg, so tried to umount the dm device to confirm I had reproduced
the problem, and when umount failed to exit I tried this:

  $ dmsetup reload backup --table "0 11720531968 error"
  $ dmsetup suspend --noflush --nolockfs backup

These two worked fine now.  "dmsetup suspend" was locking up before,
this time it worked.

  $ umount /mnt/backup
  umount: /mnt/backup: not mounted

The dm instance is no longer mounted.

  $ mdadm --manage --stop /dev/md10
  mdadm: Cannot get exclusive access to /dev/md10:Perhaps a running
    process, mounted filesystem or active volume group?

I can't restart the underlying RAID array though, as the dm instance is
still holding onto the devices.

  $ dmsetup remove --force backup
  device-mapper: remove ioctl on backup failed: Device or resource busy
  Command failed

I don't appear to be able to shut down the dm device either.  I tried
to umount the device before any of this, and the umount process has
frozen (despite it seeming to have unmounted successfully), so this is
probably what the kernel thinks is using the device.  Although the table
has been replace by the "error" target, the umount process is not
returning and appears to be frozen inside the kernel (because killall
-9 doesn't work.)

Strangely I can still read and write to the underlying device
(/dev/md10), it is only processes accessing /dev/mapper/backup that
freeze.

Any suggestions?  I imagine "dmsetup remove --deferred" won't help if
the umount process is holding the device open and never terminates.  It
still looks like once you get an I/O error, the dm device locks up and
a reboot is the only way to get it to let go of the storage device
underlying the dm device.

Not sure if this helps, but this is where 'sync' and 'umount' lock up
when the system is in this state:

sync            D ffff880121ff7e00     0 23685  23671 0x00000004
 ffff880121ff7e00 ffff88040d7a28c0 ffff88040b498a30 dead000000100100
 ffff880121ff8000 ffff8800d96ff068 ffff8800d96ff080 ffffffff81213800
 ffff8800d96ff068 ffff880121ff7e20 ffffffff81588377 ffff880037bbc068
Call Trace:
 [<ffffffff81213800>] ? SyS_tee+0x400/0x400
 [<ffffffff81588377>] schedule+0x37/0x90
 [<ffffffff8158a805>] rwsem_down_read_failed+0xd5/0x120
 [<ffffffff8120cf64>] ? sync_inodes_sb+0x184/0x1e0
 [<ffffffff812d7b24>] call_rwsem_down_read_failed+0x14/0x30
 [<ffffffff8158a1d7>] ? down_read+0x17/0x20
 [<ffffffff811e49d4>] iterate_supers+0xa4/0x120
 [<ffffffff81213b94>] sys_sync+0x44/0xb0
 [<ffffffff8158bfae>] system_call_fastpath+0x12/0x71

umount          R  running task        0 23669  18676 0x00000004
 00000000000000cb ffff880108607d78 0000000000000000 000000000000020e
 0000000000000000 0000000000000000 ffff880108604000 ffff88040b49dbb0
 00000000000000e4 00000000000000ff 0000000000000000 ffff8800d972b800
Call Trace:
 [<ffffffffa00d13a9>] ? jbd2_log_do_checkpoint+0x19/0x4b0 [jbd2]
 [<ffffffffa00d13bd>] ? jbd2_log_do_checkpoint+0x2d/0x4b0 [jbd2]
 [<ffffffffa00d6520>] ? jbd2_journal_destroy+0x140/0x240 [jbd2]
 [<ffffffff810bc720>] ? wake_atomic_t_function+0x60/0x60
 [<ffffffffa019f6d7>] ? ext4_put_super+0x67/0x360 [ext4]
 [<ffffffff811e3216>] ? generic_shutdown_super+0x76/0x100
 [<ffffffff811e35d7>] ? kill_block_super+0x27/0x80
 [<ffffffff811e3949>] ? deactivate_locked_super+0x49/0x80
 [<ffffffff811e3dbc>] ? deactivate_super+0x6c/0x80
 [<ffffffff81201863>] ? cleanup_mnt+0x43/0xa0
 [<ffffffff81201912>] ? __cleanup_mnt+0x12/0x20
 [<ffffffff81095c54>] ? task_work_run+0xd4/0xf0
 [<ffffffff81015d25>] ? do_notify_resume+0x75/0x80
 [<ffffffff8158c17c>] ? int_signal+0x12/0x17

Looks like umount might be stuck in an infinite loop, when I run
another trace, it's slightly different:

umount          R  running task        0 23669  18676 0x00000004
 ffffffffffffff02 ffffffffa00d0f2e 0000000000000010 0000000000000292
 ffff880108607c98 0000000000000018 0000000000000000 ffff8800d972b800
 ffffffffffffff02 ffffffffa00d13c0 00000000f8eef941 0000000000000296
Call Trace:
 [<ffffffffa00d0f2e>] ? jbd2_cleanup_journal_tail+0xe/0xb0 [jbd2]
 [<ffffffffa00d13c0>] ? jbd2_log_do_checkpoint+0x30/0x4b0 [jbd2]
 [<ffffffffa00d13bd>] ? jbd2_log_do_checkpoint+0x2d/0x4b0 [jbd2]
 [<ffffffffa00d6518>] ? jbd2_journal_destroy+0x138/0x240 [jbd2]
 [<ffffffff810bc720>] ? wake_atomic_t_function+0x60/0x60
 [<ffffffffa019f6d7>] ? ext4_put_super+0x67/0x360 [ext4]
 [<ffffffff811e3216>] ? generic_shutdown_super+0x76/0x100
 [<ffffffff811e35d7>] ? kill_block_super+0x27/0x80
 [<ffffffff811e3949>] ? deactivate_locked_super+0x49/0x80
 [<ffffffff811e3dbc>] ? deactivate_super+0x6c/0x80
 [<ffffffff81201863>] ? cleanup_mnt+0x43/0xa0
 [<ffffffff81201912>] ? __cleanup_mnt+0x12/0x20
 [<ffffffff81095c54>] ? task_work_run+0xd4/0xf0
 [<ffffffff81015d25>] ? do_notify_resume+0x75/0x80
 [<ffffffff8158c17c>] ? int_signal+0x12/0x17

Thanks,
Adam.

Re: How do you force-close a dm device after a disk failure?

[Lars Ellenberg]

On Sat, Sep 19, 2015 at 07:47:52PM +1000, Adam Nielsen wrote:
> > Was this the 'ONLY' dmsetup in your listing (i.e. you reproduced case
> > again)?
> 
> This was the original instance of the problem.  Today I have rebooted
> and reproduced the problem on a fresh kernel.
> 
> > I mean - your existing reported situation was already hopeless and
> > needed reboot - as if  flushing suspend holds some mutexes - no other
> > suspend call can fix it ->  you usually have just  1 chance to fix it
> > in right way, if you go wrong way reboot is unavoidable.
> 
> That sounds like a very unforgiving buggy kernel, if you only have one
> chance to fix the problem ;-)
> 
> Here is my attempt on the fresh kernel.  I received some write errors
> in dmesg, so tried to umount the dm device to confirm I had reproduced
> the problem, and when umount failed to exit I tried this:
> 
>   $ dmsetup reload backup --table "0 11720531968 error"
>   $ dmsetup suspend --noflush --nolockfs backup

You need to *resume* to activate the new table.

> These two worked fine now.  "dmsetup suspend" was locking up before,
> this time it worked.
> 
>   $ umount /mnt/backup
>   umount: /mnt/backup: not mounted
> 
> The dm instance is no longer mounted.
> 
>   $ mdadm --manage --stop /dev/md10
>   mdadm: Cannot get exclusive access to /dev/md10:Perhaps a running
>     process, mounted filesystem or active volume group?

Also, as mentioned before, why don't you
mdadm /dev/md10 --fail /dev/sdd --remove /dev/sdd
mdadm /dev/md10 --fail /dev/sde --remove /dev/sde
(for whatever sdX members it currently has;
or maybe combine in one command line, if that is supposed to work)

Should kick out the disks from the MD,
should make md10 fail all pending (and new) requests,
should even get the stuck dm suspend going again
(the implicit "flush" one, not the --noflush one,
as that did not get stuck anyways).

> I can't restart the underlying RAID array though, as the dm instance is
> still holding onto the devices.
> 
>   $ dmsetup remove --force backup
>   device-mapper: remove ioctl on backup failed: Device or resource busy
>   Command failed

You need to *resume* the new (error) table.
Or the previous table is only suspended, but still holds references.

> I don't appear to be able to shut down the dm device either.  I tried
> to umount the device before any of this, and the umount process has
> frozen (despite it seeming to have unmounted successfully), so this is
> probably what the kernel thinks is using the device.  Although the table
> has been replace by the "error" target, the umount process is not
> returning and appears to be frozen inside the kernel (because killall
> -9 doesn't work.)
> 
> Strangely I can still read and write to the underlying device
> (/dev/md10), it is only processes accessing /dev/mapper/backup that
> freeze.

You *suspended* it. It is supposed to be frozen.

Cheers,
	Lars Ellenberg

Re: How do you force-close a dm device after a disk failure?

[Zdenek Kabelac]

Dne 21.9.2015 v 13:39 Lars Ellenberg napsal(a):
> On Sat, Sep 19, 2015 at 07:47:52PM +1000, Adam Nielsen wrote:
>>> Was this the 'ONLY' dmsetup in your listing (i.e. you reproduced case
>>> again)?
>>
>> This was the original instance of the problem.  Today I have rebooted
>> and reproduced the problem on a fresh kernel.
>>
>>> I mean - your existing reported situation was already hopeless and
>>> needed reboot - as if  flushing suspend holds some mutexes - no other
>>> suspend call can fix it ->  you usually have just  1 chance to fix it
>>> in right way, if you go wrong way reboot is unavoidable.
>>
>> That sounds like a very unforgiving buggy kernel, if you only have one
>> chance to fix the problem ;-)
>>
>> Here is my attempt on the fresh kernel.  I received some write errors
>> in dmesg, so tried to umount the dm device to confirm I had reproduced
>> the problem, and when umount failed to exit I tried this:
>>
>>    $ dmsetup reload backup --table "0 11720531968 error"
>>    $ dmsetup suspend --noflush --nolockfs backup
>
> You need to *resume* to activate the new table.
>
>> These two worked fine now.  "dmsetup suspend" was locking up before,
>> this time it worked.
>>
>>    $ umount /mnt/backup
>>    umount: /mnt/backup: not mounted
>>
>> The dm instance is no longer mounted.
>>
>>    $ mdadm --manage --stop /dev/md10
>>    mdadm: Cannot get exclusive access to /dev/md10:Perhaps a running
>>      process, mounted filesystem or active volume group?
>
> Also, as mentioned before, why don't you
> mdadm /dev/md10 --fail /dev/sdd --remove /dev/sdd
> mdadm /dev/md10 --fail /dev/sde --remove /dev/sde
> (for whatever sdX members it currently has;
> or maybe combine in one command line, if that is supposed to work)
>
> Should kick out the disks from the MD,
> should make md10 fail all pending (and new) requests,
> should even get the stuck dm suspend going again
> (the implicit "flush" one, not the --noflush one,
> as that did not get stuck anyways).
>
>> I can't restart the underlying RAID array though, as the dm instance is
>> still holding onto the devices.
>>
>>    $ dmsetup remove --force backup
>>    device-mapper: remove ioctl on backup failed: Device or resource busy
>>    Command failed
>
> You need to *resume* the new (error) table.
> Or the previous table is only suspended, but still holds references.
>


There is a condition which may prevent replacement dm table.

If the 'dm' target has in-progress bio operation and the underlying device is 
not responding (acking bio completed),  you can't suspend such targeted with 
bio-in-progress.

It's not trivial to improve this.

So if you happen to 'deadlock' in this state - there is currently no other 
help then rebooting machine if you want to get rid of such 'frozen' device.

On the other hand - from what was said -  'dropping' USB disk out of system 
should not be causing such state.

So probably more details from logs need to be know for knowing more about this.


Zdenek