Occasional dmsetup resume lockups

Occasional dmsetup resume lockups

[Richard Davies]

Hi,

We have been seeing occasional "dmsetup resume" lockups on a variety of
kernels when swapping tables.

I am wondering if we are making a simple scripting mistake
(e.g. we do not run "dmsetup suspend", but my impression is that the need
for this was replaced by INACTIVE tables?)

Alternatively, what can we do to help debug this further, please?


In more detail:

We have a script which backs up a live block device DRIVE using a
temporary dm-raid1.

While a backup is going on, there are 3 devices:

ORIGIN is a dm-crypt over a local block device (the original)
SYNC is a dm-linear over a remote iscsi device (the backup)
DRIVE is a dm-raid1 of ORIGIN and SYNC (doing the backup)

Several minutes after the raid array for DRIVE is in-sync, we run:

  blockdev --flushbufs DRIVE
  TABLE=`dmsetup --showkeys table ORIGIN`
  // We don't run dmsetup suspend here, we think it is no longer required?
  dmsetup reload DRIVE --table "$TABLE"
  dmsetup resume DRIVE

Most of the time this works, but sometimes "dmsetup resume" hangs forever.
strace shows it hanging in the DM_DEV_SUSPEND ioctl.

While a hang is ongoing:

"dmsetup info DRIVE" shows "Tables present: LIVE". The INACTIVE table is no
longer listed.

"dmsetup --showkeys table DRIVE" still shows the dm-raid1

We have seen this on kernels from 3.8.6 to 4.1.12.

There is nothing in the kernel log after the dm-raid1 rebuild logging.


Do we have a script bug, or if not, how can we help debug this further?

Thanks,

Richard.

Re: Occasional dmsetup resume lockups

[Zdenek Kabelac]

Dne 11.2.2016 v 21:47 Richard Davies napsal(a):
> Hi,
>
> We have been seeing occasional "dmsetup resume" lockups on a variety of
> kernels when swapping tables.
>
> I am wondering if we are making a simple scripting mistake
> (e.g. we do not run "dmsetup suspend", but my impression is that the need
> for this was replaced by INACTIVE tables?)
>
> Alternatively, what can we do to help debug this further, please?
>
>

There could be number of reasons - but since you mention you operate
with life device -  aren't you actually running your 'dmsetup
from a device you manipulate with - aka your root volume ?

lvm2 locks itself into memory to avoid any page swapping issues.

You could also try debugging kernel on serial console with sysrq+t
to see what's get blocked on disk access.

Regards

Zdenek

Re: Occasional dmsetup resume lockups

[Richard Davies]

Zdenek Kabelac wrote:
> There could be number of reasons - but since you mention you operate with
> live device - aren't you actually running your dmsetup from a device you
> manipulate with - aka your root volume ?

No. It is different partitions.

The live device which we are running dmsetup on is the storage for a VM.
Under the dm-raid1 DRIVE, this is a dm-crypt ORIGIN over an LVM LV over md
RAID of some partitions.

The root filesystem including the dmsetup binary is on md RAID of different
partitions of the same physical disks.

> lvm2 locks itself into memory to avoid any page swapping issues.

Sorry, I don't understand if this is relevant?

> You could also try debugging kernel on serial console with sysrq+t to see
> what's get blocked on disk access.

OK, we will try this and share when we have it.

Richard.