dmsetup remove gives EBUSY (even though it shouldn't be busy)

dmsetup remove gives EBUSY (even though it shouldn't be busy)

[Johannes Bauer]

Hi list,

apologies if I'm at the wrong place, but I think this is dm related
(even though in practice the device I'm talking about is using the
dm-crypt target).

This is my problem: I have a open dm-crypt mapping (LUKS) on which I do
perform some block operations (copying). There are no mounted file
systems on the copied device handles at any time. Let's say my device is
called under /dev/mapper/foobar and I'm writing to that block device.

The file descriptor that points to the /dev/mapper/foobar is then closed
via close(2). Then I call sync(2).

Afterwards I fork/exec the command "dmsetup remove foobar". This yields:

device-mapper: remove ioctl on foobar failed: Device or resource busy
Command failed

And results in return code 1. I'm only getting this behavior with slow
disks (i.e. cannot reproduce with a loop device that points to /dev/shm
as underlying block device). And I'm guessing there's some
buffers/caches still in use (i.e. pending writes) which is why dmsetup
returns EBUSY. What I don't understand is why the sync(2) call does not
block until all pending writes have been flushed.

If I do insert sleep(10) before my "dmsetup remove" call, it works
smoothly -- but this is hardly a solution.

Do you have any pointers on why this happens and how I can cleanly solve
this?

Thank you very much,
Cheers,
Johannes

Re: dmsetup remove gives EBUSY (even though it shouldn't be busy)

[Milan Broz]

On 05/25/2015 12:31 AM, Johannes Bauer wrote:
> This is my problem: I have a open dm-crypt mapping (LUKS) on which I do
> perform some block operations (copying). There are no mounted file
> systems on the copied device handles at any time. Let's say my device is
> called under /dev/mapper/foobar and I'm writing to that block device.
> 
> The file descriptor that points to the /dev/mapper/foobar is then closed
> via close(2). Then I call sync(2).
> 
> Afterwards I fork/exec the command "dmsetup remove foobar". This yields:
> 
> device-mapper: remove ioctl on foobar failed: Device or resource busy
> Command failed

Usually this is caused by running blkid (automatically running from udev rules)
which react to close-on-write on the device and keep the device open while scanning.
(You can try to run lsof, there is a chance you will see the process blocking
this device.)

Recent device mapper tries to retry removal operation, so if you use
recent dmsetup (or cryptsetup close command) you should see that it
tries several times.

First option is to disable WATCH rule in udev rules and check if this helps.
(Unfortunately this has some side effects.)

Better solution is probably explicitly call "udevadm settle" instead
of sleep(). Still not perfect (it waits for all devices/udev processes),
but I am afraid there is nothing better yet to synchronize with these scans.

Milan

Re: dmsetup remove gives EBUSY (even though it shouldn't be busy)

[Johannes Bauer]

Hey Milan,

sorry for the late answer -- I thought nobody replied to the topic
because I had the grouping setting messed up in my mail client. Only now
that I looked in the messages again because of something totally
unrelated I saw your response.

On 25.05.2015 14:38, Milan Broz wrote:

>> Afterwards I fork/exec the command "dmsetup remove foobar". This yields:
>>
>> device-mapper: remove ioctl on foobar failed: Device or resource busy
>> Command failed
> 
> Usually this is caused by running blkid (automatically running from udev rules)
> which react to close-on-write on the device and keep the device open while scanning.
> (You can try to run lsof, there is a chance you will see the process blocking
> this device.)

Ah, yes, indeed I do see a blkid process there.

> Recent device mapper tries to retry removal operation, so if you use
> recent dmsetup (or cryptsetup close command) you should see that it
> tries several times.

Yes, I saw that option, but in some cases it wouldn't wait long enough
in my case and still return EBUSY.

> First option is to disable WATCH rule in udev rules and check if this helps.
> (Unfortunately this has some side effects.)
> 
> Better solution is probably explicitly call "udevadm settle" instead
> of sleep(). Still not perfect (it waits for all devices/udev processes),
> but I am afraid there is nothing better yet to synchronize with these scans.

Hm, I don't think I want a change in the udev config, so I'm going to
navigate around this issue somhow. In any case my main worry was that I
was doing something wrong which would cause this behavior. That udev
calls blkid which causes the EBUSY actually is a good thing (because I
can reasonably assume that blkid will terminate after a rather short
amount of time).

So I thank you for your response, clearing this up.

Best regards,
Johannes