Re: The dev node can't be released at once after stopping raid

[Xiao Ni <xni@redhat.com>]

----- Original Message -----
> From: "NeilBrown" <neilb@suse.com>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org
> Sent: Thursday, August 31, 2017 12:36:08 PM
> Subject: Re: The dev node can't be released at once after stopping raid
> 
> On Wed, Aug 30 2017, Xiao Ni wrote:
> 
> > Hi Neil
> >
> > I have searched in history emails and there have many topics like this.
> > Sorry for talking
> > about this again. But it looks like the situation I encountered is
> > different. There is 1 second
> > window between stop the raid device and delete the node /dev/md0. The
> > /dev/md0 node can be
> > removed successfully after 1 second.
> 
> I think you are saying that /dev/md0 gets deleted 1 second after the
> device is stopped.  I assume that is a delay in udev processing of
> events.
> 
> When you say "can be"  I assume you mean "is being".
> ie. if you say
>    "The node can be removed after 1 second", it seems to imply that if
>    you try to remove it earlier, the unlink() will fail.
> If you say
>   "The node is being removed after 1 seconds", that suggests that the
>   removal happens automatically, but there is a delay between the device
>   stopping and the removal happening.

Yes, it's this situation. The node is being removed after 1 seconds.

> 
> >
> > There is no process that open the /dev/md0 after mdadm -S /dev/md0:
> >
> > mdadm -CR /dev/md0 -l1 -n2 /dev/loop0 /dev/loop1 --assume-clean
> > dmesg:
> > [36416.860525] Opened by mdadm, pid is 3523
> > [36416.984160] md/raid1:md0: active with 2 out of 2 mirrors
> > [36416.984181] md0: detected capacity change from 0 to 523239424
> > [36416.984219] Released by mdadm, pid is 3523
> > [36416.984228] remove_and_add_spares
> > [36416.991588] Opened by mdadm, pid is 3541
> > [36416.997183] Released by mdadm, pid is 3541
> > [36417.001376] Opened by systemd-udevd, pid is 3525
> > [36417.007128] Released by systemd-udevd, pid is 3525
> >
> > udev:
> > KERNEL[36419.830817] add      /devices/virtual/bdi/9:0 (bdi)
> > KERNEL[36419.831045] add      /devices/virtual/block/md0 (block)
> > UDEV  [36419.832911] add      /devices/virtual/bdi/9:0 (bdi)
> > UDEV  [36419.836380] add      /devices/virtual/block/md0 (block)
> > KERNEL[36419.877705] change   /devices/virtual/block/loop0 (block)
> > KERNEL[36419.878057] change   /devices/virtual/block/loop0 (block)
> > KERNEL[36419.926761] change   /devices/virtual/block/loop1 (block)
> > KERNEL[36419.927015] change   /devices/virtual/block/loop1 (block)
> > UDEV  [36419.953112] change   /devices/virtual/block/loop0 (block)
> > UDEV  [36419.953141] change   /devices/virtual/block/loop1 (block)
> > KERNEL[36419.954765] change   /devices/virtual/block/md0 (block)
> > UDEV  [36419.955973] change   /devices/virtual/block/loop0 (block)
> > UDEV  [36419.962799] change   /devices/virtual/block/loop1 (block)
> > UDEV  [36419.982934] change   /devices/virtual/block/md0 (block)
> >
> > mdadm -S /dev/md0
> > dmesg:
> > [36493.068054] Opened by mdadm, pid is 3552
> > [36493.072051] Released by mdadm, pid is 3552
> > [36493.076123] Opened by mdadm, pid is 3552
> > [36493.080073] md0: detected capacity change from 523239424 to 0
> > [36493.080077] md: md0 stopped.
> > [36493.273011] Released by mdadm, pid is 3552
> > udev:
> > KERNEL[36496.300219] remove   /devices/virtual/bdi/9:0 (bdi)
> > KERNEL[36496.300335] remove   /devices/virtual/block/md0 (block)
> > UDEV  [36496.300736] remove   /devices/virtual/bdi/9:0 (bdi)
> > UDEV  [36496.301812] remove   /devices/virtual/block/md0 (block)
> 
> I don't see any 1 second delay here.
> I can see a 3 second delay between "Released by mdadm, pid = 3552" and
> the UDEV remove event.  Is that what you are referring to?

Ah, how do you calculate 3 second? 36496 - 36493? 

I did the test again, the dmesg and udev are:
dmesg
[ 2988.821730] Opened by mdadm, pid is 3174
[ 2988.825827] Released by mdadm, pid is 3174
[ 2988.830112] Opened by mdadm, pid is 3174
[ 2988.834200] md: md0 stopped.
[ 2988.834397] Released by mdadm, pid is 3174
udev
KERNEL[2989.150258] remove   /devices/virtual/bdi/9:0 (bdi)
KERNEL[2989.150334] remove   /devices/virtual/block/md0 (block)
UDEV  [2989.150491] remove   /devices/virtual/bdi/9:0 (bdi)
UDEV  [2989.151587] remove   /devices/virtual/block/md0 (block)


The test script is:
[root@dell-per210-01 ~]# cat test.sh 
#!/bin/sh
mdadm -CR /dev/md0 -l1 -n2 /dev/loop0  /dev/loop1 --assume-clean
mdadm -S /dev/md0
ls /dev/md0
sleep 1
ls /dev/md0

The result is:
[root@dell-per210-01 ~]# sh test.sh 
...
mdadm: stopped /dev/md0
/dev/md0
ls: cannot access /dev/md0: No such file or directory

> 
> >
> > There are only REMOVE events during command mdadm -S /dev/md0.
> 
> The remove events seems to happen *after* "mdadm -S /dev/md0", or did
> "mdadm -S /dev/md0" take 3 seconds to run?
> 
> >
> > I tried to create a lvm and remove it to check whether lvm has this problem
> > or not.
> >
> > pvcreate /dev/md0
> > vgcreate vg /dev/md0
> > lvcreate -L 100M -n test vg
> > lvremove vg/test -y
> > ls /dev/mapper/vg-test
> > ls /dev/dm-3
> >
> > The node /dev/mapper/vg-test and /dev/dm-3 can be removed in time. There is
> > no time
> > window. So it looks like it's a problem of md. Could you give some
> > suggestions about
> > this? What should I do next?
> 
> Maybe lvremove explicitly unlinks the files in /dev, I don't know.

I did a test. mdadm unlink /run/mdadm/map.lock during mdadm -S. Can mdadm
unlink explicitly too? I added this line and this problem can be fixed.

diff --git a/Manage.c b/Manage.c
index b82a729..04994b3 100644
--- a/Manage.c
+++ b/Manage.c
@@ -482,6 +482,7 @@ done:
        map_lock(&map);
        map_remove(&map, devnm);
        map_unlock(&map);
+       unlink(devname);
 out:
        sysfs_free(mdi);
 

> 
> >
> > If it's not a bug, why there is a 1 second window?
> 
> As I said, probably because udev is slow.
> Why do you think this is a problem?  Why do you care about 1 second
> window.  If I don't know how why this matters, I cannot help you.

There is a bug https://bugzilla.redhat.com/show_bug.cgi?id=1444434. 
Another tool(blivet) stops raid device and the device node still exists.
Then it calls mdadm -S xxx again and it fails. So I ask myself why
/dev/mdxxx can't be removed immediately after command mdadm -S. 

In topic "MD Remnants After –stop", you said the REMOVE events are 
generated by "md_free() -> del_gendisk() ->  blk_unregister_queue()".
When mdadm -S return, the REMOVE events should be generated already,
right?

I always have a question. Who is responsible for removing the device
node under /dev/ directory? The function unlink()?

> 
> NeilBrown
>