Re: MD Remnants After --stop

[NeilBrown <neilb@suse.com>]

[-- Attachment #1: Type: text/plain, Size: 4146 bytes --]

On Sun, Nov 27 2016, Marc Smith wrote:

> So, I modified mdopen.c to look like this:
>
> --- a/mdopen.c    2016-11-25 17:04:25.782299330 -0500
> +++ b/mdopen.c    2016-11-26 10:57:35.883621355 -0500
> @@ -416,7 +416,7 @@
>   */
>  int open_mddev(char *dev, int report_errors)
>  {
> -    int mdfd = open(dev, O_RDWR);
> +    int mdfd = open(dev, O_RDONLY);
>      if (mdfd < 0 && errno == EACCES)
>          mdfd = open(dev, O_RDONLY);
>      if (mdfd < 0) {
>
>
> And now, when running 'mdadm --stop' here is what I see...
>
> From the output 'udevadm monitor -pku':
>
> --snip--
> KERNEL[297486.536908] offline
> /kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e (dlm)
> ACTION=offline
> DEVPATH=/kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e
> LOCKSPACE=62fccfd6-605f-19e6-be6d-99a1e3cb987e
> SEQNUM=3651
> SUBSYSTEM=dlm
>
> UDEV  [297486.537541] offline
> /kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e (dlm)
> ACTION=offline
> DEVPATH=/kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e
> LOCKSPACE=62fccfd6-605f-19e6-be6d-99a1e3cb987e
> SEQNUM=3651
> SUBSYSTEM=dlm
> USEC_INITIALIZED=7486537404
>
> KERNEL[297486.538325] remove
> /kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e (dlm)
> ACTION=remove
> DEVPATH=/kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e
> LOCKSPACE=62fccfd6-605f-19e6-be6d-99a1e3cb987e
> SEQNUM=3652
> SUBSYSTEM=dlm
>
> UDEV  [297486.538644] remove
> /kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e (dlm)
> ACTION=remove
> DEVPATH=/kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e
> LOCKSPACE=62fccfd6-605f-19e6-be6d-99a1e3cb987e
> SEQNUM=3652
> SUBSYSTEM=dlm
> USEC_INITIALIZED=86538345
> --snip--
>
> And from the kernel log:
>
> --snip--
> [297504.958244] md127: detected capacity change from 73340747776 to 0
> [297504.958249] md: md127 stopped.
> [297504.958884] dlm: 62fccfd6-605f-19e6-be6d-99a1e3cb987e: leaving the
> lockspace group...
> [297504.959004] udevd[487]: seq 3651 queued, 'offline' 'dlm'
> [297504.959161] udevd[487]: seq 3651 forked new worker [5392]
> [297504.959417] udevd[5392]: seq 3651 running
> [297504.959474] udevd[5392]: no db file to read
> /run/udev/data/+dlm:62fccfd6-605f-19e6-be6d-99a1e3cb987e: No such file
> or directory
> [297504.959524] udevd[5392]: passed device to netlink monitor 0x2251c30
> [297504.959527] udevd[5392]: seq 3651 processed
> [297504.960101] dlm: 62fccfd6-605f-19e6-be6d-99a1e3cb987e: group event done 0 0
> [297504.960299] dlm: 62fccfd6-605f-19e6-be6d-99a1e3cb987e:
> release_lockspace final free
> [297504.960329] md: unbind<dm-0>
> [297504.960448] udevd[487]: seq 3652 queued, 'remove' 'dlm'
> [297504.960500] udevd[487]: passed 214 byte device to netlink monitor 0x224b130
> [297504.960584] udevd[5392]: seq 3652 running
> [297504.960606] udevd[5392]: no db file to read
> /run/udev/data/+dlm:62fccfd6-605f-19e6-be6d-99a1e3cb987e: No such file
> or directory
> [297504.967168] md: export_rdev(dm-0)
> [297504.967231] md: unbind<dm-1>
> [297504.975176] md: export_rdev(dm-1)
> --snip--
>
> So that did get rid of the synthesized CHANGE event, but still no
> REMOVE event. =)
>
> Still trying to rule-out there isn't anything strange with my Linux
> distro / setup... but I assume even if udev was mishandling something,
> we should still be seeing a REMOVE event on '--stop'.
>

Getting rid of the CHANGE events is good as they can muddy the waters a
bit.

The lack of a REMOVE event suggests that something it still holding a
reference to the 'md' kobject.  There aren't many references though.
One is held when the array is active.  That is dropped by
mddev_delayed_delete() which is called from a work-queue.  So it can be
delayed a little bit, but not much.
The other references are held by member devices, and dropped when the
devices are unbound from the array - and the kernel log shows the
"unbind" messages.

In either case, if there is a reference outstanding, there should be
some evidence in the /sys/mdXX/md directory.  Either a 'dev-XX'
directory or the various bitmap attributes..

Can you report the result of

  find /sys/block/md127/md
and
  find /sys/block/md127/md -type file | xargs grep .

after stopping the array?

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]