Re: MD Remnants After --stop

[NeilBrown <neilb@suse.com>]

[-- Attachment #1: Type: text/plain, Size: 7582 bytes --]

On Thu, Nov 24 2016, Marc Smith wrote:

> On Tue, Nov 22, 2016 at 6:51 PM, NeilBrown <neilb@suse.com> wrote:
>> On Wed, Nov 23 2016, Marc Smith wrote:
>>
>>> Hi,
>>>
>>> Sorry, I'm not trying to beat a dead horse here, but I do feel
>>> something has changed... I just tested with Linux 4.5.2 and when
>>> stopping an md array (with mdadm --stop) the entry in /sys/block/ is
>>> removed, and even the /dev/mdXXX and /dev/md/name link are removed
>>> properly.
>>>
>>> When testing with Linux 4.9-rc3, the entries in /sys/block/ still
>>> remain (array_state attribute value is "clear") after using mdadm
>>> --stop and the /dev/mdXXX device exists (the /dev/md/name link is
>>> removed, by udev I assume).
>>
>> With the latest (git) mdadm, when events are reported by "udevadm monitor"??
>>
>> I only see remove events, and the entries from /dev and /sys are
>> removed.
>>
>> If I could reproduce your problem, I would fix it...
>
> On one set of hosts I can reliably reproduce this issue, and on
> another system I could previously reproduce this, but now seems to be
> working fine... I haven't found the connection (same distro / kernel
> versions on all hosts).
>
> # mdadm --version
> mdadm - v3.4-100-g52a9408 - 26th October 2016
>
>
> From 'mdadm --stop /dev/md/blah1' (non-clustered RAID0 array):
>
> --snip--
> # udevadm monitor -pku
> calling: monitor
> monitor will print the received events for:
> UDEV - the event which udev sends out after rule processing
> KERNEL - the kernel uevent
>
> KERNEL[32930.834312] change   /devices/virtual/block/md126 (block)
> ACTION=change
> DEVNAME=/dev/md126
> DEVPATH=/devices/virtual/block/md126
> DEVTYPE=disk
> MAJOR=9
> MINOR=126
> SEQNUM=3678
> SUBSYSTEM=block
>
> UDEV  [32930.836032] change   /devices/virtual/block/md126 (block)
> ACTION=change
> DEVNAME=/dev/md126
> DEVPATH=/devices/virtual/block/md126
> DEVTYPE=disk
> MAJOR=9
> MINOR=126
> SEQNUM=3678
> SUBSYSTEM=block
> SYSTEMD_READY=0
> USEC_INITIALIZED=843336612

Using the recent version of mdadm, and a kernel newer than 4.6 (commit
399146b8) a CHANGE event should only be generated when the array is
started. To see a CHANGE of stop, you must have an older mdadm or and
older kernel.... unless something else is synthesizing a CHANGE event. 

When using a clustered array, you can also see a CHANGE event when a new
disk is being added to an array.

> --snip--
>
> Kernel logs from that:
> --snip--
> [32928.465695] md126: detected capacity change from 146681102336 to 0
> [32928.465699] md: md126 stopped.
> [32928.465702] md: unbind<dm-3>
> [32928.465798] udevd[499]: inotify event: 8 for /dev/md126
> [32928.465964] udevd[499]: device /dev/md126 closed, synthesising 'change'
> [32928.466029] udevd[499]: seq 3678 queued, 'change' 'block'
> [32928.466129] udevd[499]: seq 3678 forked new worker [27035]
> [32928.466357] udevd[27035]: seq 3678 running
> [32928.466423] udevd[27035]: removing watch on '/dev/md126'
> [32928.466492] udevd[27035]: IMPORT 'probe-bcache -o udev /dev/md126'
> /usr/lib/udev/rules.d/69-bcache.rules:16
> [32928.466712] udevd[27036]: starting 'probe-bcache -o udev /dev/md126'
> [32928.467540] udevd[27035]: 'probe-bcache -o udev /dev/md126' [27036]
> exit with return code 0
> [32928.467564] udevd[27035]: update old name,
> '/dev/disk/by-id/md-name-tgtnode1.parodyne.com:blah1' no longer
> belonging to '/devices/virtual/block/md126'
> [32928.470851] md: export_rdev(dm-3)
> [32928.470920] md: unbind<dm-2>
> [32928.478843] md: export_rdev(dm-2)
> --snip--
>
>
> From 'mdadm --stop /dev/md/asdf1' (clustered RAID1 array):
>
> --snip--
> # udevadm monitor -pku
> calling: monitor
> monitor will print the received events for:
> UDEV - the event which udev sends out after rule processing
> KERNEL - the kernel uevent
>
> KERNEL[34402.247229] change   /devices/virtual/block/md127 (block)
> ACTION=change
> DEVNAME=/dev/md127
> DEVPATH=/devices/virtual/block/md127
> DEVTYPE=disk
> MAJOR=9
> MINOR=127
> SEQNUM=3679
> SUBSYSTEM=block
>
> KERNEL[34402.247885] offline
> /kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e (dlm)
> ACTION=offline
> DEVPATH=/kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e
> LOCKSPACE=62fccfd6-605f-19e6-be6d-99a1e3cb987e
> SEQNUM=3680
> SUBSYSTEM=dlm
>
> UDEV  [34402.248269] offline
> /kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e (dlm)
> ACTION=offline
> DEVPATH=/kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e
> LOCKSPACE=62fccfd6-605f-19e6-be6d-99a1e3cb987e
> SEQNUM=3680
> SUBSYSTEM=dlm
> USEC_INITIALIZED=402248230
>
> KERNEL[34402.248841] remove
> /kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e (dlm)
> ACTION=remove
> DEVPATH=/kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e
> LOCKSPACE=62fccfd6-605f-19e6-be6d-99a1e3cb987e
> SEQNUM=3681
> SUBSYSTEM=dlm
>
> UDEV  [34402.248899] change   /devices/virtual/block/md127 (block)
> ACTION=change
> DEVNAME=/dev/md127
> DEVPATH=/devices/virtual/block/md127
> DEVTYPE=disk
> MAJOR=9
> MINOR=127
> SEQNUM=3679
> SUBSYSTEM=block
> SYSTEMD_READY=0
> USEC_INITIALIZED=1273670
>
> UDEV  [34402.248990] remove
> /kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e (dlm)
> ACTION=remove
> DEVPATH=/kernel/dlm/62fccfd6-605f-19e6-be6d-99a1e3cb987e
> LOCKSPACE=62fccfd6-605f-19e6-be6d-99a1e3cb987e
> SEQNUM=3681
> SUBSYSTEM=dlm
> USEC_INITIALIZED=2248905
> --snip--
>
>
> Kernel logs from that:
> --snip--
> [34399.753876] udevd[499]: inotify event: 8 for /dev/md127
> [34399.765389] md127: detected capacity change from 73340747776 to 0
> [34399.765393] md: md127 stopped.
> [34399.765579] udevd[499]: device /dev/md127 closed, synthesising 'change'

That is weird.  udev is synthesizing a CHANGE event.  How nice of it
(not!).
It doesn't do this for dm, only other devices.

This happens when an open-for-write is closed.
If you edit mdopen.c and change the O_RDWR in open_mddev() to O_RDONLY,
this change will go away.


> [34399.765656] udevd[499]: seq 3679 queued, 'change' 'block'
> [34399.765751] udevd[499]: seq 3679 forked new worker [6317]
> [34399.765878] udevd[6317]: seq 3679 running
> [34399.765943] udevd[6317]: removing watch on '/dev/md127'
> [34399.766012] udevd[6317]: IMPORT 'probe-bcache -o udev /dev/md127'
> /usr/lib/udev/rules.d/69-bcache.rules:16
> [34399.766259] udevd[6318]: starting 'probe-bcache -o udev /dev/md127'
> [34399.766295] dlm: 62fccfd6-605f-19e6-be6d-99a1e3cb987e: leaving the
> lockspace group...
> [34399.766421] udevd[499]: seq 3680 queued, 'offline' 'dlm'
> [34399.766549] udevd[499]: seq 3680 forked new worker [6319]
> [34399.767080] dlm: 62fccfd6-605f-19e6-be6d-99a1e3cb987e: group event done 0 0
> [34399.767297] dlm: 62fccfd6-605f-19e6-be6d-99a1e3cb987e:
> release_lockspace final free
> [34399.767320] md: unbind<dm-1>
> [34399.795574] md: export_rdev(dm-1)
> [34399.795640] md: unbind<dm-0>
> [34399.803565] md: export_rdev(dm-0)
> --snip--
>
>
> On the other machines where the md array stopped correctly (removing
> the entries from /dev and /sys) I do see the 'remove' events with
> "udevadm monitor". What produces those remove events? Is that
> something directly from the mdadm tool, or indirectly as part of the
> stop/tear-down that mdadm initiates?

REMOVE events are generated by

md_free() -> del_gendisk() ->  blk_unregister_queue()

which should happen when the md object is finally released.
If there is no CHANGE event, then nothing should stop this from
happening.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]