Re: [REGRESSION] 4.11-rc: systemd doesn't see most devices

[Greg KH <gregkh@linuxfoundation.org>]

On Tue, Apr 11, 2017 at 04:31:48PM -0400, Theodore Ts'o wrote:
> On Tue, Apr 11, 2017 at 05:38:35PM +0200, Greg KH wrote:
> > I haven't seen this at all, nor heard of it.  As systemctl only gets
> > what udev reports to it, have you tried using 'udevadm' to monitor your
> > devices when you plug them in, to ensure it is really seeing them?
> 
> The problem is that the problem happens at boot, so I can't really use
> "udevadm monitor" --- so I'm not sure whats the best way to debug
> this.  I can seen some journalctl logs which do show that it's not
> detecting the dm-crypt volume --- but that's insane, because my root
> partition is also on the dm-crypt, and it was unlocked in the initrd.
> So systemd and udev might not think it exists, but it most definitely
> does --- or the boot wouldn't have been able to proceed at all.  
> 
> In any case, here is the "udevadm info -e" output from a good and bad
> boot, as well as a dmesg from a good and a bad boot.  I'm not seeing
> anything obvious, but it does seem interesting that "udevadm info -e"
> shows a lot of devices which "systemctl | grep device" doesn't.  Is
> there any recent change in the kernel's interfaces that udev depends
> on that might make a difference?  For that matter, what does udev
> depend on?  Should I be looking at differences in sysfs?  Or does udev
> use something else?

I don't know how things get from udevd to systemd, sorry, you should ask
on the systemd mailing list, it's been a long time since I looked at
that codebase.

If udevadm is showing the same data, I don't think it's a udev or kernel
issue here, and given your strange git bisect, it must be a timing thing
somewhere (as your kernel builds shouldn't have mattered if you have no
staging drivers enabled.)

udevd uses netlink to receive the hotplug events, and then feeds them
back into the kernel and uses netlink again to send the messages out to
whomever is listening for them (like systemd).  As this is all failing
during boot, is this an initramfs issue somehow?

I don't think netlink has changed anything recently, but I haven't
looked at the kernel code path either in a long time, sorry.

heisenburgs, no fun to debug :(

greg k-h