Re: [BUG] 4.14 cannot find configured disks/partitions

Re: [BUG] 4.14 cannot find configured disks/partitions

[Bob Tracy]

On Tue, Nov 28, 2017 at 11:49:55PM -0600, Bob Tracy wrote:
> (...)
> 
> Upon trying to reboot on my 4.13 kernel, I discovered *it's* now broken
> as well, thanks to a recent udev update :-(.  Now I get a bunch of
> timeouts for all the filesystems (including the swap partition) not
> mounted immediately at boot time.  "journalctl -xb" is littered with
> applicable messages of the form
> 
> dev-sdaX.device: Job dev-sdaX.device/start timed out.
> Timed out waiting for device dev-sdaX.device.
> 
> In systemd's emergency mode, I can manually mount (or swapon as
> appropriate) the various sdaX partitions.

I *thought* this particular scenario sounded familiar.  Nearly three
years ago, this problem showed up when "systemd" and "udev" became
dependent on CONFIG_FHANDLE being set in kernel builds.  I didn't
suddenly start unsetting it again, so what changed in 2017?

--Bob

Re: [BUG] 4.14 cannot find configured disks/partitions

[Michael Cree]

On Tue, Nov 28, 2017 at 11:49:55PM -0600, Bob Tracy wrote:
> On Thu, Nov 23, 2017 at 11:10:10PM -0600, Bob Tracy wrote:
> > Perhaps the subject isn't entirely accurate, but that's what seems to be
> > the case.  After loading the initial ramdisk, the boot process stalls
> > (loops indefinitely) with "mdadm" complaining about not being able to scan 
> > the disks defined in its configuration file, which is bone-stock.  What
> > makes this particularly infuriating is, I don't have anything configured
> > for "mdadm" to worry about, which is what's normally found, i.e.,
> > nothing of interest.
> > 
> > The 4.13 kernel loads and boots just fine with exactly the same
> > up-to-date (unstable/experimental) userspace libraries and utilities.
> > 
> > Bottom line: the 4.14 kernel doesn't seem to detect my SCSI disks for
> > some reason.  All the recent patches (end of October time frame) for PCI
> > on alpha (I *think* mostly having to do with addressing some weird kind
> > of libata conflict) didn't make any difference.

Just got a chance to try out 4.14 on an Alpha and look like firmware
for built-in drivers are being treated differently.  I presume you
have a qlogic scsi driver.  Make sure qlogic/1040.bin (or whatever is
appropriate for your scsi driver) is listed under Drivers->Generic
Driver Options->Include in-kernel firmware blobs.

Cheers
Michael.

Re: [BUG] 4.14 cannot find configured disks/partitions

[Bob Tracy]

On Sat, Dec 02, 2017 at 10:16:28AM +1300, Michael Cree wrote:
> On Tue, Nov 28, 2017 at 11:49:55PM -0600, Bob Tracy wrote:
> > On Thu, Nov 23, 2017 at 11:10:10PM -0600, Bob Tracy wrote:
> > > Perhaps the subject isn't entirely accurate, but that's what seems to be
> > > the case.  After loading the initial ramdisk, the boot process stalls
> > > (loops indefinitely) with "mdadm" complaining about not being able to scan 
> > > the disks defined in its configuration file, which is bone-stock.  What
> > > makes this particularly infuriating is, I don't have anything configured
> > > for "mdadm" to worry about, which is what's normally found, i.e.,
> > > nothing of interest.
> > > 
> > > The 4.13 kernel loads and boots just fine with exactly the same
> > > up-to-date (unstable/experimental) userspace libraries and utilities.
> > > 
> > > Bottom line: the 4.14 kernel doesn't seem to detect my SCSI disks for
> > > some reason.  All the recent patches (end of October time frame) for PCI
> > > on alpha (I *think* mostly having to do with addressing some weird kind
> > > of libata conflict) didn't make any difference.
> 
> Just got a chance to try out 4.14 on an Alpha and look like firmware
> for built-in drivers are being treated differently.  I presume you
> have a qlogic scsi driver.  Make sure qlogic/1040.bin (or whatever is
> appropriate for your scsi driver) is listed under Drivers->Generic
> Driver Options->Include in-kernel firmware blobs.

Glad I saw this before trying a freshly-built 4.14 with a few more
kernel config options enabled per the README file for systemd.  Doesn't
mean there isn't a systemd/udev issue as well, but if the firmware for
the SCSI controller isn't getting loaded, that makes detection of the
host and attached devices pretty much a non-starter.

Thanks!
--Bob

Re: [BUG] 4.14 cannot find configured disks/partitions

[Bob Tracy]

On Fri, Dec 01, 2017 at 06:22:50PM -0600, Bob Tracy wrote:
> On Sat, Dec 02, 2017 at 10:16:28AM +1300, Michael Cree wrote:
> > Just got a chance to try out 4.14 on an Alpha and look like firmware
> > for built-in drivers are being treated differently.  I presume you
> > have a qlogic scsi driver.  Make sure qlogic/1040.bin (or whatever is
> > appropriate for your scsi driver) is listed under Drivers->Generic
> > Driver Options->Include in-kernel firmware blobs.
> 
> Glad I saw this before trying a freshly-built 4.14 with a few more
> kernel config options enabled per the README file for systemd.  Doesn't
> mean there isn't a systemd/udev issue as well, but if the firmware for
> the SCSI controller isn't getting loaded, that makes detection of the
> host and attached devices pretty much a non-starter.

That was *it*.  Up and running on 4.14 with no issues.  Have reported
success and requested closure of bug #883089.

FYI, "1040.bin" *is* the correct firmware for the Qlogic ISP1020 card
as confirmed by examination of the driver source.

Oddly enough, I've been including the firmware for the video card as
part of the kernel build since forever: the console wouldn't initialize
properly without it.  Never occurred to me the same thing might be
needed for the built-in SCSI driver, mostly because everything worked
great up through kernel version 4.13.

Thanks again.

--Bob

Re: [BUG] 4.14 cannot find configured disks/partitions

[Bob Tracy]

Debian bug #883089 opened for this issue.

Assuming there's a 4.13 or later Debian kernel for alpha available and
anyone reading this is brave enough to try it, I'd be interested in
knowing if it boots properly.  Without a "yes, it works" from someone,
I'm extremely reluctant to give it a try myself: been burned too many
times trying to boot the Debian kernels on my system :-(.

Many thanks in advance for anyone's time and trouble in helping me get
to the bottom of this issue.

--Bob

Re: [BUG] 4.14 cannot find configured disks/partitions

[Bob Tracy]

On Thu, Nov 23, 2017 at 11:10:10PM -0600, Bob Tracy wrote:
> Perhaps the subject isn't entirely accurate, but that's what seems to be
> the case.  After loading the initial ramdisk, the boot process stalls
> (loops indefinitely) with "mdadm" complaining about not being able to scan 
> the disks defined in its configuration file, which is bone-stock.  What
> makes this particularly infuriating is, I don't have anything configured
> for "mdadm" to worry about, which is what's normally found, i.e.,
> nothing of interest.
> 
> The 4.13 kernel loads and boots just fine with exactly the same
> up-to-date (unstable/experimental) userspace libraries and utilities.
> 
> Bottom line: the 4.14 kernel doesn't seem to detect my SCSI disks for
> some reason.  All the recent patches (end of October time frame) for PCI
> on alpha (I *think* mostly having to do with addressing some weird kind
> of libata conflict) didn't make any difference.
> 
> Any idea what's causing this?  Thanks...

Another follow-up...  Tried booting on 4.14.0 (final) this evening, not
expecting that the upgrade from -rc8 would make any difference.  It
didn't.  The SCSI host adapter is not being detected for whatever reason.
The only device present on the system that's visible when I do
"cat /proc/scsi/scsi" in the initramfs shell is the Toshiba IDE cdrom
device.  Normally, I'd also see the real SCSI host adapter and its
associated SCSI disk.

Upon trying to reboot on my 4.13 kernel, I discovered *it's* now broken
as well, thanks to a recent udev update :-(.  Now I get a bunch of
timeouts for all the filesystems (including the swap partition) not
mounted immediately at boot time.  "journalctl -xb" is littered with
applicable messages of the form

dev-sdaX.device: Job dev-sdaX.device/start timed out.
Timed out waiting for device dev-sdaX.device.

In systemd's emergency mode, I can manually mount (or swapon as
appropriate) the various sdaX partitions.

USB detection/startup is similarly broken.  I had to manually load the
usb core and host modules.

So...  It appears there's something currently badly amiss in the
systemd+udev universe.  Such are the hazards of running the
unstable/experimental distribution, and I suppose I should consider
myself fortunate I can even get to a mostly-functional multi-user run
state at this point.

Did I miss a discussion somewhere about "/dev/sdaX" in "/etc/fstab"
being deprecated?  Moving to UUIDs might help with the current 4.13
brokenness, but won't solve the host adapter detection issue in 4.14.

--Bob

[BUG] 4.14 cannot find configured disks/partitions

[Bob Tracy]

(Forgot to include "linux-alpha" in the original recipient list, so
the next few messages are an attempt to fix that.  Please copy
"debian-alpha" on any replies.)

Perhaps the subject isn't entirely accurate, but that's what seems to be
the case.  After loading the initial ramdisk, the boot process stalls
(loops indefinitely) with "mdadm" complaining about not being able to scan 
the disks defined in its configuration file, which is bone-stock.  What
makes this particularly infuriating is, I don't have anything configured
for "mdadm" to worry about, which is what's normally found, i.e.,
nothing of interest.

The 4.13 kernel loads and boots just fine with exactly the same
up-to-date (unstable/experimental) userspace libraries and utilities.

Bottom line: the 4.14 kernel doesn't seem to detect my SCSI disks for
some reason.  All the recent patches (end of October time frame) for PCI
on alpha (I *think* mostly having to do with addressing some weird kind
of libata conflict) didn't make any difference.

Any idea what's causing this?  Thanks...

--Bob