Who reordered my disks (probably v4.8-rc1 problem)

Who reordered my disks (probably v4.8-rc1 problem)

[Pavel Machek]

Hi!

It seems that in v4.8-rc0, /dev/sdX got reordered, and now USB devices
are probed before SATA drivers. That is pretty anti-social. It
broke my boot on my primary machine, and unfortunately due to BIOS
problems (keyboard does not work when connected through a hub) it is
less fun than it should be.

Best regards,

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Who reordered my disks (probably v4.8-rc1 problem)

[james harvey]

Use static (persistent) naming instead, /dev/disk/by-label,
/dev/disk/by-id, /dev/disk/by-uuid, and if gpt /dev/disk/by-partlabel
and /dev/disk/by-partuuid

As you found, /dev/sdX bus names are assigned in the order they are
added, which for some time has not been guaranteed to remain
consistent between kernel versions or even subsequent boots on the
same kernel.

On Sat, Aug 13, 2016 at 3:30 PM, Pavel Machek <pavel@ucw.cz> wrote:
> Hi!
>
> It seems that in v4.8-rc0, /dev/sdX got reordered, and now USB devices
> are probed before SATA drivers. That is pretty anti-social. It
> broke my boot on my primary machine, and unfortunately due to BIOS
> problems (keyboard does not work when connected through a hub) it is
> less fun than it should be.
>
> Best regards,
>
>                                                                         Pavel
> --
> (english) http://www.livejournal.com/~pavelmachek
> (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Who reordered my disks (probably v4.8-rc1 problem)

[Pavel Machek]

Hi!

> Use static (persistent) naming instead, /dev/disk/by-label,
> /dev/disk/by-id, /dev/disk/by-uuid, and if gpt /dev/disk/by-partlabel
> and /dev/disk/by-partuuid

Explain that to my bootloader. Kernel needs root= on a command line.

> As you found, /dev/sdX bus names are assigned in the order they are
> added, which for some time has not been guaranteed to remain
> consistent between kernel versions or even subsequent boots on the
> same kernel.

Well, for usb, order is not guaranteed, for SATA, it worked. So that's
a regression in v4.8-rc1.

Yes, /dev/disk/by-* is good idea, but this broke my boot, probably
broke some scripts that used for a long time... and kernel may not
break working systems.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Who reordered my disks (probably v4.8-rc1 problem)

[james harvey]

On Sun, Aug 14, 2016 at 5:09 AM, Pavel Machek <pavel@ucw.cz> wrote:
>> Use static (persistent) naming instead, /dev/disk/by-label,
>> /dev/disk/by-id, /dev/disk/by-uuid, and if gpt /dev/disk/by-partlabel
>> and /dev/disk/by-partuuid
>
> Explain that to my bootloader. Kernel needs root= on a command line.

It does need a root parameter.  Use root=PARTUUID=<partition UUID>.
Note the UUID is the PARTUUID from "blkid" which is for the partition,
not the UUID field which is for the filesystem.


>> As you found, /dev/sdX bus names are assigned in the order they are
>> added, which for some time has not been guaranteed to remain
>> consistent between kernel versions or even subsequent boots on the
>> same kernel.
>
> Well, for usb, order is not guaranteed, for SATA, it worked. So that's
> a regression in v4.8-rc1.
>
> Yes, /dev/disk/by-* is good idea, but this broke my boot, probably
> broke some scripts that used for a long time... and kernel may not
> break working systems.

It's always possible others may disagree, but I don't think something
is a regression when what breaks has long been said to not be expected
to work.

Re: Who reordered my disks (probably v4.8-rc1 problem)

[Bob Tracy]

On Sun, Aug 14, 2016 at 05:03:33AM -0400, james harvey wrote:
> Use static (persistent) naming instead, /dev/disk/by-label,
> /dev/disk/by-id, /dev/disk/by-uuid, and if gpt /dev/disk/by-partlabel
> and /dev/disk/by-partuuid

The key takeaway from the above is, for the non-GPT case, userspace
support (blkid and friends) is required to take advantage of static
(persistent) naming, i.e., an initial ramdisk of some kind is no
longer optional (boo! hiss!).

There's an implied requirement for udev as well, but that's not nearly
as distasteful.  I had to make my peace with udev on an otherwise
primitive (simple) Slackware system a few years ago when I upgraded my
system hardware: included was a sound device with an HDMI interface,
and the associated ALSA driver only supported dynamic minor devices.

--Bob

Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Pavel Machek]

Hi!

> It seems that in v4.8-rc0, /dev/sdX got reordered, and now USB devices
> are probed before SATA drivers. That is pretty anti-social. It
> broke my boot on my primary machine, and unfortunately due to BIOS
> problems (keyboard does not work when connected through a hub) it is
> less fun than it should be.

If you know which commit caused	the reordering, that would be helpful.

v4.1 seems to be ok: SATA disk is sda, as expected.

v4.4 seems to be ok: SATA disk is sda, as expected.

I'll test v4.6 next.

v4.8-rc1: SATA disk is sde, behind USB card readers. Not helpful.

Best regards,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Tom Yan]

Since when it is expected that SATA disks will always be probed before
USB disks? We can't guarantee that even if we make sure all ata
drivers are loaded before usb-storage/uas. That's why we need
consistent namings (e.g. /dev/disk/by-id/*).

On 14 August 2016 at 17:20, Pavel Machek <pavel@ucw.cz> wrote:
> Hi!
>
>> It seems that in v4.8-rc0, /dev/sdX got reordered, and now USB devices
>> are probed before SATA drivers. That is pretty anti-social. It
>> broke my boot on my primary machine, and unfortunately due to BIOS
>> problems (keyboard does not work when connected through a hub) it is
>> less fun than it should be.
>
> If you know which commit caused the reordering, that would be helpful.
>
> v4.1 seems to be ok: SATA disk is sda, as expected.
>
> v4.4 seems to be ok: SATA disk is sda, as expected.
>
> I'll test v4.6 next.
>
> v4.8-rc1: SATA disk is sde, behind USB card readers. Not helpful.
>
> Best regards,
>                                                                         Pavel
> --
> (english) http://www.livejournal.com/~pavelmachek
> (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ide" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Pavel Machek]

On Sun 2016-08-14 17:34:18, Tom Yan wrote:
> Since when it is expected that SATA disks will always be probed before
> USB disks? We can't guarantee that even if we make sure all ata
> drivers are loaded before usb-storage/uas. That's why we need
> consistent namings (e.g. /dev/disk/by-id/*).

Since SATA support was merged, certainly since v2.4, and from way
before /dev/disk/by-id existed.

People may not run udev, and you can't use /dev/disk/by-id on kernel
command line.

								Pavel

> On 14 August 2016 at 17:20, Pavel Machek <pavel@ucw.cz> wrote:
> > Hi!
> >
> >> It seems that in v4.8-rc0, /dev/sdX got reordered, and now USB devices
> >> are probed before SATA drivers. That is pretty anti-social. It
> >> broke my boot on my primary machine, and unfortunately due to BIOS
> >> problems (keyboard does not work when connected through a hub) it is
> >> less fun than it should be.
> >
> > If you know which commit caused the reordering, that would be helpful.
> >
> > v4.1 seems to be ok: SATA disk is sda, as expected.
> >
> > v4.4 seems to be ok: SATA disk is sda, as expected.
> >
> > I'll test v4.6 next.
> >
> > v4.8-rc1: SATA disk is sde, behind USB card readers. Not helpful.
> >
> > Best regards,
> >                                                                         Pavel
> > --
> > (english) http://www.livejournal.com/~pavelmachek
> > (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-ide" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Tom Yan]

On 14 August 2016 at 18:01, Pavel Machek <pavel@ucw.cz> wrote:
>
> Since SATA support was merged, certainly since v2.4, and from way
> before /dev/disk/by-id existed.

I have no idea how "SATA before USB" had been done in the past (if it
was ever a thing in the kernel), but that has not been the case since
at least v3.0 AFAIR.

>
> People may not run udev, and you can't use /dev/disk/by-id on kernel
> command line.
>

No, but you can always use root=PARTUUID=, that's built into the
kernel. (root=UUID= requires udev or so though).

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Tom Yan]

On 14 August 2016 at 18:07, Tom Yan <tom.ty89@gmail.com> wrote:
> On 14 August 2016 at 18:01, Pavel Machek <pavel@ucw.cz> wrote:
>>
>> Since SATA support was merged, certainly since v2.4, and from way
>> before /dev/disk/by-id existed.
>
> I have no idea how "SATA before USB" had been done in the past (if it
> was ever a thing in the kernel), but that has not been the case since
> at least v3.0 AFAIR.
>
>>
>> People may not run udev, and you can't use /dev/disk/by-id on kernel
>> command line.
>>
>
> No, but you can always use root=PARTUUID=, that's built into the
> kernel. (root=UUID= requires udev or so though).

Silly me. root=UUID= has nothing to do with udev, but `blkid` in
util-linux. At least that's how it's done in Arch/mkinitcpio.

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[David Lang]

On Sun, 14 Aug 2016, Tom Yan wrote:

> On 14 August 2016 at 18:07, Tom Yan <tom.ty89@gmail.com> wrote:
>> On 14 August 2016 at 18:01, Pavel Machek <pavel@ucw.cz> wrote:
>>>
>>> Since SATA support was merged, certainly since v2.4, and from way
>>> before /dev/disk/by-id existed.
>>
>> I have no idea how "SATA before USB" had been done in the past (if it
>> was ever a thing in the kernel), but that has not been the case since
>> at least v3.0 AFAIR.
>>
>>>
>>> People may not run udev, and you can't use /dev/disk/by-id on kernel
>>> command line.
>>>
>>
>> No, but you can always use root=PARTUUID=, that's built into the
>> kernel. (root=UUID= requires udev or so though).
>
> Silly me. root=UUID= has nothing to do with udev, but `blkid` in
> util-linux. At least that's how it's done in Arch/mkinitcpio.
>

The rule is "don't break working systems", not "but we are allowed to break 
systems, see it says here not to depend on this"

Drive ordering has been stable since the 0.1 kernel [1]

It takes a lot longer to detect USB drives, why in the world would they be 
detected before hard-wired drives?

I expect that Linus' response is going to be very quotable.

David Lang


[1] given stable hardware and no new drivers becoming involved

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Tom Yan]

Btw, why hasn't this been CC'd to linux-scsi at the very least? The
SCSI disk (sd) driver is obviously the sociopath here. It should
really differentiate disks from libata and usb-storage/uas, and wait
for at least a minute to see if there's gonna be an ATA drive popping
up before enumerating disks from the latter. Sociopaths like me that
put the root filesystem on an UAS drive should really be ignored.
Wait, there's NVMe! Problem solved.

On 14 August 2016 at 10:26, David Lang <david@lang.hm> wrote:
> On Sun, 14 Aug 2016, Tom Yan wrote:
>
>> On 14 August 2016 at 18:07, Tom Yan <tom.ty89@gmail.com> wrote:
>>>
>>> On 14 August 2016 at 18:01, Pavel Machek <pavel@ucw.cz> wrote:
>>>>
>>>>
>>>> Since SATA support was merged, certainly since v2.4, and from way
>>>> before /dev/disk/by-id existed.
>>>
>>>
>>> I have no idea how "SATA before USB" had been done in the past (if it
>>> was ever a thing in the kernel), but that has not been the case since
>>> at least v3.0 AFAIR.
>>>
>>>>
>>>> People may not run udev, and you can't use /dev/disk/by-id on kernel
>>>> command line.
>>>>
>>>
>>> No, but you can always use root=PARTUUID=, that's built into the
>>> kernel. (root=UUID= requires udev or so though).
>>
>>
>> Silly me. root=UUID= has nothing to do with udev, but `blkid` in
>> util-linux. At least that's how it's done in Arch/mkinitcpio.
>>
>
> The rule is "don't break working systems", not "but we are allowed to break
> systems, see it says here not to depend on this"
>
> Drive ordering has been stable since the 0.1 kernel [1]
>
> It takes a lot longer to detect USB drives, why in the world would they be
> detected before hard-wired drives?
>
> I expect that Linus' response is going to be very quotable.
>
> David Lang
>
>
> [1] given stable hardware and no new drivers becoming involved

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Oliver Neukum]

On Sun, 2016-08-14 at 10:53 +0000, Tom Yan wrote:
> Btw, why hasn't this been CC'd to linux-scsi at the very least? The
> SCSI disk (sd) driver is obviously the sociopath here. It should
> really differentiate disks from libata and usb-storage/uas, and wait
> for at least a minute to see if there's gonna be an ATA drive popping
> up before enumerating disks from the latter. Sociopaths like me that
> put the root filesystem on an UAS drive should really be ignored.
> Wait, there's NVMe! Problem solved.

We most certainly cannot introduce such a delay. If this really
must be done, we need synchronisation primitives between the
subsystemes, or we need stable names in kernel space.

	Regards
		Oliver

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Greg KH]

On Sun, Aug 14, 2016 at 03:26:45AM -0700, David Lang wrote:
> On Sun, 14 Aug 2016, Tom Yan wrote:
> 
> > On 14 August 2016 at 18:07, Tom Yan <tom.ty89-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > > On 14 August 2016 at 18:01, Pavel Machek <pavel-+ZI9xUNit7I@public.gmane.org> wrote:
> > > > 
> > > > Since SATA support was merged, certainly since v2.4, and from way
> > > > before /dev/disk/by-id existed.
> > > 
> > > I have no idea how "SATA before USB" had been done in the past (if it
> > > was ever a thing in the kernel), but that has not been the case since
> > > at least v3.0 AFAIR.
> > > 
> > > > 
> > > > People may not run udev, and you can't use /dev/disk/by-id on kernel
> > > > command line.
> > > > 
> > > 
> > > No, but you can always use root=PARTUUID=, that's built into the
> > > kernel. (root=UUID= requires udev or so though).
> > 
> > Silly me. root=UUID= has nothing to do with udev, but `blkid` in
> > util-linux. At least that's how it's done in Arch/mkinitcpio.
> > 
> 
> The rule is "don't break working systems", not "but we are allowed to break
> systems, see it says here not to depend on this"
> 
> Drive ordering has been stable since the 0.1 kernel [1]

Drive probing order of USB has always been non-deterministic, so while I
agree that it is not good to break existing systems at all, perhaps this
is on the edge of what works vs. doesn't work?

I know my USB drives always seem to come up in random order, which is
why tools like udev were invented :)

> It takes a lot longer to detect USB drives, why in the world would they be
> detected before hard-wired drives?

Depends, some hard-wired drives take much longer to find than USB ones.

That being said, it would be great if the original reporter could use
'git bisect' and let the linux-usb and linux-scsi mailing list know what
the offending patch is, and we can take it from there.

thanks,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Greg KH]

On Sun, Aug 14, 2016 at 03:26:45AM -0700, David Lang wrote:
> On Sun, 14 Aug 2016, Tom Yan wrote:
> 
> > On 14 August 2016 at 18:07, Tom Yan <tom.ty89@gmail.com> wrote:
> > > On 14 August 2016 at 18:01, Pavel Machek <pavel@ucw.cz> wrote:
> > > > 
> > > > Since SATA support was merged, certainly since v2.4, and from way
> > > > before /dev/disk/by-id existed.
> > > 
> > > I have no idea how "SATA before USB" had been done in the past (if it
> > > was ever a thing in the kernel), but that has not been the case since
> > > at least v3.0 AFAIR.
> > > 
> > > > 
> > > > People may not run udev, and you can't use /dev/disk/by-id on kernel
> > > > command line.
> > > > 
> > > 
> > > No, but you can always use root=PARTUUID=, that's built into the
> > > kernel. (root=UUID= requires udev or so though).
> > 
> > Silly me. root=UUID= has nothing to do with udev, but `blkid` in
> > util-linux. At least that's how it's done in Arch/mkinitcpio.
> > 
> 
> The rule is "don't break working systems", not "but we are allowed to break
> systems, see it says here not to depend on this"
> 
> Drive ordering has been stable since the 0.1 kernel [1]

Drive probing order of USB has always been non-deterministic, so while I
agree that it is not good to break existing systems at all, perhaps this
is on the edge of what works vs. doesn't work?

I know my USB drives always seem to come up in random order, which is
why tools like udev were invented :)

> It takes a lot longer to detect USB drives, why in the world would they be
> detected before hard-wired drives?

Depends, some hard-wired drives take much longer to find than USB ones.

That being said, it would be great if the original reporter could use
'git bisect' and let the linux-usb and linux-scsi mailing list know what
the offending patch is, and we can take it from there.

thanks,

greg k-h

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Pavel Machek]

Hi!

> > > > I have no idea how "SATA before USB" had been done in the past (if it
> > > > was ever a thing in the kernel), but that has not been the case since
> > > > at least v3.0 AFAIR.
> > > > 
> > > > > 
> > > > > People may not run udev, and you can't use /dev/disk/by-id on kernel
> > > > > command line.
> > > > > 
> > > > 
> > > > No, but you can always use root=PARTUUID=, that's built into the
> > > > kernel. (root=UUID= requires udev or so though).
> > > 
> > > Silly me. root=UUID= has nothing to do with udev, but `blkid` in
> > > util-linux. At least that's how it's done in Arch/mkinitcpio.
> > 
> > The rule is "don't break working systems", not "but we are allowed to break
> > systems, see it says here not to depend on this"
> > 
> > Drive ordering has been stable since the 0.1 kernel [1]
> 
> Drive probing order of USB has always been non-deterministic, so while I
> agree that it is not good to break existing systems at all, perhaps this
> is on the edge of what works vs. doesn't work?

Yeah, USB order is known to be random. But root=/dev/sda (when sda is
on SATA) is very old, and it would be good to keep it.

> I know my USB drives always seem to come up in random order, which is
> why tools like udev were invented :)
> 
> > It takes a lot longer to detect USB drives, why in the world would they be
> > detected before hard-wired drives?
> 
> Depends, some hard-wired drives take much longer to find than USB ones.
> 
> That being said, it would be great if the original reporter could use
> 'git bisect' and let the linux-usb and linux-scsi mailing list know what
> the offending patch is, and we can take it from there.

Original reporter is me :-(.

Yes, I can do bisect, if required. I'd like some kind of confirmation
that it happens on other systems...

Best regards,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Greg KH]

On Sun, Aug 14, 2016 at 02:03:27PM +0200, Pavel Machek wrote:
> Hi!
> 
> > > > > I have no idea how "SATA before USB" had been done in the past (if it
> > > > > was ever a thing in the kernel), but that has not been the case since
> > > > > at least v3.0 AFAIR.
> > > > > 
> > > > > > 
> > > > > > People may not run udev, and you can't use /dev/disk/by-id on kernel
> > > > > > command line.
> > > > > > 
> > > > > 
> > > > > No, but you can always use root=PARTUUID=, that's built into the
> > > > > kernel. (root=UUID= requires udev or so though).
> > > > 
> > > > Silly me. root=UUID= has nothing to do with udev, but `blkid` in
> > > > util-linux. At least that's how it's done in Arch/mkinitcpio.
> > > 
> > > The rule is "don't break working systems", not "but we are allowed to break
> > > systems, see it says here not to depend on this"
> > > 
> > > Drive ordering has been stable since the 0.1 kernel [1]
> > 
> > Drive probing order of USB has always been non-deterministic, so while I
> > agree that it is not good to break existing systems at all, perhaps this
> > is on the edge of what works vs. doesn't work?
> 
> Yeah, USB order is known to be random. But root=/dev/sda (when sda is
> on SATA) is very old, and it would be good to keep it.
> 
> > I know my USB drives always seem to come up in random order, which is
> > why tools like udev were invented :)
> > 
> > > It takes a lot longer to detect USB drives, why in the world would they be
> > > detected before hard-wired drives?
> > 
> > Depends, some hard-wired drives take much longer to find than USB ones.
> > 
> > That being said, it would be great if the original reporter could use
> > 'git bisect' and let the linux-usb and linux-scsi mailing list know what
> > the offending patch is, and we can take it from there.
> 
> Original reporter is me :-(.
> 
> Yes, I can do bisect, if required. I'd like some kind of confirmation
> that it happens on other systems...

Please use git-bisect, you are the first one reporting this.

thanks,

greg k-h

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Alan Stern]

On Sun, 14 Aug 2016, Pavel Machek wrote:

> > That being said, it would be great if the original reporter could use
> > 'git bisect' and let the linux-usb and linux-scsi mailing list know what
> > the offending patch is, and we can take it from there.
> 
> Original reporter is me :-(.
> 
> Yes, I can do bisect, if required. I'd like some kind of confirmation
> that it happens on other systems...

Can you post a kernel log from one of those unsuccessful boots
(netconsole or something or the sort)?

Alan Stern

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Alan Stern]

On Sun, 14 Aug 2016, Pavel Machek wrote:

> > That being said, it would be great if the original reporter could use
> > 'git bisect' and let the linux-usb and linux-scsi mailing list know what
> > the offending patch is, and we can take it from there.
> 
> Original reporter is me :-(.
> 
> Yes, I can do bisect, if required. I'd like some kind of confirmation
> that it happens on other systems...

Can you post a kernel log from one of those unsuccessful boots
(netconsole or something or the sort)?

Alan Stern

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Pavel Machek]

On Sun 2016-08-14 10:56:54, Alan Stern wrote:
> On Sun, 14 Aug 2016, Pavel Machek wrote:
> 
> > > That being said, it would be great if the original reporter could use
> > > 'git bisect' and let the linux-usb and linux-scsi mailing list know what
> > > the offending patch is, and we can take it from there.
> > 
> > Original reporter is me :-(.
> > 
> > Yes, I can do bisect, if required. I'd like some kind of confirmation
> > that it happens on other systems...
> 
> Can you post a kernel log from one of those unsuccessful boots
> (netconsole or something or the sort)?

Even scrollback does not seem to work well on modern kernels. So
sorry, no easy way to get a kernel log :-(.
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Pavel Machek]

Hi!

On Sun 2016-08-14 18:17:39, Tom Yan wrote:
> On 14 August 2016 at 18:07, Tom Yan <tom.ty89@gmail.com> wrote:
> > On 14 August 2016 at 18:01, Pavel Machek <pavel@ucw.cz> wrote:
> >>
> >> Since SATA support was merged, certainly since v2.4, and from way
> >> before /dev/disk/by-id existed.
> >
> > I have no idea how "SATA before USB" had been done in the past (if it
> > was ever a thing in the kernel), but that has not been the case since
> > at least v3.0 AFAIR.

It is the case in v4.6. We had change hda->sda for SATA drives long
time ago, it was stable since that.

> > No, but you can always use root=PARTUUID=, that's built into the
> > kernel. (root=UUID= requires udev or so though).
> 
> Silly me. root=UUID= has nothing to do with udev, but `blkid` in
> util-linux. At least that's how it's done in Arch/mkinitcpio.

I'd rather not mess with initrd, and initrd was not required in the
past.

kernel-parameters.txt only mentions UUID= in connection with
resume. Is the documentation correct?
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Tom Yan]

On 14 August 2016 at 11:10, Pavel Machek <pavel@ucw.cz> wrote:
>
> It is the case in v4.6. We had change hda->sda for SATA drives long
> time ago, it was stable since that.

Not for me. It has been like forever (even if it wasn't the fact) that
the disk order is not consistent among boots. Only that would
logically make sense anyway. How do you think we can sanely make sure
that the SCSI disk driver starts enumerating "SCSI disks" from
usb-storage/uas only when all of those from libata probed? Wait for 1
second? 1 minute? or 1 hour? It's an irrational thing to do no matter
how long or how short it is.

>
> I'd rather not mess with initrd, and initrd was not required in the
> past.

Well, that's your choice, use PARTUUID then.

>
> kernel-parameters.txt only mentions UUID= in connection with
> resume. Is the documentation correct?

That's `PARTUUID=`. See what PARTUUID is referring to in the linked
comment below. (Note that when we say UUID instead of PARTUUID, we are
usually referring to the filesystem UUID.)

...

resume= [SWSUSP]
Specify the partition device for software suspend
Format:
{/dev/<dev> | PARTUUID=<uuid> | <int>:<int> | <hex>}

...

root= [KNL] Root filesystem
See name_to_dev_t comment in init/do_mounts.c.

...

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/init/do_mounts.c?h=v4.8-rc1#n182

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[One Thousand Gnomes]

On Sun, 14 Aug 2016 17:34:18 +0800
Tom Yan <tom.ty89@gmail.com> wrote:

> Since when it is expected that SATA disks will always be probed before
> USB disks? We can't guarantee that even if we make sure all ata
> drivers are loaded before usb-storage/uas. That's why we need
> consistent namings (e.g. /dev/disk/by-id/*).

It hasn't. It isn't even true on older kernels and can depend on all
sorts of things. I have boxes that will put the USB first, and a 10
second google search says I'm not alone

http://serverfault.com/questions/322946/ensure-usb-disk-is-never-sda-even-when-booting-from-it

So your "regression" goes back to at least 2011.

Alan

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Pavel Machek]

On Sun 2016-08-14 11:20:44, Pavel Machek wrote:
> Hi!
> 
> > It seems that in v4.8-rc0, /dev/sdX got reordered, and now USB devices
> > are probed before SATA drivers. That is pretty anti-social. It
> > broke my boot on my primary machine, and unfortunately due to BIOS
> > problems (keyboard does not work when connected through a hub) it is
> > less fun than it should be.
> 
> If you know which commit caused	the reordering, that would be helpful.
> 
> v4.1 seems to be ok: SATA disk is sda, as expected.
> 
> v4.4 seems to be ok: SATA disk is sda, as expected.

v4.6 seems to be ok.

v4.7 SATA disk is sde, behind USB card readers. Not helpful.

Ouch. So we have 'stable' kernel with this. Much more fun :-(.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Pavel Machek]

On Sun 2016-08-14 12:14:38, Pavel Machek wrote:
> On Sun 2016-08-14 11:20:44, Pavel Machek wrote:
> > Hi!
> > 
> > > It seems that in v4.8-rc0, /dev/sdX got reordered, and now USB devices
> > > are probed before SATA drivers. That is pretty anti-social. It
> > > broke my boot on my primary machine, and unfortunately due to BIOS
> > > problems (keyboard does not work when connected through a hub) it is
> > > less fun than it should be.
> > 
> > If you know which commit caused	the reordering, that would be helpful.
> > 
> > v4.1 seems to be ok: SATA disk is sda, as expected.
> > 
> > v4.4 seems to be ok: SATA disk is sda, as expected.
> 
> v4.6 seems to be ok.

v4.7-rc1 seems to be ok?! 9956d572
v4.7-rc3 seems still to be ok.
v4.7-rc4: still ok.
v4.7-rc5: ok.
v4.7-rc6: problem

Bisecting now, 8 steps to go. As it was introduced between -rc5 and
-rc6, bisect should not be too scary.

Best regards,
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Pavel Machek]

On Sun 2016-08-14 18:06:53, Pavel Machek wrote:
> On Sun 2016-08-14 12:14:38, Pavel Machek wrote:
> > On Sun 2016-08-14 11:20:44, Pavel Machek wrote:
> > > Hi!
> > > 
> > > > It seems that in v4.8-rc0, /dev/sdX got reordered, and now USB devices
> > > > are probed before SATA drivers. That is pretty anti-social. It
> > > > broke my boot on my primary machine, and unfortunately due to BIOS
> > > > problems (keyboard does not work when connected through a hub) it is
> > > > less fun than it should be.
> > > 
> > > If you know which commit caused	the reordering, that would be helpful.
> > > 
> > > v4.1 seems to be ok: SATA disk is sda, as expected.
> > > 
> > > v4.4 seems to be ok: SATA disk is sda, as expected.
> > 
> > v4.6 seems to be ok.
> 
> v4.7-rc1 seems to be ok?! 9956d572
> v4.7-rc3 seems still to be ok.
> v4.7-rc4: still ok.
> v4.7-rc5: ok.
> v4.7-rc6: problem
> 
> Bisecting now, 8 steps to go. As it was introduced between -rc5 and
> -rc6, bisect should not be too scary.

Huh. Scary.

I bisected it to

[ba34a65324259082dc6d2924cb82d562db1c6a6b] drm/i915: Avoid early
timeout during AUX transfers

pavel@amd:/data/l/linux$ git bisect log
# bad: [a99cde438de0c4c0cecc1d1af1a55a75b10bfdef] Linux 4.7-rc6
# good: [a8ef0490c3583c3a83fa52ebe60abb8fd7d3c2a4] Merge commit
'4c2e07c6a29e0129e975727b9f57eede813eea85' into bisect-v4.7
git bisect start 'a99cde438de0c4c0cecc1d1af1a55a75b10bfdef'
'a8ef0490c3583c3a83fa52ebe60abb8fd7d3c2a4'
# good: [4c2e07c6a29e0129e975727b9f57eede813eea85] Linux 4.7-rc5
git bisect good 4c2e07c6a29e0129e975727b9f57eede813eea85
# good: [751ad819b0bf443ad8963eb7bfbd533e6a463973] Merge tag
'mac80211-for-davem-2016-06-29-v2' of
git://git.kernel.org/pub/scm/linux/kernel/git/jberg/mac80211
git bisect good 751ad819b0bf443ad8963eb7bfbd533e6a463973
# good: [aa7a6c8e5252ba28f36a8f87b9acd6a726aa3ae5] Merge tag
'iommu-fixes-v4.7-rc5' of
git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu
git bisect good aa7a6c8e5252ba28f36a8f87b9acd6a726aa3ae5
# good: [dbdc3bb74faeec5fd92e28c15c945045d5aab426] Merge tag
'acpi-4.7-rc6' of
git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
git bisect good dbdc3bb74faeec5fd92e28c15c945045d5aab426
# good: [467ce7693f5111c11daeb75e02eba2ab9ee6f334] Merge tag
'spi-fix-v4.7-rc5' of
git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi
git bisect good 467ce7693f5111c11daeb75e02eba2ab9ee6f334
# bad: [88c087109b5fedaf9b61839d08543fdaf9d5ec39] Merge tag
'drm-intel-fixes-2016-06-30' of
git://anongit.freedesktop.org/drm-intel into drm-fixes
git bisect bad 88c087109b5fedaf9b61839d08543fdaf9d5ec39
# bad: [cab103274352721b77fc5923a631fc63350bef8e] drm/i915: Fix
missing unlock on error in i915_ppgtt_info()
git bisect bad cab103274352721b77fc5923a631fc63350bef8e
# good: [fa7c13e5b1e2076b0ec716ed584ab76f9e65b7a6] drm/i915/hsw: Avoid
early timeout during LCPLL disable/restore
git bisect good fa7c13e5b1e2076b0ec716ed584ab76f9e65b7a6
# bad: [42482b4546336ecd93acdd95e435bafd7a4c581c] drm/i915: Add more
Kabylake PCI IDs.
git bisect bad 42482b4546336ecd93acdd95e435bafd7a4c581c
pavel@amd:/data/l/linux$

I reverted ba34a65324259082dc6d2924cb82d562db1c6a6b on top of
4.8-rc1+, and it started booting.

Now the question is... what is going on there...?
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Regression - SATA disks behind USB ones on v4.8-rc1, breaking boot. [Re: Who reordered my disks (probably v4.8-rc1 problem)]

[Hans de Goede]

Hi,

On 14-08-16 19:55, Pavel Machek wrote:
> On Sun 2016-08-14 18:06:53, Pavel Machek wrote:
>> On Sun 2016-08-14 12:14:38, Pavel Machek wrote:
>>> On Sun 2016-08-14 11:20:44, Pavel Machek wrote:
>>>> Hi!
>>>>
>>>>> It seems that in v4.8-rc0, /dev/sdX got reordered, and now USB devices
>>>>> are probed before SATA drivers. That is pretty anti-social. It
>>>>> broke my boot on my primary machine, and unfortunately due to BIOS
>>>>> problems (keyboard does not work when connected through a hub) it is
>>>>> less fun than it should be.
>>>>
>>>> If you know which commit caused	the reordering, that would be helpful.
>>>>
>>>> v4.1 seems to be ok: SATA disk is sda, as expected.
>>>>
>>>> v4.4 seems to be ok: SATA disk is sda, as expected.
>>>
>>> v4.6 seems to be ok.
>>
>> v4.7-rc1 seems to be ok?! 9956d572
>> v4.7-rc3 seems still to be ok.
>> v4.7-rc4: still ok.
>> v4.7-rc5: ok.
>> v4.7-rc6: problem
>>
>> Bisecting now, 8 steps to go. As it was introduced between -rc5 and
>> -rc6, bisect should not be too scary.
>
> Huh. Scary.
>
> I bisected it to
>
> [ba34a65324259082dc6d2924cb82d562db1c6a6b] drm/i915: Avoid early
> timeout during AUX transfers
>
> pavel@amd:/data/l/linux$ git bisect log
> # bad: [a99cde438de0c4c0cecc1d1af1a55a75b10bfdef] Linux 4.7-rc6
> # good: [a8ef0490c3583c3a83fa52ebe60abb8fd7d3c2a4] Merge commit
> '4c2e07c6a29e0129e975727b9f57eede813eea85' into bisect-v4.7
> git bisect start 'a99cde438de0c4c0cecc1d1af1a55a75b10bfdef'
> 'a8ef0490c3583c3a83fa52ebe60abb8fd7d3c2a4'
> # good: [4c2e07c6a29e0129e975727b9f57eede813eea85] Linux 4.7-rc5
> git bisect good 4c2e07c6a29e0129e975727b9f57eede813eea85
> # good: [751ad819b0bf443ad8963eb7bfbd533e6a463973] Merge tag
> 'mac80211-for-davem-2016-06-29-v2' of
> git://git.kernel.org/pub/scm/linux/kernel/git/jberg/mac80211
> git bisect good 751ad819b0bf443ad8963eb7bfbd533e6a463973
> # good: [aa7a6c8e5252ba28f36a8f87b9acd6a726aa3ae5] Merge tag
> 'iommu-fixes-v4.7-rc5' of
> git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu
> git bisect good aa7a6c8e5252ba28f36a8f87b9acd6a726aa3ae5
> # good: [dbdc3bb74faeec5fd92e28c15c945045d5aab426] Merge tag
> 'acpi-4.7-rc6' of
> git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm
> git bisect good dbdc3bb74faeec5fd92e28c15c945045d5aab426
> # good: [467ce7693f5111c11daeb75e02eba2ab9ee6f334] Merge tag
> 'spi-fix-v4.7-rc5' of
> git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi
> git bisect good 467ce7693f5111c11daeb75e02eba2ab9ee6f334
> # bad: [88c087109b5fedaf9b61839d08543fdaf9d5ec39] Merge tag
> 'drm-intel-fixes-2016-06-30' of
> git://anongit.freedesktop.org/drm-intel into drm-fixes
> git bisect bad 88c087109b5fedaf9b61839d08543fdaf9d5ec39
> # bad: [cab103274352721b77fc5923a631fc63350bef8e] drm/i915: Fix
> missing unlock on error in i915_ppgtt_info()
> git bisect bad cab103274352721b77fc5923a631fc63350bef8e
> # good: [fa7c13e5b1e2076b0ec716ed584ab76f9e65b7a6] drm/i915/hsw: Avoid
> early timeout during LCPLL disable/restore
> git bisect good fa7c13e5b1e2076b0ec716ed584ab76f9e65b7a6
> # bad: [42482b4546336ecd93acdd95e435bafd7a4c581c] drm/i915: Add more
> Kabylake PCI IDs.
> git bisect bad 42482b4546336ecd93acdd95e435bafd7a4c581c
> pavel@amd:/data/l/linux$
>
> I reverted ba34a65324259082dc6d2924cb82d562db1c6a6b on top of
> 4.8-rc1+, and it started booting.
>
> Now the question is... what is going on there...?

Timing magic, as many people have pointed out, you've been depending
on the detection order being fixed, while in reality it has not
been fixed for a long long long time.

This seemingly unrelated commit apparently manages to change
some timing in some way that the detection order changes,
could be as simple as it yielding a core and the usb core
workqueue getting that core to do detection earlier then
before.

I used to work on anaconda, the Fedora / RH installer, as
well as mkinitrd and later dracut and drive-order has
basically never been stable once we no longer had hda /
hdb for ide disks. So you really just need to fix the way
you're booting your system rather then blaming the kernel.

Regards,

Hans

Re: Who reordered my disks (probably v4.8-rc1 problem)

[Austin S. Hemmelgarn]

On 2016-08-13 15:30, Pavel Machek wrote:
> Hi!
>
> It seems that in v4.8-rc0, /dev/sdX got reordered, and now USB devices
> are probed before SATA drivers. That is pretty anti-social. It
> broke my boot on my primary machine, and unfortunately due to BIOS
> problems (keyboard does not work when connected through a hub) it is
> less fun than it should be.
>
The simple solution (which I'm surprised nobody else has suggested yet) 
is to build everything but the driver for your root device as a module. 
This will prevent anything else from getting enumerated first, and works 
regardless of whether or not your using an initrd and independently of 
what root= format you use.