guide on bisecting (was Re: [REGRESSION] kexec does firmware reboot in kernel v6.7.6)

guide on bisecting (was Re: [REGRESSION] kexec does firmware reboot in kernel v6.7.6)

[Thorsten Leemhuis]

[dropping Steve, Dave and the stable list from CC while adding LKML and
linux-docs]

On 02.03.24 17:10, Pavin Joseph wrote:
> 
> On 3/2/24 20:47, Linux regression tracking (Thorsten Leemhuis) wrote>
> Thx for testing and glad to hear. Still: if you have any feedback how to
>> make that guide even better, please let me know!
> Yes, I have some improvements in mind.

Many thx for the suggestion!

> 1. The git clone/fetch instructions in the TLDR is easy to follow, but
> there are conflicting information later on in the main section and
> reference that taken together does not work.
>
> I think it would be better to not perform shallow clones

Hmmm. Either something sent you sideways or I'm missing something, as
the main section does not use a shallow clone. It did in earlier
versions. Wondering if something was left behind I missed.

That being said: I think I might know what sent you sideways: the main
section lacked a "git remote add -t master stable
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git". :-((

> or such advanced topics could be relegated
> to its own reference section.

They are. But I'm wondering if the "this is a alternative way" aspect
was obvious enough. That's why I slightly changed that part of the
reference section in my local copy. It now looks like this:

"""

Download the sources using git
------------------------------

  *Retrieve the Linux mainline sources.*
  [:ref:`...<sources_bissbs>`]

The step-by-step guide outlines how to retrieve the Linux sources using
a full Git clone of Linus' mainline repository. There is nothing more to
say about that -- but there are two alternatives ways to retrieve the
sources that might be more suitable for your situation:

 * If you have an unreliable internet connection, you might want to use
   a 'Git bundle' to retrieve the sources.

 * If downloading the complete repository would take too long or
   requires too much storage space, consider using a shallow clone.

Downloading Linux mainline using a bundle
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Switch to you home directory and  [...]
"""

> Here's what I ended up using:
> [...]

That's round about the same as the main section, but as I said, that
"git remote add -t master stable
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git" was
missing.

> 2. The "installkernel" command is called "kernel-install" in OpenSuse,

Yeah, it looks like that, but that's not really the case. :-) In short:
on Fedora "installkernel" calls into kernel-install -- and
"installkernel" has a long history, so doing what Fedora does is likely
a wise thing for distros. And openSUSE had a "installkernel" as well,
which was part of the dracut package. Not sure if that is still the case
for current Leap and Tumbleweed. Could you check?

> and it doesn't really perform all the steps to install kernel. It calls
> dracut to create initramfs though, but that's hardly much help.

Could you please elaborate a bit on that "hardly much help", as I'm not
really sure what you exactly mean here. Are you and/or openSUSE normally
not using dracut?

> I ended up doing: [...]

Good to know, but I don't really want to add that to the guide for all
the major distros, as then things likely get messy quickly. This IMHO
better should be dealt with on the distro level, as some distros show
that calling into installkernel can make everything work.

> 3. The dependencies for kernel building in OpenSuse and other major
> distros are incomplete,

So what was missing?

> most of them have some form of package
> collection that can be provided as an alternative.

Which sometimes also install stuff that is not needed. I'm willing to
switch, but you must give me good reasons.

And in my experience doing something what is done for Fedora is likely
the best, as the name of the executable hardly change, while
"collections" come and go and sometimes change names.

But as I said, I'm not totally against switching.

> [...]
> 4. The command to build RPM package (make binrpm-pkg) fails as the
> modules are installed into "/home/<user>/linux/.../lib" while depmod
> checks for modules in "/home/<user>/linux/.../usr/lib".

That sounds like a bug that should be reported and fixed, not something
that docs should catch and work around. Could you report that?

> I think that's it, turned out not to be a gist after all. 🙂

Again, thx for the feedback, this is really helpful.

> Thank you very much for writing the updated guide, it was very helpful
> without which I don't think it would have been possible for someone like
> me to find/report this bug.

Glad to hear, as that was the goal.

Ciao, Thorsten

Re: guide on bisecting (was Re: [REGRESSION] kexec does firmware reboot in kernel v6.7.6)

[Pavin Joseph]

On 3/3/24 14:06, Thorsten Leemhuis wrote:

> That being said: I think I might know what sent you sideways: the main
> section lacked a "git remote add -t master stable
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git". :-((

Now that I read through it with a fresh pair of eyes, this is exactly 
the problem!

>> 2. The "installkernel" command is called "kernel-install" in OpenSuse,
> 
> Yeah, it looks like that, but that's not really the case. :-) In short:
> on Fedora "installkernel" calls into kernel-install -- and
> "installkernel" has a long history, so doing what Fedora does is likely
> a wise thing for distros. And openSUSE had a "installkernel" as well,
> which was part of the dracut package. Not sure if that is still the case
> for current Leap and Tumbleweed. Could you check?

It's not available even as a symlink in OpenSuse TW / Slowroll I'm afraid.

suse-pc:~ # whereis installkernel
installkernel:
suse-pc:~ # whereis kernel-install
kernel-install: /usr/bin/kernel-install 
/usr/share/man/man8/kernel-install.8.gz
suse-pc:~ # man kernel-install | grep -i installkernel
        installkernel [OPTIONS...] VERSION VMLINUZ [MAP] [INSTALLATION-DIR]
        When invoked as installkernel, this program accepts arguments as 
specified by the kernel build system's make install command. The VERSION 
and VMLINUZ parameters

>> and it doesn't really perform all the steps to install kernel. It calls
>> dracut to create initramfs though, but that's hardly much help.
> 
> Could you please elaborate a bit on that "hardly much help", as I'm not
> really sure what you exactly mean here. Are you and/or openSUSE normally
> not using dracut?

I meant that calling kernel-install in OpenSuse only seems to then call 
dracut to build an initramfs for the kernel. I can call dracut myself 
without adding an unnecessary middleman (kernel-install) in the process 
and less verbosely too: dracut --kver $(make -s kernelrelease)

Perhaps you could add generic details such as I provided in the 
reference section for distros where installkernel doesn't exist or don't 
perform all the steps required.

>> 3. The dependencies for kernel building in OpenSuse and other major
>> distros are incomplete,
> 
> So what was missing?

Sorry, I don't remember. The compile/build threw some error and I looked 
up how to install kernel building dependencies in OpenSuse only to find 
out there was a pattern for it already.

Perhaps you could list the basic dependencies in the main section and 
provide the collection/patterns in the reference section.

>> 4. The command to build RPM package (make binrpm-pkg) fails as the
>> modules are installed into "/home/<user>/linux/.../lib" while depmod
>> checks for modules in "/home/<user>/linux/.../usr/lib".
> 
> That sounds like a bug that should be reported and fixed, not something
> that docs should catch and work around. Could you report that?

Please, could you tell me where to report this bug? Kernel bugzilla? 
Which category/component? Thanks for all your help 😉

Kind regards,
Pavin Joseph.

Re: guide on bisecting (was Re: [REGRESSION] kexec does firmware reboot in kernel v6.7.6)

[Thorsten Leemhuis]

On 03.03.24 11:17, Pavin Joseph wrote:
> On 3/3/24 14:06, Thorsten Leemhuis wrote:
> 
>> That being said: I think I might know what sent you sideways: the main
>> section lacked a "git remote add -t master stable
>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git". :-((
> Now that I read through it with a fresh pair of eyes, this is exactly
> the problem!

Thx for confirming, fixed this up locally and will submit a patch to fix
this for inclusion (the text is now in -next, yeah!).

>>> 2. The "installkernel" command is called "kernel-install" in OpenSuse,
>>
>> Yeah, it looks like that, but that's not really the case. :-) In short:
>> on Fedora "installkernel" calls into kernel-install -- and
>> "installkernel" has a long history, so doing what Fedora does is likely
>> a wise thing for distros. And openSUSE had a "installkernel" as well,
>> which was part of the dracut package. Not sure if that is still the case
>> for current Leap and Tumbleweed. Could you check?
> 
> It's not available even as a symlink in OpenSuse TW / Slowroll I'm afraid.

I'm not really familiar with openSUSE, but it set up a TW container and
found that a package kernel-install-tools provides installkernel script.
Not sure how good it works though. Could you maybe test that?

>>> and it doesn't really perform all the steps to install kernel. It calls
>>> dracut to create initramfs though, but that's hardly much help.
>> Could you please elaborate a bit on that "hardly much help", as I'm not
>> really sure what you exactly mean here. Are you and/or openSUSE normally
>> not using dracut?
> I meant that calling kernel-install in OpenSuse only seems to then call
> dracut to build an initramfs for the kernel. I can call dracut myself
> without adding an unnecessary middleman (kernel-install) in the process
> and less verbosely too: dracut --kver $(make -s kernelrelease)

kernel-install is normally meant to copy the image over to /boot/ as
well afaik; maybe it did not do that in your case because you already
had placed it there manually?

> Perhaps you could add generic details such as I provided in the
> reference section for distros where installkernel doesn't exist or don't
> perform all the steps required.

Go look, they are there since the start. They differ from your
instructions though, as you put the image in /usr/lib/modules/ which
only works with distros that have kernel-install. Hmmm. Maybe it at some
point will likely be better to just use it for the manual install
instructions; but it feels a bit like it's to early for that. Not sure.

>>> 3. The dependencies for kernel building in OpenSuse and other major
>>> distros are incomplete,
>> So what was missing?
> Sorry, I don't remember.

I checked using a container and fixed this this.

> The compile/build threw some error and I looked
> up how to install kernel building dependencies in OpenSuse only to find
> out there was a pattern for it already.

I just checked, the command you provided earlier would download ~250
MByte of packages that would consume 1,5 GByte disk space, all of which
are unneeded afaics. I'm taken a bit back and forth there, but I think
sticking to just listing the packages that are actually needed might be
the better approach.

> Perhaps you could list the basic dependencies in the main section and
> provide the collection/patterns in the reference section.

I want to keep the main section distro-agnostic as much as possible;
listing package names for distros would also make it longer. I think
it's better to do that in the reference section.

>>> 4. The command to build RPM package (make binrpm-pkg) fails as the
>>> modules are installed into "/home/<user>/linux/.../lib" while depmod
>>> checks for modules in "/home/<user>/linux/.../usr/lib".
>> That sounds like a bug that should be reported and fixed, not something
>> that docs should catch and work around. Could you report that?
> Please, could you tell me where to report this bug? Kernel bugzilla?
> Which category/component?

Just sent a mail to Masahiro Yamada <masahiroy@kernel.org> with
linux-kbuild@vger.kernel.org and linux-kernel@vger.kernel.org in CC.

> Thanks for all your help 😉

Thx for your feedback, it helped making things better!

Ciao, Thorsten

Re: guide on bisecting (was Re: [REGRESSION] kexec does firmware reboot in kernel v6.7.6)

[Pavin Joseph]

On 3/4/24 11:46, Thorsten Leemhuis wrote:
> I'm not really familiar with openSUSE, but it set up a TW container and
> found that a package kernel-install-tools provides installkernel script.
> Not sure how good it works though. Could you maybe test that?

Yep, that package got me "installkernel" and it does all the 
installation as expected. I only needed to change the symlink of 
"/boot/<initrd|vmlinuz>" to point to the new ones created by it.

> kernel-install is normally meant to copy the image over to /boot/ as
> well afaik; maybe it did not do that in your case because you already
> had placed it there manually?

kernel-install doesn't do that in my testing, it also doesn't remove 
anything AFAICT. 🧐

Kind regards,
Pavin Joseph.

Re: guide on bisecting (was Re: [REGRESSION] kexec does firmware reboot in kernel v6.7.6)

[Thorsten Leemhuis]

On 04.03.24 19:03, Pavin Joseph wrote:
> On 3/4/24 11:46, Thorsten Leemhuis wrote:
>> I'm not really familiar with openSUSE, but it set up a TW container and
>> found that a package kernel-install-tools provides installkernel script.
>> Not sure how good it works though. Could you maybe test that?
> 
> Yep, that package got me "installkernel" and it does all the
> installation as expected. I only needed to change the symlink of
> "/boot/<initrd|vmlinuz>" to point to the new ones created by it.

Well, if openSUSE uses those links (other distros don't have them) then
you might want to report a bug to openSUSE.

>> kernel-install is normally meant to copy the image over to /boot/ as
>> well afaik; maybe it did not do that in your case because you already
>> had placed it there manually?
> kernel-install doesn't do that in my testing, it also doesn't remove
> anything AFAICT. 🧐

Ahh what a mess. I will slightly change the text to indicate that might
happen.

Ciao, Thorsten

Re: guide on bisecting (was Re: [REGRESSION] kexec does firmware reboot in kernel v6.7.6)

[Takashi Iwai]

On Sun, 03 Mar 2024 11:17:44 +0100,
Pavin Joseph wrote:
> 
> On 3/3/24 14:06, Thorsten Leemhuis wrote:
> 
> >> 2. The "installkernel" command is called "kernel-install" in OpenSuse,
> > 
> > Yeah, it looks like that, but that's not really the case. :-) In short:
> > on Fedora "installkernel" calls into kernel-install -- and
> > "installkernel" has a long history, so doing what Fedora does is likely
> > a wise thing for distros. And openSUSE had a "installkernel" as well,
> > which was part of the dracut package. Not sure if that is still the case
> > for current Leap and Tumbleweed. Could you check?
> 
> It's not available even as a symlink in OpenSuse TW / Slowroll I'm afraid.
> 
> suse-pc:~ # whereis installkernel
> installkernel:
> suse-pc:~ # whereis kernel-install
> kernel-install: /usr/bin/kernel-install
> /usr/share/man/man8/kernel-install.8.gz
> suse-pc:~ # man kernel-install | grep -i installkernel
>        installkernel [OPTIONS...] VERSION VMLINUZ [MAP] [INSTALLATION-DIR]
>        When invoked as installkernel, this program accepts arguments
> as specified by the kernel build system's make install command. The
> VERSION and VMLINUZ parameters

FYI, /usr/sbin/installkernel is included in kernel-install-tools
package.


Takashi