[rfc] Merge kexec-tools into the kernel tree

[rfc] Merge kexec-tools into the kernel tree

[Simon Horman]

Hi,

After all the excitement of relocating kexec-tools from
one location on kernel.org to another last week it was
suggested to me by Michael Neuling that the merging
kexec-tools into the kernel tree would be a good idea.

Given that there have been a bunch of issues with kexec
on power that this would resolve. and there is precedence
for tools in the kernel tree, this sounds entirely reasonable to me.
So with my kexec-tools maintainer hat on, I would like to start
a conversation about this.

In order to move the conversation along I have prepared a tree -
a recent clone of Linus's tree + kexec tools.

http://git.kernel.org/?p=linux/kernel/git/horms/kexec-tools-in-kernel-tree.git

I think that a good next step would be to get this tree into linux-next.
And then if all goes well, as Linus to pull the change some time in the future.





For reference the steps taken to make the tree above were roughly as follows.
Thanks to Michael for assistance with this.

git clone
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
git clone git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git

cd linux-2.6

git remote add -f kexec ../kexec-tools
git merge -s ours --no-commit kexec/master
git read-tree --prefix=tools/kexec/ -u kexec/master
git commit -m "Merge kexec-tools into subdirectory tools/kexec"

Re: [rfc] Merge kexec-tools into the kernel tree

[Américo Wang]

On Wed, Aug 04, 2010 at 04:06:48PM +0900, Simon Horman wrote:
>Hi,
>
>After all the excitement of relocating kexec-tools from
>one location on kernel.org to another last week it was
>suggested to me by Michael Neuling that the merging
>kexec-tools into the kernel tree would be a good idea.
>
>Given that there have been a bunch of issues with kexec
>on power that this would resolve. and there is precedence
>for tools in the kernel tree, this sounds entirely reasonable to me.
>So with my kexec-tools maintainer hat on, I would like to start
>a conversation about this.
>

But why? Just to draw more attentions from kernel community?

Re: [rfc] Merge kexec-tools into the kernel tree

[Eric W. Biederman]

Simon Horman <horms@verge.net.au> writes:

> Hi,
>
> After all the excitement of relocating kexec-tools from
> one location on kernel.org to another last week it was
> suggested to me by Michael Neuling that the merging
> kexec-tools into the kernel tree would be a good idea.
>
> Given that there have been a bunch of issues with kexec
> on power that this would resolve. and there is precedence
> for tools in the kernel tree, this sounds entirely reasonable to me.
> So with my kexec-tools maintainer hat on, I would like to start
> a conversation about this.

What are the issues with kexec on power?  Did someone fail to maintain
ABI compatibility?

The interface isn't even supposed to be linux specific, so I can't
imagine what would motivate moving this into the kernel tree.

I'm afraid that someone has a good answer for why their lives would be
simpler if /sbin/kexec was in the kernel tree and I will be absolutely
horrified and about someones stupidity when I hear that answer.

Eric

Re: [rfc] Merge kexec-tools into the kernel tree

[Andi Kleen]

ebiederm@xmission.com (Eric W. Biederman) writes:
>
> I'm afraid that someone has a good answer for why their lives would be
> simpler if /sbin/kexec was in the kernel tree and I will be absolutely
> horrified and about someones stupidity when I hear that answer.

Same here. In fact I guess it would be counter productive: it would
discourage a stable API between the kernel and the user tool.
We're seeing this problem with other user tools which are in 
the kernel tree.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: [rfc] Merge kexec-tools into the kernel tree

[Michael Neuling]

> > After all the excitement of relocating kexec-tools from
> > one location on kernel.org to another last week it was
> > suggested to me by Michael Neuling that the merging
> > kexec-tools into the kernel tree would be a good idea.
> >
> > Given that there have been a bunch of issues with kexec
> > on power that this would resolve. and there is precedence
> > for tools in the kernel tree, this sounds entirely reasonable to me.
> > So with my kexec-tools maintainer hat on, I would like to start
> > a conversation about this.
> 
> What are the issues with kexec on power?  Did someone fail to maintain
> ABI compatibility?
> 
> The interface isn't even supposed to be linux specific, so I can't
> imagine what would motivate moving this into the kernel tree.
> 
> I'm afraid that someone has a good answer for why their lives would be
> simpler if /sbin/kexec was in the kernel tree and I will be absolutely
> horrified and about someones stupidity when I hear that answer.

I may have misrepresented how bad it is for power to Horms.  None of the
issues would be solved by a merge, but it would make life easier IMHO.

In power we've added features to kexec which have required changes to
both the kernel and kexec-tools.  These have been backwards compatible,
so not to break to the ABI.  The problem here is getting users and
distros to take the correct versions of both sources if they want this
new feature.

Similarly with bugs.  We recently went through a round of bug fixes for
new larger power7 machines.  We found bugs in both kexec-tools and the
kernel.  That meant we had to ensure users and distros were getting
correctly updated versions of both tools.

Neither of these problems are show stoppers or power specific but I
think it would make life easier in these scenarios if the sources were
merged.  We could just tell users and distros to grab (say) 2.6.35
sources and we'd know they'd be right for both userspace and the kernel.

Also, I think kexec-tools would benefit from the same release process as
the kernel, with a merge window followed by bug fixes.  Of course,
kexec-tools doesn't need to be in the kernel for this, but it might be
easier for Horms to enforce if it was.  kexec-tools only gets a trickle
patches.

I'd also hope that kexec-tools would get some addition community
exposure and TLC if they were in the kernel sources.

My question is, why not?  What qualifies a tool to be added to tools/?
I think kexec-tools are tied to the kernel at least as much as perf is.
Certainly the ABI for the image we are booting into is not Linux
specific, but should that disqualify it from being in tools/?

Mikey

Re: [rfc] Merge kexec-tools into the kernel tree

[Eric W. Biederman]

Michael Neuling <mikey@neuling.org> writes:

>> > After all the excitement of relocating kexec-tools from
>> > one location on kernel.org to another last week it was
>> > suggested to me by Michael Neuling that the merging
>> > kexec-tools into the kernel tree would be a good idea.
>> >
>> > Given that there have been a bunch of issues with kexec
>> > on power that this would resolve. and there is precedence
>> > for tools in the kernel tree, this sounds entirely reasonable to me.
>> > So with my kexec-tools maintainer hat on, I would like to start
>> > a conversation about this.
>> 
>> What are the issues with kexec on power?  Did someone fail to maintain
>> ABI compatibility?
>> 
>> The interface isn't even supposed to be linux specific, so I can't
>> imagine what would motivate moving this into the kernel tree.
>> 
>> I'm afraid that someone has a good answer for why their lives would be
>> simpler if /sbin/kexec was in the kernel tree and I will be absolutely
>> horrified and about someones stupidity when I hear that answer.
>
> I may have misrepresented how bad it is for power to Horms.  None of the
> issues would be solved by a merge, but it would make life easier IMHO.
>
> In power we've added features to kexec which have required changes to
> both the kernel and kexec-tools.  These have been backwards compatible,
> so not to break to the ABI.  The problem here is getting users and
> distros to take the correct versions of both sources if they want this
> new feature.

I'm still scratching my head.  What new features were added recently
that required this work?  The device tree or something else?

What you are describing seems to be the case for adding any new kernel
feature.

> Similarly with bugs.  We recently went through a round of bug fixes for
> new larger power7 machines.  We found bugs in both kexec-tools and the
> kernel.  That meant we had to ensure users and distros were getting
> correctly updated versions of both tools.
>
> Neither of these problems are show stoppers or power specific but I
> think it would make life easier in these scenarios if the sources were
> merged.  We could just tell users and distros to grab (say) 2.6.35
> sources and we'd know they'd be right for both userspace and the kernel.

You are proposing optimizing for change when change should and generally
is infrequent?

> Also, I think kexec-tools would benefit from the same release process as
> the kernel, with a merge window followed by bug fixes.  Of course,
> kexec-tools doesn't need to be in the kernel for this, but it might be
> easier for Horms to enforce if it was.  kexec-tools only gets a trickle
> patches.

You would like to see a higher barrier to entry for your patches to make
it into /sbin/kexec?  Someone else to help you test so that you get fewer
buggy patches into releases?

> I'd also hope that kexec-tools would get some addition community
> exposure and TLC if they were in the kernel sources.
>
> My question is, why not?  What qualifies a tool to be added to tools/?
> I think kexec-tools are tied to the kernel at least as much as perf is.
> Certainly the ABI for the image we are booting into is not Linux
> specific, but should that disqualify it from being in tools/?

The grand and glorious vision for /sbin/kexec is that it can boot any
interesting OS kernel.  From RHEL, SLES, Fedora, Unbuntu definitely,
but also the BSDs, Mac OS X, and Windows.  I don't see how moving into
the kernel tree making that vision any easier.  Do I need a different
version of kexec to boot an Ubunutu kernel versus a RHEL kernel,
versus a kernel.org kernel?

On the flip side of it in general kexec should not be assumed to be
the only boot loader using various kernel interfaces.  So when you add
a new feature sure add the feature to /sbin/kexec but don't forget
someone eventually will want that feature in another boot loader as
well.

I can imagine arguments for putting the sources for /sbin/kexec into
the kernel tree but I don't see them being made here.

If we talk about analyzing and filtering crash dumps, I can totally
see an argument for putting something under tools/ if the authors of
mkdumpfile and crash are interested.  Those tools fundamentally really
do follow kernel internals.

I may be dense but I don't how everything will be better if sprinkled
with penguin pee.

Eric

Re: [rfc] Merge kexec-tools into the kernel tree

[Michael Neuling]

> >> > After all the excitement of relocating kexec-tools from
> >> > one location on kernel.org to another last week it was
> >> > suggested to me by Michael Neuling that the merging
> >> > kexec-tools into the kernel tree would be a good idea.
> >> >
> >> > Given that there have been a bunch of issues with kexec
> >> > on power that this would resolve. and there is precedence
> >> > for tools in the kernel tree, this sounds entirely reasonable to me.
> >> > So with my kexec-tools maintainer hat on, I would like to start
> >> > a conversation about this.
> >> 
> >> What are the issues with kexec on power?  Did someone fail to maintain
> >> ABI compatibility?
> >> 
> >> The interface isn't even supposed to be linux specific, so I can't
> >> imagine what would motivate moving this into the kernel tree.
> >> 
> >> I'm afraid that someone has a good answer for why their lives would be
> >> simpler if /sbin/kexec was in the kernel tree and I will be absolutely
> >> horrified and about someones stupidity when I hear that answer.
> >
> > I may have misrepresented how bad it is for power to Horms.  None of the
> > issues would be solved by a merge, but it would make life easier IMHO.
> >
> > In power we've added features to kexec which have required changes to
> > both the kernel and kexec-tools.  These have been backwards compatible,
> > so not to break to the ABI.  The problem here is getting users and
> > distros to take the correct versions of both sources if they want this
> > new feature.
> 
> I'm still scratching my head.  What new features were added recently
> that required this work?  The device tree or something else?

The couple that come to mind are:
 1) dynamic reconfigurable memory (added via the device tree):
    kexec-tools: cd8497a9a9e487684679b6747f7ba3f0a557328b
    kernel cf00085d8045cddd80a8aabad97de96fa8131793
 2) --reuseinitrd/retain_initrd option:
    kexec-tools: 8ec6347996ce83c369edeee4bed0498dedda6b41
    kernel: 0a7b35cb18c52d651f6ed9cd59edc979200ab880
Not recent though.

> What you are describing seems to be the case for adding any new kernel
> feature.

Yep, this is not unique to kexec.

> 
> > Similarly with bugs.  We recently went through a round of bug fixes for
> > new larger power7 machines.  We found bugs in both kexec-tools and the
> > kernel.  That meant we had to ensure users and distros were getting
> > correctly updated versions of both tools.
> >
> > Neither of these problems are show stoppers or power specific but I
> > think it would make life easier in these scenarios if the sources were
> > merged.  We could just tell users and distros to grab (say) 2.6.35
> > sources and we'd know they'd be right for both userspace and the kernel.
> 
> You are proposing optimizing for change when change should and generally
> is infrequent?

That's _part_ of the argument, yes.

> > Also, I think kexec-tools would benefit from the same release process as
> > the kernel, with a merge window followed by bug fixes.  Of course,
> > kexec-tools doesn't need to be in the kernel for this, but it might be
> > easier for Horms to enforce if it was.  kexec-tools only gets a trickle
> > patches.
> 
> You would like to see a higher barrier to entry for your patches to make
> it into /sbin/kexec?  Someone else to help you test so that you get fewer
> buggy patches into releases?

Yes and yes.

A kexec-tools merge could also benefit from gregkh's stable releases.

> > I'd also hope that kexec-tools would get some addition community
> > exposure and TLC if they were in the kernel sources.
> >
> > My question is, why not?  What qualifies a tool to be added to tools/?
> > I think kexec-tools are tied to the kernel at least as much as perf is.
> > Certainly the ABI for the image we are booting into is not Linux
> > specific, but should that disqualify it from being in tools/?
> 
> The grand and glorious vision for /sbin/kexec is that it can boot any
> interesting OS kernel.  From RHEL, SLES, Fedora, Unbuntu definitely,
> but also the BSDs, Mac OS X, and Windows.  I don't see how moving into
> the kernel tree making that vision any easier.  Do I need a different
> version of kexec to boot an Ubunutu kernel versus a RHEL kernel,
> versus a kernel.org kernel?

No I don't see a _need_ for this.

> On the flip side of it in general kexec should not be assumed to be
> the only boot loader using various kernel interfaces.  So when you add
> a new feature sure add the feature to /sbin/kexec but don't forget
> someone eventually will want that feature in another boot loader as
> well.

Yep.

> I can imagine arguments for putting the sources for /sbin/kexec into
> the kernel tree but I don't see them being made here.

Do you have any now?  :-)

> If we talk about analyzing and filtering crash dumps, I can totally
> see an argument for putting something under tools/ if the authors of
> mkdumpfile and crash are interested.  Those tools fundamentally really
> do follow kernel internals.

I agree that the argument is stronger for tools/ inclusion if internal
APIs need to be followed.  Of course perf doesn't need internals APIs
and it's in tools/.

> I may be dense but I don't how everything will be better if sprinkled
> with penguin pee.

It's more likely that I'm too dense to argue the case :-)

Mikey

Re: [rfc] Merge kexec-tools into the kernel tree

[Américo Wang]

On Thu, Aug 05, 2010 at 11:19:46AM +1000, Michael Neuling wrote:
>> If we talk about analyzing and filtering crash dumps, I can totally
>> see an argument for putting something under tools/ if the authors of
>> mkdumpfile and crash are interested.  Those tools fundamentally really
>> do follow kernel internals.
>
>I agree that the argument is stronger for tools/ inclusion if internal
>APIs need to be followed.  Of course perf doesn't need internals APIs
>and it's in tools/.
>

Well, I don't remember clearly why perf was proposed to be included into
kernel tree by Ingo, I think it was internal API/ABI issue as well.

Re: [rfc] Merge kexec-tools into the kernel tree

[Américo Wang]

On Thu, Aug 05, 2010 at 09:11:26AM +1000, Michael Neuling wrote:
>> > After all the excitement of relocating kexec-tools from
>> > one location on kernel.org to another last week it was
>> > suggested to me by Michael Neuling that the merging
>> > kexec-tools into the kernel tree would be a good idea.
>> >
>> > Given that there have been a bunch of issues with kexec
>> > on power that this would resolve. and there is precedence
>> > for tools in the kernel tree, this sounds entirely reasonable to me.
>> > So with my kexec-tools maintainer hat on, I would like to start
>> > a conversation about this.
>> 
>> What are the issues with kexec on power?  Did someone fail to maintain
>> ABI compatibility?
>> 
>> The interface isn't even supposed to be linux specific, so I can't
>> imagine what would motivate moving this into the kernel tree.
>> 
>> I'm afraid that someone has a good answer for why their lives would be
>> simpler if /sbin/kexec was in the kernel tree and I will be absolutely
>> horrified and about someones stupidity when I hear that answer.
>
>I may have misrepresented how bad it is for power to Horms.  None of the
>issues would be solved by a merge, but it would make life easier IMHO.
>
>In power we've added features to kexec which have required changes to
>both the kernel and kexec-tools.  These have been backwards compatible,
>so not to break to the ABI.  The problem here is getting users and
>distros to take the correct versions of both sources if they want this
>new feature.
>
>Similarly with bugs.  We recently went through a round of bug fixes for
>new larger power7 machines.  We found bugs in both kexec-tools and the
>kernel.  That meant we had to ensure users and distros were getting
>correctly updated versions of both tools.


I am afraid kexec-tools is not alone, there are plenty of user-space
applications which rely on some new kernel feature after a specific version.
If this could be a reason for inclusion into kernel tree, then much more
applications shall be included too. And if this happened, Linux would be
more like *BSD.

Also, this is a big reason why linux distributions exist. They are
responsible for collecting the right version of some application and the
right version of kernel that makes them work together.

So, I think  _we_ should get used to this unless Linus decides to change
this model.

>Neither of these problems are show stoppers or power specific but I
>think it would make life easier in these scenarios if the sources were
>merged.  We could just tell users and distros to grab (say) 2.6.35
>sources and we'd know they'd be right for both userspace and the kernel.
>

I think the solution is documention, or release notes, not including
it into kernel tree.

Thanks.

Re: [rfc] Merge kexec-tools into the kernel tree

[Bernhard Walle]

* Simon Horman <horms@verge.net.au> [2010-08-04 09:06]:
> 
> Given that there have been a bunch of issues with kexec
> on power that this would resolve. and there is precedence
> for tools in the kernel tree, this sounds entirely reasonable to me.
> So with my kexec-tools maintainer hat on, I would like to start
> a conversation about this.

Well, honestly I don't understand the reason behind it.

If it's about dependencies between the kexec binary and the kernel
version -- there are also dependencies between makedumpfile and crash to
the kernel version. Should we move makedumpfile and crash in the kernel
tree, too (which means that we have a GDB copy in the kernel tree)?


Regards,
Bernhard

Re: [rfc] Merge kexec-tools into the kernel tree

[Neil Horman]

On Wed, Aug 04, 2010 at 04:06:48PM +0900, Simon Horman wrote:
> Hi,
> 
> After all the excitement of relocating kexec-tools from
> one location on kernel.org to another last week it was
> suggested to me by Michael Neuling that the merging
> kexec-tools into the kernel tree would be a good idea.
> 
> Given that there have been a bunch of issues with kexec
> on power that this would resolve. and there is precedence
> for tools in the kernel tree, this sounds entirely reasonable to me.
> So with my kexec-tools maintainer hat on, I would like to start
> a conversation about this.
> 
> In order to move the conversation along I have prepared a tree -
> a recent clone of Linus's tree + kexec tools.
> 
> http://git.kernel.org/?p=linux/kernel/git/horms/kexec-tools-in-kernel-tree.git
> 
> I think that a good next step would be to get this tree into linux-next.
> And then if all goes well, as Linus to pull the change some time in the future.
> 
> 
> 
> 
> 
> For reference the steps taken to make the tree above were roughly as follows.
> Thanks to Michael for assistance with this.
> 
> git clone
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
> git clone git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git
> 
> cd linux-2.6
> 
> git remote add -f kexec ../kexec-tools
> git merge -s ours --no-commit kexec/master
> git read-tree --prefix=tools/kexec/ -u kexec/master
> git commit -m "Merge kexec-tools into subdirectory tools/kexec"
> 
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec


It seems to me that this change isn't overly usefull in solving the problems
that kexec-tools seems to chronically encounter.  The biggest kexec/kernel
problem that I seem to encounter during maintence of the package in RHEL isn't
ABI compatibility between any kexec/kernel version pair (which can often be
mitigated/controlled by merging the source trees), but rather by lack of testing
of changes.  All to often kernel code authors assume that if something they
write or modify works in the kernel during a normal boot, it can be assumed to
work just fine in the kdump kernel.  That _should_ be the case, bur rarely is
it.  And unfortunately, that won't be mitigated by tying a kexec and kernel
version together.

What I think we need is a shorter path from upstream commit to regression/bug
detection.  Something that would allow someone to validate that kexec works
properly after they have made a commit to hardware that they have in hand.
Perhaps something in the scripts directory that allows a developer to
immediately execute a kexec -e reboot and a kexec -l; sysrq-c crash reboot to
validate that their changes are working properly.  Perhaps if we added a step to
the submission checklist including the running of said script, we'd see less
hardware specific regressions.

My $0.02
Neil

Re: [rfc] Merge kexec-tools into the kernel tree

[Simon Horman]

On Wed, Aug 04, 2010 at 04:06:48PM +0900, Simon Horman wrote:
> Hi,
> 
> After all the excitement of relocating kexec-tools from
> one location on kernel.org to another last week it was
> suggested to me by Michael Neuling that the merging
> kexec-tools into the kernel tree would be a good idea.
> 
> Given that there have been a bunch of issues with kexec
> on power that this would resolve. and there is precedence
> for tools in the kernel tree, this sounds entirely reasonable to me.
> So with my kexec-tools maintainer hat on, I would like to start
> a conversation about this.

Thanks to everyone for their responses. I think we can safely conclude
that the body of opinion is that its not appropriate for kexec-tools
to be included in the kernel tree. Primarily because it isn't tied
closely to the kernel ABI. As such I withdraw this proposal.

Thanks in particular to those who offered ideas on how to resolve
some of the problems that kexec-tools faces. Lets make those ideas happen.