[Xenomai] [RFC] x86 port to kernel 3.10

[Xenomai] [RFC] x86 port to kernel 3.10

[Jan Kiszka]

Hi all,

in the process of updating our kernel support to current (and long-term)
stable 3.10, I've pushed the I-pipe master tree to that version. The
tree is available at

git://git.xenomai.org/ipipe-jki.git next-x86

Though the focus is on x86, this became a full merge, ie. I also tried
to resolve ARM and PowerPC conflicts (other archs had none). But I'm
sure I broke some things (or things were already broken after the 3.9
merge - there were suspicious unresolved conflicts). So these arch need
a careful check! Specifically watch out for merges that resolved
conflicts in their trees.

The reason for doing this full merge is that there were simply too many
merges into Linus' tree that touched x86 and other archs in the same
run. It made no sense to split them up, only pulling what affects x86.
And I guess this will also remain true in the future.

Instead of merging 3.10 in one run (like Philippe did with 3.9), I
applied the following script-driven algorithm (can share it if there is
interest): merge stepwise what went into Linus' tree, stop when there is
a conflict that requires manual actions. Go one commit back in Linus'
tree and merge this one first (to reduce the number of merges into the
I-pipe tree). Then merge the conflicting commit, resolve it manually and
resume the merges. The result looks like this:

late I-pipe fix-ups for 3.10
merge "3.10"
merge "commit B" <-- required manual conflict solving
merge "commit A" <-- merged without conflicts
...
I-pipe head for 3.8

The advantage of this pattern is that we get more bisection points in
case something subtly breaks between 3.8 and 3.10. The disadvantage is
that the I-pipe history becomes more complex than it already is. I only
did builds and tests after reaching 3.10 (therefore the later fix-ups).
This could still be improved by doing builds after each merge - but it
would extend the required time significantly. Please let me know your
opinion about this approach and how to improve it further.

To test the current state, a few Xenomai patches are required. Find the
basic set at

git://git.xenomai.org/xenomai-jki.git for-upstream

This currently leaves CAN and Analogy broken due to procfs changes.
Maybe I'll be able to look into them later. RTnet will need a massive
update, too.

Feedback, test reports etc. are welcome!

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Gilles Chanteperdrix]

On 08/21/2013 10:39 AM, Jan Kiszka wrote:
> Hi all,
> 
> in the process of updating our kernel support to current (and long-term)
> stable 3.10, I've pushed the I-pipe master tree to that version. The
> tree is available at
> 
> git://git.xenomai.org/ipipe-jki.git next-x86
> 
> Though the focus is on x86, this became a full merge, ie. I also tried
> to resolve ARM and PowerPC conflicts (other archs had none). But I'm
> sure I broke some things (or things were already broken after the 3.9
> merge - there were suspicious unresolved conflicts). So these arch need
> a careful check! Specifically watch out for merges that resolved
> conflicts in their trees.
> 

I would prefer to have the conflict markers in the git tree, so that we
know where the conflicts occur. Having to browse history to find all the
merge conflicts seems more complicated than greping for '>>>'. Of
course, it means that when all the conflicts are resolved, the kernel
should be rebased to remove the commits with the merge conflicts, so as
to keep a bisectable kernel.

-- 
                                                                Gilles.

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Jan Kiszka]

On 2013-08-21 10:44, Gilles Chanteperdrix wrote:
> On 08/21/2013 10:39 AM, Jan Kiszka wrote:
>> Hi all,
>>
>> in the process of updating our kernel support to current (and long-term)
>> stable 3.10, I've pushed the I-pipe master tree to that version. The
>> tree is available at
>>
>> git://git.xenomai.org/ipipe-jki.git next-x86
>>
>> Though the focus is on x86, this became a full merge, ie. I also tried
>> to resolve ARM and PowerPC conflicts (other archs had none). But I'm
>> sure I broke some things (or things were already broken after the 3.9
>> merge - there were suspicious unresolved conflicts). So these arch need
>> a careful check! Specifically watch out for merges that resolved
>> conflicts in their trees.
>>
> 
> I would prefer to have the conflict markers in the git tree, so that we
> know where the conflicts occur. Having to browse history to find all the
> merge conflicts seems more complicated than greping for '>>>'. Of
> course, it means that when all the conflicts are resolved, the kernel
> should be rebased to remove the commits with the merge conflicts, so as
> to keep a bisectable kernel.

As you cannot rebase merges (without losing history), this is not really
applicable. I can leave in conflicts that affect ARM or Power, sure, but
then the fix-ups will have to happen on top, taking away bisectability.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Jan Kiszka]

On 2013-08-21 10:47, Jan Kiszka wrote:
> On 2013-08-21 10:44, Gilles Chanteperdrix wrote:
>> On 08/21/2013 10:39 AM, Jan Kiszka wrote:
>>> Hi all,
>>>
>>> in the process of updating our kernel support to current (and long-term)
>>> stable 3.10, I've pushed the I-pipe master tree to that version. The
>>> tree is available at
>>>
>>> git://git.xenomai.org/ipipe-jki.git next-x86
>>>
>>> Though the focus is on x86, this became a full merge, ie. I also tried
>>> to resolve ARM and PowerPC conflicts (other archs had none). But I'm
>>> sure I broke some things (or things were already broken after the 3.9
>>> merge - there were suspicious unresolved conflicts). So these arch need
>>> a careful check! Specifically watch out for merges that resolved
>>> conflicts in their trees.
>>>
>>
>> I would prefer to have the conflict markers in the git tree, so that we
>> know where the conflicts occur. Having to browse history to find all the
>> merge conflicts seems more complicated than greping for '>>>'. Of
>> course, it means that when all the conflicts are resolved, the kernel
>> should be rebased to remove the commits with the merge conflicts, so as
>> to keep a bisectable kernel.
> 
> As you cannot rebase merges (without losing history), this is not really
> applicable. I can leave in conflicts that affect ARM or Power, sure, but
> then the fix-ups will have to happen on top, taking away bisectability.

Hmm, what we would really need is some tool to replay merges with their
conflict resolution. That must be feasible, somehow...

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Gilles Chanteperdrix]

On 08/21/2013 10:54 AM, Jan Kiszka wrote:
> On 2013-08-21 10:47, Jan Kiszka wrote:
>> On 2013-08-21 10:44, Gilles Chanteperdrix wrote:
>>> On 08/21/2013 10:39 AM, Jan Kiszka wrote:
>>>> Hi all,
>>>>
>>>> in the process of updating our kernel support to current (and long-term)
>>>> stable 3.10, I've pushed the I-pipe master tree to that version. The
>>>> tree is available at
>>>>
>>>> git://git.xenomai.org/ipipe-jki.git next-x86
>>>>
>>>> Though the focus is on x86, this became a full merge, ie. I also tried
>>>> to resolve ARM and PowerPC conflicts (other archs had none). But I'm
>>>> sure I broke some things (or things were already broken after the 3.9
>>>> merge - there were suspicious unresolved conflicts). So these arch need
>>>> a careful check! Specifically watch out for merges that resolved
>>>> conflicts in their trees.
>>>>
>>>
>>> I would prefer to have the conflict markers in the git tree, so that we
>>> know where the conflicts occur. Having to browse history to find all the
>>> merge conflicts seems more complicated than greping for '>>>'. Of
>>> course, it means that when all the conflicts are resolved, the kernel
>>> should be rebased to remove the commits with the merge conflicts, so as
>>> to keep a bisectable kernel.
>>
>> As you cannot rebase merges (without losing history), this is not really
>> applicable. I can leave in conflicts that affect ARM or Power, sure, but
>> then the fix-ups will have to happen on top, taking away bisectability.
> 
> Hmm, what we would really need is some tool to replay merges with their
> conflict resolution. That must be feasible, somehow...

You mean like git rerere? Anyway, why not just one big commit with all
the conflicts?


-- 
                                                                Gilles.

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Jan Kiszka]

On 2013-08-21 13:32, Gilles Chanteperdrix wrote:
> On 08/21/2013 10:54 AM, Jan Kiszka wrote:
>> On 2013-08-21 10:47, Jan Kiszka wrote:
>>> On 2013-08-21 10:44, Gilles Chanteperdrix wrote:
>>>> On 08/21/2013 10:39 AM, Jan Kiszka wrote:
>>>>> Hi all,
>>>>>
>>>>> in the process of updating our kernel support to current (and long-term)
>>>>> stable 3.10, I've pushed the I-pipe master tree to that version. The
>>>>> tree is available at
>>>>>
>>>>> git://git.xenomai.org/ipipe-jki.git next-x86
>>>>>
>>>>> Though the focus is on x86, this became a full merge, ie. I also tried
>>>>> to resolve ARM and PowerPC conflicts (other archs had none). But I'm
>>>>> sure I broke some things (or things were already broken after the 3.9
>>>>> merge - there were suspicious unresolved conflicts). So these arch need
>>>>> a careful check! Specifically watch out for merges that resolved
>>>>> conflicts in their trees.
>>>>>
>>>>
>>>> I would prefer to have the conflict markers in the git tree, so that we
>>>> know where the conflicts occur. Having to browse history to find all the
>>>> merge conflicts seems more complicated than greping for '>>>'. Of
>>>> course, it means that when all the conflicts are resolved, the kernel
>>>> should be rebased to remove the commits with the merge conflicts, so as
>>>> to keep a bisectable kernel.
>>>
>>> As you cannot rebase merges (without losing history), this is not really
>>> applicable. I can leave in conflicts that affect ARM or Power, sure, but
>>> then the fix-ups will have to happen on top, taking away bisectability.
>>
>> Hmm, what we would really need is some tool to replay merges with their
>> conflict resolution. That must be feasible, somehow...
> 
> You mean like git rerere? Anyway, why not just one big commit with all
> the conflicts?

For the reason given in my original message.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Gilles Chanteperdrix]

On 08/21/2013 01:46 PM, Jan Kiszka wrote:
> On 2013-08-21 13:32, Gilles Chanteperdrix wrote:
>> On 08/21/2013 10:54 AM, Jan Kiszka wrote:
>>> On 2013-08-21 10:47, Jan Kiszka wrote:
>>>> On 2013-08-21 10:44, Gilles Chanteperdrix wrote:
>>>>> On 08/21/2013 10:39 AM, Jan Kiszka wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> in the process of updating our kernel support to current (and long-term)
>>>>>> stable 3.10, I've pushed the I-pipe master tree to that version. The
>>>>>> tree is available at
>>>>>>
>>>>>> git://git.xenomai.org/ipipe-jki.git next-x86
>>>>>>
>>>>>> Though the focus is on x86, this became a full merge, ie. I also tried
>>>>>> to resolve ARM and PowerPC conflicts (other archs had none). But I'm
>>>>>> sure I broke some things (or things were already broken after the 3.9
>>>>>> merge - there were suspicious unresolved conflicts). So these arch need
>>>>>> a careful check! Specifically watch out for merges that resolved
>>>>>> conflicts in their trees.
>>>>>>
>>>>>
>>>>> I would prefer to have the conflict markers in the git tree, so that we
>>>>> know where the conflicts occur. Having to browse history to find all the
>>>>> merge conflicts seems more complicated than greping for '>>>'. Of
>>>>> course, it means that when all the conflicts are resolved, the kernel
>>>>> should be rebased to remove the commits with the merge conflicts, so as
>>>>> to keep a bisectable kernel.
>>>>
>>>> As you cannot rebase merges (without losing history), this is not really
>>>> applicable. I can leave in conflicts that affect ARM or Power, sure, but
>>>> then the fix-ups will have to happen on top, taking away bisectability.
>>>
>>> Hmm, what we would really need is some tool to replay merges with their
>>> conflict resolution. That must be feasible, somehow...
>>
>> You mean like git rerere? Anyway, why not just one big commit with all
>> the conflicts?
> 
> For the reason given in my original message.

The mail explains the how, but not the why. Having to skim through tens
of commits to find which ones may be related to a particular arch is
simply a major pain. The "old" method which was to merge the patch with
all the changes was much simpler. So, I do not see what we gain with the
new method if it makes things more complicated.

> 
> Jan
> 


-- 
                                                                Gilles.

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Jan Kiszka]

On 2013-08-21 13:49, Gilles Chanteperdrix wrote:
> On 08/21/2013 01:46 PM, Jan Kiszka wrote:
>> On 2013-08-21 13:32, Gilles Chanteperdrix wrote:
>>> On 08/21/2013 10:54 AM, Jan Kiszka wrote:
>>>> On 2013-08-21 10:47, Jan Kiszka wrote:
>>>>> On 2013-08-21 10:44, Gilles Chanteperdrix wrote:
>>>>>> On 08/21/2013 10:39 AM, Jan Kiszka wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> in the process of updating our kernel support to current (and long-term)
>>>>>>> stable 3.10, I've pushed the I-pipe master tree to that version. The
>>>>>>> tree is available at
>>>>>>>
>>>>>>> git://git.xenomai.org/ipipe-jki.git next-x86
>>>>>>>
>>>>>>> Though the focus is on x86, this became a full merge, ie. I also tried
>>>>>>> to resolve ARM and PowerPC conflicts (other archs had none). But I'm
>>>>>>> sure I broke some things (or things were already broken after the 3.9
>>>>>>> merge - there were suspicious unresolved conflicts). So these arch need
>>>>>>> a careful check! Specifically watch out for merges that resolved
>>>>>>> conflicts in their trees.
>>>>>>>
>>>>>>
>>>>>> I would prefer to have the conflict markers in the git tree, so that we
>>>>>> know where the conflicts occur. Having to browse history to find all the
>>>>>> merge conflicts seems more complicated than greping for '>>>'. Of
>>>>>> course, it means that when all the conflicts are resolved, the kernel
>>>>>> should be rebased to remove the commits with the merge conflicts, so as
>>>>>> to keep a bisectable kernel.
>>>>>
>>>>> As you cannot rebase merges (without losing history), this is not really
>>>>> applicable. I can leave in conflicts that affect ARM or Power, sure, but
>>>>> then the fix-ups will have to happen on top, taking away bisectability.
>>>>
>>>> Hmm, what we would really need is some tool to replay merges with their
>>>> conflict resolution. That must be feasible, somehow...
>>>
>>> You mean like git rerere? Anyway, why not just one big commit with all
>>> the conflicts?
>>
>> For the reason given in my original message.
> 
> The mail explains the how, but not the why. Having to skim through tens
> of commits to find which ones may be related to a particular arch is
> simply a major pain. The "old" method which was to merge the patch with
> all the changes was much simpler. So, I do not see what we gain with the
> new method if it makes things more complicated.

"The advantage of this pattern is that we get more bisection points in
case something subtly breaks between 3.8 and 3.10."

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Gilles Chanteperdrix]

On 08/21/2013 01:53 PM, Jan Kiszka wrote:
> On 2013-08-21 13:49, Gilles Chanteperdrix wrote:
>> On 08/21/2013 01:46 PM, Jan Kiszka wrote:
>>> On 2013-08-21 13:32, Gilles Chanteperdrix wrote:
>>>> On 08/21/2013 10:54 AM, Jan Kiszka wrote:
>>>>> On 2013-08-21 10:47, Jan Kiszka wrote:
>>>>>> On 2013-08-21 10:44, Gilles Chanteperdrix wrote:
>>>>>>> On 08/21/2013 10:39 AM, Jan Kiszka wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> in the process of updating our kernel support to current (and long-term)
>>>>>>>> stable 3.10, I've pushed the I-pipe master tree to that version. The
>>>>>>>> tree is available at
>>>>>>>>
>>>>>>>> git://git.xenomai.org/ipipe-jki.git next-x86
>>>>>>>>
>>>>>>>> Though the focus is on x86, this became a full merge, ie. I also tried
>>>>>>>> to resolve ARM and PowerPC conflicts (other archs had none). But I'm
>>>>>>>> sure I broke some things (or things were already broken after the 3.9
>>>>>>>> merge - there were suspicious unresolved conflicts). So these arch need
>>>>>>>> a careful check! Specifically watch out for merges that resolved
>>>>>>>> conflicts in their trees.
>>>>>>>>
>>>>>>>
>>>>>>> I would prefer to have the conflict markers in the git tree, so that we
>>>>>>> know where the conflicts occur. Having to browse history to find all the
>>>>>>> merge conflicts seems more complicated than greping for '>>>'. Of
>>>>>>> course, it means that when all the conflicts are resolved, the kernel
>>>>>>> should be rebased to remove the commits with the merge conflicts, so as
>>>>>>> to keep a bisectable kernel.
>>>>>>
>>>>>> As you cannot rebase merges (without losing history), this is not really
>>>>>> applicable. I can leave in conflicts that affect ARM or Power, sure, but
>>>>>> then the fix-ups will have to happen on top, taking away bisectability.
>>>>>
>>>>> Hmm, what we would really need is some tool to replay merges with their
>>>>> conflict resolution. That must be feasible, somehow...
>>>>
>>>> You mean like git rerere? Anyway, why not just one big commit with all
>>>> the conflicts?
>>>
>>> For the reason given in my original message.
>>
>> The mail explains the how, but not the why. Having to skim through tens
>> of commits to find which ones may be related to a particular arch is
>> simply a major pain. The "old" method which was to merge the patch with
>> all the changes was much simpler. So, I do not see what we gain with the
>> new method if it makes things more complicated.
> 
> "The advantage of this pattern is that we get more bisection points in
> case something subtly breaks between 3.8 and 3.10."

However you do it, if there are several separate commits for all the
architectures, the only bisectable architecture will be the one that was
merged first. The others are broken anyway. At least if the conflict
markers are in the tree we will know this at compilation time and not at
run time. The only way to have bisectability is with one commit with all
the architectures tested. Maybe we can do this by squashing the merge
commit with all the fixup commits made afterwards?

-- 
                                                                Gilles.

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Jan Kiszka]

On 2013-08-21 13:58, Gilles Chanteperdrix wrote:
> On 08/21/2013 01:53 PM, Jan Kiszka wrote:
>> On 2013-08-21 13:49, Gilles Chanteperdrix wrote:
>>> On 08/21/2013 01:46 PM, Jan Kiszka wrote:
>>>> On 2013-08-21 13:32, Gilles Chanteperdrix wrote:
>>>>> On 08/21/2013 10:54 AM, Jan Kiszka wrote:
>>>>>> On 2013-08-21 10:47, Jan Kiszka wrote:
>>>>>>> On 2013-08-21 10:44, Gilles Chanteperdrix wrote:
>>>>>>>> On 08/21/2013 10:39 AM, Jan Kiszka wrote:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> in the process of updating our kernel support to current (and long-term)
>>>>>>>>> stable 3.10, I've pushed the I-pipe master tree to that version. The
>>>>>>>>> tree is available at
>>>>>>>>>
>>>>>>>>> git://git.xenomai.org/ipipe-jki.git next-x86
>>>>>>>>>
>>>>>>>>> Though the focus is on x86, this became a full merge, ie. I also tried
>>>>>>>>> to resolve ARM and PowerPC conflicts (other archs had none). But I'm
>>>>>>>>> sure I broke some things (or things were already broken after the 3.9
>>>>>>>>> merge - there were suspicious unresolved conflicts). So these arch need
>>>>>>>>> a careful check! Specifically watch out for merges that resolved
>>>>>>>>> conflicts in their trees.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I would prefer to have the conflict markers in the git tree, so that we
>>>>>>>> know where the conflicts occur. Having to browse history to find all the
>>>>>>>> merge conflicts seems more complicated than greping for '>>>'. Of
>>>>>>>> course, it means that when all the conflicts are resolved, the kernel
>>>>>>>> should be rebased to remove the commits with the merge conflicts, so as
>>>>>>>> to keep a bisectable kernel.
>>>>>>>
>>>>>>> As you cannot rebase merges (without losing history), this is not really
>>>>>>> applicable. I can leave in conflicts that affect ARM or Power, sure, but
>>>>>>> then the fix-ups will have to happen on top, taking away bisectability.
>>>>>>
>>>>>> Hmm, what we would really need is some tool to replay merges with their
>>>>>> conflict resolution. That must be feasible, somehow...
>>>>>
>>>>> You mean like git rerere? Anyway, why not just one big commit with all
>>>>> the conflicts?
>>>>
>>>> For the reason given in my original message.
>>>
>>> The mail explains the how, but not the why. Having to skim through tens
>>> of commits to find which ones may be related to a particular arch is
>>> simply a major pain. The "old" method which was to merge the patch with
>>> all the changes was much simpler. So, I do not see what we gain with the
>>> new method if it makes things more complicated.
>>
>> "The advantage of this pattern is that we get more bisection points in
>> case something subtly breaks between 3.8 and 3.10."
> 
> However you do it, if there are several separate commits for all the
> architectures, the only bisectable architecture will be the one that was
> merged first. The others are broken anyway.

Not necessarily. I tried to resolve all conflicts as they showed up, so
not every state needs to be broken.

> At least if the conflict
> markers are in the tree we will know this at compilation time and not at
> run time. The only way to have bisectability is with one commit with all
> the architectures tested. Maybe we can do this by squashing the merge
> commit with all the fixup commits made afterwards?

Squashing or rebasing (I tried rebase --preserve-merges by now - not
usable) means loosing track where the merges came from.

To summarize, I think we are discussing two questions here:

1. How many merges per kernel version update (one vs. several as in my
   model)?
2. How to deal with conflicts during the merge that affect architecture
   that the merger is not (immediately or at all) testing?
    - commit them unresolved and commit resolutions on top?
    - resolve them as good as possible and leave the review/fix-up to
      the respective arch maintainer?

I don't see other options for 2. at this moment.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Gilles Chanteperdrix]

On 08/21/2013 03:39 PM, Jan Kiszka wrote:
> On 2013-08-21 13:58, Gilles Chanteperdrix wrote:
>> On 08/21/2013 01:53 PM, Jan Kiszka wrote:
>>> On 2013-08-21 13:49, Gilles Chanteperdrix wrote:
>>>> On 08/21/2013 01:46 PM, Jan Kiszka wrote:
>>>>> On 2013-08-21 13:32, Gilles Chanteperdrix wrote:
>>>>>> On 08/21/2013 10:54 AM, Jan Kiszka wrote:
>>>>>>> On 2013-08-21 10:47, Jan Kiszka wrote:
>>>>>>>> On 2013-08-21 10:44, Gilles Chanteperdrix wrote:
>>>>>>>>> On 08/21/2013 10:39 AM, Jan Kiszka wrote:
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> in the process of updating our kernel support to current (and long-term)
>>>>>>>>>> stable 3.10, I've pushed the I-pipe master tree to that version. The
>>>>>>>>>> tree is available at
>>>>>>>>>>
>>>>>>>>>> git://git.xenomai.org/ipipe-jki.git next-x86
>>>>>>>>>>
>>>>>>>>>> Though the focus is on x86, this became a full merge, ie. I also tried
>>>>>>>>>> to resolve ARM and PowerPC conflicts (other archs had none). But I'm
>>>>>>>>>> sure I broke some things (or things were already broken after the 3.9
>>>>>>>>>> merge - there were suspicious unresolved conflicts). So these arch need
>>>>>>>>>> a careful check! Specifically watch out for merges that resolved
>>>>>>>>>> conflicts in their trees.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I would prefer to have the conflict markers in the git tree, so that we
>>>>>>>>> know where the conflicts occur. Having to browse history to find all the
>>>>>>>>> merge conflicts seems more complicated than greping for '>>>'. Of
>>>>>>>>> course, it means that when all the conflicts are resolved, the kernel
>>>>>>>>> should be rebased to remove the commits with the merge conflicts, so as
>>>>>>>>> to keep a bisectable kernel.
>>>>>>>>
>>>>>>>> As you cannot rebase merges (without losing history), this is not really
>>>>>>>> applicable. I can leave in conflicts that affect ARM or Power, sure, but
>>>>>>>> then the fix-ups will have to happen on top, taking away bisectability.
>>>>>>>
>>>>>>> Hmm, what we would really need is some tool to replay merges with their
>>>>>>> conflict resolution. That must be feasible, somehow...
>>>>>>
>>>>>> You mean like git rerere? Anyway, why not just one big commit with all
>>>>>> the conflicts?
>>>>>
>>>>> For the reason given in my original message.
>>>>
>>>> The mail explains the how, but not the why. Having to skim through tens
>>>> of commits to find which ones may be related to a particular arch is
>>>> simply a major pain. The "old" method which was to merge the patch with
>>>> all the changes was much simpler. So, I do not see what we gain with the
>>>> new method if it makes things more complicated.
>>>
>>> "The advantage of this pattern is that we get more bisection points in
>>> case something subtly breaks between 3.8 and 3.10."
>>
>> However you do it, if there are several separate commits for all the
>> architectures, the only bisectable architecture will be the one that was
>> merged first. The others are broken anyway.
> 
> Not necessarily. I tried to resolve all conflicts as they showed up, so
> not every state needs to be broken.
> 
>> At least if the conflict
>> markers are in the tree we will know this at compilation time and not at
>> run time. The only way to have bisectability is with one commit with all
>> the architectures tested. Maybe we can do this by squashing the merge
>> commit with all the fixup commits made afterwards?
> 
> Squashing or rebasing (I tried rebase --preserve-merges by now - not
> usable) means loosing track where the merges came from.
> 
> To summarize, I think we are discussing two questions here:
> 
> 1. How many merges per kernel version update (one vs. several as in my
>    model)?
> 2. How to deal with conflicts during the merge that affect architecture
>    that the merger is not (immediately or at all) testing?
>     - commit them unresolved and commit resolutions on top?
>     - resolve them as good as possible and leave the review/fix-up to
>       the respective arch maintainer?
> 
> I don't see other options for 2. at this moment.

What I say, is that the two answers you propose to question 2 result in
a series of commit which can not be bisected. With 2.1, you are warned
at compilation time, because the unbisectable commits do not compile,
with 2.2 you have to test the kernel to realize that it is not working.

So, the only possibility to have a fully bisectable hisotry would be to
somehow handle all architectures at once, whether with one, or several
commits.

The result being that the new approach results in a bisectable kernel
only for one architecture, and greatly complicates the merge of the
other architectures, I am wondering about the benefits of this new approach.

-- 
                                                                Gilles.

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Jan Kiszka]

On 2013-08-21 20:23, Gilles Chanteperdrix wrote:
> On 08/21/2013 03:39 PM, Jan Kiszka wrote:
>> On 2013-08-21 13:58, Gilles Chanteperdrix wrote:
>>> On 08/21/2013 01:53 PM, Jan Kiszka wrote:
>>>> On 2013-08-21 13:49, Gilles Chanteperdrix wrote:
>>>>> On 08/21/2013 01:46 PM, Jan Kiszka wrote:
>>>>>> On 2013-08-21 13:32, Gilles Chanteperdrix wrote:
>>>>>>> On 08/21/2013 10:54 AM, Jan Kiszka wrote:
>>>>>>>> On 2013-08-21 10:47, Jan Kiszka wrote:
>>>>>>>>> On 2013-08-21 10:44, Gilles Chanteperdrix wrote:
>>>>>>>>>> On 08/21/2013 10:39 AM, Jan Kiszka wrote:
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> in the process of updating our kernel support to current (and long-term)
>>>>>>>>>>> stable 3.10, I've pushed the I-pipe master tree to that version. The
>>>>>>>>>>> tree is available at
>>>>>>>>>>>
>>>>>>>>>>> git://git.xenomai.org/ipipe-jki.git next-x86
>>>>>>>>>>>
>>>>>>>>>>> Though the focus is on x86, this became a full merge, ie. I also tried
>>>>>>>>>>> to resolve ARM and PowerPC conflicts (other archs had none). But I'm
>>>>>>>>>>> sure I broke some things (or things were already broken after the 3.9
>>>>>>>>>>> merge - there were suspicious unresolved conflicts). So these arch need
>>>>>>>>>>> a careful check! Specifically watch out for merges that resolved
>>>>>>>>>>> conflicts in their trees.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I would prefer to have the conflict markers in the git tree, so that we
>>>>>>>>>> know where the conflicts occur. Having to browse history to find all the
>>>>>>>>>> merge conflicts seems more complicated than greping for '>>>'. Of
>>>>>>>>>> course, it means that when all the conflicts are resolved, the kernel
>>>>>>>>>> should be rebased to remove the commits with the merge conflicts, so as
>>>>>>>>>> to keep a bisectable kernel.
>>>>>>>>>
>>>>>>>>> As you cannot rebase merges (without losing history), this is not really
>>>>>>>>> applicable. I can leave in conflicts that affect ARM or Power, sure, but
>>>>>>>>> then the fix-ups will have to happen on top, taking away bisectability.
>>>>>>>>
>>>>>>>> Hmm, what we would really need is some tool to replay merges with their
>>>>>>>> conflict resolution. That must be feasible, somehow...
>>>>>>>
>>>>>>> You mean like git rerere? Anyway, why not just one big commit with all
>>>>>>> the conflicts?
>>>>>>
>>>>>> For the reason given in my original message.
>>>>>
>>>>> The mail explains the how, but not the why. Having to skim through tens
>>>>> of commits to find which ones may be related to a particular arch is
>>>>> simply a major pain. The "old" method which was to merge the patch with
>>>>> all the changes was much simpler. So, I do not see what we gain with the
>>>>> new method if it makes things more complicated.
>>>>
>>>> "The advantage of this pattern is that we get more bisection points in
>>>> case something subtly breaks between 3.8 and 3.10."
>>>
>>> However you do it, if there are several separate commits for all the
>>> architectures, the only bisectable architecture will be the one that was
>>> merged first. The others are broken anyway.
>>
>> Not necessarily. I tried to resolve all conflicts as they showed up, so
>> not every state needs to be broken.
>>
>>> At least if the conflict
>>> markers are in the tree we will know this at compilation time and not at
>>> run time. The only way to have bisectability is with one commit with all
>>> the architectures tested. Maybe we can do this by squashing the merge
>>> commit with all the fixup commits made afterwards?
>>
>> Squashing or rebasing (I tried rebase --preserve-merges by now - not
>> usable) means loosing track where the merges came from.
>>
>> To summarize, I think we are discussing two questions here:
>>
>> 1. How many merges per kernel version update (one vs. several as in my
>>    model)?
>> 2. How to deal with conflicts during the merge that affect architecture
>>    that the merger is not (immediately or at all) testing?
>>     - commit them unresolved and commit resolutions on top?
>>     - resolve them as good as possible and leave the review/fix-up to
>>       the respective arch maintainer?
>>
>> I don't see other options for 2. at this moment.
> 
> What I say, is that the two answers you propose to question 2 result in
> a series of commit which can not be bisected.

s/cannot/may not/. The current one is likely not bisectable due to the
3.9 merge of Philippe which left some unresolved conflicts behind. But
if we resolve conflicts for all arches during the mege, there is a good
chance that early parts of the merge are still working and that later
parts can be made working more easily (e.g. by temporarily applying
fix-ups from the head) if bisection is required.

> With 2.1, you are warned
> at compilation time, because the unbisectable commits do not compile,
> with 2.2 you have to test the kernel to realize that it is not working.

With complex tree, you generally also have to fix-up bisection points in
order to make them work. The question is, how much work that is.

> 
> So, the only possibility to have a fully bisectable hisotry would be to
> somehow handle all architectures at once, whether with one, or several
> commits.
> 
> The result being that the new approach results in a bisectable kernel
> only for one architecture, and greatly complicates the merge of the
> other architectures, I am wondering about the benefits of this new approach.
> 

I don't see yet what complicates the merge of other archs that much.
Look at the merge commits that had conflicts on ARM (e.g. git log
--first-parent -p -c master..next-x86 arch/arm) and review the
resolutions. Many of them will likely be correct, so you no longer have
to do this job yourself. And the number of conflicts is not much higher
than when doing a single merge, it's just spread over several commits.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Gilles Chanteperdrix]

Jan Kiszka wrote:
>>> 2. How to deal with conflicts during the merge that affect architecture
>>>    that the merger is not (immediately or at all) testing?
>>>     - commit them unresolved and commit resolutions on top?
>>>     - resolve them as good as possible and leave the review/fix-up to
>>>       the respective arch maintainer?
>>>
>>> I don't see other options for 2. at this moment.
>>
>> What I say, is that the two answers you propose to question 2 result in
>> a series of commit which can not be bisected.
>
> s/cannot/may not/. The current one is likely not bisectable due to the
> 3.9 merge of Philippe which left some unresolved conflicts behind. But
> if we resolve conflicts for all arches during the mege, there is a good
> chance that early parts of the merge are still working and that later
> parts can be made working more easily (e.g. by temporarily applying
> fix-ups from the head) if bisection is required.

In my experience, with the ARM architecture, the result of the merge,
however trivial, does not work, additional work is needed.

>
>> With 2.1, you are warned
>> at compilation time, because the unbisectable commits do not compile,
>> with 2.2 you have to test the kernel to realize that it is not working.
>
> With complex tree, you generally also have to fix-up bisection points in
> order to make them work. The question is, how much work that is.

With the ARM architecture evolution pace having increased lately, this is
the major part of the work of porting the I-pipe patch from one kernel to
the next.

>
>>
>> So, the only possibility to have a fully bisectable hisotry would be to
>> somehow handle all architectures at once, whether with one, or several
>> commits.
>>
>> The result being that the new approach results in a bisectable kernel
>> only for one architecture, and greatly complicates the merge of the
>> other architectures, I am wondering about the benefits of this new
>> approach.
>>
>
> I don't see yet what complicates the merge of other archs that much.
> Look at the merge commits that had conflicts on ARM (e.g. git log
> --first-parent -p -c master..next-x86 arch/arm) and review the
> resolutions. Many of them will likely be correct, so you no longer have
> to do this job yourself. And the number of conflicts is not much higher
> than when doing a single merge, it's just spread over several commits.

Because having one set of diffs is simpler than having to skim through
several commits. Also, I want to do the merge myself, because my memory
is associative, and when I see a bug when testing the result of the merge,
which never works out of the box, it may remind me of a merge that could be
the cause of the bug. Having not done the merge myself will make the debug
harder, I will have to discover what I could already know.

So, please commit the merge conflicts and let me resolve them. There is no
perfect solution, let us take the one which does not make the job too hard.

-- 
                                              Gilles

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Jan Kiszka]

On 2013-08-22 11:37, Gilles Chanteperdrix wrote:
> 
> Jan Kiszka wrote:
>>>> 2. How to deal with conflicts during the merge that affect architecture
>>>>    that the merger is not (immediately or at all) testing?
>>>>     - commit them unresolved and commit resolutions on top?
>>>>     - resolve them as good as possible and leave the review/fix-up to
>>>>       the respective arch maintainer?
>>>>
>>>> I don't see other options for 2. at this moment.
>>>
>>> What I say, is that the two answers you propose to question 2 result in
>>> a series of commit which can not be bisected.
>>
>> s/cannot/may not/. The current one is likely not bisectable due to the
>> 3.9 merge of Philippe which left some unresolved conflicts behind. But
>> if we resolve conflicts for all arches during the mege, there is a good
>> chance that early parts of the merge are still working and that later
>> parts can be made working more easily (e.g. by temporarily applying
>> fix-ups from the head) if bisection is required.
> 
> In my experience, with the ARM architecture, the result of the merge,
> however trivial, does not work, additional work is needed.
> 
>>
>>> With 2.1, you are warned
>>> at compilation time, because the unbisectable commits do not compile,
>>> with 2.2 you have to test the kernel to realize that it is not working.
>>
>> With complex tree, you generally also have to fix-up bisection points in
>> order to make them work. The question is, how much work that is.
> 
> With the ARM architecture evolution pace having increased lately, this is
> the major part of the work of porting the I-pipe patch from one kernel to
> the next.
> 
>>
>>>
>>> So, the only possibility to have a fully bisectable hisotry would be to
>>> somehow handle all architectures at once, whether with one, or several
>>> commits.
>>>
>>> The result being that the new approach results in a bisectable kernel
>>> only for one architecture, and greatly complicates the merge of the
>>> other architectures, I am wondering about the benefits of this new
>>> approach.
>>>
>>
>> I don't see yet what complicates the merge of other archs that much.
>> Look at the merge commits that had conflicts on ARM (e.g. git log
>> --first-parent -p -c master..next-x86 arch/arm) and review the
>> resolutions. Many of them will likely be correct, so you no longer have
>> to do this job yourself. And the number of conflicts is not much higher
>> than when doing a single merge, it's just spread over several commits.
> 
> Because having one set of diffs is simpler than having to skim through
> several commits. Also, I want to do the merge myself, because my memory
> is associative, and when I see a bug when testing the result of the merge,
> which never works out of the box, it may remind me of a merge that could be
> the cause of the bug. Having not done the merge myself will make the debug
> harder, I will have to discover what I could already know.
> 
> So, please commit the merge conflicts and let me resolve them. There is no
> perfect solution, let us take the one which does not make the job too hard.
> 

OK, then I will redo the merge of 3.10 in a single commit, but
preserving the work of Philippe for 3.9.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[John Morris]

On 08/21/2013 08:39 AM, Jan Kiszka wrote:
> Squashing or rebasing (I tried rebase --preserve-merges by now - not
> usable) means loosing track where the merges came from.

I recently used Peter Wortmann's rebase --interactive script, smartened
up for preserving merges, to get through a 1200+ commit/80+ merge rebase
of over a half year's work.  Saved a ton of tears (at least).

https://github.com/scpmw/git/commit/33c707d

Explanation in this commit:

https://github.com/scpmw/git/commit/fa11202

	John

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Jan Kiszka]

On 2013-08-24 09:29, John Morris wrote:
> On 08/21/2013 08:39 AM, Jan Kiszka wrote:
>> Squashing or rebasing (I tried rebase --preserve-merges by now - not
>> usable) means loosing track where the merges came from.
> 
> I recently used Peter Wortmann's rebase --interactive script, smartened
> up for preserving merges, to get through a 1200+ commit/80+ merge rebase
> of over a half year's work.  Saved a ton of tears (at least).
> 
> https://github.com/scpmw/git/commit/33c707d
> 
> Explanation in this commit:
> 
> https://github.com/scpmw/git/commit/fa11202
> 

Cool, thanks for the hint!

Jan


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 263 bytes
Desc: OpenPGP digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20130824/46e935bd/attachment.pgp>

Re: [Xenomai] [RFC] x86 port to kernel 3.10

[Jan Kiszka]

On 2013-08-22 18:49, Jan Kiszka wrote:
> OK, then I will redo the merge of 3.10 in a single commit, but
> preserving the work of Philippe for 3.9.

Done, result can be found again at

git://git.xenomai.org/ipipe-jki.git next-x86

Gilles, Philippe, you may want to try John's suggestion and install that
patched git version. It allows to amend also merge commits (or reorder
them), thus could enable a workflow where the arch maintainers fixes up
the merges. The downside: it is awfully slow. I had to wait 15 min. or
so for git rebase -ip to give me the todo file and several minutes more
to see the changes applied (I fixed up my own merge with a forgotten hunk).

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux