Some open issues

Some open issues

[Richard Purdie]

Hi Guys,

I have a few issues with the way certain things have happened recently.

1. The FILE_PR change.

This was mentioned in an email on Wednesday with the title "[oe] [RFC]
Enable --hash-style=both for all recent gcc4 targets" at 9am. At 8pm we
have "I decided to land now as PRs are changing all the time and keeping
up with things is pretty hard...". This is not in keeping with the major
changes policy we agreed by any stretch of the imagination.

This change breaks compatibility with everyone's overlays and creates
the nightmare scenario of external OE "branches" being forced into the
change or forever being unable to sync (Openmoko and Poky spring to
mind).

What is most annoying is that given a bit longer I think we could have
done something that would have meant this was unnecessary, specifically
inserted the revision into the package at package_*.bbclass time where
we can manipulated PR as needed. This combined with a staging ABI change
would have been all that was needed. If staging ABI isn't enough, we can
insert the modified PR into STAMPS instead of the real PR or some other
magic. My point is that there are better options than FILE_PR, it just
needs some thought. The fact the testing branch had so many merge issues
should have meant a better idea was sought, not that is should be
committed ASAP.

2. The Git conversion including the BKCVS data.

I'd made it quite clear this should have been a tree graft and this
wasn't done so we're now stuck with broken history :(. This is pretty
frustrating since I'd repeatedly said not to include it and went to the
effort of gathering my conversion data and sending it to Jan who then
didn't realise what I meant by graft (though no fault of his own).

3. Merging Bitbake into OE

People are saying things like "Might be a chance to reconsider
maintaining BitBake in the OE repository.". Could people please talk to
the bitbake maintainer about things like this *before* encouraging it in
public. If we need a release lets make one (which seems to be the real
problem).

4. Bitbake changes

These should go to the bitbake list as well as the OE list and should be
discussed. I've raised issues with patches which have been ignored and
these patches have now just need committed. I'm not happy about the
process that was used :(. I know people have various commitments but we
need to try and stick to some kind of process for this kind of thing.


Where from here?

I'd actually like to a strong line on this and suggest we revert the
FILE_PR change since its badly thought out and also that we consider
redoing the git conversion ASAP and the replaying the recent commits
before its too late to get rid of the corruption in there. This is
probably going to have to go to a core team vote since its a pretty big
change to suggest but opinions are welcome.

Richard

Re: Some open issues

[Rod Whitby]

I'm not on the core team, but wanted to say that I would support the decision of the core team if they agree that Richard's proposals are worth the effort to keep the architecture and history clean.
-- Rod

-----Original Message-----
From: Richard Purdie <rpurdie@rpsys.net>
Date: Saturday, Oct 18, 2008 7:37 pm
Subject: [oe] Some open issues
To: openembedded-devel <openembedded-devel@openembedded.org>Reply-To: openembedded-devel@lists.openembedded.org

Hi Guys,
>
>I have a few issues with the way certain things have happened recently.
>
>1. The FILE_PR change.
>
>This was mentioned in an email on Wednesday with the title "[oe] [RFC] Enable --hash-style=both for all recent gcc4 targets" at 9am. At 8pm we have "I decided to land now as PRs are changing all the time and keeping up with things is pretty hard...". This is not in keeping with the major changes policy we agreed by any stretch of the imagination.
>
>This change breaks compatibility with everyone's overlays and creates the nightmare scenario of external OE "branches" being forced into the change or forever being unable to sync (Openmoko and Poky spring to
>mind).
>
>What is most annoying is that given a bit longer I think we could have done something that would have meant this was unnecessary, specifically inserted the revision into the package at package_*.bbclass time where we can manipulated PR as needed. This combined with a staging ABI change would have been all that was needed. If staging ABI isn't enough, we can insert the modified PR into STAMPS instead of the real PR or some other magic. My point is that there are better options than FILE_PR, it just needs some thought. The fact the testing branch had so many merge issues should have meant a better idea was sought, not that is should be
>committed ASAP.
>
>2. The Git conversion including the BKCVS data.
>
>I'd made it quite clear this should have been a tree graft and this wasn't done so we're now stuck with broken history :(. This is pretty frustrating since I'd repeatedly said not to include it and went to the effort of gathering my conversion data and sending it to Jan who then didn't realise what I meant by graft (though no fault of his own).
>
>3. Merging Bitbake into OE
>
>People are saying things like "Might be a chance to reconsider
>maintaining BitBake in the OE repository.". Could people please talk to the bitbake maintainer about things like this *before* encouraging it in public. If we need a release lets make one (which seems to be the real problem).
>
>4. Bitbake changes
>
>These should go to the bitbake list as well as the OE list and should be discussed. I've raised issues with patches which have been ignored and these patches have now just need committed. I'm not happy about the
>process that was used :(. I know people have various commitments but we need to try and stick to some kind of process for this kind of thing.
>
>
>Where from here?
>
>I'd actually like to a strong line on this and suggest we revert the FILE_PR change since its badly thought out and also that we consider
>redoing the git conversion ASAP and the replaying the recent commits
>before its too late to get rid of the corruption in there. This is
>probably going to have to go to a core team vote since its a pretty big change to suggest but opinions are welcome.
>
>Richard
>
>
>
>_______________________________________________
>Openembedded-devel mailing list
>Openembedded-devel@lists.openembedded.org
>http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>
>

Re: Some open issues

[Holger Freyther]

On Saturday 18 October 2008 11:03:29 Richard Purdie wrote:
> Hi Guys,
>
> I have a few issues with the way certain things have happened recently.
>
> 1. The FILE_PR change.
>
> This was mentioned in an email on Wednesday with the title "[oe] [RFC]
> Enable --hash-style=both for all recent gcc4 targets" at 9am. At 8pm we
> have "I decided to land now as PRs are changing all the time and keeping
> up with things is pretty hard...". This is not in keeping with the major
> changes policy we agreed by any stretch of the imagination.

Sorry about that.


> This change breaks compatibility with everyone's overlays and creates
> the nightmare scenario of external OE "branches" being forced into the
> change or forever being unable to sync (Openmoko and Poky spring to
> mind).

Yes, this creates an extra burden for the people that need to keep that in 
sync.


> What is most annoying is that given a bit longer I think we could have
> done something that would have meant this was unnecessary, specifically
> inserted the revision into the package at package_*.bbclass time where
> we can manipulated PR as needed. This combined with a staging ABI change
> would have been all that was needed. If staging ABI isn't enough, we can
> insert the modified PR into STAMPS instead of the real PR or some other
> magic. My point is that there are better options than FILE_PR, it just
> needs some thought. The fact the testing branch had so many merge issues
> should have meant a better idea was sought, not that is should be
> committed ASAP.

I don't agree with this. This means package names will not match with the on 
dir directory name. For me this was the strongest argument against doing it 
automatically in the packaging bbclass. I really don't like the idea that if 
I bump the DISTRO_PR the existing build directories will be recycled and 
packages will be rebuild in there. That mostly defeats the purpose of a 
deterministic tool. And I really don't like the idea of adding more python 
black magic that is mangling PR... :)



>
> 2. The Git conversion including the BKCVS data.
>
> I'd made it quite clear this should have been a tree graft and this
> wasn't done so we're now stuck with broken history :(. This is pretty
> frustrating since I'd repeatedly said not to include it and went to the
> effort of gathering my conversion data and sending it to Jan who then
> didn't realise what I meant by graft (though no fault of his own).

Should we start over from scratch and rebase what we have already added? Will 
this be the _last_ time we consider it?



> 4. Bitbake changes
>
> These should go to the bitbake list as well as the OE list and should be
> discussed. I've raised issues with patches which have been ignored and
> these patches have now just need committed. I'm not happy about the
> process that was used :(. I know people have various commitments but we
> need to try and stick to some kind of process for this kind of thing.

???? Things have been ignored... three times...


>
>
> Where from here?
>
> I'd actually like to a strong line on this and suggest we revert the
> FILE_PR change since its badly thought out and also that we consider
> redoing the git conversion ASAP and the replaying the recent commits
> before its too late to get rid of the corruption in there. This is
> probably going to have to go to a core team vote since its a pretty big
> change to suggest but opinions are welcome.

Do whatever you want. I think FILE_PR is the right thing to do, feel free to 
invest the time to redo the conversion, the scripts are public and Jan might 
be able to upload the "source" for git-fast-import, please also coordinate 
the git-rebase (we will lose merges but that seems okay, but I did merge the 
FILE_PR changes on purpose for ease of reverting), make sure no one is 
merging old history with new history (this was not a big problem with the 
trial but will be one now)...

Re: Some open issues

[Richard Purdie]

On Sat, 2008-10-18 at 14:32 +0200, Holger Freyther wrote:
> On Saturday 18 October 2008 11:03:29 Richard Purdie wrote:
> > What is most annoying is that given a bit longer I think we could have
> > done something that would have meant this was unnecessary, specifically
> > inserted the revision into the package at package_*.bbclass time where
> > we can manipulated PR as needed. This combined with a staging ABI change
> > would have been all that was needed. If staging ABI isn't enough, we can
> > insert the modified PR into STAMPS instead of the real PR or some other
> > magic. My point is that there are better options than FILE_PR, it just
> > needs some thought. The fact the testing branch had so many merge issues
> > should have meant a better idea was sought, not that is should be
> > committed ASAP.
> 
> I don't agree with this. This means package names will not match with the on 
> dir directory name. For me this was the strongest argument against doing it 
> automatically in the packaging bbclass. I really don't like the idea that if 
> I bump the DISTRO_PR the existing build directories will be recycled and 
> packages will be rebuild in there. That mostly defeats the purpose of a 
> deterministic tool. And I really don't like the idea of adding more python 
> black magic that is mangling PR... :)

We'll take this one to a separate thread.

> > 2. The Git conversion including the BKCVS data.
> >
> > I'd made it quite clear this should have been a tree graft and this
> > wasn't done so we're now stuck with broken history :(. This is pretty
> > frustrating since I'd repeatedly said not to include it and went to the
> > effort of gathering my conversion data and sending it to Jan who then
> > didn't realise what I meant by graft (though no fault of his own).
> 
> Should we start over from scratch and rebase what we have already added? Will 
> this be the _last_ time we consider it?

As far as I'm concerned, yes. This is a one off problem.

> > 4. Bitbake changes
> >
> > These should go to the bitbake list as well as the OE list and should be
> > discussed. I've raised issues with patches which have been ignored and
> > these patches have now just need committed. I'm not happy about the
> > process that was used :(. I know people have various commitments but we
> > need to try and stick to some kind of process for this kind of thing.
> 
> ???? Things have been ignored... three times...

I tried for a couple of *months* after the first versions of those
patches were published to talk to you on irc about them and you were
always too busy. I did reply on the mailing list about them but we never
got a discussion going about things. You at least *knew* I wanted too
discuss them.

To tell me I ignored them is just plain wrong.

> > Where from here?
> >
> > I'd actually like to a strong line on this and suggest we revert the
> > FILE_PR change since its badly thought out and also that we consider
> > redoing the git conversion ASAP and the replaying the recent commits
> > before its too late to get rid of the corruption in there. This is
> > probably going to have to go to a core team vote since its a pretty big
> > change to suggest but opinions are welcome.
> 
> Do whatever you want. 

This is the attitude that gets us into trouble. Its not what I want, its
a question of what is best for OE.

> I think FILE_PR is the right thing to do, feel free to 
> invest the time to redo the conversion, the scripts are public and Jan might 
> be able to upload the "source" for git-fast-import, please also coordinate 
> the git-rebase (we will lose merges but that seems okay, but I did merge the 
> FILE_PR changes on purpose for ease of reverting), make sure no one is 
> merging old history with new history (this was not a big problem with the 
> trial but will be one now)...

Right, if we do this, we need to do this carefully. If it needs to be me
that finds the time to do this, so be it, I'm the one complaining.
Obviously some tips on how to make the conversion would be good but I
will do it from scratch and blind if that pain makes you feel better.

Cheers,

Richard

Re: Some open issues

[Phil Blundell]

On Sat, 2008-10-18 at 10:03 +0100, Richard Purdie wrote:
> I'd actually like to a strong line on this and suggest
> [...] that we consider
> redoing the git conversion ASAP and the replaying the recent commits
> before its too late to get rid of the corruption in there. This is
> probably going to have to go to a core team vote since its a pretty big
> change to suggest but opinions are welcome.

Obviously I'm not a core team member, but seeing as you asked for
opinions... :-}

I do struggle to see why this is such a pressing issue right now.  As I
understand it, the "corruption" in question is basically mangled data
from the BK history.  This doesn't seem like it is obviously going to
get any worse over time, nor does it seem like the contagion is likely
to spread to other data, nor is the corruption going to be visible at
the tip of any recent checkout.  Also, we have now had several days of
activity on the live git tree, and we would presumably want to preserve
those changesets on any putative re-import.

Assuming the above to be the case, I don't fully understand either:

i) why this is a terribly big issue in the first place (i.e. what the
real-world impact of the corruption is going to be); or

ii) why, if we don't act now, it might be "too late" to solve the
problem in the future

Of course, it's entirely possible that my assumptions above are wrong.
I'd be interested to know the real situation, anyway.

p.

Re: Some open issues

[Richard Purdie]

On Sat, 2008-10-18 at 15:59 +0100, Phil Blundell wrote:
> I do struggle to see why this is such a pressing issue right now.  As I
> understand it, the "corruption" in question is basically mangled data
> from the BK history.  This doesn't seem like it is obviously going to
> get any worse over time, nor does it seem like the contagion is likely
> to spread to other data, nor is the corruption going to be visible at
> the tip of any recent checkout.  Also, we have now had several days of
> activity on the live git tree, and we would presumably want to preserve
> those changesets on any putative re-import.
> 
> Assuming the above to be the case, I don't fully understand either:
> 
> i) why this is a terribly big issue in the first place (i.e. what the
> real-world impact of the corruption is going to be); or

There is no impact on the current metadata, just the history. It depends
how much value we place on that.

At present we can't at some future date add in a good version of the
BKCVS data as easily as we could if it had been done by a tree graft
originally. The data there at the moment is pretty useless for actually
finding history (I know since I've tried to use it).

> ii) why, if we don't act now, it might be "too late" to solve the
> problem in the future

As you point out, we have a certain number of commits already. At the
moment we can probably deal with these but the longer we leave it, the
more commits and the more I'd not want to rebase things. We also have
the FILE_PR issue to consider and we need to revert that commit by some
means and find a better solution (I think even zecke agrees with that
but we'll discuss that in another thread).

Cheers,

Richard

Re: Some open issues

[Tom Rini]

On Sat, Oct 18, 2008 at 04:37:20PM +0100, Richard Purdie wrote:
> On Sat, 2008-10-18 at 15:59 +0100, Phil Blundell wrote:
[snip]
> > i) why this is a terribly big issue in the first place (i.e. what the
> > real-world impact of the corruption is going to be); or
> 
> There is no impact on the current metadata, just the history. It depends
> how much value we place on that.

And with 'git bisect' good history is very valuable for fixing
long-standing bugs.  Or at least can be.

-- 
Tom Rini

Re: Some open issues

[Koen Kooi]

On 18-10-2008 11:03, Richard Purdie wrote:
> Hi Guys,
>
> I have a few issues with the way certain things have happened recently.
>
> 1. The FILE_PR change.
>
> This was mentioned in an email on Wednesday with the title "[oe] [RFC]
> Enable --hash-style=both for all recent gcc4 targets" at 9am. At 8pm we
> have "I decided to land now as PRs are changing all the time and keeping
> up with things is pretty hard...". This is not in keeping with the major
> changes policy we agreed by any stretch of the imagination.
>
> This change breaks compatibility with everyone's overlays and creates
> the nightmare scenario of external OE "branches" being forced into the
> change or forever being unable to sync (Openmoko and Poky spring to
> mind).
>
> What is most annoying is that given a bit longer I think we could have
> done something that would have meant this was unnecessary, specifically
> inserted the revision into the package at package_*.bbclass time where
> we can manipulated PR as needed. This combined with a staging ABI change
> would have been all that was needed. If staging ABI isn't enough, we can
> insert the modified PR into STAMPS instead of the real PR or some other
> magic. My point is that there are better options than FILE_PR, it just
> needs some thought. The fact the testing branch had so many merge issues
> should have meant a better idea was sought, not that is should be
> committed ASAP.
>
> 2. The Git conversion including the BKCVS data.
>
> I'd made it quite clear this should have been a tree graft and this
> wasn't done so we're now stuck with broken history :(. This is pretty
> frustrating since I'd repeatedly said not to include it and went to the
> effort of gathering my conversion data and sending it to Jan who then
> didn't realise what I meant by graft (though no fault of his own).

I have no strong opinion on the BK stuff, but it would be nice to have 
the history in the same repo.

> 3. Merging Bitbake into OE
>
> People are saying things like "Might be a chance to reconsider
> maintaining BitBake in the OE repository.". Could people please talk to
> the bitbake maintainer about things like this *before* encouraging it in
> public. If we need a release lets make one (which seems to be the real
> problem).

I do have a strong opinion on that :) Bitbake should be kept out of the 
.dev branch. I can see how things like the new stable branch *might* 
include a copy, but let's cross that bridge when we get there.

> 4. Bitbake changes
>
> These should go to the bitbake list as well as the OE list and should be
> discussed. I've raised issues with patches which have been ignored and
> these patches have now just need committed. I'm not happy about the
> process that was used :(. I know people have various commitments but we
> need to try and stick to some kind of process for this kind of thing.

I know very little about bitbake and don't contribute to it, so no 
opinion there.

> Where from here?
>
> I'd actually like to a strong line on this and suggest we revert the
> FILE_PR change since its badly thought out and also that we consider
> redoing the git conversion ASAP and the replaying the recent commits
> before its too late to get rid of the corruption in there. This is
> probably going to have to go to a core team vote since its a pretty big
> change to suggest but opinions are welcome.

<distro hat> The most important thing is the in the end distros can have 
a way to increase a global PR</distro hat>

<oe hat>I agree with your proposal</oe hat>

regards,

Koen

Re: Some open issues

[Richard Purdie]

On Sat, 2008-10-18 at 16:37 +0100, Richard Purdie wrote:
> On Sat, 2008-10-18 at 15:59 +0100, Phil Blundell wrote:
> > i) why this is a terribly big issue in the first place (i.e. what the
> > real-world impact of the corruption is going to be); or
> 
> There is no impact on the current metadata, just the history. It depends
> how much value we place on that.
> 
> At present we can't at some future date add in a good version of the
> BKCVS data as easily as we could if it had been done by a tree graft
> originally. The data there at the moment is pretty useless for actually
> finding history (I know since I've tried to use it).
> 
> > ii) why, if we don't act now, it might be "too late" to solve the
> > problem in the future
> 
> As you point out, we have a certain number of commits already. At the
> moment we can probably deal with these but the longer we leave it, the
> more commits and the more I'd not want to rebase things. We also have
> the FILE_PR issue to consider and we need to revert that commit by some
> means and find a better solution (I think even zecke agrees with that
> but we'll discuss that in another thread).

I've run some tests and to fix things, someone needs to make a checkout
and then run:

git filter-branch --parent-filter  'test $GIT_COMMIT = e3d1546d2c1ec9fd00af7780ba41250496153b15 && echo || cat' -- --all

which will remove the bitkeeper data from the history. We can then graft
it back in with:

echo "b97239969db33bf4be1b5f5807eae4d1d2987ca1 c8e5702127e507e82e6f68a4b8c546803accea9d fc6dcc1f1f0eabd8f2a465f219ffcd2554deec04" > .git/info/grafts

(where b97239969db33bf4be1b5f5807eae4d1d2987ca1 is the revision
e3d1546d2c1ec9fd00af7780ba41250496153b15 became. Just to illustrate how
broken the bkcvs conversion is, look at the diff between the git and
bkcvs history:

git diff fc6dcc1f1f0eabd8f2a465f219ffcd2554deec04  b97239969db33bf4be1b5f5807eae4d1d2987ca1

The reason this is huge is that we seem to have ton of zero length
files. Given its so totally broken I'm keen to do the above so we need a
plan. I propose:

a) repo goes read only
b) checkout is made
c) conversion is run
d) sanity checks are made
e) new revisions are pushed
f) graft is added to the main OE repo
g) commits based on the old history are blocked
h) repo is made writable again
i) people are advised to rebase any unpushed changes

Is anyone strongly against this? If not, when should we do this?

Richard

Re: Some open issues

[Michael 'Mickey' Lauer]

Am Sunday 19 October 2008 20:06:50 schrieb Richard Purdie:
> The reason this is huge is that we seem to have ton of zero length
> files. Given its so totally broken I'm keen to do the above so we need a
> plan. I propose:
>
> a) repo goes read only
> b) checkout is made
> c) conversion is run
> d) sanity checks are made
> e) new revisions are pushed
> f) graft is added to the main OE repo
> g) commits based on the old history are blocked
> h) repo is made writable again
> i) people are advised to rebase any unpushed changes
>
> Is anyone strongly against this? If not, when should we do this?

No objection. Please synchronize with Jan Lübbe, he has lots of experience 
with the conversion. Feel free to chose the time/date when you (both) have 
resources to do that. Please announce a day before though.

Thanks,

Mickey.