[RFC] Patches exchange is bad?

[RFC] Patches exchange is bad?

[Marco Costalba]

Hi,

I would like to ask the list about something I'am thinking about,
and I'am not sure it's a good idea.

Suppose a possible scenario involves using a couple of git archives, one 
for releases and stable code, say MAIN, and one for experimental stuff  
or new development, say HEAD.

Suppose there is stuff in HEAD you don't want merged in MAIN, more,
you need to update MAIN with only a subset of patches in HEAD, peraphs 
in different order. Or simply, you are not interested to see the history 
of the HEAD tree when looking MAIN. All this points could keep you 
from merging.

I have mocked up a very simple and very stupid 'drag and drop' function.
Basically you drag some selected revs in another instance of qgit, open on 
a different archive.

What happpens in the background is that git-format-patch-script is feeded 
with the selected revs and a bunch of temporary patch files are created, 
then git-applymbox (re)creates the corresponding commits in the destination 
archive.

It is just a very thin layer above the two git scripts, the only extra work 
is the cleaning up of some info that git-format-patch-script automatically adds, 
so that the new commits look like the originals (i.e. same header and description).

I've built-up this thing mainly because I found it useful for me, but I have  
some concerns that this is the correct way to go, the way git it's meant to be used.

If there is some interest for this I can push something on SF, after a due polishing, 
but if it is deeply broken I prefer discard and eventually switch to a more consistent
workflow.

What do you think?

	Marco


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [RFC] Patches exchange is bad?

[Johannes Schindelin]

Hi,

On Tue, 16 Aug 2005, Marco Costalba wrote:

> Suppose a possible scenario involves using a couple of git archives, one 
> for releases and stable code, say MAIN, and one for experimental stuff  
> or new development, say HEAD.
> 
> Suppose there is stuff in HEAD you don't want merged in MAIN, more,
> you need to update MAIN with only a subset of patches in HEAD, peraphs 
> in different order. Or simply, you are not interested to see the history 
> of the HEAD tree when looking MAIN. All this points could keep you 
> from merging.

This looks like a good description of cherry picking. See StGIT (and to a 
certain extent, git-cherry) for that.

Ciao,
Dscho

Re: [RFC] Patches exchange is bad?

[Junio C Hamano]

Marco Costalba <mcostalba@yahoo.it> writes:

> It is just a very thin layer above the two git scripts, the only extra work 
> is the cleaning up of some info that git-format-patch-script automatically adds, 
> so that the new commits look like the originals (i.e. same header and description).

> What do you think?

All good.  The existing tools are designed to work well in the
following workflow:

    $ git format-patch --mbox master..pu
    $ cat 000[234]-*.txt >to-apply
    $ git checkout master
    $ git applymbox to-apply
    $ mv .git/refs/heads/pu .git/refs/heads/old-pu
    $ git checkout -b pu master
    $ git applymbox 0001-*.txt

to advance "master" with selected commits, possibly reordered,
from the "pu" branch, and rebase the "pu" branch using the
remaining commits.

I would like to know a bit about "git format-patch" adding extra
info that you needed to get rid of.  It shouldn't be necessary.

Re: [RFC] Patches exchange is bad?

[Martin Langhoff]

On 8/17/05, Marco Costalba <mcostalba@yahoo.it> wrote:
> What do you think?

>From what I understand, you'll want the StGIT infrastructure. If you
use git/cogito, there is an underlying  assumption that you'll want
all the patches merged across, and a simple cg-update will bring in
all the pending stuff.

cheers,


martin

Re: [RFC] Patches exchange is bad?

[Catalin Marinas]

Marco Costalba <mcostalba@yahoo.it> wrote:
> Suppose a possible scenario involves using a couple of git archives, one 
> for releases and stable code, say MAIN, and one for experimental stuff  
> or new development, say HEAD.
>
> Suppose there is stuff in HEAD you don't want merged in MAIN, more,
> you need to update MAIN with only a subset of patches in HEAD, peraphs 
> in different order. Or simply, you are not interested to see the history 
> of the HEAD tree when looking MAIN. All this points could keep you 
> from merging.

As others already recommended StGIT, I will just add a simple usage
scenario (I do this with my StGIT repository).

The MAIN/stable repository (or branch) is only managed with GIT, not
StGIT. The HEAD one is managed with StGIT (only, you can use 'stg
clone'). You can create patches, modify them etc. (I updated the
README in the latest snapshot and it contains some kind of tutorial).

Once you want a subset of these patches merged into MAIN, just pop
everything from the stack and only push those you want merged, in the
order you want (if there are some dependencies, the push will fail and
you can correct them or the order). When you are happy with the
patches pushed on the stack, just do a 'git pull <HEAD>' in the MAIN
repository. After this, doing a 'stg pull <MAIN>' in the HEAD one will
mark the patches already integrated into MAIN as empty and you can
safely remove them ('stg clean' does this automatically).

This way I found StGIT useful for maintainers as well, not only for
contributors.

-- 
Catalin

Re: [RFC] Patches exchange is bad?

[Marco Costalba]

Junio C Hamano wrote:

>
>I would like to know a bit about "git format-patch" adding extra
>info that you needed to get rid of.  It shouldn't be necessary.
>

As example, in the rev d5a63b99835017d2638e55a7e34a35a3c1e80f1f from git
the original subject is:

'   Alternate object pool mechanism updates.'

while, after the round-trip git-format-patch + git applymbox I have

'   [PATCH] Alternate object pool mechanism updates.'

The extra '[PATCH]' in the subject was introduced by git-format-patch --mbox.


Perpahs I have made something wrong.


Marco



		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: [RFC] Patches exchange is bad?

[Marco Costalba]

Martin Langhoff wrote:

>>From what I understand, you'll want the StGIT infrastructure. If you
>use git/cogito, there is an underlying  assumption that you'll want
>all the patches merged across, and a simple cg-update will bring in
>all the pending stuff.
>

My concerns are both metodologicals and practical:

1) Method: To use the 'free patching workflow' on git is something foreseen in
git design, something coherent with the fork + develop + merge cycle that it
seems, at least to me, THE way git is meant to be used. Or it is stretching 
the possibility of the tool to something technically allowed but not suggested.

2) Practical: The round trip git-format-patch + git-applymbox is the logical and
natural way to reach this goal or, also in this case, I intend to stretch some tools, 
designed for one thing, for something else?


About StGIT, I agree it is the right tool, designed for this kind of problems. But,
peraphs because I don't know it very much, I still can't figure out how to integrate 
StGit in a git GUI, like qgit is, so to have an unified and friendly view of a git 
archive and a patches stack.


Marco



__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [RFC] Patches exchange is bad?

[Daniel Barkalow]

On Tue, 16 Aug 2005, Marco Costalba wrote:

> Martin Langhoff wrote:
>
> >>From what I understand, you'll want the StGIT infrastructure. If you
> >use git/cogito, there is an underlying  assumption that you'll want
> >all the patches merged across, and a simple cg-update will bring in
> >all the pending stuff.
> >
>
> My concerns are both metodologicals and practical:
>
> 1) Method: To use the 'free patching workflow' on git is something foreseen in
> git design, something coherent with the fork + develop + merge cycle that it
> seems, at least to me, THE way git is meant to be used. Or it is stretching
> the possibility of the tool to something technically allowed but not suggested.

Patches are definitely meant to be part of how git is used; they are the
primary way of getting clean history out of messy history (that is, saving
a content change while discarding some history that isn't applicable).
There's relatively little support in git itself, but that's because the
point is to go outside the system's tracking. There have been various
discussions of more explicit support, and nobody's been able to come up
with a need.

> 2) Practical: The round trip git-format-patch + git-applymbox is the logical and
> natural way to reach this goal or, also in this case, I intend to stretch some tools,
> designed for one thing, for something else?

I'd guess that git-diff-tree + git-apply (without the rest of the
scripting) would be more effective when you're not doing anything with the
intermediate files, since it saves doing a bunch of formatting and
parsing.

	-Daniel
*This .sig left intentionally blank*

Re: [RFC] Patches exchange is bad?

[Marco Costalba]

Daniel Barkalow wrote:

>>2) Practical: The round trip git-format-patch + git-applymbox is the logical and
>>natural way to reach this goal or, also in this case, I intend to stretch some tools,
>>designed for one thing, for something else?
>
>
>I'd guess that git-diff-tree + git-apply (without the rest of the
>scripting) would be more effective when you're not doing anything with the
>intermediate files, since it saves doing a bunch of formatting and
>parsing.
>
>
It would be surely better, but I need to import also the original subject and
description. I can use git-diff-tree --pretty -p for the patch creation, but this
format is not compatible with git-apply* scripts, and the git command git-apply does
not import subject + description info.

Of course I can feed proper subject and description to git-commit but I would like 
to find something less intrusive as possible, ie. one command for patch export, one 
command for patch import.

Thanks
Marco


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [RFC] Patches exchange is bad?

[Martin Langhoff]

On 8/17/05, Marco Costalba <mcostalba@yahoo.it> wrote:
> Of course I can feed proper subject and description to git-commit but I would like
> to find something less intrusive

I don't know if it helps, but I think that StGIT is what you are
looking for, not only because you have more tools to deal with
patches, but also because patches that are in the 'stack' are actually
really malleable. You can edit and reedit the patch w its commit msg
and all, commit it to the stack, and reedit it again later. It only
becomes immutable when you commit to the underlying git repo.

cheers,


martin

Re: [RFC] Patches exchange is bad?

[Marco Costalba]

Catalin Marinas wrote:

>
>Once you want a subset of these patches merged into MAIN, just pop
>everything from the stack and only push those you want merged, in the
>order you want (if there are some dependencies, the push will fail and
>you can correct them or the order). When you are happy with the
>patches pushed on the stack, just do a 'git pull <HEAD>' in the MAIN
>repository. After this, doing a 'stg pull <MAIN>' in the HEAD one will
>mark the patches already integrated into MAIN as empty and you can
>safely remove them ('stg clean' does this automatically).
>
>This way I found StGIT useful for maintainers as well, not only for
>contributors.
>

Sorry if the answer is silly, but I still don't know well StGIT .

What you describe it's an asymmetrical or one way scenario, new code 
goes always from HEAD to MAIN. But how is the workflow if:

1) There is more then one contributor feeding MAIN and you need to update 
the StGIT patch stack from MAIN.

2) You made something terribly wrong with HEAD (I don't know what can be 
'terribly wrong') and you need to recreate a clean base from MAIN.

In this cases, if I understand correctly, you need to clone a new StGIT archive from 
MAIN and push the interesting stuff from old/broken HEAD.

Or you can always merge back pulling from MAIN as in case of two pure git archives?


Thanks
Marco



		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: [RFC] Patches exchange is bad?

[Catalin Marinas]

On Wed, 2005-08-17 at 10:35 -0700, Marco Costalba wrote:
> Sorry if the answer is silly, but I still don't know well StGIT .

It's probably because there is no document really explaining the
concepts. The Quilt documentation would be a good starting point since
StGIT uses the same ideas but implemented on top if GIT instead of a
series of GNU diff files.

> What you describe it's an asymmetrical or one way scenario, new code 
> goes always from HEAD to MAIN. But how is the workflow if:

It is asymmetrical since HEAD uses StGIT and MAIN uses plain GIT but it
is a two way data flow: 'git pull HEAD' in MAIN and 'stg pull MAIN' in
HEAD.

> 1) There is more then one contributor feeding MAIN and you need to update 
> the StGIT patch stack from MAIN.

The base of the StGIT stack in the HEAD repository (branch) should
always be the head of the MAIN repository. You update it via the 'stg
pull' command, which takes care of updating the patches so that they are
seen as individual commits on top of the base.

That's how you would normally do development on Linux using StGIT -
clone the mainline kernel, create patches in your StGIT tree and submit
them either via e-mail or ask the gatekeeper to pull directly from your
tree (assuming that you only push those patches that need to be merged).
Doing a 'stg pull' would retrieve the latest changes from the mainline
kernel (including the changes made by your patches which were merged
upstream).

> 2) You made something terribly wrong with HEAD (I don't know what can be 
> 'terribly wrong') and you need to recreate a clean base from MAIN.
> 
> In this cases, if I understand correctly, you need to clone a new StGIT archive from 
> MAIN and push the interesting stuff from old/broken HEAD.

This is not clear for me. What do you mean by 'terribly wrong'? Broken
HEAD because of a bug in StGIT or GIT? Quite unlikely but in this case
the repository would be corrupt. I would recommend periodically doing a
'stg export' to keep the series of patches in GNU diff files.

If you refer to a patch which breaks the code, you can simply pop it
from the stack and even delete it with 'stg delete'. Popping it removes
the changes it makes to HEAD and the corresponding commit won't be seen
with 'git log'. You don't need to clone a new repository since StGIT
allows you to choose which of your patches (commits) are included in
HEAD (via 'stg push' and visible with 'stg applied').

> Or you can always merge back pulling from MAIN as in case of two pure git archives?

The base of the StGIT stack is identical to MAIN. Doing a 'stg pop -a'
makes the HEAD and MAIN identical.

Please let me know if this needs further clarification.

-- 
Catalin

Re: [RFC] Patches exchange is bad?

[Johannes Schindelin]

Hi Catalin,

maybe it is time for a quick run through the typical jobs you do with 
StGIT, much like what Jeff sent the other day?

Ciao,
Dscho

Re: [RFC] Patches exchange is bad?

[Catalin Marinas]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> maybe it is time for a quick run through the typical jobs you do with 
> StGIT, much like what Jeff sent the other day?

I hope I will find some time this weekend and write some tutorials on
an StGIT wiki.

-- 
Catalin

Re: [RFC] Patches exchange is bad?

[Marco Costalba]

Marco Costalba wrote:

>>
>>This way I found StGIT useful for maintainers as well, not only for
>>contributors.
>>
>
>Sorry if the answer is silly, but I still don't know well StGIT .
>

'question' not 'answer' !!!!

I don't now if the question is silly, but the typo it is for sure!


Sorry
Marco


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [RFC] Patches exchange is bad?

[Marco Costalba]

Catalin Marinas wrote:

>
>The base of the StGIT stack in the HEAD repository (branch) should
>always be the head of the MAIN repository. 

That's the point I missed, now it's more clear.

>
>That's how you would normally do development on Linux using StGIT -
>clone the mainline kernel, create patches in your StGIT tree and submit
>them either via e-mail or ask the gatekeeper to pull directly from your
>tree (assuming that you only push those patches that need to be merged).
>Doing a 'stg pull' would retrieve the latest changes from the mainline
>kernel (including the changes made by your patches which were merged
>upstream).
>

If I uderstand correctly you never commit patches from StGIT stack directly
in your base git repository, in this example git HEAD, but you always round 
trip to MAIN. 

Then you don't have two git repository: HEAD and MAIN

Infact there is only one git repository, MAIN, cloned on your box and called HEAD and 
with a StGIT stack added on top.

>
>Please let me know if this needs further clarification.
>

I think so. 
Please, reading my notes above tell me if I thinking right ;-)

Marco


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [RFC] Patches exchange is bad?

[Catalin Marinas]

Marco Costalba <mcostalba@yahoo.it> wrote:
> Catalin Marinas wrote:
>>That's how you would normally do development on Linux using StGIT -
>>clone the mainline kernel, create patches in your StGIT tree and submit
>>them either via e-mail or ask the gatekeeper to pull directly from your
>>tree (assuming that you only push those patches that need to be merged).
>>Doing a 'stg pull' would retrieve the latest changes from the mainline
>>kernel (including the changes made by your patches which were merged
>>upstream).
>
> If I uderstand correctly you never commit patches from StGIT stack directly
> in your base git repository, in this example git HEAD, but you always round 
> trip to MAIN.

One usually doesn't maintain MAIN. That's an example for the Linux
kernel development where you can't control what get merged into MAIN.

There is a bit of confusion here since you said in a previous e-mail
that more people can commit to the stable branch. In this case, you
would need a separate repository for stable with a maintainer pulling
changes from others.

> Then you don't have two git repository: HEAD and MAIN
>
> Infact there is only one git repository, MAIN, cloned on your box
> and called HEAD and with a StGIT stack added on top.

The StGIT usage idea is that you only know what patches you have on
top of a main repository. Since you expect your patches to be merged
upstream or just updated every time the main repository changes, it
might not make sense to commit the patches onto the base.

If you want to keep a stable branch (and only you comitting to it), I
would recommend having separate branches for stable and development in
the same tree (and avoid having separate trees) rather than directly
modifying the base. StGIT doesn't support this directly (didn't think
anyone would ask) but you can do the following steps:

1. Use 'stg clone <remote repository>' to create the initial
   clone. The 'master' branch will be used for development. You can
   use 'git-init-db' and 'stg init' if you don't follow a remote
   repository

2. 'git checkout -b stable' will create the stable branch. This will
   initially be the same as the master one since there are not patches
   applied to the master stack

3. switch back to the devel branch with 'git checkout master' and do
   the normal StGIT development (create patches, commit them etc.). If
   you are tracking a remote respository, you could pull the latest
   changes with 'stg pull'

4. Once you have a set of patches applied and your stack is stable
   (some patches can be unapplied, they don't affect the head of the
   development branch), you can switch to the stable branch with 'stg
   checkout stable' and merge the changes in master with
   'git-resolve-script'.

   I think it might make more sense to simply do a 'cp
   .git/refs/heads/master .git/refs/heads/stable' instead of the
   previous paragraph since the stable branch doesn't have additional
   data from what's in master. The remote repository tracking is done
   via master.

5. You can advance the base of the stack to what's in stable with an
   'stg pull stable' but it's not clear for me whether this makes any
   sense since you lose the StGIT patches already added to stable and
   you will might later get conflicts when pulling the changes from
   the remote repository you are tracking.

For the 2, 3 and 4 steps above I will add a 'snapshot' command to
StGIT but I'm not sure why one would want step 5 since this would mean
that the devel branch needs to track both the remote repository and
the stable branch.

Some questions: do you track a remote repository? Do you need a stable
repository or branch where more people can update? If you track the
remote repository, do you want the latest remote changes merged into
the stable branch directly (i.e. is the stable branch tracking the
remote repository)?

I need a bit more clarification about your work flow.

-- 
Catalin

Re: [RFC] Patches exchange is bad?

[Marco Costalba]

Catalin Marinas wrote:

>>
>>If I uderstand correctly you never commit patches from StGIT stack directly
>>in your base git repository, in this example git HEAD, but you always round 
>>trip to MAIN.
>
When I say 'you always round trip to MAIN', I mean you send patches 
upstream and someone commits to MAIN, then you pull from MAIN.

Sorry if it was not clear.

>
>One usually doesn't maintain MAIN. That's an example for the Linux
>kernel development where you can't control what get merged into MAIN.
>
>There is a bit of confusion here since you said in a previous e-mail
>that more people can commit to the stable branch. In this case, you
>would need a separate repository for stable with a maintainer pulling
>changes from others.
>

When I said 'more people can commit to the stable branch' I meant more 
people sends patches to a mantainer that commits the patches in a stable branch.

Sorry, peraphs also this was not clearly expressed.


>>Then you don't have two git repository: HEAD and MAIN
>>
>>Infact there is only one git repository, MAIN, cloned on your box
>>and called HEAD and with a StGIT stack added on top.
>
>
>The StGIT usage idea is that you only know what patches you have on
>top of a main repository. Since you expect your patches to be merged
>upstream or just updated every time the main repository changes, it
>might not make sense to commit the patches onto the base.
>

Yes, you better explained what I was badly trying to say before.


>
>I need a bit more clarification about your work flow.
>

I was thinking at two different kind of workflow, one were you are tracking a 
remote repository ( Linux kernel project like ) and one as single developer with 
both stable and develop lines ( qgit or StGIT ;-) projects like ). 

You clarified me for both cases.

Thanks
Marco


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [RFC] Patches exchange is bad?

[Catalin Marinas]

Marco Costalba <mcostalba@yahoo.it> wrote:
> I was thinking at two different kind of workflow, one were you are
> tracking a remote repository ( Linux kernel project like ) and one
> as single developer with both stable and develop lines ( qgit or
> StGIT ;-) projects like ).

There is another option for the 2nd case, where you maintain both both
stable and development. You could use 'git push' to the stable
repository (actually, more like a mirror) and make the
StGIT-maintained repository track the changes from the stable one and
just update it with 'stg pull'

-- 
Catalin