how to filter a pull

how to filter a pull

[marc.zonzon]

I'm a git beginner (and an old user of cvs, and more recently of
subversion, and occasionally of arch)

I'm very pleased with the decentralized character of git and the ease
to manage branches. But I have a usual problem that I don't know how
to solve properly in git.

I have projects that draw some parts from two or three other
projects. But it is usually some small part, that are included, and
patched in my project. I want to follow the development of these fellow
projects.

My problem is that I can of course get a branch to host a copy of the
project (if the are under git I can clone and pull, if not I use the
native scm to import and commit in the branch). But now I cannot merge
in my development branch as I include only a small part. 

I found no way to register that I copy these part. The only one I can think of,
is to have a script to extract a sub branch  with only the appropriate
part and then pull from it (or push to my project). 
i.e. I pull from the project (if git) or update in cvs, or ..., then I
filter to extract the appropiate part, then push to my development branch.

But I suppose there are a lot of cleaner way to do it. And moreover
I'm quite sure that developers have met the same problem, and have solved it.

Sorry to ask such  a stupid question but 
(1) git naming itself stupid content tracker encourage dumb people
like me to ask stupid questions .
(2) I tried to RTFM, but could not find the appropriate page.

Thank you for any hint.

Re: how to filter a pull

[Peter Baumann]

On Thu, May 03, 2007 at 03:17:05PM +0200, marc.zonzon@gmail.com wrote:
> I'm a git beginner (and an old user of cvs, and more recently of
> subversion, and occasionally of arch)
> 
> I'm very pleased with the decentralized character of git and the ease
> to manage branches. But I have a usual problem that I don't know how
> to solve properly in git.
> 
> I have projects that draw some parts from two or three other
> projects. But it is usually some small part, that are included, and
> patched in my project. I want to follow the development of these fellow
> projects.
> 
> My problem is that I can of course get a branch to host a copy of the
> project (if the are under git I can clone and pull, if not I use the
> native scm to import and commit in the branch). But now I cannot merge
> in my development branch as I include only a small part. 
> 
> I found no way to register that I copy these part. The only one I can think of,
> is to have a script to extract a sub branch  with only the appropriate
> part and then pull from it (or push to my project). 
> i.e. I pull from the project (if git) or update in cvs, or ..., then I
> filter to extract the appropiate part, then push to my development branch.
> 
> But I suppose there are a lot of cleaner way to do it. And moreover
> I'm quite sure that developers have met the same problem, and have solved it.
> 
> Sorry to ask such  a stupid question but 
> (1) git naming itself stupid content tracker encourage dumb people
> like me to ask stupid questions .
> (2) I tried to RTFM, but could not find the appropriate page.
> 
> Thank you for any hint.

Perhaps the newly added merge strategy subtree [1] could work. Or the
just added subproject support, but this is very new and only the lowlevel machinery
is implemented right now.

-Peter

[1]: http://git.kernel.org/?p=git/git.git;a=commit;h=68faf68938ee943fc251c702f2027e4dfda354db

Re: how to filter a pull

[Junio C Hamano]

Peter Baumann <waste.manager@gmx.de> writes:

> On Thu, May 03, 2007 at 03:17:05PM +0200, marc.zonzon@gmail.com wrote:

>> I have projects that draw some parts from two or three other
>> projects. But it is usually some small part, that are included, and
>> patched in my project. I want to follow the development of these fellow
>> projects.
>> ...
> Perhaps the newly added merge strategy subtree [1] could work. Or the
> just added subproject support, but this is very new and only the lowlevel machinery
> is implemented right now.

I do not think subtree merge would work with this situation, as
it is only for cases that one side of the merge represents a
tree that is a subset of the tree of the other side of the
merge.  marc.zonzon's case does not fit the pattern.

If I were doing this 6 months from now, I would probably use
subproject to host the whole tree of other projects somewhere,
adjust the build procedure of the primary project to borrow the whole of these other
projects not just subtree -- or have appropriate symlinks in the
superproject that point into relevant subtrees in the
subprojects.

If I were doing this today, I would probably use separate
repositories, next to the primary project, to host the whole
tree of other projects, adjust the build procedure of the
primary project to borrow the whole of these other projects not
just subtree -- and/or have appropriate symlinks in the primary
project that point into relevant subtrees in the neighbouring
repositories that host these other projects.

Re: how to filter a pull

[marc zonzon]

On 5/3/07, Junio C Hamano <junkio@cox.net> wrote:

> If I were doing this 6 months from now, I would probably use
> subproject to host the whole tree of other projects somewhere,
> adjust the build procedure of the primary project to borrow the whole of these other
> projects not just subtree -- or have appropriate symlinks in the
> superproject that point into relevant subtrees in the
> subprojects.
>
> If I were doing this today, I would probably use separate
> repositories, next to the primary project, to host the whole
> tree of other projects, adjust the build procedure of the
> primary project to borrow the whole of these other projects not
> just subtree -- and/or have appropriate symlinks in the primary
> project that point into relevant subtrees in the neighbouring
> repositories that host these other projects.
>
Thank you for your explanations, I'm not sure to truly understand what
you propose, (I told you I'm a beginner and not amonq the  clever
ones!)

Do you mean that I have to do the patches I do on the fellow project
in the mirror directory, as to keep it as a link? It could be possible
when the project is managed by git (with the drawback of adding an
extra complexity to dereference the links in the repository when
archiving a release, and a lot of problem to export to a public
repository) but is not a possibility when the other project is managed
by an other scm.

I don't understand neither the first solution 'borrow the whole project'.
To better explain the problem I give a practical case.

I'm working to package software for openwrt arm-mipsel system. The
whole openwrt buildsystem is available under subversion, it is a big
tree (not so huge than the kernel you are accustomed to!)

I need in my packages to produce rc scripts and udev conf, hotplug
scripts. They depends of the base rc functions of openwrt (int etc),
the config system (in lib/config), and the hotplug scripts
(/etc/hotplug.d), and I need to patch some of these scripts.
I would like to stay in sync with the development of openwrt, the
change in the patched files are quite frequent but usualy merge
easily. Of course this is only one part of my dependencies, the main
tree that is imported in my project is the original package that I
modify to tailor it to openwrt.

My problem is how to deal with this neither major nor minor dependency.

Thanks for taking time to answer.

Marc

Re: how to filter a pull

[Alex Riesen]

marc zonzon, Thu, May 03, 2007 21:32:52 +0200:
> I need in my packages to produce rc scripts and udev conf, hotplug
> scripts. They depends of the base rc functions of openwrt (int etc),
> the config system (in lib/config), and the hotplug scripts
> (/etc/hotplug.d), and I need to patch some of these scripts.
> I would like to stay in sync with the development of openwrt, the
> change in the patched files are quite frequent but usualy merge
> easily. Of course this is only one part of my dependencies, the main
> tree that is imported in my project is the original package that I
> modify to tailor it to openwrt.

It looks like you need neither subproject nor partial checkouts
(another Git-specific feature which is just about to come into
existence). Just manage everything in one repo. Git deals with large,
big and even huge repos splendidly. It's the monstrous repos which
still make problems, but people working on them.

Re: how to filter a pull

[Karl Hasselström]

On 2007-05-03 11:21:04 -0700, Junio C Hamano wrote:

> If I were doing this today, I would probably use separate
> repositories, next to the primary project, to host the whole tree of
> other projects, adjust the build procedure of the primary project to
> borrow the whole of these other projects not just subtree -- and/or
> have appropriate symlinks in the primary project that point into
> relevant subtrees in the neighbouring repositories that host these
> other projects.

It should be straightforward (and efficient) to make a script that
takes an existing branch and makes a parallel branch that contains
only one subtree of the first branch. This derived branch can then be
used as a subproject or whatever.

Or is there some obvious reason why this wouldn't work, or would be
inconvenient?

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle

Re: how to filter a pull

[marc zonzon]

On 5/4/07, Karl Hasselström <kha@treskal.com> wrote:

> It should be straightforward (and efficient) to make a script that
> takes an existing branch and makes a parallel branch that contains
> only one subtree of the first branch. This derived branch can then be
> used as a subproject or whatever.
>
> Or is there some obvious reason why this wouldn't work, or would be
> inconvenient?

This was my initial idea,  pull and filter!, but of course it is
somewhat a hack, you pull inside git, and filter outside git, I don't
know how to keep history, it seems that once filtered the origin of
the subtree is lost. But I lack of git proficiency, I have no
experience wich git core and mainly have used it at the user
'porcelain' level.
I have had some hope that there is some mean to forward the internal
state recording of the database state.
The solution proposed by alex does not seem to be an option, if you
draw from many projects, you cannot host the whole trees because some
nodes are duplicates, and if they are outside your field you don't
bother of merging them.
Moreover I don't know how you mark what belongs to your small project
island, among this big ocean.

So if I have no further advice I think the wiser is to try the raw
script option, at least for the next 6 months the time to git, and my
own proficiency to ripen..

Marc