dependable submodules

dependable submodules

[Daniel]

I have a submodule from, example, github. I do not however want to be
reliant years from now, that this remote repository on github is in
existant. Yet, I want to be able to easily do a git pull in the
submodule as long as it's around.

Is there a way to achieve this?

IOW, I may have

    myrepo/somefile.txt
    myrepo/another_project/.git/...
    myrepo/another_project/another_project_file.txt

and I'd like to, when I clone repo, to be able to get all files from
repo independently of the remote repo's existence.

Currently, if I do `git clone --recursive myrepo new_repo` I can get
almost the behavior I want with the exception of the other project
being pulled remotely, which means I would *have* to have a network
connection to do so, but as mentioned, I don't want that dependency.

-d

Re: dependable submodules

[Junio C Hamano]

Daniel <daniel@netwalk.org> writes:

> IOW, I may have
>
>     myrepo/somefile.txt
>     myrepo/another_project/.git/...
>     myrepo/another_project/another_project_file.txt
>
> and I'd like to, when I clone repo, to be able to get all files from
> repo independently of the remote repo's existence.

Cloned repositories from your repository can say where they get from
"another_project" embedded in it in .git/config of the clone of the
superproject. The supermodule may have .gitmodules that points at github
or whereever your original upstream repository resided, but that is
supposed to be used to only give you the default.

When "another_project" you borrowed really dies and becomes useless for
anybody, you may update its entry in .gitmodules of the superproject to
point at the public repository that inherited that submodule project.
The project might be simply abandoned, in which case you end up being
that inheritor.

Re: dependable submodules

[Daniel]

On Sat, Mar 19, 2011 at 9:27 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Daniel <daniel@netwalk.org> writes:
>
>> IOW, I may have
>>
>>     myrepo/somefile.txt
>>     myrepo/another_project/.git/...
>>     myrepo/another_project/another_project_file.txt
>>
>> and I'd like to, when I clone repo, to be able to get all files from
>> repo independently of the remote repo's existence.
>
> Cloned repositories from your repository can say where they get from
> "another_project" embedded in it in .git/config of the clone of the
> superproject. The supermodule may have .gitmodules that points at github
> or whereever your original upstream repository resided, but that is
> supposed to be used to only give you the default.
>
> When "another_project" you borrowed really dies and becomes useless for
> anybody, you may update its entry in .gitmodules of the superproject to
> point at the public repository that inherited that submodule project.
> The project might be simply abandoned, in which case you end up being
> that inheritor.
>

So would it be correct to say that even though those files are
remotely fetched on a clone of the repository, they are actually
revisioned and stored in the super project?

This would have to be true for me to be able to still be able to
access those files in which case the project was abandoned (or if I
lacked internet access).

-d


-- 
"America was founded by men who understood that the threat of domestic
tyranny is as great as any threat from abroad. If we want to be worthy
of their legacy, we must resist the rush toward ever-increasing state
control of our society. Otherwise, our own government will become a
greater threat to our freedoms than any foreign terrorist."
 - Ron Paul, Texas Straight Talk, May 31, 2004

Re: dependable submodules

[Jens Lehmann]

Am 20.03.2011 05:32, schrieb Daniel:
> So would it be correct to say that even though those files are
> remotely fetched on a clone of the repository, they are actually
> revisioned and stored in the super project?

Not in the superproject, but you'll have your copy of the history
of the submodule in its own .git directory on your harddrive.

> This would have to be true for me to be able to still be able to
> access those files in which case the project was abandoned (or if I
> lacked internet access).

If the remote side vanishes you still have your local copy. And if
you need to share that with others you can push the submodule
somewhere else and adjust the .gitmodules entry accordingly.

Re: dependable submodules

[Daniel]

On Sun, Mar 20, 2011 at 5:36 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 20.03.2011 05:32, schrieb Daniel:
>> So would it be correct to say that even though those files are
>> remotely fetched on a clone of the repository, they are actually
>> revisioned and stored in the super project?
>
> Not in the superproject, but you'll have your copy of the history
> of the submodule in its own .git directory on your harddrive.
>
>> This would have to be true for me to be able to still be able to
>> access those files in which case the project was abandoned (or if I
>> lacked internet access).
>
> If the remote side vanishes you still have your local copy. And if
> you need to share that with others you can push the submodule
> somewhere else and adjust the .gitmodules entry accordingly.
>


Because I'm using it to version control my own home directory, and not
all files are checked in to to git, this would post a problem for
backup. I only back up the .git folder itself since it's all I need
(running git checkout . to restore all the files) and individually
selecting each file to restore is a lot of work.

Perhaps I just have an odd use-case.

Re: dependable submodules

[Robert Pollak]

 On 2011-03-20 Daniel wrote:
[...]
> So would it be correct to say that even though those files are
> remotely fetched on a clone of the repository, they are actually
> revisioned and stored in the super project?

You might want to try Avery Pennarun's git-subtree, which seems to
provide this functionality.
It's at https://github.com/apenwarr/git-subtree.

Re: dependable submodules

[Daniel]

On Mon, Mar 21, 2011 at 9:18 AM, Robert Pollak <robert.pollak@jku.at> wrote:
>  On 2011-03-20 Daniel wrote:
> [...]
>> So would it be correct to say that even though those files are
>> remotely fetched on a clone of the repository, they are actually
>> revisioned and stored in the super project?
>
> You might want to try Avery Pennarun's git-subtree, which seems to
> provide this functionality.
> It's at https://github.com/apenwarr/git-subtree.

I tried git-subtree. thanks but this is not what I wanted. This
removed the .git dir for the subtree and hence updating the subtree is
not easy.

I want the functionality of git submodule except that I don't want the
version checked in as part of the superproject to have to be fetched
remotely.

-d




-- 
"America was founded by men who understood that the threat of domestic
tyranny is as great as any threat from abroad. If we want to be worthy
of their legacy, we must resist the rush toward ever-increasing state
control of our society. Otherwise, our own government will become a
greater threat to our freedoms than any foreign terrorist."
 - Ron Paul, Texas Straight Talk, May 31, 2004

Re: dependable submodules

[Avery Pennarun]

On Mon, Mar 21, 2011 at 8:56 PM, Daniel <daniel@netwalk.org> wrote:
> I tried git-subtree. thanks but this is not what I wanted. This
> removed the .git dir for the subtree and hence updating the subtree is
> not easy.
>
> I want the functionality of git submodule except that I don't want the
> version checked in as part of the superproject to have to be fetched
> remotely.

Well, as the logic goes, you commit to the imported project a lot more
often than you submit upstream, so git-subtree makes the common part
much easier and the less-common part slightly harder, which is a
better tradeoff than git submodules.  But it does seem to be a matter
of preference.

If you just want the submodule's commit to be auto-fetched from the
same repo as the supermodule, I've been experimenting with a tool I've
created called 'git tie' that makes this possible.  It's not yet ready
for prime time, but you can use the same trick yourself if you're in a
hurry.  The idea is simple enough: just maintain a branch named
refs/heads/tie that is a continually updated "merge -s ours" of all
your submodule commits, all jammed together in a single branch that
nobody will ever look at.

Then when you 'git fetch origin', it'll fetch both the 'tie' branch
and all your other branches, which have references to the commits in
the 'tie' branch.  The 'tie' branch is what makes it so they'll get
auto-downloaded.  (Don't forget to push it, too, when you push
master.)

The tool I'm working on is basically just one that will auto-maintain
the 'tie' branch for you.

Oh, and the second half of this is to just use '.' in your .gitmodules
as the path to your submodule repo.  It would be nice if 'git
submodule update' would use the alternates mechanism (git clone
--reference) to get all objects from your subproject's local repo,
since we know they're there anyway.  I was going to make 'git tie' do
this part right too, so you wouldn't even need a .gitmodules file.
But either way, you'll survive, it might just be slower than optimal
if it downloads everything twice.

And there you go... I've now told you my secrets and regaled you with
vapourware.  Hope that helps more than it hurts :)

Have fun,

Avery

Re: dependable submodules

[in-gitvger]

In message <AANLkTikv+Wf_nSt0GZj0WgPjpbk6Kr_WG-ueO6US9bUM@mail.gmail.com>, Dani
el writes:

    I tried git-subtree. thanks but this is not what I wanted. This
    removed the .git dir for the subtree and hence updating the subtree is
    not easy.

    I want the functionality of git submodule except that I don't want the
    version checked in as part of the superproject to have to be fetched
    remotely.

Could "remotely" be a local shadow master?  gitslave is another option
to git-submodules and git subtree merge for provide a superproject
with attached slaves, but like git submodules it does require fetching
from "upstream".  However, there is nothing requiring that "upstream"
be the true Internet upstream.  It could be a local shadow copy of the
master.  I describe this scenario in more detail on the gitslave home
page starting with the third paragraph under "Gitslave is not
perfect".  I wrote this with gitslave in mind, but there is nothing
stopping you from using the concept with git-submodules (aside from
the pain of the submodule update pain added to the shadow repository
update pain breeding).

However, if you really really have a good reason to want *everything*
in one repo, perhaps you could play games with branches and subtree
merge.  For each slave repository, create a new branch with the
content you want (or do a merge from the slave onto the new branch to
get the history or whatever).  Then attach the local repository to
itself using git-submodules (bonus points if you can use a
repository-relative URL), set the correct SHA corresponding to the
branch of the "slave", and then (if it works, and I don't see why it
would not) you are all done.  I would hate to have to use and update
such a repository, though.

					-Seth Robertson

Re: dependable submodules

[Daniel]

On Tue, Mar 22, 2011 at 7:28 AM,  <in-gitvger@baka.org> wrote:
>
> In message <AANLkTikv+Wf_nSt0GZj0WgPjpbk6Kr_WG-ueO6US9bUM@mail.gmail.com>, Dani
> el writes:
>
>    I tried git-subtree. thanks but this is not what I wanted. This
>    removed the .git dir for the subtree and hence updating the subtree is
>    not easy.
>
>    I want the functionality of git submodule except that I don't want the
>    version checked in as part of the superproject to have to be fetched
>    remotely.
>
> Could "remotely" be a local shadow master?  gitslave is another option
> to git-submodules and git subtree merge for provide a superproject
> with attached slaves, but like git submodules it does require fetching
> from "upstream".  However, there is nothing requiring that "upstream"
> be the true Internet upstream.  It could be a local shadow copy of the
> master.  I describe this scenario in more detail on the gitslave home
> page starting with the third paragraph under "Gitslave is not
> perfect".  I wrote this with gitslave in mind, but there is nothing
> stopping you from using the concept with git-submodules (aside from
> the pain of the submodule update pain added to the shadow repository
> update pain breeding).
>
> However, if you really really have a good reason to want *everything*
> in one repo, perhaps you could play games with branches and subtree
> merge.  For each slave repository, create a new branch with the
> content you want (or do a merge from the slave onto the new branch to
> get the history or whatever).  Then attach the local repository to
> itself using git-submodules (bonus points if you can use a
> repository-relative URL), set the correct SHA corresponding to the
> branch of the "slave", and then (if it works, and I don't see why it
> would not) you are all done.  I would hate to have to use and update
> such a repository, though.
>
>                                        -Seth Robertson
>

My current setup is that I'm simply versioning my home directory with
mercurial. Some vim plugins are under git and since it's a different
version control system I can check in the plugins easily. When I need
to update those plugins, I just run git update, and check them into
mercurial again. No problem, very easy.  I'm looking into what it
would take to convert my repo to be completely git, and so far I have
no exact way of dealing with those vim plugins since via submodules I
have to rely on the external repos being present when I clone my
homedir repo. Granted, if the repos have gone away a decade from now
and my vim plugins are not present, it won't be the end of the world,
but I want to see how close I can get to my current functionality when
using git instead of mercurial.

Re: dependable submodules

[Jens Lehmann]

Am 22.03.2011 15:55, schrieb Daniel:
> My current setup is that I'm simply versioning my home directory with
> mercurial. Some vim plugins are under git and since it's a different
> version control system I can check in the plugins easily. When I need
> to update those plugins, I just run git update, and check them into
> mercurial again. No problem, very easy.  I'm looking into what it
> would take to convert my repo to be completely git, and so far I have
> no exact way of dealing with those vim plugins since via submodules I
> have to rely on the external repos being present when I clone my
> homedir repo. Granted, if the repos have gone away a decade from now
> and my vim plugins are not present, it won't be the end of the world,
> but I want to see how close I can get to my current functionality when
> using git instead of mercurial.

You could add the .git directories of the vim plugin direcories that
are under git to the .gitignore in your home directory. Then you can
just continue to do what you are used to: Go into the vim plugin dir,
run git update there, go back into your home directory and commit the
new files and push everything to your remote. You'll lose the pointers
from the vim plugin directories to their origins that way, but that
doesn't seem to be a problem for you.