partial checkouts

partial checkouts

[Chani]

[-- Attachment #1: Type: Text/Plain, Size: 2290 bytes --]

so I'm trying to convert KDE's translation helper scripts to support git 
repositories for kde modules. the translators themselves are staying in svn 
for now, but we want to start migrating kde modules over to git.

I've hit a bit of a problem, though.
inside the translators' folder, there's a folder called documentation/ - and 
inside that is a bunch of svn externals pointing to the doc/ folder of every 
kde module. (this is, then, all of kde's english documentation.)
translators occasionally need to run a script that reads from the 
documentation/ folder.

now, what happens when one of those modules switches to git? first, the svn 
external goes away. that's ok, I can work around that by having the update_xml 
script (the one translators need to run) fetch the doc folder itself, right?
but where does it fetch it from? doc/ is now a subdirectory in a git 
repository, and git doesn't support checking out only one folder, not even if 
all I want to do is read the files in it.

checking out the entire git repo isn't really feasible, because once KDE 
finishes switching to git that'll mean checking out all of KDE.
I don't think that turning every doc folder into a submodule is feasible, 
either - they sound kinda awkward to work with, and I can imagine lots of 
people getting confused and messing them up...

right now all I've thought of is one ugly hack: have a server that checks out 
all the kde git repos, pulls daily, copies all the doc/ folders into a 
documentation folder, and offers that folder up on the interwebs so that 
update_xml can rsync from it or download a tgz of it or something.
there appear to be lots of images in the documentation, so it's not a small 
download - 200mb and growing. it still hasn't finished downloading all the 
externals...

I'm kinda wondering if there'd be a way to use git-filter-branch to make a repo 
that only tracks the doc/ folder for a module - but I've no idea whether it'd 
have to be recreated from scratch every time someone changes something in the 
real repo's doc/

can anyone think of a less ugly solution?
what are the chances of git supporting this kind of partial checkout someday?

-- 
This message brought to you by eevil bananas and the number 3.
www.chani3.com

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: partial checkouts

[Johan Herland]

On Saturday 23 May 2009, Chani wrote:
> [...]
>
> right now all I've thought of is one ugly hack: have a server that checks
> out all the kde git repos, pulls daily, copies all the doc/ folders into
> a documentation folder, and offers that folder up on the interwebs so
> that update_xml can rsync from it or download a tgz of it or something.
> there appear to be lots of images in the documentation, so it's not a
> small download - 200mb and growing. it still hasn't finished downloading
> all the externals...

Do you need the doc/ folders from _all_ kde git repos, or just from those 
repos that you have currently checked out? In the latter case, you could 
solve this by adding symlinks to all the doc/ folders inside the 
documentation/ folder, and then make sure the software that traverse the 
documentation/ folder recognize and skips symlinks. Of course, this won't 
work if the translations project need _all_ doc/ folders accessible, but not 
all the kde git repos.

> I'm kinda wondering if there'd be a way to use git-filter-branch to make
> a repo that only tracks the doc/ folder for a module - but I've no idea
> whether it'd have to be recreated from scratch every time someone changes
> something in the real repo's doc/
>
> can anyone think of a less ugly solution?
> what are the chances of git supporting this kind of partial checkout
> someday?

Check out git-subtree. It can split out a subdirectory into its own repo, 
and re-integrate it back into the "parent" repo at a later date. git-subtree 
has been posted as a patch to this list a couple of times without much 
response, but it looks like an interesting alternative to submodules:
http://alumnit.ca/~apenwarr/log/?m=200904#30

If a lot of people find git-subtree useful, who knows, it might be included 
in a future git version.


Have fun! :)

...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: partial checkouts

[Chani]

[-- Attachment #1: Type: Text/Plain, Size: 2950 bytes --]

On May 23, 2009 16:34:53 Johan Herland wrote:
> On Saturday 23 May 2009, Chani wrote:
> > [...]
> >
> > right now all I've thought of is one ugly hack: have a server that checks
> > out all the kde git repos, pulls daily, copies all the doc/ folders into
> > a documentation folder, and offers that folder up on the interwebs so
> > that update_xml can rsync from it or download a tgz of it or something.
> > there appear to be lots of images in the documentation, so it's not a
> > small download - 200mb and growing. it still hasn't finished downloading
> > all the externals...
>
> Do you need the doc/ folders from _all_ kde git repos, or just from those
> repos that you have currently checked out? In the latter case, you could
> solve this by adding symlinks to all the doc/ folders inside the
> documentation/ folder, and then make sure the software that traverse the
> documentation/ folder recognize and skips symlinks. Of course, this won't
> work if the translations project need _all_ doc/ folders accessible, but
> not all the kde git repos.

nope, the translators may not have checked out *any* of them but the script 
they want to run needs *all* the docs. :( however, I've been told they also 
don't want to have to change their workflow in any way at all no matter how 
small, so we may be stuck in svn-land anyways, because you can't make an svn 
external out of something that's not in svn, and having anything other than 
svn externals would change their workflow :(

>
> > I'm kinda wondering if there'd be a way to use git-filter-branch to make
> > a repo that only tracks the doc/ folder for a module - but I've no idea
> > whether it'd have to be recreated from scratch every time someone changes
> > something in the real repo's doc/
> >
> > can anyone think of a less ugly solution?
> > what are the chances of git supporting this kind of partial checkout
> > someday?
>
> Check out git-subtree. It can split out a subdirectory into its own repo,
> and re-integrate it back into the "parent" repo at a later date.
> git-subtree has been posted as a patch to this list a couple of times
> without much response, but it looks like an interesting alternative to
> submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30
>
> If a lot of people find git-subtree useful, who knows, it might be included
> in a future git version.

looks interesting. might have been a solution until I heard about this 
requirement to not change workflow at all. :/

however, my friend told me about a project to make a git-svnserver that serves 
git repos as svn repos, and *that* would allow the translators to stay where 
they are without holding everyone else there too. know anything about that? 

mm, google turns up an email from someone claiming they have a partial 
implementation in python...

-- 
This message brought to you by eevil bananas and the number 3.
www.chani3.com

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: partial checkouts

[Nguyen Thai Ngoc Duy]

On Sun, May 24, 2009 at 7:00 AM, Chani <chanika@gmail.com> wrote:
> checking out the entire git repo isn't really feasible, because once KDE
> finishes switching to git that'll mean checking out all of KDE.
> I don't think that turning every doc folder into a submodule is feasible,
> either - they sound kinda awkward to work with, and I can imagine lots of
> people getting confused and messing them up...

If you don't mind cloning the entire git repo (which may be quite
large in KDE case, I guess), then you may want to try "sparse
checkout". The patch series is available as a topic branch "tp/sco" in
this branch:

http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst

Or I can send the patch series to you. It lets you specify what part
of the tree you want to checkout.
-- 
Duy

Re: partial checkouts

[Thomas Adam]

2009/5/24 Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
> On Sun, May 24, 2009 at 7:00 AM, Chani <chanika@gmail.com> wrote:
>> checking out the entire git repo isn't really feasible, because once KDE
>> finishes switching to git that'll mean checking out all of KDE.
>> I don't think that turning every doc folder into a submodule is feasible,
>> either - they sound kinda awkward to work with, and I can imagine lots of
>> people getting confused and messing them up...
>
> If you don't mind cloning the entire git repo (which may be quite
> large in KDE case, I guess), then you may want to try "sparse
> checkout". The patch series is available as a topic branch "tp/sco" in
> this branch:
>
> http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst
>
> Or I can send the patch series to you. It lets you specify what part
> of the tree you want to checkout.

I recall following this series when you first announced it.  Do you
have any plans to try and get it included in mainline?  ISTR there was
some more work needed to be done on it before then.

-- Thomas Adam

Re: partial checkouts

[Avery Pennarun]

On Sat, May 23, 2009 at 8:26 PM, Chani <chanika@gmail.com> wrote:
> On May 23, 2009 16:34:53 Johan Herland wrote:
>> On Saturday 23 May 2009, Chani wrote:
>> > I'm kinda wondering if there'd be a way to use git-filter-branch to make
>> > a repo that only tracks the doc/ folder for a module - but I've no idea
>> > whether it'd have to be recreated from scratch every time someone changes
>> > something in the real repo's doc/
>> >
>> > can anyone think of a less ugly solution?
>> > what are the chances of git supporting this kind of partial checkout
>> > someday?
>>
>> Check out git-subtree. It can split out a subdirectory into its own repo,
>> and re-integrate it back into the "parent" repo at a later date.
>> git-subtree has been posted as a patch to this list a couple of times
>> without much response, but it looks like an interesting alternative to
>> submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30
>
> looks interesting. might have been a solution until I heard about this
> requirement to not change workflow at all. :/

Here's something that might work.  Write a script that runs on a
server somewhere.  It checks out *all* of the git-managed KDE
components, including documentation (it's okay to do this from *one*
place, right?).  Then it loops through all the documentation
subdirectories and splits them out into individual projects/branches
with 'git subtree split'.

You could then make a *new* project, kde-docs for example, that pulls
those new doc projects back together into a single project, using 'git
subtree merge'.

People could then checkout the single kde-docs project and get *all*
the docs, then make changes and push them back upstream to you.  As
far as the average person is concerned, this is probably even easier
than svn:externals - they don't have to remember to commit in each
subdirectory separately, as git-subtree rips it all apart later.  Also
note that *end users* never have to operate the git subtree command;
all that happens behind the scenes on your server.

Later, you use 'git subtree split' to rip it apart back into the
individual projects, and 'git subtree merge' those back into their
original git repositories.

Since git-subtree is my own little toy and it's designed for exactly
this kind of workflow, I'd be happy to help you debug a script for
this if you like.  Probably I'll learn something.

Disclaimer: git-subtree hasn't been used by too many people yet and we
may end up finding some bugs.

Have fun,

Avery

Re: partial checkouts

[Nguyen Thai Ngoc Duy]

On Mon, May 25, 2009 at 1:57 AM, Thomas Adam <thomas.adam22@gmail.com> wrote:
> 2009/5/24 Nguyen Thai Ngoc Duy <pclouds@gmail.com>:
>> On Sun, May 24, 2009 at 7:00 AM, Chani <chanika@gmail.com> wrote:
>>> checking out the entire git repo isn't really feasible, because once KDE
>>> finishes switching to git that'll mean checking out all of KDE.
>>> I don't think that turning every doc folder into a submodule is feasible,
>>> either - they sound kinda awkward to work with, and I can imagine lots of
>>> people getting confused and messing them up...
>>
>> If you don't mind cloning the entire git repo (which may be quite
>> large in KDE case, I guess), then you may want to try "sparse
>> checkout". The patch series is available as a topic branch "tp/sco" in
>> this branch:
>>
>> http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst
>>
>> Or I can send the patch series to you. It lets you specify what part
>> of the tree you want to checkout.
>
> I recall following this series when you first announced it.  Do you
> have any plans to try and get it included in mainline?  ISTR there was
> some more work needed to be done on it before then.

It was pulled off because of big, intrusive and not well-designed patches.

http://mid.gmane.org/20081128192033.GF23984@spearce.org

I may submit it again when I get code insertions down to ~500 lines or
something like that ;-) Anyway you could always help by giving more
use cases to test against the current design.
-- 
Duy

Re: partial checkouts

[Chani]

[-- Attachment #1: Type: Text/Plain, Size: 1261 bytes --]

> >>
> >> Check out git-subtree. It can split out a subdirectory into its own
> >> repo, and re-integrate it back into the "parent" repo at a later date.
> >> git-subtree has been posted as a patch to this list a couple of times
> >> without much response, but it looks like an interesting alternative to
> >> submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30
> >
> > looks interesting. might have been a solution until I heard about this
> > requirement to not change workflow at all. :/
>
> Here's something that might work.  Write a script that runs on a
> server somewhere.  It checks out *all* of the git-managed KDE
> components, including documentation (it's okay to do this from *one*
> place, right?).  Then it loops through all the documentation
> subdirectories and splits them out into individual projects/branches
> with 'git subtree split'.
>

I like this idea. I like the idea of git subtree in general. :)
however, we've decided to go the "copy all documentation into svn" route for 
now (so that the translators can keep using only svn).
I'll be back when we run into the next svn-external issue (there's sure to be 
more). :)

-- 
This message brought to you by eevil bananas and the number 3.
www.chani3.com

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: partial checkouts

[Aidan Van Dyk]

If all you're looking for is a "single set of read-only docs" conglomeration of
all the KDE submodules projects documentation so you can translate them, what
about using "archive"?  Since archive takes a "tree-ish", and work on remote
upload-pack servers (if it's enabled), you can get an un-versioned "remote
partial checkout" with something like do things like:
	git archive --format=tar --prefix=git/Documentation/ --remote=git://repo.or.cz/git.git HEAD:Documentation | tar tf - | head
	git archive --format=tar --prefix=/opt/git/share/man/ --remote=git://repo.or.cz/git.git man | tar tf -



So, this could be simply scripted to give you a repo with *all* modules docs/ inside a single repo...
	for m in $MODULES
	do
		git archive --format=tar --prefix="$m/$DOCDIR/" --remote=$REMOTE_BASE/$m $BRANCH:$DOCDIR | tar xf -
		git add $m/$DOCDIR
		git commit $m/$DOCDIR -m "$m: Automatic import"
	done

Do that in a central location an dlet translators pull that...  Everythings
"duplicated", but git's object store will help out there...  You loose the
history of the original docs, but for translations, you you're not translating
the history anyways, your'e translation the docs at a point in time...

Or just dump all thd docs in a .gitignore'd subdirctory and have translators
run a script to update them.

a.


> On Sat, May 23, 2009 at 8:26 PM, Chani <chanika@gmail.com> wrote:
> > On May 23, 2009 16:34:53 Johan Herland wrote:
> >> On Saturday 23 May 2009, Chani wrote:
> >> > I'm kinda wondering if there'd be a way to use git-filter-branch to make
> >> > a repo that only tracks the doc/ folder for a module - but I've no idea
> >> > whether it'd have to be recreated from scratch every time someone changes
> >> > something in the real repo's doc/
> >> >
> >> > can anyone think of a less ugly solution?
> >> > what are the chances of git supporting this kind of partial checkout
> >> > someday?
 
> Here's something that might work.  Write a script that runs on a
> server somewhere.  It checks out all of the git-managed KDE
> components, including documentation (it's okay to do this from one
> place, right?).  Then it loops through all the documentation
> subdirectories and splits them out into individual projects/branches
> with 'git subtree split'.
> 
> You could then make a new project, kde-docs for example, that pulls
> those new doc projects back together into a single project, using 'git
> subtree merge'.
> 
> People could then checkout the single kde-docs project and get all
> the docs, then make changes and push them back upstream to you.  As
> far as the average person is concerned, this is probably even easier
> than svn:externals - they don't have to remember to commit in each
> subdirectory separately, as git-subtree rips it all apart later.  Also
> note that *end users* never have to operate the git subtree command;
> all that happens behind the scenes on your server.
> 
> Later, you use 'git subtree split' to rip it apart back into the
> individual projects, and 'git subtree merge' those back into their
> original git repositories.
> 
> Since git-subtree is my own little toy and it's designed for exactly
> this kind of workflow, I'd be happy to help you debug a script for
> this if you like.  Probably I'll learn something.
> 
> Disclaimer: git-subtree hasn't been used by too many people yet and we
> may end up finding some bugs.
> 
> Have fun,
> 
> Avery
>