Re: please pull ppc64-2.6.git

Re: please pull ppc64-2.6.git

[Linus Torvalds]

On Mon, 29 Aug 2005, Paul Mackerras wrote:
> 
> Please do a pull from:
> 
> rsync://rsync.kernel.org/pub/scm/linux/kernel/git/paulus/ppc64-2.6.git

Gaah.

This is not a valid git repository.

Guys, if you do partially populated repositories, _please_ make sure that 
you still make it a valid git repository. These days you can trivially do 
so by doing a

	echo /pub/scm/linux/kernel/git/torvalds/linux-2.6/objects > objects/info/alternates

or similar. That also makes gitweb able to show diffs etc, something it 
can't do for a broken partial repository.

Not pulled,

		Linus

Re: please pull ppc64-2.6.git

[Russell King]

On Mon, Aug 29, 2005 at 10:32:09AM -0700, Linus Torvalds wrote:
> On Mon, 29 Aug 2005, Paul Mackerras wrote:
> > Please do a pull from:
> > 
> > rsync://rsync.kernel.org/pub/scm/linux/kernel/git/paulus/ppc64-2.6.git
> 
> Gaah.
> 
> This is not a valid git repository.
> 
> Guys, if you do partially populated repositories, _please_ make sure that 
> you still make it a valid git repository. These days you can trivially do 
> so by doing a
> 
> 	echo /pub/scm/linux/kernel/git/torvalds/linux-2.6/objects > objects/info/alternates
> 
> or similar. That also makes gitweb able to show diffs etc, something it 
> can't do for a broken partial repository.

Is the expected filesystem layout documented somewhere online (_external_
to the source code) ?

The reason I stress external to the code is that some of us do not track
git developments.  (Except via the ctrl-d method in our mail readers.)

Alternatively, when changes occur to the repostory format, please can
they be marked with some obvious subject so that folk know when things
are going to break?

-- 
Russell King

Re: please pull ppc64-2.6.git

[Linus Torvalds]

On Mon, 29 Aug 2005, Russell King wrote:
> 
> Is the expected filesystem layout documented somewhere online (_external_
> to the source code) ?

Nope, I don't think so. 

> Alternatively, when changes occur to the repostory format, please can
> they be marked with some obvious subject so that folk know when things
> are going to break?

The only actual filesystem _breakage_ has been the introduction of 
pack-files (and the old, old _old_ thing where I changed the actual object 
compression/hashing order).

The "objects/info/alternates" thing is an extension, which allows you to 
have a partial object store, and point to the "rest of it", and still have 
all the tools understand it and be able to parse the totality of it. So it 
doesn't break or change old formats, it only allows a new one.

(Partial repos have always worked with the rsync protocol, and with the
client-side pulling. But that was more of an accident than anything else,
and they fundamentally were broken for any real work - gitweb can't show
anything really sane, server-side serving - whether anonymous or ssh -
doesn't work, etc etc).

		Linus

Re: please pull ppc64-2.6.git

[Russell King]

On Mon, Aug 29, 2005 at 11:02:38AM -0700, Linus Torvalds wrote:
> The "objects/info/alternates" thing is an extension, which allows you to 
> have a partial object store, and point to the "rest of it", and still have 
> all the tools understand it and be able to parse the totality of it. So it 
> doesn't break or change old formats, it only allows a new one.

Ah, ok.  I thought it was a new requirement, and I had visions of
similar complaints about my repositories.  Thanks for explaining
the situation.

-- 
Russell King

Re: please pull ppc64-2.6.git

[Paul Mackerras]

Linus Torvalds writes:

> 	echo /pub/scm/linux/kernel/git/torvalds/linux-2.6/objects > objects/info/alternates

Did that (s/linux-2.6/linux-2.6.git/ actually...)

What can one put in the alternates file?  Just an absolute path, or
does a relative path or a URL work too?

> or similar. That also makes gitweb able to show diffs etc, something it 
> can't do for a broken partial repository.

Gitweb still doesn't work; I think someone needs to update the git on
*.kernel.org.

Paul.

Re: please pull ppc64-2.6.git

[Linus Torvalds]

On Tue, 30 Aug 2005, Paul Mackerras wrote:
> 
> What can one put in the alternates file?  Just an absolute path, or
> does a relative path or a URL work too?

Only an absolute path.

URL's fundamentally do not work, and relative paths end up being parsed as
relative to where-ever the user happens to be (and some commands will 
"chdir()" into the .git directory, while others will not).

For a raw git directory like the ones on master.kernel.org, with a
relative pathname (relative to the main git directory itself) the
_pulling_ should actually happen to work, since that will be happening in
the xxx.git directory using "GIT_DIR=.".

But anything who uses GIT_DIR=<something else> from another directory
wouldn't be able to use it. That probably includes gitweb, btw.

So only do the absolute ones. Maybe we should define some well-specified 
meaning for relative ones, but it definitely isn't there now.

> > or similar. That also makes gitweb able to show diffs etc, something it 
> > can't do for a broken partial repository.
> 
> Gitweb still doesn't work; I think someone needs to update the git on
> *.kernel.org.

Hmm, yes. kernel.org is at 0.99.4, and the objects/info/alternates thing 
was done after that.

It's in 0.99.5, though, so next time kernel.org updates its git version, 
it will automagically start working.

(And I use my own git installation, so it works fine for me, and I just 
pulled the thing without problems).

			Linus

Re: please pull ppc64-2.6.git

[Christian Meder]

On Mon, 2005-08-29 at 17:20 -0700, Linus Torvalds wrote:
> 
> On Tue, 30 Aug 2005, Paul Mackerras wrote:
> > 
> > What can one put in the alternates file?  Just an absolute path, or
> > does a relative path or a URL work too?
> 
> Only an absolute path.
> 
> URL's fundamentally do not work, and relative paths end up being parsed as
> relative to where-ever the user happens to be (and some commands will 
> "chdir()" into the .git directory, while others will not).

Is alternates unthinkable with URLs (e.g. remote alternates) i.e. broken
by design or thinkable but not there yet ?

Greetings,


				Christian

-- 
Christian Meder, email: chris@absolutegiganten.org

The Way-Seeking Mind of a tenzo is actualized 
by rolling up your sleeves.

                (Eihei Dogen Zenji)

Re: please pull ppc64-2.6.git

[Junio C Hamano]

Christian Meder <chris@absolutegiganten.org> writes:

> Is alternates unthinkable with URLs (e.g. remote alternates).

In order to read an object data, the low level core GIT layer
does open()/mmap() of a file on the locally mounted filesystem.
It also does opendir()/readdir()/closedir() to find what's
available.

If you can arrange to automount via luserfs (or whatever; we do
not care much about particular implementation) so that it can do
these filesystem operations, git will happily use it.

Even in that case, the alternate you would tell git should look
like a local filesystem path --- your automounting from remote
specified with URL for such a path would happen outside GIT.

Re: please pull ppc64-2.6.git

[Sergey Vlasov]

[-- Attachment #1: Type: text/plain, Size: 1593 bytes --]

On Tue, 30 Aug 2005 15:25:22 -0700 Junio C Hamano wrote:

> Christian Meder <chris@absolutegiganten.org> writes:
> 
> > Is alternates unthinkable with URLs (e.g. remote alternates).
> 
> In order to read an object data, the low level core GIT layer
> does open()/mmap() of a file on the locally mounted filesystem.
> It also does opendir()/readdir()/closedir() to find what's
> available.
> 
> If you can arrange to automount via luserfs (or whatever; we do
> not care much about particular implementation) so that it can do
> these filesystem operations, git will happily use it.
> 
> Even in that case, the alternate you would tell git should look
> like a local filesystem path --- your automounting from remote
> specified with URL for such a path would happen outside GIT.

All this means that currently there is no clean way to publish a partial
GIT repository, unless you place it at the same server where the base
repository is located (and even in that case needing to use something
like "echo /pub/scm/linux/kernel/git/torvalds/linux-2.6/objects >
objects/info/alternates" looks like a horrible hack).

What if we create a mapping layer from URLs to local mirrors of
corresponding repositories?  Store this mapping, e.g., in
~/.git-core/mirrors, then if git reads an URL instead of an absolute
local path from objects/info/alternates, it will consult that file and
either use the specified local mirror, or complain that a local copy of
the base repository is not available.  The partial repository will then
contain no system-specific paths and could be published without
problems.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: please pull ppc64-2.6.git

[Junio C Hamano]

Sergey Vlasov <vsu@altlinux.ru> writes:

> All this means that currently there is no clean way to publish a partial
> GIT repository, unless you place it at the same server where the base
> repository is located (and even in that case needing to use something
> like "echo /pub/scm/linux/kernel/git/torvalds/linux-2.6/objects >
> objects/info/alternates" looks like a horrible hack).

There has never been a way to publish a partial GIT repository.
I personally am not convinced it is even a good idea for people
to be able to do so.

$GIT_OBJECT_DIRECTORY/info/alternates is a mechanism to solve
completely different issue -- borrowing objects locally from
different object store to save space.  This is only a local
'repository organization' issue, just like packing objects or
leaving them unpacked in a repository _should_ not make any
difference to people who are interacting with it.

The use of info/alternates is internal to the git aware server
side, be it git-daemon and git-upload-pack when somebody fetches
from it, or git-receive-pack when somebody pushes into it.  The
other end _should_ never have to care if the repository uses
info/alternates, just like it _shouldn't_ matter if the
repository is packed or unpacked.  Most importantly, for people
who are interacting with it, the repository has _everything_ it
claims to have by having pointers to head commits under refs/
hierarchy, and is _not_ partial at all.

Yes, you could peek into the remote repository filesystem by
other means, and even slurp the info/alternates file via rsync,
but as stated many times before, a lot of things that rsync
does are by accident not by design.

Some historical explanation may be helpful.  info/alternates is
merely an improvement for existing ALTERNATE_OBJECT_DIRECTORIES
mechanism.  The latter required individual processes to tell git
what other object stores to consult to find missing objects when
working on a repository whose object store is a partial object
database.  This was clearly not per-process information; it was
specific to that partial object store, and that was the reason
info/alternates was invented.  The processes do not have to have
that environment variable; instead the necessary information is
recorded in the partial object store itself.

I kept saying '_should_' because commit walkers would not
currently understand alternates, just like they had trouble with
packed repositories earlier.  This _is_ a defect, but I
personally feel that this is a problem not worth solving.

Earlier, when commit walkers did not understand packed
repositories, the only workaround was 'then do not pack your
public repository'.  This was unacceptable from storage
consumption point of view, and it had to be solved in some way.
Daniel solved it for 0.99.4.

But what info/alternates does is different.  If you care about
commit walkers (and I certainly do), there are other means to
save space without using info/alternates; first try packing, and
then hard linking object files with whichever other repository
you would have placed in info/alternates, and you are done.

Now you may argue that being able to publish truly partial
repository that does not have all the objects and depends on
some other repository, very likely to be on completely different
server (otherwise you would not be advocating for a non-local
path such as URL) a worthwhile goal.  I personally do not even
believe that a repository relying on a foreign object store is a
good idea, and much less so if you are making that repository
publicly available.

I like your local-mirror-mapping idea very much, but I do not
think it is something that is specific to git.  If you have such
a mechanism to map remote URL into local filesystem namespace
and automount/mirror them on demand, I'd like to be able to use
that when I am browsing files in my Emacs (yes, there is
'ange-ftp') or even 'cat' files from it.  Its more useful that
way than limiting its use to only when git wants to find missing
object files from its alternate object store.

Re: please pull ppc64-2.6.git

[Junio C Hamano]

Russell King <rmk@arm.linux.org.uk> writes:

> Is the expected filesystem layout documented somewhere online (_external_
> to the source code) ?

There already was a sketchy description in git(7), at
<http://www.kernel.org/pub/software/scm/git/docs/>

I've updated it a bit to describe the current status; please
look for "File/Directory Structure" section of the above URL.