Gate between git and mediawiki : remote-helpers

Gate between git and mediawiki : remote-helpers

[Claire Fousse]

Hi,

I'm a member of the team trying to establish a gate between mediawiki
powered wiki and git.

We've tried several things which seems to work. However, it is
something like git-svn and would require some commands such as git-mw
to work. Is it recommended to use remote-helpers instead of that ?

There is one problem though : nobody wants to git clone the whole
Wikipedia for instance. And we should add a functionality which allows
the user to clone only a part of a wiki. Are partial clonings doable
with remote helpers ?

Thank you for your help,

Best Regards


--
Claire Fousse

Re: Gate between git and mediawiki : remote-helpers

[Arnaud Lacurie]

This mail referred to that previous one :
http://article.gmane.org/gmane.comp.version-control.git/173991

2011/5/22 Claire Fousse <claire.fousse@gmail.com>:
> Hi,
>
> I'm a member of the team trying to establish a gate between mediawiki
> powered wiki and git.
>
> We've tried several things which seems to work. However, it is
> something like git-svn and would require some commands such as git-mw
> to work. Is it recommended to use remote-helpers instead of that ?
>
> There is one problem though : nobody wants to git clone the whole
> Wikipedia for instance. And we should add a functionality which allows
> the user to clone only a part of a wiki. Are partial clonings doable
> with remote helpers ?
>
> Thank you for your help,
>
> Best Regards
>
>
> --
> Claire Fousse
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Arnaud Lacurie

Re: Gate between git and mediawiki : remote-helpers

[Junio C Hamano]

Arnaud Lacurie <arnaud.lacurie@gmail.com> writes:

> This mail referred to that previous one :
> http://article.gmane.org/gmane.comp.version-control.git/173991
>
> 2011/5/22 Claire Fousse <claire.fousse@gmail.com>:
>> Hi,
>>
>> I'm a member of the team trying to establish a gate between mediawiki
>> powered wiki and git.
>>
>> We've tried several things which seems to work. However, it is
>> something like git-svn and would require some commands such as git-mw
>> to work. Is it recommended to use remote-helpers instead of that ?

I am not Matthew, but my gut feeling is that it largely depends on what
you are interacting with, and how you envision the result to be used in a
larger MediaWiki ecosystem.

In the case of SVN interoperability, there is an established community
that exchanges their work via:

	svn checkout svn://some.where.xz/project
        svn update
	svn commit

and although people on the git side can already participate with:

	git svn init svn://some.where.xz/project
        git svn rebase
        git svn dcommit

it is understandable that they wish to be able to say:

	git clone svn://some.where.xz/project
        git pull -s rebase
        git push

to make it feel more similar to the native git experience. Now, even if
there may be no "svn checkout/update/commit" equivalents in the workflow
of established MediaWiki users, it may be nice to be able to work with:

	git clone --vcs=mediawiki http://some.where.xz/wiki/
        git pull
        git push

>> There is one problem though : nobody wants to git clone the whole
>> Wikipedia for instance.

Then don't let them in your initial version. I do not see any problem in
that. People can gain experience with smaller projects, like so:

	git clone --vcs=mediawiki https://git.wiki.kernel.org/
        git pull
        git push

When we need narrow (in the tree dimention) or shallow (in the history
dimention) in either native or foreign transports, somebody would
eventually add proper support. Currently we do not do "narrow" even for
native transport (I have one cooking privately but it is progressing only
slowly, and I think there may be others who are interested in it).

I would suggest not to be worried too much about narrow/shallow in your
initial round.

Re: Gate between git and mediawiki : remote-helpers

[Arnaud Lacurie]

Thank you for your advice.

2011/5/22 Junio C Hamano <gitster@pobox.com>:

>
>        git clone --vcs=mediawiki http://some.where.xz/wiki/
>        git pull
>        git push
>

Is vcs a standard argument (I can't see it anywhere) or do we have to
change something in order to implement it?

Additionally, is there an example of protocol which uses
remote-helpers? It seems that in the remote-helpers directory, there
isn't anything but a git directory. Is there some other protocols than
git that already use remote helpers?

Thank you very much.

-- 
Arnaud Lacurie

Re: Gate between git and mediawiki : remote-helpers

[Matthieu Moy]

Arnaud Lacurie <arnaud.lacurie@gmail.com> writes:

> Thank you for your advice.
>
> 2011/5/22 Junio C Hamano <gitster@pobox.com>:
>
>>
>>        git clone --vcs=mediawiki http://some.where.xz/wiki/
>>        git pull
>>        git push
>>
>
> Is vcs a standard argument (I can't see it anywhere) or do we have to
> change something in order to implement it?

I don't know, but anyway, it could be

git clone mediawiki+http://some.where.xz/wiki

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: Gate between git and mediawiki : remote-helpers

[Jérémie NIKAES]

Hi, we are in a little bit of trouble trying to understand how remote
helpers work and thus how to create our own.
Trying to understand this, we created a git that we access through
http on a local server on which we push files with

git remote-http http://login@localip/git
push +master:master

The problem is we don't really understand which files are involved
with remote helpers.
We have perl scripts helping us to fetch and push files from mediawiki
and we don't know how to implement them as a remote helper. Does
anyone have some information regarding this ?

Thanks,
Regards

--
Jérémie Nikaes
Second year student at Ensimag

Re: Gate between git and mediawiki : remote-helpers

[Junio C Hamano]

Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:

>>>        git clone --vcs=mediawiki http://some.where.xz/wiki/
>...
>> Is vcs a standard argument (I can't see it anywhere) or do we have to
>> change something in order to implement it?
>
> I don't know, but anyway, it could be
>
> git clone mediawiki+http://some.where.xz/wiki

I know it doesn't ;-)

As "clone" is a moral equivalent of "init" possibly followed by some
auto-configuration followed by "fetch" and "checkout", it is a natural
consequence of already supporting the "remove.<name>.vcs" configuration
variable, and if we want to avoid the insane svn+http:// syntax, it is a
logical thing to add such support.

Re: Gate between git and mediawiki : remote-helpers

[Sverre Rabbelier]

Heya,

On Mon, May 23, 2011 at 07:14, Junio C Hamano <gitster@pobox.com> wrote:
> I know it doesn't ;-)
>
> As "clone" is a moral equivalent of "init" possibly followed by some
> auto-configuration followed by "fetch" and "checkout", it is a natural
> consequence of already supporting the "remove.<name>.vcs" configuration
> variable, and if we want to avoid the insane svn+http:// syntax, it is a
> logical thing to add such support.

We currently support either:

git clone mediawiki::http://some.where.xz/wiki

Or say:

git clone svn://example.com/some/svn/repo

-- 
Cheers,

Sverre Rabbelier

Re: Gate between git and mediawiki : remote-helpers

[Junio C Hamano]

Sverre Rabbelier <srabbelier@gmail.com> writes:

> We currently support either:
>
> git clone mediawiki::http://some.where.xz/wiki

Ok, I forgot that we settled on that double-colon prefix.

Thanks for fact-checking.

Re: Gate between git and mediawiki : remote-helpers

[Jérémie NIKAES]

Thank you for the advice regarding the project. Here is its status :

- We decided with your advice to use remote helpers, using the syntax
mediawiki::http://url.com (thanks to Sverre Rabbelier & Junio Hamano)
- We created a github at the following url
https://github.com/Bibzball/Git-Mediawiki
On it you will find
	* Our source code featuring
		- A start of a remote helper. The functions currently supported are
described on the wiki
		- The import part, for which we would like to thank Jeff King for
his initial perl code that we used and altered to realize the import
function of the remote helper
		- The export part is for now only a quick application of the
mediawiki API to export a file
	* A wiki with the basic documentation on our project
Feel free to consult it and give comments / ask questions

In order to maintain coherence between revisions (mediawiki) and
commits (git), we have to store metadata somewhere. Since we want to
keep transparency on the mediawiki's side, we want this data to be
stored in git. Git notes seem to be a good answer here. They could
contain the revision numbers that correspond to the commited files.
Also, a commit with t files on git should be t revisions on mediawiki.

We identified a few issues :
* Conflicts :
	- A git user wants to push files
	- He is allowed to do so : no new revisions were made on mediawiki
	- The git user starts to push the files one by one, using the mediawiki API
	- A mediawiki user edits one of the files queued up in the for-push
list and saves his modifications
	-> The git user should be interrupted in his process. If not, the
changes of the mediawiki user will be overridden.
 	We still don't have a solution for this problem. Our guess is that
we should send each file in 2 steps.
a) get the timestamp of the last revision of this file from mediawiki
b) send the file, if allowed to, based on this timestamp

* A tree from mediawiki is necessarily linear. The tree from git can
have multiple branches. Let's have this example :
	- The mediawiki has been cloned on our side
	- We made changes and committed them
	- We want to pull changes from the mediawiki
2 solutions from here :
	a) Pull including merge
		Pros : The tree is saved on our side. If someone wants to clone our
repository, he will keep this structure.
		Cons : Because of mediawiki's trees linearity, keeping branches on
our side forces us to pull again right after the push in order to
maintain coherance between mediawiki revisions and git commits.

	b) Pull -- rebase
		Pros : We are very close to mediawiki's structure, facilitating
synchronisation.
		Cons : Forbids other git users to clone my repository since some of
my branches might die after a rebase. This, in our opinion, kind of
goes against some of git's principles.

To you, which solution do you prefer ? Do you see any alternative ?

* We would like to have the alternative to clone only partially the
mediawiki (for example, only a few pages). Let's illustrate this one :
	- A user clones pages a, b and c
	- Later on, he wants to add another page d to his clone list
	-> The entire history/tree has to be remade as commits have to be
inserted in between others

Thank you for your support on this project. If any information
reported was not clear enough, feel free to ask questions. Any
contribution / suggestion is welcome.

Regards,
The Git-Mediawiki team, Arnaud Lacurie, David Amouyal, Claire Fousse &
Jeremie Nikaes.