[RFD] Alternative to git-based wiki: wiki as foreign VCS

[RFD] Alternative to git-based wiki: wiki as foreign VCS

[Matthieu Moy]

Hi,

I think most people on this list already faced the issue: wikis are
cool, easy to contribute, ... but it's hard to force yourself to use a
purely web-based tool to interact with something which really looks
like a version-control system.

One solution is to use a git-based wiki, like ikiwiki [1], golum
[2], ... but in many contexts, this is not applicable: because one
has an existing wiki and doesn't want to migrate, because the
git-based wiki lacks features XYZ that you rely on, or because one is
a small contributor to a large project (say, wikipedia) and cannot
force the project to change.

I'm thinking of an alternative to this: implement a foreign VCS
interface for a wiki engine. Today, one can interact with, say, SVN,
using Git (via git-svn [3]). This way, we can get most of the Git
goodness locally, and just "publish" the changes on an SVN repository.

I think that should be feasible to implement the same kind of things
to interact with, say, MediaWiki. Typical scenarios would be:

1) Work locally, possibly collaboratively on a set of pages, and then
   publish them on a wiki (let's call this "git mw set-tree" by
   analogy with "git svn set-tree").

2) Wait for user contributions on the wiki, and fetch them to the Git
   repository with one command (let's call this "git mw fetch" by
   analogy with "git svn fetch").

3) Allow one to easily download a set of files, and later get updates
   (i.e. "git clone/git pull" is far better than downloading from a
   browser)

I'm personnaly interested in this in a teaching context, since I love
Git, and I use a wiki [4] to publish some documents to my students.
Scenario 1) corresponds to teachers preparing stuffs (without
necessarily publishing drafts), and 2) corresponds to two cases:
coworker unwilling to use Git, but willing to use a wiki, and students
contributing some content. Senario 3) corresponds to the case where we
distribute a set of files (say, example pieces of code), and reference
these files from a wiki documentation. See [5] for an example.

I've already got half a solution where I publish content on GitHub,
and include them on a wiki with the "Include" extension [6]. It solves
scenario 1) partially and 3) nicely, but not 2).

Together with Sylvain Boulme (in Cc), we plan to propose a project to
students to develop a git-svn-like interface to interact with
mediawiki. Students have a bit less than a month in May-June and work
in teams of 2 to 4 students (last year, we got the textconv
functionality in "git blame" and "git gui blame", and some better
error messages with the same project).

It sounds feasible to write a usable prototype, probably re-using code
from tools interacting with mediawiki (like wikipediafs [7]), and
basing the work on git remote helpers [8]. We should be able to make
this a free software.

Among the design goals:

* No restriction on the Git workflow. Unlike "git-svn" which promotes
  a flow with SVN as the central repository, we should target a
  workflow where people merge freely using Git, and then publish/fetch
  changes from the wiki (i.e. a merge-based workflow, as opposed to a
  rebase-based one).

* Ability to import only a subset of the wiki (nobody want to "git
  clone" the whole wikipedia ;-) ). At least a manually-specified list
  of pages, and better, the content of one category.

* Ability to work interact with several wikis (e.g. a test, private
  instance, and a public instance).

And then, fancy extensions can be imagined:

* Manage non-text files as Media files uploaded to the wiki.

* Manage directories in Git as subcategories in the wiki.

Any opinions? Advices? Does this sounds like a good idea? Any pitfall
to avoid?

Thanks in advance,

Footnotes:
[1] http://ikiwiki.info/
[2] https://github.com/github/gollum
[3] http://www.kernel.org/pub/software/scm/git/docs/git-svn.html
[4] http://ensiwiki.ensimag.fr/index.php/Accueil (French)
[5] http://ensiwiki.ensimag.fr/index.php/LdB_-_Modes_d%27adressages#Mode_d.27adressage_.C2.AB.C2.A0Indirect_Registre.C2.A0.C2.BB
[6] http://www.mediawiki.org/wiki/Extension:Include
[7] http://wikipediafs.sourceforge.net/
[8] http://www.kernel.org/pub/software/scm/git/docs/git-remote-helpers.html

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [RFD] Alternative to git-based wiki: wiki as foreign VCS

[Sverre Rabbelier]

Heya,

On Thu, Feb 17, 2011 at 17:07, Matthieu Moy
<Matthieu.Moy@grenoble-inp.fr> wrote:
> Any opinions? Advices? Does this sounds like a good idea? Any pitfall
> to avoid?

Sounds like a great idea, it'd be even better if it was implemented as
a remote helper.

-- 
Cheers,

Sverre Rabbelier

Re: [RFD] Alternative to git-based wiki: wiki as foreign VCS

[Matthieu Moy]

Hi again,

So, nobody's more inspired to comment on this proposal? ;-)

Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:

> Hi,
>
> I think most people on this list already faced the issue: wikis are
> cool, easy to contribute, ... but it's hard to force yourself to use a
> purely web-based tool to interact with something which really looks
> like a version-control system.
>
> One solution is to use a git-based wiki, like ikiwiki [1], golum
> [2], ... but in many contexts, this is not applicable: because one
> has an existing wiki and doesn't want to migrate, because the
> git-based wiki lacks features XYZ that you rely on, or because one is
> a small contributor to a large project (say, wikipedia) and cannot
> force the project to change.
>
> I'm thinking of an alternative to this: implement a foreign VCS
> interface for a wiki engine. Today, one can interact with, say, SVN,
> using Git (via git-svn [3]). This way, we can get most of the Git
> goodness locally, and just "publish" the changes on an SVN repository.
>
> I think that should be feasible to implement the same kind of things
> to interact with, say, MediaWiki. Typical scenarios would be:
>
> 1) Work locally, possibly collaboratively on a set of pages, and then
>    publish them on a wiki (let's call this "git mw set-tree" by
>    analogy with "git svn set-tree").
>
> 2) Wait for user contributions on the wiki, and fetch them to the Git
>    repository with one command (let's call this "git mw fetch" by
>    analogy with "git svn fetch").
>
> 3) Allow one to easily download a set of files, and later get updates
>    (i.e. "git clone/git pull" is far better than downloading from a
>    browser)
>
> I'm personnaly interested in this in a teaching context, since I love
> Git, and I use a wiki [4] to publish some documents to my students.
> Scenario 1) corresponds to teachers preparing stuffs (without
> necessarily publishing drafts), and 2) corresponds to two cases:
> coworker unwilling to use Git, but willing to use a wiki, and students
> contributing some content. Senario 3) corresponds to the case where we
> distribute a set of files (say, example pieces of code), and reference
> these files from a wiki documentation. See [5] for an example.
>
> I've already got half a solution where I publish content on GitHub,
> and include them on a wiki with the "Include" extension [6]. It solves
> scenario 1) partially and 3) nicely, but not 2).
>
> Together with Sylvain Boulme (in Cc), we plan to propose a project to
> students to develop a git-svn-like interface to interact with
> mediawiki. Students have a bit less than a month in May-June and work
> in teams of 2 to 4 students (last year, we got the textconv
> functionality in "git blame" and "git gui blame", and some better
> error messages with the same project).
>
> It sounds feasible to write a usable prototype, probably re-using code
> from tools interacting with mediawiki (like wikipediafs [7]), and
> basing the work on git remote helpers [8]. We should be able to make
> this a free software.
>
> Among the design goals:
>
> * No restriction on the Git workflow. Unlike "git-svn" which promotes
>   a flow with SVN as the central repository, we should target a
>   workflow where people merge freely using Git, and then publish/fetch
>   changes from the wiki (i.e. a merge-based workflow, as opposed to a
>   rebase-based one).
>
> * Ability to import only a subset of the wiki (nobody want to "git
>   clone" the whole wikipedia ;-) ). At least a manually-specified list
>   of pages, and better, the content of one category.
>
> * Ability to work interact with several wikis (e.g. a test, private
>   instance, and a public instance).
>
> And then, fancy extensions can be imagined:
>
> * Manage non-text files as Media files uploaded to the wiki.
>
> * Manage directories in Git as subcategories in the wiki.
>
> Any opinions? Advices? Does this sounds like a good idea? Any pitfall
> to avoid?
>
> Thanks in advance,
>
> Footnotes:
> [1] http://ikiwiki.info/
> [2] https://github.com/github/gollum
> [3] http://www.kernel.org/pub/software/scm/git/docs/git-svn.html
> [4] http://ensiwiki.ensimag.fr/index.php/Accueil (French)
> [5] http://ensiwiki.ensimag.fr/index.php/LdB_-_Modes_d%27adressages#Mode_d.27adressage_.C2.AB.C2.A0Indirect_Registre.C2.A0.C2.BB
> [6] http://www.mediawiki.org/wiki/Extension:Include
> [7] http://wikipediafs.sourceforge.net/
> [8] http://www.kernel.org/pub/software/scm/git/docs/git-remote-helpers.html

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [RFD] Alternative to git-based wiki: wiki as foreign VCS

[Jeff King]

On Tue, Feb 22, 2011 at 11:33:19AM +0100, Matthieu Moy wrote:

> So, nobody's more inspired to comment on this proposal? ;-)

No, just busy. :)

> > One solution is to use a git-based wiki, like ikiwiki [1], golum
> > [2], ... but in many contexts, this is not applicable: because one
> > has an existing wiki and doesn't want to migrate, because the
> > git-based wiki lacks features XYZ that you rely on, or because one is
> > a small contributor to a large project (say, wikipedia) and cannot
> > force the project to change.
> >
> > I'm thinking of an alternative to this: implement a foreign VCS
> > interface for a wiki engine. Today, one can interact with, say, SVN,
> > using Git (via git-svn [3]). This way, we can get most of the Git
> > goodness locally, and just "publish" the changes on an SVN repository.

I think this a great idea. I made some first steps with the import I did
here:

  https://github.com/peff/wikitest

I'll include my quick-and-dirty mediawiki fast-exporter at the end of
this mail.  I'm sure it probably has some bugs or corner-cases it
doesn't handle, but maybe it can help as a starting point.

And then of course we'd need incremental fetching (which I planned to do
by hitting the "recent changes" API in mediawiki). And some method for
pushing changes back up, which I didn't even look at.

> > * Ability to import only a subset of the wiki (nobody want to "git
> >   clone" the whole wikipedia ;-) ). At least a manually-specified list
> >   of pages, and better, the content of one category.

Neat idea. One thing that might be useful for a site like wikipedia is
"fetch this page, and any pages it links to, pages they link to, and so
on, to a recursion depth of N". So if you are interested in some topic
you could get related topics. But that's a lot harder, since it means
fetching and parsing the mediawiki, whereas the rest can be done through
the API.

Anyway, here is my mediawiki fast-exporter. Like I said, quick and
dirty.

-- >8 --
#!/usr/bin/perl

use strict;
use MediaWiki::API;
use DB_File;
use Storable qw(freeze thaw);
use DateTime::Format::ISO8601;
use Encode qw(encode_utf8);

my $url = shift;

my $mw = MediaWiki::API->new;
$mw->{config}->{api_url} = "$url/api.php";

my $pages = $mw->list({
    action => 'query',
    list => 'allpages',
    aplimit => 500,
});

# Keep everything in a db so we are restartable.
my $revdb = tie my %revisions, 'DB_File', 'revisions.db';
print STDERR "Fetching revisions...\n";
my $n = 1;
foreach my $page (@$pages) {
  my $id = $page->{pageid};

  print STDERR "$n/", scalar(@$pages), ": $page->{title}\n";
  $n++;

  next if exists $revisions{$id};

  my $q = {
    action => 'query',
    prop => 'revisions',
    rvprop => 'content|timestamp|comment|user|ids',
    rvlimit => 10,
    pageids => $page->{pageid},
  };
  my $p;
  while (1) {
    my $r = $mw->api($q);

    # Write out all content to files.
    foreach my $rev (@{$r->{query}->{pages}->{$id}->{revisions}}) {
      my $fn = "$rev->{revid}.rev";
      open(my $fh, '>', $fn)
        or die "unable to open $fn: $!";
      binmode $fh, ':utf8';
      print $fh $rev->{'*'};
      close($fh);
      delete $rev->{'*'};
    }

    # And then save the rest, appending if necessary.
    if (defined $p) {
      push @{$p->{revisions}}, @{$r->{query}->{pages}->{$id}->{revisions}};
    }
    else {
      $p = $r->{query}->{pages}->{$id};
    }

    # And continue or quit, depending on the output.
    last unless $r->{'query-continue'};
    $q->{rvstartid} = $r->{'query-continue'}->{revisions}->{rvstartid};
  }

  print STDERR "  Fetched ", scalar(@{$p->{revisions}}), " revisions.\n";
  $revisions{$id} = freeze($p);
  $revdb->sync;
}

# Make a flat list of all page revisions, so we can
# interleave them in date order.
my @revisions = map {
  my $page = thaw($revisions{$_});
  my @revisions = @{$page->{revisions}};
  delete $page->{revisions};
  $_->{page} = $page foreach @revisions;
  @revisions
} keys(%revisions);

print STDERR "Writing export data...\n";
binmode STDOUT, ':binary';
$n = 1;
foreach my $rev (sort { $a->{timestamp} cmp $b->{timestamp} } @revisions) {
  my $user = $rev->{user} || 'Anonymous';
  my $dt = DateTime::Format::ISO8601->parse_datetime($rev->{timestamp});
  my $fn = "$rev->{revid}.rev";
  my $size = -s $fn;
  my $comment = defined $rev->{comment} ? $rev->{comment} : '';
  my $title = $rev->{page}->{title};
  $title =~ y/ /_/;

  print STDERR "$n/", scalar(@revisions), ": $rev->{page}->{title}\n";
  $n++;

  print "commit refs/remotes/origin/master\n";
  print "committer $user <none\@example.com> ", $dt->epoch, " +0000\n";
  print "data ", bytes::length(encode_utf8($comment)), "\n", encode_utf8($comment);
  print "M 644 inline $title.wiki\n";
  print "data $size\n";
  open(my $fh, '<', $fn)
    or die "unable to open $fn: $!";
  binmode $fh, ':binary';
  while (read($fh, my $buf, 4096)) {
    print $buf;
  }
}

Re: [RFD] Alternative to git-based wiki: wiki as foreign VCS

[Enrico Weigelt]

* Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> wrote:
> Hi again,
> 
> So, nobody's more inspired to comment on this proposal? ;-)

Wow.

Where can I get a "git rules the world"-tshirt ? ;-)


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------