netfilter moving to git

netfilter moving to git

[Patrick McHardy]

I've finally managed to complete the git setup on netfilter.org
and created an iptables repository for testing.

gitweb is running at git.netfilter.org (http and https), the
git-daemon URL for the repositories is shown in gitweb.

The repository might change in case we manage to import the missin
branches and fix up the tags (they dates appear to be messed up for
some reason), but please feel free to test already and let me know
of any problems.

I'll send another announcement when we've completed the switch to
git so people know when to stop using SVN.

Re: netfilter moving to git

[Jan Engelhardt]

On Tuesday 2008-05-13 15:24, Patrick McHardy wrote:

> I've finally managed to complete the git setup on netfilter.org
> and created an iptables repository for testing.
>
> gitweb is running at git.netfilter.org (http and https), the
> git-daemon URL for the repositories is shown in gitweb.
>
> The repository might change in case we manage to import the missin
> branches and fix up the tags (they dates appear to be messed up for
> some reason), but please feel free to test already and let me know
> of any problems.

All of that I have long done in my copy of the git repository,
announced already around mid-February. - git://dev.medozas.de/xtables
(Never mind the repository's name, it's the official SVN-top-off-the-head)

Re: netfilter moving to git

[Patrick McHardy]

Jan Engelhardt wrote:
> On Tuesday 2008-05-13 15:24, Patrick McHardy wrote:
> 
>> I've finally managed to complete the git setup on netfilter.org
>> and created an iptables repository for testing.
>>
>> gitweb is running at git.netfilter.org (http and https), the
>> git-daemon URL for the repositories is shown in gitweb.
>>
>> The repository might change in case we manage to import the missin
>> branches and fix up the tags (they dates appear to be messed up for
>> some reason), but please feel free to test already and let me know
>> of any problems.
> 
> All of that I have long done in my copy of the git repository,
> announced already around mid-February. - git://dev.medozas.de/xtables
> (Never mind the repository's name, it's the official SVN-top-off-the-head)


Feel free to send me instructions for repeating these steps,
but I certainly won't replace our repository by an external
one.

Re: netfilter moving to git

[Jan Engelhardt]

On Tuesday 2008-05-13 16:17, Patrick McHardy wrote:
> Jan Engelhardt wrote:
>> On Tuesday 2008-05-13 15:24, Patrick McHardy wrote:
>> 
>> > I've finally managed to complete the git setup on netfilter.org
>> > and created an iptables repository for testing.
>> >
>> > gitweb is running at git.netfilter.org (http and https), the
>> > git-daemon URL for the repositories is shown in gitweb.
>> >
>> > The repository might change in case we manage to import the missin
>> > branches and fix up the tags (they dates appear to be messed up for
>> > some reason), but please feel free to test already and let me know
>> > of any problems.
>> 
>> All of that I have long done in my copy of the git repository,
>> announced already around mid-February. - git://dev.medozas.de/xtables
>> (Never mind the repository's name, it's the official SVN-top-off-the-head)
>
> Feel free to send me instructions for repeating these steps,

while (!done && !looks_good) {
	git-filter-branch
}

Seriously, there is no automated way to properly populate the Author,
Committer, Date and Email fields, especially if the practice how it
is effectively stored in the logs has changed over time.
http://dev.medozas.de/files/iptables.aux/ is the base scripts
(running do_rewrite), plus hand fixups (do_rewrite2) where the
automated tools fail to catch every case.

> but I certainly won't replace our repository by an external one.

This is the one from Yasuyuki, kept in-sync with the SVN.

There is no difference in the blobs:
diff -dpru -x .svn -x .git iptables.svncheckout iptables.git;
(account for $Id$ expansion)

or the directory structure:
cd iptables.svncheckout/;
for i in `find [a-z]* -type d`; do
	diff -dpru <(git ls-tree HEAD $i/) \
		   <(cd ../xtables/; git ls-tree HEAD $i/);
done | less -S

Only changes so far are logs to fix obvious spelling mistakes or
metadata fixups (separation of author--committer), or 
perhaps just give a better message for the older commits like "argh".

[something like
	for ((n = 0; n < 1244; ++n); do
	diff -dpru <(cd iptables.gitified; git log HEAD~"$n"!) \
	<(cd xtables; git log HEAD~"$n"!)
	done;
should make it visible]

Fact is there has been going a lot of work into keeping it in 
git-style shape. It's alright if you do not trust it, but then
again, it would be a waste to run with a crippled-metadata repo
or to do the conversion again.

Re: netfilter moving to git

[Patrick McHardy]

Jan Engelhardt wrote:
> On Tuesday 2008-05-13 16:17, Patrick McHardy wrote:
>>>
>>> All of that I have long done in my copy of the git repository,
>>> announced already around mid-February. - git://dev.medozas.de/xtables
>>> (Never mind the repository's name, it's the official SVN-top-off-the-head)
 >>
>> Feel free to send me instructions for repeating these steps,
> 
> while (!done && !looks_good) {
> 	git-filter-branch
> }
> 
> Seriously, there is no automated way to properly populate the Author,
> Committer, Date and Email fields, especially if the practice how it
> is effectively stored in the logs has changed over time.
> http://dev.medozas.de/files/iptables.aux/ is the base scripts
> (running do_rewrite), plus hand fixups (do_rewrite2) where the
> automated tools fail to catch every case.

I don't think its particulary important to fix up the commiter
names (and strip the certificate-mess), it you really need to
know, its easy to find out.

>> but I certainly won't replace our repository by an external one.
> 
> This is the one from Yasuyuki, kept in-sync with the SVN.
> 
> There is no difference in the blobs:
> diff -dpru -x .svn -x .git iptables.svncheckout iptables.git;
> (account for $Id$ expansion)
> 
> or the directory structure:
> cd iptables.svncheckout/;
> for i in `find [a-z]* -type d`; do
> 	diff -dpru <(git ls-tree HEAD $i/) \
> 		   <(cd ../xtables/; git ls-tree HEAD $i/);
> done | less -S

I took the last one from Yasuyuki and resynced it, might have
missed a more current version though.

> Only changes so far are logs to fix obvious spelling mistakes or
> metadata fixups (separation of author--committer), or 
> perhaps just give a better message for the older commits like "argh".
> 
> [something like
> 	for ((n = 0; n < 1244; ++n); do
> 	diff -dpru <(cd iptables.gitified; git log HEAD~"$n"!) \
> 	<(cd xtables; git log HEAD~"$n"!)
> 	done;
> should make it visible]
> 
> Fact is there has been going a lot of work into keeping it in 
> git-style shape. It's alright if you do not trust it, but then
> again, it would be a waste to run with a crippled-metadata repo
> or to do the conversion again.

True, but as a matter of principle, I can't use an untrusted repository.

And anyways, I also don't see anything wrong with simply starting with
a clean history by only importing the latest version, In that case I'd
have no problem using your fixed-up repository as historical reference.

Re: netfilter moving to git

[Jan Engelhardt]

On Tuesday 2008-05-13 17:19, Patrick McHardy wrote:
>> 
>> Fact is there has been going a lot of work into keeping it in git-style
>> shape. It's alright if you do not trust it, but then
>> again, it would be a waste to run with a crippled-metadata repo
>> or to do the conversion again.
>
> True, but as a matter of principle, I can't use an untrusted repository.
>
> And anyways, I also don't see anything wrong with simply starting with
> a clean history by only importing the latest version, In that case I'd
> have no problem using your fixed-up repository as historical reference.
>
I think this idea came up back in February already (starting from a
new slate), I am fine with that.

Though, what's wrong with stitching the first version of the
new repository to the last version of the old one?

Re: netfilter moving to git

[Patrick McHardy]

Jan Engelhardt wrote:
> On Tuesday 2008-05-13 17:19, Patrick McHardy wrote:
>>> Fact is there has been going a lot of work into keeping it in git-style
>>> shape. It's alright if you do not trust it, but then
>>> again, it would be a waste to run with a crippled-metadata repo
>>> or to do the conversion again.
>> True, but as a matter of principle, I can't use an untrusted repository.
>>
>> And anyways, I also don't see anything wrong with simply starting with
>> a clean history by only importing the latest version, In that case I'd
>> have no problem using your fixed-up repository as historical reference.
>>
> I think this idea came up back in February already (starting from a
> new slate), I am fine with that.
> 
> Though, what's wrong with stitching the first version of the
> new repository to the last version of the old one?


That would work, but Yasuyuki already provided me with the
information for properly tagging the tree, so I'll keep the
current iptables repository. Thanks anyway.

Re: netfilter moving to git

[Jan Engelhardt]

On Wednesday 2008-05-14 18:28, Patrick McHardy wrote:
>> > And anyways, I also don't see anything wrong with simply starting with
>> > a clean history by only importing the latest version, In that case I'd
>> > have no problem using your fixed-up repository as historical reference.
>> >
>> I think this idea came up back in February already (starting from a
>> new slate), I am fine with that.
>> 
>> Though, what's wrong with stitching the first version of the
>> new repository to the last version of the old one?
>
> That would work, but Yasuyuki already provided me with the
> information for properly tagging the tree, so I'll keep the
> current iptables repository. Thanks anyway.

"The tags - they do nothing!"

There was confusion (in the git layer) how to handle the svn path
reordering that took place over time, which Yasuyuki did a great job
of clearing up. But making it a good git repository does not stop
after importing and, probably git-tagging the SVN branches^W tags^W
twigs. I mean hey, try to work with it like with the standards that
git and linux set, like

$ git shortlog v1.4.0..HEAD --
fatal: bad revision 'v1.4.0..HEAD'

Oh shoot, the tags are not the usual ones. Well, deducting a point
and continuing the quest:

$ git shortlog svn_t_iptables_1_4_0..HEAD --
fatal: bad revision 'svn_t_iptables_1_4_0..HEAD'

Eats, shoot again, and leaves. Giving in on another deducted point:

$ git shortlog svn_t_iptables_1_4_0rc1..HEAD -- | grep -v '^ '
/C=EU/ST=EU/CN=Pablo Neira Ayuso/... (4):
/C=EU/ST=EU/CN=Patrick McHardy/... (107):
/C=JP/ST=JP/CN=Yasuyuki Kozakai/... (29):

Hm yeah X.509, not exactly git but.. *sigh*. Even so, the
contributors should get their share in the release shortlog which,
ignoring that moving to git was just recent, would also happen too
seldom if things continue their way in that 6 month cycle.

$ git shortlog v1.4.0-rc1..HEAD | grep -v '^ ' | grep -v '^$' | wc -l
23

I am not saying you should take this repository, that is for you to
decide. Just whether I should continue on that official iptables or
not because obviously here is a difference in how to run things.

Re: netfilter moving to git

[Patrick McHardy]

Jan Engelhardt wrote:
> There was confusion (in the git layer) how to handle the svn path
> reordering that took place over time, which Yasuyuki did a great job
> of clearing up. But making it a good git repository does not stop
> after importing and, probably git-tagging the SVN branches^W tags^W
> twigs. I mean hey, try to work with it like with the standards that
> git and linux set, like
> 
> $ git shortlog v1.4.0..HEAD --
> fatal: bad revision 'v1.4.0..HEAD'
> 
> Oh shoot, the tags are not the usual ones. Well, deducting a point
> and continuing the quest:
> 
> $ git shortlog svn_t_iptables_1_4_0..HEAD --
> fatal: bad revision 'svn_t_iptables_1_4_0..HEAD'

There is no 1.4.0 tag in SVN, so there's none in git.

> $ git shortlog svn_t_iptables_1_4_0rc1..HEAD -- | grep -v '^ '
> /C=EU/ST=EU/CN=Pablo Neira Ayuso/... (4):
> /C=EU/ST=EU/CN=Patrick McHardy/... (107):
> /C=JP/ST=JP/CN=Yasuyuki Kozakai/... (29):
> 
> Hm yeah X.509, not exactly git but.. *sigh*. Even so, the
> contributors should get their share in the release shortlog which,
> ignoring that moving to git was just recent, would also happen too
> seldom if things continue their way in that 6 month cycle.

Not sure what you're trying to say, what does the release cycle
have to do with anything?

Re: netfilter moving to git

[Jan Engelhardt]

On Thursday 2008-05-15 16:33, Patrick McHardy wrote:

> Jan Engelhardt wrote:
>> There was confusion (in the git layer) how to handle the svn path
>> reordering that took place over time, which Yasuyuki did a great job
>> of clearing up. But making it a good git repository does not stop
>> after importing and, probably git-tagging the SVN branches^W tags^W
>> twigs. I mean hey, try to work with it like with the standards that
>> git and linux set, like
>> 
>> $ git shortlog v1.4.0..HEAD --
>> fatal: bad revision 'v1.4.0..HEAD'
>> 
>> Oh shoot, the tags are not the usual ones. Well, deducting a point
>> and continuing the quest:
>> 
>> $ git shortlog svn_t_iptables_1_4_0..HEAD --
>> fatal: bad revision 'svn_t_iptables_1_4_0..HEAD'
>
> There is no 1.4.0 tag in SVN, so there's none in git.

Who would have thought! And there are still the branches,
which are not actually branches. See now why importing just is not
enough.

>> $ git shortlog svn_t_iptables_1_4_0rc1..HEAD -- | grep -v '^ '
>> /C=EU/ST=EU/CN=Pablo Neira Ayuso/... (4):
>> /C=EU/ST=EU/CN=Patrick McHardy/... (107):
>> /C=JP/ST=JP/CN=Yasuyuki Kozakai/... (29):
>> 
>> Hm yeah X.509, not exactly git but.. *sigh*. Even so, the
>> contributors should get their share in the release shortlog which,
>> ignoring that moving to git was just recent, would also happen too
>> seldom if things continue their way in that 6 month cycle.
>
> Not sure what you're trying to say, what does the release cycle
> have to do with anything?

Not specifically related to git, but it is the second thing I am
really unhappy about because it creates a long turnaround time for
users to actually receive the software.

Re: netfilter moving to git

[Patrick McHardy]

Jan Engelhardt wrote:
> Not specifically related to git, but it is the second thing I am
> really unhappy about because it creates a long turnaround time for
> users to actually receive the software.
>   

I'm aware of that and we're moving towards one release per kernel
release. I'll release a -rc by Monday.

Re: netfilter moving to git

[Jan Engelhardt]

On Tuesday 2008-05-13 16:17, Patrick McHardy wrote:
>> 
>> All of that I have long done in my copy of the git repository,
>> announced already around mid-February. - git://dev.medozas.de/xtables
>> (Never mind the repository's name, it's the official SVN-top-off-the-head)
>
> Feel free to send me instructions for repeating these steps,
> but I certainly won't replace our repository by an external
> one.
>
diff -dpru <(git log --pretty="format:%T" origin/master) <(git log
--pretty="format:%T" xt/master)

Result:

--- /dev/fd/63  2008-05-17 09:30:26.625861648 +0200
+++ /dev/fd/62  2008-05-17 09:30:26.629859699 +0200
@@ -1,4 +1,3 @@
-7b98d6e3798565dec77aa46ceb4f11d17f791ef7
 0d663371e39a2e4a7290ec6c5b27aaa406a8751e
 6c5a04c5b09fa13303547fba07f726c953347173
 8ae6e4457a0d70a40b573b606488744a62a7a3bb

Which proves it is the same, tree-wise, minus the newest commit
that was done.
And the log difference you can see in
http://dev.medozas.de/files/.iptables-xtables.txt
I suggest you take a look.

Re: netfilter moving to git

[Patrick McHardy]

Jan Engelhardt wrote:
> And the log difference you can see in
> http://dev.medozas.de/files/.iptables-xtables.txt
> I suggest you take a look.
>   

The committer and author name fixups certainly look nicer, but
things like:

-    Installs libxt_*.so to DEST_IPT_LIBIDR and link libip[6]t_*.so to it.
+    Install libxt_*.so to DEST_IPT_LIBIDR and link libip[6]t_*.so to it

are not good because they make it harder to find discussions that
happend before a patch was committed.

Re: netfilter moving to git

[Jan Engelhardt]

On Sunday 2008-05-18 04:25, Patrick McHardy wrote:
> Jan Engelhardt wrote:
>> And the log difference you can see in
>> http://dev.medozas.de/files/.iptables-xtables.txt
>> I suggest you take a look.
>
> The committer and author name fixups certainly look nicer,

well would you accept a script that filters the repository
to put proper committer/author in place?

> things like:
>
> -    Installs libxt_*.so to DEST_IPT_LIBIDR and link libip[6]t_*.so to it.
> +    Install libxt_*.so to DEST_IPT_LIBIDR and link libip[6]t_*.so to it
>
> are not good because they make it harder to find discussions that
> happend before a patch was committed.

If you could pick another log message to demonstrate that it's bad--
here, it's just an "s", for a patch that afaicr was not really
posted anyway.
Also, [patch N/M] is stripped from subjects for good practice,
because that is how lkml does it. In nf-land, that seems to have
slipped in again and again :(

Re: netfilter moving to git

[Patrick McHardy]

Jan Engelhardt wrote:
> On Sunday 2008-05-18 04:25, Patrick McHardy wrote:
>> Jan Engelhardt wrote:
>>> And the log difference you can see in
>>> http://dev.medozas.de/files/.iptables-xtables.txt
>>> I suggest you take a look.
>> The committer and author name fixups certainly look nicer,
> 
> well would you accept a script that filters the repository
> to put proper committer/author in place?

Mhh .. I guess I would if its soon, before too many people
have started using that tree.

Re: netfilter moving to git

[Jan Engelhardt]

On Wednesday 2008-05-21 11:43, Patrick McHardy wrote:
>> well would you accept a script that filters the repository
>> to put proper committer/author in place?
>
> Mhh .. I guess I would if its soon, before too many people
> have started using that tree.
>

http://dev.medozas.de/iptables-rw.tar.bz2

cd into the working directory, run `git status` (really necessary,
can't be automated in the script for some stupid reason),
and then run ../path/to/runme3.
It also fixes the tags; be sure to remove the svn_t_ pseudobranchtagswhatever
afterwards.

Re: netfilter moving to git

[Patrick McHardy]

Jan Engelhardt wrote:
> On Wednesday 2008-05-21 11:43, Patrick McHardy wrote:
>>> well would you accept a script that filters the repository
>>> to put proper committer/author in place?
>> Mhh .. I guess I would if its soon, before too many people
>> have started using that tree.
>>
> 
> http://dev.medozas.de/iptables-rw.tar.bz2

This gives me a 404 error.

> cd into the working directory, run `git status` (really necessary,
> can't be automated in the script for some stupid reason),
> and then run ../path/to/runme3.
> It also fixes the tags; be sure to remove the svn_t_ pseudobranchtagswhatever
> afterwards.

I'll try, thanks.

Re: netfilter moving to git

[Jan Engelhardt]

On Wednesday 2008-05-21 18:40, Patrick McHardy wrote:
> Jan Engelhardt wrote:
>> On Wednesday 2008-05-21 11:43, Patrick McHardy wrote:
>> > > well would you accept a script that filters the repository
>> > > to put proper committer/author in place?
>> > Mhh .. I guess I would if its soon, before too many people
>> > have started using that tree.
>> >
>> 
>> http://dev.medozas.de/iptables-rw.tar.bz2
>
> This gives me a 404 error.

Whoops, needs /files/ in the middle:
	http://dev.medozas.de/files/iptables-rw.tar.bz2

>> cd into the working directory, run `git status` (really necessary,
>> can't be automated in the script for some stupid reason),
>> and then run ../path/to/runme3.
>> It also fixes the tags; be sure to remove the svn_t_ pseudobranchtagswhatever
>> afterwards.
>
> I'll try, thanks.
>
>
>

Re: netfilter moving to git

[Patrick McHardy]

Jan Engelhardt wrote:
> On Wednesday 2008-05-21 18:40, Patrick McHardy wrote:
>> Jan Engelhardt wrote:
>>> On Wednesday 2008-05-21 11:43, Patrick McHardy wrote:
>>>>> well would you accept a script that filters the repository
>>>>> to put proper committer/author in place?
>>>> Mhh .. I guess I would if its soon, before too many people
>>>> have started using that tree.
>>>>
>>> http://dev.medozas.de/iptables-rw.tar.bz2
>> This gives me a 404 error.
> 
> Whoops, needs /files/ in the middle:
> 	http://dev.medozas.de/files/iptables-rw.tar.bz2

Thanks. It does a lot of rewrites, but then fails with:

../runme3: line 7: /tmp/ipt-log/iptables.aux/add-tags: No such file or 
directory

That file seems to be missing from the archive.

Re: netfilter moving to git

[Jan Engelhardt]

On Thursday 2008-05-22 16:21, Patrick McHardy wrote:
>> 
>> Whoops, needs /files/ in the middle:
>>  http://dev.medozas.de/files/iptables-rw.tar.bz2
>
> Thanks. It does a lot of rewrites, but then fails with:
>
> ../runme3: line 7: /tmp/ipt-log/iptables.aux/add-tags: No such file or
> directory
>
> That file seems to be missing from the archive.
>
I added it now. (I used selective wildcards because there are other rewrite
scripts in the original directory, like do_rewrite2 which is an interactive
rewrite.)

Re: netfilter moving to git

[Patrick McHardy]

Jan Engelhardt wrote:
> On Thursday 2008-05-22 16:21, Patrick McHardy wrote:
>>> Whoops, needs /files/ in the middle:
>>>  http://dev.medozas.de/files/iptables-rw.tar.bz2
>> Thanks. It does a lot of rewrites, but then fails with:
>>
>> ../runme3: line 7: /tmp/ipt-log/iptables.aux/add-tags: No such file or
>> directory
>>
>> That file seems to be missing from the archive.
>>
> I added it now. (I used selective wildcards because there are other rewrite
> scripts in the original directory, like do_rewrite2 which is an interactive
> rewrite.)


This worked, and in indeed looks a lot better, thanks.
I'll push it after some sanity-tests.

Re: netfilter moving to git

[Patrick McHardy]

Jan Engelhardt wrote:
> On Thursday 2008-05-22 16:21, Patrick McHardy wrote:
>>> Whoops, needs /files/ in the middle:
>>>  http://dev.medozas.de/files/iptables-rw.tar.bz2
>> Thanks. It does a lot of rewrites, but then fails with:
>>
>> ../runme3: line 7: /tmp/ipt-log/iptables.aux/add-tags: No such file or
>> directory
>>
>> That file seems to be missing from the archive.
>>
> I added it now. (I used selective wildcards because there are other rewrite
> scripts in the original directory, like do_rewrite2 which is an interactive
> rewrite.)

One thing I forgot - are you OK with putting your scripts
and files somewhere public on the netfilter servers?
I would like people to be able to recreate identical
SVN imports themselves for verification.

Re: netfilter moving to git

[Jan Engelhardt]

On Thursday 2008-05-22 16:47, Patrick McHardy wrote:
> Jan Engelhardt wrote:
>> On Thursday 2008-05-22 16:21, Patrick McHardy wrote:
>> > > Whoops, needs /files/ in the middle:
>> > >  http://dev.medozas.de/files/iptables-rw.tar.bz2
>> > Thanks. It does a lot of rewrites, but then fails with:
>> >
>> > ../runme3: line 7: /tmp/ipt-log/iptables.aux/add-tags: No such file or
>> > directory
>> >
>> > That file seems to be missing from the archive.
>> >
>> I added it now. (I used selective wildcards because there are other rewrite
>> scripts in the original directory, like do_rewrite2 which is an interactive
>> rewrite.)
>
> One thing I forgot - are you OK with putting your scripts
> and files somewhere public on the netfilter servers?
> I would like people to be able to recreate identical
> SVN imports themselves for verification.
>
Just push the tarball, I release it in the public domain.

Re: netfilter moving to git

[Jan Engelhardt]

On Thursday 2008-05-22 16:47, Patrick McHardy wrote:
> Jan Engelhardt wrote:
>> On Thursday 2008-05-22 16:21, Patrick McHardy wrote:
>> > > Whoops, needs /files/ in the middle:
>> > >  http://dev.medozas.de/files/iptables-rw.tar.bz2
>> > Thanks. It does a lot of rewrites, but then fails with:
>> >
>> > ../runme3: line 7: /tmp/ipt-log/iptables.aux/add-tags: No such file or
>> > directory
>> >
>> > That file seems to be missing from the archive.
>> >
>> I added it now. (I used selective wildcards because there are other rewrite
>> scripts in the original directory, like do_rewrite2 which is an interactive
>> rewrite.)
>
> One thing I forgot - are you OK with putting your scripts
> and files somewhere public on the netfilter servers?
> I would like people to be able to recreate identical
> SVN imports themselves for verification.
>
Nothing happened so far?