user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[Pablo Neira Ayuso]

Jamal,

I'm cc'ing netfilter-devel.

jamal wrote:
> Hi Pablo,
> 
> On Sat, 2009-05-16 at 12:37 +0200, Pablo Neira Ayuso wrote:
>> Hi Jamal!
>>
>> Jan Engelhardt sent a couple of patches a couple of weeks ago. He is
>> changing the API that is being exported by iptables. I think that your
>> tc ipt thing is concerned. Could you let us know your opinion? The main
>> problem that I see is possible ABI breakages.
> 
> I am really hoping that we are going to end up with a stable API.
> 
> With tc the desire is even stronger because we try to maintain 
> backward compatibility i.e an old tc should still be able to 
> use current ipt and a newer tc should still be able to use older
> kernels. So breaking ABI in such a way that this becomes hard 
> to achieve IMO should be once in a blue-moon-sighting event.

Indeed, I agree.

> [Essentially with the last changes, this backward 
> compatibility broke and i had to change even the name of the 
> binary so people dont continue to use it on old scripts.]

Yes, aware of those.

> I have had to "discover" that the ABI was broken 7 different
> iptable releases in the past, when my users whine (it seems to 
> happen with some strong coincidence when i am busy at work); 
> this is an unfair expectation of me and i hope at least you
> guys get to improve on.

Now that we have a public API for extensions and your tc ipt thing uses
it, I would not expect more breakages as long as we're aware of it.

> maybe we can have some simple working relationship rules:
> 
> a) whoever breaks ABI (hopefully there are good reasons
> to intentionally break it) should also fix ipt, just like they
> fix iptables. 
> b) whoever breaks the ABI should consider both backward and
> forward compatibility. I will review patches to make sure this
> is still so.

I think that, now that we have a public API, ABI should be not broken.
In case that some interface has limitations, we can add a new one while
keeping the old. The result is not nice since you end having two
functions that do the same with different interfaces but this doesn't
break binary backward compatibility. Of course, the change would need
some justification, eg. some client code do need something that the
current interface does not allow in any way.

With this policy, design errors accumulate along time so we learn the
lesson from our own mistakes and, then, we work on a new version 2 of
the API to resolve the accumulated issues after some time. This is how
I'm managing existing netfilter libraries. This policy makes the
developement of libraries/public interfaces slower but I think that
users are way happier (no binary breakages).

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[jamal]

Pablo,

On Sun, 2009-05-17 at 16:46 +0200, Pablo Neira Ayuso wrote:

> Now that we have a public API for extensions and your tc ipt thing uses
> it, I would not expect more breakages as long as we're aware of it.
[..]
> I think that, now that we have a public API, ABI should be not broken.

That would be fantastic.

> In case that some interface has limitations, we can add a new one while
> keeping the old. 

And maybe spit out some warning somewhere that the interface being used
is deprecated (the way the kernel does it so well currently).

> The result is not nice since you end having two
> functions that do the same with different interfaces but this doesn't
> break binary backward compatibility. 

Would be an improvement - currently i have to find out over configure
scripts at compile time i.e write a little program using old API and if
doesnt compile, try the next thing etc.
I think if you also depracate over a period of time the old interface
by giving ample warnings the API users are surely going to migrate.

> Of course, the change would need
> some justification, eg. some client code do need something that the
> current interface does not allow in any way.

Indeed.

> With this policy, design errors accumulate along time so we learn the
> lesson from our own mistakes and, then, we work on a new version 2 of
> the API to resolve the accumulated issues after some time. This is how
> I'm managing existing netfilter libraries. This policy makes the
> developement of libraries/public interfaces slower but I think that
> users are way happier (no binary breakages).

I think there will always be challenges and lessons learnt but i dont
see the approach you outline as making "development slower" - it just
forces whoever introduces a new interface thinks at least twice ;->
An evolution instead of a revolution is a lot more acceptable
i.e build on existing interface and give it more features than totaly
changing it like you change your socks.

cheers,
jamal 

Re: user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[Pablo Neira Ayuso]

jamal wrote:
> On Sun, 2009-05-17 at 16:46 +0200, Pablo Neira Ayuso wrote:
> 
>> Now that we have a public API for extensions and your tc ipt thing uses
>> it, I would not expect more breakages as long as we're aware of it.
> [..]
>> I think that, now that we have a public API, ABI should be not broken.
> 
> That would be fantastic.
> 
>> In case that some interface has limitations, we can add a new one while
>> keeping the old. 
> 
> And maybe spit out some warning somewhere that the interface being used
> is deprecated (the way the kernel does it so well currently).

There is an __attribute__((deprecated)) in gcc that displays a message
during compilation time to warn that the software is using obsolete
interfaces. Then, we can remove it 1 or 2 years later.

>> The result is not nice since you end having two
>> functions that do the same with different interfaces but this doesn't
>> break binary backward compatibility. 
> 
> Would be an improvement - currently i have to find out over configure
> scripts at compile time i.e write a little program using old API and if
> doesnt compile, try the next thing etc.
> I think if you also depracate over a period of time the old interface
> by giving ample warnings the API users are surely going to migrate.

Yes, that will help, however some lazy users wait for the breakage to
move on, but who cares about them ;)

>> Of course, the change would need
>> some justification, eg. some client code do need something that the
>> current interface does not allow in any way.
> 
> Indeed.
> 
>> With this policy, design errors accumulate along time so we learn the
>> lesson from our own mistakes and, then, we work on a new version 2 of
>> the API to resolve the accumulated issues after some time. This is how
>> I'm managing existing netfilter libraries. This policy makes the
>> developement of libraries/public interfaces slower but I think that
>> users are way happier (no binary breakages).
> 
> I think there will always be challenges and lessons learnt but i dont
> see the approach you outline as making "development slower" - it just
> forces whoever introduces a new interface thinks at least twice ;->

Right :)

> An evolution instead of a revolution is a lot more acceptable
> i.e build on existing interface and give it more features than totaly
> changing it like you change your socks.

In my case, it's been four years with the current library APIs and
following an evolution approach as you have mentioned (adding new
functions and obsoleting old ones but keeping them in the tree for a
while). Now I'm doing more like a full re-design that needs the
"revolution", no way to keep backward compatibility to resolve several
issues.

Still, people would be able to have both version 1 and 2 of the
libraries installed in their computer so they can keep attached to old
versions without breaking binary backward compatibility. I've been
discussing this with a friend of mine that maintains a couple of
critical libraries in the gnome project, it was nice to expose him my
ideas and see that I was on the right track.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[jamal]

On Sun, 2009-05-17 at 17:40 +0200, Pablo Neira Ayuso wrote:
> 
> There is an __attribute__((deprecated)) in gcc that displays a message
> during compilation time to warn that the software is using obsolete
> interfaces. Then, we can remove it 1 or 2 years later.

Also something runtime maybe on syslog (example what packet socket
used to do - complaining about tcpdump until they changed it).
I did forget to mention one thing which may sound extreme,
Documentation:
A document aptly named "ABI breakage" which outlines all revisions and
what they break. Maybe even an extra document called
"Deprecation" which announces what is going to break and when.

> In my case, it's been four years with the current library APIs and
> following an evolution approach as you have mentioned (adding new
> functions and obsoleting old ones but keeping them in the tree for a
> while). Now I'm doing more like a full re-design that needs the
> "revolution", no way to keep backward compatibility to resolve several
> issues.

I think there is room for revolutions with the caveat of ample warnings.
Like you said if you have been giving warning for 2 years about
something deprecating then you are absolved of dropping the interface.

> Still, people would be able to have both version 1 and 2 of the
> libraries installed in their computer so they can keep attached to old
> versions without breaking binary backward compatibility. I've been
> discussing this with a friend of mine that maintains a couple of
> critical libraries in the gnome project, it was nice to expose him my
> ideas and see that I was on the right track.

Good motivation to migrate from version 1 to 2 is always a key 
factor. Give me some compelling reason to migrate. At the risk of
sounding politically  incorrect: Example, IPV4 works just fine; add 
NAT and you address the major  sticking point and i dont care about
the fact that IPV6 has the insurance that i can slice bread with it
someday when i need to. OTOH, pay me some $$ per packet
(a .01 Canadian penny would do) and i will migrate to IPV6;->

On Sun, 2009-05-17 at 18:01 +0200, Jan Engelhardt wrote:

> For the kernel modules, it uses a separate compat_xtables.c glue
> module that hides the hacks needed for older versions.
> 
> Would tc profit from something similar for libxtables? It would gain
> the capability to work with iptableses potentially older than the
> reference point. But is that truly required? Upgrading userspace is a
> lot easier than the kernel. If a newer tc is installed on one's
> system, the user might just as well do so for iptables in the same
> run.

If you give me a backward compat i would be happier. I think providing
backward compat in general is important for the reason that the API is
public.
If it is not public, then dont put it in libxtables otherwise I will
use it (and you should not complain;->)
The other side of the coin (point i am trying to make to Pablo above) is
once i get to be backward compat, I have zero reason to upgrade.
The only reason i would want to upgrade is the fear factor that in time
my old binaries will not work. And for that it is polite to give the
user ample warnings...

cheers,
jamal

Re: user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[Jan Engelhardt]

On Sunday 2009-05-17 18:48, jamal wrote:
>
>If it is not public, then dont put it in libxtables otherwise I will
>use it (and you should not complain;->)

Then let me submit this one...

parent e55cc4aaa6e35448c14370e5261c3387d26b257d (v1.4.3.2-3-ge55cc4a)
commit ec1794f80034c4cc0c99937d6208dcdaa3d87267
Author: Jan Engelhardt <jengelh@medozas.de>
Date:   Sun May 17 19:01:10 2009 +0200

Move functions out of xtables.c which would contain the public API
---
 Makefile.am                     |   14 +++++++-------
 xtables.c => xtables_internal.c |    0
 2 files changed, 7 insertions(+), 7 deletions(-)
 rename xtables.c => xtables_internal.c (100%)

diff --git a/Makefile.am b/Makefile.am
index fd99098..5c883ba 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -18,17 +18,17 @@ lib_LTLIBRARIES           += libiptc/libiptc.la
 libiptc_libiptc_la_SOURCES = libiptc/libip4tc.c libiptc/libip6tc.c
 libiptc_libiptc_la_LDFLAGS = -version-info 0:0:0
 
-lib_LTLIBRARIES      += libxtables.la
-libxtables_la_SOURCES = xtables.c
-libxtables_la_LDFLAGS = -version-info ${libxtables_vcurrent}:0:${libxtables_vage}
-libxtables_la_LIBADD  = -ldl
+lib_LTLIBRARIES      += libxtables_internal.la
+libxtables_internal_la_SOURCES = xtables_internal.c
+libxtables_internal_la_LDFLAGS = -version-info ${libxtables_vcurrent}:0:${libxtables_vage}
+libxtables_internal_la_LIBADD  = -ldl
 
 # iptables, dynamic
 iptables_SOURCES          = iptables-standalone.c iptables.c
 iptables_LDFLAGS          = -rdynamic
-iptables_LDADD            = libiptc/libiptc.la extensions/libext4.a libxtables.la -lm
+iptables_LDADD            = libiptc/libiptc.la extensions/libext4.a libxtables_internal.la -lm
 
-iptables_xml_LDADD        = libxtables.la
+iptables_xml_LDADD        = libxtables_internal.la
 iptables_multi_SOURCES    = iptables-multi.c iptables-save.c \
                             iptables-restore.c iptables-xml.c \
                             iptables-standalone.c iptables.c
@@ -54,7 +54,7 @@ iptables_xml_SOURCES      = iptables-xml.c
 # ip6tables, dynamic
 ip6tables_SOURCES         = ip6tables-standalone.c ip6tables.c
 ip6tables_LDFLAGS         = -rdynamic
-ip6tables_LDADD           = libiptc/libiptc.la extensions/libext6.a libxtables.la -lm
+ip6tables_LDADD           = libiptc/libiptc.la extensions/libext6.a libxtables_internal.la -lm
 
 ip6tables_multi_SOURCES   = ip6tables-multi.c ip6tables-save.c \
                             ip6tables-restore.c ip6tables-standalone.c \
diff --git a/xtables.c b/xtables_internal.c
similarity index 100%
rename from xtables.c
rename to xtables_internal.c
--==== 

>The other side of the coin (point i am trying to make to Pablo above) is
>once i get to be backward compat, I have zero reason to upgrade.

If you have no reason to upgrade, you could use the exact iptables version
m_ipt was written for.

Re: user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[jamal]

On Sun, 2009-05-17 at 19:06 +0200, Jan Engelhardt wrote:
> On Sunday 2009-05-17 18:48, jamal wrote:
> >
> >If it is not public, then dont put it in libxtables otherwise I will
> >use it (and you should not complain;->)
> 
> Then let me submit this one...

[..]
> If you have no reason to upgrade, you could use the exact iptables version
> m_ipt was written for.

Ok, next steps is to convince all distros to continue support iptables.
If they did there would be no problem.
Surely, apart from trying to make the world a better place, you have to
feel some responsibility in this as well, no?

cheers,
jamal

Re: user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[Pablo Neira Ayuso]

jamal wrote:
> On Sun, 2009-05-17 at 17:40 +0200, Pablo Neira Ayuso wrote:
>> There is an __attribute__((deprecated)) in gcc that displays a message
>> during compilation time to warn that the software is using obsolete
>> interfaces. Then, we can remove it 1 or 2 years later.
> 
> Also something runtime maybe on syslog (example what packet socket
> used to do - complaining about tcpdump until they changed it).
> I did forget to mention one thing which may sound extreme,
> Documentation:
> A document aptly named "ABI breakage" which outlines all revisions and
> what they break. Maybe even an extra document called
> "Deprecation" which announces what is going to break and when.

That seems fine, although I'm not sure that users usually look at these
sort of files. I think that they wait until things break so they have to
put out the fire :).

>> In my case, it's been four years with the current library APIs and
>> following an evolution approach as you have mentioned (adding new
>> functions and obsoleting old ones but keeping them in the tree for a
>> while). Now I'm doing more like a full re-design that needs the
>> "revolution", no way to keep backward compatibility to resolve several
>> issues.
> 
> I think there is room for revolutions with the caveat of ample warnings.
> Like you said if you have been giving warning for 2 years about
> something deprecating then you are absolved of dropping the interface.
> 
>> Still, people would be able to have both version 1 and 2 of the
>> libraries installed in their computer so they can keep attached to old
>> versions without breaking binary backward compatibility. I've been
>> discussing this with a friend of mine that maintains a couple of
>> critical libraries in the gnome project, it was nice to expose him my
>> ideas and see that I was on the right track.
> 
> Good motivation to migrate from version 1 to 2 is always a key 
> factor. Give me some compelling reason to migrate. At the risk of
> sounding politically  incorrect: Example, IPV4 works just fine; add 
> NAT and you address the major  sticking point and i dont care about
> the fact that IPV6 has the insurance that i can slice bread with it
> someday when i need to. OTOH, pay me some $$ per packet
> (a .01 Canadian penny would do) and i will migrate to IPV6;->

Well, shiny new features is the only thing that I can "sell" in this
case. Something like: "if you migrate from version 1 to 2, you will get
these set of new features in return". Of course, if they don't need
them, it's a hard sell ;).

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[Jan Engelhardt]

On Sunday 2009-05-17 16:46, Pablo Neira Ayuso wrote:
>jamal wrote:
>> On Sat, 2009-05-16 at 12:37 +0200, Pablo Neira Ayuso wrote:
>>
>> maybe we can have some simple working relationship rules:
>> 
>> a) whoever breaks ABI (hopefully there are good reasons
>> to intentionally break it) should also fix ipt, just like they
>> fix iptables. 
>> b) whoever breaks the ABI should consider both backward and
>> forward compatibility. I will review patches to make sure this
>> is still so.
>
>I think that, now that we have a public API, ABI should be not broken.
>In case that some interface has limitations, we can add a new one while
>keeping the old. The result is not nice since you end having two
>functions that do the same with different interfaces but this doesn't
>break binary backward compatibility. Of course, the change would need
>some justification, eg. some client code do need something that the
>current interface does not allow in any way.
>
>With this policy, design errors accumulate along time so we learn the
>lesson from our own mistakes and, then, we work on a new version 2 of
>the API to resolve the accumulated issues after some time. This is how
>I'm managing existing netfilter libraries. This policy makes the
>developement of libraries/public interfaces slower but I think that
>users are way happier (no binary breakages).

With that policy we end up with the same crap that is happening
in the kernel.

Just because we magically managed to stuff lots of functions into an
.so file does not make it public. It just so happens to save a bunch
of kilobytes in all of the binaries that were previously statically
linked to xtables.c, and was deemed one way to figure out how to deal
with intrusive m_ipt. Sorry, but in all efforts that went in so far,
I discharge libxtables having a stable API. For that, the iptables
code is not beauty enough yet.

Now, I had to just think of Xtables-addons that has a similar issue.
For the kernel modules, it uses a separate compat_xtables.c glue
module that hides the hacks needed for older versions.

Would tc profit from something similar for libxtables? It would gain
the capability to work with iptableses potentially older than the
reference point. But is that truly required? Upgrading userspace is a
lot easier than the kernel. If a newer tc is installed on one's
system, the user might just as well do so for iptables in the same
run.

Re: user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[Pablo Neira Ayuso]

Jan Engelhardt wrote:
> On Sunday 2009-05-17 16:46, Pablo Neira Ayuso wrote:
>> With this policy, design errors accumulate along time so we learn the
>> lesson from our own mistakes and, then, we work on a new version 2 of
>> the API to resolve the accumulated issues after some time. This is how
>> I'm managing existing netfilter libraries. This policy makes the
>> developement of libraries/public interfaces slower but I think that
>> users are way happier (no binary breakages).
> 
> With that policy we end up with the same crap that is happening
> in the kernel.

I didn't say that this was nice, it's functional :)

> Just because we magically managed to stuff lots of functions into an
> .so file does not make it public. It just so happens to save a bunch
> of kilobytes in all of the binaries that were previously statically
> linked to xtables.c, and was deemed one way to figure out how to deal
> with intrusive m_ipt. Sorry, but in all efforts that went in so far,
> I discharge libxtables having a stable API. For that, the iptables
> code is not beauty enough yet.

I think that we should prioritize backward compatibility versus beauty.

> Now, I had to just think of Xtables-addons that has a similar issue.

Indeed, actually I think that a stable ABI would make this easier for you.

> For the kernel modules, it uses a separate compat_xtables.c glue
> module that hides the hacks needed for older versions.
> 
> Would tc profit from something similar for libxtables? It would gain
> the capability to work with iptableses potentially older than the
> reference point. But is that truly required? Upgrading userspace is a
> lot easier than the kernel. If a newer tc is installed on one's
> system, the user might just as well do so for iptables in the same
> run.

Sorry, I don't like that policy at all.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

Re: user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[Jan Engelhardt]

On Sunday 2009-05-17 18:12, Pablo Neira Ayuso wrote:
>Jan wrote:
>>jamal wrote:
>>> With this policy, design errors accumulate along time so we learn the        
>>> lesson from our own mistakes and[...]
>>                                                                               
>> With that policy we end up with the same crap that is happening               
>> in the kernel.                                                                
>                                                                                
>I didn't say that this was nice, it's functional :)

I am saying if we had learnt something then we should not do the same
in userspace.

>> Just because we magically managed to stuff lots of functions into an
>> .so file does not make it public. It just so happens to save a bunch
>> of kilobytes in all of the binaries that were previously statically
>> linked to xtables.c, and was deemed one way to figure out how to deal
>> with intrusive m_ipt. Sorry, but in all efforts that went in so far,
>> I discharge libxtables having a stable API. For that, the iptables
>> code is not beauty enough yet.
>
>I think that we should prioritize backward compatibility versus beauty.

And put a halt on evolvement. Compatibility is the ultimate killer
argument against about anything. Not that I am fuzzing over the
smallish int->bool changes, but potential larger changes.

>> Now, I had to just think of Xtables-addons that has a similar issue.
>
>Indeed, actually I think that a stable ABI would make this easier for you.

Not really. The currently release cycle of iptables is long enough.

Re: user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[jamal]

On Sun, 2009-05-17 at 18:39 +0200, Jan Engelhardt wrote:

> Compatibility is the ultimate killer argument against about anything. 

Once you have user of an API, that arguement becomes hard to defend.
It is certainly a lesson learnt on the development of the
internet: You cant just replace IPV4, TCP, NAT and a lot of other
protocols and practises that people are using. You can incrementally
"fix them". You can the parallel provide the new way (IPv6).

cheers,
jamal

Re: user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[Jan Engelhardt]

On Sunday 2009-05-17 18:59, jamal wrote:
>On Sun, 2009-05-17 at 18:39 +0200, Jan Engelhardt wrote:
>
>> Compatibility is the ultimate killer argument against about anything. 
>
>Once you have user of an API, that arguement becomes hard to defend.
>It is certainly a lesson learnt on the development of the
>internet: You cant just replace IPV4, TCP, NAT and a lot of other
>protocols and practises that people are using. You can incrementally
>"fix them". You can the parallel provide the new way (IPv6).

The hurdle with IPv6 is just like patches that would add a copy of a
function with a slightly different signature or semantics: it would
be met with reluctance. I already given up on submission of
Xtables-addons modules to upstream for the same reason because there
are so much "no no"s coming back. I don't get it.

Re: user-space xtables ABI [was Re: [Fwd: Re: iptables pull request]]

[jamal]

On Sun, 2009-05-17 at 19:11 +0200, Jan Engelhardt wrote:

> The hurdle with IPv6 is just like patches that would add a copy of a
> function with a slightly different signature or semantics: 
> it would be met with reluctance. 

I think v6 is met with reluctance because it is a _revolution_.
It is a slight exageration on my part to equate the API churning
challenge to be of the same magnitude as protocols deployed in the
internet, however, conceptually the pain for someone who has to deal
with API changes is similar.
Imagine a user re-learning to remember the v6 addressing semantics
(analogy would be a user of an API having to learn new semantics).
And then factor in people who have already invested a gazillion $ and
time in V4 applications, equipment and infrastructure (anology will be
some corp who has deployed applications using old API). 
NAT, OTOH, would be analogous to a patch on top of V4 which solves an
immediate problem - hence an evolution. NAT is so popular that even the
IETF has given up on trying to get the cat back into the bag.

What i am trying to say is: netfilter (hate to use marketing-speak) has
created an ecosystem; and with huge power comes huge responsibility;
every time ypu make small changes like APIs, you are providing guidance
to the distros about what you are no longer willing to support and
unintentionally screwing me (make me do more work).

> I already given up on submission of
> Xtables-addons modules to upstream for the same reason because there
> are so much "no no"s coming back. I don't get it.

It doesnt sound right to reject them if they dont kill backward compat.

cheers,
jamal