ipset save and restore

ipset save and restore

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1271 bytes --]

Hi.

I wanted to use ipset (ipset v6.11, protocol version: 6) to load it's
sets from a cluster wide distributed file, e.g. at boot, or every time
that file changes.


Now unfortunately it seems that ipset restore doesn't work as e.g.
iptables restore does and seems to me therefore pretty much useless.

ipset restore < file
gives me errors about the sets already existing, but even with -exist it
doesn't help a lot, because entries removed from the file, are not
removed from the actual ipsets.

So it seems as if ipset restore is not what the manpage describes
(restore a session) but rather an additive merge of another session to
the current one.

Of course I understand that it could not delete sets which are in use,
but at least it could empty them.


Now when I use the following instead:
ipset flush
ipset destroy
ipset restore < file

To first flush all entries possibly no longer in the current version of
file and to destroy all sets that got removed and that are no longer
used... and only then reload the rules.... I have of course some time
where even the sets that are in use are empty... and my connections will
fail.


So is there some workaround or who is the whole thing intended to be
used?


Thanks,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3811 bytes --]

Re: ipset save and restore

[Jozsef Kadlecsik]

On Wed, 19 Dec 2012, Christoph Anton Mitterer wrote:

> I wanted to use ipset (ipset v6.11, protocol version: 6) to load it's
> sets from a cluster wide distributed file, e.g. at boot, or every time
> that file changes.

That's two different operations: first the sets do not exist and in the 
second case those exist (and in use).
 
> Now unfortunately it seems that ipset restore doesn't work as e.g.
> iptables restore does and seems to me therefore pretty much useless.
> 
> ipset restore < file
> gives me errors about the sets already existing, but even with -exist it
> doesn't help a lot, because entries removed from the file, are not
> removed from the actual ipsets.
> 
> So it seems as if ipset restore is not what the manpage describes
> (restore a session) but rather an additive merge of another session to
> the current one.
>
> Of course I understand that it could not delete sets which are in use,
> but at least it could empty them.

Restore mode is flexible. If you want the sets to be emptied first, then 
start with the flush command (in the file).
 
> Now when I use the following instead:
> ipset flush
> ipset destroy
> ipset restore < file

Why do you destroy the sets? If the sets are in use then you cannot delete 
them at all.
 
> To first flush all entries possibly no longer in the current version of
> file and to destroy all sets that got removed and that are no longer
> used... and only then reload the rules.... I have of course some time
> where even the sets that are in use are empty... and my connections will
> fail.

If you are not concerned that for a very small time the sets are empty, 
then simply start with the flush command in the restore file and that's 
all (I don't see why you'd want to destroy those).

If you want to avoid the time while the sets are empty, then use the 
sequence of

- restore into a temporary set
- swap the set with the temporary one
- destroy the temporary set

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset save and restore

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 2114 bytes --]

Hey Jozsef. 

On Wed, 2012-12-19 at 22:01 +0100, Jozsef Kadlecsik wrote:
> > Of course I understand that it could not delete sets which are in use,
> > but at least it could empty them.
> Restore mode is flexible. If you want the sets to be emptied first, then 
> start with the flush command (in the file).
Phew... I'd rather say it's less powerful... if someone really want's
fine grained control, he can still use the single commands.

restore is even documented in the ipset manpage to do what e.g. iptables
restore does... getting the session back... but it doesn't.
I think that should definitely be documented better.


> > Now when I use the following instead:
> > ipset flush
> > ipset destroy
> > ipset restore < file
> 
> Why do you destroy the sets?
Well if you have long running systems and sets get unused one probably
want's to release their memory.

btw: "ipset destroy" fails if a single set still contains entries...
wouldn't it be better if all empty sets are destroyed and it only fails
on those which are not?



> If the sets are in use then you cannot delete 
> them at all.
I understand that this might be necessary from a technical point of
view,.. but wouldn't it be simply possible to consider non-existing sets
to be empty sets?


> If you are not concerned that for a very small time the sets are empty, 
> then simply start with the flush command in the restore file and that's 
> all (I don't see why you'd want to destroy those).
Well I'd have expected that everybody would be concerned... when you
have some high speed network you surely will loose packets...


> If you want to avoid the time while the sets are empty, then use the 
> sequence of
> 
> - restore into a temporary set
> - swap the set with the temporary one
> - destroy the temporary set
Thanks... this is at least a way... but unfortunately someone that
implies some additional programming effort...

Actually the above would be what one would expect from the restore
command, I guess.

No chance that the semantics are changed here?



Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3811 bytes --]

Re: ipset save and restore

[Jozsef Kadlecsik]

On Wed, 19 Dec 2012, Christoph Anton Mitterer wrote:

> On Wed, 2012-12-19 at 22:01 +0100, Jozsef Kadlecsik wrote:
> > > Of course I understand that it could not delete sets which are in use,
> > > but at least it could empty them.
> > Restore mode is flexible. If you want the sets to be emptied first, then 
> > start with the flush command (in the file).
> Phew... I'd rather say it's less powerful... if someone really want's
> fine grained control, he can still use the single commands.
> 
> restore is even documented in the ipset manpage to do what e.g. iptables
> restore does... getting the session back... but it doesn't.
> I think that should definitely be documented better.

Restore executes the commands specified in the saved session file. If the 
current state is not appropriate for the commands (set already exist, 
etc.), then it'll fail. I'll add this to the manpage.
  
> > > Now when I use the following instead:
> > > ipset flush
> > > ipset destroy
> > > ipset restore < file
> > 
> > Why do you destroy the sets?
> Well if you have long running systems and sets get unused one probably
> want's to release their memory.

But then why do you want to restore them, if those sets are unused?
 
> btw: "ipset destroy" fails if a single set still contains entries...

That's not true. Maybe you mix "set contains entries" with "set used in a 
rule"?

> wouldn't it be better if all empty sets are destroyed and it only fails
> on those which are not?

I don't understand your sentence at all...
 
> > If the sets are in use then you cannot delete 
> > them at all.
> I understand that this might be necessary from a technical point of
> view,.. but wouldn't it be simply possible to consider non-existing sets
> to be empty sets?

Non-existing sets are ... non-existent. 
  
> > If you are not concerned that for a very small time the sets are empty, 
> > then simply start with the flush command in the restore file and that's 
> > all (I don't see why you'd want to destroy those).
> Well I'd have expected that everybody would be concerned... when you
> have some high speed network you surely will loose packets...

That can happen for new connections only (unless you don't use conntrack 
at all.)
 
> > If you want to avoid the time while the sets are empty, then use the 
> > sequence of
> > 
> > - restore into a temporary set
> > - swap the set with the temporary one
> > - destroy the temporary set
> Thanks... this is at least a way... but unfortunately someone that
> implies some additional programming effort...
> 
> Actually the above would be what one would expect from the restore
> command, I guess.
> 
> No chance that the semantics are changed here?

No, not really.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset save and restore

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 4096 bytes --]

On Thu, 2012-12-20 at 00:24 +0100, Jozsef Kadlecsik wrote:
> > restore is even documented in the ipset manpage to do what e.g. iptables
> > restore does... getting the session back... but it doesn't.
> > I think that should definitely be documented better.
> Restore executes the commands specified in the saved session file. If the 
> current state is not appropriate for the commands (set already exist, 
> etc.), then it'll fail. I'll add this to the manpage.
The manpage says:
Restore a saved session generated by save. The save session can be fed
from stdin.

From this I personally would expect the same behaviour as what I get
from iptables-restore,... which includes e.g. "deleting" no longer used
entries.


> > Well if you have long running systems and sets get unused one probably
> > want's to release their memory.
> But then why do you want to restore them, if those sets are unused?
Uhm? I don't want to restore them...
I have a set file, which get's changed in some way... e.g. new sets and
entries added,... some sets and entries removed.
Then I distribute this file across the cluster and want to have it
loaded.


> > btw: "ipset destroy" fails if a single set still contains entries...
> That's not true. Maybe you mix "set contains entries" with "set used in a 
> rule"?
Yes and now ;) ... at least when with rule you mean an iptables rules

I have ipsets that are not used in any iptables rules at all.
But the some sets have References, because they are included in another
of type list:set .

So none of the sets is really used by iptables, and I'd expect I could
destroy them.
But then it says: "Set cannot be destroyed: it is in use by a kernel
component" (which is obviously ipsets itself).
Of course when first doing a flush... it succeeds as you say, because
all list:set sets are emptied.



> > wouldn't it be better if all empty sets are destroyed and it only fails
> > on those which are not?
> I don't understand your sentence at all...
In the above case I had e.g.
setA of type list:set containing (setB and setC).
setB of type bitmap:ip containing nothing
setC of type bitmap:ip containing nothing
setD of type bitmap:ip containing nothing
(non of them used anyhow in iptables rules)

If I know call ipset destroy,.. it fails with the aforementioned
error... and all sets exist as before, while at least setA and setD
could have been destroyed as they is nowhere used.


> > > If the sets are in use then you cannot delete 
> > > them at all.
> > I understand that this might be necessary from a technical point of
> > view,.. but wouldn't it be simply possible to consider non-existing sets
> > to be empty sets?
> Non-existing sets are ... non-existent.
What I meant is:
Ideally it shouldn't be impossible to delete a set that is in use by
iptables rules... such a set could be "deleted" and the iptables set
match and target should simply consider such a set to be empty and
neither fill it with new entries.


> > > If you are not concerned that for a very small time the sets are empty, 
> > > then simply start with the flush command in the restore file and that's 
> > > all (I don't see why you'd want to destroy those).
> > Well I'd have expected that everybody would be concerned... when you
> > have some high speed network you surely will loose packets...
> 
> That can happen for new connections only (unless you don't use conntrack 
> at all.)
Which is already a major limitation, IMHO,... and it may not be desired
or possible to do connection tracking in each situation...

 
> > Actually the above would be what one would expect from the restore
> > command, I guess.
> > 
> > No chance that the semantics are changed here?
> 
> No, not really.
Any special reason? Nothing speaks at least against adding another
restore command which does a proper restore.
The current restore seems to be not much more then a
sed "s/^/ipset /" | sh
and I guess anybody can do this himself, not needing a special ipset
restore command for it.


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3811 bytes --]

Re: ipset save and restore

[Jozsef Kadlecsik]

On Thu, 20 Dec 2012, Christoph Anton Mitterer wrote:

> On Thu, 2012-12-20 at 00:24 +0100, Jozsef Kadlecsik wrote:
> > > restore is even documented in the ipset manpage to do what e.g. iptables
> > > restore does... getting the session back... but it doesn't.
> > > I think that should definitely be documented better.
> > Restore executes the commands specified in the saved session file. If the 
> > current state is not appropriate for the commands (set already exist, 
> > etc.), then it'll fail. I'll add this to the manpage.
> The manpage says:
> Restore a saved session generated by save. The save session can be fed
> from stdin.
> 
> From this I personally would expect the same behaviour as what I get
> from iptables-restore,... which includes e.g. "deleting" no longer used
> entries.

No, "ipset restore" is not identical with iptables-restore. For example 
with the latter you are not able to incrementally update existing 
rulesets.
 
> > > Well if you have long running systems and sets get unused one probably
> > > want's to release their memory.
> > But then why do you want to restore them, if those sets are unused?
> Uhm? I don't want to restore them...
> I have a set file, which get's changed in some way... e.g. new sets and
> entries added,... some sets and entries removed.
> Then I distribute this file across the cluster and want to have it
> loaded.

Then you know exactly which sets are unused and can selectively destroy 
those. Or you can flush/destroy all of them and recreate. Or destroy the 
unused one and swap the used ones with new sets.
  
> > > btw: "ipset destroy" fails if a single set still contains entries...
> > That's not true. Maybe you mix "set contains entries" with "set used in a 
> > rule"?
> Yes and now ;) ... at least when with rule you mean an iptables rules
> 
> I have ipsets that are not used in any iptables rules at all.
> But the some sets have References, because they are included in another
> of type list:set .

You did not mention that you use list type of sets... Those keep 
references to the elements, yes, and the elements cannot be destroyed 
while they are members.
 
> So none of the sets is really used by iptables, and I'd expect I could
> destroy them.

No, because those are members of a set. 

> But then it says: "Set cannot be destroyed: it is in use by a kernel
> component" (which is obviously ipsets itself).
> Of course when first doing a flush... it succeeds as you say, because
> all list:set sets are emptied.
> 
> > > wouldn't it be better if all empty sets are destroyed and it only fails
> > > on those which are not?
> > I don't understand your sentence at all...
> In the above case I had e.g.
> setA of type list:set containing (setB and setC).
> setB of type bitmap:ip containing nothing
> setC of type bitmap:ip containing nothing
> setD of type bitmap:ip containing nothing
> (non of them used anyhow in iptables rules)
> 
> If I know call ipset destroy,.. it fails with the aforementioned
> error... and all sets exist as before, while at least setA and setD
> could have been destroyed as they is nowhere used.

If you want to destroy all those sets, then first you have to flush at 
least setA.

> > > > If the sets are in use then you cannot delete 
> > > > them at all.
> > > I understand that this might be necessary from a technical point of
> > > view,.. but wouldn't it be simply possible to consider non-existing sets
> > > to be empty sets?
> > Non-existing sets are ... non-existent.
> What I meant is:
> Ideally it shouldn't be impossible to delete a set that is in use by
> iptables rules... such a set could be "deleted" and the iptables set
> match and target should simply consider such a set to be empty and
> neither fill it with new entries.

No, that'd be totally incosistent, no way.
 
> > > > If you are not concerned that for a very small time the sets are empty, 
> > > > then simply start with the flush command in the restore file and that's 
> > > > all (I don't see why you'd want to destroy those).
> > > Well I'd have expected that everybody would be concerned... when you
> > > have some high speed network you surely will loose packets...
> > 
> > That can happen for new connections only (unless you don't use conntrack 
> > at all.)
> Which is already a major limitation, IMHO,... and it may not be desired
> or possible to do connection tracking in each situation...
>   
> > > Actually the above would be what one would expect from the restore
> > > command, I guess.
> > > 
> > > No chance that the semantics are changed here?
> > 
> > No, not really.
> Any special reason? Nothing speaks at least against adding another
> restore command which does a proper restore.
> The current restore seems to be not much more then a
> sed "s/^/ipset /" | sh
> and I guess anybody can do this himself, not needing a special ipset
> restore command for it.

Nothing prevents you to write a few liner in any scripting language, which 
either adds a "flush" command or wraps "create, add" into "create temp, 
add to temp, swap, destroy temp".

ipset is a basic tool to manipulate the sets and it's an interface to all 
low level commands, nothing more.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset save and restore

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1843 bytes --]

On Thu, 2012-12-20 at 13:00 +0100, Jozsef Kadlecsik wrote:
> No, "ipset restore" is not identical with iptables-restore. For example 
> with the latter you are not able to incrementally update existing 
> rulesets.
Well incrementally updating would also mean removing entries that are no
longer existing.... otherwise it's only making a union.

And as I said before... anyone who wants to do this can easily sed
"s/^/ipset //" | sh


> Nothing prevents you to write a few liner in any scripting language, which 
> either adds a "flush" command or wraps "create, add" into "create temp, 
> add to temp, swap, destroy temp".
Of course one can do this and actually I'm doing right now... but it's
quite unhandy if anybody has to do this just to get a proper restore.

And when doing it "proper" it's not just a one liner... it has to work
with different names for the actions like create, creat, crea, etc. and
one always risks to break things (and even get security issues by that)
when the syntax of ipset should change or be extended.
With temp names there's the issue, that (I guess) there is a maximum
name for set names, so one cannot just add all new sets as
tmp_<originalname> as this might already fail

Last but not least, while this allows atomic swaps of single set... one
cannot swap the whole ipset configuration in one atomic step.


ipset's current restore should actually be called exec, or merge, or
load...




btw: I might have noticed a bug(?) in at least 6.11
With the following output of store:
# ipset list | grep ^Name

I'd have expected to get the same sets with
list -names
but it gives me one twice:
# ipset list -name
lrz_lcg_se_cn
lrz_lcg_se_pn
lrz_lcg_ce_cn
lrz_lcg_ce_compn
lrz_lcg_ce_compn
lrz_lcg_se
lrz_lcg_ce
DROPPED

Is that expected?


Thanks,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3811 bytes --]

Re: ipset save and restore

[Jozsef Kadlecsik]

On Thu, 20 Dec 2012, Christoph Anton Mitterer wrote:

> On Thu, 2012-12-20 at 13:00 +0100, Jozsef Kadlecsik wrote:
> > No, "ipset restore" is not identical with iptables-restore. For example 
> > with the latter you are not able to incrementally update existing 
> > rulesets.
> Well incrementally updating would also mean removing entries that are no
> longer existing.... otherwise it's only making a union.

By "update" I meant adding/deleting entries, creating/swapping/destroying 
sets.
 
> And as I said before... anyone who wants to do this can easily sed
> "s/^/ipset //" | sh
>  
> > Nothing prevents you to write a few liner in any scripting language, which 
> > either adds a "flush" command or wraps "create, add" into "create temp, 
> > add to temp, swap, destroy temp".
> Of course one can do this and actually I'm doing right now... but it's
> quite unhandy if anybody has to do this just to get a proper restore.
> 
> And when doing it "proper" it's not just a one liner... it has to work
> with different names for the actions like create, creat, crea, etc. and
> one always risks to break things (and even get security issues by that)
> when the syntax of ipset should change or be extended.
> With temp names there's the issue, that (I guess) there is a maximum
> name for set names, so one cannot just add all new sets as
> tmp_<originalname> as this might already fail
> 
> Last but not least, while this allows atomic swaps of single set... one
> cannot swap the whole ipset configuration in one atomic step. 

That's not possible in the current framework: individual sets can be 
changed atomically by swapping (not counting add/del elements), whole 
config doesn't.
 
> ipset's current restore should actually be called exec, or merge, or
> load...

Maybe, but I'm not going to change it, in order to keep backward 
compatibility.
 
> btw: I might have noticed a bug(?) in at least 6.11
> With the following output of store:
> # ipset list | grep ^Name
> 
> I'd have expected to get the same sets with
> list -names
> but it gives me one twice:
> # ipset list -name
> lrz_lcg_se_cn
> lrz_lcg_se_pn
> lrz_lcg_ce_cn
> lrz_lcg_ce_compn
> lrz_lcg_ce_compn
> lrz_lcg_se
> lrz_lcg_ce
> DROPPED
> 
> Is that expected?

It's seems not right, but it's hard to tell without the actual sets.
Send me the data in private so that I could reproduce it.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary