Re: Adding element to interval map consumes entire memory

[Pablo Neira Ayuso <pablo@netfilter.org>]

Hi Richard,

On Mon, Dec 12, 2016 at 04:43:33PM +0100, Richard Mörbitz wrote:
> 
> > interval code is buggy, I remember to have seen a large memory
> > allocation being triggered in libgmp calls.
> 
> These allocations are triggered from the expr_to_intervals function in
> segtree.c - three times, 500 MB each. I have attached the full valgrind
> leak summary to the mail.

I think I found the problem, we have an underflow triggering the
allocation of a huge bitmask, see patch:

http://patchwork.ozlabs.org/patch/705279/

Quickly tested with your example ruleset.

BTW, if you have some spare cycles, I would really appreciate if you
can send a shell test, similar to:

nftables/tests/shell/testcases/sets/0012add_delete_many_elements_0
nftables/tests/shell/testcases/sets/0013add_delete_many_elements_0

It would be great to cover intervals and maps too.

> I also want to point out that calculating overlapping intervals has
> bugs; trying to add a non-overlapping interval can result in the error
> "interval overlaps with an existing one" (function set_overlap,
> segtree.c). However, this should probably become a different thread.

Are you running nft from git.netfilter.org? I just would like to make
sure you're not seeing anything that is already fixed.

I have also posted this patch:

http://patchwork.ozlabs.org/patch/705278/

So nft doesn't complain on exact overlaps to keep it consistent with
non-interval sets. Probably you refering to this?

> > If you can hand over an example that I can use to reproduce I'd
> > appreciate, I understand this may require some confidentiality, so
> > feel free to send me a file with randomized addresses or such.
> 
> I have attached a dummy ruleset that represents the one we use in size
> and shape. You can read it (nft -f test.ruleset) without problems. If
> you attempt to add another map element (say, nft add element nat2
> subnettoip {0.0.0.0/24: 0.0.0.0}) you get the error I have described.
> Of course it depends on the memory of the machine you are using, but you
> should see memory consumption going up drastically.

Thanks for providing the example to reproduce it.