Re: Adding element to interval map consumes entire memory

[Pablo Neira Ayuso <pablo@netfilter.org>]

On Sun, Dec 11, 2016 at 03:20:16PM +0100, Richard Mörbitz wrote:
> Hello,
> 
> first of all: the build in use is the current master
> (c89a0801d07740eff531412fe35ea2c9faad82b0).
> 
> We have a test setup running which consists of one table ("nat2") and an
> interval mapping ("subnettoip") of the type ipv4_addr -> ipv4_addr. The
> mapping is quite large (~16000 elements). One requirement of the system
> is that elements can be added and deleted during runtime.
> 
> With that map constructed, adding a new element is not possible. NFT
> will terminate during reallocation, because no memory is left.
> 
> GDB trace:
> http://pastebin.com/s7eyNEsH
> 
> Valgrind leak check:
> http://pastebin.com/fkG5UQig
> 
> Note that the test machine only has 2 GB of RAM, 1.6 of which are
> consumed by nft. So one question is: is it even possible to have enough
> memory in the final system, such that the required operation can be
> performed?
> 
> The second question would be: why is it necessary to allocate that much
> memory? As I have found out by reading the code, all map elemtents are
> cached before performing the operation; they are even sorted. Is that
> really necessary for operations like adding map entries?

interval code is buggy, I remember to have seen a large memory
allocation being triggered in libgmp calls.

If you can hand over an example that I can use to reproduce I'd
appreciate, I understand this may require some confidentiality, so
feel free to send me a file with randomized addresses or such.

Thanks.