netfilter-devel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Pablo Neira Ayuso <pablo@netfilter.org>
To: "Richard Mörbitz" <richard.moerbitz@tu-dresden.de>
Cc: netfilter-devel@vger.kernel.org
Subject: Re: Adding element to interval map consumes entire memory
Date: Tue, 13 Dec 2016 01:48:49 +0100	[thread overview]
Message-ID: <20161213004849.GA15100@salvia> (raw)
In-Reply-To: <3acabb81-c5b5-2004-18ce-8b5242f07921@tu-dresden.de>

Hi Richard,

On Mon, Dec 12, 2016 at 04:43:33PM +0100, Richard Mörbitz wrote:
> 
> > interval code is buggy, I remember to have seen a large memory
> > allocation being triggered in libgmp calls.
> 
> These allocations are triggered from the expr_to_intervals function in
> segtree.c - three times, 500 MB each. I have attached the full valgrind
> leak summary to the mail.

I think I found the problem, we have an underflow triggering the
allocation of a huge bitmask, see patch:

http://patchwork.ozlabs.org/patch/705279/

Quickly tested with your example ruleset.

BTW, if you have some spare cycles, I would really appreciate if you
can send a shell test, similar to:

nftables/tests/shell/testcases/sets/0012add_delete_many_elements_0
nftables/tests/shell/testcases/sets/0013add_delete_many_elements_0

It would be great to cover intervals and maps too.

> I also want to point out that calculating overlapping intervals has
> bugs; trying to add a non-overlapping interval can result in the error
> "interval overlaps with an existing one" (function set_overlap,
> segtree.c). However, this should probably become a different thread.

Are you running nft from git.netfilter.org? I just would like to make
sure you're not seeing anything that is already fixed.

I have also posted this patch:

http://patchwork.ozlabs.org/patch/705278/

So nft doesn't complain on exact overlaps to keep it consistent with
non-interval sets. Probably you refering to this?

> > If you can hand over an example that I can use to reproduce I'd
> > appreciate, I understand this may require some confidentiality, so
> > feel free to send me a file with randomized addresses or such.
> 
> I have attached a dummy ruleset that represents the one we use in size
> and shape. You can read it (nft -f test.ruleset) without problems. If
> you attempt to add another map element (say, nft add element nat2
> subnettoip {0.0.0.0/24: 0.0.0.0}) you get the error I have described.
> Of course it depends on the memory of the machine you are using, but you
> should see memory consumption going up drastically.

Thanks for providing the example to reproduce it.

  parent reply	other threads:[~2016-12-13  0:49 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-12-11 14:20 Adding element to interval map consumes entire memory Richard Mörbitz
2016-12-11 19:28 ` Pablo Neira Ayuso
     [not found]   ` <3acabb81-c5b5-2004-18ce-8b5242f07921@tu-dresden.de>
2016-12-13  0:48     ` Pablo Neira Ayuso [this message]
2016-12-14 23:52       ` Richard Mörbitz
2016-12-15 22:02         ` Pablo Neira Ayuso
2016-12-15 22:47           ` Pablo Neira Ayuso

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20161213004849.GA15100@salvia \
    --to=pablo@netfilter.org \
    --cc=netfilter-devel@vger.kernel.org \
    --cc=richard.moerbitz@tu-dresden.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).