From: Pablo Neira Ayuso <pablo@netfilter.org>
To: "Richard Mörbitz" <richard.moerbitz@tu-dresden.de>
Cc: netfilter-devel@vger.kernel.org
Subject: Re: Adding element to interval map consumes entire memory
Date: Tue, 13 Dec 2016 01:48:49 +0100 [thread overview]
Message-ID: <20161213004849.GA15100@salvia> (raw)
In-Reply-To: <3acabb81-c5b5-2004-18ce-8b5242f07921@tu-dresden.de>
Hi Richard,
On Mon, Dec 12, 2016 at 04:43:33PM +0100, Richard Mörbitz wrote:
>
> > interval code is buggy, I remember to have seen a large memory
> > allocation being triggered in libgmp calls.
>
> These allocations are triggered from the expr_to_intervals function in
> segtree.c - three times, 500 MB each. I have attached the full valgrind
> leak summary to the mail.
I think I found the problem, we have an underflow triggering the
allocation of a huge bitmask, see patch:
http://patchwork.ozlabs.org/patch/705279/
Quickly tested with your example ruleset.
BTW, if you have some spare cycles, I would really appreciate if you
can send a shell test, similar to:
nftables/tests/shell/testcases/sets/0012add_delete_many_elements_0
nftables/tests/shell/testcases/sets/0013add_delete_many_elements_0
It would be great to cover intervals and maps too.
> I also want to point out that calculating overlapping intervals has
> bugs; trying to add a non-overlapping interval can result in the error
> "interval overlaps with an existing one" (function set_overlap,
> segtree.c). However, this should probably become a different thread.
Are you running nft from git.netfilter.org? I just would like to make
sure you're not seeing anything that is already fixed.
I have also posted this patch:
http://patchwork.ozlabs.org/patch/705278/
So nft doesn't complain on exact overlaps to keep it consistent with
non-interval sets. Probably you refering to this?
> > If you can hand over an example that I can use to reproduce I'd
> > appreciate, I understand this may require some confidentiality, so
> > feel free to send me a file with randomized addresses or such.
>
> I have attached a dummy ruleset that represents the one we use in size
> and shape. You can read it (nft -f test.ruleset) without problems. If
> you attempt to add another map element (say, nft add element nat2
> subnettoip {0.0.0.0/24: 0.0.0.0}) you get the error I have described.
> Of course it depends on the memory of the machine you are using, but you
> should see memory consumption going up drastically.
Thanks for providing the example to reproduce it.
next prev parent reply other threads:[~2016-12-13 0:49 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-12-11 14:20 Adding element to interval map consumes entire memory Richard Mörbitz
2016-12-11 19:28 ` Pablo Neira Ayuso
[not found] ` <3acabb81-c5b5-2004-18ce-8b5242f07921@tu-dresden.de>
2016-12-13 0:48 ` Pablo Neira Ayuso [this message]
2016-12-14 23:52 ` Richard Mörbitz
2016-12-15 22:02 ` Pablo Neira Ayuso
2016-12-15 22:47 ` Pablo Neira Ayuso
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20161213004849.GA15100@salvia \
--to=pablo@netfilter.org \
--cc=netfilter-devel@vger.kernel.org \
--cc=richard.moerbitz@tu-dresden.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).