Re: [PATCH nft] json: collapse set element commands from parser

[Pablo Neira Ayuso <pablo@netfilter.org>]

Hi Phil,

On Tue, Nov 12, 2024 at 09:52:35PM +0100, Phil Sutter wrote:
> Hi Pablo,
> 
> On Thu, Oct 31, 2024 at 11:04:11PM +0100, Pablo Neira Ayuso wrote:
> > Side note: While profiling, I can still see lots json objects, this
> > results in memory consumption that is 5 times than native
> > representation. Error reporting is also lagging behind, it should be
> > possible to add a json_t pointer to struct location to relate
> > expressions and json objects.
> 
> I can't quite reproduce this. When restoring a ruleset with ~12.7k
> elements in individual standard syntax commands, valgrind prints:
> 
> | HEAP SUMMARY:
> |     in use at exit: 59,802 bytes in 582 blocks
> |   total heap usage: 954,970 allocs,
> |                     954,388 frees,
> |                  18,300,874 bytes allocated
> 
> Repeating the same in JSON syntax, I get:
> 
> | HEAP SUMMARY:
> |     in use at exit: 61,592 bytes in 647 blocks
> |   total heap usage: 1,200,164 allocs,
> |                     1,199,517 frees,
> |                    38,612,257 bytes allocated
> 
> So this is 38MB vs 18MB? At least far from the mentioned 5 times. Would
> you mind sharing how you got to that number?
> 
> Please kindly find my reproducers attached for reference.

I am using valgrind --tool=massif to measure memory consumption in
userspace.

I used these two files:

- set-init.json-nft, to create the table and set.
- set-65535.nft-json, to create a small set with 64K elements.

then I run:

valgrind --tool=massif nft -f set-65535.nft-json

there is a tool:

ms_print massif.out.XYZ

At "peak time" in heap memory consumption, I can see 60% is consumed
in json objects.

I am looking at the commands and expressions to reduce memory
consumption there. The result of that work will also help json
support.