On Wed, Nov 13, 2024 at 12:01:07PM +0100, Pablo Neira Ayuso wrote:
> Hi Phil,
> 
> On Tue, Nov 12, 2024 at 09:52:35PM +0100, Phil Sutter wrote:
> > Hi Pablo,
> > 
> > On Thu, Oct 31, 2024 at 11:04:11PM +0100, Pablo Neira Ayuso wrote:
> > > Side note: While profiling, I can still see lots json objects, this
> > > results in memory consumption that is 5 times than native
> > > representation. Error reporting is also lagging behind, it should be
> > > possible to add a json_t pointer to struct location to relate
> > > expressions and json objects.
> > 
> > I can't quite reproduce this. When restoring a ruleset with ~12.7k
> > elements in individual standard syntax commands, valgrind prints:
> > 
> > | HEAP SUMMARY:
> > |     in use at exit: 59,802 bytes in 582 blocks
> > |   total heap usage: 954,970 allocs,
> > |                     954,388 frees,
> > |                  18,300,874 bytes allocated
> > 
> > Repeating the same in JSON syntax, I get:
> > 
> > | HEAP SUMMARY:
> > |     in use at exit: 61,592 bytes in 647 blocks
> > |   total heap usage: 1,200,164 allocs,
> > |                     1,199,517 frees,
> > |                    38,612,257 bytes allocated
> > 
> > So this is 38MB vs 18MB? At least far from the mentioned 5 times. Would
> > you mind sharing how you got to that number?
> > 
> > Please kindly find my reproducers attached for reference.
> 
> I am using valgrind --tool=massif to measure memory consumption in
> userspace.
> 
> I used these two files:
> 
> - set-init.json-nft, to create the table and set.
> - set-65535.nft-json, to create a small set with 64K elements.

Forgot attachments.

> then I run:
> 
> valgrind --tool=massif nft -f set-65535.nft-json
> 
> there is a tool:
> 
> ms_print massif.out.XYZ
> 
> At "peak time" in heap memory consumption, I can see 60% is consumed
> in json objects.
> 
> I am looking at the commands and expressions to reduce memory
> consumption there. The result of that work will also help json
> support.
>