From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38267) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZzXGh-0004Uc-50 for qemu-devel@nongnu.org; Thu, 19 Nov 2015 17:01:27 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZzXGd-0007WW-Uo for qemu-devel@nongnu.org; Thu, 19 Nov 2015 17:01:27 -0500 Received: from mail-wm0-x230.google.com ([2a00:1450:400c:c09::230]:32808) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZzXGd-0007WS-P9 for qemu-devel@nongnu.org; Thu, 19 Nov 2015 17:01:23 -0500 Received: by wmec201 with SMTP id c201so46827515wme.0 for ; Thu, 19 Nov 2015 14:01:23 -0800 (PST) Sender: Paolo Bonzini References: <1447946948-12489-1-git-send-email-armbru@redhat.com> <1447946948-12489-5-git-send-email-armbru@redhat.com> From: Paolo Bonzini Message-ID: <564E46AD.4030901@redhat.com> Date: Thu, 19 Nov 2015 23:01:17 +0100 MIME-Version: 1.0 In-Reply-To: <1447946948-12489-5-git-send-email-armbru@redhat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v2 4/4] json-streamer: Limit number of tokens in addition to total size List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Markus Armbruster , qemu-devel@nongnu.org Cc: lcapitulino@redhat.com On 19/11/2015 16:29, Markus Armbruster wrote: > Commit 29c75dd "json-streamer: limit the maximum recursion depth and > maximum token count" attempts to guard against excessive heap usage by > limiting total token size (it says "token count", but that's a lie). > > Total token size is a rather imprecise predictor of heap usage: many > small tokens use more space than few large tokens with the same input > size, because there's a constant per-token overhead. > > Tighten this up: limit the token count to 128Ki. > > If you think 128Ki is too stingy: check-qjson's large_dict test eats a > sweet 500MiB on my machine to parse ~100K tokens. How much of this is freed before the start of the parse? > Signed-off-by: Markus Armbruster > Reviewed-by: Eric Blake > --- > qobject/json-streamer.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c > index 2bd22a7..8752834 100644 > --- a/qobject/json-streamer.c > +++ b/qobject/json-streamer.c > @@ -19,6 +19,7 @@ > #include "qapi/qmp/json-streamer.h" > > #define MAX_TOKEN_SIZE (64ULL << 20) > +#define MAX_TOKEN_COUNT (128ULL << 10) > #define MAX_NESTING (1ULL << 10) > > static void json_message_process_token(JSONLexer *lexer, QString *token, JSONTokenType type, int x, int y) > @@ -64,6 +65,7 @@ static void json_message_process_token(JSONLexer *lexer, QString *token, JSONTok > parser->bracket_count == 0)) { > goto out_emit; > } else if (parser->token_size > MAX_TOKEN_SIZE || > + qlist_size(parser->tokens) > MAX_TOKEN_COUNT || This is O(n^2). I'd rather skip this patch, fix the memory hog and possibly decrease MAX_TOKEN_SIZE a bit. Paolo > parser->bracket_count + parser->brace_count > MAX_NESTING) { > /* Security consideration, we limit total memory allocated per object > * and the maximum recursion depth that a message can force. >