From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38103) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZrqIf-0001Ec-IV for qemu-devel@nongnu.org; Thu, 29 Oct 2015 12:43:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZrqIc-0004Ur-Ai for qemu-devel@nongnu.org; Thu, 29 Oct 2015 12:43:41 -0400 Received: from mx1.redhat.com ([209.132.183.28]:38760) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZrqIc-0004UY-26 for qemu-devel@nongnu.org; Thu, 29 Oct 2015 12:43:38 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 59E04C0A5256 for ; Thu, 29 Oct 2015 16:43:37 +0000 (UTC) References: <1446122683-2355-1-git-send-email-armbru@redhat.com> <1446122683-2355-5-git-send-email-armbru@redhat.com> From: Eric Blake Message-ID: <56324CB8.5060303@redhat.com> Date: Thu, 29 Oct 2015 10:43:36 -0600 MIME-Version: 1.0 In-Reply-To: <1446122683-2355-5-git-send-email-armbru@redhat.com> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="jr50fSDGqCPJ79dUfU3OJTSvDRu56oaFi" Subject: Re: [Qemu-devel] [PATCH 4/4] json-streamer: Limit number of tokens in addition to total size List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Markus Armbruster , qemu-devel@nongnu.org Cc: lcapitulino@redhat.com This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --jr50fSDGqCPJ79dUfU3OJTSvDRu56oaFi Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 10/29/2015 06:44 AM, Markus Armbruster wrote: > Commit 29c75dd "json-streamer: limit the maximum recursion depth and > maximum token count" attempts to guard against excessive heap usage by > limiting total token size (it says "token count", but that's a lie). >=20 > Total token size is a rather imprecise predictor of heap usage: many > small tokens use more space than few large tokens with the same input > size, because there's a constant per-token overhead. >=20 > Tighten this up: limit the token count to 128Ki. >=20 > If you think 128Ki is too stingy: check-qjson's large_dict test eats a > sweet 500MiB and pegs a core for four minutes on my machine to parse > ~100K tokens. Absurdly wasteful. Sounds like we have some quadratic (or worse) scaling in the parser. Worth fixing some day, but I also agree that we don't have to tackle it in this series. I'm assuming you temporarily patched check-qjson to use larger constants when you hit your ~100K token testing? Because I am definitely seeing a lot of execution time spent on large_dict when running tests/check-qjson by hand, in relation to all the other tests of that file, but not minutes worth. Care to post the diff you played with? >=20 > Signed-off-by: Markus Armbruster > --- > qobject/json-streamer.c | 2 ++ > 1 file changed, 2 insertions(+) Reviewed-by: Eric Blake >=20 > diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c > index 755c74d..df2b4c1 100644 > --- a/qobject/json-streamer.c > +++ b/qobject/json-streamer.c > @@ -19,6 +19,7 @@ > #include "qapi/qmp/json-streamer.h" > =20 > #define MAX_TOKEN_SIZE (64ULL << 20) > +#define MAX_TOKEN_COUNT (128ULL << 10) > #define MAX_NESTING (1ULL << 10) > =20 > static void json_message_process_token(JSONLexer *lexer, QString *toke= n, JSONTokenType type, int x, int y) > @@ -64,6 +65,7 @@ static void json_message_process_token(JSONLexer *lex= er, QString *token, JSONTok > parser->bracket_count =3D=3D 0)) { > goto out_emit; > } else if (parser->token_size > MAX_TOKEN_SIZE || > + qlist_size(parser->tokens) > MAX_TOKEN_COUNT || > parser->bracket_count + parser->brace_count > MAX_NESTI= NG) { > /* Security consideration, we limit total memory allocated per= object > * and the maximum recursion depth that a message can force. >=20 --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --jr50fSDGqCPJ79dUfU3OJTSvDRu56oaFi Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCAAGBQJWMky4AAoJEKeha0olJ0NqiZsH/2QHVL++l2j7dXt4iC/Gi4gN n3crWXqcQCwVp0MZJG+z9e8ObcTYzcCp3tU3HJWaOp+lqSobopZC2CWVl0EKx7+z r0KheqT3BW8jJvT6ek+8wC3gkQgLGEAIaj60B7e8jcbQH+RxFu9NMRqdDuPBSmWH Im/pXVCYqPHE+JRUOwXQ/i/p3HwbjEVUtn9fBu9uq6hGd+NazbuZBHVu834GNcgT aDIDmGFEqYi8xiU+HuNvVG2OMxlCGRXxNfOQcR+/TflhTIZKp4OLo3Aaa4qPxQMT C6tT6/hlnzWGpG3Eievsi6iKu94RO2aDU55msRoTmKkVO75pbQ50nIzXcypq4/Y= =RzTX -----END PGP SIGNATURE----- --jr50fSDGqCPJ79dUfU3OJTSvDRu56oaFi--