From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from cantor2.suse.de ([195.135.220.15]:48195 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754853AbaDKTAQ (ORCPT ); Fri, 11 Apr 2014 15:00:16 -0400 Date: Fri, 11 Apr 2014 21:00:14 +0200 From: "Luis R. Rodriguez" To: Julia Lawall Cc: SF Markus Elfring , Johannes Berg , linux-kernel@vger.kernel.org, backports@vger.kernel.org, cocci@systeme.lip6.fr Subject: Re: [Cocci] [PATCH] coccinelle: add pycocci wrapper for multithreaded support Message-ID: <20140411190014.GL14815@wotan.suse.de> (sfid-20140411_210018_826879_8A653678) References: <1397152097-315-1-git-send-email-mcgrof@do-not-panic.com> <1397152289.4757.28.camel@jlt4.sipsolutions.net> <20140410175748.GK14815@wotan.suse.de> <534783DB.4080200@users.sourceforge.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="z9sQuz+HmDh2hVO4" In-Reply-To: Sender: backports-owner@vger.kernel.org List-ID: --z9sQuz+HmDh2hVO4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Apr 11, 2014 at 08:01:04AM +0200, Julia Lawall wrote: >=20 >=20 > On Fri, 11 Apr 2014, SF Markus Elfring wrote: >=20 > > > I checked the profile results, the reason the jobs finish is some thr= eads > > > had no work or little work. > >=20 > > Could you find out during the data processing which parts or files > > result in a special application behaviour you would like to point out h= ere? >=20 > I don't understand the question at all, but since the various files have= =20 > different properties, it is hard to determine automatically in advance ho= w=20 > much work Coccinelle will need to do on each one. For the person who might work on enhancing multithreading support, I'd wond= er if there could be gains of actually putting an effort out first to evaluate which files have one rule hit and then adding the file to an activity file = lis to later be spread between the threads. As you note though it is hard to determine this in advance though given that each rule express any change. I think one small change which could help, and likely not incur a drastic immediate change on architecture could be to not let theads take files / jo= bs list of files, but instead just take say: work_task_n =3D (files / jobs) / 10 The list of files needing work could then be kept on a list protected against the threads, and each thread will only die if all the files have been worked on already. This would enable keeping number_cpu threads only, as each CPU would indeed be busy all the time. BTW is the patch Acked-by Julia? Can we commit it :) Luis --z9sQuz+HmDh2hVO4 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iQIcBAEBAgAGBQJTSDu+AAoJEPep4JnvMe6zZgoP/2Sc84WZBgEzJnTiy2zZrvSw pXkdOBnRjonma9F+Rh7leBLYXi20ZzU0Uu46wDKACo55yz8dRPIstp9zAIQut3Ce QmdOCC3Ti2iYBz+hbdbz8EinGKXYLOzeBZOG1p9Bk8YxDEyQ/AGqYvVJhR502UrQ 1nSJU3s/GtNaU7mxU42Xxbs/iOAV8lM+Z6RYulfgcVKVv8bZFMNIIsfvv8+iAfua VvKz1m01zTrM+5lbv8Fj3+AeT0YksOgP6lRyNOzYHY5qzFZz83B3yoxkrkEvXfUp yexn5hCpSOdno6RupUMnOMiCQJj7MsTQvmRS5zcU6nwNsfVhw2Wczh3sJWXxet3t IzvLF2qxNgAbgbfDzkLRNeWbjoz/aLnfxy7GO3gWcx4TPfcd2I8SHtRynHJey9Q9 GaZ6FIByUePe0g4b+XADCSz88TB26VpiMk2Q9+Xb3hGGkNMmQifUTVt3gPgTgVq2 MKPxDo41BZkQ+xtWtDVWYcmk7usyOVj1FHIl/lSa8kaEZkiwLHkp0hRVgeTPOaJb 9B4Bxu1K6Ioldawrr5yvbTH9+Fg2K0S+Ls5U05z2ppiGvXh8f9X5fKXh+eqCwgp7 Ga6C7tFMidXkKaqNKCI93u4GCrSf1vZHHS+i/83NiBCBZy344Ajw0/qt86APf2au b7CaTxyMnrWXKCxU06K+ =m0g6 -----END PGP SIGNATURE----- --z9sQuz+HmDh2hVO4-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: mcgrof@suse.com (Luis R. Rodriguez) Date: Fri, 11 Apr 2014 21:00:14 +0200 Subject: [Cocci] [PATCH] coccinelle: add pycocci wrapper for multithreaded support In-Reply-To: References: <1397152097-315-1-git-send-email-mcgrof@do-not-panic.com> <1397152289.4757.28.camel@jlt4.sipsolutions.net> <20140410175748.GK14815@wotan.suse.de> <534783DB.4080200@users.sourceforge.net> Message-ID: <20140411190014.GL14815@wotan.suse.de> To: cocci@systeme.lip6.fr List-Id: cocci@systeme.lip6.fr On Fri, Apr 11, 2014 at 08:01:04AM +0200, Julia Lawall wrote: > > > On Fri, 11 Apr 2014, SF Markus Elfring wrote: > > > > I checked the profile results, the reason the jobs finish is some threads > > > had no work or little work. > > > > Could you find out during the data processing which parts or files > > result in a special application behaviour you would like to point out here? > > I don't understand the question at all, but since the various files have > different properties, it is hard to determine automatically in advance how > much work Coccinelle will need to do on each one. For the person who might work on enhancing multithreading support, I'd wonder if there could be gains of actually putting an effort out first to evaluate which files have one rule hit and then adding the file to an activity file lis to later be spread between the threads. As you note though it is hard to determine this in advance though given that each rule express any change. I think one small change which could help, and likely not incur a drastic immediate change on architecture could be to not let theads take files / jobs list of files, but instead just take say: work_task_n = (files / jobs) / 10 The list of files needing work could then be kept on a list protected against the threads, and each thread will only die if all the files have been worked on already. This would enable keeping number_cpu threads only, as each CPU would indeed be busy all the time. BTW is the patch Acked-by Julia? Can we commit it :) Luis -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: