From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
	id S1754047AbbDNUF0 (ORCPT <rfc822;w@1wt.eu>);
	Tue, 14 Apr 2015 16:05:26 -0400
Received: from casper.infradead.org ([85.118.1.10]:59071 "EHLO
	casper.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
	with ESMTP id S1752782AbbDNUFS (ORCPT
	<rfc822;linux-kernel@vger.kernel.org>);
	Tue, 14 Apr 2015 16:05:18 -0400
Message-ID: <552D72F5.3060404@infradead.org>
Date: Tue, 14 Apr 2015 13:05:09 -0700
From: Randy Dunlap <rdunlap@infradead.org>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0
MIME-Version: 1.0
To: Alexey Dobriyan <adobriyan@gmail.com>, mmarek@suse.cz,
        akpm@linux-foundation.org
CC: linux-kernel@vger.kernel.org
Subject: Re: [PATCH] tags: much faster, parallel "make tags"
References: <20150414172047.GA5641@p183.telecom.by>
In-Reply-To: <20150414172047.GA5641@p183.telecom.by>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 04/14/15 10:20, Alexey Dobriyan wrote:
> ctags is single-threaded program. Split list of files to be tagged into
> equal parts, 1 part for each CPU and then merge the results.
> 
> Speedup on one 2-way box I have is ~143 s => ~99 s (-31%).
> On another 4-way box: ~120 s => ~65 s (-46%!).
> 
> Resulting "tags" files aren't byte-for-byte identical because ctags
> program numbers anon struct and enum declarations with "__anonNNN"
> symbols. If those lines are removed, "tags" file becomes byte-for-byte
> identical with those generated with current code.
> 
> Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
> ---
> 
>  scripts/tags.sh |   34 ++++++++++++++++++++++++++++++++--
>  1 file changed, 32 insertions(+), 2 deletions(-)
> 
> --- a/scripts/tags.sh
> +++ b/scripts/tags.sh
> @@ -152,7 +152,24 @@ dogtags()
>  
>  exuberant()
>  {
> -	all_target_sources | xargs $1 -a                        \
> +	NR_CPUS=1
> +	if [ -e /proc/cpuinfo ]; then
> +		NR_CPUS=$(grep -e '^processor	: ' /proc/cpuinfo | wc -l)

That grep is rather arch-specific. If an arch does not have that string
(with an embedded tab), won't NR_CPUS be zero?  so at least, set it back to 1?

or (if 'getconf' is installed):
NR_CPUS = `getconf _NPROCESSORS_ONLN`

> +	fi
> +
> +	rm -f .make-tags.src.* .make-tags.*
> +
> +	all_target_sources >.make-tags.src
> +	# seems like Useless Use of cat(1) but not really
> +	NR_LINES=$(cat .make-tags.src | wc -l)
> +	NR_LINES=$((($NR_LINES + $NR_CPUS - 1) / $NR_CPUS))
> +
> +	split -a 6 -d -l $NR_LINES .make-tags.src .make-tags.src.
> +
> +	for i in .make-tags.src.*; do
> +		N=$(echo $i | sed -e 's/.*\.//')
> +		# -u: don't sort now, sort later
> +		cat $i | xargs $1 -a -f .make-tags.$N -u	\
>  	-I __initdata,__exitdata,__initconst,			\
>  	-I __cpuinitdata,__initdata_memblock			\
>  	-I __refdata,__attribute,__maybe_unused,__always_unused \
> @@ -211,7 +228,20 @@ exuberant()
>  	--regex-c='/DEFINE_PCI_DEVICE_TABLE\((\w*)/\1/v/'		\
>  	--regex-c='/(^\s)OFFSET\((\w*)/\2/v/'				\
>  	--regex-c='/(^\s)DEFINE\((\w*)/\2/v/'				\
> -	--regex-c='/DEFINE_HASHTABLE\((\w*)/\1/v/'
> +	--regex-c='/DEFINE_HASHTABLE\((\w*)/\1/v/'			\
> +	&
> +	done
> +	wait
> +	rm -f .make-tags.src .make-tags.src.*
> +
> +	# write header
> +	$1 -f tags /dev/null
> +	# remove header
> +	for i in .make-tags.*; do
> +		sed -i -e '/^!/d' $i
> +	done
> +	sort .make-tags.* >>tags
> +	rm -f .make-tags.*
>  
>  	all_kconfigs | xargs $1 -a                              \
>  	--langdef=kconfig --language-force=kconfig              \
> --


-- 
~Randy