[RFC 1/4] scripts: add stackusage script

[RFC 1/4] scripts: add stackusage script

[Rasmus Villemoes]

The current checkstack.pl script has a few problems, stemming from the
overly simplistic attempt at parsing objdump output with regular
expresions: For example, on x86_64 it doesn't take the push
instruction into account, making it consistently underestimate the
real stack use, and it also doesn't capture stack pointer adjustments
of exactly 128 bytes [1].

Since newer gcc (>= 4.6) knows about -fstack-usage, we might as well
take the information straight from the horse's mouth. This patch
introduces scripts/stackusage, which is a simple wrapper for running
make with EXTRA_CFLAGS set to -fstack-usage. Example use is

scripts/stackusage -o out.su -- -j8 fs/ext4/

Arguments after -- are passed to make. Afterwards, we find all newly
created .su files, massage them a little, sort by stack use and
concatenate the result to a single output file.

[1] Since gcc encodes that by

48 83 c4 80             add    $0xffffffffffffff80,%rsp

and not

48 81 ec 80 00 00 00    sub    $0x80,%rsp

since -128 fits in an imm8.

Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
---
 scripts/stackusage | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)
 create mode 100755 scripts/stackusage

diff --git a/scripts/stackusage b/scripts/stackusage
new file mode 100755
index 000000000000..d631af648ae7
--- /dev/null
+++ b/scripts/stackusage
@@ -0,0 +1,40 @@
+#!/bin/sh
+
+outfile=""
+now=`date +%s`
+
+while [ $# -gt 0 ]
+do
+    case "$1" in
+        -o)
+	    outfile="$2"
+	    echo "$outfile"
+	    shift 2;;
+	-h)
+	    echo "usage: $0 [-o outfile] -- <make options/args>"
+	    exit 0;;
+	--)
+	    shift
+	    break;;
+	-*)
+            echo >&2 "usage: $0 [-o outfile] -- <make options/args>"
+	    exit 1;;
+	*)  break;;
+    esac
+done
+
+if [ -z "$outfile" ]
+then
+    outfile=`mktemp --tmpdir stackusage.$$.XXXX`
+fi
+
+make EXTRA_CFLAGS="-fstack-usage" "$@"
+
+# Prepend directory name to file names, remove line/column
+# information, make file/function/size/type properly tab-separated.
+find . -name '*.su' -newermt "@${now}" -print |                 \
+    xargs perl -MFile::Basename -pe                             \
+      '$d = dirname($ARGV); s#([^:]+):([0-9]+:){2}#$d/$1\t#;' | \
+    sort -k3,3nr > "${outfile}"
+
+echo "$0: output written to ${outfile}"
-- 
2.1.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC 1/4] scripts: add stackusage script

[Michal Marek]

On 2015-06-21 03:10, Rasmus Villemoes wrote:
> The current checkstack.pl script has a few problems, stemming from the
> overly simplistic attempt at parsing objdump output with regular
> expresions: For example, on x86_64 it doesn't take the push
> instruction into account, making it consistently underestimate the
> real stack use, and it also doesn't capture stack pointer adjustments
> of exactly 128 bytes [1].
> 
> Since newer gcc (>= 4.6) knows about -fstack-usage, we might as well
> take the information straight from the horse's mouth. This patch
> introduces scripts/stackusage, which is a simple wrapper for running
> make with EXTRA_CFLAGS set to -fstack-usage. Example use is
> 
> scripts/stackusage -o out.su -- -j8 fs/ext4/
> 
> Arguments after -- are passed to make. Afterwards, we find all newly
> created .su files, massage them a little, sort by stack use and
> concatenate the result to a single output file.
> 
> [1] Since gcc encodes that by
> 
> 48 83 c4 80             add    $0xffffffffffffff80,%rsp
> 
> and not
> 
> 48 81 ec 80 00 00 00    sub    $0x80,%rsp
> 
> since -128 fits in an imm8.
> 
> Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
> ---
>  scripts/stackusage | 40 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 40 insertions(+)
>  create mode 100755 scripts/stackusage
> 
> diff --git a/scripts/stackusage b/scripts/stackusage
> new file mode 100755
> index 000000000000..d631af648ae7
> --- /dev/null
> +++ b/scripts/stackusage
> @@ -0,0 +1,40 @@
> +#!/bin/sh
> +
> +outfile=""
> +now=`date +%s`
> +
> +while [ $# -gt 0 ]
> +do
> +    case "$1" in
> +        -o)
> +	    outfile="$2"
> +	    echo "$outfile"
> +	    shift 2;;
> +	-h)
> +	    echo "usage: $0 [-o outfile] -- <make options/args>"
> +	    exit 0;;
> +	--)
> +	    shift
> +	    break;;
> +	-*)
> +            echo >&2 "usage: $0 [-o outfile] -- <make options/args>"
> +	    exit 1;;
> +	*)  break;;
> +    esac
> +done
> +
> +if [ -z "$outfile" ]
> +then
> +    outfile=`mktemp --tmpdir stackusage.$$.XXXX`
> +fi
> +
> +make EXTRA_CFLAGS="-fstack-usage" "$@"

EXTRA_CFLAGS is reserved for use by Makefiles, please use KCFLAGS
instead. I wonder whether it worked at all, because EXTRA_CFLAGS is
reset by scripts/Makefile.build.

Michal

Re: [RFC 1/4] scripts: add stackusage script

[Rasmus Villemoes]

On Tue, Jun 23 2015, Michal Marek <mmarek@suse.cz> wrote:

>
> EXTRA_CFLAGS is reserved for use by Makefiles, please use KCFLAGS
> instead.

Will do, thanks. Maybe Documentation/SubmitChecklist and
Documentation/development-process/4.Coding wants to be updated.

> I wonder whether it worked at all, because EXTRA_CFLAGS is
> reset by scripts/Makefile.build.

It did indeed work, but only because I passed the setting on the make
command line and not via the environment. I think one needs an override
directive to actually override the settings from the command line:

$ cat Makefile 
VAR1 := a1
VAR2 := a2
override VAR3 := a3
override VAR4 := a4

all:
        @echo $(VAR1) $(VAR2) $(VAR3) $(VAR4)
$ VAR1=b1 VAR3=b3 make VAR2=b2 VAR4=b4 all
a1 b2 a3 a4

[RFC 2/4] .gitignore: add *.su pattern

[Rasmus Villemoes]

Ignore the *.su files generated by using the gcc option -fstack-usage.

Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 4ad4a98b884b..9e51ead66a55 100644
--- a/.gitignore
+++ b/.gitignore
@@ -36,6 +36,7 @@
 modules.builtin
 Module.symvers
 *.dwo
+*.su
 
 #
 # Top-level generic files
-- 
2.1.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

[RFC 3/4] kbuild: remove *.su files generated by -fstack-usage

[Rasmus Villemoes]

Make sure 'make clean' removes *.su files generated by the gcc option
-fstack-usage.

Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index 40a8b068ac26..5cd04968d914 100644
--- a/Makefile
+++ b/Makefile
@@ -1430,6 +1430,7 @@ clean: $(clean-dirs)
 		\( -name '*.[oas]' -o -name '*.ko' -o -name '.*.cmd' \
 		-o -name '*.ko.*' \
 		-o -name '*.dwo'  \
+		-o -name '*.su'  \
 		-o -name '.*.d' -o -name '.*.tmp' -o -name '*.mod.c' \
 		-o -name '*.symtypes' -o -name 'modules.order' \
 		-o -name modules.builtin -o -name '.tmp_*.o.*' \
-- 
2.1.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

[RFC 4/4] scripts: add stackdelta script

[Rasmus Villemoes]

This adds a simple perl script for reading two files as produced by
the stackusage script and computing the changes in stack usage. For
example:

$ scripts/stackusage -o /tmp/old.su -- CC=gcc-4.7 -j8 fs/ext4/
$ scripts/stackusage -o /tmp/new.su -- CC=gcc-5.0 -j8 fs/ext4/
$ scripts/stackdelta /tmp/{old,new}.su | sort -k5,5g

shows that gcc 5.0 generally produces less stack-hungry code than gcc
4.7. Obviously, the script can also be used for measuring the effect
of commits, .config tweaks or whatnot.

Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
---
 scripts/stackdelta | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
 create mode 100755 scripts/stackdelta

diff --git a/scripts/stackdelta b/scripts/stackdelta
new file mode 100755
index 000000000000..18417098a378
--- /dev/null
+++ b/scripts/stackdelta
@@ -0,0 +1,55 @@
+#!/usr/bin/perl
+
+# Read two files producesd by the stackusage script, and show the
+# delta between them.
+#
+# Currently, only shows changes for functions listed in both files. We
+# could add an option to show also functions which have vanished or
+# appeared (which would often be due to gcc making other inlining
+# decisions).
+#
+# Another possible option would be a minimum absolute value for the
+# delta.
+
+sub read_stack_usage_file {
+    my %su;
+    my $f = shift;
+    open(my $fh, '<', $f)
+	or die "cannot open $f: $!";
+    while (<$fh>) {
+	chomp;
+	my ($file, $func, $size, $type) = split;
+	# Old versions of gcc (at least 4.7) have an annoying quirk in
+	# that a (static) function whose name has been changed into
+	# for example ext4_find_unwritten_pgoff.isra.11 will show up
+	# in the .su file with a name of just "11". Since such a
+	# numeric suffix is likely to change across different
+	# commits/compilers/.configs or whatever else we're trying to
+	# tweak, we can't really track those functions, so we just
+	# silently skip them.
+	#
+	# Newer gcc (at least 5.0) report the full name, so again,
+	# since the suffix is likely to change, we strip it.
+	next if $func =~ m/^[0-9]+$/;
+	$func =~ s/\..*$//;
+	$su{"${file}\t${func}"} = {size => $size, type => $type};
+    }
+    close($fh);
+    return \%su;
+}
+
+@ARGV == 2
+    or die "usage: $0 <old> <new>";
+
+my $old = read_stack_usage_file($ARGV[0]);
+my $new = read_stack_usage_file($ARGV[1]);
+my @common = grep {exists $new->{$_}} keys %$old;
+print join(",", keys %old) . "\n";
+for (@common) {
+    my $x = $old->{$_}{size};
+    my $y = $new->{$_}{size};
+    my $delta = $y - $x;
+    if ($delta) {
+	printf "%s\t%d\t%d\t%+d\n", $_, $x, $y, $delta;
+    }
+}
-- 
2.1.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC 4/4] scripts: add stackdelta script

[Rasmus Villemoes]

On Sun, Jun 21 2015, Rasmus Villemoes <linux@rasmusvillemoes.dk> wrote:

> +
> +my $old = read_stack_usage_file($ARGV[0]);
> +my $new = read_stack_usage_file($ARGV[1]);
> +my @common = grep {exists $new->{$_}} keys %$old;
> +print join(",", keys %old) . "\n";

Bah, that print is leftover from debugging. Please pretend it's not
there.

Rasmus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

[RFC v2 1/4] scripts: add stackusage script

[Rasmus Villemoes]

The current checkstack.pl script has a few problems, stemming from the
overly simplistic attempt at parsing objdump output with regular
expresions: For example, on x86_64 it doesn't take the push
instruction into account, making it consistently underestimate the
real stack use, and it also doesn't capture stack pointer adjustments
of exactly 128 bytes [1].

Since newer gcc (>= 4.6) knows about -fstack-usage, we might as well
take the information straight from the horse's mouth. This patch
introduces scripts/stackusage, which is a simple wrapper for running
make with KCFLAGS set to -fstack-usage. Example use is

scripts/stackusage -o out.su -j8 lib/

The script understands "-o foo" for writing to 'foo' and -h for a
trivial help text; anything else is passed to make.

Afterwards, we find all newly created .su files, massage them a
little, sort by stack use and write the result to a single output
file.

Note that the function names printed by (at least) gcc 4.7 are
sometimes useless. For example, the first three lines of out.su
generated above are

./lib/decompress_bunzip2.c:155  get_next_block  448     static
./lib/decompress_unlzma.c:537   unlzma  336     static
./lib/vsprintf.c:616    8       304     static

That function '8' is really the static symbol_string(), but it has
been subject to 'interprocedural scalar replacement of aggregates', so
its name in the object file is 'symbol_string.isra.8'. gcc 5.0 doesn't
have this problem; it uses the full name as seen in the object file.

[1] Since gcc encodes that by

48 83 c4 80             add    $0xffffffffffffff80,%rsp

and not

48 81 ec 80 00 00 00    sub    $0x80,%rsp

since -128 fits in an imm8.

Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
---
 scripts/stackusage | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100755 scripts/stackusage

diff --git a/scripts/stackusage b/scripts/stackusage
new file mode 100755
index 000000000000..b30dfa6b3a0a
--- /dev/null
+++ b/scripts/stackusage
@@ -0,0 +1,33 @@
+#!/bin/sh
+
+outfile=""
+now=`date +%s`
+
+while [ $# -gt 0 ]
+do
+    case "$1" in
+        -o)
+	    outfile="$2"
+	    shift 2;;
+	-h)
+	    echo "usage: $0 [-o outfile] <make options/args>"
+	    exit 0;;
+	*)  break;;
+    esac
+done
+
+if [ -z "$outfile" ]
+then
+    outfile=`mktemp --tmpdir stackusage.$$.XXXX`
+fi
+
+KCFLAGS="${KCFLAGS} -fstack-usage" make "$@"
+
+# Prepend directory name to file names, remove column information,
+# make file:column/function/size/type properly tab-separated.
+find . -name '*.su' -newermt "@${now}" -print |                     \
+    xargs perl -MFile::Basename -pe                                 \
+        '$d = dirname($ARGV); s#([^:]+:[0-9]+):[0-9]+:#$d/$1\t#;' | \
+    sort -k3,3nr > "${outfile}"
+
+echo "$0: output written to ${outfile}"
-- 
2.1.3

[RFC v2 2/4] .gitignore: add *.su pattern

[Rasmus Villemoes]

Ignore the *.su files generated by using the gcc option -fstack-usage.

Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 4ad4a98b884b..9e51ead66a55 100644
--- a/.gitignore
+++ b/.gitignore
@@ -36,6 +36,7 @@
 modules.builtin
 Module.symvers
 *.dwo
+*.su
 
 #
 # Top-level generic files
-- 
2.1.3

[RFC v2 3/4] kbuild: remove *.su files generated by -fstack-usage

[Rasmus Villemoes]

Make sure 'make clean' removes *.su files generated by the gcc option
-fstack-usage.

Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index 6c6f14628f32..81f1d74848cc 100644
--- a/Makefile
+++ b/Makefile
@@ -1429,6 +1429,7 @@ clean: $(clean-dirs)
 		\( -name '*.[oas]' -o -name '*.ko' -o -name '.*.cmd' \
 		-o -name '*.ko.*' \
 		-o -name '*.dwo'  \
+		-o -name '*.su'  \
 		-o -name '.*.d' -o -name '.*.tmp' -o -name '*.mod.c' \
 		-o -name '*.symtypes' -o -name 'modules.order' \
 		-o -name modules.builtin -o -name '.tmp_*.o.*' \
-- 
2.1.3

[RFC v2 4/4] scripts: add stackdelta script

[Rasmus Villemoes]

This adds a simple perl script for reading two files as produced by
the stackusage script and computing the changes in stack usage. For
example:

$ scripts/stackusage -o /tmp/old.su CC=gcc-4.7 -j8 fs/ext4/
$ scripts/stackusage -o /tmp/new.su CC=gcc-5.0 -j8 fs/ext4/
$ scripts/stackdelta /tmp/{old,new}.su | sort -k5,5g

shows that gcc 5.0 generally produces less stack-hungry code than gcc
4.7. Obviously, the script can also be used for measuring the effect
of commits, .config tweaks or whatnot.

Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
---
 scripts/stackdelta | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 59 insertions(+)
 create mode 100755 scripts/stackdelta

diff --git a/scripts/stackdelta b/scripts/stackdelta
new file mode 100755
index 000000000000..4730dfa7780f
--- /dev/null
+++ b/scripts/stackdelta
@@ -0,0 +1,59 @@
+#!/usr/bin/perl
+
+# Read two files producesd by the stackusage script, and show the
+# delta between them.
+#
+# Currently, only shows changes for functions listed in both files. We
+# could add an option to show also functions which have vanished or
+# appeared (which would often be due to gcc making other inlining
+# decisions).
+#
+# Another possible option would be a minimum absolute value for the
+# delta.
+#
+# A third possibility is for sorting by delta, but that can be
+# achieved by piping to sort -k5,5g.
+
+sub read_stack_usage_file {
+    my %su;
+    my $f = shift;
+    open(my $fh, '<', $f)
+	or die "cannot open $f: $!";
+    while (<$fh>) {
+	chomp;
+	my ($file, $func, $size, $type) = split;
+	# Old versions of gcc (at least 4.7) have an annoying quirk in
+	# that a (static) function whose name has been changed into
+	# for example ext4_find_unwritten_pgoff.isra.11 will show up
+	# in the .su file with a name of just "11". Since such a
+	# numeric suffix is likely to change across different
+	# commits/compilers/.configs or whatever else we're trying to
+	# tweak, we can't really track those functions, so we just
+	# silently skip them.
+	#
+	# Newer gcc (at least 5.0) report the full name, so again,
+	# since the suffix is likely to change, we strip it.
+	next if $func =~ m/^[0-9]+$/;
+	$func =~ s/\..*$//;
+	# Line numbers are likely to change; strip those.
+	$file =~ s/:[0-9]+$//;
+	$su{"${file}\t${func}"} = {size => $size, type => $type};
+    }
+    close($fh);
+    return \%su;
+}
+
+@ARGV == 2
+    or die "usage: $0 <old> <new>";
+
+my $old = read_stack_usage_file($ARGV[0]);
+my $new = read_stack_usage_file($ARGV[1]);
+my @common = sort grep {exists $new->{$_}} keys %$old;
+for (@common) {
+    my $x = $old->{$_}{size};
+    my $y = $new->{$_}{size};
+    my $delta = $y - $x;
+    if ($delta) {
+	printf "%s\t%d\t%d\t%+d\n", $_, $x, $y, $delta;
+    }
+}
-- 
2.1.3