Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

[John Keeping <john@keeping.me.uk>]

On Wed, Aug 06, 2014 at 07:32:14AM +0200, Steffen Prohaska wrote:
[...]
> The expectation on the process size is tested using /usr/bin/time.  An
> alternative would have been tcsh, which could be used to print memory
> information as follows:
> 
>     tcsh -c 'set time=(0 "%M"); <cmd>'
> 
> Although the logic could perhaps be simplified with tcsh, I chose to use
> 'time' to avoid a dependency on tcsh.
[...]
> @@ -190,6 +196,58 @@ test_expect_success 'required filter clean failure' '
>  	test_must_fail git add test.fc
>  '
>  
> +# Handle differences in /usr/bin/time.
> +#
> +#  - Linux: call with '-v'.
> +#    output: <spaces><description>:<space><value-in-KBytes>
> +#
> +#  - Mac: call with '-l'.
> +#    output: <spaces><value-in-Bytes><spaces><description>
> +#    Strip three digits to get to KB (base 10 is good enough).
> +#
> +case $(uname -s) in
> +Linux)
> +	test_set_prereq HAVE_MAX_MEM_USAGE
> +	max_mem_usage_KB () {
> +	    /usr/bin/time -v "$@" 2>&1 |
> +	    grep 'Maximum resident set size' |
> +	    cut -d ':' -f 2
> +	}
> +	;;
> +Darwin)
> +	test_set_prereq HAVE_MAX_MEM_USAGE
> +	max_mem_usage_KB () {
> +		/usr/bin/time -l "$@" 2>&1 |
> +		grep 'maximum resident set size' |
> +		sed -e 's/  */ /' |
> +		cut -d ' ' -f 2 |
> +		sed -e 's/...$//'
> +	}
> +	;;
> +esac
> +
> +max_mem_usage_is_lt_KB () {
> +	limit=$1
> +	shift
> +	mem_usage=$(max_mem_usage_KB "$@")
> +	if [ $mem_usage -lt $limit ]; then
> +		true
> +	else
> +		printf 'Command used too much memory (expected limit %dKB, actual usage %dKB).\n' \
> +			$limit $mem_usage
> +		false
> +	fi
> +}
> +
> +test_expect_success HAVE_MAX_MEM_USAGE \
> +'filtering large input to small output should use little memory' '
> +	git config filter.devnull.clean "cat >/dev/null" &&
> +	git config filter.devnull.required true &&
> +	for i in $(test_seq 1 30); do printf "%1048576d" 1; done >30MB &&
> +	echo "30MB filter=devnull" >.gitattributes &&
> +	max_mem_usage_is_lt_KB 15000 git add 30MB
> +'

This test fails for me:

-- 8< --
expecting success: 
        git config filter.devnull.clean "cat >/dev/null" &&
        git config filter.devnull.required true &&
        for i in $(test_seq 1 30); do printf "%1048576d" 1; done >30MB &&
        echo "30MB filter=devnull" >.gitattributes &&
        max_mem_usage_is_lt_KB 15000 git add 30MB

Command used too much memory (expected limit 15000KB, actual usage 15808KB).
not ok 8 - filtering large input to small output should use little memory
-- >8 --

This is on Linux 3.16 x86_64 with GCC 4.8.3 and glibc 2.19.  My GCC also
has "-fstack-protector" and "-D_FORTIFY_SOURCE=2" enabled by default;
turning those off for Git decreased the memory usage by about 500KB but
not enough to make the test pass.  Of course, all of the libraries Git
is linking are also compiled with those flags...

Is the 15MB limit supposed to be imposed somewhere or is it just a guide
of how much memory we expect Git to use in this scenario?