Re: [PATCH 2/3] t: add "test-tool genbytes" to avoid slow tr(1) on macOS

[Collin Funk <collin.funk1@gmail.com>]

Kristofer Karlsson <krka@spotify.com> writes:

> Nice trick, and it would work for t4205, which does not care about
> the byte value.  I tried it for the 2147483649 bytes there on Linux:
>
>   printf           time   peak memory
>   dash             6.5s   2.1GB
>   bash             6.0s   3MB
>   coreutils        3.7s   2.1GB
>
> It is definitely fast enough (on the variants I tested).
> The annoying thing (but not a real blocker) is the extra memory
> usage required and I am not sure I fully trust that all
> implementations will behave the same for absurdly large padding.
>
> (Particularly for paddings larger than 2^31, as you noted too.)
>
> It feels a bit like a hack, and tweaking the existing test-tool
> feels like a more sane option that is easier to reason about.
>
> A gotcha here is that the printf hack only works for producing
> "0", so for the cases we truly need NUL characters it does not
> help us, and I think that is also a point in favor of
> simply generalizing test-tool genzeros instead.

I've mentioned this idea on coreutils or gnulib's mailing lists before,
but I forget where and when. I've wanted to create a printf variant that
doesn't allocate much memory and will just use putc_unlocked for each
padding character, which would avoid the memory issue you bring up.

Most platforms will be limited to INT_MAX there, since that is what
POSIX specifies is supplied with '*' [1]:

    A field width, or precision, or both, may be indicated by an
    <asterisk> ('*'). In this case an argument of type int supplies the
    field width or precision.

E.g., in an invocation like this where the field width is supplied with
another argument:

    $ printf '%0*d\n' 5 1
    00001

I think my original reason for wanting a different printf variant was
escaping that limit.

For generating bytes where the values don't matter, in GNU coreutils we
frequently use 'yes' for that. As a result, the implementation is very
overengineered. E.g., it will use zero-copy since coreutils-9.11. Before
that, it would reuse ARGV if it was contiguous and the arguments used to
determine the printed string were long enough such that we weren't calling
read/write on a tiny string, or it would generate a large buffer before
using read and write. Here is the performance from before and after
zero-copy was introduced on a beefy Power10 machine, stolen from the
commit message [2]:

   IBM,9043-MRX $ taskset 1 yes | taskset 2 pv > /dev/null
   ... [11.6GiB/s]
   IBM,9043-MRX $ taskset 1 src/yes | taskset 2 pv > /dev/null
   ... [175GiB/s]

This was introduced around the time of the copy-fail vulnerability and
various zero-copy vulnerabilities, where there was serious consideration
given to nerfing splice and vmsplice. Luckily for us, that never
happened.

FreeBSD, and therefore, I assume, macOS use something similar to our
previous read/write method, which is fast enough. I think OpenBSD and
NetBSD use puts on ARGV or the "y" string, so it's relatively slow.

GNU 'tr' doesn't handle multi-byte characters, so you can adjust that
output if needed and remain relatively quick:

    $ timeout 10 yes 0 | LC_ALL=C tr -d '\n' | pv -r > /dev/null
    [ 616MiB/s]

I'm not sure if macOS 'tr' handles multibyte characters. But it might be
worth checking if they do, and if they keep a fast case for single-byte
locales. Doing that without duplicating code is quite tricky, which is
part of the reason why we haven't done it in GNU 'tr' yet (but recent
releases have improved multi-byte character support for other utilities,
so hopefully it is handled soon).

Probably more information than you need, but hopefully it helps you
generate slop sometimes needed for testing. :)

Collin

[1] https://pubs.opengroup.org/onlinepubs/9799919799/functions/printf.html
[2] https://github.com/coreutils/coreutils/commit/2b1c059e6a06eebbb721d010b1221ec54200cc33