From: Jeff King <peff@peff.net>
To: git@vger.kernel.org
Cc: "René Scharfe" <l.s.r@web.de>, "Johannes Sixt" <j6t@kdbg.org>,
"Junio C Hamano" <gitster@pobox.com>
Subject: [PATCH v4 1/5] t9300: factor out portable "head -c" replacement
Date: Thu, 30 Jun 2016 05:07:54 -0400 [thread overview]
Message-ID: <20160630090753.GA17463@sigill.intra.peff.net> (raw)
In-Reply-To: <20160630090614.GA16725@sigill.intra.peff.net>
In shell scripts it is sometimes useful to be able to read
exactly N bytes from a pipe. Doing this portably turns out
to be surprisingly difficult.
We want a solution that:
- is portable
- never reads more than N bytes due to buffering (which
would mean those bytes are not available to the next
program to read from the same pipe)
- handles partial reads by looping until N bytes or read
(or we see EOF)
- is resilient to stray signals giving us EINTR while
trying to read (even though we don't send them, things
like SIGWINCH could cause apparently-random failures)
Some possible solutions are:
- "head -c" is not portable, and implementations may
buffer (though GNU head does not)
- "read -N" is a bash-ism, and thus not portable
- "dd bs=$n count=1" does not handle partial reads. GNU dd
has iflags=fullblock, but that is not portable
- "dd bs=1 count=$n" fixes the partial read problem (all
reads are 1-byte, so there can be no partial response).
It does make a lot of write() calls, but for our tests
that's unlikely to matter. It's fairly portable. We
already use it in our tests, and it's unlikely that
implementations would screw up any of our criteria. The
most unknown one would be signal handling.
- perl can do a sysread() loop pretty easily. On my Linux
system, at least, it seems to restart the read() call
automatically. If that turns out not to be portable,
though, it would be easy for us to handle it.
That makes the perl solution the least bad (because we
conveniently omitted "length of code" as a criterion).
It's also what t9300 is currently using, so we can just pull
the implementation from there.
Signed-off-by: Jeff King <peff@peff.net>
---
t/t9300-fast-import.sh | 23 +++--------------------
t/test-lib-functions.sh | 14 ++++++++++++++
2 files changed, 17 insertions(+), 20 deletions(-)
diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 74d740d..2e0ba3e 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -7,23 +7,6 @@ test_description='test git fast-import utility'
. ./test-lib.sh
. "$TEST_DIRECTORY"/diff-lib.sh ;# test-lib chdir's into trash
-# Print $1 bytes from stdin to stdout.
-#
-# This could be written as "head -c $1", but IRIX "head" does not
-# support the -c option.
-head_c () {
- perl -e '
- my $len = $ARGV[1];
- while ($len > 0) {
- my $s;
- my $nread = sysread(STDIN, $s, $len);
- die "cannot read: $!" unless defined($nread);
- print $s;
- $len -= $nread;
- }
- ' - "$1"
-}
-
verify_packs () {
for p in .git/objects/pack/*.pack
do
@@ -2481,7 +2464,7 @@ test_expect_success PIPE 'R: copy using cat-file' '
read blob_id type size <&3 &&
echo "$blob_id $type $size" >response &&
- head_c $size >blob <&3 &&
+ test_copy_bytes $size >blob <&3 &&
read newline <&3 &&
cat <<-EOF &&
@@ -2524,7 +2507,7 @@ test_expect_success PIPE 'R: print blob mid-commit' '
EOF
read blob_id type size <&3 &&
- head_c $size >actual <&3 &&
+ test_copy_bytes $size >actual <&3 &&
read newline <&3 &&
echo
@@ -2559,7 +2542,7 @@ test_expect_success PIPE 'R: print staged blob within commit' '
echo "cat-blob $to_get" &&
read blob_id type size <&3 &&
- head_c $size >actual <&3 &&
+ test_copy_bytes $size >actual <&3 &&
read newline <&3 &&
echo deleteall
diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index 48884d5..90856d6 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -961,3 +961,17 @@ test_env () {
done
)
}
+
+# Read up to "$1" bytes (or to EOF) from stdin and write them to stdout.
+test_copy_bytes () {
+ perl -e '
+ my $len = $ARGV[1];
+ while ($len > 0) {
+ my $s;
+ my $nread = sysread(STDIN, $s, $len);
+ die "cannot read: $!" unless defined($nread);
+ print $s;
+ $len -= $nread;
+ }
+ ' - "$1"
+}
--
2.9.0.317.g65b4e7c
next prev parent reply other threads:[~2016-06-30 9:14 UTC|newest]
Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-30 9:06 [PATCH v4 0/5] friendlier handling of overflows in archive-tar Jeff King
2016-06-30 9:07 ` Jeff King [this message]
2016-07-01 4:45 ` [PATCH v4 1/5] t9300: factor out portable "head -c" replacement Eric Sunshine
2016-07-01 17:23 ` Junio C Hamano
2016-07-01 18:01 ` Jeff King
2016-06-30 9:08 ` [PATCH v4 2/5] t5000: test tar files that overflow ustar headers Jeff King
2016-07-14 15:47 ` Johannes Schindelin
2016-07-14 16:45 ` Johannes Sixt
2016-07-14 17:08 ` Junio C Hamano
2016-07-14 20:52 ` Johannes Sixt
2016-07-14 21:32 ` Jeff King
2016-07-14 22:30 ` Junio C Hamano
2016-07-14 22:38 ` Jeff King
2016-07-15 13:37 ` Torsten Bögershausen
2016-07-15 13:46 ` Jeff King
2016-07-14 22:26 ` Junio C Hamano
2016-07-14 18:24 ` Jeff King
2016-07-14 18:21 ` Jeff King
2016-07-14 20:00 ` Junio C Hamano
2016-07-14 20:03 ` Junio C Hamano
2016-07-14 20:14 ` Jeff King
2016-07-14 20:09 ` Junio C Hamano
2016-07-14 20:10 ` Jeff King
2016-07-14 20:22 ` Junio C Hamano
2016-07-14 20:27 ` Jeff King
2016-07-14 20:34 ` Junio C Hamano
2016-07-14 20:43 ` [PATCH v2 0/2] ulong may only be 32-bit wide Junio C Hamano
2016-07-14 20:43 ` [PATCH v2 1/2] t0006: skip "far in the future" test when unsigned long is not long enough Junio C Hamano
2016-07-14 20:43 ` [PATCH v2 2/2] archive-tar: huge offset and future timestamps would not work on 32-bit Junio C Hamano
2016-07-14 22:20 ` Jeff King
2016-07-14 22:36 ` Junio C Hamano
2016-07-16 6:28 ` Duy Nguyen
2016-07-15 15:10 ` [PATCH v4 2/5] t5000: test tar files that overflow ustar headers Johannes Schindelin
2016-07-15 16:49 ` Junio C Hamano
2016-06-30 9:09 ` [PATCH v4 3/5] archive-tar: write extended headers for file sizes >= 8GB Jeff King
2016-07-14 16:48 ` Johannes Sixt
2016-07-14 17:11 ` Junio C Hamano
2016-07-14 18:16 ` Jeff King
2016-07-15 2:59 ` Torsten Bögershausen
2016-06-30 9:09 ` [PATCH v4 4/5] archive-tar: write extended headers for far-future mtime Jeff King
2016-06-30 9:09 ` [PATCH v4 5/5] archive-tar: drop return value Jeff King
2016-06-30 9:14 ` [PATCH v4 6/5] t5000: use test_match_signal Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160630090753.GA17463@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=j6t@kdbg.org \
--cc=l.s.r@web.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).