[PATCH 0/2] t: work around bugs in Dash v0.5.13

[PATCH 0/2] t: work around bugs in Dash v0.5.13

[Patrick Steinhardt]

Hi,

while testing my `set -e` change for our test suite I was also playing
around with Dash v0.5.13.1 and noticed that multiple of our tests fail
with that version. As it turns out, some of these test failures are
actual bugs in Dash itself that have been introduced in v0.5.13.

There's ultimately two bugs:

  - Dash inserts CTLMBCHAR byte sequences into heredocs around multibyte
    characters. This bug still exists with the current "master" branch
    of Dash.

  - Dash may swallow some bytes when reading data, which has already
    been fixed.

I've Cc'd Herbert, maintainer of Dash.

Thanks!

Patrick

---
Patrick Steinhardt (2):
      t: work around multibyte bug in quoted heredocs with Dash v0.5.13
      t9300: work around partial read bug in Dash v0.5.13

 t/t0300-credentials.sh   |  2 +-
 t/t3430-rebase-merges.sh |  4 ++--
 t/t3902-quoted.sh        | 14 +++++++-------
 t/t4014-format-patch.sh  |  8 ++++----
 t/t4201-shortlog.sh      |  2 +-
 t/t9001-send-email.sh    |  6 +++---
 t/t9300-fast-import.sh   | 32 ++++++++++++++------------------
 7 files changed, 32 insertions(+), 36 deletions(-)


---
base-commit: 270e10ad6dda3379ea0da7efd11e4fbf2cd7a325
change-id: 20260401-pks-tests-with-dash-ea71cbb36958

[PATCH 1/2] t: work around multibyte bug in quoted heredocs with Dash v0.5.13

[Patrick Steinhardt]

When executing our test suite with Dash v0.5.13.2 one can observe
several test failures that all have the same symptoms: we have a quoted
heredoc that contains multibyte characters, but the final data does not
match what we actually wanted to write. One such example is in t0300,
where we see the diffs like the following:

  --- expect-stdout	2026-04-01 07:25:45.249919440 +0000
  +++ stdout	2026-04-01 07:25:45.254919509 +0000
  @@ -1,5 +1,5 @@
   protocol=https
   host=example.com
  -path=perú.git
  +path=perú.git
   username=foo
   password=bar

While seemingly the same, the data that we've written via the heredoc
contains some invisible bytes. The expected hex representation of the
string is:

  7065 72c3 ba2e 6769 74                 per...git

But what we actually get instead is this string:

  7065 7285 02c3 ba02 852e 6769 74       per.......git

What's important to note here is that the multibyte character exists in
both versions. But in the broken version we see that the bytes are
wrapped in a sequence of "85 02" and "02 85". This is the CTLMBCHAR byte
sequence of Dash, which it uses internally to quote multibyte sequences.

As it turns out, this bug was introduced in c5bf970 (expand: Add
multi-byte support to pmatch, 2024-06-02), which adds multibyte support
to more contexts of Dash. One of these contexts seems to be in heredocs,
and Dash _does_ correctly unquote these multibyte sequences when using
an unquoted heredoc. But the bug seems to be that this unquoting does
not happen in quoted heredocs, and the bug still exists on the latest
"master" branch.

For now, work around the bug by using unquoted heredocs instead.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0300-credentials.sh   |  2 +-
 t/t3430-rebase-merges.sh |  4 ++--
 t/t3902-quoted.sh        | 14 +++++++-------
 t/t4014-format-patch.sh  |  8 ++++----
 t/t4201-shortlog.sh      |  2 +-
 t/t9001-send-email.sh    |  6 +++---
 6 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
index 07aa834d33..fda6760955 100755
--- a/t/t0300-credentials.sh
+++ b/t/t0300-credentials.sh
@@ -675,7 +675,7 @@ test_expect_success 'match percent-encoded values' '
 test_expect_success 'match percent-encoded UTF-8 values in path' '
 	test_config credential.https://example.com.useHttpPath true &&
 	test_config credential.https://example.com/perú.git.helper "$HELPER" &&
-	check fill <<-\EOF
+	check fill <<-EOF
 	url=https://example.com/per%C3%BA.git
 	--
 	protocol=https
diff --git a/t/t3430-rebase-merges.sh b/t/t3430-rebase-merges.sh
index cc627e34a7..d3dffbb830 100755
--- a/t/t3430-rebase-merges.sh
+++ b/t/t3430-rebase-merges.sh
@@ -507,9 +507,9 @@ test_expect_success 'octopus merges' '
 	git rebase -i --force-rebase -r HEAD^^ &&
 	test "Hank" = "$(git show -s --format=%an HEAD)" &&
 	test "$before" != $(git rev-parse HEAD) &&
-	test_cmp_graph HEAD^^.. <<-\EOF
+	test_cmp_graph HEAD^^.. <<-EOF
 	*-.   Tüntenfüsch
-	|\ \
+	|\\ \\
 	| | * three
 	| * | two
 	| |/
diff --git a/t/t3902-quoted.sh b/t/t3902-quoted.sh
index f528008c36..c3b45d991e 100755
--- a/t/t3902-quoted.sh
+++ b/t/t3902-quoted.sh
@@ -60,16 +60,16 @@ With SP in it
 "\346\277\261\351\207\216\347\264\224"
 EOF
 
-cat >expect.raw <<\EOF
+cat >expect.raw <<EOF
 Name
-"Name and a\nLF"
-"Name and an\tHT"
-"Name\""
+"Name and a\\nLF"
+"Name and an\\tHT"
+"Name\\""
 With SP in it
-"濱野\t純"
-"濱野\n純"
+"濱野\\t純"
+"濱野\\n純"
 濱野 純
-"濱野\"純"
+"濱野\\"純"
 濱野/file
 濱野純
 EOF
diff --git a/t/t4014-format-patch.sh b/t/t4014-format-patch.sh
index bcdb944017..d22b7f348e 100755
--- a/t/t4014-format-patch.sh
+++ b/t/t4014-format-patch.sh
@@ -1285,7 +1285,7 @@ test_expect_success 'format-patch wraps extremely long from-header (rfc2047)' '
 	check_author "Foö Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar"
 '
 
-cat >expect <<'EOF'
+cat >expect <<EOF
 From: Foö Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar
  Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo
  Bar Foo Bar Foo Bar Foo Bar <author@example.com>
@@ -1300,7 +1300,7 @@ test_expect_success 'format-patch wraps extremely long from-header (non-ASCII wi
 	test_cmp expect actual
 '
 
-cat >expect <<'EOF'
+cat >expect <<EOF
 Subject: [PATCH] Foö
 EOF
 test_expect_success 'subject lines are unencoded with --no-encode-email-headers' '
@@ -1312,7 +1312,7 @@ test_expect_success 'subject lines are unencoded with --no-encode-email-headers'
 	test_cmp expect actual
 '
 
-cat >expect <<'EOF'
+cat >expect <<EOF
 Subject: [PATCH] Foö
 EOF
 test_expect_success 'subject lines are unencoded with format.encodeEmailHeaders=false' '
@@ -1531,7 +1531,7 @@ test_expect_success 'in-body headers trigger content encoding' '
 	test_env GIT_AUTHOR_NAME="éxötìc" test_commit exotic &&
 	test_when_finished "git reset --hard HEAD^" &&
 	git format-patch -1 --stdout --from >patch &&
-	cat >expect <<-\EOF &&
+	cat >expect <<-EOF &&
 	From: C O Mitter <committer@example.com>
 	Content-Type: text/plain; charset=UTF-8
 
diff --git a/t/t4201-shortlog.sh b/t/t4201-shortlog.sh
index 5f23fc147b..d73c9f5204 100755
--- a/t/t4201-shortlog.sh
+++ b/t/t4201-shortlog.sh
@@ -105,7 +105,7 @@ test_expect_success 'output from user-defined format is re-wrapped' '
 '
 
 test_expect_success !MINGW,ICONV 'shortlog wrapping' '
-	cat >expect <<\EOF &&
+	cat >expect <<EOF &&
 A U Thor (5):
       Test
       This is a very, very long first line for the commit message to see if
diff --git a/t/t9001-send-email.sh b/t/t9001-send-email.sh
index 24f6c76aee..3612d32d39 100755
--- a/t/t9001-send-email.sh
+++ b/t/t9001-send-email.sh
@@ -1649,7 +1649,7 @@ test_expect_success $PREREQ 'To headers from files reset each patch' '
 '
 
 test_expect_success $PREREQ 'setup expect' '
-cat >email-using-8bit <<\EOF
+cat >email-using-8bit <<EOF
 From fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001
 Message-ID: <bogus-message-id@example.com>
 From: author@example.com
@@ -1735,7 +1735,7 @@ test_expect_success $PREREQ '--8bit-encoding overrides sendemail.8bitEncoding' '
 '
 
 test_expect_success $PREREQ 'setup expect' '
-	cat >email-using-8bit <<-\EOF
+	cat >email-using-8bit <<-EOF
 	From fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001
 	Message-ID: <bogus-message-id@example.com>
 	From: author@example.com
@@ -1764,7 +1764,7 @@ test_expect_success $PREREQ '--8bit-encoding also treats subject' '
 '
 
 test_expect_success $PREREQ 'setup expect' '
-	cat >email-using-8bit <<-\EOF
+	cat >email-using-8bit <<-EOF
 	From fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001
 	Message-ID: <bogus-message-id@example.com>
 	From: A U Thor <author@example.com>

-- 
2.53.0.1185.g05d4b7b318.dirty

Re: [PATCH 1/2] t: work around multibyte bug in quoted heredocs with Dash v0.5.13

[Eric Sunshine]

On Wed, Apr 1, 2026 at 6:59 AM Patrick Steinhardt <ps@pks.im> wrote:
> When executing our test suite with Dash v0.5.13.2 one can observe
> several test failures that all have the same symptoms: we have a quoted
> heredoc that contains multibyte characters, but the final data does not
> match what we actually wanted to write. One such example is in t0300,
> where we see the diffs like the following:
> [...]
> For now, work around the bug by using unquoted heredocs instead.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
> diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
> @@ -675,7 +675,7 @@ test_expect_success 'match percent-encoded values' '
>  test_expect_success 'match percent-encoded UTF-8 values in path' '
>         test_config credential.https://example.com.useHttpPath true &&
>         test_config credential.https://example.com/perú.git.helper "$HELPER" &&
> -       check fill <<-\EOF
> +       check fill <<-EOF
>         url=https://example.com/per%C3%BA.git

Oof, this is the sort of change which cries out for an in-code
comment, since we can't expect that future reviewers will remember
this specific workaround in this specific script, and we can't expect
that people working on this code will think to check the history to
learn why the heredoc is unquoted. Without such a comment, someone
working on this file for some purpose or another (or even a GSoC
applicant looking for a microproject) will come along and "modernize"
this script by adding back the heredoc quoting which this patch
removes.

To prevent someone from "breaking" your "fix", at minimum, please add
an explanatory comment somewhere near the top of the script which
pretty much duplicates the information from the commit message:

    # *DONTQUOTE*: ...explanation of dash bug ...

and reference that comment at each location you "fix":

    # NOTE: Don't quote heredoc; see *DONTQUOTE* above.
    check fill <<-EOF
    ...

Re: [PATCH 1/2] t: work around multibyte bug in quoted heredocs with Dash v0.5.13

[Patrick Steinhardt]

On Wed, Apr 01, 2026 at 12:21:41PM -0400, Eric Sunshine wrote:
> On Wed, Apr 1, 2026 at 6:59 AM Patrick Steinhardt <ps@pks.im> wrote:
> > When executing our test suite with Dash v0.5.13.2 one can observe
> > several test failures that all have the same symptoms: we have a quoted
> > heredoc that contains multibyte characters, but the final data does not
> > match what we actually wanted to write. One such example is in t0300,
> > where we see the diffs like the following:
> > [...]
> > For now, work around the bug by using unquoted heredocs instead.
> >
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> > diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
> > @@ -675,7 +675,7 @@ test_expect_success 'match percent-encoded values' '
> >  test_expect_success 'match percent-encoded UTF-8 values in path' '
> >         test_config credential.https://example.com.useHttpPath true &&
> >         test_config credential.https://example.com/perú.git.helper "$HELPER" &&
> > -       check fill <<-\EOF
> > +       check fill <<-EOF
> >         url=https://example.com/per%C3%BA.git
> 
> Oof, this is the sort of change which cries out for an in-code
> comment, since we can't expect that future reviewers will remember
> this specific workaround in this specific script, and we can't expect
> that people working on this code will think to check the history to
> learn why the heredoc is unquoted. Without such a comment, someone
> working on this file for some purpose or another (or even a GSoC
> applicant looking for a microproject) will come along and "modernize"
> this script by adding back the heredoc quoting which this patch
> removes.
> 
> To prevent someone from "breaking" your "fix", at minimum, please add
> an explanatory comment somewhere near the top of the script which
> pretty much duplicates the information from the commit message:
> 
>     # *DONTQUOTE*: ...explanation of dash bug ...
> 
> and reference that comment at each location you "fix":
> 
>     # NOTE: Don't quote heredoc; see *DONTQUOTE* above.
>     check fill <<-EOF
>     ...

True indeed. I'll just add a short comment to the individual callsites.
Thanks!

Patrick

[PATCH 2/2] t9300: work around partial read bug in Dash v0.5.13

[Patrick Steinhardt]

When executing t9300 with Dash v0.5.13.1 we can see that the test hangs
completely with the following (condensed) trace:

  git fast-import
  + error=1
  + read output
  + cat input
  + echo checkpoint
  + echo progress checkpoint
  + test rogress checkpoint = progress checkpoint
  + test rogress checkpoint = UNEXPECTED
  + echo cruft: rogress checkpoint
  cruft: rogress checkpoint
  + read output
  + test  = progress checkpoint
  + test  = UNEXPECTED
  + echo cruft:
  cruft:
  + read output

Basically, what's happening here is that we spawn git-fast-import(1) and
wait for it to output a certain string, "progress checkpoint". Curiously
though, what we end up reading is "rogress checkpoint" -- so the first
byte of the expected string is missing.

Same as in the preceding commit, this seems to be a bug in Dash itself
that bisects to c5bf970 (expand: Add multi-byte support to pmatch,
2024-06-02). But other than in the preceding commit, this bug has
already been fixed upstream in 079059a (input: Fix heap-buffer-overflow
in preadbuffer on long lines, 2026-02-11), which is part of v0.5.13.2.

For now though, work around the bug by waiting for the expected output
in a different way. There is no good reason why one version should work
better than the other, but at least the new version doesn't exhibit the
bug. And, if you ask me, it's also slightly easier to read.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t9300-fast-import.sh | 32 ++++++++++++++------------------
 1 file changed, 14 insertions(+), 18 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 5685cce6fe..479437760b 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -3635,25 +3635,21 @@ background_import_then_checkpoint () {
 		echo "progress checkpoint"
 	) >&8 &
 
-	error=1 ;# assume the worst
-	while read output <&9
-	do
-		if test "$output" = "progress checkpoint"
-		then
-			error=0
-			break
-		elif test "$output" = "UNEXPECTED"
-		then
-			break
-		fi
-		# otherwise ignore cruft
-		echo >&2 "cruft: $output"
-	done
+	last=$(
+		while read output <&9
+		do
+			if test "$output" = "progress checkpoint" || test "$output" = "UNEXPECTED"
+			then
+				echo "$output"
+				break
+			else
+				# otherwise ignore cruft
+				echo >&2 "cruft: $output"
+			fi
+		done
+	)
 
-	if test $error -eq 1
-	then
-		false
-	fi
+	test "$last" = "progress checkpoint"
 }
 
 background_import_still_running () {

-- 
2.53.0.1185.g05d4b7b318.dirty

[PATCH v2 0/2] t: work around bugs in Dash v0.5.13

[Patrick Steinhardt]

Hi,

while testing my `set -e` change for our test suite I was also playing
around with Dash v0.5.13.1 and noticed that multiple of our tests fail
with that version. As it turns out, some of these test failures are
actual bugs in Dash itself that have been introduced in v0.5.13.

There's ultimately two bugs:

  - Dash inserts CTLMBCHAR byte sequences into heredocs around multibyte
    characters. This bug still exists with the current "master" branch
    of Dash.

  - Dash may swallow some bytes when reading data, which has already
    been fixed.

I've Cc'd Herbert, maintainer of Dash.

Changes in v2:
  - Add comment to heredocs explaining why they should never be quoted.
  - Link to v1: https://patch.msgid.link/20260401-pks-tests-with-dash-v1-0-d70b5040aa5d@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (2):
      t: work around multibyte bug in quoted heredocs with Dash v0.5.13
      t9300: work around partial read bug in Dash v0.5.13

 t/t0300-credentials.sh   |  4 +++-
 t/t3430-rebase-merges.sh |  6 ++++--
 t/t3902-quoted.sh        | 16 +++++++++-------
 t/t4014-format-patch.sh  | 16 ++++++++++++----
 t/t4201-shortlog.sh      |  4 +++-
 t/t9001-send-email.sh    | 12 +++++++++---
 t/t9300-fast-import.sh   | 32 ++++++++++++++------------------
 7 files changed, 54 insertions(+), 36 deletions(-)

Range-diff versus v1:

1:  192b5b4330 ! 1:  5367e05e0f t: work around multibyte bug in quoted heredocs with Dash v0.5.13
    @@ t/t0300-credentials.sh: test_expect_success 'match percent-encoded values' '
      	test_config credential.https://example.com.useHttpPath true &&
      	test_config credential.https://example.com/perú.git.helper "$HELPER" &&
     -	check fill <<-\EOF
    ++	# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
    ++	# that contain multibyte chars.
     +	check fill <<-EOF
      	url=https://example.com/per%C3%BA.git
      	--
    @@ t/t3430-rebase-merges.sh: test_expect_success 'octopus merges' '
      	test "Hank" = "$(git show -s --format=%an HEAD)" &&
      	test "$before" != $(git rev-parse HEAD) &&
     -	test_cmp_graph HEAD^^.. <<-\EOF
    ++	# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
    ++	# that contain multibyte chars.
     +	test_cmp_graph HEAD^^.. <<-EOF
      	*-.   Tüntenfüsch
     -	|\ \
    @@ t/t3902-quoted.sh: With SP in it
      EOF
      
     -cat >expect.raw <<\EOF
    ++# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
    ++# that contain multibyte chars.
     +cat >expect.raw <<EOF
      Name
     -"Name and a\nLF"
    @@ t/t4014-format-patch.sh: test_expect_success 'format-patch wraps extremely long
      '
      
     -cat >expect <<'EOF'
    ++# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
    ++# that contain multibyte chars.
     +cat >expect <<EOF
      From: Foö Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar
       Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo
    @@ t/t4014-format-patch.sh: test_expect_success 'format-patch wraps extremely long
      '
      
     -cat >expect <<'EOF'
    ++# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
    ++# that contain multibyte chars.
     +cat >expect <<EOF
      Subject: [PATCH] Foö
      EOF
    @@ t/t4014-format-patch.sh: test_expect_success 'subject lines are unencoded with -
      '
      
     -cat >expect <<'EOF'
    ++# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
    ++# that contain multibyte chars.
     +cat >expect <<EOF
      Subject: [PATCH] Foö
      EOF
    @@ t/t4014-format-patch.sh: test_expect_success 'in-body headers trigger content en
      	test_when_finished "git reset --hard HEAD^" &&
      	git format-patch -1 --stdout --from >patch &&
     -	cat >expect <<-\EOF &&
    ++	# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
    ++	# that contain multibyte chars.
     +	cat >expect <<-EOF &&
      	From: C O Mitter <committer@example.com>
      	Content-Type: text/plain; charset=UTF-8
    @@ t/t4201-shortlog.sh: test_expect_success 'output from user-defined format is re-
      
      test_expect_success !MINGW,ICONV 'shortlog wrapping' '
     -	cat >expect <<\EOF &&
    ++	# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
    ++	# that contain multibyte chars.
     +	cat >expect <<EOF &&
      A U Thor (5):
            Test
    @@ t/t9001-send-email.sh: test_expect_success $PREREQ 'To headers from files reset
      
      test_expect_success $PREREQ 'setup expect' '
     -cat >email-using-8bit <<\EOF
    ++# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
    ++# that contain multibyte chars.
     +cat >email-using-8bit <<EOF
      From fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001
      Message-ID: <bogus-message-id@example.com>
    @@ t/t9001-send-email.sh: test_expect_success $PREREQ '--8bit-encoding overrides se
      
      test_expect_success $PREREQ 'setup expect' '
     -	cat >email-using-8bit <<-\EOF
    ++	# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
    ++	# that contain multibyte chars.
     +	cat >email-using-8bit <<-EOF
      	From fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001
      	Message-ID: <bogus-message-id@example.com>
    @@ t/t9001-send-email.sh: test_expect_success $PREREQ '--8bit-encoding also treats
      
      test_expect_success $PREREQ 'setup expect' '
     -	cat >email-using-8bit <<-\EOF
    ++	# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
    ++	# that contain multibyte chars.
     +	cat >email-using-8bit <<-EOF
      	From fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001
      	Message-ID: <bogus-message-id@example.com>
2:  db99459b3f = 2:  364d37fcc7 t9300: work around partial read bug in Dash v0.5.13

---
base-commit: 270e10ad6dda3379ea0da7efd11e4fbf2cd7a325
change-id: 20260401-pks-tests-with-dash-ea71cbb36958

[PATCH v2 1/2] t: work around multibyte bug in quoted heredocs with Dash v0.5.13

[Patrick Steinhardt]

When executing our test suite with Dash v0.5.13.2 one can observe
several test failures that all have the same symptoms: we have a quoted
heredoc that contains multibyte characters, but the final data does not
match what we actually wanted to write. One such example is in t0300,
where we see the diffs like the following:

  --- expect-stdout	2026-04-01 07:25:45.249919440 +0000
  +++ stdout	2026-04-01 07:25:45.254919509 +0000
  @@ -1,5 +1,5 @@
   protocol=https
   host=example.com
  -path=perú.git
  +path=perú.git
   username=foo
   password=bar

While seemingly the same, the data that we've written via the heredoc
contains some invisible bytes. The expected hex representation of the
string is:

  7065 72c3 ba2e 6769 74                 per...git

But what we actually get instead is this string:

  7065 7285 02c3 ba02 852e 6769 74       per.......git

What's important to note here is that the multibyte character exists in
both versions. But in the broken version we see that the bytes are
wrapped in a sequence of "85 02" and "02 85". This is the CTLMBCHAR byte
sequence of Dash, which it uses internally to quote multibyte sequences.

As it turns out, this bug was introduced in c5bf970 (expand: Add
multi-byte support to pmatch, 2024-06-02), which adds multibyte support
to more contexts of Dash. One of these contexts seems to be in heredocs,
and Dash _does_ correctly unquote these multibyte sequences when using
an unquoted heredoc. But the bug seems to be that this unquoting does
not happen in quoted heredocs, and the bug still exists on the latest
"master" branch.

For now, work around the bug by using unquoted heredocs instead.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0300-credentials.sh   |  4 +++-
 t/t3430-rebase-merges.sh |  6 ++++--
 t/t3902-quoted.sh        | 16 +++++++++-------
 t/t4014-format-patch.sh  | 16 ++++++++++++----
 t/t4201-shortlog.sh      |  4 +++-
 t/t9001-send-email.sh    | 12 +++++++++---
 6 files changed, 40 insertions(+), 18 deletions(-)

diff --git a/t/t0300-credentials.sh b/t/t0300-credentials.sh
index 07aa834d33..64ead1571a 100755
--- a/t/t0300-credentials.sh
+++ b/t/t0300-credentials.sh
@@ -675,7 +675,9 @@ test_expect_success 'match percent-encoded values' '
 test_expect_success 'match percent-encoded UTF-8 values in path' '
 	test_config credential.https://example.com.useHttpPath true &&
 	test_config credential.https://example.com/perú.git.helper "$HELPER" &&
-	check fill <<-\EOF
+	# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
+	# that contain multibyte chars.
+	check fill <<-EOF
 	url=https://example.com/per%C3%BA.git
 	--
 	protocol=https
diff --git a/t/t3430-rebase-merges.sh b/t/t3430-rebase-merges.sh
index cc627e34a7..84b2d0e664 100755
--- a/t/t3430-rebase-merges.sh
+++ b/t/t3430-rebase-merges.sh
@@ -507,9 +507,11 @@ test_expect_success 'octopus merges' '
 	git rebase -i --force-rebase -r HEAD^^ &&
 	test "Hank" = "$(git show -s --format=%an HEAD)" &&
 	test "$before" != $(git rev-parse HEAD) &&
-	test_cmp_graph HEAD^^.. <<-\EOF
+	# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
+	# that contain multibyte chars.
+	test_cmp_graph HEAD^^.. <<-EOF
 	*-.   Tüntenfüsch
-	|\ \
+	|\\ \\
 	| | * three
 	| * | two
 	| |/
diff --git a/t/t3902-quoted.sh b/t/t3902-quoted.sh
index f528008c36..8660ec5cb0 100755
--- a/t/t3902-quoted.sh
+++ b/t/t3902-quoted.sh
@@ -60,16 +60,18 @@ With SP in it
 "\346\277\261\351\207\216\347\264\224"
 EOF
 
-cat >expect.raw <<\EOF
+# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
+# that contain multibyte chars.
+cat >expect.raw <<EOF
 Name
-"Name and a\nLF"
-"Name and an\tHT"
-"Name\""
+"Name and a\\nLF"
+"Name and an\\tHT"
+"Name\\""
 With SP in it
-"濱野\t純"
-"濱野\n純"
+"濱野\\t純"
+"濱野\\n純"
 濱野 純
-"濱野\"純"
+"濱野\\"純"
 濱野/file
 濱野純
 EOF
diff --git a/t/t4014-format-patch.sh b/t/t4014-format-patch.sh
index bcdb944017..fc37a7ea42 100755
--- a/t/t4014-format-patch.sh
+++ b/t/t4014-format-patch.sh
@@ -1285,7 +1285,9 @@ test_expect_success 'format-patch wraps extremely long from-header (rfc2047)' '
 	check_author "Foö Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar"
 '
 
-cat >expect <<'EOF'
+# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
+# that contain multibyte chars.
+cat >expect <<EOF
 From: Foö Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar
  Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo Bar Foo
  Bar Foo Bar Foo Bar Foo Bar <author@example.com>
@@ -1300,7 +1302,9 @@ test_expect_success 'format-patch wraps extremely long from-header (non-ASCII wi
 	test_cmp expect actual
 '
 
-cat >expect <<'EOF'
+# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
+# that contain multibyte chars.
+cat >expect <<EOF
 Subject: [PATCH] Foö
 EOF
 test_expect_success 'subject lines are unencoded with --no-encode-email-headers' '
@@ -1312,7 +1316,9 @@ test_expect_success 'subject lines are unencoded with --no-encode-email-headers'
 	test_cmp expect actual
 '
 
-cat >expect <<'EOF'
+# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
+# that contain multibyte chars.
+cat >expect <<EOF
 Subject: [PATCH] Foö
 EOF
 test_expect_success 'subject lines are unencoded with format.encodeEmailHeaders=false' '
@@ -1531,7 +1537,9 @@ test_expect_success 'in-body headers trigger content encoding' '
 	test_env GIT_AUTHOR_NAME="éxötìc" test_commit exotic &&
 	test_when_finished "git reset --hard HEAD^" &&
 	git format-patch -1 --stdout --from >patch &&
-	cat >expect <<-\EOF &&
+	# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
+	# that contain multibyte chars.
+	cat >expect <<-EOF &&
 	From: C O Mitter <committer@example.com>
 	Content-Type: text/plain; charset=UTF-8
 
diff --git a/t/t4201-shortlog.sh b/t/t4201-shortlog.sh
index 5f23fc147b..9f41d56d9a 100755
--- a/t/t4201-shortlog.sh
+++ b/t/t4201-shortlog.sh
@@ -105,7 +105,9 @@ test_expect_success 'output from user-defined format is re-wrapped' '
 '
 
 test_expect_success !MINGW,ICONV 'shortlog wrapping' '
-	cat >expect <<\EOF &&
+	# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
+	# that contain multibyte chars.
+	cat >expect <<EOF &&
 A U Thor (5):
       Test
       This is a very, very long first line for the commit message to see if
diff --git a/t/t9001-send-email.sh b/t/t9001-send-email.sh
index 24f6c76aee..e7ab645a3d 100755
--- a/t/t9001-send-email.sh
+++ b/t/t9001-send-email.sh
@@ -1649,7 +1649,9 @@ test_expect_success $PREREQ 'To headers from files reset each patch' '
 '
 
 test_expect_success $PREREQ 'setup expect' '
-cat >email-using-8bit <<\EOF
+# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
+# that contain multibyte chars.
+cat >email-using-8bit <<EOF
 From fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001
 Message-ID: <bogus-message-id@example.com>
 From: author@example.com
@@ -1735,7 +1737,9 @@ test_expect_success $PREREQ '--8bit-encoding overrides sendemail.8bitEncoding' '
 '
 
 test_expect_success $PREREQ 'setup expect' '
-	cat >email-using-8bit <<-\EOF
+	# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
+	# that contain multibyte chars.
+	cat >email-using-8bit <<-EOF
 	From fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001
 	Message-ID: <bogus-message-id@example.com>
 	From: author@example.com
@@ -1764,7 +1768,9 @@ test_expect_success $PREREQ '--8bit-encoding also treats subject' '
 '
 
 test_expect_success $PREREQ 'setup expect' '
-	cat >email-using-8bit <<-\EOF
+	# NOTE: do not quote this heredoc, Dash 0.5.13 has a bug with heredocs
+	# that contain multibyte chars.
+	cat >email-using-8bit <<-EOF
 	From fe6ecc66ece37198fe5db91fa2fc41d9f4fe5cc4 Mon Sep 17 00:00:00 2001
 	Message-ID: <bogus-message-id@example.com>
 	From: A U Thor <author@example.com>

-- 
2.53.0.1323.g189a785ab5.dirty

[PATCH] parser: Fix multi-byte output in here-doc with quoted delimiter

[Herbert Xu]

On Thu, Apr 02, 2026 at 08:51:18AM +0200, Patrick Steinhardt wrote:
> When executing our test suite with Dash v0.5.13.2 one can observe
> several test failures that all have the same symptoms: we have a quoted
> heredoc that contains multibyte characters, but the final data does not
> match what we actually wanted to write. One such example is in t0300,
> where we see the diffs like the following:
> 
>   --- expect-stdout	2026-04-01 07:25:45.249919440 +0000
>   +++ stdout	2026-04-01 07:25:45.254919509 +0000
>   @@ -1,5 +1,5 @@
>    protocol=https
>    host=example.com
>   -path=perú.git
>   +path=perú.git
>    username=foo
>    password=bar

Thanks for the report.

This patch should fix the problem.  Please let me know if there are
any more oustanding issues.

---8<---
For a here-document with a quoted delimiter, multi-byte characters
should be written out as is with no escaping.  Fix this by checking
for syntax == SQSYNTAX (the only time readtoken1 gets called with
SQSYNTAX is for such a here-document) before calling getmbc in
readtoken1.

Reported-by: Patrick Steinhardt <ps@pks.im>
Fixes: b12f136cc704 ("builtin: Process multi-byte characters in read(1)")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

diff --git a/src/parser.c b/src/parser.c
index bea4148..412e876 100644
--- a/src/parser.c
+++ b/src/parser.c
@@ -998,9 +998,13 @@ static char *dollarsq_escape(char *out)
 STATIC int
 readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)
 {
-	struct synstack synbase = { .syntax = syntax };
+	struct synstack synbase = {
+		.dblquote = syntax == DQSYNTAX,
+		.syntax = syntax,
+	};
 	int chkeofmark = checkkwd & CHKEOFMARK;
 	struct synstack *synstack = &synbase;
+	bool sqheredoc = syntax == SQSYNTAX;
 	struct nodelist *bqlist = NULL;
 	int dollarsq = 0;
 	int c = firstc;
@@ -1009,9 +1013,6 @@ readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)
 	size_t len;
 	char *out;
 
-	if (syntax == DQSYNTAX)
-		synstack->dblquote = 1;
-
 	STARTSTACKSTR(out);
 	loop: {	/* for each line, until end of word */
 #if ATTY
@@ -1035,7 +1036,8 @@ readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)
 				      out);
 			fieldsplitting = synstack->syntax == BASESYNTAX &&
 					 !synstack->varnest ? 4 : 0;
-			ml = getmbc(c, out, fieldsplitting);
+			ml = getmbc(c, out, fieldsplitting |
+					    (sqheredoc ? 2 : 0));
 			if (ml == 1) {
 				if (out == stackblock())
 					return TBLANK;
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH] parser: Fix multi-byte output in here-doc with quoted delimiter

[Herbert Xu]

On Thu, May 07, 2026 at 01:57:30PM +0800, Herbert Xu wrote:
> On Thu, Apr 02, 2026 at 08:51:18AM +0200, Patrick Steinhardt wrote:
> > When executing our test suite with Dash v0.5.13.2 one can observe
> > several test failures that all have the same symptoms: we have a quoted
> > heredoc that contains multibyte characters, but the final data does not
> > match what we actually wanted to write. One such example is in t0300,
> > where we see the diffs like the following:
> > 
> >   --- expect-stdout	2026-04-01 07:25:45.249919440 +0000
> >   +++ stdout	2026-04-01 07:25:45.254919509 +0000
> >   @@ -1,5 +1,5 @@
> >    protocol=https
> >    host=example.com
> >   -path=perú.git
> >   +path=perú.git
> >    username=foo
> >    password=bar
> 
> Thanks for the report.
> 
> This patch should fix the problem.  Please let me know if there are
> any more oustanding issues.

Oops, I forgot to cc the mailing list.  Sorry for the resend.

---8<---
For a here-document with a quoted delimiter, multi-byte characters
should be written out as is with no escaping.  Fix this by checking
for syntax == SQSYNTAX (the only time readtoken1 gets called with
SQSYNTAX is for such a here-document) before calling getmbc in
readtoken1.

Reported-by: Patrick Steinhardt <ps@pks.im>
Fixes: b12f136cc704 ("builtin: Process multi-byte characters in read(1)")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

diff --git a/src/parser.c b/src/parser.c
index bea4148..412e876 100644
--- a/src/parser.c
+++ b/src/parser.c
@@ -998,9 +998,13 @@ static char *dollarsq_escape(char *out)
 STATIC int
 readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)
 {
-	struct synstack synbase = { .syntax = syntax };
+	struct synstack synbase = {
+		.dblquote = syntax == DQSYNTAX,
+		.syntax = syntax,
+	};
 	int chkeofmark = checkkwd & CHKEOFMARK;
 	struct synstack *synstack = &synbase;
+	bool sqheredoc = syntax == SQSYNTAX;
 	struct nodelist *bqlist = NULL;
 	int dollarsq = 0;
 	int c = firstc;
@@ -1009,9 +1013,6 @@ readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)
 	size_t len;
 	char *out;
 
-	if (syntax == DQSYNTAX)
-		synstack->dblquote = 1;
-
 	STARTSTACKSTR(out);
 	loop: {	/* for each line, until end of word */
 #if ATTY
@@ -1035,7 +1036,8 @@ readtoken1(int firstc, char const *syntax, char *eofmark, int striptabs)
 				      out);
 			fieldsplitting = synstack->syntax == BASESYNTAX &&
 					 !synstack->varnest ? 4 : 0;
-			ml = getmbc(c, out, fieldsplitting);
+			ml = getmbc(c, out, fieldsplitting |
+					    (sqheredoc ? 2 : 0));
 			if (ml == 1) {
 				if (out == stackblock())
 					return TBLANK;
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

[PATCH v2 2/2] t9300: work around partial read bug in Dash v0.5.13

[Patrick Steinhardt]

When executing t9300 with Dash v0.5.13.1 we can see that the test hangs
completely with the following (condensed) trace:

  git fast-import
  + error=1
  + read output
  + cat input
  + echo checkpoint
  + echo progress checkpoint
  + test rogress checkpoint = progress checkpoint
  + test rogress checkpoint = UNEXPECTED
  + echo cruft: rogress checkpoint
  cruft: rogress checkpoint
  + read output
  + test  = progress checkpoint
  + test  = UNEXPECTED
  + echo cruft:
  cruft:
  + read output

Basically, what's happening here is that we spawn git-fast-import(1) and
wait for it to output a certain string, "progress checkpoint". Curiously
though, what we end up reading is "rogress checkpoint" -- so the first
byte of the expected string is missing.

Same as in the preceding commit, this seems to be a bug in Dash itself
that bisects to c5bf970 (expand: Add multi-byte support to pmatch,
2024-06-02). But other than in the preceding commit, this bug has
already been fixed upstream in 079059a (input: Fix heap-buffer-overflow
in preadbuffer on long lines, 2026-02-11), which is part of v0.5.13.2.

For now though, work around the bug by waiting for the expected output
in a different way. There is no good reason why one version should work
better than the other, but at least the new version doesn't exhibit the
bug. And, if you ask me, it's also slightly easier to read.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t9300-fast-import.sh | 32 ++++++++++++++------------------
 1 file changed, 14 insertions(+), 18 deletions(-)

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index 5685cce6fe..479437760b 100755
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -3635,25 +3635,21 @@ background_import_then_checkpoint () {
 		echo "progress checkpoint"
 	) >&8 &
 
-	error=1 ;# assume the worst
-	while read output <&9
-	do
-		if test "$output" = "progress checkpoint"
-		then
-			error=0
-			break
-		elif test "$output" = "UNEXPECTED"
-		then
-			break
-		fi
-		# otherwise ignore cruft
-		echo >&2 "cruft: $output"
-	done
+	last=$(
+		while read output <&9
+		do
+			if test "$output" = "progress checkpoint" || test "$output" = "UNEXPECTED"
+			then
+				echo "$output"
+				break
+			else
+				# otherwise ignore cruft
+				echo >&2 "cruft: $output"
+			fi
+		done
+	)
 
-	if test $error -eq 1
-	then
-		false
-	fi
+	test "$last" = "progress checkpoint"
 }
 
 background_import_still_running () {

-- 
2.53.0.1323.g189a785ab5.dirty