All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/2] utf8: two minor fixes
@ 2026-11-16 18:56 Dirk Gouders
  2026-11-16 18:57 ` [PATCH 1/2] utf8: respect bounds in utf8_strnwidth() Dirk Gouders
  2026-11-16 18:57 ` [PATCH 2/2] utf8: fix comment for utf8_strnwidth() Dirk Gouders
  0 siblings, 2 replies; 8+ messages in thread
From: Dirk Gouders @ 2026-11-16 18:56 UTC (permalink / raw)
  To: git; +Cc: Dirk Gouders

A patch here on the list made me look at
display_mode_esc_sequence_len() in utf8.c and I found two details in
the file that are perhaps worth a fix:

1) utf8_strnwidth() disregards the given len argument when fed
   with proper string and len.

2) The comment for utf8_strnwidth() is incorrect: it states the
   function returns strlen() for cases when it actually returns its
   len argument.

Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
Dirk Gouders (2):
      utf8: respect bounds in utf8_strnwidth()
      utf8: fix comment for utf8_strnwidth()

 t/unit-tests/u-utf8-width.c | 11 +++++++++++
 utf8.c                      | 12 ++++++++----
 2 files changed, 19 insertions(+), 4 deletions(-)


---
base-commit: 7fb6055e678f800320d8ab8cd8f201d9493d1ba0
change-id: 20261116-master-69b171612823


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH 1/2] utf8: respect bounds in utf8_strnwidth()
  2026-11-16 18:56 [PATCH 0/2] utf8: two minor fixes Dirk Gouders
@ 2026-11-16 18:57 ` Dirk Gouders
  2026-11-17  0:02   ` Junio C Hamano
  2026-11-16 18:57 ` [PATCH 2/2] utf8: fix comment for utf8_strnwidth() Dirk Gouders
  1 sibling, 1 reply; 8+ messages in thread
From: Dirk Gouders @ 2026-11-16 18:57 UTC (permalink / raw)
  To: git; +Cc: Dirk Gouders

utf8_strnwidth() takes a len argument, but because
display_mode_esc_sequence_len() works without an explicit boundary,
there are cases when utf8_strnwidth() operates beyond the given len
boundary.

Fix this by adding a boundary-check after looping with
display_mode_esc_sequence_len() to ensure we operate strictly within
the given len.

Further, add three corner case tests of which two will fail without
the fix:
------------------------------------------------------------------------
not ok 257 - utf8_width::strnwidth_ansi
    ---
    reason: |
      Expected comparison to hold: 0 == utf8_strnwidth("\033[31mhello", 4, 1)
      0 != 1
    at:
      file: 't/unit-tests/u-utf8-width.c'
      line: 11
      function: 'test_utf8_width__strnwidth_ansi'
    ...
------------------------------------------------------------------------

With the fix:
------------------------------------------------------------------------
ok 257 - utf8_width::strnwidth_ansi
------------------------------------------------------------------------

Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
 t/unit-tests/u-utf8-width.c | 11 +++++++++++
 utf8.c                      | 10 +++++++---
 2 files changed, 18 insertions(+), 3 deletions(-)

diff --git a/t/unit-tests/u-utf8-width.c b/t/unit-tests/u-utf8-width.c
index 86e09c3574..60538fc26f 100644
--- a/t/unit-tests/u-utf8-width.c
+++ b/t/unit-tests/u-utf8-width.c
@@ -2,6 +2,17 @@
 #include "utf8.h"
 #include "strbuf.h"
 
+/*
+ * Test utf8_strnwidth for strings with SGR sequences and corner case length
+ * arguments.
+ */
+void test_utf8_width__strnwidth_ansi(void)
+{
+	cl_assert_equal_i(0, utf8_strnwidth("\033[31mhello", 4, 1));
+	cl_assert_equal_i(0, utf8_strnwidth("\033[31mhello", 5, 1));
+	cl_assert_equal_i(1, utf8_strnwidth("\033[31mhello", 6, 1));
+}
+
 /*
  * Test utf8_strnwidth with various Chinese strings
  * Chinese characters typically have a width of 2 columns when displayed
diff --git a/utf8.c b/utf8.c
index 96460cc414..7e050bb2c4 100644
--- a/utf8.c
+++ b/utf8.c
@@ -217,9 +217,13 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
 		int glyph_width;
 		size_t skip;
 
-		while (skip_ansi &&
-		       (skip = display_mode_esc_sequence_len(string)) != 0)
-			string += skip;
+		if (skip_ansi) {
+			while ((skip = display_mode_esc_sequence_len(string)) != 0)
+				string += skip;
+
+			if (string >= orig + len)
+				break;
+		}
 
 		glyph_width = utf8_width(&string, NULL);
 		if (glyph_width > 0)

-- 
2.55.0


^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH 2/2] utf8: fix comment for utf8_strnwidth()
  2026-11-16 18:56 [PATCH 0/2] utf8: two minor fixes Dirk Gouders
  2026-11-16 18:57 ` [PATCH 1/2] utf8: respect bounds in utf8_strnwidth() Dirk Gouders
@ 2026-11-16 18:57 ` Dirk Gouders
  1 sibling, 0 replies; 8+ messages in thread
From: Dirk Gouders @ 2026-11-16 18:57 UTC (permalink / raw)
  To: git; +Cc: Dirk Gouders

The comment for utf8_strnwidth() states it returns strlen() for cases
when the given string doesn't look like valid utf8, but it returns its
argument len in such cases.

Fix the comment accordingly.

Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
 utf8.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/utf8.c b/utf8.c
index 7e050bb2c4..e6d1510b60 100644
--- a/utf8.c
+++ b/utf8.c
@@ -205,7 +205,7 @@ int utf8_width(const char **start, size_t *remainder_p)
 
 /*
  * Returns the total number of columns required by a null-terminated
- * string, assuming that the string is utf8.  Returns strlen() instead
+ * string, assuming that the string is utf8.  Returns len instead
  * if the string does not look like a valid utf8 string.
  */
 int utf8_strnwidth(const char *string, size_t len, int skip_ansi)

-- 
2.55.0


^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: [PATCH 1/2] utf8: respect bounds in utf8_strnwidth()
  2026-11-16 18:57 ` [PATCH 1/2] utf8: respect bounds in utf8_strnwidth() Dirk Gouders
@ 2026-11-17  0:02   ` Junio C Hamano
  2026-11-17 15:15     ` Dirk Gouders
  2026-11-17 16:18     ` Junio C Hamano
  0 siblings, 2 replies; 8+ messages in thread
From: Junio C Hamano @ 2026-11-17  0:02 UTC (permalink / raw)
  To: Dirk Gouders; +Cc: git

Dirk Gouders <dirk@gouders.net> writes:

> utf8_strnwidth() takes a len argument, but because
> display_mode_esc_sequence_len() works without an explicit boundary,
> there are cases when utf8_strnwidth() operates beyond the given len
> boundary.

The problem is well spotted, but isn't the blame slightly misplaced?
The preimage with missing context looks like this:

    int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
    {
            const char *orig = string;
            size_t width = 0;

            while (string && string < orig + len) {
                    int glyph_width;
                    size_t skip;

                    while (skip_ansi &&
                           (skip = display_mode_esc_sequence_len(string)) != 0)
                            string += skip;

                    glyph_width = utf8_width(&string, NULL);
                    if (glyph_width > 0)
                            width += glyph_width;
            }

Even if display_mode_esc_sequence_len() were updated to take the
remaining number of bytes in string (which I think *is* a good
change, by the way), the loop it is in may cause the entirety of the
string up to the "len" bytes to be skipped.  Because the original
lacks a guard before the call to utf8_width(), however, we end up
calling it even if string points beyond the given region of bytes
orig[0,len).

If you blame the lack of such a guard before utf8_width() (or after
the loop that has display_mode_esc_sequence_len()), it would match
the proposed fix.  But I doubt it is entirely fair to blame the lack
of an explicit boundary in display_mode_esc_sequence_len() for the
breakage you are addressing.

> Fix this by adding a boundary-check after looping with
> display_mode_esc_sequence_len() to ensure we operate strictly within
> the given len.

If the incoming "string" is at the end of an mmapped region (beyond
the end of which nothing is mapped), reading string[len] would give
you SIGBUS, so display_mode_esc_sequence_len() would already have
taken you into trouble.  In other words, even though such a check
before utf8_width() is better than nothing, in a sense it is too
late, no?


> diff --git a/utf8.c b/utf8.c
> index 96460cc414..7e050bb2c4 100644
> --- a/utf8.c
> +++ b/utf8.c
> @@ -217,9 +217,13 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
>  		int glyph_width;
>  		size_t skip;
>  
> -		while (skip_ansi &&
> -		       (skip = display_mode_esc_sequence_len(string)) != 0)
> -			string += skip;
> +		if (skip_ansi) {
> +			while ((skip = display_mode_esc_sequence_len(string)) != 0)
> +				string += skip;
> +
> +			if (string >= orig + len)
> +				break;
> +		}
>  
>  		glyph_width = utf8_width(&string, NULL);
>  		if (glyph_width > 0)

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 1/2] utf8: respect bounds in utf8_strnwidth()
  2026-11-17  0:02   ` Junio C Hamano
@ 2026-11-17 15:15     ` Dirk Gouders
  2026-11-17 16:18     ` Junio C Hamano
  1 sibling, 0 replies; 8+ messages in thread
From: Dirk Gouders @ 2026-11-17 15:15 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: git

On Mon, Nov 16 2026, Junio C Hamano wrote:

> Dirk Gouders <dirk@gouders.net> writes:
>
>> utf8_strnwidth() takes a len argument, but because
>> display_mode_esc_sequence_len() works without an explicit boundary,
>> there are cases when utf8_strnwidth() operates beyond the given len
>> boundary.
>
> The problem is well spotted, but isn't the blame slightly misplaced?
> The preimage with missing context looks like this:
>
>     int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
>     {
>             const char *orig = string;
>             size_t width = 0;
>
>             while (string && string < orig + len) {
>                     int glyph_width;
>                     size_t skip;
>
>                     while (skip_ansi &&
>                            (skip = display_mode_esc_sequence_len(string)) != 0)
>                             string += skip;
>
>                     glyph_width = utf8_width(&string, NULL);
>                     if (glyph_width > 0)
>                             width += glyph_width;
>             }
>
> Even if display_mode_esc_sequence_len() were updated to take the
> remaining number of bytes in string (which I think *is* a good
> change, by the way), the loop it is in may cause the entirety of the
> string up to the "len" bytes to be skipped.  Because the original
> lacks a guard before the call to utf8_width(), however, we end up
> calling it even if string points beyond the given region of bytes
> orig[0,len).
>
> If you blame the lack of such a guard before utf8_width() (or after
> the loop that has display_mode_esc_sequence_len()), it would match
> the proposed fix.  But I doubt it is entirely fair to blame the lack
> of an explicit boundary in display_mode_esc_sequence_len() for the
> breakage you are addressing.

Yes, I'm not sure if that term is correct but the main problem seems to
be a "contract mismatch": a function that accepts a boundary argument
calls one function display_mode_esc_sequence_len() that doesn't have an
explicit boundary and for a second one utf8_width() it doesn't use the
optional boundary argument.

>> Fix this by adding a boundary-check after looping with
>> display_mode_esc_sequence_len() to ensure we operate strictly within
>> the given len.
>
> If the incoming "string" is at the end of an mmapped region (beyond
> the end of which nothing is mapped), reading string[len] would give
> you SIGBUS, so display_mode_esc_sequence_len() would already have
> taken you into trouble.  In other words, even though such a check
> before utf8_width() is better than nothing, in a sense it is too
> late, no?

Correct.

I am currently preparing a v2 that adresses all the above.
I will try to give that v2 sufficient time to hopefully reduce errors in
my workflow and not steal other people's time.

Regards,

Dirk

>> diff --git a/utf8.c b/utf8.c
>> index 96460cc414..7e050bb2c4 100644
>> --- a/utf8.c
>> +++ b/utf8.c
>> @@ -217,9 +217,13 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
>>  		int glyph_width;
>>  		size_t skip;
>>  
>> -		while (skip_ansi &&
>> -		       (skip = display_mode_esc_sequence_len(string)) != 0)
>> -			string += skip;
>> +		if (skip_ansi) {
>> +			while ((skip = display_mode_esc_sequence_len(string)) != 0)
>> +				string += skip;
>> +
>> +			if (string >= orig + len)
>> +				break;
>> +		}
>>  
>>  		glyph_width = utf8_width(&string, NULL);
>>  		if (glyph_width > 0)

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 1/2] utf8: respect bounds in utf8_strnwidth()
  2026-11-17  0:02   ` Junio C Hamano
  2026-11-17 15:15     ` Dirk Gouders
@ 2026-11-17 16:18     ` Junio C Hamano
  2026-11-17 17:57       ` Dirk Gouders
  1 sibling, 1 reply; 8+ messages in thread
From: Junio C Hamano @ 2026-11-17 16:18 UTC (permalink / raw)
  To: Dirk Gouders; +Cc: git

Junio C Hamano <gitster@pobox.com> writes:

> The problem is well spotted, but isn't the blame slightly misplaced?
> ...
> Even if display_mode_esc_sequence_len() were updated to take the
> remaining number of bytes in string (which I think *is* a good
> change, by the way), ...

An update to display_mode_esc_sequence_len() to take the length
limit, including its fallout, is shown below, and it does not look
too bad.  Not even compile-tested, but it should be sufficient as an
illustration of the idea.

This does not make your "after display_mode_esc_sequence_len()
skips, make sure if we still have remaining string to grab one UTF-8
character" change unnecessary, as I said.


 pretty.c |  3 ++-
 utf8.c   | 27 ++++++++++++++++++---------
 utf8.h   |  2 +-
 3 files changed, 21 insertions(+), 11 deletions(-)

diff --git c/pretty.c w/pretty.c
index 0bc8ad8a9a..a20fd61909 100644
--- c/pretty.c
+++ w/pretty.c
@@ -1834,7 +1834,8 @@ static size_t format_and_pad_commit(struct strbuf *sb, /* in UTF-8 */
 			while (p > sb->buf && ch - p < 10 && *p != '\033')
 				p--;
 			if (*p != '\033' ||
-			    ch + 1 - p != display_mode_esc_sequence_len(p))
+			    ch + 1 - p != display_mode_esc_sequence_len(p,
+								ch + 1 - p))
 				break;
 			/*
 			 * got a good ansi sequence, put it back to
diff --git c/utf8.c w/utf8.c
index 96460cc414..544c5329f3 100644
--- c/utf8.c
+++ w/utf8.c
@@ -16,16 +16,23 @@ struct interval {
 	ucs_char_t last;
 };
 
-size_t display_mode_esc_sequence_len(const char *s)
+/*
+ * Return the length of the ANSI display mode escape sequence (e.g.,
+ * "\033[1;31m") that appears at the beginning of the "len" bytes
+ * of string "s", or 0 if the bytes do not begin with a complete
+ * such sequence.  Never looks at s[len] or beyond.
+ */
+size_t display_mode_esc_sequence_len(const char *s, size_t len)
 {
-	const char *p = s;
-	if (*p++ != '\033')
+	const char *p = s, *end = s + len;
+
+	if (end <= p || *p++ != '\033')
 		return 0;
-	if (*p++ != '[')
+	if (end <= p || *p++ != '[')
 		return 0;
-	while (isdigit(*p) || *p == ';')
+	while (p < end && (isdigit(*p) || *p == ';'))
 		p++;
-	if (*p++ != 'm')
+	if (end <= p || *p++ != 'm')
 		return 0;
 	return p - s;
 }
@@ -218,7 +225,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
 		size_t skip;
 
 		while (skip_ansi &&
-		       (skip = display_mode_esc_sequence_len(string)) != 0)
+		       (skip = display_mode_esc_sequence_len(string,
+							     orig + len - string)) != 0)
 			string += skip;
 
 		glyph_width = utf8_width(&string, NULL);
@@ -279,6 +287,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
 {
 	int indent, w, assume_utf8 = 1;
 	const char *bol, *space, *start = text;
+	const char *end = text + strlen(text);
 	size_t orig_len = buf->len;
 
 	if (width <= 0) {
@@ -299,7 +308,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
 		char c;
 		size_t skip;
 
-		while ((skip = display_mode_esc_sequence_len(text)))
+		while ((skip = display_mode_esc_sequence_len(text, end - text)))
 			text += skip;
 
 		c = *text;
@@ -378,7 +387,7 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
 		int glyph_width;
 		size_t n;
 
-		while ((n = display_mode_esc_sequence_len(src))) {
+		while ((n = display_mode_esc_sequence_len(src, end - src))) {
 			strbuf_add(&dst, src, n);
 			src += n;
 		}
diff --git c/utf8.h w/utf8.h
index cf8ecb0f21..fb2227a9e2 100644
--- c/utf8.h
+++ w/utf8.h
@@ -5,7 +5,7 @@ struct strbuf;
 
 typedef unsigned int ucs_char_t;  /* assuming 32bit int */
 
-size_t display_mode_esc_sequence_len(const char *s);
+size_t display_mode_esc_sequence_len(const char *s, size_t len);
 int utf8_width(const char **start, size_t *remainder_p);
 int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
 int utf8_strwidth(const char *string);

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: [PATCH 1/2] utf8: respect bounds in utf8_strnwidth()
  2026-11-17 16:18     ` Junio C Hamano
@ 2026-11-17 17:57       ` Dirk Gouders
  2026-11-17 18:11         ` Junio C Hamano
  0 siblings, 1 reply; 8+ messages in thread
From: Dirk Gouders @ 2026-11-17 17:57 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: git

On Tue, Nov 17 2026, Junio C Hamano wrote:

> Junio C Hamano <gitster@pobox.com> writes:
>
>> The problem is well spotted, but isn't the blame slightly misplaced?
>> ...
>> Even if display_mode_esc_sequence_len() were updated to take the
>> remaining number of bytes in string (which I think *is* a good
>> change, by the way), ...
>
> An update to display_mode_esc_sequence_len() to take the length
> limit, including its fallout, is shown below, and it does not look
> too bad.  Not even compile-tested, but it should be sufficient as an
> illustration of the idea.
>
> This does not make your "after display_mode_esc_sequence_len()
> skips, make sure if we still have remaining string to grab one UTF-8
> character" change unnecessary, as I said.

My v2 would have introduced two new functions (see attached) to replace
display_mode_esc_sequence_len():

• size_t ansi_sgr_nlen(const char *str, size_t len);
• size_t ansi_sgr_nlen_multi(const char *s, size_t len);

The latter one to help (most of all) users who want to eat *all* leading
SGR sequences so that they don't need loops.

But I saw you already posted a patch and that is also fine, of course.

Dirk

diff --git a/utf8.c b/utf8.c
index 96460cc414..1e3887480f 100644
--- a/utf8.c
+++ b/utf8.c
@@ -30,6 +30,44 @@ size_t display_mode_esc_sequence_len(const char *s)
 	return p - s;
 }
 
+size_t ansi_sgr_nlen(const char *str, size_t len)
+{
+	const char *p = str;
+	const char *p_max = str + len;
+
+	if (len < 3)
+		return 0;
+
+	if (*p++ != '\033')
+		return 0;
+	if (*p++ != '[')
+		return 0;
+
+	while (p < p_max && (isdigit(*p) || *p == ';'))
+		p++;
+
+	if (p == p_max || *p != 'm')
+		return 0;
+
+	return p - str;
+}
+
+size_t ansi_sgr_nlen_multi(const char *s, size_t len)
+{
+	size_t sgr_len = 0;
+
+	while (sgr_len < len) {
+		size_t count = ansi_sgr_nlen(s + sgr_len, len - sgr_len);
+
+		if (!count)
+			break;
+
+		sgr_len += count;
+	}
+
+	return sgr_len;
+}
+
 /* auxiliary function for binary search in interval table */
 static int bisearch(ucs_char_t ucs, const struct interval *table, int max)
 {

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: [PATCH 1/2] utf8: respect bounds in utf8_strnwidth()
  2026-11-17 17:57       ` Dirk Gouders
@ 2026-11-17 18:11         ` Junio C Hamano
  0 siblings, 0 replies; 8+ messages in thread
From: Junio C Hamano @ 2026-11-17 18:11 UTC (permalink / raw)
  To: Dirk Gouders; +Cc: git

Dirk Gouders <dirk@gouders.net> writes:

>> An update to display_mode_esc_sequence_len() to take the length
>> limit, including its fallout, is shown below, and it does not look
>> too bad.  Not even compile-tested, but it should be sufficient as an
>> illustration of the idea.
>>
>> This does not make your "after display_mode_esc_sequence_len()
>> skips, make sure if we still have remaining string to grab one UTF-8
>> character" change unnecessary, as I said.
>
> My v2 would have introduced two new functions (see attached) to replace
> display_mode_esc_sequence_len():

A two-function arrangement would be a good idea for maintainability.

Add display_mode_esc_sequence_len_counted() to use the counted
string API, with display_mode_esc_sequence_len() keeping the
original calling convention (i.e., we call strlen(s) then call the
new helper).  I was too lazy to do the function pair in the
illustration patch, which caused all existing callers to be
affected, but given that there is another topic or two that are just
being proposed that would add more callers to this function, it
probably makes sense to keep a function with the NUL-terminated
string API as before, and add the counted interface for callers that
care, like the one you are fixing.

Thanks.

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2026-11-17 18:11 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 18:56 [PATCH 0/2] utf8: two minor fixes Dirk Gouders
2026-11-16 18:57 ` [PATCH 1/2] utf8: respect bounds in utf8_strnwidth() Dirk Gouders
2026-11-17  0:02   ` Junio C Hamano
2026-11-17 15:15     ` Dirk Gouders
2026-11-17 16:18     ` Junio C Hamano
2026-11-17 17:57       ` Dirk Gouders
2026-11-17 18:11         ` Junio C Hamano
2026-11-16 18:57 ` [PATCH 2/2] utf8: fix comment for utf8_strnwidth() Dirk Gouders

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.