* [PATCH 0/2] utf8: two minor fixes
@ 2026-11-16 18:56 Dirk Gouders
2026-11-16 18:57 ` [PATCH 1/2] utf8: respect bounds in utf8_strnwidth() Dirk Gouders
` (4 more replies)
0 siblings, 5 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-16 18:56 UTC (permalink / raw)
To: git; +Cc: Dirk Gouders
A patch here on the list made me look at
display_mode_esc_sequence_len() in utf8.c and I found two details in
the file that are perhaps worth a fix:
1) utf8_strnwidth() disregards the given len argument when fed
with proper string and len.
2) The comment for utf8_strnwidth() is incorrect: it states the
function returns strlen() for cases when it actually returns its
len argument.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
Dirk Gouders (2):
utf8: respect bounds in utf8_strnwidth()
utf8: fix comment for utf8_strnwidth()
t/unit-tests/u-utf8-width.c | 11 +++++++++++
utf8.c | 12 ++++++++----
2 files changed, 19 insertions(+), 4 deletions(-)
---
base-commit: 7fb6055e678f800320d8ab8cd8f201d9493d1ba0
change-id: 20261116-master-69b171612823
^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH 1/2] utf8: respect bounds in utf8_strnwidth()
2026-11-16 18:56 [PATCH 0/2] utf8: two minor fixes Dirk Gouders
@ 2026-11-16 18:57 ` Dirk Gouders
2026-11-17 0:02 ` Junio C Hamano
2026-11-16 18:57 ` [PATCH 2/2] utf8: fix comment for utf8_strnwidth() Dirk Gouders
` (3 subsequent siblings)
4 siblings, 1 reply; 41+ messages in thread
From: Dirk Gouders @ 2026-11-16 18:57 UTC (permalink / raw)
To: git; +Cc: Dirk Gouders
utf8_strnwidth() takes a len argument, but because
display_mode_esc_sequence_len() works without an explicit boundary,
there are cases when utf8_strnwidth() operates beyond the given len
boundary.
Fix this by adding a boundary-check after looping with
display_mode_esc_sequence_len() to ensure we operate strictly within
the given len.
Further, add three corner case tests of which two will fail without
the fix:
------------------------------------------------------------------------
not ok 257 - utf8_width::strnwidth_ansi
---
reason: |
Expected comparison to hold: 0 == utf8_strnwidth("\033[31mhello", 4, 1)
0 != 1
at:
file: 't/unit-tests/u-utf8-width.c'
line: 11
function: 'test_utf8_width__strnwidth_ansi'
...
------------------------------------------------------------------------
With the fix:
------------------------------------------------------------------------
ok 257 - utf8_width::strnwidth_ansi
------------------------------------------------------------------------
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
t/unit-tests/u-utf8-width.c | 11 +++++++++++
utf8.c | 10 +++++++---
2 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/t/unit-tests/u-utf8-width.c b/t/unit-tests/u-utf8-width.c
index 86e09c3574..60538fc26f 100644
--- a/t/unit-tests/u-utf8-width.c
+++ b/t/unit-tests/u-utf8-width.c
@@ -2,6 +2,17 @@
#include "utf8.h"
#include "strbuf.h"
+/*
+ * Test utf8_strnwidth for strings with SGR sequences and corner case length
+ * arguments.
+ */
+void test_utf8_width__strnwidth_ansi(void)
+{
+ cl_assert_equal_i(0, utf8_strnwidth("\033[31mhello", 4, 1));
+ cl_assert_equal_i(0, utf8_strnwidth("\033[31mhello", 5, 1));
+ cl_assert_equal_i(1, utf8_strnwidth("\033[31mhello", 6, 1));
+}
+
/*
* Test utf8_strnwidth with various Chinese strings
* Chinese characters typically have a width of 2 columns when displayed
diff --git a/utf8.c b/utf8.c
index 96460cc414..7e050bb2c4 100644
--- a/utf8.c
+++ b/utf8.c
@@ -217,9 +217,13 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
int glyph_width;
size_t skip;
- while (skip_ansi &&
- (skip = display_mode_esc_sequence_len(string)) != 0)
- string += skip;
+ if (skip_ansi) {
+ while ((skip = display_mode_esc_sequence_len(string)) != 0)
+ string += skip;
+
+ if (string >= orig + len)
+ break;
+ }
glyph_width = utf8_width(&string, NULL);
if (glyph_width > 0)
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH 2/2] utf8: fix comment for utf8_strnwidth()
2026-11-16 18:56 [PATCH 0/2] utf8: two minor fixes Dirk Gouders
2026-11-16 18:57 ` [PATCH 1/2] utf8: respect bounds in utf8_strnwidth() Dirk Gouders
@ 2026-11-16 18:57 ` Dirk Gouders
2026-11-18 21:08 ` [PATCH v2 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
` (2 subsequent siblings)
4 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-16 18:57 UTC (permalink / raw)
To: git; +Cc: Dirk Gouders
The comment for utf8_strnwidth() states it returns strlen() for cases
when the given string doesn't look like valid utf8, but it returns its
argument len in such cases.
Fix the comment accordingly.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
utf8.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/utf8.c b/utf8.c
index 7e050bb2c4..e6d1510b60 100644
--- a/utf8.c
+++ b/utf8.c
@@ -205,7 +205,7 @@ int utf8_width(const char **start, size_t *remainder_p)
/*
* Returns the total number of columns required by a null-terminated
- * string, assuming that the string is utf8. Returns strlen() instead
+ * string, assuming that the string is utf8. Returns len instead
* if the string does not look like a valid utf8 string.
*/
int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH 1/2] utf8: respect bounds in utf8_strnwidth()
2026-11-16 18:57 ` [PATCH 1/2] utf8: respect bounds in utf8_strnwidth() Dirk Gouders
@ 2026-11-17 0:02 ` Junio C Hamano
2026-11-17 15:15 ` Dirk Gouders
2026-11-17 16:18 ` Junio C Hamano
0 siblings, 2 replies; 41+ messages in thread
From: Junio C Hamano @ 2026-11-17 0:02 UTC (permalink / raw)
To: Dirk Gouders; +Cc: git
Dirk Gouders <dirk@gouders.net> writes:
> utf8_strnwidth() takes a len argument, but because
> display_mode_esc_sequence_len() works without an explicit boundary,
> there are cases when utf8_strnwidth() operates beyond the given len
> boundary.
The problem is well spotted, but isn't the blame slightly misplaced?
The preimage with missing context looks like this:
int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
{
const char *orig = string;
size_t width = 0;
while (string && string < orig + len) {
int glyph_width;
size_t skip;
while (skip_ansi &&
(skip = display_mode_esc_sequence_len(string)) != 0)
string += skip;
glyph_width = utf8_width(&string, NULL);
if (glyph_width > 0)
width += glyph_width;
}
Even if display_mode_esc_sequence_len() were updated to take the
remaining number of bytes in string (which I think *is* a good
change, by the way), the loop it is in may cause the entirety of the
string up to the "len" bytes to be skipped. Because the original
lacks a guard before the call to utf8_width(), however, we end up
calling it even if string points beyond the given region of bytes
orig[0,len).
If you blame the lack of such a guard before utf8_width() (or after
the loop that has display_mode_esc_sequence_len()), it would match
the proposed fix. But I doubt it is entirely fair to blame the lack
of an explicit boundary in display_mode_esc_sequence_len() for the
breakage you are addressing.
> Fix this by adding a boundary-check after looping with
> display_mode_esc_sequence_len() to ensure we operate strictly within
> the given len.
If the incoming "string" is at the end of an mmapped region (beyond
the end of which nothing is mapped), reading string[len] would give
you SIGBUS, so display_mode_esc_sequence_len() would already have
taken you into trouble. In other words, even though such a check
before utf8_width() is better than nothing, in a sense it is too
late, no?
> diff --git a/utf8.c b/utf8.c
> index 96460cc414..7e050bb2c4 100644
> --- a/utf8.c
> +++ b/utf8.c
> @@ -217,9 +217,13 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
> int glyph_width;
> size_t skip;
>
> - while (skip_ansi &&
> - (skip = display_mode_esc_sequence_len(string)) != 0)
> - string += skip;
> + if (skip_ansi) {
> + while ((skip = display_mode_esc_sequence_len(string)) != 0)
> + string += skip;
> +
> + if (string >= orig + len)
> + break;
> + }
>
> glyph_width = utf8_width(&string, NULL);
> if (glyph_width > 0)
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH 1/2] utf8: respect bounds in utf8_strnwidth()
2026-11-17 0:02 ` Junio C Hamano
@ 2026-11-17 15:15 ` Dirk Gouders
2026-11-17 16:18 ` Junio C Hamano
1 sibling, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-17 15:15 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
On Mon, Nov 16 2026, Junio C Hamano wrote:
> Dirk Gouders <dirk@gouders.net> writes:
>
>> utf8_strnwidth() takes a len argument, but because
>> display_mode_esc_sequence_len() works without an explicit boundary,
>> there are cases when utf8_strnwidth() operates beyond the given len
>> boundary.
>
> The problem is well spotted, but isn't the blame slightly misplaced?
> The preimage with missing context looks like this:
>
> int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
> {
> const char *orig = string;
> size_t width = 0;
>
> while (string && string < orig + len) {
> int glyph_width;
> size_t skip;
>
> while (skip_ansi &&
> (skip = display_mode_esc_sequence_len(string)) != 0)
> string += skip;
>
> glyph_width = utf8_width(&string, NULL);
> if (glyph_width > 0)
> width += glyph_width;
> }
>
> Even if display_mode_esc_sequence_len() were updated to take the
> remaining number of bytes in string (which I think *is* a good
> change, by the way), the loop it is in may cause the entirety of the
> string up to the "len" bytes to be skipped. Because the original
> lacks a guard before the call to utf8_width(), however, we end up
> calling it even if string points beyond the given region of bytes
> orig[0,len).
>
> If you blame the lack of such a guard before utf8_width() (or after
> the loop that has display_mode_esc_sequence_len()), it would match
> the proposed fix. But I doubt it is entirely fair to blame the lack
> of an explicit boundary in display_mode_esc_sequence_len() for the
> breakage you are addressing.
Yes, I'm not sure if that term is correct but the main problem seems to
be a "contract mismatch": a function that accepts a boundary argument
calls one function display_mode_esc_sequence_len() that doesn't have an
explicit boundary and for a second one utf8_width() it doesn't use the
optional boundary argument.
>> Fix this by adding a boundary-check after looping with
>> display_mode_esc_sequence_len() to ensure we operate strictly within
>> the given len.
>
> If the incoming "string" is at the end of an mmapped region (beyond
> the end of which nothing is mapped), reading string[len] would give
> you SIGBUS, so display_mode_esc_sequence_len() would already have
> taken you into trouble. In other words, even though such a check
> before utf8_width() is better than nothing, in a sense it is too
> late, no?
Correct.
I am currently preparing a v2 that adresses all the above.
I will try to give that v2 sufficient time to hopefully reduce errors in
my workflow and not steal other people's time.
Regards,
Dirk
>> diff --git a/utf8.c b/utf8.c
>> index 96460cc414..7e050bb2c4 100644
>> --- a/utf8.c
>> +++ b/utf8.c
>> @@ -217,9 +217,13 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
>> int glyph_width;
>> size_t skip;
>>
>> - while (skip_ansi &&
>> - (skip = display_mode_esc_sequence_len(string)) != 0)
>> - string += skip;
>> + if (skip_ansi) {
>> + while ((skip = display_mode_esc_sequence_len(string)) != 0)
>> + string += skip;
>> +
>> + if (string >= orig + len)
>> + break;
>> + }
>>
>> glyph_width = utf8_width(&string, NULL);
>> if (glyph_width > 0)
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH 1/2] utf8: respect bounds in utf8_strnwidth()
2026-11-17 0:02 ` Junio C Hamano
2026-11-17 15:15 ` Dirk Gouders
@ 2026-11-17 16:18 ` Junio C Hamano
2026-11-17 17:57 ` Dirk Gouders
1 sibling, 1 reply; 41+ messages in thread
From: Junio C Hamano @ 2026-11-17 16:18 UTC (permalink / raw)
To: Dirk Gouders; +Cc: git
Junio C Hamano <gitster@pobox.com> writes:
> The problem is well spotted, but isn't the blame slightly misplaced?
> ...
> Even if display_mode_esc_sequence_len() were updated to take the
> remaining number of bytes in string (which I think *is* a good
> change, by the way), ...
An update to display_mode_esc_sequence_len() to take the length
limit, including its fallout, is shown below, and it does not look
too bad. Not even compile-tested, but it should be sufficient as an
illustration of the idea.
This does not make your "after display_mode_esc_sequence_len()
skips, make sure if we still have remaining string to grab one UTF-8
character" change unnecessary, as I said.
pretty.c | 3 ++-
utf8.c | 27 ++++++++++++++++++---------
utf8.h | 2 +-
3 files changed, 21 insertions(+), 11 deletions(-)
diff --git c/pretty.c w/pretty.c
index 0bc8ad8a9a..a20fd61909 100644
--- c/pretty.c
+++ w/pretty.c
@@ -1834,7 +1834,8 @@ static size_t format_and_pad_commit(struct strbuf *sb, /* in UTF-8 */
while (p > sb->buf && ch - p < 10 && *p != '\033')
p--;
if (*p != '\033' ||
- ch + 1 - p != display_mode_esc_sequence_len(p))
+ ch + 1 - p != display_mode_esc_sequence_len(p,
+ ch + 1 - p))
break;
/*
* got a good ansi sequence, put it back to
diff --git c/utf8.c w/utf8.c
index 96460cc414..544c5329f3 100644
--- c/utf8.c
+++ w/utf8.c
@@ -16,16 +16,23 @@ struct interval {
ucs_char_t last;
};
-size_t display_mode_esc_sequence_len(const char *s)
+/*
+ * Return the length of the ANSI display mode escape sequence (e.g.,
+ * "\033[1;31m") that appears at the beginning of the "len" bytes
+ * of string "s", or 0 if the bytes do not begin with a complete
+ * such sequence. Never looks at s[len] or beyond.
+ */
+size_t display_mode_esc_sequence_len(const char *s, size_t len)
{
- const char *p = s;
- if (*p++ != '\033')
+ const char *p = s, *end = s + len;
+
+ if (end <= p || *p++ != '\033')
return 0;
- if (*p++ != '[')
+ if (end <= p || *p++ != '[')
return 0;
- while (isdigit(*p) || *p == ';')
+ while (p < end && (isdigit(*p) || *p == ';'))
p++;
- if (*p++ != 'm')
+ if (end <= p || *p++ != 'm')
return 0;
return p - s;
}
@@ -218,7 +225,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
size_t skip;
while (skip_ansi &&
- (skip = display_mode_esc_sequence_len(string)) != 0)
+ (skip = display_mode_esc_sequence_len(string,
+ orig + len - string)) != 0)
string += skip;
glyph_width = utf8_width(&string, NULL);
@@ -279,6 +287,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
{
int indent, w, assume_utf8 = 1;
const char *bol, *space, *start = text;
+ const char *end = text + strlen(text);
size_t orig_len = buf->len;
if (width <= 0) {
@@ -299,7 +308,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
char c;
size_t skip;
- while ((skip = display_mode_esc_sequence_len(text)))
+ while ((skip = display_mode_esc_sequence_len(text, end - text)))
text += skip;
c = *text;
@@ -378,7 +387,7 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
int glyph_width;
size_t n;
- while ((n = display_mode_esc_sequence_len(src))) {
+ while ((n = display_mode_esc_sequence_len(src, end - src))) {
strbuf_add(&dst, src, n);
src += n;
}
diff --git c/utf8.h w/utf8.h
index cf8ecb0f21..fb2227a9e2 100644
--- c/utf8.h
+++ w/utf8.h
@@ -5,7 +5,7 @@ struct strbuf;
typedef unsigned int ucs_char_t; /* assuming 32bit int */
-size_t display_mode_esc_sequence_len(const char *s);
+size_t display_mode_esc_sequence_len(const char *s, size_t len);
int utf8_width(const char **start, size_t *remainder_p);
int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
int utf8_strwidth(const char *string);
^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH 1/2] utf8: respect bounds in utf8_strnwidth()
2026-11-17 16:18 ` Junio C Hamano
@ 2026-11-17 17:57 ` Dirk Gouders
2026-11-17 18:11 ` Junio C Hamano
0 siblings, 1 reply; 41+ messages in thread
From: Dirk Gouders @ 2026-11-17 17:57 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
On Tue, Nov 17 2026, Junio C Hamano wrote:
> Junio C Hamano <gitster@pobox.com> writes:
>
>> The problem is well spotted, but isn't the blame slightly misplaced?
>> ...
>> Even if display_mode_esc_sequence_len() were updated to take the
>> remaining number of bytes in string (which I think *is* a good
>> change, by the way), ...
>
> An update to display_mode_esc_sequence_len() to take the length
> limit, including its fallout, is shown below, and it does not look
> too bad. Not even compile-tested, but it should be sufficient as an
> illustration of the idea.
>
> This does not make your "after display_mode_esc_sequence_len()
> skips, make sure if we still have remaining string to grab one UTF-8
> character" change unnecessary, as I said.
My v2 would have introduced two new functions (see attached) to replace
display_mode_esc_sequence_len():
• size_t ansi_sgr_nlen(const char *str, size_t len);
• size_t ansi_sgr_nlen_multi(const char *s, size_t len);
The latter one to help (most of all) users who want to eat *all* leading
SGR sequences so that they don't need loops.
But I saw you already posted a patch and that is also fine, of course.
Dirk
diff --git a/utf8.c b/utf8.c
index 96460cc414..1e3887480f 100644
--- a/utf8.c
+++ b/utf8.c
@@ -30,6 +30,44 @@ size_t display_mode_esc_sequence_len(const char *s)
return p - s;
}
+size_t ansi_sgr_nlen(const char *str, size_t len)
+{
+ const char *p = str;
+ const char *p_max = str + len;
+
+ if (len < 3)
+ return 0;
+
+ if (*p++ != '\033')
+ return 0;
+ if (*p++ != '[')
+ return 0;
+
+ while (p < p_max && (isdigit(*p) || *p == ';'))
+ p++;
+
+ if (p == p_max || *p != 'm')
+ return 0;
+
+ return p - str;
+}
+
+size_t ansi_sgr_nlen_multi(const char *s, size_t len)
+{
+ size_t sgr_len = 0;
+
+ while (sgr_len < len) {
+ size_t count = ansi_sgr_nlen(s + sgr_len, len - sgr_len);
+
+ if (!count)
+ break;
+
+ sgr_len += count;
+ }
+
+ return sgr_len;
+}
+
/* auxiliary function for binary search in interval table */
static int bisearch(ucs_char_t ucs, const struct interval *table, int max)
{
^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH 1/2] utf8: respect bounds in utf8_strnwidth()
2026-11-17 17:57 ` Dirk Gouders
@ 2026-11-17 18:11 ` Junio C Hamano
0 siblings, 0 replies; 41+ messages in thread
From: Junio C Hamano @ 2026-11-17 18:11 UTC (permalink / raw)
To: Dirk Gouders; +Cc: git
Dirk Gouders <dirk@gouders.net> writes:
>> An update to display_mode_esc_sequence_len() to take the length
>> limit, including its fallout, is shown below, and it does not look
>> too bad. Not even compile-tested, but it should be sufficient as an
>> illustration of the idea.
>>
>> This does not make your "after display_mode_esc_sequence_len()
>> skips, make sure if we still have remaining string to grab one UTF-8
>> character" change unnecessary, as I said.
>
> My v2 would have introduced two new functions (see attached) to replace
> display_mode_esc_sequence_len():
A two-function arrangement would be a good idea for maintainability.
Add display_mode_esc_sequence_len_counted() to use the counted
string API, with display_mode_esc_sequence_len() keeping the
original calling convention (i.e., we call strlen(s) then call the
new helper). I was too lazy to do the function pair in the
illustration patch, which caused all existing callers to be
affected, but given that there is another topic or two that are just
being proposed that would add more callers to this function, it
probably makes sense to keep a function with the NUL-terminated
string API as before, and add the counted interface for callers that
care, like the one you are fixing.
Thanks.
^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v2 0/4] utf8: bounds-related fixes and a doc fix
2026-11-16 18:56 [PATCH 0/2] utf8: two minor fixes Dirk Gouders
2026-11-16 18:57 ` [PATCH 1/2] utf8: respect bounds in utf8_strnwidth() Dirk Gouders
2026-11-16 18:57 ` [PATCH 2/2] utf8: fix comment for utf8_strnwidth() Dirk Gouders
@ 2026-11-18 21:08 ` Dirk Gouders
2026-11-18 21:08 ` [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted() Dirk Gouders
` (3 more replies)
2026-11-20 14:44 ` [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 0/5] " Dirk Gouders
4 siblings, 4 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-18 21:08 UTC (permalink / raw)
To: git; +Cc: Dirk Gouders, Junio C Hamano
A patch here on the list made me look at
display_mode_esc_sequence_len() in utf8.c and I found two details in
the file that are perhaps worth a fix:
1) utf8_strnwidth() disregards the given len argument when fed
with proper string and len.
2) The comment for utf8_strnwidth() is incorrect: it states the
function returns strlen() for cases when it actually returns its
len argument.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
Changes in v2:
- Add a patch by Junio that introduces a bounded function display_mode_esc_sequence_len_counted()
I added some tests for this new function.
- 2/2 became 2/4
- 3/4 is the modified 1/2 and uses the new
display_mode_esc_sequence_len_counted() so that the while loop stays
within the given bounds.
The commit message was adjusted.
One of the tests was wrong: "\033" has width 0, so the sum must be 3 instead of 4.
- 4/4 is an addition to fix a similar problem with utf8_width().
That function is used without its optional bounds argument and
can violate boundaries as well. Test for such cases are added.
- Link to v1: https://patch.msgid.link/20261116-master-v1-0-dc2a50ce0aa0@gouders.net
---
Dirk Gouders (3):
utf8: fix comment for utf8_strnwidth()
utf8: respect string boundaries after skipping ANSI escape sequences
utf8: respect boundaries in utf8_strnwidth()
Junio C Hamano (1):
utf8: add display_mode_esc_sequence_len_counted()
Makefile | 1 +
t/meson.build | 1 +
t/unit-tests/u-utf8-ansi.c | 14 +++++++++++++
t/unit-tests/u-utf8-width.c | 20 ++++++++++++++++++
utf8.c | 50 +++++++++++++++++++++++++++++++++------------
utf8.h | 1 +
6 files changed, 74 insertions(+), 13 deletions(-)
Range-diff versus v1:
-: ---------- > 1: 3b2ab53858 utf8: add display_mode_esc_sequence_len_counted()
2: aab04775f6 = 2: efeff5cfd7 utf8: fix comment for utf8_strnwidth()
1: 21baf82731 ! 3: 152210dd15 utf8: respect bounds in utf8_strnwidth()
@@ Metadata
Author: Dirk Gouders <dirk@gouders.net>
## Commit message ##
- utf8: respect bounds in utf8_strnwidth()
+ utf8: respect string boundaries after skipping ANSI escape sequences
- utf8_strnwidth() takes a len argument, but because
- display_mode_esc_sequence_len() works without an explicit boundary,
- there are cases when utf8_strnwidth() operates beyond the given len
- boundary.
+ utf8_strnwidth() can skip ANSI escape sequences and it is possible
+ that such sequences end the string within the given boundary.
- Fix this by adding a boundary-check after looping with
- display_mode_esc_sequence_len() to ensure we operate strictly within
- the given len.
+ Check for such situations before proceeding and calling utf8_width().
- Further, add three corner case tests of which two will fail without
- the fix:
- ------------------------------------------------------------------------
- not ok 257 - utf8_width::strnwidth_ansi
- ---
- reason: |
- Expected comparison to hold: 0 == utf8_strnwidth("\033[31mhello", 4, 1)
- 0 != 1
- at:
- file: 't/unit-tests/u-utf8-width.c'
- line: 11
- function: 'test_utf8_width__strnwidth_ansi'
- ...
- ------------------------------------------------------------------------
-
- With the fix:
- ------------------------------------------------------------------------
- ok 257 - utf8_width::strnwidth_ansi
- ------------------------------------------------------------------------
+ Add tests that test such corner cases.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
@@ t/unit-tests/u-utf8-width.c
+ */
+void test_utf8_width__strnwidth_ansi(void)
+{
-+ cl_assert_equal_i(0, utf8_strnwidth("\033[31mhello", 4, 1));
++ cl_assert_equal_i(3, utf8_strnwidth("\033[31mhello", 4, 1));
+ cl_assert_equal_i(0, utf8_strnwidth("\033[31mhello", 5, 1));
+ cl_assert_equal_i(1, utf8_strnwidth("\033[31mhello", 6, 1));
+}
@@ utf8.c: int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
size_t skip;
- while (skip_ansi &&
-- (skip = display_mode_esc_sequence_len(string)) != 0)
+- (skip = display_mode_esc_sequence_len_counted(string,
+- orig + len - string)) != 0)
- string += skip;
-+ if (skip_ansi) {
-+ while ((skip = display_mode_esc_sequence_len(string)) != 0)
++ if (skip_ansi ) {
++ while ((skip = display_mode_esc_sequence_len_counted(string,
++ orig + len - string)) != 0)
+ string += skip;
+
+ if (string >= orig + len)
-: ---------- > 4: 044d188fa8 utf8: respect boundaries in utf8_strnwidth()
---
base-commit: 7fb6055e678f800320d8ab8cd8f201d9493d1ba0
change-id: 20261116-master-69b171612823
^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-18 21:08 ` [PATCH v2 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
@ 2026-11-18 21:08 ` Dirk Gouders
2026-11-19 5:54 ` Andy Koppe
2026-11-19 19:59 ` Junio C Hamano
2026-11-18 21:08 ` [PATCH v2 2/4] utf8: fix comment for utf8_strnwidth() Dirk Gouders
` (2 subsequent siblings)
3 siblings, 2 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-18 21:08 UTC (permalink / raw)
To: git; +Cc: Dirk Gouders, Junio C Hamano
From: Junio C Hamano <gitster@pobox.com>
display_mode_esc_sequence_len() scans for an ANSI display mode
escape sequence (e.g. "ESC [ 1 ; 31 m") at the start of a string,
relying on NUL termination to stop. Some callers, however, work on
a counted buffer: utf8_strnwidth() takes <string, len> and must not
scan past it, while strbuf_utf8_replace() and
strbuf_add_wrapped_text() know (or can cheaply learn) the end.
Add display_mode_esc_sequence_len_counted() taking <ptr, len>,
which never reads at or beyond ptr[len], and make
display_mode_esc_sequence_len() a thin wrapper for NUL-terminated
strings.
Use the counted variant in utf8_strnwidth() for correctness, and in
strbuf_utf8_replace() and strbuf_add_wrapped_text() (computing the
end once upfront in the latter) to avoid measuring the remaining
text in each iteration.
Also, add some tests for the new function.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
Makefile | 1 +
t/meson.build | 1 +
t/unit-tests/u-utf8-ansi.c | 14 ++++++++++++++
utf8.c | 35 ++++++++++++++++++++++++++---------
utf8.h | 1 +
5 files changed, 43 insertions(+), 9 deletions(-)
diff --git a/Makefile b/Makefile
index 4c84eea6e2..3c8dc9ca40 100644
--- a/Makefile
+++ b/Makefile
@@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec
CLAR_TEST_SUITES += u-trailer
CLAR_TEST_SUITES += u-urlmatch-normalization
CLAR_TEST_SUITES += u-utf8-width
+CLAR_TEST_SUITES += u-utf8-ansi
CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES))
CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
diff --git a/t/meson.build b/t/meson.build
index 1cf1d95f70..e224df58c8 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -27,6 +27,7 @@ clar_test_suites = [
'unit-tests/u-trailer.c',
'unit-tests/u-urlmatch-normalization.c',
'unit-tests/u-utf8-width.c',
+ 'unit-tests/u-utf8-ansi.c',
]
clar_sources = [
diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c
new file mode 100644
index 0000000000..c19d8bce78
--- /dev/null
+++ b/t/unit-tests/u-utf8-ansi.c
@@ -0,0 +1,14 @@
+#include "unit-test.h"
+#include "utf8.h"
+#include "strbuf.h"
+
+/*
+ * Test len limited functions for ANSI SGR lengths.
+ */
+void test_utf8_ansi__sgr_len(void)
+{
+ cl_assert_equal_i(0, display_mode_esc_sequence_len_counted("\033[m", 2));
+ cl_assert_equal_i(3, display_mode_esc_sequence_len_counted("\033[m", 3));
+ cl_assert_equal_i(5, display_mode_esc_sequence_len_counted("\033[31m", 5));
+ cl_assert_equal_i(7, display_mode_esc_sequence_len_counted("\033[0;31m", 7));
+}
diff --git a/utf8.c b/utf8.c
index 96460cc414..d0546ab31f 100644
--- a/utf8.c
+++ b/utf8.c
@@ -16,20 +16,33 @@ struct interval {
ucs_char_t last;
};
-size_t display_mode_esc_sequence_len(const char *s)
+/*
+ * Return the length of the ANSI display mode escape sequence (e.g.,
+ * "\033[1;31m") that appears at the beginning of the "len" bytes
+ * of string "s", or 0 if the bytes do not begin with a complete
+ * such sequence. Never looks at s[len] or beyond.
+ */
+size_t display_mode_esc_sequence_len_counted(const char *s, size_t len)
{
- const char *p = s;
- if (*p++ != '\033')
+ const char *p = s, *end = s + len;
+
+ if (end <= p || *p++ != '\033')
return 0;
- if (*p++ != '[')
+ if (end <= p || *p++ != '[')
return 0;
- while (isdigit(*p) || *p == ';')
+ while (p < end && (isdigit(*p) || *p == ';'))
p++;
- if (*p++ != 'm')
+ if (end <= p || *p++ != 'm')
return 0;
return p - s;
}
+/* Same as above, but for a NUL-terminated string "s". */
+size_t display_mode_esc_sequence_len(const char *s)
+{
+ return display_mode_esc_sequence_len_counted(s, strlen(s));
+}
+
/* auxiliary function for binary search in interval table */
static int bisearch(ucs_char_t ucs, const struct interval *table, int max)
{
@@ -218,7 +231,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
size_t skip;
while (skip_ansi &&
- (skip = display_mode_esc_sequence_len(string)) != 0)
+ (skip = display_mode_esc_sequence_len_counted(string,
+ orig + len - string)) != 0)
string += skip;
glyph_width = utf8_width(&string, NULL);
@@ -279,6 +293,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
{
int indent, w, assume_utf8 = 1;
const char *bol, *space, *start = text;
+ const char *end = text + strlen(text);
size_t orig_len = buf->len;
if (width <= 0) {
@@ -299,7 +314,8 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
char c;
size_t skip;
- while ((skip = display_mode_esc_sequence_len(text)))
+ while ((skip = display_mode_esc_sequence_len_counted(text,
+ end - text)))
text += skip;
c = *text;
@@ -378,7 +394,8 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
int glyph_width;
size_t n;
- while ((n = display_mode_esc_sequence_len(src))) {
+ while ((n = display_mode_esc_sequence_len_counted(src,
+ end - src))) {
strbuf_add(&dst, src, n);
src += n;
}
diff --git a/utf8.h b/utf8.h
index cf8ecb0f21..57ca2ab270 100644
--- a/utf8.h
+++ b/utf8.h
@@ -6,6 +6,7 @@ struct strbuf;
typedef unsigned int ucs_char_t; /* assuming 32bit int */
size_t display_mode_esc_sequence_len(const char *s);
+size_t display_mode_esc_sequence_len_counted(const char *s, size_t len);
int utf8_width(const char **start, size_t *remainder_p);
int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
int utf8_strwidth(const char *string);
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v2 2/4] utf8: fix comment for utf8_strnwidth()
2026-11-18 21:08 ` [PATCH v2 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
2026-11-18 21:08 ` [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted() Dirk Gouders
@ 2026-11-18 21:08 ` Dirk Gouders
2026-11-18 22:40 ` Andy Koppe
2026-11-18 21:08 ` [PATCH v2 3/4] utf8: respect string boundaries after skipping ANSI escape sequences Dirk Gouders
2026-11-18 21:08 ` [PATCH v2 4/4] utf8: respect boundaries in utf8_strnwidth() Dirk Gouders
3 siblings, 1 reply; 41+ messages in thread
From: Dirk Gouders @ 2026-11-18 21:08 UTC (permalink / raw)
To: git; +Cc: Dirk Gouders
The comment for utf8_strnwidth() states it returns strlen() for cases
when the given string doesn't look like valid utf8, but it returns its
argument len in such cases.
Fix the comment accordingly.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
utf8.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/utf8.c b/utf8.c
index d0546ab31f..b43f84a03d 100644
--- a/utf8.c
+++ b/utf8.c
@@ -218,7 +218,7 @@ int utf8_width(const char **start, size_t *remainder_p)
/*
* Returns the total number of columns required by a null-terminated
- * string, assuming that the string is utf8. Returns strlen() instead
+ * string, assuming that the string is utf8. Returns len instead
* if the string does not look like a valid utf8 string.
*/
int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v2 3/4] utf8: respect string boundaries after skipping ANSI escape sequences
2026-11-18 21:08 ` [PATCH v2 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
2026-11-18 21:08 ` [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted() Dirk Gouders
2026-11-18 21:08 ` [PATCH v2 2/4] utf8: fix comment for utf8_strnwidth() Dirk Gouders
@ 2026-11-18 21:08 ` Dirk Gouders
2026-11-18 21:49 ` Junio C Hamano
2026-11-18 21:08 ` [PATCH v2 4/4] utf8: respect boundaries in utf8_strnwidth() Dirk Gouders
3 siblings, 1 reply; 41+ messages in thread
From: Dirk Gouders @ 2026-11-18 21:08 UTC (permalink / raw)
To: git; +Cc: Dirk Gouders
utf8_strnwidth() can skip ANSI escape sequences and it is possible
that such sequences end the string within the given boundary.
Check for such situations before proceeding and calling utf8_width().
Add tests that test such corner cases.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
t/unit-tests/u-utf8-width.c | 11 +++++++++++
utf8.c | 12 ++++++++----
2 files changed, 19 insertions(+), 4 deletions(-)
diff --git a/t/unit-tests/u-utf8-width.c b/t/unit-tests/u-utf8-width.c
index 86e09c3574..5e2b636db9 100644
--- a/t/unit-tests/u-utf8-width.c
+++ b/t/unit-tests/u-utf8-width.c
@@ -2,6 +2,17 @@
#include "utf8.h"
#include "strbuf.h"
+/*
+ * Test utf8_strnwidth for strings with SGR sequences and corner case length
+ * arguments.
+ */
+void test_utf8_width__strnwidth_ansi(void)
+{
+ cl_assert_equal_i(3, utf8_strnwidth("\033[31mhello", 4, 1));
+ cl_assert_equal_i(0, utf8_strnwidth("\033[31mhello", 5, 1));
+ cl_assert_equal_i(1, utf8_strnwidth("\033[31mhello", 6, 1));
+}
+
/*
* Test utf8_strnwidth with various Chinese strings
* Chinese characters typically have a width of 2 columns when displayed
diff --git a/utf8.c b/utf8.c
index b43f84a03d..1b88324ea3 100644
--- a/utf8.c
+++ b/utf8.c
@@ -230,10 +230,14 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
int glyph_width;
size_t skip;
- while (skip_ansi &&
- (skip = display_mode_esc_sequence_len_counted(string,
- orig + len - string)) != 0)
- string += skip;
+ if (skip_ansi ) {
+ while ((skip = display_mode_esc_sequence_len_counted(string,
+ orig + len - string)) != 0)
+ string += skip;
+
+ if (string >= orig + len)
+ break;
+ }
glyph_width = utf8_width(&string, NULL);
if (glyph_width > 0)
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v2 4/4] utf8: respect boundaries in utf8_strnwidth()
2026-11-18 21:08 ` [PATCH v2 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
` (2 preceding siblings ...)
2026-11-18 21:08 ` [PATCH v2 3/4] utf8: respect string boundaries after skipping ANSI escape sequences Dirk Gouders
@ 2026-11-18 21:08 ` Dirk Gouders
2026-11-18 21:50 ` Junio C Hamano
3 siblings, 1 reply; 41+ messages in thread
From: Dirk Gouders @ 2026-11-18 21:08 UTC (permalink / raw)
To: git; +Cc: Dirk Gouders
utf8_strnwidth() can be called with boundaries that effectively cause
incomplete multibyte sequences and we should return "len" in such
cases.
But, utf8_width is used without its optional boundary argument and
will thus work beyond the given boundary.
Fix this by using the boundary argument for utf8_width.
Add some tests that cover such situations.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
t/unit-tests/u-utf8-width.c | 9 +++++++++
utf8.c | 5 ++++-
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/t/unit-tests/u-utf8-width.c b/t/unit-tests/u-utf8-width.c
index 5e2b636db9..c4df4dd6a6 100644
--- a/t/unit-tests/u-utf8-width.c
+++ b/t/unit-tests/u-utf8-width.c
@@ -2,6 +2,15 @@
#include "utf8.h"
#include "strbuf.h"
+/*
+ * Test utf8_strnwidth for cjk characters invalidated by len argument.
+ */
+void test_utf8_width__strnwidth_cjk_chopped(void)
+{
+ cl_assert_equal_i(1, utf8_strnwidth("三bytes long, 二columns wide", 1, 1));
+ cl_assert_equal_i(2, utf8_strnwidth("三bytes long, 二columns wide", 3, 1));
+}
+
/*
* Test utf8_strnwidth for strings with SGR sequences and corner case length
* arguments.
diff --git a/utf8.c b/utf8.c
index 1b88324ea3..8d4b159e1d 100644
--- a/utf8.c
+++ b/utf8.c
@@ -229,6 +229,7 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
while (string && string < orig + len) {
int glyph_width;
size_t skip;
+ size_t remainder;
if (skip_ansi ) {
while ((skip = display_mode_esc_sequence_len_counted(string,
@@ -239,7 +240,9 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
break;
}
- glyph_width = utf8_width(&string, NULL);
+ remainder = orig + len - string;
+
+ glyph_width = utf8_width(&string, &remainder);
if (glyph_width > 0)
width += glyph_width;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH v2 3/4] utf8: respect string boundaries after skipping ANSI escape sequences
2026-11-18 21:08 ` [PATCH v2 3/4] utf8: respect string boundaries after skipping ANSI escape sequences Dirk Gouders
@ 2026-11-18 21:49 ` Junio C Hamano
0 siblings, 0 replies; 41+ messages in thread
From: Junio C Hamano @ 2026-11-18 21:49 UTC (permalink / raw)
To: Dirk Gouders; +Cc: git
Dirk Gouders <dirk@gouders.net> writes:
> - while (skip_ansi &&
> - (skip = display_mode_esc_sequence_len_counted(string,
> - orig + len - string)) != 0)
> - string += skip;
> + if (skip_ansi ) {
Drop excess SP after the variable.
> + while ((skip = display_mode_esc_sequence_len_counted(string,
> + orig + len - string)) != 0)
> + string += skip;
Overlong line. You can (and are encouraged to) omit comparison with
zero "!= 0" to lose 5 columns but it still is overly long.
if (skip_ansi) {
while ((skip = display_mode_esc_sequence_len_counted(
string, orig + len - string)))
string += skip;
or something, perhaps.
> + if (string >= orig + len)
> + break;
> + }
>
> glyph_width = utf8_width(&string, NULL);
> if (glyph_width > 0)
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 4/4] utf8: respect boundaries in utf8_strnwidth()
2026-11-18 21:08 ` [PATCH v2 4/4] utf8: respect boundaries in utf8_strnwidth() Dirk Gouders
@ 2026-11-18 21:50 ` Junio C Hamano
0 siblings, 0 replies; 41+ messages in thread
From: Junio C Hamano @ 2026-11-18 21:50 UTC (permalink / raw)
To: Dirk Gouders; +Cc: git
Dirk Gouders <dirk@gouders.net> writes:
> utf8_strnwidth() can be called with boundaries that effectively cause
> incomplete multibyte sequences and we should return "len" in such
> cases.
>
> But, utf8_width is used without its optional boundary argument and
> will thus work beyond the given boundary.
>
> Fix this by using the boundary argument for utf8_width.
> Add some tests that cover such situations.
>
> Signed-off-by: Dirk Gouders <dirk@gouders.net>
> ---
> t/unit-tests/u-utf8-width.c | 9 +++++++++
> utf8.c | 5 ++++-
> 2 files changed, 13 insertions(+), 1 deletion(-)
Well spotted, and nicely done.
>
> diff --git a/t/unit-tests/u-utf8-width.c b/t/unit-tests/u-utf8-width.c
> index 5e2b636db9..c4df4dd6a6 100644
> --- a/t/unit-tests/u-utf8-width.c
> +++ b/t/unit-tests/u-utf8-width.c
> @@ -2,6 +2,15 @@
> #include "utf8.h"
> #include "strbuf.h"
>
> +/*
> + * Test utf8_strnwidth for cjk characters invalidated by len argument.
> + */
> +void test_utf8_width__strnwidth_cjk_chopped(void)
> +{
> + cl_assert_equal_i(1, utf8_strnwidth("三bytes long, 二columns wide", 1, 1));
> + cl_assert_equal_i(2, utf8_strnwidth("三bytes long, 二columns wide", 3, 1));
> +}
> +
> /*
> * Test utf8_strnwidth for strings with SGR sequences and corner case length
> * arguments.
> diff --git a/utf8.c b/utf8.c
> index 1b88324ea3..8d4b159e1d 100644
> --- a/utf8.c
> +++ b/utf8.c
> @@ -229,6 +229,7 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
> while (string && string < orig + len) {
> int glyph_width;
> size_t skip;
> + size_t remainder;
>
> if (skip_ansi ) {
> while ((skip = display_mode_esc_sequence_len_counted(string,
> @@ -239,7 +240,9 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
> break;
> }
>
> - glyph_width = utf8_width(&string, NULL);
> + remainder = orig + len - string;
> +
> + glyph_width = utf8_width(&string, &remainder);
> if (glyph_width > 0)
> width += glyph_width;
> }
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 2/4] utf8: fix comment for utf8_strnwidth()
2026-11-18 21:08 ` [PATCH v2 2/4] utf8: fix comment for utf8_strnwidth() Dirk Gouders
@ 2026-11-18 22:40 ` Andy Koppe
2026-11-19 5:58 ` Dirk Gouders
0 siblings, 1 reply; 41+ messages in thread
From: Andy Koppe @ 2026-11-18 22:40 UTC (permalink / raw)
To: Dirk Gouders; +Cc: git
Dirk Gouders wrote:
>
> The comment for utf8_strnwidth() states it returns strlen() for cases
> when the given string doesn't look like valid utf8, but it returns its
> argument len in such cases.
>
> Fix the comment accordingly.
…
> /*
> * Returns the total number of columns required by a null-terminated
> - * string, assuming that the string is utf8. Returns strlen() instead
> + * string, assuming that the string is utf8. Returns len instead
> * if the string does not look like a valid utf8 string.
> */
> int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
"null-terminated" is wrong too.
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-18 21:08 ` [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted() Dirk Gouders
@ 2026-11-19 5:54 ` Andy Koppe
2026-11-19 7:43 ` Dirk Gouders
2026-11-19 22:32 ` Dirk Gouders
2026-11-19 19:59 ` Junio C Hamano
1 sibling, 2 replies; 41+ messages in thread
From: Andy Koppe @ 2026-11-19 5:54 UTC (permalink / raw)
To: Dirk Gouders; +Cc: git, Junio C Hamano
Dirk Gouders wrote:
> +/* Same as above, but for a NUL-terminated string "s". */
> +size_t display_mode_esc_sequence_len(const char *s)
> +{
> + return display_mode_esc_sequence_len_counted(s, strlen(s));
> +}
This is a performance regression compared to the original function.
Calls that previously returned immediately because *s != '\033', or
after a few bytes of scanning, now scan the entire string with strlen.
It turns scanning a whole string for SGRs with repeated calls to the
function from linear to quadratic.
Therefore I think that if the original implementation isn't going to be
kept, it would be better not to provide the wrapper, and thereby make
callers think about what they want to pass as the length.
Uses on strbufs can of course compute it with `sb.buf + sb.len - s`,
but they don't actually need this safety net, because strbufs are
guaranteed to be NUL-terminated and the function stops at NULs.
And the length calculation itself adds a source of errors.
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 2/4] utf8: fix comment for utf8_strnwidth()
2026-11-18 22:40 ` Andy Koppe
@ 2026-11-19 5:58 ` Dirk Gouders
0 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-19 5:58 UTC (permalink / raw)
To: Andy Koppe; +Cc: git
On Wed, Nov 18 2026, Andy Koppe wrote:
> Dirk Gouders wrote:
>>
>> The comment for utf8_strnwidth() states it returns strlen() for cases
>> when the given string doesn't look like valid utf8, but it returns its
>> argument len in such cases.
>>
>> Fix the comment accordingly.
> …
>> /*
>> * Returns the total number of columns required by a null-terminated
>> - * string, assuming that the string is utf8. Returns strlen() instead
>> + * string, assuming that the string is utf8. Returns len instead
>> * if the string does not look like a valid utf8 string.
>> */
>> int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
>
> "null-terminated" is wrong too.
Yes, I will fix that, too. Thank you.
Dirk
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-19 5:54 ` Andy Koppe
@ 2026-11-19 7:43 ` Dirk Gouders
2026-11-19 20:58 ` Andy Koppe
2026-11-19 22:32 ` Dirk Gouders
1 sibling, 1 reply; 41+ messages in thread
From: Dirk Gouders @ 2026-11-19 7:43 UTC (permalink / raw)
To: Andy Koppe; +Cc: git, Junio C Hamano
On Thu, Nov 19 2026, Andy Koppe wrote:
> Dirk Gouders wrote:
>> +/* Same as above, but for a NUL-terminated string "s". */
>> +size_t display_mode_esc_sequence_len(const char *s)
>> +{
>> + return display_mode_esc_sequence_len_counted(s, strlen(s));
>> +}
>
> This is a performance regression compared to the original function.
> Calls that previously returned immediately because *s != '\033', or
> after a few bytes of scanning, now scan the entire string with strlen.
> It turns scanning a whole string for SGRs with repeated calls to the
> function from linear to quadratic.
>
> Therefore I think that if the original implementation isn't going to be
> kept, it would be better not to provide the wrapper, and thereby make
> callers think about what they want to pass as the length.
>
> Uses on strbufs can of course compute it with `sb.buf + sb.len - s`,
> but they don't actually need this safety net, because strbufs are
> guaranteed to be NUL-terminated and the function stops at NULs.
> And the length calculation itself adds a source of errors.
There is only one user in pretty.c that uses the wrapper.
I can change it to use the _counted version and we could then drop the
wrapper. Junio's v1 already included such a change and I could steal it
from there.
Would that be OK?
Dirk
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-18 21:08 ` [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted() Dirk Gouders
2026-11-19 5:54 ` Andy Koppe
@ 2026-11-19 19:59 ` Junio C Hamano
2026-11-19 20:28 ` Dirk Gouders
1 sibling, 1 reply; 41+ messages in thread
From: Junio C Hamano @ 2026-11-19 19:59 UTC (permalink / raw)
To: Dirk Gouders; +Cc: git
Dirk Gouders <dirk@gouders.net> writes:
> diff --git a/t/meson.build b/t/meson.build
> index 1cf1d95f70..e224df58c8 100644
> --- a/t/meson.build
> +++ b/t/meson.build
> @@ -27,6 +27,7 @@ clar_test_suites = [
> 'unit-tests/u-trailer.c',
> 'unit-tests/u-urlmatch-normalization.c',
> 'unit-tests/u-utf8-width.c',
> + 'unit-tests/u-utf8-ansi.c',
> ]
Should this list be reordered? "make -C t check-meson" complains.
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-19 19:59 ` Junio C Hamano
@ 2026-11-19 20:28 ` Dirk Gouders
0 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-19 20:28 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
On Thu, Nov 19 2026, Junio C Hamano wrote:
> Dirk Gouders <dirk@gouders.net> writes:
>
>> diff --git a/t/meson.build b/t/meson.build
>> index 1cf1d95f70..e224df58c8 100644
>> --- a/t/meson.build
>> +++ b/t/meson.build
>> @@ -27,6 +27,7 @@ clar_test_suites = [
>> 'unit-tests/u-trailer.c',
>> 'unit-tests/u-urlmatch-normalization.c',
>> 'unit-tests/u-utf8-width.c',
>> + 'unit-tests/u-utf8-ansi.c',
>> ]
>
> Should this list be reordered? "make -C t check-meson" complains.
Ah yes, thank you. The list in meson.txt must be sorted (check-meson
diff's the output of ls(1) and the list in meson.build).
I will fix the sorting order with v3.
Dirk
PS: The purpose of 154ce05cce (Makefile: detect missing Meson tests,
2024-12-13) wasn't to enforce a sorted order and maybe it would help to
put a comment in meson.build.
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-19 7:43 ` Dirk Gouders
@ 2026-11-19 20:58 ` Andy Koppe
2026-11-19 21:04 ` Andy Koppe
0 siblings, 1 reply; 41+ messages in thread
From: Andy Koppe @ 2026-11-19 20:58 UTC (permalink / raw)
To: Dirk Gouders; +Cc: git, Junio C Hamano
Dirk Gouders wrote:
>
> On Thu, Nov 19 2026, Andy Koppe wrote:
>
> > Dirk Gouders wrote:
> >> +/* Same as above, but for a NUL-terminated string "s". */
> >> +size_t display_mode_esc_sequence_len(const char *s)
> >> +{
> >> + return display_mode_esc_sequence_len_counted(s, strlen(s));
> >> +}
> >
> > This is a performance regression compared to the original function.
> > Calls that previously returned immediately because *s != '\033', or
> > after a few bytes of scanning, now scan the entire string with strlen.
> > It turns scanning a whole string for SGRs with repeated calls to the
> > function from linear to quadratic.
> >
> > Therefore I think that if the original implementation isn't going to be
> > kept, it would be better not to provide the wrapper, and thereby make
> > callers think about what they want to pass as the length.
> >
> > Uses on strbufs can of course compute it with `sb.buf + sb.len - s`,
> > but they don't actually need this safety net, because strbufs are
> > guaranteed to be NUL-terminated and the function stops at NULs.
> > And the length calculation itself adds a source of errors.
>
> There is only one user in pretty.c that uses the wrapper.
> I can change it to use the _counted version and we could then drop the
> wrapper. Junio's v1 already included such a change and I could steal it
> from there.
>
> Would that be OK?
No objection from me. I would change the make_line_colors_self_contained()
function I'm proposing in [1] to the _counted variant anyway if the original
display_mode_esc_sequence_len() is going away, to avoid the performance issue
with the wrapper.
[1] https://lore.kernel.org/git/e9f88b533dfbd10528c9151e2e87207419d0253d.1794905511.git.andy.koppe@gmail.com
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-19 20:58 ` Andy Koppe
@ 2026-11-19 21:04 ` Andy Koppe
2026-11-19 21:33 ` Dirk Gouders
0 siblings, 1 reply; 41+ messages in thread
From: Andy Koppe @ 2026-11-19 21:04 UTC (permalink / raw)
To: Dirk Gouders; +Cc: git, Junio C Hamano
Andy Koppe wrote:
> Dirk Gouders wrote:
> > There is only one user in pretty.c that uses the wrapper.
> > I can change it to use the _counted version and we could then drop the
> > wrapper. Junio's v1 already included such a change and I could steal it
> > from there.
> >
> > Would that be OK?
>
> No objection from me. I would change the make_line_colors_self_contained()
> function I'm proposing in [1] to the _counted variant anyway if the original
> display_mode_esc_sequence_len() is going away, to avoid the performance issue
> with the wrapper.
>
> [1] https://lore.kernel.org/git/e9f88b533dfbd10528c9151e2e87207419d0253d.1794905511.git.andy.koppe@gmail.com
ps: I guess you might as well just drop the _counted suffix then, and
basically just add the len parameter to display_mode_esc_sequence_len().
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-19 21:04 ` Andy Koppe
@ 2026-11-19 21:33 ` Dirk Gouders
0 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-19 21:33 UTC (permalink / raw)
To: Andy Koppe; +Cc: git, Junio C Hamano
On Thu, Nov 19 2026, Andy Koppe wrote:
> Andy Koppe wrote:
>> Dirk Gouders wrote:
>> > There is only one user in pretty.c that uses the wrapper.
>> > I can change it to use the _counted version and we could then drop the
>> > wrapper. Junio's v1 already included such a change and I could steal it
>> > from there.
>> >
>> > Would that be OK?
>>
>> No objection from me. I would change the make_line_colors_self_contained()
>> function I'm proposing in [1] to the _counted variant anyway if the original
>> display_mode_esc_sequence_len() is going away, to avoid the performance issue
>> with the wrapper.
>>
>> [1] https://lore.kernel.org/git/e9f88b533dfbd10528c9151e2e87207419d0253d.1794905511.git.andy.koppe@gmail.com
>
> ps: I guess you might as well just drop the _counted suffix then, and
> basically just add the len parameter to display_mode_esc_sequence_len().
That was the plan. I just checked v1 and all this would actually mean to use
Junio's v1 instead of v2.
Dirk
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-19 5:54 ` Andy Koppe
2026-11-19 7:43 ` Dirk Gouders
@ 2026-11-19 22:32 ` Dirk Gouders
2026-11-19 22:59 ` Dirk Gouders
2026-11-20 10:16 ` Andy Koppe
1 sibling, 2 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-19 22:32 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Andy Koppe
On Thu, Nov 19 2026, Andy Koppe wrote:
> Dirk Gouders wrote:
>> +/* Same as above, but for a NUL-terminated string "s". */
>> +size_t display_mode_esc_sequence_len(const char *s)
>> +{
>> + return display_mode_esc_sequence_len_counted(s, strlen(s));
>> +}
>
> This is a performance regression compared to the original function.
> Calls that previously returned immediately because *s != '\033', or
> after a few bytes of scanning, now scan the entire string with strlen.
> It turns scanning a whole string for SGRs with repeated calls to the
> function from linear to quadratic.
How about creating a v3 for display_mode_esc_sequence_len() similar to
utf8_width() to accept an optional remainder argument?
If NULL is given, display_mode_esc_sequence_len() assumes a
null-terminated string, otherwise the length is given in *remainder.
The function can then update the remainder and return the length.
Just an idea.
Dirk
> Therefore I think that if the original implementation isn't going to be
> kept, it would be better not to provide the wrapper, and thereby make
> callers think about what they want to pass as the length.
>
> Uses on strbufs can of course compute it with `sb.buf + sb.len - s`,
> but they don't actually need this safety net, because strbufs are
> guaranteed to be NUL-terminated and the function stops at NULs.
> And the length calculation itself adds a source of errors.
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-19 22:32 ` Dirk Gouders
@ 2026-11-19 22:59 ` Dirk Gouders
2026-11-20 10:16 ` Andy Koppe
1 sibling, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-19 22:59 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Andy Koppe
On Thu, Nov 19 2026, Dirk Gouders wrote:
> On Thu, Nov 19 2026, Andy Koppe wrote:
>
>> Dirk Gouders wrote:
>>> +/* Same as above, but for a NUL-terminated string "s". */
>>> +size_t display_mode_esc_sequence_len(const char *s)
>>> +{
>>> + return display_mode_esc_sequence_len_counted(s, strlen(s));
>>> +}
>>
>> This is a performance regression compared to the original function.
>> Calls that previously returned immediately because *s != '\033', or
>> after a few bytes of scanning, now scan the entire string with strlen.
>> It turns scanning a whole string for SGRs with repeated calls to the
>> function from linear to quadratic.
>
> How about creating a v3 for display_mode_esc_sequence_len() similar to
> utf8_width() to accept an optional remainder argument?
>
> If NULL is given, display_mode_esc_sequence_len() assumes a
> null-terminated string, otherwise the length is given in *remainder.
> The function can then update the remainder and return the length.
>
> Just an idea.
Something like this untested example (please ignore the _counted):
diff --git a/utf8.c b/utf8.c
index 8d4b159e1d..fe6f288179 100644
--- a/utf8.c
+++ b/utf8.c
@@ -22,18 +22,24 @@ struct interval {
* of string "s", or 0 if the bytes do not begin with a complete
* such sequence. Never looks at s[len] or beyond.
*/
-size_t display_mode_esc_sequence_len_counted(const char *s, size_t len)
+size_t display_mode_esc_sequence_len_counted(const char *s, size_t *len)
{
- const char *p = s, *end = s + len;
+ const char *p = s, *end = NULL;
- if (end <= p || *p++ != '\033')
+ if (len)
+ end = s + *len;
+
+ if ((end && end <= p) || *p++ != '\033')
return 0;
- if (end <= p || *p++ != '[')
+ if ((end && end <= p) || *p++ != '[')
return 0;
- while (p < end && (isdigit(*p) || *p == ';'))
+ while (((end && p < end) || (!end && *p)) && (isdigit(*p) || *p == ';'))
p++;
- if (end <= p || *p++ != 'm')
+ if ((end && end <= p) || (!end && !*p) || *p++ != 'm')
return 0;
+
+ if (len)
+ *len -= p - s;
return p - s;
}
^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-19 22:32 ` Dirk Gouders
2026-11-19 22:59 ` Dirk Gouders
@ 2026-11-20 10:16 ` Andy Koppe
2026-11-20 16:36 ` Junio C Hamano
1 sibling, 1 reply; 41+ messages in thread
From: Andy Koppe @ 2026-11-20 10:16 UTC (permalink / raw)
To: Dirk Gouders; +Cc: Junio C Hamano, git
On Thu, 19 Nov 2026 at 22:33, Dirk Gouders wrote:
>
> On Thu, Nov 19 2026, Andy Koppe wrote:
>
> > Dirk Gouders wrote:
> >> +/* Same as above, but for a NUL-terminated string "s". */
> >> +size_t display_mode_esc_sequence_len(const char *s)
> >> +{
> >> + return display_mode_esc_sequence_len_counted(s, strlen(s));
> >> +}
> >
> > This is a performance regression compared to the original function.
> > Calls that previously returned immediately because *s != '\033', or
> > after a few bytes of scanning, now scan the entire string with strlen.
> > It turns scanning a whole string for SGRs with repeated calls to the
> > function from linear to quadratic.
>
> How about creating a v3 for display_mode_esc_sequence_len() similar to
> utf8_width() to accept an optional remainder argument?
>
> If NULL is given, display_mode_esc_sequence_len() assumes a
> null-terminated string, otherwise the length is given in *remainder.
> The function can then update the remainder and return the length.
Callers could already sidestep the length check by passing SIZE_MAX, and
the number of consumed bytes is returned, so I don't think that's needed.
Andy
^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix
2026-11-16 18:56 [PATCH 0/2] utf8: two minor fixes Dirk Gouders
` (2 preceding siblings ...)
2026-11-18 21:08 ` [PATCH v2 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
@ 2026-11-20 14:44 ` Dirk Gouders
2026-11-20 14:44 ` [PATCH v3 1/4] utf8: introduce bounds for display_mode_esc_sequence_len() Dirk Gouders
` (4 more replies)
2026-11-20 15:11 ` [PATCH v4 0/5] " Dirk Gouders
4 siblings, 5 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-20 14:44 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Dirk Gouders, Andy Koppe
A patch here on the list made me look at
display_mode_esc_sequence_len() in utf8.c and I found two details in
the file that are perhaps worth a fix:
1) utf8_strnwidth() disregards the given len argument when fed
with proper string and len.
2) The comment for utf8_strnwidth() is incorrect: it states the
function returns strlen() for cases when it actually returns its
len argument.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
Changes in v3:
- 1/5: * fix sorting order in t/meson.build
* fix risk of O(n²) degradation:
drop the wrapper function and introduce an end pointer to
display_mode_esc_sequence_len()
(I was so used to len arguments for string handling functions
that I only now realized the beauty of optionally optional
end pointers for string handling. I know: beauty is in the
eye of the beholder ;-))
* try to fix an overly long line
- 2/5: also fix a further error concerning "null-terminated"
- 3/5: * drop excess SP in if-statement
- 5/5: this actually is a POC, not fixing a bug:
users of display_mode_esc_sequence_len() often need the length
of all leading SGR sequences and here is a function that does that.
I mentioned such a function in the discussion for v1.
- Link to v2: https://patch.msgid.link/20261118-master-v2-0-3f2aaec15d31@gouders.net
Changes in v2:
- Add a patch by Junio that introduces a bounded function display_mode_esc_sequence_len_counted()
I added some tests for this new function.
- 2/2 became 2/4
- 3/4 is the modified 1/2 and uses the new
display_mode_esc_sequence_len_counted() so that the while loop stays
within the given bounds.
The commit message was adjusted.
One of the tests was wrong: "\033" has width 0, so the sum must be 3 instead of 4.
- 4/4 is an addition to fix a similar problem with utf8_width().
That function is used without its optional bounds argument and
can violate boundaries as well. Test for such cases are added.
- Link to v1: https://patch.msgid.link/20261116-master-v1-0-dc2a50ce0aa0@gouders.net
---
Dirk Gouders (3):
utf8: fix comment for utf8_strnwidth()
utf8: respect string boundaries after skipping ANSI escape sequences
utf8: respect boundaries in utf8_strnwidth()
Junio C Hamano (1):
utf8: introduce bounds for display_mode_esc_sequence_len()
Makefile | 1 +
pretty.c | 2 +-
t/meson.build | 1 +
t/unit-tests/u-utf8-ansi.c | 85 +++++++++++++++++++++++++++++++++++++++++++++
t/unit-tests/u-utf8-width.c | 20 +++++++++++
utf8.c | 78 +++++++++++++++++++++++++++++------------
utf8.h | 3 +-
7 files changed, 165 insertions(+), 25 deletions(-)
Range-diff versus v2:
1: 60979dc4d6 < -: ---------- utf8: add display_mode_esc_sequence_len_counted()
2: b8dbf5c5bf < -: ---------- utf8: fix comment for utf8_strnwidth()
-: ---------- > 1: 5b938b3fa0 utf8: introduce bounds for display_mode_esc_sequence_len()
-: ---------- > 2: 5ef8308d51 utf8: fix comment for utf8_strnwidth()
3: 12a99d1c8d ! 3: bd7c9b0b01 utf8: respect string boundaries after skipping ANSI escape sequences
@@ utf8.c: int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
size_t skip;
- while (skip_ansi &&
-- (skip = display_mode_esc_sequence_len_counted(string,
-- orig + len - string)) != 0)
+- (skip = display_mode_esc_sequence_len(string, end)))
- string += skip;
-+ if (skip_ansi ) {
-+ while ((skip = display_mode_esc_sequence_len_counted(string,
-+ orig + len - string)) != 0)
++ if (skip_ansi) {
++ while ((skip = display_mode_esc_sequence_len(string, end)))
+ string += skip;
+
-+ if (string >= orig + len)
++ if (string >= end)
+ break;
+ }
4: 471c3d6888 < -: ---------- utf8: respect boundaries in utf8_strnwidth()
-: ---------- > 4: a6cf4c99dc utf8: respect boundaries in utf8_strnwidth()
---
base-commit: 7fb6055e678f800320d8ab8cd8f201d9493d1ba0
change-id: 20261116-master-69b171612823
^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v3 1/4] utf8: introduce bounds for display_mode_esc_sequence_len()
2026-11-20 14:44 ` [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
@ 2026-11-20 14:44 ` Dirk Gouders
2026-11-20 14:44 ` [PATCH v3 2/4] utf8: fix comment for utf8_strnwidth() Dirk Gouders
` (3 subsequent siblings)
4 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-20 14:44 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Dirk Gouders, Andy Koppe
From: Junio C Hamano <gitster@pobox.com>
display_mode_esc_sequence_len() scans for an ANSI display mode
escape sequence (e.g. "ESC [ 1 ; 31 m") at the start of a string,
relying on NUL termination to stop. Some callers, however, work on
a counted buffer: utf8_strnwidth() takes <string, len> and must not
scan past it, while strbuf_utf8_replace() and
strbuf_add_wrapped_text() know (or can cheaply learn) the end.
Introduce an optional end pointer to display_mode_esc_sequence_len()
which then never reads *end or beyond when end is not NULL.
If end is NULL, display_mode_esc_sequence_len() assumes a
NUL terminated string as before.
Andy Koppe pointed out the risk of performance degrading to O(n²)
when functions process strings and have to check each position for an
escape sequence [1]. This change doesn't carry this risk.
Further, add thorough tests for both end pointer cases.
[1] https://lore.kernel.org/git/CAHWeT-aWx1VwUjM+hZ--jjSAbcMenz5Mw11y9ouFavGndUMrSA@mail.gmail.com/
Cc: Andy Koppe <andy.koppe@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
Makefile | 1 +
pretty.c | 2 +-
t/meson.build | 1 +
t/unit-tests/u-utf8-ansi.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++
utf8.c | 34 ++++++++++++++++----------
utf8.h | 2 +-
6 files changed, 85 insertions(+), 14 deletions(-)
diff --git a/Makefile b/Makefile
index 4c84eea6e2..db8f7f5c60 100644
--- a/Makefile
+++ b/Makefile
@@ -1563,6 +1563,7 @@ CLAR_TEST_SUITES += u-string-list
CLAR_TEST_SUITES += u-strvec
CLAR_TEST_SUITES += u-trailer
CLAR_TEST_SUITES += u-urlmatch-normalization
+CLAR_TEST_SUITES += u-utf8-ansi
CLAR_TEST_SUITES += u-utf8-width
CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES))
diff --git a/pretty.c b/pretty.c
index d8a9f370f6..58dfeb61f4 100644
--- a/pretty.c
+++ b/pretty.c
@@ -1848,7 +1848,7 @@ static size_t format_and_pad_commit(struct strbuf *sb, /* in UTF-8 */
while (p > sb->buf && ch - p < 10 && *p != '\033')
p--;
if (*p != '\033' ||
- ch + 1 - p != display_mode_esc_sequence_len(p))
+ ch + 1 - p != display_mode_esc_sequence_len(p, ch + 1))
break;
/*
* got a good ansi sequence, put it back to
diff --git a/t/meson.build b/t/meson.build
index 1cf1d95f70..ae7479a98e 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -26,6 +26,7 @@ clar_test_suites = [
'unit-tests/u-strvec.c',
'unit-tests/u-trailer.c',
'unit-tests/u-urlmatch-normalization.c',
+ 'unit-tests/u-utf8-ansi.c',
'unit-tests/u-utf8-width.c',
]
diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c
new file mode 100644
index 0000000000..cb75c8612b
--- /dev/null
+++ b/t/unit-tests/u-utf8-ansi.c
@@ -0,0 +1,59 @@
+#include "unit-test.h"
+#include "utf8.h"
+#include "strbuf.h"
+
+/*
+ * Test display_mode_esc_sequence_len() for NUL terminated strings.
+ */
+void test_utf8_ansi__sgr_len_null(void)
+{
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("", NULL));
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("\033", NULL));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("\033[", NULL));
+ cl_assert_equal_i(3, display_mode_esc_sequence_len("\033[m", NULL));
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("\033[3", NULL));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("\033[31", NULL));
+ cl_assert_equal_i(5, display_mode_esc_sequence_len("\033[31m", NULL));
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("\033[31;", NULL));
+ cl_assert_equal_i(6, display_mode_esc_sequence_len("\033[31;m", NULL));
+ cl_assert_equal_i(7, display_mode_esc_sequence_len("\033[31;0m", NULL));
+}
+
+/*
+ * Test display_mode_esc_sequence_len() with end pointers.
+ */
+void test_utf8_ansi__sgr_len_end_ptr(void)
+{
+ char *sgr_1 = (char *)"\033[m";
+ char *sgr_2 = (char *)"\033[31m";
+ char *sgr_3 = (char *)"\033[31;m";
+ char *sgr_4 = (char *)"\033[31;0m";
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("", ""));
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_1, sgr_1 + 1));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_1, sgr_1 + 2));
+ cl_assert_equal_i(3, display_mode_esc_sequence_len(sgr_1, sgr_1 + 3));
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_2, sgr_2 + 1));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_2, sgr_2 + 2));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_2, sgr_2 + 3));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_2, sgr_2 + 4));
+ cl_assert_equal_i(5, display_mode_esc_sequence_len(sgr_2, sgr_2 + 5));
+
+ cl_assert_equal_i(6, display_mode_esc_sequence_len(sgr_3, sgr_3 + sizeof(sgr_3) - 1));
+ cl_assert_equal_i(7, display_mode_esc_sequence_len(sgr_4, sgr_4 + sizeof(sgr_4) - 1));
+}
+
+/*
+ * Test display_mode_esc_sequence_len() for swapped start and end pointers.
+ */
+void test_utf8_ansi__sgr_len_start_end_swap(void)
+{
+ char *sgr_1 = (char *)"\033[m";
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_1 + 3, sgr_1));
+}
diff --git a/utf8.c b/utf8.c
index 96460cc414..48aa189624 100644
--- a/utf8.c
+++ b/utf8.c
@@ -16,18 +16,27 @@ struct interval {
ucs_char_t last;
};
-size_t display_mode_esc_sequence_len(const char *s)
+/*
+ * Return the length of the ANSI display mode escape sequence (e.g.,
+ * "\033[1;31m") that appears at the beginning of the string starting at start,
+ * or 0 if the string does not begin with a complete such sequence.
+ * If end is NULL, assume a NUL terminated string, else never look at *end or
+ * beyond.
+ */
+size_t display_mode_esc_sequence_len(const char *start, const char *end)
{
- const char *p = s;
- if (*p++ != '\033')
+ const char *p = start;
+
+ if ((end && end <= p) || *p++ != '\033')
return 0;
- if (*p++ != '[')
+ if ((end && end <= p) || *p++ != '[')
return 0;
- while (isdigit(*p) || *p == ';')
+ while (((end && p < end) || (!end && *p)) && (isdigit(*p) || *p == ';'))
p++;
- if (*p++ != 'm')
+ if ((end && end <= p) || (!end && !*p) || *p++ != 'm')
return 0;
- return p - s;
+
+ return p - start;
}
/* auxiliary function for binary search in interval table */
@@ -210,15 +219,15 @@ int utf8_width(const char **start, size_t *remainder_p)
*/
int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
{
- const char *orig = string;
+ const char *end = string + len;
size_t width = 0;
- while (string && string < orig + len) {
+ while (string && string < end) {
int glyph_width;
size_t skip;
while (skip_ansi &&
- (skip = display_mode_esc_sequence_len(string)) != 0)
+ (skip = display_mode_esc_sequence_len(string, end)))
string += skip;
glyph_width = utf8_width(&string, NULL);
@@ -279,6 +288,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
{
int indent, w, assume_utf8 = 1;
const char *bol, *space, *start = text;
+ const char *end = text + strlen(text);
size_t orig_len = buf->len;
if (width <= 0) {
@@ -299,7 +309,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
char c;
size_t skip;
- while ((skip = display_mode_esc_sequence_len(text)))
+ while ((skip = display_mode_esc_sequence_len(text, end)))
text += skip;
c = *text;
@@ -378,7 +388,7 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
int glyph_width;
size_t n;
- while ((n = display_mode_esc_sequence_len(src))) {
+ while ((n = display_mode_esc_sequence_len(src, end))) {
strbuf_add(&dst, src, n);
src += n;
}
diff --git a/utf8.h b/utf8.h
index cf8ecb0f21..7eb7bf042a 100644
--- a/utf8.h
+++ b/utf8.h
@@ -5,7 +5,7 @@ struct strbuf;
typedef unsigned int ucs_char_t; /* assuming 32bit int */
-size_t display_mode_esc_sequence_len(const char *s);
+size_t display_mode_esc_sequence_len(const char *start, const char *end);
int utf8_width(const char **start, size_t *remainder_p);
int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
int utf8_strwidth(const char *string);
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v3 2/4] utf8: fix comment for utf8_strnwidth()
2026-11-20 14:44 ` [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
2026-11-20 14:44 ` [PATCH v3 1/4] utf8: introduce bounds for display_mode_esc_sequence_len() Dirk Gouders
@ 2026-11-20 14:44 ` Dirk Gouders
2026-11-20 14:44 ` [PATCH v3 3/4] utf8: respect string boundaries after skipping ANSI escape sequences Dirk Gouders
` (2 subsequent siblings)
4 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-20 14:44 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Dirk Gouders, Andy Koppe
The comment for utf8_strnwidth() states it works on a NUL terminated
string whereas the string is bounded by len.
It also states it returns strlen() for cases when the given string
doesn't look like valid utf8, but it returns its argument len in such
cases.
Fix the comment accordingly.
Helped-by: Andy Koppe <andy.koppe@gmail.com>
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
utf8.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/utf8.c b/utf8.c
index 48aa189624..35b2bb8290 100644
--- a/utf8.c
+++ b/utf8.c
@@ -213,8 +213,8 @@ int utf8_width(const char **start, size_t *remainder_p)
}
/*
- * Returns the total number of columns required by a null-terminated
- * string, assuming that the string is utf8. Returns strlen() instead
+ * Returns the total number of columns required by a string terminated by len,
+ * assuming that the string is utf8. Returns len instead
* if the string does not look like a valid utf8 string.
*/
int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v3 3/4] utf8: respect string boundaries after skipping ANSI escape sequences
2026-11-20 14:44 ` [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
2026-11-20 14:44 ` [PATCH v3 1/4] utf8: introduce bounds for display_mode_esc_sequence_len() Dirk Gouders
2026-11-20 14:44 ` [PATCH v3 2/4] utf8: fix comment for utf8_strnwidth() Dirk Gouders
@ 2026-11-20 14:44 ` Dirk Gouders
2026-11-20 14:44 ` [PATCH v3 4/4] utf8: respect boundaries in utf8_strnwidth() Dirk Gouders
2026-11-20 15:09 ` [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
4 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-20 14:44 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Dirk Gouders
utf8_strnwidth() can skip ANSI escape sequences and it is possible
that such sequences end the string within the given boundary.
Check for such situations before proceeding and calling utf8_width().
Add tests that test such corner cases.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
t/unit-tests/u-utf8-width.c | 11 +++++++++++
utf8.c | 10 +++++++---
2 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/t/unit-tests/u-utf8-width.c b/t/unit-tests/u-utf8-width.c
index 86e09c3574..5e2b636db9 100644
--- a/t/unit-tests/u-utf8-width.c
+++ b/t/unit-tests/u-utf8-width.c
@@ -2,6 +2,17 @@
#include "utf8.h"
#include "strbuf.h"
+/*
+ * Test utf8_strnwidth for strings with SGR sequences and corner case length
+ * arguments.
+ */
+void test_utf8_width__strnwidth_ansi(void)
+{
+ cl_assert_equal_i(3, utf8_strnwidth("\033[31mhello", 4, 1));
+ cl_assert_equal_i(0, utf8_strnwidth("\033[31mhello", 5, 1));
+ cl_assert_equal_i(1, utf8_strnwidth("\033[31mhello", 6, 1));
+}
+
/*
* Test utf8_strnwidth with various Chinese strings
* Chinese characters typically have a width of 2 columns when displayed
diff --git a/utf8.c b/utf8.c
index 35b2bb8290..493f723e3b 100644
--- a/utf8.c
+++ b/utf8.c
@@ -226,9 +226,13 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
int glyph_width;
size_t skip;
- while (skip_ansi &&
- (skip = display_mode_esc_sequence_len(string, end)))
- string += skip;
+ if (skip_ansi) {
+ while ((skip = display_mode_esc_sequence_len(string, end)))
+ string += skip;
+
+ if (string >= end)
+ break;
+ }
glyph_width = utf8_width(&string, NULL);
if (glyph_width > 0)
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v3 4/4] utf8: respect boundaries in utf8_strnwidth()
2026-11-20 14:44 ` [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
` (2 preceding siblings ...)
2026-11-20 14:44 ` [PATCH v3 3/4] utf8: respect string boundaries after skipping ANSI escape sequences Dirk Gouders
@ 2026-11-20 14:44 ` Dirk Gouders
2026-11-20 15:09 ` [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
4 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-20 14:44 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Dirk Gouders
utf8_strnwidth() can be called with boundaries that effectively cause
incomplete multibyte sequences and we should return "len" in such
cases.
But, utf8_width is used without its optional boundary argument and
will thus work beyond the given boundary.
Fix this by using the boundary argument for utf8_width.
Add some tests that cover such situations.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
t/unit-tests/u-utf8-ansi.c | 26 ++++++++++++++++++++++++++
t/unit-tests/u-utf8-width.c | 9 +++++++++
utf8.c | 40 +++++++++++++++++++++++++++++-----------
utf8.h | 1 +
4 files changed, 65 insertions(+), 11 deletions(-)
diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c
index cb75c8612b..909643d07d 100644
--- a/t/unit-tests/u-utf8-ansi.c
+++ b/t/unit-tests/u-utf8-ansi.c
@@ -2,6 +2,32 @@
#include "utf8.h"
#include "strbuf.h"
+/*
+ * Test the greedy function display_mode_esc_sequence_len_all().
+ */
+void test_utf8_ansi__sgr_len_greedy(void)
+{
+ char *s1 = (char *)"no ESC sequence";
+ char *s2 = (char *)"\033[m";
+ char *s3 = (char *)"\033[0;31m";
+ char *s4 = (char *)"\033[0m\033[31m";
+
+ /* no sequences */
+ cl_assert_equal_i(0, display_mode_esc_sequence_len_all("", ""));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len_all(s1, s1 + 5));
+
+ /* single sequences */
+ cl_assert_equal_i(3, display_mode_esc_sequence_len_all(s2, s2 + 3));
+ cl_assert_equal_i(7, display_mode_esc_sequence_len_all(s3, s3 + 7));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len_all(s3, s3 + 3));
+
+ /* multiple sequences */
+ cl_assert_equal_i(4, display_mode_esc_sequence_len_all(s4, s4 + 4));
+ cl_assert_equal_i(9, display_mode_esc_sequence_len_all(s4, s4 + 9));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len_all(s4, s4 + 3));
+ cl_assert_equal_i(4, display_mode_esc_sequence_len_all(s4, s4 + 8));
+}
+
/*
* Test display_mode_esc_sequence_len() for NUL terminated strings.
*/
diff --git a/t/unit-tests/u-utf8-width.c b/t/unit-tests/u-utf8-width.c
index 5e2b636db9..c4df4dd6a6 100644
--- a/t/unit-tests/u-utf8-width.c
+++ b/t/unit-tests/u-utf8-width.c
@@ -2,6 +2,15 @@
#include "utf8.h"
#include "strbuf.h"
+/*
+ * Test utf8_strnwidth for cjk characters invalidated by len argument.
+ */
+void test_utf8_width__strnwidth_cjk_chopped(void)
+{
+ cl_assert_equal_i(1, utf8_strnwidth("三bytes long, 二columns wide", 1, 1));
+ cl_assert_equal_i(2, utf8_strnwidth("三bytes long, 二columns wide", 3, 1));
+}
+
/*
* Test utf8_strnwidth for strings with SGR sequences and corner case length
* arguments.
diff --git a/utf8.c b/utf8.c
index 493f723e3b..9478985cbb 100644
--- a/utf8.c
+++ b/utf8.c
@@ -39,6 +39,26 @@ size_t display_mode_esc_sequence_len(const char *start, const char *end)
return p - start;
}
+/*
+ * Count the lenght of all leading ANSI display mode escape sequences that
+ * are a prefix of the string at start up to end.
+ */
+size_t display_mode_esc_sequence_len_all(const char *start, const char *end)
+{
+ size_t seq_len = 0;
+
+ while (start + seq_len < end) {
+ size_t count = display_mode_esc_sequence_len(start + seq_len, end);
+
+ if (!count)
+ break;
+
+ seq_len += count;
+ }
+
+ return seq_len;
+}
+
/* auxiliary function for binary search in interval table */
static int bisearch(ucs_char_t ucs, const struct interval *table, int max)
{
@@ -224,17 +244,18 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
while (string && string < end) {
int glyph_width;
- size_t skip;
+ size_t remainder;
if (skip_ansi) {
- while ((skip = display_mode_esc_sequence_len(string, end)))
- string += skip;
+ string += display_mode_esc_sequence_len_all(string, end);
if (string >= end)
break;
}
- glyph_width = utf8_width(&string, NULL);
+ remainder = end - string;
+
+ glyph_width = utf8_width(&string, &remainder);
if (glyph_width > 0)
width += glyph_width;
}
@@ -311,10 +332,8 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
for (;;) {
char c;
- size_t skip;
- while ((skip = display_mode_esc_sequence_len(text, end)))
- text += skip;
+ text += display_mode_esc_sequence_len_all(text, end);
c = *text;
if (!c || isspace(c)) {
@@ -392,10 +411,9 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
int glyph_width;
size_t n;
- while ((n = display_mode_esc_sequence_len(src, end))) {
- strbuf_add(&dst, src, n);
- src += n;
- }
+ n = display_mode_esc_sequence_len_all(src, end);
+ strbuf_add(&dst, src, n);
+ src += n;
if (src >= end)
break;
diff --git a/utf8.h b/utf8.h
index 7eb7bf042a..fdde5af782 100644
--- a/utf8.h
+++ b/utf8.h
@@ -6,6 +6,7 @@ struct strbuf;
typedef unsigned int ucs_char_t; /* assuming 32bit int */
size_t display_mode_esc_sequence_len(const char *start, const char *end);
+size_t display_mode_esc_sequence_len_all(const char *start, const char *end);
int utf8_width(const char **start, size_t *remainder_p);
int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
int utf8_strwidth(const char *string);
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix
2026-11-20 14:44 ` [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
` (3 preceding siblings ...)
2026-11-20 14:44 ` [PATCH v3 4/4] utf8: respect boundaries in utf8_strnwidth() Dirk Gouders
@ 2026-11-20 15:09 ` Dirk Gouders
4 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-20 15:09 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Andy Koppe
I used --reflect >5 times but with the final "simple" polish I squased
two commits -- and did not --reflect one more time...
Please disregard this series.
Dirk
On Fri, Nov 20 2026, Dirk Gouders wrote:
> A patch here on the list made me look at
> display_mode_esc_sequence_len() in utf8.c and I found two details in
> the file that are perhaps worth a fix:
>
> 1) utf8_strnwidth() disregards the given len argument when fed
> with proper string and len.
>
> 2) The comment for utf8_strnwidth() is incorrect: it states the
> function returns strlen() for cases when it actually returns its
> len argument.
>
> Signed-off-by: Dirk Gouders <dirk@gouders.net>
> ---
> Changes in v3:
> - 1/5: * fix sorting order in t/meson.build
> * fix risk of O(n²) degradation:
> drop the wrapper function and introduce an end pointer to
> display_mode_esc_sequence_len()
> (I was so used to len arguments for string handling functions
> that I only now realized the beauty of optionally optional
> end pointers for string handling. I know: beauty is in the
> eye of the beholder ;-))
> * try to fix an overly long line
> - 2/5: also fix a further error concerning "null-terminated"
> - 3/5: * drop excess SP in if-statement
> - 5/5: this actually is a POC, not fixing a bug:
> users of display_mode_esc_sequence_len() often need the length
> of all leading SGR sequences and here is a function that does that.
> I mentioned such a function in the discussion for v1.
> - Link to v2: https://patch.msgid.link/20261118-master-v2-0-3f2aaec15d31@gouders.net
>
> Changes in v2:
> - Add a patch by Junio that introduces a bounded function display_mode_esc_sequence_len_counted()
> I added some tests for this new function.
> - 2/2 became 2/4
> - 3/4 is the modified 1/2 and uses the new
> display_mode_esc_sequence_len_counted() so that the while loop stays
> within the given bounds.
> The commit message was adjusted.
> One of the tests was wrong: "\033" has width 0, so the sum must be 3 instead of 4.
> - 4/4 is an addition to fix a similar problem with utf8_width().
> That function is used without its optional bounds argument and
> can violate boundaries as well. Test for such cases are added.
> - Link to v1: https://patch.msgid.link/20261116-master-v1-0-dc2a50ce0aa0@gouders.net
>
> ---
> Dirk Gouders (3):
> utf8: fix comment for utf8_strnwidth()
> utf8: respect string boundaries after skipping ANSI escape sequences
> utf8: respect boundaries in utf8_strnwidth()
>
> Junio C Hamano (1):
> utf8: introduce bounds for display_mode_esc_sequence_len()
>
> Makefile | 1 +
> pretty.c | 2 +-
> t/meson.build | 1 +
> t/unit-tests/u-utf8-ansi.c | 85 +++++++++++++++++++++++++++++++++++++++++++++
> t/unit-tests/u-utf8-width.c | 20 +++++++++++
> utf8.c | 78 +++++++++++++++++++++++++++++------------
> utf8.h | 3 +-
> 7 files changed, 165 insertions(+), 25 deletions(-)
>
> Range-diff versus v2:
>
> 1: 60979dc4d6 < -: ---------- utf8: add display_mode_esc_sequence_len_counted()
> 2: b8dbf5c5bf < -: ---------- utf8: fix comment for utf8_strnwidth()
> -: ---------- > 1: 5b938b3fa0 utf8: introduce bounds for display_mode_esc_sequence_len()
> -: ---------- > 2: 5ef8308d51 utf8: fix comment for utf8_strnwidth()
> 3: 12a99d1c8d ! 3: bd7c9b0b01 utf8: respect string boundaries after skipping ANSI escape sequences
> @@ utf8.c: int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
> size_t skip;
>
> - while (skip_ansi &&
> -- (skip = display_mode_esc_sequence_len_counted(string,
> -- orig + len - string)) != 0)
> +- (skip = display_mode_esc_sequence_len(string, end)))
> - string += skip;
> -+ if (skip_ansi ) {
> -+ while ((skip = display_mode_esc_sequence_len_counted(string,
> -+ orig + len - string)) != 0)
> ++ if (skip_ansi) {
> ++ while ((skip = display_mode_esc_sequence_len(string, end)))
> + string += skip;
> +
> -+ if (string >= orig + len)
> ++ if (string >= end)
> + break;
> + }
>
> 4: 471c3d6888 < -: ---------- utf8: respect boundaries in utf8_strnwidth()
> -: ---------- > 4: a6cf4c99dc utf8: respect boundaries in utf8_strnwidth()
>
> ---
> base-commit: 7fb6055e678f800320d8ab8cd8f201d9493d1ba0
> change-id: 20261116-master-69b171612823
^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v4 0/5] utf8: bounds-related fixes and a doc fix
2026-11-16 18:56 [PATCH 0/2] utf8: two minor fixes Dirk Gouders
` (3 preceding siblings ...)
2026-11-20 14:44 ` [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
@ 2026-11-20 15:11 ` Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 1/5] utf8: introduce bounds for display_mode_esc_sequence_len() Dirk Gouders
` (4 more replies)
4 siblings, 5 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-20 15:11 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Dirk Gouders, Andy Koppe
A patch here on the list made me look at
display_mode_esc_sequence_len() in utf8.c and I found two details in
the file that are perhaps worth a fix:
1) utf8_strnwidth() disregards the given len argument when fed
with proper string and len.
2) The comment for utf8_strnwidth() is incorrect: it states the
function returns strlen() for cases when it actually returns its
len argument.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
Changes in v4:
- Resend un-squashed v3
Changes in v3:
- 1/5: * fix sorting order in t/meson.build
* fix risk of O(n²) degradation:
drop the wrapper function and introduce an end pointer to
display_mode_esc_sequence_len()
(I was so used to len arguments for string handling functions
that I only now realized the beauty of optionally optional
end pointers for string handling. I know: beauty is in the
eye of the beholder ;-))
* try to fix an overly long line
- 2/5: also fix a further error concerning "null-terminated"
- 3/5: * drop excess SP in if-statement
- 5/5: this actually is a POC, not fixing a bug:
users of display_mode_esc_sequence_len() often need the length
of all leading SGR sequences and here is a function that does that.
I mentioned such a function in the discussion for v1.
- Link to v2: https://patch.msgid.link/20261118-master-v2-0-3f2aaec15d31@gouders.net
Changes in v2:
- Add a patch by Junio that introduces a bounded function display_mode_esc_sequence_len_counted()
I added some tests for this new function.
- 2/2 became 2/4
- 3/4 is the modified 1/2 and uses the new
display_mode_esc_sequence_len_counted() so that the while loop stays
within the given bounds.
The commit message was adjusted.
One of the tests was wrong: "\033" has width 0, so the sum must be 3 instead of 4.
- 4/4 is an addition to fix a similar problem with utf8_width().
That function is used without its optional bounds argument and
can violate boundaries as well. Test for such cases are added.
- Link to v1: https://patch.msgid.link/20261116-master-v1-0-dc2a50ce0aa0@gouders.net
---
Dirk Gouders (4):
utf8: fix comment for utf8_strnwidth()
utf8: respect string boundaries after skipping ANSI escape sequences
utf8: respect boundaries in utf8_strnwidth()
utf8: introduce display_mode_esc_sequence_len_all()
Junio C Hamano (1):
utf8: introduce bounds for display_mode_esc_sequence_len()
Makefile | 1 +
pretty.c | 2 +-
t/meson.build | 1 +
t/unit-tests/u-utf8-ansi.c | 85 +++++++++++++++++++++++++++++++++++++++++++++
t/unit-tests/u-utf8-width.c | 20 +++++++++++
utf8.c | 78 +++++++++++++++++++++++++++++------------
utf8.h | 3 +-
7 files changed, 165 insertions(+), 25 deletions(-)
Range-diff versus v3:
1: 122df176a3 = 1: ce2736da52 utf8: introduce bounds for display_mode_esc_sequence_len()
2: 6cb2da91ae = 2: ec70d5e553 utf8: fix comment for utf8_strnwidth()
3: d62360d690 = 3: 0e09e16822 utf8: respect string boundaries after skipping ANSI escape sequences
-: ---------- > 4: f49010e6c5 utf8: respect boundaries in utf8_strnwidth()
4: caca770f8b ! 5: eb428dfc47 utf8: respect boundaries in utf8_strnwidth()
@@ Metadata
Author: Dirk Gouders <dirk@gouders.net>
## Commit message ##
- utf8: respect boundaries in utf8_strnwidth()
+ utf8: introduce display_mode_esc_sequence_len_all()
- utf8_strnwidth() can be called with boundaries that effectively cause
- incomplete multibyte sequences and we should return "len" in such
- cases.
+ Most users of display_mode_esc_sequence_len() need to know the length
+ of *all* leading ANSI display mode escape sequences and use
+ display_mode_esc_sequence_len() in a loop.
- But, utf8_width is used without its optional boundary argument and
- will thus work beyond the given boundary.
+ Introduce display_mode_esc_sequence_len_all() for convenience and
+ to reduce the risk of bugs.
- Fix this by using the boundary argument for utf8_width.
- Add some tests that cover such situations.
+ Add tests for this function as well.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
@@ t/unit-tests/u-utf8-ansi.c
* Test display_mode_esc_sequence_len() for NUL terminated strings.
*/
- ## t/unit-tests/u-utf8-width.c ##
-@@
- #include "utf8.h"
- #include "strbuf.h"
-
-+/*
-+ * Test utf8_strnwidth for cjk characters invalidated by len argument.
-+ */
-+void test_utf8_width__strnwidth_cjk_chopped(void)
-+{
-+ cl_assert_equal_i(1, utf8_strnwidth("三bytes long, 二columns wide", 1, 1));
-+ cl_assert_equal_i(2, utf8_strnwidth("三bytes long, 二columns wide", 3, 1));
-+}
-+
- /*
- * Test utf8_strnwidth for strings with SGR sequences and corner case length
- * arguments.
-
## utf8.c ##
@@ utf8.c: size_t display_mode_esc_sequence_len(const char *start, const char *end)
return p - start;
@@ utf8.c: size_t display_mode_esc_sequence_len(const char *start, const char *end)
static int bisearch(ucs_char_t ucs, const struct interval *table, int max)
{
@@ utf8.c: int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
-
- while (string && string < end) {
- int glyph_width;
-- size_t skip;
-+ size_t remainder;
+ size_t remainder;
if (skip_ansi) {
- while ((skip = display_mode_esc_sequence_len(string, end)))
@@ utf8.c: int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
if (string >= end)
break;
- }
-
-- glyph_width = utf8_width(&string, NULL);
-+ remainder = end - string;
-+
-+ glyph_width = utf8_width(&string, &remainder);
- if (glyph_width > 0)
- width += glyph_width;
- }
@@ utf8.c: void strbuf_add_wrapped_text(struct strbuf *buf,
for (;;) {
---
base-commit: 7fb6055e678f800320d8ab8cd8f201d9493d1ba0
change-id: 20261116-master-69b171612823
^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v4 1/5] utf8: introduce bounds for display_mode_esc_sequence_len()
2026-11-20 15:11 ` [PATCH v4 0/5] " Dirk Gouders
@ 2026-11-20 15:11 ` Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 2/5] utf8: fix comment for utf8_strnwidth() Dirk Gouders
` (3 subsequent siblings)
4 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-20 15:11 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Dirk Gouders, Andy Koppe
From: Junio C Hamano <gitster@pobox.com>
display_mode_esc_sequence_len() scans for an ANSI display mode
escape sequence (e.g. "ESC [ 1 ; 31 m") at the start of a string,
relying on NUL termination to stop. Some callers, however, work on
a counted buffer: utf8_strnwidth() takes <string, len> and must not
scan past it, while strbuf_utf8_replace() and
strbuf_add_wrapped_text() know (or can cheaply learn) the end.
Introduce an optional end pointer to display_mode_esc_sequence_len()
which then never reads *end or beyond when end is not NULL.
If end is NULL, display_mode_esc_sequence_len() assumes a
NUL terminated string as before.
Andy Koppe pointed out the risk of performance degrading to O(n²)
when functions process strings and have to check each position for an
escape sequence [1]. This change doesn't carry this risk.
Further, add thorough tests for both end pointer cases.
[1] https://lore.kernel.org/git/CAHWeT-aWx1VwUjM+hZ--jjSAbcMenz5Mw11y9ouFavGndUMrSA@mail.gmail.com/
Cc: Andy Koppe <andy.koppe@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
Makefile | 1 +
pretty.c | 2 +-
t/meson.build | 1 +
t/unit-tests/u-utf8-ansi.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++
utf8.c | 34 ++++++++++++++++----------
utf8.h | 2 +-
6 files changed, 85 insertions(+), 14 deletions(-)
diff --git a/Makefile b/Makefile
index 4c84eea6e2..db8f7f5c60 100644
--- a/Makefile
+++ b/Makefile
@@ -1563,6 +1563,7 @@ CLAR_TEST_SUITES += u-string-list
CLAR_TEST_SUITES += u-strvec
CLAR_TEST_SUITES += u-trailer
CLAR_TEST_SUITES += u-urlmatch-normalization
+CLAR_TEST_SUITES += u-utf8-ansi
CLAR_TEST_SUITES += u-utf8-width
CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES))
diff --git a/pretty.c b/pretty.c
index d8a9f370f6..58dfeb61f4 100644
--- a/pretty.c
+++ b/pretty.c
@@ -1848,7 +1848,7 @@ static size_t format_and_pad_commit(struct strbuf *sb, /* in UTF-8 */
while (p > sb->buf && ch - p < 10 && *p != '\033')
p--;
if (*p != '\033' ||
- ch + 1 - p != display_mode_esc_sequence_len(p))
+ ch + 1 - p != display_mode_esc_sequence_len(p, ch + 1))
break;
/*
* got a good ansi sequence, put it back to
diff --git a/t/meson.build b/t/meson.build
index 1cf1d95f70..ae7479a98e 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -26,6 +26,7 @@ clar_test_suites = [
'unit-tests/u-strvec.c',
'unit-tests/u-trailer.c',
'unit-tests/u-urlmatch-normalization.c',
+ 'unit-tests/u-utf8-ansi.c',
'unit-tests/u-utf8-width.c',
]
diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c
new file mode 100644
index 0000000000..cb75c8612b
--- /dev/null
+++ b/t/unit-tests/u-utf8-ansi.c
@@ -0,0 +1,59 @@
+#include "unit-test.h"
+#include "utf8.h"
+#include "strbuf.h"
+
+/*
+ * Test display_mode_esc_sequence_len() for NUL terminated strings.
+ */
+void test_utf8_ansi__sgr_len_null(void)
+{
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("", NULL));
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("\033", NULL));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("\033[", NULL));
+ cl_assert_equal_i(3, display_mode_esc_sequence_len("\033[m", NULL));
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("\033[3", NULL));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("\033[31", NULL));
+ cl_assert_equal_i(5, display_mode_esc_sequence_len("\033[31m", NULL));
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("\033[31;", NULL));
+ cl_assert_equal_i(6, display_mode_esc_sequence_len("\033[31;m", NULL));
+ cl_assert_equal_i(7, display_mode_esc_sequence_len("\033[31;0m", NULL));
+}
+
+/*
+ * Test display_mode_esc_sequence_len() with end pointers.
+ */
+void test_utf8_ansi__sgr_len_end_ptr(void)
+{
+ char *sgr_1 = (char *)"\033[m";
+ char *sgr_2 = (char *)"\033[31m";
+ char *sgr_3 = (char *)"\033[31;m";
+ char *sgr_4 = (char *)"\033[31;0m";
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len("", ""));
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_1, sgr_1 + 1));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_1, sgr_1 + 2));
+ cl_assert_equal_i(3, display_mode_esc_sequence_len(sgr_1, sgr_1 + 3));
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_2, sgr_2 + 1));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_2, sgr_2 + 2));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_2, sgr_2 + 3));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_2, sgr_2 + 4));
+ cl_assert_equal_i(5, display_mode_esc_sequence_len(sgr_2, sgr_2 + 5));
+
+ cl_assert_equal_i(6, display_mode_esc_sequence_len(sgr_3, sgr_3 + sizeof(sgr_3) - 1));
+ cl_assert_equal_i(7, display_mode_esc_sequence_len(sgr_4, sgr_4 + sizeof(sgr_4) - 1));
+}
+
+/*
+ * Test display_mode_esc_sequence_len() for swapped start and end pointers.
+ */
+void test_utf8_ansi__sgr_len_start_end_swap(void)
+{
+ char *sgr_1 = (char *)"\033[m";
+
+ cl_assert_equal_i(0, display_mode_esc_sequence_len(sgr_1 + 3, sgr_1));
+}
diff --git a/utf8.c b/utf8.c
index 96460cc414..48aa189624 100644
--- a/utf8.c
+++ b/utf8.c
@@ -16,18 +16,27 @@ struct interval {
ucs_char_t last;
};
-size_t display_mode_esc_sequence_len(const char *s)
+/*
+ * Return the length of the ANSI display mode escape sequence (e.g.,
+ * "\033[1;31m") that appears at the beginning of the string starting at start,
+ * or 0 if the string does not begin with a complete such sequence.
+ * If end is NULL, assume a NUL terminated string, else never look at *end or
+ * beyond.
+ */
+size_t display_mode_esc_sequence_len(const char *start, const char *end)
{
- const char *p = s;
- if (*p++ != '\033')
+ const char *p = start;
+
+ if ((end && end <= p) || *p++ != '\033')
return 0;
- if (*p++ != '[')
+ if ((end && end <= p) || *p++ != '[')
return 0;
- while (isdigit(*p) || *p == ';')
+ while (((end && p < end) || (!end && *p)) && (isdigit(*p) || *p == ';'))
p++;
- if (*p++ != 'm')
+ if ((end && end <= p) || (!end && !*p) || *p++ != 'm')
return 0;
- return p - s;
+
+ return p - start;
}
/* auxiliary function for binary search in interval table */
@@ -210,15 +219,15 @@ int utf8_width(const char **start, size_t *remainder_p)
*/
int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
{
- const char *orig = string;
+ const char *end = string + len;
size_t width = 0;
- while (string && string < orig + len) {
+ while (string && string < end) {
int glyph_width;
size_t skip;
while (skip_ansi &&
- (skip = display_mode_esc_sequence_len(string)) != 0)
+ (skip = display_mode_esc_sequence_len(string, end)))
string += skip;
glyph_width = utf8_width(&string, NULL);
@@ -279,6 +288,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
{
int indent, w, assume_utf8 = 1;
const char *bol, *space, *start = text;
+ const char *end = text + strlen(text);
size_t orig_len = buf->len;
if (width <= 0) {
@@ -299,7 +309,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
char c;
size_t skip;
- while ((skip = display_mode_esc_sequence_len(text)))
+ while ((skip = display_mode_esc_sequence_len(text, end)))
text += skip;
c = *text;
@@ -378,7 +388,7 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
int glyph_width;
size_t n;
- while ((n = display_mode_esc_sequence_len(src))) {
+ while ((n = display_mode_esc_sequence_len(src, end))) {
strbuf_add(&dst, src, n);
src += n;
}
diff --git a/utf8.h b/utf8.h
index cf8ecb0f21..7eb7bf042a 100644
--- a/utf8.h
+++ b/utf8.h
@@ -5,7 +5,7 @@ struct strbuf;
typedef unsigned int ucs_char_t; /* assuming 32bit int */
-size_t display_mode_esc_sequence_len(const char *s);
+size_t display_mode_esc_sequence_len(const char *start, const char *end);
int utf8_width(const char **start, size_t *remainder_p);
int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
int utf8_strwidth(const char *string);
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v4 2/5] utf8: fix comment for utf8_strnwidth()
2026-11-20 15:11 ` [PATCH v4 0/5] " Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 1/5] utf8: introduce bounds for display_mode_esc_sequence_len() Dirk Gouders
@ 2026-11-20 15:11 ` Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 3/5] utf8: respect string boundaries after skipping ANSI escape sequences Dirk Gouders
` (2 subsequent siblings)
4 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-20 15:11 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Dirk Gouders, Andy Koppe
The comment for utf8_strnwidth() states it works on a NUL terminated
string whereas the string is bounded by len.
It also states it returns strlen() for cases when the given string
doesn't look like valid utf8, but it returns its argument len in such
cases.
Fix the comment accordingly.
Helped-by: Andy Koppe <andy.koppe@gmail.com>
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
utf8.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/utf8.c b/utf8.c
index 48aa189624..35b2bb8290 100644
--- a/utf8.c
+++ b/utf8.c
@@ -213,8 +213,8 @@ int utf8_width(const char **start, size_t *remainder_p)
}
/*
- * Returns the total number of columns required by a null-terminated
- * string, assuming that the string is utf8. Returns strlen() instead
+ * Returns the total number of columns required by a string terminated by len,
+ * assuming that the string is utf8. Returns len instead
* if the string does not look like a valid utf8 string.
*/
int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v4 3/5] utf8: respect string boundaries after skipping ANSI escape sequences
2026-11-20 15:11 ` [PATCH v4 0/5] " Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 1/5] utf8: introduce bounds for display_mode_esc_sequence_len() Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 2/5] utf8: fix comment for utf8_strnwidth() Dirk Gouders
@ 2026-11-20 15:11 ` Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 4/5] utf8: respect boundaries in utf8_strnwidth() Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 5/5] utf8: introduce display_mode_esc_sequence_len_all() Dirk Gouders
4 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-20 15:11 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Dirk Gouders
utf8_strnwidth() can skip ANSI escape sequences and it is possible
that such sequences end the string within the given boundary.
Check for such situations before proceeding and calling utf8_width().
Add tests that test such corner cases.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
t/unit-tests/u-utf8-width.c | 11 +++++++++++
utf8.c | 10 +++++++---
2 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/t/unit-tests/u-utf8-width.c b/t/unit-tests/u-utf8-width.c
index 86e09c3574..5e2b636db9 100644
--- a/t/unit-tests/u-utf8-width.c
+++ b/t/unit-tests/u-utf8-width.c
@@ -2,6 +2,17 @@
#include "utf8.h"
#include "strbuf.h"
+/*
+ * Test utf8_strnwidth for strings with SGR sequences and corner case length
+ * arguments.
+ */
+void test_utf8_width__strnwidth_ansi(void)
+{
+ cl_assert_equal_i(3, utf8_strnwidth("\033[31mhello", 4, 1));
+ cl_assert_equal_i(0, utf8_strnwidth("\033[31mhello", 5, 1));
+ cl_assert_equal_i(1, utf8_strnwidth("\033[31mhello", 6, 1));
+}
+
/*
* Test utf8_strnwidth with various Chinese strings
* Chinese characters typically have a width of 2 columns when displayed
diff --git a/utf8.c b/utf8.c
index 35b2bb8290..493f723e3b 100644
--- a/utf8.c
+++ b/utf8.c
@@ -226,9 +226,13 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
int glyph_width;
size_t skip;
- while (skip_ansi &&
- (skip = display_mode_esc_sequence_len(string, end)))
- string += skip;
+ if (skip_ansi) {
+ while ((skip = display_mode_esc_sequence_len(string, end)))
+ string += skip;
+
+ if (string >= end)
+ break;
+ }
glyph_width = utf8_width(&string, NULL);
if (glyph_width > 0)
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v4 4/5] utf8: respect boundaries in utf8_strnwidth()
2026-11-20 15:11 ` [PATCH v4 0/5] " Dirk Gouders
` (2 preceding siblings ...)
2026-11-20 15:11 ` [PATCH v4 3/5] utf8: respect string boundaries after skipping ANSI escape sequences Dirk Gouders
@ 2026-11-20 15:11 ` Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 5/5] utf8: introduce display_mode_esc_sequence_len_all() Dirk Gouders
4 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-20 15:11 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Dirk Gouders
utf8_strnwidth() can be called with boundaries that effectively cause
incomplete multibyte sequences and we should return "len" in such
cases.
But, utf8_width is used without its optional boundary argument and
will thus work beyond the given boundary.
Fix this by using the boundary argument for utf8_width.
Add some tests that cover such situations.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
t/unit-tests/u-utf8-width.c | 9 +++++++++
utf8.c | 6 ++++--
2 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/t/unit-tests/u-utf8-width.c b/t/unit-tests/u-utf8-width.c
index 5e2b636db9..c4df4dd6a6 100644
--- a/t/unit-tests/u-utf8-width.c
+++ b/t/unit-tests/u-utf8-width.c
@@ -2,6 +2,15 @@
#include "utf8.h"
#include "strbuf.h"
+/*
+ * Test utf8_strnwidth for cjk characters invalidated by len argument.
+ */
+void test_utf8_width__strnwidth_cjk_chopped(void)
+{
+ cl_assert_equal_i(1, utf8_strnwidth("三bytes long, 二columns wide", 1, 1));
+ cl_assert_equal_i(2, utf8_strnwidth("三bytes long, 二columns wide", 3, 1));
+}
+
/*
* Test utf8_strnwidth for strings with SGR sequences and corner case length
* arguments.
diff --git a/utf8.c b/utf8.c
index 493f723e3b..291f34db7e 100644
--- a/utf8.c
+++ b/utf8.c
@@ -224,7 +224,7 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
while (string && string < end) {
int glyph_width;
- size_t skip;
+ size_t remainder;
if (skip_ansi) {
while ((skip = display_mode_esc_sequence_len(string, end)))
@@ -234,7 +234,9 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
break;
}
- glyph_width = utf8_width(&string, NULL);
+ remainder = end - string;
+
+ glyph_width = utf8_width(&string, &remainder);
if (glyph_width > 0)
width += glyph_width;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v4 5/5] utf8: introduce display_mode_esc_sequence_len_all()
2026-11-20 15:11 ` [PATCH v4 0/5] " Dirk Gouders
` (3 preceding siblings ...)
2026-11-20 15:11 ` [PATCH v4 4/5] utf8: respect boundaries in utf8_strnwidth() Dirk Gouders
@ 2026-11-20 15:11 ` Dirk Gouders
4 siblings, 0 replies; 41+ messages in thread
From: Dirk Gouders @ 2026-11-20 15:11 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Dirk Gouders
Most users of display_mode_esc_sequence_len() need to know the length
of *all* leading ANSI display mode escape sequences and use
display_mode_esc_sequence_len() in a loop.
Introduce display_mode_esc_sequence_len_all() for convenience and
to reduce the risk of bugs.
Add tests for this function as well.
Signed-off-by: Dirk Gouders <dirk@gouders.net>
---
t/unit-tests/u-utf8-ansi.c | 26 ++++++++++++++++++++++++++
utf8.c | 34 +++++++++++++++++++++++++---------
utf8.h | 1 +
3 files changed, 52 insertions(+), 9 deletions(-)
diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c
index cb75c8612b..909643d07d 100644
--- a/t/unit-tests/u-utf8-ansi.c
+++ b/t/unit-tests/u-utf8-ansi.c
@@ -2,6 +2,32 @@
#include "utf8.h"
#include "strbuf.h"
+/*
+ * Test the greedy function display_mode_esc_sequence_len_all().
+ */
+void test_utf8_ansi__sgr_len_greedy(void)
+{
+ char *s1 = (char *)"no ESC sequence";
+ char *s2 = (char *)"\033[m";
+ char *s3 = (char *)"\033[0;31m";
+ char *s4 = (char *)"\033[0m\033[31m";
+
+ /* no sequences */
+ cl_assert_equal_i(0, display_mode_esc_sequence_len_all("", ""));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len_all(s1, s1 + 5));
+
+ /* single sequences */
+ cl_assert_equal_i(3, display_mode_esc_sequence_len_all(s2, s2 + 3));
+ cl_assert_equal_i(7, display_mode_esc_sequence_len_all(s3, s3 + 7));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len_all(s3, s3 + 3));
+
+ /* multiple sequences */
+ cl_assert_equal_i(4, display_mode_esc_sequence_len_all(s4, s4 + 4));
+ cl_assert_equal_i(9, display_mode_esc_sequence_len_all(s4, s4 + 9));
+ cl_assert_equal_i(0, display_mode_esc_sequence_len_all(s4, s4 + 3));
+ cl_assert_equal_i(4, display_mode_esc_sequence_len_all(s4, s4 + 8));
+}
+
/*
* Test display_mode_esc_sequence_len() for NUL terminated strings.
*/
diff --git a/utf8.c b/utf8.c
index 291f34db7e..9478985cbb 100644
--- a/utf8.c
+++ b/utf8.c
@@ -39,6 +39,26 @@ size_t display_mode_esc_sequence_len(const char *start, const char *end)
return p - start;
}
+/*
+ * Count the lenght of all leading ANSI display mode escape sequences that
+ * are a prefix of the string at start up to end.
+ */
+size_t display_mode_esc_sequence_len_all(const char *start, const char *end)
+{
+ size_t seq_len = 0;
+
+ while (start + seq_len < end) {
+ size_t count = display_mode_esc_sequence_len(start + seq_len, end);
+
+ if (!count)
+ break;
+
+ seq_len += count;
+ }
+
+ return seq_len;
+}
+
/* auxiliary function for binary search in interval table */
static int bisearch(ucs_char_t ucs, const struct interval *table, int max)
{
@@ -227,8 +247,7 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
size_t remainder;
if (skip_ansi) {
- while ((skip = display_mode_esc_sequence_len(string, end)))
- string += skip;
+ string += display_mode_esc_sequence_len_all(string, end);
if (string >= end)
break;
@@ -313,10 +332,8 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
for (;;) {
char c;
- size_t skip;
- while ((skip = display_mode_esc_sequence_len(text, end)))
- text += skip;
+ text += display_mode_esc_sequence_len_all(text, end);
c = *text;
if (!c || isspace(c)) {
@@ -394,10 +411,9 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
int glyph_width;
size_t n;
- while ((n = display_mode_esc_sequence_len(src, end))) {
- strbuf_add(&dst, src, n);
- src += n;
- }
+ n = display_mode_esc_sequence_len_all(src, end);
+ strbuf_add(&dst, src, n);
+ src += n;
if (src >= end)
break;
diff --git a/utf8.h b/utf8.h
index 7eb7bf042a..fdde5af782 100644
--- a/utf8.h
+++ b/utf8.h
@@ -6,6 +6,7 @@ struct strbuf;
typedef unsigned int ucs_char_t; /* assuming 32bit int */
size_t display_mode_esc_sequence_len(const char *start, const char *end);
+size_t display_mode_esc_sequence_len_all(const char *start, const char *end);
int utf8_width(const char **start, size_t *remainder_p);
int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
int utf8_strwidth(const char *string);
--
2.55.0
^ permalink raw reply related [flat|nested] 41+ messages in thread
* Re: [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-20 10:16 ` Andy Koppe
@ 2026-11-20 16:36 ` Junio C Hamano
2026-11-20 18:03 ` Andy Koppe
0 siblings, 1 reply; 41+ messages in thread
From: Junio C Hamano @ 2026-11-20 16:36 UTC (permalink / raw)
To: Andy Koppe; +Cc: Dirk Gouders, git
Andy Koppe <andy.koppe@gmail.com> writes:
> Callers could already sidestep the length check by passing SIZE_MAX, and
> the number of consumed bytes is returned, so I don't think that's needed.
I like the simplicity of the approach. Thanks.
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted()
2026-11-20 16:36 ` Junio C Hamano
@ 2026-11-20 18:03 ` Andy Koppe
0 siblings, 0 replies; 41+ messages in thread
From: Andy Koppe @ 2026-11-20 18:03 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Dirk Gouders, git
On Fri, 20 Nov 2026 at 16:36, Junio C Hamano <gitster@pobox.com> wrote:
>
> Andy Koppe <andy.koppe@gmail.com> writes:
>
> > Callers could already sidestep the length check by passing SIZE_MAX, and
> > the number of consumed bytes is returned, so I don't think that's needed.
>
> I like the simplicity of the approach. Thanks.
Except, I was wrong, sorry. The 'end = s + len' would be undefined
behaviour with SIZE_MAX. Callers do have to pass a valid length.
But it didn't sound like there are use cases that don't know the
remaining length anyway and can't tolerate calling strlen().
^ permalink raw reply [flat|nested] 41+ messages in thread
end of thread, other threads:[~2026-11-20 18:03 UTC | newest]
Thread overview: 41+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 18:56 [PATCH 0/2] utf8: two minor fixes Dirk Gouders
2026-11-16 18:57 ` [PATCH 1/2] utf8: respect bounds in utf8_strnwidth() Dirk Gouders
2026-11-17 0:02 ` Junio C Hamano
2026-11-17 15:15 ` Dirk Gouders
2026-11-17 16:18 ` Junio C Hamano
2026-11-17 17:57 ` Dirk Gouders
2026-11-17 18:11 ` Junio C Hamano
2026-11-16 18:57 ` [PATCH 2/2] utf8: fix comment for utf8_strnwidth() Dirk Gouders
2026-11-18 21:08 ` [PATCH v2 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
2026-11-18 21:08 ` [PATCH v2 1/4] utf8: add display_mode_esc_sequence_len_counted() Dirk Gouders
2026-11-19 5:54 ` Andy Koppe
2026-11-19 7:43 ` Dirk Gouders
2026-11-19 20:58 ` Andy Koppe
2026-11-19 21:04 ` Andy Koppe
2026-11-19 21:33 ` Dirk Gouders
2026-11-19 22:32 ` Dirk Gouders
2026-11-19 22:59 ` Dirk Gouders
2026-11-20 10:16 ` Andy Koppe
2026-11-20 16:36 ` Junio C Hamano
2026-11-20 18:03 ` Andy Koppe
2026-11-19 19:59 ` Junio C Hamano
2026-11-19 20:28 ` Dirk Gouders
2026-11-18 21:08 ` [PATCH v2 2/4] utf8: fix comment for utf8_strnwidth() Dirk Gouders
2026-11-18 22:40 ` Andy Koppe
2026-11-19 5:58 ` Dirk Gouders
2026-11-18 21:08 ` [PATCH v2 3/4] utf8: respect string boundaries after skipping ANSI escape sequences Dirk Gouders
2026-11-18 21:49 ` Junio C Hamano
2026-11-18 21:08 ` [PATCH v2 4/4] utf8: respect boundaries in utf8_strnwidth() Dirk Gouders
2026-11-18 21:50 ` Junio C Hamano
2026-11-20 14:44 ` [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
2026-11-20 14:44 ` [PATCH v3 1/4] utf8: introduce bounds for display_mode_esc_sequence_len() Dirk Gouders
2026-11-20 14:44 ` [PATCH v3 2/4] utf8: fix comment for utf8_strnwidth() Dirk Gouders
2026-11-20 14:44 ` [PATCH v3 3/4] utf8: respect string boundaries after skipping ANSI escape sequences Dirk Gouders
2026-11-20 14:44 ` [PATCH v3 4/4] utf8: respect boundaries in utf8_strnwidth() Dirk Gouders
2026-11-20 15:09 ` [PATCH v3 0/4] utf8: bounds-related fixes and a doc fix Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 0/5] " Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 1/5] utf8: introduce bounds for display_mode_esc_sequence_len() Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 2/5] utf8: fix comment for utf8_strnwidth() Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 3/5] utf8: respect string boundaries after skipping ANSI escape sequences Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 4/5] utf8: respect boundaries in utf8_strnwidth() Dirk Gouders
2026-11-20 15:11 ` [PATCH v4 5/5] utf8: introduce display_mode_esc_sequence_len_all() Dirk Gouders
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox