* [PATCH] utf8: make display_mode_esc_sequence_len() take a length
@ 2026-11-17 16:50 Junio C Hamano
2026-11-17 20:06 ` [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() Junio C Hamano
0 siblings, 1 reply; 15+ messages in thread
From: Junio C Hamano @ 2026-11-17 16:50 UTC (permalink / raw)
To: git; +Cc: Dirk Gouders
display_mode_esc_sequence_len() scans for an ANSI display mode
escape sequence (e.g. "ESC [ 1 ; 31 m"), relying on NUL termination
to stop. Some callers, however, use a counted <ptr, len> buffer or
need to stop at a specific position.
Teach it to take a buffer length and never read at or beyond
ptr[len], returning the length of a complete sequence at the start
or 0 otherwise.
Update the callers to pass the remaining byte count. For
strbuf_add_wrapped_text(), which takes a NUL-terminated string,
compute the end once upfront instead of calling strlen() each time.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
* Copy of https://lore.kernel.org/git/xmqqse0zfn2n.fsf@gitster.g/
with a proposed commit log message.
pretty.c | 3 ++-
utf8.c | 27 ++++++++++++++++++---------
utf8.h | 2 +-
3 files changed, 21 insertions(+), 11 deletions(-)
diff --git a/pretty.c b/pretty.c
index 0bc8ad8a9a..a20fd61909 100644
--- a/pretty.c
+++ b/pretty.c
@@ -1834,7 +1834,8 @@ static size_t format_and_pad_commit(struct strbuf *sb, /* in UTF-8 */
while (p > sb->buf && ch - p < 10 && *p != '\033')
p--;
if (*p != '\033' ||
- ch + 1 - p != display_mode_esc_sequence_len(p))
+ ch + 1 - p != display_mode_esc_sequence_len(p,
+ ch + 1 - p))
break;
/*
* got a good ansi sequence, put it back to
diff --git a/utf8.c b/utf8.c
index 96460cc414..544c5329f3 100644
--- a/utf8.c
+++ b/utf8.c
@@ -16,16 +16,23 @@ struct interval {
ucs_char_t last;
};
-size_t display_mode_esc_sequence_len(const char *s)
+/*
+ * Return the length of the ANSI display mode escape sequence (e.g.,
+ * "\033[1;31m") that appears at the beginning of the "len" bytes
+ * of string "s", or 0 if the bytes do not begin with a complete
+ * such sequence. Never looks at s[len] or beyond.
+ */
+size_t display_mode_esc_sequence_len(const char *s, size_t len)
{
- const char *p = s;
- if (*p++ != '\033')
+ const char *p = s, *end = s + len;
+
+ if (end <= p || *p++ != '\033')
return 0;
- if (*p++ != '[')
+ if (end <= p || *p++ != '[')
return 0;
- while (isdigit(*p) || *p == ';')
+ while (p < end && (isdigit(*p) || *p == ';'))
p++;
- if (*p++ != 'm')
+ if (end <= p || *p++ != 'm')
return 0;
return p - s;
}
@@ -218,7 +225,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
size_t skip;
while (skip_ansi &&
- (skip = display_mode_esc_sequence_len(string)) != 0)
+ (skip = display_mode_esc_sequence_len(string,
+ orig + len - string)) != 0)
string += skip;
glyph_width = utf8_width(&string, NULL);
@@ -279,6 +287,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
{
int indent, w, assume_utf8 = 1;
const char *bol, *space, *start = text;
+ const char *end = text + strlen(text);
size_t orig_len = buf->len;
if (width <= 0) {
@@ -299,7 +308,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
char c;
size_t skip;
- while ((skip = display_mode_esc_sequence_len(text)))
+ while ((skip = display_mode_esc_sequence_len(text, end - text)))
text += skip;
c = *text;
@@ -378,7 +387,7 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
int glyph_width;
size_t n;
- while ((n = display_mode_esc_sequence_len(src))) {
+ while ((n = display_mode_esc_sequence_len(src, end - src))) {
strbuf_add(&dst, src, n);
src += n;
}
diff --git a/utf8.h b/utf8.h
index cf8ecb0f21..fb2227a9e2 100644
--- a/utf8.h
+++ b/utf8.h
@@ -5,7 +5,7 @@ struct strbuf;
typedef unsigned int ucs_char_t; /* assuming 32bit int */
-size_t display_mode_esc_sequence_len(const char *s);
+size_t display_mode_esc_sequence_len(const char *s, size_t len);
int utf8_width(const char **start, size_t *remainder_p);
int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
int utf8_strwidth(const char *string);
--
2.56.1-740-gf38658f3d0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-17 16:50 [PATCH] utf8: make display_mode_esc_sequence_len() take a length Junio C Hamano
@ 2026-11-17 20:06 ` Junio C Hamano
2026-11-17 21:26 ` Dirk Gouders
2026-11-17 23:08 ` Andy Koppe
0 siblings, 2 replies; 15+ messages in thread
From: Junio C Hamano @ 2026-11-17 20:06 UTC (permalink / raw)
To: git; +Cc: Dirk Gouders
display_mode_esc_sequence_len() scans for an ANSI display mode
escape sequence (e.g. "ESC [ 1 ; 31 m") at the start of a string,
relying on NUL termination to stop. Some callers, however, work on
a counted buffer: utf8_strnwidth() takes <string, len> and must not
scan past it, while strbuf_utf8_replace() and
strbuf_add_wrapped_text() know (or can cheaply learn) the end.
Add display_mode_esc_sequence_len_counted() taking <ptr, len>,
which never reads at or beyond ptr[len], and make
display_mode_esc_sequence_len() a thin wrapper for NUL-terminated
strings.
Use the counted variant in utf8_strnwidth() for correctness, and in
strbuf_utf8_replace() and strbuf_add_wrapped_text() (computing the
end once upfront in the latter) to avoid measuring the remaining
text in each iteration.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
* As there are new patches that want to add new callers to this
function, let's keep the original with the same API while adding
a new helper function that allows us to limit the scan.
utf8.c | 35 ++++++++++++++++++++++++++---------
utf8.h | 1 +
2 files changed, 27 insertions(+), 9 deletions(-)
diff --git a/utf8.c b/utf8.c
index 96460cc414..d0546ab31f 100644
--- a/utf8.c
+++ b/utf8.c
@@ -16,20 +16,33 @@ struct interval {
ucs_char_t last;
};
-size_t display_mode_esc_sequence_len(const char *s)
+/*
+ * Return the length of the ANSI display mode escape sequence (e.g.,
+ * "\033[1;31m") that appears at the beginning of the "len" bytes
+ * of string "s", or 0 if the bytes do not begin with a complete
+ * such sequence. Never looks at s[len] or beyond.
+ */
+size_t display_mode_esc_sequence_len_counted(const char *s, size_t len)
{
- const char *p = s;
- if (*p++ != '\033')
+ const char *p = s, *end = s + len;
+
+ if (end <= p || *p++ != '\033')
return 0;
- if (*p++ != '[')
+ if (end <= p || *p++ != '[')
return 0;
- while (isdigit(*p) || *p == ';')
+ while (p < end && (isdigit(*p) || *p == ';'))
p++;
- if (*p++ != 'm')
+ if (end <= p || *p++ != 'm')
return 0;
return p - s;
}
+/* Same as above, but for a NUL-terminated string "s". */
+size_t display_mode_esc_sequence_len(const char *s)
+{
+ return display_mode_esc_sequence_len_counted(s, strlen(s));
+}
+
/* auxiliary function for binary search in interval table */
static int bisearch(ucs_char_t ucs, const struct interval *table, int max)
{
@@ -218,7 +231,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
size_t skip;
while (skip_ansi &&
- (skip = display_mode_esc_sequence_len(string)) != 0)
+ (skip = display_mode_esc_sequence_len_counted(string,
+ orig + len - string)) != 0)
string += skip;
glyph_width = utf8_width(&string, NULL);
@@ -279,6 +293,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
{
int indent, w, assume_utf8 = 1;
const char *bol, *space, *start = text;
+ const char *end = text + strlen(text);
size_t orig_len = buf->len;
if (width <= 0) {
@@ -299,7 +314,8 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
char c;
size_t skip;
- while ((skip = display_mode_esc_sequence_len(text)))
+ while ((skip = display_mode_esc_sequence_len_counted(text,
+ end - text)))
text += skip;
c = *text;
@@ -378,7 +394,8 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
int glyph_width;
size_t n;
- while ((n = display_mode_esc_sequence_len(src))) {
+ while ((n = display_mode_esc_sequence_len_counted(src,
+ end - src))) {
strbuf_add(&dst, src, n);
src += n;
}
diff --git a/utf8.h b/utf8.h
index cf8ecb0f21..57ca2ab270 100644
--- a/utf8.h
+++ b/utf8.h
@@ -6,6 +6,7 @@ struct strbuf;
typedef unsigned int ucs_char_t; /* assuming 32bit int */
size_t display_mode_esc_sequence_len(const char *s);
+size_t display_mode_esc_sequence_len_counted(const char *s, size_t len);
int utf8_width(const char **start, size_t *remainder_p);
int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
int utf8_strwidth(const char *string);
--
2.56.1-740-gf38658f3d0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-17 20:06 ` [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() Junio C Hamano
@ 2026-11-17 21:26 ` Dirk Gouders
2026-11-17 21:43 ` Junio C Hamano
2026-11-17 23:08 ` Andy Koppe
1 sibling, 1 reply; 15+ messages in thread
From: Dirk Gouders @ 2026-11-17 21:26 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
On Tue, Nov 17 2026, Junio C Hamano wrote:
> display_mode_esc_sequence_len() scans for an ANSI display mode
> escape sequence (e.g. "ESC [ 1 ; 31 m") at the start of a string,
> relying on NUL termination to stop. Some callers, however, work on
> a counted buffer: utf8_strnwidth() takes <string, len> and must not
> scan past it, while strbuf_utf8_replace() and
> strbuf_add_wrapped_text() know (or can cheaply learn) the end.
>
> Add display_mode_esc_sequence_len_counted() taking <ptr, len>,
> which never reads at or beyond ptr[len], and make
> display_mode_esc_sequence_len() a thin wrapper for NUL-terminated
> strings.
>
> Use the counted variant in utf8_strnwidth() for correctness, and in
> strbuf_utf8_replace() and strbuf_add_wrapped_text() (computing the
> end once upfront in the latter) to avoid measuring the remaining
> text in each iteration.
>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---
> * As there are new patches that want to add new callers to this
> function, let's keep the original with the same API while adding
> a new helper function that allows us to limit the scan.
>
> utf8.c | 35 ++++++++++++++++++++++++++---------
> utf8.h | 1 +
> 2 files changed, 27 insertions(+), 9 deletions(-)
>
> diff --git a/utf8.c b/utf8.c
> index 96460cc414..d0546ab31f 100644
> --- a/utf8.c
> +++ b/utf8.c
> @@ -16,20 +16,33 @@ struct interval {
> ucs_char_t last;
> };
>
> -size_t display_mode_esc_sequence_len(const char *s)
> +/*
> + * Return the length of the ANSI display mode escape sequence (e.g.,
> + * "\033[1;31m") that appears at the beginning of the "len" bytes
> + * of string "s", or 0 if the bytes do not begin with a complete
> + * such sequence. Never looks at s[len] or beyond.
> + */
> +size_t display_mode_esc_sequence_len_counted(const char *s, size_t len)
> {
> - const char *p = s;
> - if (*p++ != '\033')
> + const char *p = s, *end = s + len;
You have better knowledge of the users of this function: would an early
return perhaps handle a significant amount of cases?
/* The shortest SGR sequence is "\033[m". */
if (len < 3)
return 0;
And, why not add some tests (the untested function I showed had
an off-by-one error):
diff --git a/Makefile b/Makefile
index 4c84eea6e2..3c8dc9ca40 100644
--- a/Makefile
+++ b/Makefile
@@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec
CLAR_TEST_SUITES += u-trailer
CLAR_TEST_SUITES += u-urlmatch-normalization
CLAR_TEST_SUITES += u-utf8-width
+CLAR_TEST_SUITES += u-utf8-ansi
CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES))
CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
diff --git a/t/meson.build b/t/meson.build
index 1cf1d95f70..e224df58c8 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -27,6 +27,7 @@ clar_test_suites = [
'unit-tests/u-trailer.c',
'unit-tests/u-urlmatch-normalization.c',
'unit-tests/u-utf8-width.c',
+ 'unit-tests/u-utf8-ansi.c',
]
clar_sources = [
diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c
new file mode 100644
index 0000000000..c19d8bce78
--- /dev/null
+++ b/t/unit-tests/u-utf8-ansi.c
@@ -0,0 +1,14 @@
+#include "unit-test.h"
+#include "utf8.h"
+#include "strbuf.h"
+
+/*
+ * Test len limited functions for ANSI SGR lengths.
+ */
+void test_utf8_ansi__sgr_len(void)
+{
+ cl_assert_equal_i(0, display_mode_esc_sequence_len_counted("\033[m", 2));
+ cl_assert_equal_i(3, display_mode_esc_sequence_len_counted("\033[m", 3));
+ cl_assert_equal_i(5, display_mode_esc_sequence_len_counted("\033[31m", 5));
+ cl_assert_equal_i(7, display_mode_esc_sequence_len_counted("\033[0;31m", 7));
+}
Regards,
Dirk
> + if (end <= p || *p++ != '\033')
> return 0;
> - if (*p++ != '[')
> + if (end <= p || *p++ != '[')
> return 0;
> - while (isdigit(*p) || *p == ';')
> + while (p < end && (isdigit(*p) || *p == ';'))
> p++;
> - if (*p++ != 'm')
> + if (end <= p || *p++ != 'm')
> return 0;
> return p - s;
> }
>
> +/* Same as above, but for a NUL-terminated string "s". */
> +size_t display_mode_esc_sequence_len(const char *s)
> +{
> + return display_mode_esc_sequence_len_counted(s, strlen(s));
> +}
> +
> /* auxiliary function for binary search in interval table */
> static int bisearch(ucs_char_t ucs, const struct interval *table, int max)
> {
> @@ -218,7 +231,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi)
> size_t skip;
>
> while (skip_ansi &&
> - (skip = display_mode_esc_sequence_len(string)) != 0)
> + (skip = display_mode_esc_sequence_len_counted(string,
> + orig + len - string)) != 0)
> string += skip;
>
> glyph_width = utf8_width(&string, NULL);
> @@ -279,6 +293,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
> {
> int indent, w, assume_utf8 = 1;
> const char *bol, *space, *start = text;
> + const char *end = text + strlen(text);
> size_t orig_len = buf->len;
>
> if (width <= 0) {
> @@ -299,7 +314,8 @@ void strbuf_add_wrapped_text(struct strbuf *buf,
> char c;
> size_t skip;
>
> - while ((skip = display_mode_esc_sequence_len(text)))
> + while ((skip = display_mode_esc_sequence_len_counted(text,
> + end - text)))
> text += skip;
>
> c = *text;
> @@ -378,7 +394,8 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width,
> int glyph_width;
> size_t n;
>
> - while ((n = display_mode_esc_sequence_len(src))) {
> + while ((n = display_mode_esc_sequence_len_counted(src,
> + end - src))) {
> strbuf_add(&dst, src, n);
> src += n;
> }
> diff --git a/utf8.h b/utf8.h
> index cf8ecb0f21..57ca2ab270 100644
> --- a/utf8.h
> +++ b/utf8.h
> @@ -6,6 +6,7 @@ struct strbuf;
> typedef unsigned int ucs_char_t; /* assuming 32bit int */
>
> size_t display_mode_esc_sequence_len(const char *s);
> +size_t display_mode_esc_sequence_len_counted(const char *s, size_t len);
> int utf8_width(const char **start, size_t *remainder_p);
> int utf8_strnwidth(const char *string, size_t len, int skip_ansi);
> int utf8_strwidth(const char *string);
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-17 21:26 ` Dirk Gouders
@ 2026-11-17 21:43 ` Junio C Hamano
2026-11-17 22:03 ` Dirk Gouders
0 siblings, 1 reply; 15+ messages in thread
From: Junio C Hamano @ 2026-11-17 21:43 UTC (permalink / raw)
To: Dirk Gouders; +Cc: git
Dirk Gouders <dirk@gouders.net> writes:
> You have better knowledge of the users of this function: would an early
> return perhaps handle a significant amount of cases?
>
> /* The shortest SGR sequence is "\033[m". */
> if (len < 3)
> return 0;
Do you have any benchmarks? I don't.
I find it of dubious performance merit to run this comparison for a
long string as it is eaten one UTF-8 characters at a time by the
caller, and when the length is actually shorter than 3, then
runnning the remainder of the function for a few steps should not be
10x more expensive than the above comparison, no?
> And, why not add some tests (the untested function I showed had
> an off-by-one error):
Sounds good. Should I just steal what you sent out?
> diff --git a/Makefile b/Makefile
> index 4c84eea6e2..3c8dc9ca40 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec
> CLAR_TEST_SUITES += u-trailer
> CLAR_TEST_SUITES += u-urlmatch-normalization
> CLAR_TEST_SUITES += u-utf8-width
> +CLAR_TEST_SUITES += u-utf8-ansi
> CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
> CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES))
> CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
> diff --git a/t/meson.build b/t/meson.build
> index 1cf1d95f70..e224df58c8 100644
> --- a/t/meson.build
> +++ b/t/meson.build
> @@ -27,6 +27,7 @@ clar_test_suites = [
> 'unit-tests/u-trailer.c',
> 'unit-tests/u-urlmatch-normalization.c',
> 'unit-tests/u-utf8-width.c',
> + 'unit-tests/u-utf8-ansi.c',
> ]
>
> clar_sources = [
> diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c
> new file mode 100644
> index 0000000000..c19d8bce78
> --- /dev/null
> +++ b/t/unit-tests/u-utf8-ansi.c
> @@ -0,0 +1,14 @@
> +#include "unit-test.h"
> +#include "utf8.h"
> +#include "strbuf.h"
> +
> +/*
> + * Test len limited functions for ANSI SGR lengths.
> + */
> +void test_utf8_ansi__sgr_len(void)
> +{
> + cl_assert_equal_i(0, display_mode_esc_sequence_len_counted("\033[m", 2));
> + cl_assert_equal_i(3, display_mode_esc_sequence_len_counted("\033[m", 3));
> + cl_assert_equal_i(5, display_mode_esc_sequence_len_counted("\033[31m", 5));
> + cl_assert_equal_i(7, display_mode_esc_sequence_len_counted("\033[0;31m", 7));
> +}
Thanks.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-17 21:43 ` Junio C Hamano
@ 2026-11-17 22:03 ` Dirk Gouders
2026-11-18 1:33 ` Junio C Hamano
0 siblings, 1 reply; 15+ messages in thread
From: Dirk Gouders @ 2026-11-17 22:03 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
On Tue, Nov 17 2026, Junio C Hamano wrote:
> Dirk Gouders <dirk@gouders.net> writes:
>
>> You have better knowledge of the users of this function: would an early
>> return perhaps handle a significant amount of cases?
>>
>> /* The shortest SGR sequence is "\033[m". */
>> if (len < 3)
>> return 0;
>
> Do you have any benchmarks? I don't.
>
> I find it of dubious performance merit to run this comparison for a
> long string as it is eaten one UTF-8 characters at a time by the
> caller, and when the length is actually shorter than 3, then
> runnning the remainder of the function for a few steps should not be
> 10x more expensive than the above comparison, no?
Meanwhile, I thought: even if the test would be a good idea, the
"\033"-test probably is the one that handles most cases.
Maybe I will look closer, but not now. Correctness is more important
and I need to work on v2.
>> And, why not add some tests (the untested function I showed had
>> an off-by-one error):
>
> Sounds good. Should I just steal what you sent out?
That wouldn't be stealing, of course you can use it.
Regards,
Dirk
>> diff --git a/Makefile b/Makefile
>> index 4c84eea6e2..3c8dc9ca40 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec
>> CLAR_TEST_SUITES += u-trailer
>> CLAR_TEST_SUITES += u-urlmatch-normalization
>> CLAR_TEST_SUITES += u-utf8-width
>> +CLAR_TEST_SUITES += u-utf8-ansi
>> CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X)
>> CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES))
>> CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
>> diff --git a/t/meson.build b/t/meson.build
>> index 1cf1d95f70..e224df58c8 100644
>> --- a/t/meson.build
>> +++ b/t/meson.build
>> @@ -27,6 +27,7 @@ clar_test_suites = [
>> 'unit-tests/u-trailer.c',
>> 'unit-tests/u-urlmatch-normalization.c',
>> 'unit-tests/u-utf8-width.c',
>> + 'unit-tests/u-utf8-ansi.c',
>> ]
>>
>> clar_sources = [
>> diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c
>> new file mode 100644
>> index 0000000000..c19d8bce78
>> --- /dev/null
>> +++ b/t/unit-tests/u-utf8-ansi.c
>> @@ -0,0 +1,14 @@
>> +#include "unit-test.h"
>> +#include "utf8.h"
>> +#include "strbuf.h"
>> +
>> +/*
>> + * Test len limited functions for ANSI SGR lengths.
>> + */
>> +void test_utf8_ansi__sgr_len(void)
>> +{
>> + cl_assert_equal_i(0, display_mode_esc_sequence_len_counted("\033[m", 2));
>> + cl_assert_equal_i(3, display_mode_esc_sequence_len_counted("\033[m", 3));
>> + cl_assert_equal_i(5, display_mode_esc_sequence_len_counted("\033[31m", 5));
>> + cl_assert_equal_i(7, display_mode_esc_sequence_len_counted("\033[0;31m", 7));
>> +}
>
> Thanks.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-17 20:06 ` [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() Junio C Hamano
2026-11-17 21:26 ` Dirk Gouders
@ 2026-11-17 23:08 ` Andy Koppe
2026-11-17 23:25 ` Junio C Hamano
1 sibling, 1 reply; 15+ messages in thread
From: Andy Koppe @ 2026-11-17 23:08 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Dirk Gouders
Junio C Hamano wrote:
> - while (isdigit(*p) || *p == ';')
> + while (p < end && (isdigit(*p) || *p == ';'))
While you're there, could you add colons (':') to that?
ECMA-48 §5.4.2 defines them as sub-separators within the
semicolon-separated parameters of control sequences. Colons are
referred to as "3/10" there, aka ASCII 0x3A:
https://ecma-international.org/wp-content/uploads/ECMA-48_5th_edition_june_1991.pdf#page=26
Maybe more practically relevant, xterm and its many
more-or-less-compatible brethren accept colons in SGR sequences:
"xterm allows either colons (standard) or semicolons (legacy) to
separate the subparameters (but after the first colon, colons must be
used)"
https://invisible-island.net/xterm/ctlseqs/ctlseqs.html
Andy
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-17 23:08 ` Andy Koppe
@ 2026-11-17 23:25 ` Junio C Hamano
2026-11-18 6:59 ` Andy Koppe
0 siblings, 1 reply; 15+ messages in thread
From: Junio C Hamano @ 2026-11-17 23:25 UTC (permalink / raw)
To: Andy Koppe; +Cc: git, Dirk Gouders
Andy Koppe <andy.koppe@gmail.com> writes:
> Junio C Hamano wrote:
>
>> - while (isdigit(*p) || *p == ';')
>> + while (p < end && (isdigit(*p) || *p == ';'))
>
> While you're there, could you add colons (':') to that?
Do we generate such a sequence that uses subseparators?
IIRC, this code is not counting the display width of end-user
supplied string. Rather, it gets string we paint in a buffer and
then counts how much display column it occupies.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-17 22:03 ` Dirk Gouders
@ 2026-11-18 1:33 ` Junio C Hamano
2026-11-18 7:37 ` Dirk Gouders
0 siblings, 1 reply; 15+ messages in thread
From: Junio C Hamano @ 2026-11-18 1:33 UTC (permalink / raw)
To: Dirk Gouders; +Cc: git
Dirk Gouders <dirk@gouders.net> writes:
> Meanwhile, I thought: even if the test would be a good idea, the
> "\033"-test probably is the one that handles most cases.
>
> Maybe I will look closer, but not now. Correctness is more important
> and I need to work on v2.
It would be nice if you can take this patch over and make it your
[PATCH v2 1/2].
Thanks.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-17 23:25 ` Junio C Hamano
@ 2026-11-18 6:59 ` Andy Koppe
2026-11-18 7:43 ` Dirk Gouders
0 siblings, 1 reply; 15+ messages in thread
From: Andy Koppe @ 2026-11-18 6:59 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Dirk Gouders
Junio C Hamano wrote:
>
> Andy Koppe <andy.koppe@gmail.com> writes:
>
> > Junio C Hamano wrote:
> >
> >> - while (isdigit(*p) || *p == ';')
> >> + while (p < end && (isdigit(*p) || *p == ';'))
> >
> > While you're there, could you add colons (':') to that?
>
> Do we generate such a sequence that uses subseparators?
>
> IIRC, this code is not counting the display width of end-user
> supplied string. Rather, it gets string we paint in a buffer and
> then counts how much display column it occupies.
You're right that Git itself doesn't generate them (although future
enhancements might change that).
However, display_mode_esc_sequence_len() already operates on expanded
custom formats and message text, e.g. in rewrap_message_tail() via
strbuf_wrap(). And on stdin in git-column. They could all contain SGR
sequences with colons, whose width currently isn't ignored as
intended. (And my proposed make_line_colors_self_contained() wouldn't
reset or replay them.)
The check could be changed to `*p >= '0' && *p<=';'` to cheaply include colons.
Andy
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-18 1:33 ` Junio C Hamano
@ 2026-11-18 7:37 ` Dirk Gouders
0 siblings, 0 replies; 15+ messages in thread
From: Dirk Gouders @ 2026-11-18 7:37 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
On Tue, Nov 17 2026, Junio C Hamano wrote:
> Dirk Gouders <dirk@gouders.net> writes:
>
>> Meanwhile, I thought: even if the test would be a good idea, the
>> "\033"-test probably is the one that handles most cases.
>>
>> Maybe I will look closer, but not now. Correctness is more important
>> and I need to work on v2.
>
> It would be nice if you can take this patch over and make it your
> [PATCH v2 1/2].
OK, I will do so, thank you.
Regards,
Dirk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-18 6:59 ` Andy Koppe
@ 2026-11-18 7:43 ` Dirk Gouders
2026-11-18 9:48 ` Andy Koppe
0 siblings, 1 reply; 15+ messages in thread
From: Dirk Gouders @ 2026-11-18 7:43 UTC (permalink / raw)
To: Andy Koppe; +Cc: Junio C Hamano, git
On Wed, Nov 18 2026, Andy Koppe wrote:
> Junio C Hamano wrote:
>>
>> Andy Koppe <andy.koppe@gmail.com> writes:
>>
>> > Junio C Hamano wrote:
>> >
>> >> - while (isdigit(*p) || *p == ';')
>> >> + while (p < end && (isdigit(*p) || *p == ';'))
>> >
>> > While you're there, could you add colons (':') to that?
>>
>> Do we generate such a sequence that uses subseparators?
>>
>> IIRC, this code is not counting the display width of end-user
>> supplied string. Rather, it gets string we paint in a buffer and
>> then counts how much display column it occupies.
>
> You're right that Git itself doesn't generate them (although future
> enhancements might change that).
>
> However, display_mode_esc_sequence_len() already operates on expanded
> custom formats and message text, e.g. in rewrap_message_tail() via
> strbuf_wrap(). And on stdin in git-column. They could all contain SGR
> sequences with colons, whose width currently isn't ignored as
> intended. (And my proposed make_line_colors_self_contained() wouldn't
> reset or replay them.)
>
> The check could be changed to `*p >= '0' && *p<=';'` to cheaply include colons.
I would suggest to not mix these changes simply for bisection purposes.
And as far as I know, we would need an FSM to identify valid sequences
if we are going to allow colons.
Regards,
Dirk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-18 7:43 ` Dirk Gouders
@ 2026-11-18 9:48 ` Andy Koppe
2026-11-18 18:38 ` Dirk Gouders
0 siblings, 1 reply; 15+ messages in thread
From: Andy Koppe @ 2026-11-18 9:48 UTC (permalink / raw)
To: Dirk Gouders; +Cc: Junio C Hamano, git
Dirk Gouders <wrote:
>
> On Wed, Nov 18 2026, Andy Koppe wrote:
>
> > Junio C Hamano wrote:
> >>
> >> Andy Koppe <andy.koppe@gmail.com> writes:
> >>
> >> > Junio C Hamano wrote:
> >> >
> >> >> - while (isdigit(*p) || *p == ';')
> >> >> + while (p < end && (isdigit(*p) || *p == ';'))
> >> >
> >> > While you're there, could you add colons (':') to that?
> >>
> >> Do we generate such a sequence that uses subseparators?
> >>
> >> IIRC, this code is not counting the display width of end-user
> >> supplied string. Rather, it gets string we paint in a buffer and
> >> then counts how much display column it occupies.
> >
> > You're right that Git itself doesn't generate them (although future
> > enhancements might change that).
> >
> > However, display_mode_esc_sequence_len() already operates on expanded
> > custom formats and message text, e.g. in rewrap_message_tail() via
> > strbuf_wrap(). And on stdin in git-column. They could all contain SGR
> > sequences with colons, whose width currently isn't ignored as
> > intended. (And my proposed make_line_colors_self_contained() wouldn't
> > reset or replay them.)
> >
> > The check could be changed to `*p >= '0' && *p<=';'` to cheaply include colons.
>
> I would suggest to not mix these changes simply for bisection purposes.
Fair, and there's testing work as well of course. It was silly of me
to suggest just sneaking that change in.
> And as far as I know, we would need an FSM to identify valid sequences
> if we are going to allow colons.
No. The parameter field can be any mix of digits, colons and
semicolons. Semicolons separate parameters, colons separate
subparameters, and individual (sub)parameters can be empty, which is
equivalent to zero. If the field consists of nothing but colons,
that's one parameter with multiple empty subparameters.
Whether any particular SGR sequence means anything to a terminal is
another matter of course, but git doesn't try to determine that for
colon-less sequences either, and nor should it. Well-behaved terminals
ignore valid sequences they don't understand.
Andy
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-18 9:48 ` Andy Koppe
@ 2026-11-18 18:38 ` Dirk Gouders
2026-11-18 19:12 ` Andy Koppe
0 siblings, 1 reply; 15+ messages in thread
From: Dirk Gouders @ 2026-11-18 18:38 UTC (permalink / raw)
To: Andy Koppe; +Cc: Junio C Hamano, git
On Wed, Nov 18 2026, Andy Koppe wrote:
>> Dirk Gouders <wrote:
>> And as far as I know, we would need an FSM to identify valid sequences
>> if we are going to allow colons.
> No. The parameter field can be any mix of digits, colons and
> semicolons. Semicolons separate parameters, colons separate
> subparameters, and individual (sub)parameters can be empty, which is
> equivalent to zero. If the field consists of nothing but colons,
> that's one parameter with multiple empty subparameters.
My statement was a bit misleading, because
display_mode_esc_sequence_len() already is an FSM
accepting the language "\033\[[0-9;]+m". My concern was that I doubt
that simply adding ":" to the alphabet and as valid input to the state
"parameter" (without introducing one or more new states) would still
let it accept only valid SGR sequences.
It seems I can learn something: is "\033[:;m" a valid SGR sequence?
But, such a discussion is difficult without defining what language you
want to accept. I guess the two links were meant as a definition and I
have to confess I didn't yet read them.
Regards,
Dirk
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-18 18:38 ` Dirk Gouders
@ 2026-11-18 19:12 ` Andy Koppe
2026-11-18 19:29 ` Dirk Gouders
0 siblings, 1 reply; 15+ messages in thread
From: Andy Koppe @ 2026-11-18 19:12 UTC (permalink / raw)
To: Dirk Gouders; +Cc: Junio C Hamano, git
Dirk Gouders wrote:
>
> On Wed, Nov 18 2026, Andy Koppe wrote:
> >> And as far as I know, we would need an FSM to identify valid sequences
> >> if we are going to allow colons.
>
> > No. The parameter field can be any mix of digits, colons and
> > semicolons. Semicolons separate parameters, colons separate
> > subparameters, and individual (sub)parameters can be empty, which is
> > equivalent to zero. If the field consists of nothing but colons,
> > that's one parameter with multiple empty subparameters.
>
> My statement was a bit misleading, because
> display_mode_esc_sequence_len() already is an FSM
> accepting the language "\033\[[0-9;]+m". My concern was that I doubt
> that simply adding ":" to the alphabet and as valid input to the state
> "parameter" (without introducing one or more new states) would still
> let it accept only valid SGR sequences.
"\033\[[0-9:;]*m" is the complete regex for SGRs. (Of course, it's not
that simple if you wanted to actually collect the (sub)parameter
values, but git doesn't need to.)
> It seems I can learn something: is "\033[:;m" a valid SGR sequence?
Yep. That's two parameters, with the first one consisting of two empty
subparameters, and the second one an empty parameter.
> But, such a discussion is difficult without defining what language you
> want to accept. I guess the two links were meant as a definition and I
> have to confess I didn't yet read them.
To be fair, ECMA-48 isn't the most readable, but once you realise that
3/0 stands for 0x30 (aka '0') and 3/11 for 0x3B (aka ';'), it's not
too bad.
Andy
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted()
2026-11-18 19:12 ` Andy Koppe
@ 2026-11-18 19:29 ` Dirk Gouders
0 siblings, 0 replies; 15+ messages in thread
From: Dirk Gouders @ 2026-11-18 19:29 UTC (permalink / raw)
To: Andy Koppe; +Cc: Junio C Hamano, git
On Wed, Nov 18 2026, Andy Koppe wrote:
> Dirk Gouders wrote:
>>
>> On Wed, Nov 18 2026, Andy Koppe wrote:
>
>> >> And as far as I know, we would need an FSM to identify valid sequences
>> >> if we are going to allow colons.
>>
>> > No. The parameter field can be any mix of digits, colons and
>> > semicolons. Semicolons separate parameters, colons separate
>> > subparameters, and individual (sub)parameters can be empty, which is
>> > equivalent to zero. If the field consists of nothing but colons,
>> > that's one parameter with multiple empty subparameters.
>>
>> My statement was a bit misleading, because
>> display_mode_esc_sequence_len() already is an FSM
>> accepting the language "\033\[[0-9;]+m". My concern was that I doubt
>> that simply adding ":" to the alphabet and as valid input to the state
>> "parameter" (without introducing one or more new states) would still
>> let it accept only valid SGR sequences.
>
> "\033\[[0-9:;]*m" is the complete regex for SGRs. (Of course, it's not
> that simple if you wanted to actually collect the (sub)parameter
> values, but git doesn't need to.)
>
>> It seems I can learn something: is "\033[:;m" a valid SGR sequence?
>
> Yep. That's two parameters, with the first one consisting of two empty
> subparameters, and the second one an empty parameter.
OK, I didn't know that, thank you.
>> But, such a discussion is difficult without defining what language you
>> want to accept. I guess the two links were meant as a definition and I
>> have to confess I didn't yet read them.
>
> To be fair, ECMA-48 isn't the most readable, but once you realise that
> 3/0 stands for 0x30 (aka '0') and 3/11 for 0x3B (aka ';'), it's not
> too bad.
That notation is fine for me, I will definitely read that document.
Regards,
Dirk
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2026-11-18 19:29 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-17 16:50 [PATCH] utf8: make display_mode_esc_sequence_len() take a length Junio C Hamano
2026-11-17 20:06 ` [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() Junio C Hamano
2026-11-17 21:26 ` Dirk Gouders
2026-11-17 21:43 ` Junio C Hamano
2026-11-17 22:03 ` Dirk Gouders
2026-11-18 1:33 ` Junio C Hamano
2026-11-18 7:37 ` Dirk Gouders
2026-11-17 23:08 ` Andy Koppe
2026-11-17 23:25 ` Junio C Hamano
2026-11-18 6:59 ` Andy Koppe
2026-11-18 7:43 ` Dirk Gouders
2026-11-18 9:48 ` Andy Koppe
2026-11-18 18:38 ` Dirk Gouders
2026-11-18 19:12 ` Andy Koppe
2026-11-18 19:29 ` Dirk Gouders
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox