* [PATCH] utf8: make display_mode_esc_sequence_len() take a length @ 2026-11-17 16:50 Junio C Hamano 2026-11-17 20:06 ` [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() Junio C Hamano 0 siblings, 1 reply; 15+ messages in thread From: Junio C Hamano @ 2026-11-17 16:50 UTC (permalink / raw) To: git; +Cc: Dirk Gouders display_mode_esc_sequence_len() scans for an ANSI display mode escape sequence (e.g. "ESC [ 1 ; 31 m"), relying on NUL termination to stop. Some callers, however, use a counted <ptr, len> buffer or need to stop at a specific position. Teach it to take a buffer length and never read at or beyond ptr[len], returning the length of a complete sequence at the start or 0 otherwise. Update the callers to pass the remaining byte count. For strbuf_add_wrapped_text(), which takes a NUL-terminated string, compute the end once upfront instead of calling strlen() each time. Signed-off-by: Junio C Hamano <gitster@pobox.com> --- * Copy of https://lore.kernel.org/git/xmqqse0zfn2n.fsf@gitster.g/ with a proposed commit log message. pretty.c | 3 ++- utf8.c | 27 ++++++++++++++++++--------- utf8.h | 2 +- 3 files changed, 21 insertions(+), 11 deletions(-) diff --git a/pretty.c b/pretty.c index 0bc8ad8a9a..a20fd61909 100644 --- a/pretty.c +++ b/pretty.c @@ -1834,7 +1834,8 @@ static size_t format_and_pad_commit(struct strbuf *sb, /* in UTF-8 */ while (p > sb->buf && ch - p < 10 && *p != '\033') p--; if (*p != '\033' || - ch + 1 - p != display_mode_esc_sequence_len(p)) + ch + 1 - p != display_mode_esc_sequence_len(p, + ch + 1 - p)) break; /* * got a good ansi sequence, put it back to diff --git a/utf8.c b/utf8.c index 96460cc414..544c5329f3 100644 --- a/utf8.c +++ b/utf8.c @@ -16,16 +16,23 @@ struct interval { ucs_char_t last; }; -size_t display_mode_esc_sequence_len(const char *s) +/* + * Return the length of the ANSI display mode escape sequence (e.g., + * "\033[1;31m") that appears at the beginning of the "len" bytes + * of string "s", or 0 if the bytes do not begin with a complete + * such sequence. Never looks at s[len] or beyond. + */ +size_t display_mode_esc_sequence_len(const char *s, size_t len) { - const char *p = s; - if (*p++ != '\033') + const char *p = s, *end = s + len; + + if (end <= p || *p++ != '\033') return 0; - if (*p++ != '[') + if (end <= p || *p++ != '[') return 0; - while (isdigit(*p) || *p == ';') + while (p < end && (isdigit(*p) || *p == ';')) p++; - if (*p++ != 'm') + if (end <= p || *p++ != 'm') return 0; return p - s; } @@ -218,7 +225,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi) size_t skip; while (skip_ansi && - (skip = display_mode_esc_sequence_len(string)) != 0) + (skip = display_mode_esc_sequence_len(string, + orig + len - string)) != 0) string += skip; glyph_width = utf8_width(&string, NULL); @@ -279,6 +287,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf, { int indent, w, assume_utf8 = 1; const char *bol, *space, *start = text; + const char *end = text + strlen(text); size_t orig_len = buf->len; if (width <= 0) { @@ -299,7 +308,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf, char c; size_t skip; - while ((skip = display_mode_esc_sequence_len(text))) + while ((skip = display_mode_esc_sequence_len(text, end - text))) text += skip; c = *text; @@ -378,7 +387,7 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width, int glyph_width; size_t n; - while ((n = display_mode_esc_sequence_len(src))) { + while ((n = display_mode_esc_sequence_len(src, end - src))) { strbuf_add(&dst, src, n); src += n; } diff --git a/utf8.h b/utf8.h index cf8ecb0f21..fb2227a9e2 100644 --- a/utf8.h +++ b/utf8.h @@ -5,7 +5,7 @@ struct strbuf; typedef unsigned int ucs_char_t; /* assuming 32bit int */ -size_t display_mode_esc_sequence_len(const char *s); +size_t display_mode_esc_sequence_len(const char *s, size_t len); int utf8_width(const char **start, size_t *remainder_p); int utf8_strnwidth(const char *string, size_t len, int skip_ansi); int utf8_strwidth(const char *string); -- 2.56.1-740-gf38658f3d0 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-17 16:50 [PATCH] utf8: make display_mode_esc_sequence_len() take a length Junio C Hamano @ 2026-11-17 20:06 ` Junio C Hamano 2026-11-17 21:26 ` Dirk Gouders 2026-11-17 23:08 ` Andy Koppe 0 siblings, 2 replies; 15+ messages in thread From: Junio C Hamano @ 2026-11-17 20:06 UTC (permalink / raw) To: git; +Cc: Dirk Gouders display_mode_esc_sequence_len() scans for an ANSI display mode escape sequence (e.g. "ESC [ 1 ; 31 m") at the start of a string, relying on NUL termination to stop. Some callers, however, work on a counted buffer: utf8_strnwidth() takes <string, len> and must not scan past it, while strbuf_utf8_replace() and strbuf_add_wrapped_text() know (or can cheaply learn) the end. Add display_mode_esc_sequence_len_counted() taking <ptr, len>, which never reads at or beyond ptr[len], and make display_mode_esc_sequence_len() a thin wrapper for NUL-terminated strings. Use the counted variant in utf8_strnwidth() for correctness, and in strbuf_utf8_replace() and strbuf_add_wrapped_text() (computing the end once upfront in the latter) to avoid measuring the remaining text in each iteration. Signed-off-by: Junio C Hamano <gitster@pobox.com> --- * As there are new patches that want to add new callers to this function, let's keep the original with the same API while adding a new helper function that allows us to limit the scan. utf8.c | 35 ++++++++++++++++++++++++++--------- utf8.h | 1 + 2 files changed, 27 insertions(+), 9 deletions(-) diff --git a/utf8.c b/utf8.c index 96460cc414..d0546ab31f 100644 --- a/utf8.c +++ b/utf8.c @@ -16,20 +16,33 @@ struct interval { ucs_char_t last; }; -size_t display_mode_esc_sequence_len(const char *s) +/* + * Return the length of the ANSI display mode escape sequence (e.g., + * "\033[1;31m") that appears at the beginning of the "len" bytes + * of string "s", or 0 if the bytes do not begin with a complete + * such sequence. Never looks at s[len] or beyond. + */ +size_t display_mode_esc_sequence_len_counted(const char *s, size_t len) { - const char *p = s; - if (*p++ != '\033') + const char *p = s, *end = s + len; + + if (end <= p || *p++ != '\033') return 0; - if (*p++ != '[') + if (end <= p || *p++ != '[') return 0; - while (isdigit(*p) || *p == ';') + while (p < end && (isdigit(*p) || *p == ';')) p++; - if (*p++ != 'm') + if (end <= p || *p++ != 'm') return 0; return p - s; } +/* Same as above, but for a NUL-terminated string "s". */ +size_t display_mode_esc_sequence_len(const char *s) +{ + return display_mode_esc_sequence_len_counted(s, strlen(s)); +} + /* auxiliary function for binary search in interval table */ static int bisearch(ucs_char_t ucs, const struct interval *table, int max) { @@ -218,7 +231,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi) size_t skip; while (skip_ansi && - (skip = display_mode_esc_sequence_len(string)) != 0) + (skip = display_mode_esc_sequence_len_counted(string, + orig + len - string)) != 0) string += skip; glyph_width = utf8_width(&string, NULL); @@ -279,6 +293,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf, { int indent, w, assume_utf8 = 1; const char *bol, *space, *start = text; + const char *end = text + strlen(text); size_t orig_len = buf->len; if (width <= 0) { @@ -299,7 +314,8 @@ void strbuf_add_wrapped_text(struct strbuf *buf, char c; size_t skip; - while ((skip = display_mode_esc_sequence_len(text))) + while ((skip = display_mode_esc_sequence_len_counted(text, + end - text))) text += skip; c = *text; @@ -378,7 +394,8 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width, int glyph_width; size_t n; - while ((n = display_mode_esc_sequence_len(src))) { + while ((n = display_mode_esc_sequence_len_counted(src, + end - src))) { strbuf_add(&dst, src, n); src += n; } diff --git a/utf8.h b/utf8.h index cf8ecb0f21..57ca2ab270 100644 --- a/utf8.h +++ b/utf8.h @@ -6,6 +6,7 @@ struct strbuf; typedef unsigned int ucs_char_t; /* assuming 32bit int */ size_t display_mode_esc_sequence_len(const char *s); +size_t display_mode_esc_sequence_len_counted(const char *s, size_t len); int utf8_width(const char **start, size_t *remainder_p); int utf8_strnwidth(const char *string, size_t len, int skip_ansi); int utf8_strwidth(const char *string); -- 2.56.1-740-gf38658f3d0 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-17 20:06 ` [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() Junio C Hamano @ 2026-11-17 21:26 ` Dirk Gouders 2026-11-17 21:43 ` Junio C Hamano 2026-11-17 23:08 ` Andy Koppe 1 sibling, 1 reply; 15+ messages in thread From: Dirk Gouders @ 2026-11-17 21:26 UTC (permalink / raw) To: Junio C Hamano; +Cc: git On Tue, Nov 17 2026, Junio C Hamano wrote: > display_mode_esc_sequence_len() scans for an ANSI display mode > escape sequence (e.g. "ESC [ 1 ; 31 m") at the start of a string, > relying on NUL termination to stop. Some callers, however, work on > a counted buffer: utf8_strnwidth() takes <string, len> and must not > scan past it, while strbuf_utf8_replace() and > strbuf_add_wrapped_text() know (or can cheaply learn) the end. > > Add display_mode_esc_sequence_len_counted() taking <ptr, len>, > which never reads at or beyond ptr[len], and make > display_mode_esc_sequence_len() a thin wrapper for NUL-terminated > strings. > > Use the counted variant in utf8_strnwidth() for correctness, and in > strbuf_utf8_replace() and strbuf_add_wrapped_text() (computing the > end once upfront in the latter) to avoid measuring the remaining > text in each iteration. > > Signed-off-by: Junio C Hamano <gitster@pobox.com> > --- > * As there are new patches that want to add new callers to this > function, let's keep the original with the same API while adding > a new helper function that allows us to limit the scan. > > utf8.c | 35 ++++++++++++++++++++++++++--------- > utf8.h | 1 + > 2 files changed, 27 insertions(+), 9 deletions(-) > > diff --git a/utf8.c b/utf8.c > index 96460cc414..d0546ab31f 100644 > --- a/utf8.c > +++ b/utf8.c > @@ -16,20 +16,33 @@ struct interval { > ucs_char_t last; > }; > > -size_t display_mode_esc_sequence_len(const char *s) > +/* > + * Return the length of the ANSI display mode escape sequence (e.g., > + * "\033[1;31m") that appears at the beginning of the "len" bytes > + * of string "s", or 0 if the bytes do not begin with a complete > + * such sequence. Never looks at s[len] or beyond. > + */ > +size_t display_mode_esc_sequence_len_counted(const char *s, size_t len) > { > - const char *p = s; > - if (*p++ != '\033') > + const char *p = s, *end = s + len; You have better knowledge of the users of this function: would an early return perhaps handle a significant amount of cases? /* The shortest SGR sequence is "\033[m". */ if (len < 3) return 0; And, why not add some tests (the untested function I showed had an off-by-one error): diff --git a/Makefile b/Makefile index 4c84eea6e2..3c8dc9ca40 100644 --- a/Makefile +++ b/Makefile @@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec CLAR_TEST_SUITES += u-trailer CLAR_TEST_SUITES += u-urlmatch-normalization CLAR_TEST_SUITES += u-utf8-width +CLAR_TEST_SUITES += u-utf8-ansi CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X) CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES)) CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o diff --git a/t/meson.build b/t/meson.build index 1cf1d95f70..e224df58c8 100644 --- a/t/meson.build +++ b/t/meson.build @@ -27,6 +27,7 @@ clar_test_suites = [ 'unit-tests/u-trailer.c', 'unit-tests/u-urlmatch-normalization.c', 'unit-tests/u-utf8-width.c', + 'unit-tests/u-utf8-ansi.c', ] clar_sources = [ diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c new file mode 100644 index 0000000000..c19d8bce78 --- /dev/null +++ b/t/unit-tests/u-utf8-ansi.c @@ -0,0 +1,14 @@ +#include "unit-test.h" +#include "utf8.h" +#include "strbuf.h" + +/* + * Test len limited functions for ANSI SGR lengths. + */ +void test_utf8_ansi__sgr_len(void) +{ + cl_assert_equal_i(0, display_mode_esc_sequence_len_counted("\033[m", 2)); + cl_assert_equal_i(3, display_mode_esc_sequence_len_counted("\033[m", 3)); + cl_assert_equal_i(5, display_mode_esc_sequence_len_counted("\033[31m", 5)); + cl_assert_equal_i(7, display_mode_esc_sequence_len_counted("\033[0;31m", 7)); +} Regards, Dirk > + if (end <= p || *p++ != '\033') > return 0; > - if (*p++ != '[') > + if (end <= p || *p++ != '[') > return 0; > - while (isdigit(*p) || *p == ';') > + while (p < end && (isdigit(*p) || *p == ';')) > p++; > - if (*p++ != 'm') > + if (end <= p || *p++ != 'm') > return 0; > return p - s; > } > > +/* Same as above, but for a NUL-terminated string "s". */ > +size_t display_mode_esc_sequence_len(const char *s) > +{ > + return display_mode_esc_sequence_len_counted(s, strlen(s)); > +} > + > /* auxiliary function for binary search in interval table */ > static int bisearch(ucs_char_t ucs, const struct interval *table, int max) > { > @@ -218,7 +231,8 @@ int utf8_strnwidth(const char *string, size_t len, int skip_ansi) > size_t skip; > > while (skip_ansi && > - (skip = display_mode_esc_sequence_len(string)) != 0) > + (skip = display_mode_esc_sequence_len_counted(string, > + orig + len - string)) != 0) > string += skip; > > glyph_width = utf8_width(&string, NULL); > @@ -279,6 +293,7 @@ void strbuf_add_wrapped_text(struct strbuf *buf, > { > int indent, w, assume_utf8 = 1; > const char *bol, *space, *start = text; > + const char *end = text + strlen(text); > size_t orig_len = buf->len; > > if (width <= 0) { > @@ -299,7 +314,8 @@ void strbuf_add_wrapped_text(struct strbuf *buf, > char c; > size_t skip; > > - while ((skip = display_mode_esc_sequence_len(text))) > + while ((skip = display_mode_esc_sequence_len_counted(text, > + end - text))) > text += skip; > > c = *text; > @@ -378,7 +394,8 @@ void strbuf_utf8_replace(struct strbuf *sb_src, int pos, int width, > int glyph_width; > size_t n; > > - while ((n = display_mode_esc_sequence_len(src))) { > + while ((n = display_mode_esc_sequence_len_counted(src, > + end - src))) { > strbuf_add(&dst, src, n); > src += n; > } > diff --git a/utf8.h b/utf8.h > index cf8ecb0f21..57ca2ab270 100644 > --- a/utf8.h > +++ b/utf8.h > @@ -6,6 +6,7 @@ struct strbuf; > typedef unsigned int ucs_char_t; /* assuming 32bit int */ > > size_t display_mode_esc_sequence_len(const char *s); > +size_t display_mode_esc_sequence_len_counted(const char *s, size_t len); > int utf8_width(const char **start, size_t *remainder_p); > int utf8_strnwidth(const char *string, size_t len, int skip_ansi); > int utf8_strwidth(const char *string); ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-17 21:26 ` Dirk Gouders @ 2026-11-17 21:43 ` Junio C Hamano 2026-11-17 22:03 ` Dirk Gouders 0 siblings, 1 reply; 15+ messages in thread From: Junio C Hamano @ 2026-11-17 21:43 UTC (permalink / raw) To: Dirk Gouders; +Cc: git Dirk Gouders <dirk@gouders.net> writes: > You have better knowledge of the users of this function: would an early > return perhaps handle a significant amount of cases? > > /* The shortest SGR sequence is "\033[m". */ > if (len < 3) > return 0; Do you have any benchmarks? I don't. I find it of dubious performance merit to run this comparison for a long string as it is eaten one UTF-8 characters at a time by the caller, and when the length is actually shorter than 3, then runnning the remainder of the function for a few steps should not be 10x more expensive than the above comparison, no? > And, why not add some tests (the untested function I showed had > an off-by-one error): Sounds good. Should I just steal what you sent out? > diff --git a/Makefile b/Makefile > index 4c84eea6e2..3c8dc9ca40 100644 > --- a/Makefile > +++ b/Makefile > @@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec > CLAR_TEST_SUITES += u-trailer > CLAR_TEST_SUITES += u-urlmatch-normalization > CLAR_TEST_SUITES += u-utf8-width > +CLAR_TEST_SUITES += u-utf8-ansi > CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X) > CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES)) > CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o > diff --git a/t/meson.build b/t/meson.build > index 1cf1d95f70..e224df58c8 100644 > --- a/t/meson.build > +++ b/t/meson.build > @@ -27,6 +27,7 @@ clar_test_suites = [ > 'unit-tests/u-trailer.c', > 'unit-tests/u-urlmatch-normalization.c', > 'unit-tests/u-utf8-width.c', > + 'unit-tests/u-utf8-ansi.c', > ] > > clar_sources = [ > diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c > new file mode 100644 > index 0000000000..c19d8bce78 > --- /dev/null > +++ b/t/unit-tests/u-utf8-ansi.c > @@ -0,0 +1,14 @@ > +#include "unit-test.h" > +#include "utf8.h" > +#include "strbuf.h" > + > +/* > + * Test len limited functions for ANSI SGR lengths. > + */ > +void test_utf8_ansi__sgr_len(void) > +{ > + cl_assert_equal_i(0, display_mode_esc_sequence_len_counted("\033[m", 2)); > + cl_assert_equal_i(3, display_mode_esc_sequence_len_counted("\033[m", 3)); > + cl_assert_equal_i(5, display_mode_esc_sequence_len_counted("\033[31m", 5)); > + cl_assert_equal_i(7, display_mode_esc_sequence_len_counted("\033[0;31m", 7)); > +} Thanks. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-17 21:43 ` Junio C Hamano @ 2026-11-17 22:03 ` Dirk Gouders 2026-11-18 1:33 ` Junio C Hamano 0 siblings, 1 reply; 15+ messages in thread From: Dirk Gouders @ 2026-11-17 22:03 UTC (permalink / raw) To: Junio C Hamano; +Cc: git On Tue, Nov 17 2026, Junio C Hamano wrote: > Dirk Gouders <dirk@gouders.net> writes: > >> You have better knowledge of the users of this function: would an early >> return perhaps handle a significant amount of cases? >> >> /* The shortest SGR sequence is "\033[m". */ >> if (len < 3) >> return 0; > > Do you have any benchmarks? I don't. > > I find it of dubious performance merit to run this comparison for a > long string as it is eaten one UTF-8 characters at a time by the > caller, and when the length is actually shorter than 3, then > runnning the remainder of the function for a few steps should not be > 10x more expensive than the above comparison, no? Meanwhile, I thought: even if the test would be a good idea, the "\033"-test probably is the one that handles most cases. Maybe I will look closer, but not now. Correctness is more important and I need to work on v2. >> And, why not add some tests (the untested function I showed had >> an off-by-one error): > > Sounds good. Should I just steal what you sent out? That wouldn't be stealing, of course you can use it. Regards, Dirk >> diff --git a/Makefile b/Makefile >> index 4c84eea6e2..3c8dc9ca40 100644 >> --- a/Makefile >> +++ b/Makefile >> @@ -1564,6 +1564,7 @@ CLAR_TEST_SUITES += u-strvec >> CLAR_TEST_SUITES += u-trailer >> CLAR_TEST_SUITES += u-urlmatch-normalization >> CLAR_TEST_SUITES += u-utf8-width >> +CLAR_TEST_SUITES += u-utf8-ansi >> CLAR_TEST_PROG = $(UNIT_TEST_BIN)/unit-tests$(X) >> CLAR_TEST_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(CLAR_TEST_SUITES)) >> CLAR_TEST_OBJS += $(UNIT_TEST_DIR)/clar/clar.o >> diff --git a/t/meson.build b/t/meson.build >> index 1cf1d95f70..e224df58c8 100644 >> --- a/t/meson.build >> +++ b/t/meson.build >> @@ -27,6 +27,7 @@ clar_test_suites = [ >> 'unit-tests/u-trailer.c', >> 'unit-tests/u-urlmatch-normalization.c', >> 'unit-tests/u-utf8-width.c', >> + 'unit-tests/u-utf8-ansi.c', >> ] >> >> clar_sources = [ >> diff --git a/t/unit-tests/u-utf8-ansi.c b/t/unit-tests/u-utf8-ansi.c >> new file mode 100644 >> index 0000000000..c19d8bce78 >> --- /dev/null >> +++ b/t/unit-tests/u-utf8-ansi.c >> @@ -0,0 +1,14 @@ >> +#include "unit-test.h" >> +#include "utf8.h" >> +#include "strbuf.h" >> + >> +/* >> + * Test len limited functions for ANSI SGR lengths. >> + */ >> +void test_utf8_ansi__sgr_len(void) >> +{ >> + cl_assert_equal_i(0, display_mode_esc_sequence_len_counted("\033[m", 2)); >> + cl_assert_equal_i(3, display_mode_esc_sequence_len_counted("\033[m", 3)); >> + cl_assert_equal_i(5, display_mode_esc_sequence_len_counted("\033[31m", 5)); >> + cl_assert_equal_i(7, display_mode_esc_sequence_len_counted("\033[0;31m", 7)); >> +} > > Thanks. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-17 22:03 ` Dirk Gouders @ 2026-11-18 1:33 ` Junio C Hamano 2026-11-18 7:37 ` Dirk Gouders 0 siblings, 1 reply; 15+ messages in thread From: Junio C Hamano @ 2026-11-18 1:33 UTC (permalink / raw) To: Dirk Gouders; +Cc: git Dirk Gouders <dirk@gouders.net> writes: > Meanwhile, I thought: even if the test would be a good idea, the > "\033"-test probably is the one that handles most cases. > > Maybe I will look closer, but not now. Correctness is more important > and I need to work on v2. It would be nice if you can take this patch over and make it your [PATCH v2 1/2]. Thanks. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-18 1:33 ` Junio C Hamano @ 2026-11-18 7:37 ` Dirk Gouders 0 siblings, 0 replies; 15+ messages in thread From: Dirk Gouders @ 2026-11-18 7:37 UTC (permalink / raw) To: Junio C Hamano; +Cc: git On Tue, Nov 17 2026, Junio C Hamano wrote: > Dirk Gouders <dirk@gouders.net> writes: > >> Meanwhile, I thought: even if the test would be a good idea, the >> "\033"-test probably is the one that handles most cases. >> >> Maybe I will look closer, but not now. Correctness is more important >> and I need to work on v2. > > It would be nice if you can take this patch over and make it your > [PATCH v2 1/2]. OK, I will do so, thank you. Regards, Dirk ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-17 20:06 ` [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() Junio C Hamano 2026-11-17 21:26 ` Dirk Gouders @ 2026-11-17 23:08 ` Andy Koppe 2026-11-17 23:25 ` Junio C Hamano 1 sibling, 1 reply; 15+ messages in thread From: Andy Koppe @ 2026-11-17 23:08 UTC (permalink / raw) To: Junio C Hamano; +Cc: git, Dirk Gouders Junio C Hamano wrote: > - while (isdigit(*p) || *p == ';') > + while (p < end && (isdigit(*p) || *p == ';')) While you're there, could you add colons (':') to that? ECMA-48 §5.4.2 defines them as sub-separators within the semicolon-separated parameters of control sequences. Colons are referred to as "3/10" there, aka ASCII 0x3A: https://ecma-international.org/wp-content/uploads/ECMA-48_5th_edition_june_1991.pdf#page=26 Maybe more practically relevant, xterm and its many more-or-less-compatible brethren accept colons in SGR sequences: "xterm allows either colons (standard) or semicolons (legacy) to separate the subparameters (but after the first colon, colons must be used)" https://invisible-island.net/xterm/ctlseqs/ctlseqs.html Andy ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-17 23:08 ` Andy Koppe @ 2026-11-17 23:25 ` Junio C Hamano 2026-11-18 6:59 ` Andy Koppe 0 siblings, 1 reply; 15+ messages in thread From: Junio C Hamano @ 2026-11-17 23:25 UTC (permalink / raw) To: Andy Koppe; +Cc: git, Dirk Gouders Andy Koppe <andy.koppe@gmail.com> writes: > Junio C Hamano wrote: > >> - while (isdigit(*p) || *p == ';') >> + while (p < end && (isdigit(*p) || *p == ';')) > > While you're there, could you add colons (':') to that? Do we generate such a sequence that uses subseparators? IIRC, this code is not counting the display width of end-user supplied string. Rather, it gets string we paint in a buffer and then counts how much display column it occupies. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-17 23:25 ` Junio C Hamano @ 2026-11-18 6:59 ` Andy Koppe 2026-11-18 7:43 ` Dirk Gouders 0 siblings, 1 reply; 15+ messages in thread From: Andy Koppe @ 2026-11-18 6:59 UTC (permalink / raw) To: Junio C Hamano; +Cc: git, Dirk Gouders Junio C Hamano wrote: > > Andy Koppe <andy.koppe@gmail.com> writes: > > > Junio C Hamano wrote: > > > >> - while (isdigit(*p) || *p == ';') > >> + while (p < end && (isdigit(*p) || *p == ';')) > > > > While you're there, could you add colons (':') to that? > > Do we generate such a sequence that uses subseparators? > > IIRC, this code is not counting the display width of end-user > supplied string. Rather, it gets string we paint in a buffer and > then counts how much display column it occupies. You're right that Git itself doesn't generate them (although future enhancements might change that). However, display_mode_esc_sequence_len() already operates on expanded custom formats and message text, e.g. in rewrap_message_tail() via strbuf_wrap(). And on stdin in git-column. They could all contain SGR sequences with colons, whose width currently isn't ignored as intended. (And my proposed make_line_colors_self_contained() wouldn't reset or replay them.) The check could be changed to `*p >= '0' && *p<=';'` to cheaply include colons. Andy ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-18 6:59 ` Andy Koppe @ 2026-11-18 7:43 ` Dirk Gouders 2026-11-18 9:48 ` Andy Koppe 0 siblings, 1 reply; 15+ messages in thread From: Dirk Gouders @ 2026-11-18 7:43 UTC (permalink / raw) To: Andy Koppe; +Cc: Junio C Hamano, git On Wed, Nov 18 2026, Andy Koppe wrote: > Junio C Hamano wrote: >> >> Andy Koppe <andy.koppe@gmail.com> writes: >> >> > Junio C Hamano wrote: >> > >> >> - while (isdigit(*p) || *p == ';') >> >> + while (p < end && (isdigit(*p) || *p == ';')) >> > >> > While you're there, could you add colons (':') to that? >> >> Do we generate such a sequence that uses subseparators? >> >> IIRC, this code is not counting the display width of end-user >> supplied string. Rather, it gets string we paint in a buffer and >> then counts how much display column it occupies. > > You're right that Git itself doesn't generate them (although future > enhancements might change that). > > However, display_mode_esc_sequence_len() already operates on expanded > custom formats and message text, e.g. in rewrap_message_tail() via > strbuf_wrap(). And on stdin in git-column. They could all contain SGR > sequences with colons, whose width currently isn't ignored as > intended. (And my proposed make_line_colors_self_contained() wouldn't > reset or replay them.) > > The check could be changed to `*p >= '0' && *p<=';'` to cheaply include colons. I would suggest to not mix these changes simply for bisection purposes. And as far as I know, we would need an FSM to identify valid sequences if we are going to allow colons. Regards, Dirk ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-18 7:43 ` Dirk Gouders @ 2026-11-18 9:48 ` Andy Koppe 2026-11-18 18:38 ` Dirk Gouders 0 siblings, 1 reply; 15+ messages in thread From: Andy Koppe @ 2026-11-18 9:48 UTC (permalink / raw) To: Dirk Gouders; +Cc: Junio C Hamano, git Dirk Gouders <wrote: > > On Wed, Nov 18 2026, Andy Koppe wrote: > > > Junio C Hamano wrote: > >> > >> Andy Koppe <andy.koppe@gmail.com> writes: > >> > >> > Junio C Hamano wrote: > >> > > >> >> - while (isdigit(*p) || *p == ';') > >> >> + while (p < end && (isdigit(*p) || *p == ';')) > >> > > >> > While you're there, could you add colons (':') to that? > >> > >> Do we generate such a sequence that uses subseparators? > >> > >> IIRC, this code is not counting the display width of end-user > >> supplied string. Rather, it gets string we paint in a buffer and > >> then counts how much display column it occupies. > > > > You're right that Git itself doesn't generate them (although future > > enhancements might change that). > > > > However, display_mode_esc_sequence_len() already operates on expanded > > custom formats and message text, e.g. in rewrap_message_tail() via > > strbuf_wrap(). And on stdin in git-column. They could all contain SGR > > sequences with colons, whose width currently isn't ignored as > > intended. (And my proposed make_line_colors_self_contained() wouldn't > > reset or replay them.) > > > > The check could be changed to `*p >= '0' && *p<=';'` to cheaply include colons. > > I would suggest to not mix these changes simply for bisection purposes. Fair, and there's testing work as well of course. It was silly of me to suggest just sneaking that change in. > And as far as I know, we would need an FSM to identify valid sequences > if we are going to allow colons. No. The parameter field can be any mix of digits, colons and semicolons. Semicolons separate parameters, colons separate subparameters, and individual (sub)parameters can be empty, which is equivalent to zero. If the field consists of nothing but colons, that's one parameter with multiple empty subparameters. Whether any particular SGR sequence means anything to a terminal is another matter of course, but git doesn't try to determine that for colon-less sequences either, and nor should it. Well-behaved terminals ignore valid sequences they don't understand. Andy ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-18 9:48 ` Andy Koppe @ 2026-11-18 18:38 ` Dirk Gouders 2026-11-18 19:12 ` Andy Koppe 0 siblings, 1 reply; 15+ messages in thread From: Dirk Gouders @ 2026-11-18 18:38 UTC (permalink / raw) To: Andy Koppe; +Cc: Junio C Hamano, git On Wed, Nov 18 2026, Andy Koppe wrote: >> Dirk Gouders <wrote: >> And as far as I know, we would need an FSM to identify valid sequences >> if we are going to allow colons. > No. The parameter field can be any mix of digits, colons and > semicolons. Semicolons separate parameters, colons separate > subparameters, and individual (sub)parameters can be empty, which is > equivalent to zero. If the field consists of nothing but colons, > that's one parameter with multiple empty subparameters. My statement was a bit misleading, because display_mode_esc_sequence_len() already is an FSM accepting the language "\033\[[0-9;]+m". My concern was that I doubt that simply adding ":" to the alphabet and as valid input to the state "parameter" (without introducing one or more new states) would still let it accept only valid SGR sequences. It seems I can learn something: is "\033[:;m" a valid SGR sequence? But, such a discussion is difficult without defining what language you want to accept. I guess the two links were meant as a definition and I have to confess I didn't yet read them. Regards, Dirk ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-18 18:38 ` Dirk Gouders @ 2026-11-18 19:12 ` Andy Koppe 2026-11-18 19:29 ` Dirk Gouders 0 siblings, 1 reply; 15+ messages in thread From: Andy Koppe @ 2026-11-18 19:12 UTC (permalink / raw) To: Dirk Gouders; +Cc: Junio C Hamano, git Dirk Gouders wrote: > > On Wed, Nov 18 2026, Andy Koppe wrote: > >> And as far as I know, we would need an FSM to identify valid sequences > >> if we are going to allow colons. > > > No. The parameter field can be any mix of digits, colons and > > semicolons. Semicolons separate parameters, colons separate > > subparameters, and individual (sub)parameters can be empty, which is > > equivalent to zero. If the field consists of nothing but colons, > > that's one parameter with multiple empty subparameters. > > My statement was a bit misleading, because > display_mode_esc_sequence_len() already is an FSM > accepting the language "\033\[[0-9;]+m". My concern was that I doubt > that simply adding ":" to the alphabet and as valid input to the state > "parameter" (without introducing one or more new states) would still > let it accept only valid SGR sequences. "\033\[[0-9:;]*m" is the complete regex for SGRs. (Of course, it's not that simple if you wanted to actually collect the (sub)parameter values, but git doesn't need to.) > It seems I can learn something: is "\033[:;m" a valid SGR sequence? Yep. That's two parameters, with the first one consisting of two empty subparameters, and the second one an empty parameter. > But, such a discussion is difficult without defining what language you > want to accept. I guess the two links were meant as a definition and I > have to confess I didn't yet read them. To be fair, ECMA-48 isn't the most readable, but once you realise that 3/0 stands for 0x30 (aka '0') and 3/11 for 0x3B (aka ';'), it's not too bad. Andy ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() 2026-11-18 19:12 ` Andy Koppe @ 2026-11-18 19:29 ` Dirk Gouders 0 siblings, 0 replies; 15+ messages in thread From: Dirk Gouders @ 2026-11-18 19:29 UTC (permalink / raw) To: Andy Koppe; +Cc: Junio C Hamano, git On Wed, Nov 18 2026, Andy Koppe wrote: > Dirk Gouders wrote: >> >> On Wed, Nov 18 2026, Andy Koppe wrote: > >> >> And as far as I know, we would need an FSM to identify valid sequences >> >> if we are going to allow colons. >> >> > No. The parameter field can be any mix of digits, colons and >> > semicolons. Semicolons separate parameters, colons separate >> > subparameters, and individual (sub)parameters can be empty, which is >> > equivalent to zero. If the field consists of nothing but colons, >> > that's one parameter with multiple empty subparameters. >> >> My statement was a bit misleading, because >> display_mode_esc_sequence_len() already is an FSM >> accepting the language "\033\[[0-9;]+m". My concern was that I doubt >> that simply adding ":" to the alphabet and as valid input to the state >> "parameter" (without introducing one or more new states) would still >> let it accept only valid SGR sequences. > > "\033\[[0-9:;]*m" is the complete regex for SGRs. (Of course, it's not > that simple if you wanted to actually collect the (sub)parameter > values, but git doesn't need to.) > >> It seems I can learn something: is "\033[:;m" a valid SGR sequence? > > Yep. That's two parameters, with the first one consisting of two empty > subparameters, and the second one an empty parameter. OK, I didn't know that, thank you. >> But, such a discussion is difficult without defining what language you >> want to accept. I guess the two links were meant as a definition and I >> have to confess I didn't yet read them. > > To be fair, ECMA-48 isn't the most readable, but once you realise that > 3/0 stands for 0x30 (aka '0') and 3/11 for 0x3B (aka ';'), it's not > too bad. That notation is fine for me, I will definitely read that document. Regards, Dirk ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2026-11-18 19:29 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-17 16:50 [PATCH] utf8: make display_mode_esc_sequence_len() take a length Junio C Hamano 2026-11-17 20:06 ` [PATCH v2] utf8: add display_mode_esc_sequence_len_counted() Junio C Hamano 2026-11-17 21:26 ` Dirk Gouders 2026-11-17 21:43 ` Junio C Hamano 2026-11-17 22:03 ` Dirk Gouders 2026-11-18 1:33 ` Junio C Hamano 2026-11-18 7:37 ` Dirk Gouders 2026-11-17 23:08 ` Andy Koppe 2026-11-17 23:25 ` Junio C Hamano 2026-11-18 6:59 ` Andy Koppe 2026-11-18 7:43 ` Dirk Gouders 2026-11-18 9:48 ` Andy Koppe 2026-11-18 18:38 ` Dirk Gouders 2026-11-18 19:12 ` Andy Koppe 2026-11-18 19:29 ` Dirk Gouders
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox