Re: [PATCH] precompose_utf8: use a flex array for d_name

[Ihar Hrachyshka <ihar.hrachyshka@gmail.com>]

On 7/3/26 4:40 AM, Patrick Steinhardt wrote:
> On Thu, Jul 02, 2026 at 10:35:54PM -0400, Ihar Hrachyshka wrote:
>> On macOS, git status may abort while reading a directory entry
>> whose UTF-8 name grows past NAME_MAX bytes:
>>
>>    __chk_fail_overflow
>>    __strlcpy_chk
>>    precompose_utf8_readdir
>>    read_directory_recursive
>>    wt_status_collect
>>    cmd_status
>>
>> The precompose wrapper already reallocates dirent_prec_psx for
>> long names, but d_name is declared as char[NAME_MAX + 1]. A
>> fortified libc can still see that declared object size and reject a
>> larger strlcpy bound, even though the allocation was grown.
>>
>> Make d_name a FLEX_ARRAY and size allocations from offsetof(). That
>> matches the actual object layout with the dynamic allocation, so the
>> fortified copy sees a destination whose size can grow with max_name_len.
>>
>> Add a regression test that creates a 261-byte non-ASCII basename and
>> runs status with core.precomposeunicode enabled.
> Hm. Why does macOS even allow you to create a file that has a basename
> longer than NAME_MAX? Does macOS count unicode characters specially?


Yes, macOS file names can exceed NAME_MAX bytes because the real dirent 
limit in system headers is:

#define __DARWIN_MAXPATHLEN 1024

#define __DARWIN_STRUCT_DIRENTRY { \
char d_name[__DARWIN_MAXPATHLEN]; /* entry name (up to MAXPATHLEN bytes) 
*/ \
}

(for a very old 32-bit ABI it's 256 but it's not really relevant)

This in-memory limit may be further capped by file system. For HFS+, 
it's 255 16-bit Unicode characters (as per on-disk format). For APFS, 
on-disk theoretically allows up to 1022 UTF-8 bytes, but my testing 
suggests they still enforce the same 255 character limit somewhere in 
kernel API layer. (Which means that they could later expand the maximum 
filename length further without changing the on-disk format.)

So effectively, today on Darwin, the real limit is "up to 255 2-byte 
code points", not bytes. Which is potentially beyond NAME_MAX.

...that said, Linux readdir() doesn't guarantee NAME_MAX limit either. 
 From readdir(3):

"""

         Note that while the call

             fpathconf(fd, _PC_NAME_MAX)

         returns the value 255 for most filesystems, on some filesystems
         (e.g., CIFS, Windows SMB servers), the null-terminated filename
         that is (correctly) returned in .d_name can actually exceed this
         size.  In such cases, the .d_reclen field will contain a value
         that exceeds the size of the glibc dirent structure shown above.

"""


The man page also advises against using sizeof() against dirent structs. 
(Which is what we currently do - against our own MacOS helper dirent 
struct.)


As a side note, it probably means neither Darwin nor Linux readdir() is 
POSIX compliant, because, as per:

https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/dirent.h.html


"The array d_name in each of these structures is of unspecified size, 
but shall contain a filename of at most {NAME_MAX} bytes followed by a 
terminating null byte."


>> diff --git a/compat/precompose_utf8.c b/compat/precompose_utf8.c
>> index 1711794..8077f62 100644
>> --- a/compat/precompose_utf8.c
>> +++ b/compat/precompose_utf8.c
>> @@ -19,6 +19,11 @@ typedef char *iconv_ibp;
>>   static const char *repo_encoding = "UTF-8";
>>   static const char *path_encoding = "UTF-8-MAC";
>>   
>> +static size_t dirent_prec_psx_size(size_t max_name_len)
>> +{
>> +	return st_add(offsetof(dirent_prec_psx, d_name), max_name_len);
>> +}
>> +
>>   static size_t has_non_ascii(const char *s, size_t maxlen, size_t *strlen_c)
>>   {
>>   	const uint8_t *ptr = (const uint8_t *)s;
>> @@ -114,8 +119,8 @@ const char *precompose_argv_prefix(int argc, const char **argv, const char *pref
>>   PREC_DIR *precompose_utf8_opendir(const char *dirname)
>>   {
>>   	PREC_DIR *prec_dir = xmalloc(sizeof(PREC_DIR));
>> -	prec_dir->dirent_nfc = xmalloc(sizeof(dirent_prec_psx));
>> -	prec_dir->dirent_nfc->max_name_len = sizeof(prec_dir->dirent_nfc->d_name);
>> +	prec_dir->dirent_nfc = xmalloc(dirent_prec_psx_size(NAME_MAX + 1));
>> +	prec_dir->dirent_nfc->max_name_len = NAME_MAX + 1;
> We have the `FLEX_ALLOC_MEM()` macro that would probably be a better fit
> compared to introducing `dirent_prec_psx_size()`.
>
> Also, when converting this to a flex array, can't we do better here and
> allocate the structures with the right size? Otherwise, I expect that we
> overallocate most of the entrise.


As I understand it, this is a *per-directory* buffer that starts from 
NAME_MAX + 1, then gets expanded as entries with names longer than 
NAME_MAX + 1 are encountered. It is reused for next entries.


>> @@ -145,8 +150,7 @@ struct dirent_prec_psx *precompose_utf8_readdir(PREC_DIR *prec_dir)
>>   		int ret_errno = errno;
>>   
>>   		if (new_maxlen > prec_dir->dirent_nfc->max_name_len) {
>> -			size_t new_len = sizeof(dirent_prec_psx) + new_maxlen -
>> -				sizeof(prec_dir->dirent_nfc->d_name);
>> +			size_t new_len = dirent_prec_psx_size(new_maxlen);
>>   
>>   			prec_dir->dirent_nfc = xrealloc(prec_dir->dirent_nfc, new_len);
>>   			prec_dir->dirent_nfc->max_name_len = new_maxlen;
> Okay, here we indeed have to realloc though, and thus we can't quite
> avoid `dirent_prec_psx_size()`. Too bad.
>
> Thanks!
>
> Patrick