Re: alx-0096r1 - string (and nonstring) copying

[Alejandro Colomar <alx@kernel.org>]

[-- Attachment #1: Type: text/plain, Size: 15971 bytes --]

Hi Mark,

> Date: 2026-07-29 10:27:15-0700
> From: Mark Harris <mark.hsj@gmail.com>
>
>  Alejandro Colomar wrote:
> > >  4.3 BSD (1986) then also added the mem*()
> > > functions for System V compatibility.  Later the C89 committee chose
> > > to standardize the mem*() functions but moved them from their own
> > > header file memory.h (introduced by System V) to string.h with the
> > > str*() functions.
> >
> > Wow!  This is very interesting.  So, after all, we shouldn't blame old
> > Unix systems, but the C Committee!!  Another very important issue to add
> > to the list of bad inventions of the committee.  :)
> >
> > I wish we could re-introduce the <memory.h> header, and move these back
> > there (of course, with compat declarations in <string.h>).
> 
> Since you propose to add functions like memtostr() and strtomem(), I
> would think that having them both in the same header file would be
> beneficial.  Otherwise it would add yet another area of potential
> confusion for users:  Do I need to include memory.h or string.h for
> strtomem()?

Actually, I'd use a third header file for these hybrids: <nonstring.h>.
All of the current strn*() functions belong there.
They don't belong in either <memory.h> or <string.h>.

>  Does it depend on the source or destination argument?  I
> thought the goal was to reduce confusion.
> 
> > > That is, memtostr(str, mem, n), introducing a whole
> > > new area of potential confusion that was not present in any existing
> > > mem*() or str*() functions.  If reducing potential confusion is that
> > > important I would have expected maybe strfrommem(str, mem, n) or
> > > similar.
> >
> > Hmmmm, interesting suggestion!  That makes it resemble the strfromd(3)
> > family, with which it actually has some (minor) relation.  (It might be
> > a bit
> >
> > strfrommemcat() would be a weird name, because the cat is next to mem
> > instead of str.  Should it be strcatfrommem()?
> 
> I think that strcatfrommem() is better than strfrommemcat().

Agree.

>  Although
> I am not convinced that it is worth changing the name of a
> well-established 49-year-old function just because the name may
> confuse some people, especially if it may end up just exchanging one
> area of confusion for a different one that people haven't even yet
> learned to watch out for.
> 
> > On the other hand, there's precedent in strtol(3) having the arguments
> > reversed compared to the name.  Programmers are used to having the
> > destination in the first parameter.
> 
> strtol() is fine, because str is first in the name and is also the
> first argument.

Ouch, I was wrong.  Every time I read the strtol() manual page, I get
confused by the 'nptr' argument, which looks like it'd be a pointer in
which to store the number.  I'll change it to 's'.

>  What makes memtostr() confusing is that mem is
> mentioned first in the name but it is the second argument, and str is
> mentioned second but it is the first argument.
> 
> > [...]
> > > >     shadow; strtcpy(), strncpy(3)
> > > >         The shadow project more or less agrees with the Linux kernel in
> > > >         this regard.  It has implemented strtcpy(), and uses strncpy(3).
> > > >         It doesn't have memtostr() yet, because it happens to always
> > > >         allocate the buffer at the same time, using strndup(3) --which
> > > >         is essentially malloc(3)+memtostr()--, but that will change
> > > >         soon, because (non-VLA) arrays are safer, as they allow
> > > >         validation of source and destination sizes at compile time.
> > > >
> > > >         The implementations are independent, and happened to be
> > > >         fundamentally identical.  It seems that good implementations of
> > > >         string and nonstring copying code converge to this set of
> > > >         interfaces.
> > > >
> > > >         Here's a naive implementation of strtcpy():
> > > >
> > > >                 ssize_t
> > > >                 strtcpy(char *restrict dst, const char *restrict src, size_t dsize)
> > > >                 {
> > > >                         bool    trunc;
> > > >                         size_t  dlen, slen;
> > > >
> > > >                         if (dsize == 0)  // UB
> > > >                                 abort();
> > > >
> > > >                         slen = strnlen(src, dsize);
> > > >                         trunc = (slen == dsize);
> > > >                         dlen = slen - trunc;
> > > >
> > > >                         stpcpy(mempcpy(dst, src, dlen), "");
> > > >
> > > >                         if (trunc) {
> > > >                                 errno = E2BIG;
> > > >                                 return -1;
> > > >                         }
> > > >
> > > >                         return slen;
> > > >                 }
> > >
> > > ISO C does not have ssize_t or E2BIG.
> >
> > I'm aware; this is a push to add ssize_t.  In the wording, I have
> > specified strtcpy() as not-necessarily setting errno, though, since it's
> > not strictly necessary.
> >
> > >  Also this API does not handle
> > > the case of slen too large for ssize_t, which may not be an issue for
> > > Linux, however ISO C has to support a much wider range of systems.
> > > The same issues apply to strtcat().
> >
> > I believe sizes beyond ssize_t are not valid.  At least, for a ssize_t
> > that matches ptrdiff_t.  No object may be larger than PTRDIFF_MAX,
> > because that would mean that pointer subtraction would be undefined:
> >
> >         char *first = a;
> >         char *end = a + sizeof(a);
> >
> >         end - first;  // Overflows ptrdiff_t
> >
> > That's because pointer subtraction happens on ptrdiff_t instead of
> > size_t.  Thus, we don't need to care about sizes larger than SSIZE_MAX.
> 
> ISO C allows objects to have a size up to SIZE_MAX (§ 6.2.5).  ISO C
> does not limit objects to size PTRDIFF_MAX, it just says that the
> pointer subtraction is undefined if the result is not representable in
> an object of type ptrdiff_t (§ 6.5.7).  It is the responsibility of
> your code to avoid undefined behavior.  Glibc malloc() deliberately
> fails on sizes > PTRDIFF_MAX to help clients to avoid this undefined
> behavior, but that is not an ISO C requirement and it is still
> possible to obtain such an object in other ways.

Yup.

>  Also some systems
> (e.g. with segmented or banked memory models) have a ptrdiff_t that is
> larger than size_t.

I'm interested in those systems.  Do they support POSIX.1-2024 and/or
ISO C23?  We were discussing the possibility of standardizing ssize_t in
ISO C, and also adding a requirement that it's as large as size_t, and
also potentially adding a requirement that it's the same type as
ptrdiff_t.

It would be good to know if those systems that have a larger ptrdiff_t
are still alive or can be ignored/forced to adapt.

> > [...]
> > > >     System V; memccpy(3)
> > > >         memccpy(3) was invented in System V (like the other mem*()
> > > >         functions).  The System V sources are not public (AFAIK), so
> > > >         it's not known what this function was designed for.  It was
> > > >         later added to the BSDs and glibc for compatibility with SysV,
> > > >         but nobody really knew what this function is good for.  It
> > > >         doesn't seem ergonomic for any basic functionality.
> > >
> > > It is for implementing functions like fgets(), which needs to copy the
> > > next '\n'-delimited line from a buffer filled by read() (so not a
> > > null-terminated string).
> >
> > Hmmmm, do you have any such code?  I'd be interested in reviewing it.
> > I still suspect it might have been better if it didn't copy the
> > terminator.
> 
> Sure.
> https://github.com/illumos/illumos-gate/blob/4b44494cae63d4bf0c7d3f34828503e68d1c0e69/usr/src/lib/libc/port/stdio/fgets.c#L44

Thanks!

Let's now consider an alternative world where there was a function with
semantics similar to memccpy(3) with one difference: it wouldn't copy
the delimiter.  Let's call such a function memcpyc().  Here's a naive
implementation of memccpy(3):

	void *
	memccpy(void *dst, const void *src, int c, size_t n)
	{
		unsigned char  *d = dst;
		unsigned char  *s = src;

		for (size_t i = 0; i < n; i++) {
			*d++ = *s++;
			if (*s == c)
				return d;
		}
		return NULL;
	}

Here's a naive implementation of the hypothetical memcpyc():

	void *
	memcpyc(void *dst, const void *src, int c, size_t n)
	{
		unsigned char  *d = dst;
		unsigned char  *s = src;

		for (size_t i = 0; i < n; i++) {
			if (*s == c)
				return d;
			*d++ = *s++;
		}
		return NULL;
	}

(I've written both of the above quickly, without compiling nor testing;
 I might have made a mistake.)

With such a function, the fgets(3) implementation from Illumos gate
would change in this manner:

	diff --git i/usr/src/lib/libc/port/stdio/fgets.c w/usr/src/lib/libc/port/stdio/fgets.c
	index 9401217410ca..ac1f330ea46e 100644
	--- i/usr/src/lib/libc/port/stdio/fgets.c
	+++ w/usr/src/lib/libc/port/stdio/fgets.c
	@@ -82,9 +82,12 @@ fgets(char *buf, int size, FILE *iop)
					break;          /* nothing left to read */
			}
			n = (int)(size < iop->_cnt ? size : iop->_cnt);
	-               if ((p = memccpy(ptr, (char *)iop->_ptr, '\n',
	+               if ((p = memcpyc(ptr, (char *)iop->_ptr, '\n',
			    (size_t)n)) != NULL)
	+               {
	+                       p = stpcpy(p, "\n");
				n = (int)(p - ptr);
	+               }
			ptr += n;
			iop->_cnt -= n;
			iop->_ptr += n;

IMO, this wouldn't make it more complex.  However, such a hypothetical
memcpyc() would be much more useful elsewhere.  For example, it would
simplify the two unique uses of memccpy(3) in FreeBSD:

	diff --git i/bin/sh/parser.c w/bin/sh/parser.c
	index 0c1b7a91c257..713a35ad8cd8 100644
	--- i/bin/sh/parser.c
	+++ w/bin/sh/parser.c
	@@ -2116,7 +2116,7 @@ getprompt(void *unused __unused)
				if (fmt[0] != '}') {
					char *end;
	 
	-                               end = memccpy(tfmt, fmt, '}', sizeof(tfmt));
	+                               end = memcpyc(tfmt, fmt, '}', sizeof(tfmt));
					if (end == NULL) {
						/*
						 * Format too long or no '}', so
	@@ -2129,7 +2129,7 @@ getprompt(void *unused __unused)
						fmt--;
						break;
					}
	-                               *--end = '\0'; /* Ignore the copy of '}'. */
	+                               *end = '\0'; /* Ignore the copy of '}'. */
					fmt += end - tfmt;
				}
				now = localtime(&(time_t){time(NULL)});
	diff --git i/lib/libc/amd64/string/strncat.c w/lib/libc/amd64/string/strncat.c
	index 2c63ab50b3c3..4a1e8c0119ac 100644
	--- i/lib/libc/amd64/string/strncat.c
	+++ w/lib/libc/amd64/string/strncat.c
	@@ -19,13 +19,13 @@ strncat(char *dest, const char *src, size_t n)
		char *endptr;
	 
		len = strlen(dest);
	-       endptr = __memccpy(dest + len, src, '\0', n);
	+       endptr = memcpyc(dest + len, src, '\0', n);
	 
		/* avoid an extra branch */
		if (endptr == NULL)
	-               endptr = dest + len + n + 1;
	+               endptr = dest + len + n;
	 
	-       endptr[-1] = '\0';
	+       *endptr = '\0';
	 
		return (dest);
	 }

memccpy(3) is a bad design, which forces complex code, prone to
off-by-one bugs.

> > [...]
> > > >     POSIX; memccpy(3)
> > > >         POSIX standardized memccpy(3) just because it was in System V.
> > > >         POSIX derives from Issue 1 of the SVID.  It's not surprising
> > > >         that it's there.  Especially, this function was always in POSIX,
> > > >         and the early revisions of POSIX are known to have strong
> > > >         preference for SysV functions, regardless of their quality or
> > > >         widespread use.
> > > >
> > > >         Interestingly, POSIX mentions that memccpy(3) does not check for
> > > >         overflow.
> > > >
> > > >         > The memccpy() function does not check for the overflow of the
> > > >         > receiving memory area.
> > > >
> > > >         This is because the 4th parameter to memccpy(3) is not the size
> > > >         of the destination buffer, but the size of the source buffer.
> > > >         It is assumed that the destination buffer is large enough.
> > >
> > > That is silly; the maximum size applies to both the source and
> > > destination, like memcpy().  Obviously if one is smaller than the
> > > other, you must not pass in a size that is larger than the smaller of
> > > the two, as with memcpy().
> > >
> > > >
> > > >         This hints that the original (System V) authors of the function
> > > >         didn't consider copying strings as a use case for this function.
> > >
> > > It's a "mem" function, not a "str" function, so it should be clear
> > > that it is intended for processing byte arrays and not null-terminated
> > > strings.  That said, if you wanted to I guess you could use it to
> > > implement a function like your strtcpy() pretty easily:
> > >
> > >     if (dsize == 0) abort();
> > >     char *p = memccpy(dst, src, '\0', dsize);
> > >     return p ? p-dst-1 : (dst[dsize-1] = '\0', -1);
> >
> > FWIW, I wouldn't call this 'pretty easily', but this is a bit
> > subjective.  I'd possibly make bugs in the code above.
> 
> It is shorter than your "naive" strtcpy() implementation,

I can golf too, but lines of code isn't a good measure of simplicity,
unless you limit to certain common guidelines such as avoiding that
tricky comma operator in the third line.

Ignoring the abort() line, which is identical to my implementation, I'd
write it as this, for a fair comparison:

	char  *p;

	p = memccpy(dst, src, '\0', dsize);
	if (p == NULL) {
		dst[dsize-1] = '\0';
		return -1;
	}

	return p-dst-1;

That's only slightly shorter than my implementation of strtcpy() with
strnlen(3)+mempcpy(3) (ignoring also errno).

Moreover, it's full of '-1's, which give plenty of opportunities of
making a mistake (adding one '-1' too much, or forgetting to put one),
and resulting in a bug.

> and doesn't
> depend on an undocumented non-C23 function.

You mean mempcpy(3), I guess.  I am working on standardizing it.  In any
case, you can use good ol' memcpy(3).  After all, it's as simple as
this:

	void *
	mempcpy(void *dest, const void *src, size_t n)
	{
		return (char *)memcpy(dest, src, n) + n;
	}

Here's an implementation using exlusively old POSIX functions:

	ssize_t
	strtcpy(char *restrict dst, const char *restrict src, size_t dsize)
	{
		bool    trunc;
		size_t  dlen, slen;

		if (dsize == 0)  // UB
			abort();

		slen = strnlen(src, dsize);
		trunc = (slen == dsize);
		dlen = slen - trunc;

		strcpy(memcpy(dst, src, dlen) + dlen, "");

		if (trunc) {
			errno = E2BIG;
			return -1;
		}

		return slen;
	}

Now, let's do something interesting: let's see how this would be
implemented with the memcpyc() I've implemented above.

	char  *p, *nul;

	p = memcpyc(dst, src, '\0', dsize);
	nul = p ?: dst+dsize-1;
	strcpy(nul, "");
	
	return p ? p-dst : -1;

It still contains one '-1' too much for my taste, but it'd be more
acceptable.  Actually, memcpyc() would be better for implementing
stpecpy() --similar to Plan9's strecpy(2)--:

	char *
	stpecpy(char *dst, const char *end, const char *restrict src)
	{
		char  *p;

		if (dst == NULL)
			return NULL;

		p = memcpyc(dst, src, '\0', end - dst);
		strcpy(p ?: end-1, "");

		return p;
	}

And then this stpecpy() could be used for implementing strtcpy():

	ssize_t
	strtcpy(char *restrict dst, const char *restrict src, size_t dsize)
	{
		char  *p;

		p = stpecpy(dst, dst + dsize, src);
		return p ? p-dst : -1;
	}


Have a lovely day!
Alex

-- 
<https://www.alejandro-colomar.es>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]