Re: [PATCH 1/5] string: add strtok/strtokv

[Tobias Waldekranz <tobias@waldekranz.com>]

On fre, sep 05, 2025 at 18:28, Ahmad Fatoum <a.fatoum@pengutronix.de> wrote:
> Hi,
>
> On 9/4/25 3:35 PM, Tobias Waldekranz wrote:
>> On tor, sep 04, 2025 at 13:00, Ahmad Fatoum <a.fatoum@pengutronix.de> wrote:
>>> Hello Tobias,
>>>
>>> On 8/28/25 5:05 PM, Tobias Waldekranz wrote:
>>>> Add an implementation of libc's standard strtok(3), which is useful
>>>> for tokenizing strings.
>>>
>>> strtok was previously removed in favor of strsep as it doesn't suffer
>>> from re-entrancy issues (poller and bthreads can run during delays). If
>>> you want to allow escapes, there's also strsep_unescaped.
>> 
>> Aha, my bad. I did not realize that there was more than one thread of
>> execution.
>
> The pollers are run during delay loops for stuff like feeding a
> watchdog, blinking a heartbeat LED or polling network link state.
> Bthreads are currently only used for the USB mass storage gadget[1] and
> out-of-tree for baredoom, so it can be played while booting..

V-e-r-y cool way to demo Barebox, BTW :)

>> strsep() is not quite the same thing though, I am really after the
>> strtok()'s behavior of skipping empty tokens.
>
> Ah, right. strsep is used in a loop, where it's just an extra check to
> skip empty tokens.
>
>> How would you feel about adding strtok_r() instead?
>
> You are not using anyways though, so what does it matter compared to
>
> while ((token = strsep(&sep, delims))) {
>         if (!*token)
>                 continue;
>
> ?

I just thought it might be a good hint for future readers: "we're
following strtok() semantics here". Anyway, it is not important, I'll
drop it in v2.

>>>> Also, add a version that will collect all tokens from a string into an
>>>> array, which is useful in situations where you need to know how many
>>>> tokens there are, and when a token's relative position in the order is
>>>> significant.
>>>
>>> We have the inverse as strjoin, but not this. Maybe call it strsplit
>>> instead?
>> 
>> If you accept my strtok_r() suggestion, do you still think strsplit() is
>> a better name, or is there value in signaling the underlying strtok()
>> behavior?
>
> I can see the argument that strjoin(strsplit(s)) should be s.
>
> Ok, let's keep it at strtokv. Some final bikeshedding: Would it be
> cleaner to return the string argument and have the length be the pointer
> argument?

Agreed, the array does seem like the "primary" return value. I'll swap
them for v2.

> [1]: Implementing stackful coroutines was less of a hassle than
> rewriting a complex state machine implemented as a kthread..
>
> Cheers,
> Ahmad
>
>> 
>>> Cheers,
>>> Ahmad
>>>
>>>>
>>>> Signed-off-by: Tobias Waldekranz <tobias@waldekranz.com>
>>>> ---
>>>>  include/string.h |  2 ++
>>>>  lib/string.c     | 66 ++++++++++++++++++++++++++++++++++++++++++++++++
>>>>  2 files changed, 68 insertions(+)
>>>>
>>>> diff --git a/include/string.h b/include/string.h
>>>> index 71affe48b6..c8df8540d8 100644
>>>> --- a/include/string.h
>>>> +++ b/include/string.h
>>>> @@ -8,6 +8,8 @@
>>>>  void *mempcpy(void *dest, const void *src, size_t count);
>>>>  int strtobool(const char *str, int *val);
>>>>  char *strsep_unescaped(char **, const char *, char *);
>>>> +char *strtok(char *str, const char *delim);
>>>> +int strtokv(char *str, const char *delim, char ***vecp);
>>>>  char *stpcpy(char *dest, const char *src);
>>>>  bool strends(const char *str, const char *postfix);
>>>>  
>>>> diff --git a/lib/string.c b/lib/string.c
>>>> index 73637cd971..be7e65eb45 100644
>>>> --- a/lib/string.c
>>>> +++ b/lib/string.c
>>>> @@ -593,6 +593,72 @@ char *strsep_unescaped(char **s, const char *ct, char *delim)
>>>>          return sbegin;
>>>>  }
>>>>  
>>>> +/**
>>>> + * strtok - extract tokens from string
>>>> + * @str:	string to split
>>>> + * @delim:	set of delimiter characters
>>>> + *
>>>> + * The strtok() function breaks up a string into zero or more nonempty
>>>> + * tokens.  On the first call, the string to be parsed should be
>>>> + * specified in @str.  In each subsequent call that should parse the
>>>> + * same string, @str must be NULL.
>>>> + *
>>>> + * @delim specifies a set of bytes that delimit the tokens in the
>>>> + * string.
>>>> + *
>>>> + * Each call to strtok() returns a pointer to a string containing the
>>>> + * next token.  This is done by replacing the first delimiter with a
>>>> + * NUL character, the operation is thus destructive to the string. If
>>>> + * no more tokens are found, strtok() returns NULL.
>>>> + */
>>>> +char *strtok(char *str, const char *delim)
>>>> +{
>>>> +	static char *cursor;
>>>> +
>>>> +	if (str)
>>>> +		cursor = str;
>>>> +
>>>> +	if (!cursor)
>>>> +		return NULL;
>>>> +
>>>> +	cursor += strspn(cursor, delim);
>>>> +	if (*cursor == '\0') {
>>>> +		cursor = NULL;
>>>> +		return NULL;
>>>> +	}
>>>> +
>>>> +	return strsep(&cursor, delim);
>>>> +}
>>>> +EXPORT_SYMBOL(strtok);
>>>> +
>>>> +/**
>>>> + * strtokv - split string into array of tokens based on a delimiter set
>>>> + * @str:	string to split
>>>> + * @delim:	set of delimiter characters
>>>> + * @vecp:	array of tokens
>>>> + *
>>>> + * Split @str into tokens delimited by @delim, using strtok(), and
>>>> + * store the allocated token array in @vecp, which the caller is
>>>> + * responsible for freeing.
>>>> + *
>>>> + * Return: The number of tokens in the array.
>>>> + */
>>>> +int strtokv(char *str, const char *delim, char ***vecp)
>>>> +{
>>>> +	char *tok, **vec = NULL;
>>>> +	int cnt = 0;
>>>> +
>>>> +
>>>> +	for (tok = strtok(str, delim); tok; tok = strtok(NULL, delim)) {
>>>> +		vec = xrealloc(vec, (cnt + 1) * sizeof(*vec));
>>>> +		vec[cnt++] = tok;
>>>> +	}
>>>> +
>>>> +	*vecp = vec;
>>>> +	return cnt;
>>>> +}
>>>> +EXPORT_SYMBOL(strtokv);
>>>> +
>>>>  #ifndef __HAVE_ARCH_STRSWAB
>>>>  /**
>>>>   * strswab - swap adjacent even and odd bytes in %NUL-terminated string
>>>
>>> -- 
>>> Pengutronix e.K.                  |                             |
>>> Steuerwalder Str. 21              | http://www.pengutronix.de/  |
>>> 31137 Hildesheim, Germany         | Phone: +49-5121-206917-0    |
>>> Amtsgericht Hildesheim, HRA 2686  | Fax:   +49-5121-206917-5555 |
>> 
>
> -- 
> Pengutronix e.K.                  |                             |
> Steuerwalder Str. 21              | http://www.pengutronix.de/  |
> 31137 Hildesheim, Germany         | Phone: +49-5121-206917-0    |
> Amtsgericht Hildesheim, HRA 2686  | Fax:   +49-5121-206917-5555 |