strtok - strings without delimiters

strtok - strings without delimiters

[Georg Sauthoff]

Hi,

the strtok man-page does not specify what happens if `str` does not
contain any `delim`iters.

If you don't already know strtok one might assume then either:

- the delimiter-less string is considered the next token and thus
  returned upon the first call to strtok
- on the first call NULL is returned because there are not any tokens

Related issue: is the non-empty `str`-suffix after the last `delim`
returned as a token or not?

I know how strtok works - but perhaps it makes sense to explicitly
address these issues in the Linux man page.

Best regards
Georg
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strtok - strings without delimiters

[Michael Kerrisk]

Hello Georg,

On 05/12/13 22:22, Georg Sauthoff wrote:
> Hi,
> 
> the strtok man-page does not specify what happens if `str` does not
> contain any `delim`iters.
> 
> If you don't already know strtok one might assume then either:
> 
> - the delimiter-less string is considered the next token and thus
>   returned upon the first call to strtok
> - on the first call NULL is returned because there are not any tokens
> 
> Related issue: is the non-empty `str`-suffix after the last `delim`
> returned as a token or not?

Thanks for the report, though I must say that it would have been a 
little more helpful if you had answered the rhetorical questions you 
ask... (It would have aided me as I reminded myself of the details.)

> I know how strtok works - 

In which case a patch would have made the bug report even better still!

> but perhaps it makes sense to explicitly
> address these issues in the Linux man page.

Anyway, I agree that the page could be clearer. I've added the following text:

       A sequence of calls to strtok() that operate on the same string
       maintains  a  pointer  that  determines the point from which to
       start searching for the next token.  The first call to strtok()
       sets  this  pointer  to  point to the first byte of the string.
       The start of the next token is determined by  scanning  forward
       for  the  next  nondelimiter  byte  in  str.  If such a byte is
       found, it is taken as the start of the next token.  If no  such
       byte  is  found,  then  there  are no more tokens, and strtok()
       returns NULL.  (A string that is empty or  that  contains  only
       delimiter  will thus cause strtok() to return NULL on the first
       call.)

       The end of each token is found by scanning forward until either
       the  next delimiter byte is found or until the terminating null
       byte ('\0') is encountered.  If a delimiter byte is  found,  it
       is overwritten with a null byte to terminate the current token,
       and strtok() saves  a  pointer  to  the  following  byte;  that
       pointer  will  be used as the starting point when searching for
       the next token.  In this case, strtok() returns  a  pointer  to
       the start of the found token.

       From  the above description, it follows thatt a sequence of two
       or more contiguous delimiter bytes in the parsed string is con‐
       sidered  to  be a single delimiter, and that delimiter bytes at
       the start or end of the string are ignored.  Put  another  way:
       the  tokens  returned  by strtok() are always nonempty strings.
       Thus, for example, given  the  string  "aaa;;bbb,",  successive
       calls  to strtok() that specify the delimiter string ";," would
       return the strings "aaa" and "bbb", and then a NULL pointer.

Sufficient?

Cheers,

Michael
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strtok - strings without delimiters

[Georg Sauthoff]

On Sun, May 19, 2013 at 10:46:49PM +0200, Michael Kerrisk wrote:

Hi,

> On 05/12/13 22:22, Georg Sauthoff wrote:
> > the strtok man-page does not specify what happens if `str` does not
> > contain any `delim`iters.

> > If you don't already know strtok one might assume then either:

> > - the delimiter-less string is considered the next token and thus
> >   returned upon the first call to strtok
> > - on the first call NULL is returned because there are not any tokens

> > Related issue: is the non-empty `str`-suffix after the last `delim`
> > returned as a token or not?
 
> Thanks for the report, though I must say that it would have been a 
> little more helpful if you had answered the rhetorical questions you 
> ask... (It would have aided me as I reminded myself of the details.)

oh, sorry, you are right.

> > I know how strtok works - 
>
> In which case a patch would have made the bug report even better still!

Ok, next time.

> > but perhaps it makes sense to explicitly
> > address these issues in the Linux man page.
 
> Anyway, I agree that the page could be clearer. I've added the following text:
 
>        A sequence of calls to strtok() that operate on the same string
>        maintains  a  pointer  that  determines the point from which to
>        start searching for the next token.  The first call to strtok()
>        sets  this  pointer  to  point to the first byte of the string.

>        The start of the next token is determined by  scanning  forward
>        for  the  next  nondelimiter  byte  in  str.  If such a byte is
>        found, it is taken as the start of the next token.  If no  such
>        byte  is  found,  then  there  are no more tokens, and strtok()
>        returns NULL.  (A string that is empty or  that  contains  only
>        delimiter  will thus cause strtok() to return NULL on the first

plural, delimiters?

>        call.)
> 
>        The end of each token is found by scanning forward until either
>        the  next delimiter byte is found or until the terminating null
>        byte ('\0') is encountered.  If a delimiter byte is  found,  it
>        is overwritten with a null byte to terminate the current token,
>        and strtok() saves  a  pointer  to  the  following  byte;  that
>        pointer  will  be used as the starting point when searching for
>        the next token.  In this case, strtok() returns  a  pointer  to
>        the start of the found token.

>        From  the above description, it follows thatt a sequence of two

typo: /thatt/that/

>        or more contiguous delimiter bytes in the parsed string is con‐
>        sidered  to  be a single delimiter, and that delimiter bytes at
>        the start or end of the string are ignored.  Put  another  way:
>        the  tokens  returned  by strtok() are always nonempty strings.
>        Thus, for example, given  the  string  "aaa;;bbb,",  successive
>        calls  to strtok() that specify the delimiter string ";," would
>        return the strings "aaa" and "bbb", and then a NULL pointer.
 
> Sufficient?

Yes!

Best regards
Georg
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: strtok - strings without delimiters

[Michael Kerrisk]

Hi Georg,

On 06/01/13 22:58, Georg Sauthoff wrote:
> On Sun, May 19, 2013 at 10:46:49PM +0200, Michael Kerrisk wrote:

[...]

>>> but perhaps it makes sense to explicitly
>>> address these issues in the Linux man page.
>  
>> Anyway, I agree that the page could be clearer. I've added the following text:
>  
>>        A sequence of calls to strtok() that operate on the same string
>>        maintains  a  pointer  that  determines the point from which to
>>        start searching for the next token.  The first call to strtok()
>>        sets  this  pointer  to  point to the first byte of the string.
> 
>>        The start of the next token is determined by  scanning  forward
>>        for  the  next  nondelimiter  byte  in  str.  If such a byte is
>>        found, it is taken as the start of the next token.  If no  such
>>        byte  is  found,  then  there  are no more tokens, and strtok()
>>        returns NULL.  (A string that is empty or  that  contains  only
>>        delimiter  will thus cause strtok() to return NULL on the first
> 
> plural, delimiters?

Fixed now.
>>        call.)
>>
>>        The end of each token is found by scanning forward until either
>>        the  next delimiter byte is found or until the terminating null
>>        byte ('\0') is encountered.  If a delimiter byte is  found,  it
>>        is overwritten with a null byte to terminate the current token,
>>        and strtok() saves  a  pointer  to  the  following  byte;  that
>>        pointer  will  be used as the starting point when searching for
>>        the next token.  In this case, strtok() returns  a  pointer  to
>>        the start of the found token.
> 
>>        From  the above description, it follows thatt a sequence of two
> 
> typo: /thatt/that/

Looks like I already fixed that one.

>>        or more contiguous delimiter bytes in the parsed string is con‐
>>        sidered  to  be a single delimiter, and that delimiter bytes at
>>        the start or end of the string are ignored.  Put  another  way:
>>        the  tokens  returned  by strtok() are always nonempty strings.
>>        Thus, for example, given  the  string  "aaa;;bbb,",  successive
>>        calls  to strtok() that specify the delimiter string ";," would
>>        return the strings "aaa" and "bbb", and then a NULL pointer.
>  
>> Sufficient?
> 
> Yes!

Good. thank you for checking the text.

Cheers,

Michael

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html