[PATCH] scanf.3: Do not mention the ERANGE error

[PATCH] scanf.3: Do not mention the ERANGE error

[Ian Abbott]

The `scanf()` function does not intentionally set `errno` to `ERANGE`.
That is just a side effect of the code that it uses to perform
conversions.  It also does not work as reliably as indicated in the
'man' page when the target integer type is narrower than `long`.
Typically (at least in glibc) for target integer types narrower than
`long`, the number has to exceed the range of `long` (for signed
conversions) or `unsigned long` (for unsigned conversions) for `errno`
to be set to `ERANGE`.

Documenting `ERANGE` in the ERRORS section kind of implies that
`scanf()` should return `EOF` when an integer overflow is encountered,
which it doesn't (and doing so would violate the C standard).

Just remove any mention of the `ERANGE` error to avoid confusion.

Fixes: 646af540e467 ("Add an ERRORS section documenting at least some of the errors that may occur for scanf().")
Cc: Michael Kerrisk <mtk.manpages@gmail.com>
Signed-off-by: Ian Abbott <abbotti@mev.co.uk>
---
 man3/scanf.3 | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/man3/scanf.3 b/man3/scanf.3
index ba470a5c1..c5ff59f45 100644
--- a/man3/scanf.3
+++ b/man3/scanf.3
@@ -576,10 +576,6 @@ is NULL.
 .TP
 .B ENOMEM
 Out of memory.
-.TP
-.B ERANGE
-The result of an integer conversion would exceed the size
-that can be stored in the corresponding integer type.
 .SH ATTRIBUTES
 For an explanation of the terms used in this section, see
 .BR attributes (7).
@@ -609,9 +605,6 @@ The functions
 and
 .BR sscanf ()
 conform to C89 and C99 and POSIX.1-2001.
-These standards do not specify the
-.B ERANGE
-error.
 .PP
 The
 .B q
-- 
2.35.1

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 2007 bytes --]

Hi Ian,

On 12/8/22 13:34, Ian Abbott wrote:
> The `scanf()` function does not intentionally set `errno` to `ERANGE`.
> That is just a side effect of the code that it uses to perform
> conversions.  It also does not work as reliably as indicated in the
> 'man' page when the target integer type is narrower than `long`.
> Typically (at least in glibc) for target integer types narrower than
> `long`, the number has to exceed the range of `long` (for signed
> conversions) or `unsigned long` (for unsigned conversions) for `errno`
> to be set to `ERANGE`.
> 
> Documenting `ERANGE` in the ERRORS section kind of implies that
> `scanf()` should return `EOF` when an integer overflow is encountered,
> which it doesn't (and doing so would violate the C standard).
> 
> Just remove any mention of the `ERANGE` error to avoid confusion.
> 
> Fixes: 646af540e467 ("Add an ERRORS section documenting at least some of the errors that may occur for scanf().")
> Cc: Michael Kerrisk <mtk.manpages@gmail.com>
> Signed-off-by: Ian Abbott <abbotti@mev.co.uk>

I see.  How about saying something like "it may also fail for any of any errors 
that functions used to perform the conversions may fail"?

Cheers,

Alex

> ---
>   man3/scanf.3 | 7 -------
>   1 file changed, 7 deletions(-)
> 
> diff --git a/man3/scanf.3 b/man3/scanf.3
> index ba470a5c1..c5ff59f45 100644
> --- a/man3/scanf.3
> +++ b/man3/scanf.3
> @@ -576,10 +576,6 @@ is NULL.
>   .TP
>   .B ENOMEM
>   Out of memory.
> -.TP
> -.B ERANGE
> -The result of an integer conversion would exceed the size
> -that can be stored in the corresponding integer type.
>   .SH ATTRIBUTES
>   For an explanation of the terms used in this section, see
>   .BR attributes (7).
> @@ -609,9 +605,6 @@ The functions
>   and
>   .BR sscanf ()
>   conform to C89 and C99 and POSIX.1-2001.
> -These standards do not specify the
> -.B ERANGE
> -error.
>   .PP
>   The
>   .B q

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Ian Abbott]

On 09/12/2022 18:59, Alejandro Colomar wrote:
> Hi Ian,
> 
> On 12/8/22 13:34, Ian Abbott wrote:
>> The `scanf()` function does not intentionally set `errno` to `ERANGE`.
>> That is just a side effect of the code that it uses to perform
>> conversions.  It also does not work as reliably as indicated in the
>> 'man' page when the target integer type is narrower than `long`.
>> Typically (at least in glibc) for target integer types narrower than
>> `long`, the number has to exceed the range of `long` (for signed
>> conversions) or `unsigned long` (for unsigned conversions) for `errno`
>> to be set to `ERANGE`.
>>
>> Documenting `ERANGE` in the ERRORS section kind of implies that
>> `scanf()` should return `EOF` when an integer overflow is encountered,
>> which it doesn't (and doing so would violate the C standard).
>>
>> Just remove any mention of the `ERANGE` error to avoid confusion.
>>
>> Fixes: 646af540e467 ("Add an ERRORS section documenting at least some 
>> of the errors that may occur for scanf().")
>> Cc: Michael Kerrisk <mtk.manpages@gmail.com>
>> Signed-off-by: Ian Abbott <abbotti@mev.co.uk>
> 
> I see.  How about saying something like "it may also fail for any of any 
> errors that functions used to perform the conversions may fail"?

It depends what you mean by "fail".  These errors do not make scanf 
return EOF.  Technically, the behavior is undefined if the result of the 
conversion cannot be represented in the object being assigned to by 
scanf.  (In the case of glibc, that probably results in either the 
integer object being set to a truncated version of the input integer, or 
the integer object being set to a truncated version of LONG_MIN or 
LONG_MAX, depending on the actual number.)

Setting errno to 0 before calling scanf and expecting errno to have a 
meaningful value when scanf returns something other than EOF is bogus usage.

> 
> Cheers,
> 
> Alex

Cheers,
Ian

> 
>> ---
>>   man3/scanf.3 | 7 -------
>>   1 file changed, 7 deletions(-)
>>
>> diff --git a/man3/scanf.3 b/man3/scanf.3
>> index ba470a5c1..c5ff59f45 100644
>> --- a/man3/scanf.3
>> +++ b/man3/scanf.3
>> @@ -576,10 +576,6 @@ is NULL.
>>   .TP
>>   .B ENOMEM
>>   Out of memory.
>> -.TP
>> -.B ERANGE
>> -The result of an integer conversion would exceed the size
>> -that can be stored in the corresponding integer type.
>>   .SH ATTRIBUTES
>>   For an explanation of the terms used in this section, see
>>   .BR attributes (7).
>> @@ -609,9 +605,6 @@ The functions
>>   and
>>   .BR sscanf ()
>>   conform to C89 and C99 and POSIX.1-2001.
>> -These standards do not specify the
>> -.B ERANGE
>> -error.
>>   .PP
>>   The
>>   .B q
> 

-- 
-=( Ian Abbott <abbotti@mev.co.uk> || MEV Ltd. is a company  )=-
-=( registered in England & Wales.  Regd. number: 02862268.  )=-
-=( Regd. addr.: S11 & 12 Building 67, Europa Business Park, )=-
-=( Bird Hall Lane, STOCKPORT, SK3 0XA, UK. || www.mev.co.uk )=-

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 2570 bytes --]

Hi Ian,

On 12/9/22 20:28, Ian Abbott wrote:
> On 09/12/2022 18:59, Alejandro Colomar wrote:
>> On 12/8/22 13:34, Ian Abbott wrote:
>>> The `scanf()` function does not intentionally set `errno` to `ERANGE`.
>>> That is just a side effect of the code that it uses to perform
>>> conversions.  It also does not work as reliably as indicated in the
>>> 'man' page when the target integer type is narrower than `long`.
>>> Typically (at least in glibc) for target integer types narrower than
>>> `long`, the number has to exceed the range of `long` (for signed
>>> conversions) or `unsigned long` (for unsigned conversions) for `errno`
>>> to be set to `ERANGE`.
>>>
>>> Documenting `ERANGE` in the ERRORS section kind of implies that
>>> `scanf()` should return `EOF` when an integer overflow is encountered,
>>> which it doesn't (and doing so would violate the C standard).
>>>
>>> Just remove any mention of the `ERANGE` error to avoid confusion.
>>>
>>> Fixes: 646af540e467 ("Add an ERRORS section documenting at least some of the 
>>> errors that may occur for scanf().")
>>> Cc: Michael Kerrisk <mtk.manpages@gmail.com>
>>> Signed-off-by: Ian Abbott <abbotti@mev.co.uk>
>>
>> I see.  How about saying something like "it may also fail for any of any 
>> errors that functions used to perform the conversions may fail"?
> 
> It depends what you mean by "fail".  These errors do not make scanf return EOF.  

Just to clarify.  Does scanf(3) _never_ fail (EOF) due to ERANGE?  Or is it that 
ERANGE sometimes makes it fail, sometimes not?

If it's the former, I agree with your patch.  When a function hasn't reported 
failure, errno is unspecified.

If it's the latter, I'd write something about it.


> Technically, the behavior is undefined if the result of the conversion cannot be 
> represented in the object being assigned to by scanf.  (In the case of glibc, 
> that probably results in either the integer object being set to a truncated 
> version of the input integer, or the integer object being set to a truncated 
> version of LONG_MIN or LONG_MAX, depending on the actual number.)

Hmm, UB.  Under UB, anything can change, so error reporting is already 
unreliable.  If EOF+ERANGE can _only_ happen under UB, I'd rather remove the 
paragraph.  Please confirm.

> 
> Setting errno to 0 before calling scanf and expecting errno to have a meaningful 
> value when scanf returns something other than EOF is bogus usage.

Yep, that's bogus.


Cheers,

Alex

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Zack Weinberg]

On 2022-12-09 2:33 PM, Alejandro Colomar via Libc-alpha wrote:
>> Technically, the behavior is undefined if the result of the conversion 
>> cannot be represented in the object being assigned to by scanf.  (In 
>> the case of glibc, that probably results in either the integer object 
>> being set to a truncated version of the input integer, or the integer 
>> object being set to a truncated version of LONG_MIN or LONG_MAX, 
>> depending on the actual number.)
> 
> Hmm, UB.  Under UB, anything can change, so error reporting is already 
> unreliable.  If EOF+ERANGE can _only_ happen under UB, I'd rather remove 
> the paragraph.  Please confirm.
			
BUGS

The `scanf` functions have undefined behavior if numeric input 
overflows.  This means it is *impossible* to detect malformed input 
reliably using these functions.

Many input specifications (e.g. `%s`, `%[^\n]`) read a sequence of 
characters into a destination buffer whose size is unspecified; any use 
of such specifications renders `scanf` every bit as dangerous as `gets`.

Best practice is not to use any of these functions at all.

zw (no, this is not a joke)

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 2292 bytes --]

[CC += Ian]

Hi Zack,

On 12/9/22 22:41, Zack Weinberg via Libc-alpha wrote:
> On 2022-12-09 2:33 PM, Alejandro Colomar via Libc-alpha wrote:
>>> Technically, the behavior is undefined if the result of the conversion cannot 
>>> be represented in the object being assigned to by scanf.  (In the case of 
>>> glibc, that probably results in either the integer object being set to a 
>>> truncated version of the input integer, or the integer object being set to a 
>>> truncated version of LONG_MIN or LONG_MAX, depending on the actual number.)
>>
>> Hmm, UB.  Under UB, anything can change, so error reporting is already 
>> unreliable.  If EOF+ERANGE can _only_ happen under UB, I'd rather remove the 
>> paragraph.  Please confirm.
> 
> BUGS
> 
> The `scanf` functions have undefined behavior if numeric input overflows.  This 
> means it is *impossible* to detect malformed input reliably using these functions.
> 
> Many input specifications (e.g. `%s`, `%[^\n]`) read a sequence of characters 
> into a destination buffer whose size is unspecified; any use of such 
> specifications renders `scanf` every bit as dangerous as `gets`.

Thanks for reminding that!  Since I don't use these functions, I don't remember 
how bad they are :)

> 
> Best practice is not to use any of these functions at all.
> 
> zw (no, this is not a joke)

I'm inclined to add that in that manual page.  Is there anything that can be 
saved from that page, or should we burn it all?  To be more specific:

-  Are there any functions in that page that are still useful for any corner 
cases, or are they all useless?
-  Are there any conversion specifiers that can be used safely?

Or the converse questions:

-  Which conversion specifiers (or modifiers) are impossible to use safely as 
gets(3) and should therefore be marked as deprecated in the manual page (and 
probably warned in GCC)?
-  Which functions in that page are impossible to use safely and should 
therefore be marked as deprecated?

Would you please mark them as [[deprecated]] in glibc too?  This is not 
essential to me, since I can mark them as deprecated in the manual pages without 
that happening, but it'd help.

Cheers,

Alex

> 

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 2621 bytes --]



On 12/11/22 16:58, Alejandro Colomar wrote:
> [CC += Ian]
> 
> Hi Zack,
> 
> On 12/9/22 22:41, Zack Weinberg via Libc-alpha wrote:
>> On 2022-12-09 2:33 PM, Alejandro Colomar via Libc-alpha wrote:
>>>> Technically, the behavior is undefined if the result of the conversion 
>>>> cannot be represented in the object being assigned to by scanf.  (In the 
>>>> case of glibc, that probably results in either the integer object being set 
>>>> to a truncated version of the input integer, or the integer object being set 
>>>> to a truncated version of LONG_MIN or LONG_MAX, depending on the actual 
>>>> number.)
>>>
>>> Hmm, UB.  Under UB, anything can change, so error reporting is already 
>>> unreliable.  If EOF+ERANGE can _only_ happen under UB, I'd rather remove the 
>>> paragraph.  Please confirm.
>>
>> BUGS
>>
>> The `scanf` functions have undefined behavior if numeric input overflows.  
>> This means it is *impossible* to detect malformed input reliably using these 
>> functions.
>>
>> Many input specifications (e.g. `%s`, `%[^\n]`) read a sequence of characters 
>> into a destination buffer whose size is unspecified; any use of such 
>> specifications renders `scanf` every bit as dangerous as `gets`.
> 
> Thanks for reminding that!  Since I don't use these functions, I don't remember 
> how bad they are :)
> 
>>
>> Best practice is not to use any of these functions at all.
>>
>> zw (no, this is not a joke)
> 
> I'm inclined to add that in that manual page.  Is there anything that can be 
> saved from that page, or should we burn it all?  To be more specific:
> 
> -  Are there any functions in that page that are still useful for any corner 
> cases, or are they all useless?
> -  Are there any conversion specifiers that can be used safely?
> 
> Or the converse questions:
> 
> -  Which conversion specifiers (or modifiers) are impossible to use safely as 
> gets(3) and should therefore be marked as deprecated in the manual page (and 
> probably warned in GCC)?
> -  Which functions in that page are impossible to use safely and should 
> therefore be marked as deprecated?

This includes functions that can only be used safely for the most basic behavior 
for which better functions such as fgets(3) are superior.  I'd also deprecate those.

> 
> Would you please mark them as [[deprecated]] in glibc too?  This is not 
> essential to me, since I can mark them as deprecated in the manual pages without 
> that happening, but it'd help.
> 
> Cheers,
> 
> Alex
> 
>>
> 

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Zack Weinberg]

Alejandro Colomar <alx.manpages@gmail.com> writes:
> On 12/9/22 22:41, Zack Weinberg via Libc-alpha wrote:
>> The `scanf` functions have undefined behavior if numeric input
>> overflows.  This means it is *impossible* to detect malformed input
>> reliably using these functions.
>> Many input specifications (e.g. `%s`, `%[^\n]`) read a sequence of
>> characters into a destination buffer whose size is unspecified; any
>> use of such specifications renders `scanf` every bit as dangerous as
>> `gets`.
…
>> Best practice is not to use any of these functions at all.
…
> I'm inclined to add that in that manual page.  Is there anything that
> can be saved from that page, or should we burn it all?  To be more
> specific:
>
> -  Are there any functions in that page that are still useful for any
>    corner cases, or are they all useless?
> -  Are there any conversion specifiers that can be used safely?

Hmm, this turns out to be a bit of a rabbit hole.

There are two major design-level problems with the scanf family.
The more important one is that string input conversions (%s, %[…])
will read an unlimited number of characters by default, oblivious to
the size of the destination buffer — exactly the same design flaw as
‘gets’.  They do stop scanning at _any_ whitespace (not just \n) so,
if you’re trying to craft exploit code, there are more byte values that
must be avoided, but this is only a minor obstacle.  They can, however,
be used safely, either by supplying a field width that accurately
reflects the size of the destination buffer, or by using the ‘m’
modifier (a POSIX extension), which directs scanf to allocate the
right amount of space for the string with malloc.

(Field widths are awkward to use because you have to write them as
decimal constants _inside the format string_, which makes them more
likely to get out of sync with the actual size of the buffer than,
say, the buffer-size argument to ‘fgets’, but this is not a fatal
flaw in and of itself.)

The other design-level issue affects all of the numeric conversions:
if the result of (abstract, infinite-precision) numeric input conversion
does not fit in the variable supplied to hold the result of that conversion,
the behavior is undefined.  The manpage says that you get an ERANGE error
in this case, and that may be the behavior _glibc_ guarantees (I don’t
actually know for sure), but in the modern era of compilers drawing
inferences from undefined behavior, a guarantee by one C library is
not good enough.

That covers everything except %c and %n, which are safe but somewhat
pointless in isolation.

> Or the converse questions:
>
> -  Which conversion specifiers (or modifiers) are impossible to use
>    safely as gets(3) and should therefore be marked as deprecated in
>    the manual page (and probably warned in GCC)?
> -  Which functions in that page are impossible to use safely and
>    should therefore be marked as deprecated?
>
> Would you please mark them as [[deprecated]] in glibc too?

I don’t think glibc should unilaterally deprecate any function that’s
specified by ISO C.  And, the scanf family *can* be used safely with
sufficient care — read entire lines of input with getline, then split
them up into fields with sscanf using only %ms and %m[…], and finally
parse all numeric fields by hand with strtoX — the issue is more that,
if you limit yourself to the set of scanf operations that are 100% safe,
you’re left only with stuff that is arguably *easier* to do with <string.h>
and <regex.h> functions.

In a more sober tone of voice I suggest this text for the manpage:

.SH BUGS
By default, the 
.IR %s " and " %[
conversions will read an
.I unlimited
number of characters from the input.
In this mode they are just as unsafe as the infamous
.BR gets (3).
One should always specify either a field width,
or the
.I m
modifier,
with every use of
.IR %s " or " %[ .
.PP
If a numeric input conversion produces a value
that is not representable in the type of the corresponding argument
(e.g. if 99999 is to be stored in an
.IR "unsigned short" ),
ISO C says that the behavior is undefined.
The GNU C Library guarantees to treat this condition as a
.IR "matching failure" ,
but portable code should avoid using the numeric conversions.

I also suggest that GCC should add diagnostics to -Wformat and/or
-Wformat-security to catch use of %s and %[ with no size specified;
if glibc doesn’t already treat numeric overflow as a matching failure,
it should be changed to do so; and maybe someone should write up a
proposal for the C standard to make the same change.

zw

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 8332 bytes --]

Hi Zack!

On 12/12/22 03:11, Zack Weinberg wrote:
> Alejandro Colomar <alx.manpages@gmail.com> writes:
>> On 12/9/22 22:41, Zack Weinberg via Libc-alpha wrote:
>>> The `scanf` functions have undefined behavior if numeric input
>>> overflows.  This means it is *impossible* to detect malformed input
>>> reliably using these functions.
>>> Many input specifications (e.g. `%s`, `%[^\n]`) read a sequence of
>>> characters into a destination buffer whose size is unspecified; any
>>> use of such specifications renders `scanf` every bit as dangerous as
>>> `gets`.
> …
>>> Best practice is not to use any of these functions at all.
> …
>> I'm inclined to add that in that manual page.  Is there anything that
>> can be saved from that page, or should we burn it all?  To be more
>> specific:
>>
>> -  Are there any functions in that page that are still useful for any
>>     corner cases, or are they all useless?
>> -  Are there any conversion specifiers that can be used safely?
> 
> Hmm, this turns out to be a bit of a rabbit hole.
> 
> There are two major design-level problems with the scanf family.
> The more important one is that string input conversions (%s, %[…])
> will read an unlimited number of characters by default, oblivious to
> the size of the destination buffer — exactly the same design flaw as
> ‘gets’.  They do stop scanning at _any_ whitespace (not just \n) so,
> if you’re trying to craft exploit code, there are more byte values that
> must be avoided, but this is only a minor obstacle.  They can, however,
> be used safely, either by supplying a field width that accurately
> reflects the size of the destination buffer, or by using the ‘m’
> modifier (a POSIX extension), which directs scanf to allocate the
> right amount of space for the string with malloc.

Okay, so %s and $[ are at least usable.  Useful?  I don't know.  Probably 
fgets(3) and then either <string.h> or <regex.h> functions or taking 
unterminated strings (pointer plus length) is a much better idea.

But not enough to deprecate the specifiers; probably better to warn on GCC.

> 
> (Field widths are awkward to use because you have to write them as
> decimal constants _inside the format string_, which makes them more
> likely to get out of sync with the actual size of the buffer than,
> say, the buffer-size argument to ‘fgets’, but this is not a fatal
> flaw in and of itself.)

Yeah; it's an almost useless feature, but not a fatal flaw.  Any programmer 
would probably intuitively know that it's bad.  So, no action is required here.

> 
> The other design-level issue affects all of the numeric conversions:
> if the result of (abstract, infinite-precision) numeric input conversion
> does not fit in the variable supplied to hold the result of that conversion,
> the behavior is undefined.  The manpage says that you get an ERANGE error
> in this case, and that may be the behavior _glibc_ guarantees (I don’t
> actually know for sure), but in the modern era of compilers drawing
> inferences from undefined behavior, a guarantee by one C library is
> not good enough.

This, to me, is enough to mark them as deprecated in the manual page.  Anyway, 
deprecating something is not removing it.  It's just saying "hey, you shouldn't 
be using that; it's bad, and don't expect that ISO C will keep it around next 
century".

Something that results in undefined behavior without control of the programmer 
is as bad as gets(3) (okay, not as bad; gets(3) just was a red carpet for 
attacks, but fundamentally, yes).

So, I'll apply the diff shown at the bottom of the page for this.

> 
> That covers everything except %c and %n, which are safe but somewhat
> pointless in isolation.
> 
>> Or the converse questions:
>>
>> -  Which conversion specifiers (or modifiers) are impossible to use
>>     safely as gets(3) and should therefore be marked as deprecated in
>>     the manual page (and probably warned in GCC)?
>> -  Which functions in that page are impossible to use safely and
>>     should therefore be marked as deprecated?
>>
>> Would you please mark them as [[deprecated]] in glibc too?
> 
> I don’t think glibc should unilaterally deprecate any function that’s
> specified by ISO C.

Okay.  The man-pages are not that restricted, since they won't affect code at 
all, and only the minds of the programmers, which is more powerful.  So, I'll 
mark as deprecated at least the integer conversion specifiers.

>  And, the scanf family *can* be used safely with
> sufficient care — read entire lines of input with getline,

If getline(3) _is necessary_ to be safe, then I would deprecate the stream 
functions, and keep only the "s" variants.  Is it?

In fact, I'd say that even if it's not necessary to be safe, there are no good 
reasons to use [f]scanf(3) at all.  I'm very much considering deprecating them 
in the manual page.

> then split
> them up into fields with sscanf using only %ms and %m[…], and finally
> parse all numeric fields by hand with strtoX — the issue is more that,
> if you limit yourself to the set of scanf operations that are 100% safe,
> you’re left only with stuff that is arguably *easier* to do with <string.h>
> and <regex.h> functions.
> 
> In a more sober tone of voice I suggest this text for the manpage:
> 
> .SH BUGS
> By default, the
> .IR %s " and " %[
> conversions will read an
> .I unlimited
> number of characters from the input.
> In this mode they are just as unsafe as the infamous
> .BR gets (3).
> One should always specify either a field width,
> or the
> .I m
> modifier,
> with every use of
> .IR %s " or " %[ .
> .PP
> If a numeric input conversion produces a value
> that is not representable in the type of the corresponding argument
> (e.g. if 99999 is to be stored in an
> .IR "unsigned short" ),
> ISO C says that the behavior is undefined.
> The GNU C Library guarantees to treat this condition as a
> .IR "matching failure" ,
> but portable code should avoid using the numeric conversions.

That makes sense to me.  Would you mind sending a patch?  :)

> 
> I also suggest that GCC should add diagnostics to -Wformat and/or
> -Wformat-security to catch use of %s and %[ with no size specified;
> if glibc doesn’t already treat numeric overflow as a matching failure,
> it should be changed to do so; and maybe someone should write up a
> proposal for the C standard to make the same change.

Acked-by: Alejandro Colomar <alx@kernel.org>

> 
> zw

Cheers,

Alex

---


diff --git a/man3/scanf.3 b/man3/scanf.3
index ba470a5c1..0041d5573 100644
--- a/man3/scanf.3
+++ b/man3/scanf.3
@@ -386,6 +386,7 @@ .SS Conversions
  and assignment does not occur.
  .TP
  .B d
+.IR Deprecated .
  Matches an optionally signed decimal integer;
  the next pointer must be a pointer to
  .IR int .
@@ -400,6 +401,7 @@ .SS Conversions
  .\" is silently ignored, causing old programs to fail mysteriously.)
  .TP
  .B i
+.IR Deprecated .
  Matches an optionally signed integer; the next pointer must be a pointer to
  .IR int .
  The integer is read in base 16 if it begins with
@@ -412,15 +414,18 @@ .SS Conversions
  Only characters that correspond to the base are used.
  .TP
  .B o
+.IR Deprecated .
  Matches an unsigned octal integer; the next pointer must be a pointer to
  .IR "unsigned int" .
  .TP
  .B u
+.IR Deprecated .
  Matches an unsigned decimal integer; the next pointer must be a
  pointer to
  .IR "unsigned int" .
  .TP
  .B x
+.IR Deprecated .
  Matches an unsigned hexadecimal integer
  (that may optionally begin with a prefix of
  .I 0x
@@ -431,27 +436,33 @@ .SS Conversions
  .IR "unsigned int" .
  .TP
  .B X
+.IR Deprecated .
  Equivalent to
  .BR x .
  .TP
  .B f
+.IR Deprecated .
  Matches an optionally signed floating-point number; the next pointer must
  be a pointer to
  .IR float .
  .TP
  .B e
+.IR Deprecated .
  Equivalent to
  .BR f .
  .TP
  .B g
+.IR Deprecated .
  Equivalent to
  .BR f .
  .TP
  .B E
+.IR Deprecated .
  Equivalent to
  .BR f .
  .TP
  .B a
+.IR Deprecated .
  (C99) Equivalent to
  .BR f .
  .TP



-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Zack Weinberg]

Alejandro Colomar <alx.manpages@gmail.com> writes:
> Okay, so %s and $[ are at least usable.  Useful?  I don't know.  Probably 
> fgets(3) and then either <string.h> or <regex.h> functions or taking 
> unterminated strings (pointer plus length) is a much better idea.

Yeah, agreed.

>> The other design-level issue affects all of the numeric conversions:
>> if the result of (abstract, infinite-precision) numeric input conversion
>> does not fit in the variable supplied to hold the result of that conversion,
>> the behavior is undefined.  The manpage says that you get an ERANGE error
>> in this case, and that may be the behavior _glibc_ guarantees (I don’t
>> actually know for sure), but in the modern era of compilers drawing
>> inferences from undefined behavior, a guarantee by one C library is
>> not good enough.
>
> This, to me, is enough to mark them as deprecated in the manual page.  Anyway, 
> deprecating something is not removing it.  It's just saying "hey, you shouldn't 
> be using that; it's bad, and don't expect that ISO C will keep it around next 
> century".

In my lexicon “deprecated” is a very strong statement, possibly because
I’m used to seeing it in the context of standards where it means “we
think we should never have added this in the first place, there’s no
plausible way to fix it, but we have to keep it around for backward
compatibility.”

The scanf numeric conversions could be fixed with a one-sentence edit to
the C standard: change the last sentence of http://port70.net/~nsz/c/c11/n1570.html#7.21.6.2p10
from “If this object does not have an appropriate type, or if the result
of the conversion cannot be represented in the object, the behavior is
undefined” to “If this object does not have an appropriate type, the
behavior is undefined.  If the result of the conversion cannot be
represented in the object, the execution of the directive fails; this
condition is a matching failure.”  And, even if the C committee doesn’t
want to make that change, open-source C libraries can and should do it
unilaterally, as a documented implementation extension.  I think that’s
a better plan than declaring most uses of *scanf “deprecated.”

>>  And, the scanf family *can* be used safely with
>> sufficient care — read entire lines of input with getline,
>
> If getline(3) _is necessary_ to be safe, then I would deprecate the stream 
> functions, and keep only the "s" variants.  Is it?

Oh, right, the _third_ headache with fscanf.

Yes, I think it would be fair to say that it is almost always a mistake
to use the scanf variants that read directly from a FILE.  The issue
here is, at its root, that people new to C _expect_ a scanf call to read
an entire line of input, but it doesn’t. This is especially problematic
for interactive input — they try to use plain scanf to read numeric
input, don’t realize that `scanf("%d", &arg)` doesn’t consume the \n in
the terminal’s line buffer _after_ the number, and get very confused
when a subsequent getchar() reads that \n instead of the ‘y’ or ‘n’ they
were expecting as a response to the _next_ prompt.  But it’s _also_ a
problem for error recovery, because scanf will stop in the middle of the
line when a matching failure occurs, and if you naively assumed it would
throw away the rest of the line, you get an error cascade.

The recommended practice to avoid this trap, is that you should use one
of the functions that _does_ read an entire line of input, i.e. fgets or
getline, and then parse the line as a string.  It would make sense for
the [f]scanf manpage to say that.

>> In a more sober tone of voice I suggest this text for the manpage:
…
> That makes sense to me.  Would you mind sending a patch?  :)

I do not have time to do that anytime soon.  Also, maybe glibc’s
behavior on numeric input overflow should be fixed first.

zw

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 5300 bytes --]

Hi Zack,

On 12/14/22 03:13, Zack Weinberg wrote:
> Alejandro Colomar <alx.manpages@gmail.com> writes:
>> Okay, so %s and $[ are at least usable.  Useful?  I don't know.  Probably
>> fgets(3) and then either <string.h> or <regex.h> functions or taking
>> unterminated strings (pointer plus length) is a much better idea.
> 
> Yeah, agreed.
> 
>>> The other design-level issue affects all of the numeric conversions:
>>> if the result of (abstract, infinite-precision) numeric input conversion
>>> does not fit in the variable supplied to hold the result of that conversion,
>>> the behavior is undefined.  The manpage says that you get an ERANGE error
>>> in this case, and that may be the behavior _glibc_ guarantees (I don’t
>>> actually know for sure), but in the modern era of compilers drawing
>>> inferences from undefined behavior, a guarantee by one C library is
>>> not good enough.
>>
>> This, to me, is enough to mark them as deprecated in the manual page.  Anyway,
>> deprecating something is not removing it.  It's just saying "hey, you shouldn't
>> be using that; it's bad, and don't expect that ISO C will keep it around next
>> century".
> 
> In my lexicon “deprecated” is a very strong statement, possibly because
> I’m used to seeing it in the context of standards where it means “we
> think we should never have added this in the first place, there’s no
> plausible way to fix it, but we have to keep it around for backward
> compatibility.”
> 
> The scanf numeric conversions could be fixed with a one-sentence edit to
> the C standard: change the last sentence of http://port70.net/~nsz/c/c11/n1570.html#7.21.6.2p10
> from “If this object does not have an appropriate type, or if the result
> of the conversion cannot be represented in the object, the behavior is
> undefined” to “If this object does not have an appropriate type, the
> behavior is undefined.  If the result of the conversion cannot be
> represented in the object, the execution of the directive fails; this
> condition is a matching failure.”  And, even if the C committee doesn’t
> want to make that change, open-source C libraries can and should do it
> unilaterally, as a documented implementation extension.  I think that’s
> a better plan than declaring most uses of *scanf “deprecated.”

Yeah, if you have plans to fix it, I'm fine removing the deprecation now. :)

> 
>>>   And, the scanf family *can* be used safely with
>>> sufficient care — read entire lines of input with getline,
>>
>> If getline(3) _is necessary_ to be safe, then I would deprecate the stream
>> functions, and keep only the "s" variants.  Is it?
> 
> Oh, right, the _third_ headache with fscanf.
> 
> Yes, I think it would be fair to say that it is almost always a mistake
> to use the scanf variants that read directly from a FILE.  The issue
> here is, at its root, that people new to C _expect_ a scanf call to read
> an entire line of input, but it doesn’t. This is especially problematic
> for interactive input — they try to use plain scanf to read numeric
> input, don’t realize that `scanf("%d", &arg)` doesn’t consume the \n in
> the terminal’s line buffer _after_ the number, and get very confused
> when a subsequent getchar() reads that \n instead of the ‘y’ or ‘n’ they
> were expecting as a response to the _next_ prompt.  But it’s _also_ a
> problem for error recovery, because scanf will stop in the middle of the
> line when a matching failure occurs, and if you naively assumed it would
> throw away the rest of the line, you get an error cascade.
> 
> The recommended practice to avoid this trap, is that you should use one
> of the functions that _does_ read an entire line of input, i.e. fgets or
> getline, and then parse the line as a string.  It would make sense for
> the [f]scanf manpage to say that.

Please clarify; do you consider [v][f]scanf something that "we think we should 
never have added this in the first place, there’s no plausible way to fix it, 
but we have to keep it around for backward compatibility"?

> 
>>> In a more sober tone of voice I suggest this text for the manpage:
> …
>> That makes sense to me.  Would you mind sending a patch?  :)
> 
> I do not have time to do that anytime soon.  Also, maybe glibc’s
> behavior on numeric input overflow should be fixed first.

That also makes sense ;)

In short:

(1)  Numeric conversion specifiers are broken but can be fixed, and you plan to 
fix them.

      (1.1)  I'll revert the deprecation warning now; since they are only broken 
because the _current_ standard and implementations are broken, but not by 
inherent design problems.

      (1.2)  When you fix the implementation to not be UB anymore, it will also 
make sense to revert the patch that removed the ERANGE error, since you'll need 
to report it.

(2)  For the string conversion specifiers, there are ways to use them safely, 
and you plan to add a way to specify a size at runtime to the function, so it 
will be even better in the future.  No action required.

(3)  [v][f]scanf seem to be really broken by design.  Please confirm.

Cheers,

Alex

> 
> zw

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Ian Abbott]

On 2022-12-14 10:47, Alejandro Colomar wrote:
> On 12/14/22 03:13, Zack Weinberg wrote:
>> Alejandro Colomar <alx.manpages@gmail.com> writes:
>>>> In a more sober tone of voice I suggest this text for the manpage:
>> …
>>> That makes sense to me.  Would you mind sending a patch?  :)
>>
>> I do not have time to do that anytime soon.  Also, maybe glibc’s
>> behavior on numeric input overflow should be fixed first.
> 
> That also makes sense ;)
> 
> In short:
> 
> (1)  Numeric conversion specifiers are broken but can be fixed, and you 
> plan to fix them.
> 
>       (1.1)  I'll revert the deprecation warning now; since they are 
> only broken because the _current_ standard and implementations are 
> broken, but not by inherent design problems.
> 
>       (1.2)  When you fix the implementation to not be UB anymore, it 
> will also make sense to revert the patch that removed the ERANGE error, 
> since you'll need to report it.

And would ERANGE cause scanf to return EOF in the fixed implementation? 
That seems like it would break a lot of existing code (even though it is 
currently UB).  It would probably be better to silently set errno to 
ERANGE without returning EOF, and to set the integer object's value to 
the maximum or minimum value for its type (as it currently does for 
signed/unsigned long).

-- 
-=( Ian Abbott <abbotti@mev.co.uk> || MEV Ltd. is a company  )=-
-=( registered in England & Wales.  Regd. number: 02862268.  )=-
-=( Regd. addr.: S11 & 12 Building 67, Europa Business Park, )=-
-=( Bird Hall Lane, STOCKPORT, SK3 0XA, UK. || www.mev.co.uk )=-

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Zack Weinberg]

On Wed, 14 Dec 2022 06:03:07 -0500, Ian Abbott wrote:
> And would ERANGE cause scanf to return EOF in the fixed
> implementation?

In the counterfactual universe where I actually patch scanf, yes.

> That seems like it would break a lot of existing code
> (even though it is currently UB).  It would probably be better to
> silently set errno to ERANGE without returning EOF

My intuition says it’s the other way around.  What existing code do
you expect will break?

zw

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Zack Weinberg]

On Wed, 14 Dec 2022 05:47:12 -0500, Alejandro Colomar wrote:
> On 12/14/22 03:13, Zack Weinberg wrote:
> > The scanf numeric conversions could be fixed with a one-sentence edit to
> > the C standard: change the last sentence of http://port70.net/~nsz/c/c11/n1570.html#7.21.6.2p10
> > from “If this object does not have an appropriate type, or if the result
> > of the conversion cannot be represented in the object, the behavior is
> > undefined” to “If this object does not have an appropriate type, the
> > behavior is undefined.  If the result of the conversion cannot be
> > represented in the object, the execution of the directive fails; this
> > condition is a matching failure.”  And, even if the C committee doesn’t
> > want to make that change, open-source C libraries can and should do it
> > unilaterally, as a documented implementation extension.  I think that’s
> > a better plan than declaring most uses of *scanf “deprecated.”
> 
> Yeah, if you have plans to fix it, I'm fine removing the deprecation now. :)

To be clear, I personally don’t have plans to do any of the actual
programming or standard-changing work involved here.  :-)

> > Yes, I think it would be fair to say that it is almost always a mistake
> > to use the scanf variants that read directly from a FILE.  The issue
> > here is, at its root, that people new to C _expect_ a scanf call to read
> > an entire line of input, but it doesn’t. This is especially problematic
> > for interactive input ― they try to use plain scanf to read numeric
> > input, don’t realize that `scanf("%d", &arg)` doesn’t consume the \n in
> > the terminal’s line buffer _after_ the number, and get very confused
> > when a subsequent getchar() reads that \n instead of the ‘y’ or ‘n’ they
> > were expecting as a response to the _next_ prompt.  But it’s _also_ a
> > problem for error recovery, because scanf will stop in the middle of the
> > line when a matching failure occurs, and if you naively assumed it would
> > throw away the rest of the line, you get an error cascade.
> > 
> > The recommended practice to avoid this trap, is that you should use one
> > of the functions that _does_ read an entire line of input, i.e. fgets or
> > getline, and then parse the line as a string.  It would make sense for
> > the [f]scanf manpage to say that.
> 
> Please clarify; do you consider [v][f]scanf something that "we think
> we should never have added this in the first place, there’s no
> plausible way to fix it, but we have to keep it around for backward
> compatibility"?

_I_ wouldn’t have added them in the first place, but I care more than
the average about robustness in the face of unexpected input, even for
“throwaway” programs.  I doubt the C committee would be prepared to
say the same thing.  They _can_ be used legitimately, and they can
even be used in ways that meet my robustness standards if you go
to enough trouble.  It’s just that (IMNSHO) there are better ways to
reach those standards.

In terms of text for the [v][f]scanf manpage, maybe something like

[NOTES? BUGS?]
    When scanf() or fscanf() report a _matching failure_, all of the
    text that was matched successfully has still been read from
    _stdin_ or the _stream_ (respectively), and so have an
    unpredictable number of characters associated with the conversion
    that failed to match.  The latter characters are lost.  This may
    make it difficult to recover from invalid input.

    One way to make recovery easier is to separate reading from
    parsing: use fgets() or getline() to read an entire line of text
    into a string, then use sscanf() to parse the string.  If a
    _matching failure_ occurs, you can try sscanf() again with a
    different _format_; the equivalent is not possible using fscanf().

    _Successful_ calls to scanf() and fscanf() frequently consume
    either more, or fewer, characters from the input than was
    expected.  For example, assuming the next six characters readable
    from `stdin` are `"123\n a"‘, `scanf("%d", &val)` will consume the
    digits but _not_ the newline, and `scanf("%d\n", &val)‘ will
    consume the digits, the newline, _and_ the space.  Either of these
    is likely to cause trouble when mixing calls to scanf() with calls
    to fgets() or fgetc().  As above, it helps to read entire lines of
    text with fgets() or getline() and then parse them with sscanf().

> In short:
> 
> (1)  Numeric conversion specifiers are broken but can be fixed…

Yes.

> …and you plan to fix them.

No :)

>      (1.1)  I'll revert the deprecation warning now; since they are
> only broken because the _current_ standard and implementations are
> broken, but not by inherent design problems.

OK.

>      (1.2) When [someone fixes] the implementation to not be UB
> anymore, it will also make sense to revert the patch that removed
> the ERANGE error, since you'll need to report it.

Yes.

> (2)  For the string conversion specifiers, there are ways to use them
> safely.

Yes.

> and you plan to add a way to specify a size at runtime to the
> function,

No again :)

> so it will be even better in the future.  No action required.

Concur.

> (3)  [v][f]scanf seem to be really broken by design.  Please confirm.

See above.

If you remind me where to find the git repo for the manpages, I _may_
have time to write a patch for all this sometime next week.

zw

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 5528 bytes --]

Hi Zack,

On 12/29/22 07:39, Zack Weinberg wrote:
>> Yeah, if you have plans to fix it, I'm fine removing the deprecation now. :)
> 
> To be clear, I personally don’t have plans to do any of the actual
> programming or standard-changing work involved here.  :-)

Ah, no, I meant more that the whole set of glibc maintainers had that in mind, 
as a long term plan (like 10 years maybe?).  If you don't think these functions 
will be saved (even if we can) because it's not worth it, maybe we can just kill 
them.  But I'll defer that decission to after some fixes to the documentation.

>> Please clarify; do you consider [v][f]scanf something that "we think
>> we should never have added this in the first place, there’s no
>> plausible way to fix it, but we have to keep it around for backward
>> compatibility"?
> 
> _I_ wouldn’t have added them in the first place, but I care more than
> the average about robustness in the face of unexpected input, even for
> “throwaway” programs.  I doubt the C committee would be prepared to
> say the same thing.

I share your concerns about robustness of code, and am prepared to defend at 
least some partial/unofficial deprecation.  I'll clarify below.

> They _can_ be used legitimately, and they can
> even be used in ways that meet my robustness standards if you go
> to enough trouble.

That's not so clear to me.

>  It’s just that (IMNSHO) there are better ways to
> reach those standards.
> 
> In terms of text for the [v][f]scanf manpage, maybe something like
> 
> [NOTES? BUGS?]
>      When scanf() or fscanf() report a _matching failure_, all of the
>      text that was matched successfully has still been read from
>      _stdin_ or the _stream_ (respectively), and so have an
>      unpredictable number of characters associated with the conversion
>      that failed to match.  The latter characters are lost.  This may
>      make it difficult to recover from invalid input.
> 
>      One way to make recovery easier is to separate reading from
>      parsing: use fgets() or getline() to read an entire line of text
>      into a string, then use sscanf() to parse the string.  If a
>      _matching failure_ occurs, you can try sscanf() again with a
>      different _format_; the equivalent is not possible using fscanf().

This reads more or less as: "the only way to use scanf(3) is not to use it; use 
fgets(3)/getline(3) + sscanf(3) instead".

> 
>      _Successful_ calls to scanf() and fscanf() frequently consume
>      either more, or fewer, characters from the input than was
>      expected.  For example, assuming the next six characters readable
>      from `stdin` are `"123\n a"‘, `scanf("%d", &val)` will consume the
>      digits but _not_ the newline, and `scanf("%d\n", &val)‘ will
>      consume the digits, the newline, _and_ the space.  Either of these
>      is likely to cause trouble when mixing calls to scanf() with calls
>      to fgets() or fgetc().  As above, it helps to read entire lines of
>      text with fgets() or getline() and then parse them with sscanf().

And this reads as "even \"successful\" calls to scanf(3) are doomed; really, 
never use it".  :)

> 
>> In short:
>>
>> (1)  Numeric conversion specifiers are broken but can be fixed…
> 
> Yes.
> 
>> …and you plan to fix them.
> 
> No :)
> 
>>       (1.1)  I'll revert the deprecation warning now; since they are
>> only broken because the _current_ standard and implementations are
>> broken, but not by inherent design problems.
> 
> OK.
> 
>>       (1.2) When [someone fixes] the implementation to not be UB
>> anymore, it will also make sense to revert the patch that removed
>> the ERANGE error, since you'll need to report it.
> 
> Yes.
> 
>> (2)  For the string conversion specifiers, there are ways to use them
>> safely.
> 
> Yes.
> 
>> and you plan to add a way to specify a size at runtime to the
>> function,
> 
> No again :)
> 
>> so it will be even better in the future.  No action required.
> 
> Concur.
> 
>> (3)  [v][f]scanf seem to be really broken by design.  Please confirm.
> 
> See above.

Before you start writing patches, I'm considering the following, which is my way 
to say don't use these functions without deprecating them:

Split FILE and char* functions into separate manual pages.  In the one for 
[v]sscanf(3), I'd keep the current documentation.  In the one for FILE 
functions, I'd keep it very short, defering to sscanf(3) for documentation of 
things like conversion specifiers, and that page would only cover the 
bugs^Wdifferences that apply only to FILE functions.

I'll prepare some patches and show them for discussion in linux-man@.

> 
> If you remind me where to find the git repo for the manpages, I _may_
> have time to write a patch for all this sometime next week.

man-pages' README:

Versions
        Tarballs of releases starting from 2.00 are available at
        <https://mirrors.edge.kernel.org/pub/linux/docs/man-pages/>.

        The git(1) repository can be found at:
        <git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git>

        A secondary git(1) repository can be found at:
        <git://www.alejandro-colomar.es/src/alx/linux/man-pages/man-pages.git>

See also
        man-pages(7)

    Website
        <http://www.kernel.org/doc/man-pages/index.html>.


> 
> zw


Cheers,

Alex

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Zack Weinberg]

On Thu, 29 Dec 2022 05:47:06 -0500, Alejandro Colomar wrote:
> On 12/29/22 07:39, Zack Weinberg wrote:
> > To be clear, I personally don’t have plans to do any of the actual
> > programming or standard-changing work involved here.  :-)
> 
> Ah, no, I meant more that the whole set of glibc maintainers had that
> in mind, as a long term plan (like 10 years maybe?).

Oh, OK.  Yeah, changes to the standard can easily take that long.

> Before you start writing patches, I'm considering the following, which
> is my way to say don't use these functions without deprecating them:
> 
> Split FILE and char* functions into separate manual pages.  In the one
> for [v]sscanf(3), I'd keep the current documentation.  In the one for
> FILE functions, I'd keep it very short, defering to sscanf(3) for
> documentation of things like conversion specifiers, and that page
> would only cover the bugs^Wdifferences that apply only to FILE
> functions.

That seems like a good way forward to me.

zw

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 1263 bytes --]

Hi Zack,

On 12/29/22 17:35, Zack Weinberg wrote:
> On Thu, 29 Dec 2022 05:47:06 -0500, Alejandro Colomar wrote:
>> On 12/29/22 07:39, Zack Weinberg wrote:
>>> To be clear, I personally don’t have plans to do any of the actual
>>> programming or standard-changing work involved here.  :-)
>>
>> Ah, no, I meant more that the whole set of glibc maintainers had that
>> in mind, as a long term plan (like 10 years maybe?).
> 
> Oh, OK.  Yeah, changes to the standard can easily take that long.
> 
>> Before you start writing patches, I'm considering the following, which
>> is my way to say don't use these functions without deprecating them:
>>
>> Split FILE and char* functions into separate manual pages.  In the one
>> for [v]sscanf(3), I'd keep the current documentation.  In the one for
>> FILE functions, I'd keep it very short, defering to sscanf(3) for
>> documentation of things like conversion specifiers, and that page
>> would only cover the bugs^Wdifferences that apply only to FILE
>> functions.
> 
> That seems like a good way forward to me.

I've done the splitting.  If you would like to prepare any patches for adding 
BUGS, I'll take them :)

Cheers,

Alex

> 
> zw

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Ian Abbott]

On 12/12/2022 02:11, Zack Weinberg wrote:
> (Field widths are awkward to use because you have to write them as
> decimal constants _inside the format string_, which makes them more
> likely to get out of sync with the actual size of the buffer than,
> say, the buffer-size argument to ‘fgets’, but this is not a fatal
> flaw in and of itself.)

It's a shame that scanf's maximum field width couldn't be specified 
using an integer parameter in the same was as printf's minimum field 
width, but the '*' flag was already taken!

-- 
-=( Ian Abbott <abbotti@mev.co.uk> || MEV Ltd. is a company  )=-
-=( registered in England & Wales.  Regd. number: 02862268.  )=-
-=( Regd. addr.: S11 & 12 Building 67, Europa Business Park, )=-
-=( Bird Hall Lane, STOCKPORT, SK3 0XA, UK. || www.mev.co.uk )=-

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Zack Weinberg]

Ian Abbott <abbotti@mev.co.uk> writes:

> On 12/12/2022 02:11, Zack Weinberg wrote:
>> Field widths are awkward to use because you have to write them as
>> decimal constants _inside the format string_…
>
> It's a shame that scanf's maximum field width couldn't be specified
> using an integer parameter in the same was as printf's minimum field
> width, but the '*' flag was already taken!

Yup.  I suppose we could make up another flag … ‘@’ isn’t used for
anything …

zw

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Ian Abbott]

On 14/12/2022 02:18, Zack Weinberg wrote:
> Ian Abbott <abbotti@mev.co.uk> writes:
> 
>> On 12/12/2022 02:11, Zack Weinberg wrote:
>>> Field widths are awkward to use because you have to write them as
>>> decimal constants _inside the format string_…
>>
>> It's a shame that scanf's maximum field width couldn't be specified
>> using an integer parameter in the same was as printf's minimum field
>> width, but the '*' flag was already taken!
> 
> Yup.  I suppose we could make up another flag … ‘@’ isn’t used for
> anything …

'@' isn't included in C's basic character set though.  '&' is available.

-- 
-=( Ian Abbott <abbotti@mev.co.uk> || MEV Ltd. is a company  )=-
-=( registered in England & Wales.  Regd. number: 02862268.  )=-
-=( Regd. addr.: S11 & 12 Building 67, Europa Business Park, )=-
-=( Bird Hall Lane, STOCKPORT, SK3 0XA, UK. || www.mev.co.uk )=-

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 903 bytes --]

Hi Ian & Zack,

On 12/14/22 11:22, Ian Abbott wrote:
> On 14/12/2022 02:18, Zack Weinberg wrote:
>> Ian Abbott <abbotti@mev.co.uk> writes:
>>
>>> On 12/12/2022 02:11, Zack Weinberg wrote:
>>>> Field widths are awkward to use because you have to write them as
>>>> decimal constants _inside the format string_…
>>>
>>> It's a shame that scanf's maximum field width couldn't be specified
>>> using an integer parameter in the same was as printf's minimum field
>>> width, but the '*' flag was already taken!
>>
>> Yup.  I suppose we could make up another flag … ‘@’ isn’t used for
>> anything …
> 
> '@' isn't included in C's basic character set though.  '&' is available.

Just a curious question from an ignorant:  what's the difference between the 
basic character set and the source character set?

Thanks,

Alex

> 

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Ian Abbott]

On 2022-12-14 10:39, Alejandro Colomar wrote:
> Hi Ian & Zack,
> 
> On 12/14/22 11:22, Ian Abbott wrote:
>> On 14/12/2022 02:18, Zack Weinberg wrote:
>>> Ian Abbott <abbotti@mev.co.uk> writes:
>>>
>>>> On 12/12/2022 02:11, Zack Weinberg wrote:
>>>>> Field widths are awkward to use because you have to write them as
>>>>> decimal constants _inside the format string_…
>>>>
>>>> It's a shame that scanf's maximum field width couldn't be specified
>>>> using an integer parameter in the same was as printf's minimum field
>>>> width, but the '*' flag was already taken!
>>>
>>> Yup.  I suppose we could make up another flag … ‘@’ isn’t used for
>>> anything …
>>
>> '@' isn't included in C's basic character set though.  '&' is available.
> 
> Just a curious question from an ignorant:  what's the difference between 
> the basic character set and the source character set?

The source character set may contain locale-specific characters outside 
the basic source character set.

Actually, there are two basic character sets - the basic source 
character set and the basic execution character set (which includes the 
basic source character set plus a few control characters).  The source 
character set and/or execution character set may contain 
locale-specific, extended characters outside the basic character set.

https://port70.net/~nsz/c/c11/n1570.html#5.2.1

-- 
-=( Ian Abbott <abbotti@mev.co.uk> || MEV Ltd. is a company  )=-
-=( registered in England & Wales.  Regd. number: 02862268.  )=-
-=( Regd. addr.: S11 & 12 Building 67, Europa Business Park, )=-
-=( Bird Hall Lane, STOCKPORT, SK3 0XA, UK. || www.mev.co.uk )=-

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 1420 bytes --]



On 12/14/22 11:52, Ian Abbott wrote:
>>>
>>> '@' isn't included in C's basic character set though.  '&' is available.
>>
>> Just a curious question from an ignorant:  what's the difference between the 
>> basic character set and the source character set?
> 
> The source character set may contain locale-specific characters outside the 
> basic source character set.
> 
> Actually, there are two basic character sets - the basic source character set 
> and the basic execution character set (which includes the basic source character 
> set plus a few control characters).  The source character set and/or execution 
> character set may contain locale-specific, extended characters outside the basic 
> character set.
> 
> https://port70.net/~nsz/c/c11/n1570.html#5.2.1

I still have a small doubt.  C23 added '@' to the source character set, but 
seems to be a second-class citizen:



  The execution character set may also contain multibyte characters, which
need not have the same encoding as for the source character set. For both 
character sets, the following
shall hold:
— The basic character set, @, $, and ` shall be present and each character shall 
be encoded as a
single byte.



What's the difference, and why isn't it part of the basic character set?  Maybe 
because not all keyboards have those three characters?
> 

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Ian Abbott]

On 14/12/2022 11:23, Alejandro Colomar wrote:
> 
> 
> On 12/14/22 11:52, Ian Abbott wrote:
>>>>
>>>> '@' isn't included in C's basic character set though.  '&' is 
>>>> available.
>>>
>>> Just a curious question from an ignorant:  what's the difference 
>>> between the basic character set and the source character set?
>>
>> The source character set may contain locale-specific characters 
>> outside the basic source character set.
>>
>> Actually, there are two basic character sets - the basic source 
>> character set and the basic execution character set (which includes 
>> the basic source character set plus a few control characters).  The 
>> source character set and/or execution character set may contain 
>> locale-specific, extended characters outside the basic character set.
>>
>> https://port70.net/~nsz/c/c11/n1570.html#5.2.1
> 
> I still have a small doubt.  C23 added '@' to the source character set, 
> but seems to be a second-class citizen:
> 
> 
> 
>   The execution character set may also contain multibyte characters, which
> need not have the same encoding as for the source character set. For 
> both character sets, the following
> shall hold:
> — The basic character set, @, $, and ` shall be present and each 
> character shall be encoded as a
> single byte.
> 
> What's the difference, and why isn't it part of the basic character 
> set?  Maybe because not all keyboards have those three characters?

I think the inability to type certain characters in the basic source 
character set is the reason why the language contains the horrible 
trigraph sequences (no longer valid since the C23 final draft N3054), 
and the slightly less horrible digraph tokens.

Here is the rationale for inclusion of @ and $ in the source and 
execution character sets, but ` is only mentioned briefly as an also-ran 
at the end of the document in section "Do we also want to add ` in the 
same way as @ and $?":

https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2701.htm

The rationale for exclusion of @ and $ characters from the basic 
character set is given in this paragraph from the document:

"""
By requiring @ and $ in the source and execution character set we, reach 
the goal of making them useable in comments and string literals. By not 
adding them to the basic source character set, we protect the freedom of 
implementations of allowing or disallowing them in identifiers, and 
avoid inconsistency or incompability regarding the use of universal 
character names (currently the use of universal character names for 
characters in the basic source character set is not allowed, so adding 
characters to the basic source character set without lifting that 
restriction could break existing code).
"""

I guess it was decided to add all three proposed characters during the 
Jan/Feb 2022 virtual meeting of WG14 as mentioned here:

https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2913.htm

The first C2x draft that incorporated the change is this one:

https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2912.pdf

-- 
-=( Ian Abbott <abbotti@mev.co.uk> || MEV Ltd. is a company  )=-
-=( registered in England & Wales.  Regd. number: 02862268.  )=-
-=( Regd. addr.: S11 & 12 Building 67, Europa Business Park, )=-
-=( Bird Hall Lane, STOCKPORT, SK3 0XA, UK. || www.mev.co.uk )=-

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Joseph Myers]

On Wed, 14 Dec 2022, Alejandro Colomar via Libc-alpha wrote:

> What's the difference, and why isn't it part of the basic character set?

Apart from the point discussed about how making them part of the basic 
character set would interact with other rules involving that character 
set, the basic source character set consists of those characters that have 
some specified role in the C syntax.  There is no specified role for $ @ ` 
in the C syntax - they can only be used in string or character literals 
(modulo the question of whether $ should still be allowed as an 
implementation-defined character in identifiers, see N3046) and have no 
special role to play there.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Ian Abbott]

On 09/12/2022 19:33, Alejandro Colomar wrote:
> Hi Ian,
> 
> On 12/9/22 20:28, Ian Abbott wrote:
>> On 09/12/2022 18:59, Alejandro Colomar wrote:
>>> On 12/8/22 13:34, Ian Abbott wrote:
>>>> The `scanf()` function does not intentionally set `errno` to `ERANGE`.
>>>> That is just a side effect of the code that it uses to perform
>>>> conversions.  It also does not work as reliably as indicated in the
>>>> 'man' page when the target integer type is narrower than `long`.
>>>> Typically (at least in glibc) for target integer types narrower than
>>>> `long`, the number has to exceed the range of `long` (for signed
>>>> conversions) or `unsigned long` (for unsigned conversions) for `errno`
>>>> to be set to `ERANGE`.
>>>>
>>>> Documenting `ERANGE` in the ERRORS section kind of implies that
>>>> `scanf()` should return `EOF` when an integer overflow is encountered,
>>>> which it doesn't (and doing so would violate the C standard).
>>>>
>>>> Just remove any mention of the `ERANGE` error to avoid confusion.
>>>>
>>>> Fixes: 646af540e467 ("Add an ERRORS section documenting at least 
>>>> some of the errors that may occur for scanf().")
>>>> Cc: Michael Kerrisk <mtk.manpages@gmail.com>
>>>> Signed-off-by: Ian Abbott <abbotti@mev.co.uk>
>>>
>>> I see.  How about saying something like "it may also fail for any of 
>>> any errors that functions used to perform the conversions may fail"?
>>
>> It depends what you mean by "fail".  These errors do not make scanf 
>> return EOF. 
> 
> Just to clarify.  Does scanf(3) _never_ fail (EOF) due to ERANGE?  Or is 
> it that ERANGE sometimes makes it fail, sometimes not?

The glibc implementation certainly doesn't return EOF when ERANGE is 
detected.  __vfscanf_internal() in stdio-common/vfscan-internal.c does 
not contain any code to deal with ERANGE - it's just a side-effect of 
the calls to __strtol_internal(), __strtoul_internal(), 
__strtoll_internal(), or __strtoull_internal().

> If it's the former, I agree with your patch.  When a function hasn't 
> reported failure, errno is unspecified.
> 
> If it's the latter, I'd write something about it.

For the glibc implementation, it's the former.

>> Technically, the behavior is undefined if the result of the conversion 
>> cannot be represented in the object being assigned to by scanf.  (In 
>> the case of glibc, that probably results in either the integer object 
>> being set to a truncated version of the input integer, or the integer 
>> object being set to a truncated version of LONG_MIN or LONG_MAX, 
>> depending on the actual number.)
> 
> Hmm, UB.  Under UB, anything can change, so error reporting is already 
> unreliable.  If EOF+ERANGE can _only_ happen under UB, I'd rather remove 
> the paragraph.  Please confirm.

Yes, it is UB as per C17 7.21.6 paragraph 10: "[...] Unless assignment 
suppression was indicated by a *, the result of the conversion is placed 
in the object pointed to by the first argument following the format 
argument that has not already received a conversion result. If this 
object does not have an appropriate type, or if the result of the 
conversion cannot be represented in the object, the behavior is undefined."

>> Setting errno to 0 before calling scanf and expecting errno to have a 
>> meaningful value when scanf returns something other than EOF is bogus 
>> usage.
> 
> Yep, that's bogus.
> 
> 
> Cheers,
> 
> Alex

Best regards,
Ian

-- 
-=( Ian Abbott <abbotti@mev.co.uk> || MEV Ltd. is a company  )=-
-=( registered in England & Wales.  Regd. number: 02862268.  )=-
-=( Regd. addr.: S11 & 12 Building 67, Europa Business Park, )=-
-=( Bird Hall Lane, STOCKPORT, SK3 0XA, UK. || www.mev.co.uk )=-

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 2126 bytes --]

Hi Ian & Zack,

On 12/8/22 13:34, Ian Abbott wrote:
> The `scanf()` function does not intentionally set `errno` to `ERANGE`.
> That is just a side effect of the code that it uses to perform
> conversions.  It also does not work as reliably as indicated in the
> 'man' page when the target integer type is narrower than `long`.
> Typically (at least in glibc) for target integer types narrower than
> `long`, the number has to exceed the range of `long` (for signed
> conversions) or `unsigned long` (for unsigned conversions) for `errno`
> to be set to `ERANGE`.
> 
> Documenting `ERANGE` in the ERRORS section kind of implies that
> `scanf()` should return `EOF` when an integer overflow is encountered,
> which it doesn't (and doing so would violate the C standard).
> 
> Just remove any mention of the `ERANGE` error to avoid confusion.
> 
> Fixes: 646af540e467 ("Add an ERRORS section documenting at least some of the errors that may occur for scanf().")
> Cc: Michael Kerrisk <mtk.manpages@gmail.com>
> Signed-off-by: Ian Abbott <abbotti@mev.co.uk>

I applied the patch; thanks!  And also applied another one explicitly marking 
all those numeric conversion specifiers as deprecated:

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=a15d34326c581eab107bf05782cc60d8ebdcad69>

Cheers,

Alex

> ---
>   man3/scanf.3 | 7 -------
>   1 file changed, 7 deletions(-)
> 
> diff --git a/man3/scanf.3 b/man3/scanf.3
> index ba470a5c1..c5ff59f45 100644
> --- a/man3/scanf.3
> +++ b/man3/scanf.3
> @@ -576,10 +576,6 @@ is NULL.
>   .TP
>   .B ENOMEM
>   Out of memory.
> -.TP
> -.B ERANGE
> -The result of an integer conversion would exceed the size
> -that can be stored in the corresponding integer type.
>   .SH ATTRIBUTES
>   For an explanation of the terms used in this section, see
>   .BR attributes (7).
> @@ -609,9 +605,6 @@ The functions
>   and
>   .BR sscanf ()
>   conform to C89 and C99 and POSIX.1-2001.
> -These standards do not specify the
> -.B ERANGE
> -error.
>   .PP
>   The
>   .B q

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Eric Biggers]

Hi Alejandro,

On Mon, Dec 12, 2022 at 12:33:08PM +0100, Alejandro Colomar wrote:
> > Documenting `ERANGE` in the ERRORS section kind of implies that
> > `scanf()` should return `EOF` when an integer overflow is encountered,
> > which it doesn't (and doing so would violate the C standard).
> > 
> > Just remove any mention of the `ERANGE` error to avoid confusion.
> > 
> > Fixes: 646af540e467 ("Add an ERRORS section documenting at least some of the errors that may occur for scanf().")
> > Cc: Michael Kerrisk <mtk.manpages@gmail.com>
> > Signed-off-by: Ian Abbott <abbotti@mev.co.uk>
> 
> I applied the patch; thanks!  And also applied another one explicitly
> marking all those numeric conversion specifiers as deprecated:
> 
> <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=a15d34326c581eab107bf05782cc60d8ebdcad69>
> 
> Cheers,
> 

This is commonly used functionality with no replacement suggested.  Should it
really be deprecated?  Is the undefined behavior here a real world issue
anywhere, or is this just a theoretical issue based on interpretation of the C
standard?  Perhaps instead of a deprecation, this just deserves a mention in the
BUGS section of the man page, and a request to the C standard committee to fix
the text in the standard to make the behavior defined?

I'll note that memcpy() is not deprecated, even though the C standard has a bug
where 'memcpy(dst, src, 0)' is undefined if either pointer is NULL.  That's a
very similar sort of issue.  The fix, of course, is for implementations to
ignore the defective standard and make the behavior defined...

- Eric

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 3285 bytes --]



On 1/20/23 05:09, Eric Biggers wrote:
> Hi Alejandro,
> 

[...]

>>
>> <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=a15d34326c581eab107bf05782cc60d8ebdcad69>
>>
>> Cheers,
>>
> 
> This is commonly used functionality with no replacement suggested.

Maybe we should suggest using strtol(3) in BUGS.

>  Should it
> really be deprecated?

While the interface of sscanf(3) numeric conversions is not mis-designed and 
could be fixed, it is not correctly implemented, nor even standardized.

I think it's correct to deprecate unless there's a clear effort to fix it.

This is similar but different to bzero(3).  bzero(3) was broken or slow in some 
implementations.  That's probably why it was never added to ISO C, and why POSIX 
later removed it.  The API wasn't bad, and in fact it's great, I prefer it over 
memset(3).  The difference between bzero(3) and sscanf(3) is that bzero(3) has 
now been fixed, and is _the best_ API, while sscanf(3) is still broken.

>  Is the undefined behavior here a real world issue
> anywhere, or is this just a theoretical issue based on interpretation of the C
> standard?

All implementations of sscanf(3) produce Undefined Behavior (UB), AFAIK.  How 
much you consider UB to be a real-world issue differs for each programmer, but I 
tend to consider all UB to be as bad as nasal demons.  I'm not saying UB 
shouldn't exist, just that you shouldn't invoke it.  And a function that is used 
for scanning user input is one of those places where you really want to avoid 
invoking UB.

>  Perhaps instead of a deprecation, this just deserves a mention in the
> BUGS section of the man page, and a request to the C standard committee to fix
> the text in the standard to make the behavior defined?

I could change the "deprecated" statements by "see bugs", but I don't think it's 
a reasonable difference.  However, yes, I should still add something to BUGS.

While I'm doing some effort to fix implementations and standards in other APIs 
that matter to me, such as sockaddr_storage, or bzero(3), I don't use sscanf(3), 
so I'm sorry that I won't do the same for it.

However, if somebody really wants to use that function, and would like to fix 
it, I encourage that effort.  If the function is fixed, which shouldn't be that 
hard, I'm fine removing the messages against its usage in the manual.

While that doesn't happen, I prefer strongly recommending against their usage in 
the manual.  And dict(1) seems to say that the verb for that is "to deprecate" :)

> 
> I'll note that memcpy() is not deprecated, even though the C standard has a bug
> where 'memcpy(dst, src, 0)' is undefined if either pointer is NULL.  That's a
> very similar sort of issue.  The fix, of course, is for implementations to
> ignore the defective standard and make the behavior defined...

Right.  memcpy(3) has a bug in the standard.  However, implementations do the 
Right Thing (tm).  If implementations did the right thing for sscanf(3), that 
would be enough to remove the recommendation against it.  But my understanding 
is that the sscanf(3) implementation is not free of that problem.

> 
> - Eric

Cheers,

Alex

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[G. Branden Robinson]

[-- Attachment #1: Type: text/plain, Size: 5155 bytes --]

[re-ordering the mail I'm quoting]

Hi Alex,

I have some observations on your deprecation initiative and people's
reactions to it.

At 2023-01-20T14:12:07+0100, Alejandro Colomar wrote:
> All implementations of sscanf(3) produce Undefined Behavior (UB),
> AFAIK.  How much you consider UB to be a real-world issue differs for
> each programmer, but I tend to consider all UB to be as bad as nasal
> demons.  I'm not saying UB shouldn't exist, just that you shouldn't
> invoke it.  And a function that is used for scanning user input is one
> of those places where you really want to avoid invoking UB.

If there are common idioms that result in UB, it might be worth
documenting this in the man page, with a citation to the relevant
clause of the standard that declares it thus.  I agree that UB is
something to be avoided and I think most other programmers do too.  The
advantage to this approach is that if they disagree, they can take their
argument to the standards body instead of litigating it with you.

> This is similar but different to bzero(3).  bzero(3) was broken or
> slow in some implementations.  That's probably why it was never added
> to ISO C, and why POSIX later removed it.  The API wasn't bad, and in
> fact it's great, I prefer it over memset(3).  The difference between
> bzero(3) and sscanf(3) is that bzero(3) has now been fixed,

I still don't share your preference here.  The exposure of a more
general interface (memset) by a general-purpose library when the
implementation otherwise has no additional implementation cost is the
correct choice.  If a given programmer's use cases are restricted such
that one of the arguments to a general-purpose function is constant,
then that is exactly the time for them to write a macro or function
specific to their project to hide the complexity.

If you tilt your head right, this is similar to one of the ways closures
are used in other languages.

> I could change the "deprecated" statements by "see bugs",

I think you've hit upon one of the core drivers of resistance here.  A
problem with calling something "deprecated" is that it's often unstated
_who_ is doing the deprecation.  Traditionally, I think the Linux
man-pages have tended only to use this term in reference to one of the
standards bodies (WG14 or the Austin Group) formally employing it.

(Maybe I'm wrong, and Linux man-pages _has_ deprecated things in its own
authorial voice...but if other people also don't know that, it doesn't
matter, and confusion remains.)

So I suggest you adopt a new phrase, like "discouraged by Linux
man-pages", to characterize the authorial voice here.  Some people will
ignore your advice either way, but at least they'll know who they're
ignoring.[1]

> However, if somebody really wants to use that function, and would like
> to fix it, I encourage that effort.  If the function is fixed, which
> shouldn't be that hard, I'm fine removing the messages against its
> usage in the manual.
> 
> While that doesn't happen, I prefer strongly recommending against
> their usage in the manual.  And dict(1) seems to say that the verb for
> that is "to deprecate" :)

Your dictionary is correct but social knowledge, a.k.a. tradition and
folklore, impose a context on the discussion.  Sometimes dumb things
become tradition (like calculating factorials or Fibonacci numbers with
recursive functions[2])--we don't have to acquiesce to that, but we will
have to document and sometimes defend our rejection of them.

> Right.  memcpy(3) has a bug in the standard.  However, implementations
> do the Right Thing (tm).  If implementations did the right thing for
> sscanf(3), that would be enough to remove the recommendation against
> it.  But my understanding is that the sscanf(3) implementation is not
> free of that problem.

This is a good opportunity to say so in these terms.  "Linux man-pages
discourages use of sscanf [under the conditions XXX] until
implementations are corrected to avoid undefined behavior [cite URL
here]."[3]

Regards,
Branden

[1] In groff_man(7), I admit I have not taken my own advice, and use the
    term "deprecated" in a subsection heading.  I have two defenses for
    this.  (1) I reorganized the man page along those lines 5-6 years
    ago, when I had less practice at writing technical documentation,
    and (2) the man(7) macros are not formally standardized anywhere
    anyway.  There is no "official" body with which to conflict, or with
    whom groff can be confused by the reader.

    After groff 1.23 is released (good news, I heard from Bertrand last
    weekend) I hope to add the SunOS extension "SB" to the deprecation
    list now that Solaris's death seems irreversible.

[2] https://sleeplessafternoon.wordpress.com/2013/03/26/examples-of-recursion-the-good-the-bad-and-the-silly/

    For the mathematically or algorithmically inclined, I also
    recommend "The Genuine Sieve of Eratosthenes", by Melissa O'Neill.

    https://www.cs.hmc.edu/~oneill/papers/Sieve-JFP.pdf

[3] groff_man(7) gives you UR/UE, so use them!  >:-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 9012 bytes --]

Hi Branden!

On 1/20/23 18:55, G. Branden Robinson wrote:
> [re-ordering the mail I'm quoting]
> 
> Hi Alex,
> 
> I have some observations on your deprecation initiative and people's
> reactions to it.

Sure :)

> 
> At 2023-01-20T14:12:07+0100, Alejandro Colomar wrote:
>> All implementations of sscanf(3) produce Undefined Behavior (UB),
>> AFAIK.  How much you consider UB to be a real-world issue differs for
>> each programmer, but I tend to consider all UB to be as bad as nasal
>> demons.  I'm not saying UB shouldn't exist, just that you shouldn't
>> invoke it.  And a function that is used for scanning user input is one
>> of those places where you really want to avoid invoking UB.
> 
> If there are common idioms that result in UB, it might be worth
> documenting this in the man page, with a citation to the relevant
> clause of the standard that declares it thus.

Okay.  See proposed diff below

>  I agree that UB is
> something to be avoided and I think most other programmers do too.  The
> advantage to this approach is that if they disagree, they can take their
> argument to the standards body instead of litigating it with you.

:)

> 
>> This is similar but different to bzero(3).  bzero(3) was broken or
>> slow in some implementations.  That's probably why it was never added
>> to ISO C, and why POSIX later removed it.  The API wasn't bad, and in
>> fact it's great, I prefer it over memset(3).  The difference between
>> bzero(3) and sscanf(3) is that bzero(3) has now been fixed,
> 
> I still don't share your preference here.  The exposure of a more
> general interface (memset) by a general-purpose library when the
> implementation otherwise has no additional implementation cost is the
> correct choice.

While I share your interest in general-purpose over specialized, and that's in 
essence the essence of Unix, I also believe that encapsulation is very necessary 
for writing readable code.

Your (and many others') proposal of having a project-specific macro for bzero(3) 
seems reasonable in absence of a standard name for it.  However, having a 
POSIX-blessed (until recently) name for such an interface, I'd prefer sticking 
to it.  Otherwise, we risk having bzero(), memzero(), zerobytes(), zero(), ... 
which is not crazy, but hey, I prefer less moving parts when reading code :)

As for removing from POSIX a function just because it's not generic... I have in 
mind a long list of such features that are equally trivial and unnecessary (and 
in some cases, they hurt unlike bzero(3), IMO), yet they haven't died.  For a 
representative, let me present our friend:

	printf(3)

Oh boy, tell me it hurts your fingers writing fprintf(stdout, ) but not memset(, 
0, ).  At least with fprintf(3) it's obvious the ordering of the parameters and 
I don't need to check the man page.


>  If a given programmer's use cases are restricted such

It's not a single given programmer.  memset(3) is likely to be the most obvious 
case where the thin wrapper is what you want to call.  There are many uses for 
fprintf(3), there are many uses for other such functions that have a thin 
wrapper in the same libc, but memset(3)?  How much you've (or any code you know) 
used it with something other than constant expression 0?

> that one of the arguments to a general-purpose function is constant,
> then that is exactly the time for them to write a macro or function
> specific to their project to hide the complexity.
> 
> If you tilt your head right, this is similar to one of the ways closures
> are used in other languages.

I'm fine with the function being implemented as a macro, although it would be 
better to have it as an inline function, so that -Os can produce smaller code if 
needed.  In general, I don't like macros unless there's a need to avoid type 
conversions; for example for keeping arrays as arrays.

> 
>> I could change the "deprecated" statements by "see bugs",
> 
> I think you've hit upon one of the core drivers of resistance here.  A
> problem with calling something "deprecated" is that it's often unstated
> _who_ is doing the deprecation.  Traditionally, I think the Linux
> man-pages have tended only to use this term in reference to one of the
> standards bodies (WG14 or the Austin Group) formally employing it.

There are some pages which have single-handedly deprecated features with no 
standard or group doing so.  I remember having seen a few pages do that, but 
they are all from prehistoric times, when standards didn't mean so much (or 
maybe there weren't such standards).

> 
> (Maybe I'm wrong, and Linux man-pages _has_ deprecated things in its own
> authorial voice...but if other people also don't know that, it doesn't
> matter, and confusion remains.)

Yes, they did.  Well, confusion always happens when things change.  I expect 
that to settle down.  However, I'll try to improve my methods for deprecating 
broken stuff as much as I can so we can reduce the confusion.

> 
> So I suggest you adopt a new phrase, like "discouraged by Linux
> man-pages", to characterize the authorial voice here.  Some people will
> ignore your advice either way, but at least they'll know who they're
> ignoring.[1]

I like deprecating.  I want such a strong term.  I'll try to clarify that it's 
the man-pages that do the deprecation, and not a standards body.

> 
>> However, if somebody really wants to use that function, and would like
>> to fix it, I encourage that effort.  If the function is fixed, which
>> shouldn't be that hard, I'm fine removing the messages against its
>> usage in the manual.
>>
>> While that doesn't happen, I prefer strongly recommending against
>> their usage in the manual.  And dict(1) seems to say that the verb for
>> that is "to deprecate" :)
> 
> Your dictionary is correct but social knowledge, a.k.a. tradition and
> folklore, impose a context on the discussion.  Sometimes dumb things
> become tradition (like calculating factorials or Fibonacci numbers with
> recursive functions[2])--we don't have to acquiesce to that, but we will
> have to document and sometimes defend our rejection of them.
> 
>> Right.  memcpy(3) has a bug in the standard.  However, implementations
>> do the Right Thing (tm).  If implementations did the right thing for
>> sscanf(3), that would be enough to remove the recommendation against
>> it.  But my understanding is that the sscanf(3) implementation is not
>> free of that problem.
> 
> This is a good opportunity to say so in these terms.  "Linux man-pages
> discourages use of sscanf [under the conditions XXX] until
> implementations are corrected to avoid undefined behavior [cite URL
> here]."[3]
> 
> Regards,
> Branden
> 
> [1] In groff_man(7), I admit I have not taken my own advice, and use the
>      term "deprecated" in a subsection heading.  I have two defenses for
>      this.  (1) I reorganized the man page along those lines 5-6 years
>      ago, when I had less practice at writing technical documentation,
>      and (2) the man(7) macros are not formally standardized anywhere
>      anyway.  There is no "official" body with which to conflict, or with
>      whom groff can be confused by the reader.
> 
>      After groff 1.23 is released (good news, I heard from Bertrand last
>      weekend)

Nice :)

> I hope to add the SunOS extension "SB" to the deprecation
>      list now that Solaris's death seems irreversible.
> 
> [2] https://sleeplessafternoon.wordpress.com/2013/03/26/examples-of-recursion-the-good-the-bad-and-the-silly/
> 
>      For the mathematically or algorithmically inclined, I also
>      recommend "The Genuine Sieve of Eratosthenes", by Melissa O'Neill.
> 
>      https://www.cs.hmc.edu/~oneill/papers/Sieve-JFP.pdf
> 
> [3] groff_man(7) gives you UR/UE, so use them!  >:-)

How about the following?

Cheers,

Alex

---

diff --git a/man3/sscanf.3 b/man3/sscanf.3
index 26a02521b..870c6f54b 100644
--- a/man3/sscanf.3
+++ b/man3/sscanf.3
@@ -653,6 +653,25 @@ .SS The 'a' assignment-allocation modifier
  .I gcc\~\-std=c99
  etc.).
  .SH BUGS
+.SS Numeric conversion specifiers
+Use of the numeric conversion specifiers produces Undefined Behavior
+for invalid input.
+See
+.UR https://port70.net/\:%7Ensz/\:c/\:c11/\:n1570.html\:#7.21.6.2p10
+C11 7.21.6.2/10
+.UE .
+This is a bug in the ISO C standard,
+and not an inherent design issue with the API.
+However,
+current implementations are not safe from that bug,
+so it is not recommended to use them.
+Instead,
+programs should use functions such as
+.BR strtol (3)
+to parse numeric input.
+This manual page deprecates use of the numeric conversion specifiers
+until they are fixed by ISO C.
+.SS Nonstandard modifiers
  These functions are fully C99 conformant, but provide the
  additional modifiers
  .B q


-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] scanf.3: Do not mention the ERANGE error

[Eric Biggers]

On Fri, Jan 20, 2023 at 02:12:07PM +0100, Alejandro Colomar wrote:
> >  Is the undefined behavior here a real world issue
> > anywhere, or is this just a theoretical issue based on interpretation of the C
> > standard?
> 
> All implementations of sscanf(3) produce Undefined Behavior (UB), AFAIK.
> How much you consider UB to be a real-world issue differs for each
> programmer, but I tend to consider all UB to be as bad as nasal demons.  I'm
> not saying UB shouldn't exist, just that you shouldn't invoke it.  And a
> function that is used for scanning user input is one of those places where
> you really want to avoid invoking UB.

Well, according to the C standard, the behavior is undefined when the value
scanned by an integer conversion specifier does not fit in the type.  That's
clearly a bug in the standard.  Obviously, implementations need not implement
that bug, because it is stupid -- much stupider than other cases of UB in the
standard.  It's not unreasonable to focus on what implementations actually do.

In general, compilers optimize code assuming that undefined behavior never
occurs.  However, this specific type of undefined behavior is pretty obscure,
and it is hard to think of any compiler optimization that would apply to it.
I'm doubtful that any has been implemented.

UBSAN does not detect this as undefined behavior either.  (Tested with
'sscanf("99999999999999999999", "%d", &x' with gcc 12.2.0 and clang 14.0.6.)

The remaining question, then, would be whether any actual sscanf()
implementation would actually do something "bad" given one of these "undefined"
inputs.  I think the main pitfall would be that a naive implementation of %d and
other *signed* integer conversion specifiers would execute a signed integer
multiplication that overflows.  That's a more well established undefined
behavior.  Though, some implementations make that behavior defined too.

Unsigned specifiers such as %u should fare better, as unsigned integer overflow
has defined behavior.  I.e., it would be much harder to write an implementation
of %u that invokes undefined behavior due to the value being too large.

It might be fair to say that behavior here is de facto implementation-defined,
despite the standard saying undefined...

Anyway, if you do go the hard-line route of "undefined is undefined, so let's
deprecate", you need to make it clear (a) who is doing the deprecation, (b) what
the actual issue is, and (c) what the replacement is.

- Eric