Status for bug 71211? [random(4): clarify utility and volume]

Status for bug 71211? [random(4): clarify utility and volume]

[Carl Winbäck]

Hello Michael & Co,

I would like to bring your attention to bug report 71211, ”random(4):
clarify utility and volume”.

https://bugzilla.kernel.org/show_bug.cgi?id=71211

This report was filed over a year ago but still hasn’t received any response.

Michael, do you have the time to take a look?

Perhaps you, or someone else on the linux-man list, are familiar with
CSPRNGs and have some ideas on how to reword this man page?

Unfortunately I’m not at all an expert in this area, so I’m afraid I
don’t have any specific suggestions regarding how to change this man
page. But I still think it would be helpful to the Linux community if
it could be improved, and as a result, hopefully cause less confusion
regarding getting random numbers on Linux.


Looking forward to hear your thoughts on this.

Best regards,
Carl Winbäck
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status for bug 71211? [random(4): clarify utility and volume]

[Laurent Georget]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hello,

the text of this man page has been the subject of endless controversies
for ages. CSPRNGs are confusing and the page unfortunately adds a
little to the confusion. The newer getrandom(2) page is better, and I
(personally, this doesn't engage Michael nor any other author) think
that the random(4) page should be rewritten in the same spirit.
(getrandom is a system call used to get a random number, by default,
it's more or less equivalent than reading from /dev/urandom if you call
it without flags and for less than 256 bytes).

Compare this in random(4)

> The kernel random-number generator is designed to produce a small
> amount of high-quality seed material to seed a cryptographic pseudo-
> random number generator (CPRNG). It is designed for security, not
> speed, and is poorly suited to generating large amounts of random
> data. Users should be very economical in the amount of seed material
> that they read from /dev/urandom (and /dev/random); unnecessarily
> reading large quantities of data from this device will have a
> negative impact on other users of the device.
with this in getrandom(2)

> *getrandom*() relies on entropy gathered from device drivers and other
> sources of environmental noise.  Unnecessarily reading large
> quantities of data will have a negative impact on other users of the
> //dev/random/ and //dev/urandom/ devices.  Therefore, *getrandom*() should
> not be used for Monte Carlo simulations or other programs/algorithms
> which are doing probabilistic sampling.
This says exactly the same thing, but getrandom(2) is less misleading.
First note that the man page for random says that /dev/urandom is
"poorly suited to generating large amounts of random data", not
"poorly suited to generating large amounts of *cryptographic* random data".
Basically, you can use /dev/urandom for all cryptographic purposes,
because you never need so many bits at once when doing cryptography.
Even generating several 16-bytes (128-bits) UIDs per minute if you need
them to be cryptographically secure (you may want to think about this
requirement, is it strict?) is not that much. A Monte-Carlo simulation
requires, say (I don't know exactly let's make a rough guess) around
several MB of random numbers per minute. That's 4 or 5 orders of
magnitude higher than your UIDs use case. This would be a wrong
usage of /dev/urandom for two reasons:
- - it would be awfully slow
- - you don't need cryptographically secure random numbers, so there's
no need to deplete the entropy pool.
Next, compare this in random(4):

> If  you    are  unsure  about  whether  you  should  use  /dev/random  or
> /dev/urandom,  then  probably you want to use the latter.  As a general
> rule, /dev/urandom should be  used  for    everything  except  long-lived
> GPG/SSL/SSH keys.
with this in getrandom(2):

> Unless  you  are     doing    long-term key generation (and perhaps not even
> then), you probably shouldn't be using GRND_RANDOM.  The     cryptographic
> algorithms  used for /dev/urandom are quite conservative, and so should
> be sufficient for all purposes.    The  disadvantage  of  GRND_RANDOM  is
> that  it     can block.  Furthermore, dealing with the partially fulfilled
> getrandom() requests that can occur when     using    GRND_RANDOM  increases
> code complexity.
Again, the two man pages say the same. getrandom(2) is more nuanced,
though.

To answer your question about how much you can ask /dev/urandom for :

in random(4) :

> if any program reads more than 256 bits (32 bytes) from the kernel    random
> pool  per  invocation, or per reasonable reseed interval (not less than
> one minute), that should be taken as a sign that its  cryptography  is
> not skillfully implemented.
In getrandom(2):

> Calling getrandom() to read /dev/urandom for small values  (<= 256 bytes)
> of buflen is the preferred mode of usage.
Furthermore, it's impossible to read more than 32MB from /dev/urandom
per invocation.

So, actually, the random(4) page is very conservative about the reading
limit.

My final conclusion: as long as you use /dev/urandom for cryptographic
purposes only, you should be ok, because you will never need *a lot* of
random data anyway in any sensible program. For non-cryptographic usage,
seed a user-space PRNG with a few bytes from /dev/urandom and you will
be good.

Laurent



Le 26/07/2015 22:20, Carl Winbäck a écrit :
> Hello Michael & Co,
>
> I would like to bring your attention to bug report 71211, ”random(4):
> clarify utility and volume”.
>
> https://bugzilla.kernel.org/show_bug.cgi?id=71211
>
> This report was filed over a year ago but still hasn’t received any response.
>
> Michael, do you have the time to take a look?
>
> Perhaps you, or someone else on the linux-man list, are familiar with
> CSPRNGs and have some ideas on how to reword this man page?
>
> Unfortunately I’m not at all an expert in this area, so I’m afraid I
> don’t have any specific suggestions regarding how to change this man
> page. But I still think it would be helpful to the Linux community if
> it could be improved, and as a result, hopefully cause less confusion
> regarding getting random numbers on Linux.
>
>
> Looking forward to hear your thoughts on this.
>
> Best regards,
> Carl Winbäck
> --
> To unsubscribe from this list: send the line "unsubscribe linux-man" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlW2XawACgkQRTidSplJch4jJQD/d4LOrShlXmQx5RClVCdj7sKL
6n7SQhlCIjfqvi86JDcA/28cCtYZ8HKL1RgWkgEjGmWoIH6ZA+AKJjgnmugk1wFj
=ff9U
-----END PGP SIGNATURE-----

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status for bug 71211? [random(4): clarify utility and volume]

[Michael Kerrisk (man-pages)]

Nikos,

This was an earlier mail from Laurent Georget. I bring
you into this thread in case there's any of Laurent's comments
that may be helpful as inspiration for your patch.

See also https://bugzilla.kernel.org/show_bug.cgi?id=71211

Cheers,

Michael

On 07/27/2015 06:34 PM, Laurent Georget wrote:
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> Hello,
> 
> the text of this man page has been the subject of endless controversies
> for ages. CSPRNGs are confusing and the page unfortunately adds a
> little to the confusion. The newer getrandom(2) page is better, and I
> (personally, this doesn't engage Michael nor any other author) think
> that the random(4) page should be rewritten in the same spirit.
> (getrandom is a system call used to get a random number, by default,
> it's more or less equivalent than reading from /dev/urandom if you call
> it without flags and for less than 256 bytes).
> 
> Compare this in random(4)
> 
>> The kernel random-number generator is designed to produce a small
>> amount of high-quality seed material to seed a cryptographic pseudo-
>> random number generator (CPRNG). It is designed for security, not
>> speed, and is poorly suited to generating large amounts of random
>> data. Users should be very economical in the amount of seed material
>> that they read from /dev/urandom (and /dev/random); unnecessarily
>> reading large quantities of data from this device will have a
>> negative impact on other users of the device.
> with this in getrandom(2)
> 
>> *getrandom*() relies on entropy gathered from device drivers and other
>> sources of environmental noise.  Unnecessarily reading large
>> quantities of data will have a negative impact on other users of the
>> //dev/random/ and //dev/urandom/ devices.  Therefore, *getrandom*() should
>> not be used for Monte Carlo simulations or other programs/algorithms
>> which are doing probabilistic sampling.
> This says exactly the same thing, but getrandom(2) is less misleading.
> First note that the man page for random says that /dev/urandom is
> "poorly suited to generating large amounts of random data", not
> "poorly suited to generating large amounts of *cryptographic* random data".
> Basically, you can use /dev/urandom for all cryptographic purposes,
> because you never need so many bits at once when doing cryptography.
> Even generating several 16-bytes (128-bits) UIDs per minute if you need
> them to be cryptographically secure (you may want to think about this
> requirement, is it strict?) is not that much. A Monte-Carlo simulation
> requires, say (I don't know exactly let's make a rough guess) around
> several MB of random numbers per minute. That's 4 or 5 orders of
> magnitude higher than your UIDs use case. This would be a wrong
> usage of /dev/urandom for two reasons:
> - - it would be awfully slow
> - - you don't need cryptographically secure random numbers, so there's
> no need to deplete the entropy pool.
> Next, compare this in random(4):
> 
>> If  you    are  unsure  about  whether  you  should  use  /dev/random  or
>> /dev/urandom,  then  probably you want to use the latter.  As a general
>> rule, /dev/urandom should be  used  for    everything  except  long-lived
>> GPG/SSL/SSH keys.
> with this in getrandom(2):
> 
>> Unless  you  are     doing    long-term key generation (and perhaps not even
>> then), you probably shouldn't be using GRND_RANDOM.  The     cryptographic
>> algorithms  used for /dev/urandom are quite conservative, and so should
>> be sufficient for all purposes.    The  disadvantage  of  GRND_RANDOM  is
>> that  it     can block.  Furthermore, dealing with the partially fulfilled
>> getrandom() requests that can occur when     using    GRND_RANDOM  increases
>> code complexity.
> Again, the two man pages say the same. getrandom(2) is more nuanced,
> though.
> 
> To answer your question about how much you can ask /dev/urandom for :
> 
> in random(4) :
> 
>> if any program reads more than 256 bits (32 bytes) from the kernel    random
>> pool  per  invocation, or per reasonable reseed interval (not less than
>> one minute), that should be taken as a sign that its  cryptography  is
>> not skillfully implemented.
> In getrandom(2):
> 
>> Calling getrandom() to read /dev/urandom for small values  (<= 256 bytes)
>> of buflen is the preferred mode of usage.
> Furthermore, it's impossible to read more than 32MB from /dev/urandom
> per invocation.
> 
> So, actually, the random(4) page is very conservative about the reading
> limit.
> 
> My final conclusion: as long as you use /dev/urandom for cryptographic
> purposes only, you should be ok, because you will never need *a lot* of
> random data anyway in any sensible program. For non-cryptographic usage,
> seed a user-space PRNG with a few bytes from /dev/urandom and you will
> be good.
> 
> Laurent
> 
> 
> 
> Le 26/07/2015 22:20, Carl Winbäck a écrit :
>> Hello Michael & Co,
>>
>> I would like to bring your attention to bug report 71211, ”random(4):
>> clarify utility and volume”.
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=71211
>>
>> This report was filed over a year ago but still hasn’t received any response.
>>
>> Michael, do you have the time to take a look?
>>
>> Perhaps you, or someone else on the linux-man list, are familiar with
>> CSPRNGs and have some ideas on how to reword this man page?
>>
>> Unfortunately I’m not at all an expert in this area, so I’m afraid I
>> don’t have any specific suggestions regarding how to change this man
>> page. But I still think it would be helpful to the Linux community if
>> it could be improved, and as a result, hopefully cause less confusion
>> regarding getting random numbers on Linux.
>>
>>
>> Looking forward to hear your thoughts on this.
>>
>> Best regards,
>> Carl Winbäck
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-man" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
> 
> iF4EAREIAAYFAlW2XawACgkQRTidSplJch4jJQD/d4LOrShlXmQx5RClVCdj7sKL
> 6n7SQhlCIjfqvi86JDcA/28cCtYZ8HKL1RgWkgEjGmWoIH6ZA+AKJjgnmugk1wFj
> =ff9U
> -----END PGP SIGNATURE-----
> 
> 


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status for bug 71211? [random(4): clarify utility and volume]

[Nikos Mavrogiannopoulos]

On Wed, 2016-11-09 at 16:27 +0100, Michael Kerrisk (man-pages) wrote:
> Nikos,
> 
> This was an earlier mail from Laurent Georget. I bring
> you into this thread in case there's any of Laurent's comments
> that may be helpful as inspiration for your patch.
> 
> See also https://bugzilla.kernel.org/show_bug.cgi?id=71211

I think that's a nice comment. The text referred to applies to old
kernels not new ones (especially not after the recent rewrite), and I
think it documents and opinion rather than a fact. I am inclined to
simply drop the referred paragraph. Any better suggestions?

regards,
Nikos

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Michael Kerrisk (man-pages)]

[was: Re: Status for bug 71211? [random(4): clarify utility and volume]]

Hello Nikos,

On 11/10/2016 09:42 AM, Nikos Mavrogiannopoulos wrote:
> On Wed, 2016-11-09 at 16:27 +0100, Michael Kerrisk (man-pages) wrote:
>> Nikos,
>>
>> This was an earlier mail from Laurent Georget. I bring
>> you into this thread in case there's any of Laurent's comments
>> that may be helpful as inspiration for your patch.
>>
>> See also https://bugzilla.kernel.org/show_bug.cgi?id=71211
> 
> I think that's a nice comment. The text referred to applies to old
> kernels not new ones (especially not after the recent rewrite), and I
> think it documents and opinion rather than a fact. I am inclined to
> simply drop the referred paragraph. Any better suggestions?

Dropping the paragraph appears too strong too me. Surely we want 
to maintain a recommendation not to consume too much data from
/dev/urandom?

Part of the problem is I think that recommendations
on how to consume randomness are spread across two places.
What do folk think of the patch below?

Cheers,

Michael


    getrandom.2, random.4: Consolidate and improve discussion on usage of randomness
    
    Currently, recommendations on how to consumer randomness are
    spread across both getrandom(2) and random(4) and the general
    opinion seems to be that the text in getrandom(2) does a
    somewhat better job. Consolidate the discussion to a single
    page (getrandom(2)) and address some of the concerns
    expressed about the existing text in random(4).
    
    Signed-off-by: Michael Kerrisk <mtk.manpages-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>

diff --git a/man2/getrandom.2 b/man2/getrandom.2
index db3a0f6..32c55bd 100644
--- a/man2/getrandom.2
+++ b/man2/getrandom.2
@@ -41,20 +41,6 @@ with up to
 random bytes.
 These bytes can be used to seed user-space random number generators
 or for cryptographic purposes.
-.PP
-.BR getrandom ()
-relies on entropy gathered from device drivers and other sources of
-environmental noise.
-Unnecessarily reading large quantities of data will have a negative impact
-on other users of the
-.I /dev/random
-and
-.I /dev/urandom
-devices.
-Therefore,
-.BR getrandom ()
-should not be used for Monte Carlo simulations or other
-programs/algorithms which are doing probabilistic sampling.
 
 By default,
 .BR getrandom ()
@@ -277,19 +263,75 @@ a return of fewer bytes than requested should never happen,
 but the careful programmer will check for this anyway!
 .SS Choice of random device
 Unless you are doing long-term key generation (and perhaps not even
-then), you probably shouldn't be using
-.B GRND_RANDOM.
-The cryptographic algorithms used for
-.I /dev/urandom
+then), you probably shouldn't be using the
+.BR getrandom ()
+.BR GRND_RANDOM
+flag or the
+.IR /dev/random
+device.
+
+Instead, use
+.IR /dev/urandom ;
+the cryptographic algorithms used for
+.IR /dev/urandom ;
 are quite conservative, and so should be sufficient for all purposes.
+
 The disadvantage of
 .B GRND_RANDOM
-is that it can block.
+and reads from
+.I dev/random
+is that the operation can block.
 Furthermore, dealing with the partially fulfilled
-.BR getrandom ()
 requests that can occur when using
 .B GRND_RANDOM
+or when reading from
+.I /dev/random
 increases code complexity.
+.\"
+.SS Usage recommendations
+The kernel random-number generator
+relies on entropy gathered from device drivers and other sources of
+environmental noise.
+It is designed to produce a small
+amount of high-quality seed material to seed a
+cryptographic pseudo-random number generator (CPRNG).
+It is designed for security, not speed, and is poorly
+suited to generating large amounts of cryptographic random data.
+Users should be very economical in the amount of seed
+material that they consume via
+.BR getrandom (),
+.IR /dev/urandom ,
+and
+.IR /dev/random .
+Consuming unnecessarily large quantities of data via these interfaces
+will have a negative impact on other consumers of randomness.
+
+These interfaces should not be used to provide large quantities
+of data for Monte Carlo simulations or other
+programs/algorithms which are doing probabilistic sampling.
+And indeed, such usage is unnecessary (and will be slow):
+instead, use these interfaces to provide a small amount of
+data used to seed a user-space pseudo-random number generator
+for use by such applications.
+.\"
+.SS Generating cryptographic keys
+The amount of seed material required to generate a cryptographic key
+equals the effective key size of the key.
+For example, a 3072-bit RSA
+or Diffie-Hellman private key has an effective key size of 128 bits
+(it requires about 2^128 operations to break) so a key generator
+needs only 128 bits (16 bytes) of seed material from
+.IR /dev/random .
+
+While some safety margin above that minimum is reasonable, as a guard
+against flaws in the CPRNG algorithm, no cryptographic primitive
+available today can hope to promise more than 256 bits of security,
+so if any program reads more than 256 bits (32 bytes) from the kernel
+random pool per invocation, or per reasonable reseed interval (not less
+than one minute), that should be taken as a sign that its cryptography is
+.I not
+skillfully implemented.
+.\"
 .SS Emulating OpenBSD's getentropy()
 The
 .BR getentropy ()
diff --git a/man4/random.4 b/man4/random.4
index 736cbde..a027ab8 100644
--- a/man4/random.4
+++ b/man4/random.4
@@ -138,35 +138,9 @@ may block, users will usually want to open it in nonblocking mode
 and provide some sort of user notification if the desired
 entropy is not immediately available.
 
-The kernel random-number generator is designed to produce a small
-amount of high-quality seed material to seed a
-cryptographic pseudo-random number generator (CPRNG).
-It is designed for security, not speed, and is poorly
-suited to generating large amounts of random data.
-Users should be very economical in the amount of seed
-material that they read from
-.IR /dev/urandom
-(and
-.IR /dev/random );
-unnecessarily reading large quantities of data from this device will have
-a negative impact on other users of the device.
-
-The amount of seed material required to generate a cryptographic key
-equals the effective key size of the key.
-For example, a 3072-bit RSA
-or Diffie-Hellman private key has an effective key size of 128 bits
-(it requires about 2^128 operations to break) so a key generator
-needs only 128 bits (16 bytes) of seed material from
-.IR /dev/random .

Re: [patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Nikos Mavrogiannopoulos]

On Thu, 2016-11-10 at 15:23 +0100, Michael Kerrisk (man-pages) wrote:
> [was: Re: Status for bug 71211? [random(4): clarify utility and
> volume]]
> 
> Hello Nikos,
> 
> On 11/10/2016 09:42 AM, Nikos Mavrogiannopoulos wrote:
> > 
> > On Wed, 2016-11-09 at 16:27 +0100, Michael Kerrisk (man-pages)
> > wrote:
> > > 
> > > Nikos,
> > > 
> > > This was an earlier mail from Laurent Georget. I bring
> > > you into this thread in case there's any of Laurent's comments
> > > that may be helpful as inspiration for your patch.
> > > 
> > > See also https://bugzilla.kernel.org/show_bug.cgi?id=71211
> > 
> > I think that's a nice comment. The text referred to applies to old
> > kernels not new ones (especially not after the recent rewrite), and
> > I
> > think it documents and opinion rather than a fact. I am inclined to
> > simply drop the referred paragraph. Any better suggestions?
> 
> Dropping the paragraph appears too strong too me. Surely we want 
> to maintain a recommendation not to consume too much data from
> /dev/urandom?

Stephan Mueller or Ted should be able to provide more info for the new
code. I think in the new versions of /dev/urandom the amount consumed
shouldn't cause issues or affect other users.

However, I agree that overall, that this is a low level interface and
it should be treated as such. I.e., I'd expect applications to use
their crypto libraries' interfaces rather than getrandom directly. The
reason is not only about being slow, but about having to take care
about EINTR, short reads, quirks such as for early boot, etc [0]. 

regards,
Nikos

[0]. I've tried to write down some argumentation at:
http://nmav.gnutls.org/2016/10/random-generator-linux.html

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Laurent Georget]

[-- Attachment #1.1: Type: text/plain, Size: 2533 bytes --]



Le 10/11/2016 à 15:35, Nikos Mavrogiannopoulos a écrit :
> On Thu, 2016-11-10 at 15:23 +0100, Michael Kerrisk (man-pages) wrote:
>> [was: Re: Status for bug 71211? [random(4): clarify utility and
>> volume]]
>>
>> Hello Nikos,
>>
>> On 11/10/2016 09:42 AM, Nikos Mavrogiannopoulos wrote:
>>>
>>> On Wed, 2016-11-09 at 16:27 +0100, Michael Kerrisk (man-pages)
>>> wrote:
>>>>
>>>> Nikos,
>>>>
>>>> This was an earlier mail from Laurent Georget. I bring
>>>> you into this thread in case there's any of Laurent's comments
>>>> that may be helpful as inspiration for your patch.
>>>>
>>>> See also https://bugzilla.kernel.org/show_bug.cgi?id=71211
>>>
>>> I think that's a nice comment. The text referred to applies to old
>>> kernels not new ones (especially not after the recent rewrite), and
>>> I
>>> think it documents and opinion rather than a fact. I am inclined to
>>> simply drop the referred paragraph. Any better suggestions?
>>
>> Dropping the paragraph appears too strong too me. Surely we want 
>> to maintain a recommendation not to consume too much data from
>> /dev/urandom?
> 
> Stephan Mueller or Ted should be able to provide more info for the new
> code. I think in the new versions of /dev/urandom the amount consumed
> shouldn't cause issues or affect other users.
> 
> However, I agree that overall, that this is a low level interface and
> it should be treated as such. I.e., I'd expect applications to use
> their crypto libraries' interfaces rather than getrandom directly. The
> reason is not only about being slow, but about having to take care
> about EINTR, short reads, quirks such as for early boot, etc [0].

I agree. For non-cryptographic uses, it's always better to use a userspace
random-number generator such as the generators provided by the standard
library of Java, Python, etc. or by OpenSSL. For cryptographic purposes,
check what available libraries have first, and if needs be, getrandom is
fine for all reasonable usesand avoid the hassle of opening and closing
/dev/urandom.

The warning about the limits of /dev/urandom proposed in the other patch
(about short reads and EINTR) seems sufficient to me. It's also already
written that up to 256 bytes is always fine, more than 32MB at once is
impossible. I don't think we need to stress out anything else.

Cheers,

Laurent

> 
> regards,
> Nikos
> 
> [0]. I've tried to write down some argumentation at:
> http://nmav.gnutls.org/2016/10/random-generator-linux.html
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Michael Kerrisk (man-pages)]

Nikos, Laurent,

On 11/10/2016 04:10 PM, Laurent Georget wrote:
> 
> 
> Le 10/11/2016 à 15:35, Nikos Mavrogiannopoulos a écrit :
>> On Thu, 2016-11-10 at 15:23 +0100, Michael Kerrisk (man-pages) wrote:
>>> [was: Re: Status for bug 71211? [random(4): clarify utility and
>>> volume]]
>>>
>>> Hello Nikos,
>>>
>>> On 11/10/2016 09:42 AM, Nikos Mavrogiannopoulos wrote:
>>>>
>>>> On Wed, 2016-11-09 at 16:27 +0100, Michael Kerrisk (man-pages)
>>>> wrote:
>>>>>
>>>>> Nikos,
>>>>>
>>>>> This was an earlier mail from Laurent Georget. I bring
>>>>> you into this thread in case there's any of Laurent's comments
>>>>> that may be helpful as inspiration for your patch.
>>>>>
>>>>> See also https://bugzilla.kernel.org/show_bug.cgi?id=71211
>>>>
>>>> I think that's a nice comment. The text referred to applies to old
>>>> kernels not new ones (especially not after the recent rewrite), and
>>>> I
>>>> think it documents and opinion rather than a fact. I am inclined to
>>>> simply drop the referred paragraph. Any better suggestions?
>>>
>>> Dropping the paragraph appears too strong too me. Surely we want 
>>> to maintain a recommendation not to consume too much data from
>>> /dev/urandom?
>>
>> Stephan Mueller or Ted should be able to provide more info for the new
>> code. I think in the new versions of /dev/urandom the amount consumed
>> shouldn't cause issues or affect other users.
>>
>> However, I agree that overall, that this is a low level interface and
>> it should be treated as such. I.e., I'd expect applications to use
>> their crypto libraries' interfaces rather than getrandom directly. The
>> reason is not only about being slow, but about having to take care
>> about EINTR, short reads, quirks such as for early boot, etc [0].
> 
> I agree. For non-cryptographic uses, it's always better to use a userspace
> random-number generator such as the generators provided by the standard
> library of Java, Python, etc. or by OpenSSL. For cryptographic purposes,
> check what available libraries have first, and if needs be, getrandom is
> fine for all reasonable usesand avoid the hassle of opening and closing
> /dev/urandom.
> 
> The warning about the limits of /dev/urandom proposed in the other patch
> (about short reads and EINTR) seems sufficient to me. It's also already
> written that up to 256 bytes is always fine, more than 32MB at once is
> impossible. I don't think we need to stress out anything else.

So, I must admit that after your respective mails, I'm still not
clear. Do you think I should keep this patch, change it, or
discard it?

Cheers,

Michael



-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Nikos Mavrogiannopoulos]

On Thu, 2016-11-10 at 19:16 +0100, Michael Kerrisk (man-pages) wrote:
> Nikos, Laurent,
> So, I must admit that after your respective mails, I'm still not
> clear. Do you think I should keep this patch, change it, or
> discard it?

It is a bit confusing to me. The sentences:
"When  reading  from  /dev/urandom  (GRND_RANDOM  is  not set),
getrandom()"

and

"The behavior when a call to getrandom() that is
blocked  while  reading  from  /dev/urandom"

seem to imply that getrandom() is a wrapper over /dev/urandom (i.e.,
internally it opens the device reads etc). That's not the case the
system call doesn't go through /dev/urandom, although the pools behind
are the same.

maybe saying the /dev/urandom pool instead, but I find that even that
could confuse someone.

So while the text is better and more precise in other aspects than
before I think it is a bit confusing the mix of getrandom() with
/dev/urandom and /dev/random. Maybe copy the text back and separate the
descriptions even if they are very similar at the moment? 

regards,
Nikos

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Michael Kerrisk (man-pages)]

Hi Nikos,

On 11 November 2016 at 08:41, Nikos Mavrogiannopoulos <nmav-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> On Thu, 2016-11-10 at 19:16 +0100, Michael Kerrisk (man-pages) wrote:
>> Nikos, Laurent,
>> So, I must admit that after your respective mails, I'm still not
>> clear. Do you think I should keep this patch, change it, or
>> discard it?
>
> It is a bit confusing to me. The sentences:
> "When  reading  from  /dev/urandom  (GRND_RANDOM  is  not set),
> getrandom()"
>
> and
>
> "The behavior when a call to getrandom() that is
> blocked  while  reading  from  /dev/urandom"
>
> seem to imply that getrandom() is a wrapper over /dev/urandom (i.e.,
> internally it opens the device reads etc). That's not the case the
> system call doesn't go through /dev/urandom, although the pools behind
> are the same.

I agree that this language is a bit confusing. But that language was
not introduced by my patch.

> maybe saying the /dev/urandom pool instead, but I find that even that
> could confuse someone.
>
> So while the text is better and more precise in other aspects than
> before I think it is a bit confusing the mix of getrandom() with
> /dev/urandom and /dev/random. Maybe copy the text back and separate the
> descriptions even if they are very similar at the moment?

I'm reluctant to duplicate text in two places. I think that that
duplication os prt of the reason why we have the current mess.

I'll instead try to fine tune the text to remove the implication that
getrandom() is reading from the devices. Take a look at the current
draft of the getrandom(2) page in the branch at
http://git.kernel.org/cgit/docs/man-pages/man-pages.git/log/?h=draft_random

The text is also rendered below, but may be a little less readable
without the formatting, since I user the terms "urandom pool" and
"random pool" with the words "urandom" and "random"
italicized/underlined.

Cheers,

Michael

NAME
       getrandom - obtain a series of random bytes

SYNOPSIS
       #include <linux/random.h>

       int getrandom(void *buf, size_t buflen, unsigned int flags);

DESCRIPTION
       The  getrandom()  system  call  fills the buffer pointed to by buf
       with up to buflen random bytes.  These bytes can be used  to  seed
       user-space random number generators or for cryptographic purposes.

       By default, getrandom() draws entropy from the urandom pool (i.e.,
       the same source as the /dev/urandom device).  This behavior can be
       changed via the flags argument.

       If the urandom pool has been initialized, reads of up to 256 bytes
       will always return as many bytes as  requested  and  will  not  be
       interrupted  by signals.  No such guarantees apply for larger buf‐
       fer sizes.  For example, if the call is interrupted  by  a  signal
       handler, it may return a partially filled buffer, or fail with the
       error EINTR.

       If the urandom pool has not yet been initialized, then getrandom()
       will block, unless GRND_NONBLOCK is specified in flags.

       The  flags argument is a bit mask that can contain zero or more of
       the following values ORed together:

       GRND_RANDOM
              If this bit is set, then random bytes are  drawn  from  the
              random  pool  (i.e.,  the  same  source  as the /dev/random
              device) instead of the urandom pool.  The  random  pool  is
              limited  based  on  the  entropy  that can be obtained from
              environmental noise.  If the number of available  bytes  in
              the  random pool is less than requested in buflen, the call
              returns just the available  random  bytes.   If  no  random
              bytes  are  available, the behavior depends on the presence
              of GRND_NONBLOCK in the flags argument.

       GRND_NONBLOCK
              By default, when reading from the random pool,  getrandom()
              blocks  if  no random bytes are available, and when reading
              from the urandom pool, it blocks if the  entropy  pool  has
              not  yet  been  initialized.   If the GRND_NONBLOCK flag is
              set, then getrandom() does not block in  these  cases,  but
              instead immediately returns -1 with errno set to EAGAIN.

RETURN VALUE
       On  success,  getrandom()  returns  the  number of bytes that were
       copied to the buffer buf.  This may be less  than  the  number  of
       bytes  requested via buflen if either GRND_RANDOM was specified in
       flags and insufficient entropy was present in the random  pool  or
       the system call was interrupted by a signal.

       On error, -1 is returned, and errno is set appropriately.

ERRORS
       EAGAIN The  requested  entropy  was not available, and getrandom()
              would have blocked if the GRND_NONBLOCK flag was not set.

       EFAULT The address referred to by buf is  outside  the  accessible
              address space.

       EINTR  The  call  was  interrupted  by  a  signal handler; see the
              description of how  interrupted  read(2)  calls  on  "slow"
              devices are handled with and without the SA_RESTART flag in
              the signal(7) man page.

       EINVAL An invalid flag was specified in flags.

VERSIONS
       getrandom() was introduced in version 3.17 of the Linux kernel.

CONFORMING TO
       This system call is Linux-specific.

NOTES
       Unlike /dev/random and /dev/random, getrandom() does  not  involve
       the  use  of pathnames or file descriptors.  Thus, getrandom() can
       be useful in cases where chroot(2) makes /dev pathnames invisible,
       and where an application (e.g., a daemon during start-up) closes a
       file descriptor for one of  these  files  that  was  opened  by  a
       library.

   Maximum number of bytes returned
       As of Linux 3.19 the following limits apply:

       *  When reading from the urandom pool, a maximum of 33554431 bytes
          is returned by a single call to getrandom()  on  systems  where
          int has a size of 32 bits.

       *  When  reading  from  the random pool, a maximum of 512 bytes is
          returned.

   Initialization of the entropy pool
       The kernel collects bits of entropy from the environment.  When  a
       sufficient  number  of random bits has been collected, the urandom
       entropy pool is considered to be initialized.  This state is  nor‐
       mally reached early in the system bootstrap phase.

   Interruption by a signal handler
       When  reading  from  the  urandom  pool  (GRND_RANDOM is not set),
       getrandom() will block until the entropy pool has been initialized
       (unless  the  GRND_NONBLOCK  flag was specified).  If a request is
       made to read a large number of bytes (more than 256),  getrandom()
       will  block  until those bytes have been generated and transferred
       from kernel memory to buf.  When  reading  from  the  random  pool
       (GRND_RANDOM  is  set),  getrandom()  will block until some random
       bytes become available (unless the GRND_NONBLOCK flag  was  speci‐
       fied).

       The  behavior  when  a  call  to getrandom() that is blocked while
       reading from the urandom pool is interrupted by a  signal  handler
       depends  on  the initialization state of the entropy buffer and on
       the request size, buflen.  If the entropy is not yet  initialized,
       then the call will fail with the EINTR error.  If the entropy pool
       has been initialized and the request size is large (buflen > 256),
       the  call either succeeds, returning a partially filled buffer, or
       fails with the error EINTR.  If the entropy pool has been initial‐
       ized  and  the request size is small (buflen <= 256), then getran‐
       dom() will not fail with EINTR.  Instead, it will  return  all  of
       the bytes that have been requested.

       When  reading  from the random pool, blocking requests of any size
       can be interrupted by a signal handler (the call  fails  with  the
       error EINTR).

       Using  getrandom()  to  read small buffers (<= 256 bytes) from the
       urandom pool is the preferred mode of usage.

       The special treatment of small values of buflen was  designed  for
       compatibility with OpenBSD's getentropy() system call.

       The  user  of  getrandom()  must always check the return value, to
       determine whether either an error occurred  or  fewer  bytes  than
       requested  were  returned.   In  the case where GRND_RANDOM is not
       specified and buflen is less than or equal to  256,  a  return  of
       fewer  bytes  than  requested should never happen, but the careful
       programmer will check for this anyway!

   Choice of random device
       Unless you are doing long-term key  generation  (and  perhaps  not
       even then), you probably shouldn't be using the GRND_RANDOM or the
       /dev/random device.

       Instead, use either getrandom() without the  GRND_RANDOM  flag  or
       the  /dev/urandom  device.   The cryptographic algorithms used for
       the urandom pool are quite conservative, and so should  be  suffi‐
       cient for all purposes.

       The disadvantage of GRND_RANDOM and reads from /dev/random is that
       the operation can block.  Furthermore, dealing with the  partially
       fulfilled  requests  that can occur when using GRND_RANDOM or when
       reading from /dev/random increases code complexity.

   Usage recommendations
       The kernel random-number generator relies on entropy gathered from
       device  drivers  and  other sources of environmental noise.  It is
       designed to produce a small amount of high-quality  seed  material
       to seed a cryptographic pseudorandom number generator (CPRNG).  It
       is designed for security, not speed, and is poorly suited to  gen‐
       erating  large amounts of cryptographic random data.  Users should
       be very economical in the amount of seed material that  they  con‐
       sume  via  getrandom(),  /dev/urandom, and /dev/random.  Consuming
       unnecessarily large quantities of data via these  interfaces  will
       have a negative impact on other consumers of randomness.

       These interfaces should not be used to provide large quantities of
       data for Monte  Carlo  simulations  or  other  programs/algorithms
       which are doing probabilistic sampling.  And indeed, such usage is
       unnecessary (and will be slow): instead, use these  interfaces  to
       provide  a  small amount of data used to seed a user-space pseudo‐
       random number generator for use by such applications.

   Generating cryptographic keys
       The amount of seed material required to generate  a  cryptographic
       key  equals  the  effective  key  size of the key.  For example, a
       3072-bit RSA or Diffie-Hellman private key has  an  effective  key
       size  of 128 bits (it requires about 2^128 operations to break) so
       a key generator needs only 128 bits (16 bytes)  of  seed  material
       from /dev/random.

       While  some  safety  margin above that minimum is reasonable, as a
       guard against flaws in the CPRNG algorithm, no cryptographic prim‐
       itive  available  today  can hope to promise more than 256 bits of
       security, so if any program reads more than 256  bits  (32  bytes)
       from  the  kernel  random  pool  per invocation, or per reasonable
       reseed interval (not less than one minute), that should  be  taken
       as a sign that its cryptography is not skillfully implemented.

   Emulating OpenBSD's getentropy()
       The  getentropy() system call in OpenBSD can be emulated using the
       following function:

           int
           getentropy(void *buf, size_t buflen)
           {
               int ret;

               if (buflen > 256)
                   goto failure;
               ret = getrandom(buf, buflen, 0);
               if (ret < 0)
                   return ret;
               if (ret == buflen)
                   return 0;
           failure:
               errno = EIO;
               return -1;
           }

BUGS
       As of Linux 3.19, the following bug exists:

       *  Depending on CPU load, getrandom() does not react to interrupts
          before reading all bytes requested.

SEE ALSO
       random(4), urandom(4), signal(7)

>
> regards,
> Nikos
>



-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Laurent Georget]

[-- Attachment #1.1: Type: text/plain, Size: 1825 bytes --]



Le 11/11/2016 à 11:47, Michael Kerrisk (man-pages) a écrit :
> Hi Nikos,
> 
> On 11 November 2016 at 08:41, Nikos Mavrogiannopoulos <nmav-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>> On Thu, 2016-11-10 at 19:16 +0100, Michael Kerrisk (man-pages) wrote:
>>> Nikos, Laurent,
>>> So, I must admit that after your respective mails, I'm still not
>>> clear. Do you think I should keep this patch, change it, or
>>> discard it?
>>
>> It is a bit confusing to me. The sentences:
>> "When  reading  from  /dev/urandom  (GRND_RANDOM  is  not set),
>> getrandom()"
>>
>> and
>>
>> "The behavior when a call to getrandom() that is
>> blocked  while  reading  from  /dev/urandom"
>>
>> seem to imply that getrandom() is a wrapper over /dev/urandom (i.e.,
>> internally it opens the device reads etc). That's not the case the
>> system call doesn't go through /dev/urandom, although the pools behind
>> are the same.
> 
> I agree that this language is a bit confusing. But that language was
> not introduced by my patch.
> 
>> maybe saying the /dev/urandom pool instead, but I find that even that
>> could confuse someone.
>>
>> So while the text is better and more precise in other aspects than
>> before I think it is a bit confusing the mix of getrandom() with
>> /dev/urandom and /dev/random. Maybe copy the text back and separate the
>> descriptions even if they are very similar at the moment?
> 
> I'm reluctant to duplicate text in two places. I think that that
> duplication os prt of the reason why we have the current mess.

So, maybe all this discussion about which interface to choose, expected
usage, etc. should go to a random.7 man page? This would be the logical
location to detail the differences about the three interfaces. What do
you think?

Cheers,
Laurent


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Laurent Georget]

>> I'm reluctant to duplicate text in two places. I think that that
>> duplication os prt of the reason why we have the current mess.
> 
> So, maybe all this discussion about which interface to choose, expected
> usage, etc. should go to a random.7 man page? This would be the logical
> location to detail the differences about the three interfaces. What do
> you think?

To follow up on this, what do you think of the following patch? I do not
propose it for inclusion as is but more as a kind of RFC. Would it be useful
to have this kind of table to sum up in one place the differences between
getrandom(), /dev/random and /dev/urandom?

Note that this is my first attempt to make tables in man pages so I have no
idea if I did things right or not.

Cheers,
Laurent

---
 man2/getrandom.2 | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 75 insertions(+)

diff --git a/man2/getrandom.2 b/man2/getrandom.2
index 32c55bd..b337415 100644
--- a/man2/getrandom.2
+++ b/man2/getrandom.2
@@ -313,6 +313,81 @@ And indeed, such usage is unnecessary (and will be slow):
 instead, use these interfaces to provide a small amount of
 data used to seed a user-space pseudo-random number generator
 for use by such applications.
+
+.\"
+.SS Comparison between getrandom, /dev/urandom and /dev/random
+
+.TS
+allbox;
+lb lb lb lb.
+Interface	Pool	Blocking behavior	Behavior in early boot time
+T{
+.I /dev/random
+T}	Blocking pool	T{
+Blocks when the entropy estimate is too low until there is enough entropy again
+T}	T{
+Blocks until enough entropy is gathered
+T}
+T{
+.I /dev/urandom
+T}	T{
+Cryptographically-secure Random Number Generator (CRNG) output
+T}	T{
+Does not block once the CRNG is ready
+T}	T{
+Returns output from uninitialized CRNG (possibly low entropy and not suitable for cryptography)
+T}
+T{
+.BR getrandom ()
+T}	T{
+Same as
+.I /dev/urandom
+T}	T{
+Does not block once the pool is ready
+T}	T{
+Blocks until the pool is ready
+T}
+T{
+.BR getrandom ()
+with
+.B GRND_RANDOM
+T}	T{
+Same as
+.I /dev/random
+T}	T{
+Blocks when the entropy estimate is too low until there is enough entropy again
+T}	T{
+Blocks until the pool is ready
+T}
+T{
+.BR getrandom ()
+with
+.B GRND_NONBLOCK
+T}	T{
+Same as
+.I /dev/urandom
+T}	T{
+Does not block
+T}	T{
+Returns -EAGAIN if the pool is not ready
+T}
+T{
+.BR getrandom ()
+with
+.B GRND_RANDOM
+and
+.B GRND_NONBLOCK
+T}	T{
+Same as
+.I /dev/random
+T}	T{
+Returns -EAGAIN if not enough entropy is available
+T}	T{
+Returns -EAGAIN if the pool is not ready
+T}
+.TE
+
+
 .\"
 .SS Generating cryptographic keys
 The amount of seed material required to generate a cryptographic key
-- 
2.10.1
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Nikos Mavrogiannopoulos]

On Fri, 2016-11-11 at 13:08 +0100, Laurent Georget wrote:

> +.I /dev/urandom
> +T}	T{
> +Cryptographically-secure Random Number Generator (CRNG) output
> +T}	T{
> +Does not block once the CRNG is ready
> +T}	T{
> +Returns output from uninitialized CRNG (possibly low entropy and not
> suitable for cryptography)

I'd make that specific, and mention early boot explicitly, otherwise it
implies that this always returns from an uninitialized CRNG. This is a
limitation that applies only for applications started on early boot;
for the majority of applications this is not applicable.

regards,
Nikos

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Laurent Georget]

[-- Attachment #1.1: Type: text/plain, Size: 1028 bytes --]



Le 11/11/2016 à 18:02, Nikos Mavrogiannopoulos a écrit :
> On Fri, 2016-11-11 at 13:08 +0100, Laurent Georget wrote:
> 
>> +.I /dev/urandom
>> +T}	T{
>> +Cryptographically-secure Random Number Generator (CRNG) output
>> +T}	T{
>> +Does not block once the CRNG is ready
>> +T}	T{
>> +Returns output from uninitialized CRNG (possibly low entropy and not
>> suitable for cryptography)
> 
> I'd make that specific, and mention early boot explicitly, otherwise it
> implies that this always returns from an uninitialized CRNG. This is a
> limitation that applies only for applications started on early boot;
> for the majority of applications this is not applicable.

The title of the last column is "Behavior in early boot time". We can
rephrase the content as "Even if the CRNG is not ready yet, returns
output from it anyway (possibly low entropy and not suitable for
cryptography)". Does that sound better?

I got the third column wrong by the way, please read  "Never blocks".

Cheers,
Laurent


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Michael Kerrisk (man-pages)]

On 11/11/2016 06:35 PM, Laurent Georget wrote:
> 
> 
> Le 11/11/2016 à 18:02, Nikos Mavrogiannopoulos a écrit :
>> On Fri, 2016-11-11 at 13:08 +0100, Laurent Georget wrote:
>>
>>> +.I /dev/urandom
>>> +T}	T{
>>> +Cryptographically-secure Random Number Generator (CRNG) output
>>> +T}	T{
>>> +Does not block once the CRNG is ready
>>> +T}	T{
>>> +Returns output from uninitialized CRNG (possibly low entropy and not
>>> suitable for cryptography)
>>
>> I'd make that specific, and mention early boot explicitly, otherwise it
>> implies that this always returns from an uninitialized CRNG. This is a
>> limitation that applies only for applications started on early boot;
>> for the majority of applications this is not applicable.
> 
> The title of the last column is "Behavior in early boot time". We can
> rephrase the content as "Even if the CRNG is not ready yet, returns
> output from it anyway (possibly low entropy and not suitable for
> cryptography)". Does that sound better?

I think the existing text was okay.

> I got the third column wrong by the way, please read  "Never blocks".

Noted!

Cheers,

Michael


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Michael Kerrisk (man-pages)]

On 11/11/2016 01:08 PM, Laurent Georget wrote:
>>> I'm reluctant to duplicate text in two places. I think that that
>>> duplication os prt of the reason why we have the current mess.
>>
>> So, maybe all this discussion about which interface to choose, expected
>> usage, etc. should go to a random.7 man page? This would be the logical
>> location to detail the differences about the three interfaces. What do
>> you think?
> 
> To follow up on this, what do you think of the following patch? I do not
> propose it for inclusion as is but more as a kind of RFC. Would it be useful
> to have this kind of table to sum up in one place the differences between
> getrandom(), /dev/random and /dev/urandom?
> 
> Note that this is my first attempt to make tables in man pages so I have no
> idea if I did things right or not.

I like this. I'll incorporate it in random(7) :-).

Cheers,

Michael


> 
> Cheers,
> Laurent
> 
> ---
>  man2/getrandom.2 | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 75 insertions(+)
> 
> diff --git a/man2/getrandom.2 b/man2/getrandom.2
> index 32c55bd..b337415 100644
> --- a/man2/getrandom.2
> +++ b/man2/getrandom.2
> @@ -313,6 +313,81 @@ And indeed, such usage is unnecessary (and will be slow):
>  instead, use these interfaces to provide a small amount of
>  data used to seed a user-space pseudo-random number generator
>  for use by such applications.
> +
> +.\"
> +.SS Comparison between getrandom, /dev/urandom and /dev/random
> +
> +.TS
> +allbox;
> +lb lb lb lb.
> +Interface	Pool	Blocking behavior	Behavior in early boot time
> +T{
> +.I /dev/random
> +T}	Blocking pool	T{
> +Blocks when the entropy estimate is too low until there is enough entropy again
> +T}	T{
> +Blocks until enough entropy is gathered
> +T}
> +T{
> +.I /dev/urandom
> +T}	T{
> +Cryptographically-secure Random Number Generator (CRNG) output
> +T}	T{
> +Does not block once the CRNG is ready
> +T}	T{
> +Returns output from uninitialized CRNG (possibly low entropy and not suitable for cryptography)
> +T}
> +T{
> +.BR getrandom ()
> +T}	T{
> +Same as
> +.I /dev/urandom
> +T}	T{
> +Does not block once the pool is ready
> +T}	T{
> +Blocks until the pool is ready
> +T}
> +T{
> +.BR getrandom ()
> +with
> +.B GRND_RANDOM
> +T}	T{
> +Same as
> +.I /dev/random
> +T}	T{
> +Blocks when the entropy estimate is too low until there is enough entropy again
> +T}	T{
> +Blocks until the pool is ready
> +T}
> +T{
> +.BR getrandom ()
> +with
> +.B GRND_NONBLOCK
> +T}	T{
> +Same as
> +.I /dev/urandom
> +T}	T{
> +Does not block
> +T}	T{
> +Returns -EAGAIN if the pool is not ready
> +T}
> +T{
> +.BR getrandom ()
> +with
> +.B GRND_RANDOM
> +and
> +.B GRND_NONBLOCK
> +T}	T{
> +Same as
> +.I /dev/random
> +T}	T{
> +Returns -EAGAIN if not enough entropy is available
> +T}	T{
> +Returns -EAGAIN if the pool is not ready
> +T}
> +.TE
> +
> +
>  .\"
>  .SS Generating cryptographic keys
>  The amount of seed material required to generate a cryptographic key
> 


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [patch] getrandom.2, random.4: Consolidate and improve discussion on usage of randomness

[Michael Kerrisk (man-pages)]

On 11/11/2016 12:01 PM, Laurent Georget wrote:
> 
> 
> Le 11/11/2016 à 11:47, Michael Kerrisk (man-pages) a écrit :
>> Hi Nikos,
>>
>> On 11 November 2016 at 08:41, Nikos Mavrogiannopoulos <nmav-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

[...]

>>> So while the text is better and more precise in other aspects than
>>> before I think it is a bit confusing the mix of getrandom() with
>>> /dev/urandom and /dev/random. Maybe copy the text back and separate the
>>> descriptions even if they are very similar at the moment?
>>
>> I'm reluctant to duplicate text in two places. I think that that
>> duplication os prt of the reason why we have the current mess.
> 
> So, maybe all this discussion about which interface to choose, expected
> usage, etc. should go to a random.7 man page? This would be the logical
> location to detail the differences about the three interfaces. What do
> you think?

That seems a reasonable suggestion.

Cheers,

Michael


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status for bug 71211? [random(4): clarify utility and volume]

[Michael Kerrisk (man-pages)]

[Adding the original reporter of the bug to this thread]

On 11/09/2016 04:27 PM, Michael Kerrisk (man-pages) wrote:
> Nikos,
> 
> This was an earlier mail from Laurent Georget. I bring
> you into this thread in case there's any of Laurent's comments
> that may be helpful as inspiration for your patch.
> 
> See also https://bugzilla.kernel.org/show_bug.cgi?id=71211
> 
> Cheers,
> 
> Michael
> 
> On 07/27/2015 06:34 PM, Laurent Georget wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Hello,
>>
>> the text of this man page has been the subject of endless controversies
>> for ages. CSPRNGs are confusing and the page unfortunately adds a
>> little to the confusion. The newer getrandom(2) page is better, and I
>> (personally, this doesn't engage Michael nor any other author) think
>> that the random(4) page should be rewritten in the same spirit.
>> (getrandom is a system call used to get a random number, by default,
>> it's more or less equivalent than reading from /dev/urandom if you call
>> it without flags and for less than 256 bytes).
>>
>> Compare this in random(4)
>>
>>> The kernel random-number generator is designed to produce a small
>>> amount of high-quality seed material to seed a cryptographic pseudo-
>>> random number generator (CPRNG). It is designed for security, not
>>> speed, and is poorly suited to generating large amounts of random
>>> data. Users should be very economical in the amount of seed material
>>> that they read from /dev/urandom (and /dev/random); unnecessarily
>>> reading large quantities of data from this device will have a
>>> negative impact on other users of the device.
>> with this in getrandom(2)
>>
>>> *getrandom*() relies on entropy gathered from device drivers and other
>>> sources of environmental noise.  Unnecessarily reading large
>>> quantities of data will have a negative impact on other users of the
>>> //dev/random/ and //dev/urandom/ devices.  Therefore, *getrandom*() should
>>> not be used for Monte Carlo simulations or other programs/algorithms
>>> which are doing probabilistic sampling.
>> This says exactly the same thing, but getrandom(2) is less misleading.
>> First note that the man page for random says that /dev/urandom is
>> "poorly suited to generating large amounts of random data", not
>> "poorly suited to generating large amounts of *cryptographic* random data".
>> Basically, you can use /dev/urandom for all cryptographic purposes,
>> because you never need so many bits at once when doing cryptography.
>> Even generating several 16-bytes (128-bits) UIDs per minute if you need
>> them to be cryptographically secure (you may want to think about this
>> requirement, is it strict?) is not that much. A Monte-Carlo simulation
>> requires, say (I don't know exactly let's make a rough guess) around
>> several MB of random numbers per minute. That's 4 or 5 orders of
>> magnitude higher than your UIDs use case. This would be a wrong
>> usage of /dev/urandom for two reasons:
>> - - it would be awfully slow
>> - - you don't need cryptographically secure random numbers, so there's
>> no need to deplete the entropy pool.
>> Next, compare this in random(4):
>>
>>> If  you    are  unsure  about  whether  you  should  use  /dev/random  or
>>> /dev/urandom,  then  probably you want to use the latter.  As a general
>>> rule, /dev/urandom should be  used  for    everything  except  long-lived
>>> GPG/SSL/SSH keys.
>> with this in getrandom(2):
>>
>>> Unless  you  are     doing    long-term key generation (and perhaps not even
>>> then), you probably shouldn't be using GRND_RANDOM.  The     cryptographic
>>> algorithms  used for /dev/urandom are quite conservative, and so should
>>> be sufficient for all purposes.    The  disadvantage  of  GRND_RANDOM  is
>>> that  it     can block.  Furthermore, dealing with the partially fulfilled
>>> getrandom() requests that can occur when     using    GRND_RANDOM  increases
>>> code complexity.
>> Again, the two man pages say the same. getrandom(2) is more nuanced,
>> though.
>>
>> To answer your question about how much you can ask /dev/urandom for :
>>
>> in random(4) :
>>
>>> if any program reads more than 256 bits (32 bytes) from the kernel    random
>>> pool  per  invocation, or per reasonable reseed interval (not less than
>>> one minute), that should be taken as a sign that its  cryptography  is
>>> not skillfully implemented.
>> In getrandom(2):
>>
>>> Calling getrandom() to read /dev/urandom for small values  (<= 256 bytes)
>>> of buflen is the preferred mode of usage.
>> Furthermore, it's impossible to read more than 32MB from /dev/urandom
>> per invocation.
>>
>> So, actually, the random(4) page is very conservative about the reading
>> limit.
>>
>> My final conclusion: as long as you use /dev/urandom for cryptographic
>> purposes only, you should be ok, because you will never need *a lot* of
>> random data anyway in any sensible program. For non-cryptographic usage,
>> seed a user-space PRNG with a few bytes from /dev/urandom and you will
>> be good.
>>
>> Laurent
>>
>>
>>
>> Le 26/07/2015 22:20, Carl Winbäck a écrit :
>>> Hello Michael & Co,
>>>
>>> I would like to bring your attention to bug report 71211, ”random(4):
>>> clarify utility and volume”.
>>>
>>> https://bugzilla.kernel.org/show_bug.cgi?id=71211
>>>
>>> This report was filed over a year ago but still hasn’t received any response.
>>>
>>> Michael, do you have the time to take a look?
>>>
>>> Perhaps you, or someone else on the linux-man list, are familiar with
>>> CSPRNGs and have some ideas on how to reword this man page?
>>>
>>> Unfortunately I’m not at all an expert in this area, so I’m afraid I
>>> don’t have any specific suggestions regarding how to change this man
>>> page. But I still think it would be helpful to the Linux community if
>>> it could be improved, and as a result, hopefully cause less confusion
>>> regarding getting random numbers on Linux.
>>>
>>>
>>> Looking forward to hear your thoughts on this.
>>>
>>> Best regards,
>>> Carl Winbäck
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-man" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2
>>
>> iF4EAREIAAYFAlW2XawACgkQRTidSplJch4jJQD/d4LOrShlXmQx5RClVCdj7sKL
>> 6n7SQhlCIjfqvi86JDcA/28cCtYZ8HKL1RgWkgEjGmWoIH6ZA+AKJjgnmugk1wFj
>> =ff9U
>> -----END PGP SIGNATURE-----
>>
>>
> 
> 


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status for bug 71211? [random(4): clarify utility and volume]

[Michael Kerrisk (man-pages)]

[also adding in other people CCed on bug 71211]

On 07/27/2015 06:34 PM, Laurent Georget wrote:
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> Hello,
> 
> the text of this man page has been the subject of endless controversies
> for ages. CSPRNGs are confusing and the page unfortunately adds a
> little to the confusion. The newer getrandom(2) page is better, and I
> (personally, this doesn't engage Michael nor any other author) think
> that the random(4) page should be rewritten in the same spirit.
> (getrandom is a system call used to get a random number, by default,
> it's more or less equivalent than reading from /dev/urandom if you call
> it without flags and for less than 256 bytes).
> 
> Compare this in random(4)
> 
>> The kernel random-number generator is designed to produce a small
>> amount of high-quality seed material to seed a cryptographic pseudo-
>> random number generator (CPRNG). It is designed for security, not
>> speed, and is poorly suited to generating large amounts of random
>> data. Users should be very economical in the amount of seed material
>> that they read from /dev/urandom (and /dev/random); unnecessarily
>> reading large quantities of data from this device will have a
>> negative impact on other users of the device.
> with this in getrandom(2)
> 
>> *getrandom*() relies on entropy gathered from device drivers and other
>> sources of environmental noise.  Unnecessarily reading large
>> quantities of data will have a negative impact on other users of the
>> //dev/random/ and //dev/urandom/ devices.  Therefore, *getrandom*() should
>> not be used for Monte Carlo simulations or other programs/algorithms
>> which are doing probabilistic sampling.
> This says exactly the same thing, but getrandom(2) is less misleading.
> First note that the man page for random says that /dev/urandom is
> "poorly suited to generating large amounts of random data", not
> "poorly suited to generating large amounts of *cryptographic* random data".
> Basically, you can use /dev/urandom for all cryptographic purposes,
> because you never need so many bits at once when doing cryptography.
> Even generating several 16-bytes (128-bits) UIDs per minute if you need
> them to be cryptographically secure (you may want to think about this
> requirement, is it strict?) is not that much. A Monte-Carlo simulation
> requires, say (I don't know exactly let's make a rough guess) around
> several MB of random numbers per minute. That's 4 or 5 orders of
> magnitude higher than your UIDs use case. This would be a wrong
> usage of /dev/urandom for two reasons:
> - - it would be awfully slow
> - - you don't need cryptographically secure random numbers, so there's
> no need to deplete the entropy pool.
> Next, compare this in random(4):
> 
>> If  you    are  unsure  about  whether  you  should  use  /dev/random  or
>> /dev/urandom,  then  probably you want to use the latter.  As a general
>> rule, /dev/urandom should be  used  for    everything  except  long-lived
>> GPG/SSL/SSH keys.
> with this in getrandom(2):
> 
>> Unless  you  are     doing    long-term key generation (and perhaps not even
>> then), you probably shouldn't be using GRND_RANDOM.  The     cryptographic
>> algorithms  used for /dev/urandom are quite conservative, and so should
>> be sufficient for all purposes.    The  disadvantage  of  GRND_RANDOM  is
>> that  it     can block.  Furthermore, dealing with the partially fulfilled
>> getrandom() requests that can occur when     using    GRND_RANDOM  increases
>> code complexity.
> Again, the two man pages say the same. getrandom(2) is more nuanced,
> though.
> 
> To answer your question about how much you can ask /dev/urandom for :
> 
> in random(4) :
> 
>> if any program reads more than 256 bits (32 bytes) from the kernel    random
>> pool  per  invocation, or per reasonable reseed interval (not less than
>> one minute), that should be taken as a sign that its  cryptography  is
>> not skillfully implemented.
> In getrandom(2):
> 
>> Calling getrandom() to read /dev/urandom for small values  (<= 256 bytes)
>> of buflen is the preferred mode of usage.
> Furthermore, it's impossible to read more than 32MB from /dev/urandom
> per invocation.
> 
> So, actually, the random(4) page is very conservative about the reading
> limit.
> 
> My final conclusion: as long as you use /dev/urandom for cryptographic
> purposes only, you should be ok, because you will never need *a lot* of
> random data anyway in any sensible program. For non-cryptographic usage,
> seed a user-space PRNG with a few bytes from /dev/urandom and you will
> be good.
> 
> Laurent
> 
> 
> 
> Le 26/07/2015 22:20, Carl Winbäck a écrit :
>> Hello Michael & Co,
>>
>> I would like to bring your attention to bug report 71211, ”random(4):
>> clarify utility and volume”.
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=71211
>>
>> This report was filed over a year ago but still hasn’t received any response.
>>
>> Michael, do you have the time to take a look?
>>
>> Perhaps you, or someone else on the linux-man list, are familiar with
>> CSPRNGs and have some ideas on how to reword this man page?
>>
>> Unfortunately I’m not at all an expert in this area, so I’m afraid I
>> don’t have any specific suggestions regarding how to change this man
>> page. But I still think it would be helpful to the Linux community if
>> it could be improved, and as a result, hopefully cause less confusion
>> regarding getting random numbers on Linux.
>>
>>
>> Looking forward to hear your thoughts on this.
>>
>> Best regards,
>> Carl Winbäck
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-man" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
> 
> iF4EAREIAAYFAlW2XawACgkQRTidSplJch4jJQD/d4LOrShlXmQx5RClVCdj7sKL
> 6n7SQhlCIjfqvi86JDcA/28cCtYZ8HKL1RgWkgEjGmWoIH6ZA+AKJjgnmugk1wFj
> =ff9U
> -----END PGP SIGNATURE-----
> 
> 


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status for bug 71211? [random(4): clarify utility and volume]

[Laurent Georget]

Hello,

the text of this man page has been the subject of endless controversies
for ages. CSPRNGs are confusing and the page unfortunately adds a
little to the confusion. The newer getrandom(2) page is better, and I
(personally, this doesn't engage Michael nor any other author) think
that the random(4) page should be rewritten in the same spirit.
(getrandom is a system call used to get a random number, by default,
it's more or less equivalent than reading from /dev/urandom if you call
it without flags and for less than 256 bytes).

Compare this in random(4)

> The kernel random-number generator is designed to produce a small
> amount of high-quality seed material to seed a cryptographic pseudo-
> random number generator (CPRNG). It is designed for security, not
> speed, and is poorly suited to generating large amounts of random
> data. Users should be very economical in the amount of seed material
> that they read from /dev/urandom (and /dev/random); unnecessarily
> reading large quantities of data from this device will have a
> negative impact on other users of the device.

with this in getrandom(2)

> *getrandom*() relies on entropy gathered from device drivers and other
> sources of environmental noise.  Unnecessarily reading large
> quantities of data will have a negative impact on other users of the
> //dev/random/ and //dev/urandom/ devices.  Therefore, *getrandom*() should
> not be used for Monte Carlo simulations or other programs/algorithms
> which are doing probabilistic sampling.

This says exactly the same thing, but getrandom(2) is less misleading.
First note that the man page for random says that /dev/urandom is
"poorly suited to generating large amounts of random data", not
"poorly suited to generating large amounts of *cryptographic* random data".
Basically, you can use /dev/urandom for all cryptographic purposes,
because you never need so many bits at once when doing cryptography.
Even generating several 16-bytes (128-bits) UIDs per minute if you need
them to be cryptographically secure (you may want to think about this
requirement, is it strict?) is not that much. A Monte-Carlo simulation
requires, say (I don't know exactly let's make a rough guess) around
several MB of random numbers per minute. That's 4 or 5 orders of
magnitude higher than your UIDs use case. This would be a wrong
usage of /dev/urandom for two reasons:
- it would be awfully slow
- you don't need cryptographically secure random numbers, so there's
no need to deplete the entropy pool.

Next, compare this in random(4):

> If  you    are  unsure  about  whether  you  should  use  /dev/random  or
> /dev/urandom,  then  probably you want to use the latter.  As a general
> rule, /dev/urandom should be  used  for    everything  except  long-lived
> GPG/SSL/SSH keys.

with this in getrandom(2):

> Unless  you  are     doing    long-term key generation (and perhaps not even
> then), you probably shouldn't be using GRND_RANDOM.  The     cryptographic
> algorithms  used for /dev/urandom are quite conservative, and so should
> be sufficient for all purposes.    The  disadvantage  of  GRND_RANDOM  is
> that  it     can block.  Furthermore, dealing with the partially fulfilled
> getrandom() requests that can occur when     using    GRND_RANDOM  increases
> code complexity.

Again, the two man pages say the same. getrandom(2) is more nuanced,
though.

To answer your question about how much you can ask /dev/urandom for:

in random(4) :

> if any program reads more than 256 bits (32 bytes) from the kernel    random
> pool  per  invocation, or per reasonable reseed interval (not less than
> one minute), that should be taken as a sign that its  cryptography  is
> not skillfully implemented.

In getrandom(2):

> Calling getrandom() to read /dev/urandom for small values  (<= 256 bytes)
> of buflen is the preferred mode of usage.

Furthermore, it's impossible to read more than 32MB from /dev/urandom
at once.

So, actually, the random(4) page is very conservative about the reading
limit.

My final conclusion: as long as you use /dev/urandom for cryptographic
purposes only, you should be ok, because you will never need *a lot* of
random data anyway in any sensible program. For non-cryptographic usage,
seed a user-space PRNG with a few bytes from /dev/urandom (or something
else like time(NULL) or a known constant) and you will be good.

Laurent



Le 26/07/2015 22:20, Carl Winbäck a écrit :
> Hello Michael & Co,
>
> I would like to bring your attention to bug report 71211, ”random(4):
> clarify utility and volume”.
>
> https://bugzilla.kernel.org/show_bug.cgi?id=71211
>
> This report was filed over a year ago but still hasn’t received any response.
>
> Michael, do you have the time to take a look?
>
> Perhaps you, or someone else on the linux-man list, are familiar with
> CSPRNGs and have some ideas on how to reword this man page?
>
> Unfortunately I’m not at all an expert in this area, so I’m afraid I
> don’t have any specific suggestions regarding how to change this man
> page. But I still think it would be helpful to the Linux community if
> it could be improved, and as a result, hopefully cause less confusion
> regarding getting random numbers on Linux.
>
>
> Looking forward to hear your thoughts on this.
>
> Best regards,
> Carl Winbäck
> --
> To unsubscribe from this list: send the line "unsubscribe linux-man" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html