dom0 and domU /dev/urandom generating too less entropy

dom0 and domU /dev/urandom generating too less entropy

[Stephan Seitz]

[-- Attachment #1.1.1: Type: text/plain, Size: 2180 bytes --]

Hi there,

I've recently seen problems after migrating physical servers into
paravirtualized domU's.

The migrated systems vary from debian woody, sarge, ubuntu >=breezy,
each system with it's own but manageable problems.

On thing in common is: /dev/urandom generates too less entropy for
e.g. ssh-keygen.
Last days, i found even sshd itself dying by too less entropy:

sshd[26134]: fatal: Couldn't obtain random bytes (error 604389476)

We're using an own build derived from the 3.1.0 tarball, but without
any substantial changes to the code:
The currently used kernel has been heavily patched, but this issue
doesn't seem to be kernel-specific.
We tried the 2.6.18 (xensource 3.1.0) as well as different distro
kernels.

host                   :
release                : 2.6.20-100-server
version                : #2 SMP Sat Jun 2 12:18:40 UTC 2007
machine                : i686
nr_cpus                : 4
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 2394
hw_caps                : bfebfbff:20100000:00000000:00000140:0000e3bd:00000000:00000001
total_memory           : 8190
free_memory            : 1
xen_major              : 3
xen_minor              : 1
xen_extra              : .0
xen_caps               : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xf5800000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu4)
cc_compile_by          : root
cc_compile_domain      : halo.local
cc_compile_date        : Wed May 23 02:33:53 CEST 2007
xend_config_format     : 4


Do you know about a workaround, or maybe the possibility for another (xen-specific) RNG
besides of /dev/urandom ?

Thanks in advance!


-- 
Stephan Seitz
Senior System Administrator

*netz-haut* e.K.
multimediale kommunikation

zweierweg 22
97074 würzburg

fon: +49 931 2876247
fax: +49 931 2876248

web: www.netz-haut.de <http://www.netz-haut.de/>

registriergericht: amtsgericht würzburg, hra 5054

[-- Attachment #1.1.2: s.seitz.vcf --]
[-- Type: text/x-vcard, Size: 335 bytes --]

begin:vcard
fn:Stephan Seitz
n:Seitz;Stephan
org:netz-haut e.K.
adr:;;Zweierweg 22;Wuerzburg;Bayern;97074;Deutschland
email;internet:s.seitz@netz-haut.de
title:Senior System Administrator
tel;work:+49-931-287-6247
tel;fax:+49-931-287-6248
x-mozilla-html:FALSE
url:http://www.schwarz-mode.de/files/
version:2.1
end:vcard


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: dom0 and domU /dev/urandom generating too less entropy

[Keir Fraser]

On 10/10/07 21:00, "Stephan Seitz" <s.seitz@netz-haut.de> wrote:

> Do you know about a workaround, or maybe the possibility for another
> (xen-specific) RNG
> besides of /dev/urandom ?

I'm surprised you see failures. By my understanding, /dev/urandom is always
supposed to return the request number of bytes, but their randomness depends
on the amount of entropy currently in the pool. Perhaps sshd explicitly
interrogates urandom to find out how much entropy it has gathered?

Anyway, the domU kernel gathers entropy from the interrupt delivery times of
the netfront and blkfront drivers. This is similar to what a native kernel
does. It's not clear how we can easily improve on that without e.g.,
plumbing through a hardware RNG to domUs.

 -- Keir

Re: [Xen-users] Re: dom0 and domU /dev/urandom generating too less entropy

[Robbie Dinn]

Keir Fraser wrote:
> On 10/10/07 21:00, "Stephan Seitz" <s.seitz@netz-haut.de> wrote:
> 
>> Do you know about a workaround, or maybe the possibility for another
>> (xen-specific) RNG
>> besides of /dev/urandom ?
> 
> I'm surprised you see failures. By my understanding, /dev/urandom is always
> supposed to return the request number of bytes, but their randomness depends
> on the amount of entropy currently in the pool. Perhaps sshd explicitly
> interrogates urandom to find out how much entropy it has gathered?
I haven't checked (I am too laxy to strace it) but I believe that sshd
is using /dev/random not /dev/urandom. You can see how much entropy is
available by cat'ing /proc/sys/kernel/random/entropy_avail .
> 
> Anyway, the domU kernel gathers entropy from the interrupt delivery times of
> the netfront and blkfront drivers. This is similar to what a native kernel
> does. It's not clear how we can easily improve on that without e.g.,
> plumbing through a hardware RNG to domUs.

I had a similar problem on a mail server providing a pop3 service. Every
time a client machine connected to the pop3 daemon (cyrus imap actually),
it consumed entropy. More entropy was consumed for each connection
than was provided by the packets arriving. The machine ran of entropy
and stopped providing bytes via /dev/random. The pop3 daemon ground
to a halt because it was waiting to read bytes from /dev/random.

The work around was to feed entropy into the random number generator.
There is a user space tool to do this called 'rngd'.

The correct way to do this would be, as you say, to get the the entropy
from outside the domU. I used a dirty hack instead, I ran

/sbin/rngd --rng-device=/dev/urandom

Yes is wrong and evil but it got me up and running again.

Re: Re: [Xen-devel] dom0 and domU /dev/urandom generating too less entropy

[Robbie Dinn]

Robbie Dinn wrote:
> Keir Fraser wrote:
>> On 10/10/07 21:00, "Stephan Seitz" <s.seitz@netz-haut.de> wrote:
>>
>>> Do you know about a workaround, or maybe the possibility for another
>>> (xen-specific) RNG
>>> besides of /dev/urandom ?
>> I'm surprised you see failures. By my understanding, /dev/urandom is always
>> supposed to return the request number of bytes, but their randomness depends
>> on the amount of entropy currently in the pool. Perhaps sshd explicitly
>> interrogates urandom to find out how much entropy it has gathered?
> I haven't checked (I am too laxy to strace it) but I believe that sshd
> is using /dev/random not /dev/urandom. You can see how much entropy is
> available by cat'ing /proc/sys/kernel/random/entropy_avail .

No I am wrong. I did an strace of sshd and it does read /dev/urandom
not /dev/random as I claimed.

read(5, "\0\0\0\1\212\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"..., 752) = 752
open("/dev/urandom", O_RDONLY|O_NONBLOCK|O_NOCTTY) = 3
fstat64(3, {st_mode=S_IFCHR|0644, st_rdev=makedev(1, 9), ...}) = 0
poll([{fd=3, events=POLLIN, revents=POLLIN}], 1, 10) = 1
read(3, "p:\326\232y\326f\364<#L\204(<\244\"\275,r\263\r%Z\f\304"..., 32) = 32
close(3)                                = 0

Sorry for the noise.

>> Anyway, the domU kernel gathers entropy from the interrupt delivery times of
>> the netfront and blkfront drivers. This is similar to what a native kernel
>> does. It's not clear how we can easily improve on that without e.g.,
>> plumbing through a hardware RNG to domUs.
> 
> I had a similar problem on a mail server providing a pop3 service. Every
> time a client machine connected to the pop3 daemon (cyrus imap actually),
> it consumed entropy. More entropy was consumed for each connection
> than was provided by the packets arriving. The machine ran of entropy
s/machine ran of entropy/machine ran out of entropy/

> and stopped providing bytes via /dev/random. The pop3 daemon ground
> to a halt because it was waiting to read bytes from /dev/random.
> 
> The work around was to feed entropy into the random number generator.
> There is a user space tool to do this called 'rngd'.
> 
> The correct way to do this would be, as you say, to get the the entropy
> from outside the domU. I used a dirty hack instead, I ran
> 
> /sbin/rngd --rng-device=/dev/urandom
> 
> Yes is wrong and evil but it got me up and running again.
s/Yes is wrong/Yes, it is wrong/

Re: [Xen-users] Re: dom0 and domU /dev/urandom generating too less entropy

[Stephan Seitz]

[-- Attachment #1.1.1: Type: text/plain, Size: 990 bytes --]

> Robbie Dinn wrote:
>> Keir Fraser wrote:

Thanks for your suggestions Keir and Robbie,

but it looks like /dev/urandom is used by sshd, but somewhere inside of sshd the stream is checked against a minimum
entropy. I don't think this is a an issue by getting blocked via /dev/random.

I'll try the userspace rngd, but - as you said - this shouldn't be the preferred method of getting more entropy into
the system.

Another thing i'll try (later at night, as the domU's are productive) is, to configure the domU's using the same pCPU's
targetting that the pCPU's cache is filled only sub-optimal and the latency is higher ... (maybe ;) )

I'll report here later.

Greetings

Stephan




-- 
Stephan Seitz
Senior System Administrator

*netz-haut* e.K.
multimediale kommunikation

zweierweg 22
97074 würzburg

fon: +49 931 2876247
fax: +49 931 2876248

web: www.netz-haut.de <http://www.netz-haut.de/>

registriergericht: amtsgericht würzburg, hra 5054

[-- Attachment #1.1.2: s.seitz.vcf --]
[-- Type: text/x-vcard, Size: 335 bytes --]

begin:vcard
fn:Stephan Seitz
n:Seitz;Stephan
org:netz-haut e.K.
adr:;;Zweierweg 22;Wuerzburg;Bayern;97074;Deutschland
email;internet:s.seitz@netz-haut.de
title:Senior System Administrator
tel;work:+49-931-287-6247
tel;fax:+49-931-287-6248
x-mozilla-html:FALSE
url:http://www.schwarz-mode.de/files/
version:2.1
end:vcard


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel