[uml-devel] Strange behaviour of hwrng

[uml-devel] Strange behaviour of hwrng

[Gerd Stolpmann]

Hi list,

I recently tested the hwrng driver. In principal, it works, but it
sometimes eats up all host CPU time. In particular, I can see that the
UML system does a (blocking) read on /dev/hwrng, and that the host
system loops while reading from /dev/random which almost always returns
-EAGAIN. (Found that out with strace, in the hope the output is
correct.)

Nevertheless rngd seems to work, because when killing with SIGUSR1 it
shows some reasonable statistics.

My test setup is as follows:

- Host system: 2.6.12.6 with SKAS3 v8.2 patch
- UML system: 2.6.14-rc5, in SKAS3 mode.
  Using rngd from Debian Sarge.
  TLS is disabled.

Let me know if you need more information.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Telefon: 06151/153855                  Telefax: 06151/997714
------------------------------------------------------------



-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Strange behaviour of hwrng

[Blaisorblade]

On Friday 21 October 2005 20:24, Gerd Stolpmann wrote:
> Hi list,
>
> I recently tested the hwrng driver. In principal, it works, but it
> sometimes eats up all host CPU time. In particular, I can see that the
> UML system does a (blocking) read on /dev/hwrng,

Could you please elaborate on that? Who is doing the read, the rng tools? 
Also, are they doing only a single read (in which case, it's we the ones who 
are looping and we're broken) or they are astonished by our -EAGAIN and keep 
trying (in which case, they are broken)?

Actually, from looking at the code (arch/um/drivers/random.c:rng_dev_read()) 
it appears that we _do_ loop ourselves, if the read is blocking. Not sure how 
to handle that.

We could return maybe -EIO, possibly when a rate limit is exceeded (not 
trivial to do, though - I must learn using timers first).
> and that the host 
> system loops while reading from /dev/random which almost always returns
> -EAGAIN. (Found that out with strace, in the hope the output is
> correct.)

Well, if the host hasn't enough entropy, it's reasonable for it to return 
-EAGAIN.

And we should do the same (the loop is actually executed by the UML code, 
right?).

However, probably the tools inside UML don't expect a lot to get -EAGAIN from 
an hardware generator. So possibly they are not ready to handle that well.

> Nevertheless rngd seems to work, because when killing with SIGUSR1 it
> shows some reasonable statistics.

> My test setup is as follows:

> - Host system: 2.6.12.6 with SKAS3 v8.2 patch
> - UML system: 2.6.14-rc5, in SKAS3 mode.
>   Using rngd from Debian Sarge.
>   TLS is disabled.

> Let me know if you need more information.

> Gerd

-- 
Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade

	

	
		
___________________________________ 
Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB 
http://mail.yahoo.it



-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Strange behaviour of hwrng

[Gerd Stolpmann]

Am Freitag, den 21.10.2005, 20:42 +0200 schrieb Blaisorblade:
> On Friday 21 October 2005 20:24, Gerd Stolpmann wrote:
> > Hi list,
> >
> > I recently tested the hwrng driver. In principal, it works, but it
> > sometimes eats up all host CPU time. In particular, I can see that the
> > UML system does a (blocking) read on /dev/hwrng,
> 
> Could you please elaborate on that? Who is doing the read, the rng tools? 

Yes, one of the rngd threads is doing the read:

play:~# strace -p 517
Process 517 attached - interrupt to quit
read(3,  <unfinished ...>

top shows it consumes 12.5% CPU time (on the UML system) although it is
blocking, I guess it is system time accounted for this process. On the
host system, UML consumes more than 70% CPU time.

> Also, are they doing only a single read (in which case, it's we the ones who 
> are looping and we're broken) or they are astonished by our -EAGAIN and keep 
> trying (in which case, they are broken)?

Obviously, they don't get the -EAGAIN, so they are only doing a single
read. (We should see the -EAGAIN with strace if it occurred, shouldn'd
we.)

> Actually, from looking at the code (arch/um/drivers/random.c:rng_dev_read()) 
> it appears that we _do_ loop ourselves, if the read is blocking. Not sure how 
> to handle that.

I think this is the problem.

> We could return maybe -EIO, possibly when a rate limit is exceeded (not 
> trivial to do, though - I must learn using timers first).

Maybe an opportunity...

> > and that the host 
> > system loops while reading from /dev/random which almost always returns
> > -EAGAIN. (Found that out with strace, in the hope the output is
> > correct.)
> 
> Well, if the host hasn't enough entropy, it's reasonable for it to return 
> -EAGAIN.

Of course, it it will do most of the time.

> And we should do the same (the loop is actually executed by the UML code, 
> right?).

If the descriptor is non-blocking.

> However, probably the tools inside UML don't expect a lot to get -EAGAIN from 
> an hardware generator. So possibly they are not ready to handle that well.

Obviously, the real hardware generators block when there is not enough
entropy, and rngd was written for them. I think there are two ways of
fixing the problem:

(1) Change rngd. If it sees -EAGAIN, it sleeps for a moment. This is not
easy, because it has to find out the available bit rate of entropy, in
order to determine a reasonable frequency of polling /dev/hwrng.

(2) Fix /dev/hwrng such that it blocks when it is out of entropy. Looks
like the same problem, only within the kernel.

Gerd

> 
> > Nevertheless rngd seems to work, because when killing with SIGUSR1 it
> > shows some reasonable statistics.
> 
> > My test setup is as follows:
> 
> > - Host system: 2.6.12.6 with SKAS3 v8.2 patch
> > - UML system: 2.6.14-rc5, in SKAS3 mode.
> >   Using rngd from Debian Sarge.
> >   TLS is disabled.
> 
> > Let me know if you need more information.
> 
> > Gerd
> 
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Telefon: 06151/153855                  Telefax: 06151/997714
------------------------------------------------------------



-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Strange behaviour of hwrng

[Blaisorblade]

On Friday 21 October 2005 21:06, Gerd Stolpmann wrote:
> Am Freitag, den 21.10.2005, 20:42 +0200 schrieb Blaisorblade:
> > On Friday 21 October 2005 20:24, Gerd Stolpmann wrote:
> > > Hi list,
> > >
> > > I recently tested the hwrng driver. In principal, it works, but it
> > > sometimes eats up all host CPU time. In particular, I can see that the
> > > UML system does a (blocking) read on /dev/hwrng,
> >
> > Could you please elaborate on that? Who is doing the read, the rng tools?
>
> Yes, one of the rngd threads is doing the read:
>
> play:~# strace -p 517
> Process 517 attached - interrupt to quit
> read(3,  <unfinished ...>
>
> top shows it consumes 12.5% CPU time (on the UML system) although it is
> blocking, I guess it is system time accounted for this process. On the
> host system, UML consumes more than 70% CPU time.

> > Also, are they doing only a single read (in which case, it's we the ones
> > who are looping and we're broken) or they are astonished by our -EAGAIN
> > and keep trying (in which case, they are broken)?
>
> Obviously, they don't get the -EAGAIN, so they are only doing a single
> read. (We should see the -EAGAIN with strace if it occurred, shouldn'd
> we.)
Yes, we'd see it. Anyway, after seeing rngd is doing a blocking read, I 
guessed that.
> > Actually, from looking at the code
> > (arch/um/drivers/random.c:rng_dev_read()) it appears that we _do_ loop
> > ourselves, if the read is blocking. Not sure how to handle that.
>
> I think this is the problem.
Yes, agreed.
> > We could return maybe -EIO, possibly when a rate limit is exceeded (not
> > trivial to do, though - I must learn using timers first).
>
> Maybe an opportunity...
>
> > > and that the host
> > > system loops while reading from /dev/random which almost always returns
> > > -EAGAIN. (Found that out with strace, in the hope the output is
> > > correct.)
> >
> > Well, if the host hasn't enough entropy, it's reasonable for it to return
> > -EAGAIN.
>
> Of course, it it will do most of the time.
Well, the host should have more entropy than the guest. But if the host hasn't 
a real hardware generator, yes, that will happen. For the normal rngd 
context, the hardware is expected to give a continous flow.
> > And we should do the same (the loop is actually executed by the UML code,
> > right?).

> If the descriptor is non-blocking.
Theoretically yes, in practice this limitation is a problem.
> > However, probably the tools inside UML don't expect a lot to get -EAGAIN
> > from an hardware generator. So possibly they are not ready to handle that
> > well.

> Obviously, the real hardware generators block when there is not enough
> entropy, and rngd was written for them. I think there are two ways of
> fixing the problem:

> (1) Change rngd. If it sees -EAGAIN, it sleeps for a moment. This is not
> easy, because it has to find out the available bit rate of entropy, in
> order to determine a reasonable frequency of polling /dev/hwrng.
We could implement poll() for that purpose, 
> (2) Fix /dev/hwrng such that it blocks when it is out of entropy. Looks
> like the same problem, only within the kernel.

Proper solution: use a separate _host_ thread, which either sits blocked in a 
blocking read or calls poll() on the host /dev/random, and have it do the 
work. It's not easy however.
The ubd driver works currently this way, but the thing is not trivial to get 
right (and the UBD rewrite in the works has shown that).

A simpler solution is to increase the parameter passed to schedule_timeout. We 
currently sleep for one hundredth of second (actually one jiffy).

But only if need_resched(), which is wrong. I.e. if a timeslice lasts 80 ms 
(as is possible) we'll sleep 1 ms every 80 ones, which is unfair. Also, 
there's no need to continously poll the host /dev/random.

So, in rng_dev_read, in this piece of code:
                       if(need_resched()){
                                current->state = TASK_INTERRUPTIBLE;
                                schedule_timeout(1);
                        }

Remove the need_resched() checking, and retest. You can also test increasing 
the param to schedule timeout (it's currently 10 ms, but it's better to use 
HZ to mean a second and, say, HZ/100 to mean 10ms).

In any case, give me an answer on this.
Bye!
-- 
Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade

	

	
		
___________________________________ 
Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB 
http://mail.yahoo.it



-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Strange behaviour of hwrng

[Jeff Dike]

On Sat, Oct 22, 2005 at 02:52:00AM +0200, Blaisorblade wrote:
> Proper solution: use a separate _host_ thread, which either sits blocked in a 
> blocking read or calls poll() on the host /dev/random, and have it do the 
> work. 

Yeah, unfortunately.  The real right thing to do is give the host random
driver fasync support (I'm not sure how possible this is, however, given the
nature of the hardware).  Then we could register an IRQ for having some 
randomness and do this right.

> It's not easy however.

It's fairly trivial.  Look at the SIGIO workaround thread - this is essentially
the same thing.

> The ubd driver works currently this way, but the thing is not trivial to get 
> right (and the UBD rewrite in the works has shown that).

The complexity there is due to entirely different causes, namely making sure
that requests are sent out in order and that pieces of the bitmap are written
out when they should be, and not before.

				Jeff


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Strange behaviour of hwrng

[Blaisorblade]

On Saturday 22 October 2005 05:11, Jeff Dike wrote:
> On Sat, Oct 22, 2005 at 02:52:00AM +0200, Blaisorblade wrote:

> It's fairly trivial.  Look at the SIGIO workaround thread - this is
> essentially the same thing.
_Yes_, but see below.
> > The ubd driver works currently this way, but the thing is not trivial to
> > get right (and the UBD rewrite in the works has shown that).

> The complexity there is due to entirely different causes, namely making
> sure that requests are sent out in order and that pieces of the bitmap are
> written out when they should be, and not before.
I also remember the deadlock due to the pipe buffer used for communication 
filling completely... that's what could be related.
-- 
Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade

	

	
		
___________________________________ 
Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB 
http://mail.yahoo.it



-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Strange behaviour of hwrng

[Gerd Stolpmann]

Am Samstag, den 22.10.2005, 02:52 +0200 schrieb Blaisorblade:
> > Obviously, the real hardware generators block when there is not enough
> > entropy, and rngd was written for them. I think there are two ways of
> > fixing the problem:
> 
> > (1) Change rngd. If it sees -EAGAIN, it sleeps for a moment. This is not
> > easy, because it has to find out the available bit rate of entropy, in
> > order to determine a reasonable frequency of polling /dev/hwrng.
> We could implement poll() for that purpose, 
> > (2) Fix /dev/hwrng such that it blocks when it is out of entropy. Looks
> > like the same problem, only within the kernel.
> 
> Proper solution: use a separate _host_ thread, which either sits blocked in a 
> blocking read or calls poll() on the host /dev/random, and have it do the 
> work. It's not easy however.
> The ubd driver works currently this way, but the thing is not trivial to get 
> right (and the UBD rewrite in the works has shown that).
> 
> A simpler solution is to increase the parameter passed to schedule_timeout. We 
> currently sleep for one hundredth of second (actually one jiffy).
> 
> But only if need_resched(), which is wrong. I.e. if a timeslice lasts 80 ms 
> (as is possible) we'll sleep 1 ms every 80 ones, which is unfair. Also, 
> there's no need to continously poll the host /dev/random.
> 
> So, in rng_dev_read, in this piece of code:
>                        if(need_resched()){
>                                 current->state = TASK_INTERRUPTIBLE;
>                                 schedule_timeout(1);
>                         }
> 
> Remove the need_resched() checking, and retest. You can also test increasing 
> the param to schedule timeout (it's currently 10 ms, but it's better to use 
> HZ to mean a second and, say, HZ/100 to mean 10ms).

Ok, I tested that with schedule_timeout(HZ), i.e. it polls once per
second. This looks good enough for the case that the host /dev/random is
slow. rngd reports that it gets about 60 random bits/sec, which is about
the same as before with busy waiting.

The code looks now:

                        if(1){
                                current->state = TASK_INTERRUPTIBLE;
                                schedule_timeout(HZ);
                        }

It would be interesting how well this works for a hardware entropy
generator. I don't have such a board, however.

Another question is how fair we are. It may happen that host processes
like sshd starve that need entropy only from time to time, because all
randomness was pulled out of /dev/random from greedy UML kernels.
Ideally, the UMLs take only entropy when the entropy pool is filled to a
certain watermark at minimum.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Telefon: 06151/153855                  Telefax: 06151/997714
------------------------------------------------------------



-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.
Get Certified Today * Register for a JBoss Training Course
Free Certification Exam for All Training Attendees Through End of 2005
Visit http://www.jboss.com/services/certification for more information
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel