[uml-devel] /dev/random problems _not_ solved in 2.6.26

[uml-devel] /dev/random problems _not_ solved in 2.6.26

[Stanislav Meduna]

Hi,

this is just to let you know that the /dev/random problems
are still not solved in the 2.6.26 version. A small dictionary
attack on sshd (in my case 260 tries over 12 minutes)
are able to produce

auth.log.0:Aug  3 05:30:35 dirk sshd[1825]: fatal: Couldn't obtain random bytes (error 
604389476)

and the sshd dies (followed by apache on the first subsequent
SSL request and so on).


Unfortunately I can't help you in tracing the uml process
or trying patches - this is a virtual server at my provider
and I don't have any access there. I could try to get
the .config file.


# uname -a
Linux dirk 2.6.26 #1 Wed Jul 30 10:56:10 CEST 2008 i686 GNU/Linux

# cat /proc/cpuinfo
processor       : 0
vendor_id       : User Mode Linux
model name      : UML
mode            : skas
host            : Linux seldon-base 2.6.23.17 #2 SMP Sat Jun 21 15:04:22 CEST 2008 i686
bogomips        : 4771.02

# cat /proc/sys/kernel/random/entropy_avail
293

After exhausting the entropy with

   cat /dev/random | od -cx

the entropy only slooowly gets up, one byte per several
seconds or so with me typing in the ssh console...


Regards
-- 
                                 Stano


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] /dev/random problems _not_ solved in 2.6.26

[Jeff Dike]

On Mon, Aug 04, 2008 at 02:40:01PM +0200, Stanislav Meduna wrote:
> this is just to let you know that the /dev/random problems
> are still not solved in the 2.6.26 version. A small dictionary
> attack on sshd (in my case 260 tries over 12 minutes)
> are able to produce
> 
> auth.log.0:Aug  3 05:30:35 dirk sshd[1825]: fatal: Couldn't obtain random bytes (error 
> 604389476)
> 
> and the sshd dies (followed by apache on the first subsequent
> SSL request and so on).

Off the top of my head, you need the following:
    2.6.26 UML, which you do
    rngtools installed in the UML
    sufficient entropy on the host

This last one shouldn't be taken for granted.  I had problems with
this when debugging the random driver changes - on my laptop with me
typing on it, mousing, disk going, and network traffic going in and
out.  It would be worse on a colocated, no-keyboard, no-mouse, server,
which are notorious for this problem.

> Unfortunately I can't help you in tracing the uml process
> or trying patches - this is a virtual server at my provider
> and I don't have any access there. I could try to get
> the .config file.

You could ask them about how their servers get their randomness.

If nothing else works, I suppose a nasty workaround might be to
replace /dev/random with dev/urandom, but that has security
implications.

    	      	   	 Jeff

-- 
Work email - jdike at linux dot intel dot com

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] /dev/random problems _not_ solved in 2.6.26

[Stanislav Meduna]

Jeff Dike wrote:

> If nothing else works, I suppose a nasty workaround might be to
> replace /dev/random with dev/urandom, but that has security
> implications.

I am aware of the implications, but unfortunately it did
not help either.

# ls -l /dev/*random*
crw-rw-rw- 1 root root 1, 8 2008-08-05 17:41 /dev/oldrandom
lrwxrwxrwx 1 root root    7 2008-08-05 17:41 /dev/random -> urandom
crw-rw-rw- 1 root root 1, 9 2008-08-05 17:42 /dev/urandom

# grep random /var/log/auth.log
Aug  5 19:53:50 dirk sshd[3117]: fatal: Couldn't obtain random bytes (error 604389476)

# cat /proc/sys/kernel/random/entropy_avail
335

I am now talking to the host owner - he is running quite
a few UML guests, each in chroot environment, and one of the
problems is that his entropy gets sucked out...

Now he has replaced random with urandom in my chroot jail,
so the UML's random is his urandom. Let's see what happens...

-- 
                                     Stano


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] /dev/random problems _not_ solved in 2.6.26

[Brock, Anthony - NET]

> -----Original Message-----
> I am aware of the implications, but unfortunately it did
> not help either.
> 
> # ls -l /dev/*random*
> crw-rw-rw- 1 root root 1, 8 2008-08-05 17:41 /dev/oldrandom
> lrwxrwxrwx 1 root root    7 2008-08-05 17:41 /dev/random -> urandom
> crw-rw-rw- 1 root root 1, 9 2008-08-05 17:42 /dev/urandom
> 
> # grep random /var/log/auth.log
> Aug  5 19:53:50 dirk sshd[3117]: fatal: Couldn't obtain random bytes
> (error 604389476)
> 
> # cat /proc/sys/kernel/random/entropy_avail
> 335
> 
> I am now talking to the host owner - he is running quite
> a few UML guests, each in chroot environment, and one of the
> problems is that his entropy gets sucked out...
> 
> Now he has replaced random with urandom in my chroot jail,
> so the UML's random is his urandom. Let's see what happens...

Please keep me appraised of what you find. Our servers never had this
problem until upgrading to a 2.6.2x version of the kernel. SSHD, Apache
and several other software packages worked fine under previous versions
(and still do when we downgrade them). Its also worth noting that this
behavior has never been exhibited on the host system. We've only seen it
on our virtual systems.

I keep meaning to go back and search for exactly which version
introduced the changed behavior, but I don't have time. Instead we now
have scripts that automatically restart processes that use OpenSSL when
they die. We tried shifting to urandom earlier to see if it would
resolve the issue, but didn't see a change in behavior. I'm interested
in knowing if your approach will function better.

Tony


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] /dev/random problems _not_ solved in 2.6.26

[Stanislav Meduna]

Brock, Anthony - NET wrote:

>> Now he has replaced random with urandom in my chroot jail,
>> so the UML's random is his urandom. Let's see what happens...
> 
> Please keep me appraised of what you find.

Look like this works - at least there were no problems over the night.

>                                                  Instead we now
> have scripts that automatically restart processes that use OpenSSL when
> they die.

Tried this too. Note that if you are checking the affected
ports it can actually make the situation worse - I tried this
using monit checking for ssh and https and as it creates
a SSL connection to do this, it needs the entropy, both
the client and the server.

> We tried shifting to urandom earlier to see if it would
> resolve the issue, but didn't see a change in behavior.

I looked into the openssl code (vanilla, dunno whether
there are OS-specific patches, I am using Debian testing).

It tries to use both /dev/urandom, /dev/random and
/dev/srandom, and it filters them according to major/minor -
so linking the /dev/* on the guest won't help, it picks one.
However, this one should be the urandom first...

It then select-s(!) for the device, 10 ms in each step.
If anything fails, it proceeds to the next device and
tries the same there too.

There is a suspicious code sequence:

  if (select(fd+1,&fset,NULL,NULL,&t) >= 0)
  ...
  usec = t.tv_usec;
  ...
  /* Some Unixen will update t in select(), some
     won't.  For those who won't, or if we
     didn't use select() in the first place,
     give up here, otherwise, we will do
     this once again for the remaining
     time. */
   if (usec == 10*1000)
     usec = 0;


 From the static code inspection it looks like one
of the following situations on urandom produces this:

- select returns -1 with errno other than EINTR/EAGAIN

- read on the device returns -1 or (!!!) 0 with errno
   other than EINTR/EAGAIN

- select select's successfully and immediately, leaves
   the time not slept unchanged in the time argument (which
   is IMHO fully legal, if it finds the bytes immediately)
   _and_ the read does not get all the needed bytes.

I don't have time now to exactly debug what is wrong,
but at least the last case looks as a bug in openssl,
triggered by some change in the select or read-behaviour.
I'll contact the openssl developers about it.


Linking the random to urandom on the host should make
the situation better, as it gives the guest a chance
for trying "another" device...

Regards
-- 
                                    Stano

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] /dev/random problems .. or FP registers corruption?!

[Stanislav Meduna]

Hi,

Well, I am now puzzled quite a bit more. Another problem
on this UML box is that if I do tail -f on some file,
I ocassionally get an assertion:

tail: xnanosleep.c:67: xnanosleep: Assertion `0 <= seconds' failed.

This seems to happen more often (or at all) when I am
doing something using the openssl - tail -f on an apache
access log runs fine when I am doing http accesses
and fails quite repeatedly when I am doing https
or starting/terminating new SSL connection from/to the box.

I also have a few strace-s from the failing sshd,
where I can't really understand how the code got there.

These two have one thing in common - a double variable
that got initialized long ago and at the time of the
problem it looks like it mysteriously changed value:
"entropy" in the case of openssl and "sleep_interval"
in tail.

Am I seeing ghosts? Anyone got other mysterious
problems with current UML kernel? Could it be that
some state-saving method is corrupting fp registers
or something like that? Was there some change in the
UML / vanilla kernel recently?

To the openssl-ers: I am starting to think that openssl
is fine here and that the problem is indeed in the
UML kernel.


Regards
-- 
                                  Stano

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] /dev/random problems .. or FP registers corruption?!

[Stanislav Meduna]

Stanislav Meduna wrote:

> Am I seeing ghosts? Anyone got other mysterious
> problems with current UML kernel? Could it be that
> some state-saving method is corrupting fp registers
> or something like that? Was there some change in the
> UML / vanilla kernel recently?

Confirmed: I can reproduce openssl changing a double
value in unrelated process. I was not able to come
up with a code not relying on external software yet:

===
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <time.h>
#include <signal.h>

testdblchg()
{
   volatile double x0 = 2.0;
   volatile double x1;
   int iter = 0;

   fprintf(stderr, "Starting loop, pid=%d\n", getpid());

   while(1)
   {
     ++iter;
     x1 = x0;
     x0 += 1.0;
     if (x0 != x1 + 1)
     {
       kill(getppid(), SIGTERM);
       fprintf(stderr, "Double variable changed! pid=%d, iteration=%d, should=%g, 
is=%g\n", getpid(), iter, x1+1, x0);
       exit(1);
     }
     sleep(1);
   }
}

testcrypto()
{
   while(1)
   {
     system("openssl genrsa -out /dev/null 4096");
     sleep(1);
   }
}

main()
{
   if (fork() == 0)
     testdblchg();

   testcrypto();
}
===

Output:
$ ./a.out
Starting loop, pid=16920
Generating RSA private key, 4096 bit long modulus
......++
.......................................................................................................................................................................++
e is 65537 (0x10001)
Generating RSA private key, 4096 bit long modulus
...................................Double variable changed! pid=16920, iteration=15, 
should=nan, is=nan
Terminated

$ uname -a
Linux dirk 2.6.26 #1 Wed Jul 30 10:56:10 CEST 2008 i686 GNU/Linux

$ cat /proc/cpuinfo
processor       : 0
vendor_id       : User Mode Linux
model name      : UML
mode            : skas
host            : Linux seldon-base 2.6.23.17 #2 SMP Sat Jun 21 15:04:22 CEST 2008 i686
bogomips        : 4771.02

$ gcc -v
Using built-in specs.
Target: i486-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.3.1-2' 
--with-bugurl=file:///usr/share/doc/gcc-4.3/README.Bugs 
--enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared 
--with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix 
--enable-nls --with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3 
--enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr 
--enable-targets=all --enable-cld --enable-checking=release --build=i486-linux-gnu 
--host=i486-linux-gnu --target=i486-linux-gnu
Thread model: posix
gcc version 4.3.1 (Debian 4.3.1-2)

$ dpkg -l libc6
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name           Version        Description
+++-==============-==============-============================================
ii  libc6          2.7-10         GNU C Library: Shared libraries


Regards
-- 
                                            Stano

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

[uml-devel] FP registers corruption

[Stanislav Meduna]

Erm.. after debugging the FP corruption for several hours
trying to isolate the cause I stumbled upon a patch from
Jeff Dike from February:

http://article.gmane.org/gmane.linux.uml.devel/11412

Did this or equivalent patch get into the mainline kernel?
Browsing through the source I don't think so :( If not,
where is the best place to fetch the UML tree with the
patches that the developers seem to find necessary,
applied?

Thanks
-- 
                                     Stano


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] FP registers corruption

[Jeff Dike]

On Fri, Aug 08, 2008 at 02:35:39PM +0200, Stanislav Meduna wrote:
> Erm.. after debugging the FP corruption for several hours
> trying to isolate the cause I stumbled upon a patch from
> Jeff Dike from February:
>
> http://article.gmane.org/gmane.linux.uml.devel/11412
>
> Did this or equivalent patch get into the mainline kernel?
> Browsing through the source I don't think so :( If not,
> where is the best place to fetch the UML tree with the
> patches that the developers seem to find necessary,
> applied?

It's there.  I just eyeballed the source.  It's commit
2f56debd77a8f52f1ac1d3c3d89cc7ce5e083230 - the only way you wouldn't
have it is by having a too-old version of UML.

     	      	       	       Jeff

-- 
Work email - jdike at linux dot intel dot com

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] FP registers corruption

[Stanislav Meduna]

Jeff Dike wrote:

>> Did this or equivalent patch get into the mainline kernel?
>> Browsing through the source I don't think so :( If not,
>> where is the best place to fetch the UML tree with the
>> patches that the developers seem to find necessary,
>> applied?
> 
> It's there.  I just eyeballed the source.  It's commit
> 2f56debd77a8f52f1ac1d3c3d89cc7ce5e083230 - the only way you wouldn't
> have it is by having a too-old version of UML.

Yes, you're right - sorry. I probably looked at the wrong
version (I used some diff-ing web interface) :( However,
the problem is still here.

I tried to isolate the exact version, but I was not able
to compile/run all the UML versions in my test environment
(current Ubuntu) - obviously the UML is quite sensitive
to the host kernel as well.

So the only thing I can say is that I cannot reproduce
it with neither 2.6.24.7 nor 2.6.26.2 guest in my environment.
The only instance I am seeing it is the 2.6.26 guest on
2.6.23.17 host at my provider :(

The owner of the host machine where the problem is easily
reproducible is back from the vacation, so I hope to have
some more information soon; however I am not sure I will
be able to convince him to update the host kernel too
(that means scheduled downtime etc., ...).

Any hint what should I grep and code-review for? The
symptoms of the bug look very similar to the page-fault
induced one, so the cause is maybe also similar/the same.
I played with the scheduler (sched_yield) etc and it does
not seem that it is the 'normal' context switch who is
corrupting the registers, more like it is some interrupt
handler or something like that.

Any other folks seeing ssh or spache/ssl crashing: could
you please post the UML and host versions and any other
relevant informatio such as number of the UML kernels
running, whether the machine is under memory pressure,
devices used (e.g. heavy networking usage), ...?

Thanks
-- 
                                      Stano

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] FP registers corruption

[Jeff Dike]

On Tue, Aug 26, 2008 at 09:20:56AM +0200, Stanislav Meduna wrote:
> I tried to isolate the exact version, but I was not able
> to compile/run all the UML versions in my test environment
> (current Ubuntu) - obviously the UML is quite sensitive
> to the host kernel as well.

It's not.  Any 2.6 UML should run on a any 2.6 host.  If you are
having problems that look like version skew problems, say what they
are.

> So the only thing I can say is that I cannot reproduce
> it with neither 2.6.24.7 nor 2.6.26.2 guest in my environment.
> The only instance I am seeing it is the 2.6.26 guest on
> 2.6.23.17 host at my provider :(

In an earlier email, you provide a nice test case.  However, I'm
running that here, and getting page after page of successful key
generation.

Is there anything else that might be relevant to reproducing this?

   	 	       	   Jeff

-- 
Work email - jdike at linux dot intel dot com

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] FP registers corruption

[Stanislav Meduna]

Jeff Dike wrote:

> It's not.  Any 2.6 UML should run on a any 2.6 host.  If you are
> having problems that look like version skew problems, say what they
> are.

My host is current Ubuntu and I was unable to test 2.6.25.x
there due to
   Locating the top of the address space ... Address 0x0 no good?
which AFAIK was a known problem caused by some ubuntu-specific
patch.

2.6.26 crashes here with

Locating the bottom of the address space ... 0x10000
Locating the top of the address space ... Checking that ptrace can change system call 
numbers...0xc0000000
Core dump limits :
         soft - 0
         hard - NONE
OK
Checking syscall emulation patch for ptrace...0xc0000000
Core dump limits :
         soft - 0
         hard - NONE
OK
Checking advanced syscall emulation patch for ptrace...0xc0000000
Core dump limits :
         soft - 0
         hard - NONE
OK
0xc0000000
Core dump limits :
         soft - 0
         hard - NONE
Checking for tmpfs mount on /dev/shm...OK
Checking PROT_EXEC mmap in /dev/shm/...OK
Checking for the skas3 patch in the host:
   - /proc/mm...not found: No such file or directory
   - PTRACE_FAULTINFO...not found
OK
   - PTRACE_LDT...not found
OK
OK
UML running in SKAS0 mode
Linux version 2.6.26 (stano@maroon) (gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)) #2 Mon Aug 
11 23:50:44 CEST 2008
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 6096
Kernel command line: root=/dev/ubda ubd0=/home/stano/uml/test.rootfs mem=24M con=pty 
con0=fd:0,fd:1 con1=xterm eth0=daemon,,unix,/var/run/uml-utilities/uml_switch.ctl
PID hash table entries: 128 (order: 7, 512 bytes)
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 21684k available
SLUB: Genslabs=12, HWalign=128, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Mount-cache hash table entries: 512
Checking for host processor cmov support...Yes
Checking that host ptys support output SIGIO...Yes
net_namespace: 624 bytes
NET: Registered protocol family 16
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NET: Registered protocol family 1
mconsole (version 2) initialized on /home/stano/.uml/umltest/mconsole
Checking host MADV_REMOVE support...OK
ubd: Synchronous mode
UML Watchdog Timer
Host TLS support detected
Detected host type: i386 (GDT indexes 6 to 9)
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
msgmni has been set to 42
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
brd: module loaded
loop: module loaded
SoftDog: cannot register miscdev on minor=130 (err=-16)
Mirror/redirect action on
TCP cubic registered
NET: Registered protocol family 17
Initialized stdio console driver
Console initialized on /dev/tty0
console [tty0] enabled
Initializing software serial port version 1
Choosing a random ethernet address for device eth0
Netdevice 0 (9a:fd:f3:b2:aa:10) : daemon backend (uml_switch version 3) - 
unix:/var/run/uml-utilities/uml_switch.ctl
console [mc-1] enabled
  ubda: unknown partition table
VFS: Mounted root (ext2 filesystem) readonly.
Stub registers -
         0 - 0
         1 - 0
         2 - 0
         3 - 0
         4 - 0
         5 - 0
         6 - 0
         7 - 0
         8 - 0
         9 - 0
         10 - 0
         11 - ffffffff
         12 - 0
         13 - 73
         14 - 10202
         15 - 0
         16 - 7b
wait_stub_done : failed to wait for SIGTRAP, pid = 24625, n = 24625, errno = 25, status = 
0xb7f
Kernel panic - not syncing: Attempted to kill init!

EIP: 0073:[<08189151>] CPU: 0 Not tainted ESP: 007b:bfb22dc0 EFLAGS: 00000246
     Not tainted
EAX: 00000000 EBX: 0000602c ECX: 00000013 EDX: 0000602c
ESI: 00006028 EDI: 0828e818 EBP: bfb2379a DS: 007b ES: 007b
09463bd8:  [<0808322b>] notifier_call_chain+0x25/0x48
09463bfc:  [<08083283>] atomic_notifier_call_chain+0x15/0x19
09463c14:  [<0807011b>] panic+0x52/0xce
09463c30:  [<08072ec0>] do_exit+0x5c/0x5bf
09463c4c:  [<08078767>] recalc_sigpending+0x16/0x25
09463c64:  [<0807349c>] do_group_exit+0x79/0xa5
09463c7c:  [<0807af6d>] get_signal_to_deliver+0x286/0x29b
09463ca0:  [<08058dc6>] do_signal+0x230/0x2e7
09463d00:  [<0807961e>] send_signal+0x197/0x1ae
09463d1c:  [<08065a5a>] set_signals+0x18/0x28
09463d24:  [<080799f7>] force_sig_info+0x9f/0xa9
09463d44:  [<08079a10>] force_sig+0xf/0x13
09463d68:  [<0805a26d>] fatal_sigsegv+0x1b/0x20
09463d78:  [<08067df1>] wait_stub_done+0xa1/0xa7
09463da4:  [<08067681>] do_syscall_stub+0x9c/0x16b
09463dc4:  [<08068d6d>] write_ldt_entry+0xef/0x14f
09463e0c:  [<08069482>] init_new_ldt+0x193/0x2d5
09463e2c:  [<080935b5>] __alloc_pages_internal+0x7d/0x341
09463e88:  [<0805a6e3>] init_new_context+0xc7/0x108
09463ea4:  [<080ade0f>] bprm_mm_init+0x41/0x169
09463ed0:  [<080aea02>] do_execve+0x6b/0x19a
09463ed8:  [<080a9114>] do_sys_open+0x96/0xba
09463eec:  [<0805738d>] execve1+0x21/0x39
09463f0c:  [<080573f5>] um_execve+0x14/0x33
09463f14:  [<0805907a>] kernel_execve+0x2b/0x34
09463f30:  [<080570d5>] run_init_process+0x15/0x19
09463f40:  [<0805717e>] init_post+0xa5/0xcf
09463f48:  [<080485ab>] kernel_init+0x1e5/0x1ef
09463f58:  [<080553b6>] tcp_congestion_default+0x0/0xc
09463fbc:  [<08064cd4>] run_kernel_thread+0x38/0x3f
09463fd8:  [<08064cb7>] run_kernel_thread+0x1b/0x3f
09463fe4:  [<0805838e>] new_thread_handler+0x53/0x79
09463fe8:  [<080483c6>] kernel_init+0x0/0x1ef


2.6.26.2 has no problems.

> In an earlier email, you provide a nice test case.  However, I'm
> running that here, and getting page after page of successful key
> generation.

Same here with 2.6.26.2 :(

> Is there anything else that might be relevant to reproducing this?

Well - the difference to my test environment is that
- the host machine is a SMP one (I am running the host inside
   vmware)
- there are quite a few UML guests running (AFAIK 5 to 8 -
   no space for testing this here yet)

Config should be the same, I used /proc/config.gz from
the failing machine.

dirk:~# uname -a
Linux dirk 2.6.26 #1 Wed Jul 30 10:56:10 CEST 2008 i686 GNU/Linux
dirk:~# cat /proc/cpuinfo
processor       : 0
vendor_id       : User Mode Linux
model name      : UML
mode            : skas
host            : Linux seldon-base 2.6.23.17 #2 SMP Sat Jun 21 15:04:22 CEST 2008 i686
bogomips        : 4771.02

-- 
                                         Stano

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] FP registers corruption

[Jeff Dike]

On Tue, Aug 26, 2008 at 08:07:20PM +0200, Stanislav Meduna wrote:
> My host is current Ubuntu and I was unable to test 2.6.25.x
> there due to
>   Locating the top of the address space ... Address 0x0 no good?
> which AFAIK was a known problem caused by some ubuntu-specific
> patch.

You can work around this by putting a 0 into /proc/sys/vm/mmap_min_addr.

> 2.6.26 crashes here with

This one is fixed in 2.6.26.2, as you note below.

> 2.6.26.2 has no problems.
>
>> In an earlier email, you provide a nice test case.  However, I'm
>> running that here, and getting page after page of successful key
>> generation.
>
> Same here with 2.6.26.2 :(

Hmmm.  If it's not a problem there, I'm tempted to call it fixed,
whatever it was.  Can you check current 2.6.27-rc?

	    	      	  	Jeff

-- 
Work email - jdike at linux dot intel dot com

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] FP registers corruption

[Stanislav Meduna]

Jeff Dike wrote:

> You can work around this by putting a 0 into /proc/sys/vm/mmap_min_addr.

Permission denied here - looks quite hard readonly on this kernel
even for root.

>> Same here with 2.6.26.2 :(
> 
> Hmmm.  If it's not a problem there, I'm tempted to call it fixed,
> whatever it was.  Can you check current 2.6.27-rc?

OK, I'll try to convince my provider to test 2.6.26.2 and
possibly 27-rc soon - I'll let you know of the result.

Thanks
-- 
                                   Stano

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] FP registers corruption

[Jeff Dike]

On Tue, Aug 26, 2008 at 08:51:51PM +0200, Stanislav Meduna wrote:
>> You can work around this by putting a 0 into /proc/sys/vm/mmap_min_addr.
>
> Permission denied here - looks quite hard readonly on this kernel
> even for root.

Hmmm.  The actual fix is below,

       	   	      	 Jeff

-- 
Work email - jdike at linux dot intel dot com


commit 40fb16a360d9c6459afee91dc793c1e3374feb94
Author: Tom Spink <tspink@gmail.com>
Date:   Thu Jun 5 22:46:12 2008 -0700

    uml: deal with inaccessible address space start
    
    This patch makes os_get_task_size locate the bottom of the address space,
    as well as the top.  This is for systems which put a lower limit on mmap
    addresses.  It works by manually scanning pages from zero onwards until a
    valid page is found.
    
    Because the bottom of the address space may not be zero, it's not
    sufficient to assume the top of the address space is the size of the
    address space.  The size is the difference between the top address and
    bottom address.
    
    [jdike@addtoit.com: changed the name to reflect that this function is
    supposed to return the top of the process address space, not its size and
    changed the return value to reflect that.  Also some minor formatting
    changes]
    Signed-off-by: Tom Spink <tspink@gmail.com>
    Signed-off-by: Jeff Dike <jdike@linux.intel.com>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

diff --git a/arch/um/include/os.h b/arch/um/include/os.h
index e2716ac..db5be46 100644
--- a/arch/um/include/os.h
+++ b/arch/um/include/os.h
@@ -299,6 +299,6 @@ extern int os_arch_prctl(int pid, int code, unsigned long *addr);
 extern int get_pty(void);
 
 /* sys-$ARCH/task_size.c */
-extern unsigned long os_get_task_size(void);
+extern unsigned long os_get_top_address(void);
 
 #endif
diff --git a/arch/um/kernel/um_arch.c b/arch/um/kernel/um_arch.c
index 9db85b2..8d84250 100644
--- a/arch/um/kernel/um_arch.c
+++ b/arch/um/kernel/um_arch.c
@@ -274,7 +274,7 @@ int __init linux_main(int argc, char **argv)
 	if (have_root == 0)
 		add_arg(DEFAULT_COMMAND_LINE);
 
-	host_task_size = os_get_task_size();
+	host_task_size = os_get_top_address();
 	/*
 	 * TASK_SIZE needs to be PGDIR_SIZE aligned or else exit_mmap craps
 	 * out
diff --git a/arch/um/os-Linux/sys-i386/task_size.c b/arch/um/os-Linux/sys-i386/task_size.c
index ccb49b0..be04c1e 100644
--- a/arch/um/os-Linux/sys-i386/task_size.c
+++ b/arch/um/os-Linux/sys-i386/task_size.c
@@ -63,7 +63,7 @@ static int page_ok(unsigned long page)
 	return ok;
 }
 
-unsigned long os_get_task_size(void)
+unsigned long os_get_top_address(void)
 {
 	struct sigaction sa, old;
 	unsigned long bottom = 0;
@@ -76,9 +76,9 @@ unsigned long os_get_task_size(void)
 	 * hosts, but shouldn't hurt otherwise.
 	 */
 	unsigned long top = 0xffffd000 >> UM_KERN_PAGE_SHIFT;
-	unsigned long test;
+	unsigned long test, original;
 
-	printf("Locating the top of the address space ... ");
+	printf("Locating the bottom of the address space ... ");
 	fflush(stdout);
 
 	/*
@@ -89,16 +89,31 @@ unsigned long os_get_task_size(void)
 	sigemptyset(&sa.sa_mask);
 	sa.sa_flags = SA_NODEFER;
 	if (sigaction(SIGSEGV, &sa, &old)) {
-		perror("os_get_task_size");
+		perror("os_get_top_address");
 		exit(1);
 	}
 
-	if (!page_ok(bottom)) {
-		fprintf(stderr, "Address 0x%x no good?\n",
-			bottom << UM_KERN_PAGE_SHIFT);
+	/* Manually scan the address space, bottom-up, until we find
+	 * the first valid page (or run out of them).
+	 */
+	for (bottom = 0; bottom < top; bottom++) {
+		if (page_ok(bottom))
+			break;
+	}
+
+	/* If we've got this far, we ran out of pages. */
+	if (bottom == top) {
+		fprintf(stderr, "Unable to determine bottom of address "
+			"space.\n");
 		exit(1);
 	}
 
+	printf("0x%x\n", bottom << UM_KERN_PAGE_SHIFT);
+	printf("Locating the top of the address space ... ");
+	fflush(stdout);
+
+	original = bottom;
+
 	/* This could happen with a 4G/4G split */
 	if (page_ok(top))
 		goto out;
@@ -114,7 +129,7 @@ unsigned long os_get_task_size(void)
 out:
 	/* Restore the old SIGSEGV handling */
 	if (sigaction(SIGSEGV, &old, NULL)) {
-		perror("os_get_task_size");
+		perror("os_get_top_address");
 		exit(1);
 	}
 	top <<= UM_KERN_PAGE_SHIFT;
diff --git a/arch/um/os-Linux/sys-x86_64/task_size.c b/arch/um/os-Linux/sys-x86_64/task_size.c
index fad6f57..26a0dd1 100644
--- a/arch/um/os-Linux/sys-x86_64/task_size.c
+++ b/arch/um/os-Linux/sys-x86_64/task_size.c
@@ -1,4 +1,4 @@
-unsigned long os_get_task_size(unsigned long shift)
+unsigned long os_get_top_address(unsigned long shift)
 {
 	/* The old value of CONFIG_TOP_ADDR */
 	return 0x7fc0000000;

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] FP registers corruption

[Stanislav Meduna]

Stanislav Meduna wrote:

>>> Same here with 2.6.26.2 :(
>>
>> Hmmm.  If it's not a problem there, I'm tempted to call it fixed,
>> whatever it was.  Can you check current 2.6.27-rc?
> 
> OK, I'll try to convince my provider to test 2.6.26.2 and
> possibly 27-rc soon - I'll let you know of the result.

Sorry for the late response. My provider compiled 2.6.26.3
and the problem is still here :(

I'll try to get the host config from him and to reproduce
it in a clean environment with the similar config as he has
(2.6.23.17 + skas3, i tried with current and without skas).
Is the skas3 patch still maintained?

He doesn't want to switch to SKAS4 yet, as it is marked
experimental and requires building from the same tree,
which he does not want.

-- 
                             Stano

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] FP registers corruption

[Stanislav Meduna]

[-- Attachment #1: Type: text/plain, Size: 1263 bytes --]

Stanislav Meduna wrote:

> I'll try to get the host config from him and to reproduce
> it in a clean environment with the similar config as he has
> (2.6.23.17 + skas3, i tried with current and without skas).
> Is the skas3 patch still maintained?

An update: I can definitely reproduce the problem using

- 2.6.23.17

- http://user-mode-linux.sourceforge.net/skas3-2.6.23.patch

- a patch from Marek Zelem that adds files missing
  from your patch (see attach)


Now we don't know whether the files added could cause
the problem - they were taken from a version a few weeks
older. There was following conversation between you
and Marek in June:

> > On Sun, Jun 22, 2008 at 03:37:12PM +0200, Marek Zelem wrote:
> >  I just want to inform you that skas3 patch available at
> > http://user-mode-linux.sourceforge.net/skas3-2.6.23.patch
> > does not contain new files (e.g. proc_mm.c, proc_mm.h, ...).

> Oops, thanks for letting me know.  I'm testing a fixed patch
> and will update the site when that's done.

Do you have the current version that applies cleanly
to the 2.6.23.17? I'd like to test with it before
declaring it as a definitive bug, it is well possible
that there was some hidden incompatibility.

Regards
-- 
                                 Stano

[-- Attachment #2: skas3-2.6.23.patch.new_files.gz --]
[-- Type: application/x-gzip, Size: 4249 bytes --]

[-- Attachment #3: Type: text/plain, Size: 363 bytes --]

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

[-- Attachment #4: Type: text/plain, Size: 194 bytes --]

_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] FP registers corruption

[Stanislav Meduna]

Stanislav Meduna wrote:

>> I'll try to get the host config from him and to reproduce
>> it in a clean environment with the similar config as he has
>> (2.6.23.17 + skas3, i tried with current and without skas).
>> Is the skas3 patch still maintained?
> 
> An update: I can definitely reproduce the problem using
> 
> - 2.6.23.17
> 
> - http://user-mode-linux.sourceforge.net/skas3-2.6.23.patch
> 
> - a patch from Marek Zelem that adds files missing
>   from your patch

Another piece of information: the FP registers only get corrupted
for skas3; running the guest with mode=skas0 or host with CONFIG_PROC_MM
disabled works (or at least did not fail in a few minutes - the skas3
fails in seconds).

Regards
-- 
                                   Stano


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel