[uml-devel] Problem with newer (>2.6.14) UML kernels hanging under Xen DomU/Centos 4.4

[uml-devel] Problem with newer (>2.6.14) UML kernels hanging under Xen DomU/Centos 4.4

[Matt Willsher]

I've been trying to get UML running under a Xen DomU running Centos
4.4, which I understand is a reasonable thing to do.

When I started out I was using a download 2.6.12 kernel and the image
booted up fine. I then tried 2.6.18.2, which is hanging after VFS:
Mounted root (ext3 filesystem) readonly. I've had someone check my
kernel and the image on another machine and it works without problem.
I've tried some other kernels, and 2.6.14 seemed ok, 2.6.16 wasn't.

I've had some problems with TLS under Xen running Berkely DB, so have
rebuilt glibc with native threading disabled, but /lib/tls is still
there because Bdb complains if it's removed.

Below I've give the process list at the time, backtraces of each of
these processes and the output of the ./linux ubd0=root_fs_router
command.  Let me know if more output is required

Any help is very much appreciated.

Thanks,
Matt

The UML processes that are running at this time are:

matt     15425 15415  0 11:22 pts/1    00:00:00 ./linux
ubd0=/home/uml/router/root_fs_router
matt     15430 15415  0 11:22 pts/1    00:00:00 ./linux
ubd0=/home/uml/router/root_fs_router
matt     15431 15415 43 11:22 pts/1    00:07:04 [linux]
matt     15415 17607 50 11:22 pts/1    00:08:09 ./linux
ubd0=/home/uml/router/root_fs_router
matt     15424 15415  0 11:22 pts/1    00:00:00 ./linux
ubd0=/home/uml/router/root_fs_router

15431 and 15415 are taking 100% CPU between then.

backtraces of those PIDs result in:

PID 15425:
#0  0xb7eaddbe in __read_nocancel () from /lib/tls/i686/libc.so.6
#1  0x08070510 in file_io () at include/linux/sched.h:1439
#2  0x0806d5b7 in io_thread (arg=0x0) at arch/um/drivers/ubd_kern.c:1374
#3  0xb7ebbcbe in clone () from /lib/tls/i686/libc.so.6


PID 15430:
#0  0xb7eb2e3a in poll () from /lib/tls/i686/libc.so.6
#1  0x08072b55 in write_sigio_thread () at include/linux/sched.h:1439
#2  0xb7ebbcbe in clone () from /lib/tls/i686/libc.so.6

PID 15431:
Operation not permitted at attach

PID 15415:
#0  0xb7eb5bff in ptrace (request=3086049268) at
../sysdeps/unix/sysv/linux/ptrace.c:104
#1  0x08077352 in userspace () at include/linux/sched.h:1439
#2  0x08061c79 in new_thread_handler (sig=10) at include/asm/thread_info.h:46
#3  <signal handler called>
#4  0xb7e29041 in kill () at ../string/bits/string2.h:1000
#5  0x0828b580 in init_thread_union ()
#6  0x080735aa in set_signals () at include/linux/sched.h:1439
#7  0x08077b78 in new_thread () at include/linux/sched.h:1439
#8  0x00000000 in ?? ()

PID 15424:
#0  0xb7eaddbe in __read_nocancel () from /lib/tls/i686/libc.so.6
#1  0x08070510 in file_io () at include/linux/sched.h:1439
#2  0x0806f95e in not_aio_thread () at include/linux/sched.h:1439
#3  0xb7ebbcbe in clone () from /lib/tls/i686/libc.so.6


Output from ./linux ubd0=./root_fs_router:

[42949372.960000] Linux version 2.6.18.2 (root@localhost)
(gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)) #4 Sat Nov 18 11:13:34 GMT 2006
[42949372.960000] Built 1 zonelists.  Total pages: 8192
[42949372.960000] Kernel command line:
ubd0=/home/uml/router/root_fs_router root=98:0
[42949372.960000] PID hash table entries: 256 (order: 8, 1024 bytes)
[42949372.960000] ------------------------
[42949372.960000] | Locking API testsuite:
[42949372.960000]
----------------------------------------------------------------------------
[42949372.960000]                                  | spin |wlock
|rlock |mutex | wsem | rsem |
[42949372.960000]
--------------------------------------------------------------------------
[42949372.960000]                      A-A deadlock:failed|failed|  ok
 |failed|failed|failed|
[42949372.960000]                  A-B-B-A deadlock:failed|failed|  ok
 |failed|failed|failed|
[42949372.960000]              A-B-B-C-C-A deadlock:failed|failed|  ok
 |failed|failed|failed|
[42949372.960000]              A-B-C-A-B-C deadlock:failed|failed|  ok
 |failed|failed|failed|
[42949372.960000]          A-B-B-C-C-D-D-A deadlock:failed|failed|  ok
 |failed|failed|failed|
[42949372.960000]          A-B-C-D-B-D-D-A deadlock:failed|failed|  ok
 |failed|failed|failed|
[42949372.960000]          A-B-C-D-B-C-D-A deadlock:failed|failed|  ok
 |failed|failed|failed|
[42949372.960000]                     double unlock:  ok  |  ok
|failed|  ok  |failed|failed|
[42949372.960000]                   initialize
held:failed|failed|failed|failed|failed|failed|
[42949372.960000]                  bad unlock order:  ok  |  ok  |  ok
 |  ok  |  ok  |  ok  |
[42949372.960000]
--------------------------------------------------------------------------
[42949372.960000]               recursive read-lock:             |  ok
 |             |failed|
[42949372.960000]            recursive read-lock #2:             |  ok
 |             |failed|
[42949372.960000]             mixed read-write-lock:
|failed|             |failed|
[42949372.960000]             mixed write-read-lock:
|failed|             |failed|
[42949372.960000]
--------------------------------------------------------------------------
[42949372.960000]      hard-irqs-on + irq-safe-A/12:failed|failed|  ok  |
[42949372.960000]      soft-irqs-on + irq-safe-A/12:failed|failed|  ok  |
[42949372.960000]      hard-irqs-on + irq-safe-A/21:failed|failed|  ok  |
[42949372.960000]      soft-irqs-on + irq-safe-A/21:failed|failed|  ok  |
[42949372.960000]        sirq-safe-A => hirqs-on/12:failed|failed|  ok  |
[42949372.960000]        sirq-safe-A => hirqs-on/21:failed|failed|  ok  |
[42949372.960000]          hard-safe-A + irqs-on/12:failed|failed|  ok  |
[42949372.960000]          soft-safe-A + irqs-on/12:failed|failed|  ok  |
[42949372.960000]          hard-safe-A + irqs-on/21:failed|failed|  ok  |
[42949372.960000]          soft-safe-A + irqs-on/21:failed|failed|  ok  |
[42949372.960000]     hard-safe-A + unsafe-B #1/123:failed|failed|  ok  |
[42949372.960000]     soft-safe-A + unsafe-B #1/123:failed|failed|  ok  |
[42949372.960000]     hard-safe-A + unsafe-B #1/132:failed|failed|  ok  |
[42949372.960000]     soft-safe-A + unsafe-B #1/132:failed|failed|  ok  |
[42949372.960000]     hard-safe-A + unsafe-B #1/213:failed|failed|  ok  |
[42949372.960000]     soft-safe-A + unsafe-B #1/213:failed|failed|  ok  |
[42949372.960000]     hard-safe-A + unsafe-B #1/231:failed|failed|  ok  |
[42949372.960000]     soft-safe-A + unsafe-B #1/231:failed|failed|  ok  |
[42949372.960000]     hard-safe-A + unsafe-B #1/312:failed|failed|  ok  |
[42949372.960000]     soft-safe-A + unsafe-B #1/312:failed|failed|  ok  |
[42949372.960000]     hard-safe-A + unsafe-B #1/321:failed|failed|  ok  |
[42949372.960000]     soft-safe-A + unsafe-B #1/321:failed|failed|  ok  |
[42949372.960000]     hard-safe-A + unsafe-B #2/123:failed|failed|  ok  |
[42949372.960000]     soft-safe-A + unsafe-B #2/123:failed|failed|  ok  |
[42949372.960000]     hard-safe-A + unsafe-B #2/132:failed|failed|  ok  |
[42949372.960000]     soft-safe-A + unsafe-B #2/132:failed|failed|  ok  |
[42949372.960000]     hard-safe-A + unsafe-B #2/213:failed|failed|  ok  |
[42949372.960000]     soft-safe-A + unsafe-B #2/213:failed|failed|  ok  |
[42949372.960000]     hard-safe-A + unsafe-B #2/231:failed|failed|  ok  |
[42949372.960000]     soft-safe-A + unsafe-B #2/231:failed|failed|  ok  |
[42949372.960000]     hard-safe-A + unsafe-B #2/312:failed|failed|  ok  |
[42949372.960000]     soft-safe-A + unsafe-B #2/312:failed|failed|  ok  |
[42949372.960000]     hard-safe-A + unsafe-B #2/321:failed|failed|  ok  |
[42949372.960000]     soft-safe-A + unsafe-B #2/321:failed|failed|  ok  |
[42949372.960000]       hard-irq lock-inversion/123:failed|failed|  ok  |
[42949372.960000]       soft-irq lock-inversion/123:failed|failed|  ok  |
[42949372.960000]       hard-irq lock-inversion/132:failed|failed|  ok  |
[42949372.960000]       soft-irq lock-inversion/132:failed|failed|  ok  |
[42949372.960000]       hard-irq lock-inversion/213:failed|failed|  ok  |
[42949372.960000]       soft-irq lock-inversion/213:failed|failed|  ok  |
[42949372.960000]       hard-irq lock-inversion/231:failed|failed|  ok  |
[42949372.960000]       soft-irq lock-inversion/231:failed|failed|  ok  |
[42949372.960000]       hard-irq lock-inversion/312:failed|failed|  ok  |
[42949372.960000]       soft-irq lock-inversion/312:failed|failed|  ok  |
[42949372.960000]       hard-irq lock-inversion/321:failed|failed|  ok  |
[42949372.960000]       soft-irq lock-inversion/321:failed|failed|  ok  |
[42949372.960000]       hard-irq read-recursion/123:  ok  |
[42949372.960000]       soft-irq read-recursion/123:  ok  |
[42949372.960000]       hard-irq read-recursion/132:  ok  |
[42949372.960000]       soft-irq read-recursion/132:  ok  |
[42949372.960000]       hard-irq read-recursion/213:  ok  |
[42949372.960000]       soft-irq read-recursion/213:  ok  |
[42949372.960000]       hard-irq read-recursion/231:  ok  |
[42949372.960000]       soft-irq read-recursion/231:  ok  |
[42949372.960000]       hard-irq read-recursion/312:  ok  |
[42949372.960000]       soft-irq read-recursion/312:  ok  |
[42949372.960000]       hard-irq read-recursion/321:  ok  |
[42949372.960000]       soft-irq read-recursion/321:  ok  |
[42949372.960000] --------------------------------------------------------
[42949372.960000] 142 out of 218 testcases failed, as expected. |
[42949372.960000] ----------------------------------------------------
[42949372.960000] Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
[42949372.960000] Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
[42949372.960000] Memory: 29408k available
[42949373.480000] Mount-cache hash table entries: 512
[42949373.480000] Checking for host processor cmov support...Yes
[42949373.480000] Checking for host processor xmm support...No
[42949373.480000] Checking that host ptys support output SIGIO...Yes
[42949373.480000] Checking that host ptys support SIGIO on close...No,
enabling workaround
[42949373.480000] /usr/include/linux/aio_abi.h not present during build
[42949373.480000] 2.6 host AIO support not used - falling back to I/O thread
[42949373.480000] NET: Registered protocol family 16
[42949373.480000] NET: Registered protocol family 2
[42949373.660000] IP route cache hash table entries: 256 (order: -2, 1024 bytes)
[42949373.660000] TCP established hash table entries: 1024 (order: 2,
16384 bytes)
[42949373.660000] TCP bind hash table entries: 512 (order: 1, 10240 bytes)
[42949373.660000] TCP: Hash tables configured (established 1024 bind 512)
[42949373.660000] TCP reno registered
[42949373.660000] Checking host MADV_REMOVE support...OK
[42949373.660000] mconsole (version 2) initialized on
/home/matt/.uml/i5b2KQ/mconsole
[42949373.660000] Host TLS support detected
[42949373.660000] Detected host type: i386
[42949373.660000] VFS: Disk quotas dquot_6.5.1
[42949373.660000] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[42949373.660000] Initializing Cryptographic API
[42949373.660000] io scheduler noop registered
[42949373.660000] io scheduler anticipatory registered (default)
[42949373.660000] io scheduler deadline registered
[42949373.660000] io scheduler cfq registered
[42949373.660000] loop: loaded (max 8 devices)
[42949373.660000] nbd: registered device at major 43
[42949373.660000] PPP generic driver version 2.4.2
[42949373.660000] SLIP: version 0.8.4-NET3.019-NEWTTY (dynamic
channels, max=256).
[42949373.660000] tun: Universal TUN/TAP device driver, 1.6
[42949373.660000] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[42949373.660000] ip_conntrack version 2.4 (229 buckets, 1832 max) -
228 bytes per conntrack
[42949374.160000] NET: Registered protocol family 1
[42949374.160000] NET: Registered protocol family 17
[42949375.000000] Bridge firewalling registered
[42949375.000000] 802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
[42949375.000000] All bugs added by David S. Miller <davem@redhat.com>
[42949375.000000] Initialized stdio console driver
[42949375.000000] Console initialized on /dev/tty0
[42949375.000000] Initializing software serial port version 1
[42949375.000000]  ubda: unknown partition table
[42949375.000000] kjournald starting.  Commit interval 5 seconds
[42949375.000000] EXT3-fs: mounted filesystem with ordered data mode.
[42949375.000000] VFS: Mounted root (ext3 filesystem) readonly.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Problem with newer (>2.6.14) UML kernels hanging under Xen DomU/Centos 4.4

[Blaisorblade]

On Saturday 18 November 2006 12:45, Matt Willsher wrote:
> I've been trying to get UML running under a Xen DomU running Centos
> 4.4, which I understand is a reasonable thing to do.
>
> When I started out I was using a download 2.6.12 kernel and the image
> booted up fine. I then tried 2.6.18.2, which is hanging after VFS:
> Mounted root (ext3 filesystem) readonly. I've had someone check my
> kernel and the image on another machine and it works without problem.
> I've tried some other kernels, and 2.6.14 seemed ok, 2.6.16 wasn't.
>
> I've had some problems with TLS under Xen running Berkely DB, so have
> rebuilt glibc with native threading disabled, but /lib/tls is still
> there because Bdb complains if it's removed.
>
> Below I've give the process list at the time, backtraces of each of
> these processes and the output of the ./linux ubd0=root_fs_router
> command.  Let me know if more output is required
>
> Any help is very much appreciated.

I have two suggestions for you to try:
* reduce CONFIG_HOST_VM_SPLIT for UML guest kernels (Xen consumes part of the 
available memory for UML, which causes crashes)

* try if booting with mode=tt helps - around 2.6.14 or 2.6.15 skas0 support 
was introduced (a very big change!) and become the new default, so that's the 
1st likely culprit.

-- 
Inform me of my mistakes, so I can add them to my list!
Paolo Giarrusso, aka Blaisorblade
http://www.user-mode-linux.org/~blaisorblade
Chiacchiera con i tuoi amici in tempo reale! 
 http://it.yahoo.com/mail_it/foot/*http://it.messenger.yahoo.com 


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Problem with newer (>2.6.14) UML kernels hanging under Xen DomU/Centos 4.4

[Matt Willsher]

On 11/21/06, Blaisorblade <blaisorblade@yahoo.it> wrote:
> On Saturday 18 November 2006 12:45, Matt Willsher wrote:
> > I've been trying to get UML running under a Xen DomU running Centos
> > 4.4, which I understand is a reasonable thing to do.
> >
> > When I started out I was using a download 2.6.12 kernel and the image
> > booted up fine. I then tried 2.6.18.2, which is hanging after VFS:
> > Mounted root (ext3 filesystem) readonly. I've had someone check my
> > kernel and the image on another machine and it works without problem.
> > I've tried some other kernels, and 2.6.14 seemed ok, 2.6.16 wasn't.

> I have two suggestions for you to try:
> * reduce CONFIG_HOST_VM_SPLIT for UML guest kernels (Xen consumes part of the
> available memory for UML, which causes crashes)
>
> * try if booting with mode=tt helps - around 2.6.14 or 2.6.15 skas0 support
> was introduced (a very big change!) and become the new default, so that's the
> 1st likely culprit.

Thank you for your suggestions.

I must correct myself. 2.6.14 does indeed hang in the same way as
later kernels. I think I tried 2.6.13 resulting in the same hang but I
can't be sure (I've tried rather a lot of kernels this last week!)

I've applied your latest patch bundle to the 2.6.18.3 tree, and set
the VM_HOST split to 1/3. Booting with this, the guest kernel hangs at
the same point. With mode=tt it crashes outright. I've posted the
output from the run below. I've also got an strace output, which I can
post if you want, but thought I'd check first as it's 380K.

The same command line used on the below works with a 2.6.12.2-bb11
kernel. I've also tried init=/bin/sh, again with the same results as
below.

Checking that ptrace can change system call numbers...OK
Checking syscall emulation patch for ptrace...OK
Checking advanced syscall emulation patch for ptrace...OK
Checking for tmpfs mount on /dev/shm...OK
Checking PROT_EXEC mmap in /dev/shm/...OK
UML running in TT mode
tracing thread pid = 1350
Checking that ptrace can change system call numbers...OK
Checking syscall emulation patch for ptrace...OK
Checking advanced syscall emulation patch for ptrace...OK
Linux version 2.6.18.3-bb2 (root@localhost) (gcc version 3.4.6
20060404 (Red Hat 3.4.6-3)) #3 Tue Nov 21 19:06:28 GMT 2006
Built 1 zonelists.  Total pages: 16384
Kernel command line: ubd0=/home/uml/images/ano_router_root_fs
ubd1=/home/uml/images/ano_router_swap_fs mode=tt mem=64M con=null
con0=fd:0,fd:1 con1=port:9000 eth0=tuntap,umltun0 eth1=tuntap,umltun1
root=98:0
PID hash table entries: 512 (order: 9, 2048 bytes)
Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
Memory: 61164k available
Mount-cache hash table entries: 512
Checking for host processor cmov support...Yes
Checking for host processor xmm support...No
Checking that host ptys support output SIGIO...Yes
Checking that host ptys support SIGIO on close...No, enabling workaround
/usr/include/linux/aio_abi.h not present during build
2.6 host AIO support not used - falling back to I/O thread
Failed to initialize umid "anoroute", trying with a random umid
make_umid - mkstemp(/home/uml/.uml/MyLtrF) failed: Permission denied
NET: Registered protocol family 16
NET: Registered protocol family 2
IP route cache hash table entries: 512 (order: -1, 2048 bytes)
TCP established hash table entries: 2048 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 2048 bind 1024)
TCP reno registered
Checking host MADV_REMOVE support...OK
make_umid - mkstemp(/home/uml/.uml/i5cVhQ) failed: Permission denied
ubd: Synchronous mode
Netdevice 0 : TUN/TAP backend -
Netdevice 1 : TUN/TAP backend -
Host TLS support detected
Detected host type: i386
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
NET: Registered protocol family 1
NET: Registered protocol family 17
Initialized stdio console driver
Console initialized on /dev/tty0
Initializing software serial port version 1
 ubda: unknown partition table
 ubdb: unknown partition table
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Kernel panic - not syncing: No init found.  Try passing init= option to kernel.

EIP: 0073:[<2017abc1>] CPU: 0 Not tainted ESP: 007b:203357d4 EFLAGS: 00000216
    Not tainted
EAX: 00000000 EBX: 00000566 ECX: 0000000a EDX: 20014df2
ESI: 00000000 EDI: 00000000 EBP: 00000000 DS: 007b ES: 007b
20333bdc:  [<20048c2c>] notifier_call_chain+0x28/0x3e
20333bf0:  [<2000e193>] init+0x0/0x175
20333bf8:  [<2003643b>] panic+0x70/0xff
20333c10:  [<2000e2a0>] init+0x10d/0x175
20333c2c:  [<20026c66>] run_kernel_thread+0x52/0x5c
20333c3c:  [<2000e193>] init+0x0/0x175
20333c4c:  [<20026c39>] run_kernel_thread+0x25/0x5c
20333cd4:  [<2000e193>] init+0x0/0x175
20333cdc:  [<20014dc1>] new_thread_handler+0x13f/0x170
20333ce0:  [<2000e193>] init+0x0/0x175
20333d48:  [<20014df2>] new_thread_proc+0x0/0x43
20333d5c:  [<2017abc1>] __kill+0x11/0x20

 remove_umid_dir - remove_files_and_dir failed with err = -13

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Problem with newer (>2.6.14) UML kernels hanging under Xen DomU/Centos 4.4

[Blaisorblade]

On Tuesday 21 November 2006 20:23, Matt Willsher wrote:
> On 11/21/06, Blaisorblade <blaisorblade@yahoo.it> wrote:
> > On Saturday 18 November 2006 12:45, Matt Willsher wrote:
> > > I've been trying to get UML running under a Xen DomU running Centos
> > > 4.4, which I understand is a reasonable thing to do.
> > >
> > > When I started out I was using a download 2.6.12 kernel and the image
> > > booted up fine. I then tried 2.6.18.2, which is hanging after VFS:
> > > Mounted root (ext3 filesystem) readonly. I've had someone check my
> > > kernel and the image on another machine and it works without problem.
> > > I've tried some other kernels, and 2.6.14 seemed ok, 2.6.16 wasn't.
> >
> > I have two suggestions for you to try:
> > * reduce CONFIG_HOST_VM_SPLIT for UML guest kernels (Xen consumes part of
> > the available memory for UML, which causes crashes)
> >
> > * try if booting with mode=tt helps - around 2.6.14 or 2.6.15 skas0
> > support was introduced (a very big change!) and become the new default,
> > so that's the 1st likely culprit.
>
> Thank you for your suggestions.
>
> I must correct myself. 2.6.14 does indeed hang in the same way as
> later kernels. I think I tried 2.6.13 resulting in the same hang but I
> can't be sure (I've tried rather a lot of kernels this last week!)
>
> I've applied your latest patch bundle to the 2.6.18.3 tree, and set
> the VM_HOST split to 1/3. Booting with this, the guest kernel hangs at
> the same point. With mode=tt it crashes outright.

mode=tt can possibly work only on _old_ kernel, say the first one where skas0 
was introduced i.e. (if my guess is right) the first one breaking under Xen.

> I've posted the 
> output from the run below. I've also got an strace output, which I can
> post if you want, but thought I'd check first as it's 380K.

The output makes me suspicious of some misconfiguration - I've never seen a 
bug giving a misleading "no init found". And it happens that even smart users 
use different command lines, say, because they use a script for the default 
kernel and do not use it for the other one.

/home/uml/.uml ($HOME/.uml) seems not writable according to messages below, 
which is _very_ strange. If the error message is wrong it should be checked 
in strace output, looking for EPERM or EACCESS on access to files mentioned 
below I guess (but I'll not have the time to even give a look to it, maybe 
someone else will be able or you can give a look yourself).

> The same command line used on the below works with a 2.6.12.2-bb11
> kernel. I've also tried init=/bin/sh, again with the same results as
> below.
>
> Checking that ptrace can change system call numbers...OK
> Checking syscall emulation patch for ptrace...OK
> Checking advanced syscall emulation patch for ptrace...OK
> Checking for tmpfs mount on /dev/shm...OK
> Checking PROT_EXEC mmap in /dev/shm/...OK
> UML running in TT mode
> tracing thread pid = 1350
> Checking that ptrace can change system call numbers...OK
> Checking syscall emulation patch for ptrace...OK
> Checking advanced syscall emulation patch for ptrace...OK
> Linux version 2.6.18.3-bb2 (root@localhost) (gcc version 3.4.6
> 20060404 (Red Hat 3.4.6-3)) #3 Tue Nov 21 19:06:28 GMT 2006
> Built 1 zonelists.  Total pages: 16384
> Kernel command line: ubd0=/home/uml/images/ano_router_root_fs
> ubd1=/home/uml/images/ano_router_swap_fs mode=tt mem=64M con=null
> con0=fd:0,fd:1 con1=port:9000 eth0=tuntap,umltun0 eth1=tuntap,umltun1
> root=98:0
> PID hash table entries: 512 (order: 9, 2048 bytes)
> Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
> Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
> Memory: 61164k available
> Mount-cache hash table entries: 512
> Checking for host processor cmov support...Yes
> Checking for host processor xmm support...No
> Checking that host ptys support output SIGIO...Yes
> Checking that host ptys support SIGIO on close...No, enabling workaround
> /usr/include/linux/aio_abi.h not present during build
> 2.6 host AIO support not used - falling back to I/O thread
> Failed to initialize umid "anoroute", trying with a random umid
> make_umid - mkstemp(/home/uml/.uml/MyLtrF) failed: Permission denied
> NET: Registered protocol family 16
> NET: Registered protocol family 2
> IP route cache hash table entries: 512 (order: -1, 2048 bytes)
> TCP established hash table entries: 2048 (order: 1, 8192 bytes)
> TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
> TCP: Hash tables configured (established 2048 bind 1024)
> TCP reno registered
> Checking host MADV_REMOVE support...OK
> make_umid - mkstemp(/home/uml/.uml/i5cVhQ) failed: Permission denied
> ubd: Synchronous mode
> Netdevice 0 : TUN/TAP backend -
> Netdevice 1 : TUN/TAP backend -
> Host TLS support detected
> Detected host type: i386
> Initializing Cryptographic API
> io scheduler noop registered
> io scheduler anticipatory registered
> io scheduler deadline registered
> io scheduler cfq registered (default)
> NET: Registered protocol family 1
> NET: Registered protocol family 17
> Initialized stdio console driver
> Console initialized on /dev/tty0
> Initializing software serial port version 1
>  ubda: unknown partition table
>  ubdb: unknown partition table
> kjournald starting.  Commit interval 5 seconds
> EXT3-fs: mounted filesystem with ordered data mode.
> VFS: Mounted root (ext3 filesystem) readonly.
> Kernel panic - not syncing: No init found.  Try passing init= option to
> kernel.

-- 
Inform me of my mistakes, so I can add them to my list!
Paolo Giarrusso, aka Blaisorblade
http://www.user-mode-linux.org/~blaisorblade
Chiacchiera con i tuoi amici in tempo reale! 
 http://it.yahoo.com/mail_it/foot/*http://it.messenger.yahoo.com 


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Problem with newer (>2.6.14) UML kernels hanging under Xen DomU/Centos 4.4

[Matt Willsher]

On 11/22/06, Blaisorblade <blaisorblade@yahoo.it> wrote:
> On Tuesday 21 November 2006 20:23, Matt Willsher wrote:
> > On 11/21/06, Blaisorblade <blaisorblade@yahoo.it> wrote:
> >
> > I've applied your latest patch bundle to the 2.6.18.3 tree, and set
> > the VM_HOST split to 1/3. Booting with this, the guest kernel hangs at
> > the same point. With mode=tt it crashes outright.
>
> mode=tt can possibly work only on _old_ kernel, say the first one where skas0
> was introduced i.e. (if my guess is right) the first one breaking under Xen.

From the output below is the line:

UML running in TT mode

Does that not indicate that it is running in TT mode? The kernel
option for TT mode is set. I tried to disable SKAS support but the
kernel won't compile with out that set (check_host_supports_tls not
found, as reported on the user mailing list last month). I will
however try with 2.6.13 and 2.6.14.

> The output makes me suspicious of some misconfiguration - I've never seen a
> bug giving a misleading "no init found". And it happens that even smart users
> use different command lines, say, because they use a script for the default
> kernel and do not use it for the other one.

The same script was used for both. I copied my start script and simply
changed the kernel file being called and added the mode=tt option. I
was particular about this. I also used the same base .config file for
the 2.6.12 kernel and the 2.6.18 kernel, running menuconfig to
sanitise it, and only adjusted the options you've recommended.

> /home/uml/.uml ($HOME/.uml) seems not writable according to messages below,
> which is _very_ strange.

That was my fault. I accidentally ran the binary as root before
running it as the unpriviledged user. I fixed the permissions and the
error went away but I still get the same crash.

As far as further debugging from me, I'm at the limits of my abilities
in that area. I'm happy do what every anyone requests of me if I'm
given a few pointers in the right direction. I'm keen to see this
issue resolved.

Thanks again for you time on this,

Matt

> > The same command line used on the below works with a 2.6.12.2-bb11
> > kernel. I've also tried init=/bin/sh, again with the same results as
> > below.
> >
> > Checking that ptrace can change system call numbers...OK
> > Checking syscall emulation patch for ptrace...OK
> > Checking advanced syscall emulation patch for ptrace...OK
> > Checking for tmpfs mount on /dev/shm...OK
> > Checking PROT_EXEC mmap in /dev/shm/...OK
> > UML running in TT mode
> > tracing thread pid = 1350
> > Checking that ptrace can change system call numbers...OK
> > Checking syscall emulation patch for ptrace...OK
> > Checking advanced syscall emulation patch for ptrace...OK
> > Linux version 2.6.18.3-bb2 (root@localhost) (gcc version 3.4.6
> > 20060404 (Red Hat 3.4.6-3)) #3 Tue Nov 21 19:06:28 GMT 2006
> > Built 1 zonelists.  Total pages: 16384
> > Kernel command line: ubd0=/home/uml/images/ano_router_root_fs
> > ubd1=/home/uml/images/ano_router_swap_fs mode=tt mem=64M con=null
> > con0=fd:0,fd:1 con1=port:9000 eth0=tuntap,umltun0 eth1=tuntap,umltun1
> > root=98:0
> > PID hash table entries: 512 (order: 9, 2048 bytes)
> > Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
> > Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
> > Memory: 61164k available
> > Mount-cache hash table entries: 512
> > Checking for host processor cmov support...Yes
> > Checking for host processor xmm support...No
> > Checking that host ptys support output SIGIO...Yes
> > Checking that host ptys support SIGIO on close...No, enabling workaround
> > /usr/include/linux/aio_abi.h not present during build
> > 2.6 host AIO support not used - falling back to I/O thread
> > Failed to initialize umid "anoroute", trying with a random umid
> > make_umid - mkstemp(/home/uml/.uml/MyLtrF) failed: Permission denied
> > NET: Registered protocol family 16
> > NET: Registered protocol family 2
> > IP route cache hash table entries: 512 (order: -1, 2048 bytes)
> > TCP established hash table entries: 2048 (order: 1, 8192 bytes)
> > TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
> > TCP: Hash tables configured (established 2048 bind 1024)
> > TCP reno registered
> > Checking host MADV_REMOVE support...OK
> > make_umid - mkstemp(/home/uml/.uml/i5cVhQ) failed: Permission denied
> > ubd: Synchronous mode
> > Netdevice 0 : TUN/TAP backend -
> > Netdevice 1 : TUN/TAP backend -
> > Host TLS support detected
> > Detected host type: i386
> > Initializing Cryptographic API
> > io scheduler noop registered
> > io scheduler anticipatory registered
> > io scheduler deadline registered
> > io scheduler cfq registered (default)
> > NET: Registered protocol family 1
> > NET: Registered protocol family 17
> > Initialized stdio console driver
> > Console initialized on /dev/tty0
> > Initializing software serial port version 1
> >  ubda: unknown partition table
> >  ubdb: unknown partition table
> > kjournald starting.  Commit interval 5 seconds
> > EXT3-fs: mounted filesystem with ordered data mode.
> > VFS: Mounted root (ext3 filesystem) readonly.
> > Kernel panic - not syncing: No init found.  Try passing init= option to
> > kernel.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Problem with newer (>2.6.14) UML kernels hanging under Xen DomU/Centos 4.4

[Jeff Dike]

On Wed, Nov 22, 2006 at 08:04:59AM +0000, Matt Willsher wrote:
> UML running in TT mode
> 
> Does that not indicate that it is running in TT mode? The kernel
> option for TT mode is set. I tried to disable SKAS support but the
> kernel won't compile with out that set (check_host_supports_tls not
> found, as reported on the user mailing list last month). I will
> however try with 2.6.13 and 2.6.14.

Disable CONFIG_MODE_TT.  Its presence in the binary, even if it's not running,
has caused boot failures.

				Jeff

-- 
Work email - jdike at linux dot intel dot com

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Problem with newer (>2.6.14) UML kernels hanging under Xen DomU/Centos 4.4

[Jeff Dike]

On Wed, Nov 22, 2006 at 08:04:59AM +0000, Matt Willsher wrote:
> UML running in TT mode

Sorry about that last suggestion - I missed that you were trying an
old UML.

I have to repeat what BB said about the "init not found" error.  I
never seen that not be a filesystem problem.  Can you try booting that
filesystem in an environment which is known to work, to make sure that
it is sane?

				Jeff
-- 
Work email - jdike at linux dot intel dot com

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

[uml-devel] Fwd: Re: Re: Problem with newer (>2.6.14) UML kernels hanging under Xen DomU/Centos 4.4

[Matt Willsher]

On 11/22/06, Jeff Dike <jdike@addtoit.com> wrote:
> On Wed, Nov 22, 2006 at 08:04:59AM +0000, Matt Willsher wrote:
> > UML running in TT mode
>
> Sorry about that last suggestion - I missed that you were trying an
> old UML.

The TT mode on my last mail was on the latest kernel (2.6.18.3) with
the BB patch sets applied. I recompiled with TT disabled and let it
run in SKAS0 mode and it hung as per my initial mail.

> I have to repeat what BB said about the "init not found" error.  I
> never seen that not be a filesystem problem.  Can you try booting that
> filesystem in an environment which is known to work, to make sure that
> it is sane?

This was only in TT mode on the most recent kernel. The image I was
using was one that I am successfully using under 2.6.12.6-bs11.

I've not tried 2.6.13 or 2.6.14 again in TT mode, but I'm not sure
what that would prove? If it would aid in troubleshooting the problem
of running under the latest kernel in SKAS0 mode, give TT mode on the
new kernels is obsolete, I am of course happy to try that.

My initial searching into this problem returned a reference to a VDSO
issue, which is also documented on the new site on the problems page.
I'm not sure if this could be the cause of the problem as the UML I'm
running is under a Xen DomU, and I have no control over this kernel
and /proc/config.gz is not available.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Problem with newer (>2.6.14) UML kernels hanging under Xen DomU/Centos 4.4

[Jeff Dike]

On Fri, Nov 24, 2006 at 09:10:50PM +0000, Matt Willsher wrote:
> The very same image, uncopied. I have a directory tree set up and
> scripts I use and all I did was to change the script to call the newer
> kernel.

OK, a couple experiments to try:
	add "init=/bin/bash" to the command line and see what that does

	find a statically linked shell in the filesystem and use that
as the argument to "init=".

				Jeff
-- 
Work email - jdike at linux dot intel dot com

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Problem with newer (>2.6.14) UML kernels hanging under Xen DomU/Centos 4.4

[Jeff Dike]

On Fri, Nov 24, 2006 at 09:10:50PM +0000, Matt Willsher wrote:
> The very same image, uncopied. I have a directory tree set up and
> scripts I use and all I did was to change the script to call the newer
> kernel.

And the exact same UML binary?  I'm grabbing at straws here, in case
somehow the broken one is lacking binfmt support for /sbin/init.

				Jeff

-- 
Work email - jdike at linux dot intel dot com

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Problem with newer (>2.6.14) UML kernels hanging under Xen DomU/Centos 4.4

[Matt Willsher]

The problem I'm having

On 12/1/06, Jeff Dike <jdike@addtoit.com> wrote:
> On Fri, Nov 24, 2006 at 09:10:50PM +0000, Matt Willsher wrote:
> > The very same image, uncopied. I have a directory tree set up and
> > scripts I use and all I did was to change the script to call the newer
> > kernel.
>
> And the exact same UML binary?  I'm grabbing at straws here, in case
> somehow the broken one is lacking binfmt support for /sbin/init.
>

Just to recap:

TT mode support under 2.6.12 works fine and I can boot a custom build
kernel with it. A downloaded, known good 2.6.12 also boots a know good
linux disk image.

2.6.18, either compiled by myself, or a downloaded known good doesn't
boot either my known good disk image as used with 2.6.12 or one
downloaded from the web.  The 2.6.18 that doesn't work on my machine
has been used successfully on someone elses environment boot my known
good image.

My guess is that this issue is related to SKAS0 support in the newer
kernels. I've been meaning to find the earliest kernel that ships with
SKAS0 and try both TT mode and SKAS0 mode to see what happens there.

http://user-mode-linux.sourceforge.net/new/problems.html#vdso seems to
be the problem I'm having but I'm running UML under Xen, and there is
not /proc/config.gz to check the kernel config, nor do I access to
Dom0. so I have no way to check CONFIG_COMPAT_VDSO or have it changed
on my host.

I'm willing to have a go at debugging using gdb etc. but I really
don't know what I'm looking for there.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel