Xen 3.1 - Can't run Fedora Core 1 PV

Xen 3.1 - Can't run Fedora Core 1 PV

[Nick Craig-Wood]

I can't run a Fedora Core 1 image PV - it SEGVs on start.  This is a
show stopper for me upgrading to xen 3.1 as we have quite a few FC1
images.

I've written a bug report here, but I'm not sure anyone ever looks at
bugzilla!

  http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=993

Here are the details...

Xen 3.1 runs most of my domUs just fine, windows etc, but it doesn't
run a Fedora Core 1 image.  It hangs at 100% cpu usage just after
"Freeing kernel memory".

If I mount the FC1 image on dom0 and chroot into it I get

  # chroot . /bin/bash
  Segmentation fault

I've put a small ext2 filing system here (built from a virgin FC1
image) which demonstrates the problem

  http://www.craig-wood.com/nick/pub/fedora_demo.gz

Mounting it and chrooting from dom0 under Xen 3.1 gives the SEGV.
Doing the same from Xen 3.0.3 , Xen 3.0.4 and Debian testing works
fine.

I've verified that the same thing happens on the binary download from
the xen web site (xen-3.1.0-install-x86_32p.tgz) dated June 1st so it
isn't just my dodgy compiling skills!

-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Keir Fraser]

Is this using an up-to-date kernel image? I.e., we think the problem occurs
when we hit the init task?

 -- Keir

On 12/7/07 11:46, "Nick Craig-Wood" <nick@craig-wood.com> wrote:

> I can't run a Fedora Core 1 image PV - it SEGVs on start.  This is a
> show stopper for me upgrading to xen 3.1 as we have quite a few FC1
> images.
> 
> I've written a bug report here, but I'm not sure anyone ever looks at
> bugzilla!
> 
>   http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=993
> 
> Here are the details...
> 
> Xen 3.1 runs most of my domUs just fine, windows etc, but it doesn't
> run a Fedora Core 1 image.  It hangs at 100% cpu usage just after
> "Freeing kernel memory".
> 
> If I mount the FC1 image on dom0 and chroot into it I get
> 
>   # chroot . /bin/bash
>   Segmentation fault
> 
> I've put a small ext2 filing system here (built from a virgin FC1
> image) which demonstrates the problem
> 
>   http://www.craig-wood.com/nick/pub/fedora_demo.gz
> 
> Mounting it and chrooting from dom0 under Xen 3.1 gives the SEGV.
> Doing the same from Xen 3.0.3 , Xen 3.0.4 and Debian testing works
> fine.
> 
> I've verified that the same thing happens on the binary download from
> the xen web site (xen-3.1.0-install-x86_32p.tgz) dated June 1st so it
> isn't just my dodgy compiling skills!

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Nick Craig-Wood]

On Thu, Jul 12, 2007 at 03:55:55AM -0700, Keir Fraser wrote:
> On 12/7/07 11:46, "Nick Craig-Wood" <nick@craig-wood.com> wrote:
> 
> > I've put a small ext2 filing system here (built from a virgin FC1
> > image) which demonstrates the problem
> > 
> >   http://www.craig-wood.com/nick/pub/fedora_demo.gz
> > 
> > Mounting it and chrooting from dom0 under Xen 3.1 gives the SEGV.
> > Doing the same from Xen 3.0.3 , Xen 3.0.4 and Debian testing works
> > fine.
> 
> What exactly do you mean? I can mount the thing and chroot into it in dom0:
> what do I need to do next?

OK, I get a SEGV at that point using xen-3.1-testing or the xen 3.1
binary release.

Which xen version are you using?

On Thu, Jul 12, 2007 at 03:52:40AM -0700, Keir Fraser wrote:
> Is this using an up-to-date kernel image? I.e., we think the problem occurs
> when we hit the init task?

I think it is using the as released fedora core 1 image.

-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Keir Fraser]

On 12/7/07 12:53, "Nick Craig-Wood" <nick@craig-wood.com> wrote:

>> What exactly do you mean? I can mount the thing and chroot into it in dom0:
>> what do I need to do next?
> 
> OK, I get a SEGV at that point using xen-3.1-testing or the xen 3.1
> binary release.
> 
> Which xen version are you using?

Do you mean you do 'chroot /mnt /sbin/init'?

 -- Keir

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Keir Fraser]

My current guess is that this is a kernel version issue. Have you tried
using your old (pre-3.1) domU kernel with the FC1 image on Xen 3.1?

 -- Keir

On 12/7/07 11:52, "Keir Fraser" <keir@xensource.com> wrote:

> Is this using an up-to-date kernel image? I.e., we think the problem occurs
> when we hit the init task?
> 
>  -- Keir
> 
> On 12/7/07 11:46, "Nick Craig-Wood" <nick@craig-wood.com> wrote:
> 
>> I can't run a Fedora Core 1 image PV - it SEGVs on start.  This is a
>> show stopper for me upgrading to xen 3.1 as we have quite a few FC1
>> images.
>> 
>> I've written a bug report here, but I'm not sure anyone ever looks at
>> bugzilla!
>> 
>>   http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=993
>> 
>> Here are the details...
>> 
>> Xen 3.1 runs most of my domUs just fine, windows etc, but it doesn't
>> run a Fedora Core 1 image.  It hangs at 100% cpu usage just after
>> "Freeing kernel memory".
>> 
>> If I mount the FC1 image on dom0 and chroot into it I get
>> 
>>   # chroot . /bin/bash
>>   Segmentation fault
>> 
>> I've put a small ext2 filing system here (built from a virgin FC1
>> image) which demonstrates the problem
>> 
>>   http://www.craig-wood.com/nick/pub/fedora_demo.gz
>> 
>> Mounting it and chrooting from dom0 under Xen 3.1 gives the SEGV.
>> Doing the same from Xen 3.0.3 , Xen 3.0.4 and Debian testing works
>> fine.
>> 
>> I've verified that the same thing happens on the binary download from
>> the xen web site (xen-3.1.0-install-x86_32p.tgz) dated June 1st so it
>> isn't just my dodgy compiling skills!
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Nick Craig-Wood]

On Thu, Jul 12, 2007 at 01:44:29PM +0100, Keir Fraser wrote:
> On 12/7/07 12:53, "Nick Craig-Wood" <nick@craig-wood.com> wrote:
> >> What exactly do you mean? I can mount the thing and chroot into it in dom0:
> >> what do I need to do next?
> > 
> > OK, I get a SEGV at that point using xen-3.1-testing or the xen 3.1
> > binary release.
> > 
> > Which xen version are you using?
> 
> Do you mean you do 'chroot /mnt /sbin/init'?

Here is the exact sequence I've used

# mkdir mnt
# mount -o loop fedora_demo mnt/
# cd mnt/
# chroot .
Segmentation fault
# cd ..
# umount mnt/

The above fails for xen 3.1 but succeeds (you get a shell) for xen 3.0.4 and 3.0.3

-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Nick Craig-Wood]

On Thu, Jul 12, 2007 at 02:11:19PM +0100, Keir Fraser wrote:
> My current guess is that this is a kernel version issue. Have you tried
> using your old (pre-3.1) domU kernel with the FC1 image on Xen 3.1?

Yes you are correct.  I can boot an fc1 image using xen-3.1
hypervisor, xen-3.1 dom0 and a xen-3.0.4 domU.

What exactly do you mean by "this is a kernel version issue"?

Thanks

Nick
-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Ian Campbell]

On Thu, 2007-07-12 at 18:19 +0100, Nick Craig-Wood wrote:
> On Thu, Jul 12, 2007 at 01:44:29PM +0100, Keir Fraser wrote:
> > On 12/7/07 12:53, "Nick Craig-Wood" <nick@craig-wood.com> wrote:
> > >> What exactly do you mean? I can mount the thing and chroot into it in dom0:
> > >> what do I need to do next?
> > > 
> > > OK, I get a SEGV at that point using xen-3.1-testing or the xen 3.1
> > > binary release.
> > > 
> > > Which xen version are you using?
> > 
> > Do you mean you do 'chroot /mnt /sbin/init'?
> 
> Here is the exact sequence I've used
> 
> # mkdir mnt
> # mount -o loop fedora_demo mnt/
> # cd mnt/
> # chroot .
> Segmentation fault
> # cd ..
> # umount mnt/
> 
> The above fails for xen 3.1 but succeeds (you get a shell) for xen 3.0.4 and 3.0.3

FC1 had some odd stuff going on with the vdso -- it comes up any time
upstream tries to touch the compat vdso code.

You could try enabling/disabling COMPAT_VDSO in your kernel 
configuration, see what happens.

It'd also be interesting to know which libc is getting linked in within
the chroot.

Ian.

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Keir Fraser]

On 12/7/07 18:51, "Ian Campbell" <Ian.Campbell@XenSource.com> wrote:

>> The above fails for xen 3.1 but succeeds (you get a shell) for xen 3.0.4 and
>> 3.0.3
> 
> FC1 had some odd stuff going on with the vdso -- it comes up any time
> upstream tries to touch the compat vdso code.
> 
> You could try enabling/disabling COMPAT_VDSO in your kernel
> configuration, see what happens.
> 
> It'd also be interesting to know which libc is getting linked in within
> the chroot.

Yes, it's definitely VDSO related as it crashes in linux-gate.so. 3.0.4
kernels work just fine with 3.1, so that is the correct fix for this
situation.

 -- Keir

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Nick Craig-Wood]

On Fri, Jul 13, 2007 at 07:31:32AM +0100, Keir Fraser wrote:
> On 12/7/07 18:51, "Ian Campbell" <Ian.Campbell@XenSource.com> wrote:
> 
> >> The above fails for xen 3.1 but succeeds (you get a shell) for xen 3.0.4 and
> >> 3.0.3
> > 
> > FC1 had some odd stuff going on with the vdso -- it comes up any time
> > upstream tries to touch the compat vdso code.
> > 
> > You could try enabling/disabling COMPAT_VDSO in your kernel
> > configuration, see what happens.
> > 
> > It'd also be interesting to know which libc is getting linked in within
> > the chroot.
> 
> Yes, it's definitely VDSO related as it crashes in linux-gate.so. 3.0.4
> kernels work just fine with 3.1, so that is the correct fix for this
> situation.

OK, I'll give that a go.  It will have to wait until next week, but
I'll report back the results!

-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Nick Craig-Wood]

On Fri, Jul 13, 2007 at 07:31:32AM +0100, Keir Fraser wrote:
> On 12/7/07 18:51, "Ian Campbell" <Ian.Campbell@XenSource.com> wrote:
> 
> >> The above fails for xen 3.1 but succeeds (you get a shell) for xen 3.0.4 and
> >> 3.0.3
> > 
> > FC1 had some odd stuff going on with the vdso -- it comes up any time
> > upstream tries to touch the compat vdso code.
> > 
> > You could try enabling/disabling COMPAT_VDSO in your kernel
> > configuration, see what happens.
> > 
> > It'd also be interesting to know which libc is getting linked in within
> > the chroot.
> 
> Yes, it's definitely VDSO related as it crashes in linux-gate.so. 3.0.4
> kernels work just fine with 3.1, so that is the correct fix for this
> situation.

I've now tried this.

Firstly the COMPAT_VDSO wasn't appearing in the menu

# from arch/i386/Kconfig 

config COMPAT_VDSO
        bool "Compat VDSO support"
        depends on !X86_XEN
        default y
        help
          Map the VDSO to the predictable old-style address too.
        ---help---
          Say N here if you are running a sufficiently recent glibc
          version (2.3.3 or later), to remove the high-mapped
          VDSO mapping and to exclusively use the randomized VDSO.

          If unsure, say Y.

endmenu

I removed the "depends on !X86_XEN" line, configured COMPAT_VDSO but
the build then failed here

  CC      arch/i386/kernel/asm-offsets.s
arch/i386/kernel/asm-offsets.c: In function 'foo':
arch/i386/kernel/asm-offsets.c:79: warning: asm operand 0 probably doesn't match constraints
arch/i386/kernel/asm-offsets.c:79: error: impossible constraint in 'asm'

So I guess it was there for a reason!  Configuring COMPAT_VDSO off
causes it to compile again.

Any ideas?

Thanks

-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Ian Campbell]

On Thu, 2007-07-19 at 12:59 +0100, Nick Craig-Wood wrote:
> Any ideas? 

Could you try reverting
http://xenbits.xensource.com/xen-unstable.hg?rev/98eacf1b075c

I wonder if the broken glibc referred to in the comment is FC1's. Seems
likely.

Ian.

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Nick Craig-Wood]

On Thu, Jul 19, 2007 at 01:29:31PM +0100, Ian Campbell wrote:
> On Thu, 2007-07-19 at 12:59 +0100, Nick Craig-Wood wrote:
> > Any ideas? 
> 
> Could you try reverting
> http://xenbits.xensource.com/xen-unstable.hg?rev/98eacf1b075c

No change in symptoms unfortunately :-(

The FC1 domU produces :-

Linux version 2.6.18-u8a-p4 (root@dev0001) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Thu Jul 19 13:22:35 BST 2007
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 0000000008800000 (usable)
0MB HIGHMEM available.
136MB LOWMEM available.
NX (Execute Disable) protection: active
Allocating PCI resources starting at 10000000 (gap: 08800000:f7800000)
Detected 2992.776 MHz processor.
Built 1 zonelists.  Total pages: 34816
Kernel command line: root=/dev/sda1 ro panic=15
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 1024 (order: 10, 4096 bytes)
Xen reported: 2992.504 MHz processor.
Console: colour dummy device 80x25
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Software IO TLB disabled
vmalloc area: c9000000-f53fe000, maxmem 2d7fe000
Memory: 124768k/139264k available (3034k kernel code, 6144k reserved, 829k data, 188k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 5987.76 BogoMIPS (lpj=29938837)
Mount-cache hash table entries: 512
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
Checking 'hlt' instruction... OK.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 20k freed
Brought up 1 CPUs
migration_cost=0
NET: Registered protocol family 16
xen_mem: Initialising balloon driver.
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 4096)
TCP reno registered
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
fuse init (API version 7.7)
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered (default)
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
loop: loaded (max 8 devices)
nbd: registered device at major 43
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
PPP BSD Compression module registered
PPP MPPE Compression module registered
NET: Registered protocol family 24
SLIP: version 0.8.4-NET3.019-NEWTTY (dynamic channels, max=256) (6 bit encapsulation enabled).
CSLIP: code copyright 1989 Regents of the University of California.
SLIP linefill/keepalive option.
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
Xen virtual console successfully installed as tty1
Event-channel device installed.
netfront: Initialising virtual ethernet driver.
aoe: aoe_init: AoE v22 initialised.
i8042.c: No controller found.
mice: PS/2 mouse device common for all mice
GACT probability on
Mirror/redirect action on
Simple TC action Loaded
netem: version 1.2
u32 classifier
    Performance counters on
    input device check on
    Actions configured
Netfilter messages via NETLINK v0.30.
IPv4 over IPv4 tunneling driver
GRE over IPv4 tunneling driver
ip_conntrack version 2.4 (1088 buckets, 8704 max) - 224 bytes per conntrack
xen-vbd: registered block device major 8
netfront: device eth0 has copying receive path.
ctnetlink v0.90: registering with nfnetlink.
ip_conntrack_pptp version 3.1 loaded
ip_nat_pptp version 3.0 loaded
ip_tables: (C) 2000-2006 Netfilter Core Team
ClusterIP Version 0.8 loaded successfully
arp_tables: (C) 2002 David S. Miller
IPVS: Registered protocols (TCP, UDP, AH, ESP)
IPVS: Connection hash table configured (size=4096, memory=32Kbytes)
IPVS: ipvs loaded.
IPVS: [rr] scheduler registered.
IPVS: [wrr] scheduler registered.
IPVS: [lc] scheduler registered.
IPVS: [wlc] scheduler registered.
IPVS: [lblc] scheduler registered.
IPVS: [lblcr] scheduler registered.
IPVS: [dh] scheduler registered.
IPVS: [sh] scheduler registered.
IPVS: [sed] scheduler registered.
IPVS: [nq] scheduler registered.
IPVS: ftp: loaded support on port[0] = 21
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
ip6_tables: (C) 2000-2006 Netfilter Core Team
NET: Registered protocol family 17
NET: Registered protocol family 15
Using IPI Shortcut mode
XENBUS: Device with no driver: device/console/0
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 188k freed

It hangs here with 100% CPU usage.

-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Ian Campbell]

On Thu, 2007-07-19 at 14:33 +0100, Nick Craig-Wood wrote:
> On Thu, Jul 19, 2007 at 01:29:31PM +0100, Ian Campbell wrote:
> > On Thu, 2007-07-19 at 12:59 +0100, Nick Craig-Wood wrote:
> > > Any ideas? 
> > 
> > Could you try reverting
> > http://xenbits.xensource.com/xen-unstable.hg?rev/98eacf1b075c
> 
> No change in symptoms unfortunately :-(

Hmm, I'm starting to clutch at straws now then :-(

Could you try adding "vdso=0" to the kernel command line.

Ian.

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Nick Craig-Wood]

On Thu, Jul 19, 2007 at 02:50:12PM +0100, Ian Campbell wrote:
> On Thu, 2007-07-19 at 14:33 +0100, Nick Craig-Wood wrote:
> > On Thu, Jul 19, 2007 at 01:29:31PM +0100, Ian Campbell wrote:
> > > On Thu, 2007-07-19 at 12:59 +0100, Nick Craig-Wood wrote:
> > > > Any ideas? 
> > > 
> > > Could you try reverting
> > > http://xenbits.xensource.com/xen-unstable.hg?rev/98eacf1b075c
> > 
> > No change in symptoms unfortunately :-(
> 
> Hmm, I'm starting to clutch at straws now then :-(
> 
> Could you try adding "vdso=0" to the kernel command line.

A good clutch!

That fixes the problem.  The original kernel I started with works fine
with vdso=0 as does the xen 3.1 binary release.

As I understand it vdso enables use of the faster sysenter rather than
int80 for sys calls.  So disabling it will slow down syscalls slightly
but otherwise shouldn't matter.  Is that right?

Thanks for all your help

Nick
-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick

Re: Xen 3.1 - Can't run Fedora Core 1 PV

[Ian Campbell]

On Thu, 2007-07-19 at 17:34 +0100, Nick Craig-Wood wrote:
> That fixes the problem.  The original kernel I started with works fine
> with vdso=0 as does the xen 3.1 binary release.

Phew!

> As I understand it vdso enables use of the faster sysenter rather than
> int80 for sys calls.  So disabling it will slow down syscalls slightly
> but otherwise shouldn't matter.  Is that right?

On native yes but under Xen sysenter isn't available so you are using
int80 anyway.

Cheers,
Ian.

"Network %s doesn't exist" in xenapi_create.py

[Ramon Caceres]

Hello.

How do I arrange for a new network bridge in dom0 to appear in the 
"networks" dictionary created by the following statement in line 417 of 
recent versions of xen-unstable.hg/tools/python/xen/xm/xenapi_create.py? 

                networks = dict([(record['name_label'], ref)
                                 for ref, record in
 server.xenapi.network.get_all_records().items()])
 
I am creating a bridge named br0.101 and adding to it a VLAN interface 
named peth0.101.  This bridge is in addition to the default eth0 bridge 
created by /xen/scripts/network-bridge.  I use the following sequence in a 
modified version of network-bridge:

        # Create VLAN interface
        vconfig add ${pdev} ${vlan}
        vlanif=${pdev}.${vlan}
        ip link set ${vlanif} up

        # Create bridge
        br="br${netnum}.${vlan}"
        create_bridge ${br}

        # Add VLAN interface to bridge
        setup_bridge_port ${vlanif}
        add_to_bridge2 ${br} ${vlanif}
        do_ifup ${br}

The new bridge and interface show up as expected in the output of brctl, 
ifconfig, and ip, as well as in the /sys/class/net directory.   However, 
when I try to connect a domU to the bridge by specifying "vif = [ 
'bridge=br0.101' ]", I get this:

        Error: Network br0.101 doesn't exist

If I change that to "vif = [ 'bridge=eth0' ]" or "vif = [ '' ]" then the 
domU comes up without error and connects to the eth0 bridge as normal.

The VLAN-related bridges were working for several months on xen-unstable 
changeset 14887 from April 21, but the above problem surfaced when I 
updated to 15521 from July 10.  I haven't been able to find what I'm 
missing to make my new bridge appear in that "networks" dictionary the way 
that the default bridge does.

Any help would be greatly appreciated.

--Ramón
_____________________________
Ramón Cáceres
Research Staff Member
IBM T.J. Watson Research Center