"hwcap 0 nosegneg" doesnt work with paravirt_ops xen as of 2.6.23.9

"hwcap 0 nosegneg" doesnt work with paravirt_ops xen as of 2.6.23.9

[Andres Freund]

[-- Attachment #1: Type: text/plain, Size: 2039 bytes --]

Hi all,

after converting a host from using a ubuntu patched 2.6.22 domU to using 
2.6.23 xen via paravirt_ops (no configuration change except devicename change 
in fstab, and another getty running) ldconfig no longer picks up the nosegneg 
libc leading to segfaults for some programs.

/etc/ld.so.conf.d/xen.conf:
hwcap 0 nosegneg

On ubuntu's 2.6.22-xen:
ldd /usr/bin/perl
        linux-gate.so.1 =>  (0xf57fe000)
        libdl.so.2 => /lib/tls/i686/nosegneg/libdl.so.2 (0xb7f90000)
        libm.so.6 => /lib/tls/i686/nosegneg/libm.so.6 (0xb7f6b000)
        libpthread.so.0 => /lib/tls/i686/nosegneg/libpthread.so.0 (0xb7f53000)
        libc.so.6 => /lib/tls/i686/nosegneg/libc.so.6 (0xb7e05000)
        libcrypt.so.1 => /lib/tls/i686/nosegneg/libcrypt.so.1 (0xb7dd7000)
        /lib/ld-linux.so.2 (0xb7f9d000)

On selfcompiled 2.6.23.9:
ldd /usr/bin/perl
        linux-gate.so.1 =>  (0xf57fc000)
        libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7f3b000)
        libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7f16000)
        libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7efe000)
        libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7db3000)
        libcrypt.so.1 => /lib/tls/i686/cmov/libcrypt.so.1 (0xb7d85000)
        /lib/ld-linux.so.2 (0xb7f46000)


I now discover, that after some playing around (during discovering what the 
problem is) i must have changed something, because ldd still shows the use of 
the i686/cmov libraries, but applications reliably causing crashes earlier, 
dont do so anymore.

Perhaps Im all wrong, and paravirt_ops doesnt need the nosegneg versions and 
just a ldconfig or so (which seems strange because the result before and 
after the ldconfig were the same).

Greetings, 

Andres Freund


PS: I did this conversion mainly, because I was getting quite random crashes 
inside the domU. I guess, as this was a vendor kernel + xensource xen kernel, 
this isnt appropriate for lkml? As well as the crashes of the Dom0 (ubuntu 
again)?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: "hwcap 0 nosegneg" doesnt work with paravirt_ops xen as of 2.6.23.9

[Jeremy Fitzhardinge]

Andres Freund wrote:
> after converting a host from using a ubuntu patched 2.6.22 domU to using 
> 2.6.23 xen via paravirt_ops (no configuration change except devicename change 
> in fstab, and another getty running) ldconfig no longer picks up the nosegneg 
> libc leading to segfaults for some programs.
>   

It shouldn't ever lead to segfaults.  At worst, you should get poor
performance because the hypervisor emulates the -ve segment offset.


> /etc/ld.so.conf.d/xen.conf:
> hwcap 0 nosegneg
>   

This looks OK.  If this doesn't work, then it suggests there's something
wrong with the kernel vdso which is not exporting the nonegseg
(pseudo-)hardware capability.  But we've gone over this several times
now, and I was fairly sure we'd finally got it right - works for me, at
least.

The alternative is that there's some bug on the userspace side in your
version of Ubuntu...

Roland?

> On ubuntu's 2.6.22-xen:
> ldd /usr/bin/perl
>         linux-gate.so.1 =>  (0xf57fe000)
>         libdl.so.2 => /lib/tls/i686/nosegneg/libdl.so.2 (0xb7f90000)
>         libm.so.6 => /lib/tls/i686/nosegneg/libm.so.6 (0xb7f6b000)
>         libpthread.so.0 => /lib/tls/i686/nosegneg/libpthread.so.0 (0xb7f53000)
>         libc.so.6 => /lib/tls/i686/nosegneg/libc.so.6 (0xb7e05000)
>         libcrypt.so.1 => /lib/tls/i686/nosegneg/libcrypt.so.1 (0xb7dd7000)
>         /lib/ld-linux.so.2 (0xb7f9d000)
>
> On selfcompiled 2.6.23.9:
> ldd /usr/bin/perl
>         linux-gate.so.1 =>  (0xf57fc000)
>         libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7f3b000)
>         libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7f16000)
>         libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7efe000)
>         libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7db3000)
>         libcrypt.so.1 => /lib/tls/i686/cmov/libcrypt.so.1 (0xb7d85000)
>         /lib/ld-linux.so.2 (0xb7f46000)
>
>
> I now discover, that after some playing around (during discovering what the 
> problem is) i must have changed something, because ldd still shows the use of 
> the i686/cmov libraries, but applications reliably causing crashes earlier, 
> dont do so anymore.
>   

As I said, you shouldn't be seeing crashes either way.  If you are, it
suggests a hypervisor bug.  Does "xm dmesg" show anything?

> Perhaps Im all wrong, and paravirt_ops doesnt need the nosegneg versions and 
> just a ldconfig or so (which seems strange because the result before and 
> after the ldconfig were the same).
>   

nosegneg is still necessary, and it should still be working as it did
before.

> PS: I did this conversion mainly, because I was getting quite random crashes 
> inside the domU. I guess, as this was a vendor kernel + xensource xen kernel, 
> this isnt appropriate for lkml? As well as the crashes of the Dom0 (ubuntu 
> again)?
>   

Yeah, anything which isn't 2.6.18-xen isn't officially "Xen project",
but its worth reporting the bugs to xen-devel (and Ubuntu, of course). 
And to me, for that matter.  I won't be able to help directly, but the
crashes may highlight things we need to be careful about in the pvops
Xen support.

    J

Re: "hwcap 0 nosegneg" doesnt work with paravirt_ops xen as of 2.6.23.9

[Andres Freund]

[-- Attachment #1: Type: text/plain, Size: 4337 bytes --]

Hi Jeremy,

On Wednesday 28 November 2007, Jeremy Fitzhardinge wrote in "Re: "hwcap 0 
nosegneg" doesnt work with paravirt_ops xen as of 2.6.23.9":
> Andres Freund wrote:
> > after converting a host from using a ubuntu patched 2.6.22 domU to using
> > 2.6.23 xen via paravirt_ops (no configuration change except devicename
> > change in fstab, and another getty running) ldconfig no longer picks up
> > the nosegneg libc leading to segfaults for some programs.
> It shouldn't ever lead to segfaults.  At worst, you should get poor
> performance because the hypervisor emulates the -ve segment offset.
strace -ff 
perl /usr/bin/debsums --generate=nocheck -sp /var/cache/apt/archives
...
open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 4
fstat64(4, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(4, F_SETFD, FD_CLOEXEC)         = 0
getdents64(4, /* 3 entries */, 4096)    = 72
getdents64(4, /* 0 entries */, 4096)    = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 3238 detached

root@test:~# gdb perl5.8.8                 
GNU gdb 6.6-debian
...
Starting 
program: /usr/bin/perl5.8.8 /usr/bin/debsums --generate=nocheck -sp /var/cache/apt/archives
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New Thread -1209940304 (LWP 3356)]
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1209940304 (LWP 3356)]
0xb7eda3d7 in readdir64_r () from /lib/tls/i686/cmov/libc.so.6
(gdb) bt
#0  0xb7eda3d7 in readdir64_r () from /lib/tls/i686/cmov/libc.so.6
#1  0x080fdca5 in Perl_pp_readdir ()
#2  0x080c0d29 in Perl_runops_standard ()
#3  0x0806727a in perl_run ()
#4  0x08063732 in main ()

This was before running an ldconfig. After running, it still selects the wrong 
libraries, but doesnt crash anymore (hm?).


> > /etc/ld.so.conf.d/xen.conf:
> > hwcap 0 nosegneg
> This looks OK.  If this doesn't work, then it suggests there's something
> wrong with the kernel vdso which is not exporting the nonegseg
> (pseudo-)hardware capability.  But we've gone over this several times
> now, and I was fairly sure we'd finally got it right - works for me, at
> least.

> The alternative is that there's some bug on the userspace side in your
> version of Ubuntu...
The thing is, that the same image, aside from the neccessary config changes 
(sda1->xvda1, xvc0->hvc0), recognizes the right libraries. So I dont see an 
obvious problem there.

> > I now discover, that after some playing around (during discovering what
> > the problem is) i must have changed something, because ldd still shows
> > the use of the i686/cmov libraries, but applications reliably causing
> > crashes earlier, dont do so anymore.
> As I said, you shouldn't be seeing crashes either way.  If you are, it
> suggests a hypervisor bug.  Does "xm dmesg" show anything?
(XEN) traps.c:1698:d16 Domain attempted WRMSR 0000008b from 00000056:00000000 
to 00000000:00000000.
(XEN) traps.c:1698:d17 Domain attempted WRMSR 00000404 from 00000000:00000001 
to ffffffff:ffffffff.
And some more of it. But why does a ldconfig, which doesnt change the 
libraries loaded, change that?

For all debug data of the crashes see above.

> > PS: I did this conversion mainly, because I was getting quite random
> > crashes inside the domU. I guess, as this was a vendor kernel + xensource
> > xen kernel, this isnt appropriate for lkml? As well as the crashes of the
> > Dom0 (ubuntu again)?
> Yeah, anything which isn't 2.6.18-xen isn't officially "Xen project",
> but its worth reporting the bugs to xen-devel (and Ubuntu, of course).
> And to me, for that matter.  I won't be able to help directly, but the
> crashes may highlight things we need to be careful about in the pvops
> Xen support.
Ok, will do so in the next days and will CC you at that.

Btw, is there a roadmap how in-kernel Xen is planned to develop?

Greetings, 

Andres

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: "hwcap 0 nosegneg" doesnt work with paravirt_ops xen as of 2.6.23.9

[Roland McGrath]

> > /etc/ld.so.conf.d/xen.conf:
> > hwcap 0 nosegneg
> 
> This looks OK. [...]

No, it does not match this:

	/* Bit used for the pseudo-hwcap for non-negative segments.  We use
	   bit 1 to avoid bugs in some versions of glibc when bit 0 is
	   used; the choice is otherwise arbitrary. */
	#define VDSO_NOTE_NONEGSEG_BIT	1

You really want "hwcap 1 nosegneg" given that.
Yes, we've been over this several times already.


Thanks,
Roland

Re: "hwcap 0 nosegneg" doesnt work with paravirt_ops xen as of 2.6.23.9

[Andres Freund]

[-- Attachment #1: Type: text/plain, Size: 1198 bytes --]

Hi,

On Wednesday 28 November 2007, Roland McGrath wrote in "Re: "hwcap 0 nosegneg" 
doesnt work with paravirt_ops xen as of 2.6.23.9":
> > > /etc/ld.so.conf.d/xen.conf:
> > > hwcap 0 nosegneg
> >
> > This looks OK. [...]
>
> No, it does not match this:
> 	/* Bit used for the pseudo-hwcap for non-negative segments.  We use
> 	   bit 1 to avoid bugs in some versions of glibc when bit 0 is
> 	   used; the choice is otherwise arbitrary. */
> 	#define VDSO_NOTE_NONEGSEG_BIT	1
> You really want "hwcap 1 nosegneg" given that.
> Yes, we've been over this several times already.
Thanks for the information. I guess it would be nice to document this 
somewhere (except in code)... At least I could not find any information about 
it beside some patch (Not directed at you, Roland). Any idea where?

Besides, it has one Problem: It is not backwards compatible. Using it on 
earlier Xen Versions yields the normal tls versions (xen 3.1). Isnt the 
userbase of existing xen installations bigger than the one of some point in 
FC5 test releases?
Ugh. Leaning a bit far out of the door with that one considering my knowledge 
about it... Sorry about that.

Greetings,

Andres

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]