[uml-devel] glibc and NPTL, problem, workarounds, etc.

[uml-devel] glibc and NPTL, problem, workarounds, etc.

[William Stearns]

Good day, all,
	Zaphod had a rather nasty filesystem crash last week.  It happened 
around the time that two other events happened; the raid array forgot it 
had drive 3 (requiring me to remove and re-add the drive and start a raid 
rebuild), and one of the users was attempting to upgrade glibc on debian.  
This put in a new version of glibc, along with the NPTL (Native Posix 
Threading Libraries) that don't work with UML.  From that point, that user 
could log in with ssh, but could no longer run any applications because 
bash refused to fork.

	There are a couple of workarounds for this problem.  First, when 
installing any root filesystem, check afterwards to see if there's a 
directory called /lib/tls/ .  If so, rename or delete it.  This removes 
the NPTL extensions to glibc but leaves glibc itself, so applications 
continue to work; they just never try to use the NPTL extensions.
	To make sure it's never used again, do the following as root:

rm -rf /lib/tls/
mkdir /lib/tls/
chmod 000 /lib/tls/
chattr +i /lib/tls/

	Now even root can't write to that directory.  Any attempt to 
install a new version of glibc will probably fail because rpm/apt can't 
write to that directory, but it's better than having a non-functioning 
system.

	In trying to run the above commands while fixing his system, I ran
into the problem where the commands I wanted to run loaded glibc and the
NPTL libaries, and therefore refused to run (the error message I got on
fork mentioned "nptl").  In fact, running single user mode (starting up
uml with "ro single init=/bin/bash") didn't do any better since bash
needed to fork to run the above commands and the fork failed.
	The workaround to the above is this:

export LD_ASSUME_KERNEL=2.4.1
exec /bin/bash

	which replaces your current bash with a new one that won't try to 
load the nptl libraries.  Now you can run commands even though the 
/lib/tls/ directory is there - this would be an excellent time to remove 
them.  :-)

	Jeff, I have a seriously hard time believing that simply running 
nptl binaries could cause journal errors out on the host.  I'm much more 
inclined to believe that the raid rebuild might have briefly set the block 
devices read-only, long enough for the journal to give up because there's 
no place to put the data.  I just thought I'd bring up the closesness in 
time in case it triggered any connections for you.
	I do have some of the host journal errors if they're useful, but I 
suspect they're not.
	Cheers,
	- Bill

---------------------------------------------------------------------------
	"Computers let you make more mistakes faster than any other
invention in human history, with the possible exception of handguns and
tequila." 
	-- Mitch Radcliffe
(Courtesy of Hugo van der Kooij <hvdkooij@caiw.nl>)
--------------------------------------------------------------------------
William Stearns (wstearns@pobox.com).  Mason, Buildkernel, freedups, p0f,
rsync-backup, ssh-keyinstall, dns-check, more at:   http://www.stearns.org
--------------------------------------------------------------------------


-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] glibc and NPTL, problem, workarounds, etc.

[Steven Pritchard]

On Wed, Sep 08, 2004 at 03:02:03PM -0400, William Stearns wrote:
> 	The workaround to the above is this:
> 
> export LD_ASSUME_KERNEL=2.4.1
> exec /bin/bash

Couldn't you just boot with LD_ASSUME_KERNEL=2.4.1 on the kernel
command line?  ISTR that anything not processed by the kernel ends up
in init's environment.

Unfortunately, I don't have an easy way to test this at the moment...

Steve
-- 
Steven Pritchard - K&S Pritchard Enterprises, Inc.
Email: steve@kspei.com             http://www.kspei.com/
Phone: (618)398-7360               Mobile: (618)567-7320


-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] glibc and NPTL, problem, workarounds, etc.

[BlaisorBlade]

On Wednesday 08 September 2004 21:02, William Stearns wrote:
> Good day, all,
> 	Zaphod had a rather nasty filesystem crash last week.  It happened
> around the time that two other events happened; the raid array forgot it
> had drive 3 (requiring me to remove and re-add the drive and start a raid
> rebuild), and one of the users was attempting to upgrade glibc on debian.
> This put in a new version of glibc, along with the NPTL (Native Posix
> Threading Libraries) that don't work with UML.  From that point, that user
> could log in with ssh, but could no longer run any applications because
> bash refused to fork.

> 	There are a couple of workarounds for this problem.  First, when
> installing any root filesystem, check afterwards to see if there's a
> directory called /lib/tls/ .  If so, rename or delete it.  This removes
> the NPTL extensions to glibc but leaves glibc itself, so applications
> continue to work; they just never try to use the NPTL extensions.
> 	To make sure it's never used again, do the following as root:

There is another patch which should make the work - (I never tested it, but it 
should work). Basically, uname will state it is a "i586" rather than "i686", 
so NPTL is not enabled:

http://www.suse.de/~kraxel/uml/patches/uml-pretend-to-be-i586

> rm -rf /lib/tls/
> mkdir /lib/tls/
> chmod 000 /lib/tls/
> chattr +i /lib/tls/

> 	Now even root can't write to that directory.  Any attempt to
> install a new version of glibc will probably fail because rpm/apt can't
> write to that directory, but it's better than having a non-functioning
> system.

> 	In trying to run the above commands while fixing his system, I ran
> into the problem where the commands I wanted to run loaded glibc and the
> NPTL libaries, and therefore refused to run (the error message I got on
> fork mentioned "nptl").  In fact, running single user mode (starting up
> uml with "ro single init=/bin/bash") didn't do any better since bash
> needed to fork to run the above commands and the fork failed.
> 	The workaround to the above is this:
>
> export LD_ASSUME_KERNEL=2.4.1
> exec /bin/bash

> 	which replaces your current bash with a new one that won't try to
> load the nptl libraries.  Now you can run commands even though the
> /lib/tls/ directory is there - this would be an excellent time to remove
> them.  :-)

> 	Jeff, I have a seriously hard time believing that simply running
> nptl binaries could cause journal errors out on the host.

You got them on the *physical* host? Yup!

> I'm much more 
> inclined to believe that the raid rebuild might have briefly set the block
> devices read-only
No, it shouldn't have - I didn't write the RAID code, but I'm sure that 
anything such is not allowed to happen.

Did the machine poweroff/crash/oops during the rebuild?

I've lately discovered (and fixed) a big number of problems with the SKAS 
patch, and somebody reported a host crash while running Java in UML which 
could be connected to one of the SKAS bugs.
> , long enough for the journal to give up because there's 
> no place to put the data. 
Ok, there is something more similar that could, actually, have happened. The 
device does not become "read only" at all - the stripe which is being 
processed will probably be locked for a while, which means that the datas 
will wait to be written for a while. Then, if memory is exhausted, the 
journal will get errors.
> I just thought I'd bring up the closesness in 
> time in case it triggered any connections for you.

> 	I do have some of the host journal errors if they're useful, but I
> suspect they're not.
Please post the messages (without flooding us with debug datas - post only 
what is easily human-readable, and redirect the rest to the LKML, the 
filesystem ML and to the RAID ml, if any).

Also, which is the filesystem you use?

Bye
-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM. 
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] glibc and NPTL, problem, workarounds, etc.

[Chris Wedgwood]

On Sun, Sep 12, 2004 at 08:20:51PM +0200, BlaisorBlade wrote:

> There is another patch which should make the work - (I never tested
> it, but it should work). Basically, uname will state it is a "i586"
> rather than "i686",
> so NPTL is not enabled:
>
> http://www.suse.de/~kraxel/uml/patches/uml-pretend-to-be-i586

This apparently only works because people[1] ship either i386 or i686
libc and only the latter has NPTL support --- but there is nothing
preventing i586 NPTL support in the future...

  --cw

[1] RH & Debian checked, maybe this isn't true for gentoo in which
    case it will break there?


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM. 
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] glibc and NPTL, problem, workarounds, etc.

[BlaisorBlade]

On Sunday 12 September 2004 21:02, Chris Wedgwood wrote:
> On Sun, Sep 12, 2004 at 08:20:51PM +0200, BlaisorBlade wrote:
> > There is another patch which should make the work - (I never tested
> > it, but it should work). Basically, uname will state it is a "i586"
> > rather than "i686",
> > so NPTL is not enabled:

> > http://www.suse.de/~kraxel/uml/patches/uml-pretend-to-be-i586

> This apparently only works because people[1] ship either i386 or i686
> libc and only the latter has NPTL support --- but there is nothing
> preventing i586 NPTL support in the future...
Are you sure? I remember that NPTL needs using some new instructions of i686
(maybe CMOVxx) for futexes, to be atomic. 
However, if I'm wrong, we could even pretend to be i386.
>   --cw

> [1] RH & Debian checked, maybe this isn't true for gentoo in which
>     case it will break there?

Well, you compile Gentoo for your own processor - you might maybe even enable 
-mregparm for everything (which even breaks binary compatibility everything). 
I guess if you build Gentoo on an i386, it will either refuse to build or 
build an i386 glibc. Otherwise, it does not make sense for them to ship i386 
glibc.

That said, you must just be careful when building the guest root_fs, with 
Gentoo.
-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM. 
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] glibc and NPTL, problem, workarounds, etc.

[Chris Wedgwood]

On Sun, Sep 12, 2004 at 09:27:12PM +0200, BlaisorBlade wrote:

> Are you sure? I remember that NPTL needs using some new instructions
> of i686 (maybe CMOVxx) for futexes, to be atomic.

i586 and later i486 will suffice

> However, if I'm wrong, we could even pretend to be i386.

NPTL can be made to work too there I'm told, with a little more
effort.  I'm not sure if anyone will ever bother...

> Well, you compile Gentoo for your own processor - you might maybe
> even enable -mregparm for everything (which even breaks binary
> compatibility everything).

i was thinking gentoo on a i585 would maybe build /lib/tls/i586


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM. 
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel