Re: kernel: ldt allocation failed

Re: kernel: ldt allocation failed

[Dan Maas]

> We suddenly started seeing freezing problems on a number of machines
> in the past couple of days. There is no pattern as far as I can tell.
> It has happened while running OpenGL apps, netscape, or even when not
> doing anything.

> Software:
>     - NVIDIA drivers 1.0-1541

Sorry, we do not have the source to NVIDIA's driver, so we cannot help you
debug this problem. Please contact NVIDIA support.

Regards,
Dan

Re: kernel: ldt allocation failed

[Kiril Vidimce]

On Fri, 7 Dec 2001, Dan Maas wrote:
> > We suddenly started seeing freezing problems on a number of machines
> > in the past couple of days. There is no pattern as far as I can tell.
> > It has happened while running OpenGL apps, netscape, or even when not
> > doing anything.
> 
> > Software:
> >     - NVIDIA drivers 1.0-1541
> 
> Sorry, we do not have the source to NVIDIA's driver, so we cannot help you
> debug this problem. Please contact NVIDIA support.

I don't see how one can magically tell that this is an NVIDIA problem. 
I am contacting NVIDIA at the same time and it's possible that the 
problem is with their drivers. However, there is something going on 
in the kernel and I imagine that even if the NVIDIA drivers are 
triggering the problem, there are other modules/apps that can bring 
about the same behavior.

KV
--
  ___________________________________________________________________
  Studio Tools                                        vkire@pixar.com
  Pixar Animation Studios                        http://www.pixar.com/

Re: kernel: ldt allocation failed

[Alan Cox]

> I don't see how one can magically tell that this is an NVIDIA problem. 

We don't know. But since we don't have their source and they have our
source only they can tell you.

> in the kernel and I imagine that even if the NVIDIA drivers are 
> triggering the problem, there are other modules/apps that can bring 
> about the same behavior.

Possibly, but you'll have to ask Nvidia to debug it for you. If you can
reproduce a bug by 
	-	removing the nvidia modules so they wont be loaded
	-	hard booting the machine
	-	triggering the bug without loading the nvidia drivers

then please report it. If not, its between you and nvidia.

Re: kernel: ldt allocation failed

[Kiril Vidimce]

On Fri, 7 Dec 2001, Alan Cox wrote:
> > I don't see how one can magically tell that this is an NVIDIA problem. 
> 
> We don't know. But since we don't have their source and they have our
> source only they can tell you.
> 
> > in the kernel and I imagine that even if the NVIDIA drivers are 
> > triggering the problem, there are other modules/apps that can bring 
> > about the same behavior.
> 
> Possibly, but you'll have to ask Nvidia to debug it for you. If you can
> reproduce a bug by 
> 	-	removing the nvidia modules so they wont be loaded
> 	-	hard booting the machine
> 	-	triggering the bug without loading the nvidia drivers
> 
> then please report it. If not, its between you and nvidia.

Without turning this into a religous debate, is there any way to
find out what those messages really mean? When does one run into
an ldt allocation failure? What is ldt? I am not just trying to
solve this problem; I also want to understand what is going on
in the kernel.

Thanks for any explanations.

KV
--
  ___________________________________________________________________
  Studio Tools                                        vkire@pixar.com
  Pixar Animation Studios                        http://www.pixar.com/

Re: kernel: ldt allocation failed

[Alan Cox]

> Without turning this into a religous debate, is there any way to
> find out what those messages really mean? When does one run into
> an ldt allocation failure? What is ldt? I am not just trying to
> solve this problem; I also want to understand what is going on
> in the kernel.

Have a look in arch/i386/kernel/*.c. LDT is a descriptor table of segment
registers. Its generally used by emulators to run things like win16 binaries
and sometimes by threaded apps to do thread specific address spaces by
giving each thread a different %fs or %gs segment register

Re: kernel: ldt allocation failed

[James Davies]

The nVidia kernel drivers are open source. You can get them from
ftp://ftp.nvidia.com/pub/drivers/english/XFree86_40/1.0-2313/NVIDIA_kernel-1.0-2313.tar.gz

the 0.9 serious of drivers were buggy and crashed a lot, earning them a
bad rep. But the 1.0 series are faster and more stable than their
windows counterparts- I havn't had one crash, even with a faulty card
that kills windows constantly. 


On Fri, 2001-12-07 at 19:15, Alan Cox wrote:
> > I don't see how one can magically tell that this is an NVIDIA problem. 
> 
> We don't know. But since we don't have their source and they have our
> source only they can tell you.
> 
> > in the kernel and I imagine that even if the NVIDIA drivers are 
> > triggering the problem, there are other modules/apps that can bring 
> > about the same behavior.
> 
> Possibly, but you'll have to ask Nvidia to debug it for you. If you can
> reproduce a bug by 
> 	-	removing the nvidia modules so they wont be loaded
> 	-	hard booting the machine
> 	-	triggering the bug without loading the nvidia drivers
> 
> then please report it. If not, its between you and nvidia.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/



_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

Re: kernel: ldt allocation failed

[Alan Cox]

> The nVidia kernel drivers are open source. You can get them from
> ftp://ftp.nvidia.com/pub/drivers/english/XFree86_40/1.0-2313/NVIDIA_kernel-1.0-2313.tar.gz

The Nvidia drivers are not open source. Please stop pedalling this myth. All
you are doing is risking confusing other unfortunates into buying nvidia
hardware and finding there is no open 3D support.

> bad rep. But the 1.0 series are faster and more stable than their
> windows counterparts- I havn't had one crash, even with a faulty card
> that kills windows constantly. 

So tell me why I keep getting people who remove the Nvidia drivers and have
their machine work. I'm sure some of them are hardware related issues, but
all of them ?

Alan

Re: kernel: ldt allocation failed

[James Davies]

On Fri, 2001-12-07 at 19:43, Alan Cox wrote:
> > The nVidia kernel drivers are open source. You can get them from
> > ftp://ftp.nvidia.com/pub/drivers/english/XFree86_40/1.0-2313/NVIDIA_kernel-1.0-2313.tar.gz
> 
> The Nvidia drivers are not open source. Please stop pedalling this myth. All
> you are doing is risking confusing other unfortunates into buying nvidia
> hardware and finding there is no open 3D support.

I wan't trying to state that it is GPL or similar, but the kernel driver
source code IS available for free under a restrictive license. the GLX
drivers are closed. 

> > bad rep. But the 1.0 series are faster and more stable than their
> > windows counterparts- I havn't had one crash, even with a faulty card
> > that kills windows constantly. 
> 
> So tell me why I keep getting people who remove the Nvidia drivers and have
> their machine work. I'm sure some of them are hardware related issues, but
> all of them ?

I can't comment on the stability in all configurations, but I've found
them to work quite well with my box since 1.0 came out. the 0.9 serious
were notorious for crashing though.
 
> Alan



_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

Re: kernel: ldt allocation failed

[Dave Jones]

On 7 Dec 2001, James Davies wrote:

> source code IS available for free under a restrictive license. the GLX
> drivers are closed.

Bzzzt...

>From the tarball you posted a link to..

-rwxr-xr-x    1 davej    users      889615 Nov 27 20:39 Module-nvkernel*

(davej@noodles:NVIDIA_kernel-1.0-2313)$ file Module-nvkernel
Module-nvkernel: ELF 32-bit LSB relocatable, Intel 80386, version 1, not
stripped

No-one but nvidia knows whats in this.

regards,
Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: kernel: ldt allocation failed

[James Davies]

Ok I see my mistake now. While the kernel driver contains source code,
it is only enough to link with the current kernel correctly, and the
rest binary. Sorry.

On Fri, 2001-12-07 at 23:41, Dave Jones wrote:
> On 7 Dec 2001, James Davies wrote:
> 
> > source code IS available for free under a restrictive license. the GLX
> > drivers are closed.
> 
> Bzzzt...
> 
> >From the tarball you posted a link to..
> 
> -rwxr-xr-x    1 davej    users      889615 Nov 27 20:39 Module-nvkernel*
> 
> (davej@noodles:NVIDIA_kernel-1.0-2313)$ file Module-nvkernel
> Module-nvkernel: ELF 32-bit LSB relocatable, Intel 80386, version 1, not
> stripped
> 
> No-one but nvidia knows whats in this.
> 
> regards,
> Dave.
> 
> -- 
> | Dave Jones.        http://www.codemonkey.org.uk
> | SuSE Labs



_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

Re: kernel: ldt allocation failed

[Alan Cox]

> > The Nvidia drivers are not open source. Please stop pedalling this myth. All
> > you are doing is risking confusing other unfortunates into buying nvidia
> > hardware and finding there is no open 3D support.
> 
> I wan't trying to state that it is GPL or similar, but the kernel driver
> source code IS available for free under a restrictive license. the GLX
> drivers are closed. 

No. The kernel driver source code is not available. Stop trying to mislead
people.

Alan

Re: kernel: ldt allocation failed

[Anton Altaparmakov]

At 09:15 07/12/01, Alan Cox wrote:
> > I don't see how one can magically tell that this is an NVIDIA problem.
>
>We don't know. But since we don't have their source and they have our
>source only they can tell you.

Not doing any debugging is fine.

However, looking at 2.4.16/arch/i386/kernel/process.c::copy_segments() 
which generates this message it seems odd: It returns void, yet it can fail 
because it is doing a vmalloc().

When the vmalloc() fails, the new_mm->context.segments is set to NULL and 
the function returns.

That seems wrong, no? Shouldn't there be a panic() when the allocation 
fails at least? Or even better the function should perhaps return an error 
code?

Considering there is only one caller (kernel/fork.c::copy_mm()) it would be 
easy to modify copy_mm() to handle a returned error code gracefully and 
goto fail_nomem, which would in turn result in kernel/fork.c::do_fork(), 
the only caller of copy_mm(), cleaning up properly and returning an error code.

Or have I missed something and the situation where the ldt is missing can 
be recovered from? - I would think (without looking into this in the kernel 
code) that loosing the local descriptor table would be rather detrimental 
on the first context switch to the new process created by fork... And 
considering all kinds of errors are being handled in this code path, except 
for the vmalloc() failure, it seems like a good idea to add the appropriate 
fail mechanism.

If nvidia is causing this to get triggered they will likely run into 
problems elsewhere anyway and we don't care but we should get the kernel 
working. As it is AFAICS we have a potential DOS, just get the box close to 
OOM and start calling man 2 fork and/or man 3 clone and you could trigger 
this with a finite probability.

Best regards,

         Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: kernel: ldt allocation failed

[Alan Cox]

> When the vmalloc() fails, the new_mm->context.segments is set to NULL and 
> the function returns.
> 
> That seems wrong, no? Shouldn't there be a panic() when the allocation 
> fails at least? Or even better the function should perhaps return an error 
> code?
> 
> Considering there is only one caller (kernel/fork.c::copy_mm()) it would be 
> easy to modify copy_mm() to handle a returned error code gracefully and 

That sounds like an appropriate change.

Re: kernel: ldt allocation failed

[Denis Vlasenko]

On 7 December 2001 07:45, Anton Altaparmakov wrote:
> However, looking at 2.4.16/arch/i386/kernel/process.c::copy_segments()
> which generates this message it seems odd: It returns void, yet it can fail
> because it is doing a vmalloc().
>
> When the vmalloc() fails, the new_mm->context.segments is set to NULL and
> the function returns.
>
> That seems wrong, no? Shouldn't there be a panic() when the allocation
> fails at least? Or even better the function should perhaps return an error
> code?
>
> Considering there is only one caller (kernel/fork.c::copy_mm()) it would be
> easy to modify copy_mm() to handle a returned error code gracefully and
> goto fail_nomem, which would in turn result in kernel/fork.c::do_fork(),
> the only caller of copy_mm(), cleaning up properly and returning an error
> code.

I am ignorant on the subject, but why LDT is used in Linux at all?
LDT register can be set to 0, this can speed up task switch time and save 
some memory used for LDT.

I see a i386 specific syscall (kernel/ldt.c:sys_modify_ldt()) and 
        /*
         * default LDT is a single-entry callgate to lcall7 for iBCS
         * and a callgate to lcall27 for Solaris/x86 binaries
         */
        set_call_gate(&default_ldt[0],lcall7);
        set_call_gate(&default_ldt[4],lcall27);
in kernel/traps.c:trap_init().

Is it used elsewhere?
--
vda

Re: kernel: ldt allocation failed

[Alan Cox]

> I am ignorant on the subject, but why LDT is used in Linux at all?
> LDT register can be set to 0, this can speed up task switch time and save 
> some memory used for LDT.

Wine and certain threaded applications

Re: kernel: ldt allocation failed

[Christoph Hellwig]

In article <E16YSLg-00056Z-00@the-village.bc.nu> you wrote:
>> I am ignorant on the subject, but why LDT is used in Linux at all?
>> LDT register can be set to 0, this can speed up task switch time and save 
>> some memory used for LDT.
>
> Wine and certain threaded applications

Xenix/286 binary emulation.

Re: kernel: ldt allocation failed

[Anton Altaparmakov]

At 09:45 07/12/01, Anton Altaparmakov wrote:
>At 09:15 07/12/01, Alan Cox wrote:
>> > I don't see how one can magically tell that this is an NVIDIA problem.
>>
>>We don't know. But since we don't have their source and they have our
>>source only they can tell you.
>
>Not doing any debugging is fine.
>
>However, looking at 2.4.16/arch/i386/kernel/process.c::copy_segments() 
>which generates this message it seems odd: It returns void, yet it can 
>fail because it is doing a vmalloc().
>
>When the vmalloc() fails, the new_mm->context.segments is set to NULL and 
>the function returns.
>
>That seems wrong, no? Shouldn't there be a panic() when the allocation 
>fails at least? Or even better the function should perhaps return an error 
>code?
>
>Considering there is only one caller (kernel/fork.c::copy_mm()) it would 
>be easy to modify copy_mm() to handle a returned error code gracefully and 
>goto fail_nomem, which would in turn result in kernel/fork.c::do_fork(), 
>the only caller of copy_mm(), cleaning up properly and returning an error code.
>
>Or have I missed something and the situation where the ldt is missing can 
>be recovered from? - I would think (without looking into this in the 
>kernel code) that loosing the local descriptor table would be rather 
>detrimental on the first context switch to the new process created by 
>fork... And considering all kinds of errors are being handled in this code 
>path, except for the vmalloc() failure, it seems like a good idea to add 
>the appropriate fail mechanism.
>
>If nvidia is causing this to get triggered they will likely run into 
>problems elsewhere anyway and we don't care but we should get the kernel 
>working. As it is AFAICS we have a potential DOS, just get the box close 
>to OOM and start calling man 2 fork and/or man 3 clone and you could trigger

s/man 3 clone/man 2 clone/

>  this with a finite probability.
>
>Best regards,
>
>         Anton
>
>
>--
>   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
>--
>Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
>Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
>ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Linux NTFS Maintainer / WWW: http://linux-ntfs.sf.net/
ICQ: 8561279 / WWW: http://www-stu.christs.cam.ac.uk/~aia21/

Re: kernel: ldt allocation failed

[Andi Kleen]

Denis Vlasenko <vda@port.imtp.ilyichevsk.odessa.ua> writes:
> 
> I am ignorant on the subject, but why LDT is used in Linux at all?
> LDT register can be set to 0, this can speed up task switch time and save 
> some memory used for LDT.

glibc thread local data uses an LDT for the segment register.

glibc 2.3 seems to plan to use segment register based thread local data for 
even non threaded programs, so it would be a good idea to optimize LDT 
allocation a bit (= not allocate 64K of vmalloc space every time 
sys_modify_ldt is called - there is only 8MB of it) 

-Andi

Re: kernel: ldt allocation failed

[Dave Jones]

On Wed, Feb 06, 2002 at 02:19:07PM +0100, Andi Kleen wrote:

 > glibc 2.3 seems to plan to use segment register based thread local data for 
 > even non threaded programs, so it would be a good idea to optimize LDT 
 > allocation a bit (= not allocate 64K of vmalloc space every time 
 > sys_modify_ldt is called - there is only 8MB of it) 

 Manfred Spraul had a patch that did this.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: kernel: ldt allocation failed

[Alan Cox]

> glibc 2.3 seems to plan to use segment register based thread local data for 
> even non threaded programs, so it would be a good idea to optimize LDT 
> allocation a bit (= not allocate 64K of vmalloc space every time 
> sys_modify_ldt is called - there is only 8MB of it) 

I think it would be a good idea to modify the glibc authors in that case.
The ldt costs real performance on task switches. It would be very dumb of
glibc to use it except when justified in the bigger picture - ie threaded
apps

Re: kernel: ldt allocation failed

[Andi Kleen]

On Wed, Feb 06, 2002 at 02:13:45PM +0000, Alan Cox wrote:
> > glibc 2.3 seems to plan to use segment register based thread local data for 
> > even non threaded programs, so it would be a good idea to optimize LDT 
> > allocation a bit (= not allocate 64K of vmalloc space every time 
> > sys_modify_ldt is called - there is only 8MB of it) 
> 
> I think it would be a good idea to modify the glibc authors in that case.
> The ldt costs real performance on task switches. It would be very dumb of
> glibc to use it except when justified in the bigger picture - ie threaded
> apps

Are you sure it does? LGDT with non zero argument shouldn't be that costly. 
The %fs switching adds some locked cycles for reloading the segment cache, 
but because Windows uses that I would it expect to be reasonably optimized 
on CPUs. 

I actually tried to complain because on x86-64 it is more costly, but to
no avail. 

-Andi

Re: kernel: ldt allocation failed

[Alan Cox]

> Are you sure it does? LGDT with non zero argument shouldn't be that costly. 
> The %fs switching adds some locked cycles for reloading the segment cache, 
> but because Windows uses that I would it expect to be reasonably optimized 
> on CPUs. 

Its measurable, even on an Athlon. 

> I actually tried to complain because on x86-64 it is more costly, but to
> no avail. 

The more I see from glibc the more I realise that Linus is right - it needs
replacing 

Re: kernel: ldt allocation failed

[bert hubert]

On Wed, Feb 06, 2002 at 02:34:06PM +0000, Alan Cox wrote:

> The more I see from glibc the more I realise that Linus is right - it needs
> replacing 

Felix Leitner, the diet-libc man, loves to point out the difference between
his popen.c and the one in glibc. The dietlibc version is obviously correct
and 37 lines. The glibc version is 337 lines and full of goobledygook.

Regards,

bert

-- 
http://www.PowerDNS.com          Versatile DNS Software & Services
http://www.tk                              the dot in .tk
Netherlabs BV / Rent-a-Nerd.nl           - Nerd Available -
Linux Advanced Routing & Traffic Control: http://ds9a.nl/lartc

Re: kernel: ldt allocation failed

[Bill Davidsen]

On Wed, 6 Feb 2002, Alan Cox wrote:

> I think it would be a good idea to modify the glibc authors in that case.

  Did you mean "notify," or are you REALLY serious about this ;-)

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: kernel: ldt allocation failed

[Alan Cox]

> On Wed, 6 Feb 2002, Alan Cox wrote:
> > I think it would be a good idea to modify the glibc authors in that case.
>   Did you mean "notify," or are you REALLY serious about this ;-)

Can we have a kernel FAQ entry about non-US humour 8)

Actually Jakub has already replied to my email with a glibc folks paper
on the thread stuff (confusingly called tls just to confuse everyone who
reads internet standards) and useful discussion can I hope go from that
point.

Alan

Re: kernel: ldt allocation failed

[John Levon]

On Wed, Feb 06, 2002 at 04:08:45PM +0000, Alan Cox wrote:

> Actually Jakub has already replied to my email with a glibc folks paper
> on the thread stuff (confusingly called tls just to confuse everyone who

where can we find this ?

thanks
john

-- 
"Mathemeticians stand on each other's shoulders while computer scientists
 stand on each other's toes."
	- Richard Hamming

Re: kernel: ldt allocation failed

[Alan Cox]

> where can we find this ?
http://people.redhat.com/drepper/tls.pdf

Re: kernel: ldt allocation failed

[Denis Vlasenko]

On 6 February 2002 11:19, Andi Kleen wrote:
> Denis Vlasenko <vda@port.imtp.ilyichevsk.odessa.ua> writes:
> > I am ignorant on the subject, but why LDT is used in Linux at all?
> > LDT register can be set to 0, this can speed up task switch time and save
> > some memory used for LDT.
>
> glibc thread local data uses an LDT for the segment register.
>
> glibc 2.3 seems to plan to use segment register based thread local data for
> even non threaded programs, so it would be a good idea to optimize LDT
> allocation a bit (= not allocate 64K of vmalloc space every time
> sys_modify_ldt is called - there is only 8MB of it)

What do they use on arches without LDT or equivalent?
--
vda

Re: kernel: ldt allocation failed

[Jakub Jelinek]

On Wed, Feb 06, 2002 at 04:02:25PM -0200, Denis Vlasenko wrote:
> On 6 February 2002 11:19, Andi Kleen wrote:
> > Denis Vlasenko <vda@port.imtp.ilyichevsk.odessa.ua> writes:
> > > I am ignorant on the subject, but why LDT is used in Linux at all?
> > > LDT register can be set to 0, this can speed up task switch time and save
> > > some memory used for LDT.
> >
> > glibc thread local data uses an LDT for the segment register.
> >
> > glibc 2.3 seems to plan to use segment register based thread local data for
> > even non threaded programs, so it would be a good idea to optimize LDT
> > allocation a bit (= not allocate 64K of vmalloc space every time
> > sys_modify_ldt is called - there is only 8MB of it)
> 
> What do they use on arches without LDT or equivalent?

Most sane architectures reserve a thread pointer register (%g6 resp. %g7 on
sparc, tp on ia64, ppc will use %r2, alpha uses a fast pall call as thread
"register", s390 uses user access register 0 (and s390x uar 0 and 1), etc.).
On register starved ia32 there aren't too many spare registers, so %gs is
used instead.

	Jakub

Re: kernel: ldt allocation failed

[Maciej W. Rozycki]

On Wed, 6 Feb 2002, Jakub Jelinek wrote:

> Most sane architectures reserve a thread pointer register (%g6 resp. %g7 on
> sparc, tp on ia64, ppc will use %r2, alpha uses a fast pall call as thread
> "register", s390 uses user access register 0 (and s390x uar 0 and 1), etc.).
> On register starved ia32 there aren't too many spare registers, so %gs is
> used instead.

 Actually really sane architectures, such as MIPS, have no unused
registers floating around just in case someone needs one in the next ten
years or so.  They require an ABI change which can only be justified if
the benefit is large.  So far I failed to see the benefit, but hopefully
it's only a fault of mine.

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: kernel: ldt allocation failed

[H. Peter Anvin]

Followup to:  <20020206101231.X21624@devserv.devel.redhat.com>
By author:    Jakub Jelinek <jakub@redhat.com>
In newsgroup: linux.dev.kernel
> 
> Most sane architectures reserve a thread pointer register (%g6 resp. %g7 on
> sparc, tp on ia64, ppc will use %r2, alpha uses a fast pall call as thread
> "register", s390 uses user access register 0 (and s390x uar 0 and 1), etc.).
> On register starved ia32 there aren't too many spare registers, so %gs is
> used instead.
> 

x86-64, interestingly, retains vestigial meaning of the %fs and %gs
registers (but no others) to use as a base pointer for this reason
alone.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: kernel: ldt allocation failed

[Jakub Jelinek]

On Wed, Feb 06, 2002 at 12:19:37PM -0800, H. Peter Anvin wrote:
> Followup to:  <20020206101231.X21624@devserv.devel.redhat.com>
> By author:    Jakub Jelinek <jakub@redhat.com>
> In newsgroup: linux.dev.kernel
> > 
> > Most sane architectures reserve a thread pointer register (%g6 resp. %g7 on
> > sparc, tp on ia64, ppc will use %r2, alpha uses a fast pall call as thread
> > "register", s390 uses user access register 0 (and s390x uar 0 and 1), etc.).
> > On register starved ia32 there aren't too many spare registers, so %gs is
> > used instead.
> > 
> 
> x86-64, interestingly, retains vestigial meaning of the %fs and %gs
> registers (but no others) to use as a base pointer for this reason
> alone.

Well, on x86-64 this is purely x86-64 ABI designers decision, they could
pick one of %r8 - %r15 and use that as thread pointer instead (and were
recommended to do so).

	Jakub

Re: kernel: ldt allocation failed

[H. Peter Anvin]

Jakub Jelinek wrote:

>>>
>>x86-64, interestingly, retains vestigial meaning of the %fs and %gs
>>registers (but no others) to use as a base pointer for this reason
>>alone.
> 
> Well, on x86-64 this is purely x86-64 ABI designers decision, they could
> pick one of %r8 - %r15 and use that as thread pointer instead (and were
> recommended to do so).
> 


Wasting an architectural register is still bloody expensive, even if 1
among 16 is less than 1 among 8.  They'll be using %gs as thread pointer
in userspace and CPU pointer in kernel space.

	-hpa

Re: kernel: ldt allocation failed

[yodaiken]

On Wed, Feb 06, 2002 at 10:12:31AM -0500, Jakub Jelinek wrote:
> Most sane architectures reserve a thread pointer register (%g6 resp. %g7 on
> sparc, tp on ia64, ppc will use %r2, alpha uses a fast pall call as thread
> "register", s390 uses user access register 0 (and s390x uar 0 and 1), etc.).
> On register starved ia32 there aren't too many spare registers, so %gs is
> used instead.

So the x86 designers have provided all sorts of shadow registers and extensive
high speed caches and the glibc developers deliberately choose to defeat all that
expensive optimization?

Re: kernel: ldt allocation failed

[H. Peter Anvin]

Followup to:  <20020206132144.A29162@hq.fsmlabs.com>
By author:    yodaiken@fsmlabs.com
In newsgroup: linux.dev.kernel
>
> On Wed, Feb 06, 2002 at 10:12:31AM -0500, Jakub Jelinek wrote:
> > Most sane architectures reserve a thread pointer register (%g6 resp. %g7 on
> > sparc, tp on ia64, ppc will use %r2, alpha uses a fast pall call as thread
> > "register", s390 uses user access register 0 (and s390x uar 0 and 1), etc.).
> > On register starved ia32 there aren't too many spare registers, so %gs is
> > used instead.
> 
> So the x86 designers have provided all sorts of shadow registers and extensive
> high speed caches and the glibc developers deliberately choose to defeat all that
> expensive optimization?
> 

Uhm... none of that "expensive optimization" will help you here,
because you need *architectural storage*.  Archtectural storage is
always a fundamentally limited resource, regardless of how many shadow
registers you add.

However, an x86 has a whole additional register file which is rarely
used these days -- the segment register file.  The segment register
file is mainly useful when the main usage is address offsetting, since
it provides an extra input into the address adder.  For this
particular purpose, using it is very much the sane thing to do.

As far as %gs switching is concerned, using %gs doesn't automatically
mean using the LDT.  There are two ways you can avoid setting up LDTs
in single-threaded apps, and still allow an ABI compatible with the
threaded apps:

a) For single-threaded apps, define %gs == %ds.  Less than ideal for
   several reasons, but no kernel mods needed.

b) Have the kernel provide another GDT value which can be used by the
   single-threaded apps.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: kernel: ldt allocation failed

[Jakub Jelinek]

On Wed, Feb 06, 2002 at 01:00:12PM -0800, H. Peter Anvin wrote:
> Uhm... none of that "expensive optimization" will help you here,
> because you need *architectural storage*.  Archtectural storage is
> always a fundamentally limited resource, regardless of how many shadow
> registers you add.
> 
> However, an x86 has a whole additional register file which is rarely
> used these days -- the segment register file.  The segment register
> file is mainly useful when the main usage is address offsetting, since
> it provides an extra input into the address adder.  For this
> particular purpose, using it is very much the sane thing to do.
> 
> As far as %gs switching is concerned, using %gs doesn't automatically
> mean using the LDT.  There are two ways you can avoid setting up LDTs
> in single-threaded apps, and still allow an ABI compatible with the
> threaded apps:
> 
> a) For single-threaded apps, define %gs == %ds.  Less than ideal for
>    several reasons, but no kernel mods needed.

This is not possible, since then %gs:0 (which is TLS base) cannot be read.
We would have to change the TLS ABI (thus become incompatible e.g. with Sun)
- note that these sequences are emitted by compiler or linker (the latter in
code transformations) and pick some hardcoded constant, say %gs:0x1000.
This would mean though that every single non-threaded app would need to
mmap a page at 4KB. If this offset was not constant, it would slow down all
TLS accesses.

> b) Have the kernel provide another GDT value which can be used by the
>    single-threaded apps.

Like above, a fixed address for mmap would have to be chosen, but the
advantage would be that the TLS ABI would need no changing.
Simply kernel would add 0x33 to GDT as 4KB -> 0xc0000000 user data segment
and apps could put that value into %gs if not using threads.

But I think there is c) and d).
c) is just minor modification of current ldt handling in kernel, which would
   mean a single entry LDT (residing in task_struct) could be used instead
   of vmalloced one - this has the disadvantage of lldt on almost every
   context switch
d) default to a single-entry per-cpu LDT, which only non-linux personality
   apps and apps needing more than 1 LDT entry (threaded apps, wine/dosemu/...)
   would not use. Non-linux apps would use current default_ldt and those
   needing more than one LDT would use the current vmalloced mm private area.
   If a task would be using this per-cpu LDT (common case), context switch
   would do lldt only if previous task was not using the per-cpu LDT
   (unlikely) and just store task_struct->thread.ldt_word_0 and ldt_word_1
   into the per-cpu LDT (dunno how expensive is that, but IMHO it should be
   cheaper than full load_LDT).

	Jakub

Re: kernel: ldt allocation failed

[H. Peter Anvin]

Jakub Jelinek wrote:

> 
>>b) Have the kernel provide another GDT value which can be used by the
>>   single-threaded apps.
>>
> 
> Like above, a fixed address for mmap would have to be chosen, but the
> advantage would be that the TLS ABI would need no changing.
> Simply kernel would add 0x33 to GDT as 4KB -> 0xc0000000 user data segment
> and apps could put that value into %gs if not using threads.
> 
> But I think there is c) and d).
> c) is just minor modification of current ldt handling in kernel, which would
>    mean a single entry LDT (residing in task_struct) could be used instead
>    of vmalloced one - this has the disadvantage of lldt on almost every
>    context switch
> d) default to a single-entry per-cpu LDT, which only non-linux personality
>    apps and apps needing more than 1 LDT entry (threaded apps, wine/dosemu/...)
>    would not use. Non-linux apps would use current default_ldt and those
>    needing more than one LDT would use the current vmalloced mm private area.
>    If a task would be using this per-cpu LDT (common case), context switch
>    would do lldt only if previous task was not using the per-cpu LDT
>    (unlikely) and just store task_struct->thread.ldt_word_0 and ldt_word_1
>    into the per-cpu LDT (dunno how expensive is that, but IMHO it should be
>    cheaper than full load_LDT).
> 


Actually d) is probably better done by allowing CPUs to put *one* entry in
the GDT instead of requesting an LDT.  Since the LDT takes up a GDT entry
anyway, this should be simple enough.

	-hpa

Re: kernel: ldt allocation failed

[Jakub Jelinek]

On Wed, Feb 06, 2002 at 02:05:52PM -0800, H. Peter Anvin wrote:
> >>b) Have the kernel provide another GDT value which can be used by the
> >>   single-threaded apps.
> >>
> > 
> > Like above, a fixed address for mmap would have to be chosen, but the
> > advantage would be that the TLS ABI would need no changing.
> > Simply kernel would add 0x33 to GDT as 4KB -> 0xc0000000 user data segment
> > and apps could put that value into %gs if not using threads.
> > 
> > But I think there is c) and d).
> > c) is just minor modification of current ldt handling in kernel, which would
> >    mean a single entry LDT (residing in task_struct) could be used instead
> >    of vmalloced one - this has the disadvantage of lldt on almost every
> >    context switch
> > d) default to a single-entry per-cpu LDT, which only non-linux personality
> >    apps and apps needing more than 1 LDT entry (threaded apps, wine/dosemu/...)
> >    would not use. Non-linux apps would use current default_ldt and those
> >    needing more than one LDT would use the current vmalloced mm private area.
> >    If a task would be using this per-cpu LDT (common case), context switch
> >    would do lldt only if previous task was not using the per-cpu LDT
> >    (unlikely) and just store task_struct->thread.ldt_word_0 and ldt_word_1
> >    into the per-cpu LDT (dunno how expensive is that, but IMHO it should be
> >    cheaper than full load_LDT).
> > 
> 
> 
> Actually d) is probably better done by allowing CPUs to put *one* entry in
> the GDT instead of requesting an LDT.  Since the LDT takes up a GDT entry
> anyway, this should be simple enough.

On UP sure, on SMP you'd need to allocate as many GDT entries as there are
CPUs and special case this in __switch_to too (something like
        loadsegment(fs, next->fs);
	if ((next->gs & 7) == 3 && next->gs >= 0x30 && next->gs < 0x30 + NR_CPUS * 8)
	  next->gs = smp_processor_id() * 8 + 0x33;
        loadsegment(gs, next->gs);
) which is kind of ugly (and userland must not save/restore %gs itself then).
Unlike d) with LDT, where unmodified glibc could work with older kernels
too, thus would mean strict kernel minimum version requirement (with LDT d)
it would be just an optimization).

	Jakub

Re: kernel: ldt allocation failed

[H. Peter Anvin]

Followup to:  <20020206172025.H21624@devserv.devel.redhat.com>
By author:    Jakub Jelinek <jakub@redhat.com>
In newsgroup: linux.dev.kernel
>
> Unlike d) with LDT, where unmodified glibc could work with older kernels
> too, thus would mean strict kernel minimum version requirement (with LDT d)
> it would be just an optimization).
> 

Just make it a fallback option.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: kernel: ldt allocation failed

[Alan Cox]

> This is not possible, since then %gs:0 (which is TLS base) cannot be read.
> We would have to change the TLS ABI (thus become incompatible e.g. with Sun)

Sun who have canned their x86 product it seems. I don't feel "the standard
requires we suck" is an appropriate justification for anything. If there 
is not a sane way to follow the standard - break it. If there is a sane way
then all fair and good, find it and use it


Alan

Re: kernel: ldt allocation failed

[Jakub Jelinek]

On Thu, Feb 07, 2002 at 12:21:22AM +0000, Alan Cox wrote:
> > This is not possible, since then %gs:0 (which is TLS base) cannot be read.
> > We would have to change the TLS ABI (thus become incompatible e.g. with Sun)
> 
> Sun who have canned their x86 product it seems. I don't feel "the standard
> requires we suck" is an appropriate justification for anything. If there 
> is not a sane way to follow the standard - break it. If there is a sane way
> then all fair and good, find it and use it

We have changed it already (e.g,. for regparm(1), fewer relocs, shorter insn
sequences, etc.), but with exception of 2 non-dynamic relocs (which get
different numbers) we are still compatible.
But as written later, just using a different GDT descriptor could avoid
having to change the ABI, but would still have the undesirable property of
requiring every app to mmap a new page at fixed location.

	Jakub

Re: kernel: ldt allocation failed

[H. Peter Anvin]

Jakub Jelinek wrote:

> 
> We have changed it already (e.g,. for regparm(1), fewer relocs, shorter insn
> sequences, etc.), but with exception of 2 non-dynamic relocs (which get
> different numbers) we are still compatible.
> But as written later, just using a different GDT descriptor could avoid
> having to change the ABI, but would still have the undesirable property of
> requiring every app to mmap a new page at fixed location.

>

It's not that bad, though, especially if your fixed location is just
beneath the standard mmap base address (e.g. 0x3f000000 -- leave some room
for future expansion -- or thereabouts on the standard 3:1 split space.)

Do you support tlsalloc() or whatever it is called?

	-hpa

Re: kernel: ldt allocation failed

[H. Peter Anvin]

Alan Cox wrote:

>>This is not possible, since then %gs:0 (which is TLS base) cannot be read.
>>We would have to change the TLS ABI (thus become incompatible e.g. with Sun)
>>
> 
> Sun who have canned their x86 product it seems. I don't feel "the standard
> requires we suck" is an appropriate justification for anything. If there 
> is not a sane way to follow the standard - break it. If there is a sane way
> then all fair and good, find it and use it
> 


I do have to agree that zero-basing the TLS pointer seems saner than not
doing so.

	-hpa

Re: kernel: ldt allocation failed

[Alan Cox]

> > allocation a bit (= not allocate 64K of vmalloc space every time
> > sys_modify_ldt is called - there is only 8MB of it)
> 
> What do they use on arches without LDT or equivalent?

Generally on such platforms you have enough registers to use a register
for your thread specific storage. In fact even the kernel does that - you'll
find 'current' on some platforms is a global register variable

kernel: ldt allocation failed

[Kiril Vidimce]

We suddenly started seeing freezing problems on a number of machines
in the past couple of days. There is no pattern as far as I can tell.
It has happened while running OpenGL apps, netscape, or even when not 
doing anything. The machine will usually just hang and while it's 
still pingable, it's totally unresponsive and you can't remotely log in.
The 2.4.3 kernel usually hangs forever; the machines with 2.4.9
kernels usually come back within 10-15 secs.

Every time this happens we see the following message in the system log:

Dec  6 21:29:00 stranger kernel: ldt allocation failed

The machines are:

Hardware:
    - IBM Intellistation M Pro 
    - dual 2 GHz P4's
    - 2 GB RAM
    - NVIDIA Quadro DCC card

Software:
    - Red Hat 7.1
    - kernel 2.4.3smp or 2.4.9smp
    - XFree 4.1
    - Ximian Gnome 1.4
    - NVIDIA drivers 1.0-1541

Once this problem occurs, even if the hang is temporary, the machine
is extremely flakey. Almost any app will start causing ldt allocation
failure messages in the system log. Only a reboot really helps.

Questions:
- Does this ring a bell to anybody? What is ldt anyway and what would 
  cause an allocation to fail?

- I still keep one of these messages in this state. Is there anything
  I can probe to get useful debugging info?

Any help would greatly be appreciated.

Thanks,
KV
--
  ___________________________________________________________________
  Studio Tools                                        vkire@pixar.com
  Pixar Animation Studios                        http://www.pixar.com/

Re: kernel: ldt allocation failed

[Jeffrey H. Ingber]

This problem is very familiar.  It was fixed in an -ac patch against
2.4.9 (although I don't recall the exact one), and is included in
subsequent Linus kernels.  Without this fix, I experienced similar
problems with X on multiprocessor workstations.

Jeffrey H. Ingber (jhingber _at_ ix.netcom.com)

On Fri, 2001-12-07 at 00:40, Kiril Vidimce wrote:
> 
> We suddenly started seeing freezing problems on a number of machines
> in the past couple of days. There is no pattern as far as I can tell.
> It has happened while running OpenGL apps, netscape, or even when not 
> doing anything. The machine will usually just hang and while it's 
> still pingable, it's totally unresponsive and you can't remotely log in.
> The 2.4.3 kernel usually hangs forever; the machines with 2.4.9
> kernels usually come back within 10-15 secs.
> 
> Every time this happens we see the following message in the system log:
> 
> Dec  6 21:29:00 stranger kernel: ldt allocation failed
> 
> The machines are:
> 
> Hardware:
>     - IBM Intellistation M Pro 
>     - dual 2 GHz P4's
>     - 2 GB RAM
>     - NVIDIA Quadro DCC card
> 
> Software:
>     - Red Hat 7.1
>     - kernel 2.4.3smp or 2.4.9smp
>     - XFree 4.1
>     - Ximian Gnome 1.4
>     - NVIDIA drivers 1.0-1541
> 
> Once this problem occurs, even if the hang is temporary, the machine
> is extremely flakey. Almost any app will start causing ldt allocation
> failure messages in the system log. Only a reboot really helps.
> 
> Questions:
> - Does this ring a bell to anybody? What is ldt anyway and what would 
>   cause an allocation to fail?
> 
> - I still keep one of these messages in this state. Is there anything
>   I can probe to get useful debugging info?
> 
> Any help would greatly be appreciated.
> 
> Thanks,
> KV
> --
>   ___________________________________________________________________
>   Studio Tools                                        vkire@pixar.com
>   Pixar Animation Studios                        http://www.pixar.com/
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/