booting a kernel compiled with -mregparm=0

booting a kernel compiled with -mregparm=0

[Tigran Aivazian]

Hello,

I am trying to boot a 2.6.x kernel (on x86_64) compiled with -mregparm=0 
and it does not boot, i.e. hangs at the very first stage.

I know this breaks ABI/x86_64 but the reason I need to compile such a 
kernel is because kdb on x86_64 cannot show the function arguments and the 
only way to make it work that I found was to pass all arguments on the 
stack (then kdb works fine and shows correct values for all arguments). 
But obviously the module and the kernel need to match, otherwise it will 
panic easily; and so I have to use the same argument passing convention in 
the kernel. This is obviously for debugging only, nobody would ever ship 
such "incorrectly" compiled module anywhere.

So, I have to find a "boundary" between the parts of the kernel that can 
be safely compiled with -mregparm=0 and which must stay as they are. Any 
ideas as to what to do in this situation?

Kind regards
Tigran

Re: [discuss] booting a kernel compiled with -mregparm=0

[Jan Hubicka]

> Hello,
> 
> I am trying to boot a 2.6.x kernel (on x86_64) compiled with -mregparm=0 
> and it does not boot, i.e. hangs at the very first stage.
> 
> I know this breaks ABI/x86_64 but the reason I need to compile such a 
> kernel is because kdb on x86_64 cannot show the function arguments and the 
> only way to make it work that I found was to pass all arguments on the 
> stack (then kdb works fine and shows correct values for all arguments). 
> But obviously the module and the kernel need to match, otherwise it will 
> panic easily; and so I have to use the same argument passing convention in 
> the kernel. This is obviously for debugging only, nobody would ever ship 
> such "incorrectly" compiled module anywhere.
> 
> So, I have to find a "boundary" between the parts of the kernel that can 
> be safely compiled with -mregparm=0 and which must stay as they are. Any 
> ideas as to what to do in this situation?

Actually -mregparm=0 is not supposed to be even accepted by x86-64
compiler (I've disabled the function attribute but apparently missed
this one) and even if GCC produced valid code by miracle, you will get
into trouble with hand written assembly.

Honza
> 
> Kind regards
> Tigran
> 

Re: [discuss] booting a kernel compiled with -mregparm=0

[linux-os]

On Fri, 14 Jan 2005, Jan Hubicka wrote:

>> Hello,
>>
>> I am trying to boot a 2.6.x kernel (on x86_64) compiled with -mregparm=0
>> and it does not boot, i.e. hangs at the very first stage.
>>
>> I know this breaks ABI/x86_64 but the reason I need to compile such a
>> kernel is because kdb on x86_64 cannot show the function arguments and the
>> only way to make it work that I found was to pass all arguments on the
>> stack (then kdb works fine and shows correct values for all arguments).
>> But obviously the module and the kernel need to match, otherwise it will
>> panic easily; and so I have to use the same argument passing convention in
>> the kernel. This is obviously for debugging only, nobody would ever ship
>> such "incorrectly" compiled module anywhere.
>>
>> So, I have to find a "boundary" between the parts of the kernel that can
>> be safely compiled with -mregparm=0 and which must stay as they are. Any
>> ideas as to what to do in this situation?
>
> Actually -mregparm=0 is not supposed to be even accepted by x86-64
> compiler (I've disabled the function attribute but apparently missed
> this one) and even if GCC produced valid code by miracle, you will get
> into trouble with hand written assembly.

Huh? That's the default for a 'C' compiler (not to pass parameters
in registers). The parameters are passed on the stack as the default!
The return values don't count. They are, by default passed in eax
or edx-eax pair for a long long.

-mregparm=0 should therefore be a no-op unless overridden by
an __attribute__ macro.

I recall that Linus changed some code when it was discovered that
there was a bug in linux-2.6.8/arch/i386/kernel/semaphore.c

This forced some parameters to be passed in registers, using
the __attribute__((regparm(3))) macro in ../asm/linkage.h. This
might not be a GoodThing(tm) with x86_64.


>
> Honza
>>
>> Kind regards
>> Tigran
>>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.

Re: [discuss] booting a kernel compiled with -mregparm=0

[H. Peter Anvin]

Followup to:  <Pine.LNX.4.61.0501141613500.6747@chaos.analogic.com>
By author:    linux-os <linux-os@analogic.com>
In newsgroup: linux.dev.kernel
> >
> > Actually -mregparm=0 is not supposed to be even accepted by x86-64
> > compiler (I've disabled the function attribute but apparently missed
> > this one) and even if GCC produced valid code by miracle, you will get
> > into trouble with hand written assembly.
> 
> Huh? That's the default for a 'C' compiler (not to pass parameters
> in registers). The parameters are passed on the stack as the default!
> The return values don't count. They are, by default passed in eax
> or edx-eax pair for a long long.
> 

Dear Wrongbot,

It depends on the architecture ABI.  This is the case for the i386
ABI, but definitely *NOT* for x86-64.

	-hpa

Re: [discuss] booting a kernel compiled with -mregparm=0

[Tigran Aivazian]

On Sat, 15 Jan 2005, H. Peter Anvin wrote:
> It depends on the architecture ABI.  This is the case for the i386
> ABI, but definitely *NOT* for x86-64.

<begin stuff superseded by later findings, but left here to clarify the 
train of thoughts>
Yes, precisely. The ABI/x86_64 defines this behaviour explicitly. However, 
that would mean the ABI was designed without giving thought to debugging 
via kdb.

When I said "2-3 weeks of work" I didn't fully realize the complexity of 
the problem. It is actually more like several months of research work and 
then (most likely) coming to the conclusion that the code to simulate the 
cpu (by disassembling the functions to track down where those registers 
went in each function) is just too complex to be written.

So, this means there is no way for kdb on x86_64 to show the parameter 
values for each function in the back trace. Any chance of changing the 
ABI/x86_64 to do the right (i.e. passing via stack like on i386) thing 
now? Then kdb would automatically support it via normal ar-handling code.

Also, we should NOT claim that Linux has been ported to x86_64 
architecture yet, because the port is not clean, i.e. doesn't allow 
breaking ABI like it allows on i386, by means of CONFIG_REGPARM 
optimization. Anyone working on fixing this?

I cc'd Linus as I cannot believe he agreed with allowing such an 
optimization to be a default and standard thing accepted by the Linux 
kernel. (But I may be wrong, especially since Linus isn't particularly 
fond of kdb anyway :)
<end of stuff declared as superceded>

Actually, having cc'd Linus made me think very _carefully_ about what I 
say and I went and checked how the userspace does it, as I couldn't 
believe that such fine piece of software as gdb would be broken as well. 
And to my surprize I discovered that gdb (when a program is compiled with 
-g) works fine! I.e. it shows the function arguments correctly. And 
disassembling the functions shows that although the arguments are passed 
via registers (as ABI demands) they are also saved somewhere on the stack. 
Hmmm, interesting, then -g compiled Linux kernel should also be useable, 
with perhaps some tweaks to kdb to decode these frames correctly, right?

I just want someone who knows the answer to confirm the last statement and 
so then I can go ahead and (try to) implement the solution by mimicking 
gdb behaviour (or help Jack Vogel implement it if he prefers doing it 
himself as defacto kdb/x86_64 maintainer)

Kind regards
Tigran

Re: [discuss] booting a kernel compiled with -mregparm=0

[Arjan van de Ven]

On Mon, 2005-01-17 at 09:30 +0000, Tigran Aivazian wrote:
> I cc'd Linus as I cannot believe he agreed with allowing such an 
> optimization to be a default and standard thing accepted by the Linux 
> kernel. (But I may be wrong, especially since Linus isn't particularly 
> fond of kdb anyway :)

I don't see a problem, have you ever seen ia64??
> Actually, having cc'd Linus made me think very _carefully_ about what I 
> say and I went and checked how the userspace does it, as I couldn't 
> believe that such fine piece of software as gdb would be broken as well. 
> And to my surprize I discovered that gdb (when a program is compiled with 
> -g) works fine! I.e. it shows the function arguments correctly. And 


so why don't you use kgdb instead of kdb ?

Re: [discuss] booting a kernel compiled with -mregparm=0

[Tigran Aivazian]

On Mon, 17 Jan 2005, Arjan van de Ven wrote:
>> Actually, having cc'd Linus made me think very _carefully_ about what I
>> say and I went and checked how the userspace does it, as I couldn't
>> believe that such fine piece of software as gdb would be broken as well.
>> And to my surprize I discovered that gdb (when a program is compiled with
>> -g) works fine! I.e. it shows the function arguments correctly. And
>
>
> so why don't you use kgdb instead of kdb ?

If kdb was some dead unmaintained piece of software then, yes, I would 
follow your advice and switch to kgdb. But kdb is a very nice and actively 
maintained piece of work, so it should be fixed to show the parameter 
values correctly in the backtrace.

Kind regards
Tigran

Re: [discuss] booting a kernel compiled with -mregparm=0

[H. Peter Anvin]

Tigran Aivazian wrote:
> On Mon, 17 Jan 2005, Arjan van de Ven wrote:
> 
>>> Actually, having cc'd Linus made me think very _carefully_ about what I
>>> say and I went and checked how the userspace does it, as I couldn't
>>> believe that such fine piece of software as gdb would be broken as well.
>>> And to my surprize I discovered that gdb (when a program is compiled 
>>> with
>>> -g) works fine! I.e. it shows the function arguments correctly. And
>>
>> so why don't you use kgdb instead of kdb ?
> 
> If kdb was some dead unmaintained piece of software then, yes, I would 
> follow your advice and switch to kgdb. But kdb is a very nice and 
> actively maintained piece of work, so it should be fixed to show the 
> parameter values correctly in the backtrace.

That's a kdb maintainer issue.  The x86-64 folks have nicely provided a 
set of libraries to do backtraces, etc.  Your previous rant is just so 
far off base it's not even funny.

	-hpa

Re: [discuss] booting a kernel compiled with -mregparm=0

[Andi Kleen]

"H. Peter Anvin" <hpa@zytor.com> writes:

> Tigran Aivazian wrote:
>> On Mon, 17 Jan 2005, Arjan van de Ven wrote:
>>
>>>> Actually, having cc'd Linus made me think very _carefully_ about what I
>>>> say and I went and checked how the userspace does it, as I couldn't
>>>> believe that such fine piece of software as gdb would be broken as well.
>>>> And to my surprize I discovered that gdb (when a program is
>>>> compiled with
>>>> -g) works fine! I.e. it shows the function arguments correctly. And
>>>
>>> so why don't you use kgdb instead of kdb ?
>> If kdb was some dead unmaintained piece of software then, yes, I
>> would follow your advice and switch to kgdb. But kdb is a very nice
>> and actively maintained piece of work, so it should be fixed to show
>> the parameter values correctly in the backtrace.
>
> That's a kdb maintainer issue.  The x86-64 folks have nicely provided
> a set of libraries to do backtraces, etc.  Your previous rant is just
> so far off base it's not even funny.

To be fair there isn't a nice library for it on x86-64.  There
is libunwind on IA64, but afaik nobody ported it to x86-64 yet.

Just various projects have their own private unwind
implementation. The kernel including KDB has always lived with
imprecise backtraces and no argument printing. I don't think it has
been a show stopper so far.  If you really want the arguments you can
always use kgdb.

However I'm not sure we really want libunwind in the kernel anyways
(not even in KDB ;-) If anything better something stripped down and 
simple which libunwind isn't.

Unfortunately dwarf2 is not exactly a simple spec so implementing
a new backtracer for the kernel is not a trivial task. 

-Andi

Re: [discuss] booting a kernel compiled with -mregparm=0

[H. Peter Anvin]

Andi Kleen wrote:
> 
> To be fair there isn't a nice library for it on x86-64.  There
> is libunwind on IA64, but afaik nobody ported it to x86-64 yet.
> 
> Just various projects have their own private unwind
> implementation. The kernel including KDB has always lived with
> imprecise backtraces and no argument printing. I don't think it has
> been a show stopper so far.  If you really want the arguments you can
> always use kgdb.
> 
> However I'm not sure we really want libunwind in the kernel anyways
> (not even in KDB ;-) If anything better something stripped down and 
> simple which libunwind isn't.
> 
> Unfortunately dwarf2 is not exactly a simple spec so implementing
> a new backtracer for the kernel is not a trivial task. 
> 

Seems like the unwinder should be running client-side, like it does on 
kgdb.  Or does kdb not have a client at all?  (If so, I have no sympathy 
for it.)

	-hpa

Re: [discuss] booting a kernel compiled with -mregparm=0

[Andi Kleen]

> Seems like the unwinder should be running client-side, like it does on 
> kgdb.  Or does kdb not have a client at all?  (If so, I have no sympathy 
> for it.)

kdb runs 100% in the kernel.

-Andi

Re: [discuss] booting a kernel compiled with -mregparm=0

[Tigran Aivazian]

On Mon, 17 Jan 2005, H. Peter Anvin wrote:
> Or does kdb not have a client at all?  (If so, I have no sympathy for it.)

Peter, it was very easy for you to call my emails "rants" and "not even 
funny" but the above statement is displaying complete ignorance of what 
kdb actually is :) So, instead of "patronizing" your fellow-hacker, please 
listen to what he has to say below:

I already solved this paricular problem. And the solution is (but don't 
tell me you knew it, for then why didn't you tell anyone) simply --- 
compile the kernel with -g and that includes enough debug information to 
be able to decode the stack content correctly. And yes, kdb does show the 
correct argument values now. No changes to kdb are necessary and no need 
to do the work with dwarf2 implementation etc etc.

However, this highlighted a more serious problem in the x86_64 kernel (or 
more likely in the kdb patch) --- the kernel compiled with -g panics when 
you try to return from kdb after hitting a breakpoint. This is a bug and 
I'll investigate to find out the reason why it panics. (I hope it is not 
an "assumption" of the x86_64 port that one must never compile the kernel 
with -g either...)

Kind regards
Tigran

Re: [discuss] booting a kernel compiled with -mregparm=0

[Arjan van de Ven]

On Tue, 2005-01-18 at 11:25 +0000, Tigran Aivazian wrote:

> However, this highlighted a more serious problem in the x86_64 kernel (or 
> more likely in the kdb patch) --- the kernel compiled with -g panics when 
> you try to return from kdb after hitting a breakpoint. This is a bug and 
> I'll investigate to find out the reason why it panics. (I hope it is not 
> an "assumption" of the x86_64 port that one must never compile the kernel 
> with -g either...)

That is isn't; for example the Fedora Core 2 and 3 and the Red Hat
Enterprise Linux 4 kernels are compiled wit -g.

Re: [discuss] booting a kernel compiled with -mregparm=0

[Tigran Aivazian]

On Tue, 18 Jan 2005, Tigran Aivazian wrote:
> I already solved this paricular problem. And the solution is (but don't tell 
> me you knew it, for then why didn't you tell anyone) simply --- compile the 
> kernel with -g and that includes enough debug information to be able to 
> decode the stack content correctly. And yes, kdb does show the correct 
> argument values now. No changes to kdb are necessary and no need to do the 
> work with dwarf2 implementation etc etc.

actually I am very surprized that it worked (because looking at the code I 
concluded that it should NOT). So I need to retest in all cases to make 
sure this is not a coincidence but a solid fact...

Kind regards
Tigran

Re: [discuss] booting a kernel compiled with -mregparm=0

[Tigran Aivazian]

Sorry, it was a "red herring", namely my module's Makefile still had 
"-mregparm=0" in CFLAGS, so that is why kdb showed it's arguments 
correctly (and then paniced on the way out).

So we still have to deal with DWARF2 data then... Ok, will look into it.

Kind regards
Tigran

On Tue, 18 Jan 2005, Tigran Aivazian wrote:

> On Tue, 18 Jan 2005, Tigran Aivazian wrote:
>> I already solved this paricular problem. And the solution is (but don't 
>> tell me you knew it, for then why didn't you tell anyone) simply --- 
>> compile the kernel with -g and that includes enough debug information to be 
>> able to decode the stack content correctly. And yes, kdb does show the 
>> correct argument values now. No changes to kdb are necessary and no need to 
>> do the work with dwarf2 implementation etc etc.
>
> actually I am very surprized that it worked (because looking at the code I 
> concluded that it should NOT). So I need to retest in all cases to make sure 
> this is not a coincidence but a solid fact...
>
> Kind regards
> Tigran
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [discuss] booting a kernel compiled with -mregparm=0

[Andi Kleen]

Tigran Aivazian <tigran@veritas.com> writes:
>
> When I said "2-3 weeks of work" I didn't fully realize the complexity
> of the problem. It is actually more like several months of research
> work and then (most likely) coming to the conclusion that the code to
> simulate the cpu (by disassembling the functions to track down where
> those registers went in each function) is just too complex to be
> written.

Did you actually ever read the ABI? 

The ABI supported way is to read the DWARF2 unwind tables. For that
you would a dwarf2 reader.  gdb does that in user space, and libgcc2
also does it for exception unwinding. IA64 has an in kernel dwarf2
reader library (and ia64 kdb uses it), although it would probably need
some work to make it work on x86-64.

So far nobody wanted it enough to do the porting work though.


-Andi

Re: [discuss] booting a kernel compiled with -mregparm=0

[Tigran Aivazian]

Hi Andi,

On Mon, 17 Jan 2005, Andi Kleen wrote:
> The ABI supported way is to read the DWARF2 unwind tables. For that
> you would a dwarf2 reader.  gdb does that in user space, and libgcc2
> also does it for exception unwinding. IA64 has an in kernel dwarf2
> reader library (and ia64 kdb uses it), although it would probably need
> some work to make it work on x86-64.
>
> So far nobody wanted it enough to do the porting work though.

Thank you for the pointer. I will look at it and see if I can try to port 
it to x86_64, since that is the standard and official way. I admit that I 
didn't realize that the "magic stuff" which gdb does is in fact a dwarf2 
implementation (and this is what's missing in kdb on x86_64).

Kind regards
Tigran

Re: [discuss] booting a kernel compiled with -mregparm=0

[Linus Torvalds]

On Mon, 17 Jan 2005, Tigran Aivazian wrote:
> 
> So, this means there is no way for kdb on x86_64 to show the parameter 
> values for each function in the back trace. Any chance of changing the 
> ABI/x86_64 to do the right (i.e. passing via stack like on i386) thing 
> now? Then kdb would automatically support it via normal ar-handling code.

"right thing"? You have a very strange definition of "right". The x86 is 
pretty much the only architecture still in use that passes everything on 
the stack by default, and even there it's considered pretty painful (and 
is not true of FP arguments).

If a debugger cannot handle arguments in registers, it by definition 
cannot handle things like alpha/ppc/mips/xxx, so I'd say that the 
debugger is seriously broken.

		Linus

Re: [discuss] booting a kernel compiled with -mregparm=0

[Keith Owens]

On Mon, 17 Jan 2005 09:30:17 +0000 (GMT), 
Tigran Aivazian <tigran@veritas.com> wrote:
>Hmmm, interesting, then -g compiled Linux kernel should also be useable, 
>with perhaps some tweaks to kdb to decode these frames correctly, right?

kdb on i386 uses heuristics to guess at what parameters have been
passed on stack.  When the parameters are passed by register, there is
not enough information in the code to work out which parameters have
been passed nor what gcc has done with them once the function has been
entered.

Why use heuristics and guess?  Because when kdb was started we were
still using a.out and stabs.  In those days there was no way for code
in the running kernel to access the kernel's debugging information to
track the parameter and register usage.

Since then the world has moved on.  The IA64 ABI mandates that the
information required to do backtrace is stored in the running kernel,
and kdb uses that unwind data, including tracking parameters passed in
registers.

Nobody has been concerned enough about the backtraces on i386 and
x86_64 to add the required unwind data to the kernel for those
platforms.  If you want to extract the dwarf data from a kernel
compiled with -g, include the dwarf data in the running kernel and add
a dwarf unwinder to the kernel then I will happily accept patches to
kdb.  Don't forget about support for adding and removing unwind data as
modules are loaded and unloaded.

To hpa: kdb is designed to run completely in kernel.  It is not (repeat
not) intended to be a remote client debugger.

BTW, even on IA64 which has unwind data, we still get problems because
the unwind data only says what parameters are passed in registers, it
says nothing about register reuse.  gcc can reuse a parameter register
if the parameter value is no longer required, for example :-

  void function foo(struct bar *b)
  {
    // On entry, ia64 r32 contains *b
    int i = b->i;
    // If b is not used after this point, gcc can reuse r32 for the value of i.
    // Debugging after this point will show a misleading value in r32 for b.
    ...
  }

Re: [discuss] booting a kernel compiled with -mregparm=0

[Andi Kleen]

Keith Owens <kaos@sgi.com> writes:

> Nobody has been concerned enough about the backtraces on i386 and
> x86_64 to add the required unwind data to the kernel for those
> platforms.  If you want to extract the dwarf data from a kernel
> compiled with -g, include the dwarf data in the running kernel and add
> a dwarf unwinder to the kernel then I will happily accept patches to
> kdb.  Don't forget about support for adding and removing unwind data as

It would be pretty easy to do.

The x86-64 ABI actually includes unwind data (without other dwarf
data) by default in all executables. However it wasn't needed in the
kernel so far so I turned it off to save some disk space:

If you want it without -g just remove the 

ifneq ($(CONFIG_DEBUG_INFO),y)
CFLAGS += -fno-asynchronous-unwind-tables
endif

in arch/x86_64/Makefile. Then to actually use it in the running kernel
you would need to change the unwind segment in the vmlinux.lds.S
to be loaded instead of discarded at link time (one liner change too)

And something to map it for modules (i haven't looked at that, but 
I suppose if ia64 has the infrastructure it shouldn't be hard to port)

I wouldn't be opposed to a new CONFIG_RUNTIME_UNWIND that does all
this. However without an working unwinder in kernel it's not very useful.

>
> BTW, even on IA64 which has unwind data, we still get problems because
> the unwind data only says what parameters are passed in registers, it
> says nothing about register reuse.  gcc can reuse a parameter register
> if the parameter value is no longer required, for example :-

This is no different from stack based parameters where the stack slot
of the parameter can be overwritten by the callee too.
You just will have to live with that.

-Andi