[PATCH] mkelf32: Correct sh_link

[PATCH] mkelf32: Correct sh_link

[Christoph Egger]

[-- Attachment #1: Type: text/plain, Size: 111 bytes --]


Hi!

According to the ELF Spec, sh_link points to the String table.
The attached patch fixes this.

Christoph

[-- Attachment #2: xen-mkelf32.diff --]
[-- Type: text/x-diff, Size: 720 bytes --]

diff -r 886f7312257e xen/arch/x86/boot/mkelf32.c
--- a/xen/arch/x86/boot/mkelf32.c	Tue Jan 02 14:19:47 2007 +0000
+++ b/xen/arch/x86/boot/mkelf32.c	Fri Dec 22 14:22:04 2006 +0100
@@ -72,7 +72,7 @@ static Elf32_Shdr out_shdr[] = {
       DYNAMICALLY_FILLED,                    /* sh_addr */
       RAW_OFFSET,                            /* sh_offset */
       DYNAMICALLY_FILLED,                    /* sh_size */
-      0,                                     /* sh_link */
+      2,                                     /* sh_link */
       0,                                     /* sh_info */
       64,                                    /* sh_addralign */
       0                                      /* sh_entsize */

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] mkelf32: Correct sh_link

[Keir Fraser]

SHT_PROGBITS sections do not use the sh_link field. It's used by
symbol-table sections, relocation-entry sections, and suchlike, of which we
don't have any. And .shstrtab section is never linked to.

 -- Keir

On 3/1/07 09:30, "Christoph Egger" <Christoph.Egger@amd.com> wrote:

> 
> Hi!
> 
> According to the ELF Spec, sh_link points to the String table.
> The attached patch fixes this.
> 
> Christoph
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH] mkelf32: Correct sh_link

[grel]

I am a newbie and I would like to know how to apply this patch.
Can someone provide me with steps.


Christoph Egger-2 wrote:
> 
> 
> Hi!
> 
> According to the ELF Spec, sh_link points to the String table.
> The attached patch fixes this.
> 
> Christoph
> 
> diff -r 886f7312257e xen/arch/x86/boot/mkelf32.c
> --- a/xen/arch/x86/boot/mkelf32.c	Tue Jan 02 14:19:47 2007 +0000
> +++ b/xen/arch/x86/boot/mkelf32.c	Fri Dec 22 14:22:04 2006 +0100
> @@ -72,7 +72,7 @@ static Elf32_Shdr out_shdr[] = {
>        DYNAMICALLY_FILLED,                    /* sh_addr */
>        RAW_OFFSET,                            /* sh_offset */
>        DYNAMICALLY_FILLED,                    /* sh_size */
> -      0,                                     /* sh_link */
> +      2,                                     /* sh_link */
>        0,                                     /* sh_info */
>        64,                                    /* sh_addralign */
>        0                                      /* sh_entsize */
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 

-- 
View this message in context: http://www.nabble.com/-PATCH--mkelf32%3A-Correct-sh_link-tf2912492.html#a8141336
Sent from the Xen - Dev mailing list archive at Nabble.com.

Re: [PATCH] mkelf32: Correct sh_link

[Christoph Egger]

On Wednesday 03 January 2007 14:57, Keir Fraser wrote:
> SHT_PROGBITS sections do not use the sh_link field. It's used by
> symbol-table sections, relocation-entry sections, and suchlike, of which we
> don't have any. And .shstrtab section is never linked to.

Why do you strip the symbol table and re-add it in an unusual format
when linking the bootable ELF image? Is this a historical relict?

Christoph

Re: [PATCH] mkelf32: Correct sh_link

[Christoph Egger]

On Wednesday 03 January 2007 16:11, Christoph Egger wrote:
> On Wednesday 03 January 2007 14:57, Keir Fraser wrote:
> > SHT_PROGBITS sections do not use the sh_link field. It's used by
> > symbol-table sections, relocation-entry sections, and suchlike, of which
> > we don't have any. And .shstrtab section is never linked to.
>
> Why do you strip the symbol table and re-add it in an unusual format
> when linking the bootable ELF image? Is this a historical relict?

Ah, sorry! I actually meant: Why do you strip the symbol and string tables
and re-add the string table in an unusal format?
Is this a historical relict?

Christoph

Re: [PATCH] mkelf32: Correct sh_link

[Keir Fraser]

On 3/1/07 15:11, "Christoph Egger" <Christoph.Egger@amd.com> wrote:

> On Wednesday 03 January 2007 14:57, Keir Fraser wrote:
>> SHT_PROGBITS sections do not use the sh_link field. It's used by
>> symbol-table sections, relocation-entry sections, and suchlike, of which we
>> don't have any. And .shstrtab section is never linked to.
> 
> Why do you strip the symbol table and re-add it in an unusual format
> when linking the bootable ELF image? Is this a historical relict?

No bootloader would load a symbol table for us (where would it load it? How
would it tell us about it?), so we have to internalise it into the PROGBITS
section somehow. Stealing the Linux code to do this made sense.

 -- Keir

Re: [PATCH] mkelf32: Correct sh_link

[Keir Fraser]

On 3/1/07 15:14, "Christoph Egger" <Christoph.Egger@amd.com> wrote:

>> Why do you strip the symbol table and re-add it in an unusual format
>> when linking the bootable ELF image? Is this a historical relict?
> 
> Ah, sorry! I actually meant: Why do you strip the symbol and string tables
> and re-add the string table in an unusal format?
> Is this a historical relict?

It isn't the symbol-name string table: look at how short it is! It's the
section-header string table. I believe all Elf images require one of those
(not least because some section names have special meaning).

 -- Keir

Re: [PATCH] mkelf32: Correct sh_link

[Christoph Egger]

On Wednesday 03 January 2007 16:20, Keir Fraser wrote:
> On 3/1/07 15:11, "Christoph Egger" <Christoph.Egger@amd.com> wrote:
> > On Wednesday 03 January 2007 14:57, Keir Fraser wrote:
> >> SHT_PROGBITS sections do not use the sh_link field. It's used by
> >> symbol-table sections, relocation-entry sections, and suchlike, of which
> >> we don't have any. And .shstrtab section is never linked to.
> >
> > Why do you strip the symbol table and re-add it in an unusual format
> > when linking the bootable ELF image? Is this a historical relict?
>
> No bootloader would load a symbol table for us (where would it load it? How
> would it tell us about it?), so we have to internalise it into the PROGBITS
> section somehow. Stealing the Linux code to do this made sense.

The multiboot header tells us where to find the symbol and string table.
Then we can load it ourself.

Look into the NetBSD code how this would work:
http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/i386/include/multiboot.h?annotate=1.4
http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/i386/i386/multiboot.c?annotate=1.9

;-)

Would you accept a patch, that makes XEN bootable as an ELF image with
ELF symbol-name and string tables? If yes, then I will work on this.

Christoph

Re: [PATCH] mkelf32: Correct sh_link

[Keir Fraser]

On 3/1/07 15:38, "Christoph Egger" <Christoph.Egger@amd.com> wrote:

> Would you accept a patch, that makes XEN bootable as an ELF image with
> ELF symbol-name and string tables? If yes, then I will work on this.

What we have now works and produces a reasonably compressed symbol table. I
don't see any value in changing it.

 -- Keir

Re: [PATCH] mkelf32: Correct sh_link

[John Levon]

On Wed, Jan 03, 2007 at 04:21:35PM +0000, Keir Fraser wrote:

> On 3/1/07 15:38, "Christoph Egger" <Christoph.Egger@amd.com> wrote:
> 
> > Would you accept a patch, that makes XEN bootable as an ELF image with
> > ELF symbol-name and string tables? If yes, then I will work on this.
> 
> What we have now works and produces a reasonably compressed symbol table. I
> don't see any value in changing it.

It seems like it would simplify things. And it would certainly help us
as we want to ship the binary with (compressed) debug info, which is
tricky right now due to this unusual behaviour...

regards
john

Re: [PATCH] mkelf32: Correct sh_link

[Keir Fraser]

On 3/1/07 6:36 pm, "John Levon" <levon@movementarian.org> wrote:

>> What we have now works and produces a reasonably compressed symbol table. I
>> don't see any value in changing it.
> 
> It seems like it would simplify things. And it would certainly help us
> as we want to ship the binary with (compressed) debug info, which is
> tricky right now due to this unusual behaviour...

You can ship xen-syms if users want extra debug info (e.g., attach a gdb
client). What exactly is your requirement?

 -- Keir

Re: [PATCH] mkelf32: Correct sh_link

[John Levon]

On Wed, Jan 03, 2007 at 07:14:47PM +0000, Keir Fraser wrote:

> >> What we have now works and produces a reasonably compressed symbol table. I
> >> don't see any value in changing it.
> > 
> > It seems like it would simplify things. And it would certainly help us
> > as we want to ship the binary with (compressed) debug info, which is
> > tricky right now due to this unusual behaviour...
> 
> You can ship xen-syms if users want extra debug info (e.g., attach a gdb
> client).

Indeed, and that's what we're doing currently. But this is problematic
as it's very easy for other files to get lost, not be handed over etc.
and working backwards to the right version is very rarely feasible...
Thus we'd like to have a single binary (or more importantly dump!) that
has everything in place.

regards,
john

Re: [PATCH] mkelf32: Correct sh_link

[Keir Fraser]

On 3/1/07 7:20 pm, "John Levon" <levon@movementarian.org> wrote:

>> You can ship xen-syms if users want extra debug info (e.g., attach a gdb
>> client).
> 
> Indeed, and that's what we're doing currently. But this is problematic
> as it's very easy for other files to get lost, not be handed over etc.
> and working backwards to the right version is very rarely feasible...
> Thus we'd like to have a single binary (or more importantly dump!) that
> has everything in place.

That would potentially be nice, but I don't know of any tools that would
allow us to symbolically debug a dump without having a separate source of
symbol information (like xen-syms). The problems you mention would typically
be resolved by non-technical means anyway (offical release builds for which
you keep a copy of the bits, which can then be used to augment the
information from a raw dump). I wouldn't deep-dive into a core dump for a
build of dubious heritage where the builder hadn't kept the sources and
build bits! :-)

 -- Keir

Re: [PATCH] mkelf32: Correct sh_link

[John Levon]

On Wed, Jan 03, 2007 at 08:06:12PM +0000, Keir Fraser wrote:

> > Indeed, and that's what we're doing currently. But this is problematic
> > as it's very easy for other files to get lost, not be handed over etc.
> > and working backwards to the right version is very rarely feasible...
> > Thus we'd like to have a single binary (or more importantly dump!) that
> > has everything in place.
> 
> That would potentially be nice, but I don't know of any tools that would
> allow us to symbolically debug a dump without having a separate source of
> symbol information (like xen-syms).

This is exactly what is available on Solaris both for userspace core
dumps and kernel crash dumps. We'd like to be able to extend this to
Xen. As far as I know, there's nothing preventing, say, Red Hat's
'crash' doing the same.

> The problems you mention would typically
> be resolved by non-technical means anyway (offical release builds for which
> you keep a copy of the bits, which can then be used to augment the
> information from a raw dump).

Bitter experience tends to show this doesn't work well especially in
development environments.

> I wouldn't deep-dive into a core dump for a
> build of dubious heritage where the builder hadn't kept the sources and
> build bits! :-)

All too common alas :) Wasting a small amount of time debugging a dodgy
build is usually better than letting a real intermittent bug slip by...

regards,
john

Re: [PATCH] mkelf32: Correct sh_link

[Keir Fraser]

On 3/1/07 8:11 pm, "John Levon" <levon@movementarian.org> wrote:

>> That would potentially be nice, but I don't know of any tools that would
>> allow us to symbolically debug a dump without having a separate source of
>> symbol information (like xen-syms).
> 
> This is exactly what is available on Solaris both for userspace core
> dumps and kernel crash dumps. We'd like to be able to extend this to
> Xen. As far as I know, there's nothing preventing, say, Red Hat's
> 'crash' doing the same.

Well, extracting symbols from the existing Xen format isn't hard. We could
add that, but really it's not as good as source-level debugging, being able
look at local variables up the call stack, etc. I don't think it is
expecting much to require xen-syms to be kept around for xen images that are
still being tested. Another trick might be to allow production of a xen
image with the full xen-syms image appended -- then a crash dump would have
all the info required to do full source-level debugging (except the source
tree, but you do get a changeset number from the core dump which will
suffice if you use revision control). That would be easy actually -- just
get mkelf32 to append the xen-syms file and set a couple of values at a
known offset in the main xen image, easily picked up by crashdump tools and
by Xen itself.

 -- Keir

Re: [PATCH] mkelf32: Correct sh_link

[John Levon]

On Wed, Jan 03, 2007 at 09:21:08PM +0000, Keir Fraser wrote:

> >> That would potentially be nice, but I don't know of any tools that would
> >> allow us to symbolically debug a dump without having a separate source of
> >> symbol information (like xen-syms).
> > 
> > This is exactly what is available on Solaris both for userspace core
> > dumps and kernel crash dumps. We'd like to be able to extend this to
> > Xen. As far as I know, there's nothing preventing, say, Red Hat's
> > 'crash' doing the same.
> 
> Well, extracting symbols from the existing Xen format isn't hard.

It was more of an answer to your question as to why there would be any
point in shipping a standard symtab rather than the Linux thing.
Essentially it makes things much easier for us as our debug info relies
on a standard symtab.

> suffice if you use revision control). That would be easy actually -- just
> get mkelf32 to append the xen-syms file and set a couple of values at a
> known offset in the main xen image, easily picked up by crashdump tools and
> by Xen itself.

I had in mind something very much like this, we'd just append the
symtab,strtab, and our debug info on to the image and set something to
point to the location. This was mostly a workaround of the difficulties
with building those sections straight into the file in the normal ELF
manner, though.

I think the question might more be: what do we gain from being
'unusual'? A significantly smaller string size for the symbol names
might be a reasonable answer, I don't really know how much of a gain
that stuff is. An entertaining logic puzzle in the makefiles, perhaps
not ;)

Of course it would still be a /little/ unusual because you'd want the
symtab etc. as allocated PROGBITS.

regards,
john

Re: [PATCH] mkelf32: Correct sh_link

[Keir Fraser]

On 3/1/07 9:32 pm, "John Levon" <levon@movementarian.org> wrote:

> I had in mind something very much like this, we'd just append the
> symtab,strtab, and our debug info on to the image and set something to
> point to the location. This was mostly a workaround of the difficulties
> with building those sections straight into the file in the normal ELF
> manner, though.

We could build them in as ordinary sections in our loadable xen image, but
the bootloader wouldn't load them into memory for us let alone point us at
them. So I think we have to cook all the stuff you're interested in into
PROGBITS (unless you really only want symtab/strtab, but if you're going
this far why not keep stuff like the dwarf debug info as well?).

One nice thing about 'dumbly' appending xen-syms is that the whole image is
then extractable to a file post-crash and can be manipulated by the
developer independently of tools like RH's crash. It's also simple to
implement and doesn't mess with the basic structure of our loadable Xen elf
image (which has acquired various tweaks and workarounds for multiboot
loader bugs).

 -- Keir

Re: [PATCH] mkelf32: Correct sh_link

[Mark Williamson]

> I am a newbie and I would like to know how to apply this patch.
> Can someone provide me with steps.

Save the patch xen-mkelf32.diff to your hard drive

Get a xen repository and change into it:

cd xen-unstable.hg (or whatever you called it)
patch -p0 < /path/to/xen-mkelf32.diff

Should work.

Cheers,
Mark

> Christoph Egger-2 wrote:
> > Hi!
> >
> > According to the ELF Spec, sh_link points to the String table.
> > The attached patch fixes this.
> >
> > Christoph
> >
> > diff -r 886f7312257e xen/arch/x86/boot/mkelf32.c
> > --- a/xen/arch/x86/boot/mkelf32.c	Tue Jan 02 14:19:47 2007 +0000
> > +++ b/xen/arch/x86/boot/mkelf32.c	Fri Dec 22 14:22:04 2006 +0100
> > @@ -72,7 +72,7 @@ static Elf32_Shdr out_shdr[] = {
> >        DYNAMICALLY_FILLED,                    /* sh_addr */
> >        RAW_OFFSET,                            /* sh_offset */
> >        DYNAMICALLY_FILLED,                    /* sh_size */
> > -      0,                                     /* sh_link */
> > +      2,                                     /* sh_link */
> >        0,                                     /* sh_info */
> >        64,                                    /* sh_addralign */
> >        0                                      /* sh_entsize */
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: [PATCH] mkelf32: Correct sh_link

[Christoph Egger]

On Wednesday 03 January 2007 20:20, John Levon wrote:
> On Wed, Jan 03, 2007 at 07:14:47PM +0000, Keir Fraser wrote:
> > >> What we have now works and produces a reasonably compressed symbol
> > >> table. I don't see any value in changing it.
> > >
> > > It seems like it would simplify things. And it would certainly help us
> > > as we want to ship the binary with (compressed) debug info, which is
> > > tricky right now due to this unusual behaviour...
> >
> > You can ship xen-syms if users want extra debug info (e.g., attach a gdb
> > client).
>
> Indeed, and that's what we're doing currently. But this is problematic
> as it's very easy for other files to get lost, not be handed over etc.
> and working backwards to the right version is very rarely feasible...
> Thus we'd like to have a single binary (or more importantly dump!) that
> has everything in place.

Further, when you boot on a serial console how do you connect gdb
over a serial console? You can't use a serial console for two things
concurrently. Then a xen-kernel debugger is useful. And yeah, that needs
standard symtabs and strtabs.
And with a kernel debugger, you don't need to worry about what John pointed
out.

regards,
John

Re: [PATCH] mkelf32: Correct sh_link

[Keir Fraser]

On 4/1/07 08:30, "Christoph Egger" <Christoph.Egger@amd.com> wrote:

>> Indeed, and that's what we're doing currently. But this is problematic
>> as it's very easy for other files to get lost, not be handed over etc.
>> and working backwards to the right version is very rarely feasible...
>> Thus we'd like to have a single binary (or more importantly dump!) that
>> has everything in place.
> 
> Further, when you boot on a serial console how do you connect gdb
> over a serial console? You can't use a serial console for two things
> concurrently. Then a xen-kernel debugger is useful. And yeah, that needs
> standard symtabs and strtabs.
> And with a kernel debugger, you don't need to worry about what John pointed
> out.

Gdbstub and console can share a serial port. When gdbstub is entered then
the gdbstub steals the serial port. Console messages are then wrapped as gdb
messages. Given that this setup works and requires minimal code in Xen, I
have no interest in accepting an in-kernel debugger into the tree.

 -- Keir

Re: [PATCH] mkelf32: Correct sh_link

[Christoph Egger]

On Thursday 04 January 2007 10:58, Keir Fraser wrote:
> On 4/1/07 08:30, "Christoph Egger" <Christoph.Egger@amd.com> wrote:
> >> Indeed, and that's what we're doing currently. But this is problematic
> >> as it's very easy for other files to get lost, not be handed over etc.
> >> and working backwards to the right version is very rarely feasible...
> >> Thus we'd like to have a single binary (or more importantly dump!) that
> >> has everything in place.
> >
> > Further, when you boot on a serial console how do you connect gdb
> > over a serial console? You can't use a serial console for two things
> > concurrently. Then a xen-kernel debugger is useful. And yeah, that needs
> > standard symtabs and strtabs.
> > And with a kernel debugger, you don't need to worry about what John
> > pointed out.
>
> Gdbstub and console can share a serial port. When gdbstub is entered then
> the gdbstub steals the serial port. Console messages are then wrapped as
> gdb messages. Given that this setup works and requires minimal code in Xen,
> I have no interest in accepting an in-kernel debugger into the tree.

What about xen/common/keyhandler.c ? Isn't this a very tiny
in-kernel debugger? If YOU are fine with gdb, why is it there?
And it ALWAYS gets compiled in.

We already have gdb, xentrace, xencrash, loglevels and more. Nonetheless,
many developers still add/remove tons of printk()'s for debugging purpose.

I count this as a sign, that there is a demand for better debugging 
facilities. gdb is a very powerful general purpose debugger.
So the core question is, what is really needed and what not.

_Some_ examples to think about:
Is a command line useful? Should it be possible to lookup symbols
for its addresses? Is it useful to reprint xen messages?
Is it better to put lots of printk() to figure out which virtual cpus and 
interrupts are mapped to which physical ones or just tell the debugger "show 
me this" ?

Re: [PATCH] mkelf32: Correct sh_link

[John Levon]

On Wed, Jan 03, 2007 at 09:40:46PM +0000, Keir Fraser wrote:

> > I had in mind something very much like this, we'd just append the
> > symtab,strtab, and our debug info on to the image and set something to
> > point to the location. This was mostly a workaround of the difficulties
> > with building those sections straight into the file in the normal ELF
> > manner, though.
> 
> We could build them in as ordinary sections in our loadable xen image, but
> the bootloader wouldn't load them into memory for us let alone point us at
> them.

Unless I'm misunderstanding badly isn't this just a case of changing the
mapfile to stick them into the single program header.

> One nice thing about 'dumbly' appending xen-syms is that the whole image is
> then extractable to a file post-crash and can be manipulated by the
> developer independently of tools like RH's crash. It's also simple to
> implement and doesn't mess with the basic structure of our loadable Xen elf
> image (which has acquired various tweaks and workarounds for multiboot
> loader bugs).

It's perhaps a bit silly to have .text twice etc.though. I'm not
familiar with the tweaks you mention so I'm sure you're right there.

john