New module loader makes kernel debugging much harder

New module loader makes kernel debugging much harder

[Keith Owens]

The new module loader makes kernel debugging much harder.

There is no section information available, which means ...

* ksymoops cannot decode oops in modules.  Without section data, there
  is no way to reliably determine where module symbols were loaded.

* kgdb cannot debug modules.  gdb needs to know which modules are
  loaded and where each section of each module starts.

* kdb cannot display the section for a symbol.  Which means the user
  cannot do
    objdump -j <section_name> --adjust-vma=<start_address> module
  to map the code to the original object and source.

The complete lack of kernel and module symbols (no /proc/ksyms) means
that ksymoops is now useless on 2.5 kernels.  If you get an oops in a
kernel built without kksymoops, there is no way to decode the oops.

Big step backwards, Rusty.

Re: New module loader makes kernel debugging much harder

[John Levon]

On Sat, Nov 23, 2002 at 01:20:10PM +1100, Keith Owens wrote:

> The complete lack of kernel and module symbols (no /proc/ksyms) means
> that ksymoops is now useless on 2.5 kernels.  If you get an oops in a
> kernel built without kksymoops, there is no way to decode the oops.

Additionally, module profiling is not possible any more.

> Big step backwards, Rusty.

Somebody (this includes Rusty himself) needs to come up with a workable
solution. For my own needs, the start address of the mapped module is
good enough, but it seems you need more than that. I'd be quite happy
with the re-introduction of /proc/ksyms as it would mean no userspace
code changes for me :)

regards
john

-- 
Khendon's Law: If the same point is made twice by the same person,
the thread is over.

Re: New module loader makes kernel debugging much harder

[Keith Owens]

On Sat, 23 Nov 2002 02:35:13 +0000, 
John Levon <levon@movementarian.org> wrote:
>On Sat, Nov 23, 2002 at 01:20:10PM +1100, Keith Owens wrote:
>
>> The complete lack of kernel and module symbols (no /proc/ksyms) means
>> that ksymoops is now useless on 2.5 kernels.  If you get an oops in a
>> kernel built without kksymoops, there is no way to decode the oops.
>
>Additionally, module profiling is not possible any more.
>
>> Big step backwards, Rusty.
>
>Somebody (this includes Rusty himself) needs to come up with a workable
>solution. For my own needs, the start address of the mapped module is
>good enough [snip]

Only if you assume that the .text is at a known offset from the start
of the module.  There are multiple programs that need to know where
each section really is, instead of making assumptions about how a
module is laid out.

Re: New module loader makes kernel debugging much harder

[John Levon]

On Sat, Nov 23, 2002 at 01:43:20PM +1100, Keith Owens wrote:

> Only if you assume that the .text is at a known offset from the start
> of the module.  There are multiple programs that need to know where
> each section really is, instead of making assumptions about how a
> module is laid out.

Yes, sorry, that is what I meant.

regards
john

-- 
Khendon's Law: If the same point is made twice by the same person,
the thread is over.

Re: New module loader makes kernel debugging much harder

[Keith Owens]

On Sat, 23 Nov 2002 16:23:46 +0000, 
John Levon <levon@movementarian.org> wrote:
>On Sat, Nov 23, 2002 at 01:43:20PM +1100, Keith Owens wrote:
>
>> Only if you assume that the .text is at a known offset from the start
>> of the module.  There are multiple programs that need to know where
>> each section really is, instead of making assumptions about how a
>> module is laid out.
>
>Yes, sorry, that is what I meant.

Even for module profiling, we need section data available.

Although all module code currently goes in a single text area, there is
no guarantee that will always be the case.  In the past we have had
multiple text areas in modules (out of line lock code used its own
section for a long time) and future changes could require multiple text
sections.  To do profiling correctly, you need to know where all the
text sections are, i.e. the module loader has to publish symbols and
section data.  Loosing that data is a big step backwards.

Re: New module loader makes kernel debugging much harder

[John Levon]

On Sun, Nov 24, 2002 at 11:49:32AM +1100, Keith Owens wrote:

> Even for module profiling, we need section data available.

I know, that's why I agreed with you...

> Although all module code currently goes in a single text area, there is
> no guarantee that will always be the case.  In the past we have had
> multiple text areas in modules (out of line lock code used its own
> section for a long time) and future changes could require multiple text
> sections.  To do profiling correctly, you need to know where all the
> text sections are, i.e. the module loader has to publish symbols and
> section data.  Loosing that data is a big step backwards.

Actually all *I* need is the address of one of the sections, as long as
they're simply mapped in we can work backwards given the sample file
and the offset of that section. So this already worked for when the
locking code was a separate section. But yeah, it would be helpful for
other uses to have more info available.

regards
john

-- 
Khendon's Law: If the same point is made twice by the same person,
the thread is over.

Re: New module loader makes kernel debugging much harder

[Keith Owens]

On Sun, 24 Nov 2002 01:06:17 +0000, 
John Levon <levon@movementarian.org> wrote:
>On Sun, Nov 24, 2002 at 11:49:32AM +1100, Keith Owens wrote:
>> Although all module code currently goes in a single text area, there is
>> no guarantee that will always be the case.  In the past we have had
>> multiple text areas in modules (out of line lock code used its own
>> section for a long time) and future changes could require multiple text
>> sections.  To do profiling correctly, you need to know where all the
>> text sections are, i.e. the module loader has to publish symbols and
>> section data.  Loosing that data is a big step backwards.
>
>Actually all *I* need is the address of one of the sections, as long as
>they're simply mapped in we can work backwards given the sample file
         ^^^^^^^^^^^^^
>and the offset of that section. So this already worked for when the
>locking code was a separate section. But yeah, it would be helpful for
>other uses to have more info available.

How do you know if the sections are "simply mapped"?  The module loader
could assign different sections to different mappings, there is no
guarantee that they are contiguous.  They were contiguous using the 2.4
module loader but only because the syscall only allowed for a single
area.  The new loader can assign sections anywhere it likes.

One possibility the new loader opens up is the ability to replicate the
pure module data (rodata and text) for each node of a NUMA box.  There
is already an option to replicate the kernel text on each node to cut
down inter-node traffic.  Replicating the pure module data would be
nice as well.  I guarantee that will result in something that is not
"simply mapped".

Re: New module loader makes kernel debugging much harder

[John Levon]

On Sun, Nov 24, 2002 at 12:18:46PM +1100, Keith Owens wrote:

> How do you know if the sections are "simply mapped"?  The module loader
> could assign different sections to different mappings, there is no
> guarantee that they are contiguous.  They were contiguous using the 2.4
> module loader but only because the syscall only allowed for a single
> area.  The new loader can assign sections anywhere it likes.

Mmmm, good point. So I will need full section info too. And some
userspace changes to cope with such ...

thanks
john
-- 
Khendon's Law: If the same point is made twice by the same person,
the thread is over.

Re: New module loader makes kernel debugging much harder

[Rusty Russell]

On Sun, 24 Nov 2002 12:18:46 +1100
Keith Owens <kaos@ocs.com.au> wrote:
> One possibility the new loader opens up is the ability to replicate the
> pure module data (rodata and text) for each node of a NUMA box.  There
> is already an option to replicate the kernel text on each node to cut
> down inter-node traffic.  Replicating the pure module data would be
> nice as well.  I guarantee that will result in something that is not
> "simply mapped".

Ewww, you are a sick lad 8)

But I'm not sure that there is any interface which wouldn't break horribly
when faced with that possibility.

The patch to restore /proc/ksyms is trivial.  As is the addition of a start
entry /proc/modules.  When combined with rth's simplified loader, it should
be sufficient for both ksymoops and oprofile.

kgdb needs a patch to work, anyway: you might want to restore /proc/ksyms
in that patch?  (I don't use kgdb, so my ignorance here is complete).

Rusty.
-- 
   there are those who do and those who hang on and you don't see too
   many doers quoting their contemporaries.  -- Larry McVoy

Re: New module loader makes kernel debugging much harder

[Jamie Lokier]

Also "depmod" is now broken, so even with Rusty's replacement modutils
a system won't demand load modules properly.  Or is there a working
"depmod" and I missed it somewhere?

-- Jamie