Userspace Test Framework for module loader porting

Userspace Test Framework for module loader porting

[Rusty Russell]

The userspace test framework I used to develop module loading on
different archs is up at:

	http://www.kernel.org/pub/linux/kernel/people/rusty/modules/module-test-framework.tar.gz 

I found it much easier to use for each arch than doing the
crash/reboot cycle (and you can use a real debugger).

BTW, the change to use shared objects for modules is going to be a 2.7
thing: after 10 architectures, MIPS toolchain issues made it
non-trivial.  So the current stuff is what is going to be there for
2.6, so no point waiting 8)

Please report any problems!
Rusty.
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: Userspace Test Framework for module loader porting

[David Mosberger]

>>>>> On Mon, 06 Jan 2003 13:27:02 +1100, Rusty Russell <rusty@rustcorp.com.au> said:

  Rusty> BTW, the change to use shared objects for modules is going to
  Rusty> be a 2.7 thing: after 10 architectures, MIPS toolchain issues
  Rusty> made it non-trivial.  So the current stuff is what is going
  Rusty> to be there for 2.6, so no point waiting 8)

What about all the problems that Richard Henderson pointed out with
the original in-kernel module loader?  Were those solved?  My gut
feeling is that we really want shared objects for kernel modules on
ia64 (and probably alpha?).

	--david

Re: Userspace Test Framework for module loader porting

[Richard Henderson]

On Mon, Jan 06, 2003 at 11:38:20AM -0800, David Mosberger wrote:
> What about all the problems that Richard Henderson pointed out with
> the original in-kernel module loader?  Were those solved?

Yes.  The most important one for correctness was

ChangeSet@1.838.130.13, 2003-01-01 19:02:38-08:00, rusty@rustcorp.com.au
  [PATCH] Modules 3/3: Sort sections
  
  RTH's final complaint (so far 8) was that we should sort the module
  sections: archs might require some sections to be adjacent, so they can
  all be reached by a relative pointer (ie.  GOT pointer).  This
  implements that reordering, and simplfies the module interface for
  architectures as well.
  
  Previously an arch could specify it wanted extra space, but not where
  that space would be.  The new method (used only by PPC so far) is to
  allocate an empty section (in asm/module.h or by setting LDFLAGS_MODULE
  to use an arch specific linker script), and expand that to the desired
  size in "module_frob_arch_sections()".


> My gut feeling is that we really want shared objects for kernel
> modules on ia64 (and probably alpha?).

Well, most everyone wants it.  Except that MIPS is terminally
broken.  They need a rewrite of bfd/elfxx-mips.c in order to
be able to do non-pic ET_DYN images.  Which leaves the rest of
us out in the cold.


r~

Re: Userspace Test Framework for module loader porting

[David Mosberger]

>>>>> On Mon, 6 Jan 2003 14:41:04 -0800, Richard Henderson <rth@twiddle.net> said:

  Rich> On Mon, Jan 06, 2003 at 11:38:20AM -0800, David Mosberger
  Rich> wrote:

  >> What about all the problems that Richard Henderson pointed out
  >> with the original in-kernel module loader?  Were those solved?

  Rich> Yes.

OK, that's good.

Rusty, have you maintained the ia64 support of your in-kernel loader?
To be honest, I have less than zero interest in maintaining such code.
I'd rather prefer the old (user-level loader) or the new shared-object
loader.  (Of course, if someone else wants to volunteer, that would be
fine, too... ;-)

	--david

Re: Userspace Test Framework for module loader porting

[Rusty Russell]

In message <15898.8498.519625.200668@napali.hpl.hp.com> you write:
> >>>>> On Mon, 6 Jan 2003 14:41:04 -0800, Richard Henderson <rth@twiddle.net> 
said:
> 
>   Rich> On Mon, Jan 06, 2003 at 11:38:20AM -0800, David Mosberger
>   Rich> wrote:
> 
>   >> What about all the problems that Richard Henderson pointed out
>   >> with the original in-kernel module loader?  Were those solved?
> 
>   Rich> Yes.
> 
> OK, that's good.
> 
> Rusty, have you maintained the ia64 support of your in-kernel loader?

No, but I can update it next week when I'm back in the office.  Should
be trivial to do; I'll pass it by Mike Stephens if he's willing,
though, since he wrote the original code I cribbed off.

> I'd rather prefer the old (user-level loader) 

Really?  Because it already exists (and is maintained by someone else)
or for some other reason?

> or the new shared-object loader.

I thought about letting archs choose which one they wanted to use, but
it would really mess up the core code.  Of course, the transition
won't break userspace (kind of the whole point of the in-kernel module
loader).

Cheers!
Rusty.
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: Userspace Test Framework for module loader porting

[David Mosberger]

>>>>> On Wed, 08 Jan 2003 22:44:15 +1100, Rusty Russell <rusty@rustcorp.com.au> said:

  >> I'd rather prefer the old (user-level loader)
  >> or the new shared-object loader.

  Rusty> Really?  Because it already exists (and is maintained by
  Rusty> someone else) or for some other reason?

Yeah, I'm lazy: I don't really want to have to deal with two new
module loaders: one for 2.6, soon to be followed by one for 2.7.  But
if someone volunteers to do and _maintain_ an interim kernel loader,
that's fine with me.

  Rusty> I thought about letting archs choose which one they wanted to
  Rusty> use, but it would really mess up the core code.  Of course,
  Rusty> the transition won't break userspace (kind of the whole point
  Rusty> of the in-kernel module loader).

But it would be more in keeping with the Linux philosophy: do the
Right Thing, fix up "broken" stuff by doing whatever is necessary.
I'm also a bit worried about changing module loaders so often.  Yeah,
once you switch to a kernel-loader, presumably users won't be
affected, but (kernel-module) developers will be.

	--david

Re: Userspace Test Framework for module loader porting

[Rusty Russell]

In message <15903.30632.576801.904652@napali.hpl.hp.com> you write:
> >>>>> On Wed, 08 Jan 2003 22:44:15 +1100, Rusty Russell <rusty@rustcorp.com.a
u> said:
> Yeah, I'm lazy: I don't really want to have to deal with two new
> module loaders: one for 2.6, soon to be followed by one for 2.7.  But
> if someone volunteers to do and _maintain_ an interim kernel loader,
> that's fine with me.

Well, "soon" here is > 12 months away, of course.  And most of it
involves removing, rather than adding, code.

>   Rusty> I thought about letting archs choose which one they wanted to
>   Rusty> use, but it would really mess up the core code.  Of course,
>   Rusty> the transition won't break userspace (kind of the whole point
>   Rusty> of the in-kernel module loader).
> 
> But it would be more in keeping with the Linux philosophy: do the
> Right Thing, fix up "broken" stuff by doing whatever is necessary.

I think you missed the "work around what we can't change" (eg. always
initializing per-cpu variables because Sparc's toolchain is broken, or
adding that crazy restart stuff so we didn't have to create a one-arg
re-enterable nanosleep then make glibc use it).

And, of course, the other Golden Rule: "if it's not x86, it doesn't
matter" 8)

> I'm also a bit worried about changing module loaders so often.  Yeah,
> once you switch to a kernel-loader, presumably users won't be
> affected, but (kernel-module) developers will be.

While ET_DYN modules are a reasonably serious win for ia64 (and
probably hppa) (ie. -300 lines or so), they're a minor win for alpha
and ppc64 (-100 lines or so), and no real change for arm, i386, ppc,
sparc, and sparc64.  It's a lose for x86_64 (toolchain fixes, unless
they want to use -fPIC for modules), mips and mips64 (major toolchain
fixes, unless they want to use -fPIC for modules and stop using r28
for current inside modules).

So, if I were ia64 maintainer, I'd be lobbying for ET_DYN modules now,
too, but I don't it's a big enough general win to outweigh the other
problems.

Sorry,
Rusty.
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: Userspace Test Framework for module loader porting

[Maciej W. Rozycki]

On Mon, 13 Jan 2003, Rusty Russell wrote:

> > I'm also a bit worried about changing module loaders so often.  Yeah,
> > once you switch to a kernel-loader, presumably users won't be
> > affected, but (kernel-module) developers will be.
> 
> While ET_DYN modules are a reasonably serious win for ia64 (and
> probably hppa) (ie. -300 lines or so), they're a minor win for alpha
> and ppc64 (-100 lines or so), and no real change for arm, i386, ppc,
> sparc, and sparc64.  It's a lose for x86_64 (toolchain fixes, unless
> they want to use -fPIC for modules), mips and mips64 (major toolchain
> fixes, unless they want to use -fPIC for modules and stop using r28
> for current inside modules).

 Hmm, what's the gain from using shared objects as opposed to relocatables
and why is there any for non-PIC objects at all?  After all, the reason
for using shared objects is to handle shared PIC code, for which
relocation is indeed easier to perform -- you only need to relocate the
GOT and data sections, the latter being rare and with a limited number of
relocation types.  For non-PIC shared objects the GOT and data sections
still need to be relocated, but text sections do as well, introducting all
the relocation types back.  The end result is almost no different from
operating on relocatables. 

 From the MIPS pov, adding non-PIC support to tools would mean defining
R_MIPS_26 (with some effort these could mostly be avoided for modules less
than 128kB in size), R_MIPS_32, R_MIPS_HI16 and R_MIPS_LO16 for shared
objects.  Additionally R_MIPS_64, R_MIPS_HIGHEST and R_MIPS_HIGHER for
MIPS64.  A few ABI issues should be resolved, specifically how to create
the GOT and the dynamic section, possibly others, as the MIPS ABI
supplement doesn't define handling of non-PIC shared objects.  An
implementation for the BFD, though likely tiresome, shouldn't be difficult
in principle, but tricky issues may arrise due to BFD's subtleties.  And
the overall processing gain would be questionable, IMO. 

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Userspace Test Framework for module loader porting

[Rusty Russell]

In message <Pine.GSO.3.96.1030113114240.25230B-100000@delta.ds2.pg.gda.pl> you 
write:
> On Mon, 13 Jan 2003, Rusty Russell wrote:
> 
> > > I'm also a bit worried about changing module loaders so often.  Yeah,
> > > once you switch to a kernel-loader, presumably users won't be
> > > affected, but (kernel-module) developers will be.
> 
> Hmm, what's the gain from using shared objects as opposed to relocatables
> and why is there any for non-PIC objects at all?

[ Other maintainers dropped from CC, they're probably not that interested ].

Basically it comes down to ET_DYNs being designed to be easy to load.
For most archs, new sections don't have to be allocated (eg. plt,
GOT).  Sections don't have to be sorted, or layed out.  As you point
out, relocations are bundled nicely together, etc.

On some architectures, making the linker do more of the work makes a
significant difference (eg. in ia64 it generates the stub code to jump
out of modules).  The module being layed out continuously, however,
means we need a "vmalloc_truncate" to free init sections (except for
archs which use their own alloc functions anyway).

Like most things, it's a tradeoff.  19 archs makes it even more complex.
Rusty.
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

return-type for search_extable()

[David Mosberger]

Hi Rusty,

Could you please change the return-type of search_extable() to
something that allows a bit more flexibility?  The value returned by
this function is "something that lets architecture-specific code
recover from a memory-managment-fault".  This may or may not be the
same as an exception_table_entry.  For example, on ia64, I want to
return an already-relocated fixup-word.  Perhaps the cleanest way to
fix this would be to have:

	exception_fixup_t search_extable (...);

By default, you could then use

	typedef struct exception_table_entry *exception_fixup_t;

and on ia64 I could use:

	typedef long exception_fixup_t.

The only restriction on exception_fixup_t would be that it's a type
that can be tested for being equal to zero and, if it is zero, it
would mean that there is no exception-table entry.

Alternatively, we could make search_extable() just always return a
"void *" or a "long", but that's less clear and less type-safe.

	--david

Re: return-type for search_extable()

[Rusty Russell]

In message <15921.33955.645830.709868@napali.hpl.hp.com> you write:
> Hi Rusty,
> 
> Could you please change the return-type of search_extable() to
> something that allows a bit more flexibility?  The value returned by
> this function is "something that lets architecture-specific code
> recover from a memory-managment-fault".  This may or may not be the
> same as an exception_table_entry.  For example, on ia64, I want to
> return an already-relocated fixup-word.  Perhaps the cleanest way to
> fix this would be to have:
> 
> 	exception_fixup_t search_extable (...);
> 
> By default, you could then use
> 
> 	typedef struct exception_table_entry *exception_fixup_t;
> 
> and on ia64 I could use:
> 
> 	typedef long exception_fixup_t.

Sure.  Of course, you could just cast the return value to a long
inside ia64, but that's pretty unclear.

Hmm, I actually just tried to do this, and like all typedefs, it's
icky.  search_exception_table() belongs in kernel.h, not module.h
where it is now.  But that means kernel.h has to include asm/uaccess.h
to get exception_fixup_t.

Maybe the cleanest way really is to simply document that your
search_extable() returns the fixup word... 8(

Rusty.
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: return-type for search_extable()

[David Mosberger]

>>>>> On Tue, 28 Jan 2003 15:53:20 +1100, Rusty Russell <rusty@rustcorp.com.au> said:

  Rusty> Hmm, I actually just tried to do this, and like all typedefs,
  Rusty> it's icky.  search_exception_table() belongs in kernel.h, not
  Rusty> module.h where it is now.  But that means kernel.h has to
  Rusty> include asm/uaccess.h to get exception_fixup_t.

Yeah, that is a bit ugly.

  Rusty> Maybe the cleanest way really is to simply document that your
  Rusty> search_extable() returns the fixup word... 8(

It turns out that on Dec 18th laster year, HJ Lu fixed the gas bug
that prevented me from using the place-relative relocations.  I ended
up biting the bullet and making the bleeding edge assembler a
pre-requisite for building 2.5.59.  I try to avoid that, but in this
case, it was better than the alternatives.

Anyhow, it means that we don't need a separate type anymore for ia64.

Thanks,

	--david

Re: Userspace Test Framework for module loader porting

[Rusty Russell]

In message <15897.56108.201340.229554@napali.hpl.hp.com> you write:
> >>>>> On Mon, 06 Jan 2003 13:27:02 +1100, Rusty Russell <rusty@rustcorp.com.a
u> said:
> 
>   Rusty> BTW, the change to use shared objects for modules is going to
>   Rusty> be a 2.7 thing: after 10 architectures, MIPS toolchain issues
>   Rusty> made it non-trivial.  So the current stuff is what is going
>   Rusty> to be there for 2.6, so no point waiting 8)
> 
> What about all the problems that Richard Henderson pointed out with
> the original in-kernel module loader?  Were those solved?

Yes.  Richard withdrew the one about allocation (we used a special
per-arch allocator, he missed it), the one about common section
ordering was fixed by RTH noticing we use -fno-common, so all that
logic was dropped anyway, and his section ordering query was fixed by
his section ordering patch (which changed the arch interface
slightly).

> My gut feeling is that we really want shared objects for kernel
> modules on ia64 (and probably alpha?).

It's certainly simpler.  My ia64 current implementation is around 500
lines, vs 225 lines for the shared version.

But I'm not prepared to shaft an arch over it this late in the cycle,
and the current scheme puts less pressure on toolchains (several other
binutils "issues" were discovered in implementing
modules-as-shared-objects).

Hope that clarifies,
Rusty.
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.