Re: [Linux-ia64] Re: [PATCH] head.S fix for unusual load addrs

Jack Steiner
2003-05-09T19:31:54Z

>
> On Wed, May 07, 2003 at 04:24:09PM -0700, Luck, Tony wrote:
> > 1) My patch (posted around October last year) which picked virtual addresses
> > in the wild blue yonder (initial versions used 0xe002000000000000, later ones
> > used 0xffffffff00000000) for the link address for the kernel. Elilo can load
> > kernel at any suitably aligned physical address, and head.S establishes the
> > mappings using itr[0] and dtr[0].
> >
> > pros) provided separate maps for kernel text and data, so supported kernel text
> > replication too.
> > cons) __pa() no longer works on kernel addresses, use new __tpa() instead.
> > Some ugly runtime patching of kernel code to get physical address of
> > swapper_pg_dir into the TLB miss code.

The __tpa macros are ugly but they are fully contained within the ia64 part
of the tree. (IIRC, the old scheduler had a reference but the O(1) scheduler doe not).
In our tree, there are currently only 12 references to __tpa. All are
in boottime initialization code., mostly in mca.c. Although I would
rather not have __tpa, this doesnt seem too bad.

> >
> > 2) I think SGI are currently running a modified version of #1 without the text
> > replication support, and that provides a mapping from the normal virtual
> > address that the kernel is linked for (0xe00000000044000000) to whatever physical
> > address it was actually loaded at ... at least I think that's what Jack said.

Right.

> >
> > pros) simpler than my patch
> >
> > cons) Still needs __tpa() instead of __pa() for kernel addresses.
> >
> > 3) David's suggestion of boot-time relocation. Probably simplest to implement
> > this in elilo, but if you are really good at PIC asm code it could be done in
> > the kernel startup sequence.
> >
> > pros) Just like linking kernel at a new address.
> > Avoids the __tpa() issue.
> > Doesn't invalidate any assumptions about how to get from virtual to
> > physical addresses and back again.
> >
> > cons) Nobody has implemented it.
> doesn't address text replication concerns, while (1) and (2)
> do
>
> So, is there any consensus on the best path to pursue? Chris Wedgwood
> is working on option #3, and I've got Tony's patch trimmed down to #2
> (with one piece missing--ia64_switch_to runtime patching), but none of
> these are in either 2.4 or 2.5 yet. Maybe for 2.4 we should do #2 or
> #3 and for 2.5 we could implement #1 with the virtual offsets Tony
> mentioned earlier?

As far as I can tell, #1 is the only solution that will support kernel text replication.
We did experiments to measure the effect of kernel text replication. On AIM7,
(granted, not the best benchmark), we see a small but consistent
improvement in performance with text replication enabled. (Data attached below).
The biggest difference is cputime, not throughput.

This was from a relatively small system with very good remote-to-local latency
ratios. As system sizes increase, I expect the benefit of text replication will
increase.

I think that whatever solution is adopted, it need to accommadate text replication.

If we are going to do #1 for 2.5, it seems like #2 is the best solution for 2.4.

#3 requires changes elilo & the way System.map is used. These changes wont apply
to 2.5. #3 also will introduce some confusion since the System.map that is
generated at build time cant be used without knowing the physical address
where elilo actually loaded the kernel. The address could change based on minor
changes in the configuration (amount of memory, devices, nodes, etc).

------

AIM7
Kernel Text replication:
Tasks Jobs/Min JTI Real CPU Jobs/sec/task
200 18110.2 84 64.3 619.6 1.5092
500 23932.5 76 121.6 2947.1 0.7977
1000 24402.0 71 238.5 9890.6 0.4067
2000 23895.4 71 487.1 25537.8 0.1991
3000 22848.8 75 764.2 42549.7 0.1269
4000 20420.4 78 1140.0 64237.4 0.0851

Baseline
Tasks Jobs/Min JTI Real CPU Jobs/sec/task
200 18008.8 84 64.6 624.1 1.5007
500 23102.2 76 126.0 4238.1 0.7701
1000 23886.4 71 243.7 10507.3 0.3981
2000 23669.9 72 491.8 26120.2 0.1972
3000 22344.9 75 781.4 42965.0 0.1241
4000 20256.9 78 1149.2 64711.0 0.0844

>
> Thanks,
> Jesse
>
> _______________________________________________
> Linux-IA64 mailing list
> Linux-IA64@linuxia64.org
> http://lists.linuxia64.org/lists/listinfo/linux-ia64
>

--
Thanks

Jack Steiner (651-683-5302) (vnet 233-5302) steiner@sgi.com