Here's a proof of concept patch (against 2.5.67) that covers
most of the features we've been talking about in this thread.
This builds and boots both UP and SMP on a Tiger.

Kernel text and data are linked into region 5 at address
0xA000000100000000, leaving the bottom 4G of region 5 available
for miscellaneous stuff as suggested by David.  It will be possible
to extend this patch to split text and data into separate sections
to allow kernel text replication, but I left that out for now to
keep this patch to the bare minimum.

This version still uses the __tpa() macro, but converting it to
use the ia64_tpa() inline function is just a matter of getting
all the types (argument and return value) so that the compiler
doesn't spit warnings all over the screen during build.

This patch doesn't fixup fs/proc/kcore.c (my old patch didn't
apply cleanly ... and much of the changes it would have made are
no longer needed as kernel is now at a lower address than modules).

Here's a file by file description of the changes:

arch/ia64/kernel/efi_stub.S
	calls to ia64_switch_mode replaced by calls to
	ia64_switch_mode_phys or ia64_switch_mode_virt
	as appropriate
arch/ia64/kernel/entry.S
	ia64_switch_to() doesn't need to test whether stack
	overlaps DTR[0] (it can't, kernel and stack are in
	different regions now).
arch/ia64/kernel/head.S
	Initialize all the kernel region registers here (we
	now have to do rr[5] and rr[7], so I moved rr[6] up
	to the top and dropped the EARLY_PRINTK #ifdef for
	symmetry).  Map itr[0]/dtr[0] based on actual load
	address.  Map dtr[2] to the region 7 address of the
	stack, and set IA64_KR(CURRENT_STACK) to correct
	granule number.
	Replace ia64_switch_mode() from a function that toggles
	virtual mode on/off with separate functions to go from
	virt to phys mode, and from phys to virt mode.
arch/ia64/kernel/ia64_ksyms.c
	Export zero_page_memmap_ptr
arch/ia64/kernel/ivt.S
	Provide labels on code that needs to be patched with the
	physical address of swapper_pg_dir
arch/ia64/kernel/mca.c
	Lots of __pa() need to be __tpa()
arch/ia64/kernel/pal.S
	calls to ia64_switch_mode replaced by calls to
	ia64_switch_mode_phys or ia64_switch_mode_virt
	as appropriate
arch/ia64/kernel/setup.c
	Get correct page numbers when marking start/end of kernel.
	Code to patch ivt.S with physical address of swapper_pg_dir.
	Pass region 7 virtual address of per-cpu area to
	ia64_mmu_init() [UP code would have passed a region 5 kernel
	address, which chokes the __pa() call in ia64_mmu_init()]	
arch/ia64/kernel/smpboot.c
	Use __tpa() to get physical address of entry point to start
	other cpus
arch/ia64/mm/init.c
	free_initmem() needs to be smarter about addresses
	initialization of rr[5] and rr[6] moved to head.S
	setup zero_page_memmap_ptr
arch/ia64/vmlinux.lds.S
	Set LOAD_OFFSET to a sane default value for DIG machines,
	kernel loaded at 64MB ... on machines where this patch is
	really needed, elilo will have to ignore the load address
	and put the kernel anyplace there is suitably aligned memory.
	Use LOAD_OFFSET rather than PAGE_OFFSET throughout the rest
	of the file.
include/asm-ia64/page.h
	define __tpa() and __imva() macros
include/asm-ia64/pgtable.h
	Move VMALLOC_START up above kernel (region 5 base + 8GB)
	New definition of ZERO_PAGE uses zero_page_memmap_ptr
	Drop KERNEL_TR_PAGE_NUM define, it is meaningless now.
include/asm-ia64/system.h
	KERNEL_START moves to region 5 base + 4GB