Re: [Fastboot] [PATCH] kexec on ia64

Re: [Fastboot] [PATCH] kexec on ia64

[Khalid Aziz]

On Mon, 2004-11-15 at 13:32, Khalid Aziz wrote:
> Here is what I am working on next:
> 
> 1. Save EFI memory map before it is trimmed.
> 
> 2. Fix up /proc/iomem on ia64 so we can enable validating memory range
> in kexec tools.
> 
> 3. Add a /proc interface to enable reboots on panic and INIT (and
> possibly MCA) to be kexec reboots.
> 
> 4. Add initrd support.

And

5. Port the patch to 2.6.9 kernel :) Or 2.6.10 if I do not get to it
soon enough.

-- 
Khalid

==================================
Khalid Aziz                                Linux and Open Source Lab
(970)898-9214                                        Hewlett-Packard
khalid_aziz@hp.com                                  Fort Collins, CO

"The Linux kernel is subject to relentless development" 
				- Alessandro Rubini

Re: [Fastboot] [PATCH] kexec on ia64

[Khalid Aziz]

Another limitation I forgot to mention. I have not added support for
compressed kernel to kexec-tools yet. It is on my list of things to do
next.

--
Khalid

On Mon, 2004-11-15 at 13:32, Khalid Aziz wrote:
> I have been able to get kexec working on ia64. I am attaching the kernel
> patch and kexec-tools patch. For the kernel patch, start with 2.6.8
> kernel from kernel.org, apply ia64 patch
> <http://www.kernel.org/pub/linux/kernel/ports/ia64/v2.6/linux-2.6.8-ia64-040901.diff.bz2>, apply Eric' 2.6.8.1-kexec3 patch <http://www.xmission.com/~ebiederm/files/kexec/2.6.8.1-kexec3> and apply attached 2.6.8.1-kexec3-ia64.diff patch. For kexec-tools, apply attached kexec-tools-1.98-ia64.diff patch to Eric's kexec-tools 1.98 sources <http://www.xmission.com/~ebiederm/files/kexec/kexec-tools-1.98.tgz>.
> 
> At this point, I have done minimal testing. Here is what I know does not
> work currently:
> 
> 1. No support for initrd for kexec'd kernel
> 
> 2. No support for new kernel parameters for kexec'd kernel.
> 
> 3. If a kernel is booted up with "mem=" or "max_addr=" to restrict the
> amount of memory, a kernel kexec'd from this kernel will only see the
> same amount of memory as this one. This is not only due to the new
> kernel being kexec'd with the same parameter, but also becuase the EFI
> memory map as passed to the kernel by ELILO gets trimmed very early on
> by the kernel. I have tried adding code to save the memory map early on
> and then pass this saved memory map to kexec'd kernel, but apparently I
> still am not saving it early enough. I wait until bootmem allocator has
> been initailized so I can allocate memory to save unmolested EFI memory
> map in. In the process of initializing bootmem allocator, kernel calls
> efi-Memory_map_walk() which in turn trims the memory map. I am looking
> into allocating memory out of the EFI memory map before the first
> efi_mem_map_walk() happens, so I can save pristine EFI memmap for use
> later by kexec.
> 
> Here is what I have not tested yet:
> 
> 1. I am not sure if  ACPI subsystem is happy in kexec'd kernel. I have
> not seen any problems, but I have not tested it enough either.
> 
> 2. Stability of kexec'd kernel over long term. It ran fine for an hour
> not doing much :)
> 
> Here is what I am working on next:
> 
> 1. Save EFI memory map before it is trimmed.
> 
> 2. Fix up /proc/iomem on ia64 so we can enable validating memory range
> in kexec tools.
> 
> 3. Add a /proc interface to enable reboots on panic and INIT (and
> possibly MCA) to be kexec reboots.
> 
> 4. Add initrd support.
> 
> Any feedback on these patches is welcome. Any patch to fix problems in
> these patches is very much appreciated :)
-- 
Khalid Aziz                                 Linux and Open Source Lab
(970)898-9214                                         Hewlett-Packard
khalid_aziz@hp.com                                   Fort Collins, CO

Re: [Fastboot] [PATCH] kexec on ia64

[Eric W. Biederman]

Khalid Aziz <khalid_aziz@hp.com> writes:

> Add kexec support on ia64.

This looks like a starting place but this patch needs some
more work.

> Signed-off-by: Khalid Aziz <khalid.aziz@hp.com>
> ---
>
> diff -urNp linux-2.6.16/arch/ia64/hp/common/sba_iommu.c
> linux-2.6.16-kexec/arch/ia64/hp/common/sba_iommu.c
> --- linux-2.6.16/arch/ia64/hp/common/sba_iommu.c 2006-03-19 22:53:29.000000000
> -0700
> +++ linux-2.6.16-kexec/arch/ia64/hp/common/sba_iommu.c 2006-03-27
> 15:42:47.000000000 -0700
> @@ -1624,6 +1624,28 @@ ioc_iova_init(struct ioc *ioc)
>  	READ_REG(ioc->ioc_hpa + IOC_IBASE);
>  }
>  
> +#ifdef CONFIG_KEXEC
> +void
> +ioc_iova_disable(void)
> +{
> +	struct ioc *ioc;
> +
> +	ioc = ioc_list;
> +
> +	while (ioc != NULL) {
> +		/* Disable IOVA translation */
> + WRITE_REG(ioc->ibase & 0xfffffffffffffffe, ioc->ioc_hpa + IOC_IBASE);
> +		READ_REG(ioc->ioc_hpa + IOC_IBASE);
> +
> +		/* Clear I/O TLB of any possible entries */
> + WRITE_REG(ioc->ibase | (get_iovp_order(ioc->iov_size) + iovp_shift),
> ioc->ioc_hpa + IOC_PCOM);
> +		READ_REG(ioc->ioc_hpa + IOC_PCOM);
> +
> +		ioc = ioc->next;
> +	}
> +}
> +#endif
> +
>  static void __init
>  ioc_resource_init(struct ioc *ioc)
>  {
> diff -urNp linux-2.6.16/arch/ia64/Kconfig linux-2.6.16-kexec/arch/ia64/Kconfig
> --- linux-2.6.16/arch/ia64/Kconfig	2006-03-19 22:53:29.000000000 -0700
> +++ linux-2.6.16-kexec/arch/ia64/Kconfig 2006-03-27 15:42:47.000000000 -0700
> @@ -376,6 +376,23 @@ config IA64_PALINFO
>  config SGI_SN
>  	def_bool y if (IA64_SGI_SN2 || IA64_GENERIC)
>  
> +config KEXEC
> +	bool "kexec system call (EXPERIMENTAL)"
> +	depends on EXPERIMENTAL
> +	help
> +	  kexec is a system call that implements the ability to shutdown your
> +	  current kernel, and to start another kernel.  It is like a reboot
> +	  but it is indepedent of the system firmware.   And like a reboot
> +	  you can start any kernel with it, not just Linux.
> +
> +	  The name comes from the similiarity to the exec system call.
> +
> +	  It is an ongoing process to be certain the hardware in a machine
> +	  is properly shutdown, so do not be surprised if this code does not
> +	  initially work for you.  It may help to enable device hotplugging
> +	  support.  As of this writing the exact hardware interface is
> +	  strongly in flux, so no good recommendation can be made.
> +
>  source "drivers/firmware/Kconfig"
>  
>  source "fs/Kconfig.binfmt"
> diff -urNp linux-2.6.16/arch/ia64/kernel/crash.c
> linux-2.6.16-kexec/arch/ia64/kernel/crash.c
> --- linux-2.6.16/arch/ia64/kernel/crash.c 1969-12-31 17:00:00.000000000 -0700
> +++ linux-2.6.16-kexec/arch/ia64/kernel/crash.c 2006-03-27 15:49:44.000000000
> -0700
> @@ -0,0 +1,43 @@
> +/*
> + * arch/ia64/kernel/crash.c
> + *
> + * Architecture specific (ia64) functions for kexec based crash dumps.
> + *
> + * Created by: Khalid Aziz <khalid.aziz@hp.com>
> + *
> + * Copyright (C) 2005 Hewlett-Packard Development Company, L.P.
> + *
> + */
> +#include <linux/init.h>
> +#include <linux/types.h>
> +#include <linux/kernel.h>
> +#include <linux/smp.h>
> +#include <linux/irq.h>
> +#include <linux/reboot.h>
> +#include <linux/kexec.h>
> +#include <linux/irq.h>
> +#include <linux/delay.h>
> +#include <linux/elf.h>
> +#include <linux/elfcore.h>
> +#include <linux/device.h>
> +
> +void
> +machine_crash_shutdown(struct pt_regs *pt)
> +{
> +	/* This function is only called after the system
> +	 * has paniced or is otherwise in a critical state.
> +	 * The minimum amount of code to allow a kexec'd kernel
> +	 * to run successfully needs to happen here.
> +	 *
> +	 * In practice this means shooting down the other cpus in
> +	 * an SMP system.
> +	 */
> +	if (in_interrupt()) {
> +		terminate_irqs();
> +		ia64_eoi();
> +	}
> +	system_state = SYSTEM_RESTART;
> +	device_shutdown();
> +	system_state = SYSTEM_BOOTING;
> +	machine_shutdown();
> +}

machine_crash_shutdown must not call device_shutdown.  That has
been shown to way exceed the minimum necessary to shutdown a system.
I would prefer this to be a noop stub that doesn't work at all than
something like this that does way too much, and makes people think
the code will work.

As for terminate_irqs on x86 we do that on bootup not in the middle
of a crash shutdown.  The apics and xapics are close enough you
should be able to do the same on ia64.

You display remarkable faith in a kernel that has paniced.

> +#ifdef CONFIG_PCI
> +void machine_shutdown(void)
> +{
> +	struct pci_dev *dev;
> +	irq_desc_t *idesc;
> +	cpumask_t mask = CPU_MASK_NONE;
> +
> +	/* Disable all PCI devices */
> +	list_for_each_entry(dev, &pci_devices, global_list) {
> +		if (!(dev->is_enabled))
> +			continue;
> +		idesc = irq_descp(dev->irq);
> +		if (!idesc)
> +			continue;
> +		cpu_set(0, mask);
> +		disable_irq_nosync(dev->irq);
> +		idesc->handler->end(dev->irq);
> +		idesc->handler->set_affinity(dev->irq, mask);
> +		idesc->action = NULL;
> +		pci_disable_device(dev);
> +		pci_set_power_state(dev, 0);
> +	}
> +}
> +#endif

This is peculiar but almost sane.  We don't do this on x86,
because devices are peculiar enough that no generic sequence works.
What you have above belongs in the shutdown methods of the pci
devices.  There is no way to get this right in the general case.

some of the irq disable logic may in fact be sane.

Unless there is a good reason not to machine_shutdown needs
to be called from machine_restart.  So the code is routinely
used and tested.

Having machine_shutdown only build when you have PCI present
and then not making KEXEC depend on PCI is wrong.

The #ifdef needs to move inside machine_shutdown.

> +
> +/*
> + * Do not allocate memory (or fail in any way) in machine_kexec().
> + * We are past the point of no return, committed to rebooting now. 
> + */
> +void machine_kexec(struct kimage *image)
> +{
> +	unsigned long indirection_page;
> +	relocate_new_kernel_t rnk;
> +	unsigned long pta, impl_va_bits;
> +	void *pal_addr = efi_get_pal_addr();
> + unsigned long code_addr = (unsigned
> long)page_address(image->control_code_page);
> +
> +#ifdef CONFIG_HOTPLUG_CPU
> +	int cpu;
> +
> +	for_each_online_cpu(cpu) {
> +		if (cpu != smp_processor_id())
> +			cpu_down(cpu);
> +	}
> +#elif CONFIG_SMP
> +	smp_call_function(kexec_stop_this_cpu, (void *)image->start, 0, 0);
> +#endif

This CPU and HOTPUG_CPU stuff belongs in machine shutdown.

> +
> +	ia64_set_itv(1<<16);
> +	/* Interrupts aren't acceptable while we reboot */
> +	local_irq_disable();
> +
> +	/* set kr0 to the appropriate address */
> +	set_io_base();
> +
> +	/* Disable VHPT */
> +	impl_va_bits = ffz(~(local_cpu_data->unimpl_va_mask | (7UL << 61)));
> +	pta = POW2(61) - POW2(vmlpt_bits);
> +	ia64_set_pta(pta | (0 << 8) | (vmlpt_bits << 2) | 0);
> +
> +#ifdef CONFIG_IA64_HP_ZX1
> +	ioc_iova_disable();
> +#endif

This also looks like it needs to be part of machine_shutdown.
I have no confidence in ioc_iova_disable when the machine is crashing.
Basically anything that touches a pointer is likely to be bad.

> +	/* now execute the control code.
> +	 * We will start by executing the control code linked into the 
> + * kernel as opposed to the code we copied in control code buffer * page. When
> this code switches to physical mode, we will start
> +	 * executing the code in control code buffer page. Reason for
> +	 * doing this is we start code execution in virtual address space.
> +	 * If we were to try to execute the newly copied code in virtual
> +	 * address space, we will need to make an ITLB entry to avoid ITLB 
> +	 * miss. By executing the code linked into kernel, we take advantage
> +	 * of the ITLB entry already in place for kernel and avoid making
> +	 * a new entry.
> +	 */
> +	indirection_page = image->head & PAGE_MASK;
> +
> +	rnk = (relocate_new_kernel_t)&code_addr;
> +	(*rnk)(indirection_page, image->start, ia64_boot_param,
> +		     GRANULEROUNDDOWN((unsigned long) pal_addr));
> +	BUG();
> +	for (;;)
> +		;
> +}


Eric

RE: [Fastboot] [PATCH] kexec on ia64

[Zou, Nanhai]

> -----Original Message-----
> From: linux-ia64-owner@vger.kernel.org
> [mailto:linux-ia64-owner@vger.kernel.org] On Behalf Of Eric W. Biederman
> Sent: 2006Äê4ÔÂ5ÈÕ 2:14
> To: Khalid Aziz
> Cc: LKML; Fastboot mailing list; Linux ia64
> Subject: Re: [Fastboot] [PATCH] kexec on ia64
> 
> Khalid Aziz <khalid_aziz@hp.com> writes:
> 
> > Add kexec support on ia64.
> 
> This looks like a starting place but this patch needs some
> more work.
> 
Eric,
	Khalid is also merging my ia64 kdump patch posted in http://lkml.org/lkml/2006/3/14/46.
	Hopefully those issues you pointed out will be solved once the kdump patch is merged. 

Thanks
Zou Nan hai

RE: [Fastboot] [PATCH] kexec on ia64

[Zou, Nanhai]

> -----Original Message-----
> From: KAMEZAWA Hiroyuki [mailto:kamezawa.hiroyu@jp.fujitsu.com]
> Sent: 2006年4月5日 9:13
> To: Zou, Nanhai
> Cc: ebiederm@xmission.com; khalid_aziz@hp.com;
> linux-kernel@vger.kernel.org; fastboot@lists.osdl.org;
> linux-ia64@vger.kernel.org
> Subject: Re: [Fastboot] [PATCH] kexec on ia64
> 
> On Wed, 5 Apr 2006 08:36:07 +0800
> "Zou, Nanhai" <nanhai.zou@intel.com> wrote:
> 
> > > -----Original Message-----
> > > From: linux-ia64-owner@vger.kernel.org
> > > [mailto:linux-ia64-owner@vger.kernel.org] On Behalf Of Eric W. Biederman
> > > Sent: 2006年4月5日 2:14
> > > To: Khalid Aziz
> > > Cc: LKML; Fastboot mailing list; Linux ia64
> > > Subject: Re: [Fastboot] [PATCH] kexec on ia64
> > >
> > > Khalid Aziz <khalid_aziz@hp.com> writes:
> > >
> > > > Add kexec support on ia64.
> > >
> > > This looks like a starting place but this patch needs some
> > > more work.
> > >
> > Eric,
> > 	Khalid is also merging my ia64 kdump patch posted in
> http://lkml.org/lkml/2006/3/14/46.
> > 	Hopefully those issues you pointed out will be solved once the kdump patch
> is merged.
> >
> Hi, I have a question about kexec/kdump.
> 
> How does kdump know memory layout (of old kernel) now ?
> 
> I'm working for memory hotplug. When memory is hot-added, memory layout
> changes.
> But I think there is no code to manage memory layout information of added memory.
> 
 It reads memory layout from /proc/iomem...,
 If memory is hotpluged, I think we need a reload of kdump.


> Thanks,
> - Kame
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Fastboot] [PATCH] kexec on ia64

[KAMEZAWA Hiroyuki]

On Wed, 5 Apr 2006 09:13:36 +0800
"Zou, Nanhai" <nanhai.zou@intel.com> wrote:
> > I'm working for memory hotplug. When memory is hot-added, memory layout
> > changes.
> > But I think there is no code to manage memory layout information of added memory.
> > 
>  It reads memory layout from /proc/iomem...,
>  If memory is hotpluged, I think we need a reload of kdump.
> 
If /proc/iomem is updated at hotplug event (this is not updated now),
is there no problem ?

calling insert_resource like  efi_initialize_iomem_resources() is good ?

-Kame

RE: [Fastboot] [PATCH] kexec on ia64

[Zou, Nanhai]

> -----Original Message-----
> From: KAMEZAWA Hiroyuki [mailto:kamezawa.hiroyu@jp.fujitsu.com]
> Sent: 2006Äê4ÔÂ5ÈÕ 9:28
> To: Zou, Nanhai
> Cc: ebiederm@xmission.com; khalid_aziz@hp.com; linux-kernel@vger.kernel.org;
> fastboot@lists.osdl.org; linux-ia64@vger.kernel.org
> Subject: Re: [Fastboot] [PATCH] kexec on ia64
> 
> On Wed, 5 Apr 2006 09:13:36 +0800
> "Zou, Nanhai" <nanhai.zou@intel.com> wrote:
> > > I'm working for memory hotplug. When memory is hot-added, memory layout
> > > changes.
> > > But I think there is no code to manage memory layout information of added
> memory.
> > >
> >  It reads memory layout from /proc/iomem...,
> >  If memory is hotpluged, I think we need a reload of kdump.
> >
> If /proc/iomem is updated at hotplug event (this is not updated now),
> is there no problem ?
> 
 The crash dumping kernel also needs a reload, because the physical memory list is read and saved at kdump kernel loading time instead of crashing time.

Zou Nan hai

Re: [Fastboot] [PATCH] kexec on ia64

[Eric W. Biederman]

KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> writes:

> Hi, I have a question about kexec/kdump.
>
> How does kdump know memory layout (of old kernel) now ?
>
> I'm working for memory hotplug. When memory is hot-added, memory layout changes.
> But I think there is no code to manage memory layout information of added
> memory.

It is passed from one kernel to another, and it is memorized when you load
the crash dump kernel.  If your memory layout changes you need to reload
the crash dump kernel from user space with the appropriate hotplug script.  

Unless this happens often it shouldn't be a problem. 

And yes this does leave a small race during which kexec on panic won't
work.

Eric

Re: [Fastboot] [PATCH] kexec on ia64

[KAMEZAWA Hiroyuki]

On Tue, 04 Apr 2006 20:49:49 -0600
ebiederm@xmission.com (Eric W. Biederman) wrote:

> KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> writes:
> 
> > Hi, I have a question about kexec/kdump.
> >
> > How does kdump know memory layout (of old kernel) now ?
> >
> > I'm working for memory hotplug. When memory is hot-added, memory layout changes.
> > But I think there is no code to manage memory layout information of added
> > memory.
> 
> It is passed from one kernel to another, and it is memorized when you load
> the crash dump kernel.  If your memory layout changes you need to reload
> the crash dump kernel from user space with the appropriate hotplug script.  
> 
> Unless this happens often it shouldn't be a problem. 
> 

> And yes this does leave a small race during which kexec on panic won't
> work.

Hmm.. Okay. 
Before reloading kdump kernel, kdump continues to use old information.
(when adding, it's not be big problem.)

Thank you.
- Kame

Re: [Fastboot] [PATCH] kexec on ia64

[Khalid Aziz]

On Tue, 2006-04-04 at 12:13 -0600, Eric W. Biederman wrote:
> Khalid Aziz <khalid_aziz@hp.com> writes:
> > +void
> > +machine_crash_shutdown(struct pt_regs *pt)
> > +{
> > +	/* This function is only called after the system
> > +	 * has paniced or is otherwise in a critical state.
> > +	 * The minimum amount of code to allow a kexec'd kernel
> > +	 * to run successfully needs to happen here.
> > +	 *
> > +	 * In practice this means shooting down the other cpus in
> > +	 * an SMP system.
> > +	 */
> > +	if (in_interrupt()) {
> > +		terminate_irqs();
> > +		ia64_eoi();
> > +	}
> > +	system_state = SYSTEM_RESTART;
> > +	device_shutdown();
> > +	system_state = SYSTEM_BOOTING;
> > +	machine_shutdown();
> > +}
> 
> machine_crash_shutdown must not call device_shutdown.  That has
> been shown to way exceed the minimum necessary to shutdown a system.
> I would prefer this to be a noop stub that doesn't work at all than
> something like this that does way too much, and makes people think
> the code will work.
> 
> As for terminate_irqs on x86 we do that on bootup not in the middle
> of a crash shutdown.  The apics and xapics are close enough you
> should be able to do the same on ia64.
> 
> You display remarkable faith in a kernel that has paniced.

I will look into eliminating this as much as possible.

> Having machine_shutdown only build when you have PCI present
> and then not making KEXEC depend on PCI is wrong.
> 
> The #ifdef needs to move inside machine_shutdown.

Fixed.

> 
> > +
> > +/*
> > + * Do not allocate memory (or fail in any way) in machine_kexec().
> > + * We are past the point of no return, committed to rebooting now. 
> > + */
> > +void machine_kexec(struct kimage *image)
> > +{
> > +	unsigned long indirection_page;
> > +	relocate_new_kernel_t rnk;
> > +	unsigned long pta, impl_va_bits;
> > +	void *pal_addr = efi_get_pal_addr();
> > + unsigned long code_addr = (unsigned
> > long)page_address(image->control_code_page);
> > +
> > +#ifdef CONFIG_HOTPLUG_CPU
> > +	int cpu;
> > +
> > +	for_each_online_cpu(cpu) {
> > +		if (cpu != smp_processor_id())
> > +			cpu_down(cpu);
> > +	}
> > +#elif CONFIG_SMP
> > +	smp_call_function(kexec_stop_this_cpu, (void *)image->start, 0, 0);
> > +#endif
> 
> This CPU and HOTPUG_CPU stuff belongs in machine shutdown.

Moved to machine_shutdown().

> 
> > +
> > +	ia64_set_itv(1<<16);
> > +	/* Interrupts aren't acceptable while we reboot */
> > +	local_irq_disable();
> > +
> > +	/* set kr0 to the appropriate address */
> > +	set_io_base();
> > +
> > +	/* Disable VHPT */
> > +	impl_va_bits = ffz(~(local_cpu_data->unimpl_va_mask | (7UL << 61)));
> > +	pta = POW2(61) - POW2(vmlpt_bits);
> > +	ia64_set_pta(pta | (0 << 8) | (vmlpt_bits << 2) | 0);
> > +
> > +#ifdef CONFIG_IA64_HP_ZX1
> > +	ioc_iova_disable();
> > +#endif
> 
> This also looks like it needs to be part of machine_shutdown.
> I have no confidence in ioc_iova_disable when the machine is crashing.
> Basically anything that touches a pointer is likely to be bad.

I have moved above code to machine_shutdown. I would prefer to delay
disabling VHPT as much as possible, but since machine_kexec gets called
soon after machine_shutdown and we should be executing kernel code
strictly at this point which uses pinned TR entries, disabling VHPT
should not have any deleterious effect.

> 
> > +	/* now execute the control code.
> > +	 * We will start by executing the control code linked into the 
> > + * kernel as opposed to the code we copied in control code buffer * page. When
> > this code switches to physical mode, we will start
> > +	 * executing the code in control code buffer page. Reason for
> > +	 * doing this is we start code execution in virtual address space.
> > +	 * If we were to try to execute the newly copied code in virtual
> > +	 * address space, we will need to make an ITLB entry to avoid ITLB 
> > +	 * miss. By executing the code linked into kernel, we take advantage
> > +	 * of the ITLB entry already in place for kernel and avoid making
> > +	 * a new entry.
> > +	 */
> > +	indirection_page = image->head & PAGE_MASK;
> > +
> > +	rnk = (relocate_new_kernel_t)&code_addr;
> > +	(*rnk)(indirection_page, image->start, ia64_boot_param,
> > +		     GRANULEROUNDDOWN((unsigned long) pal_addr));
> > +	BUG();
> > +	for (;;)
> > +		;
> > +}
> 
> 
> Eric

Thanks for the review.

-- 
Khalid

==================================
Khalid Aziz                       Open Source and Linux Organization
(970)898-9214                                        Hewlett-Packard
khalid.aziz@hp.com                                  Fort Collins, CO

"The Linux kernel is subject to relentless development" 
                                - Alessandro Rubini