kexec - not happening on mipsel?

kexec - not happening on mipsel?

[Andrew Sharp]

We could sure make use of kexec for quick reboots (it bloody takes
forever for the PROM to set up ECC memory).  The config option is in
Kconfig; I haven't checked the kernel source.  But the userspace package
kexec-tools barfs on the mipsel arch in configure.

Is anybody using this on MIPS?  Do the kernel portions work for anyone?

Cheers,

a


kexec-tools-1.101-kdump10$ dpkg-buildpackage -us -uc -nc -b -d -rfakeroot
dpkg-buildpackage: source package is kexec-tools
dpkg-buildpackage: source version is 1.101-kdump10-2
dpkg-buildpackage: source changed by Khalid Aziz <khalid@debian.org>
dpkg-buildpackage: host architecture mipsel
dpkg-buildpackage: source version without epoch 1.101-kdump10-2
 debian/rules build
dh_testdir
# Add here commands to configure the package.
(cd kexec-tools-1.101; ./configure --prefix=/usr --sbindir=/sbin --mandir=/usr/share/man --datadir=/usr/share)
checking build system type... mipsel-unknown-linux-gnu
checking host system type... mipsel-unknown-linux-gnu
configure: error:  unsupported architecture mipsel
make: *** [configure-stamp] Error 1

Re: kexec - not happening on mipsel?

[Andrew Sharp]

On Wed, 8 Aug 2007 17:08:46 -0700 Andrew Sharp <andy.sharp@onstor.com>
wrote:

> We could sure make use of kexec for quick reboots (it bloody takes
> forever for the PROM to set up ECC memory).  The config option is in
> Kconfig; I haven't checked the kernel source.  But the userspace
> package kexec-tools barfs on the mipsel arch in configure.

Answering my own question on this part at least, it didn't take long to
find this patch, so we'll see how that goes.

http://lists.infradead.org/pipermail/kexec/2007-June/000214.html

> Is anybody using this on MIPS?  Do the kernel portions work for
> anyone?

Re: kexec - not happening on mipsel?

[Ralf Baechle]

On Wed, Aug 08, 2007 at 06:41:20PM -0700, Andrew Sharp wrote:

> > We could sure make use of kexec for quick reboots (it bloody takes
> > forever for the PROM to set up ECC memory).  The config option is in
> > Kconfig; I haven't checked the kernel source.  But the userspace
> > package kexec-tools barfs on the mipsel arch in configure.
> 
> Answering my own question on this part at least, it didn't take long to
> find this patch, so we'll see how that goes.
> 
> http://lists.infradead.org/pipermail/kexec/2007-June/000214.html
> 
> > Is anybody using this on MIPS?  Do the kernel portions work for
> > anyone?

I recently noticed the kernel part was fairly broken, did (if ever) only
work for 32-bit machines.  I fixed what I could but I don't have the
necessary test setup.  See

  http://www.linux-mips.org/git?p=linux.git;a=commit;h=bb73f9d8ee3133800da546832ca7f09d3f27695e

  Ralf

Re: kexec - not happening on mipsel?

[Nicolas Schichan]

On Thursday 09 August 2007 14:35:30 Ralf Baechle wrote:

Hi,

> I recently noticed the kernel part was fairly broken, did (if ever) only
> work for 32-bit machines.  I fixed what I could but I don't have the
> necessary test setup.  See
> 
> http://www.linux-mips.org/git?p=linux.git;a=commit;h=bb73f9d8ee3133800da546
>832ca7f09d3f27695e

My appologies for this, I should have taken more care about 64 bit portability 
when writing this code.

I have also been trying to address the cache problem in the kexec code. please 
see the attached patch.

I now use __flush_cache_all(), since flush_cache_all() is mostly a noop on r4k 
if cpu_has_dc_aliases evaluates to 0 (and it is the case on the MIPS cpu I 
have access to). To avoid caching problem in the relocation code, I disable 
the cache in KSEG0 (using the K0 field in CP0 CONFIG register) before jumping 
to the relocation code. I don't know how clean this solution is, but this 
avoids having to add non-portable cache flush code to relocate_kernel.S.

Regards,

Signed-off-by: Nicolas Schichan <nschichan@freebox.fr>

--- linux/arch/mips/kernel/machine_kexec.c	(revision 5939)
+++ linux/arch/mips/kernel/machine_kexec.c	(revision 5941)
@@ -50,8 +50,10 @@
 	reboot_code_buffer =
 	  (unsigned long)page_address(image->control_code_page);
 
+	printk(KERN_INFO "reboot code is at %08lx\n", reboot_code_buffer);
+
 	kexec_start_address = image->start;
-	kexec_indirection_page = phys_to_virt(image->head & PAGE_MASK);
+	kexec_indirection_page = (long)phys_to_virt(image->head & PAGE_MASK);
 
 	memcpy((void*)reboot_code_buffer, relocate_new_kernel,
 	       relocate_new_kernel_size);
@@ -75,11 +77,17 @@
 	 */
 	local_irq_disable();
 
-	flush_icache_range(reboot_code_buffer,
-			   reboot_code_buffer + KEXEC_CONTROL_CODE_SIZE);
+	__flush_cache_all();
+	flush_icache_all();
 
-	printk("Will call new kernel at %08x\n", image->start);
-	printk("Bye ...\n");
-	flush_cache_all();
+	/*
+	 * avoid cache operation related headache in
+	 * relocate_kernel.S: disable caches in kseg0, the new kernel
+	 * will take care to re-enable cache in kseg0.
+	 */
+	change_c0_config(CONF_CM_CMASK, CONF_CM_UNCACHED);
+
+	printk(KERN_INFO "Will call new kernel at %08lx\n", image->start);
+	printk(KERN_INFO "Bye ...\n");
 	((void (*)(void))reboot_code_buffer)();
 }

-- 
Nicolas Schichan

Re: kexec - not happening on mipsel?

[Ralf Baechle]

On Fri, Aug 10, 2007 at 06:57:15PM +0200, Nicolas Schichan wrote:

> I now use __flush_cache_all(), since flush_cache_all() is mostly a noop on r4k 
> if cpu_has_dc_aliases evaluates to 0 (and it is the case on the MIPS cpu I 
> have access to). To avoid caching problem in the relocation code, I disable 
> the cache in KSEG0 (using the K0 field in CP0 CONFIG register) before jumping 
> to the relocation code. I don't know how clean this solution is, but this 
> avoids having to add non-portable cache flush code to relocate_kernel.S.


> 
> Regards,
> 
> Signed-off-by: Nicolas Schichan <nschichan@freebox.fr>
> 
> --- linux/arch/mips/kernel/machine_kexec.c	(revision 5939)
> +++ linux/arch/mips/kernel/machine_kexec.c	(revision 5941)
> @@ -50,8 +50,10 @@
>  	reboot_code_buffer =
>  	  (unsigned long)page_address(image->control_code_page);
>  
> +	printk(KERN_INFO "reboot code is at %08lx\n", reboot_code_buffer);
> +
>  	kexec_start_address = image->start;
> -	kexec_indirection_page = phys_to_virt(image->head & PAGE_MASK);
> +	kexec_indirection_page = (long)phys_to_virt(image->head & PAGE_MASK);
>  
>  	memcpy((void*)reboot_code_buffer, relocate_new_kernel,
>  	       relocate_new_kernel_size);
> @@ -75,11 +77,17 @@
>  	 */
>  	local_irq_disable();
>  
> -	flush_icache_range(reboot_code_buffer,
> -			   reboot_code_buffer + KEXEC_CONTROL_CODE_SIZE);
> +	__flush_cache_all();
> +	flush_icache_all();

flush_icache_all() is a nop except on VIVT caches.  Anyway, following a
__flush_cache_all it's pointless unless you want to flush caches a little
harder.

> -	printk("Will call new kernel at %08x\n", image->start);
> -	printk("Bye ...\n");
> -	flush_cache_all();
> +	/*
> +	 * avoid cache operation related headache in
> +	 * relocate_kernel.S: disable caches in kseg0, the new kernel
> +	 * will take care to re-enable cache in kseg0.
> +	 */
> +	change_c0_config(CONF_CM_CMASK, CONF_CM_UNCACHED);

And that line is the kernel's one way ticket to hell on some platforms.

There is a hazard barrier missing here.

Only KSEG0 (or CKSEG0 in 64-bit parlance) is affected by changing Config.K0
field.  But 64-bit kernels do not necessarily run in one of the 32-bit
compatibility segments (R8000 doesn't but that's an academic counter
example) - if they exist at all there is no guarantee that there is any RAM
mapped in them (IP30 and others).

On IP27 the kernel will run in CKSEG0 and if you switch that to 0 it'll mean
BRIDGE ASIC will start looking at the uncached attribute which will be
defaulted to 0 meaning that in CKSEG0 the CPU will no longer address RAM but
the ECC and Backdoor Directory information.

Basically it boils down to be very, very careful about cache modes
on MIPS.  What the kernel does on bootup only happens to work because for
all the platforms that have potencial issues with the change of the CCA
for KSEG0 we know that Linux will set the CCA to the same value that is
already in that field.  So that mode switch is sort of useless for most
platforms except a few where firmware doesn't initialize Config0.K0.

  Ralf

Re: kexec - not happening on mipsel?

[Nicolas Schichan]

On Tuesday 14 August 2007 11:47:38 Ralf Baechle wrote:

Hi,

> flush_icache_all() is a nop except on VIVT caches.  Anyway, following a
> __flush_cache_all it's pointless unless you want to flush caches a little
> harder.

The updated patch only uses __flush_cache_all().

> > -	printk("Will call new kernel at %08x\n", image->start);
> > -	printk("Bye ...\n");
> > -	flush_cache_all();
> > +	/*
> > +	 * avoid cache operation related headache in
> > +	 * relocate_kernel.S: disable caches in kseg0, the new kernel
> > +	 * will take care to re-enable cache in kseg0.
> > +	 */
> > +	change_c0_config(CONF_CM_CMASK, CONF_CM_UNCACHED);
>
> And that line is the kernel's one way ticket to hell on some platforms.
>
> There is a hazard barrier missing here.

Just for my culture, could you please specify which kind of barrier macro 
should I use here ? should it be a back_to_back_c0_hazard() ?

> Only KSEG0 (or CKSEG0 in 64-bit parlance) is affected by changing Config.K0
> field.  But 64-bit kernels do not necessarily run in one of the 32-bit
> compatibility segments (R8000 doesn't but that's an academic counter
> example) - if they exist at all there is no guarantee that there is any RAM
> mapped in them (IP30 and others).
>
> On IP27 the kernel will run in CKSEG0 and if you switch that to 0 it'll
> mean BRIDGE ASIC will start looking at the uncached attribute which will be
> defaulted to 0 meaning that in CKSEG0 the CPU will no longer address RAM
> but the ECC and Backdoor Directory information.
>
> Basically it boils down to be very, very careful about cache modes
> on MIPS.  What the kernel does on bootup only happens to work because for
> all the platforms that have potencial issues with the change of the CCA
> for KSEG0 we know that Linux will set the CCA to the same value that is
> already in that field.  So that mode switch is sort of useless for most
> platforms except a few where firmware doesn't initialize Config0.K0.

Ok, updated patch does not try to disable caching in KSEG0 using Config0.K0. 

Hoewever I so feel a bit unsafe now because D-Cache is not wrote-back and 
I-Cache is not invalidated in relocate_kernel.S, before jumping to the new 
kernel. This happens to work on my board, but I think that it is mostly 
because of luck. Maybe using KSEG1 or XKPHYS (not sure about this one, I am 
not familiar with 64bit mips) when fixing the indirection list addresses 
should be safer.

However, updated patch follows.

Regards.

Signed-off-by: Nicolas Schichan <nschichan@freebox.fr>

--- linux/arch/mips/kernel/machine_kexec.c	(revision 5975)
+++ linux/arch/mips/kernel/machine_kexec.c	(revision 5976)
@@ -78,11 +78,8 @@
 	 */
 	local_irq_disable();
 
-	flush_icache_range(reboot_code_buffer,
-			   reboot_code_buffer + KEXEC_CONTROL_CODE_SIZE);
-
 	printk("Will call new kernel at %08lx\n", image->start);
 	printk("Bye ...\n");
-	flush_cache_all();
+	__flush_cache_all();
 	((noretfun_t) reboot_code_buffer)();
 }

-- 
Nicolas Schichan

Re: kexec - not happening on mipsel?

[Ralf Baechle]

On Mon, Aug 20, 2007 at 03:57:38PM +0200, Nicolas Schichan wrote:

> The updated patch only uses __flush_cache_all().

Applied, thanks.

> > There is a hazard barrier missing here.
> 
> Just for my culture, could you please specify which kind of barrier macro 
> should I use here ? should it be a back_to_back_c0_hazard() ?

According to the MIPS architecture, a mfc0 instruction that is immediately
following a mtc0 instruction where both are accessing the same cp0
register, has undefined operation.  Not all processors have this hazard and
for those those which have it the resolution before version 2 of the
architecture did differ between MIPS implementation.
back_to_back_c0_hazard() will solve this problem.  For example:

	write_c0_status(read_c0_status() & ~ ST0_BEV);
	write_c0_status(read_c0_status() | ST0_IE);

could possibly be compiled into such a mtc0 mfc0 sequence.  Throwing in a
back_to_back_c0_hazard() line will fix that.

	write_c0_status(read_c0_status() & ~ ST0_BEV);
	back_to_back_c0_hazard();
	write_c0_status(read_c0_status() | ST0_IE);

On a modern processor back_to_back_c0_hazard() will expand into a EHB,
on an antique to whatever serves the equivalent job.

Changing Config.K0 also creates an "instruction hazard".  Even when the
mtc0 has been graduated in the pipeline it needs to be ensured that no
instructions are being fetched and executed based on the old value.

My description is somewhat to short to be accurate, I suggest you read
up on it in the MIPS R2 architecture specification, See MIPS Run and of
course in the manual for your specific processor.

> Hoewever I so feel a bit unsafe now because D-Cache is not wrote-back and 
> I-Cache is not invalidated in relocate_kernel.S, before jumping to the new 
> kernel. This happens to work on my board, but I think that it is mostly 
> because of luck. Maybe using KSEG1 or XKPHYS (not sure about this one, I am 
> not familiar with 64bit mips) when fixing the indirection list addresses 
> should be safer.

KSEG1 would have the same issues as KSEG0 configured to uncached.

But really, why making things more complicated than required.  A flush
should do the trick.

Another open end in your patch would be SMP; normally Linux expects that
on bootup all processors are under firmware control but that is not the
case when a new kernel is initializing after being kexeced.

  Ralf