Kirkwood, kernel 3.2, vmalloc region overlap, not starting up

Kirkwood, kernel 3.2, vmalloc region overlap, not starting up

[Nils Faerber]

Hello!
I am in the process of upgrading board support for my TK71 board from
2.6.38 to 3.2 and am now halted by this strange issue...

The board support for the TK71 is almost identical to the 3.2 included
board files for e.g. the Sheevaplug or the rd88f6281 reference design.
After applying my board patch and re-inserting my machine ID again the
kernel freezes very early.

After enabling earlyprintk I get the following:

Uncompressing Linux... done, booting the kernel.

[    0.000000] Linux version 3.2.0-00012-g980d683-dirty (nils at moi) (gcc
version 4.4.2 (GCC) ) #30 Tue Feb 7 15:18:57 CET 2012
[    0.000000] CPU: Feroceon 88FR131 [56251311] revision 1 (ARMv5TE),
cr=00053977
[    0.000000] CPU: VIVT data cache, VIVT instruction cache

[    0.000000] Machine: TK71 Kirkwood based Q7 formfactor board

[    0.000000] bootconsole [earlycon0] enabled

[    0.000000] Ignoring RAM at 00000000-0fffffff (vmalloc region
overlap).
[    0.000000] Memory policy: ECC disabled, Data cache writeback

[    0.000000] Kernel panic - not syncing: ERROR: Failed to allocate
0x1000 bytes below 0x0.
[    0.000000]

[    0.000000] Backtrace:

[    0.000000] [<c000c2a0>] (dump_backtrace+0x0/0x110) from [<c0319b84>]
(dump_stack+0x18/0x1c)

The overlapping region in question is exactly my whole RAM so no
surprise the kernel will not start-up much further. The whole stuff is
working properly with 2.6.38.
I am using U-Boot as bootloader - "U-Boot 2010.03-01252-gcb89b82-dirty".

Since the only board/platform specific patch is in my board file I
compared that very carefully to the other Kirkwood boards that exist in
the 3.2 kernel - but I did not find any major difference and especially
nothing that could explain this memory region problem.

Are there any other changes that I am missing from 2.6.38 to 3.2 that
could explain the above?
What needs to be done to resolve this?

Any hint would be very much appreciated.

Many thanks!

Cheers
  nils

-- 
kernel concepts GbR      Tel: +49-271-771091-12
Sieghuetter Hauptweg 48  Fax: +49-271-771091-19
D-57072 Siegen           Mob: +49-176-21024535
http://www.kernelconcepts.de

Kirkwood, kernel 3.2, vmalloc region overlap, not starting up

[Andrew Lunn]

Hi Nils

There was a thread a while back about CONFIG_ARM_PATCH_PHYS_VIRT
causing problem with some kirkwood boards. Maybe try disabling this
and set CONFIG_PHYS_OFFSET=0x0.

However, it could be a different problem...

	 Andrew

Kirkwood, kernel 3.2, vmalloc region overlap, not starting up

[Nils Faerber]

Am 07.02.2012 16:29, schrieb Andrew Lunn:
> Hi Nils
Hey Andrew!

> There was a thread a while back about CONFIG_ARM_PATCH_PHYS_VIRT
> causing problem with some kirkwood boards. Maybe try disabling this
> and set CONFIG_PHYS_OFFSET=0x0.
> 
> However, it could be a different problem...

Nope, nothing elese, it worked exactly as you described it!
(Though it took me some minutes to find the options in menuconfig).

Here it is happily up again:

root at tk71:~# uname -a
Linux tk71 3.2.0-00012-g980d683-dirty #31 Tue Feb 7 17:04:50 CET 2012
armv5tel GNU/Linux


Thanks a million!
Should we ever meet in person - I now officially owe you a beer (or
other drink at your choice :)

Apart from my joy - the basic reason for that change would be still
interesting? Does this only apply to Kirwoods or my special board?

> 	 Andrew
Thanks again!
Cheers
  nils

-- 
kernel concepts GbR      Tel: +49-271-771091-12
Sieghuetter Hauptweg 48  Fax: +49-271-771091-19
D-57072 Siegen           Mob: +49-176-21024535
http://www.kernelconcepts.de

Kirkwood, kernel 3.2, vmalloc region overlap, not starting up

[Russell King - ARM Linux]

On Tue, Feb 07, 2012 at 05:10:52PM +0100, Nils Faerber wrote:
> Apart from my joy - the basic reason for that change would be still
> interesting? Does this only apply to Kirwoods or my special board?

I don't think it's related to the uboot caching issues (but it could
be).

It needs someone to cook up a patch to find out what's going on, but
alas I don't have time at the moment, because I'm busy writing ARM
kautobuild V2 with kboottest functionality at the moment.

Kirkwood, kernel 3.2, vmalloc region overlap, not starting up

[Andrew Lunn]

> Apart from my joy - the basic reason for that change would be still
> interesting? Does this only apply to Kirwoods or my special board?

I think the conversation fizzled out without coming to a conclusion
what is wrong. Look back in the archives and see what you can find.
The problem hits kirkwood, but i don't know if it also affects other
Orion based processors. u-boot also play a role in the problem...

      Andrew

Kirkwood, kernel 3.2, vmalloc region overlap, not starting up

[Russell King - ARM Linux]

On Tue, Feb 07, 2012 at 05:18:38PM +0100, Andrew Lunn wrote:
> > Apart from my joy - the basic reason for that change would be still
> > interesting? Does this only apply to Kirwoods or my special board?
> 
> I think the conversation fizzled out without coming to a conclusion
> what is wrong. Look back in the archives and see what you can find.
> The problem hits kirkwood, but i don't know if it also affects other
> Orion based processors. u-boot also play a role in the problem...

I think what's going on (what's suggested by the uboot patch) is that
uboot leaves the L2 cache enabled, and then passes control over to
the kernel decompressor.

The kernel decompressor starts, and enables the L1 cache.  This
unexpectedly enables the L2 cache as well.  However, the decompressor
is not L2 cache aware, and only cleans the L1 cache.

The decompressor finishes, flushes the L1 cache, and disables the L1
cache (and, because of the way things work, disables the dirty L2
cache.)

We're now in the situation where the kernel text/data may or may not
be in RAM, and therefore may or may not be executable.

I don't think that's Nils problem.

But why you've both identified that turning off the V2P stuff fixes it,
I'm not sure.  It needs debugging (it needs me or someone else to cook
up a patch to get some information out of - explicitly - Nils system)
so that we can work out what's going on there.

The theory is, the V2P stuff should be able to calculate the physical
offset irrespective of the kernel placement, and so having it enabled
should be totally transparent.  Obviously, something is going wrong.
As yet we don't know what's causing Nils problem.

If no one gets there before me, then I'll post a patch for Nils to run
with to get some information from his kernel sometime in the next few
days, assuming these messages don't get too buried and I don't forget.

Kirkwood, kernel 3.2, vmalloc region overlap, not starting up

[Johannes Stezenbach]

Hi Nils,

On Tue, Feb 07, 2012 at 05:10:52PM +0100, Nils Faerber wrote:
> (Though it took me some minutes to find the options in menuconfig).

If you look closely at the top of the menuconfig screen
it says "Press ... </> for Search".

HTH
Johannes

Kirkwood, kernel 3.2, vmalloc region overlap, not starting up

[Nils Faerber]

Am 07.02.2012 17:51, schrieb Russell King - ARM Linux:
> On Tue, Feb 07, 2012 at 05:18:38PM +0100, Andrew Lunn wrote:
>>> Apart from my joy - the basic reason for that change would be still
>>> interesting? Does this only apply to Kirwoods or my special board?
>>
>> I think the conversation fizzled out without coming to a conclusion
>> what is wrong. Look back in the archives and see what you can find.
>> The problem hits kirkwood, but i don't know if it also affects other
>> Orion based processors. u-boot also play a role in the problem...
> 
> I think what's going on (what's suggested by the uboot patch) is that
> uboot leaves the L2 cache enabled, and then passes control over to
> the kernel decompressor.
> 
> The kernel decompressor starts, and enables the L1 cache.  This
> unexpectedly enables the L2 cache as well.  However, the decompressor
> is not L2 cache aware, and only cleans the L1 cache.
> 
> The decompressor finishes, flushes the L1 cache, and disables the L1
> cache (and, because of the way things work, disables the dirty L2
> cache.)
> 
> We're now in the situation where the kernel text/data may or may not
> be in RAM, and therefore may or may not be executable.
> 
> I don't think that's Nils problem.
> 
> But why you've both identified that turning off the V2P stuff fixes it,
> I'm not sure.  It needs debugging (it needs me or someone else to cook
> up a patch to get some information out of - explicitly - Nils system)
> so that we can work out what's going on there.
> 
> The theory is, the V2P stuff should be able to calculate the physical
> offset irrespective of the kernel placement, and so having it enabled
> should be totally transparent.  Obviously, something is going wrong.
> As yet we don't know what's causing Nils problem.
> 
> If no one gets there before me, then I'll post a patch for Nils to run
> with to get some information from his kernel sometime in the next few
> days, assuming these messages don't get too buried and I don't forget.

If you have anything for me to try I will gladly do it, no problem!

The same issue should also hit most of the other Kirkwood based systems
so finding a fix for my TK71 should also resolve issues on quite some
other boards.


Cheers
  nils

-- 
kernel concepts GbR      Tel: +49-271-771091-12
Sieghuetter Hauptweg 48  Fax: +49-271-771091-19
D-57072 Siegen           Mob: +49-176-21024535
http://www.kernelconcepts.de

Kirkwood, kernel 3.2, vmalloc region overlap, not starting up

[Russell King - ARM Linux]

On Tue, Feb 07, 2012 at 03:41:54PM +0100, Nils Faerber wrote:
> Hello!
> I am in the process of upgrading board support for my TK71 board from
> 2.6.38 to 3.2 and am now halted by this strange issue...
> 
> The board support for the TK71 is almost identical to the 3.2 included
> board files for e.g. the Sheevaplug or the rd88f6281 reference design.
> After applying my board patch and re-inserting my machine ID again the
> kernel freezes very early.
> 
> After enabling earlyprintk I get the following:
> 
> Uncompressing Linux... done, booting the kernel.
> 
> [    0.000000] Linux version 3.2.0-00012-g980d683-dirty (nils at moi) (gcc
> version 4.4.2 (GCC) ) #30 Tue Feb 7 15:18:57 CET 2012
> [    0.000000] CPU: Feroceon 88FR131 [56251311] revision 1 (ARMv5TE),
> cr=00053977
> [    0.000000] CPU: VIVT data cache, VIVT instruction cache
> 
> [    0.000000] Machine: TK71 Kirkwood based Q7 formfactor board
> 
> [    0.000000] bootconsole [earlycon0] enabled
> 
> [    0.000000] Ignoring RAM at 00000000-0fffffff (vmalloc region
> overlap).

I think the problem may be due to something screwing up the v:p code
fixups.  So, let's try to find out what the values the code decided on
were, and how some of the __va() macros are behaving.  Could you try
booting with this patch applied?

If this is cache related, this _may_ change the behaviour and make the
bug go away...

diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
index 10303b3..98213f0 100644
--- a/arch/arm/kernel/setup.c
+++ b/arch/arm/kernel/setup.c
@@ -916,6 +916,9 @@ void __init setup_arch(char **cmdline_p)
 {
 	struct machine_desc *mdesc;
 
+	pr_info("PHYS_OFFSET: 0x%08lx PAGE_OFFSET: 0x%08lx __pa(PAGE_OFFSET): 0x%08lx\n",
+		PHYS_OFFSET, PAGE_OFFSET, __virt_to_phys(PAGE_OFFSET));
+
 	setup_processor();
 	mdesc = setup_machine_fdt(__atags_pointer);
 	if (!mdesc)
diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
index f77f1db..143430f 100644
--- a/arch/arm/mm/mmu.c
+++ b/arch/arm/mm/mmu.c
@@ -821,7 +821,7 @@ void __init sanity_check_meminfo(void)
 	for (i = 0, j = 0; i < meminfo.nr_banks; i++) {
 		struct membank *bank = &meminfo.bank[j];
 		*bank = meminfo.bank[i];
-
+pr_info("block %u: pa0x%08lx va%p +0x%08lx\n", bank->start, __va(bank->start), bank->size);
 		if (bank->start > ULONG_MAX)
 			highmem = 1;