Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Prarit Bhargava]

Prarit Bhargava wrote:
> Émeric Maschino wrote:
> 
>> Hi,
>>
>> There's something wrong with kernel 2.6.16-1.2097_FC6. The last displayed
>> message is "Freeing unused kernel memory: 400kB freed" and then nothing.
>> kernel-2.6.16-1.2088_FC6 runs fine.
>>
> 
> I saw an error spit out while the RPM install was assembling the initrd. 
>  I'll look into it.

Looks like upstream 2.6.16-1 doesn't boot either.

Adding linux-ia64 to the list ...

P.

> 
> P.
> 

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Prarit Bhargava]

Prarit Bhargava wrote:
> Émeric Maschino wrote:
> 
>> Hi,
>>
>> There's something wrong with kernel 2.6.16-1.2097_FC6. The last displayed
>> message is "Freeing unused kernel memory: 400kB freed" and then nothing.
>> kernel-2.6.16-1.2088_FC6 runs fine.
>>
> 
> I saw an error spit out while the RPM install was assembling the initrd. 
>  I'll look into it.
> 


Looks like upstream 2.6.16-1 doesn't boot either.

Adding linux-ia64 to the list ...

P.

> P.
> 

RE: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Chen, Kenneth W]

Prarit Bhargava wrote on Wednesday, March 29, 2006 8:34 AM
> Looks like upstream 2.6.16-1 doesn't boot either.
> 
> Adding linux-ia64 to the list ...

Just tried upstream 2.6.16.1 on my Intel tiger4 machine.  It boots OK.
What kind of problem do you have?  Where does it hang?

- Ken

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Prarit Bhargava]

Chen, Kenneth W wrote:
> Prarit Bhargava wrote on Wednesday, March 29, 2006 8:34 AM
> 
>>Looks like upstream 2.6.16-1 doesn't boot either.
>>
>>Adding linux-ia64 to the list ...
> 
> 
> Just tried upstream 2.6.16.1 on my Intel tiger4 machine.  It boots OK.
> What kind of problem do you have?  Where does it hang?
> 

I'm trying to get a hold of Emeric to find out what system he's using :)

But it looks like it is hanging on Altix's at

Calling initcall 0xa0000001006ad230: bictcp_register+0x0/0x40()
TCP bic registered
Calling initcall 0xa0000001006b1160: xfrm_user_init+0x0/0xe0()
Initializing IPsec netlink socket
Calling initcall 0xa0000001006ade70: af_unix_init+0x0/0x120()
NET: Registered protocol family 1
Calling initcall 0xa0000001006adfc0: packet_init+0x0/0x100()
NET: Registered protocol family 17
Calling initcall 0xa0000001006a9670: acpi_poweroff_init+0x0/0x100()
Calling initcall 0xa0000001006af2d0: seqgen_init+0x0/0x40()
Calling initcall 0xa0000001006b18d0: early_uart_console_switch+0x0/0x120()
Calling initcall 0xa0000001006a8e10: net_random_reseed+0x0/0x100()
Freeing unused kernel memory: 400kB freed

P.

> - Ken
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Greg Edwards]

On Wed, Mar 29, 2006 at 01:55:26PM -0500, Prarit Bhargava wrote:
| But it looks like it is hanging on Altix's at
| 
| Calling initcall 0xa0000001006ad230: bictcp_register+0x0/0x40()
| TCP bic registered
| Calling initcall 0xa0000001006b1160: xfrm_user_init+0x0/0xe0()
| Initializing IPsec netlink socket
| Calling initcall 0xa0000001006ade70: af_unix_init+0x0/0x120()
| NET: Registered protocol family 1
| Calling initcall 0xa0000001006adfc0: packet_init+0x0/0x100()
| NET: Registered protocol family 17
| Calling initcall 0xa0000001006a9670: acpi_poweroff_init+0x0/0x100()
| Calling initcall 0xa0000001006af2d0: seqgen_init+0x0/0x40()
| Calling initcall 0xa0000001006b18d0: early_uart_console_switch+0x0/0x120()
| Calling initcall 0xa0000001006a8e10: net_random_reseed+0x0/0x100()
| Freeing unused kernel memory: 400kB freed

Is it hung, or did it MCA?  Check the leds (ctrl-t to L2 from console,
then 'leds').  If you repeat this procedure, you should see the LED
values changing.

Greg

RE: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Chen, Kenneth W]

Prarit Bhargava wrote on Wednesday, March 29, 2006 10:55 AM
> > Just tried upstream 2.6.16.1 on my Intel tiger4 machine.  It boots OK.
> > What kind of problem do you have?  Where does it hang?
> > 
> 
> I'm trying to get a hold of Emeric to find out what system he's using :)
> 
> But it looks like it is hanging on Altix's at
> 
> Calling initcall 0xa0000001006ad230: bictcp_register+0x0/0x40()
> TCP bic registered
> Calling initcall 0xa0000001006b1160: xfrm_user_init+0x0/0xe0()
> Initializing IPsec netlink socket
> Calling initcall 0xa0000001006ade70: af_unix_init+0x0/0x120()
> NET: Registered protocol family 1
> Calling initcall 0xa0000001006adfc0: packet_init+0x0/0x100()
> NET: Registered protocol family 17
> Calling initcall 0xa0000001006a9670: acpi_poweroff_init+0x0/0x100()
> Calling initcall 0xa0000001006af2d0: seqgen_init+0x0/0x40()
> Calling initcall 0xa0000001006b18d0: early_uart_console_switch+0x0/0x120()
> Calling initcall 0xa0000001006a8e10: net_random_reseed+0x0/0x100()
> Freeing unused kernel memory: 400kB freed

I also tried yesterday's snapshot 2.6.16-git17 on my tiger4, it doesn't
boot :-(

So far we have reports on:

(1) 2.6.16.1 doesn't boot on altix
(2) 2.6.16-git17 doesn't boot on tiger4
(3) Jes's strange .section causing some hangs.

Hmm, I don't know what's going on ....

- Ken

RE: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Luck, Tony]

> I also tried yesterday's snapshot 2.6.16-git17 on my tiger4, it doesn't
> boot :-(
>
> So far we have reports on:
>
> (1) 2.6.16.1 doesn't boot on altix
> (2) 2.6.16-git17 doesn't boot on tiger4
> (3) Jes's strange .section causing some hangs.

Tiger still boots with my release tree (just has seven patches that
haven't been sent to Linus yet, including Russ' fix for my merge
goof that created item "3" above).  I haven't pulled in from Linus
in about three days, so I'm about 458 commits behind the bleeding
edge.  So a "git bisect" between Linus head and the point where I last
merged (commit 64bc0430) might yield the problem (for Ken's tiger
boot issue).

-Tony

Here's what's in release/test that hasn't gone to Linus yet:

 arch/ia64/kernel/iosapic.c          |  279 +++++++++++++++++++++---------------
 arch/ia64/kernel/vmlinux.lds.S      |   18 +-
 arch/ia64/mm/init.c                 |    8 -
 arch/ia64/mm/tlb.c                  |   12 -
 arch/ia64/sn/kernel/sn2/sn_hwperf.c |    8 -
 5 files changed, 198 insertions(+), 127 deletions(-)

Chen, Kenneth W:
      [IA64] optimize flush_tlb_range on large numa box

Dean Roe:
      [IA64-SGI] fix for-loop in sn_hwperf_geoid_to_cnode()

hawkes@sgi.com:
      [IA64-SGI] sn_hwperf use of num_online_cpus()

Russ Anderson:
      [IA64] Move __mca_table out of the __init section

Satoru Takeuchi:
      [IA64] correct some messages and fixes some minor things
      [IA64] simplify some condition checks in iosapic_check_gsi_range

Zhang, Yanmin:
      [IA64] lazy_mmu_prot_update needs to be aware of huge pages

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Grant Grundler]

On Wed, Mar 29, 2006 at 11:55:13AM -0800, Chen, Kenneth W wrote:
> I also tried yesterday's snapshot 2.6.16-git17 on my tiger4, it doesn't
> boot :-(
> 
> So far we have reports on:
> 
> (1) 2.6.16.1 doesn't boot on altix
> (2) 2.6.16-git17 doesn't boot on tiger4
> (3) Jes's strange .section causing some hangs.

(4) James Bottomley couldn't boot his ZX2000 (HP ZX1 chipset) from git
yesterday either.
He had to update firmware to 2.31 version because of some changes
in 2.6.16 SAL usage exposed an older SAL bug. Alex Williamson had
dug up the details and I'm blanking on them right now.
But his zx2000 is still not able to boot.

Could be multiple, unrelated problems being reported now.

grant

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on

[James Bottomley]

On Wed, 2006-03-29 at 12:51 -0800, Grant Grundler wrote:
> (4) James Bottomley couldn't boot his ZX2000 (HP ZX1 chipset) from git
> yesterday either.
> He had to update firmware to 2.31 version because of some changes
> in 2.6.16 SAL usage exposed an older SAL bug. Alex Williamson had
> dug up the details and I'm blanking on them right now.
> But his zx2000 is still not able to boot.
> 
> Could be multiple, unrelated problems being reported now.

No, I can confirm that reverting

[IA64] MCA Recovery: kernel context recovery table

fixes my boot problems.

I actually got slightly further than Jes with the analysis.  The problem
occurs because something in this changeset makes code that dynamically
loads the TLS libraries segfault.  If you look, Jes, you'll find init is
continually segfaulting and respawning.

James

RE: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Luck, Tony]

> Tiger still boots with my release tree (just has seven patches that
> haven't been sent to Linus yet, including Russ' fix for my merge
> goof that created item "3" above).  I haven't pulled in from Linus
>in about three days, so I'm about 458 commits behind the bleeding
> edge.  So a "git bisect" between Linus head and the point where I last
> merged (commit 64bc0430) might yield the problem (for Ken's tiger
> boot issue).

Just started the bisection ... and the current tip of Linus' git
tree (f3cab8a0) booted just fine on tiger!  There are 85 commits
in there since git-17 was cut, but a quick glance at gitk doesn't
show any obvious fixes or reverts that might affect ia64.

-Tony

RE: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Émeric Maschino]

This is my turn to fill in the report: yesterday's Rawhide kernel
2.6.16-1.2097_FC6 (based on 2.6.16-git9, git10 and git 13) and today's Rawhide
kernel 2.6.16-1.2102_FC6 (based on 2.6.16-git14, 2.6.16-git15) don't boot on hp
workstation zx6000.

   Émeric

> So far we have reports on:
>
> (1) 2.6.16.1 doesn't boot on altix
> (2) 2.6.16-git17 doesn't boot on tiger4
> (3) Jes's strange .section causing some hangs.
>
> Hmm, I don't know what's going on ....
>
> - Ken

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Jack Steiner]

On Wed, Mar 29, 2006 at 12:58:40PM -0800, Luck, Tony wrote:
> > Tiger still boots with my release tree (just has seven patches that
> > haven't been sent to Linus yet, including Russ' fix for my merge
> > goof that created item "3" above).  I haven't pulled in from Linus
> >in about three days, so I'm about 458 commits behind the bleeding
> > edge.  So a "git bisect" between Linus head and the point where I last
> > merged (commit 64bc0430) might yield the problem (for Ken's tiger
> > boot issue).
> 
> Just started the bisection ... and the current tip of Linus' git
> tree (f3cab8a0) booted just fine on tiger!  There are 85 commits
> in there since git-17 was cut, but a quick glance at gitk doesn't
> show any obvious fixes or reverts that might affect ia64.
> 

I don't know if this helps or not.

I ran Jes's kernel on the simulator. Unfortunately, he sent me a stripped
kernel so I have no symbol table.

The kernel blows up right after printing:

	...
	Virtual mem_map starts at 0xa0007fffd5f2c000
	Built 1 zonelists
	Kernel command line: root=/dev/hda2 init=/bin/bash console=ttyS0
	PID hash table entries: 1024 (order: 10, 32768 bytes)
	Console: colour dummy device 80x25

The failure is an MCA caused by a cache hit on a memory reference to an
uncached address. The simulator detects this error & stops.

The code that took the failure was memcopy (or equiv). I recognized the
code from the prefetchs and ld/st sequence. The data appears to be
an ACPI table that is being copied into kernel memory. The current
reference is using uncached addresses but 15M instructions in the past, the
table was referenced cached.

Does this ring any bells with anyone?  This failure may 
occur only in our simulator environment, so don't spent any
time on this unless it sounds familar.

As soon as I get a symbol table, I'll know a lot more about the
failure.

---
Jack

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Jes Sorensen]

>>>>> "Tony" = Luck, Tony <tony.luck@intel.com> writes:

>> I also tried yesterday's snapshot 2.6.16-git17 on my tiger4, it
>> doesn't boot :-(
>> 
>> So far we have reports on:
>> 
>> (1) 2.6.16.1 doesn't boot on altix (2) 2.6.16-git17 doesn't boot on
>> tiger4 (3) Jes's strange .section causing some hangs.

Tony> Tiger still boots with my release tree (just has seven patches
Tony> that haven't been sent to Linus yet, including Russ' fix for my
Tony> merge goof that created item "3" above).  I haven't pulled in
Tony> from Linus in about three days, so I'm about 458 commits behind
Tony> the bleeding edge.  So a "git bisect" between Linus head and the
Tony> point where I last merged (commit 64bc0430) might yield the
Tony> problem (for Ken's tiger boot issue).

Tony,

Please note that if I compile the kernel on a box with an old
installation and old toolchain the kernel boots, same source tree
built on a system with a new installation and it doesn't boot.

Cheers,
Jes

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Jes Sorensen]

>>>>> "Jack" = Jack Steiner <steiner@sgi.com> writes:

Jack> 	...  Virtual mem_map starts at 0xa0007fffd5f2c000 Built 1
Jack> zonelists Kernel command line: root=/dev/hda2 init=/bin/bash
Jack> console=ttyS0 PID hash table entries: 1024 (order: 10, 32768
Jack> bytes) Console: colour dummy device 80x25

Jack> The failure is an MCA caused by a cache hit on a memory
Jack> reference to an uncached address. The simulator detects this
Jack> error & stops.

Hi Jack,

Sorry about the missing symbol table, was a bit late.

I am not sure it's the same thing, as my kernel gets a lot further and
normally spawns init before exploding.

Jack> The code that took the failure was memcopy (or equiv). I
Jack> recognized the code from the prefetchs and ld/st sequence. The
Jack> data appears to be an ACPI table that is being copied into
Jack> kernel memory. The current reference is using uncached addresses
Jack> but 15M instructions in the past, the table was referenced
Jack> cached.

Jack> Does this ring any bells with anyone?  This failure may occur
Jack> only in our simulator environment, so don't spent any time on
Jack> this unless it sounds familar.

Jack> As soon as I get a symbol table, I'll know a lot more about the
Jack> failure.

I put one up on attica for you ~jes/System.map-jack

It seems the kernel explodes in __copy_user in your case.

Cheers,
Jes

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Jes Sorensen]

>>>>> "Prarit" = Prarit Bhargava <prarit@redhat.com> writes:

Prarit> Looks like upstream 2.6.16-1 doesn't boot either.

Prarit> Adding linux-ia64 to the list ...

Please don't cross post to linux-ia64 when you have a fscked up list
on the CC, the fedora list is one of those hopeless subscribers only
things ;-(

Jes

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Keith Owens]

Jes Sorensen (on 30 Mar 2006 03:57:04 -0500) wrote:
>>>>>> "Tony" = Luck, Tony <tony.luck@intel.com> writes:
>
>>> I also tried yesterday's snapshot 2.6.16-git17 on my tiger4, it
>>> doesn't boot :-(
>>> 
>>> So far we have reports on:
>>> 
>>> (1) 2.6.16.1 doesn't boot on altix (2) 2.6.16-git17 doesn't boot on
>>> tiger4 (3) Jes's strange .section causing some hangs.
>
>Tony> Tiger still boots with my release tree (just has seven patches
>Tony> that haven't been sent to Linus yet, including Russ' fix for my
>Tony> merge goof that created item "3" above).  I haven't pulled in
>Tony> from Linus in about three days, so I'm about 458 commits behind
>Tony> the bleeding edge.  So a "git bisect" between Linus head and the
>Tony> point where I last merged (commit 64bc0430) might yield the
>Tony> problem (for Ken's tiger boot issue).
>
>Tony,
>
>Please note that if I compile the kernel on a box with an old
>installation and old toolchain the kernel boots, same source tree
>built on a system with a new installation and it doesn't boot.

This sounds like a random data overwrite which depends on the exact
kernel layout.  Somebody else in SGI (Prarit ?) said that they compiled
your failing tree on the same platform and their kernel worked.  Since
the build userid and the pathname to the kernel source tree are
embedded in the binary, changing either of those will disturb the
kernel layout, possibly making the problem move.

Jes, try renaming your source tree to something with a different
pathname length and rebuild the kernel.  Does the problem change or go
away?

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Jes Sorensen]

>>>>> "Keith" = Keith Owens <kaos@sgi.com> writes:

Keith> Jes Sorensen (on 30 Mar 2006 03:57:04 -0500) wrote:
>> Please note that if I compile the kernel on a box with an old
>> installation and old toolchain the kernel boots, same source tree
>> built on a system with a new installation and it doesn't boot.

Keith> This sounds like a random data overwrite which depends on the
Keith> exact kernel layout.  Somebody else in SGI (Prarit ?) said that
Keith> they compiled your failing tree on the same platform and their
Keith> kernel worked.  Since the build userid and the pathname to the
Keith> kernel source tree are embedded in the binary, changing either
Keith> of those will disturb the kernel layout, possibly making the
Keith> problem move.

Keith> Jes, try renaming your source tree to something with a
Keith> different pathname length and rebuild the kernel.  Does the
Keith> problem change or go away?

Moved it to /tmp/a and built there - no go.

Cheers,
Jes

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Jes Sorensen]

>>>>> "James" = James Bottomley <James.Bottomley@SteelEye.com> writes:

James> No, I can confirm that reverting

James> [IA64] MCA Recovery: kernel context recovery table

James> fixes my boot problems.

James> I actually got slightly further than Jes with the analysis.
James> The problem occurs because something in this changeset makes
James> code that dynamically loads the TLS libraries segfault.  If you
James> look, Jes, you'll find init is continually segfaulting and
James> respawning.

Yay!

I am not the only one seeing this! I am not insane!

I bet that if you revert the patch above but then add these two lines
to arch/ia64/kernel/gate.S you'll see the same failure:

	.section "__mca_table", "a"
	.previous

Tracked it down further. If I take the gate.o file from a  working
kernel and copy it into the broken tree and run make again, the
resulting kernel boots.

I compared the code between a working and a broken gate.o with objdump
and it's identical. However the good file is 3552 bytes whereas the
broken one is 3648.

objdump -h reveals one extra empty section which seems to be what
makes all the difference. The below output is from a broken gate.o,
the only difference I can find comparing it to the good one is that
idx 4 is missing in the good one and the following ones are
renamed. Do we have some strange restrictions in the linker on the
number of sections it can handle?

Cheers,
Jes


gate.o:     file format elf64-ia64-little

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         000003b0  0000000000000000  0000000000000000  00000040  2**5
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .data         00000000  0000000000000000  0000000000000000  000003f0  2**0
                  CONTENTS, ALLOC, LOAD, DATA
  2 .bss          00000000  0000000000000000  0000000000000000  000003f0  2**0
                  ALLOC
  3 __ex_table    00000000  0000000000000000  0000000000000000  000003f0  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 __mca_table   00000000  0000000000000000  0000000000000000  000003f0  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .data.patch.vtop 00000000  0000000000000000  0000000000000000  000003f0  2**0
                  CONTENTS, ALLOC, LOAD, DATA
  6 .data.patch.mckinley_e9 00000004  0000000000000000  0000000000000000  000003f0  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, DATA
  7 .data.patch.fsyscall_table 00000004  0000000000000000  0000000000000000  000003f4  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, DATA
  8 .data.patch.brl_fsys_bubble_down 00000004  0000000000000000  0000000000000000  000003f8  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, DATA
  9 .IA_64.unwind_info 000000b8  0000000000000000  0000000000000000  00000400  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 10 .IA_64.unwind 00000048  0000000000000000  0000000000000000  000004b8  2**3
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Jes Sorensen]

>>>>> "Jes" = Jes Sorensen <jes@sgi.com> writes:

Jes> Tracked it down further. If I take the gate.o file from a working
Jes> kernel and copy it into the broken tree and run make again, the
Jes> resulting kernel boots.

Jes> I compared the code between a working and a broken gate.o with
Jes> objdump and it's identical. However the good file is 3552 bytes
Jes> whereas the broken one is 3648.

Looks like something goes wrong with some linking references due to
this. I ran objdump on the two kernels I get when replacing gate.o and
the diff is as attached below. Not sure if this is an ia64 specific
bug or we're just 'lucky'.

Cheers,
Jes

--- ../good/vmlinux.s	2006-03-30 15:13:11.000000000 +0200
+++ ../broken/vmlinux.s	2006-03-30 15:11:57.000000000 +0200
@@ -4981,8 +4981,8 @@
 a00000010000bd96:	30 01 44 44 08 40 	            mov.m r19=ar.bsp
 a00000010000bd9c:	01 00 40 00       	            mov f10ð
 a00000010000bda0:	05 00 00 00 01 00 	[MLX]       nop.m 0x0
-a00000010000bda6:	00 00 00 00 20 c0 	            movl r14=0xa000000000010640;;
-a00000010000bdac:	01 24 30 68 
+a00000010000bda6:	00 00 00 00 00 c0 	            movl r14=0x10080;;
+a00000010000bdac:	01 20 04 60 
 a00000010000bdb0:	08 00 00 32 2a 04 	[MMI]       mov.m ar25=r0
 a00000010000bdb6:	00 00 80 54 08 e0 	            mov.m ar.ccv=r0
 a00000010000bdbc:	e0 08 00 07       	            mov b7=r14
@@ -8192,8 +8192,8 @@
 a00000010000fe06:	00 01 00 00 20 c0 	            movl r14=0xa00000010000bc60
 a00000010000fe0c:	01 06 e0 6d 
 a00000010000fe10:	05 00 00 00 01 00 	[MLX]       nop.m 0x0
-a00000010000fe16:	00 00 00 00 20 80 	            movl r28=0xa000000000010620;;
-a00000010000fe1c:	03 22 30 68 
+a00000010000fe16:	00 00 00 00 00 80 	            movl r28=0x10060;;
+a00000010000fe1c:	03 26 00 60 
 a00000010000fe20:	01 10 00 20 00 21 	[MII]       mov r2=r16
 a00000010000fe26:	00 21 42 0c 42 e0 	            adds r16ƒ6,r16
 a00000010000fe2c:	03 00 cc 00       	            mov r31=pr;;
@@ -1563052,7 +1563052,7 @@
 a00000010091856c:	03 73 ff 00       	            data8 0x1fee607fe
 a000000100918570:	73 ec 02 f0 05 44 	[MBB] (p35) mov pmd[r120]=r0
 a000000100918576:	ee 02 f0 06 46 e8 	            data8 0x1181bc00bb9
-a00000010091857c:	02 f0 03 4c       	            br.ctop.sptk.few.clr a0000000ff957570 <__kernel_sigtramp+0xff946e50>;;
+a00000010091857c:	02 f0 03 4c       	            br.ctop.sptk.few.clr a0000000ff957570 <__end_gate_brl_fsys_bubble_down_patchlist+0xff957064>;;
 a000000100918580:	e4 02 f0 01 36 e6 	[MLX] (p23) mov r0=-2090116
 a000000100918586:	02 f0 02 38 e1 02 	            data8 0x1be0edc605
 a00000010091858c:	e3 76 f0 0d 
@@ -1563061,7 +1563061,7 @@
 a00000010091859c:	03 73 ff 00       	            data8 0x1fee607fe
 a0000001009185a0:	73 ec 00 f0 05 44 	[MBB] (p35) mov pmd[r120]=r0
 a0000001009185a6:	ee 00 f0 06 46 e8 	            data8 0x1181bc003b9
-a0000001009185ac:	00 f0 03 4c       	            br.ctop.sptk.few.clr a0000000ff9575a0 <__kernel_sigtramp+0xff946e80>;;
+a0000001009185ac:	00 f0 03 4c       	            br.ctop.sptk.few.clr a0000000ff9575a0 <__end_gate_brl_fsys_bubble_down_patchlist+0xff957094>;;
 a0000001009185b0:	e4 00 f0 01 36 e6 	[MLX] (p07) mov r0=-2090116
 a0000001009185b6:	00 f0 02 38 e1 00 	            data8 0x1cdf2edc601
 a0000001009185bc:	e3 76 f9 e6 

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Jack Steiner]

> I don't know if this helps or not.
> 
> I ran Jes's kernel on the simulator. Unfortunately, he sent me a stripped
> kernel so I have no symbol table.
> 
> The kernel blows up right after printing:
> 
> 	...
> 	Virtual mem_map starts at 0xa0007fffd5f2c000
> 	Built 1 zonelists
> 	Kernel command line: root=/dev/hda2 init=/bin/bash console=ttyS0
> 	PID hash table entries: 1024 (order: 10, 32768 bytes)
> 	Console: colour dummy device 80x25
> 
> The failure is an MCA caused by a cache hit on a memory reference to an
> uncached address. The simulator detects this error & stops.
> 
> The code that took the failure was memcopy (or equiv). I recognized the
> code from the prefetchs and ld/st sequence. The data appears to be
> an ACPI table that is being copied into kernel memory. The current
> reference is using uncached addresses but 15M instructions in the past, the
> table was referenced cached.
 
I found the problem that is causing the cache/uncached collision. This is
most likely NOT what is causing the problem you are currently chasing, but
it is a problem never-the-less.

(Background: I noticed that recent IA64 kernels are making references to the
same address using both cached & uncached addresses. The kernel fails to
boot on the simulator because it detects a cache-hit on an uncached
reference. Mixing cached & uncached references is not supported).


acpi_os_map_memory() has changed the order of checking the EFI attributes on
pages being mapped. In 2.6.15, if the EFI map indicates that an address
supports both cached & uncached references, acpi_os_map_memory() would map
it as cached.

The most recent kernel has changed this behavior so that the address will be
referenced as uncached.


OLD:
        acpi_os_map_memory
                if (EFI_MEMORY_WB & efi_mem_attributes(phys)) {
                        *virt = (void __iomem *)phys_to_virt(phys);
                } else {
                        *virt = ioremap(phys, size);
                }

NEW
        acpi_os_map_memory() unconditionally calls ioremap() to do the
        mapping

        ioremap
                if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
                         return __ioremap(offset, size);

                if (efi_mem_attribute_range(offset, size, EFI_MEMORY_WB))
                         return phys_to_virt(offset);


Earlier in the boot, the ACPI tables are unconditionally references as cached:

        char *__acpi_map_table(unsigned long phys_addr, unsigned long size)
        {
                return __va(phys_addr);
        }



I suspect there was a reason for this change. I'll add Bjorn to the cc
list. 

Is this problem unique to SN systems. The BIOS reports that most
memory ranges support both CACHED & UNCACHED references. I _think_
this is correct.

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Prarit Bhargava]

Hello all,

> ---
>  arch/ia64/kernel/gate.lds.S |    1 +
>  include/asm-ia64/asmmacro.h |    4 ++++
>  2 files changed, 5 insertions(+)
> 

I applied this patch and compiled using gcc-4.1.0-3.

I now see hundreds of warnings like:

WARNING: drivers/atm/he.o - Section mismatch: reference to 
.init.text:he_start from .text after 'he_init_one' (at offset 0x81a2)
WARNING: drivers/block/cpqarray.o - Section mismatch: reference to 
.init.text:cpqarray_init_one from .data.rel.local between 
'cpqarray_pci_driver' (at offset 0x20) and 'smart1_access'

Just an FYI,

P.

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[David Mosberger-Tang]

On 3/30/06, Jack Steiner <steiner@sgi.com> wrote:

> Is this problem unique to SN systems

No, the same will happen on all other systems (that I know of).

> The BIOS reports that most
> memory ranges support both CACHED & UNCACHED references. I _think_
> this is correct.

That's correct.  The map shows the ways the page *can* be mapped, not
the way it *should* be mapped.

It's strange that ACPI would prefer to use WC when WB mapping is
possible.  They definitely need to pick one way and stick with it
though.  As you say, mapping the same page with different cacheability
is a no-no (and at least in theory, it should cause an MCA even on
real hardware).

  --david

--
Mosberger Consulting LLC, http://www.mosberger-consulting.com/

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Jack Steiner]

On Thu, Mar 30, 2006 at 08:43:43AM -0700, David Mosberger-Tang wrote:
> On 3/30/06, Jack Steiner <steiner@sgi.com> wrote:
> 
> > Is this problem unique to SN systems
> 
> No, the same will happen on all other systems (that I know of).
> 
> > The BIOS reports that most
> > memory ranges support both CACHED & UNCACHED references. I _think_
> > this is correct.
> 
> That's correct.  The map shows the ways the page *can* be mapped, not
> the way it *should* be mapped.
> 
> It's strange that ACPI would prefer to use WC when WB mapping is
> possible.  They definitely need to pick one way and stick with it
> though.  As you say, mapping the same page with different cacheability
> is a no-no (and at least in theory, it should cause an MCA even on
> real hardware).

It does, at least on our chipset. If the chipset detects a simultaneous
UC & C reference, it generates a BUS error. It is surprising how
quickly this MCA occurs when we break the rules.



> 
>   --david
> 
> --
> Mosberger Consulting LLC, http://www.mosberger-consulting.com/

-- 
Jack

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[David Mosberger-Tang]

On 3/30/06, Jack Steiner <steiner@sgi.com> wrote:
> On Thu, Mar 30, 2006 at 08:43:43AM -0700, David Mosberger-Tang wrote:
> > (and at least in theory, it should cause an MCA even on
> > real hardware).
>
> It does, at least on our chipset. If the chipset detects a simultaneous
> UC & C reference, it generates a BUS error. It is surprising how
> quickly this MCA occurs when we break the rules.

That's good.  While MCAs are a pain, it's a lot better than having to
track down data corruption due to (in-)coherency.

 --david
--
Mosberger Consulting LLC, http://www.mosberger-consulting.com/

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[H. J. Lu]

On Thu, Mar 30, 2006 at 10:42:00AM -0500, Prarit Bhargava wrote:
> Hello all,
> 
> >---
> > arch/ia64/kernel/gate.lds.S |    1 +
> > include/asm-ia64/asmmacro.h |    4 ++++
> > 2 files changed, 5 insertions(+)
> >
> 
> I applied this patch and compiled using gcc-4.1.0-3.
> 
> I now see hundreds of warnings like:
> 
> WARNING: drivers/atm/he.o - Section mismatch: reference to 
> .init.text:he_start from .text after 'he_init_one' (at offset 0x81a2)
> WARNING: drivers/block/cpqarray.o - Section mismatch: reference to 
> .init.text:cpqarray_init_one from .data.rel.local between 
> 'cpqarray_pci_driver' (at offset 0x20) and 'smart1_access'
> 

Where did they come from?


H.J.

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Prarit Bhargava]

> 
> Where did they come from?
> 

During the 'make compressed' phase ...

P.

> 
> H.J.
> 

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[H. J. Lu]

On Thu, Mar 30, 2006 at 11:38:32AM -0500, Prarit Bhargava wrote:
> 
> >
> >Where did they come from?
> >
> 
> During the 'make compressed' phase ...
> 

Which program generated them?


H.J.

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Prarit Bhargava]

> 
> Which program generated them?
> 

[root@altix2 ~]# gcc --version
gcc (GCC) 4.1.0 20060304 (Red Hat 4.1.0-3)

P.

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Bjorn Helgaas]

On Thursday 30 March 2006 08:16, Jack Steiner wrote:
> acpi_os_map_memory() has changed the order of checking the EFI attributes on
> pages being mapped. In 2.6.15, if the EFI map indicates that an address
> supports both cached & uncached references, acpi_os_map_memory() would map
> it as cached.

> OLD:
>         acpi_os_map_memory
>                 if (EFI_MEMORY_WB & efi_mem_attributes(phys)) {
>                         *virt = (void __iomem *)phys_to_virt(phys);
>                 } else {
>                         *virt = ioremap(phys, size);
>                 }
> 
> NEW
>         acpi_os_map_memory() unconditionally calls ioremap() to do the
>         mapping
> 
>         ioremap
>                 if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
>                          return __ioremap(offset, size);
> 
>                 if (efi_mem_attribute_range(offset, size, EFI_MEMORY_WB))
>                          return phys_to_virt(offset);

I debated whether to check for UC or WB first, and I think I got it
wrong.  My reasoning for UC first was that it was a smaller change
for the typical case CSR mapping case, because ioremap() used to
*always* return UC mappings.

But efi_memmap_init() collects full granules of WB memory, without
regard for whether they also support UC.  So if ioremap() needs to
work for main memory (which is slightly strange in itself, but seems
to be the precedent), it needs to prefer WB mappings when possible.

> Earlier in the boot, the ACPI tables are unconditionally references as cached:
> 
>         char *__acpi_map_table(unsigned long phys_addr, unsigned long size)
>         {
>                 return __va(phys_addr);
>         }

This seems suspicious to me.  But we look at some of these ACPI tables
prior to calling efi_memmap_init(), so it might not be safe to use
ioremap() yet.  It's conceivable that a platform could place ACPI
tables in UC-only ROM, and this would blow up.  But I guess we
haven't tripped over that yet.

Does the patch below solve your immediate problem?


[IA64] ioremap() should prefer WB over UC

efi_memmap_init() collects full granules of WB memory, without
regard for whether they also support UC.  So in order for ioremap()
to work for main memory, it must prefer WB mappings when possible.

Signed-off-by: Bjorn Helgaas <bjorn.helgaas@hp.com>

Index: work-mm5/arch/ia64/mm/ioremap.c
=================================--- work-mm5.orig/arch/ia64/mm/ioremap.c	2006-03-23 10:22:40.000000000 -0700
+++ work-mm5/arch/ia64/mm/ioremap.c	2006-03-30 09:36:57.000000000 -0700
@@ -21,12 +21,12 @@
 void __iomem *
 ioremap (unsigned long offset, unsigned long size)
 {
-	if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
-		return __ioremap(offset, size);
-
 	if (efi_mem_attribute_range(offset, size, EFI_MEMORY_WB))
 		return phys_to_virt(offset);
 
+	if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
+		return __ioremap(offset, size);
+
 	/*
 	 * Someday this should check ACPI resources so we
 	 * can do the right thing for hot-plugged regions.

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Jack Steiner]

On Thu, Mar 30, 2006 at 09:53:39AM -0700, Bjorn Helgaas wrote:
> On Thursday 30 March 2006 08:16, Jack Steiner wrote:
> > acpi_os_map_memory() has changed the order of checking the EFI attributes on
> > pages being mapped. In 2.6.15, if the EFI map indicates that an address
> > supports both cached & uncached references, acpi_os_map_memory() would map
> > it as cached.
> 
> > OLD:
> >         acpi_os_map_memory
> >                 if (EFI_MEMORY_WB & efi_mem_attributes(phys)) {
> >                         *virt = (void __iomem *)phys_to_virt(phys);
> >                 } else {
> >                         *virt = ioremap(phys, size);
> >                 }
> > 
> > NEW
> >         acpi_os_map_memory() unconditionally calls ioremap() to do the
> >         mapping
> > 
> >         ioremap
> >                 if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
> >                          return __ioremap(offset, size);
> > 
> >                 if (efi_mem_attribute_range(offset, size, EFI_MEMORY_WB))
> >                          return phys_to_virt(offset);
> 
> I debated whether to check for UC or WB first, and I think I got it
> wrong.  My reasoning for UC first was that it was a smaller change
> for the typical case CSR mapping case, because ioremap() used to
> *always* return UC mappings.
> 
> But efi_memmap_init() collects full granules of WB memory, without
> regard for whether they also support UC.  So if ioremap() needs to
> work for main memory (which is slightly strange in itself, but seems
> to be the precedent), it needs to prefer WB mappings when possible.
> 
> > Earlier in the boot, the ACPI tables are unconditionally references as cached:
> > 
> >         char *__acpi_map_table(unsigned long phys_addr, unsigned long size)
> >         {
> >                 return __va(phys_addr);
> >         }
> 
> This seems suspicious to me.  But we look at some of these ACPI tables
> prior to calling efi_memmap_init(), so it might not be safe to use
> ioremap() yet.  It's conceivable that a platform could place ACPI
> tables in UC-only ROM, and this would blow up.  But I guess we
> haven't tripped over that yet.
> 
> Does the patch below solve your immediate problem?

Looks good. 


> 
> 
> [IA64] ioremap() should prefer WB over UC
> 
> efi_memmap_init() collects full granules of WB memory, without
> regard for whether they also support UC.  So in order for ioremap()
> to work for main memory, it must prefer WB mappings when possible.
> 
> Signed-off-by: Bjorn Helgaas <bjorn.helgaas@hp.com>
> 
> Index: work-mm5/arch/ia64/mm/ioremap.c
> =================================> --- work-mm5.orig/arch/ia64/mm/ioremap.c	2006-03-23 10:22:40.000000000 -0700
> +++ work-mm5/arch/ia64/mm/ioremap.c	2006-03-30 09:36:57.000000000 -0700
> @@ -21,12 +21,12 @@
>  void __iomem *
>  ioremap (unsigned long offset, unsigned long size)
>  {
> -	if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
> -		return __ioremap(offset, size);
> -
>  	if (efi_mem_attribute_range(offset, size, EFI_MEMORY_WB))
>  		return phys_to_virt(offset);
>  
> +	if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
> +		return __ioremap(offset, size);
> +
>  	/*
>  	 * Someday this should check ACPI resources so we
>  	 * can do the right thing for hot-plugged regions.

-- 
Thanks

Jack Steiner (steiner@sgi.com)          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.

RE: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Luck, Tony]

> I applied this patch and compiled using gcc-4.1.0-3.
> 
> I now see hundreds of warnings like:
> 
> WARNING: drivers/atm/he.o - Section mismatch: reference to 
> .init.text:he_start from .text after 'he_init_one' (at offset 0x81a2)
> WARNING: drivers/block/cpqarray.o - Section mismatch: reference to 
> .init.text:cpqarray_init_one from .data.rel.local between 
> 'cpqarray_pci_driver' (at offset 0x20) and 'smart1_access'

Just tried with vanilla gcc 4.1.0, and I don't see these
warnings.

BTW, these messages presumably come from scripts/mod/modpost.c

-Tony

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Prarit Bhargava]

Prarit Bhargava wrote:
>>
>> Which program generated them?
>>
> 
> [root@altix2 ~]# gcc --version
> gcc (GCC) 4.1.0 20060304 (Red Hat 4.1.0-3)
> 

Humph.  I can't get this to happen again :/.  Oh well ... if anyone else 
sees this issue please let me know.

P.

> P.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [Fedora-ia64-list] kernel 2.6.16-1.2097_FC6 unbootable on Itanium

[Grant Grundler]

On Thu, Mar 30, 2006 at 09:16:11AM -0600, Jack Steiner wrote:
...
> The most recent kernel has changed this behavior so that the address will be
> referenced as uncached.
...
> NEW
>         acpi_os_map_memory() unconditionally calls ioremap() to do the
>         mapping
> 
>         ioremap
>                 if (efi_mem_attribute_range(offset, size, EFI_MEMORY_UC))
>                          return __ioremap(offset, size);

Could this be related to the issue of ioremap() vs ioremap_nocache()
that Mathew Wilcox proposed a patch for?

	http://lkml.org/lkml/2006/3/30/251

grant