INIT dumps broken?

INIT dumps broken?

[David Mosberger]

I noticed that with kernel v2.6.9-rc1 INIT dumps appear not to work
anymore.  The machine just hangs, without ever (visibly) control
reaching the OS INIT handler.  v2.6.7-rc3 was OK.  Anybody know what
broke?

	--david

RE: INIT dumps broken?

[Luck, Tony]

>I noticed that with kernel v2.6.9-rc1 INIT dumps appear not to work
>anymore.  The machine just hangs, without ever (visibly) control
>reaching the OS INIT handler.  v2.6.7-rc3 was OK.  Anybody know what
>broke?

Nothing immediately obvious.  mca_asm.S hasn't been touched in that
interval.  mca.c has had a few changes for cpe polling, and some
cleanup in the split of things between mca_init() and mca_late_init(),
but the INIT handlers are registered in mca_init() before any code
changes to that routine.

-Tony

Re: INIT dumps broken?

[Keith Owens]

On Tue, 14 Sep 2004 07:50:55 -0700, 
David Mosberger <davidm@napali.hpl.hp.com> wrote:
>I noticed that with kernel v2.6.9-rc1 INIT dumps appear not to work
>anymore.  The machine just hangs, without ever (visibly) control
>reaching the OS INIT handler.  v2.6.7-rc3 was OK.  Anybody know what
>broke?

cpu 0 is using init_task's original address.

2.6.7-rc3, kdb ps A

Task Addr               Pid   Parent [*] cpu State Thread             Command
0xe0000430148dc000        0        0  1    0   I  0xe0000430148dc4b0 *swapper
0xe00004b43be70000        0        1  1    1   I  0xe00004b43be704b0  swapper
0xe00004b43bd68000        0        1  1    2   I  0xe00004b43bd684b0  swapper
0xe0000430067b0000        0        1  1    3   I  0xe0000430067b04b0  swapper

2.6.8.1, kdb ps A

Task Addr               Pid   Parent [*] cpu State Thread             Command
0xa0000001008e8000        0        0  1    0   I  0xa0000001008e84d0 *swapper   <=0xe00004b43bd38000        0        1  1    1   I  0xe00004b43bd384d0  swapper
0xe00004b43bd00000        0        1  1    2   I  0xe00004b43bd004d0  swapper
0xe00004b0037e0000        0        1  1    3   I  0xe00004b0037e04d0  swapper

The use of 0xa.. for cpu 0 is causing an MCA when cpu 0 receives INIT.

Re: INIT dumps broken?

[David Mosberger]

>>>>> On Wed, 15 Sep 2004 17:16:47 +1000, Keith Owens <kaos@sgi.com> said:

  Keith> The use of 0xa.. for cpu 0 is causing an MCA when cpu 0
  Keith> receives INIT.

The MINSTATE_PHYS-version of MINSTATE_GET_CURRENT() looks broken to
me.  Perhaps changing "dep reg=0,reg,61,3" to a "tpa" is all that's
needed there.  Can someone check this out?  I probably won't get to it
today and will be on vacation starting tomorrow.

	--david

Re: INIT dumps broken?

[Keith Owens]

On Wed, 15 Sep 2004 00:56:15 -0700, 
David Mosberger <davidm@napali.hpl.hp.com> wrote:
>>>>>> On Wed, 15 Sep 2004 17:16:47 +1000, Keith Owens <kaos@sgi.com> said:
>
>  Keith> The use of 0xa.. for cpu 0 is causing an MCA when cpu 0
>  Keith> receives INIT.
>
>The MINSTATE_PHYS-version of MINSTATE_GET_CURRENT() looks broken to
>me.  Perhaps changing "dep reg=0,reg,61,3" to a "tpa" is all that's
>needed there.  Can someone check this out?  I probably won't get to it
>today and will be on vacation starting tomorrow.

Agreed that MINSTATE_PHYS is broken.  Not sure that tpa is reliable
during INIT/MCA, especially as one of the possible causes of an MCA is
an invalid tlb.

Can anybody describe why cpu 0 changed from 0xe... to 0xa... for the
idle task between 2.6.7-rc3 and 2.6.8?

RE: INIT dumps broken?

[Luck, Tony]

>Can anybody describe why cpu 0 changed from 0xe... to 0xa... for the
>idle task between 2.6.7-rc3 and 2.6.8?

Because the initialized lists in the init_task structure were all
messed up.  When we moved the kernel to region 5 we originally tried
to keep the task for proc 0 in region 7 ... so head.S converted from
the region 5 address that &init_task now gave us to a region 7 1:1
mapped address.  But the initialized list_head structures inside the
init_task used the region 5 addresses ... and this upset the other
list macros (e.g. the actually empty list didn't appear empty anymore
because it apparently didn't point to itself because of the mix of
region 5 and region 7 addresses.  There were a few patch attempts to
try any rescue the "address via region 7" solution, but the kernel is
now littered with "&init_task" references in generic code ... so we
changed head.S to drop the region5->region7 conversion.

I'll take a look at fixing the INIT code today.

-Tony

Re: INIT dumps broken?

[Kazuto Miyoshi]

Hello, 

> Can anybody describe why cpu 0 changed from 0xe... to 0xa... for the
> idle task between 2.6.7-rc3 and 2.6.8?

init_task is initialized by INIT_TASK macro and it fills the init_task
with 0xa.. value, which is computed at compilation time. 
Thus, we observed 0xa.. link lists from 0xe.. mapping and it caused
panic with components which refer swapper.

Actually, there was different approach which is to initialize init_task
again with 0xe.. value after boot, but it requries future maintainance
of the re-initialization with the changes of task_struct.
That is the reason we proposed the patch to maintain coherent view of
task_struct.

Currently we do not have consistent way to solve the problem,
but we will try to think about fixing the problem.

BTW, I tried INIT on my box but it seemed to produce correct task
dump to the console. I will look at it more closely...

Best Regards,
--
Kazuto Miyoshi, NEC
miyoshi@hpc.bs1.fc.nec.co.jp

RE: INIT dumps broken?

[Luck, Tony]

>BTW, I tried INIT on my box but it seemed to produce correct task
>dump to the console. I will look at it more closely...

The problem depends on which task is running on the monarch at
the time of the INIT.  Expanding the SAVE_MIN_WITH_COVER macro
in ia64_monarch_init_handler, one of the first things done is
to get "current" ... since MINSTATE_PHYS is defined, we use the
version that reads:
   mov reg=IA64_KR(Current);;dep reg=0,reg,61,3

So if our task is anything other than the initial "init_task",
we'll take a nice region 7 address from ar.k6, and convert to
physical by zapping bits [63:61].  But we happen to be running
the init_task, then ar.k6 contains a region 5 address, and we
create a garbage address by clearing the high bits, which can
lead to an MCA (if the address points somewhere bad) or to
stomping on some random place (if the address happens to be a
legal physical address).

Perhaps MIN_STATE_GET_CURRENT() needs to be smarter and handle
the region 5 case?  Or perhaps Russ will factor this into a newer
version of his percpu save area patch????

-Tony

Re: INIT dumps broken?

[Russ Anderson]

Tony Luck wrote:
>
> The problem depends on which task is running on the monarch at
> the time of the INIT.  Expanding the SAVE_MIN_WITH_COVER macro
> in ia64_monarch_init_handler, one of the first things done is
> to get "current" ... since MINSTATE_PHYS is defined, we use the
> version that reads:
>    mov reg=IA64_KR(Current);;dep reg=0,reg,61,3
>
> So if our task is anything other than the initial "init_task",
> we'll take a nice region 7 address from ar.k6, and convert to
> physical by zapping bits [63:61].  But we happen to be running
> the init_task, then ar.k6 contains a region 5 address, and we
> create a garbage address by clearing the high bits, which can
> lead to an MCA (if the address points somewhere bad) or to
> stomping on some random place (if the address happens to be a
> legal physical address).
>
> Perhaps MIN_STATE_GET_CURRENT() needs to be smarter and handle
> the region 5 case?  Or perhaps Russ will factor this into a newer
> version of his percpu save area patch????

Correctly understanding the proper way to translate addresses for
the various regions would be the first step.

The top 3 bits of the virtual address is the virtual region number.  

If it is a region 7 (0xe) address, then masking off the top 3 bits 
yields the correct physical address.

If it is a region 5 (0xa) address, then a "tpa" instruction (using 
the TLB) should yield the correct physical address (assuming there is
a valid TLB entry for that virtual address).

Would that be the correct way to translate the addresses?

Are there exceptions, such as region 7 addresses where masking off 
the top 3 bits would not yield the correct physical address?

How should region 5 addresses without a valid TLB mapping be handled?

Are there any other regions that need handling, and if so, how?

Thanks,
-- 
Russ Anderson, OS RAS/Partitioning Project Lead  
SGI - Silicon Graphics Inc          rja@sgi.com

RE: INIT dumps broken?

[Luck, Tony]

>Correctly understanding the proper way to translate addresses for
>the various regions would be the first step.
>
>The top 3 bits of the virtual address is the virtual region number.  
>
>If it is a region 7 (0xe) address, then masking off the top 3 bits 
>yields the correct physical address.
>
>If it is a region 5 (0xa) address, then a "tpa" instruction (using 
>the TLB) should yield the correct physical address (assuming there is
>a valid TLB entry for that virtual address).
>
>Would that be the correct way to translate the addresses?
>
>Are there exceptions, such as region 7 addresses where masking off 
>the top 3 bits would not yield the correct physical address?
>
>How should region 5 addresses without a valid TLB mapping be handled?
>
>Are there any other regions that need handling, and if so, how?


For this case we can use the "tpa" (and I'm just about to check-in
a change to do that).  The TLB is functional for an INIT signal, we
don't use this call in the MCA path when the TLB might be broken.

Here's a breakdown of kernel virtual address space (in 2.6 kernel,
there are a few differences in 2.4):

Region 7:  Almost all of this is mapped 1:1 with physical memory
using the mapping "virtual = physical + PAGE_OFFSET", so you can
almost always turn a region 7 address into a physical address by
just subtracting PAGE_OFFSET ... or more simply by clearing bits
{63:61} (since PAGE_OFFSET is 0xE000000000000000).  The exception
in region 7 is the top 64Kbytes (0xffffffffffff0000-0xffffffffffffffff)
which is the percpu area hard-wired by DTR[1] in the TLB to a
different physical address on each cpu.

Region 6: All mapped 1:1 with uncacheable physical memory using
the mapping "virtual = physical + 0xC000000000000000"

Region 5:  Lots of different stuff in here.  Going from low to
high:
0xA000000000000000 : unmapped (redzone above user space) ... or I
thought it was ... I see some symbols in here, hmmm?
0xA000000000010000 : gate page
0xA000000100000000 : base of kernel. mapped by ITR[0] & DTR[0]
0xA000000200000000 : VMALLOC_START. Virtual playground for vmalloc
                     mapped in PAGE_SIZE discreet pieces.
0xA0007xxxxxxxxxxx : If you have CONFIG_VIRT_MEM_MAP on, and if the
                     kernel is using it, then the top of the vmalloc
                     area is used for page structs (mem_map/vmem_map).
0xA000800000000000 : end of vmalloc area (when PAGE_SIZE\x16k)
0xBFFCxxxxxxxxxxxx : VHPT for ptes mapped in region 5


Whether you can use "tpa" on an address depends on a few factors. If
psr.dt is 1, then most cases will work (possibly involving a TLB fault
to get the answer).  When psr.dt is 0 (e.g. in our INIT handler), then
we check the DTLB ... if that fails then there will be a AltDTLB fault
or a DataNestedTLB fault (depending on whether psr.ic is set).  In
general in these cases we only want to use "tpa" for addresses that
are locked in the TLB by one of the DTR[] registers.

-Tony

Re: INIT dumps broken?

[Christian Hildner]

Russ Anderson schrieb:

>Correctly understanding the proper way to translate addresses for
>the various regions would be the first step.
>
>The top 3 bits of the virtual address is the virtual region number.  
>
>If it is a region 7 (0xe) address, then masking off the top 3 bits 
>yields the correct physical address.
>
>If it is a region 5 (0xa) address, then a "tpa" instruction (using 
>the TLB) should yield the correct physical address (assuming there is
>a valid TLB entry for that virtual address).
>
>Would that be the correct way to translate the addresses?
>
If you can rely on a working ivt and uncorrupted page tables, yes.

>Are there exceptions, such as region 7 addresses where masking off 
>the top 3 bits would not yield the correct physical address?
>
No. Region 7 is per definition one to one mapped. Because of that the 
kernel went from region 7 (2.4.x) to region 5 (2.6.x) to allow having a 
relocatable kernel at a fixed virtual address.

>How should region 5 addresses without a valid TLB mapping be handled?
>
If you don't accept tlb misses thrown by tpa then you have to first make 
sure that the address is in the page table mapped segment and second 
walk the page tables in physical mode (see ivt.S nested TLB handler).

>Are there any other regions that need handling, and if so, how?
>
There shouldn't be other regions needed. Nevertheless you could also 
walk the page tables for them.

Christian

Re: INIT dumps broken?

[Kazuto Miyoshi]

Hello, 

Thank you for detailed explanation.

> So if our task is anything other than the initial "init_task",
> we'll take a nice region 7 address from ar.k6, and convert to
> physical by zapping bits [63:61].  But we happen to be running
> the init_task, then ar.k6 contains a region 5 address, and we
> create a garbage address by clearing the high bits, which can
> lead to an MCA (if the address points somewhere bad) or to
> stomping on some random place (if the address happens to be a
> legal physical address).

I checked the INIT log and it showed r13 and xip had actually 0xa..
address, the "stomping on some random place" scenario is probable.
I tried reducing memory from 2GB and it reproduced the MCA.

Best Regards,
--
Kazuto Miyoshi, NEC
miyoshi@hpc.bs1.fc.nec.co.jp

Re: INIT dumps broken?

[Takao Indoh]

On Thu, 16 Sep 2004 00:14:11 +1000, Keith Owens wrote:

>On Wed, 15 Sep 2004 00:56:15 -0700, 
>David Mosberger <davidm@napali.hpl.hp.com> wrote:
>>>>>>> On Wed, 15 Sep 2004 17:16:47 +1000, Keith Owens <kaos@sgi.com> said:
>>
>>  Keith> The use of 0xa.. for cpu 0 is causing an MCA when cpu 0
>>  Keith> receives INIT.
>>
>>The MINSTATE_PHYS-version of MINSTATE_GET_CURRENT() looks broken to
>>me.  Perhaps changing "dep reg=0,reg,61,3" to a "tpa" is all that's
>>needed there.  Can someone check this out?  I probably won't get to it
>>today and will be on vacation starting tomorrow.
>
>Agreed that MINSTATE_PHYS is broken.  Not sure that tpa is reliable
>during INIT/MCA, especially as one of the possible causes of an MCA is
>an invalid tlb.
>
>Can anybody describe why cpu 0 changed from 0xe... to 0xa... for the
>idle task between 2.6.7-rc3 and 2.6.8?

MINSTATE_END_SAVE_MIN_PHYS is also wrong?

#define MINSTATE_END_SAVE_MIN_PHYS
        or r12=r12,r14;         /* make sp a kernel virtual address */
        or r13=r13,r14;         /* make `current' a kernel virtual address */
        ;;

In the case of swapper on cpu0, it calculates wrong address.
If address of swapper on cpu0 is a000000100634000, this macro
sets r13 to e000000100634000.


Regards,
Takao Indoh

Re: INIT dumps broken?

[Keith Owens]

On Mon, 27 Sep 2004 22:34:15 +0900, 
Takao Indoh <indou.takao@soft.fujitsu.com> wrote:
>On Thu, 16 Sep 2004 00:14:11 +1000, Keith Owens wrote:
>>Agreed that MINSTATE_PHYS is broken.  Not sure that tpa is reliable
>>during INIT/MCA, especially as one of the possible causes of an MCA is
>>an invalid tlb.
>>
>>Can anybody describe why cpu 0 changed from 0xe... to 0xa... for the
>>idle task between 2.6.7-rc3 and 2.6.8?
>
>MINSTATE_END_SAVE_MIN_PHYS is also wrong?
>
>#define MINSTATE_END_SAVE_MIN_PHYS
>        or r12=r12,r14;         /* make sp a kernel virtual address */
>        or r13=r13,r14;         /* make `current' a kernel virtual address */
>        ;;
>
>In the case of swapper on cpu0, it calculates wrong address.
>If address of swapper on cpu0 is a000000100634000, this macro
>sets r13 to e000000100634000.

Fixed in Tony's bk tree since September 16.

Re: INIT dumps broken?

[Takao Indoh]

On Tue, 28 Sep 2004 09:03:45 +1000, Keith Owens wrote:

>>MINSTATE_END_SAVE_MIN_PHYS is also wrong?
>>
>>#define MINSTATE_END_SAVE_MIN_PHYS
>>        or r12=r12,r14;         /* make sp a kernel virtual address */
>>        or r13=r13,r14;         /* make `current' a kernel virtual address
>>  */
>>        ;;
>>
>>In the case of swapper on cpu0, it calculates wrong address.
>>If address of swapper on cpu0 is a000000100634000, this macro
>>sets r13 to e000000100634000.
>
>Fixed in Tony's bk tree since September 16.

I found a patch you had posted. Thanks.

Regards,
Takao Indoh