Re: [crash, bisected] Re: [PATCH 3/4] x86_64: Fold pda into per cpu area

[Mike Travis <travis@sgi.com>]

Eric W. Biederman wrote:
> Jeremy Fitzhardinge <jeremy@goop.org> writes:
> 
>> Christoph Lameter wrote:
>>> On Fri, 20 Jun 2008, Jeremy Fitzhardinge wrote:
>>>
>>>
>>>>> The loader setup for the percpu section changes with zero basing. Maybe that
>>>>> has bad side effects
>>>>>
>>>> How does it work?  The symbols in the percpu segment are 0-based, but where
>>>> does the data for the sections which correspond to that segment go?
>>>>
>>> Its loaded at __per_cpu_load but the symbols have addresses starting at 0.
>>>
>> Yes, which leads to an odd-looking ELF file where the Phdrs aren't sorted by
>> virtual address order.  I'm wondering what would happen if a bootloader that
>> actually understood ELF files tried to load it as an actual ELF
>> file...
> 
> Well /sbin/kexec looks at the physical addresses not the virtual ones
> so that may not be a problem.
> 
>>>> So the question is what kernel virtual address is it being loaded to?
>>>> __per_cpu_load is ffffffff808d1000, so ffffffff808d6000 is what you'd
>>>> expect...
>>>>
>>> Correct.
>>>
>> Well, reading back from that address got zeros, so something is
>> amiss.
> 
> Weird.
> 
>>>> Hm, but what happens when this gets converted to bzImage?  Hm, looks OK, I
>>>> think.
>>>>
>>>> BTW, I think __per_cpu_load will cause trouble if you make a relocatable
>>>> kernel, being an absolute symbol.  But I have relocation off at the moment.
>>>>
>>> Hmmm.... we could add the relocation offset to __per_cpu_load? __per_cpu_load
>>> is used very sparingly. Basically only useful during early boot and when a new
>>> per cpu area has to be setup. In that case we want to copy from __per_cpu_load
>>> to the newly allocated percpu area.
>>>
>> Yes, it should be fairly easy to manually relocate it by applying the (load -
>> link) offset to it.
> 
> For x86_64 all kernels are built relocatable as the only cost was
> changing the physical addresses in the initial page tables.  The
> virtual address always remain the same but the physical addresses
> change. So that could be part of what is going on.
> 
> Is this a change that only got tested on x86_32?

I'm only testing this on x86_64.  The zero-based percpu/pda changes worked fine
up until just recently.  At first it was one of Ingo's "randconfig" config files
that was tripping it up, but lately it's not working on any config.

> 
> As long as we are not changing the way the kernel virtual address
> are actually being used we should be ok with a change to make the pda
> 0 based.  Still it is an area you need to be especially careful with.

The major gotcha's seem to be in referencing the per_cpu symbol directly though
I've examined them all and nothing seems amiss.

> 
> 
> Eric

Thanks,
Mike