Re: [Xen-users] ARM: Xen on Vexpress

Re: [Xen-users] ARM: Xen on Vexpress

[Julien Grall]

(CCing Xen-devel)

On 06/16/2014 05:32 PM, Jeenu Viswambharan wrote:
> On Thu, Jun 12, 2014 at 15:50:02, Jeenu Viswambharan wrote:
>>>>   - What does DOM0 see as its PC at entry?
>>>
>>> The load address of the kernel image, since the zImage protocol
>>> requires us to enter the kernel at that offset.
>>
>> I managed to break at eret before guest entry, and the PC is
>> 0xafc00000 itself. It does fail to boot bare metal too from that
>> address. I'll see if I can find what's going wrong.
> 
> I've been told that loading Linux at that high an offset from the memory
> base wouldn't work after all, even on bare metal. I therefore set
> kernel_addr_r to 0x80008000, which makes Xen load DOM0 at 0x8fc00000. I
> can see Linux starting to boot from this address on bare metal, but With
> Xen I'm seeing "uncompression error" (not printed on to the console, but
> through debugger inspection).
> 
> I haven't pinned down why exactly it's throwing this. While I'm at it,
> I'd appreciate any helpful pointers.

Did you try to use CONFIG_VMSPLIT_3G rather than CONFIG_VMSPLIT_2G as I
suggested last week?

Using CONFIG_VMSPLIT_3G works correctly with 3.15. It looks like another
issue in the pa - virt delta when it's positive.

IIRC, it's because linux is assuming that delta is always negative (see
comment on __PV_BITS_31_24 in arch/arm/include/asm/memory.h).

At first glance, handling positive offset will be hard because, their
are rely on the compiler to provide the right layout for the
instruction. Positive number is not encoded the same way. I will have to
spend a bit more time to fully understand this code.

With Ian multiple bank support patch for DOM0, I think we can safely
assume that the delta will (nearly?) always be the same.

Regards,

-- 
Julien Grall

Re: [Xen-users] ARM: Xen on Vexpress

[Ian Campbell]

On Tue, 2014-06-17 at 18:50 +0100, Julien Grall wrote:
> (CCing Xen-devel)
> 
> On 06/16/2014 05:32 PM, Jeenu Viswambharan wrote:
> > On Thu, Jun 12, 2014 at 15:50:02, Jeenu Viswambharan wrote:
> >>>>   - What does DOM0 see as its PC at entry?
> >>>
> >>> The load address of the kernel image, since the zImage protocol
> >>> requires us to enter the kernel at that offset.
> >>
> >> I managed to break at eret before guest entry, and the PC is
> >> 0xafc00000 itself. It does fail to boot bare metal too from that
> >> address. I'll see if I can find what's going wrong.
> > 
> > I've been told that loading Linux at that high an offset from the memory
> > base wouldn't work after all, even on bare metal. I therefore set
> > kernel_addr_r to 0x80008000, which makes Xen load DOM0 at 0x8fc00000. I
> > can see Linux starting to boot from this address on bare metal, but With
> > Xen I'm seeing "uncompression error" (not printed on to the console, but
> > through debugger inspection).
> > 
> > I haven't pinned down why exactly it's throwing this. While I'm at it,
> > I'd appreciate any helpful pointers.
> 
> Did you try to use CONFIG_VMSPLIT_3G rather than CONFIG_VMSPLIT_2G as I
> suggested last week?
> 
> Using CONFIG_VMSPLIT_3G works correctly with 3.15. It looks like another
> issue in the pa - virt delta when it's positive.
> 
> IIRC, it's because linux is assuming that delta is always negative (see
> comment on __PV_BITS_31_24 in arch/arm/include/asm/memory.h).
> 
> At first glance, handling positive offset will be hard because, their
> are rely on the compiler to provide the right layout for the
> instruction. Positive number is not encoded the same way. I will have to
> spend a bit more time to fully understand this code.
> 
> With Ian multiple bank support patch for DOM0, I think we can safely
> assume that the delta will (nearly?) always be the same.

That would be a coincidence of the implementation, not a guarantee which
we would make.

If there is an issue with the pa - virt delta handling then we should
raise it with Linux upstream.

Ian.

Re: [Xen-users] ARM: Xen on Vexpress

[Julien Grall]

On 18/06/14 09:29, Ian Campbell wrote:
> On Tue, 2014-06-17 at 18:50 +0100, Julien Grall wrote:
>> (CCing Xen-devel)
>>
>> On 06/16/2014 05:32 PM, Jeenu Viswambharan wrote:
>>> On Thu, Jun 12, 2014 at 15:50:02, Jeenu Viswambharan wrote:
>>>>>>    - What does DOM0 see as its PC at entry?
>>>>>
>>>>> The load address of the kernel image, since the zImage protocol
>>>>> requires us to enter the kernel at that offset.
>>>>
>>>> I managed to break at eret before guest entry, and the PC is
>>>> 0xafc00000 itself. It does fail to boot bare metal too from that
>>>> address. I'll see if I can find what's going wrong.
>>>
>>> I've been told that loading Linux at that high an offset from the memory
>>> base wouldn't work after all, even on bare metal. I therefore set
>>> kernel_addr_r to 0x80008000, which makes Xen load DOM0 at 0x8fc00000. I
>>> can see Linux starting to boot from this address on bare metal, but With
>>> Xen I'm seeing "uncompression error" (not printed on to the console, but
>>> through debugger inspection).
>>>
>>> I haven't pinned down why exactly it's throwing this. While I'm at it,
>>> I'd appreciate any helpful pointers.
>>
>> Did you try to use CONFIG_VMSPLIT_3G rather than CONFIG_VMSPLIT_2G as I
>> suggested last week?
>>
>> Using CONFIG_VMSPLIT_3G works correctly with 3.15. It looks like another
>> issue in the pa - virt delta when it's positive.
>>
>> IIRC, it's because linux is assuming that delta is always negative (see
>> comment on __PV_BITS_31_24 in arch/arm/include/asm/memory.h).
>>
>> At first glance, handling positive offset will be hard because, their
>> are rely on the compiler to provide the right layout for the
>> instruction. Positive number is not encoded the same way. I will have to
>> spend a bit more time to fully understand this code.
>>
>> With Ian multiple bank support patch for DOM0, I think we can safely
>> assume that the delta will (nearly?) always be the same.
>
> That would be a coincidence of the implementation, not a guarantee which
> we would make.
>
> If there is an issue with the pa - virt delta handling then we should
> raise it with Linux upstream.

I will raise again the issue to Linux. But I start to think this is a 
restriction by Linux but I can't find a clear comment about it. Not much 
(maybe none of) ARM platform has the start physical address very high.

Regards,

-- 
Julien Grall

Re: [Xen-users] ARM: Xen on Vexpress

[Ian Campbell]

On Wed, 2014-06-18 at 09:40 +0100, Julien Grall wrote:
> 
> On 18/06/14 09:29, Ian Campbell wrote:
> > On Tue, 2014-06-17 at 18:50 +0100, Julien Grall wrote:
> >> (CCing Xen-devel)
> >>
> >> On 06/16/2014 05:32 PM, Jeenu Viswambharan wrote:
> >>> On Thu, Jun 12, 2014 at 15:50:02, Jeenu Viswambharan wrote:
> >>>>>>    - What does DOM0 see as its PC at entry?
> >>>>>
> >>>>> The load address of the kernel image, since the zImage protocol
> >>>>> requires us to enter the kernel at that offset.
> >>>>
> >>>> I managed to break at eret before guest entry, and the PC is
> >>>> 0xafc00000 itself. It does fail to boot bare metal too from that
> >>>> address. I'll see if I can find what's going wrong.
> >>>
> >>> I've been told that loading Linux at that high an offset from the memory
> >>> base wouldn't work after all, even on bare metal. I therefore set
> >>> kernel_addr_r to 0x80008000, which makes Xen load DOM0 at 0x8fc00000. I
> >>> can see Linux starting to boot from this address on bare metal, but With
> >>> Xen I'm seeing "uncompression error" (not printed on to the console, but
> >>> through debugger inspection).
> >>>
> >>> I haven't pinned down why exactly it's throwing this. While I'm at it,
> >>> I'd appreciate any helpful pointers.
> >>
> >> Did you try to use CONFIG_VMSPLIT_3G rather than CONFIG_VMSPLIT_2G as I
> >> suggested last week?
> >>
> >> Using CONFIG_VMSPLIT_3G works correctly with 3.15. It looks like another
> >> issue in the pa - virt delta when it's positive.
> >>
> >> IIRC, it's because linux is assuming that delta is always negative (see
> >> comment on __PV_BITS_31_24 in arch/arm/include/asm/memory.h).
> >>
> >> At first glance, handling positive offset will be hard because, their
> >> are rely on the compiler to provide the right layout for the
> >> instruction. Positive number is not encoded the same way. I will have to
> >> spend a bit more time to fully understand this code.
> >>
> >> With Ian multiple bank support patch for DOM0, I think we can safely
> >> assume that the delta will (nearly?) always be the same.
> >
> > That would be a coincidence of the implementation, not a guarantee which
> > we would make.
> >
> > If there is an issue with the pa - virt delta handling then we should
> > raise it with Linux upstream.
> 
> I will raise again the issue to Linux.

Thank you.

>  But I start to think this is a 
> restriction by Linux but I can't find a clear comment about it. Not much 
> (maybe none of) ARM platform has the start physical address very high.

I wouldn't be terribly surprised if there were platforms with RAM
starting at 3GB, which would break with VMSPLIT_2G and VMSPLIT_1G and
I'm sure there are platforms with ram at 2GB which would fail with
VMSPLIT_1G (vexpress falls into this category, I think).

In the days before multiplatform this was fine -- the restriction could
be expressed in Kconfig, but these days I don't think that flies...

Ian.