Oops in guest after ioremap() on ARMv7

Oops in guest after ioremap() on ARMv7

[David Vrabel]

When running the linux kernel on the ARMv7 envelope model as a guest
under the Xen hypervisor there is a oops (see below for an example of
the page translation fault) when trying to access ioremap()'d memory.

The same kernel works fine when not running under the hypervisor.

It's a 3.2.0-rc5+ kernel with the two additional linux-arch-arm
branches: arm-arch/vexpress and arm-arch/arm-lpae.

Calling flush_cache_all() in flush_cache_vmap() makes it work.  What
isn't being correctly flushed?  I see that flush_pmd_entry() and
cpu_v7_set_pte_ext() already flush the L1 and L2 translation table
entries and I can't think of anything else that would need to be flushed
(unless the mapped virtual addresses need to be flushed as well?)

The "Barrier Litmus Tests and Cookbook" says that a TLB flush and a
branch predictor flush are required after a translation table entry
update.  This seems not to be done but adding this didn't seem to help
(and using local_flush_tlb_all()) in flush_cache_vmap() didn't help either).

I don't see anything in the hypervisor that could be causing this as the
fault is occurring at stage 1 and not stage 2 translation.

David

[    0.200952] Unable to handle kernel paging request at virtual address
c8804050
[    0.203234] pgd = c0004000
[    0.204147] [c8804050] *pgd=8781a811, *pte=1a000653, *ppte=1a000453
[    0.206120] Internal error: Oops: 7 [#1]
[    0.207381] CPU: 0    Tainted: G        W     (3.2.0-rc5.dv #123)
[    0.209338] PC is at v2m_init+0x44/0x100
[    0.210616] LR is at ioremap_page_range+0x100/0x164
[    0.212197] pc : [<c0325130>]    lr : [<c013e278>]    psr: a0000013
[    0.212235] sp : c781dfa0  ip : 00000001  fp : 00000000
[    0.215880] r10: 00000000  r9 : 00000000  r8 : 00000000
[    0.217570] r7 : 00000013  r6 : 00000000  r5 : c0321f1c  r4 : c033d184
[    0.219680] r3 : c8804000  r2 : c0325128  r1 : 1a000613  r0 : c8804000
[    0.221756] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM
Segment kernel
[    0.224159] Control: 10c53c7d  Table: 80004059  DAC: 00000015
[    0.225990] Process swapper (pid: 1, stack limit = 0xc781c268)
[    0.227858] Stack: (0xc781dfa0 to 0xc781e000)
[    0.229325] dfa0: c03250ec c033d184 c0321f1c c0321f38 c781c000
c031f7c0 c033d430 33369eb8
[    0.231955] dfc0: 00000000 00000000 00000080 c033d184 c033d430
c0009eb8 00000013 00000000
[    0.234652] dfe0: 00000000 c031f908 00000000 00000000 c031f890
c0009eb8 dfdfdfcf cfdfdfdf
[    0.237304] [<c0325130>] (v2m_init+0x44/0x100) from [<c0321f38>]
(customize_machine+0x1c/0x28)
[    0.240099] [<c0321f38>] (customize_machine+0x1c/0x28) from
[<c031f7c0>] (do_one_initcall+0x90/0x160)
[    0.243080] [<c031f7c0>] (do_one_initcall+0x90/0x160) from
[<c031f908>] (kernel_init+0x78/0x114)
[    0.245958] [<c031f908>] (kernel_init+0x78/0x114) from [<c0009eb8>]
(kernel_thread_exit+0x0/0x8)
[    0.248823] Code: e3a02000 ebf3a9b7 e2503000 0a00000a (e5933050)

Oops in guest after ioremap() on ARMv7

[Catalin Marinas]

On Thu, Dec 22, 2011 at 12:08:07PM +0000, David Vrabel wrote:
> When running the linux kernel on the ARMv7 envelope model as a guest
> under the Xen hypervisor there is a oops (see below for an example of
> the page translation fault) when trying to access ioremap()'d memory.
> 
> The same kernel works fine when not running under the hypervisor.
> 
> It's a 3.2.0-rc5+ kernel with the two additional linux-arch-arm
> branches: arm-arch/vexpress and arm-arch/arm-lpae.
> 
> Calling flush_cache_all() in flush_cache_vmap() makes it work.  What
> isn't being correctly flushed?  I see that flush_pmd_entry() and
> cpu_v7_set_pte_ext() already flush the L1 and L2 translation table
> entries and I can't think of anything else that would need to be flushed
> (unless the mapped virtual addresses need to be flushed as well?)
> 
> The "Barrier Litmus Tests and Cookbook" says that a TLB flush and a
> branch predictor flush are required after a translation table entry
> update.  This seems not to be done but adding this didn't seem to help
> (and using local_flush_tlb_all()) in flush_cache_vmap() didn't help either).
> 
> I don't see anything in the hypervisor that could be causing this as the
> fault is occurring at stage 1 and not stage 2 translation.

Interesting error, I don't have an immediate idea of what might be
wrong, just some questions.

What's the value of the VTCR register for this guest? Are the
translation table walks marked as cacheable? Also, are the page table
attributes Normal Cacheable in the stage 2 translation? The processor
chooses the more restrictive attribute between stage 1 and stage 2.

-- 
Catalin

Oops in guest after ioremap() on ARMv7

[David Vrabel]

On 22/12/11 14:49, Catalin Marinas wrote:
> On Thu, Dec 22, 2011 at 12:08:07PM +0000, David Vrabel wrote:
>> When running the linux kernel on the ARMv7 envelope model as a guest
>> under the Xen hypervisor there is a oops (see below for an example of
>> the page translation fault) when trying to access ioremap()'d memory.

The translation tables for userspace seem to be also affected.  The
program repeatedly faults with a translation fault on the same address.
 Putting a cache_flush_all() after the call to handle_mm_fault() in
__do_page_fault() makes userspace work as well.

>> The same kernel works fine when not running under the hypervisor.
>>
>> It's a 3.2.0-rc5+ kernel with the two additional linux-arch-arm
>> branches: arm-arch/vexpress and arm-arch/arm-lpae.
>>
>> Calling flush_cache_all() in flush_cache_vmap() makes it work.  What
>> isn't being correctly flushed?  I see that flush_pmd_entry() and
>> cpu_v7_set_pte_ext() already flush the L1 and L2 translation table
>> entries and I can't think of anything else that would need to be flushed
>> (unless the mapped virtual addresses need to be flushed as well?)
>>
>> The "Barrier Litmus Tests and Cookbook" says that a TLB flush and a
>> branch predictor flush are required after a translation table entry
>> update.  This seems not to be done but adding this didn't seem to help
>> (and using local_flush_tlb_all()) in flush_cache_vmap() didn't help either).
>>
>> I don't see anything in the hypervisor that could be causing this as the
>> fault is occurring at stage 1 and not stage 2 translation.
> 
> Interesting error, I don't have an immediate idea of what might be
> wrong, just some questions.
> 
> What's the value of the VTCR register for this guest? Are the
> translation table walks marked as cacheable? Also, are the page table
> attributes Normal Cacheable in the stage 2 translation? The processor
> chooses the more restrictive attribute between stage 1 and stage 2.

VTCR = 0x80002558 which is: Outer Shareable; Normal memory, outer
write-back write-allocate cacheable; Normal memory, inner write-back,
write-allocate cacheable.

L3 TT entries for stage 2 have the following attributes:
Outer-Shareable; Normal, inner write-back cachable; Normal, outer
write-back cacheable.

These look sensible to me.

David

Oops in guest after ioremap() on ARMv7

[Catalin Marinas]

On Thu, Dec 22, 2011 at 04:38:23PM +0000, David Vrabel wrote:
> On 22/12/11 14:49, Catalin Marinas wrote:
> > On Thu, Dec 22, 2011 at 12:08:07PM +0000, David Vrabel wrote:
> >> When running the linux kernel on the ARMv7 envelope model as a guest
> >> under the Xen hypervisor there is a oops (see below for an example of
> >> the page translation fault) when trying to access ioremap()'d memory.
> 
> The translation tables for userspace seem to be also affected.  The
> program repeatedly faults with a translation fault on the same address.
>  Putting a cache_flush_all() after the call to handle_mm_fault() in
> __do_page_fault() makes userspace work as well.

With the classic page tables, on A15 we need this patch:

http://git.kernel.org/?p=linux/kernel/git/cmarinas/linux.git;a=commitdiff_plain;h=27cbbe6b1e17fa0b954edd37e26d601bdd7766a6

But that's to do with TLBs rather than cache and it only shows on real
hardware rather than model.

> >> The same kernel works fine when not running under the hypervisor.
> >>
> >> It's a 3.2.0-rc5+ kernel with the two additional linux-arch-arm
> >> branches: arm-arch/vexpress and arm-arch/arm-lpae.
> >>
> >> Calling flush_cache_all() in flush_cache_vmap() makes it work.  What
> >> isn't being correctly flushed?  I see that flush_pmd_entry() and
> >> cpu_v7_set_pte_ext() already flush the L1 and L2 translation table
> >> entries and I can't think of anything else that would need to be flushed
> >> (unless the mapped virtual addresses need to be flushed as well?)
> >>
> >> The "Barrier Litmus Tests and Cookbook" says that a TLB flush and a
> >> branch predictor flush are required after a translation table entry
> >> update.  This seems not to be done but adding this didn't seem to help
> >> (and using local_flush_tlb_all()) in flush_cache_vmap() didn't help either).
> >>
> >> I don't see anything in the hypervisor that could be causing this as the
> >> fault is occurring at stage 1 and not stage 2 translation.
> > 
> > Interesting error, I don't have an immediate idea of what might be
> > wrong, just some questions.
> > 
> > What's the value of the VTCR register for this guest? Are the
> > translation table walks marked as cacheable? Also, are the page table
> > attributes Normal Cacheable in the stage 2 translation? The processor
> > chooses the more restrictive attribute between stage 1 and stage 2.
> 
> VTCR = 0x80002558 which is: Outer Shareable; Normal memory, outer
> write-back write-allocate cacheable; Normal memory, inner write-back,
> write-allocate cacheable.
> 
> L3 TT entries for stage 2 have the following attributes:
> Outer-Shareable; Normal, inner write-back cachable; Normal, outer
> write-back cacheable.
> 
> These look sensible to me.

They look fine (UP system). BTW, I assume that the hypervisor also
flushes the caches and TLBs for the stage 2 translation tables.

It could as well be a model bug but people are on holiday at the moment
(and I'm off shortly as well, until 3rd of January). Could you try to
disable the cacheability of the page table walks for both stage 1 (TTBRx
with classic page tables or TTBCR with LPAE) and stage 2 (VTCR)? Since
Linux does the correct cache flushing and I assume the hypervisor as
well, this may work around possible model bug.

-- 
Catalin

Oops in guest after ioremap() on ARMv7

[David Vrabel]

On 22/12/11 18:13, Catalin Marinas wrote:
> On Thu, Dec 22, 2011 at 04:38:23PM +0000, David Vrabel wrote:
>> On 22/12/11 14:49, Catalin Marinas wrote:
>>> On Thu, Dec 22, 2011 at 12:08:07PM +0000, David Vrabel wrote:
>>>> When running the linux kernel on the ARMv7 envelope model as a guest
>>>> under the Xen hypervisor there is a oops (see below for an example of
>>>> the page translation fault) when trying to access ioremap()'d memory.
>>
>> The translation tables for userspace seem to be also affected.  The
>> program repeatedly faults with a translation fault on the same address.
>>  Putting a cache_flush_all() after the call to handle_mm_fault() in
>> __do_page_fault() makes userspace work as well.
> 
> With the classic page tables, on A15 we need this patch:
> 
> http://git.kernel.org/?p=linux/kernel/git/cmarinas/linux.git;a=commitdiff_plain;h=27cbbe6b1e17fa0b954edd37e26d601bdd7766a6
> 
> But that's to do with TLBs rather than cache and it only shows on real
> hardware rather than model.
> 
>>>> The same kernel works fine when not running under the hypervisor.
>>>>
>>>> It's a 3.2.0-rc5+ kernel with the two additional linux-arch-arm
>>>> branches: arm-arch/vexpress and arm-arch/arm-lpae.
>>>>
>>>> Calling flush_cache_all() in flush_cache_vmap() makes it work.  What
>>>> isn't being correctly flushed?  I see that flush_pmd_entry() and
>>>> cpu_v7_set_pte_ext() already flush the L1 and L2 translation table
>>>> entries and I can't think of anything else that would need to be flushed
>>>> (unless the mapped virtual addresses need to be flushed as well?)
>>>>
>>>> The "Barrier Litmus Tests and Cookbook" says that a TLB flush and a
>>>> branch predictor flush are required after a translation table entry
>>>> update.  This seems not to be done but adding this didn't seem to help
>>>> (and using local_flush_tlb_all()) in flush_cache_vmap() didn't help either).
>>>>
>>>> I don't see anything in the hypervisor that could be causing this as the
>>>> fault is occurring at stage 1 and not stage 2 translation.
>>>
>>> Interesting error, I don't have an immediate idea of what might be
>>> wrong, just some questions.
>>>
>>> What's the value of the VTCR register for this guest? Are the
>>> translation table walks marked as cacheable? Also, are the page table
>>> attributes Normal Cacheable in the stage 2 translation? The processor
>>> chooses the more restrictive attribute between stage 1 and stage 2.
>>
>> VTCR = 0x80002558 which is: Outer Shareable; Normal memory, outer
>> write-back write-allocate cacheable; Normal memory, inner write-back,
>> write-allocate cacheable.
>>
>> L3 TT entries for stage 2 have the following attributes:
>> Outer-Shareable; Normal, inner write-back cachable; Normal, outer
>> write-back cacheable.
>>
>> These look sensible to me.
> 
> They look fine (UP system). BTW, I assume that the hypervisor also
> flushes the caches and TLBs for the stage 2 translation tables.

I think so. Cc'ing Ian Campbell who knows the hypervisor side better
than me.

> It could as well be a model bug but people are on holiday at the moment
> (and I'm off shortly as well, until 3rd of January). Could you try to
> disable the cacheability of the page table walks for both stage 1 (TTBRx
> with classic page tables or TTBCR with LPAE) and stage 2 (VTCR)? Since
> Linux does the correct cache flushing and I assume the hypervisor as
> well, this may work around possible model bug.

I can try this but probably not until the new year.

David

Oops in guest after ioremap() on ARMv7

[Ian Campbell]

On Thu, 2011-12-22 at 18:33 +0000, David Vrabel wrote:
> On 22/12/11 18:13, Catalin Marinas wrote:
> > On Thu, Dec 22, 2011 at 04:38:23PM +0000, David Vrabel wrote:
> >> On 22/12/11 14:49, Catalin Marinas wrote:
> >>> On Thu, Dec 22, 2011 at 12:08:07PM +0000, David Vrabel wrote:
> >>>> When running the linux kernel on the ARMv7 envelope model as a guest
> >>>> under the Xen hypervisor there is a oops (see below for an example of
> >>>> the page translation fault) when trying to access ioremap()'d memory.
> >>
> >> The translation tables for userspace seem to be also affected.  The
> >> program repeatedly faults with a translation fault on the same address.
> >>  Putting a cache_flush_all() after the call to handle_mm_fault() in
> >> __do_page_fault() makes userspace work as well.
> > 
> > With the classic page tables, on A15 we need this patch:
> > 
> > http://git.kernel.org/?p=linux/kernel/git/cmarinas/linux.git;a=commitdiff_plain;h=27cbbe6b1e17fa0b954edd37e26d601bdd7766a6
> > 
> > But that's to do with TLBs rather than cache and it only shows on real
> > hardware rather than model.
> > 
> >>>> The same kernel works fine when not running under the hypervisor.
> >>>>
> >>>> It's a 3.2.0-rc5+ kernel with the two additional linux-arch-arm
> >>>> branches: arm-arch/vexpress and arm-arch/arm-lpae.
> >>>>
> >>>> Calling flush_cache_all() in flush_cache_vmap() makes it work.  What
> >>>> isn't being correctly flushed?  I see that flush_pmd_entry() and
> >>>> cpu_v7_set_pte_ext() already flush the L1 and L2 translation table
> >>>> entries and I can't think of anything else that would need to be flushed
> >>>> (unless the mapped virtual addresses need to be flushed as well?)
> >>>>
> >>>> The "Barrier Litmus Tests and Cookbook" says that a TLB flush and a
> >>>> branch predictor flush are required after a translation table entry
> >>>> update.  This seems not to be done but adding this didn't seem to help
> >>>> (and using local_flush_tlb_all()) in flush_cache_vmap() didn't help either).
> >>>>
> >>>> I don't see anything in the hypervisor that could be causing this as the
> >>>> fault is occurring at stage 1 and not stage 2 translation.
> >>>
> >>> Interesting error, I don't have an immediate idea of what might be
> >>> wrong, just some questions.
> >>>
> >>> What's the value of the VTCR register for this guest? Are the
> >>> translation table walks marked as cacheable? Also, are the page table
> >>> attributes Normal Cacheable in the stage 2 translation? The processor
> >>> chooses the more restrictive attribute between stage 1 and stage 2.
> >>
> >> VTCR = 0x80002558 which is: Outer Shareable; Normal memory, outer
> >> write-back write-allocate cacheable; Normal memory, inner write-back,
> >> write-allocate cacheable.
> >>
> >> L3 TT entries for stage 2 have the following attributes:
> >> Outer-Shareable; Normal, inner write-back cachable; Normal, outer
> >> write-back cacheable.
> >>
> >> These look sensible to me.
> > 
> > They look fine (UP system). BTW, I assume that the hypervisor also
> > flushes the caches and TLBs for the stage 2 translation tables.
> 
> I think so. Cc'ing Ian Campbell who knows the hypervisor side better
> than me.

At the moment we build the entire p2m before we ever load the VTTBR or
enable stage-2 translations in the HCR. Is that sufficient or do we also
need to flush something?

Obviously we will need to make sure we do appropriate flushes when we
start needing to change the p2m of a running guest etc. Currently our
write_pte does a flush with DCCMVAC and in general our global flushes
are at the more aggressive end of the scale (correctness before
optimisation ;-)).

BTW the Xen code is all at
http://xenbits.xen.org/gitweb/?p=people/ianc/xen-unstable.git;a=tree;h=refs/heads/devel/arm;hb=devel/arm
interesting code from this PoV is likely to be
xen/arch/arm/{p2m,domain_build}.c and xen/include/asm-arm/page.h

Ian.

> 
> > It could as well be a model bug but people are on holiday at the moment
> > (and I'm off shortly as well, until 3rd of January). Could you try to
> > disable the cacheability of the page table walks for both stage 1 (TTBRx
> > with classic page tables or TTBCR with LPAE) and stage 2 (VTCR)? Since
> > Linux does the correct cache flushing and I assume the hypervisor as
> > well, this may work around possible model bug.
> 
> I can try this but probably not until the new year.
> 
> David

Oops in guest after ioremap() on ARMv7

[Catalin Marinas]

On Fri, Dec 23, 2011 at 12:00:25PM +0000, Ian Campbell wrote:
> On Thu, 2011-12-22 at 18:33 +0000, David Vrabel wrote:
> > On 22/12/11 18:13, Catalin Marinas wrote:
> > > On Thu, Dec 22, 2011 at 04:38:23PM +0000, David Vrabel wrote:
> > >> On 22/12/11 14:49, Catalin Marinas wrote:
> > >>> What's the value of the VTCR register for this guest? Are the
> > >>> translation table walks marked as cacheable? Also, are the page table
> > >>> attributes Normal Cacheable in the stage 2 translation? The processor
> > >>> chooses the more restrictive attribute between stage 1 and stage 2.
> > >>
> > >> VTCR = 0x80002558 which is: Outer Shareable; Normal memory, outer
> > >> write-back write-allocate cacheable; Normal memory, inner write-back,
> > >> write-allocate cacheable.

BTW, it would be better to mark the stage 2 table walks as inner
shareable, i.e. VTCR = 0x80003558. That's because in case of SMP you
would want page table accesses to snoop the caches of the other CPUs.

> > >> L3 TT entries for stage 2 have the following attributes:
> > >> Outer-Shareable; Normal, inner write-back cachable; Normal, outer
> > >> write-back cacheable.
> > >>
> > >> These look sensible to me.
> > > 
> > > They look fine (UP system). BTW, I assume that the hypervisor also
> > > flushes the caches and TLBs for the stage 2 translation tables.
> > 
> > I think so. Cc'ing Ian Campbell who knows the hypervisor side better
> > than me.
> 
> At the moment we build the entire p2m before we ever load the VTTBR or
> enable stage-2 translations in the HCR. Is that sufficient or do we also
> need to flush something?

If the model does not correctly implement cacheable page table walks for
either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate
that it should), the hypervisor would need to clean the D-cache to the
point of unification (not necessary to go to the point of coherency)
before any of the state 2 translation tables are used.

> Obviously we will need to make sure we do appropriate flushes when we
> start needing to change the p2m of a running guest etc. Currently our
> write_pte does a flush with DCCMVAC and in general our global flushes
> are at the more aggressive end of the scale (correctness before
> optimisation ;-)).

You can use DCCMVAU but could also check the coherent walk bits in
ID_MMFR3 (assuming that the model works correctly).

-- 
Catalin

Oops in guest after ioremap() on ARMv7

[Ian Campbell]

On Tue, 2012-01-03 at 16:50 +0000, Catalin Marinas wrote:
> On Fri, Dec 23, 2011 at 12:00:25PM +0000, Ian Campbell wrote:
> > On Thu, 2011-12-22 at 18:33 +0000, David Vrabel wrote:
> > > On 22/12/11 18:13, Catalin Marinas wrote:
> > > > On Thu, Dec 22, 2011 at 04:38:23PM +0000, David Vrabel wrote:
> > > >> On 22/12/11 14:49, Catalin Marinas wrote:
> > > >>> What's the value of the VTCR register for this guest? Are the
> > > >>> translation table walks marked as cacheable? Also, are the page table
> > > >>> attributes Normal Cacheable in the stage 2 translation? The processor
> > > >>> chooses the more restrictive attribute between stage 1 and stage 2.
> > > >>
> > > >> VTCR = 0x80002558 which is: Outer Shareable; Normal memory, outer
> > > >> write-back write-allocate cacheable; Normal memory, inner write-back,
> > > >> write-allocate cacheable.
> 
> BTW, it would be better to mark the stage 2 table walks as inner
> shareable, i.e. VTCR = 0x80003558. That's because in case of SMP you
> would want page table accesses to snoop the caches of the other CPUs.

I did this. It didn't make any difference to the problem (I don't think
you were suggesting it would).

> > > >> L3 TT entries for stage 2 have the following attributes:
> > > >> Outer-Shareable; Normal, inner write-back cachable; Normal, outer
> > > >> write-back cacheable.
> > > >>
> > > >> These look sensible to me.
> > > > 
> > > > They look fine (UP system). BTW, I assume that the hypervisor also
> > > > flushes the caches and TLBs for the stage 2 translation tables.
> > > 
> > > I think so. Cc'ing Ian Campbell who knows the hypervisor side better
> > > than me.
> > 
> > At the moment we build the entire p2m before we ever load the VTTBR or
> > enable stage-2 translations in the HCR. Is that sufficient or do we also
> > need to flush something?
> 
> If the model does not correctly implement cacheable page table walks for
> either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate
> that it should), the hypervisor would need to clean the D-cache to the
> point of unification (not necessary to go to the point of coherency)
> before any of the state 2 translation tables are used.

I added a copy of Linux's v7_flush_dcache_all after building the p2m but
just before loading the VTTBR and it didn't help.

I turned off cacheability in VTCR and that didn't help either.

Then I turned off cacheability in the Linux TTBR{0,1} (by frobbing both
TTB_FLAGS_UP and TTB_FLAGS_SMP to use FOO_NC everywhere) and this _did_
make a difference -- the kernel now boots.

I then tested just that change by itself and it seems to have done the
trick.

Does that indicate a model bug?

FWIW ID_MMFR3 is 0x02102111 on the model I've got so you are right that
it should to cacheable walks.

Cheers,
Ian.

> > Obviously we will need to make sure we do appropriate flushes when we
> > start needing to change the p2m of a running guest etc. Currently our
> > write_pte does a flush with DCCMVAC and in general our global flushes
> > are at the more aggressive end of the scale (correctness before
> > optimisation ;-)).
> 
> You can use DCCMVAU but could also check the coherent walk bits in
> ID_MMFR3 (assuming that the model works correctly).
> 

Oops in guest after ioremap() on ARMv7

[Catalin Marinas]

On 19 January 2012 17:35, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Tue, 2012-01-03 at 16:50 +0000, Catalin Marinas wrote:
>> On Fri, Dec 23, 2011 at 12:00:25PM +0000, Ian Campbell wrote:
>> > At the moment we build the entire p2m before we ever load the VTTBR or
>> > enable stage-2 translations in the HCR. Is that sufficient or do we also
>> > need to flush something?
>>
>> If the model does not correctly implement cacheable page table walks for
>> either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate
>> that it should), the hypervisor would need to clean the D-cache to the
>> point of unification (not necessary to go to the point of coherency)
>> before any of the state 2 translation tables are used.
>
> I added a copy of Linux's v7_flush_dcache_all after building the p2m but
> just before loading the VTTBR and it didn't help.
>
> I turned off cacheability in VTCR and that didn't help either.
>
> Then I turned off cacheability in the Linux TTBR{0,1} (by frobbing both
> TTB_FLAGS_UP and TTB_FLAGS_SMP to use FOO_NC everywhere) and this _did_
> make a difference -- the kernel now boots.
>
> I then tested just that change by itself and it seems to have done the
> trick.
>
> Does that indicate a model bug?

It is possible. The scenario I'm thinking of is that with cacheable
PTWs enabled in TTBR, the model wrongly decides to use the
intermediate physical address (IPA) to look up the caches and gets the
wrong information.

I'll take this to the model guys but most likely they'll ask for an
image to load and just run. Could you provide such simple image
(minimal filesystem)? I'm not familiar with Xen and building it.

(I'll first ask the model guys, maybe they can spot the error easily
without additional debugging).

-- 
Catalin

Oops in guest after ioremap() on ARMv7

[Ian Campbell]

On Tue, 2012-01-31 at 14:40 +0000, Catalin Marinas wrote:
> On 19 January 2012 17:35, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Tue, 2012-01-03 at 16:50 +0000, Catalin Marinas wrote:
> >> On Fri, Dec 23, 2011 at 12:00:25PM +0000, Ian Campbell wrote:
> >> > At the moment we build the entire p2m before we ever load the VTTBR or
> >> > enable stage-2 translations in the HCR. Is that sufficient or do we also
> >> > need to flush something?
> >>
> >> If the model does not correctly implement cacheable page table walks for
> >> either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate
> >> that it should), the hypervisor would need to clean the D-cache to the
> >> point of unification (not necessary to go to the point of coherency)
> >> before any of the state 2 translation tables are used.
> >
> > I added a copy of Linux's v7_flush_dcache_all after building the p2m but
> > just before loading the VTTBR and it didn't help.
> >
> > I turned off cacheability in VTCR and that didn't help either.
> >
> > Then I turned off cacheability in the Linux TTBR{0,1} (by frobbing both
> > TTB_FLAGS_UP and TTB_FLAGS_SMP to use FOO_NC everywhere) and this _did_
> > make a difference -- the kernel now boots.
> >
> > I then tested just that change by itself and it seems to have done the
> > trick.
> >
> > Does that indicate a model bug?
> 
> It is possible. The scenario I'm thinking of is that with cacheable
> PTWs enabled in TTBR, the model wrongly decides to use the
> intermediate physical address (IPA) to look up the caches and gets the
> wrong information.
> 
> I'll take this to the model guys but most likely they'll ask for an
> image to load and just run. Could you provide such simple image
> (minimal filesystem)?

Sure. It'd be a Xen image & a Linux kernel image. The filesystem I'm
using is currently the one which comes from your 3rd party whatsit pack
but this bug stops us getting as far as trying to mount it anyway.

>  I'm not familiar with Xen and building it.
> 
> (I'll first ask the model guys, maybe they can spot the error easily
> without additional debugging).

OK, I'll wait till I hear back before pull it together.

Ian.

Oops in guest after ioremap() on ARMv7

[Ian Campbell]

Hi Catalin,

On Tue, 2012-01-31 at 14:40 +0000, Catalin Marinas wrote:
> On 19 January 2012 17:35, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Tue, 2012-01-03 at 16:50 +0000, Catalin Marinas wrote:
> >> On Fri, Dec 23, 2011 at 12:00:25PM +0000, Ian Campbell wrote:
> >> > At the moment we build the entire p2m before we ever load the VTTBR or
> >> > enable stage-2 translations in the HCR. Is that sufficient or do we also
> >> > need to flush something?
> >>
> >> If the model does not correctly implement cacheable page table walks for
> >> either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate
> >> that it should), the hypervisor would need to clean the D-cache to the
> >> point of unification (not necessary to go to the point of coherency)
> >> before any of the state 2 translation tables are used.
> >
> > I added a copy of Linux's v7_flush_dcache_all after building the p2m but
> > just before loading the VTTBR and it didn't help.
> >
> > I turned off cacheability in VTCR and that didn't help either.
> >
> > Then I turned off cacheability in the Linux TTBR{0,1} (by frobbing both
> > TTB_FLAGS_UP and TTB_FLAGS_SMP to use FOO_NC everywhere) and this _did_
> > make a difference -- the kernel now boots.
> >
> > I then tested just that change by itself and it seems to have done the
> > trick.
> >
> > Does that indicate a model bug?
> 
> It is possible. The scenario I'm thinking of is that with cacheable
> PTWs enabled in TTBR, the model wrongly decides to use the
> intermediate physical address (IPA) to look up the caches and gets the
> wrong information.
> 
> I'll take this to the model guys but most likely they'll ask for an
> image to load and just run. Could you provide such simple image
> (minimal filesystem)? I'm not familiar with Xen and building it.
> 
> (I'll first ask the model guys, maybe they can spot the error easily
> without additional debugging).

Did they conclude anything? Can I provide any more info?

Ian.

Oops in guest after ioremap() on ARMv7

[Catalin Marinas]

On Thu, Feb 16, 2012 at 05:01:42PM +0000, Ian Campbell wrote:
> On Tue, 2012-01-31 at 14:40 +0000, Catalin Marinas wrote:
> > On 19 January 2012 17:35, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > > On Tue, 2012-01-03 at 16:50 +0000, Catalin Marinas wrote:
> > >> On Fri, Dec 23, 2011 at 12:00:25PM +0000, Ian Campbell wrote:
> > >> > At the moment we build the entire p2m before we ever load the VTTBR or
> > >> > enable stage-2 translations in the HCR. Is that sufficient or do we also
> > >> > need to flush something?
> > >>
> > >> If the model does not correctly implement cacheable page table walks for
> > >> either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate
> > >> that it should), the hypervisor would need to clean the D-cache to the
> > >> point of unification (not necessary to go to the point of coherency)
> > >> before any of the state 2 translation tables are used.
> > >
> > > I added a copy of Linux's v7_flush_dcache_all after building the p2m but
> > > just before loading the VTTBR and it didn't help.
> > >
> > > I turned off cacheability in VTCR and that didn't help either.
> > >
> > > Then I turned off cacheability in the Linux TTBR{0,1} (by frobbing both
> > > TTB_FLAGS_UP and TTB_FLAGS_SMP to use FOO_NC everywhere) and this _did_
> > > make a difference -- the kernel now boots.
> > >
> > > I then tested just that change by itself and it seems to have done the
> > > trick.
> > >
> > > Does that indicate a model bug?
> > 
> > It is possible. The scenario I'm thinking of is that with cacheable
> > PTWs enabled in TTBR, the model wrongly decides to use the
> > intermediate physical address (IPA) to look up the caches and gets the
> > wrong information.
> > 
> > I'll take this to the model guys but most likely they'll ask for an
> > image to load and just run. Could you provide such simple image
> > (minimal filesystem)? I'm not familiar with Xen and building it.
> > 
> > (I'll first ask the model guys, maybe they can spot the error easily
> > without additional debugging).
> 
> Did they conclude anything? Can I provide any more info?

No, they didn't :(. They would like to run some image showing the
problem so they can trace the internal state of the model. Unfortunately
I didn't have time to look into this.

Is it possible for you to create an ELF image that can be loaded onto
the model with some instructions to show the failure?

Thanks.

-- 
Catalin

Oops in guest after ioremap() on ARMv7

[Ian Campbell]

On Thu, 2012-02-16 at 17:18 +0000, Catalin Marinas wrote:
> On Thu, Feb 16, 2012 at 05:01:42PM +0000, Ian Campbell wrote:
> > On Tue, 2012-01-31 at 14:40 +0000, Catalin Marinas wrote:
> > > On 19 January 2012 17:35, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > > > On Tue, 2012-01-03 at 16:50 +0000, Catalin Marinas wrote:
> > > >> On Fri, Dec 23, 2011 at 12:00:25PM +0000, Ian Campbell wrote:
> > > >> > At the moment we build the entire p2m before we ever load the VTTBR or
> > > >> > enable stage-2 translations in the HCR. Is that sufficient or do we also
> > > >> > need to flush something?
> > > >>
> > > >> If the model does not correctly implement cacheable page table walks for
> > > >> either stage 1 or stage 2 translation (though ID_MMFR3[23:20] indicate
> > > >> that it should), the hypervisor would need to clean the D-cache to the
> > > >> point of unification (not necessary to go to the point of coherency)
> > > >> before any of the state 2 translation tables are used.
> > > >
> > > > I added a copy of Linux's v7_flush_dcache_all after building the p2m but
> > > > just before loading the VTTBR and it didn't help.
> > > >
> > > > I turned off cacheability in VTCR and that didn't help either.
> > > >
> > > > Then I turned off cacheability in the Linux TTBR{0,1} (by frobbing both
> > > > TTB_FLAGS_UP and TTB_FLAGS_SMP to use FOO_NC everywhere) and this _did_
> > > > make a difference -- the kernel now boots.
> > > >
> > > > I then tested just that change by itself and it seems to have done the
> > > > trick.
> > > >
> > > > Does that indicate a model bug?
> > > 
> > > It is possible. The scenario I'm thinking of is that with cacheable
> > > PTWs enabled in TTBR, the model wrongly decides to use the
> > > intermediate physical address (IPA) to look up the caches and gets the
> > > wrong information.
> > > 
> > > I'll take this to the model guys but most likely they'll ask for an
> > > image to load and just run. Could you provide such simple image
> > > (minimal filesystem)? I'm not familiar with Xen and building it.
> > > 
> > > (I'll first ask the model guys, maybe they can spot the error easily
> > > without additional debugging).
> > 
> > Did they conclude anything? Can I provide any more info?
> 
> No, they didn't :(. They would like to run some image showing the
> problem so they can trace the internal state of the model. Unfortunately
> I didn't have time to look into this.
> 
> Is it possible for you to create an ELF image that can be loaded onto
> the model with some instructions to show the failure?

Yes I'll do that (tomorrow hopefully) and contact you off list with a
pointer to the binaries etc.

Thanks!

Ian.

Oops in guest after ioremap() on ARMv7

[Russell King - ARM Linux]

On Thu, Dec 22, 2011 at 06:13:56PM +0000, Catalin Marinas wrote:
> It could as well be a model bug but people are on holiday at the moment
> (and I'm off shortly as well, until 3rd of January). Could you try to
> disable the cacheability of the page table walks for both stage 1 (TTBRx
> with classic page tables or TTBCR with LPAE) and stage 2 (VTCR)? Since
> Linux does the correct cache flushing and I assume the hypervisor as
> well, this may work around possible model bug.

Note that Linux, through and through, expects that page table entries are
not negatively cached.  Non-present page table entries are expected to be
refetched without requiring any kind of TLB flushing at any point.

If later hardware is, then it's incompatible with how Linux works (and
with every other CPU which Linux supports.)  It'd mean a new raft of
endless problems which only affect ARM.  So, I'd recommend getting rid
of this feature in hardware, not only because of this reason, but also
because it's absolutely insane: negative caching probably does _more_
to hurt performance by taking up valuable positive TLB caching space.
Not only that, but non-present page table entries aren't used very often,
and when they are they're normally part of a slow path.

So I suspect that the amount of traction you'd get to solve such an issue
in mainline would be fairly minimal.

Oops in guest after ioremap() on ARMv7

[Catalin Marinas]

On 22 December 2011 19:29, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Thu, Dec 22, 2011 at 06:13:56PM +0000, Catalin Marinas wrote:
>> It could as well be a model bug but people are on holiday at the moment
>> (and I'm off shortly as well, until 3rd of January). Could you try to
>> disable the cacheability of the page table walks for both stage 1 (TTBRx
>> with classic page tables or TTBCR with LPAE) and stage 2 (VTCR)? Since
>> Linux does the correct cache flushing and I assume the hypervisor as
>> well, this may work around possible model bug.
>
> Note that Linux, through and through, expects that page table entries are
> not negatively cached. ?Non-present page table entries are expected to be
> refetched without requiring any kind of TLB flushing at any point.

The TLB on ARM does not do any negative caching. Full translations are
only stored if the page table walk succeeds. You may get some partial
caching (like level 1) if there is a valid entry in the page table but
that's not negative caching. For this scenario we have the
(pmd|pte)_free_tlb() implementation.

What caching attributes in TTBR/TTBCR/VTCR mean is looking up the page
table in the L1 cache so that we don't need to do extra cache
flushing.

-- 
Catalin