Re: Does HIGHMEM work on 32-bit MIPS ports?

Re: Does HIGHMEM work on 32-bit MIPS ports?

[David VomLehn]

 > I've written MIPS highmem support in late 2002 for a customer who
 > back then wasn't interested in being the first through the 64-bit
 > minefield. Which back then certainly was justified - but there are
 > now fairly stable 64-bit Linux kernels available so if you happen to
 > be running on 64-bit hardware don't even spend a nanosecond on
 > thinking about 32-bit highmem kernels.  Highmem fundamentally sucks
 > rocks through a straw.
 >
 > Coming back to your question.  Highmem was only ever tested to work
 > on SB1 and somewhat later PMC-Sierra RM9000 cores, both being 64-bit.
 > With the increasing maturity of 64-bit Linux interest in these went
 > away and as the result the highmem code started a slow bitrot -
 > unnoticed for many moons.

Hmm, this is not good. I've got a MIPS 24Kc processor with a very 
awkward memory layout. Any hints?

 >   Ralf
-- 
David VomLehn, dvomlehn@cisco.com
The opinions expressed herein are likely mine, but might not be my 
employer's...

Re: Does HIGHMEM work on 32-bit MIPS ports?

[Maciej W. Rozycki]

On Wed, 20 Feb 2008, David VomLehn wrote:

> Hmm, this is not good. I've got a MIPS 24Kc processor with a very awkward
> memory layout. Any hints?

 What does it mean "very awkward"?  What sort of problems do you have that 
you are trying to solve?

  Maciej

Re: Does HIGHMEM work on 32-bit MIPS ports?

[David VomLehn]

>> The function __flush_dcache_page (in arch/mips/mm/cache.c) was simply 
>> returning if the struct page* argument it was given indicated we had a 
>> page in high memory, so the dcache was never being flushed. This is an 
>> obvious Bad Thing.
> 
> Sort of.  It could be argued that the flushing of highmem pages should be
> done on kunmap but I haven't researched that into depth.

I find it hard to get my brain around caching, but my expectation is that you
would not normally need to flush the dcache because, with physical tagging, this
all happens automatically. The case where you do need to flush the cache is
where you may do DMA on the memory you mapped, which is only part of why you use
kmap_atomic. So, most of the flushes in kunmap_atomic would be unnecessary.
People expect to have to flush the dcache before doing DMA, so they'd call
flush_dcache_page/__flush_dcache_page.

>> We then have the following:
>>
>>    addr = (unsigned long) page_address(page);
>>    flush_data_cache_page(addr);
>>
>> Additional Modification #2: If the page is in high memory, it may not 
>> have a kernel mapping, in which case page_address() will return NULL. 
>> So, I've modified the code to only call flush_data_cache_page() if the
>> page_address() doesn't return NULL.
> 
> This assumes that kunmap and kunmap_atomic flush the cache.

I think we've already taken care of the kunmap_atomic case above. For the kunmap
case, if you call __flush_dcache_page and page_address returns NULL, then don't
you have to be working with a user-mapped page? If you do DMA from a user-mapped
page, then I don't know how you come up with a virtual address to use to flush
the cache.

> copy_user_highpage, copy_to_user_page and copy_from_user_page could use some review for correctness for the highmem case.

Thanks! I'll look at these, too.

>   Ralf

To me, this is one of the hairiest corners of the the kernel. I appreciate the
time you're taking to respond!

-- 
David VomLehn, dvomlehn@cisco.com
The opinions expressed herein are likely mine, but might not be my employer's...

Re: Does HIGHMEM work on 32-bit MIPS ports?

[David VomLehn]

We've made significant progress in getting HIGHMEM to work on our 24K 
processor, but things do not completely work yet. Since I don't yet have 
confidence that we know everything that's going on, I"m not ready to 
submit a full-blown patch, but here's what we've done so far. Please 
send comments/suggestions...

The function __flush_dcache_page (in arch/mips/mm/cache.c) was simply 
returning if the struct page* argument it was given indicated we had a 
page in high memory, so the dcache was never being flushed. This is an 
obvious Bad Thing.

Our first modification was to expand the check for high memory. If the 
page had a temporary mapping, i.e. it was mapped through kmap_atomic(), 
we call flush_data_cache_page(). We then immediately return:

    if (PageHighMem(page)) {
        addr = (unsigned long)kmap_atomic_to_vaddr(page);
        if (addr != 0) {
            flush_data_cache_page(addr);
        }
        return;
    }

(kmap_atomic_to_vaddr() returns the virtual address if the page is 
mapped with kmap_atomic(), otherwise it returns NULL). This change by 
itself is enough to be able to boot with NFS most of the time. I think 
it is not sufficient for permanently mapped kernel pages (those mapped 
with kmap_high()). So, I made two other modifications.

Additional Modification #1: To me, it looks like the return should be 
moved to right after the call to flush_data_cache_page() so that we only 
return immediately for temporary kernel mappings.

The next section of code, which I think already works correctly with 
high memory, is:

    if (mapping && !mapping_mapped(mapping)) {
        SetPageDcacheDirty(page);
        return;
    }

We then have the following:

    addr = (unsigned long) page_address(page);
    flush_data_cache_page(addr);

Additional Modification #2: If the page is in high memory, it may not 
have a kernel mapping, in which case page_address() will return NULL. 
So, I've modified the code to only call flush_data_cache_page() if the 
page_address() doesn't return NULL.

With the two additional modifications above, thing are still not 
completely reliable. So, two questions:

   1. Does what we've done so far make sense?
   2. Since the behavior is still somewhat flaky, I'm still missing
      something. Any suggestions?

-- 
David VomLehn, dvomlehn@cisco.com
The opinions expressed herein are likely mine, but might not be my employer's...

Re: Does HIGHMEM work on 32-bit MIPS ports?

[Ralf Baechle]

On Wed, Mar 05, 2008 at 02:55:48PM -0800, David VomLehn wrote:

> We've made significant progress in getting HIGHMEM to work on our 24K 
> processor, but things do not completely work yet. Since I don't yet have 
> confidence that we know everything that's going on, I"m not ready to submit 
> a full-blown patch, but here's what we've done so far. Please send 
> comments/suggestions...

Even a work in progress patch would be useful.

> The function __flush_dcache_page (in arch/mips/mm/cache.c) was simply 
> returning if the struct page* argument it was given indicated we had a page 
> in high memory, so the dcache was never being flushed. This is an obvious 
> Bad Thing.

Sort of.  It could be argued that the flushing of highmem pages should
be done on kunmap but I haven't researched that into depth.

> Our first modification was to expand the check for high memory. If the page 
> had a temporary mapping, i.e. it was mapped through kmap_atomic(), we call 
> flush_data_cache_page(). We then immediately return:
>
>    if (PageHighMem(page)) {
>        addr = (unsigned long)kmap_atomic_to_vaddr(page);
>        if (addr != 0) {
>            flush_data_cache_page(addr);
>        }
>        return;
>    }
>
> (kmap_atomic_to_vaddr() returns the virtual address if the page is mapped 
> with kmap_atomic(), otherwise it returns NULL). This change by itself is 
> enough to be able to boot with NFS most of the time. I think it is not 
> sufficient for permanently mapped kernel pages (those mapped with 
> kmap_high()). So, I made two other modifications.
>
> Additional Modification #1: To me, it looks like the return should be moved 
> to right after the call to flush_data_cache_page() so that we only return 
> immediately for temporary kernel mappings.
>
> The next section of code, which I think already works correctly with high 
> memory, is:
>
>    if (mapping && !mapping_mapped(mapping)) {
>        SetPageDcacheDirty(page);
>        return;
>    }
>
> We then have the following:
>
>    addr = (unsigned long) page_address(page);
>    flush_data_cache_page(addr);
>
> Additional Modification #2: If the page is in high memory, it may not have 
> a kernel mapping, in which case page_address() will return NULL. So, I've 
> modified the code to only call flush_data_cache_page() if the 
> page_address() doesn't return NULL.

This assumes that kunmap and kunmap_atomic flush the cache.

> With the two additional modifications above, thing are still not completely 
> reliable. So, two questions:
>
>   1. Does what we've done so far make sense?
>   2. Since the behavior is still somewhat flaky, I'm still missing
>      something. Any suggestions?

copy_user_highpage, copy_to_user_page and copy_from_user_page could use
some review for correctness for the highmem case.

  Ralf

Re: Does HIGHMEM work on 32-bit MIPS ports?

[David VomLehn]

> On Wed, 20 Feb 2008, David VomLehn wrote:
>
> > Hmm, this is not good. I've got a MIPS 24Kc processor with a very
> > awkward memory layout. Any hints?
>
>  What does it mean "very awkward"?  What sort of problems do you have 
> that you are trying to solve?
>
>   Maciej
Specifically, we have two banks of memory. The first starts at 
0x10000000, which is no big deal (other than wasting the page map 
entries for the first chunk of memory). The second starts at 0x60000000, so:

   1. We have to access it with high memory, and
   2. There is a huge section of unused page map entries.

As it turns out, it is starting to look like we're making progress with 
the problem with high memory: In __flush_dcache_page, nothing is done if 
PageHighMem() returns true. Not surprisingly, this leads to Bad 
Things(tm). What we are working through now is making sure that we 
understand exactly what *should* be happening.

-- 
David VomLehn, dvomlehn@cisco.com
The opinions expressed herein are likely mine, but might not be my employer's...

Does HIGHMEM work on 32-bit MIPS ports?

[David VomLehn]

As we continue to investigate using high memory on MIPS, we keep coming 
up with odd results. The basic mapping of high memory seems to be 
working correctly, and if we use an INITRAMFS root filesystem, things 
seem to work. Things also seem to work with an NFS root filesystem if we 
disable preemption, though we get someone squirrelly behavior in some 
minor ways. Has anyone else successfully been able to use high memory on 
a 32-bit MIPS Linux port?

Any feedback would be helpful.

-- 
David VomLehn, dvomlehn@cisco.com
The opinions expressed herein are likely mine, but might not be my employer's...

Re: Does HIGHMEM work on 32-bit MIPS ports?

[Ralf Baechle]

On Tue, Feb 19, 2008 at 08:09:45PM -0800, David VomLehn wrote:

> As we continue to investigate using high memory on MIPS, we keep coming up 
> with odd results. The basic mapping of high memory seems to be working 
> correctly, and if we use an INITRAMFS root filesystem, things seem to work. 
> Things also seem to work with an NFS root filesystem if we disable 
> preemption, though we get someone squirrelly behavior in some minor ways. 
> Has anyone else successfully been able to use high memory on a 32-bit MIPS 
> Linux port?

I've written MIPS highmem support in late 2002 for a customer who back
then wasn't interested in being the first through the 64-bit minefield.
Which back then certainly was justified - but there are now fairly
stable 64-bit Linux kernels available so if you happen to be running on
64-bit hardware don't even spend a nanosecond on thinking about 32-bit
highmem kernels.  Highmem fundamentally sucks rocks through a straw.

Coming back to your question.  Highmem was only ever tested to work on
SB1 and somewhat later PMC-Sierra RM9000 cores, both being 64-bit.  With
the increasing maturity of 64-bit Linux interest in these went away and
as the result the highmem code started a slow bitrot - unnoticed for many
moons.

  Ralf