Broadcom Swarm support

Broadcom Swarm support

[Aurelien Jarno]

Hi all,

I am still trying to get a Broadcom Swarm boot on a recent kernel. I
have made some progress, but I am now stuck on another problem.

I am using a lmo 2.6.30 kernel, using the defconfig configuration, with
minor tweaks to get the IDE and EXT3 support working.

The boot process works correctly until it reaches userland:

| hda: max request size: 128KiB
| hda: 78165360 sectors (40020 MB) w/2048KiB Cache, CHS=65535/16/63
| hda: cache flushes not supported
|  hda: hda1 hda2
| eth0 (sb1250-mac): not using net_device_ops yet
| sb1250-mac.0: registered as eth0
| eth0: enabling TCP rcv checksum
| eth0: SiByte Ethernet at 0x10064000, address: 00:02:4c:fe:51:e0
| eth1 (sb1250-mac): not using net_device_ops yet
| sb1250-mac.1: registered as eth1
| eth1: enabling TCP rcv checksum
| eth1: SiByte Ethernet at 0x10065000, address: 00:02:4c:fe:51:e1
| usbmon: debugfs is not available
| ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
| TCP cubic registered
| NET: Registered protocol family 17
| NET: Registered protocol family 15
| RPC: Registered udp transport module.
| RPC: Registered tcp transport module.
| kjournald starting.  Commit interval 5 seconds
| EXT3-fs: mounted filesystem with writeback data mode.
| VFS: Mounted root (ext3 filesystem) readonly on device 3:2.
| Freeing unused kernel memory: 184k freed
| Kernel panic - not syncing: Attempted to kill init!
| Rebooting in 5 seconds..Passing control back to CFE...

This is with /sbin/init, but I get the exact same problem with /bin/sh.
Also the file is found, as otherwise the message is different. The same
partition when used with an older kernel (2.6.18) works perfectly.

I have the problem with both big and little endian, and with and without
SMP support.

Does anyone is able to run a recent kernel on such a board ? If yes,
could you please share the configuration file? Thanks in advance.

Best regards,
Aurelien

-- 
Aurelien Jarno	                        GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net

RE: Broadcom Swarm support

[Kaz Kylheku]

Aurelien Jarno wrote:
> Hi all,
> 
> I am still trying to get a Broadcom Swarm boot on a recent kernel. I
> have made some progress, but I am now stuck on another problem.
> 
> I am using a lmo 2.6.30 kernel, using the defconfig 

[ snip ]

> | Kernel panic - not syncing: Attempted to kill init!
> | Rebooting in 5 seconds..Passing control back to CFE...

What kernel were you running prior to trying 30?

When I migrated from 2.6.17 to 2.6.26, on a Broadcom
1480 based board, I discovered that there is some kind
of instruction cache problem, which causes userland to
fetch garbage instead of code from its mmap-ped executables.
I could not get init to execute successfully.

Sorry, I can no longer remember whether this problem was
SMP specific or not (like what you're experiencing);
it might have been.

At some point in the kernel history, Ralfie decided that
the flush_icache_page function is unnecessary and
turned it into a MIPS-wide noop. But the SB1 core, which has
a VIVT instruction cache, it appears that there
is some kind of issue whereby when it
is handling a fault for a not-present virtual page,
it somehow ends up with bad data in the instruction
cache---perhaps an inconsistent state due to not having
been able to complete the fetch, but having initiated
a cache update on the expectation that the fetch
will complete. It seems that the the fault handler
is expected to do a flush.

Anyway, see if you can work this patch (based on 2.6.26)
into your kernel, and report whether it makes any difference.

Index: include/asm-mips/cacheflush.h
===================================================================
--- include/asm-mips/cacheflush.h	(revision 2677)
+++ include/asm-mips/cacheflush.h	(revision 2678)
@@ -37,6 +37,7 @@
 	unsigned long start, unsigned long end);
 extern void (*flush_cache_page)(struct vm_area_struct *vma, unsigned
long page, unsigned long pfn);
 extern void __flush_dcache_page(struct page *page);
+extern void __flush_icache_page(struct vm_area_struct *vma, struct page
*page);
 
 static inline void flush_dcache_page(struct page *page)
 {
@@ -57,11 +58,6 @@
 		__flush_anon_page(page, vmaddr);
 }
 
-static inline void flush_icache_page(struct vm_area_struct *vma,
-	struct page *page)
-{
-}
-
 extern void (*flush_icache_range)(unsigned long start, unsigned long
end);
 
 extern void (*__flush_cache_vmap)(void);
@@ -93,6 +89,13 @@
 extern void (*local_flush_data_cache_page)(void * addr);
 extern void (*flush_data_cache_page)(unsigned long addr);
 
+static inline void flush_icache_page(struct vm_area_struct *vma,
+	struct page *page)
+{
+        __flush_icache_page(vma, page);
+}
+
+
 /*
  * This flag is used to indicate that the page pointed to by a pte
  * is dirty and requires cleaning before returning it to the user.
Index: arch/mips/mm/cache.c
===================================================================
--- arch/mips/mm/cache.c	(revision 2677)
+++ arch/mips/mm/cache.c	(revision 2678)
@@ -93,6 +93,14 @@
 
 EXPORT_SYMBOL(__flush_dcache_page);
 
+void __flush_icache_page(struct vm_area_struct *vma, struct page *page)
+{
+	if (vma->vm_flags & VM_EXEC)
+		flush_icache_range((unsigned long) page_address(page),
PAGE_SIZE); 
+}
+
+EXPORT_SYMBOL(__flush_icache_page);
+
 void __flush_anon_page(struct page *page, unsigned long vmaddr)
 {
 	unsigned long addr = (unsigned long) page_address(page);

RE: Broadcom Swarm support

[Kaz Kylheku]

Aurelien Jarno wrote:
> Hi all,
> 
> I am still trying to get a Broadcom Swarm boot on a recent kernel. I
> have made some progress, but I am now stuck on another problem.
> 
> I am using a lmo 2.6.30 kernel, using the defconfig 

[ snip ]

> | Kernel panic - not syncing: Attempted to kill init!
> | Rebooting in 5 seconds..Passing control back to CFE...

What kernel were you running prior to trying 30?

When I migrated from 2.6.17 to 2.6.26, on a Broadcom
1480 based board, I discovered that there is some kind
of instruction cache problem, which causes userland to
fetch garbage instead of code from its mmap-ped executables.
I could not get init to execute successfully.

Sorry, I can no longer remember whether this problem was
SMP specific or not (like what you're experiencing);
it might have been.

At some point in the kernel history, Ralfie decided that
the flush_icache_page function is unnecessary and
turned it into a MIPS-wide noop. But the SB1 core, which has
a VIVT instruction cache, it appears that there
is some kind of issue whereby when it
is handling a fault for a not-present virtual page,
it somehow ends up with bad data in the instruction
cache---perhaps an inconsistent state due to not having
been able to complete the fetch, but having initiated
a cache update on the expectation that the fetch
will complete. It seems that the the fault handler
is expected to do a flush.

Anyway, see if you can work this patch (based on 2.6.26)
into your kernel, and report whether it makes any difference.

Index: include/asm-mips/cacheflush.h
===================================================================
--- include/asm-mips/cacheflush.h	(revision 2677)
+++ include/asm-mips/cacheflush.h	(revision 2678)
@@ -37,6 +37,7 @@
 	unsigned long start, unsigned long end);
 extern void (*flush_cache_page)(struct vm_area_struct *vma, unsigned
long page, unsigned long pfn);
 extern void __flush_dcache_page(struct page *page);
+extern void __flush_icache_page(struct vm_area_struct *vma, struct page
*page);
 
 static inline void flush_dcache_page(struct page *page)
 {
@@ -57,11 +58,6 @@
 		__flush_anon_page(page, vmaddr);
 }
 
-static inline void flush_icache_page(struct vm_area_struct *vma,
-	struct page *page)
-{
-}
-
 extern void (*flush_icache_range)(unsigned long start, unsigned long
end);
 
 extern void (*__flush_cache_vmap)(void);
@@ -93,6 +89,13 @@
 extern void (*local_flush_data_cache_page)(void * addr);
 extern void (*flush_data_cache_page)(unsigned long addr);
 
+static inline void flush_icache_page(struct vm_area_struct *vma,
+	struct page *page)
+{
+        __flush_icache_page(vma, page);
+}
+
+
 /*
  * This flag is used to indicate that the page pointed to by a pte
  * is dirty and requires cleaning before returning it to the user.
Index: arch/mips/mm/cache.c
===================================================================
--- arch/mips/mm/cache.c	(revision 2677)
+++ arch/mips/mm/cache.c	(revision 2678)
@@ -93,6 +93,14 @@
 
 EXPORT_SYMBOL(__flush_dcache_page);
 
+void __flush_icache_page(struct vm_area_struct *vma, struct page *page)
+{
+	if (vma->vm_flags & VM_EXEC)
+		flush_icache_range((unsigned long) page_address(page),
PAGE_SIZE); 
+}
+
+EXPORT_SYMBOL(__flush_icache_page);
+
 void __flush_anon_page(struct page *page, unsigned long vmaddr)
 {
 	unsigned long addr = (unsigned long) page_address(page);

icache flushing (Re: Broadcom Swarm support)

[Atsushi Nemoto]

On Wed, 24 Jun 2009 15:18:24 -0700, "Kaz Kylheku" <KKylheku@zeugmasystems.com> wrote:
> At some point in the kernel history, Ralfie decided that
> the flush_icache_page function is unnecessary and
> turned it into a MIPS-wide noop. But the SB1 core, which has
> a VIVT instruction cache, it appears that there
> is some kind of issue whereby when it
> is handling a fault for a not-present virtual page,
> it somehow ends up with bad data in the instruction
> cache---perhaps an inconsistent state due to not having
> been able to complete the fetch, but having initiated
> a cache update on the expectation that the fetch
> will complete. It seems that the the fault handler
> is expected to do a flush.

Looking at current code, I also have some questions aboud icache
flushing.

* flush_cache_mm does not flush icache.  Is it OK?

* flush_cache_{vmap,vunmap} does not flush icache.  When icache used
  by modules flushed after unloading?

* __update_cache, copy_user_highpage does not flush icache even if
  !cpu_has_ic_fills_f_dc.  Is it OK?

* free_initmem does not flush icache.  When these init pages are
  reused, how corresponding icache will be flushed?

I suppose flushing icache in flush_icache_page() will hide real bugs
somewhere else...

---
Atsushi Nemoto

Re: Broadcom Swarm support

[Aurelien Jarno]

On Wed, Jun 24, 2009 at 03:18:24PM -0700, Kaz Kylheku wrote:
> Aurelien Jarno wrote:
> > Hi all,
> > 
> > I am still trying to get a Broadcom Swarm boot on a recent kernel. I
> > have made some progress, but I am now stuck on another problem.
> > 
> > I am using a lmo 2.6.30 kernel, using the defconfig 
> 
> [ snip ]
> 
> > | Kernel panic - not syncing: Attempted to kill init!
> > | Rebooting in 5 seconds..Passing control back to CFE...
> 
> What kernel were you running prior to trying 30?

I am currently stuck with 2.6.18. I have tested a lot of kernels between
2.6.18 and 2.6.30, but they all have a different problem. Kernels 2.6.26
to 2.6.30 suffer from a similar problem, though from some versions it
hangs instead of panicking.

> When I migrated from 2.6.17 to 2.6.26, on a Broadcom
> 1480 based board, I discovered that there is some kind
> of instruction cache problem, which causes userland to
> fetch garbage instead of code from its mmap-ped executables.
> I could not get init to execute successfully.
> 
> Sorry, I can no longer remember whether this problem was
> SMP specific or not (like what you're experiencing);
> it might have been.
> 
> At some point in the kernel history, Ralfie decided that
> the flush_icache_page function is unnecessary and
> turned it into a MIPS-wide noop. But the SB1 core, which has
> a VIVT instruction cache, it appears that there
> is some kind of issue whereby when it
> is handling a fault for a not-present virtual page,
> it somehow ends up with bad data in the instruction
> cache---perhaps an inconsistent state due to not having
> been able to complete the fetch, but having initiated
> a cache update on the expectation that the fetch
> will complete. It seems that the the fault handler
> is expected to do a flush.
> 
> Anyway, see if you can work this patch (based on 2.6.26)
> into your kernel, and report whether it makes any difference.

I have applied the patch on 2.6.30 after a few changes, and it works!!!

Thanks a lot!

> Index: include/asm-mips/cacheflush.h
> ===================================================================
> --- include/asm-mips/cacheflush.h	(revision 2677)
> +++ include/asm-mips/cacheflush.h	(revision 2678)
> @@ -37,6 +37,7 @@
>  	unsigned long start, unsigned long end);
>  extern void (*flush_cache_page)(struct vm_area_struct *vma, unsigned
> long page, unsigned long pfn);
>  extern void __flush_dcache_page(struct page *page);
> +extern void __flush_icache_page(struct vm_area_struct *vma, struct page
> *page);
>  
>  static inline void flush_dcache_page(struct page *page)
>  {
> @@ -57,11 +58,6 @@
>  		__flush_anon_page(page, vmaddr);
>  }
>  
> -static inline void flush_icache_page(struct vm_area_struct *vma,
> -	struct page *page)
> -{
> -}
> -
>  extern void (*flush_icache_range)(unsigned long start, unsigned long
> end);
>  
>  extern void (*__flush_cache_vmap)(void);
> @@ -93,6 +89,13 @@
>  extern void (*local_flush_data_cache_page)(void * addr);
>  extern void (*flush_data_cache_page)(unsigned long addr);
>  
> +static inline void flush_icache_page(struct vm_area_struct *vma,
> +	struct page *page)
> +{
> +        __flush_icache_page(vma, page);
> +}
> +
> +
>  /*
>   * This flag is used to indicate that the page pointed to by a pte
>   * is dirty and requires cleaning before returning it to the user.
> Index: arch/mips/mm/cache.c
> ===================================================================
> --- arch/mips/mm/cache.c	(revision 2677)
> +++ arch/mips/mm/cache.c	(revision 2678)
> @@ -93,6 +93,14 @@
>  
>  EXPORT_SYMBOL(__flush_dcache_page);
>  
> +void __flush_icache_page(struct vm_area_struct *vma, struct page *page)
> +{
> +	if (vma->vm_flags & VM_EXEC)
> +		flush_icache_range((unsigned long) page_address(page),
> PAGE_SIZE); 
> +}
> +
> +EXPORT_SYMBOL(__flush_icache_page);
> +
>  void __flush_anon_page(struct page *page, unsigned long vmaddr)
>  {
>  	unsigned long addr = (unsigned long) page_address(page);
> 
> 

-- 
Aurelien Jarno	                        GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net

Re: Broadcom Swarm support

[Ralf Baechle]

On Wed, Jun 24, 2009 at 03:18:24PM -0700, Kaz Kylheku wrote:
> From: Kaz Kylheku <KKylheku@zeugmasystems.com>
> Date: Wed, 24 Jun 2009 15:18:24 -0700
> To: Aurelien Jarno <aurelien@aurel32.net>, linux-mips@linux-mips.org
> Subject: RE: Broadcom Swarm support
> Content-Type: text/plain;
> 	charset="us-ascii"
> 
> Aurelien Jarno wrote:
> > Hi all,
> > 
> > I am still trying to get a Broadcom Swarm boot on a recent kernel. I
> > have made some progress, but I am now stuck on another problem.
> > 
> > I am using a lmo 2.6.30 kernel, using the defconfig 
> 
> [ snip ]
> 
> > | Kernel panic - not syncing: Attempted to kill init!
> > | Rebooting in 5 seconds..Passing control back to CFE...
> 
> What kernel were you running prior to trying 30?
> 
> When I migrated from 2.6.17 to 2.6.26, on a Broadcom
> 1480 based board, I discovered that there is some kind
> of instruction cache problem, which causes userland to
> fetch garbage instead of code from its mmap-ped executables.
> I could not get init to execute successfully.
> 
> Sorry, I can no longer remember whether this problem was
> SMP specific or not (like what you're experiencing);
> it might have been.
> 
> At some point in the kernel history, Ralfie decided that
> the flush_icache_page function is unnecessary and
> turned it into a MIPS-wide noop. But the SB1 core, which has
> a VIVT instruction cache, it appears that there
> is some kind of issue whereby when it
> is handling a fault for a not-present virtual page,
> it somehow ends up with bad data in the instruction
> cache---perhaps an inconsistent state due to not having
> been able to complete the fetch, but having initiated
> a cache update on the expectation that the fetch
> will complete. It seems that the the fault handler
> is expected to do a flush.
> 
> Anyway, see if you can work this patch (based on 2.6.26)
> into your kernel, and report whether it makes any difference.

The functionality of flush_icache_page() is being handled update_mmu_cache.

The SB1 isn't the only MIPS CPU featuring VIVT caches - there are also
the MIPS 20K and 25K CPUs and the virtually identical SB1A core.  I've
tested Linux on 20K and 25K just days ago and I get months of uptime on my
BCM1480 box.

What type of boot devices are you using, is it the onboard IDE controller
by any chance?

  Ralf

Re: Broadcom Swarm support

[Aurelien Jarno]

On Sat, Jun 27, 2009 at 12:24:32AM +0100, Ralf Baechle wrote:
> On Wed, Jun 24, 2009 at 03:18:24PM -0700, Kaz Kylheku wrote:
> > From: Kaz Kylheku <KKylheku@zeugmasystems.com>
> > Date: Wed, 24 Jun 2009 15:18:24 -0700
> > To: Aurelien Jarno <aurelien@aurel32.net>, linux-mips@linux-mips.org
> > Subject: RE: Broadcom Swarm support
> > Content-Type: text/plain;
> > 	charset="us-ascii"
> > 
> > Aurelien Jarno wrote:
> > > Hi all,
> > > 
> > > I am still trying to get a Broadcom Swarm boot on a recent kernel. I
> > > have made some progress, but I am now stuck on another problem.
> > > 
> > > I am using a lmo 2.6.30 kernel, using the defconfig 
> > 
> > [ snip ]
> > 
> > > | Kernel panic - not syncing: Attempted to kill init!
> > > | Rebooting in 5 seconds..Passing control back to CFE...
> > 
> > What kernel were you running prior to trying 30?
> > 
> > When I migrated from 2.6.17 to 2.6.26, on a Broadcom
> > 1480 based board, I discovered that there is some kind
> > of instruction cache problem, which causes userland to
> > fetch garbage instead of code from its mmap-ped executables.
> > I could not get init to execute successfully.
> > 
> > Sorry, I can no longer remember whether this problem was
> > SMP specific or not (like what you're experiencing);
> > it might have been.
> > 
> > At some point in the kernel history, Ralfie decided that
> > the flush_icache_page function is unnecessary and
> > turned it into a MIPS-wide noop. But the SB1 core, which has
> > a VIVT instruction cache, it appears that there
> > is some kind of issue whereby when it
> > is handling a fault for a not-present virtual page,
> > it somehow ends up with bad data in the instruction
> > cache---perhaps an inconsistent state due to not having
> > been able to complete the fetch, but having initiated
> > a cache update on the expectation that the fetch
> > will complete. It seems that the the fault handler
> > is expected to do a flush.
> > 
> > Anyway, see if you can work this patch (based on 2.6.26)
> > into your kernel, and report whether it makes any difference.
> 
> The functionality of flush_icache_page() is being handled update_mmu_cache.
> 
> The SB1 isn't the only MIPS CPU featuring VIVT caches - there are also
> the MIPS 20K and 25K CPUs and the virtually identical SB1A core.  I've
> tested Linux on 20K and 25K just days ago and I get months of uptime on my
> BCM1480 box.
> 
> What type of boot devices are you using, is it the onboard IDE controller
> by any chance?
> 

Yes, booting on the IDE controller.


-- 
Aurelien Jarno	                        GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net

Re: Broadcom Swarm support

[Ralf Baechle]

On Sat, Jun 27, 2009 at 07:10:26AM +0200, Aurelien Jarno wrote:

> Yes, booting on the IDE controller.

Chances are it's a bug in the PIO IDE driver or it's interaction with
update_mmu_cache().  If I'm right you should not see the issue if you
boot of another block device with DMA like a PCI PATA/SATA card.

Which generally is a sane thing to do - the onboard controller is a
quick hack to demonstrate the capabilities of the BCM1250's GPIO features;
in practical terms it totally sucks but I SATA card solves that.  If
you do that, get a 64-bit card.  32-bit DMA PCI cards have other issues
in Sibyte systems.

  Ralf

Re: Broadcom Swarm support

[Atsushi Nemoto]

On Sat, 27 Jun 2009 00:24:32 +0100, Ralf Baechle <ralf@linux-mips.org> wrote:
> > At some point in the kernel history, Ralfie decided that
> > the flush_icache_page function is unnecessary and
> > turned it into a MIPS-wide noop. But the SB1 core, which has
> > a VIVT instruction cache, it appears that there
> > is some kind of issue whereby when it
> > is handling a fault for a not-present virtual page,
> > it somehow ends up with bad data in the instruction
> > cache---perhaps an inconsistent state due to not having
> > been able to complete the fetch, but having initiated
> > a cache update on the expectation that the fetch
> > will complete. It seems that the the fault handler
> > is expected to do a flush.
> > 
> > Anyway, see if you can work this patch (based on 2.6.26)
> > into your kernel, and report whether it makes any difference.
> 
> The functionality of flush_icache_page() is being handled update_mmu_cache.

A bit off-topic question.  The update_mmu_cache (or __update_cache)
itself does not flush icache.  When icache is invalidated (especially
VIPT case) ?

---
Atsushi Nemoto

Re: Broadcom Swarm support

[Ralf Baechle]

On Sat, Jun 27, 2009 at 10:59:33PM +0900, Atsushi Nemoto wrote:

> A bit off-topic question.  The update_mmu_cache (or __update_cache)
> itself does not flush icache.  When icache is invalidated (especially
> VIPT case) ?

Not off-topic at all in this thread.

The I-cache for page just being loaded is clean so no flushing needed.  It
is clean because when the page has been unmapped it was flushed or because
the CPU switched to a fresh ASID.

The reason for this bug is that when data is being shoveled around by the
processor (as opposed to DMA) as on PIO block devices it'll end up sitting
in the D-cache so I-cache refills will grab stale data from S-cache or
memory.

  Ralf

Re: Broadcom Swarm support

[Atsushi Nemoto]

On Sat, 27 Jun 2009 16:48:11 +0100, Ralf Baechle <ralf@linux-mips.org> wrote:
> On Sat, Jun 27, 2009 at 10:59:33PM +0900, Atsushi Nemoto wrote:
> 
> > A bit off-topic question.  The update_mmu_cache (or __update_cache)
> > itself does not flush icache.  When icache is invalidated (especially
> > VIPT case) ?
> 
> Not off-topic at all in this thread.
> 
> The I-cache for page just being loaded is clean so no flushing needed.  It
> is clean because when the page has been unmapped it was flushed or because
> the CPU switched to a fresh ASID.

Then, flush_cache_range or flush_cache_page should be called then the
page was unmmapped, right?  How about flush_cache_mm?  It does not
flush icache currently.

And how about kernel __init code pages?  These pages are just freed on
free_initmem.  Also how about code pages used by a module which is to
be unloaded from kernel?

> The reason for this bug is that when data is being shoveled around by the
> processor (as opposed to DMA) as on PIO block devices it'll end up sitting
> in the D-cache so I-cache refills will grab stale data from S-cache or
> memory.

Yes, I suppose so on this swarm case, but I'm just thinking of other
case breaking icache coherency...

---
Atsushi Nemoto

Re: Broadcom Swarm support

[Atsushi Nemoto]

On Sun, 28 Jun 2009 01:09:06 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> wrote:
> Then, flush_cache_range or flush_cache_page should be called then the
                                                               ~~~~ when
> page was unmmapped, right?  How about flush_cache_mm?  It does not
> flush icache currently.

Oops, excuse me for my poor writing.

---
Atsushi Nemoto

Re: Broadcom Swarm support

[Ralf Baechle]

On Sun, Jun 28, 2009 at 01:09:06AM +0900, Atsushi Nemoto wrote:

> > The I-cache for page just being loaded is clean so no flushing needed.  It
> > is clean because when the page has been unmapped it was flushed or because
> > the CPU switched to a fresh ASID.
> 
> Then, flush_cache_range or flush_cache_page should be called then the
> page was unmmapped, right?  How about flush_cache_mm?  It does not
> flush icache currently.

If that is being called then we're either about to terminate a process or
to exec a new process.  In either case flush_tlb_cache (on VIVT I-cache)
will drop the tlb context which effectively is a full I-cache flush.

> And how about kernel __init code pages?  These pages are just freed on
> free_initmem.  Also how about code pages used by a module which is to
> be unloaded from kernel?

Init code pages won't be used to store code that will be executed at
KSEG0 or XKPHYS addresses so I-cache coherency is not of interest.

For modules the I-cache is being flushed on loading of the module, see
calls to flush_icache_range() in kernel/module.c so I-cache coherency is
not of concern on module unload.

> > The reason for this bug is that when data is being shoveled around by the
> > processor (as opposed to DMA) as on PIO block devices it'll end up sitting
> > in the D-cache so I-cache refills will grab stale data from S-cache or
> > memory.
> 
> Yes, I suppose so on this swarm case, but I'm just thinking of other
> case breaking icache coherency...

Be careful with that - you might succeed ;-)

  Ralf

Re: Broadcom Swarm support

[Atsushi Nemoto]

On Mon, 29 Jun 2009 20:08:10 +0100, Ralf Baechle <ralf@linux-mips.org> wrote:
> > Then, flush_cache_range or flush_cache_page should be called then the
> > page was unmmapped, right?  How about flush_cache_mm?  It does not
> > flush icache currently.
> 
> If that is being called then we're either about to terminate a process or
> to exec a new process.  In either case flush_tlb_cache (on VIVT I-cache)
> will drop the tlb context which effectively is a full I-cache flush.

Thanks, I see.  Then, how about VIPT I-cache case?

> > And how about kernel __init code pages?  These pages are just freed on
> > free_initmem.  Also how about code pages used by a module which is to
> > be unloaded from kernel?
> 
> Init code pages won't be used to store code that will be executed at
> KSEG0 or XKPHYS addresses so I-cache coherency is not of interest.
> 
> For modules the I-cache is being flushed on loading of the module, see
> calls to flush_icache_range() in kernel/module.c so I-cache coherency is
> not of concern on module unload.

Same here.  A physical page used to __init code might be reused for
user code page, so explicit flush (invalide) is required for VIPT
case, no?

---
Atsushi Nemoto

Re: Broadcom Swarm support

[Atsushi Nemoto]

On Tue, 30 Jun 2009 23:00:40 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> wrote:
> > > And how about kernel __init code pages?  These pages are just freed on
> > > free_initmem.  Also how about code pages used by a module which is to
> > > be unloaded from kernel?
> > 
> > Init code pages won't be used to store code that will be executed at
> > KSEG0 or XKPHYS addresses so I-cache coherency is not of interest.
> > 
> > For modules the I-cache is being flushed on loading of the module, see
> > calls to flush_icache_range() in kernel/module.c so I-cache coherency is
> > not of concern on module unload.
> 
> Same here.  A physical page used to __init code might be reused for
> user code page, so explicit flush (invalide) is required for VIPT
> case, no?

I tracked some icache flushes after free_initmem() on a VIPT CPU.

* free_initmem()
* r4k_flush_cache_range(start=0x7fff7000, end=0x7fff8000)
	backtraces:
	#0  0x801263c0 in r4k_flush_cache_range ()
	#1  0x801bc640 in move_page_tables ()
	#2  0x801d317c in setup_arg_pages ()
	#3  0x80210af8 in load_elf_binary ()
	#4  0x801d2200 in search_binary_handler ()
	#5  0x801d3640 in do_execve ()
	#6  0x80119d58 in sys_execve ()
	#7  0x801022d0 in handle_sys ()
	#8  0x80111018 in init_post ()
* r4k_flush_cache_range(start=0x2aac9000, end=0x2aada000)
	backtraces:
	#0  0x801263c0 in r4k_flush_cache_range ()
	#1  0x801b4abc in unmap_vmas ()
	#2  0x801b9798 in unmap_region ()
	#3  0x801bac44 in do_munmap ()
	#4  0x80210538 in elf_map ()
	#5  0x80211054 in load_elf_binary ()
	#6  0x801d2200 in search_binary_handler ()
	#7  0x801d3640 in do_execve ()
	#8  0x80119d58 in sys_execve ()
	#9  0x801022d0 in handle_sys ()
	#10 0x80111018 in init_post ()
* some page faults from outside [0x2aac9000, 0x2aada000] range
* r4k_flush_cache_range(start=0x2ab19000, end=0x2ab29000)
	backtraces:
	#0  0x801263c0 in r4k_flush_cache_range ()
	#1  0x801bc178 in mprotect_fixup ()
	#2  0x801bc560 in sys_mprotect ()
	#3  0x801022d0 in handle_sys ()

The first icache flush is for stack (arg pages).  The second one is
for unmap holes in ELF image (I do not understand details...).  Anyway
both flushes did not intend to flush text pages for the first
userspace program.

Since current implementation of r4k_flush_cache_range flushes all
icache regardless of its range arguments, all icaches used for __init
pages are flushed effectively, but it seems just a sort of luck for
me.

Also flusing icache in flush_cache_range is relatively young code ---
the commit 2eaa7e ("Handle I-cache coherency in flush_cache_range()")
on Feb 2008.  I'm wondering how icache were flushed before that
commit...

---
Atsushi Nemoto

Re: Broadcom Swarm support

[Ralf Baechle]

On Wed, Jun 24, 2009 at 03:18:24PM -0700, Kaz Kylheku wrote:

> +void __flush_icache_page(struct vm_area_struct *vma, struct page *page)
> +{
> +	if (vma->vm_flags & VM_EXEC)
> +		flush_icache_range((unsigned long) page_address(page),
> PAGE_SIZE); 
> +}

Flush_icache_range takes two arguments, start and end address.  Both
addresses are the virtual addresses at which the code will run.  Iow both
arguments are wrong.  The result is that for start address values normally
passed to flush_icache_range it will optimize the flush into a full cache
flush of I-cache and D-cache - iow you're lucky.

  Ralf

RE: Broadcom Swarm support

[Kaz Kylheku]

Ralf Baechle wrote, on June 27, 2009 8:49 AM
> On Wed, Jun 24, 2009 at 03:18:24PM -0700, Kaz Kylheku wrote:
> 
> > +void __flush_icache_page(struct vm_area_struct *vma, 
> struct page *page)
> > +{
> > +	if (vma->vm_flags & VM_EXEC)
> > +		flush_icache_range((unsigned long) page_address(page),
> > PAGE_SIZE); 
> > +}
> 
> Flush_icache_range takes two arguments, start and end address.  Both
> addresses are the virtual addresses at which the code will 
> run.

Thanks for pointing that out. After wiping some egg of my face,
I'm now testing this change:

 void __flush_icache_page(struct vm_area_struct *vma, struct page *page)
 {
        if (vma->vm_flags & VM_EXEC)
-               flush_icache_range((unsigned long) page_address(page),
PAGE_SIZE);
+               flush_icache_range(vma->vm_start, vma->vm_end);
 }

I see a small performance improvement, suggesting that it's flushing
cache lines more precisely.

But I am now more confused, because it turns out that my kernel still
runs
fine now if I turn the function into a noop!

We must have fixed something since the time I was getting this kernel
running. Or it could be that I'm using a CF from a different
manufacturer.

Still, there is the observation that this flush_icache_page restoration
also gets Aurelien's Swarm up and running.

RE: Broadcom Swarm support

[Kaz Kylheku]

Ralf Baechle wrote, on June 27, 2009 8:49 AM
> On Wed, Jun 24, 2009 at 03:18:24PM -0700, Kaz Kylheku wrote:
> 
> > +void __flush_icache_page(struct vm_area_struct *vma, 
> struct page *page)
> > +{
> > +	if (vma->vm_flags & VM_EXEC)
> > +		flush_icache_range((unsigned long) page_address(page),
> > PAGE_SIZE); 
> > +}
> 
> Flush_icache_range takes two arguments, start and end address.  Both
> addresses are the virtual addresses at which the code will 
> run.

Thanks for pointing that out. After wiping some egg of my face,
I'm now testing this change:

 void __flush_icache_page(struct vm_area_struct *vma, struct page *page)
 {
        if (vma->vm_flags & VM_EXEC)
-               flush_icache_range((unsigned long) page_address(page),
PAGE_SIZE);
+               flush_icache_range(vma->vm_start, vma->vm_end);
 }

I see a small performance improvement, suggesting that it's flushing
cache lines more precisely.

But I am now more confused, because it turns out that my kernel still
runs
fine now if I turn the function into a noop!

We must have fixed something since the time I was getting this kernel
running. Or it could be that I'm using a CF from a different
manufacturer.

Still, there is the observation that this flush_icache_page restoration
also gets Aurelien's Swarm up and running.