[U-Boot] i.MX51: FEC: Cache coherency problem?

[David Jander <david.jander@protonic.nl>]

On Mon, 18 Jul 2011 21:46:28 +0530
Aneesh V <aneesh@ti.com> wrote:

> Hi David,
> 
> On Monday 18 July 2011 08:48 PM, David Jander wrote:
> >
> > Hi all,
> >
> > I am busy debugging a problem with the i.MX51 FEC ethernet driver, that
> > stopped working after upgrading u-boot. Before the upgrade I used
> > v2010.06-rc3, which worked fine.
> > I gave up on trying to find the difference beween the old version and this
> > one that broke it.... due to BSP issues, git bisecting seems a monumental
> > task I am not considering yet.
> > The funny part is that it seems to work fine if I disable data-caches!
> > With data caches enabled, it hangs in the following while loop in
> > fec_send(), at line 592:
> >
> > ...
> > 584         /*
> > 585          * Enable SmartDMA transmit task
> > 586          */
> > 587         fec_tx_task_enable(fec);
> > 588
> > 589         /*
> > 590          * wait until frame is sent .
> > 591          */
> > 592         while (readw(&fec->tbd_base[fec->tbd_index].status)&
> > FEC_TBD_READY) { 593                 udelay(1);
> > 594         }
> > ...
> >
> > If I change this code in the following way, the while loop exits
> > successfully:
> >
> > ...
> > 584         /*
> > 585          * Enable SmartDMA transmit task
> > 586          */
> > 587         flush_cache(&fec->tbd_base[fec->tbd_index], 4);
> > 588         fec_tx_task_enable(fec);
> > 589         flash_dcache_all();
> > 590
> > 591         /*
> > 592          * wait until frame is sent .
> > 593          */
> > 594         while (readw(&fec->tbd_base[fec->tbd_index].status)&
> > FEC_TBD_READY) { 595                 udelay(1);
> > 596         }
> > ...
> >
> > Notice the cache flush calls at line 587 and 589. With these, sending
> > succeeds. The driver still hangs in receiving afterwards, but at least this
> > part seems to work. If I remove either of the two added lines, it stops
> > working again.
> >
> > What is going on here? Why did this work with caches enabled before??
> 
> Are you sure caches were enabled before? Until recently absence of
> CONFIG_SYS_DCACHE_OFF did not mean that your D-cache is enabled because
> you also had to call the function dcache_enable(). Some platforms were
> calling this from board file or some drivers but not all.

We did. I am always calling dcache_enable(), icache_enable() and
l2_cache_enable() from board code. The latter function I implemented myself,
because nor u-boot nor the kernel does otherwise activate l2-cache, which has a
big performance impact.

Just in case, at the moment I am testing with only the L1 caches enabled.

> The following patch changed this for ARM:
> 
> commit c2dd0d45540397704de9b13287417d21049d34c6
> armv7: integrate cache maintenance support
> 
> In this patch I added a call to dcache_enable() at the beginning of
> board_init_r() for ARM(i.e. as soon as relocation is over). As a result
> D-cache will be enabled for all ARM platforms now unless
> CONFIG_SYS_DCACHE_OFF is set.

Yes, I noticed that.

> Looks like this is a real coherency issue that is brought out because
> d-cache is really enabled for you now.

AFAICS, it was always enabled for our board. At least I did always call
i/dcache_enable() in board setup code.... was it broken before? Are the L1
caches enabled by (mainline) linux code?

Best regards,

-- 
David Jander
Protonic Holland.