arm64: Question about barriers with the mmu off

arm64: Question about barriers with the mmu off

[Wonhyuk Yang]

Hi, I have a question about dmb barriers in arm64's head.S.
In the head.S, I could see the pattern below several times.

str w0, [x1]
dmb sys
dc ivac, x1   // Invalidate potentially stale cache line

I found that,
Commit(fix cache flushing and barriers in set_cpu_boot_mode_flag)
explained the code.

> This patch reworks the broken flushing code so that we:
>
> (1) Use a DMB to order the strongly-ordered write of the cacheline
> against the subsequent cache-maintenance operation (by-VA
> operations only hazard against normal, cacheable accesses).
>
> (2) Use a single dc ivac instruction to invalidate any clean lines
> containing a stale copy of the line after it has been updated.
> Use a DMB to order the strongly-> ordered write of the cacheline

But I can't  understand why the store operation should precede the
dc operation.

Is there any problem, if the dc operation precedes the store operation?

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: arm64: Question about barriers with the mmu off

[Valdis Klētnieks]

[-- Attachment #1.1: Type: text/plain, Size: 520 bytes --]

On Mon, 16 Nov 2020 20:58:52 +0900, Wonhyuk Yang said:

> str w0, [x1]
So we dirtied the cache line.
> dmb sys
> dc ivac, x1   // Invalidate potentially stale cache line

So we invalidate it.

> Is there any problem, if the dc operation precedes the store operation?

If you swap them, you get...

dc ivac,x1   // invalidate a cache line that's probably OK
str w0,[x1   // and now we do a store that leaves a possibly stale cache line

In other words, if you swap them you may leave an un-invalidated
stale cache line.


[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

arm64: Question about barriers with the mmu off

[Wonhyuk Yang]

[-- Attachment #1.1: Type: text/plain, Size: 442 bytes --]

On Tue, Nov 17, 2020 at 11:14 AM Valdis Klētnieks <valdis.kletnieks@vt.edu>
wrote:

> If you swap them, you get...
>
> dc ivac,x1   // invalidate a cache line that's probably OK
> str w0,[x1   // and now we do a store that leaves a possibly stale cache
line
>
> In other words, if you swap them you may leave an un-invalidated
> stale cache line.

You mean, even if STCLR_EL1.{C, M} is cleared, store doesn't bypass the
cache?

[-- Attachment #1.2: Type: text/html, Size: 581 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: arm64: Question about barriers with the mmu off

[Valdis Klētnieks]

[-- Attachment #1.1: Type: text/plain, Size: 655 bytes --]

On Tue, 17 Nov 2020 12:00:32 +0900, Wonhyuk Yang said:
> > If you swap them, you get...
> >
> > dc ivac,x1   // invalidate a cache line that's probably OK
> > str w0,[x1   // and now we do a store that leaves a possibly stale cache line
> >
> > In other words, if you swap them you may leave an un-invalidated
> > stale cache line.
>
> You mean, even if STCLR_EL1.{C, M} is cleared, store doesn't bypass the
> cache?

That's the problem.  The store bypasses the cache line, and the next reference
that uses the cache can get stale data. So you have to flush the cache line
so the next reference has to refresh the cache on the memory read.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: arm64: Question about barriers with the mmu off

[Wonhyuk Yang]

On Tue, Nov 17, 2020 at 12:45 PM Valdis Klētnieks
<valdis.kletnieks@vt.edu> wrote:
>
> > > If you swap them, you get...
> > >
> > > dc ivac,x1   // invalidate a cache line that's probably OK
> > > str w0,[x1   // and now we do a store that leaves a possibly stale cache line
> > >
> > > In other words, if you swap them you may leave an un-invalidated
> > > stale cache line.
> >
> > You mean, even if STCLR_EL1.{C, M} is cleared, store doesn't bypass the
> > cache?
>
> That's the problem.  The store bypasses the cache line, and the next reference
> that uses the cache can get stale data. So you have to flush the cache line
> so the next reference has to refresh the cache on the memory read.

Yes, I understood that I have to flush the cache before the cacheable
read(mmu on).
But I'm not fully understand your explanation below.

> > > dc ivac,x1   // invalidate a cache line that's probably OK
> > > str w0,[x1   // and now we do a store that leaves a possibly stale cache line

Could you explain me why the store still leaves stale cache?
We invalidated the cacheline and store will not make footprint in the cache.

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: arm64: Question about barriers with the mmu off

[Valdis Klētnieks]

[-- Attachment #1.1: Type: text/plain, Size: 571 bytes --]

On Tue, 17 Nov 2020 14:08:02 +0900, Wonhyuk Yang said:

> > > > dc ivac,x1   // invalidate a cache line that's probably OK
> > > > str w0,[x1   // and now we do a store that leaves a possibly stale cache line

> Could you explain me why the store still leaves stale cache?
> We invalidated the cacheline and store will not make footprint in the cache.

There's a race condition...

Invalidate the cache line....  then another CPU manages to fetch the cache line.
and then we do a store that doesn't update the cache - and the other CPU
is still looking at the old data.


[-- Attachment #1.2: Type: application/pgp-signature, Size: 832 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: arm64: Question about barriers with the mmu off

[Wonhyuk Yang]

On Tue, Nov 17, 2020 at 3:25 PM Valdis Klētnieks
<valdis.kletnieks@vt.edu> wrote:
>
> > > > > dc ivac,x1   // invalidate a cache line that's probably OK
> > > > > str w0,[x1   // and now we do a store that leaves a possibly stale cache line
>
> > Could you explain me why the store still leaves stale cache?
> > We invalidated the cacheline and store will not make footprint in the cache.
>
> There's a race condition...
>
> Invalidate the cache line....  then another CPU manages to fetch the cache line.
> and then we do a store that doesn't update the cache - and the other CPU
> is still looking at the old data.

Oh, I didn't consider that another cpu read with cacheable.
Now I understand why the barrier is here.

Thank you for your help.

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies