[PATCH] riscv: dts: thead: set dma-noncoherent to soc bus

[PATCH] riscv: dts: thead: set dma-noncoherent to soc bus

[Jisheng Zhang]

riscv select ARCH_DMA_DEFAULT_COHERENT by default, and th1520 isn't
dma coherent, so set dma-noncoherent to reflect this fact.

Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
---
 arch/riscv/boot/dts/thead/th1520.dtsi | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/riscv/boot/dts/thead/th1520.dtsi b/arch/riscv/boot/dts/thead/th1520.dtsi
index 56a73134b49e..58108f0eb3fd 100644
--- a/arch/riscv/boot/dts/thead/th1520.dtsi
+++ b/arch/riscv/boot/dts/thead/th1520.dtsi
@@ -139,6 +139,7 @@ soc {
 		interrupt-parent = <&plic>;
 		#address-cells = <2>;
 		#size-cells = <2>;
+		dma-noncoherent;
 		ranges;
 
 		plic: interrupt-controller@ffd8000000 {
-- 
2.40.1

Re: [PATCH] riscv: dts: thead: set dma-noncoherent to soc bus

[Jisheng Zhang]

On Sun, Aug 20, 2023 at 07:53:53PM +0800, Jisheng Zhang wrote:
> riscv select ARCH_DMA_DEFAULT_COHERENT by default, and th1520 isn't
> dma coherent, so set dma-noncoherent to reflect this fact.

correct typo in linux-riscv maillist addr
Add linux-riscv, sorry.

> 
> Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
> ---
>  arch/riscv/boot/dts/thead/th1520.dtsi | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/riscv/boot/dts/thead/th1520.dtsi b/arch/riscv/boot/dts/thead/th1520.dtsi
> index 56a73134b49e..58108f0eb3fd 100644
> --- a/arch/riscv/boot/dts/thead/th1520.dtsi
> +++ b/arch/riscv/boot/dts/thead/th1520.dtsi
> @@ -139,6 +139,7 @@ soc {
>  		interrupt-parent = <&plic>;
>  		#address-cells = <2>;
>  		#size-cells = <2>;
> +		dma-noncoherent;
>  		ranges;
>  
>  		plic: interrupt-controller@ffd8000000 {
> -- 
> 2.40.1
> 

Re: [PATCH] riscv: dts: thead: set dma-noncoherent to soc bus

[Drew Fustini]

On Sun, Aug 20, 2023 at 07:53:53PM +0800, Jisheng Zhang wrote:
> riscv select ARCH_DMA_DEFAULT_COHERENT by default, and th1520 isn't
> dma coherent, so set dma-noncoherent to reflect this fact.
> 
> Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
> ---
>  arch/riscv/boot/dts/thead/th1520.dtsi | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/riscv/boot/dts/thead/th1520.dtsi b/arch/riscv/boot/dts/thead/th1520.dtsi
> index 56a73134b49e..58108f0eb3fd 100644
> --- a/arch/riscv/boot/dts/thead/th1520.dtsi
> +++ b/arch/riscv/boot/dts/thead/th1520.dtsi
> @@ -139,6 +139,7 @@ soc {
>  		interrupt-parent = <&plic>;
>  		#address-cells = <2>;
>  		#size-cells = <2>;
> +		dma-noncoherent;
>  		ranges;
>  
>  		plic: interrupt-controller@ffd8000000 {
> -- 
> 2.40.1
> 

Tested-by: Drew Fustini <dfustini@baylibre.com>

I tried this on the BeagleV Ahead. They system booted as expected and I
did not notice any problems in the boot log.

Are there other patches such as the dwmac series that I should test this
with?

Also, I think this might have missed patchwork since the original email
had a typo for the list, so it might be good to resend it?

Thanks,
Drew

Re: [PATCH] riscv: dts: thead: set dma-noncoherent to soc bus

[Jisheng Zhang]

On Sun, Sep 03, 2023 at 04:32:21PM -0700, Drew Fustini wrote:
> On Sun, Aug 20, 2023 at 07:53:53PM +0800, Jisheng Zhang wrote:
> > riscv select ARCH_DMA_DEFAULT_COHERENT by default, and th1520 isn't
> > dma coherent, so set dma-noncoherent to reflect this fact.
> > 
> > Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
> > ---
> >  arch/riscv/boot/dts/thead/th1520.dtsi | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/arch/riscv/boot/dts/thead/th1520.dtsi b/arch/riscv/boot/dts/thead/th1520.dtsi
> > index 56a73134b49e..58108f0eb3fd 100644
> > --- a/arch/riscv/boot/dts/thead/th1520.dtsi
> > +++ b/arch/riscv/boot/dts/thead/th1520.dtsi
> > @@ -139,6 +139,7 @@ soc {
> >  		interrupt-parent = <&plic>;
> >  		#address-cells = <2>;
> >  		#size-cells = <2>;
> > +		dma-noncoherent;
> >  		ranges;
> >  
> >  		plic: interrupt-controller@ffd8000000 {
> > -- 
> > 2.40.1
> > 
> 
> Tested-by: Drew Fustini <dfustini@baylibre.com>
> 
> I tried this on the BeagleV Ahead. They system booted as expected and I
> did not notice any problems in the boot log.
> 
> Are there other patches such as the dwmac series that I should test this
> with?

Hmm, this patch is necessary to test dwmac, emmc ;)

> 
> Also, I think this might have missed patchwork since the original email
> had a typo for the list, so it might be good to resend it?

This patch was sent after sending PR to soc people, so it was late for
linux-6.6-rc1, but this patch is a fix, I may send it as a fix once
linux-6.6-rc1 is out.

Re: [PATCH] riscv: dts: thead: set dma-noncoherent to soc bus

[Xi Ruoyao]

On Mon, 2023-09-04 at 22:45 +0800, Jisheng Zhang wrote:
> On Sun, Sep 03, 2023 at 04:32:21PM -0700, Drew Fustini wrote:
> > On Sun, Aug 20, 2023 at 07:53:53PM +0800, Jisheng Zhang wrote:
> > > riscv select ARCH_DMA_DEFAULT_COHERENT by default, and th1520 isn't
> > > dma coherent, so set dma-noncoherent to reflect this fact.
> > > 
> > > Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
> > > ---
> > >  arch/riscv/boot/dts/thead/th1520.dtsi | 1 +
> > >  1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/arch/riscv/boot/dts/thead/th1520.dtsi b/arch/riscv/boot/dts/thead/th1520.dtsi
> > > index 56a73134b49e..58108f0eb3fd 100644
> > > --- a/arch/riscv/boot/dts/thead/th1520.dtsi
> > > +++ b/arch/riscv/boot/dts/thead/th1520.dtsi
> > > @@ -139,6 +139,7 @@ soc {
> > >                 interrupt-parent = <&plic>;
> > >                 #address-cells = <2>;
> > >                 #size-cells = <2>;
> > > +               dma-noncoherent;
> > >                 ranges;
> > >  
> > >                 plic: interrupt-controller@ffd8000000 {
> > > -- 
> > > 2.40.1
> > > 
> > 
> > Tested-by: Drew Fustini <dfustini@baylibre.com>
> > 
> > I tried this on the BeagleV Ahead. They system booted as expected and I
> > did not notice any problems in the boot log.
> > 
> > Are there other patches such as the dwmac series that I should test this
> > with?
> 
> Hmm, this patch is necessary to test dwmac, emmc ;)

Drew: does this fix the "broken DMA" issue you've mentioned in the EMMC
support patch?

-- 
Xi Ruoyao <xry111@xry111.site>
School of Aerospace Science and Technology, Xidian University

Re: [PATCH] riscv: dts: thead: set dma-noncoherent to soc bus

[Drew Fustini]

On Tue, Sep 05, 2023 at 12:28:41AM +0800, Xi Ruoyao wrote:
> On Mon, 2023-09-04 at 22:45 +0800, Jisheng Zhang wrote:
> > On Sun, Sep 03, 2023 at 04:32:21PM -0700, Drew Fustini wrote:
> > > On Sun, Aug 20, 2023 at 07:53:53PM +0800, Jisheng Zhang wrote:
> > > > riscv select ARCH_DMA_DEFAULT_COHERENT by default, and th1520 isn't
> > > > dma coherent, so set dma-noncoherent to reflect this fact.
> > > > 
> > > > Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
> > > > ---
> > > >  arch/riscv/boot/dts/thead/th1520.dtsi | 1 +
> > > >  1 file changed, 1 insertion(+)
> > > > 
> > > > diff --git a/arch/riscv/boot/dts/thead/th1520.dtsi b/arch/riscv/boot/dts/thead/th1520.dtsi
> > > > index 56a73134b49e..58108f0eb3fd 100644
> > > > --- a/arch/riscv/boot/dts/thead/th1520.dtsi
> > > > +++ b/arch/riscv/boot/dts/thead/th1520.dtsi
> > > > @@ -139,6 +139,7 @@ soc {
> > > >                 interrupt-parent = <&plic>;
> > > >                 #address-cells = <2>;
> > > >                 #size-cells = <2>;
> > > > +               dma-noncoherent;
> > > >                 ranges;
> > > >  
> > > >                 plic: interrupt-controller@ffd8000000 {
> > > > -- 
> > > > 2.40.1
> > > > 
> > > 
> > > Tested-by: Drew Fustini <dfustini@baylibre.com>
> > > 
> > > I tried this on the BeagleV Ahead. They system booted as expected and I
> > > did not notice any problems in the boot log.
> > > 
> > > Are there other patches such as the dwmac series that I should test this
> > > with?
> > 
> > Hmm, this patch is necessary to test dwmac, emmc ;)
> 
> Drew: does this fix the "broken DMA" issue you've mentioned in the EMMC
> support patch?

After I sent my reply yesterday, I then tried applying this patch on top
of my eMMC series. I removed SDHCI_QUIRK_BROKEN_DMA and I discovered
that SDMA did not immediately break and the mmcblk0 partitions appeared
in /dev [1]. Note - I am using CONFIG_INITRAMFS_SOURCE set to my
buidlroot rootfs.cpio so that the system will boot okay regardless of
the emmc.

I was able to mount the emmc boot partition /dev/mmcblk0p2. However,
after unmounting it, I noticed that I could not mount it again:

  # mount /dev/mmcblk0p2 /mnt/p2 
  [   41.684899] EXT4-fs (mmcblk0p2): mounted filesystem 6ea98ccf-2178-48a9-a1ad-0eb454f57268 r/w with ordered data mode
  # ls -la /mnt/p2
  total 22696
  drwxr-xr-x    5 root     root          4096 Sep  4  2023 .
  drwxr-xr-x    4 root     root             0 Jan  1 00:00 ..
  -rw-rw-r--    1 1000     1000      26455552 Sep  4  2023 Image
  drwxr-xr-x    2 root     root          4096 Jul 29  2023 extlinux
  <snip>
  # umount /mnt/p2
  [   52.101003] EXT4-fs (mmcblk0p2): unmounting filesystem 6ea98ccf-2178-48a9-a1ad-0eb454f57268.
  # mount /dev/mmcblk0p2 /mnt/p2
  [   58.826078] JBD2: no valid journal superblock found
  [   58.831041] EXT4-fs (mmcblk0p2): Could not load journal inode
  mount: mounting /dev/mmcblk0p2 on /mnt/p2 failed: Invalid argument

I was completely unable to mount the root fs partition /dev/mmcblk0p3:

  # mount /dev/mmcblk0p2 /mnt/p2
  [   58.826078] JBD2: no valid journal superblock found
  [   58.831041] EXT4-fs (mmcblk0p2): Could not load journal inode
  mount: mounting /dev/mmcblk0p2 on /mnt/p2 failed: Invalid argument
  # umount /mnt/p2
  umount: can't unmount /mnt/p2: Invalid argument

I suspect there is some corruption of the buffers happening. I see there
are also these patch series from Jisheng so I will try applying them and
see if there is any difference:

  [PATCH v2 0/2] riscv: errata: improve T-Head CMO
  [PATCH] riscv: mm: update T-Head memory type definitions

As for ADMA, when I removed SDHCI_QUIRK_BROKEN_ADMA, something fails
silently [2] during boot and the mmcblk0 partitions do not appear under
/dev. 

-Drew

[1] https://gist.github.com/pdp7/6d27c57f8280fe1ae35e585d66e73f61
[2] https://gist.github.com/pdp7/23e2bcc2aba09b5c88eb579b61ea45ce

Re: [PATCH] riscv: dts: thead: set dma-noncoherent to soc bus

[Drew Fustini]

On Mon, Sep 04, 2023 at 10:45:43PM +0800, Jisheng Zhang wrote:
> On Sun, Sep 03, 2023 at 04:32:21PM -0700, Drew Fustini wrote:
> > On Sun, Aug 20, 2023 at 07:53:53PM +0800, Jisheng Zhang wrote:
> > > riscv select ARCH_DMA_DEFAULT_COHERENT by default, and th1520 isn't
> > > dma coherent, so set dma-noncoherent to reflect this fact.
> > > 
> > > Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
> > > ---
> > >  arch/riscv/boot/dts/thead/th1520.dtsi | 1 +
> > >  1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/arch/riscv/boot/dts/thead/th1520.dtsi b/arch/riscv/boot/dts/thead/th1520.dtsi
> > > index 56a73134b49e..58108f0eb3fd 100644
> > > --- a/arch/riscv/boot/dts/thead/th1520.dtsi
> > > +++ b/arch/riscv/boot/dts/thead/th1520.dtsi
> > > @@ -139,6 +139,7 @@ soc {
> > >  		interrupt-parent = <&plic>;
> > >  		#address-cells = <2>;
> > >  		#size-cells = <2>;
> > > +		dma-noncoherent;
> > >  		ranges;
> > >  
> > >  		plic: interrupt-controller@ffd8000000 {
> > > -- 
> > > 2.40.1
> > > 
> > 
> > Tested-by: Drew Fustini <dfustini@baylibre.com>
> > 
> > I tried this on the BeagleV Ahead. They system booted as expected and I
> > did not notice any problems in the boot log.
> > 
> > Are there other patches such as the dwmac series that I should test this
> > with?
> 
> Hmm, this patch is necessary to test dwmac, emmc ;)

Would you be able to share list of patches you are currently using so
that I can apply them all?

Do you have any changes in the device tree that you are using that is
not in mainline?

I added the linux,cma node [1] with compatible "shared-dma-pool" from 
light.dtsi in the thead sdk, but that did not seem to make a difference.

thanks,
drew

[1] https://git.beagleboard.org/beaglev-ahead/kernel/-/blob/master/arch/riscv/boot/dts/thead/light.dtsi#L88

Re: [PATCH] riscv: dts: thead: set dma-noncoherent to soc bus

[Drew Fustini]

On Mon, Sep 04, 2023 at 10:52:39AM -0700, Drew Fustini wrote:
> On Tue, Sep 05, 2023 at 12:28:41AM +0800, Xi Ruoyao wrote:
> > On Mon, 2023-09-04 at 22:45 +0800, Jisheng Zhang wrote:
> > > On Sun, Sep 03, 2023 at 04:32:21PM -0700, Drew Fustini wrote:
> > > > On Sun, Aug 20, 2023 at 07:53:53PM +0800, Jisheng Zhang wrote:
> > > > > riscv select ARCH_DMA_DEFAULT_COHERENT by default, and th1520 isn't
> > > > > dma coherent, so set dma-noncoherent to reflect this fact.
> > > > > 
> > > > > Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
> > > > > ---
> > > > >  arch/riscv/boot/dts/thead/th1520.dtsi | 1 +
> > > > >  1 file changed, 1 insertion(+)
> > > > > 
> > > > > diff --git a/arch/riscv/boot/dts/thead/th1520.dtsi b/arch/riscv/boot/dts/thead/th1520.dtsi
> > > > > index 56a73134b49e..58108f0eb3fd 100644
> > > > > --- a/arch/riscv/boot/dts/thead/th1520.dtsi
> > > > > +++ b/arch/riscv/boot/dts/thead/th1520.dtsi
> > > > > @@ -139,6 +139,7 @@ soc {
> > > > >                 interrupt-parent = <&plic>;
> > > > >                 #address-cells = <2>;
> > > > >                 #size-cells = <2>;
> > > > > +               dma-noncoherent;
> > > > >                 ranges;
> > > > >  
> > > > >                 plic: interrupt-controller@ffd8000000 {
> > > > > -- 
> > > > > 2.40.1
> > > > > 
> > > > 
> > > > Tested-by: Drew Fustini <dfustini@baylibre.com>
> > > > 
> > > > I tried this on the BeagleV Ahead. They system booted as expected and I
> > > > did not notice any problems in the boot log.
> > > > 
> > > > Are there other patches such as the dwmac series that I should test this
> > > > with?
> > > 
> > > Hmm, this patch is necessary to test dwmac, emmc ;)
> > 
> > Drew: does this fix the "broken DMA" issue you've mentioned in the EMMC
> > support patch?
> 
> After I sent my reply yesterday, I then tried applying this patch on top
> of my eMMC series. I removed SDHCI_QUIRK_BROKEN_DMA and I discovered
> that SDMA did not immediately break and the mmcblk0 partitions appeared
> in /dev [1]. Note - I am using CONFIG_INITRAMFS_SOURCE set to my
> buidlroot rootfs.cpio so that the system will boot okay regardless of
> the emmc.
> 
> I was able to mount the emmc boot partition /dev/mmcblk0p2. However,
> after unmounting it, I noticed that I could not mount it again:
> 
>   # mount /dev/mmcblk0p2 /mnt/p2 
>   [   41.684899] EXT4-fs (mmcblk0p2): mounted filesystem 6ea98ccf-2178-48a9-a1ad-0eb454f57268 r/w with ordered data mode
>   # ls -la /mnt/p2
>   total 22696
>   drwxr-xr-x    5 root     root          4096 Sep  4  2023 .
>   drwxr-xr-x    4 root     root             0 Jan  1 00:00 ..
>   -rw-rw-r--    1 1000     1000      26455552 Sep  4  2023 Image
>   drwxr-xr-x    2 root     root          4096 Jul 29  2023 extlinux
>   <snip>
>   # umount /mnt/p2
>   [   52.101003] EXT4-fs (mmcblk0p2): unmounting filesystem 6ea98ccf-2178-48a9-a1ad-0eb454f57268.
>   # mount /dev/mmcblk0p2 /mnt/p2
>   [   58.826078] JBD2: no valid journal superblock found
>   [   58.831041] EXT4-fs (mmcblk0p2): Could not load journal inode
>   mount: mounting /dev/mmcblk0p2 on /mnt/p2 failed: Invalid argument
> 
> I was completely unable to mount the root fs partition /dev/mmcblk0p3:
> 
>   # mount /dev/mmcblk0p2 /mnt/p2
>   [   58.826078] JBD2: no valid journal superblock found
>   [   58.831041] EXT4-fs (mmcblk0p2): Could not load journal inode
>   mount: mounting /dev/mmcblk0p2 on /mnt/p2 failed: Invalid argument
>   # umount /mnt/p2
>   umount: can't unmount /mnt/p2: Invalid argument
> 
> I suspect there is some corruption of the buffers happening. I see there
> are also these patch series from Jisheng so I will try applying them and
> see if there is any difference:
> 
>   [PATCH v2 0/2] riscv: errata: improve T-Head CMO
>   [PATCH] riscv: mm: update T-Head memory type definitions

Good news, I applied "riscv: errata: improve T-Head CMO" [1] and SDMA is
now working correctly. I can mount both boot and rootfs paritions and
then is no problem umounting and then remounting [2].

Note - I didn't 'riscv: mm: update T-Head memory type definitions' yet
but I will do that next and make sure everything is still working.

I'll post an update to my eMMC series.

-Drew

[1] https://lore.kernel.org/all/20230827090813.1353-1-jszhang@kernel.org/
[2] https://gist.github.com/pdp7/c72c91530c46ef560f4d1940ea1964c9