[PATCH 0/2] spl, rk3399: fix FIT image loadingg out-of-range

[PATCH 0/2] spl, rk3399: fix FIT image loadingg out-of-range

[Jerome Forissier]

The patches in this series address an issue I met when trying to enable
FIT signature verification by SPL on a RockPi4B board.

- The first patch avoids a buffer overflow when writing to INTMEM1 SRAM
- The second one addresses reliability issues I had with back-to-back
MMC reads from the microSD card. By allowing DMA operation, the issue is
gone and generally speaking DMA is preferred over FIFO mode anyways.

Jerome Forissier (2):
  spl: fit: add config option for temporary buffer when loading image
  rockchip: rk3399: enable spl-fifo-mode for sdmmc only when needed

 arch/arm/dts/rk3399-u-boot.dtsi |  2 +
 common/spl/Kconfig              | 45 +++++++++++++++++++
 common/spl/spl_fit.c            | 79 ++++++++++++++++++++++++++++++---
 3 files changed, 120 insertions(+), 6 deletions(-)

-- 
2.34.1

[PATCH 1/2] spl: fit: add config option for temporary buffer when loading image

[Jerome Forissier]

When the load address of a FIT image isn't properly aligned,
spl_load_fit_image() may write past the end of the destination buffer. It
is not an issue in many cases because the memory happens to be writeable
and nothing important is present in the overflow. On RockPi4 however there
is a configuration where a TF-A image (bl31_0xff3b0000.bin) has to be
loaded into a 8K range of SRAM memory, between 0xff3b0000 and 0xff3b2000.
The end address is a hard limit, because due to the way the hardware is
wired, the addresses wrap and any overflow gets written back to 0xff3b0000
thus overwriting previous data.

To address this problem, introduce a helper function which loads data using
a temporary buffer allocated on the stack. The size of the buffer is
defined by SPL_LOAD_FIT_IMAGE_BUFFER_SIZE (default 0x0 or disabled).

Co-developed-by: Xavier Drudis Ferran <xdrudis@tinet.cat>
Signed-off-by: Xavier Drudis Ferran <xdrudis@tinet.cat>
Signed-off-by: Jerome Forissier <jerome.forissier@linaro.org>
---
 common/spl/Kconfig   | 45 +++++++++++++++++++++++++
 common/spl/spl_fit.c | 79 ++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 118 insertions(+), 6 deletions(-)

diff --git a/common/spl/Kconfig b/common/spl/Kconfig
index 50ff113cab..d95141a6f6 100644
--- a/common/spl/Kconfig
+++ b/common/spl/Kconfig
@@ -1339,6 +1339,51 @@ config SPL_OPENSBI_LOAD_ADDR
 	help
 	  Load address of the OpenSBI binary.
 
+config SPL_LOAD_FIT_IMAGE_BUFFER_SIZE
+	hex "Read unaligned external FIT images to a temporary buffer in SPL"
+	default 0x0
+	depends on SPL_LOAD_FIT
+	help
+	  An aligned FIT image is such that it starts at the beginning
+	  of a block in media and has a load_addr in its FIT header
+	  that is DMA aligned in RAM. These aligned images can be read
+	  directly from media to RAM. Unaligned external FIT images
+	  are those that need reading some extra data before and/or after
+	  the image because they don't occupy fully the blocks they're
+	  in in media, or their destination is not DMA aligned and must
+	  be read somewhere aligned before copying them to load_addr.
+
+	  With this option set to 0x0 full blocks will just be read in
+	  the closest DMA aligned address, and the unaligned image
+	  inside those read blocks will later be copied to
+	  load_addr. Meanwhile memory outside [load_addr,
+	  load_addr+length) will have been written. That's no problem
+	  when the block length, image size and load_addr have been
+	  taken into account when laying out memory.
+
+	  But in some cases not all memory is writable, or undesired
+	  effects arise when writing outside [load_addr,
+	  load_addr+length). For instance, in RK3399, one of the
+	  images is ATF2, of size 8KiB which should be loaded into
+	  INTMEM1, at address 0xff3b0000. This address is DMA aligned
+	  but tight. It maps to a 8KiB SRAM area. If the image on
+	  media is not block aligned some extra bytes get read before
+	  and after, and everything extra written in
+	  0xff3b2000-0xff3bffff corrupts SRAM at
+	  0xff3b0000-0xff3b1fff before the image is memcpyied in place.
+
+	  With this option set to a nonzero value a DMA aligned buffer
+	  will be allocated on the stack and the image will be read
+	  in chuncks of blocks to this buffer, with each chunk being
+	  memcpyied to [load_addr,load_addr+length), so never writing
+	  outside the destination area.
+
+	  The advantage of enabling this option is safety, and the
+	  disadvantage is more stack use and slower image load (one read
+	  per chunk instead of just one).
+
+	  The default is 0x0 to replicate previous behaviour.
+
 config TPL
 	bool
 	depends on SUPPORT_TPL
diff --git a/common/spl/spl_fit.c b/common/spl/spl_fit.c
index 1bbf824684..56775fd744 100644
--- a/common/spl/spl_fit.c
+++ b/common/spl/spl_fit.c
@@ -12,6 +12,7 @@
 #include <log.h>
 #include <malloc.h>
 #include <mapmem.h>
+#include <memalign.h>
 #include <spl.h>
 #include <sysinfo.h>
 #include <asm/cache.h>
@@ -218,6 +219,64 @@ static int get_aligned_image_size(struct spl_load_info *info, int data_size,
 	return (data_size + info->bl_len - 1) / info->bl_len;
 }
 
+#if (CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE != 0x0)
+static int load_with_tmpbuf(struct spl_load_info *info, ulong load_addr,
+			    ulong sector, int offs, size_t len)
+{
+	ALLOC_CACHE_ALIGN_BUFFER(u8, buf,
+				 CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE);
+	void *dst = (void *)load_addr;
+	int nsect = (len + offs + info->bl_len - 1) / info->bl_len;
+	int bufsect = (CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE) / info->bl_len;
+	size_t sz, tail = 0;
+
+	if (offs) {
+		sz = info->bl_len - offs;
+		if (sz > len)
+			sz = len;
+		if (info->read(info, sector, 1, buf) != 1)
+			return -EIO;
+		memcpy(dst, buf + offs, sz);
+		dst += sz;
+		sector++;
+		nsect--;
+	}
+
+	if (nsect) {
+		tail = (len + offs) % info->bl_len;
+		nsect--;
+	}
+
+	while (nsect) {
+		int n = nsect;
+
+		if (n > bufsect)
+			n = bufsect;
+		if (info->read(info, sector, n, buf) != n)
+			return -EIO;
+		sz = n * info->bl_len;
+		memcpy(dst, buf, sz);
+		dst += sz;
+		sector += n;
+		nsect -= n;
+	}
+
+	if (tail) {
+		if (info->read(info, sector, 1, buf) != 1)
+			return -EIO;
+		memcpy(dst, buf, tail);
+	}
+
+	return 0;
+}
+#else
+static int load_with_tmpbuf(struct spl_load_info *info, ulong load_addr,
+			    ulong sector, int offs, size_t len)
+{
+	return -ENOENT;
+}
+#endif
+
 /**
  * spl_load_fit_image(): load the image described in a certain FIT node
  * @info:	points to information about the device to load data from
@@ -236,6 +295,7 @@ static int spl_load_fit_image(struct spl_load_info *info, ulong sector,
 			      const struct spl_fit_info *ctx, int node,
 			      struct spl_image_info *image_info)
 {
+	int ret;
 	int offset;
 	size_t length;
 	int len;
@@ -298,15 +358,22 @@ static int spl_load_fit_image(struct spl_load_info *info, ulong sector,
 
 		overhead = get_aligned_image_overhead(info, offset);
 		nr_sectors = get_aligned_image_size(info, length, offset);
-
-		if (info->read(info,
-			       sector + get_aligned_image_offset(info, offset),
-			       nr_sectors, src_ptr) != nr_sectors)
-			return -EIO;
+		sector += get_aligned_image_offset(info, offset);
+		if (CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE) {
+			ret = load_with_tmpbuf(info, load_addr, sector,
+					       overhead, len);
+			if (ret < 0)
+				return ret;
+			src = (void *)load_addr;
+		} else {
+			if (info->read(info, sector, nr_sectors,
+				       src_ptr) != nr_sectors)
+				return -EIO;
+			src = src_ptr + overhead;
+		}
 
 		debug("External data: dst=%p, offset=%x, size=%lx\n",
 		      src_ptr, offset, (unsigned long)length);
-		src = src_ptr + overhead;
 	} else {
 		/* Embedded data */
 		if (fit_image_get_data(fit, node, &data, &length)) {
-- 
2.34.1

[PATCH 2/2] rockchip: rk3399: enable spl-fifo-mode for sdmmc only when needed

[Jerome Forissier]

Commit 5c606ca35c42 ("rockchip: rk3399: enable spl-fifo-mode for sdmmc")
mentions that the RK3399 SoC can't do DMA between SDMMC and SRAM.
According to the TRM "7.3.2 Embedded SRAM access path" [1], only the
8KB SRAM at 0xff3b0000 (INTMEM1) is in this situation. The 192KB SRAM
can be accessed by both DMA controllers.

Assuming the only use case for writing from MMC to INTMEM1 is loading
a FIT image, and with the introduction of a temporary buffer for that
purpose (CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE, which is required
anyways to ensure the destination boundaries are enforced), then
spl-fifo-mode is not needed anymore and DMA can be enabled safely.

Link: [1] https://www.rockchip.fr/Rockchip%20RK3399%20TRM%20V1.4%20Part1.pdf
CC: Deepak Das <deepakdas.linux@gmail.com>
Signed-off-by: Jerome Forissier <jerome.forissier@linaro.org>
---
 arch/arm/dts/rk3399-u-boot.dtsi | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/arm/dts/rk3399-u-boot.dtsi b/arch/arm/dts/rk3399-u-boot.dtsi
index 716b9a433a..a1b6d6f007 100644
--- a/arch/arm/dts/rk3399-u-boot.dtsi
+++ b/arch/arm/dts/rk3399-u-boot.dtsi
@@ -124,8 +124,10 @@
 &sdmmc {
 	u-boot,dm-pre-reloc;
 
+#ifndef CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE
 	/* mmc to sram can't do dma, prevent aborts transferring TF-A parts */
 	u-boot,spl-fifo-mode;
+#endif
 };
 
 &spi1 {
-- 
2.34.1

Re: [PATCH 2/2] rockchip: rk3399: enable spl-fifo-mode for sdmmc only when needed

[Jerome Forissier]

On 6/9/22 08:23, Jerome Forissier wrote:
> Commit 5c606ca35c42 ("rockchip: rk3399: enable spl-fifo-mode for sdmmc")
> mentions that the RK3399 SoC can't do DMA between SDMMC and SRAM.
> According to the TRM "7.3.2 Embedded SRAM access path" [1], only the
> 8KB SRAM at 0xff3b0000 (INTMEM1) is in this situation. The 192KB SRAM
> can be accessed by both DMA controllers.
> 
> Assuming the only use case for writing from MMC to INTMEM1 is loading
> a FIT image, and with the introduction of a temporary buffer for that
> purpose (CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE, which is required
> anyways to ensure the destination boundaries are enforced), then
> spl-fifo-mode is not needed anymore and DMA can be enabled safely.
> 
> Link: [1] https://www.rockchip.fr/Rockchip%20RK3399%20TRM%20V1.4%20Part1.pdf
> CC: Deepak Das <deepakdas.linux@gmail.com>
> Signed-off-by: Jerome Forissier <jerome.forissier@linaro.org>
> ---
>  arch/arm/dts/rk3399-u-boot.dtsi | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/arch/arm/dts/rk3399-u-boot.dtsi b/arch/arm/dts/rk3399-u-boot.dtsi
> index 716b9a433a..a1b6d6f007 100644
> --- a/arch/arm/dts/rk3399-u-boot.dtsi
> +++ b/arch/arm/dts/rk3399-u-boot.dtsi
> @@ -124,8 +124,10 @@
>  &sdmmc {
>  	u-boot,dm-pre-reloc;
>  
> +#ifndef CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE

Oops, that should rather be:

+#if (CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE == 0)

>  	/* mmc to sram can't do dma, prevent aborts transferring TF-A parts */
>  	u-boot,spl-fifo-mode;
> +#endif
>  };
>  
>  &spi1 {

Re: [PATCH 2/2] rockchip: rk3399: enable spl-fifo-mode for sdmmc only when needed

[Xavier Drudis Ferran]

El Tue, Jun 14, 2022 at 11:16:42AM -0700, Jerome Forissier deia:
> Oops, that should rather be:
> 
> +#if (CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE == 0)
>

I tested with this change, not that my opinion counts much, but
anyway:

Reviewed-by: Xavier Drudis Ferran <xdrudis@tinet.cat>
Tested-by: Xavier Drudis Ferran <xdrudis@tinet.cat>

This changes reduce some 0.2 s the boot time of my Rock Pi 4B. 

Before this series, just with u-boot/master from today 
   c18e5fb055 dtoc: Update test_src_scan.py for new tegra compatibles
plus the patches I sent to this list (I can't boot from MMC without
them)
   https://lists.denx.de/pipermail/u-boot/2022-June/485497.html
and bootstage configured, I get :

Timer summary in microseconds (10 records):
       Mark    Elapsed  Stage
  1,903,436  1,903,436  board_init_f
  1,903,436          0  board_init_f
  2,900,331    996,895  board_init_r
  4,091,657  1,191,326  id=64
  4,930,650    838,993  id=65
  4,930,827        177  main_loop
  7,946,715  3,015,888  bootm_start
  9,010,637  1,063,922  id=15
  9,010,639          2  start_kernel

Accumulated time:
                22,700  dm_r
               479,397  dm_f

With that plus 1/2 in this series and 
CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE=0x3000
I get something similar (slightly slower because of >=3 correct reads 
instead of 1 overwriting read per image): 

Timer summary in microseconds (10 records):
       Mark    Elapsed  Stage
  1,979,414  1,979,414  board_init_f
  1,979,414          0  board_init_f
  2,976,429    997,015  board_init_r
  4,166,623  1,190,194  id=64
  5,005,623    839,000  id=65
  5,005,800        177  main_loop
  8,020,791  3,014,991  bootm_start
  9,084,116  1,063,325  id=15
  9,084,118          2  start_kernel

Accumulated time:
                22,699  dm_r
               479,480  dm_f

With that plus this 2/2 it's faster (while safer) than initially.

Timer summary in microseconds (10 records):
       Mark    Elapsed  Stage
  1,709,384  1,709,384  board_init_f
  1,709,384          0  board_init_f
  2,706,192    996,808  board_init_r
  3,895,269  1,189,077  id=64
  4,733,786    838,517  id=65
  4,733,963        177  main_loop
  7,751,063  3,017,100  bootm_start
  8,814,449  1,063,386  id=15
  8,814,451          2  start_kernel

Accumulated time:
                22,703  dm_r
               479,520  dm_f


With this change your 2/2 patch becomes 

--- 

From: Jerome Forissier <jerome.forissier@linaro.org>
Date: Thu, 9 Jun 2022 17:23:22 +0200
Subject: [PATCH] rockchip: rk3399: enable spl-fifo-mode for sdmmc only when
 needed

Commit 5c606ca35c42 ("rockchip: rk3399: enable spl-fifo-mode for sdmmc")
mentions that the RK3399 SoC can't do DMA between SDMMC and SRAM.
According to the TRM "7.3.2 Embedded SRAM access path" [1], only the
8KB SRAM at 0xff3b0000 (INTMEM1) is in this situation. The 192KB SRAM
can be accessed by both DMA controllers.

Assuming the only use case for writing from MMC to INTMEM1 is loading
a FIT image, and with the introduction of a temporary buffer for that
purpose (CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE, which is required
anyways to ensure the destination boundaries are enforced), then
spl-fifo-mode is not needed anymore and DMA can be enabled safely.

Link: [1] https://www.rockchip.fr/Rockchip%20RK3399%20TRM%20V1.4%20Part1.pdf
CC: Deepak Das <deepakdas.linux@gmail.com>
Signed-off-by: Jerome Forissier <jerome.forissier@linaro.org>
---
 arch/arm/dts/rk3399-u-boot.dtsi | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/arm/dts/rk3399-u-boot.dtsi b/arch/arm/dts/rk3399-u-boot.dtsi
index 716b9a433a..e0bb230022 100644
--- a/arch/arm/dts/rk3399-u-boot.dtsi
+++ b/arch/arm/dts/rk3399-u-boot.dtsi
@@ -124,9 +124,10 @@
 &sdmmc {
 	u-boot,dm-pre-reloc;
 
+#if (CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE == 0)
 	/* mmc to sram can't do dma, prevent aborts transferring TF-A parts */
 	u-boot,spl-fifo-mode;
+#endif
 };
 
 &spi1 {
-- 
2.20.1

Re: [PATCH 2/2] rockchip: rk3399: enable spl-fifo-mode for sdmmc only when needed

[Kever Yang]

On 2022/6/9 23:23, Jerome Forissier wrote:
> Commit 5c606ca35c42 ("rockchip: rk3399: enable spl-fifo-mode for sdmmc")
> mentions that the RK3399 SoC can't do DMA between SDMMC and SRAM.
> According to the TRM "7.3.2 Embedded SRAM access path" [1], only the
> 8KB SRAM at 0xff3b0000 (INTMEM1) is in this situation. The 192KB SRAM
> can be accessed by both DMA controllers.
>
> Assuming the only use case for writing from MMC to INTMEM1 is loading
> a FIT image, and with the introduction of a temporary buffer for that
> purpose (CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE, which is required
> anyways to ensure the destination boundaries are enforced), then
> spl-fifo-mode is not needed anymore and DMA can be enabled safely.
>
> Link: [1] https://www.rockchip.fr/Rockchip%20RK3399%20TRM%20V1.4%20Part1.pdf
> CC: Deepak Das <deepakdas.linux@gmail.com>
> Signed-off-by: Jerome Forissier <jerome.forissier@linaro.org>

Reviewed-by: Kever Yang <kever.yang@rock-chips.com>


Thanks,

- Kever

> ---
>   arch/arm/dts/rk3399-u-boot.dtsi | 2 ++
>   1 file changed, 2 insertions(+)
>
> diff --git a/arch/arm/dts/rk3399-u-boot.dtsi b/arch/arm/dts/rk3399-u-boot.dtsi
> index 716b9a433a..a1b6d6f007 100644
> --- a/arch/arm/dts/rk3399-u-boot.dtsi
> +++ b/arch/arm/dts/rk3399-u-boot.dtsi
> @@ -124,8 +124,10 @@
>   &sdmmc {
>   	u-boot,dm-pre-reloc;
>   
> +#ifndef CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE
>   	/* mmc to sram can't do dma, prevent aborts transferring TF-A parts */
>   	u-boot,spl-fifo-mode;
> +#endif
>   };
>   
>   &spi1 {

Re: [PATCH 2/2] rockchip: rk3399: enable spl-fifo-mode for sdmmc only when needed

[Jerome Forissier]

On 6/29/22 05:01, Kever Yang wrote:
> 
> On 2022/6/9 23:23, Jerome Forissier wrote:
>> Commit 5c606ca35c42 ("rockchip: rk3399: enable spl-fifo-mode for sdmmc")
>> mentions that the RK3399 SoC can't do DMA between SDMMC and SRAM.
>> According to the TRM "7.3.2 Embedded SRAM access path" [1], only the
>> 8KB SRAM at 0xff3b0000 (INTMEM1) is in this situation. The 192KB SRAM
>> can be accessed by both DMA controllers.
>>
>> Assuming the only use case for writing from MMC to INTMEM1 is loading
>> a FIT image, and with the introduction of a temporary buffer for that
>> purpose (CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE, which is required
>> anyways to ensure the destination boundaries are enforced), then
>> spl-fifo-mode is not needed anymore and DMA can be enabled safely.
>>
>> Link: [1] https://www.rockchip.fr/Rockchip%20RK3399%20TRM%20V1.4%20Part1.pdf
>> CC: Deepak Das <deepakdas.linux@gmail.com>
>> Signed-off-by: Jerome Forissier <jerome.forissier@linaro.org>
> 
> Reviewed-by: Kever Yang <kever.yang@rock-chips.com>

Thanks. I will post a v2 with the corrected conditional [#if (xxx == 0)].

-- 
Jerome
> 
> Thanks,
> 
> - Kever
> 
>> ---
>>   arch/arm/dts/rk3399-u-boot.dtsi | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/arch/arm/dts/rk3399-u-boot.dtsi b/arch/arm/dts/rk3399-u-boot.dtsi
>> index 716b9a433a..a1b6d6f007 100644
>> --- a/arch/arm/dts/rk3399-u-boot.dtsi
>> +++ b/arch/arm/dts/rk3399-u-boot.dtsi
>> @@ -124,8 +124,10 @@
>>   &sdmmc {
>>       u-boot,dm-pre-reloc;
>>   +#ifndef CONFIG_SPL_LOAD_FIT_IMAGE_BUFFER_SIZE
>>       /* mmc to sram can't do dma, prevent aborts transferring TF-A parts */
>>       u-boot,spl-fifo-mode;
>> +#endif
>>   };
>>     &spi1 {