Partial reconfiguration on ZynqMP

Partial reconfiguration on ZynqMP

[Simon Doppler]

Hi everyone,

We are currently trying to use partial reconfiguration for our ZynqMP-
based system (part is xczu6eg) and device-tree overlay based
reconfiguration crashes.
Our Linux system is using 5.10.0-xilinx-v2021.1.

The reconfigurable partition is decoupled from the rest of the design
with an AXI-shutdown manager IP and a DFX decoupler (the
shutdown_requested signal of the AXI shutdown manager is used to drive
the request_shutdown of the DFX decoupler).

The relevant parts of the device tree are:

---- BEGIN SNIP ----
fpga_full: fpga-full {
		status = "okay";
		reconf_region: reconf_region0 {
			  compatible = "fpga-region";
			  fpga-bridges = <&reconf_bridge0>;
		};
       };

reconf_bridge0: reconf_bridge@a0080000  {
			status = "okay";
			compatible = "xlnx,dfx-axi-shutdown-manager-
1.00";
			reg = <0x0 0xa0080000 0x0 0x10>;
			clock-names = "aclk";
			clocks = <&zynqmp_clk 73>;
			bridge-enable = <1>;
		};
---- END SNIP ----

To reconfigure, I tested the following cases:

1. manual reconfiguration: WORKS
   - decouple by writing to the register directly
   - reconfigure directly with the fpga-manager
   - recouple by writing directly to the register
   
2. full reconfiguration with device-tree overlay: WORKS
   - we feed the full bitstream to the FPGA manager
   - using the following device-tree overlay:
---- BEGIN SNIP ----
/dts-v1/;
/plugin/;

&reconf_region {
	firmware-name = "full.bit";
};
---- BEGIN SNIP ----

3. partial reconfiguration with device-tree overlay:
   - we feed the full bitstream to the FPGA manager
   - using the following device-tree overlay:
---- BEGIN SNIP ----
/dts-v1/;
/plugin/;

&reconf_region {
	partial-fpga-config;
	fpga-bridges = <&reconf_bridge0>;
	firmware-name = "partial.bit";
};
---- BEGIN SNIP ----

In this case, the whole system crashes (even the platform cable does
not respond anymore).
When setting an ILA trigger on the write valid signal of the AXI
control path of the AXI Shutdown manager, the ILA appears to trigger
but crashes before the data is shown in the hardware manager window of
Vivado.

When the partial bitstream written by the overlay is already loaded (by
hand, as is case 1), the reconfiguration using an overlay works and the
AXI Shutdown manager is written to as expected.

Do you have an idea on what is going wrong here ?
As I manage to do the reconfiguration by hand, I suspect a
configuration/device-tree/software error.

Best regards

Simon Doppler

RE: Partial reconfiguration on ZynqMP

[Nava kishore Manne]

Hi ,
	Please find my response inline.

> > -----Original Message-----
> > From: Simon Doppler <simon.doppler@fastree3d.com>
> > Sent: Wednesday, December 22, 2021 2:14 PM
> > To: mdf@kernel.org; trix@redhat.com; Michal Simek
> > <michals@xilinx.com>; linux-fpga@vger.kernel.org
> > Cc: Adrian Fiergolski <adrian.fiergolski@fastree3d.com>
> > Subject: Partial reconfiguration on ZynqMP
> >
> > Hi everyone,
> >
> > We are currently trying to use partial reconfiguration for our ZynqMP-
> > based system (part is xczu6eg) and device-tree overlay based
> > reconfiguration crashes.
> > Our Linux system is using 5.10.0-xilinx-v2021.1.
> >
> > The reconfigurable partition is decoupled from the rest of the design
> > with an AXI-shutdown manager IP and a DFX decoupler (the
> > shutdown_requested signal of the AXI shutdown manager is used to drive
> > the request_shutdown of the DFX decoupler).
> >
> > The relevant parts of the device tree are:
> >
> > ---- BEGIN SNIP ----
> > fpga_full: fpga-full {
> > 		status = "okay";
> > 		reconf_region: reconf_region0 {
> > 			  compatible = "fpga-region";
> > 			  fpga-bridges = <&reconf_bridge0>;
> > 		};
> >        };
> >
> > reconf_bridge0: reconf_bridge@a0080000  {
> > 			status = "okay";
> > 			compatible = "xlnx,dfx-axi-shutdown-manager-
> 1.00";
> > 			reg = <0x0 0xa0080000 0x0 0x10>;
> > 			clock-names = "aclk";
> > 			clocks = <&zynqmp_clk 73>;
> > 			bridge-enable = <1>;
> > 		};
> > ---- END SNIP ----
> >
> > To reconfigure, I tested the following cases:
> >
> > 1. manual reconfiguration: WORKS
> >    - decouple by writing to the register directly
> >    - reconfigure directly with the fpga-manager
> >    - recouple by writing directly to the register
> >
> > 2. full reconfiguration with device-tree overlay: WORKS
> >    - we feed the full bitstream to the FPGA manager
> >    - using the following device-tree overlay:
> > ---- BEGIN SNIP ----
> > /dts-v1/;
> > /plugin/;
> >
> > &reconf_region {
> > 	firmware-name = "full.bit";
> > };
> > ---- BEGIN SNIP ----
> >
> > 3. partial reconfiguration with device-tree overlay:
> >    - we feed the full bitstream to the FPGA manager
> >    - using the following device-tree overlay:
> > ---- BEGIN SNIP ----
> > /dts-v1/;
> > /plugin/;
> >
> > &reconf_region {
> > 	partial-fpga-config;
> > 	fpga-bridges = <&reconf_bridge0>;
> > 	firmware-name = "partial.bit";
> > };
> > ---- BEGIN SNIP ----
> >
> > In this case, the whole system crashes (even the platform cable does
> > not respond anymore).
> > When setting an ILA trigger on the write valid signal of the AXI
> > control path of the AXI Shutdown manager, the ILA appears to trigger
> > but crashes before the data is shown in the hardware manager window of
> Vivado.
> >
> > When the partial bitstream written by the overlay is already loaded
> > (by hand, as is case 1), the reconfiguration using an overlay works
> > and the AXI Shutdown manager is written to as expected.
> >
> > Do you have an idea on what is going wrong here ?
> > As I manage to do the reconfiguration by hand, I suspect a
> > configuration/device-tree/software error.
> >

In the case 2 use case could you please provide more details (if possible please share the log)
needs to cross verify whether the axi shutdown manager is written/configured as expected.

Regards,
Navakishore.

Re: Partial reconfiguration on ZynqMP

[Simon Doppler]

Hi,

See my reply below.

On Thu, 2021-12-23 at 06:03 +0000, Nava kishore Manne wrote:
> Hi ,
> 	Please find my response inline.
> 
> > > -----Original Message-----
> > > From: Simon Doppler <simon.doppler@fastree3d.com>
> > > Sent: Wednesday, December 22, 2021 2:14 PM
> > > To: mdf@kernel.org; trix@redhat.com; Michal Simek
> > > <michals@xilinx.com>; linux-fpga@vger.kernel.org
> > > Cc: Adrian Fiergolski <adrian.fiergolski@fastree3d.com>
> > > Subject: Partial reconfiguration on ZynqMP
> > > 
> > > Hi everyone,
> > > 
> > > We are currently trying to use partial reconfiguration for our
> > > ZynqMP-
> > > based system (part is xczu6eg) and device-tree overlay based
> > > reconfiguration crashes.
> > > Our Linux system is using 5.10.0-xilinx-v2021.1.
> > > 
> > > The reconfigurable partition is decoupled from the rest of the
> > > design
> > > with an AXI-shutdown manager IP and a DFX decoupler (the
> > > shutdown_requested signal of the AXI shutdown manager is used to
> > > drive
> > > the request_shutdown of the DFX decoupler).
> > > 
> > > The relevant parts of the device tree are:
> > > 
> > > ---- BEGIN SNIP ----
> > > fpga_full: fpga-full {
> > > 		status = "okay";
> > > 		reconf_region: reconf_region0 {
> > > 			  compatible = "fpga-region";
> > > 			  fpga-bridges = <&reconf_bridge0>;
> > > 		};
> > >        };
> > > 
> > > reconf_bridge0: reconf_bridge@a0080000  {
> > > 			status = "okay";
> > > 			compatible = "xlnx,dfx-axi-shutdown-manager-
> > 1.00";
> > > 			reg = <0x0 0xa0080000 0x0 0x10>;
> > > 			clock-names = "aclk";
> > > 			clocks = <&zynqmp_clk 73>;
> > > 			bridge-enable = <1>;
> > > 		};
> > > ---- END SNIP ----
> > > 
> > > To reconfigure, I tested the following cases:
> > > 
> > > 1. manual reconfiguration: WORKS
> > >    - decouple by writing to the register directly
> > >    - reconfigure directly with the fpga-manager
> > >    - recouple by writing directly to the register
> > > 
> > > 2. full reconfiguration with device-tree overlay: WORKS
> > >    - we feed the full bitstream to the FPGA manager
> > >    - using the following device-tree overlay:
> > > ---- BEGIN SNIP ----
> > > /dts-v1/;
> > > /plugin/;
> > > 
> > > &reconf_region {
> > > 	firmware-name = "full.bit";
> > > };
> > > ---- BEGIN SNIP ----
> > > 
> > > 3. partial reconfiguration with device-tree overlay:
> > >    - we feed the full bitstream to the FPGA manager
> > >    - using the following device-tree overlay:
> > > ---- BEGIN SNIP ----
> > > /dts-v1/;
> > > /plugin/;
> > > 
> > > &reconf_region {
> > > 	partial-fpga-config;
> > > 	fpga-bridges = <&reconf_bridge0>;
> > > 	firmware-name = "partial.bit";
> > > };
> > > ---- BEGIN SNIP ----
> > > 
> > > In this case, the whole system crashes (even the platform cable
> > > does
> > > not respond anymore).
> > > When setting an ILA trigger on the write valid signal of the AXI
> > > control path of the AXI Shutdown manager, the ILA appears to
> > > trigger
> > > but crashes before the data is shown in the hardware manager
> > > window of
> > Vivado.
> > > When the partial bitstream written by the overlay is already
> > > loaded
> > > (by hand, as is case 1), the reconfiguration using an overlay
> > > works
> > > and the AXI Shutdown manager is written to as expected.
> > > 
> > > Do you have an idea on what is going wrong here ?
> > > As I manage to do the reconfiguration by hand, I suspect a
> > > configuration/device-tree/software error.
> > > 
> 
> In the case 2 use case could you please provide more details (if
> possible please share the log)
> needs to cross verify whether the axi shutdown manager is
> written/configured as expected.
> 

In case 2, the bridge is not triggered as I replace the full bitstream
including the bridge. The relevant parts of the log are below:

U-Boot SPL 2021.01 (Oct 12 2021 - 09:28:42 +0000)
[....]
U-Boot 2021.01 (Oct 12 2021 - 09:28:42 +0000)
[....]
Scanning mmc 1:1...
Found U-Boot script /boot.scr
509 bytes read in 6 ms (82 KiB/s)
## Executing script at 20000000
26510890 bytes read in 1759 ms (14.4 MiB/s)
  design filename = "Falcon;UserID=0XFFFFFFFF;Version=2021.1"
  part number = "xczu6eg-ffvc900-1-i"
  date = "2021/12/20"
  time = "16:29:50"
  bytes in bitstream = 26510780
zynqmp_align_dma_buffer: Align buffer at 000000000010006e to
00000000000fff80(swap 0)
49559 bytes read in 16 ms (3 MiB/s)
21019136 bytes read in 1393 ms (14.4 MiB/s)
## Flattened Device Tree blob at 00100000
   Booting using the fdt blob at 0x100000
   Loading Device Tree to 000000007de03000, end 000000007de12196 ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 5.10.0-xilinx-v2021.1 (oe-user@oe-host)
(aarch64-poky-linux-gcc (GCC) 10.2.0, GNU ld (GNU Binutils)
2.36.1.20210209) #1 SMP Tue Oct 12 09:30:57 UTC 2021
[....]
[    0.000000] Kernel command line: video=DP-1:1920x1080 xilinx-
video.is_mplane=0 root=/dev/mmcblk1p2 rw rootwait earlycon
clk_ignore_unused
[....]
[    3.217384] zynqmp-ipi-mbox mailbox@ff990400: Registered ZynqMP IPI
mbox with TX/RX channels.
[    3.217581] FPGA manager framework
[....]
[    3.439816] fpga_manager fpga0: Xilinx ZynqMP FPGA Manager
registered
[    3.440223] xlnx_pr_decoupler a0080000.user_proc_bridge: fpga bridge
[Xilinx DFX AXI shutdown mgr] registered


[....]
falcon login: root
root@falcon:~# mkdir /sys/kernel/config/device-tree/overlays/user
partialBitstream-average-fwonly.dtbo     partialBitstream-average-
nobridge.dtbo   partialBitstream-average-
nofw.dtbo       partialBitstream-average-nopartial.dtbo
root@falcon:~# cat full.dtbo > /sys/kernel/config/device-
tree/overlays/user/dtbo 
[   92.045900] fpga_manager fpga0: writing partialBitstream-average.bit 
to Xilinx ZynqMP FPGA Manager
[  104.099858] OF: overlay: WARNING: memory leak will occur if overlay
removed, property: /fpga-full/reconf_region0/fpga-bridges
[  104.111882] OF: overlay: WARNING: memory leak will occur if overlay
removed, property: /fpga-full/reconf_region0/firmware-name
root@falcon:~#

--- END LOG ---

When loading the partial bitstream, I get the following log:


U-Boot SPL 2021.01 (Oct 12 2021 - 09:28:42 +0000)
[... irrelevant U-Boot SPL log ...]
U-Boot 2021.01 (Oct 12 2021 - 09:28:42 +0000)
[... irrelevant U-Boot log ...]
Scanning mmc 1:1...
Found U-Boot script /boot.scr
509 bytes read in 6 ms (82 KiB/s)
## Executing script at 20000000
26510890 bytes read in 1786 ms (14.2 MiB/s)
  design filename = "Falcon;UserID=0XFFFFFFFF;Version=2021.1"
  part number = "xczu6eg-ffvc900-1-i"
  date = "2021/12/20"
  time = "16:29:50"
  bytes in bitstream = 26510780
zynqmp_align_dma_buffer: Align buffer at 000000000010006e to
00000000000fff80(swap 0)
49559 bytes read in 16 ms (3 MiB/s)
21019136 bytes read in 1413 ms (14.2 MiB/s)
## Flattened Device Tree blob at 00100000
   Booting using the fdt blob at 0x100000
   Loading Device Tree to 000000007de03000, end 000000007de12196 ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 5.10.0-xilinx-v2021.1 (oe-user@oe-host)
(aarch64-poky-linux-gcc (GCC) 10.2.0, GNU ld (GNU Binutils)
2.36.1.20210209) #1 SMP Tue Oct 12 09:30:57 UTC 2021
[....]
[    0.000000] Kernel command line: video=DP-1:1920x1080 xilinx-
video.is_mplane=0 root=/dev/mmcblk1p2 rw rootwait earlycon
clk_ignore_unused
[....]
[    2.372821] zynqmp-ipi-mbox mailbox@ff990400: Registered ZynqMP IPI
mbox with TX/RX channels.
[    2.373018] FPGA manager framework
[....]
[    2.595794] fpga_manager fpga0: Xilinx ZynqMP FPGA Manager
registered
[    2.596206] xlnx_pr_decoupler a0080000.user_proc_bridge: fpga bridge
[Xilinx DFX AXI shutdown mgr] registered


[....]
falcon login: root
root@falcon:~# mkdir /sys/kernel/config/device-tree/overlays/user
root@falcon:~# file /lib/firmware/partial.bit
/lib/firmware/partial.bit: Xilinx BIT data - from
Falcon;PARTIAL=TRUE;UserID=0XFFFFFFFF;COMPRESS=FALSE;Version=2021.1 -
for xczu6eg-ffvc900-1-i - built 2021/12/20(16:35:18) - data length
0xf4510
root@falcon:~# cat partial.dtbo > /sys/kernel/config/device-
tree/overlays/user/dtbo 
[  259.672997] fpga_manager fpga0: writing partialBitstream-average.bit 
to Xilinx ZynqMP FPGA Manager

[ system crash at this point ]
--- END LOG ---

The result of the Vivado Hardware manager is here: 
https://nextcloud.bfh.science/index.php/s/HWSEfzawEm2kWpa

As you can see, the System ILA triggerd on a AWVALID signal rising edge
with the 0xA0080000 address (the address of the bridge). There does not
appear to be a read to that address which would check the the
decoupling actually happened.

Is is possible that there is a delay missing when the overlay is
applied? This delay is present since I have to type the command when I
do it manually (and then it works).

> Regards,
> Navakishore.

Best regards
Simon