TFTP hangs with fragmented IP packets

TFTP hangs with fragmented IP packets

[Laurent Pinchart]

[-- Attachment #1: Type: text/plain, Size: 1458 bytes --]

Hello,

I've banged my head a few days ago trying to debug an issue with a TFTP
transfer hanging in the middle.

I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using
the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and
the board uses the EQoS ethernet controller.

The problem started occurring after rebuilding a kernel image. U-Boot
started transferring the image, and stopped in the middle, eventually
timing out. Capture network traffic showed that U-Boot was continuously
asking for retransmit of the same block, and eventually timed out.

U-Boot is configured with a default TFTP block size of 4096 bytes, which
results in the TFTP blocks being sent in one UDP packet split in three
IP packets. U-Boot is configured with IP fragmentation supprot enabled.
This works fine for all TFTP blocks until a paticular one in the middle
of the kernel image.

I've narrowed it down to a file of 1472 that can't be transferred at
all (I have attached the binary to this e-mail). Changing the value of
any of the last two bytes of the file allows transferring it correctly,
so I suspect a CRC issue, likely related to IP fragmentation. Lowering
the TFTP block size to avoid fragmentation works around the problem.

Arguably a TFTP block size of 4096 bytes should probably not be used
with a 1500 bytes MTU network, but I thought it would be useful to fix
the issue nonetheless.

I can test patches.

-- 
Regards,

Laurent Pinchart

[-- Attachment #2: test.bin --]
[-- Type: application/octet-stream, Size: 1472 bytes --]

Re: TFTP hangs with fragmented IP packets

[Ramon Fried]

On Tue, Mar 29, 2022 at 11:28 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hello,
>
> I've banged my head a few days ago trying to debug an issue with a TFTP
> transfer hanging in the middle.
>
> I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using
> the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and
> the board uses the EQoS ethernet controller.
>
> The problem started occurring after rebuilding a kernel image. U-Boot
> started transferring the image, and stopped in the middle, eventually
> timing out. Capture network traffic showed that U-Boot was continuously
> asking for retransmit of the same block, and eventually timed out.
>
> U-Boot is configured with a default TFTP block size of 4096 bytes, which
> results in the TFTP blocks being sent in one UDP packet split in three
> IP packets. U-Boot is configured with IP fragmentation supprot enabled.
> This works fine for all TFTP blocks until a paticular one in the middle
> of the kernel image.
>
> I've narrowed it down to a file of 1472 that can't be transferred at
> all (I have attached the binary to this e-mail). Changing the value of
> any of the last two bytes of the file allows transferring it correctly,
> so I suspect a CRC issue, likely related to IP fragmentation. Lowering
> the TFTP block size to avoid fragmentation works around the problem.
>
> Arguably a TFTP block size of 4096 bytes should probably not be used
> with a 1500 bytes MTU network, but I thought it would be useful to fix
> the issue nonetheless.
>
> I can test patches.
>
> --
> Regards,
>
> Laurent Pinchart
Interesting.
I will try to reproduce.
Thanks for reporting.

Re: TFTP hangs with fragmented IP packets

[Ramon Fried]

On Thu, Mar 31, 2022 at 8:43 AM Ramon Fried <rfried.dev@gmail.com> wrote:
>
> On Tue, Mar 29, 2022 at 11:28 PM Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
> >
> > Hello,
> >
> > I've banged my head a few days ago trying to debug an issue with a TFTP
> > transfer hanging in the middle.
> >
> > I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using
> > the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and
> > the board uses the EQoS ethernet controller.
> >
> > The problem started occurring after rebuilding a kernel image. U-Boot
> > started transferring the image, and stopped in the middle, eventually
> > timing out. Capture network traffic showed that U-Boot was continuously
> > asking for retransmit of the same block, and eventually timed out.
> >
> > U-Boot is configured with a default TFTP block size of 4096 bytes, which
> > results in the TFTP blocks being sent in one UDP packet split in three
> > IP packets. U-Boot is configured with IP fragmentation supprot enabled.
> > This works fine for all TFTP blocks until a paticular one in the middle
> > of the kernel image.
> >
> > I've narrowed it down to a file of 1472 that can't be transferred at
> > all (I have attached the binary to this e-mail). Changing the value of
> > any of the last two bytes of the file allows transferring it correctly,
> > so I suspect a CRC issue, likely related to IP fragmentation. Lowering
> > the TFTP block size to avoid fragmentation works around the problem.
> >
> > Arguably a TFTP block size of 4096 bytes should probably not be used
> > with a 1500 bytes MTU network, but I thought it would be useful to fix
> > the issue nonetheless.
> >
> > I can test patches.
> >
> > --
> > Regards,
> >
> > Laurent Pinchart
> Interesting.
> I will try to reproduce.
> Thanks for reporting.

Hi.
I couldn't reproduce the issue with the file you provided.
I used sandbox64, and changed the TFTP BLOCK size to 4096.
I managed to TFTP the file successfully.

=> setenv autoload no
=> setenv ethrotate no
=> setenv ethact eth0
=> setenv ipaddr
=> setenv serverip 172.23.1.137
=> env set ipaddr 172.23.1.100
=> tftpboot 0x100000 test.bin
Using host_eno1 device
TFTP from server 172.23.1.137; our IP address is 172.23.1.100
Filename 'test.bin'.
Load address: 0x100000
Loading: #
         718.8 KiB/s
done
Bytes transferred = 1472 (5c0 hex)
=>

Can you perhaps send me a network PCAP of your device failing to
digest the file ?

Thanks,
Ramon

Re: TFTP hangs with fragmented IP packets

[Laurent Pinchart]

Hi Ramon,

On Sun, Apr 03, 2022 at 02:17:26AM +0300, Ramon Fried wrote:
> On Thu, Mar 31, 2022 at 8:43 AM Ramon Fried wrote:
> > On Tue, Mar 29, 2022 at 11:28 PM Laurent Pinchart wrote:
> > >
> > > Hello,
> > >
> > > I've banged my head a few days ago trying to debug an issue with a TFTP
> > > transfer hanging in the middle.
> > >
> > > I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using
> > > the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and
> > > the board uses the EQoS ethernet controller.
> > >
> > > The problem started occurring after rebuilding a kernel image. U-Boot
> > > started transferring the image, and stopped in the middle, eventually
> > > timing out. Capture network traffic showed that U-Boot was continuously
> > > asking for retransmit of the same block, and eventually timed out.
> > >
> > > U-Boot is configured with a default TFTP block size of 4096 bytes, which
> > > results in the TFTP blocks being sent in one UDP packet split in three
> > > IP packets. U-Boot is configured with IP fragmentation supprot enabled.
> > > This works fine for all TFTP blocks until a paticular one in the middle
> > > of the kernel image.
> > >
> > > I've narrowed it down to a file of 1472 that can't be transferred at
> > > all (I have attached the binary to this e-mail). Changing the value of
> > > any of the last two bytes of the file allows transferring it correctly,
> > > so I suspect a CRC issue, likely related to IP fragmentation. Lowering
> > > the TFTP block size to avoid fragmentation works around the problem.
> > >
> > > Arguably a TFTP block size of 4096 bytes should probably not be used
> > > with a 1500 bytes MTU network, but I thought it would be useful to fix
> > > the issue nonetheless.
> > >
> > > I can test patches.
> > >
> > Interesting.
> > I will try to reproduce.
> > Thanks for reporting.
> 
> Hi.
> I couldn't reproduce the issue with the file you provided.
> I used sandbox64, and changed the TFTP BLOCK size to 4096.
> I managed to TFTP the file successfully.
> 
> => setenv autoload no
> => setenv ethrotate no
> => setenv ethact eth0
> => setenv ipaddr
> => setenv serverip 172.23.1.137
> => env set ipaddr 172.23.1.100
> => tftpboot 0x100000 test.bin
> Using host_eno1 device
> TFTP from server 172.23.1.137; our IP address is 172.23.1.100
> Filename 'test.bin'.
> Load address: 0x100000
> Loading: #
>          718.8 KiB/s
> done
> Bytes transferred = 1472 (5c0 hex)
> =>

Hmmmm... I wonder if it could be specific to the EQoS ethernet
controller in the i.MX8, maybe caused by issues in hardware CRC
calculation ? I don't have any other board I can use for testing this at
the moment as I'm travelling.

> Can you perhaps send me a network PCAP of your device failing to
> digest the file ?

I'll try to do that tomorrow.

-- 
Regards,

Laurent Pinchart

Re: TFTP hangs with fragmented IP packets

[Tim Harvey]

On Tue, Apr 5, 2022 at 4:11 PM Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Ramon,
>
> On Sun, Apr 03, 2022 at 02:17:26AM +0300, Ramon Fried wrote:
> > On Thu, Mar 31, 2022 at 8:43 AM Ramon Fried wrote:
> > > On Tue, Mar 29, 2022 at 11:28 PM Laurent Pinchart wrote:
> > > >
> > > > Hello,
> > > >
> > > > I've banged my head a few days ago trying to debug an issue with a TFTP
> > > > transfer hanging in the middle.
> > > >
> > > > I'm testing U-Boot 2022-rc5 on a Toradex Verdin i.MX8MP module (using
> > > > the verdin-imx8mp defconfig). My local network MTU is 1500 bytes, and
> > > > the board uses the EQoS ethernet controller.
> > > >
> > > > The problem started occurring after rebuilding a kernel image. U-Boot
> > > > started transferring the image, and stopped in the middle, eventually
> > > > timing out. Capture network traffic showed that U-Boot was continuously
> > > > asking for retransmit of the same block, and eventually timed out.
> > > >
> > > > U-Boot is configured with a default TFTP block size of 4096 bytes, which
> > > > results in the TFTP blocks being sent in one UDP packet split in three
> > > > IP packets. U-Boot is configured with IP fragmentation supprot enabled.
> > > > This works fine for all TFTP blocks until a paticular one in the middle
> > > > of the kernel image.
> > > >
> > > > I've narrowed it down to a file of 1472 that can't be transferred at
> > > > all (I have attached the binary to this e-mail). Changing the value of
> > > > any of the last two bytes of the file allows transferring it correctly,
> > > > so I suspect a CRC issue, likely related to IP fragmentation. Lowering
> > > > the TFTP block size to avoid fragmentation works around the problem.
> > > >
> > > > Arguably a TFTP block size of 4096 bytes should probably not be used
> > > > with a 1500 bytes MTU network, but I thought it would be useful to fix
> > > > the issue nonetheless.
> > > >
> > > > I can test patches.
> > > >
> > > Interesting.
> > > I will try to reproduce.
> > > Thanks for reporting.
> >
> > Hi.
> > I couldn't reproduce the issue with the file you provided.
> > I used sandbox64, and changed the TFTP BLOCK size to 4096.
> > I managed to TFTP the file successfully.
> >
> > => setenv autoload no
> > => setenv ethrotate no
> > => setenv ethact eth0
> > => setenv ipaddr
> > => setenv serverip 172.23.1.137
> > => env set ipaddr 172.23.1.100
> > => tftpboot 0x100000 test.bin
> > Using host_eno1 device
> > TFTP from server 172.23.1.137; our IP address is 172.23.1.100
> > Filename 'test.bin'.
> > Load address: 0x100000
> > Loading: #
> >          718.8 KiB/s
> > done
> > Bytes transferred = 1472 (5c0 hex)
> > =>
>
> Hmmmm... I wonder if it could be specific to the EQoS ethernet
> controller in the i.MX8, maybe caused by issues in hardware CRC
> calculation ? I don't have any other board I can use for testing this at
> the moment as I'm travelling.
>
> > Can you perhaps send me a network PCAP of your device failing to
> > digest the file ?
>
> I'll try to do that tomorrow.
>
> --
> Regards,
>
> Laurent Pinchart

Laurent,

I'm working with a imx8mp-venice-gw74xx [1] board with EQoS<->RGMII
PHY and am able to transfer your test file with
CONFIG_TFTP_BLOCKSIZE=4096 fine. I haven't had any issues transferring
files in general and routinely transfer files up to 513MB.

Are you running into an issue perhaps with your loadaddr position? I'm
using CONFIG_SYS_LOAD_ADDR=0x40480000.

Best Regards,

Tim
[1] https://patchwork.ozlabs.org/project/uboot/list/?series=294275