Re: [Intel-wired-lan] [PATCH net v3] ice: fix packet corruption due to extraneous page flip

[Philipp Schuster <philipp.schuster@cyberus-technology.de>]

Hi there,

We were hitting this bug pretty reliably in one of our setups and it
looks like this patch is fixing our problems. Is there any update on
that? For now, we've moved to a newer kernel (7.2.8) but it might still
be relevant to many folks out there.

I did not monitor memory growth during these tests.

A few details about my observation. Please note that I heavily used
an LLM to track down the issue and to create a reproducer. The LLM's
output and reproducer seemed valid to me:

I ran into this bug while benchmarking live migration with Cloud
Hypervisor between two hosts with Intel E810 NICs (ice, MTU 9000,
6.18.41). We migrate a VM with 120 vCPUs and 20 GiB of memory under a
write-heavy load, using TLS over 8 parallel TCP connections at about
8 GiB/s. On the receiver, up to 5 of 6 migrations failed because TLS
could not decrypt a record, and once even the plaintext record header
was broken. First I suspected our own code. But an in-process check
that decrypted 1.1 billion records again right after encryption found
no mismatch, and switching to another cipher did not help. So the data
was still correct when it left userspace. A plain-TCP test tool with
bulk traffic transferred more than 1.4 TB without any error, so the
trigger had to depend on the traffic pattern. At this point I found this
LKML thread. With the patched ice module on the receiver, my reproducer
passed all runs without a failure (432 GiB). Without the patch, 2 out of
3 runs had corrupted data.

The reproducer is a small C program (about 290 lines) that uses plain
TCP. To trigger the bug reliably, it does four things. First, it chooses
the write lengths so that frames exactly fill one or two 3 KiB Rx
buffers, and it uses TCP_NODELAY so that these frames are really sent
this way. Second, the sender is bursty and fast: 9 parallel connections,
plus 120 busy threads that simulate the vCPUs of a large VM with dirty
page tracking. Third, the receiver keeps many packets queued, because it
reads into scattered locations of a large buffer. Fourth, the stream
content is deterministic, so the receiver can check every byte. Without
the busy threads, 720 GiB went through without errors, and in single
test runs, leaving out one of the other conditions also made the
corruption disappear. Each corrupt block was at most one Rx buffer long
and contained data from the same connection, about 11 MiB later in the
stream. TCP itself reported no error.

Thanks,
Philipp