Re: bug: tpacket_snd can cause data corruption

[Frank de Brabander <debrabander@gmail.com>]

On 03-07-19 18:07, Willem de Bruijn wrote:

> On Wed, Jul 3, 2019 at 7:08 AM Frank de Brabander <debrabander@gmail.com> wrote:
>> In commit 5cd8d46e a fix was applied for data corruption in
>> tpacket_snd. A selftest was added in commit 358be656 which
>> validates this fix.
>>
>> Unfortunately this bug still persists, although since this fix less
>> likely to trigger. This bug was initially observed using a PACKET_MMAP
>> application, but can also be seen by tweaking the kernel selftest.
>>
>> By tweaking the selftest txring_overwrite.c to run
>> as an infinite loop, the data corruption will still trigger. It
>> seems to occur faster by generating interrupts (e.g. by plugging
>> in USB devices). Tested with kernel version 5.2-RC7.
>>
>> Cause for this bug is still unclear.
> The cause of the original bug is well understood.
>
> The issue you report I expect is due to background traffic. And more
> about the test than the kernel implementation.
>
> Can you reproduce the issue when running the modified test in a
> network namespace (./in_netns.sh ./txring_overwrite)?
>
> I observe the issue report outside that, but not inside. That implies
> that what we're observing is random background traffic. The modified
> test then drops the unexpected packet because it mismatches on length.
> As a result the next read (the test always sends two packets, then
> reads both) will report a data mismatch. Because it is reading the
> first test packet, but expecting the second. Output with a bit more
> data:
>
> count: 200
> count: 300
> count: 400
> count: 500
>   read: 90B != 100B
> wrong pattern: 0x61 != 0x62
> count: 600
> count: 700
> count: 800
>   read: 90B != 100B
> wrong pattern: 0x61 != 0x62
> count: 900
>   read: 90B != 100B
> wrong pattern: 0x61 != 0x62
>
> Notice the clear pattern.
>
> This does not trigger inside a network namespace, which is how
> kselftest invokes txring_override (from run_afpackettests).
I'm also able to reproduce the issue inside a network namespace.

I've added the extra logging, as seen in your output, for
mismatches on length. Running the test without ./in_netns.sh
is indeed as you describe:

read error: 66 != 100
wrong pattern: 0x61 != 0x62
read error: 66 != 100
wrong pattern: 0x61 != 0x62
read error: 74 != 100
read error: 66 != 100
wrong pattern: 0x53 != 0x61
wrong pattern: 0x53 != 0x62
read error: 66 != 100
read error: 66 != 100
read error: 66 != 100
wrong pattern: 0x61 != 0x62
read error: 95 != 100
read error: 95 != 100
wrong pattern: 0xffffffbe != 0x61
wrong pattern: 0x61 != 0x62
read error: 66 != 100

But even when running the test with ./in_netns.sh it shows
"wrong pattern", this time without length mismatches:

wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61
wrong pattern: 0x62 != 0x61

As already mentioned, it seems to trigger mainly (only ?) when
an USB device is connected. The PC I'm testing this on has an
USB hub with many ports and connected devices. When connecting
this USB hub, the amount of "wrong pattern" errors that are
shown seems to correlate to the amount of new devices
that the kernel should detect. Connecting in a single USB device
also triggers the error, but not on every attempt.

Unfortunately have not found any other way to force the
error to trigger. E.g. running stress-ng to generate CPU load or
timer interrupts does not seem to have any impact.