Re: Data corruption with 5.10.x client -> 6.5.x server

["Mantas Mikulėnas" <grawity@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2497 bytes --]

On 2023-09-24 17:44, Chuck Lever III wrote:
> 
> 
>> On Sep 24, 2023, at 10:32 AM, Mantas Mikulėnas <grawity@gmail.com> wrote:
>>
>> On 2023-09-24 16:28, Chuck Lever III wrote:
>>>> On Sep 24, 2023, at 9:07 AM, Mantas Mikulėnas <grawity@gmail.com> wrote:
>>>>
>>>> I've recently upgraded my home NFS server from 6.4.12 to 6.5.4 (running Arch Linux x86_64).
>>>>
>>>> Now, when I'm accessing the server over NFSv4.2 from a client that's running 5.10.0 (32-bit x86, Debian 11), if the mount is using sec=krb5i or sec=krb5p, trying to read a file that's <= 4092 bytes in size will return all-zero data. (That is, `hexdump -C file` shows "00 00 00...") Files that are 4093 bytes or larger seem to be unaffected.
>>>>
>>>> Only sec=krb5i/krb5p are affected by this – plain sec=krb5 (or sec=sys for that matter) seems to work without any problems.
>>>>
>>>> Newer clients (like 6.1.x or 6.4.x) don't seem to have any issues, it's only 5.10.0 that does... though it might also be that the client is 32-bit, but the same client did work previously when the server was running older kernels, so I still suspect 6.5.x on the server being the problem.
>>>>
>>>> Upgrading to 6.6.0-rc2 on the server hasn't changed anything.
>>>> The server is using Btrfs but I've tested with tmpfs as well.
>>> I'm guessing proto=tcp as well (as opposed to proto=rdma).
>>
>> Yes, it's TCP.
>>
>> (I do have RDMA set up between two of the 6.5.x server systems, but in this case all the clients I've tested were TCP-only, and the home server that I originally noticed the problem with doesn't have RDMA at all.)
>>
>>> Does the problem go away with vers=4.1 ?
>>
>> No, it doesn't (neither with 4.0).
>>
>>> Can you capture network traffic during the failure? Use sec=krb5i so
>>> we can see the RPC payloads. On the client:
>>> # tcpdump -iany -s0 -w/tmp/sniffer.pcap
>>
>> Attached. (The script I've been using for testing mounts with -o sec=krb5i, cats three files, then unmounts.)<nfs_krb5i.pcap>
> 
> I see three NFS READs in the capture.
> 
> The first READ payload is all zeroes. The second payload contains
> "Hello World (4093 bytes)" repeatedly, and the third contains
> "Hello World (4096 bytes)" repeatedly.

Right, whereas on the server, the first file is filled with "Hello World 
(4092 bytes)" as I originally tried to narrow down the issue.

Meanwhile, 6.4.x (Arch) clients don't seem to be having any problems 
with the same server, and with seemingly the same mount options.

Thanks for looking into it!

[-- Attachment #2: nfs_krb5i_working_6.4client.pcap --]
[-- Type: application/octet-stream, Size: 47504 bytes --]