Re: [PATCH v2 19/19] sunrpc: Disable splice for krb5i

Chuck Lever
2017-06-16T18:37:40Z

> On Jun 16, 2017, at 1:52 PM, bfields-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org wrote:
>
> Just repeating some comments from the bug:
>
> On Fri, Jun 16, 2017 at 11:22:54AM -0400, Chuck Lever wrote:
>> Running a multi-threaded 8KB fio test (70/30 mix), three or four out
>> of twelve of the jobs fail when using krb5i. The failure is an EIO
>> on a read.
>>
>> Troubleshooting confirmed the EIO results when the client fails to
>> verify the MIC of an NFS READ reply. Bruce suggested the problem
>> could be due to the data payload changing between the time the
>> reply's MIC was computed on the server and the time the reply was
>> actually sent.
>>
>> krb5p gets around this problem by disabling RQ_SPLICE_OK.
>
> And you verified that this does fix the problem in your case.

I've had this applied to my server for a week or so. There
hasn't been a single recurrence of the issue.

> So, I think it's a simple fix and probably the best we can do without a
> lot more work, so I'm happy applying it.
>
> That said, I'm still curious about the performance:
>
>> I would say that there is not much difference in this test.
>
> We added an extra copy to the read path and it didn't seem to affect
> throughput of streaming read much--I think that just says memory
> bandwidth isn't the bottlneck in this case? Which doesn't seem too
> surprising.

With krb5i, an additional memory copy is minor compared to the
computation needed.

I'm testing with 56Gbps networking and a tmpfs export. I'm not
exhausting the CPU on my 4-core server, even with krb5p. The
effects could be seen in a scalability test, but I don't have
anything that pushes my server that hard.

> I wonder what we should be looking for--maybe running the same test but
> also measuring CPU usage somehow.

Maybe an increase in latency. But I didn't see much change, and
the throughput numbers don't reflect any underlying increase in
per-RPC latency.

--
Chuck Lever