Re: [PATCH 4/4] nvme-tcp: switch to 'cpu' affinity scope for unbound workqueues

[Sagi Grimberg <sagi@grimberg.me>]

>> I tend to think that the io timeouts are caused by a bug, not by "non 
>> optimized" code. io timeouts are eternity for this test, which makes me
>> think we have a different issue here.
>
> I did some latency measurements for the send and receive loop, and 
> found that we are in fact starved by the receive side. The sending 
> side is pretty well limited by the 'deadline' setting, but the 
> receiving side has no such precaution, and I have seen per-queue 
> receive latencies of over 5 milliseconds.
> The worrying thing here was that only individual queues have been 
> affected; most queues had the expected latency of around 50usecs, but
> some really went over the top with 1000s of usecs. And these were the
> queues which were generating I/O timeouts.
>
> I have now modified the deadline method to cover both receive and 
> sending side, and the results were pretty good; timeouts are gone and
> even the overall performance for the 4 subsystem case has gone up.
>
> Will be posting an updated patchset shortly.

That is good to get some confirmation. I'll wait to see your patch 
(assuming that you added
count limit to the desc passed to read_sock?)

btw, the count doesn't need to be byte granular, it can also store 
jiffies and check time.
However, I'd prefer to keep it to byte count such that we can leverage 
it to do one call
directly in nvme_tcp_data_ready().