Re: [PATCH] ntb_netdev: fix sleep time mismatch

[Dave Jiang <dave.jiang@intel.com>]

On 06/12/2018 03:04 PM, Logan Gunthorpe wrote:
> 
> 
> On 12/06/18 03:59 PM, Eric Pilmore wrote:
>> I agree. There are too many interrupts and too much bounce buffering.
>> We have played around with eliminating the interrupts all together, but haven't
>> seen a noticeable improvement, although our queue size is artificially small
>> for other reasons.  We're hoping to devote more resources to benchmarking
>> and improving it. Eliminating or reducing the bounce buffering would help
>> quite a bit also. Ideally the sender could copy straight into a skb on the
>> receiving side instead of the intermediate buffering into the shared memory.
>> Again, we're hoping to get time and resources to play around with some
>> ideas.
> 
> Awesome! I'm really glad someone is looking into this! I'd really
> appreciate it if you keep the list informed if you make progress.
> 
> The only idea I had was to start by writing an ntb_transport_perf to see
> if the slow down is in ntb_transport (due to bounce buffering, etc) or
> if it's all ntb_netdev/tcp layer.

The bounce buffering definitely decreases the performance. So the idea
of directly writing to the remote skbuff would be great. It would also
resolve issue of failing to allocate a large enough DMA region for the
bounce buffer. Allen and I talked about netpoll support but never got
around to look into it. So that may be another idea for performance
enhancement. I do wonder if we will have any resistance from upstream
due to security concerns since in order to do that we'll have to open up
a very large BAR and allow access to entire memory region of the
opposing node. Not an IOMMU expert, but will we have any issues when it
comes to DMA to the remote buffers in this way? Probably not since Allen
has NTRDMA doing something similar. I guess we can see what the change
looks like. Looking forward to that though.