Re: [PATCH v1 3/9] xprtrdma: Introduce ro_unmap_sync method

[Sagi Grimberg <sagig@dev.mellanox.co.il>]

Hey Tom,

> On 11/24/2015 5:59 AM, Sagi Grimberg wrote:
>> As I see it, if we don't wait for local-invalidate to complete before
>> unmap and IO completion (and no one does)
>
> For the record, that is false. Windows quite strictly performs these
> steps,

I meant Linux ULPs. I'm not very familiar with Windows SMBD.

> and deliver millions of real 4K IOPS over SMB Direct.

That's very encouraging! Is this the client side scaling?
 From my experience, getting a storage client/initiator to scale
up to 1.5-3 MIOPs over a single HCA with limited number of cores
is really a struggle for each interrupt and cacheline.

Still the latency of each IO will see a noticable increase.

>> Waiting for local invalidate to complete would be a really big
>> sacrifice for our storage ULPs.
>
> Not waiting would also be a sacrifice, by compromising data integrity.
> It's a tough tradeoff, but if you choose to go that way it will be
> critical to be honest about the consequences to the data.

That's true. I assume that the best compromise would be remote
invalidation but some standards needs enhancements and it doesn't solve
the multi-rkey transactions.

As storage devices are becoming faster we really should try to do our
best to keep up.