Re: [PATCH] blkback: Fix block I/O latency issue

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

> >>what were the numbers when it came to high bandwidth numbers
> 
> Under high I/O workload, where the blkfront would fill up the queue as
> blkback works the queue, the I/O latency problem in question doesn't
> manifest itself and as a result this patch doesn't make much of a
> difference in terms of interrupt rate. My benchmarks didn't show any
> significant effect.

I have to rerun my benchmarks. Under high load (so 64Kb, four threads
writting as much as they can to a iSCSI disk), the IRQ rate for each
blkif went from 2-3/sec to ~5K/sec. But I did not do a good
job on capturing the submission latency to see if the I/Os get the
response back as fast (or the same) as without your patch.

And the iSCSI disk on the target side was an RAMdisk, so latency
was quite small which is not fair to your problem.

Do you have a program to measure the latency for the workload you
had encountered? I would like to run those numbers myself.

> 
> The above rationale combined with relatively high disk I/O latencies
> (compared to IRQ latency) generally minimizes excessive interrupt rate.
> Also, blkfront interrupt generation mechanism works exactly the same way
> as the patched blkback. Netfront and netback generate interrupts the same
> way as well. 
> 
> Under 'moderate' I/O workload, the rate of interrupt does go up but the
> I/O latency benefit clearly outweighs the cost extra interrupt rate (which
> isn't much for moderate I/O load anyways)
> 
> 
> Overall, advantages of this patch (I/O latency improvement) outweighs any
> potential fringe negative effects by a large margin and the fact that
> netfront, netback and blkfront already have the same interrupt generation
> design point should give us a lot of confidence.

If we can come up with a patch that does decrease the I/O latency _and_
does not cause a huge spike in interrupt generation that would be super.

I am particularly worried about the high interrupt generation when there is
a high I/O load. But that seems to be tied directly to the "disk" I have.
I am curious to how this works with drivers that are SSDs for example.

> 
> That said, I do think a comprehensive interrupt throttling mechanism for
> netback, blkback and other backend I/O drivers would be useful and should
> be pursued as a separate initiative. Such a mechanism would be

Yeah, my fear is that you will disappear once this patch is in and I will
have to go forth to do this (and I don't know when I would get to it).
Would prefer to have your help here or (better) you write the patch for
both problems right now :-)

> particularly useful for netfront-netback stack which is more susceptible
> to interrupt storms than blkfront-blkback. 'IRQ coalescing' type mechanism
> that could induce delays in the order of  10s of microsecs (certainly not
> in millisecs though) to minimize interrupt generation rate would be useful
> (similar to what NICs do).

Good point. Awaiting your patch for the netfront-netback code :-)