Injecting delays into block layer

[Oleksandr Natalenko <oleksandr@natalenko.name>]

Hi Paolo et al.

I have a strong suspect that something is going wrong when the 
underlying block device responds with a large delay. What makes me 
thinking so is that I use a VM on some cloud provider, and they have 
substantial block device latency resulting in permanently high (~20%) 
iowait. It spikes occasionally when their cluster is overloaded, and 
when that happens, the I/O in my VM may stop and never recover. This is 
a rare occasion, but it really happens.

What's worse, so far I've seen such a behaviour with BFQ only. I'm still 
testing other schedulers though.

Important note: I have no strict evidences that this is *the* case, thus 
I'm asking for some suggestions. My idea is to fire up a local VM and 
inject delays to a block device while performing some I/O from within 
the VM.

So the question is: how can those delays be injected? Using dm-delay? 
Can those delays be random?

Thanks in advance.

-- 
   Oleksandr Natalenko (post-factum)