Re: Big I/O latencies, except when iotop is hooked

[Arnd Bergmann <arnd@arndb.de>]

On Wednesday 09 May 2012, Felipe Contreras wrote:
> I've been noticing big I/O latencies when doing operations with
> notmuch (xapian database), the operations are quite simple and
> probably should not need a lot of I/O, however, they seen to take a
> long time, sometimes even seconds. But when I hook iotop (-p pid), the
> latency goes away, and every operation is reliably instantaneous
> (basically).
> 
> Do you have any ideas what might be causing this delay, and why is
> iotop making it go away?
> 
> BTW. This is an ext4 encrypted partition on a USB stick, I tried
> different mount options without any noticeable change. I tried to copy
> the data to my SSD drive and do the same operations, while it was much
> faster, it still seemed to have some delays triggered randomly. This
> is with v3.3.5.

USB sticks like most other cheap flash media tend to have long latencies
because of the effects I describe on https://lwn.net/Articles/428584/.

I don't know if you have the chance to run flashbench[1] on it (which
will destroy the data for the partition you are testing), but that
should at least tell you if it's a problem with the drive.
You can also use the information from that and combine it with
a blocktrace log and flashsim[2] to see where the actual latencies
are expected to happen.

The first thing you should check is whether the partitions are
properly aligned, using 'fdisk -l -u /dev/sdX'.

Of course none of this will tell you why iotop makes any difference.

	Arnd

[1] http://git.linaro.org/gitweb?p=people/arnd/flashbench.git
[2] http://yxit.co.uk/public/flash-performance/flashsim/