Re: invalidate the buffer heads of a block device

[Jan Kara <jack@suse.cz>]

On Mon 29-09-14 16:24:35, Thanos Makatos wrote:
> I'm looking for ways to achieve _read_ caching of a block device (usually
> iSCSI) that is attached to multiple hosts. The problem is that sometimes there
> will be some writes on that block device in a particular, known host using
> O_DIRECT, and that requires all other hosts to invalidate their buffer caches
> for that block device. I'd prefer to use standard tools/procedures rather than
> hacking things, but if I do have to implement something to solve this problem
> I'd prefer it to be something that can be accepted upstream.
> 
> It looks like that closing the block device guarantees that the buffer
> cache is invalidated, and I can guarantee that all _my_ processes close
> the block device to achieve this. However, if there is at least one other
> process I don't know of that is keeping an open file handle against that
> block device, the buffer cache won't be invalidated, and that will result
> in corruption. So this solution doesn't seem to work.
  Well, I wouldn't really advice to depend on buffer cache to be flushed
when all openers close the device.

> I had a look at the BLKFLSBUF ioctl, which seems to be designed to do the
> job, except that it doesn't work if a process has memory mapped the block
> device, and AFAIK there's no way to disallow memory mapping of a block
> device. Again, that looks like a deal-breaker.
  Yeah, plus it doesn't work when a page is dirty / under writeback
although that doesn't seem to be an issue for your usecase.

> If the above observations are correct, it seems that I have to either
> extend BLKFLSBUF to some invalidate such memory maps (I'm completely
> ignorant in that field, is it even possible?), or look for other
> solutions.
  Well, you could unmap the pages of block device that are mapped in your
new ioctl but I don't think it's easily possible to disallow userspace to
fault the pages back behind your back. And that could be a problem for you.

> Could some configuration of dm-cache, bcache, or some other component
> solve my problem? posix_fadvise with POSIX_FADV_DONTNEED seems to be just
> a hint to the kernel, there are no guarantees that cached data will be
> discarded.  I've also though of using a virtual block device driver that
> exclusively opens the actual network block device and lets user-space
> applications use the virtualised one so that I have more control and
> enforce things.
  Hum, I'm not aware of any approach which would do what you need.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR