Invalidating buffer cache in dom0

Invalidating buffer cache in dom0

[Jacob Gorm Hansen]

hi,

I have a domU that dumps some data (a checkpoint) to a virtual device. 
Afterwards, I want to create a new domU based on that data, but because 
right now my domain bootstrapper does not have a disk driver, I first 
build a ramdisk which I give as an argument to my bootstrapper.

This all works fine, but I have the problem that I have to reboot dom0 
between each run, because apparently dom0 Linux does not notice that the 
contents on disk have changed, and keep giving me stale data from its 
buffer cache when I create the ramdisk. I have tested that this is 
actually what happens by grepping for a string in the block device 
(/dev/hda5), and the value of the string changes between reboots.

Does anyone know of an ioctl or similar that I could use to make Linux 
forget about (not flush) buffer cache contents, or how to disable it?

I have tried reading from the block dev with O_DIRECT but that does not 
help.

Thanks,
Jacob

RE: Invalidating buffer cache in dom0

[Ian Pratt]

> Does anyone know of an ioctl or similar that I could use to 
> make Linux forget about (not flush) buffer cache contents, or 
> how to disable it?
> 
> I have tried reading from the block dev with O_DIRECT but 
> that does not help.

Have you tried using raw devices? See "man raw". I've never used them,
but I'd be interested to know whether this works. There's an argument
that we should be using them by default, particularly for people using
using cluster filesystems -- caching stuff in dom0 could be very bad.

Perhaps some of the ocfs2 guys that hang out on the list could comment?

Thanks,
Ian

Re: Invalidating buffer cache in dom0

[Keir Fraser]

On 28 May 2005, at 19:49, Ian Pratt wrote:

> Have you tried using raw devices? See "man raw". I've never used them,
> but I'd be interested to know whether this works. There's an argument
> that we should be using them by default, particularly for people using
> using cluster filesystems -- caching stuff in dom0 could be very bad.

This isn't a problem for domU virtual blkdevs: our backend driver 
doesn't hook into the page cache, so unless the block device it sends 
requests to loops back to the page cache then we will not be affected 
by stale buffer cache contents.

Jacob has given a domU direct access to a partition as a virtual 
blkdev. It has modified it, and the updates did go straight to disc. 
But if dom0 tries to read partition contents itself then it sees stale 
page cache data (because the backend completely circumvented the page 
cache). I'm surprised that O_DIRECT didn't fix the problem -- I suppose 
it's being 'too clever' in this case and checking for potentially dirty 
pages in the cache before going direct to disc.

  -- Keir

Re: Invalidating buffer cache in dom0

[Jacob Gorm Hansen]

Ian Pratt wrote:

> Have you tried using raw devices? See "man raw". I've never used them,
> but I'd be interested to know whether this works. There's an argument
> that we should be using them by default, particularly for people using
> using cluster filesystems -- caching stuff in dom0 could be very bad.

It is my impression that in recent 2.6 kernels raw devices and O_DIRECT 
are the same thing. Actually, I was forced to use O_DIRECT inside my 
domU for the self-checkpointing, as otherwise the buffer cache would 
inflate the working set and create additional downtime. O_DIRECT works 
really well, and minus a ton of bugs I can now self-checkpoint to disk 
as well as self-migrate via the network.

O_DIRECT works for writes. However, it seems it still tries to be a 
little too clever when reading from a blockdev.

This is not a huge prob for me, just an annoyance that I have to reboot 
all the time to flush the cache, but I guess that for a Xen system in 
general it would be nice to have consistency of block device contents 
between domUs and dom0.

thanks,
Jacob