Re: "Directly mapped persistent memory page cache"

[Ingo Molnar <mingo@kernel.org>]

* Eric W. Biederman <ebiederm@xmission.com> wrote:

> > What do you think?
> 
> The tricky bit is what happens when you reboot and run a different 
> version of the kernel, especially a kernel with things debugging 
> features like kmemcheck that increase the size of struct page.

Yes - but I think that's relatively easy to handle, as most 'weird' 
page struct usages can be cordoned off:

I.e. we could define a 64-bit "core" struct page, denote it with a 
single PG_ flag and stick with it: the only ABI is its size 
essentially, as we (lazily) re-initialize it after every bootup.

The 'extended' (often debug) part of a struct page, such as 
page->shadow on kmemcheck, can simply be handled in a special way 
based on the PG_ flag:

 - for example in the kmemcheck case no page->shadow means no leak 
   tracking: that's perfectly fine as these pages aren't part of the 
   buddy allocator and kmalloc() anyway.

 - or NUMA_BALANCING's page->_last_cpupid can be 0 as well, as these
   pages aren't (normally) NUMA-migrated.

The extended fields would have to be accessed via small wrappers, 
which return 0 if the extended part is not present, but that's pretty 
much all.

> I don't think we could have persistent struct page entries, as the 
> exact contents of the struct page entries is too volatile and too 
> different between architectures. [...]

Especially with the 2MB (and 1GB) granular lazy initialization 
approach persisting them across reboots does not seem necessary 
either.

Even main RAM is already doing lazy initialization: Mel's patches that 
do that just went into -mm.

> [...] Especially architecture changes that a pmem store is likely to 
> see such as switching between a 32bit and a 64bit kernel.

We'd not want to ABI-restrict the layout of struct page. But to say 
that there's a core 64-byte descriptor per 4K page is not an overly 
strict promise to keep.

> Further I think where in the persistent memory the struct page 
> arrays live is something we could leave up to the filesystem.  We 
> could have some reasonable constraints to make it fast but I think 
> whoever decides where things live on the persistent memory can make 
> that choice.

So the beauty of the scheme is that in its initial incarnation it's 
filesystem independent: you can create any filesystem on top of it 
seemlessly, the filesystem simply sees a linear block device that is 
1.5% smaller than the underlying storage. It won't even (normally) 
have access to the struct page areas. This kind of data space 
separation also protects against filesystem originated data 
corruption.

Now in theory a filesystem might be aware of it, but I think it's far 
more important to keep this scheme simple, robust, fast and 
predictable.

> For small persistent memories it probably make sense to allocate the 
> struct page array describing them out of ordinary ram.  For small 
> memories I don't think we are talking enough memory to worry about. 
> For TB+ persistent memories where you need 16GiB per TiB it makes 
> sense to allocate a one or several regions to store your struct page 
> arrays, as you can't count on ordinary ram having enough capacity, 
> and you may not even be talking about a system that actually has 
> ordinary ram at that point.

Correct - if there's no ordinary page cache in main DRAM then for many 
appliances ordinary RAM could be something like SRAM: really fast and 
not wasted on dirty state and IO caches - a huge, directly mapped L4 
or L5 CPU cache in essence.

Thanks,

	Ingo