Re: [PATCH 00/33] Swap over NFS -v14

[Peter Zijlstra <a.p.zijlstra@chello.nl>]

[-- Attachment #1: Type: text/plain, Size: 2911 bytes --]

On Wed, 2007-10-31 at 08:16 -0400, Jeff Garzik wrote:
> Thoughts:
> 
> 1) I absolutely agree that NFS is far more prominent and useful than any 
> network block device, at the present time.
> 
> 
> 2) Nonetheless, swap over NFS is a pretty rare case.  I view this work 
> as interesting, but I really don't see a huge need, for swapping over 
> NBD or swapping over NFS.  I tend to think swapping to a remote resource 
> starts to approach "migration" rather than merely swapping.  Yes, we can 
> do it...  but given the lack of burning need one must examine the price.

There is a large corporate demand for this, which is why I'm doing this.

The typical usage scenarios are:
 - cluster/blades, where having local disks is a cost issue (maintenance
   of failures, heat, etc)
 - virtualisation, where dumping the storage on a networked storage unit
   makes for trivial migration and what not..

But please, people who want this (I'm sure some of you are reading) do
speak up. I'm just the motivated corporate drone implementing the
feature :-)

> 3) You note
> > Swap over network has the problem that the network subsystem does not use fixed
> > sized allocations, but heavily relies on kmalloc(). This makes mempools
> > unusable.
> 
> True, but IMO there are mitigating factors that should be researched and 
> taken into account:
> 
> a) To give you some net driver background/history, most mainstream net 
> drivers were coded to allocate RX skbs of size 1538, under the theory 
> that they would all be allocating out of the same underlying slab cache. 
>   It would not be difficult to update a great many of the [non-jumbo] 
> cases to create a fixed size allocation pattern.

One issue that comes to mind is how to ensure we'd still overflow the
IP-reassembly buffers. Currently those are managed on the number of
bytes present, not the number of fragments.

One of the goals of my approach was to not rewrite the network subsystem
to accomodate this feature (and I hope I succeeded).

> b) Spare-time experiments and anecdotal evidence points to RX and TX skb 
> recycling as a potentially valuable area of research.  If you are able 
> to do something like that, then memory suddenly becomes a lot more 
> bounded and predictable.
> 
> 
> So my gut feeling is that taking a hard look at how net drivers function 
> in the field should give you a lot of good ideas that approach the 
> shared goal of making network memory allocations more predictable and 
> bounded.

Note that being bounded only comes from dropping most packets before
trying them to a socket. That is the crucial part of the RX path, to
receive all packets from the NIC (regardless their size) but to not pass
them on to the network stack - unless they belong to a 'special' socket
that promises undelayed processing.

Thanks for these ideas, I'll look into them.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]