Re: nbd, nbdkit, loopback mounts and memory management

[Wouter Verhelst <w@uter.be>]

Hi,

On Fri, Feb 15, 2019 at 10:53:32PM +0000, Richard W.M. Jones wrote:
> On Fri, Feb 15, 2019 at 10:41:26PM +0000, Richard W.M. Jones wrote:
> > On Fri, Feb 15, 2019 at 08:19:54PM +0100, Pavel Machek wrote:
> > > Hi!
> > > 
> > > I watched fosdem talk about
> > > nbdkit... https://www.youtube.com/watch?v=9E5A608xJG0 . Nice. But word
> > > of warning: I'm not sure using it read-write on localhost is safe.
> > > 
> > > In particular, user application could create a lot of dirty data
> > > quickly. If there's not enough memory for nbdkit (or nbd-client or
> > > nbd-server), you might get a deadlock.
> > 
> > Thanks for the kind words about the talk.  I've added Wouter Verhelst
> > & the NBD mailing list to CC.  Although I did the talk because the
> > subject is interesting, how I actually use nbdkit / NBD is to talk to
> > qemu and that's where I have most experience and where we (Red Hat)
> > use it in production systems.
> > 
> > However in January I spent a lot of time exercising the NBD loop-mount
> > + nbdkit case using fio in order to find contention / bottlenecks in
> > our use of threads and locks.  I didn't notice any particular problems
> > then, but it's possible my testing wasn't thorough enough.  Or that
> > fio only creates small numbers of dirty pages (because of locality in
> > its access patterns I guess?)
> > 
> > When you say it's not safe, what could happen?  What would we observe
> > if it was going wrong?
> 
> Reading more carefully I see you said we'd observe a deadlock.  I
> didn't see that, but again my testing of this wouldn't have been very
> thorough.  When I have some time I'll try creating / spooling huge
> files into an NBD loop mount to see if I can cause a deadlock.

While it's of course impossible to fully exclude the possibility of
deadlock when clearing dirty pages to the network, since Mel Gorman's
work that resulted in commit 7f338fe4540b1d0600b02314c7d885fd358e9eca
this should be extremely unlikely, and swapping over the network (NBD or
NFS or whatnot) should be reasonably safe, as well as clearing dirty
pages etc.

Additionally, nbd-client when called with -s calls memlockall() at an
appropriate moment, so that it should not be swapped out.

That only leaves the server side. Personally I haven't been able to
deadlock a reasonably recent machine using NBD, but of course YMMV.

> > > Also note that nbd.txt in Documentation/blockdev/ points to
> > > sourceforge; it should probably point to
> > > https://github.com/NetworkBlockDevice/nbd ?
> > 
> > Wouter should be able to say what the correct link should be.

The sourceforge project is still active, and is where I do the official
file releases. I also push the git repository there. For people who just
want a released version of the NBD utilities, pointing to sourceforge
isn't wrong, I would say.

GitHub is indeed used mostly for development, though.

It might be nice to rethink all that, now that we don't have a
mailinglist running at sourceforge anymore, but I don't think it's very
urgent.

-- 
To the thief who stole my anti-depressants: I hope you're happy

  -- seen somewhere on the Internet on a photo of a billboard