Re: Ceph on just two nodes being clients - reasonable?

[Tomasz Chmielewski <mangoo@wpkg.org>]

On 19.01.2011 16:57, Gregory Farnum wrote:

>> Currently, I'm running glusterfs in such a scenario (two servers, each being
>> also clients), but I wanted to give ceph a try (glusterfs has some
>> performance issues with lots of small files), also because of its nice
>> features (snapshots, rbd etc.).
> Rather than running 3 monitors you could just put a monitor on one of
> the machines -- your cluster will go down if it fails, but in a 2-node
> system it's not like resilience from one-node failure would be very
> helpful anyway.

OK, I could imagine starting the monitor on just one node i.e. with the 
help of heartbeat - so if the node with the monitor goes down, heartbeat 
starts the monitor process on the other machine.


> However, there is a serious issue with running clients and servers on
> one machine, which may or may not be a problem depending on your use
> case: Deadlock becomes a significant possibility.

Sounds like the "freezes" issue mentioned by Dong Jin Lee?


> This isn't a problem
> we've come up with a good solution for, unfortunately, but imagine
> you're writing a lot of files to Ceph. Ceph dutifully writes them and
> the kernel dutifully caches them. You also have a lot of write
> activity so the Ceph kernel client is doing local caching. Then the
> kernel comes along and says "I'm low on memory! Flush stuff to disk!"
> and the kernel client tries to flush it out...which involves creating
> another copy of the data in memory on the same machine. Uh-oh!

Uh-oh, it doesn't sound encouraging, and will likely happen sooner or later.

Would some sort of zero-copy help here? But perhaps it's not that easy 
to solve, otherwise, we wouldn't be discussing it here.

I think swapping over NFS (or, iSCSI) has a similar problem ("need to 
write, but the network buffer is full, so we can't write over network -> 
deadlock"), and there were some patches floating around some years ago 
to solve it. Not sure what's the state of it and how similar it is to 
Ceph though.


Last I checked, 2 < 3, so having a budget HA solution which just needed 
2 servers instead of 3 would be a great thing to have! ;)


-- 
Tomasz Chmielewski
http://wpkg.org