Messenger V2: multiple bind support

Messenger V2: multiple bind support

[Ricardo Dias]

Hi list,

I was wondering if it makes sense to support multiple binds with a
single messenger instance.

The use case I have in mind is when there are multiple public/cluster
networks specified in ceph.conf and we want to listen for connections
in one interface of each network. With the support for multiple binds,
we could use a single messenger instance to listen to all interfaces.

Do you think it is worth to implement such support or is the above use
case easily handled by having multiple messenger instances?

Thanks,
-- 
Ricardo Dias
Senior Software Engineer - Storage Team
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284
(AG Nürnberg)

Re: Messenger V2: multiple bind support

[Joao Eduardo Luis]

On 10/24/2017 03:36 PM, Ricardo Dias wrote:
> Hi list,
> 
> I was wondering if it makes sense to support multiple binds with a
> single messenger instance.
> 
> The use case I have in mind is when there are multiple public/cluster
> networks specified in ceph.conf and we want to listen for connections
> in one interface of each network. With the support for multiple binds,
> we could use a single messenger instance to listen to all interfaces.
> 
> Do you think it is worth to implement such support or is the above use
> case easily handled by having multiple messenger instances?

I would think multiple binds would make sense, but how would this affect 
setting the policies? Would we be sharing a single policy across 
multiple binds, or would we be allowed different policies for different 
binds?

Regardless, and not knowing the code, wouldn't this make sense for 
multiplexing multiple daemons on a single messenger? Making the 
messenger sort of a blackbox, that everyone could just use seamlessly 
without having to figure too many things out?

E.g., instead of multiplexing several osds on a messenger for the public 
network, and then the same set of osds on a messenger for the cluster 
network, we could simply have one messenger (bound to public and 
cluster) and have just that one messenger for all the osds.

Anyway, from an abstraction point-of-view it makes sense to me; no idea 
how feasible that would be though.

   -Joao

Re: Messenger V2: multiple bind support

[Gregory Farnum]

On Wed, Oct 25, 2017 at 7:43 AM Joao Eduardo Luis <joao@suse.de> wrote:
>
> On 10/24/2017 03:36 PM, Ricardo Dias wrote:
> > Hi list,
> >
> > I was wondering if it makes sense to support multiple binds with a
> > single messenger instance.
> >
> > The use case I have in mind is when there are multiple public/cluster
> > networks specified in ceph.conf and we want to listen for connections
> > in one interface of each network. With the support for multiple binds,
> > we could use a single messenger instance to listen to all interfaces.
> >
> > Do you think it is worth to implement such support or is the above use
> > case easily handled by having multiple messenger instances?
>
> I would think multiple binds would make sense, but how would this affect
> setting the policies? Would we be sharing a single policy across
> multiple binds, or would we be allowed different policies for different
> binds?
>
> Regardless, and not knowing the code, wouldn't this make sense for
> multiplexing multiple daemons on a single messenger? Making the
> messenger sort of a blackbox, that everyone could just use seamlessly
> without having to figure too many things out?
>
> E.g., instead of multiplexing several osds on a messenger for the public
> network, and then the same set of osds on a messenger for the cluster
> network, we could simply have one messenger (bound to public and
> cluster) and have just that one messenger for all the osds.
>
> Anyway, from an abstraction point-of-view it makes sense to me; no idea
> how feasible that would be though.

There's sort of two pieces here, right?
1) From the Dispatcher perspective, we want a single "Messenger" that
it talks to. (At least, I'm pretty sure. Unless we need to be making
routing decisions using internal data? I really hope that never comes
up.) That could be set up by having multiple Messenger implementations
behind a "UnifyingMessenger" abstraction layer with pretty minimal
changes.

2) From an implementation perspective, which data structures do we end
up needing to share in order to have a daemon listening on multiple
ports?

So, from the SimpleMessenger perspective, I don't think we get much
out of trying to build multi-bind into it. We'd basically get to
eliminate the duplicated DIspatchQueue, which is not very interesting.
(Modulo some flow control stuff, perhaps.) From an AsyncMessenger
implementation perspective, that calculus might shift since we're
running on a thread pool, but I'm not deeply familiar with the code
internals.

But I'm curious what brings this up. I thought you were working on the
v2 protocol — does something in that somehow hinge on the server-side
implementation of multiple networks?
-Greg

Re: Messenger V2: multiple bind support

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3557 bytes --]

On Wed, 1 Nov 2017, Gregory Farnum wrote:
> On Wed, Oct 25, 2017 at 7:43 AM Joao Eduardo Luis <joao@suse.de> wrote:
> >
> > On 10/24/2017 03:36 PM, Ricardo Dias wrote:
> > > Hi list,
> > >
> > > I was wondering if it makes sense to support multiple binds with a
> > > single messenger instance.
> > >
> > > The use case I have in mind is when there are multiple public/cluster
> > > networks specified in ceph.conf and we want to listen for connections
> > > in one interface of each network. With the support for multiple binds,
> > > we could use a single messenger instance to listen to all interfaces.
> > >
> > > Do you think it is worth to implement such support or is the above use
> > > case easily handled by having multiple messenger instances?
> >
> > I would think multiple binds would make sense, but how would this affect
> > setting the policies? Would we be sharing a single policy across
> > multiple binds, or would we be allowed different policies for different
> > binds?
> >
> > Regardless, and not knowing the code, wouldn't this make sense for
> > multiplexing multiple daemons on a single messenger? Making the
> > messenger sort of a blackbox, that everyone could just use seamlessly
> > without having to figure too many things out?
> >
> > E.g., instead of multiplexing several osds on a messenger for the public
> > network, and then the same set of osds on a messenger for the cluster
> > network, we could simply have one messenger (bound to public and
> > cluster) and have just that one messenger for all the osds.
> >
> > Anyway, from an abstraction point-of-view it makes sense to me; no idea
> > how feasible that would be though.
> 
> There's sort of two pieces here, right?
> 1) From the Dispatcher perspective, we want a single "Messenger" that
> it talks to. (At least, I'm pretty sure. Unless we need to be making
> routing decisions using internal data? I really hope that never comes
> up.) That could be set up by having multiple Messenger implementations
> behind a "UnifyingMessenger" abstraction layer with pretty minimal
> changes.
> 
> 2) From an implementation perspective, which data structures do we end
> up needing to share in order to have a daemon listening on multiple
> ports?
> 
> So, from the SimpleMessenger perspective, I don't think we get much
> out of trying to build multi-bind into it. We'd basically get to
> eliminate the duplicated DIspatchQueue, which is not very interesting.
> (Modulo some flow control stuff, perhaps.) From an AsyncMessenger
> implementation perspective, that calculus might shift since we're
> running on a thread pool, but I'm not deeply familiar with the code
> internals.

I'm afraid that the problem that's going to come up is the peer_addr 
hash_map<> for stateful_peer connections (osd to osd, etc.).  This is down 
in the messenger layer and is the main piece of shared state left (most 
everything else works with Connection refs).  I'm worried that we'll want 
or need to pull that up out of the messenger into the consumer (e.g., a 
hash_map<int,ConnectionRef> osd_peers in OSDService) and refactor the 
connection race handling a bit.  I think that will ultimately solve some 
bugs and will simplify the Messenger greatly, but it's a painful refactor.

> But I'm curious what brings this up. I thought you were working on the
> v2 protocol — does something in that somehow hinge on the server-side
> implementation of multiple networks?

The v2 protocol will run on another port and we want to answer on the old 
port+protocol for old clients too.

sage

Re: Messenger V2: multiple bind support

[Gregory Farnum]

On Wed, Nov 1, 2017 at 7:43 PM Sage Weil <sage@newdream.net> wrote:
>
> On Wed, 1 Nov 2017, Gregory Farnum wrote:
> > On Wed, Oct 25, 2017 at 7:43 AM Joao Eduardo Luis <joao@suse.de> wrote:
> > >
> > > On 10/24/2017 03:36 PM, Ricardo Dias wrote:
> > > > Hi list,
> > > >
> > > > I was wondering if it makes sense to support multiple binds with a
> > > > single messenger instance.
> > > >
> > > > The use case I have in mind is when there are multiple public/cluster
> > > > networks specified in ceph.conf and we want to listen for connections
> > > > in one interface of each network. With the support for multiple binds,
> > > > we could use a single messenger instance to listen to all interfaces.
> > > >
> > > > Do you think it is worth to implement such support or is the above use
> > > > case easily handled by having multiple messenger instances?
> > >
> > > I would think multiple binds would make sense, but how would this affect
> > > setting the policies? Would we be sharing a single policy across
> > > multiple binds, or would we be allowed different policies for different
> > > binds?
> > >
> > > Regardless, and not knowing the code, wouldn't this make sense for
> > > multiplexing multiple daemons on a single messenger? Making the
> > > messenger sort of a blackbox, that everyone could just use seamlessly
> > > without having to figure too many things out?
> > >
> > > E.g., instead of multiplexing several osds on a messenger for the public
> > > network, and then the same set of osds on a messenger for the cluster
> > > network, we could simply have one messenger (bound to public and
> > > cluster) and have just that one messenger for all the osds.
> > >
> > > Anyway, from an abstraction point-of-view it makes sense to me; no idea
> > > how feasible that would be though.
> >
> > There's sort of two pieces here, right?
> > 1) From the Dispatcher perspective, we want a single "Messenger" that
> > it talks to. (At least, I'm pretty sure. Unless we need to be making
> > routing decisions using internal data? I really hope that never comes
> > up.) That could be set up by having multiple Messenger implementations
> > behind a "UnifyingMessenger" abstraction layer with pretty minimal
> > changes.
> >
> > 2) From an implementation perspective, which data structures do we end
> > up needing to share in order to have a daemon listening on multiple
> > ports?
> >
> > So, from the SimpleMessenger perspective, I don't think we get much
> > out of trying to build multi-bind into it. We'd basically get to
> > eliminate the duplicated DIspatchQueue, which is not very interesting.
> > (Modulo some flow control stuff, perhaps.) From an AsyncMessenger
> > implementation perspective, that calculus might shift since we're
> > running on a thread pool, but I'm not deeply familiar with the code
> > internals.
>
> I'm afraid that the problem that's going to come up is the peer_addr
> hash_map<> for stateful_peer connections (osd to osd, etc.).  This is down
> in the messenger layer and is the main piece of shared state left (most
> everything else works with Connection refs).  I'm worried that we'll want
> or need to pull that up out of the messenger into the consumer (e.g., a
> hash_map<int,ConnectionRef> osd_peers in OSDService) and refactor the
> connection race handling a bit.  I think that will ultimately solve some
> bugs and will simplify the Messenger greatly, but it's a painful refactor.


You just mean here when we're selecting the correct network to use for
OSD<->OSD communications? Hmm, I'm tempted to say we could do
something pretty simple by setting a priority for the different
addresses, but you're right, there are some issues with reconnect
especially in cases where one network stops working.

>
>
> > But I'm curious what brings this up. I thought you were working on the
> > v2 protocol — does something in that somehow hinge on the server-side
> > implementation of multiple networks?
>
> The v2 protocol will run on another port and we want to answer on the old
> port+protocol for old clients too.


This is a rather narrower case than the above though, right? Because
we don't need to worry about cross-messenger reconnects (the client
will be old and use the old messenger/port, or new and use the new
one) so we can put in a very simple shim.
-Greg

Re: Messenger V2: multiple bind support

[Ricardo Dias]

[-- Attachment #1: Type: text/plain, Size: 2195 bytes --]


> On 1 Nov 2017, at 15:19, Gregory Farnum <gfarnum@redhat.com> wrote:
> 
> 
> There's sort of two pieces here, right?
> 1) From the Dispatcher perspective, we want a single "Messenger" that
> it talks to. (At least, I'm pretty sure. Unless we need to be making
> routing decisions using internal data? I really hope that never comes
> up.) That could be set up by having multiple Messenger implementations
> behind a "UnifyingMessenger" abstraction layer with pretty minimal
> changes.

I agree, with the current code the solution would be to share the same dispatcher
across multiple messenger instances.

> 
> 2) From an implementation perspective, which data structures do we end
> up needing to share in order to have a daemon listening on multiple
> ports?
> 
> So, from the SimpleMessenger perspective, I don't think we get much
> out of trying to build multi-bind into it. We'd basically get to
> eliminate the duplicated DIspatchQueue, which is not very interesting.
> (Modulo some flow control stuff, perhaps.) From an AsyncMessenger
> implementation perspective, that calculus might shift since we're
> running on a thread pool, but I'm not deeply familiar with the code
> internals.
> 
> But I'm curious what brings this up. I thought you were working on the
> v2 protocol — does something in that somehow hinge on the server-side
> implementation of multiple networks?

I remembered this while talking with Joao about why the Mons bind to a single
network interface even if in ceph.conf the public_network value contains multiple
networks.
Then I thought that maybe the mons only bind to a single interface because
it would be too cumbersome to start a single messenger for each network
interface and manage the shared dispatcher. Therefore, since I was already
making changes to the async messenger implementation, I could see if
it was easy to support multiple-bind in the messenger code and make this
functionality available to other modules with minimal code changes.

The V2 protocol implementation does not depend on the multiple bind
functionality, so I would just implement it if people would find it very
useful.

> -Greg


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 529 bytes --]