determining client and server on a connection

determining client and server on a connection

[Jeff Layton]

Hi! I'm just getting started working with ceph, and decided to tackle
fixing up the wireshark dissector which isn't working properly when you
use the kernel's fs client.

This page says that the server always sends its banner first:

    http://docs.ceph.com/docs/master/dev/network-protocol/?highlight=protocol

...but that's not true with the Linux kernel client. The client and
server send their banners and addresses concurrently, and the client
often gets there first. The wireshark dissector relies on the server
sending its banner first however, so it quickly mixes the two up and
things go south from there.

Given the way the protocol works, the only way I can see to reliably
determine client and server is to read enough bytes to get to the
client's address when the server sends it, and see whether it matches
the receiver's address/port.

Is there a simpler way to do this that I'm missing?

Also, it looks like this shouldn't be a problem for the msgr2 protocol
since the initial exchange doesn't involve sending addresses. Is that
the case?

Thanks,
-- 
Jeff Layton <jlayton@redhat.com>

Re: determining client and server on a connection

[Sage Weil]

On Mon, 20 Jun 2016, Jeff Layton wrote:
> Hi! I'm just getting started working with ceph, and decided to tackle
> fixing up the wireshark dissector which isn't working properly when you
> use the kernel's fs client.
> 
> This page says that the server always sends its banner first:
> 
>     http://docs.ceph.com/docs/master/dev/network-protocol/?highlight=protocol
> 
> ...but that's not true with the Linux kernel client. The client and
> server send their banners and addresses concurrently, and the client
> often gets there first. The wireshark dissector relies on the server
> sending its banner first however, so it quickly mixes the two up and
> things go south from there.
> 
> Given the way the protocol works, the only way I can see to reliably
> determine client and server is to read enough bytes to get to the
> client's address when the server sends it, and see whether it matches
> the receiver's address/port.

I'm not sure I follow.  The client is the one initiating the connection 
and the server is the one accepting.  Does wireshark not let you tell 
that?

The addrs are exchanged so that each end can learn what their 
effective address is, but this is a bit of a hack and not really 
ideal--hoping to reduce our reliance on this (or drop it entirely) 
with msgr2.

> Is there a simpler way to do this that I'm missing?
> 
> Also, it looks like this shouldn't be a problem for the msgr2 protocol
> since the initial exchange doesn't involve sending addresses. Is that
> the case?

It's true that it doesn't include the addr exchange.  It doesn't have any 
other explicit indication in the data flow that tells you who is the 
client vs server, though, either...

sage

Re: determining client and server on a connection

[Jeff Layton]

On Mon, 2016-06-20 at 12:21 +0000, Sage Weil wrote:
> On Mon, 20 Jun 2016, Jeff Layton wrote:
> > Hi! I'm just getting started working with ceph, and decided to tackle
> > fixing up the wireshark dissector which isn't working properly when you
> > use the kernel's fs client.
> > 
> > This page says that the server always sends its banner first:
> > 
> >     http://docs.ceph.com/docs/master/dev/network-protocol/?highlight=protocol
> > 
> > ...but that's not true with the Linux kernel client. The client and
> > server send their banners and addresses concurrently, and the client
> > often gets there first. The wireshark dissector relies on the server
> > sending its banner first however, so it quickly mixes the two up and
> > things go south from there.
> > 
> > Given the way the protocol works, the only way I can see to reliably
> > determine client and server is to read enough bytes to get to the
> > client's address when the server sends it, and see whether it matches
> > the receiver's address/port.
> 
> I'm not sure I follow.  The client is the one initiating the connection 
> and the server is the one accepting.  Does wireshark not let you tell 
> that?
> 

I don't think so, at least not that I can tell. I'll double-check
though.

> The addrs are exchanged so that each end can learn what their 
> effective address is, but this is a bit of a hack and not really 
> ideal--hoping to reduce our reliance on this (or drop it entirely) 
> with msgr2.
> 

Good.

If we do need something along those lines, it would be best to make
each peer send the same thing. Right now, the server sends its address
and then the address of the client, but the client only sends its own
address.

An impartial observer that doesn't see the socket connection has no way
to know which end is going to send what. If we had the client and
server both send both addresses (or neither) then that makes things
_much_ simpler for the dissector.

> > Is there a simpler way to do this that I'm missing?
> > 
> > Also, it looks like this shouldn't be a problem for the msgr2 protocol
> > since the initial exchange doesn't involve sending addresses. Is that
> > the case?
> 
> It's true that it doesn't include the addr exchange.  It doesn't have any 
> other explicit indication in the data flow that tells you who is the 
> client vs server, though, either...

If we can avoid sending addresses at all in the initial negotiation,
then I think that takes care of the problem.

-- 
Jeff Layton <jlayton@redhat.com>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: determining client and server on a connection

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2853 bytes --]

On Mon, 20 Jun 2016, Jeff Layton wrote:
> On Mon, 2016-06-20 at 12:21 +0000, Sage Weil wrote:
> > On Mon, 20 Jun 2016, Jeff Layton wrote:
> > > Hi! I'm just getting started working with ceph, and decided to tackle
> > > fixing up the wireshark dissector which isn't working properly when you
> > > use the kernel's fs client.
> > > 
> > > This page says that the server always sends its banner first:
> > > 
> > >     http://docs.ceph.com/docs/master/dev/network-protocol/?highlight=protocol
> > > 
> > > ...but that's not true with the Linux kernel client. The client and
> > > server send their banners and addresses concurrently, and the client
> > > often gets there first. The wireshark dissector relies on the server
> > > sending its banner first however, so it quickly mixes the two up and
> > > things go south from there.
> > > 
> > > Given the way the protocol works, the only way I can see to reliably
> > > determine client and server is to read enough bytes to get to the
> > > client's address when the server sends it, and see whether it matches
> > > the receiver's address/port.
> > 
> > I'm not sure I follow.  The client is the one initiating the connection 
> > and the server is the one accepting.  Does wireshark not let you tell 
> > that?
> > 
> 
> I don't think so, at least not that I can tell. I'll double-check
> though.
> 
> > The addrs are exchanged so that each end can learn what their 
> > effective address is, but this is a bit of a hack and not really 
> > ideal--hoping to reduce our reliance on this (or drop it entirely) 
> > with msgr2.
> > 
> 
> Good.
> 
> If we do need something along those lines, it would be best to make
> each peer send the same thing. Right now, the server sends its address
> and then the address of the client, but the client only sends its own
> address.
> 
> An impartial observer that doesn't see the socket connection has no way
> to know which end is going to send what. If we had the client and
> server both send both addresses (or neither) then that makes things
> _much_ simpler for the dissector.

Let's maybe change teh msgr2 banner to be 'ceph accept %llx %llx' and 
'ceph connect %llx %llx' or similar so that we don't have this problem 
there?

sage

> > > Is there a simpler way to do this that I'm missing?
> > > 
> > > Also, it looks like this shouldn't be a problem for the msgr2 protocol
> > > since the initial exchange doesn't involve sending addresses. Is that
> > > the case?
> > 
> > It's true that it doesn't include the addr exchange.  It doesn't have any 
> > other explicit indication in the data flow that tells you who is the 
> > client vs server, though, either...
> 
> If we can avoid sending addresses at all in the initial negotiation,
> then I think that takes care of the problem.
> 
> -- 
> Jeff Layton <jlayton@redhat.com>
> 
> 

Re: determining client and server on a connection

[Jeff Layton]

On Mon, 2016-06-20 at 12:38 +0000, Sage Weil wrote:
> On Mon, 20 Jun 2016, Jeff Layton wrote:
> > On Mon, 2016-06-20 at 12:21 +0000, Sage Weil wrote:
> > > On Mon, 20 Jun 2016, Jeff Layton wrote:
> > > > Hi! I'm just getting started working with ceph, and decided to tackle
> > > > fixing up the wireshark dissector which isn't working properly when you
> > > > use the kernel's fs client.
> > > > 
> > > > This page says that the server always sends its banner first:
> > > > 
> > > >     http://docs.ceph.com/docs/master/dev/network-protocol/?highlight=protocol
> > > > 
> > > > ...but that's not true with the Linux kernel client. The client and
> > > > server send their banners and addresses concurrently, and the client
> > > > often gets there first. The wireshark dissector relies on the server
> > > > sending its banner first however, so it quickly mixes the two up and
> > > > things go south from there.
> > > > 
> > > > Given the way the protocol works, the only way I can see to reliably
> > > > determine client and server is to read enough bytes to get to the
> > > > client's address when the server sends it, and see whether it matches
> > > > the receiver's address/port.
> > > 
> > > I'm not sure I follow.  The client is the one initiating the connection 
> > > and the server is the one accepting.  Does wireshark not let you tell 
> > > that?
> > > 
> > 
> > I don't think so, at least not that I can tell. I'll double-check
> > though.
> > 
> > > The addrs are exchanged so that each end can learn what their 
> > > effective address is, but this is a bit of a hack and not really 
> > > ideal--hoping to reduce our reliance on this (or drop it entirely) 
> > > with msgr2.
> > > 
> > 
> > Good.
> > 
> > If we do need something along those lines, it would be best to make
> > each peer send the same thing. Right now, the server sends its address
> > and then the address of the client, but the client only sends its own
> > address.
> > 
> > An impartial observer that doesn't see the socket connection has no way
> > to know which end is going to send what. If we had the client and
> > server both send both addresses (or neither) then that makes things
> > _much_ simpler for the dissector.
> 
> Let's maybe change teh msgr2 banner to be 'ceph accept %llx %llx' and 
> 'ceph connect %llx %llx' or similar so that we don't have this problem 
> there?
> 
> sage
> 

Yeah, that'd be fine too.

OTOH, does the connector/acceptor distinction really make any
difference? The only time that wireshark cares is when it's dissecting
the initial negotiation, because the inital message lengths are
different.

I guess it might be nice to know just for informational purposes
though...

> > > > Is there a simpler way to do this that I'm missing?
> > > > 
> > > > Also, it looks like this shouldn't be a problem for the msgr2 protocol
> > > > since the initial exchange doesn't involve sending addresses. Is that
> > > > the case?
> > > 
> > > It's true that it doesn't include the addr exchange.  It doesn't have any 
> > > other explicit indication in the data flow that tells you who is the 
> > > client vs server, though, either...
> > 
> > If we can avoid sending addresses at all in the initial negotiation,
> > then I think that takes care of the problem.
> > 
-- 
Jeff Layton <jlayton@redhat.com>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: determining client and server on a connection

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3062 bytes --]

On Mon, 20 Jun 2016, Jeff Layton wrote:
> On Mon, 2016-06-20 at 12:38 +0000, Sage Weil wrote:
> > On Mon, 20 Jun 2016, Jeff Layton wrote:
> > > On Mon, 2016-06-20 at 12:21 +0000, Sage Weil wrote:
> > > > On Mon, 20 Jun 2016, Jeff Layton wrote:
> > > > > Hi! I'm just getting started working with ceph, and decided to tackle
> > > > > fixing up the wireshark dissector which isn't working properly when you
> > > > > use the kernel's fs client.
> > > > > 
> > > > > This page says that the server always sends its banner first:
> > > > > 
> > > > >     http://docs.ceph.com/docs/master/dev/network-protocol/?highlight=protocol
> > > > > 
> > > > > ...but that's not true with the Linux kernel client. The client and
> > > > > server send their banners and addresses concurrently, and the client
> > > > > often gets there first. The wireshark dissector relies on the server
> > > > > sending its banner first however, so it quickly mixes the two up and
> > > > > things go south from there.
> > > > > 
> > > > > Given the way the protocol works, the only way I can see to reliably
> > > > > determine client and server is to read enough bytes to get to the
> > > > > client's address when the server sends it, and see whether it matches
> > > > > the receiver's address/port.
> > > > 
> > > > I'm not sure I follow.  The client is the one initiating the connection 
> > > > and the server is the one accepting.  Does wireshark not let you tell 
> > > > that?
> > > > 
> > > 
> > > I don't think so, at least not that I can tell. I'll double-check
> > > though.
> > > 
> > > > The addrs are exchanged so that each end can learn what their 
> > > > effective address is, but this is a bit of a hack and not really 
> > > > ideal--hoping to reduce our reliance on this (or drop it entirely) 
> > > > with msgr2.
> > > > 
> > > 
> > > Good.
> > > 
> > > If we do need something along those lines, it would be best to make
> > > each peer send the same thing. Right now, the server sends its address
> > > and then the address of the client, but the client only sends its own
> > > address.
> > > 
> > > An impartial observer that doesn't see the socket connection has no way
> > > to know which end is going to send what. If we had the client and
> > > server both send both addresses (or neither) then that makes things
> > > _much_ simpler for the dissector.
> > 
> > Let's maybe change teh msgr2 banner to be 'ceph accept %llx %llx' and 
> > 'ceph connect %llx %llx' or similar so that we don't have this problem 
> > there?
> > 
> > sage
> > 
> 
> Yeah, that'd be fine too.
> 
> OTOH, does the connector/acceptor distinction really make any
> difference? The only time that wireshark cares is when it's dissecting
> the initial negotiation, because the inital message lengths are
> different.

Yeah, maybe not...
 
> I guess it might be nice to know just for informational purposes
> though...

That's what I'm thinking.  The client vs server behave differently, but 
you'd need to infer it from context.  This way it'd be explicit.

sage

Re: determining client and server on a connection

[Jeff Layton]

On Tue, 2016-06-21 at 00:26 +0800, Haomai Wang wrote:
> 
> 
> On Mon, Jun 20, 2016 at 7:21 PM, Jeff Layton <jlayton@redhat.com> wrote:
> > Hi! I'm just getting started working with ceph, and decided to tackle
> > fixing up the wireshark dissector which isn't working properly when you
> > use the kernel's fs client.
> > 
> > This page says that the server always sends its banner first:
> > 
> >     http://docs.ceph.com/docs/master/dev/network-protocol/?highlight=protocol
> > 
> > ...but that's not true with the Linux kernel client. The client and
> > server send their banners and addresses concurrently, and the client
> > often gets there first. The wireshark dissector relies on the server
> > sending its banner first however, so it quickly mixes the two up and
> > things go south from there.
> Hmm, actually I also modify this behavior in async msgr(https://github.com/ceph/ceph/pull/9802 last commit).
> Because I found there is no real logic difference in ordering. 
>  


Yeah, sending them concurrently does seem like the right thing to do.
There's no reason to serialize them at all, AFAICT. The first draft of
the patch is here:

    https://code.wireshark.org/review/#/c/16043/1

It seems to work for me when sniffing both the kernel and FUSE mounts
now.

The patch is not terribly pretty, but I think we'll have more work to
do in the dissector once msgr2 starts getting closer to being
finalized.

-- 
Jeff Layton <jlayton@redhat.com>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html