Snooping expected connections in a user CT helper

Snooping expected connections in a user CT helper

[Kevin Cernekee]

Hi,

I am trying to extend the ssdp user helper in conntrackd to handle
event subscriptions on a UPnP control point.  The flow looks like
this:

1) Outbound multicast M-SEARCH packet (dst: 1900/udp)
 - Create expectation for unicast reply from <any host> to source port

2) Inbound unicast reply (there may be several of these from different devices)
 - Find the device's URL, e.g.
   LOCATION: http://192.168.1.123:1400/xml/device_description.xml
 - Create expectation to track connections to this host:port (tcp)

3) Outbound connection to device's web server (there will be several of these)
 - Watch for a SUBSCRIBE request
 - Find the control point's callback URL, e.g.
   CALLBACK: <http://192.168.1.124:3500/notify>
 - Create expectation to open up inbound connections to this host:port

4) Inbound connection to control point's web server
 - Once this is complete, the subscription should work

So, all of the port numbers except 1900 are dynamic, and thus each
stage depends on the result of the previous stage.  Therefore I need
the callback to inspect the traffic for stages (1)-(3).

Currently, ssdp_helper_cb() only gets called for stage (1).  Is there
something I can do when creating the expectation to tell netfilter
that I would like to receive callbacks when the unicast reply from (2)
is received?

Alternatively, is there an iptables rule that I should be creating for
this purpose?  I tried this:

    iptables -t raw -I INPUT -m helper --helper ssdp -j CT --helper ssdp

but `-m helper` did not work in the raw table, and `-j CT` did not
work in the filter table.  `-m state --state RELATED` also did not
work in the raw table.

Re: Snooping expected connections in a user CT helper

[Pablo Neira Ayuso]

Hi Kevin,

Let me comment on this to make sure I follow.

On Tue, Aug 16, 2016 at 09:51:00PM -0700, Kevin Cernekee wrote:
> Hi,
> 
> I am trying to extend the ssdp user helper in conntrackd to handle
> event subscriptions on a UPnP control point.  The flow looks like
> this:
> 
> 1) Outbound multicast M-SEARCH packet (dst: 1900/udp)
>  - Create expectation for unicast reply from <any host> to source port

This creates the permanent expectation.

> 2) Inbound unicast reply (there may be several of these from different devices)
>  - Find the device's URL, e.g.
>    LOCATION: http://192.168.1.123:1400/xml/device_description.xml
>  - Create expectation to track connections to this host:port (tcp)

This packet is matching the permanent expectation.

Now you want to attach the ssdp helper to track traffic going to
192.168.1.123:1400.

Looking at ctnetlink, it should be possible to make it via
CTA_EXPECT_HELP_NAME. Thus, by when we find a matching expectation,
the helper is set to this new connection too.

See line 1086 in nf_conntrack_core.c.

> 3) Outbound connection to device's web server (there will be several of these)
>  - Watch for a SUBSCRIBE request
>  - Find the control point's callback URL, e.g.
>    CALLBACK: <http://192.168.1.124:3500/notify>
>  - Create expectation to open up inbound connections to this host:port

This is the packet that goes over 192.168.1.123:1400 and creates an
expectation again, right?

> 4) Inbound connection to control point's web server
>  - Once this is complete, the subscription should work

... and this packet matches the expectation that was created in 3.

> So, all of the port numbers except 1900 are dynamic, and thus each
> stage depends on the result of the previous stage.  Therefore I need
> the callback to inspect the traffic for stages (1)-(3).
> 
> Currently, ssdp_helper_cb() only gets called for stage (1).  Is there
> something I can do when creating the expectation to tell netfilter
> that I would like to receive callbacks when the unicast reply from (2)
> is received?
>
> Alternatively, is there an iptables rule that I should be creating for
> this purpose?  I tried this:
> 
>     iptables -t raw -I INPUT -m helper --helper ssdp -j CT --helper ssdp
> 
> but `-m helper` did not work in the raw table, and `-j CT` did not
> work in the filter table.  `-m state --state RELATED` also did not
> work in the raw table.

You don't need this. It should be possible to set the helper that will
be assigned to the related connection (the one that matches) with the
existing code through the ctnetlink interface.

Re: Snooping expected connections in a user CT helper

[Kevin Cernekee]

On Wed, Aug 17, 2016 at 6:12 PM, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> Looking at ctnetlink, it should be possible to make it via
> CTA_EXPECT_HELP_NAME. Thus, by when we find a matching expectation,
> the helper is set to this new connection too.
>
> See line 1086 in nf_conntrack_core.c.

Thanks, that works.  After setting the helper string, my callback is
invoked for the replies.

I'm running into a couple of other issues with the existing ssdp helper, though:

1) The code does not call cthelper_add_expect().  Nor do some of the
other helpers, such as sane.  When I use a restrictive firewall
configuration, the expectation is never created (according to
`conntrack -L expect`) and all of the incoming SSDP replies are
dropped.  Adding a call to cthelper_add_expect() fixes this.  Do we
know the circumstances under which the current master branch is
expected to work properly?

2) Just noticed that the sane and tftp modules require Linux 3.12+.
My test system is running 3.8.  Does ssdp have a similar restriction,
and if so, what would need to be backported?

3) It looks like each expectation matches, at most, one new
connection.  So if my host multicasts an SSDP request and then 5 other
hosts send replies (each coming from a unique IP/port), only one of
them will match the expectation and create a state table entry.  Is
this true, and if so, what is the best way to allow all 5 replies to
be treated as related connections?

4) Ashley's email address was bouncing due to an overzealous spam
filter.  Will retry one last time.

Re: Snooping expected connections in a user CT helper

[Pablo Neira Ayuso]

On Mon, Aug 22, 2016 at 08:34:41PM -0700, Kevin Cernekee wrote:
> On Wed, Aug 17, 2016 at 6:12 PM, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> > Looking at ctnetlink, it should be possible to make it via
> > CTA_EXPECT_HELP_NAME. Thus, by when we find a matching expectation,
> > the helper is set to this new connection too.
> >
> > See line 1086 in nf_conntrack_core.c.
> 
> Thanks, that works.  After setting the helper string, my callback is
> invoked for the replies.
> 
> I'm running into a couple of other issues with the existing ssdp helper, though:
> 
> 1) The code does not call cthelper_add_expect().  Nor do some of the
> other helpers, such as sane.

If you attach the helper via "myct->exp = exp" then the expectation is
set up from nfqueue path.

cthelper_add_expect() was introduced in first place, so it is an older
way to attach expectations from userspace IIRC.

> When I use a restrictive firewall configuration, the expectation is
> never created (according to `conntrack -L expect`) and all of the
> incoming SSDP replies are dropped.  Adding a call to
> cthelper_add_expect() fixes this.  Do we know the circumstances
> under which the current master branch is expected to work properly?
> 
> 2) Just noticed that the sane and tftp modules require Linux 3.12+.
> My test system is running 3.8.  Does ssdp have a similar restriction,
> and if so, what would need to be backported?

Userspace expectation creation via nfqueue is available since 3.12.
The relevant code is under ctnetlink_nfqueue_attach_expect() in
nf_conntrack_netlink.c, if you want to follow that path, you'll have
to backport, then pull accumulated fixes by tracking my nf.git tree.

I can have a look back and see what needs to be passed to -stable (up
to 3.12) if that makes it easier for you.

> 3) It looks like each expectation matches, at most, one new
> connection.  So if my host multicasts an SSDP request and then 5 other
> hosts send replies (each coming from a unique IP/port), only one of
> them will match the expectation and create a state table entry.  Is
> this true, and if so, what is the best way to allow all 5 replies to
> be treated as related connections?

If you set the permanent expectation flag, the expectation remains
there forever, so all 5 replies will go through as related. Permanent
expectations don't get removed by when we see a matching, they remain
there as long as the master conntrack is there in place.

> 4) Ashley's email address was bouncing due to an overzealous spam
> filter.  Will retry one last time.

OK.

Re: Snooping expected connections in a user CT helper

[Kevin Cernekee]

On Tue, Aug 23, 2016 at 8:36 AM, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>> 2) Just noticed that the sane and tftp modules require Linux 3.12+.
>> My test system is running 3.8.  Does ssdp have a similar restriction,
>> and if so, what would need to be backported?
>
> Userspace expectation creation via nfqueue is available since 3.12.
> The relevant code is under ctnetlink_nfqueue_attach_expect() in
> nf_conntrack_netlink.c, if you want to follow that path, you'll have
> to backport, then pull accumulated fixes by tracking my nf.git tree.
>
> I can have a look back and see what needs to be passed to -stable (up
> to 3.12) if that makes it easier for you.

Thanks for the offer.  I have switched to a different test device
which is running Linux 3.14, and I added a note to the comments in
ssdp.c regarding the 3.12+ requirement.

>> 3) It looks like each expectation matches, at most, one new
>> connection.  So if my host multicasts an SSDP request and then 5 other
>> hosts send replies (each coming from a unique IP/port), only one of
>> them will match the expectation and create a state table entry.  Is
>> this true, and if so, what is the best way to allow all 5 replies to
>> be treated as related connections?
>
> If you set the permanent expectation flag, the expectation remains
> there forever, so all 5 replies will go through as related. Permanent
> expectations don't get removed by when we see a matching, they remain
> there as long as the master conntrack is there in place.

The patch that I sent out last night is able to handle scenarios in
which the event occurs shortly after the subscription is established.
But in my testing I am noticing two other problems:

1) Approximately two minutes after the subscription is set up, the
expectation abruptly disappears.  This even happens if I set the
timeout to 3600; it shows up in `conntrack -L expect` until the time
column drops to ~3480, then it is gone.  This may be caused by the
master conntrack expiring.  Is there a way to set up the expectation
so that it persists for the entire timeout period?

2) The timeout is not extended when there is activity on the
expectation.  It would be good if it was extended any time there is
new activity, in order to support long-lived subscriptions.

Re: Snooping expected connections in a user CT helper

[Kevin Cernekee]

On Thu, Sep 1, 2016 at 4:47 PM, Kevin Cernekee <cernekee@chromium.org> wrote:
> The patch that I sent out last night is able to handle scenarios in
> which the event occurs shortly after the subscription is established.
> But in my testing I am noticing two other problems:
>
> 1) Approximately two minutes after the subscription is set up, the
> expectation abruptly disappears.  This even happens if I set the
> timeout to 3600; it shows up in `conntrack -L expect` until the time
> column drops to ~3480, then it is gone.  This may be caused by the
> master conntrack expiring.  Is there a way to set up the expectation
> so that it persists for the entire timeout period?
>
> 2) The timeout is not extended when there is activity on the
> expectation.  It would be good if it was extended any time there is
> new activity, in order to support long-lived subscriptions.

Friendly ping...

Do you think I'm on the right track with this approach, and if so,
what is the best way to establish long-lived expectations for UPnP?

http://marc.info/?l=netfilter-devel&m=147270416910580&w=2

Thanks.

Re: Snooping expected connections in a user CT helper

[Pablo Neira Ayuso]

Hi Kevin,

On Thu, Sep 08, 2016 at 03:02:13PM -0700, Kevin Cernekee wrote:
> On Thu, Sep 1, 2016 at 4:47 PM, Kevin Cernekee <cernekee@chromium.org> wrote:
> > The patch that I sent out last night is able to handle scenarios in
> > which the event occurs shortly after the subscription is established.
> > But in my testing I am noticing two other problems:
> >
> > 1) Approximately two minutes after the subscription is set up, the
> > expectation abruptly disappears.  This even happens if I set the
> > timeout to 3600; it shows up in `conntrack -L expect` until the time
> > column drops to ~3480, then it is gone.  This may be caused by the
> > master conntrack expiring.  Is there a way to set up the expectation
> > so that it persists for the entire timeout period?

Yes, the expectation goes away if the master is gone. You can set a
larger timeout for the master using -j CT --timeout timeout-policy and
the cttimeout infrastructure.

You have to set the helper and the timeout in one go, ie.

        -j CT --helper ssdp --timeout xyz

> > 2) The timeout is not extended when there is activity on the
> > expectation.  It would be good if it was extended any time there is
> > new activity, in order to support long-lived subscriptions.

Actually, I thought permanent expectation has no timeout, but looking
at the code it seems they do. I think it makes sense to refresh it or
to keep it fixed, given that this depends on the master, it will just
go away once the master is not there anymore.

> Friendly ping...
> 
> Do you think I'm on the right track with this approach, and if so,
> what is the best way to establish long-lived expectations for UPnP?

Yes, although I didn't look your code in deep, but I think you're on
the right track.

BTW, you may also want to explore enabling zero-copy in conntrackd
userspace helper (Eric Dumazet made a patch for nfqueue in 2013).

And you may need to cherry-pick b18bcb0019c to resolve an embarrasing
leak. Fell free to submit this to -stable and keep me on Cc.

Thanks!

Re: Snooping expected connections in a user CT helper

[Pablo Neira Ayuso]

On Fri, Sep 09, 2016 at 12:18:24AM +0200, Pablo Neira Ayuso wrote:
[...]
> > > 2) The timeout is not extended when there is activity on the
> > > expectation.  It would be good if it was extended any time there is
> > > new activity, in order to support long-lived subscriptions.
> 
> Actually, I thought permanent expectation has no timeout, but looking
> at the code it seems they do. I think it makes sense to refresh it or
> to keep it fixed, given that this depends on the master, it will just
> go away once the master is not there anymore.

Revisiting this: Actually, you are in control of the expectation time
too, so you can set a larger one given this is bound by the master
conntrack if you don't want to cook a patch to update this.

The question here is what are the reasonable timeouts to set, I guess
you can extract this from this application protocol behaviour or any
standard that document it.