High accuracy bandwidth accounting?

High accuracy bandwidth accounting?

[Ed W]

Hi, I have a slightly peculiar requirement to track very accurate *per
user* traffic for a small remote userbase.  The internet connections
these users have available will be one or more of: a) circuit switched
satellite phone (ie per second billing), data volume billed (ie GPRS
style) satellite phone or a 3G cell phone - all of these will have non
trivial bandwidth costs and we want to attribute very exact costs back
on a per user basis.

To do this I'm using a small custom built embedded router, and we will
use some form of 802.11x or captive portal style user authentication but
I have two areas I need advice on solving:

1) Best way to do per user traffic accounting *per* internet gateway. ie
each gateway will have quite radically different costs to run and so we
need to also count traffic per route.  My current thinking is to use
packet marking to choose the route and my tests suggest that I can
pickup this mark via conntrack and therefore account using ulogd/pmacct
or similar?  Anyone got any thoughts on other ways to slice this or
anything I am missing?

2) How to account for traffic passing through a "proxy".  eg I want to
run a local DNS resolver, but try to match the external DNS traffic back
to the user that caused it? I think I could probably modify the code of
a suitable resolver to apply a packet mark to upstream data, but I
wasn't able to find how to apply "marks" from userspace applications
from a quick google - can someone point me to a reference?  Does anyone
have any other ideas on how I might do this?  I will also use a couple
of other proxies for http (probably Squid) and email traffic - will need
to apply a similar solution there (perhaps TPROXY with squid?)


Thanks for any ideas.

Ed W

Re: High accuracy bandwidth accounting?

[Andrew Beverley]

On Mon, 2011-05-09 at 15:12 +0100, Ed W wrote:
> Hi, I have a slightly peculiar requirement to track very accurate *per
> user* traffic for a small remote userbase.  The internet connections
> these users have available will be one or more of: a) circuit switched
> satellite phone (ie per second billing), data volume billed (ie GPRS
> style) satellite phone or a 3G cell phone - all of these will have non
> trivial bandwidth costs and we want to attribute very exact costs back
> on a per user basis.
> 
> To do this I'm using a small custom built embedded router, and we will
> use some form of 802.11x or captive portal style user authentication but
> I have two areas I need advice on solving:
> 
> 1) Best way to do per user traffic accounting *per* internet gateway. ie
> each gateway will have quite radically different costs to run and so we
> need to also count traffic per route.  My current thinking is to use
> packet marking to choose the route and my tests suggest that I can
> pickup this mark via conntrack and therefore account using ulogd/pmacct
> or similar?  Anyone got any thoughts on other ways to slice this or
> anything I am missing?

That sounds good. Using marks is a pretty flexible way of achieving most
things. I don't think I fully understand your setup though without a
diagram. How are you identifying individual users within each route? By
IP address?

> 2) How to account for traffic passing through a "proxy".  eg I want to
> run a local DNS resolver, but try to match the external DNS traffic back
> to the user that caused it?

Obviously if you just want to know what traffic is going to/from an IP
address, then you can use the INPUT/OUTPUT chains rather than the
FORWARD chain. I assume that it is not that simple though.

>  I think I could probably modify the code of
> a suitable resolver to apply a packet mark to upstream data, but I
> wasn't able to find how to apply "marks" from userspace applications
> from a quick google - can someone point me to a reference?

Not sure of a reference, but you can use:

setsockopt(fd, SOL_SOCKET, SO_MARK, ...)

Contact the netfilter-dev list for more advice on that.

>   Does anyone
> have any other ideas on how I might do this?  I will also use a couple
> of other proxies for http (probably Squid) and email traffic - will need
> to apply a similar solution there (perhaps TPROXY with squid?)

I wrote a similar patch for Squid (released in V3.2), which allows
packets to be marked before Squid, and Squid to reapply the mark on
retransmission. Marks can also be applied for locally cached files. If
it helps the patch is at:

http://bazaar.launchpad.net/~squid/squid/3-trunk/revision/10925

Andy

Re: High accuracy bandwidth accounting?

[Ed W]

On 09/05/2011 22:45, Andrew Beverley wrote:

> I wrote a similar patch for Squid (released in V3.2), which allows
> packets to be marked before Squid, and Squid to reapply the mark on
> retransmission. Marks can also be applied for locally cached files. If
> it helps the patch is at:
> 
> http://bazaar.launchpad.net/~squid/squid/3-trunk/revision/10925

Well that just happens to be a perfect example of how to implement that
mark copying for my various proxies!  Thanks

Ed W

Re: High accuracy bandwidth accounting?

[Andrew Beverley]

On Mon, 2011-05-09 at 23:07 +0100, Ed W wrote:
> On 09/05/2011 22:45, Andrew Beverley wrote:
> 
> > I wrote a similar patch for Squid (released in V3.2), which allows
> > packets to be marked before Squid, and Squid to reapply the mark on
> > retransmission. Marks can also be applied for locally cached files. If
> > it helps the patch is at:
> > 
> > http://bazaar.launchpad.net/~squid/squid/3-trunk/revision/10925
> 
> Well that just happens to be a perfect example of how to implement that
> mark copying for my various proxies!  Thanks
> 

No problem. If you write any patches for publicly available software
then please get them included upstream - would be very useful!

Andy

Re: High accuracy bandwidth accounting?

[Ed W]

On 09/05/2011 22:45, Andrew Beverley wrote:
> On Mon, 2011-05-09 at 15:12 +0100, Ed W wrote:
>> Hi, I have a slightly peculiar requirement to track very accurate *per
>> user* traffic for a small remote userbase.  The internet connections
>> these users have available will be one or more of: a) circuit switched
>> satellite phone (ie per second billing), data volume billed (ie GPRS
>> style) satellite phone or a 3G cell phone - all of these will have non
>> trivial bandwidth costs and we want to attribute very exact costs back
>> on a per user basis.
>>
>> To do this I'm using a small custom built embedded router, and we will
>> use some form of 802.11x or captive portal style user authentication but
>> I have two areas I need advice on solving:
>>
>> 1) Best way to do per user traffic accounting *per* internet gateway. ie
>> each gateway will have quite radically different costs to run and so we
>> need to also count traffic per route.  My current thinking is to use
>> packet marking to choose the route and my tests suggest that I can
>> pickup this mark via conntrack and therefore account using ulogd/pmacct
>> or similar?  Anyone got any thoughts on other ways to slice this or
>> anything I am missing?
> 
> That sounds good. Using marks is a pretty flexible way of achieving most
> things. I don't think I fully understand your setup though without a
> diagram. How are you identifying individual users within each route? By
> IP address?

The "per user" part is still under experimentation.  The main option
seem to be some kind of authentication which then marks a MAC/IP combo
as "authenticated" and we track traffic to that device (being aware of
the limitations of that). However, if I use 802.11x auth then I appear
to get my traffic automatically put into it's own vlan - this might be
interesting, but vlans don't appear to give me a whole lot of options to
filter within iptables?

The basic scenario is a small number of guys in a remote location with
only a satellite connection to the internet.  Lets imagine they are on a
tanker steaming across the Pacific say.  The crew in this scenario might
be "sold" or otherwise allocated a quantity of data that they can use,
where the owner of the satellite connection will be paying between
$10-100/MB of data (it actually tends to be one end or the other
depending on the system used - fairly pricey anyway).  However,
occasionally the unit will be within reach of an alternative connection
such as cell phone range - at that point we want to switch over to a
cheaper circuit and adjust our billing.

Essentially it's a captive portal scenario, but most captive portals
have the luxury of billing based on all traffic crossing the internal
network, whereas we want to track usage only across the expensive link
(and track each link separately).

Suggestions and comments welcome!

Ed W

Re: High accuracy bandwidth accounting?

[Ed W]

On 09/05/2011 22:45, Andrew Beverley wrote:
> I wrote a similar patch for Squid (released in V3.2), which allows
> packets to be marked before Squid, and Squid to reapply the mark on
> retransmission. Marks can also be applied for locally cached files. If
> it helps the patch is at:
> 
> http://bazaar.launchpad.net/~squid/squid/3-trunk/revision/10925


Did you find any better way to read the packet mark than using
conntrack?  I see that's how you are doing it in the patch you reference?

Thanks

Ed W

Re: High accuracy bandwidth accounting?

[Andrew Beverley]

On Wed, 2011-05-11 at 15:30 +0100, Ed W wrote:
> On 09/05/2011 22:45, Andrew Beverley wrote:
> > I wrote a similar patch for Squid (released in V3.2), which allows
> > packets to be marked before Squid, and Squid to reapply the mark on
> > retransmission. Marks can also be applied for locally cached files. If
> > it helps the patch is at:
> > 
> > http://bazaar.launchpad.net/~squid/squid/3-trunk/revision/10925
> 
> 
> Did you find any better way to read the packet mark than using
> conntrack?  I see that's how you are doing it in the patch you reference?
> 

No, I think that's the only/best way of doing it (you can't read it from
a socket). That's the advice I got from the netfilter developers anyway,
and it doesn't get much better than that.

Andy

Re: High accuracy bandwidth accounting?

[Ed W]

On 12/05/2011 01:01, Andrew Beverley wrote:
> On Wed, 2011-05-11 at 15:30 +0100, Ed W wrote:
>> On 09/05/2011 22:45, Andrew Beverley wrote:
>>> I wrote a similar patch for Squid (released in V3.2), which allows
>>> packets to be marked before Squid, and Squid to reapply the mark on
>>> retransmission. Marks can also be applied for locally cached files. If
>>> it helps the patch is at:
>>>
>>> http://bazaar.launchpad.net/~squid/squid/3-trunk/revision/10925
>>
>>
>> Did you find any better way to read the packet mark than using
>> conntrack?  I see that's how you are doing it in the patch you reference?
>>
> 
> No, I think that's the only/best way of doing it (you can't read it from
> a socket). That's the advice I got from the netfilter developers anyway,
> and it doesn't get much better than that.

I guess a feature request that the nf_mark is copied down to the "socket
mark" for arriving packets is going to be met with a "lets see your
patch"?  Did you do any investigation to see where such code might be
fitted - or even if it's a good idea?

Cheers

Ed W

Re: High accuracy bandwidth accounting?

[Andrew Beverley]

On Thu, 2011-05-12 at 23:17 +0100, Ed W wrote:
> On 12/05/2011 01:01, Andrew Beverley wrote:
> > On Wed, 2011-05-11 at 15:30 +0100, Ed W wrote:
> >> On 09/05/2011 22:45, Andrew Beverley wrote:
> >>> I wrote a similar patch for Squid (released in V3.2), which allows
> >>> packets to be marked before Squid, and Squid to reapply the mark on
> >>> retransmission. Marks can also be applied for locally cached files. If
> >>> it helps the patch is at:
> >>>
> >>> http://bazaar.launchpad.net/~squid/squid/3-trunk/revision/10925
> >>
> >>
> >> Did you find any better way to read the packet mark than using
> >> conntrack?  I see that's how you are doing it in the patch you reference?
> >>
> > 
> > No, I think that's the only/best way of doing it (you can't read it from
> > a socket). That's the advice I got from the netfilter developers anyway,
> > and it doesn't get much better than that.
> 
> I guess a feature request that the nf_mark is copied down to the "socket
> mark" for arriving packets is going to be met with a "lets see your
> patch"?  Did you do any investigation to see where such code might be
> fitted - or even if it's a good idea?

I'm afraid I've got no idea. That patch is the only bit of network
programming I have done, and the principle behind getting the mark was
suggested by Jan on the developer's list.

Probably worth a quick question to that effect to the netfilter-dev list
though.

Andy

Re: High accuracy bandwidth accounting?

[Ed W]

On 09/05/2011 15:12, Ed W wrote:
> 1) Best way to do per user traffic accounting *per* internet gateway. ie
> each gateway will have quite radically different costs to run and so we
> need to also count traffic per route.  My current thinking is to use
> packet marking to choose the route and my tests suggest that I can
> pickup this mark via conntrack and therefore account using ulogd/pmacct
> or similar?  Anyone got any thoughts on other ways to slice this or
> anything I am missing?

I have been experimenting with logging conntrack flows using ulogd2.  However, 
the numbers aren't stacking up against a simple iptables accounting rule...  
Is there an expectation that nf_conntrack should log every single byte in/out?

An example seems to be to cause a name lookup via dnsmasq. For whatever reason 
this does two simultaneous dns requests to both configured dns servers.  One 
reply comes back slightly quicker than the other and the slower reply appears 
to cause a local ICMP unreachable response to be generated.  Everything is 
logged *except* the data for the ICMP unreachable response?

So tcpdump gives me (note sizes are payload, add 28 to compare with conntrack):

22:23:40.151666 IP 10.141.36.76.25630 > 8.8.4.4.53: 11049+ AAAA? www.yahoo.co.uk. (33)
22:23:40.151993 IP 10.141.36.76.25630 > 8.8.8.8.53: 11049+ AAAA? www.yahoo.co.uk. (33)
22:23:40.850776 IP 8.8.4.4.53 > 10.141.36.76.25630: 11049 3/1/0 CNAME rc.yahoo.com., CNAME rc.g01.yahoodns.net., CNAME any-rc.a01.yahoodns.net. (178)
22:23:41.014108 IP 8.8.8.8.53 > 10.141.36.76.25630: 11049 3/1/0 CNAME rc.yahoo.com., CNAME rc.g01.yahoodns.net., CNAME any-rc.a01.yahoodns.net. (178)

22:23:41.014217 IP 10.141.36.76 > 8.8.8.8: ICMP 10.141.36.76 udp port 25630 unreachable, length 214

22:23:41.401743 IP 10.141.36.76.10248 > 8.8.4.4.53: 25285+ A? www.yahoo.co.uk. (33)
22:23:41.764124 IP 8.8.4.4.53 > 10.141.36.76.10248: 25285 4/0/0 CNAME rc.yahoo.com., CNAME rc.g01.yahoodns.net., CNAME any-rc.a01.yahoodns.net., A 77.238.178.122 (133)


However, conntrack gives me:

May  9 22:24:10 localhost [DESTROY] ORIG: SRC=10.141.36.76 DST=8.8.4.4 PROTO=UDP SPT=25630 DPT=53 PKTS=1 BYTES=61 , REPLY: SRC=8.8.4.4 DST=10.141.36.76 PROTO=UDP SPT=53 DPT=25630 PKTS=1 BYTES=206 
May  9 22:24:10 localhost [DESTROY] ORIG: SRC=10.141.36.76 DST=8.8.8.8 PROTO=UDP SPT=25630 DPT=53 PKTS=1 BYTES=61 , REPLY: SRC=8.8.8.8 DST=10.141.36.76 PROTO=UDP SPT=53 DPT=25630 PKTS=1 BYTES=206 
May  9 22:24:11 localhost [DESTROY] ORIG: SRC=10.141.36.76 DST=8.8.4.4 PROTO=UDP SPT=10248 DPT=53 PKTS=1 BYTES=61 , REPLY: SRC=8.8.4.4 DST=10.141.36.76 PROTO=UDP SPT=53 DPT=10248 PKTS=1 BYTES=161 


Basically it's missing any count for the packet with tcpdump timestamp: 22:23:41.014217 - ie the port unreachable response?  This is confirmed by looking at the iptables counts


Disregarding whether this is good protocol use or not, I need to track to byte accurate levels - do I therefore need to avoid conntrack based accounting?  Or is this a "bug" in conntrack or in ulogd?  Conntrack based accounting has lured me in because it seems to do a lot of the heavy lifting for me - would be nice if this turned out to be fixable?

Anyone know of any other reasons that conntrack accounting might not count some traffic? (corrupted packets? tcp resends?)

Just trying to get the base architecture correct from the start!

Thanks

Ed W

Re: High accuracy bandwidth accounting?

[Andrew Beverley]

On Tue, 2011-05-10 at 00:23 +0100, Ed W wrote:
> On 09/05/2011 15:12, Ed W wrote:
> > 1) Best way to do per user traffic accounting *per* internet gateway. ie
> > each gateway will have quite radically different costs to run and so we
> > need to also count traffic per route.  My current thinking is to use
> > packet marking to choose the route and my tests suggest that I can
> > pickup this mark via conntrack and therefore account using ulogd/pmacct
> > or similar?  Anyone got any thoughts on other ways to slice this or
> > anything I am missing?
> 
> I have been experimenting with logging conntrack flows using ulogd2.  However, 
> the numbers aren't stacking up against a simple iptables accounting rule...  
> Is there an expectation that nf_conntrack should log every single byte in/out?
> 
> An example seems to be to cause a name lookup via dnsmasq. For whatever reason 
> this does two simultaneous dns requests to both configured dns servers.  One 
> reply comes back slightly quicker than the other and the slower reply appears 
> to cause a local ICMP unreachable response to be generated.  Everything is 
> logged *except* the data for the ICMP unreachable response?
> 
> So tcpdump gives me (note sizes are payload, add 28 to compare with conntrack):
> 
> 22:23:40.151666 IP 10.141.36.76.25630 > 8.8.4.4.53: 11049+ AAAA? www.yahoo.co.uk. (33)
> 22:23:40.151993 IP 10.141.36.76.25630 > 8.8.8.8.53: 11049+ AAAA? www.yahoo.co.uk. (33)
> 22:23:40.850776 IP 8.8.4.4.53 > 10.141.36.76.25630: 11049 3/1/0 CNAME rc.yahoo.com., CNAME rc.g01.yahoodns.net., CNAME any-rc.a01.yahoodns.net. (178)
> 22:23:41.014108 IP 8.8.8.8.53 > 10.141.36.76.25630: 11049 3/1/0 CNAME rc.yahoo.com., CNAME rc.g01.yahoodns.net., CNAME any-rc.a01.yahoodns.net. (178)
> 
> 22:23:41.014217 IP 10.141.36.76 > 8.8.8.8: ICMP 10.141.36.76 udp port 25630 unreachable, length 214
> 
> 22:23:41.401743 IP 10.141.36.76.10248 > 8.8.4.4.53: 25285+ A? www.yahoo.co.uk. (33)
> 22:23:41.764124 IP 8.8.4.4.53 > 10.141.36.76.10248: 25285 4/0/0 CNAME rc.yahoo.com., CNAME rc.g01.yahoodns.net., CNAME any-rc.a01.yahoodns.net., A 77.238.178.122 (133)
> 
> 
> However, conntrack gives me:
> 
> May  9 22:24:10 localhost [DESTROY] ORIG: SRC=10.141.36.76 DST=8.8.4.4 PROTO=UDP SPT=25630 DPT=53 PKTS=1 BYTES=61 , REPLY: SRC=8.8.4.4 DST=10.141.36.76 PROTO=UDP SPT=53 DPT=25630 PKTS=1 BYTES=206 
> May  9 22:24:10 localhost [DESTROY] ORIG: SRC=10.141.36.76 DST=8.8.8.8 PROTO=UDP SPT=25630 DPT=53 PKTS=1 BYTES=61 , REPLY: SRC=8.8.8.8 DST=10.141.36.76 PROTO=UDP SPT=53 DPT=25630 PKTS=1 BYTES=206 
> May  9 22:24:11 localhost [DESTROY] ORIG: SRC=10.141.36.76 DST=8.8.4.4 PROTO=UDP SPT=10248 DPT=53 PKTS=1 BYTES=61 , REPLY: SRC=8.8.4.4 DST=10.141.36.76 PROTO=UDP SPT=53 DPT=10248 PKTS=1 BYTES=161 
> 
> 
> Basically it's missing any count for the packet with tcpdump timestamp: 22:23:41.014217 - ie the port unreachable response?  This is confirmed by looking at the iptables counts
> 
> 
> Disregarding whether this is good protocol use or not, I need to track to byte accurate levels - do I therefore need to avoid conntrack based accounting?  Or is this a "bug" in conntrack or in ulogd?  Conntrack based accounting has lured me in because it seems to do a lot of the heavy lifting for me - would be nice if this turned out to be fixable?
> 
> Anyone know of any other reasons that conntrack accounting might not count some traffic? (corrupted packets? tcp resends?)
> 
> Just trying to get the base architecture correct from the start!
> 

Sorry, only just got round to replying. Did you get anywhere with this?

Have you tried investigating whether the ICMP packets in question are
making it into any of the conntrack system? Maybe by using the userspace
program, or my using the conntrack match extension in iptables?

Andy

Re: High accuracy bandwidth accounting?

[Ed W]

>> I have been experimenting with logging conntrack flows using ulogd2.  However, 
>> the numbers aren't stacking up against a simple iptables accounting rule...  
>> Is there an expectation that nf_conntrack should log every single byte in/out?
>>
>> An example seems to be to cause a name lookup via dnsmasq. For whatever reason 
>> this does two simultaneous dns requests to both configured dns servers.  One 
>> reply comes back slightly quicker than the other and the slower reply appears 
>> to cause a local ICMP unreachable response to be generated.  Everything is 
>> logged *except* the data for the ICMP unreachable response?
>>
>> So tcpdump gives me (note sizes are payload, add 28 to compare with conntrack):
>>
>> 22:23:40.151666 IP 10.141.36.76.25630 > 8.8.4.4.53: 11049+ AAAA? www.yahoo.co.uk. (33)
>> 22:23:40.151993 IP 10.141.36.76.25630 > 8.8.8.8.53: 11049+ AAAA? www.yahoo.co.uk. (33)
>> 22:23:40.850776 IP 8.8.4.4.53 > 10.141.36.76.25630: 11049 3/1/0 CNAME rc.yahoo.com., CNAME rc.g01.yahoodns.net., CNAME any-rc.a01.yahoodns.net. (178)
>> 22:23:41.014108 IP 8.8.8.8.53 > 10.141.36.76.25630: 11049 3/1/0 CNAME rc.yahoo.com., CNAME rc.g01.yahoodns.net., CNAME any-rc.a01.yahoodns.net. (178)
>>
>> 22:23:41.014217 IP 10.141.36.76 > 8.8.8.8: ICMP 10.141.36.76 udp port 25630 unreachable, length 214
>>
>> 22:23:41.401743 IP 10.141.36.76.10248 > 8.8.4.4.53: 25285+ A? www.yahoo.co.uk. (33)
>> 22:23:41.764124 IP 8.8.4.4.53 > 10.141.36.76.10248: 25285 4/0/0 CNAME rc.yahoo.com., CNAME rc.g01.yahoodns.net., CNAME any-rc.a01.yahoodns.net., A 77.238.178.122 (133)
>>
>>
>> However, conntrack gives me:
>>
>> May  9 22:24:10 localhost [DESTROY] ORIG: SRC=10.141.36.76 DST=8.8.4.4 PROTO=UDP SPT=25630 DPT=53 PKTS=1 BYTES=61 , REPLY: SRC=8.8.4.4 DST=10.141.36.76 PROTO=UDP SPT=53 DPT=25630 PKTS=1 BYTES=206 
>> May  9 22:24:10 localhost [DESTROY] ORIG: SRC=10.141.36.76 DST=8.8.8.8 PROTO=UDP SPT=25630 DPT=53 PKTS=1 BYTES=61 , REPLY: SRC=8.8.8.8 DST=10.141.36.76 PROTO=UDP SPT=53 DPT=25630 PKTS=1 BYTES=206 
>> May  9 22:24:11 localhost [DESTROY] ORIG: SRC=10.141.36.76 DST=8.8.4.4 PROTO=UDP SPT=10248 DPT=53 PKTS=1 BYTES=61 , REPLY: SRC=8.8.4.4 DST=10.141.36.76 PROTO=UDP SPT=53 DPT=10248 PKTS=1 BYTES=161 
>>
>>
>> Basically it's missing any count for the packet with tcpdump timestamp: 22:23:41.014217 - ie the port unreachable response?  This is confirmed by looking at the iptables counts

> 
> Have you tried investigating whether the ICMP packets in question are
> making it into any of the conntrack system? Maybe by using the userspace
> program, or my using the conntrack match extension in iptables?

I think that these UDP packets are completely missing the entire
conntrack system?

I am using ulogd2 to dump all the connections and the ones above are all
that I see.  Therefore either there is a bug in ulogd2 which isn't
spitting out this connection or conntrack completely misses this packet?
(arguably whether should it be applied to this connection or not mind?)

I think you are leaning towards confirming this is unexpected and could
be reported as a "bug"?  I wrote up the above on the -devel list but
didn't attract any comments so far

It would be interesting to hear if this is a "bug" or expected?

Cheers

Ed W

Re: High accuracy bandwidth accounting?

[Andrew Beverley]

On Sat, 2011-05-14 at 14:36 +0100, Ed W wrote:
> >>
> >> So tcpdump gives me (note sizes are payload, add 28 to compare with conntrack):
> >>
> >> 22:23:41.014217 IP 10.141.36.76 > 8.8.8.8: ICMP 10.141.36.76 udp port 25630 unreachable, length 214
> >>
> >>
> > 
> > Have you tried investigating whether the ICMP packets in question are
> > making it into any of the conntrack system? Maybe by using the userspace
> > program, or my using the conntrack match extension in iptables?
> 
> I think that these UDP packets are completely missing the entire
> conntrack system?
> 

Okay, I've played around with this myself using a similar scenario. It
looks to me like the packets *are* making it into the conntrack system.

I tried setting a LOG target to match those packets with a ctstate of
RELATED:

iptables -A INPUT -p ICMP -m conntrack --ctstate RELATED -j LOG

And they were indeed logged. But there was no visibility of them using
the conntrack userspace program. This says to me that conntrack does
know about them, but it doesn't do anything with them, as the packets
are basically saying "no thanks" to a connection request.

I guess you could argue that the packet counters should be updated
regardless, although bear in mind that the ICMP packets are related to
the original UDP connection, so any update should happen to that
connection. You wouldn't see them as a separate connection.

> 
> I am using ulogd2 to dump all the connections and the ones above are all
> that I see.  Therefore either there is a bug in ulogd2 which isn't
> spitting out this connection or conntrack completely misses this packet?
> (arguably whether should it be applied to this connection or not mind?)

Your last comment there is the key. The ICMP packets are coming in,
RELATED to the UDP connection, but are not being applied to it, arguably
because of what they are saying.

> 
> I think you are leaning towards confirming this is unexpected and could
> be reported as a "bug"?

I wouldn't like to say whether it is a bug really...

>   I wrote up the above on the -devel list but
> didn't attract any comments so far

It might be worth trying again with a different question, along the
lines of "can conntrack be changed to count ICMP unreachable packets
related to an existing connection". If you're feeling brave, you could
have a poke around in the source code.

Andy

Re: High accuracy bandwidth accounting?

[Jan Engelhardt]

On Saturday 2011-05-14 18:29, Andrew Beverley wrote:

>On Sat, 2011-05-14 at 14:36 +0100, Ed W wrote:
>
>Okay, I've played around with this myself using a similar scenario. It
>looks to me like the packets *are* making it into the conntrack system.
>
>I tried setting a LOG target to match those packets with a ctstate of
>RELATED:
>
>iptables -A INPUT -p ICMP -m conntrack --ctstate RELATED -j LOG
>
>And they were indeed logged. But there was no visibility of them using
>the conntrack userspace program.

Does `conntrack -L` show anything for you at all? There is/was a 
short intermediate time when it would not return anything.

Re: High accuracy bandwidth accounting?

[Andrew Beverley]

On Sun, 2011-05-15 at 00:33 +0200, Jan Engelhardt wrote:
> On Saturday 2011-05-14 18:29, Andrew Beverley wrote:
> 
> >On Sat, 2011-05-14 at 14:36 +0100, Ed W wrote:
> >
> >Okay, I've played around with this myself using a similar scenario. It
> >looks to me like the packets *are* making it into the conntrack system.
> >
> >I tried setting a LOG target to match those packets with a ctstate of
> >RELATED:
> >
> >iptables -A INPUT -p ICMP -m conntrack --ctstate RELATED -j LOG
> >
> >And they were indeed logged. But there was no visibility of them using
> >the conntrack userspace program.
> 
> Does `conntrack -L` show anything for you at all? 

Yes, it shows the outgoing packets:

udp      17 23 src=10.0.10.206 dst=212.110.185.119 sport=35259 dport=53
packets=3 bytes=168 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206
sport=53 dport=35259 packets=0 bytes=0 mark=0 secmark=0 use=2

But it doesn't show the "ICMP port unreachable" packets that are sent in
reply. The question is: should it show them?

> There is/was a 
> short intermediate time when it would not return anything.

Do you mean there was a time when a particular version of conntrack
would not return anything?

Andy

Re: High accuracy bandwidth accounting?

[Jan Engelhardt]

On Sunday 2011-05-15 09:23, Andrew Beverley wrote:

>On Sun, 2011-05-15 at 00:33 +0200, Jan Engelhardt wrote:
>> On Saturday 2011-05-14 18:29, Andrew Beverley wrote:
>> 
>> >On Sat, 2011-05-14 at 14:36 +0100, Ed W wrote:
>> >
>> >Okay, I've played around with this myself using a similar scenario. It
>> >looks to me like the packets *are* making it into the conntrack system.
>> >
>> >I tried setting a LOG target to match those packets with a ctstate of
>> >RELATED:
>> >
>> >iptables -A INPUT -p ICMP -m conntrack --ctstate RELATED -j LOG
>> >
>> >And they were indeed logged. But there was no visibility of them using
>> >the conntrack userspace program.
>> 
>> Does `conntrack -L` show anything for you at all? 
>
>Yes, it shows the outgoing packets:
>
>udp      17 23 src=10.0.10.206 dst=212.110.185.119 sport=35259 dport=53
>packets=3 bytes=168 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206
>sport=53 dport=35259 packets=0 bytes=0 mark=0 secmark=0 use=2
>
>But it doesn't show the "ICMP port unreachable" packets that are sent in
>reply. The question is: should it show them?

conntrack -L shows pseudo/NFCT-style connections, not packets. As for 
ICMP port unreachable, either its classification is
- INVALID, in which case there is no CT to show
- a reply, in which case it is part of the shown CT - and the event 
monitor will show an UPDATE
- RELATED, in which case a new CT is shown - and which may disappear 
shortly after due to a short lifetime - the event monitor should show 
NEW I think.

Monitor with conntrack -E for details.

>> There is/was a 
>> short intermediate time when it would not return anything.
>
>Do you mean there was a time when a particular version of conntrack
>would not return anything?

Yes, see kernel commit cba85b532e4aabdb97f44c18987d45141fd93faa that 
(the fix) for details.

Re: High accuracy bandwidth accounting?

[Andrew Beverley]

On Sun, 2011-05-15 at 11:08 +0200, Jan Engelhardt wrote: 
> On Sunday 2011-05-15 09:23, Andrew Beverley wrote:
> 
> >On Sun, 2011-05-15 at 00:33 +0200, Jan Engelhardt wrote:
> >> On Saturday 2011-05-14 18:29, Andrew Beverley wrote:
> >> 
> >> >On Sat, 2011-05-14 at 14:36 +0100, Ed W wrote:
> >> >
> >> >Okay, I've played around with this myself using a similar scenario. It
> >> >looks to me like the packets *are* making it into the conntrack system.
> >> >
> >> >I tried setting a LOG target to match those packets with a ctstate of
> >> >RELATED:
> >> >
> >> >iptables -A INPUT -p ICMP -m conntrack --ctstate RELATED -j LOG
> >> >
> >> >And they were indeed logged. But there was no visibility of them using
> >> >the conntrack userspace program.
> >> 
> >> Does `conntrack -L` show anything for you at all? 
> >
> >Yes, it shows the outgoing packets:
> >
> >udp      17 23 src=10.0.10.206 dst=212.110.185.119 sport=35259 dport=53
> >packets=3 bytes=168 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206
> >sport=53 dport=35259 packets=0 bytes=0 mark=0 secmark=0 use=2
> >
> >But it doesn't show the "ICMP port unreachable" packets that are sent in
> >reply. The question is: should it show them?
> 
> conntrack -L shows pseudo/NFCT-style connections, not packets.

Sorry, when I say it doesn't show them, I mean they are not counted. The
conntrack entry for that connection only shows a packet count in one
direction. It shows zero in the other direction.

> As for 
> ICMP port unreachable, either its classification is
> - INVALID, in which case there is no CT to show
> - a reply, in which case it is part of the shown CT - and the event 
> monitor will show an UPDATE
> - RELATED, in which case a new CT is shown - and which may disappear 
> shortly after due to a short lifetime - the event monitor should show 
> NEW I think.

The ones in this case are coming in as RELATED.

> 
> Monitor with conntrack -E for details.
> 

conntrack -E -d 212.110.185.119 gives:

    [NEW] udp      17 30 src=10.0.10.206 dst=212.110.185.119 sport=35676 dport=53 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206 sport=53 dport=35676
[DESTROY] udp      17 src=10.0.10.206 dst=212.110.185.119 sport=35676 dport=53 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206 sport=53 dport=35676

But conntrack -E -s 212.110.185.119 gives nothing

So they do not seem to be counted anywhere.

> >> There is/was a 
> >> short intermediate time when it would not return anything.
> >
> >Do you mean there was a time when a particular version of conntrack
> >would not return anything?
> 
> Yes, see kernel commit cba85b532e4aabdb97f44c18987d45141fd93faa that 
> (the fix) for details.

I upgraded the kernel to 2.6.38.6 just in case, but the same still
happens.

Thanks,

Andy

Re: High accuracy bandwidth accounting?

[Jan Engelhardt]

On Monday 2011-05-16 08:43, Andrew Beverley wrote:
>> >
>> >Yes, it shows the outgoing packets:
>> >
>> >udp      17 23 src=10.0.10.206 dst=212.110.185.119 sport=35259 dport=53
>> >packets=3 bytes=168 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206
>> >sport=53 dport=35259 packets=0 bytes=0 mark=0 secmark=0 use=2
>> >
>> >But it doesn't show the "ICMP port unreachable" packets that are sent in
>> >reply. The question is: should it show them?
>
>Sorry, when I say it doesn't show them, I mean they are not counted.

They are:

>The ones in this case are coming in as RELATED.
>> 
>> Monitor with conntrack -E for details.
>> 
>conntrack -E -d 212.110.185.119 gives:
>
>    [NEW] udp      17 30 src=10.0.10.206 dst=212.110.185.119 sport=35676 dport=53 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206 sport=53 dport=35676

Since it is a new/related CT (with sport=35676), the packets won't be counted
towards the original ct (sport=35259).

>[DESTROY] udp      17 src=10.0.10.206 dst=212.110.185.119 sport=35676 dport=53 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206 sport=53 dport=35676

I observe that only UNREPLIED ones carry no counters - and that such is a
minority of CT events - probably hinting to nonresponsive servers.
 
>But conntrack -E -s 212.110.185.119 gives nothing

No connection was initiated (=packet first seen) from 212.110.185.119.
Note that -s filters on an NFCT tuple's address, not packet address.

Re: High accuracy bandwidth accounting?

[Andrew Beverley]

Sorry, a bit late replying...

On Mon, 2011-05-16 at 09:23 +0200, Jan Engelhardt wrote: 
> On Monday 2011-05-16 08:43, Andrew Beverley wrote:
> >> >
> >> >Yes, it shows the outgoing packets:
> >> >
> >> >udp      17 23 src=10.0.10.206 dst=212.110.185.119 sport=35259 dport=53
> >> >packets=3 bytes=168 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206
> >> >sport=53 dport=35259 packets=0 bytes=0 mark=0 secmark=0 use=2
> >> >
> >> >But it doesn't show the "ICMP port unreachable" packets that are sent in
> >> >reply. The question is: should it show them?
> >
> >Sorry, when I say it doesn't show them, I mean they are not counted.
> 
> They are:
> 
> >The ones in this case are coming in as RELATED.
> >> 
> >> Monitor with conntrack -E for details.
> >> 
> >conntrack -E -d 212.110.185.119 gives:
> >
> >    [NEW] udp      17 30 src=10.0.10.206 dst=212.110.185.119 sport=35676 dport=53 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206 sport=53 dport=35676
> 
> Since it is a new/related CT (with sport=35676), the packets won't be counted
> towards the original ct (sport=35259).

I didn't make myself clear. The 2 different connections (and 2 different
source ports) were 2 separate tests, but exhibiting the same results. 

> 
> >[DESTROY] udp      17 src=10.0.10.206 dst=212.110.185.119 sport=35676 dport=53 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206 sport=53 dport=35676
> 
> I observe that only UNREPLIED ones carry no counters - and that such is a
> minority of CT events - probably hinting to nonresponsive servers.
>  

The server responds, but with "ICMP port unreachable".

> >But conntrack -E -s 212.110.185.119 gives nothing
> 
> No connection was initiated (=packet first seen) from 212.110.185.119.

Correct, but an "ICMP port unreachable" was sent from 212.110.185.119.

To repeat the test:

iptables -A INPUT -p ICMP -m conntrack --ctstate RELATED -j LOG

dig google.com @212.110.185.119

Logged in the system logger:

[354578.512848] IN=wlan0 OUT= MAC=... SRC=212.110.185.119
DST=10.0.10.206 LEN=84 TOS=0x00 PREC=0x00 TTL=54 ID=36992 PROTO=ICMP
TYPE=3 CODE=3 [SRC=10.0.10.206 DST=212.110.185.119 LEN=56 TOS=0x00
PREC=0x00 TTL=53 ID=17592 PROTO=UDP SPT=53163 DPT=53 LEN=36 ] 

The only thing shown in conntrack -E is:

[NEW] udp      17 30 src=10.0.10.206 dst=212.110.185.119 sport=53163
dport=53 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206 sport=53
dport=53163

And conntrack -L | grep 212.110.185.119 shows (in completion):

udp      17 22 src=10.0.10.206 dst=212.110.185.119 sport=53163 dport=53
packets=3 bytes=168 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206
sport=53 dport=53163 packets=0 bytes=0 mark=0 use=1

So, the ICMP packets arrive as RELATED, but are not accounted for
looking at the conntrack accounting.

Andy

Re: High accuracy bandwidth accounting?

[Ed W]

On 15/05/2011 10:08, Jan Engelhardt wrote:
> On Sunday 2011-05-15 09:23, Andrew Beverley wrote:
> 
>> On Sun, 2011-05-15 at 00:33 +0200, Jan Engelhardt wrote:
>>> On Saturday 2011-05-14 18:29, Andrew Beverley wrote:
>>>
>>>> On Sat, 2011-05-14 at 14:36 +0100, Ed W wrote:
>>>>
>>>> Okay, I've played around with this myself using a similar scenario. It
>>>> looks to me like the packets *are* making it into the conntrack system.
>>>>
>>>> I tried setting a LOG target to match those packets with a ctstate of
>>>> RELATED:
>>>>
>>>> iptables -A INPUT -p ICMP -m conntrack --ctstate RELATED -j LOG
>>>>
>>>> And they were indeed logged. But there was no visibility of them using
>>>> the conntrack userspace program.
>>>
>>> Does `conntrack -L` show anything for you at all? 
>>
>> Yes, it shows the outgoing packets:
>>
>> udp      17 23 src=10.0.10.206 dst=212.110.185.119 sport=35259 dport=53
>> packets=3 bytes=168 [UNREPLIED] src=212.110.185.119 dst=10.0.10.206
>> sport=53 dport=35259 packets=0 bytes=0 mark=0 secmark=0 use=2
>>
>> But it doesn't show the "ICMP port unreachable" packets that are sent in
>> reply. The question is: should it show them?
> 
> conntrack -L shows pseudo/NFCT-style connections, not packets. As for 
> ICMP port unreachable, either its classification is
> - INVALID, in which case there is no CT to show
> - a reply, in which case it is part of the shown CT - and the event 
> monitor will show an UPDATE
> - RELATED, in which case a new CT is shown - and which may disappear 
> shortly after due to a short lifetime - the event monitor should show 
> NEW I think.
> 
> Monitor with conntrack -E for details.

Hi Jan, thanks for taking an interest in this.  Just to repeat the testing with 
my situation.  So to recap I'm using "dnsmasq" with two upstream dns resolvers 
defined.  For various reasons dnsmasq queries both of them simultaneously and 
keeps the answer from whichever responds first.  The second answer is met with a 
(presumably) kernel generated "unreachable" response since the UDP port has already 
been closed after the faster response is received


So my test is "nslookup www.yahoo.co.uk" and both tcpdump and conntrack -E dumps are
below:

    [NEW] udp      17 30 src=127.0.0.1 dst=127.0.0.1 sport=43972 dport=53 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=53 dport=43972
    [NEW] udp      17 30 src=10.94.230.137 dst=8.8.4.4 sport=20110 dport=53 [UNREPLIED] src=8.8.4.4 dst=10.94.230.137 sport=53 dport=20110
    [NEW] udp      17 30 src=10.94.230.137 dst=8.8.8.8 sport=20110 dport=53 [UNREPLIED] src=8.8.8.8 dst=10.94.230.137 sport=53 dport=20110
 [UPDATE] udp      17 29 src=10.94.230.137 dst=8.8.4.4 sport=20110 dport=53 src=8.8.4.4 dst=10.94.230.137 sport=53 dport=20110
 [UPDATE] udp      17 29 src=127.0.0.1 dst=127.0.0.1 sport=43972 dport=53 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=43972
    [NEW] udp      17 30 src=127.0.0.1 dst=127.0.0.1 sport=44866 dport=53 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=53 dport=44866
 [UPDATE] udp      17 30 src=127.0.0.1 dst=127.0.0.1 sport=44866 dport=53 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=44866
    [NEW] udp      17 30 src=127.0.0.1 dst=127.0.0.1 sport=33044 dport=53 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=53 dport=33044
 [UPDATE] udp      17 30 src=127.0.0.1 dst=127.0.0.1 sport=33044 dport=53 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=33044
    [NEW] udp      17 30 src=127.0.0.1 dst=127.0.0.1 sport=36565 dport=53 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=53 dport=36565
 [UPDATE] udp      17 30 src=127.0.0.1 dst=127.0.0.1 sport=36565 dport=53 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=36565
    [NEW] udp      17 30 src=127.0.0.1 dst=127.0.0.1 sport=45214 dport=53 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=53 dport=45214
    [NEW] udp      17 30 src=10.94.230.137 dst=8.8.4.4 sport=32019 dport=53 [UNREPLIED] src=8.8.4.4 dst=10.94.230.137 sport=53 dport=32019
 [UPDATE] udp      17 29 src=10.94.230.137 dst=8.8.8.8 sport=20110 dport=53 src=8.8.8.8 dst=10.94.230.137 sport=53 dport=20110
 [UPDATE] udp      17 29 src=10.94.230.137 dst=8.8.4.4 sport=32019 dport=53 src=8.8.4.4 dst=10.94.230.137 sport=53 dport=32019
 [UPDATE] udp      17 29 src=127.0.0.1 dst=127.0.0.1 sport=45214 dport=53 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=45214
    [NEW] udp      17 30 src=127.0.0.1 dst=127.0.0.1 sport=47713 dport=53 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=53 dport=47713
 [UPDATE] udp      17 30 src=127.0.0.1 dst=127.0.0.1 sport=47713 dport=53 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=47713
    [NEW] udp      17 30 src=127.0.0.1 dst=127.0.0.1 sport=37969 dport=53 [UNREPLIED] src=127.0.0.1 dst=127.0.0.1 sport=53 dport=37969
    [NEW] udp      17 30 src=10.94.230.137 dst=8.8.4.4 sport=43177 dport=53 [UNREPLIED] src=8.8.4.4 dst=10.94.230.137 sport=53 dport=43177
 [UPDATE] udp      17 29 src=10.94.230.137 dst=8.8.4.4 sport=43177 dport=53 src=8.8.4.4 dst=10.94.230.137 sport=53 dport=43177
 [UPDATE] udp      17 29 src=127.0.0.1 dst=127.0.0.1 sport=37969 dport=53 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=37969
[DESTROY] udp      17 src=127.0.0.1 dst=127.0.0.1 sport=43972 dport=53 packets=1 bytes=61 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=43972 packets=1 bytes=206
[DESTROY] udp      17 src=10.94.230.137 dst=8.8.4.4 sport=20110 dport=53 packets=1 bytes=61 src=8.8.4.4 dst=10.94.230.137 sport=53 dport=20110 packets=1 bytes=206
[DESTROY] udp      17 src=10.94.230.137 dst=8.8.8.8 sport=20110 dport=53 packets=1 bytes=61 src=8.8.8.8 dst=10.94.230.137 sport=53 dport=20110 packets=1 bytes=206
[DESTROY] udp      17 src=127.0.0.1 dst=127.0.0.1 sport=44866 dport=53 packets=1 bytes=58 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=44866 packets=1 bytes=128
[DESTROY] udp      17 src=127.0.0.1 dst=127.0.0.1 sport=33044 dport=53 packets=1 bytes=65 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=33044 packets=1 bytes=102
[DESTROY] udp      17 src=127.0.0.1 dst=127.0.0.1 sport=36565 dport=53 packets=1 bytes=69 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=36565 packets=1 bytes=69
[DESTROY] udp      17 src=127.0.0.1 dst=127.0.0.1 sport=45214 dport=53 packets=1 bytes=61 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=45214 packets=1 bytes=177
[DESTROY] udp      17 src=10.94.230.137 dst=8.8.4.4 sport=32019 dport=53 packets=1 bytes=61 src=8.8.4.4 dst=10.94.230.137 sport=53 dport=32019 packets=1 bytes=177
[DESTROY] udp      17 src=127.0.0.1 dst=127.0.0.1 sport=47713 dport=53 packets=1 bytes=73 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=47713 packets=1 bytes=110
[DESTROY] udp      17 src=127.0.0.1 dst=127.0.0.1 sport=37969 dport=53 packets=1 bytes=73 src=127.0.0.1 dst=127.0.0.1 sport=53 dport=37969 packets=1 bytes=110
[DESTROY] udp      17 src=10.94.230.137 dst=8.8.4.4 sport=43177 dport=53 packets=1 bytes=73 src=8.8.4.4 dst=10.94.230.137 sport=53 dport=43177 packets=1 bytes=110


10:49:53.325930 IP 10.94.230.137.20110 > 8.8.4.4.domain: 50338+ AAAA? www.yahoo.co.uk. (33)
10:49:53.327794 IP 10.94.230.137.20110 > 8.8.8.8.domain: 50338+ AAAA? www.yahoo.co.uk. (33)
10:49:53.996918 IP 8.8.4.4.domain > 10.94.230.137.20110: 50338 3/1/0 CNAME rc.yahoo.com., CNAME rc.g01.yahoodns.net., CNAME any-rc.a01.yahoodns.net. (178)
10:49:54.004573 IP 10.94.230.137.32019 > 8.8.4.4.domain: 33843+ A? www.yahoo.co.uk. (33)
10:49:54.126735 IP 8.8.8.8.domain > 10.94.230.137.20110: 50338 3/1/0 CNAME rc.yahoo.com., CNAME rc.g01.yahoodns.net., CNAME any-rc.a01.yahoodns.net. (178)
10:49:54.126855 IP 10.94.230.137 > 8.8.8.8: ICMP 10.94.230.137 udp port 20110 unreachable, length 214
10:49:54.340595 IP 8.8.4.4.domain > 10.94.230.137.32019: 33843 5/0/0 CNAME rc.yahoo.com., CNAME rc.g01.yahoodns.net., CNAME any-rc.a01.yahoodns.net., A 87.248.120.148, A 77.238.178.122 (149)
10:49:54.345321 IP 10.94.230.137.43177 > 8.8.4.4.domain: 34962+ PTR? 122.178.238.77.in-addr.arpa. (45)
10:49:55.030067 IP 8.8.4.4.domain > 10.94.230.137.43177: 34962 1/0/0 PTR w2.rc.vip.ird.yahoo.com. (82)



For ease of reading, here are the conntrack connections relating to 10.94.230.137:

    [NEW] udp      17 30 src=10.94.230.137 dst=8.8.4.4 sport=20110 dport=53 [UNREPLIED] src=8.8.4.4 dst=10.94.230.137 sport=53 dport=20110
    [NEW] udp      17 30 src=10.94.230.137 dst=8.8.8.8 sport=20110 dport=53 [UNREPLIED] src=8.8.8.8 dst=10.94.230.137 sport=53 dport=20110
 [UPDATE] udp      17 29 src=10.94.230.137 dst=8.8.4.4 sport=20110 dport=53 src=8.8.4.4 dst=10.94.230.137 sport=53 dport=20110
    [NEW] udp      17 30 src=10.94.230.137 dst=8.8.4.4 sport=32019 dport=53 [UNREPLIED] src=8.8.4.4 dst=10.94.230.137 sport=53 dport=32019
 [UPDATE] udp      17 29 src=10.94.230.137 dst=8.8.8.8 sport=20110 dport=53 src=8.8.8.8 dst=10.94.230.137 sport=53 dport=20110
 [UPDATE] udp      17 29 src=10.94.230.137 dst=8.8.4.4 sport=32019 dport=53 src=8.8.4.4 dst=10.94.230.137 sport=53 dport=32019
    [NEW] udp      17 30 src=10.94.230.137 dst=8.8.4.4 sport=43177 dport=53 [UNREPLIED] src=8.8.4.4 dst=10.94.230.137 sport=53 dport=43177
 [UPDATE] udp      17 29 src=10.94.230.137 dst=8.8.4.4 sport=43177 dport=53 src=8.8.4.4 dst=10.94.230.137 sport=53 dport=43177
[DESTROY] udp      17 src=10.94.230.137 dst=8.8.4.4 sport=20110 dport=53 packets=1 bytes=61 src=8.8.4.4 dst=10.94.230.137 sport=53 dport=20110 packets=1 bytes=206
[DESTROY] udp      17 src=10.94.230.137 dst=8.8.8.8 sport=20110 dport=53 packets=1 bytes=61 src=8.8.8.8 dst=10.94.230.137 sport=53 dport=20110 packets=1 bytes=206
[DESTROY] udp      17 src=10.94.230.137 dst=8.8.4.4 sport=32019 dport=53 packets=1 bytes=61 src=8.8.4.4 dst=10.94.230.137 sport=53 dport=32019 packets=1 bytes=177
[DESTROY] udp      17 src=10.94.230.137 dst=8.8.4.4 sport=43177 dport=53 packets=1 bytes=73 src=8.8.4.4 dst=10.94.230.137 sport=53 dport=43177 packets=1 bytes=110


However, I don't see this packet accounted in that list above:
10:49:54.126855 IP 10.94.230.137 > 8.8.8.8: ICMP 10.94.230.137 udp port 20110 unreachable, length 214



This feels to me like a packet which has fallen between two situations. It's not a new connection so it didn't get logged as such.  It's also kind of not obviously part of the existing connection so it doesn't get logged as such either? (but I think most use cases would prefer that it shows as part of the now closed connection?)

Grateful for any insight?

Ed W

Re: High accuracy bandwidth accounting?

[Jan Engelhardt]

On Monday 2011-05-16 16:35, Ed W wrote:
>> 
>> conntrack -L shows pseudo/NFCT-style connections, not packets. As for 
>> ICMP port unreachable, either its classification is
>> - INVALID, in which case there is no CT to show
>> - a reply, in which case it is part of the shown CT - and the event 
>> monitor will show an UPDATE
>> - RELATED, in which case a new CT is shown - and which may disappear 
>> shortly after due to a short lifetime - the event monitor should show 
>> NEW I think.
>
>I'm using "dnsmasq" with two upstream dns resolvers [...]
>The second answer is met with a 
>(presumably) kernel generated "unreachable" response since the UDP port has already 
>been closed after the faster [first] response is received
>
>So my test is "nslookup www.yahoo.co.uk" and both tcpdump and conntrack -E dumps are
>below:

Modern times - modern tools: host(1).

>For ease of reading, here are the conntrack connections relating to 10.94.230.137:
>
>    [NEW] udp      17 30 src=10.94.230.137 dst=8.8.4.4 sport=20110 dport=53 [UNREPLIED] src=8.8.4.4 dst=10.94.230.137 sport=53 dport=20110
[...]
>
>However, I don't see this packet accounted in that list above:
>10:49:54.126855 IP 10.94.230.137 > 8.8.8.8: ICMP 10.94.230.137 udp port 20110 unreachable, length 214

>This feels to me like a packet which has fallen between two
>situations. It's not a new connection so it didn't get logged as
>such. It's also kind of not obviously part of the existing
>connection so it doesn't get logged as such either?

Something like that. Check the state of this ICMP packet with -j
LOGMARK from Xtables-addons. I'd be interested in what it belongs to.

Re: High accuracy bandwidth accounting?

[Ed W]

>> So my test is "nslookup www.yahoo.co.uk" and both tcpdump and conntrack -E dumps are
>> below:
> 
> Modern times - modern tools: host(1).

Busybox...  :-(


>> This feels to me like a packet which has fallen between two
>> situations. It's not a new connection so it didn't get logged as
>> such. It's also kind of not obviously part of the existing
>> connection so it doesn't get logged as such either?
> 
> Something like that. Check the state of this ICMP packet with -j
> LOGMARK from Xtables-addons. I'd be interested in what it belongs to.

Because this is an embedded device and my build scripts are currently broken at the moment (being rewritten...), adding xtables-addons to the device is slightly tricky right now...

(I think I need some more linux development machines here...  All the other linux machines here are "production" servers... Hmm)


After some thought, this command line *seems* to reproduce the problem - would someone with LOGMARK available be kind enough to reproduce this:

	nslookup www.yahoo.co.uk & sleep 0.01 && killall nslookup


16:45:02.034285 IP 10.94.230.137.38407 > 8.8.8.8.domain: 2+ PTR? 8.8.8.8.in-addr.arpa. (38)
16:45:02.316397 IP 8.8.8.8.domain > 10.94.230.137.38407: 2 1/0/0 PTR google-public-dns-a.google.com. (82)
16:45:02.316522 IP 10.94.230.137 > 8.8.8.8: ICMP 10.94.230.137 udp port 38407 unreachable, length 118

[NEW] udp      17 30 src=10.94.230.137 dst=8.8.8.8 sport=38407 dport=53 [UNREPLIED] src=8.8.8.8 dst=10.94.230.137 sport=53 dport=38407
[UPDATE] udp      17 29 src=10.94.230.137 dst=8.8.8.8 sport=38407 dport=53 src=8.8.8.8 dst=10.94.230.137 sport=53 dport=38407
[DESTROY] udp      17 src=10.94.230.137 dst=8.8.8.8 sport=38407 dport=53 packets=1 bytes=66 src=8.8.8.8 dst=10.94.230.137 sport=53 dport=38407 packets=1 bytes=110

(Obviously s/nslookup/host/ as appropriate...)

This seems like a smaller reproducible test case?  The only caveat is that the "sleep x" be smaller than the RTT to the DNS server - using a real DNS server on the internet makes this straightforward.

Thanks

Ed W

Re: High accuracy bandwidth accounting?

[Marek Kierdelewicz]

Hi,

Firstly I'd like to say you have an interesting use case.

>An example seems to be to cause a name lookup via dnsmasq. For whatever
>reason this does two simultaneous dns requests to both configured dns
>servers.  One reply comes back slightly quicker than the other and the
>slower reply appears to cause a local ICMP unreachable response to be
>generated.  Everything is logged *except* the data for the ICMP
>unreachable response?

You should consider disabling icmp responses:
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_all

... and/or limiting/disabling unreachable responses:
echo 1 > /proc/sys/net/ipv4/icmp_ratelimit

Agressive ratelimiting will cause your router as a hop in traceroute to
show near 100% loss, but other then that you'll save bandwidth.

Best regards,
Marek Kierdelewicz