iptables --string-replace

iptables --string-replace

[Ben K]

I'd like to be able to mangle strings passing through my home router
running Openwrt in order to modify/anonymize user-agent strings. I
believe a patch further extending the iptables string extension by
providing string replace functionality was submitted by Michael Rash
back in 2004 (archived at
http://www.spinics.net/lists/netfilter/msg23791.html). This would be
ideal as I could then mangle user-agent headers without eg needing to
run an http proxy.

Does anyone know if the --string-replace functionality ever made it
into iptables? If not, what are my chances of the patch from 2004
playing nice with the current Git head revision?

Thanks very much,
Ben

(BTW what's with the mailing list rejecting HTML emails?! Are we
living in 2001?)

Re: iptables --string-replace

[Jan Engelhardt]

[fullquote due to added cc]
On Sunday 2011-01-16 23:43, Ben K wrote:

>I'd like to be able to mangle strings passing through my home router
>running Openwrt in order to modify/anonymize user-agent strings. I
>believe a patch further extending the iptables string extension by
>providing string replace functionality was submitted by Michael Rash
>back in 2004 (archived at
>http://www.spinics.net/lists/netfilter/msg23791.html). This would be
>ideal as I could then mangle user-agent headers without eg needing to
>run an http proxy.

That is not trivially possible because the UA identifier may be split 
across multiple packets, which does not exactly facilitate its 
replacement.
The only sensible choice is to use some userspace tools, and proxies 
usually do that job well.


>Does anyone know if the --string-replace functionality ever made it
>into iptables? If not, what are my chances of the patch from 2004
>playing nice with the current Git head revision?
>
>Thanks very much,
>Ben
>
>(BTW what's with the mailing list rejecting HTML emails?! Are we
>living in 2001?)

We are developers, not salesmen.

Re: iptables --string-replace

[Ben K]

Thank you for the speedy reply.

So I'm guessing the answer to the main question below, about whether
that patch ever made it into iptables, is "no".
Is this because of the limitations of string matching with regards to
packet fragmentation, as per your response, or for some other reason
(eg unwanted complexity)?

If the former, then why does iptables include the string match (but
not replace) extension, which surely suffers from the same
limitations?

>Does anyone know if the --string-replace functionality ever made it
>into iptables? If not, what are my chances of the patch from 2004
>playing nice with the current Git head revision?

Re: iptables --string-replace

[Jan Engelhardt]

Don't strip Cc, and don't top post.

On Monday 2011-01-17 00:58, Ben K wrote:
>
>>Does anyone know if the --string-replace functionality ever made it
>>into iptables? If not, what are my chances of the patch from 2004
>>playing nice with the current Git head revision?

>Thank you for the speedy reply.
>
>So I'm guessing the answer to the main question below, about whether
>that patch ever made it into iptables, is "no".
>Is this because of the limitations of string matching with regards to
>packet fragmentation, as per your response, or for some other reason
>(eg unwanted complexity)?

Matching across packets would incur unwanted complexity.

>If the former, then why does iptables include the string match (but
>not replace) extension, which surely suffers from the same
>limitations?

You'd have to ask Pablo (cc'd) who added ipt_string.c 5 years ago to
the kernel, or even Emmanuel Roger who had it added 10 years ago to
the iptables userspace part.

Re: iptables --string-replace

[Ben K]

> Don't strip Cc, and don't top post.

Sorry, missed the cc. I read the posting guidelines before mailing and
don't consider my means of quoting to be a top-post (I removed most of
the content and the quote was 'standalone', I just chose to put it at
bottom).

> Matching across packets would incur unwanted complexity.

Just curious, does the current string match implementation match
across packets? If not, then surely adding replace functionality (with
the same compromise) is not overly complex?

On Mon, Jan 17, 2011 at 12:20 PM, Jan Engelhardt <jengelh@medozas.de> wrote:

>
> On Monday 2011-01-17 00:58, Ben K wrote:
>>
>>>Does anyone know if the --string-replace functionality ever made it
>>>into iptables? If not, what are my chances of the patch from 2004
>>>playing nice with the current Git head revision?
>
>>Thank you for the speedy reply.
>>
>>So I'm guessing the answer to the main question below, about whether
>>that patch ever made it into iptables, is "no".
>>Is this because of the limitations of string matching with regards to
>>packet fragmentation, as per your response, or for some other reason
>>(eg unwanted complexity)?
>
> Matching across packets would incur unwanted complexity.
>
>>If the former, then why does iptables include the string match (but
>>not replace) extension, which surely suffers from the same
>>limitations?
>
> You'd have to ask Pablo (cc'd) who added ipt_string.c 5 years ago to
> the kernel, or even Emmanuel Roger who had it added 10 years ago to
> the iptables userspace part.
>

Re: iptables --string-replace

[Jan Engelhardt]

On Monday 2011-01-17 03:44, Ben K wrote:

>> Matching across packets would incur unwanted complexity.
>
>Just curious, does the current string match implementation match
>across packets? If not, then surely adding replace functionality (with
>the same compromise) is not overly complex?

The string match does indeed not work across packets. I do not know why 
we have it, it won't have much use for stream protocols either and was 
probably devised for datagrams. I can't say for sure what the original 
authors' intentions were. xt_string also works on the entire IP packet, 
so there is a chance for false positives if one only wants to match 
actual L7 payload.

Re: iptables --string-replace

[Amos Jeffries]

On 17/01/11 16:41, Jan Engelhardt wrote:
> On Monday 2011-01-17 03:44, Ben K wrote:
>
>>> Matching across packets would incur unwanted complexity.
>>
>> Just curious, does the current string match implementation match
>> across packets? If not, then surely adding replace functionality (with
>> the same compromise) is not overly complex?
>
> The string match does indeed not work across packets. I do not know why
> we have it, it won't have much use for stream protocols either and was
> probably devised for datagrams. I can't say for sure what the original
> authors' intentions were. xt_string also works on the entire IP packet,
> so there is a chance for false positives if one only wants to match
> actual L7 payload.

It has some favor amongst the NAT interception / transparent proxy crowd.

The use-case is to distinguish real HTTP traffic to be intercepted vs 
weird binary abusing port 80. Or the reverse, to only catch HTTP going 
over general ports like 8080 which may get anything.
In this type of case only the first dozen or so bytes are relevant and 
almost always guaranteed to be in one (first) packet.

AYJ

Re: iptables --string-replace

[Jan Engelhardt]

On Monday 2011-01-17 11:52, Amos Jeffries wrote:
>>
>> The string match does indeed not work across packets. I do not know why
>> we have it, it won't have much use for stream protocols either and was
>> probably devised for datagrams. I can't say for sure what the original
>> authors' intentions were. xt_string also works on the entire IP packet,
>> so there is a chance for false positives if one only wants to match
>> actual L7 payload.
>
> It has some favor amongst the NAT interception / transparent proxy crowd.
>
> The use-case is to distinguish real HTTP traffic to be intercepted vs weird
> binary abusing port 80. Or the reverse, to only catch HTTP going over general
> ports like 8080 which may get anything.
> In this type of case only the first dozen or so bytes are relevant and almost
> always guaranteed to be in one (first) packet.

But is it not that the first packet is a SYN which, in most cases, does not
carry any data, yet NAT decisions need to be made on this first packet.

Re: iptables --string-replace

[Pablo Neira Ayuso]

On 17/01/11 04:41, Jan Engelhardt wrote:
> On Monday 2011-01-17 03:44, Ben K wrote:
> 
>>> Matching across packets would incur unwanted complexity.
>>
>> Just curious, does the current string match implementation match
>> across packets? If not, then surely adding replace functionality (with
>> the same compromise) is not overly complex?
> 
> The string match does indeed not work across packets. I do not know why 
> we have it, it won't have much use for stream protocols either and was 
> probably devised for datagrams.

Could you tell me why is not useful for stream protocols?

> I can't say for sure what the original 
> authors' intentions were. xt_string also works on the entire IP packet, 
> so there is a chance for false positives if one only wants to match 
> actual L7 payload.

It's easy to extend it to make it start after the IP header. I'll send a
patch for this.

I guess that it's going to be hard to find some pattern that matches in
the IP header, so that false positive that you mention has a very low
probability.

Re: iptables --string-replace

[Jan Engelhardt]

On Friday 2011-01-21 11:04, Pablo Neira Ayuso wrote:

>On 17/01/11 04:41, Jan Engelhardt wrote:
>> On Monday 2011-01-17 03:44, Ben K wrote:
>> 
>>>> Matching across packets would incur unwanted complexity.
>>>
>>> Just curious, does the current string match implementation match
>>> across packets? If not, then surely adding replace functionality (with
>>> the same compromise) is not overly complex?
>> 
>> The string match does indeed not work across packets. I do not know why 
>> we have it, it won't have much use for stream protocols either and was 
>> probably devised for datagrams.
>
>Could you tell me why is not useful for stream protocols?

Because the stream may be segmented at about any point, thereby
splitting the very string the user was trying to match on
across two packets.

>
>> I can't say for sure what the original 
>> authors' intentions were. xt_string also works on the entire IP packet, 
>> so there is a chance for false positives if one only wants to match 
>> actual L7 payload.
>
>It's easy to extend it to make it start after the IP header. I'll send a
>patch for this.
>
>I guess that it's going to be hard to find some pattern that matches in
>the IP header, so that false positive that you mention has a very low
>probability.

That does of course depend on the length of the pattern.
If you only search for "GET", you could for example match
on the sequence number.

Re: iptables --string-replace

[Pablo Neira Ayuso]

On 21/01/11 11:09, Jan Engelhardt wrote:
> On Friday 2011-01-21 11:04, Pablo Neira Ayuso wrote:
> 
>> On 17/01/11 04:41, Jan Engelhardt wrote:
>>> On Monday 2011-01-17 03:44, Ben K wrote:
>>>
>>>>> Matching across packets would incur unwanted complexity.
>>>>
>>>> Just curious, does the current string match implementation match
>>>> across packets? If not, then surely adding replace functionality (with
>>>> the same compromise) is not overly complex?
>>>
>>> The string match does indeed not work across packets. I do not know why 
>>> we have it, it won't have much use for stream protocols either and was 
>>> probably devised for datagrams.
>>
>> Could you tell me why is not useful for stream protocols?
> 
> Because the stream may be segmented at about any point, thereby
> splitting the very string the user was trying to match on
> across two packets.

It's not hard to extend the textsearch infrastructure to perform
searches across packets.

Anyway, I don't believe that normal TCP traffic would experience such
splitting, in general.

And there are tons of way to defeat pattern matching filtering, this is one.

>>> I can't say for sure what the original 
>>> authors' intentions were. xt_string also works on the entire IP packet, 
>>> so there is a chance for false positives if one only wants to match 
>>> actual L7 payload.
>>
>> It's easy to extend it to make it start after the IP header. I'll send a
>> patch for this.
>>
>> I guess that it's going to be hard to find some pattern that matches in
>> the IP header, so that false positive that you mention has a very low
>> probability.
> 
> That does of course depend on the length of the pattern.

and the pattern itself.

> If you only search for "GET", you could for example match
> on the sequence number.

That's possible, although quite unlikely.

Even if we start in the payload of the packet, pattern matching
infrastructures still have false positives. This is something you'll
have to live with.

And smart guys will always find the way to by-pass it.

Re: iptables --string-replace

[Pablo Neira Ayuso]

On 17/01/11 03:44, Ben K wrote:
>> Don't strip Cc, and don't top post.
> 
> Sorry, missed the cc. I read the posting guidelines before mailing and
> don't consider my means of quoting to be a top-post (I removed most of
> the content and the quote was 'standalone', I just chose to put it at
> bottom).
> 
>> Matching across packets would incur unwanted complexity.
> 
> Just curious, does the current string match implementation match
> across packets? If not, then surely adding replace functionality (with
> the same compromise) is not overly complex?
> 
> On Mon, Jan 17, 2011 at 12:20 PM, Jan Engelhardt <jengelh@medozas.de> wrote:
> 
>>
>> On Monday 2011-01-17 00:58, Ben K wrote:
>>>
>>>> Does anyone know if the --string-replace functionality ever made it
>>>> into iptables? If not, what are my chances of the patch from 2004
>>>> playing nice with the current Git head revision?

I remember that this patch have several problems:

* it does not handle fragmented packets
* it only allows to replace strings of the same size, otherwise you have
to perform sequence number adjustments, and that complicates the whole
thing.

Re: iptables --string-replace

[/dev/rob0]

On Mon, Jan 17, 2011 at 09:43:27AM +1100, Ben K wrote:
> I'd like to be able to mangle strings passing through my home 
> router running Openwrt in order to modify/anonymize user-agent 
> strings. I believe a patch further extending the iptables string 
> extension by providing string replace functionality was submitted 
> by Michael Rash back in 2004 (archived at 
> http://www.spinics.net/lists/netfilter/msg23791.html). This would 
> be ideal as I could then mangle user-agent headers without eg 
> needing to run an http proxy.

I don't see why that's ideal. The proxy solution seems like the 
ideal, to me. While an openwrt router might not have the horsepower 
needed, neither does it have the horsepower you'll need for string 
matching. You're going to have to throw more hardware at this 
problem, however you might proceed.

> (BTW what's with the mailing list rejecting HTML emails?! Are we
> living in 2001?)

HTML posting on mailing lists is very rude. I usually ignore them. 
Gmail's implementation of it is particularly bad, converting many 
arbitrary strings into HTTP URI's.
-- 
    Offlist mail to this address is discarded unless
    "/dev/rob0" or "not-spam" is in Subject: header