Wrong MAC in redirected packet

Wrong MAC in redirected packet

[Kristian Evensen]

Hello,

I have a setup where I have a sender and a multihomed receiver
connected through a switch, and I am working on a module that
currently does more or less the same as RAWDNAT [1] (at least it is
supposed to). My goal is to redirect packets destined for one
interface on the multihomed receiver to the other, and changing the IP
address works. Unfortunately, the packet keeps the original
destination MAC-address, so it arrives at the wrong interface on the
multihomed receiver. The sender has the correct mapping between IP's
and MAC's (arp_filter is 1 on the receiver) and it works when I do the
redirection using DNAT or for example ping each interface separately.
Does anyone have any suggestions or hints? Reconstructing the SKB
would most likely solve it, but that seems a bit drastic.

My theory is that the original MAC-address is somehow added to the SKB
before it reaches the output-part of the RAW-table (which is where I
hook in), because of the sender's mapping between the original
destination IP and MAC. However, I have not been able to figure this
out. Also, I  looked at the NAT-code, but it seems to "only" change
IP-address as well. Have I overlooked something or am I correct?

Btw, RAWDNAT throws up a couple of errors on my machine, so I have not
been able to see it it does what I want to do.

Please let me know if you need more information.

Thanks,
Kristian

[1] - http://dev.medozas.de/gitweb.cgi?p=xtables-addons;a=blob;f=extensions/libxt_RAWDNAT.c;h=1d50b9188ae5e1e2b257ac15afcb2857c9353c25;hb=RAWNAT

Re: Wrong MAC in redirected packet

[Jan Engelhardt]

On Wednesday 2009-03-25 15:27, Kristian Evensen wrote:

>Hello,
>
>I have a setup where I have a sender and a multihomed receiver
>connected through a switch, and I am working on a module that
>currently does more or less the same as RAWDNAT [1] (at least it is
>supposed to). My goal is to redirect packets destined for one
>interface on the multihomed receiver to the other, and changing the IP
>address works. Unfortunately, the packet keeps the original
>destination MAC-address, so it arrives at the wrong interface on the
>multihomed receiver.

Ah that might also explain why I observed that packets get lost sometimes..

>My theory is that the original MAC-address is somehow added to the SKB
>before it reaches the output-part of the RAW-table (which is where I
>hook in),

Routing is done before rawpost, yes, but before the (traditional) raw table.
(it's in skb->dst)

>because of the sender's mapping between the original
>destination IP and MAC. However, I have not been able to figure this
>out. Also, I  looked at the NAT-code, but it seems to "only" change
>IP-address as well. Have I overlooked something or am I correct?

Most likely RAWS/DNAT should gain another option to also tweak the
MAC daddr, by calling ip_route_output_key to get a new skb->dst.

>Btw, RAWDNAT throws up a couple of errors on my machine, so I have not
>been able to see it it does what I want to do.

Which errors?

Re: Wrong MAC in redirected packet

[Kristian Evensen]

>>My theory is that the original MAC-address is somehow added to the SKB
>>before it reaches the output-part of the RAW-table (which is where I
>>hook in),
>
> Routing is done before rawpost, yes, but before the (traditional) raw table.
> (it's in skb->dst)

Ok, I assumed that the MAC-header was added at a lower layer, but that
was then wrong? For some reason I have always though skb->dst was the
IP-adress, I will look into that field.

>>because of the sender's mapping between the original
>>destination IP and MAC. However, I have not been able to figure this
>>out. Also, I  looked at the NAT-code, but it seems to "only" change
>>IP-address as well. Have I overlooked something or am I correct?
>
> Most likely RAWS/DNAT should gain another option to also tweak the
> MAC daddr, by calling ip_route_output_key to get a new skb->dst.

This sounds like a good idea. I will look at the ip_route_output_key,
try to write a function for my module and patch it into RAWNAT (unless
somebody else does it first).

>>Btw, RAWDNAT throws up a couple of errors on my machine, so I have not
>>been able to see it it does what I want to do.
>
> Which errors?
>

After struggling a bit with the compilation (I had to remove the
iptable_rawpost.o ip6table_rawpost.o in Kbuild or the compiler
complained that it couldn't find the files), I get this error when I
try to use it:

kristrev@mylatop:~/src/xtables-addons-1.12$ sudo iptables -A
PREROUTING -t raw -p udp -d 192.168.101.14 --dport 9999 -j RAWDNAT
--to-destination 192.168.100.250
iptables: Invalid argument

-Kristian
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Wrong MAC in redirected packet

[Jan Engelhardt]

On Wednesday 2009-03-25 16:00, Kristian Evensen wrote:

>>>My theory is that the original MAC-address is somehow added to the SKB
>>>before it reaches the output-part of the RAW-table (which is where I
>>>hook in),
>>
>> Routing is done before rawpost, yes, but before the (traditional) raw table.
>> (it's in skb->dst)
>
>Ok, I assumed that the MAC-header was added at a lower layer, but that
>was then wrong? For some reason I have always though skb->dst was the
>IP-adress, I will look into that field.

skb->dst contains the pointer to the neighbor (simply put), and a 
neighbor is (also simplified) what you see in `ip neigh`, i.e. MAC 

>After struggling a bit with the compilation (I had to remove the
>iptable_rawpost.o ip6table_rawpost.o in Kbuild or the compiler
>complained that it couldn't find the files), I get this error when I
>try to use it:
>
>kristrev@mylatop:~/src/xtables-addons-1.12$ sudo iptables -A
>PREROUTING -t raw -p udp -d 192.168.101.14 --dport 9999 -j RAWDNAT
>--to-destination 192.168.100.250
>iptables: Invalid argument

See `dmesg` for the details of EINVAL.

Re: Wrong MAC in redirected packet

[Kristian Evensen]

>>>>My theory is that the original MAC-address is somehow added to the SKB
>>>>before it reaches the output-part of the RAW-table (which is where I
>>>>hook in),
>>>
>>> Routing is done before rawpost, yes, but before the (traditional) raw table.
>>> (it's in skb->dst)
>>
>>Ok, I assumed that the MAC-header was added at a lower layer, but that
>>was then wrong? For some reason I have always though skb->dst was the
>>IP-adress, I will look into that field.
>
> skb->dst contains the pointer to the neighbor (simply put), and a
> neighbor is (also simplified) what you see in `ip neigh`, i.e. MAC

Thanks.

>>After struggling a bit with the compilation (I had to remove the
>>iptable_rawpost.o ip6table_rawpost.o in Kbuild or the compiler
>>complained that it couldn't find the files), I get this error when I
>>try to use it:
>>
>>kristrev@mylatop:~/src/xtables-addons-1.12$ sudo iptables -A
>>PREROUTING -t raw -p udp -d 192.168.101.14 --dport 9999 -j RAWDNAT
>>--to-destination 192.168.100.250
>>iptables: Invalid argument
>
> See `dmesg` for the details of EINVAL.

[22480.722995] xt_RAWNAT may only be used in the "raw" or "rawpost" table.

When outputting the name of the table, I see that it is corrupted. I
tried to change the _check-function to use the new signature, but then
I got compilation-errors from the register-functions. I am using the
version of RAWNAT I found here [1] (please let me know if there is a
more recent one) and xtables-1.12.

-Kristian

Re: Wrong MAC in redirected packet

[Jan Engelhardt]

On Wednesday 2009-03-25 16:17, Kristian Evensen wrote:
>>>
>>>kristrev@mylatop:~/src/xtables-addons-1.12$ sudo iptables -A
>>>PREROUTING -t raw -p udp -d 192.168.101.14 --dport 9999 -j RAWDNAT
>>>--to-destination 192.168.100.250
>>>iptables: Invalid argument
>>
>> See `dmesg` for the details of EINVAL.
>
>[22480.722995] xt_RAWNAT may only be used in the "raw" or "rawpost" table.
>
>When outputting the name of the table, I see that it is corrupted. I
>tried to change the _check-function to use the new signature, but then
>I got compilation-errors from the register-functions. I am using the
>version of RAWNAT I found here [1] (please let me know if there is a
>more recent one) and xtables-1.12.

I will look into rebasing RAWNAT to a newer Xt-a.

Re: Wrong MAC in redirected packet

[Kristian Evensen]

>>>> My theory is that the original MAC-address is somehow added to the SKB
>>>> before it reaches the output-part of the RAW-table (which is where I
>>>> hook in),
>>>>         
>>> Routing is done before rawpost, yes, but before the (traditional) raw table.
>>> (it's in skb->dst)
>>>       
>> Ok, I assumed that the MAC-header was added at a lower layer, but that
>> was then wrong? For some reason I have always though skb->dst was the
>> IP-adress, I will look into that field.
>>     
>
> skb->dst contains the pointer to the neighbor (simply put), and a 
> neighbor is (also simplified) what you see in `ip neigh`, i.e. MAC 
>
>   
I can now confirm that this works, doing another lookup and updating the 
dst solved the problem and the MAC-header is now correct. I will clean 
up the code and then patch it into RAWNAT or something similar tomorrow, 
if it is of any interest.

However, I have noticed a similar problem when using my module on 
incoming packets in PREROUTING (on the multihomed receiver), the IP 
adress is changed (accoring to my dmesg-output) but then they are not 
heard from again. I have not debugged this properly, but if anyone has 
experienced something similar, feel free :) Can it be caused by the 
wrong MAC-header (changing dst does not work on input on my machine, the 
two interfaces are not aware of eachother's MAC address) being refused 
by some part of the kernel? As always, it for some reasong works when 
using DNAT, but I have not been able to figure out why :)

-Kristian

Re: Wrong MAC in redirected packet

[Jan Engelhardt]

On Wednesday 2009-03-25 20:47, Kristian Evensen wrote:
>
> However, I have noticed a similar problem when using my module on incoming
> packets in PREROUTING (on the multihomed receiver), the IP adress is changed
> (accoring to my dmesg-output) but then they are not heard from again.

Probably dropped due to L3 or L4 checksum mismatches. While we try to
change the checksum, I have not verified it for all cases.

Re: Wrong MAC in redirected packet

[Kristian Evensen]

Jan Engelhardt skrev:
> On Wednesday 2009-03-25 20:47, Kristian Evensen wrote:
>   
>> However, I have noticed a similar problem when using my module on incoming
>> packets in PREROUTING (on the multihomed receiver), the IP adress is changed
>> (accoring to my dmesg-output) but then they are not heard from again.
>>     
>
> Probably dropped due to L3 or L4 checksum mismatches. While we try to
> change the checksum, I have not verified it for all cases.
>
>   
Hm, I am noe sure. I recalculate the checksum (using csum_replace4 for 
l3 and inet....replace4 for l4) after modifying the destination IP, and 
when used with OUTPUT the checksum is correct. Since tcpdump/Wireshark 
outputs the packets before they are modified I haven't been able to 
verify this though.

-Kristian

Re: Wrong MAC in redirected packet

[Jan Engelhardt]

On Wednesday 2009-03-25 20:56, Kristian Evensen wrote:

> Jan Engelhardt skrev:
>> On Wednesday 2009-03-25 20:47, Kristian Evensen wrote:
>>  
>>> However, I have noticed a similar problem when using my module on incoming
>>> packets in PREROUTING (on the multihomed receiver), the IP adress is changed
>>> (accoring to my dmesg-output) but then they are not heard from again.
>>>    
>>
>> Probably dropped due to L3 or L4 checksum mismatches. While we try to
>> change the checksum, I have not verified it for all cases.
>>
>>  
> Hm, I am noe sure. I recalculate the checksum (using csum_replace4 for l3 and
> inet....replace4 for l4) after modifying the destination IP, and when used with
> OUTPUT the checksum is correct. Since tcpdump/Wireshark outputs the packets
> before they are modified I haven't been able to verify this though.

You can use hexdump in the xtables module, or any other form, to
dump the checksum.

Re: Wrong MAC in redirected packet

[Kristian Evensen]

Jan Engelhardt skrev:
> On Wednesday 2009-03-25 20:56, Kristian Evensen wrote:
>
>   
>> Jan Engelhardt skrev:
>>     
>>> On Wednesday 2009-03-25 20:47, Kristian Evensen wrote:
>>>  
>>>       
>>>> However, I have noticed a similar problem when using my module on incoming
>>>> packets in PREROUTING (on the multihomed receiver), the IP adress is changed
>>>> (accoring to my dmesg-output) but then they are not heard from again.
>>>>    
>>>>         
>>> Probably dropped due to L3 or L4 checksum mismatches. While we try to
>>> change the checksum, I have not verified it for all cases.
>>>
>>>  
>>>       
>> Hm, I am noe sure. I recalculate the checksum (using csum_replace4 for l3 and
>> inet....replace4 for l4) after modifying the destination IP, and when used with
>> OUTPUT the checksum is correct. Since tcpdump/Wireshark outputs the packets
>> before they are modified I haven't been able to verify this though.
>>     
>
> You can use hexdump in the xtables module, or any other form, to
> dump the checksum.
>   
I had completely forgot that, thank you. However, tcpdump tells me that 
the checksum is ok, but I will look into it and simply compare a packet 
that has not been redirected with one that has. Thank you.

-Kristian

Re: Wrong MAC in redirected packet

[Kristian Evensen]

>>>>> However, I have noticed a similar problem when using my module on
>>>>> incoming
>>>>> packets in PREROUTING (on the multihomed receiver), the IP adress is
>>>>> changed
>>>>> (accoring to my dmesg-output) but then they are not heard from again.
>>>>>
>>>>
>>>> Probably dropped due to L3 or L4 checksum mismatches. While we try to
>>>> change the checksum, I have not verified it for all cases.
>>>>
>>>>
>>>
>>> Hm, I am noe sure. I recalculate the checksum (using csum_replace4 for l3
>>> and
>>> inet....replace4 for l4) after modifying the destination IP, and when
>>> used with
>>> OUTPUT the checksum is correct. Since tcpdump/Wireshark outputs the
>>> packets
>>> before they are modified I haven't been able to verify this though.
>>>

Turned out that it was the checksums. They were calculated correctly,
but since I processed packets on input, tcp_hdr and udp_hdr of course
returned invalid pointers. My code is now more or less RAWNAT, so if
it is of interest to anybody else I will merge them and make it
available.

-Kristian

Re: Wrong MAC in redirected packet

[Jan Engelhardt]

On Thursday 2009-03-26 11:26, Kristian Evensen wrote:
>
>Turned out that it was the checksums. They were calculated correctly,
>but since I processed packets on input, tcp_hdr and udp_hdr of course
>returned invalid pointers. My code is now more or less RAWNAT, so if
>it is of interest to anybody else I will merge them and make it
>available.

I have now rebased the RAWNAT branch to Xt-a's master and
made the rawpost tables workable with all kernel versions.
It does not have MAC addr punning yet.

Re: Wrong MAC in redirected packet

[Kristian Evensen]

Jan Engelhardt skrev:
> On Thursday 2009-03-26 11:26, Kristian Evensen wrote:
>   
>> Turned out that it was the checksums. They were calculated correctly,
>> but since I processed packets on input, tcp_hdr and udp_hdr of course
>> returned invalid pointers. My code is now more or less RAWNAT, so if
>> it is of interest to anybody else I will merge them and make it
>> available.
>>     
>
> I have now rebased the RAWNAT branch to Xt-a's master and
> made the rawpost tables workable with all kernel versions.
> It does not have MAC addr punning yet.
>   
Great, but I think I see a small bug (I left my Linux-machine at work so 
I am not able to try it out). If you use RAWDNAT in prerouting, tcp_hdr 
and udp_hdr will return the wrong values since the corresponsing values 
in the SKB are not set yet. At least, that is what I experienced in my 
own module. Please let me know if I am wrong :)

Would a better solution by to use conntrack to determine the direction, 
check the name of the chain (is that possible?) or just simply ip_hdr + 
1? :)

-Kristian

Re: Wrong MAC in redirected packet

[Jan Engelhardt]

On Thursday 2009-03-26 21:54, Kristian Evensen wrote:
> Jan Engelhardt skrev:
>> On Thursday 2009-03-26 11:26, Kristian Evensen wrote:
>>  
>>> Turned out that it was the checksums. They were calculated correctly,
>>> but since I processed packets on input, tcp_hdr and udp_hdr of course
>>> returned invalid pointers. My code is now more or less RAWNAT, so if
>>> it is of interest to anybody else I will merge them and make it
>>> available.
>>
>> I have now rebased the RAWNAT branch to Xt-a's master and
>> made the rawpost tables workable with all kernel versions.
>> It does not have MAC addr punning yet.

I am just rethinking this. There are cases where we do not want MAC punning.
(E.g. when a packet still needs to be routed through another gateway
instead of being delivered directly to the final ethernet segment.)

Is --to-mac aa:bb:cc:dd:ee:ff sufficient, or did you want --to-mac IPaddr?

Re: Wrong MAC in redirected packet

[Kristian Evensen]

>>>> Turned out that it was the checksums. They were calculated correctly,
>>>> but since I processed packets on input, tcp_hdr and udp_hdr of course
>>>> returned invalid pointers. My code is now more or less RAWNAT, so if
>>>> it is of interest to anybody else I will merge them and make it
>>>> available.
>>>
>>> I have now rebased the RAWNAT branch to Xt-a's master and
>>> made the rawpost tables workable with all kernel versions.
>>> It does not have MAC addr punning yet.
>
> I am just rethinking this. There are cases where we do not want MAC punning.
> (E.g. when a packet still needs to be routed through another gateway
> instead of being delivered directly to the final ethernet segment.)
>
> Is --to-mac aa:bb:cc:dd:ee:ff sufficient, or did you want --to-mac IPaddr?

The last one would be the easiest to use and would therefore be
desirable. However, forcing the user to write the full MAC-address
might limit scenarios where it is used wrongly.

-Kristian

Re: Wrong MAC in redirected packet

[Jan Engelhardt]

On Sunday 2009-04-26 18:22, Kristian Evensen wrote:
>>
>> I am just rethinking this. There are cases where we do not want MAC punning.
>> (E.g. when a packet still needs to be routed through another gateway
>> instead of being delivered directly to the final ethernet segment.)
>>
>> Is --to-mac aa:bb:cc:dd:ee:ff sufficient, or did you want --to-mac IPaddr?
>
>The last one would be the easiest to use and would therefore be
>desirable. However, forcing the user to write the full MAC-address
>might limit scenarios where it is used wrongly.

You know what, I am just pushing rawnat out now (merged into master
as of today), and everything else is "patches please" :-)

It does work for modifying packets-in-transit as a start. Have not
tested local address manipulation tho.