Clarification on the use of the statistic module

Clarification on the use of the statistic module

[Nestor A. Diaz]

Hi Guys.

I am using the statistics module to balance traffic coming from a
network interface into some services running on the same machine, but
distribution don't tend to be equal.

I use the following iptables script for debugging purposes, which
receives a packet and log it according to the statistic module, i also
put a default log entry in case module don't catch it.

# balance
# log prefix=`date +%Y%m%d%H%M%S`
/sbin/iptables -t nat -A prerouting_rule -i eth0 -p tcp --dport 7000 -m
state --state NEW -m statistic --mode nth --every 2 --packet 0 -j LOG
--log-prefix 20130710095901_packet_0
/sbin/iptables -t nat -A prerouting_rule -i eth0 -p tcp --dport 7000 -m
state --state NEW -m statistic --mode nth --every 2 --packet 0 -j ACCEPT
/sbin/iptables -t nat -A prerouting_rule -i eth0 -p tcp --dport 7000 -m
state --state NEW -m statistic --mode nth --every 2 --packet 1 -j LOG
--log-prefix 20130710095901_packet_1
/sbin/iptables -t nat -A prerouting_rule -i eth0 -p tcp --dport 7000 -m
state --state NEW -m statistic --mode nth --every 2 --packet 1 -j ACCEPT
# default
/sbin/iptables -t nat -A prerouting_rule -i eth0 -p tcp --dport 7000 -m
state --state NEW -j LOG --log-prefix 20130710095901_packet_2
/sbin/iptables -t nat -A prerouting_rule -i eth0 -p tcp --dport 7000 -m
state --state NEW -j ACCEPT

Then i run a series of netcat process to test the configuration:

for i in `seq 1 1000`; do echo test | nc 172.16.1.1 7000; done

The results are logged via syslog, then i use a script to process the
result produced:

# Example: Jul 10 09:37:10 asterix kernel: [3712376.281427]
20130710093659_packet_1 IN=eth0 OUT=
MAC=00:22:4d:56:ac:0c:00:27:0e:0e:b2:61:08:00 SRC=172.24.107.59
DST=172.24.107.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=54795 DF PROTO=TCP
SPT=35341 DPT=7000 WINDOW=14600 RES=0x00 SYN URGP=0
# count_packets.sh
#!/bin/bash

prefix=$1

for i in 0 1 2
do
    echo -n "${i}: "
    grep ${prefix}_packet_${i} /var/log/iptables.log | wc | awk '{print $1}'
done

And for 1000 packets i got exactly the following every time i run the
script (with a different log prefix off course)

./count_packets.sh 20130710093659
0: 500
1: 250
2: 250

When i was expecting the following result:

0: 500
1: 500
2: 0

What i am doing wrong ?

Thanks.

-- 
Typed on my key64.org keyboard

Nestor A Diaz

Re: Clarification on the use of the statistic module

[Pascal Hambourg]

Hello,

Nestor A. Diaz a écrit :
> 
> /sbin/iptables -t nat -A prerouting_rule -i eth0 -p tcp --dport 7000 -m
> state --state NEW -m statistic --mode nth --every 2 --packet 0 -j LOG
> --log-prefix 20130710095901_packet_0
> /sbin/iptables -t nat -A prerouting_rule -i eth0 -p tcp --dport 7000 -m
> state --state NEW -m statistic --mode nth --every 2 --packet 0 -j ACCEPT
> /sbin/iptables -t nat -A prerouting_rule -i eth0 -p tcp --dport 7000 -m
> state --state NEW -m statistic --mode nth --every 2 --packet 1 -j LOG
> --log-prefix 20130710095901_packet_1
> /sbin/iptables -t nat -A prerouting_rule -i eth0 -p tcp --dport 7000 -m
> state --state NEW -m statistic --mode nth --every 2 --packet 1 -j ACCEPT
> # default
> /sbin/iptables -t nat -A prerouting_rule -i eth0 -p tcp --dport 7000 -m
> state --state NEW -j LOG --log-prefix 20130710095901_packet_2
> /sbin/iptables -t nat -A prerouting_rule -i eth0 -p tcp --dport 7000 -m
> state --state NEW -j ACCEPT

Note : the state match is useless here : by definition packets in the
nat table cannot have any other state than NEW.

> ./count_packets.sh 20130710093659
> 0: 500
> 1: 250
> 2: 250

Exactly as expected.

> When i was expecting the following result:
> 
> 0: 500
> 1: 500
> 2: 0
> 
> What i am doing wrong ?

You ACCEPT the packets matching the first statistic set, so they won't
reach the second one. Therefore the second statistic set matches 50% of
the remaining packets, i.e. 50% * 50% = 25%.

Re: Clarification on the use of the statistic module

[Nestor A. Diaz]

Hi, thanks for your answer, as i understand the statistic module use a
static counter that change everytime the packet traverse the chains, i
though the counter got altered just one time while the packet traverse
the chains.

According to your suggestion if i remove the line with the "-j ACCEPT"
then the statistic log as I want and in fact it does.

However if i jump to a 'DNAT' directly, the problem persist as (50/25)
it doesn't work as i have read from some websites

# This doesn't work:

/sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
2 --packet 0 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
DNAT --to-destination 192.168.2.20:7101
/sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
2 --packet 1 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
DNAT --to-destination 192.168.2.20:7102

As solution if I want to jump to DNAT directly then i have to decrease
the 'every' option as follows which do what i want:

# This works:
/sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
2 --packet 0 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
DNAT --to-destination 192.168.2.20:7101
/sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
1 --packet 0 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
DNAT --to-destination 192.168.2.20:7102

I am experimenting with the behavior and if I jump to custom chain which
performs other operations like 'log' statistics keep working as
expected. (50/50) however if i put a 'DNAT' rule things become (50/25),
it seems DNAT affects the behavior but i don't know why,  Any
explanation for this will be appreciated.

# Still don't work:

/sbin/iptables -t nat -N custom_chain_1
/sbin/iptables -t nat -F custom_chain_1
/sbin/iptables -t nat -A custom_chain_1 -j LOG --log-prefix
20130711120831_packet_0
/sbin/iptables -t nat -A custom_chain_1 -i eth0 -s 0.0.0.0/0 -d
192.168.1.1 -p tcp --dport 7100 -j DNAT --to-destination 192.168.2.20:7101

/sbin/iptables -t nat -N custom_chain_2
/sbin/iptables -t nat -F custom_chain_2
/sbin/iptables -t nat -A custom_chain_2 -j LOG --log-prefix
20130711120831_packet_1
/sbin/iptables -t nat -A custom_chain_2 -i eth0 -s 0.0.0.0/0 -d
192.168.1.1 -p tcp --dport 7100 -j DNAT --to-destination 192.168.2.20:7102

/sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
2 --packet 0 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
custom_chain_1
/sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
2 --packet 1 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
custom_chain_2

Slds.

-- 
Typed on my key64.org keyboard

Nestor A Diaz

Re: Clarification on the use of the statistic module

[Pascal Hambourg]

Nestor A. Diaz a écrit :
> Hi, thanks for your answer, as i understand the statistic module use a
> static counter that change everytime the packet traverse the chains, i
> though the counter got altered just one time while the packet traverse
> the chains.

Each occurence of the statistic match has its own individual counter.

> According to your suggestion if i remove the line with the "-j ACCEPT"
> then the statistic log as I want and in fact it does.
> 
> However if i jump to a 'DNAT' directly, the problem persist as (50/25)
> it doesn't work as i have read from some websites

Of course. Like ACCEPT, DNAT is also a terminal target.

> As solution if I want to jump to DNAT directly then i have to decrease
> the 'every' option as follows which do what i want:
> 
> # This works:
> /sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
> 2 --packet 0 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
> DNAT --to-destination 192.168.2.20:7101
> /sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
> 1 --packet 0 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
> DNAT --to-destination 192.168.2.20:7102

You realize that "--every 1" does not make any sense and it is much
simpler to just remove the statistic match in that rule, don't you ?

> I am experimenting with the behavior and if I jump to custom chain which
> performs other operations like 'log' statistics keep working as
> expected. (50/50) however if i put a 'DNAT' rule things become (50/25),
> it seems DNAT affects the behavior but i don't know why,  Any
> explanation for this will be appreciated.

Jumping to a user-defined chain is a good idea if multiple actions are
associated to the same statistic match (e.g. LOG and DNAT). However it
won't change the fact that terminal targets such as ACCEPT, DROP,
REJECT, DNAT... prevent further rules to see the packet, thus change the
actual ratio of further statistic matches.

If the first statistic match takes 1 over N packets, then the next
statistic match will see only the remaining packets, i.e. N-1 over N,
not N. So if you want it to also take 1 over N of all packets, it means
1 over N-1 of the remaining packets. And so on. This is why you had to
decrease the 'every' option. The last rule will take all the remaining
packets without the need for a statistic match

Re: Clarification on the use of the statistic module

[Emilio Lazo Zaia]

Néstor,

First pair or rules (LOG+ACCEPT) matchs on 50% of all packets, all the 
remaining rules matchs on the other 50%, so if you use again statistic, 
it will create a new statistic starting with 50% of the total.

E.g. first packet match first pair or rules, second packet match second 
rule, third packet match again _first_ rule, fourth packet match third 
rule (50% + 25% + 25% -> 500 + 250 + 250 packets).

As Pascal said, you will need to remove the statistic match on the 
second pair.

On 11.07.2013 14:19, Nestor A. Diaz wrote:
> Hi, thanks for your answer, as i understand the statistic module use a
> static counter that change everytime the packet traverse the chains, i
> though the counter got altered just one time while the packet traverse
> the chains.
>
> According to your suggestion if i remove the line with the "-j ACCEPT"
> then the statistic log as I want and in fact it does.
>
> However if i jump to a 'DNAT' directly, the problem persist as (50/25)
> it doesn't work as i have read from some websites
>
> # This doesn't work:
>
> /sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
> 2 --packet 0 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
> DNAT --to-destination 192.168.2.20:7101
> /sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
> 2 --packet 1 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
> DNAT --to-destination 192.168.2.20:7102
>
> As solution if I want to jump to DNAT directly then i have to decrease
> the 'every' option as follows which do what i want:
>
> # This works:
> /sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
> 2 --packet 0 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
> DNAT --to-destination 192.168.2.20:7101
> /sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
> 1 --packet 0 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
> DNAT --to-destination 192.168.2.20:7102
>
> I am experimenting with the behavior and if I jump to custom chain which
> performs other operations like 'log' statistics keep working as
> expected. (50/50) however if i put a 'DNAT' rule things become (50/25),
> it seems DNAT affects the behavior but i don't know why,  Any
> explanation for this will be appreciated.
>
> # Still don't work:
>
> /sbin/iptables -t nat -N custom_chain_1
> /sbin/iptables -t nat -F custom_chain_1
> /sbin/iptables -t nat -A custom_chain_1 -j LOG --log-prefix
> 20130711120831_packet_0
> /sbin/iptables -t nat -A custom_chain_1 -i eth0 -s 0.0.0.0/0 -d
> 192.168.1.1 -p tcp --dport 7100 -j DNAT --to-destination 192.168.2.20:7101
>
> /sbin/iptables -t nat -N custom_chain_2
> /sbin/iptables -t nat -F custom_chain_2
> /sbin/iptables -t nat -A custom_chain_2 -j LOG --log-prefix
> 20130711120831_packet_1
> /sbin/iptables -t nat -A custom_chain_2 -i eth0 -s 0.0.0.0/0 -d
> 192.168.1.1 -p tcp --dport 7100 -j DNAT --to-destination 192.168.2.20:7102
>
> /sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
> 2 --packet 0 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
> custom_chain_1
> /sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
> 2 --packet 1 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
> custom_chain_2
>
> Slds.
>

Re: Clarification on the use of the statistic module

[Nestor A. Diaz]

Hi,Thank you very much for the explanation, that makes things more clear
for me now.

> Each occurence of the statistic match has its own individual counter.
>
>> According to your suggestion if i remove the line with the "-j ACCEPT"
>> then the statistic log as I want and in fact it does.
>>
>> However if i jump to a 'DNAT' directly, the problem persist as (50/25)
>> it doesn't work as i have read from some websites
> Of course. Like ACCEPT, DNAT is also a terminal target.
Ok, good to know :)

>> As solution if I want to jump to DNAT directly then i have to decrease
>> the 'every' option as follows which do what i want:
>>
>> # This works:
>> /sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
>> 2 --packet 0 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
>> DNAT --to-destination 192.168.2.20:7101
>> /sbin/iptables -t nat -A prerouting_rule -m statistic --mode nth --every
>> 1 --packet 0 -i eth0 -s 0.0.0.0/0 -d 192.168.1.1 -p tcp --dport 7100 -j
>> DNAT --to-destination 192.168.2.20:7102
> You realize that "--every 1" does not make any sense and it is much
> simpler to just remove the statistic match in that rule, don't you ?
Yes, but as a personal preference: 'Explicit is better than implicit, no
matter how obvious is'.

I usually make my own bash functions, something like this:

balance tcp eth1 192.168.1.1 7100 0.0.0.0/0 eth0 192.168.2.20 2 7101

which translates to: balance every tcp packet coming from eth1 interface
with destination ip address 192.168.1.1 and port 7100 coming from
0.0.0.0/0 then dnat  via eth0 to 192.168.2.20 ip address and balance
between 2 ports incrementing the port by one starting at 7101.

This is the bash function i use and is now working:

balance() {
    DNAT_PROT=${1}
    DNAT_IN_IFACE=${2}
    DNAT_IN_IP=${3}
    DNAT_IN_PORT=${4}
    DNAT_IN_NET=${5}
    DNAT_OUT_IFACE=${6}
    DNAT_OUT_IP=${7}
    DNAT_OUT_EVERY=${8}
    DNAT_OUT_PORT=${9}

    case $MASQMETHOD in
        netfilter)

            for i in `seq 1 $((${DNAT_OUT_EVERY})) | sort -r`
            do
            balance_port=$((${DNAT_OUT_PORT}+${DNAT_OUT_EVERY}-${i}))
                $IPTABLES -t nat -A prerouting_rule \
                    -m statistic --mode nth --every ${i} --packet 0 \
                    -i ${DNAT_IN_IFACE} -s ${DNAT_IN_NET} -d
${DNAT_IN_IP} -p ${DNAT_PROT} --dport ${DNAT_IN_PORT} \
                    -j DNAT --to-destination ${DNAT_OUT_IP}:${balance_port}
                $IPTABLES -A forwarding_rule -i ${DNAT_IN_IFACE} -o
${DNAT_OUT_IFACE} -s ${DNAT_IN_NET} -d ${DNAT_OUT_IP} -p ${DNAT_PROT}
--dport ${balance_port} -j ACCEPT
                $IPTABLES -A forwarding_rule -i ${DNAT_OUT_IFACE} -o
${DNAT_IN_IFACE} -s ${DNAT_OUT_IP} -d ${DNAT_IN_NET} -p ${DNAT_PROT}
--sport ${balance_port} -j ACCEPT
            done
            ;;
    esac
}


This way i don't have to type the whole iptables pastoril

Even the OpenWRT firewall which is one of the best shell scripts i have
seen for managing iptables always put '-t filter ' no matter if that is
the default.

>> I am experimenting with the behavior and if I jump to custom chain which
>> performs other operations like 'log' statistics keep working as
>> expected. (50/50) however if i put a 'DNAT' rule things become (50/25),
>> it seems DNAT affects the behavior but i don't know why,  Any
>> explanation for this will be appreciated.
> Jumping to a user-defined chain is a good idea if multiple actions are
> associated to the same statistic match (e.g. LOG and DNAT). However it
> won't change the fact that terminal targets such as ACCEPT, DROP,
> REJECT, DNAT... prevent further rules to see the packet, thus change the
> actual ratio of further statistic matches.
>
> If the first statistic match takes 1 over N packets, then the next
> statistic match will see only the remaining packets, i.e. N-1 over N,
> not N. So if you want it to also take 1 over N of all packets, it means
> 1 over N-1 of the remaining packets. And so on. This is why you had to
> decrease the 'every' option. The last rule will take all the remaining
> packets without the need for a statistic match

Slds.

-- 
Typed on my key64.org keyboard

Nestor A Diaz