Configuring keepalived + ipvs with persistence: doesn't balance to other node if initial persistent node fails.

Configuring keepalived + ipvs with persistence: doesn't balance to other node if initial persistent node fails.

[Jose Luis Godoy]

Description of problem:

Configuring keepalived + ipvs with persistence: doesn't balance to
other node if initial persistent node fails.

Version-Release number of selected component (if applicable):

keepalived-1.2.9-1.fc19.x86_64
ipvsadm-1.27-1.fc19.x86_64
/lib/modules/3.11.9-200.fc19.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko

How reproducible:
Always.

Steps to Reproduce:
1. /etc/keepalived/keepalived.conf -> this is part of the configuration file:
...
virtual_server 192.168.58.10 443 {
delay_loop 5
lb_algo rr
lb_kind DR
persistence_timeout 3600
persistence_granularity 255.255.255.255
protocol TCP

sorry_server 192.168.58.200 443

real_server 192.168.58.201 443 {
    weight 1
    inhibit_on_failure
    TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 1
    }
}
real_server 192.168.58.202 443 {
    weight 1
    inhibit_on_failure
    TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 1
    }
}
...

2. systemctl restart keepalived

# ipvsadm -L -n --persistent-conn

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Weight PersistConn ActiveConn InActConn
-> RemoteAddress:Port
TCP 192.168.58.10:80 rr
-> 192.168.58.201:80 1 0 0 0
-> 192.168.58.202:80 1 0 0 0
TCP 192.168.58.10:443 rr persistent 3600
-> 192.168.58.201:443 1 0 0 0
-> 192.168.58.202:443 1 0 0 0

3. At first we connect to 192.168.58.10:443 four times from the same
source address.

# ipvsadm -L -n --persistent-conn

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Weight PersistConn ActiveConn InActConn
  -> RemoteAddress:Port
TCP 192.168.58.10:80 rr
  -> 192.168.58.201:80 1 0 0 0
  -> 192.168.58.202:80 1 0 0 0
TCP 192.168.58.10:443 rr persistent 3600
  -> 192.168.58.201:443 1 1 0 4
  -> 192.168.58.202:443 1 0 0 0

4. We stop the persistent node 192.168.58.201:

# ipvsadm -L -n --persistent-conn

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Weight PersistConn ActiveConn InActConn
  -> RemoteAddress:Port
TCP 192.168.58.10:80 rr
  -> 192.168.58.201:80 1 0 0 0
  -> 192.168.58.202:80 1 0 0 0
TCP 192.168.58.10:443 rr persistent 3600
  -> 192.168.58.201:443 0 1 0 4
  -> 192.168.58.202:443 1 0 0 0

4. We connect again to 192.168.58.10:443 four times from the initial
source address.

# ipvsadm -L -n --persistent-conn

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Weight PersistConn ActiveConn InActConn
  -> RemoteAddress:Port
TCP 192.168.58.10:80 rr
  -> 192.168.58.201:80 1 0 0 0
  -> 192.168.58.202:80 1 0 0 0
TCP 192.168.58.10:443 rr persistent 3600
  -> 192.168.58.201:443 0 1 0 4
  -> 192.168.58.202:443 1 0 0 0 <<<--- There aren't connections

Expected results:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Weight PersistConn ActiveConn InActConn
  -> RemoteAddress:Port
TCP 192.168.58.10:80 rr
  -> 192.168.58.201:80 1 0 0 0
  -> 192.168.58.202:80 1 0 0 0
TCP 192.168.58.10:443 rr persistent 3600
  -> 192.168.58.201:443 0 1 0 4
  -> 192.168.58.202:443 1 0 0 4 <<<--- We must get four connections

Reference to bug in Fedora Project:

https://bugzilla.redhat.com/show_bug.cgi?id=1040405


Additional info:

The first node is stopped, IPVS doesn't balance to the other web
server node 192.168.58.202:443 and waits 3600 seconds (in this case)
to balance to it. We don't know te reason to do this, in our opinion
it must balance to the other web server node, we think it could be a
possible bug.

Can anyone explain this behaviour or is really it a bug?

Thanks!

Jose Luis

Re: Configuring keepalived + ipvs with persistence: doesn't balance to other node if initial persistent node fails.

[Graeme Fowler]

On 11/12/2013 15:26, Jose Luis Godoy wrote:
> real_server 192.168.58.201 443 {
>      weight 1
>      inhibit_on_failure

Remove inhibit_on_failure.

That keeps the server in the pool but with weight 0, meaning no *new* 
connections go to it. Existing persistent connections remain.

Alternatively, if you don't want to do that, set the sysctl 
net.ipv4.vs.expire_quiescent_template to 1, and keep inhibit_on_failure.

Both options are in effect synonyms, meaning in both cases that the 
persistence templates are removed for the VS in question when either it 
is removed from the pool (removing inhibit_on_failure) or quiesced 
(weight set to 0).

Graeme

Re: Configuring keepalived + ipvs with persistence: doesn't balance to other node if initial persistent node fails.

[Ryan O'Hara]

On Wed, Dec 11, 2013 at 04:26:18PM +0100, Jose Luis Godoy wrote:
> Description of problem:
> 
> Configuring keepalived + ipvs with persistence: doesn't balance to
> other node if initial persistent node fails.


The problem is that your using "inhibit_on_failure" for quiescent
servers and persistence. To get the behavior your are expecting:

echo 1 > /proc/sys/net/ipv4/vs/expire_quiescent_template


Ryan

> Version-Release number of selected component (if applicable):
> 
> keepalived-1.2.9-1.fc19.x86_64
> ipvsadm-1.27-1.fc19.x86_64
> /lib/modules/3.11.9-200.fc19.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko
> 
> How reproducible:
> Always.
> 
> Steps to Reproduce:
> 1. /etc/keepalived/keepalived.conf -> this is part of the configuration file:
> ...
> virtual_server 192.168.58.10 443 {
> delay_loop 5
> lb_algo rr
> lb_kind DR
> persistence_timeout 3600
> persistence_granularity 255.255.255.255
> protocol TCP
> 
> sorry_server 192.168.58.200 443
> 
> real_server 192.168.58.201 443 {
>     weight 1
>     inhibit_on_failure
>     TCP_CHECK {
>         connect_timeout 3
>         nb_get_retry 3
>         delay_before_retry 1
>     }
> }
> real_server 192.168.58.202 443 {
>     weight 1
>     inhibit_on_failure
>     TCP_CHECK {
>         connect_timeout 3
>         nb_get_retry 3
>         delay_before_retry 1
>     }
> }
> ...
> 
> 2. systemctl restart keepalived
> 
> # ipvsadm -L -n --persistent-conn
> 
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Weight PersistConn ActiveConn InActConn
> -> RemoteAddress:Port
> TCP 192.168.58.10:80 rr
> -> 192.168.58.201:80 1 0 0 0
> -> 192.168.58.202:80 1 0 0 0
> TCP 192.168.58.10:443 rr persistent 3600
> -> 192.168.58.201:443 1 0 0 0
> -> 192.168.58.202:443 1 0 0 0
> 
> 3. At first we connect to 192.168.58.10:443 four times from the same
> source address.
> 
> # ipvsadm -L -n --persistent-conn
> 
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Weight PersistConn ActiveConn InActConn
>   -> RemoteAddress:Port
> TCP 192.168.58.10:80 rr
>   -> 192.168.58.201:80 1 0 0 0
>   -> 192.168.58.202:80 1 0 0 0
> TCP 192.168.58.10:443 rr persistent 3600
>   -> 192.168.58.201:443 1 1 0 4
>   -> 192.168.58.202:443 1 0 0 0
> 
> 4. We stop the persistent node 192.168.58.201:
> 
> # ipvsadm -L -n --persistent-conn
> 
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Weight PersistConn ActiveConn InActConn
>   -> RemoteAddress:Port
> TCP 192.168.58.10:80 rr
>   -> 192.168.58.201:80 1 0 0 0
>   -> 192.168.58.202:80 1 0 0 0
> TCP 192.168.58.10:443 rr persistent 3600
>   -> 192.168.58.201:443 0 1 0 4
>   -> 192.168.58.202:443 1 0 0 0
> 
> 4. We connect again to 192.168.58.10:443 four times from the initial
> source address.
> 
> # ipvsadm -L -n --persistent-conn
> 
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Weight PersistConn ActiveConn InActConn
>   -> RemoteAddress:Port
> TCP 192.168.58.10:80 rr
>   -> 192.168.58.201:80 1 0 0 0
>   -> 192.168.58.202:80 1 0 0 0
> TCP 192.168.58.10:443 rr persistent 3600
>   -> 192.168.58.201:443 0 1 0 4
>   -> 192.168.58.202:443 1 0 0 0 <<<--- There aren't connections
> 
> Expected results:
> 
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Weight PersistConn ActiveConn InActConn
>   -> RemoteAddress:Port
> TCP 192.168.58.10:80 rr
>   -> 192.168.58.201:80 1 0 0 0
>   -> 192.168.58.202:80 1 0 0 0
> TCP 192.168.58.10:443 rr persistent 3600
>   -> 192.168.58.201:443 0 1 0 4
>   -> 192.168.58.202:443 1 0 0 4 <<<--- We must get four connections
> 
> Reference to bug in Fedora Project:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1040405
> 
> 
> Additional info:
> 
> The first node is stopped, IPVS doesn't balance to the other web
> server node 192.168.58.202:443 and waits 3600 seconds (in this case)
> to balance to it. We don't know te reason to do this, in our opinion
> it must balance to the other web server node, we think it could be a
> possible bug.
> 
> Can anyone explain this behaviour or is really it a bug?
> 
> Thanks!
> 
> Jose Luis
> --
> To unsubscribe from this list: send the line "unsubscribe lvs-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html