Re: Multiple load balancers problem

[Dmitry Akindinov <dimak@stalker.com>]

Hello,

A small addition below.

On 2012-08-25 11:37, Dmitry Akindinov wrote:
> Hello,
>
> We are currently stuck with the following ipvs problem:
>
> 1. The configuration includes a (potentially large) set of servers
> providing various services - besides HTTP (POP, IMAP, LDAP, SMTP, XMPP,
> etc.) The test setup includes just 2 servers, though.
> 2. Each server runs a stock version of CentOS 6.0
> 3. The application software (CommuniGate Pro) controls the ipvs kernel
> module using the ipvsadm commands.
> 4. On each server, iptables are configured to:
> a) disable connection tracking for VIP address(es)
> b) mark all packets coming to the VIP address(es) with the mark value of
> 100.
> 5. On the currently active load balancer, the ipvsadm is used to
> configure ipvs to load-balance packets with the marker 100:
> -A -f 100 -s rr -p 1
> -a -f 100 -r <server1> -g
> -a -f 100 -r <server2> -g
> ....
> where the active balancer itself is one of the <serverN>
> 6. All other servers (just 1 "other" server in our test config) are
> running ipvs, but with an empty rule set.
> 7. The active load balancer runs the sync daemon started with ipvsadm
> --start-daemon master
> 7. All other servers run the sync daemon started with ipvsadm
> --start-daemon backup.
>
> As a result, all servers have the duplicated ipvs connection tables. If
> the active balancer fails, some other server assumes its role by
> arp-broadcasting VIP and loading the ipvs rule set listed above.
>
> When a connection is being established to the VIP address, and the
> active load balancer directs it to itself, everything works fine.
> When a connection is being established to the VIP address, and the
> active load balancer directs it to some other server, the connection is
> established fine, and if the protocol is POP, IMAP, SMTP, the server
> prompt is sent to the client via VIP, and it is seen by client just fine.
> But when the client tries to send anything to the server, the packet
> (according to tcpdump) reaches the load balancer server, and from there
> it reaches the "other" server. Where the packet is dropped. The client
> resends that packet, it goes to the active balancer, then to the "other"
> server, and it is dropped again.
>
>
> Observations:
> *) if ipvs is switched off on that "other" server, everything works just
> fine (service ipvsadm stop)
>
> *) if ipvs is left running on that "other" server, but syncing daemon is
> switched off, everything works just fine.
> We are 95% sure that the problem appears only if the "other server" ipvs
> connection table gets a copy of this
> connection from the active balancer. If the copy is not there (the sync
> daemon was stopped when the connection
> was established, and restarted immediately after), everything works just
> fine.
>
> *) the problem exists for protocols like POP, IMAP, SMTP - where the
> server immediately sends some data (prompt) to the client, as soon as
> the connection is established.
> When the HTTP protocol is used, the problem does not exist, but only if
> the entire request is sent as one packet. If the HTTP connection is a
> "keep-alive" one, subsequent requests in the same connection do not
> reach the application either.
> I.e. it looks like the "idling" ipvs allows only one incoming data
> packet in, and only if there has been no outgoing packet on that
> connection yet.
>
> *) Sometimes (we still cannot reproduce this reliably) the ksoftirqd
> threads on the "other" server jump to 100% CPU
> utilization, and when it happens, it happens in reaction to one
> connection being established.

And when a new connection is being established, the second ksoftirqd 
thread using 100% CPU appears in the "top" output, and so on - till all 
ksoftirqd threads (8 in case of our 8-CPU test servers) are looping 
somewhere, consuming most of CPU cycles.

> Received suggestions:
> *) it was suggested that we use iptables to filter the packets to VIP
> that come from other servers in the farm (using their MAC addresses) and
> direct them directly to the local application, bypassing ipvs
> processing. We cannot do that, as servers in the farm can be added at
> any moment, and updating the list of MACs on all servers is not trivial.
> It may be easier to filter the packets that come from the router(s),
> which are less numerous and do not change that often.
> But it does not look like a good solution. If the ipvs table on
> "inactive" balancer drops packets, why would it stop dropping them when
> it becomes an "active" balancer? Just because there will be ipvs rules
> present?
>
> *) The suggestion to separate load balancer(s) and real servers won't
> work for us at all.
>
> *) We tried not to empty the ipvs table on the "other" server(s).
> Instead, we left it balancing - but with only one "real server" - this
> server itself. Now, the "active" load balancer dsitributes packets to
> itself and other servers, and when the packets hit the "other"
> server(s), they get to the ipvs again, where they are balanced again,
> but to the local server only.
>
> It looks like it does solve the problem. But now the ipvs connection
> table on the "other" server(s) is filled by both that server ipvs itself
> and by the sync-daemon. While the locally-generated connection table
> entries should be the same as corresponding entries received with the
> sync daemon, it does not look good when the same table is modified from
> two sources.
>
> Any comment, please? Should we use the last suggestion?
>
>


-- 
Best regards,
Dmitry Akindinov