[LARTC] Load balancing using connmark

[LARTC] Load balancing using connmark

[Francis Brosnan Blazquez]

Hi,

I've been implementing a load balancing solution using CONNMARK, based
on solution described by Luciano Ruete at [1]. Gracias por el post y por
apuntar en la direcciÛn correcta Luciano!

Once implemented, I've found that due to some reason packets aren't
properly marked (or improperly remarked) and sent out using the wrong
interface. 

My topo setup is:

[82.123.136.74]: eth1 : mark:0x1 --\
                                    +--[FW BOX] -- eth0: 192.168.0.53
[217.146.74.82]: eth2 : mark:0x2 --/

Using conntrack tool, shows that after a while, it starts to appear
packets marked with 0x2 or 0x1 not comming from the proper source IP.

>> conntrack -L | grep mark=2 | grep '82.123.136.74'; conntrack -L |
grep mark=1 | grep '217.146.74.82'

tcp      6 425543 ESTABLISHED src\x192.168.0.178 dstÇ.216.53.249
sport\x1552 dportD3 packetsÅ8 bytesì471 srcÇ.216.53.249
dstÇ.123.136.74 sportD3 dport\x1552 packetsá5 bytesÉ909 [ASSURED]
mark=2 use=1
tcp      6 428681 ESTABLISHED src\x192.168.0.177 dstâ.139.122.12
sport#61 dportD3 packets\x122 bytes)381 srcâ.139.122.12
dstÇ.123.136.74 sportD3 dport#61 packets\x139 bytes\x14120 [ASSURED]
mark=2 use=1

This is quite odd since solution proposed at [1] looks good. I'll cite
it here for clarity (suppose I already have all ip rule stuff
installed):

iptables -t mangle -A POSTROUTING -m mark  --mark ! 0 -j ACCEPT 
iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1
iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark

After giving a try during several days, I've found that another firewall
solution, shorewall [2], implements built-in load balacing for free by
using the following set of instructions:

iptables -t mangle -A PREROUTING -m connmark ! --mark 0/0xFF -j CONNMARK
--restore-mark --mask 0xFF

iptables -t mangle -A OUTPUT -m connmark ! --mark 0/0xFF -j CONNMARK
--restore-mark --mask 0xFF

iptables -t mangle -N routemark
iptables -t mangle -A PREROUTING -i eth1 -m mark --mark 0/0xFF -j
routemark

iptables -t mangle -A routemark -i eth1 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -i eth2 -m mark --mark 0/0xFF -j
routemark

iptables -t mangle -A routemark -i eth2 -j MARK --set-mark 2
iptables -t mangle -A routemark -m mark ! --mark 0/0xFF -j CONNMARK
--save-mark --mask 0xFF

After a bit of testing with the second solution, it seems to behave
better, doing all marking job at the PREROUTING and OUTPUT.

Did anybody find that some packages doesn't get properly routed
according to the mark with the first solution? What you do think about
the second solution?

Cheers!

[1] http://mailman.ds9a.nl/pipermail/lartc/2006q2/018964.html
[2] http://www.shorewall.net


-- 
Francis Brosnan Blazquez <francis@aspl.es>
Advanced Software Production Line, S.L.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Load balancing using connmark

[Peter Rabbitson]

Francis Brosnan Blazquez wrote:
> Hi,
> 
> I've been implementing a load balancing solution using CONNMARK, based
> on solution described by Luciano Ruete at [1]. Gracias por el post y por
> apuntar en la dirección correcta Luciano!
> 
> Once implemented, I've found that due to some reason packets aren't
> properly marked (or improperly remarked) and sent out using the wrong
> interface. 
> 
> <snip>
> 
> iptables -t mangle -A POSTROUTING -m mark  --mark ! 0 -j ACCEPT 
> iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1
> iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2
> iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark

This is wrong. POSTROUTING is exactly what is is _POST_ routing. By the
time you do your marks and stuff the kernel has _already_ assigned a
packet to an interface, and you can not alter this anymore.

> After a bit of testing with the second solution, it seems to behave
> better, doing all marking job at the PREROUTING and OUTPUT.

This is flawed too. OUTPUT suffers from the very same problem as
POSTROUTING - by the time the packets hit the NF stack the process has
already bound itself to an interface, which you can not change anymore.

Peter




_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Load balancing using connmark

[Salim S I]

[-- Attachment #1.1: Type: text/plain, Size: 1876 bytes --]

Francis Brosnan Blazquez wrote:
> Hi,
> 
> I've been implementing a load balancing solution using CONNMARK, based
> on solution described by Luciano Ruete at [1]. Gracias por el post y
por
> apuntar en la dirección correcta Luciano!
> 
> Once implemented, I've found that due to some reason packets aren't
> properly marked (or improperly remarked) and sent out using the wrong
> interface. 
> 
> <snip>
> 
> iptables -t mangle -A POSTROUTING -m mark  --mark ! 0 -j ACCEPT 
> iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1
> iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2
> iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
 
This is wrong. POSTROUTING is exactly what is is _POST_ routing. By the
time you do your marks and stuff the kernel has _already_ assigned a
packet to an interface, and you can not alter this anymore.
 
> After a bit of testing with the second solution, it seems to behave
> better, doing all marking job at the PREROUTING and OUTPUT.
 
This is flawed too. OUTPUT suffers from the very same problem as
POSTROUTING - by the time the packets hit the NF stack the process has
already bound itself to an interface, which you can not change anymore.
 
Peter
 
Disagree with Peter. The marking in postrouting table is CONNMARK. This
is for marking the connection, which has already had a route decided for
it, so that all packets of the connection passes through this interface.
This marking is done for packets with NEW state, see the check for
mark==0 in the prev. line. The restore mark in PREROUTING will restore
the connmark and route the subsequent packets.
This approach will work, but you need some sort of stateful-ness in
netfilter.
 
The second point in Brosnan Blazquez’s mail about shorewall: They seem
to be doing Policy Routing, not real load balancing.

[-- Attachment #1.2: Type: text/html, Size: 9859 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Load balancing using connmark

[Salim S I]

[-- Attachment #1.1: Type: text/plain, Size: 2446 bytes --]

On closer look, I am wrong about shorewall. It seems to be a different
approach to load balancing. They connmark the incoming packets from WAN,
rather than outgoing packets. I think it should work well, but I wonder
why this approach is not popular. There must be some drawback to it. I
can’t think of one,though.
 
-----Original Message-----
From: lartc-bounces@mailman.ds9a.nl
[mailto:lartc-bounces@mailman.ds9a.nl] On Behalf Of Salim S I
Sent: Thursday, May 10, 2007 2:15 PM
To: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Load balancing using connmark
 
Francis Brosnan Blazquez wrote:
> Hi,
> 
> I've been implementing a load balancing solution using CONNMARK, based
> on solution described by Luciano Ruete at [1]. Gracias por el post y
por
> apuntar en la dirección correcta Luciano!
> 
> Once implemented, I've found that due to some reason packets aren't
> properly marked (or improperly remarked) and sent out using the wrong
> interface. 
> 
> <snip>
> 
> iptables -t mangle -A POSTROUTING -m mark  --mark ! 0 -j ACCEPT 
> iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1
> iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2
> iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
 
This is wrong. POSTROUTING is exactly what is is _POST_ routing. By the
time you do your marks and stuff the kernel has _already_ assigned a
packet to an interface, and you can not alter this anymore.
 
> After a bit of testing with the second solution, it seems to behave
> better, doing all marking job at the PREROUTING and OUTPUT.
 
This is flawed too. OUTPUT suffers from the very same problem as
POSTROUTING - by the time the packets hit the NF stack the process has
already bound itself to an interface, which you can not change anymore.
 
Peter
 
Disagree with Peter. The marking in postrouting table is CONNMARK. This
is for marking the connection, which has already had a route decided for
it, so that all packets of the connection passes through this interface.
This marking is done for packets with NEW state, see the check for
mark==0 in the prev. line. The restore mark in PREROUTING will restore
the connmark and route the subsequent packets.
This approach will work, but you need some sort of stateful-ness in
netfilter.
 
The second point in Brosnan Blazquez’s mail about shorewall: They seem
to be doing Policy Routing, not real load balancing.

[-- Attachment #1.2: Type: text/html, Size: 12606 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Load balancing using connmark

[Francis Brosnan Blazquez]

El jue, 10-05-2007 a las 16:01 +0800, Salim S I escribió:
Hi Salim,

Thanks for your reply,

> On closer look, I am wrong about shorewall. It seems to be a different
> approach to load balancing. They connmark the incoming packets from
> WAN, rather than outgoing packets. I think it should work well, but I
> wonder why this approach is not popular. There must be some drawback
> to it. I can’t think of one,though.

I think the main advantage of shorewall solution is that it applies
connmark to incoming packets from the wan as you point, leaving load
balancing to outgoing connections to the main table.

In any case, with this second solution I don't see wrong routed packages
on wan interfaces using tcpdump, whereas with the first solution I do.
More testing is required.

Regarding to your previous reply, can you elaborate more on "...This
approach will work, but you need some sort of stateful-ness in
netfilter..."

Cheers!

-- 
Francis Brosnan Blazquez <francis@aspl.es>
Advanced Software Production Line, S.L.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

FW: [LARTC] Load balancing using connmark

[Salim S I]

-----Original Message-----
From: Salim S I [mailto:salim.si@cipherium.com.tw] 
Sent: Thursday, May 10, 2007 5:22 PM
To: 'Francis Brosnan Blazquez'
Subject: RE: [LARTC] Load balancing using connmark

"I think the main advantage of shorewall solution is that it applies
connmark to incoming packets from the wan as you point, leaving load
balancing to outgoing connections to the main table"

Actually, the main table/multipath route only routes the first packet of
a connection. The subsequent routing for that connection is done based
on connmark, for outgoing packets too. Otherwise replies to packets
coming from WAN1 may go through WAN2. The difference in the two
solutions is only in where packets are marked and which packets are
marked. Routing is the same.

For a detailed discussion on the first approach, you can refer to this
thread. 

http://mailman.ds9a.nl/pipermail/lartc/2006q2/018964.html


-----Original Message-----
From: Francis Brosnan Blazquez [mailto:francis@aspl.es] 
Sent: Thursday, May 10, 2007 5:07 PM
To: Salim S I
Cc: lartc@mailman.ds9a.nl
Subject: RE: [LARTC] Load balancing using connmark

El jue, 10-05-2007 a las 16:01 +0800, Salim S I escribió:
Hi Salim,

Thanks for your reply,

> On closer look, I am wrong about shorewall. It seems to be a different
> approach to load balancing. They connmark the incoming packets from
> WAN, rather than outgoing packets. I think it should work well, but I
> wonder why this approach is not popular. There must be some drawback
> to it. I can’t think of one,though.

I think the main advantage of shorewall solution is that it applies
connmark to incoming packets from the wan as you point, leaving load
balancing to outgoing connections to the main table.

In any case, with this second solution I don't see wrong routed packages
on wan interfaces using tcpdump, whereas with the first solution I do.
More testing is required.

Regarding to your previous reply, can you elaborate more on "...This
approach will work, but you need some sort of stateful-ness in
netfilter..."

Cheers!

-- 
Francis Brosnan Blazquez <francis@aspl.es>
Advanced Software Production Line, S.L.


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Load balancing using connmark

[Peter Warasin]

hi people

Francis Brosnan Blazquez wrote:
> I've been implementing a load balancing solution using CONNMARK, based

> After giving a try during several days, I've found that another firewall
> solution, shorewall [2], implements built-in load balacing for free by
> using the following set of instructions:

did somebody try the shorewall solution with centos 4?

with centos 4 and the first solution i always had the problem, that it
routes correctly only for passing through connections (forwarded).
connections starting from the machine or hoing to the machine
(input/output chain) had exactly the same behaviour as you stated before.

i noticed with centos 4 that packets do not pass the prerouting magle
chain if going to the local host (passing the input filter chain
thereafter). therefore certainly the mark will not be restored and there
will be no influence on the routing decision.
someone noticed similar behaviour?

peter

-- 
:: e n d i a n
:: open source - open minds

:: peter warasin
:: http://www.endian.com   :: peter@endian.com
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Load balancing using connmark

[Peter Rabbitson]

Salim S I wrote:
> Francis Brosnan Blazquez wrote:
> 
>> Hi,
> 
>> 
> 
>> I've been implementing a load balancing solution using CONNMARK, based
> 
>> on solution described by Luciano Ruete at [1]. Gracias por el post y por
> 
>> apuntar en la dirección correcta Luciano!
> 
>> 
> 
>> Once implemented, I've found that due to some reason packets aren't
> 
>> properly marked (or improperly remarked) and sent out using the wrong
> 
>> interface. 
> 
>> 
> 
>> <snip>
> 
>> 
> 
>> iptables -t mangle -A POSTROUTING -m mark  --mark ! 0 -j ACCEPT 
> 
>> iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1
> 
>> iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2
> 
>> iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
> 
>  
> 
> This is wrong. POSTROUTING is exactly what is is _POST_ routing. By the
> 
> time you do your marks and stuff the kernel has _already_ assigned a
> 
> packet to an interface, and you can not alter this anymore.
> 
>  
> 
>> After a bit of testing with the second solution, it seems to behave
> 
>> better, doing all marking job at the PREROUTING and OUTPUT.
> 
>  
> 
> This is flawed too. OUTPUT suffers from the very same problem as
> 
> POSTROUTING - by the time the packets hit the NF stack the process has
> 
> already bound itself to an interface, which you can not change anymore.
> 
>  
> 
> Peter
> 
>  
> 
> Disagree with Peter. The marking in postrouting table is CONNMARK. This
> is for marking the connection, which has already had a route decided for
> it, so that all packets of the connection passes through this interface.
> This marking is done for packets with NEW state, see the check for
> mark=0 in the prev. line. The restore mark in PREROUTING will restore
> the connmark and route the subsequent packets.
> 
> This approach will work, but you need some sort of stateful-ness in
> netfilter.
> 

Connmark is exactly the statefullness you are talking about. The problem
is that the marks by themselves do not mean anything. You mark packets
and expect iproute to classify the packet in the correct routing table
etc. CONNMARK is invisible to iproute - this is why you have only
--save-mark and --restore-mark, and the rest of the rules deal with real
MARKs.

Further you (and the OP) seem to be confused by a mix of routing tasks.
In the case of _forwarded_ traffic, you need to make sure that all
packets within a connection leave to WAN over the same interface, and
are SNATed to the same ip, so that they will come bak the same
interface. The SNATting is trivial (as it can be done in POSTROUTING
only), but you need to set all marks before the routing takes place
(which is anywhere _but_ POSTROUTING). You might mark the connection
with the proper CONNMARK. and subsequent packets might get routed
correctly, but the _first_ packet (the one that you use to set the mark)
is already assigned to an interface, and there is nothing you can do
about it.

In the case of _local_ traffic - it becomes even trickier. The problem
is that when sockets are created they already have a source IP (the
kernel determines that by looking at the default routing table, your
marks do not exist yet). So since you can not alter the socket binding,
the only way to make it leave on a different interface is by treating it
as a forwarded connection and performing NAT on it. It is arguable if
NATting locally originating connections is a good idea, but it can be
done in OUTPUT just like it is done for forwarder connections in
PREROUTING.

I hope this clarifies things a bit, feel free to point out any
inconsistencies you may find.

Peter

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Load balancing using connmark

[Peter Rabbitson]

Peter Rabbitson wrote:
> ...
> In the case of _local_ traffic - it becomes even trickier. The problem
> is that when sockets are created they already have a source IP (the
> kernel determines that by looking at the default routing table, your
> marks do not exist yet). 

This is misleading - it will happen only when the application does not
request a specific ip/interface to bind to. Only then the kernel default
table is consulted, and the best interface is determined based on the
destination that is supplied on socket creation.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Load balancing using connmark

[Salim S I]

Let me explain why the marking is done in POSTROUTING.

The first packet of any connection get routed by the multipath routing
entry. This happens AFTER PREROUTING, as you know. And this is what we
want, letting the kernel decide based on the weights. (some people do
think that we shouldn't let multipath decide routing, but thatz a
different story).

So where can this packet be marked? Obviously in POSTROUTING (so that
local pkts also can be caught). We mark it and save it.(connmark).The
mark is decoded by the chosen interface. (eg:-o WAN1 --set mark 1,-o
WAN2 --set-mark 2)

In PREROUTING, there is a restore-mark. You see

iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark.

If this packet belong to a connection that has already sent a
packet,this will restore the mark set in POSTROUTING. Then it will be
routed by the corresponding routing table.(wan1 table lookup mark1 and
wan2 table lookup mark2)
If it is a new pkt, it will be routed by multipath routing
statement,since no mark exists.

-----Original Message-----
From: lartc-bounces@mailman.ds9a.nl
[mailto:lartc-bounces@mailman.ds9a.nl] On Behalf Of Peter Rabbitson
Sent: Thursday, May 10, 2007 6:51 PM
Cc: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Load balancing using connmark

Salim S I wrote:
> Francis Brosnan Blazquez wrote:
> 
>> Hi,
> 
>> 
> 
>> I've been implementing a load balancing solution using CONNMARK,
based
> 
>> on solution described by Luciano Ruete at [1]. Gracias por el post y
por
> 
>> apuntar en la dirección correcta Luciano!
> 
>> 
> 
>> Once implemented, I've found that due to some reason packets aren't
> 
>> properly marked (or improperly remarked) and sent out using the wrong
> 
>> interface. 
> 
>> 
> 
>> <snip>
> 
>> 
> 
>> iptables -t mangle -A POSTROUTING -m mark  --mark ! 0 -j ACCEPT 
> 
>> iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1
> 
>> iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2
> 
>> iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
> 
>  
> 
> This is wrong. POSTROUTING is exactly what is is _POST_ routing. By
the
> 
> time you do your marks and stuff the kernel has _already_ assigned a
> 
> packet to an interface, and you can not alter this anymore.
> 
>  
> 
>> After a bit of testing with the second solution, it seems to behave
> 
>> better, doing all marking job at the PREROUTING and OUTPUT.
> 
>  
> 
> This is flawed too. OUTPUT suffers from the very same problem as
> 
> POSTROUTING - by the time the packets hit the NF stack the process has
> 
> already bound itself to an interface, which you can not change
anymore.
> 
>  
> 
> Peter
> 
>  
> 
> Disagree with Peter. The marking in postrouting table is CONNMARK.
This
> is for marking the connection, which has already had a route decided
for
> it, so that all packets of the connection passes through this
interface.
> This marking is done for packets with NEW state, see the check for
> mark=0 in the prev. line. The restore mark in PREROUTING will restore
> the connmark and route the subsequent packets.
> 
> This approach will work, but you need some sort of stateful-ness in
> netfilter.
> 

Connmark is exactly the statefullness you are talking about. The problem
is that the marks by themselves do not mean anything. You mark packets
and expect iproute to classify the packet in the correct routing table
etc. CONNMARK is invisible to iproute - this is why you have only
--save-mark and --restore-mark, and the rest of the rules deal with real
MARKs.

Further you (and the OP) seem to be confused by a mix of routing tasks.
In the case of _forwarded_ traffic, you need to make sure that all
packets within a connection leave to WAN over the same interface, and
are SNATed to the same ip, so that they will come bak the same
interface. The SNATting is trivial (as it can be done in POSTROUTING
only), but you need to set all marks before the routing takes place
(which is anywhere _but_ POSTROUTING). You might mark the connection
with the proper CONNMARK. and subsequent packets might get routed
correctly, but the _first_ packet (the one that you use to set the mark)
is already assigned to an interface, and there is nothing you can do
about it.

In the case of _local_ traffic - it becomes even trickier. The problem
is that when sockets are created they already have a source IP (the
kernel determines that by looking at the default routing table, your
marks do not exist yet). So since you can not alter the socket binding,
the only way to make it leave on a different interface is by treating it
as a forwarded connection and performing NAT on it. It is arguable if
NATting locally originating connections is a good idea, but it can be
done in OUTPUT just like it is done for forwarder connections in
PREROUTING.

I hope this clarifies things a bit, feel free to point out any
inconsistencies you may find.

Peter

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Load balancing using connmark

[David Ford]

Is there a good [single?] document explaining all of this and more?
What the kernel does in POST vs PRE with respect to iproute2 and
netfilter with CONNMARK and etc?

Thank you,
David

Salim S I wrote:
> Let me explain why the marking is done in POSTROUTING.
> [...]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Load balancing using connmark

[Peter Rabbitson]

Salim S I wrote:
> Let me explain why the marking is done in POSTROUTING.
> 
> want, letting the kernel decide based on the weights. (some people do
> think that we shouldn't let multipath decide routing, but thatz a
> different story).

I apologize, as I am one of these people, and subsequently assumed the
OP wanted this. In this light I agree with Salim. On an unrelated note
the OP should be aware that letting multipath do the balancing is
impractical (i.e. does not work) in real life scenarios, but this indeed
is a topic for a separate thread.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc