[LARTC] Hardware Configuration Ideas

[LARTC] Hardware Configuration Ideas

[Mihai Vlad]

Hey guys,

I am planning to buy some components for a Linux router that will handle the
Internet access of 200 computers (includes tc shaping) and some inter
sub-network routing (at least 100MBps per eth - and there are 3 eth cards).

I was thinking of a:
Pentium 4 - 3GHz
256 or 512MB RAM
Network Cards.

Now - I wonder what is more important: the processor speed or the amount of
RAM.

And can you point me to some good Network Cards that you have used and are
not so expensive. Some Intel, etc. I have no clue about this subject...


Maybe this discussion can be extended to a list of best practices to set up
a performant Linux Router from the hardware point of view.

Thanks in advance,
Mihai


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Hardware Configuration Ideas

[Nickola Kolev]

[-- Attachment #1.1: Type: text/plain, Size: 998 bytes --]

Hello, Mihai,

On Mon, 15 Aug 2005 23:53:38 +0300
"Mihai Vlad" <mihaivlad@web-profile.net> wrote:

> Hey guys,
> 
> I am planning to buy some components for a Linux router that will
> handle the Internet access of 200 computers (includes tc shaping) and
> some inter sub-network routing (at least 100MBps per eth - and there
> are 3 eth cards).
> 
> I was thinking of a:
> Pentium 4 - 3GHz
> 256 or 512MB RAM
> Network Cards.

This is more than enough - I'd suggest a PIII/1GHz with 512MB RAM for
those kind of needs, which is also a bit generous.

[ cut ]
> And can you point me to some good Network Cards that you have used
> and are not so expensive. Some Intel, etc. I have no clue about this
> subject...

Network cards - YMMV, but I'd go for Intel EtherXpress Pro/100 for the
100Mbit, and something based on BCM57xx for the gigabit speeds.

My experience dictates that for now the 2.4.30+ kernels are more
suitable for tc/QoS setups.

Hope it helped,
Nickola

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Hardware Configuration Ideas

[Taylor, Grant]

Memory will most definitely be your problem.  I think you could get away with a fairly low end processor (read < 1 GHz) but you will need a lot of memory depending on how much you want to do.  I have a router in place that I was running out of memory for the connection tracking sub system.  I ended up allocating 1 GB of RAM to just connection tracking.  In fact you need 1 GB (or very close to) to be able to track 65535 connections.  You may think this is way over kill, but not really.  Keep in mind that connections tend to hang around on average 10 minutes after they are closed b/c not all systems out there close them correctly and thus they have to time out (10 minutes).  You can get away with less RAM but you need to watch your DMESG to make sure that you don't see any issues with your connection tracking table filling up, it acts like a FIFO if memory serves.  If you are not doing much in the way if *VERY* *ADVANCED* firewalling, just basic source and / or destination IP v
alidation and / or source and / or destination port validation will not need much of a processor.  In fact I'd try it with a 500 MHz to 1 GHz system, what ever is the most economical that you can get your hands on.

Another problem that you may run in to will be filling your ARP table.  The kernel space ARP table is not very large at all, only like 64 or maybe up to 255 IP MAC pairs.  I want to say it's closer to 64.  Thus you may want to take a look at using the ARP Daemon for Linux to offload the ARP cache to thus avoiding this issue.  Basically how it works (from what I've read) is you reduce the number of times you query the ARP cache in kernel to 0 which will cause the kernel to query the user space daemon for the ARP data.  The user space daemon does it's own ARPing to make sure that it has the information to hand to the kernel.  The main advantage of the user space daemon is that it can handle LOTS of ARP entries, well beyond 1024 (I think).

Something else you might consider would be some managed switches so that you could bond your connections out of the router to them thus ensuring that a cable failure (disconnection) will not take the router down.  If you plug everything in to the managed switch and set up some VLANs you can easily do everything that you are wanting to do over the bonded connections with VLANs on top.  The VLAN interfaces in Linux look like another network interface that you can do all the routing that you want off of.

If the client systems you are going to be firewalling for are business systems I might suggest building two of these routers and setting them up as a VRRP router to ensure that the ""router is alwayse up and useable.  This is much easier through managed switches too as you don't have to cable as much to the physical routers.

In short get memory and a lower end proc to save the money for a 2nd identical router.



Grant. . . .

Mihai Vlad wrote:
> Hey guys,
> 
> I am planning to buy some components for a Linux router that will handle the
> Internet access of 200 computers (includes tc shaping) and some inter
> sub-network routing (at least 100MBps per eth - and there are 3 eth cards).
> 
> I was thinking of a:
> Pentium 4 - 3GHz
> 256 or 512MB RAM
> Network Cards.
> 
> Now - I wonder what is more important: the processor speed or the amount of
> RAM.
> 
> And can you point me to some good Network Cards that you have used and are
> not so expensive. Some Intel, etc. I have no clue about this subject...
> 
> 
> Maybe this discussion can be extended to a list of best practices to set up
> a performant Linux Router from the hardware point of view.
> 
> Thanks in advance,
> Mihai
> 
> 
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Hardware Configuration Ideas

[Peter Surda]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="windows-1252", Size: 2388 bytes --]

On Tue, 16 Aug 2005 11:38:06 -0500 "Taylor, Grant" <gtaylor@riverviewtech.net>
wrote:

>I ended up
>+allocating 1 GB of RAM to just connection tracking.  In fact you need 1 GB (or
>+very close to) to be able to track 65535 connections.
You don't. Maybe that's conntrack's default, but you can set it to a higher
number manually. The required memory is approx 400b per connection (depends on
iptables/kernel compile time options). The rather conservative default (hashsize
= 1/16384th of RAM) is for a generic system. For more info look at
ip_conntrack_core.c

65535 connections need about 25MB in RAM, so before starting iptables, do
modprobe ip_conntrack hashsize92
(contrack_max is auto-set to 8*hashsize, this is the recommended relation). In
fact my distro Shurdix automatically sets up larger hashsize than the default,
depending on system memory.

You can change conntrack_max while the module is loaded (sysctl
net.ipv4.netfilter.ip_conntrack_max), but you can't change the hashsize this
way. If the relation is other than 1:8, you might experience performance
problems (I don't know details, this is recommended on various places on the
net).

>Another problem that you may run in to will be filling your ARP table.  The
>+kernel space ARP table is not very large at all, only like 64 or maybe up to
255
>+IP MAC pairs.
This is also tunable, per sysctl, somewhere like
net.ipv4.neigh.default.gc_thresh[123]. Unfortunately poorly documented, I had to
look at the source to realize this, and I don't remember what means what.

>In short get memory and a lower end proc to save the money for a 2nd identical
>router.
While a redundant system is indeed a good idea, I recommend making sure the
router is rock stable. This doesn't necessarily require high-end / fast
hardware, it is recommended to stress test it before going live
(memtest/cpuburn/whatever).

My tip is not to use "primitive" network cards like those based on rtl8139 which
you require high bandwidth. This has the most noticeable impact on performance.
I have ok experience with 3com's, I've heard intels are even better.

Yours sincerely,
Peter

-- 
http://www.shurdix.org - Linux distribution for routers and firewalls
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Hardware Configuration Ideas

[Peter Surda]

Also,
....

On Tue, 16 Aug 2005 11:38:06 -0500 "Taylor, Grant" <gtaylor@riverviewtech.net>
wrote:

>+If you are not doing much in the way if *VERY* *ADVANCED*
>+firewalling, just basic source and / or destination IP v
>alidation and / or source and / or destination port validation will not need
>+much of a processor.  In fact I'd try it with a 500 MHz to 1 GHz system, what
>+ever is the most economical that you can get your hands on.
Yes. In fact most cases of "advanced" firewalling only mean that you have a
stupid fw-design, like hundreds/thousands of rules in one chain :-). Usually can
be optimised by using sub-chains, ipset and/or ipt_ACCOUNT.

Yours sincerely,
Peter

-- 
http://www.shurdix.org - Linux distribution for routers and firewalls
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Hardware Configuration Ideas

[Grant Taylor]

> Yes. In fact most cases of "advanced" firewalling only mean that you have a
> stupid fw-design, like hundreds/thousands of rules in one chain :-). Usually can
> be optimised by using sub-chains, ipset and/or ipt_ACCOUNT.

If someone has hundreds of rules in one chain (with out a _*VERY*_ good reason and even then) they need to be shot on the spot.  For performance reasons such a chain should be broken out in to a tree of chains an subchains that are jumped to in an attempt to minimize the number of rules that have to be traversed to get a match on any given packet.

What I was referring to by advanced firewalling was such things as running things like "-p udp -s 0.0.0.0/32 -d 255.255.255.255/32 --sport 68 --dport 67 -m addrtype --src-type broadcast -m pkttype --pkt-type broadcast" for DHCP requests. or complex SSH Brute Force prevention chains / rules, or recent lists to control what types of traffic will be valid based on what you have sent or is not valid b/c you have not sent any thing, or should packets with the reset flag have the ack flat set or not, etc.



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Hardware Configuration Ideas

[Grant Taylor]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="windows-1252", Size: 2170 bytes --]

> You don't. Maybe that's conntrack's default, but you can set it to a higher
> number manually. The required memory is approx 400b per connection (depends on
> iptables/kernel compile time options). The rather conservative default (hashsize
> = 1/16384th of RAM) is for a generic system. For more info look at
> ip_conntrack_core.c
> 
> 65535 connections need about 25MB in RAM, so before starting iptables, do
> modprobe ip_conntrack hashsize92
> (contrack_max is auto-set to 8*hashsize, this is the recommended relation). In
> fact my distro Shurdix automatically sets up larger hashsize than the default,
> depending on system memory.

Hmm, I did not have much time to solve this problem at the time and documentation was hard to come by at the time and what I did find was old.  Alass I was not subscribed to this list to ask for help either.  Note things have changed sense then.  :)

> While a redundant system is indeed a good idea, I recommend making sure the
> router is rock stable. This doesn't necessarily require high-end / fast
> hardware, it is recommended to stress test it before going live
> (memtest/cpuburn/whatever).
> 
> My tip is not to use "primitive" network cards like those based on rtl8139 which
> you require high bandwidth. This has the most noticeable impact on performance.
> I have ok experience with 3com's, I've heard intels are even better.

I would agree to both points.  I have had good luck with the rtl8139s on Cable / DSL and T1 routers but I would want something better (3C905x cards) for a much higher bandwidth installation.  The redundant (identical) system is for those cases where the cleaning crew and / or momma nature and / or Mr Murphy have their way with your box.  We have all had it happen (or will) in some way or another at some time.  It is not "if" a box will fail in some way, but rather "when".  The failure may not be any thing you could prevent.  I think the stores in Florida this year are a good example of that.



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Hardware Configuration Ideas

[Mihai Vlad]

Thank you all for your help!

The reason why I asked for your help was because I had the following
problem:

Router:
1.4 GHZ AMD
256 RAM
Realtek 8139 NIC
Slackware
2.6 kernel

No iptables rules just iproute2 routing.
I have 2 sub-networks and the router is configured as a "router on a stick".
I tested a simple routing from one sub-network to another (FTP transfer)
(the traffic enters and leaves the router using the same NIC) and the
maximum transfer rate is 50 Mbit (instead of 100 Mbit as expected).

I wonder how am I able to route a Gigabit network ... Someone told me that
he solved the problem using FreeBSD as it has a better performance on this
field...

I am bound to Linux as it has a superb implementation of QoS. You have
nothing like this in BSD.

I want to ask you: what is the maximum transfer rate that you can obtain on
your configurations (with and without heavy firewalling / marking / tc).

Thanks again for your input!

Mihai




_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Hardware Configuration Ideas

[Peter Surda]

On Wed, 17 Aug 2005 18:46:35 +0300 "Mihai Vlad" <mihaivlad@web-profile.net>
wrote:

>The reason why I asked for your help was because I had the following
>problem:
[cut]

>Realtek 8139 NIC
Yes, this is indeed a problem. Cards based on this chipset don't scale well. Try
3coms or intels. If you're short on money, dlink dfe-530tx might do too.

(I work for none of the mentioned companies :-)).

>Mihai
Yours sincerely,
Peter

-- 
http://www.shurdix.org - Linux distribution for routers and firewalls
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Hardware Configuration Ideas

[Taylor, Grant]

> Yes, this is indeed a problem. Cards based on this chipset don't scale well. Try
> 3coms or intels. If you're short on money, dlink dfe-530tx might do too.

Um, the DFE-530TX+ is a (rebranded) Realtek8139 chipset card so I would expect that the performance would be comparable to the Realtek cards.  I personally have not used them but I have heard wonderful things about DEC Tulup cards.  If you are interested I have some 3Com 905 cards I'd be willing to part with for a very reasonable price.  (Please email me off the list if you are interested in the NICs.)



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Hardware Configuration Ideas

[Carl-Daniel Hailfinger]

Taylor, Grant schrieb:
>>Yes, this is indeed a problem. Cards based on this chipset don't scale well. Try
>>3coms or intels. If you're short on money, dlink dfe-530tx might do too.
> 
> Um, the DFE-530TX+ is a (rebranded) Realtek8139 chipset card so I would
> expect that the performance would be comparable to the Realtek cards.
> I personally have not used them but I have heard wonderful things about
> DEC Tulip cards.

Some of the DFE-5..TX cards are indeed tulip based and have very good
performance for a reasonable price. Unfortunately sources on the net
disagree which cards have which chipset, so I recommend looking at them
(I bought a new tulip based card here in Germany for 7 Euros). It pays
to request a look at the chipsets of the cheaper cards, they might not
even advertise their chipsets as tulip.


Regards,
Carl-Daniel
-- 
http://www.hailfinger.org/
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Hardware Configuration Ideas

[Peter Surda]

On Wed, 17 Aug 2005 14:18:47 -0500 "Taylor, Grant" <gtaylor@riverviewtech.net>
wrote:


>Um, the DFE-530TX+ is a (rebranded) Realtek8139 chipset card so I would expect
>that the performance would be comparable to the Realtek cards.
I don't have one here, but I vaguely remember they use a different kernel module
(8139too vs. via-rhine) so I think it's a different chipset. I only mentioned
them because some time in distant past I solved some performance problems by
switching from from 8139 to the dlink, and the dlink costs less than a half of
entry-level 3com.

>Grant. . . .

-- 
http://www.shurdix.org - Linux distribution for routers and firewalls
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc