Re: Routing 600+ vlan's via linux problems (looks like arp problems)

[Willy Tarreau <w@1wt.eu>]

On Fri, May 04, 2007 at 05:48:18AM +0200, Øyvind Vågen Jægtnes wrote:
> Hi again :)
> 
> On 5/4/07, Willy Tarreau <w@1wt.eu> wrote:
> >On Thu, May 03, 2007 at 11:12:09PM +0200, Øyvind Vågen Jægtnes wrote:
> >> On 5/3/07, Jan Engelhardt <jengelh@linux01.gwdg.de> wrote:
> >> >
> >> >On May 3 2007 22:53, Willy Tarreau wrote:
> >> >>> For the rest all we see in the arp cache is (incomplete)
> >> >>
> >> >>I suspect that your arp cache is full (128 entries by default).
> >> >>Check /proc/sys/net/ipv4/neigh/gc_thresh1 (128 for me). You can
> >> >>set it as high as gc_thresh2 (512 for me), and I don't know what
> >> >>happens above.
> >> >
> >> >Above, you will perhaps need the not-so-elegant userspace arpd :-/
> >>
> >> Yes, i was suspecting that the arp cache got full, but i will try
> >> increasing it :)
> >> Would there be any huge bugs if i change these lines in arp.c:
> >>
> >>        .gc_thresh1 =   128,
> >>        .gc_thresh2 =   512,
> >>
> >> to
> >>
> >>        .gc_thresh1 =   700,
> >>        .gc_thresh2 =   700,
> >>
> >> under the definition for struct arp_tbl?
> >
> >I don't think it could cause a problem, but network people will surely
> >correct me if I'm wrong.
> 
> System is up and running perfectly now, it is routing everything at
> about 200 mbps now with only 5% load avg with the above changes to
> arp.c
> 
> So the real question now is, why is this number so low by default?
> It would probably be much better if this could be handled dynamically
> in the kernel.

I remember I read an argument against this a long time ago, but I
don't remember where. I think it was some arbitrary decision that
people using more than X ARP entries will need arpd. Most probably
the code path in the ARP updates is/was not much optimized to handle
large number of entries. Think about cable operators who may have
10-20000 entries !

> Its a Juniper M7i
> It comes default with a 5400 rpm laptop 2.5" harddrive but now we
> bought a more robust "server" 2.5" harddrive.

The "server" ones are not necessarily more robust, often they are faster.

> It still barfs on the OS
> install, so the linux is doing all the job now. Will get a juniper guy
> to come and fix :)
> 
> As a side note, i'm starting to wonder if it was worth the $20k when i
> could just have a linux machine to do the job with a clone for backup
> ;)

That's often how linux penetrates the enterprise ;-)

Willy