Re: Receive side performance issue with multi-10-GigE and NUMA

[Andrew Gallatin <gallatin@myri.com>]

Neil Horman wrote:
> On Fri, Aug 21, 2009 at 12:14:21AM -0400, Bill Fink wrote:
>> On Thu, 20 Aug 2009, Neil Horman wrote:
>>
>>> On Thu, Aug 20, 2009 at 03:50:44AM -0400, Bill Fink wrote:
>>>
>>>> When I tried an actual nuttcp performance test, even when rate limiting
>>>> to just 1 Mbps, I immediately got a kernel oops.  I tried to get a
>>>> crashdump via kexec/kdump, but the kexec kernel, instead of just
>>>> generating a crashdump, fully booted the new kernel, which was
>>>> extremely sluggish until I rebooted it through a BIOS re-init,
>>>> and never produced a crashdump.  I tried this several times and
>>>> an immediate kernel oops was always the result (with either a TCP
>>>> or UDP test).  A ping test of 1000 9000-byte packets with an interval
>>>> of 0.001 seconds (which is 72 Mbps for 1 second) on the other hand
>>>> worked just fine.
>>> The sluggishness is expected, since the kdump kernel operates out of such
>>> limited memory.  don't know why you booted to a full system rather than did a
>>> crash recovery.  Don't suppose you got a backtrace did you?
>> There was a backtrace on the screen but I didn't have a chance to
>> record it.  BTW did anyone ever think to print the backtrace in
>> reverse (first to some reserved memory and then output to the display)
>> so the more interesting parts wouldn't have scrolled off the top of
>> the screen?
>>
> The real solution is to use a console to which the output doesn't scroll off the
> screen.  Normally people use a serial console they can log, or a RAC card that
> they can record. Even on a regular vga monitor in text mode, you can set up the
> vt iirc to allow for scrolling.

Indeed. Another option when setting up a serial console is not practical
is netconsole.  I've captured a few panics this way on machines like
macs, with no serial port support (at the time).

Drew