Re: Network dies and kernel errors

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

> > Did you try that? Did that make any difference?
> 
> Not tested I did install one.
> 
> I think I found a way to keep it running.
> On the new igb driver I built from new intel source added module parameter 
> IntMode=1.
> 
> This puts it in msi mode. It was in msi-x mode.
> It's never died with that setting.
> It's up now over a day.
> No real experience with msi-x. I think it's the first time I have seen a 
> driver use msi-x interrupts.
> Maybe that gives you more ideas?

That was my thought - the MSI-X aren't somehow being ACKed properly. But I don't
know if the issue with Dom0 or Xen.
> 
> 
> > 
> > > Any ideas?
> > 
> > There is a Xen parameter called 'noirqbalance' . Try that. Also see if you
> > can limit the CPUs in the dom0 using these two arguments on Xen
> > hypervisor:
> > 
> Should I turn off the irqbalence daemon also?

Sure.

> Just in case you wonder it does with out it.
> 
> > dom0_vcpus=2 dom0_vcpus_pin=1
> > 
> > 
> > It would be interesting to narrow down _when_ you trigger this failure. B/c
> > we can pull Xen to see what the MSI's are 'xl debug-keys M' _before_ and
> > _after_ your failure to see if something is amiss.
> > 
> > Mainly to figure out if the vectors are moving around the CPUs (or not)
> > 
> > (XEN)  MSI    29 vec=21 lowest  edge   assert  log lowest dest=00000001
> > mask=0/0/-1
> > 
> > and also 'xl debug-keys i' to see if the domain has ACK-ed the interrupt:
> > (XEN)    IRQ:  29 affinity:00000000,00000000,00000000,00000001 vec:21
> > type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:275(----),
> > 
> > (the last '----' might have something else in in them - if so that is a
> > sign that dom0 hasn't picked up the event/vector).
> 
> Much of my frustration is that I have not found a way to get it to fail other 
> than waiting a long time :-(

Ah that sucks. Well, just make a nice shell script that will run those continously
(and also 'xl dmesg') and pipe the log to a file.