Re: xen dependant on pcpu 0 ?

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Tue, Oct 12, 2010 at 06:28:13PM +0200, Sander Eikelenboom wrote:
> Hi Keir,
> 
> Does xen and/or the xen console depend on physical cpu 0 ?

Usually the console for Dom0, and I think all other domains go
through CPU0. Let me CC Ian here, who has been mucking in this
area and found some bugs (and produced fixes).

Ian, that bug you found with not clearing the eventchannel - that
wouldn't have an impact here, right?

> 
> I'm still trying to solve the mystery of my machine freezing when doing:
> 
>  - videograbbing in a domU with a usb3 pci-express controller passed through (seems to cause quite a few interrupts)
>  - compiling a linux kernel with "make -j 6"
> 
> It's a 6 core AMD phenom x6.
> 
> Without cpu pinning:
> I can freeze the machine easily within a minute after starting the compile, at first xen serial console also slows down under the load (slow updates).
> When the machine freezes i can't do anything with xen serial console.
> 
> With cpu pinning:
> By not using the pcpu 0 at all for any domain, and pinning the domain with the videograbber to it's own pcpu (pcpu 5)  it seems the machine keeps running after 20 "make -j6" iterations of kernel compilation.
> Xen serial console stays responsive and doesn't slow down during the kernel compilation. The videograbber shows no problem grabbing video.
> 

AHA! So finally closer to the mystery.

Can you provide the /proc/interrupts of the Dom0?

I wonder if this is related to the isseu I had some time ago, and never got
to look at. The problem was that during heavy compilation (this is a 2 Nehelem
socket box, just running Dom0 - no guests), the keyboard and USB driver would
stop getting interrupts.  So the drivers would start polling which is quite slow,
albeit servicable, and then at some point it would pick up again.

The weirdness was that the /proc/interrupts showed absolutly _no_ interrupts on CPU0
during that time - as if Xen just forgot to update them. Jeremy suggested I try to
disable Xen IRQ balance (noirqbalance on Xen command line) in case that is it, and to my
emberrasement I haven't tried that yet.

Did you try that? I think somebody suggested that but I can't recall whether it
was for this issue?
> 
> Name                                ID  VCPU   CPU State   Time(s) CPU Affinity
> Domain-0                             0     0     3   r--    2169.7 1-4
> Domain-0                             0     1     1   -b-    2339.3 1-4
> Domain-0                             0     2     2   -b-    2358.9 1-4
> Domain-0                             0     3     3   -b-    2298.2 1-4
> Domain-0                             0     4     1   -b-    2221.9 1-4
> Domain-0                             0     5     4   -b-    2287.7 1-4
> backup                               9     0     4   -b-      10.6 1-4
> database                             1     0     4   -b-      45.3 1-4
> davical                              5     0     3   -b-       8.7 1-4
> git                                  8     0     2   -b-       7.9 1-4
> mail                                 2     0     4   -b-       8.0 1-4
> samba                                3     0     3   -b-      11.1 1-4
> security                             7     0     5   r--    1433.2 5
> www                                  4     0     1   -b-      10.2 1-4
> zabbix                               6     0     3   -b-      21.2 1-4
> 
> 
> Is there a way a deadlock could occur between hypervisor <-> dom0 <-> domU especially related to passthrough/interrupts in the context of pcpu 0 ?

I don't know, but I do know that the IRQ handling in Xen 4.0 changed significantly compared
to 3.4. I don't remember if you ever ran this setup under 3.4?
> 
> --
> Sander