Re: [xen-devel] create irq failed due to move_cleanup_count always being set

[Andrew Cooper <andrew.cooper3@citrix.com>]

Could you please avoid top posing.

On 06/01/12 06:04, Liuyongan wrote:
>    As only 33 domains were successfully created(and destroyed) before the problem occurring,  there should be enough free IRQ  number and vector number to allocate(suppose that irqs and vectors failed to deallocate). And destroy_irq() will clear move_in_progress, so move_cleanup_count must be setted?  Is this the case? 

Is it repeatably 33 domains, or was that a 1 off experiment?  Can you
confirm exactly which version of Xen you are using, including changeset
if you know it?  Without knowing your hardware, it is hard to say if
there are actually enough free IRQs, although I do agree that what you
are currently seeing is buggy behavior.

The per-cpu IDT functionality introduced in Xen-4.0 is fragile at the
best of times, and has had several bugfixes and tweaks to it which I am
not certain have actually found their way back to Xen-4.0.  Could you
try with Xen-4.1 and see if the problem persists?

~Andrew

>    Yong an
>> -----Original Message-----
>> From: Liuyongan
>> Sent: Thursday, January 05, 2012 2:14 PM
>> To: Liuyongan; xen-devel@lists.xensource.com;
>> andrew.cooper3@citrix.com; keir@xen.org
>> Cc: Qianhuibin
>> Subject: RE: [xen-devel] create irq failed due to move_cleanup_count
>> always being set
>>
>>> On 04/01/12 11:38, Andrew Cooper wrote:
>>>> On 04/01/12 04:37, Liuyongan wrote:
>>>> Hi, all
>>>>
>>>>     I'm using xen-4.0 to do a test. And when I create a domain, it
>> failed due to create_irq() failure. As only 33 domains were
>> successfully created and destroyed before I got the continuous
>> failures, and the domain just before the failure was properly
>> destroyed(at least destroy_irq() was properly called, which will clear
>> move_in_progress, according to the prink-message). So I can conclude
>> for certain that __assign_irq_vector failed due to move_cleanup_count
>> always being set.
>>> Is it always 33 domains it takes to cause the problem, or does it
>> vary?
>>> If it varies, then I think you want this patch
>>> http://xenbits.xensource.com/hg/xen-unstable.hg/rev/68b903bb1b01
>> which
>>> corrects the logic which works out which moved vectors it should
>> clean
>>> up.  Without it, stale irq numbers build up in the per-cpu irq_vector
>>> tables leading to __assign_irq_vector failing with -ENOSPC as it find
>>> find a vector to allocate.
>>   Yes, I've noticed this patch, as only 33 domains were created before
>> the failures, so vectors of a given cpu should not have been used up.
>> Besides, I got this problem after 143 domains were created another
>> time. But I could not repeat this problem manually as 4000+ domains
>> created successfully without this problem.
>>
>>>> //this is the normal case when create and destroy domain whose id is
>> 31;
>>>> (XEN) irq.c:1232:d0 bind pirq 79, irq 77, share flag:0
>>>> (XEN) irq.c:1377: dom31: pirq 79, irq 77 force unbind
>>>> (XEN) irq.c:1593: dom31: forcing unbind of pirq 79
>>>> (XEN) irq.c:223, destroy irq 77
>>>>
>>>> //domain id 32 is created and destroyed correctly also.
>>>> (XEN) irq.c:1232:d0 bind pirq 79, irq 77, share flag:0
>>>> (XEN) irq.c:1377: dom32: pirq 79, irq 77 force unbind
>>>> (XEN) irq.c:1593: dom32: forcing unbind of pirq 79
>>>> (XEN) irq.c:223, destroy irq 77
>>>>
>>>> //all the subsequent domain creation failed, below lists only 3
>> times:
>>>> (XEN) physdev.c:88: dom33: can't create irq for msi!
>>>> (XEN) physdev.c:88: dom34: can't create irq for msi!
>>>> (XEN) physdev.c:88: dom35: can't create irq for msi!
>>>>
>>>>      I think this might be a bug and might have fixed, so I compare
>> my code with 4.1.2 and search the mail list for potential patches.
>> (http://xen.markmail.org/search/?q=move_cleanup_count#query:move_cleanu
>> p_count+page:6+mid:fpkrafqbeyiauvhs+state:results) submit a patch which
>> add locks in __assign_irq_vector. Can anybody explain why this lock is
>> needed? Or is there a patch that might fix my bug? Thx.
>>> This patch fixes a problem where IOAPIC line level interrupts cease
>> for
>>> a while.  It has nothing to do with MSI interrupts.  (Also, there are
>> no
>>> locks altered, and xen-4.0-testing seems to have gained an additional
>>> hunk in hvm/vmx code unrelated to the original patch.)
>>>
>>>>     Addition message: my board is arch-x86, no domains left when
>> failed to create new ones, create_irq failure lasted one day until I
>> reboot the board, and the irq number allocated is used certainly for a
>> msi dev.
>>>> Yong an Liu
>>>> 2012.1.4
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel**************

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com