Re: Clock Interrupt not occurring. hypervisor_callback not being called. (more info)

Re: Clock Interrupt not occurring. hypervisor_callback not being called. (more info)

[Ivan Kelly]

Some more info on this problem.
The problem seems to be that evtchn_pending[0] is not being updated
fast enough after the callback is made. Therefore the callback is
running and seeing nothing as pending and discarding anything that
happens.
If i put in something that delays before the start of the inner loop
somewhere, it will work fine. It doesn't have to be a printf, can be a
for(i=0;i<100000;i++) or something similar.
evtchn_pending is set as volatile. Im not sure if this is actually
helping at all though, since I have to use the ACK compiler which is
the only thing that minix will compile with.

Does anyone have any idea on how i can get evtchn_pending to update on time?
Regards
Ivan

> <SNIP>
> PUBLIC void
> do_hypervisor_callback(struct stackframe_s *regs)
> {
>         unsigned long  l1, l2;
>         unsigned int   l1i, l2i, port;
>         int            irq;
>         shared_info_t *s = hypervisor_shared_info;
>
>         s->vcpu_data[0].evtchn_upcall_pending = 0;
>
>         l1 = x86_atomic_xchg(&s->evtchn_pending_sel, 0);
>
>         while ( l1 != 0 )
>         {
> /*              kprintf("l1: %x\n");*/
>                 l1i = x86_scan_forward(l1);
>                 l1 &= ~(1 << l1i);
>
>                 l2 = s->evtchn_pending[l1i] & ~s->evtchn_mask[l1i];
>                 while ( l2 != 0 )
>                 {
>                         kprintf("l2: %x\n", l2);
>                         l2i = x86_scan_forward(l2);
>                         l2 &= ~(1 << l2i);
>
>                         port = (l1i << 5) + l2i;
>
>                         if ( (irq = event_to_action[port]) != -1 ) {
>                                 do_event(irq, regs);
>                         }
>                 }
>         }
> }
> <ENDSNIP>
>

Re: Clock Interrupt not occurring. hypervisor_callback not being called. (more info)

[Dave Winchell]

Ivan,

How about a memory barrier right before the call to evtchn_notify
in evtchn_set_pending and one before l1 = 
x86_atomic_xchg(&s->evtchn_pending_sel, 0)
in do_hypervisor_callback?

Regards,
Dave


Ivan Kelly wrote:

>Some more info on this problem.
>The problem seems to be that evtchn_pending[0] is not being updated
>fast enough after the callback is made. Therefore the callback is
>running and seeing nothing as pending and discarding anything that
>happens.
>If i put in something that delays before the start of the inner loop
>somewhere, it will work fine. It doesn't have to be a printf, can be a
>for(i=0;i<100000;i++) or something similar.
>evtchn_pending is set as volatile. Im not sure if this is actually
>helping at all though, since I have to use the ACK compiler which is
>the only thing that minix will compile with.
>
>Does anyone have any idea on how i can get evtchn_pending to update on time?
>Regards
>Ivan
>
>  
>
>><SNIP>
>>PUBLIC void
>>do_hypervisor_callback(struct stackframe_s *regs)
>>{
>>        unsigned long  l1, l2;
>>        unsigned int   l1i, l2i, port;
>>        int            irq;
>>        shared_info_t *s = hypervisor_shared_info;
>>
>>        s->vcpu_data[0].evtchn_upcall_pending = 0;
>>
>>        l1 = x86_atomic_xchg(&s->evtchn_pending_sel, 0);
>>
>>        while ( l1 != 0 )
>>        {
>>/*              kprintf("l1: %x\n");*/
>>                l1i = x86_scan_forward(l1);
>>                l1 &= ~(1 << l1i);
>>
>>                l2 = s->evtchn_pending[l1i] & ~s->evtchn_mask[l1i];
>>                while ( l2 != 0 )
>>                {
>>                        kprintf("l2: %x\n", l2);
>>                        l2i = x86_scan_forward(l2);
>>                        l2 &= ~(1 << l2i);
>>
>>                        port = (l1i << 5) + l2i;
>>
>>                        if ( (irq = event_to_action[port]) != -1 ) {
>>                                do_event(irq, regs);
>>                        }
>>                }
>>        }
>>}
>><ENDSNIP>
>>
>>    
>>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>  
>

Re: Clock Interrupt not occurring. hypervisor_callback not being called. (more info)

[Ivan Kelly]

how would one create a memory barrier? I think the xchg asm
instruction should be a natural barrier in any case.
-Ivan

On 3/1/06, Dave Winchell <dwinchell@virtualiron.com> wrote:
> Ivan,
>
> How about a memory barrier right before the call to evtchn_notify
> in evtchn_set_pending and one before l1 =
> x86_atomic_xchg(&s->evtchn_pending_sel, 0)
> in do_hypervisor_callback?
>
> Regards,
> Dave
>
>
> Ivan Kelly wrote:
>
> >Some more info on this problem.
> >The problem seems to be that evtchn_pending[0] is not being updated
> >fast enough after the callback is made. Therefore the callback is
> >running and seeing nothing as pending and discarding anything that
> >happens.
> >If i put in something that delays before the start of the inner loop
> >somewhere, it will work fine. It doesn't have to be a printf, can be a
> >for(i=0;i<100000;i++) or something similar.
> >evtchn_pending is set as volatile. Im not sure if this is actually
> >helping at all though, since I have to use the ACK compiler which is
> >the only thing that minix will compile with.
> >
> >Does anyone have any idea on how i can get evtchn_pending to update on time?
> >Regards
> >Ivan
> >
> >
> >
> >><SNIP>
> >>PUBLIC void
> >>do_hypervisor_callback(struct stackframe_s *regs)
> >>{
> >>        unsigned long  l1, l2;
> >>        unsigned int   l1i, l2i, port;
> >>        int            irq;
> >>        shared_info_t *s = hypervisor_shared_info;
> >>
> >>        s->vcpu_data[0].evtchn_upcall_pending = 0;
> >>
> >>        l1 = x86_atomic_xchg(&s->evtchn_pending_sel, 0);
> >>
> >>        while ( l1 != 0 )
> >>        {
> >>/*              kprintf("l1: %x\n");*/
> >>                l1i = x86_scan_forward(l1);
> >>                l1 &= ~(1 << l1i);
> >>
> >>                l2 = s->evtchn_pending[l1i] & ~s->evtchn_mask[l1i];
> >>                while ( l2 != 0 )
> >>                {
> >>                        kprintf("l2: %x\n", l2);
> >>                        l2i = x86_scan_forward(l2);
> >>                        l2 &= ~(1 << l2i);
> >>
> >>                        port = (l1i << 5) + l2i;
> >>
> >>                        if ( (irq = event_to_action[port]) != -1 ) {
> >>                                do_event(irq, regs);
> >>                        }
> >>                }
> >>        }
> >>}
> >><ENDSNIP>
> >>
> >>
> >>
> >
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@lists.xensource.com
> >http://lists.xensource.com/xen-devel
> >
> >
>
>

Re: Clock Interrupt not occurring. hypervisor_callback not being called. (more info)

[Keir Fraser]

On 1 Mar 2006, at 16:17, Ivan Kelly wrote:

> The problem seems to be that evtchn_pending[0] is not being updated
> fast enough after the callback is made. Therefore the callback is
> running and seeing nothing as pending and discarding anything that
> happens.

This is certainly not happening or I think it would have been 
discovered and fixed by now. :-)

When firing an event the pending flag gets set first, then the selector 
bit, then the evtchn_upcall_pending master flag, then the upcall 
happens.

When processing an upcall, you should clear bits in the opposite order.

  -- Keir

Re: Clock Interrupt not occurring. hypervisor_callback not being called. (more info)

[Dave Winchell]

I can't explain why the atomic instructions wouldn't be sufficient.
I just thought it was worth a try to be explicit.
You should be able to create them with 'mb();'

-Dave



Ivan Kelly wrote:

>how would one create a memory barrier? I think the xchg asm
>instruction should be a natural barrier in any case.
>-Ivan
>
>On 3/1/06, Dave Winchell <dwinchell@virtualiron.com> wrote:
>  
>
>>Ivan,
>>
>>How about a memory barrier right before the call to evtchn_notify
>>in evtchn_set_pending and one before l1 =
>>x86_atomic_xchg(&s->evtchn_pending_sel, 0)
>>in do_hypervisor_callback?
>>
>>Regards,
>>Dave
>>
>>
>>Ivan Kelly wrote:
>>
>>    
>>
>>>Some more info on this problem.
>>>The problem seems to be that evtchn_pending[0] is not being updated
>>>fast enough after the callback is made. Therefore the callback is
>>>running and seeing nothing as pending and discarding anything that
>>>happens.
>>>If i put in something that delays before the start of the inner loop
>>>somewhere, it will work fine. It doesn't have to be a printf, can be a
>>>for(i=0;i<100000;i++) or something similar.
>>>evtchn_pending is set as volatile. Im not sure if this is actually
>>>helping at all though, since I have to use the ACK compiler which is
>>>the only thing that minix will compile with.
>>>
>>>Does anyone have any idea on how i can get evtchn_pending to update on time?
>>>Regards
>>>Ivan
>>>
>>>
>>>
>>>      
>>>
>>>><SNIP>
>>>>PUBLIC void
>>>>do_hypervisor_callback(struct stackframe_s *regs)
>>>>{
>>>>       unsigned long  l1, l2;
>>>>       unsigned int   l1i, l2i, port;
>>>>       int            irq;
>>>>       shared_info_t *s = hypervisor_shared_info;
>>>>
>>>>       s->vcpu_data[0].evtchn_upcall_pending = 0;
>>>>
>>>>       l1 = x86_atomic_xchg(&s->evtchn_pending_sel, 0);
>>>>
>>>>       while ( l1 != 0 )
>>>>       {
>>>>/*              kprintf("l1: %x\n");*/
>>>>               l1i = x86_scan_forward(l1);
>>>>               l1 &= ~(1 << l1i);
>>>>
>>>>               l2 = s->evtchn_pending[l1i] & ~s->evtchn_mask[l1i];
>>>>               while ( l2 != 0 )
>>>>               {
>>>>                       kprintf("l2: %x\n", l2);
>>>>                       l2i = x86_scan_forward(l2);
>>>>                       l2 &= ~(1 << l2i);
>>>>
>>>>                       port = (l1i << 5) + l2i;
>>>>
>>>>                       if ( (irq = event_to_action[port]) != -1 ) {
>>>>                               do_event(irq, regs);
>>>>                       }
>>>>               }
>>>>       }
>>>>}
>>>><ENDSNIP>
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>_______________________________________________
>>>Xen-devel mailing list
>>>Xen-devel@lists.xensource.com
>>>http://lists.xensource.com/xen-devel
>>>
>>>
>>>      
>>>
>>    
>>

Re: Clock Interrupt not occurring. hypervisor_callback not being called. (more info)

[Ivan Kelly]

> This is certainly not happening or I think it would have been
> discovered and fixed by now. :-)
Im developing on 2.0.7. Could such a thing have existed back then? in
any case, it's something im doing wrong.
>
> When processing an upcall, you should clear bits in the opposite order.
This is the order i am clearing them in. It looks to me like upcall is
being called with no event_pending bit set, though i doubt this could
even be possible. maybe im clearing the bit accidently :/
-ivan

Re: Clock Interrupt not occurring. hypervisor_callback not being called. (more info)

[Ivan Kelly]

Problem solved and I am rather annoyed. When porting XEN_BLOCK_EVENTS
and XEN_UNBLOCK_EVENTS macros, i had removed the 'b' from movb
assuming it was an at&t asm thing, like the l on movl. silly mistake,
and i shall paddle myself sufficiently for it.
Thanks for all yer help,
Ivan

On 3/1/06, Ivan Kelly <ivanbkelly@gmail.com> wrote:
> > This is certainly not happening or I think it would have been
> > discovered and fixed by now. :-)
> Im developing on 2.0.7. Could such a thing have existed back then? in
> any case, it's something im doing wrong.
> >
> > When processing an upcall, you should clear bits in the opposite order.
> This is the order i am clearing them in. It looks to me like upcall is
> being called with no event_pending bit set, though i doubt this could
> even be possible. maybe im clearing the bit accidently :/
> -ivan
>