Question about interrupt enabling/disabling in kernel exit path

Question about interrupt enabling/disabling in kernel exit path

[Lee Schermerhorn]

Hello:

I have a question about enabling interrupts in the kernel exit path.
Especially, in do_notify_resume_user().  I have work that I need to
defer to this point, but it needs to run with interrupts enabled.  I
wanted to piggy back off TIF_NOTIFY_RESUME, because that is more or less
generic code and gets handled in a C function where I can play the usual
header games:  static inline wrapper function when feature is
configured;  null macro when not.

I see a comment at the beginning of ia64_leave_kernel that says that
"work.need_resched, etc. mustn't get changed by this CPU before it
returns to user- or fsys-mode, hence we disable interrupts early on."
[This may be a stale comment.  I.e., we now check for resched needed in
current task's thread_info flags.] But, then I see later that when
CONFIG_PREEMPT is set, we do enable interrupts and then re-disable them.

Then I notice that after we return from do_notify_resume_user() we don't
recheck TIF_NEED_RESCHED, etc. in the thread_info flags.   So, maybe
this is the issue?  If I open an interrupt window in
do_notify_resume_user(), 'NEED_RESCHED might get set and we wouldn't
notice it on return to the assembly language code.  [I might even sleep
here.  Is that a problem?]

I took a look at the x86_64 and i386 versions and they always goes back
to check for resched after calling their version of do_notify_resume()
[no '_user suffix on the routine].  If this were the only issue with
enabling interrupts in do_notify_resume_user(), I could change it to
return non-zero if recheck is needed [zero in the usual case] and then
go back and recheck thread_info flags.

Regards,
Lee

Re: Question about interrupt enabling/disabling in kernel exit path

[Stephane Eranian]

Lee,

As you may have seen there is a perfmon callback from do_notify_resume().
I am also using TIF_NOTIFY_RESUME to get back into perfmon just before
leaving the kernel.

It is not true that interrupts are always off when getting there.
Last year we ran into a issue for this reason, i.e., I was assume
interrupts were always masked when getting there. But apparentlyy,
that is not always the case. I don't recall exactly what is the path
to get to this situation but I know it exists.

On Wed, Nov 09, 2005 at 01:12:05PM -0500, Lee Schermerhorn wrote:
> Hello:
> 
> I have a question about enabling interrupts in the kernel exit path.
> Especially, in do_notify_resume_user().  I have work that I need to
> defer to this point, but it needs to run with interrupts enabled.  I
> wanted to piggy back off TIF_NOTIFY_RESUME, because that is more or less
> generic code and gets handled in a C function where I can play the usual
> header games:  static inline wrapper function when feature is
> configured;  null macro when not.
> 
> I see a comment at the beginning of ia64_leave_kernel that says that
> "work.need_resched, etc. mustn't get changed by this CPU before it
> returns to user- or fsys-mode, hence we disable interrupts early on."
> [This may be a stale comment.  I.e., we now check for resched needed in
> current task's thread_info flags.] But, then I see later that when
> CONFIG_PREEMPT is set, we do enable interrupts and then re-disable them.
> 
> Then I notice that after we return from do_notify_resume_user() we don't
> recheck TIF_NEED_RESCHED, etc. in the thread_info flags.   So, maybe
> this is the issue?  If I open an interrupt window in
> do_notify_resume_user(), 'NEED_RESCHED might get set and we wouldn't
> notice it on return to the assembly language code.  [I might even sleep
> here.  Is that a problem?]
> 
> I took a look at the x86_64 and i386 versions and they always goes back
> to check for resched after calling their version of do_notify_resume()
> [no '_user suffix on the routine].  If this were the only issue with
> enabling interrupts in do_notify_resume_user(), I could change it to
> return non-zero if recheck is needed [zero in the usual case] and then
> go back and recheck thread_info flags.
> 
> Regards,
> Lee
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 

-Stephane

Re: Question about interrupt enabling/disabling in kernel exit path

[David Mosberger-Tang]

> Then I notice that after we return from do_notify_resume_user() we don't
> recheck TIF_NEED_RESCHED, etc. in the thread_info flags.

Why do you say this?  It's not true.  After processing pending work,
we go back to .work_processed_{syscall,kernel), which will again
invoke .work_pending, if necessary.

AFAIR, it's OK to reenable interrupts in notify_resume_user() because
we know we're at the top of the kernel stack (i.e., there is no risk
of nesting the kernel stack too deeply due to pending interrupts),
provided the interrupts are off again when resuming execution at
.work_processed_kernel.

  --david
--
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536

Re: Question about interrupt enabling/disabling in kernel exit path

[Lee Schermerhorn]

On Mon, 2005-11-14 at 13:52 -0800, David Mosberger-Tang wrote:
> > Then I notice that after we return from do_notify_resume_user() we don't
> > recheck TIF_NEED_RESCHED, etc. in the thread_info flags.
> 
> Why do you say this?  It's not true.  After processing pending work,
> we go back to .work_processed_{syscall,kernel), which will again
> invoke .work_pending, if necessary.

OK.  I guess I'm misinterpreting the comments on the line labeled
'.ret10:'.
Say's it's setting p6 to zero, so no recheck.  I'm not fluent [writing
nor reading] ia64 assembler, but it appeared to me that with p6 set to
zero on return from notify_resume(), the checks for more work back
at .work_processed_* would be disabled.

Are the comments at the ia64_leave_kernel entry point stale?

> 
> AFAIR, it's OK to reenable interrupts in notify_resume_user() because
> we know we're at the top of the kernel stack (i.e., there is no risk
> of nesting the kernel stack too deeply due to pending interrupts),
> provided the interrupts are off again when resuming execution at
> .work_processed_kernel.

OK.  Ken said he thought it was OK to disable.  And Stephane mentioned
that he's seen interrupts enabled when entering do_notify_resume_user()
as well.

Thanks,
Lee

RE: Question about interrupt enabling/disabling in kernel exit path

[Chen, Kenneth W]

Lee Schermerhorn wrote on Tuesday, November 15, 2005 12:21 PM
> On Mon, 2005-11-14 at 13:52 -0800, David Mosberger-Tang wrote:
> > > Then I notice that after we return from do_notify_resume_user() we don't
> > > recheck TIF_NEED_RESCHED, etc. in the thread_info flags.
> > 
> > Why do you say this?  It's not true.  After processing pending work,
> > we go back to .work_processed_{syscall,kernel), which will again
> > invoke .work_pending, if necessary.
> 
> OK.  I guess I'm misinterpreting the comments on the line labeled
> '.ret10:'.
> Say's it's setting p6 to zero, so no recheck.  I'm not fluent [writing
> nor reading] ia64 assembler, but it appeared to me that with p6 set to
> zero on return from notify_resume(), the checks for more work back
> at .work_processed_* would be disabled.
> 

Looking at the code, I think Lee is correct that upon returning from
notify_resume_user, p6 is turned off and subsequently turns off all
the check in work_pending.

- Ken

Re: Question about interrupt enabling/disabling in kernel exit path

[David Mosberger-Tang]

On 11/15/05, Chen, Kenneth W <kenneth.w.chen@intel.com> wrote:
> Lee Schermerhorn wrote on Tuesday, November 15, 2005 12:21 PM
> > On Mon, 2005-11-14 at 13:52 -0800, David Mosberger-Tang wrote:
> > > > Then I notice that after we return from do_notify_resume_user() we don't
> > > > recheck TIF_NEED_RESCHED, etc. in the thread_info flags.
> > >
> > > Why do you say this?  It's not true.  After processing pending work,
> > > we go back to .work_processed_{syscall,kernel), which will again
> > > invoke .work_pending, if necessary.
> >
> > OK.  I guess I'm misinterpreting the comments on the line labeled
> > '.ret10:'.
> > Say's it's setting p6 to zero, so no recheck.  I'm not fluent [writing
> > nor reading] ia64 assembler, but it appeared to me that with p6 set to
> > zero on return from notify_resume(), the checks for more work back
> > at .work_processed_* would be disabled.
> >
>
> Looking at the code, I think Lee is correct that upon returning from
> notify_resume_user, p6 is turned off and subsequently turns off all
> the check in work_pending.

Yes, I goofed.

The ia64 code shouldn't be different from other platforms in this
respect though.  If x86 nowadays does the re-check unconditionally, we
need to look into what happened.  Perhaps x86 got updated and we
missed that update for ia64?

  --david
--
Mosberger Consulting LLC, voice/fax: 510-744-9372,
http://www.mosberger-consulting.com/
35706 Runckel Lane, Fremont, CA 94536