Triggering graceful bitbake shutdown

Triggering graceful bitbake shutdown

[Tanu Kaskinen]

Hi,

I wrote a small script that runs bitbake sequentially with multiple
MACHINE values. Sometimes I want to stop it with ctrl-c before it has
finished, but it didn't work properly out-of-the box when running
bitbake under the script (my script would not wait for bitbake to exit,
and bitbake threw some backtrace too). I then modified SIGINT handling
in my script so that it forwards the signal to bitbake and then waits
for bitbake to exit. The problem is that bitbake ignores SIGINT. The UI
handles python's KeyboardInterrupt exception, but that exception
doesn't get raised when bitbake runs under my script. I don't know what
triggers KeyboardInterrupt in normal conditions, since SIGINT doesn't
seem to be sufficient, does python perhaps monitor stdin to catch the
actual ctrl-c keypress?

Anyway, sending SIGTERM instead of SIGINT "works". However, the SIGTERM
handler shuts down bitbake immediately rather than waiting for the
current tasks to finish. Is that less safe than the normal ctrl-c
handling? If the immediate shutdown is equally safe, then I have no
problem, but if SIGTERM can cause state corruption so that resuming
later doesn't work reliably, then I'd like to have some way for
triggering graceful bitbake shutdown from my script. Any suggestions?

Or maybe there is already some way to run bitbake for multiple
MACHINEs, and I don't need to write any script myself?

-- 
Tanu

Re: Triggering graceful bitbake shutdown

[Richard Purdie]

Hi,

On Tue, 2016-03-01 at 14:50 +0200, Tanu Kaskinen wrote:
> I wrote a small script that runs bitbake sequentially with multiple
> MACHINE values. Sometimes I want to stop it with ctrl-c before it has
> finished, but it didn't work properly out-of-the box when running
> bitbake under the script (my script would not wait for bitbake to
> exit,
> and bitbake threw some backtrace too). I then modified SIGINT
> handling
> in my script so that it forwards the signal to bitbake and then waits
> for bitbake to exit. The problem is that bitbake ignores SIGINT. The
> UI
> handles python's KeyboardInterrupt exception, but that exception
> doesn't get raised when bitbake runs under my script. I don't know
> what
> triggers KeyboardInterrupt in normal conditions, since SIGINT doesn't
> seem to be sufficient, does python perhaps monitor stdin to catch the
> actual ctrl-c keypress?
> 
> Anyway, sending SIGTERM instead of SIGINT "works". However, the
> SIGTERM
> handler shuts down bitbake immediately rather than waiting for the
> current tasks to finish. Is that less safe than the normal ctrl-c
> handling? If the immediate shutdown is equally safe, then I have no
> problem, but if SIGTERM can cause state corruption so that resuming
> later doesn't work reliably, then I'd like to have some way for
> triggering graceful bitbake shutdown from my script. Any suggestions?
> 
> Or maybe there is already some way to run bitbake for multiple
> MACHINEs, and I don't need to write any script myself?

I'm curious which bitbake process you're sending SIGINT to. If its the
knotty UI, it should shutdown gracefully. Other pieces of the system
ignore SIGINT since we really want the UI to do it. The behaviour you
mention above sounds suboptimal though.

I'm a bit pushed for time with the release stablisation at the moment
but would love to see if there is something better we can do here...

Cheers,

Richard

Re: Triggering graceful bitbake shutdown

[Tanu Kaskinen]

On Fri, 2016-03-04 at 17:58 +0000, Richard Purdie wrote:
> Hi,
> 
> On Tue, 2016-03-01 at 14:50 +0200, Tanu Kaskinen wrote:
> > I wrote a small script that runs bitbake sequentially with multiple
> > MACHINE values. Sometimes I want to stop it with ctrl-c before it has
> > finished, but it didn't work properly out-of-the box when running
> > bitbake under the script (my script would not wait for bitbake to
> > exit,
> > and bitbake threw some backtrace too). I then modified SIGINT
> > handling
> > in my script so that it forwards the signal to bitbake and then waits
> > for bitbake to exit. The problem is that bitbake ignores SIGINT. The
> > UI
> > handles python's KeyboardInterrupt exception, but that exception
> > doesn't get raised when bitbake runs under my script. I don't know
> > what
> > triggers KeyboardInterrupt in normal conditions, since SIGINT doesn't
> > seem to be sufficient, does python perhaps monitor stdin to catch the
> > actual ctrl-c keypress?
> > 
> > Anyway, sending SIGTERM instead of SIGINT "works". However, the
> > SIGTERM
> > handler shuts down bitbake immediately rather than waiting for the
> > current tasks to finish. Is that less safe than the normal ctrl-c
> > handling? If the immediate shutdown is equally safe, then I have no
> > problem, but if SIGTERM can cause state corruption so that resuming
> > later doesn't work reliably, then I'd like to have some way for
> > triggering graceful bitbake shutdown from my script. Any suggestions?
> > 
> > Or maybe there is already some way to run bitbake for multiple
> > MACHINEs, and I don't need to write any script myself?
> 
> I'm curious which bitbake process you're sending SIGINT to. If its the
> knotty UI, it should shutdown gracefully. Other pieces of the system
> ignore SIGINT since we really want the UI to do it. The behaviour you
> mention above sounds suboptimal though.

The script only sends the signal to the direct child process, which
obviously won't work if knotty is in a different process and the direct
child deliberately ignores the signal. However, I also tried sending
SIGINT to all other bitbake processes from the command line with
"kill", and none of them responded to it.

> I'm a bit pushed for time with the release stablisation at the moment
> but would love to see if there is something better we can do here...

I suppose knotty could use a custom SIGINT handler rather than relying
on KeyboardInterrupt. If I then change my script to send the signal to
the knotty process instead of the direct child, that should work.

-- 
Tanu

Re: Triggering graceful bitbake shutdown

[Tanu Kaskinen]

On Sat, 2016-03-05 at 12:41 +0200, Tanu Kaskinen wrote:
> On Fri, 2016-03-04 at 17:58 +0000, Richard Purdie wrote:
> > Hi,
> > 
> > On Tue, 2016-03-01 at 14:50 +0200, Tanu Kaskinen wrote:
> > > I wrote a small script that runs bitbake sequentially with multiple
> > > MACHINE values. Sometimes I want to stop it with ctrl-c before it has
> > > finished, but it didn't work properly out-of-the box when running
> > > bitbake under the script (my script would not wait for bitbake to
> > > exit,
> > > and bitbake threw some backtrace too). I then modified SIGINT
> > > handling
> > > in my script so that it forwards the signal to bitbake and then waits
> > > for bitbake to exit. The problem is that bitbake ignores SIGINT. The
> > > UI
> > > handles python's KeyboardInterrupt exception, but that exception
> > > doesn't get raised when bitbake runs under my script. I don't know
> > > what
> > > triggers KeyboardInterrupt in normal conditions, since SIGINT doesn't
> > > seem to be sufficient, does python perhaps monitor stdin to catch the
> > > actual ctrl-c keypress?
> > > 
> > > Anyway, sending SIGTERM instead of SIGINT "works". However, the
> > > SIGTERM
> > > handler shuts down bitbake immediately rather than waiting for the
> > > current tasks to finish. Is that less safe than the normal ctrl-c
> > > handling? If the immediate shutdown is equally safe, then I have no
> > > problem, but if SIGTERM can cause state corruption so that resuming
> > > later doesn't work reliably, then I'd like to have some way for
> > > triggering graceful bitbake shutdown from my script. Any suggestions?
> > > 
> > > Or maybe there is already some way to run bitbake for multiple
> > > MACHINEs, and I don't need to write any script myself?
> > 
> > I'm curious which bitbake process you're sending SIGINT to. If its the
> > knotty UI, it should shutdown gracefully. Other pieces of the system
> > ignore SIGINT since we really want the UI to do it. The behaviour you
> > mention above sounds suboptimal though.
> 
> The script only sends the signal to the direct child process, which
> obviously won't work if knotty is in a different process and the direct
> child deliberately ignores the signal. However, I also tried sending
> SIGINT to all other bitbake processes from the command line with
> "kill", and none of them responded to it.
> 
> > I'm a bit pushed for time with the release stablisation at the moment
> > but would love to see if there is something better we can do here...
> 
> I suppose knotty could use a custom SIGINT handler rather than relying
> on KeyboardInterrupt. If I then change my script to send the signal to
> the knotty process instead of the direct child, that should work.

Hmm... now I realized that if it's normal for knotty to run in a
separate process and still get KeyboardInterrupts, there must be some
difference in how my script spawns bitbake compared to how the "main"
bitbake process spawns the knotty process. Adding a custom SIGINT
handler to knotty may still make sense, but that should not be required
to make this work.

-- 
Tanu

Re: Triggering graceful bitbake shutdown

[Tanu Kaskinen]

On Sat, 2016-03-05 at 13:21 +0200, Tanu Kaskinen wrote:
> Hmm... now I realized that if it's normal for knotty to run in a
> separate process and still get KeyboardInterrupts, there must be some
> difference in how my script spawns bitbake compared to how the "main"
> bitbake process spawns the knotty process. Adding a custom SIGINT
> handler to knotty may still make sense, but that should not be required
> to make this work.

Sorry for the noise. The problem was that I spawned bitbake while
SIGINT was blocked in my script, and that was then inherited by the
child process. I removed the SIGINT block in the parent, and now
bitbake responds to ctrl-c just fine.

-- 
Tanu