Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp

[Joshua Watt <jpewhacker@gmail.com>]

On Thu, 2019-02-07 at 17:21 +0000, Richard Purdie wrote:
> On Thu, 2019-02-07 at 10:59 -0600, Joshua Watt wrote:
> > On Thu, Feb 7, 2019 at 10:44 AM Joshua Watt <jpewhacker@gmail.com>
> > wrote:
> > > Martin,
> > > 
> > > Do you by any chance build in Docker (or some other transient
> > > container)? We recently found an issue with building in docker
> > > where when the primary container command would exit, causing the
> > > container to terminate. When the container terminates, the kernel
> > > unceremoniously sends SIGKILL to all its processes. This plays
> > > *very* poorly with pseudo because pseudo keeps it's entire
> > > database
> > > in a in-memory sqlite database and only flushes it out to disk on
> > > a
> > > periodic interval, or when it get SIGTERM (maybe SIGQUIT?) or
> > > exits
> > > normally. By default the pseudo daemon will also hang around for
> > > 20(?) seconds waiting for a new connection from a pseudo client,
> > > meaning that when our docker container was exiting, there were
> > > potentially several pseudo dameons sitting around waiting for a
> > > client with unflushed databases in memory that suddenly got a
> > > SIGKILL and all their data was lost. This was particularly bad if
> > > you aborted a build while it was running, or some build failure
> > > occurred. Since 
> >  the pseudo database was lost, the only way to correct these errors
> > was to clean and rebuild.
> > > I think pseudo could handle this better and keep the database on
> > > disk instead of in memory, but in the short term you can hack
> > > bitbake to pass the -S flag when invoking the pseudo client which
> > > will make it tell the dameon to shutdown (and thus flush the
> > > database) if it is the last connection.
> > 
> > FWIW, this patch makes it much easier to do that
> > http://lists.openembedded.org/pipermail/bitbake-devel/2019-February/019813.html
> > Just add
> >  FAKEROOTCMD += "-S"
> > in local.conf
> 
> Should we be doing that by default? Is there a downside?

pseudo won't stick around waiting for connections from other clients.
I'm not sure if this would impact performance at all, as I'm not sure
how often a client is able to connect during the "grace period" to
prevent a new server needing to be spun up. I would image this
potentially cuts down on the spin up time for back-to-back fakeroot
tasks in the same recipe?

> 
> Keeping the database flushed to disk would likely be slow but this
> seems reasonable.

I think there are 3 options in increasing order of
complexity/risk/safety

 1. Add -S to FAKEROOTCMD by default
 2. Make pseudo flush the in memory cache every time the last client
disconnects
 3. Remove the in-memory cache from pseudo completely and make it write
to disk each time. Even if we removed all disk flushes from the sqlite
database to keep sync()'s down, it would no worse that what happens
today in any case I can think of (specifically, a power loss will cause
lost data), and in many cases it would be better (e.g. SIGKILL and
crashes in pseudo wouldn't cause data loss).

> 
> Cheers,
> 
> Richard
> 
> 
-- 
Joshua Watt <JPEWhacker@gmail.com>