[PATCH] prserv/db: Avoid fsync() calls

[PATCH] prserv/db: Avoid fsync() calls

[Richard Purdie]

If the power were to fail, it doesn't matter to us much if the data
makes it to disk or not, we'd have other problems. However an fsync()
call on a multi build autobuilder is painful so lets avoid them.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/bitbake/lib/prserv/db.py b/bitbake/lib/prserv/db.py
index 86eafe3..49f36da 100644
--- a/bitbake/lib/prserv/db.py
+++ b/bitbake/lib/prserv/db.py
@@ -220,6 +220,7 @@ class PRData(object):
                 raise e
         self.connection=sqlite3.connect(self.filename, isolation_level="EXCLUSIVE", check_same_thread = False)
         self.connection.row_factory=sqlite3.Row
+        self.connection.execute("pragma synchronous = off;")
         self._tables={}
 
     def __del__(self):

Re: [PATCH] prserv/db: Avoid fsync() calls

[Mark Hatle]

On 5/3/14, 3:18 AM, Richard Purdie wrote:
> If the power were to fail, it doesn't matter to us much if the data
> makes it to disk or not, we'd have other problems. However an fsync()
> call on a multi build autobuilder is painful so lets avoid them.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/bitbake/lib/prserv/db.py b/bitbake/lib/prserv/db.py
> index 86eafe3..49f36da 100644
> --- a/bitbake/lib/prserv/db.py
> +++ b/bitbake/lib/prserv/db.py
> @@ -220,6 +220,7 @@ class PRData(object):
>                   raise e
>           self.connection=sqlite3.connect(self.filename, isolation_level="EXCLUSIVE", check_same_thread = False)
>           self.connection.row_factory=sqlite3.Row
> +        self.connection.execute("pragma synchronous = off;")
>           self._tables={}
>
>       def __del__(self):
>
>

Does the prserver make any noticeable difference?  This may be the one case 
where I could see an fsync being desired.  If you are serving remote machines, 
once you've requested a PR, the number should be stored to disk so that it will 
be the same one that comes back again.  ... but I can easily see that in the 
middle of a build, when the PR server is on the same machine (and not shared) 
this could cause enough slow down to justify turning off the fsync.

--Mark

Re: [PATCH] prserv/db: Avoid fsync() calls

[Richard Purdie]

On Sat, 2014-05-03 at 09:12 -0700, Mark Hatle wrote:
> On 5/3/14, 3:18 AM, Richard Purdie wrote:
> > If the power were to fail, it doesn't matter to us much if the data
> > makes it to disk or not, we'd have other problems. However an fsync()
> > call on a multi build autobuilder is painful so lets avoid them.
> >
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> > diff --git a/bitbake/lib/prserv/db.py b/bitbake/lib/prserv/db.py
> > index 86eafe3..49f36da 100644
> > --- a/bitbake/lib/prserv/db.py
> > +++ b/bitbake/lib/prserv/db.py
> > @@ -220,6 +220,7 @@ class PRData(object):
> >                   raise e
> >           self.connection=sqlite3.connect(self.filename, isolation_level="EXCLUSIVE", check_same_thread = False)
> >           self.connection.row_factory=sqlite3.Row
> > +        self.connection.execute("pragma synchronous = off;")
> >           self._tables={}
> >
> >       def __del__(self):
> >
> >
> 
> Does the prserver make any noticeable difference?  This may be the one case 
> where I could see an fsync being desired.  If you are serving remote machines, 
> once you've requested a PR, the number should be stored to disk so that it will 
> be the same one that comes back again.  ... but I can easily see that in the 
> middle of a build, when the PR server is on the same machine (and not shared) 
> this could cause enough slow down to justify turning off the fsync.

Our default use case is that of the same machine and yes, these fsyncs
do made a difference in that workload for things like UI interaction
too. I stumbled into this wondering why my build wasn't starting and the
IO maxed out which made me look further.

I understand the issue of the remote server needing to write to disk
however even if its not on disk, it will be in sync with any other
questions due to the locking so it is just a power interruption issue
and I'm therefore less worried about it in most of the use cases we see
this used in.

Cheers,

Richard