[PATCH 0/1] persist_data: reconnect when DB is locked

[PATCH 0/1] persist_data: reconnect when DB is locked

[Lianhao Lu]

Fixed yocto bug #1761.

The timeout value set in sqlite3 connection only applies to the very first 
time when accessing a locked DB file.

By reconnect to the DB in 'database is locked' exception, we can leverage 
the timeout value in each retry. 

The following changes since commit 684c6ef2d08ad503ce68ac3cc273f1b6a8a56ac7:
  Paul Eggleton (1):
        bitbake/knotty: print task failure summary

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib llu/bug1761
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=llu/bug1761

Lianhao Lu (1):
  bitbake/persist_data: Reconnect when DB is locked

 bitbake/lib/bb/persist_data.py |   12 +++++++-----
 1 files changed, 7 insertions(+), 5 deletions(-)

[PATCH 1/1] bitbake/persist_data: Reconnect when DB is locked

[Lianhao Lu]

[YOCTO #1761]
Reconnect to the backend Sqlite DB in 'database is locked' exception so
the timeout can be leveraged in each time retry.

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
---
 bitbake/lib/bb/persist_data.py |   12 +++++++-----
 1 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py
index 17620ef..c4ea23b 100644
--- a/bitbake/lib/bb/persist_data.py
+++ b/bitbake/lib/bb/persist_data.py
@@ -47,9 +47,10 @@ if hasattr(sqlite3, 'enable_shared_cache'):
 @total_ordering
 class SQLTable(collections.MutableMapping):
     """Object representing a table/domain in the database"""
-    def __init__(self, cursor, table):
-        self.cursor = cursor
+    def __init__(self, cachefile, table):
+        self.cachefile = cachefile
         self.table = table
+        self.cursor = connect(self.cachefile)
 
         self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value TEXT);"
                       % table)
@@ -63,6 +64,8 @@ class SQLTable(collections.MutableMapping):
             except sqlite3.OperationalError as exc:
                 if 'database is locked' in str(exc) and count < 500:
                     count = count + 1
+                    self.cursor.close()
+                    self.cursor = connect(self.cachefile)
                     continue
                 raise
 
@@ -188,7 +191,7 @@ class PersistData(object):
         del self.data[domain][key]
 
 def connect(database):
-    return sqlite3.connect(database, timeout=30, isolation_level=None)
+    return sqlite3.connect(database, timeout=5, isolation_level=None)
 
 def persist(domain, d):
     """Convenience factory for SQLTable objects based upon metadata"""
@@ -201,5 +204,4 @@ def persist(domain, d):
 
     bb.utils.mkdirhier(cachedir)
     cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3")
-    connection = connect(cachefile)
-    return SQLTable(connection, domain)
+    return SQLTable(cachefile, domain)
-- 
1.7.0.4

Re: [PATCH 1/1] bitbake/persist_data: Reconnect when DB is locked

[Richard Purdie]

On Mon, 2012-01-16 at 12:07 +0800, Lianhao Lu wrote:
> [YOCTO #1761]
> Reconnect to the backend Sqlite DB in 'database is locked' exception so
> the timeout can be leveraged in each time retry.
> 
> Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
> ---
>  bitbake/lib/bb/persist_data.py |   12 +++++++-----
>  1 files changed, 7 insertions(+), 5 deletions(-)

Merged to master, thanks. Good to finally have a handle on why this kept
coming up! :)

Cheers,

Richard

Re: [PATCH 1/1] bitbake/persist_data: Reconnect when DB is locked

[McClintock Matthew-B29882]

This needs to be in edison-next. It cherry-pick's cleanly as an added
plus (doing more testing now). Seems like it's already in denzil-next

-M

On Sun, Jan 15, 2012 at 10:07 PM, Lianhao Lu <lianhao.lu@intel.com> wrote:
> [YOCTO #1761]
> Reconnect to the backend Sqlite DB in 'database is locked' exception so
> the timeout can be leveraged in each time retry.
>
> Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
> ---
>  bitbake/lib/bb/persist_data.py |   12 +++++++-----
>  1 files changed, 7 insertions(+), 5 deletions(-)
>
> diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py
> index 17620ef..c4ea23b 100644
> --- a/bitbake/lib/bb/persist_data.py
> +++ b/bitbake/lib/bb/persist_data.py
> @@ -47,9 +47,10 @@ if hasattr(sqlite3, 'enable_shared_cache'):
>  @total_ordering
>  class SQLTable(collections.MutableMapping):
>     """Object representing a table/domain in the database"""
> -    def __init__(self, cursor, table):
> -        self.cursor = cursor
> +    def __init__(self, cachefile, table):
> +        self.cachefile = cachefile
>         self.table = table
> +        self.cursor = connect(self.cachefile)
>
>         self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value TEXT);"
>                       % table)
> @@ -63,6 +64,8 @@ class SQLTable(collections.MutableMapping):
>             except sqlite3.OperationalError as exc:
>                 if 'database is locked' in str(exc) and count < 500:
>                     count = count + 1
> +                    self.cursor.close()
> +                    self.cursor = connect(self.cachefile)
>                     continue
>                 raise
>
> @@ -188,7 +191,7 @@ class PersistData(object):
>         del self.data[domain][key]
>
>  def connect(database):
> -    return sqlite3.connect(database, timeout=30, isolation_level=None)
> +    return sqlite3.connect(database, timeout=5, isolation_level=None)
>
>  def persist(domain, d):
>     """Convenience factory for SQLTable objects based upon metadata"""
> @@ -201,5 +204,4 @@ def persist(domain, d):
>
>     bb.utils.mkdirhier(cachedir)
>     cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3")
> -    connection = connect(cachefile)
> -    return SQLTable(connection, domain)
> +    return SQLTable(cachefile, domain)
> --
> 1.7.0.4
>
>
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel

Re: [PATCH 1/1] bitbake/persist_data: Reconnect when DB is locked

[Joshua Lock]

On 30/05/12 17:33, McClintock Matthew-B29882 wrote:
> This needs to be in edison-next. It cherry-pick's cleanly as an added
> plus (doing more testing now). Seems like it's already in denzil-next

cherry-picked to my josh/edison branch on poky-contrib - I will try and 
ensure it makes it into the edison branch of the main repository promptly.

Cheers,
Joshua
-- 
Joshua Lock
         Yocto Project
         Intel Open Source Technology Centre

Re: [PATCH 1/1] bitbake/persist_data: Reconnect when DB is locked

[McClintock Matthew-B29882]

On Thu, May 31, 2012 at 12:54 PM, Joshua Lock <josh@linux.intel.com> wrote:
> On 30/05/12 17:33, McClintock Matthew-B29882 wrote:
>>
>> This needs to be in edison-next. It cherry-pick's cleanly as an added
>> plus (doing more testing now). Seems like it's already in denzil-next
>
>
> cherry-picked to my josh/edison branch on poky-contrib - I will try and
> ensure it makes it into the edison branch of the main repository promptly.

This has resolved my issue and I've seen no new issues after quite a
bit of building.

-M