Weird migration issue

Weird migration issue

[Barros Pena, Belen]

I've been coming across a problem with migrations occasionally (and rather
randomly) when I start Toaster. At the beginning it seemed I was the only
one seeing this. But, while at ELC, someone came to me because they were
having the same problem. His trace is below. Restarting the host seems to
solve the problem for me. In his case, he tried a different host and
everything worked. 

It doesn't seem distro-related: his was CentOS, mine Ubuntu. That's all I
know. Michael Wood looked at this briefly, so he might have something to
add. It was not a problem when it was just me, but if other people are
seeing this, then I am bit more worried.

Any ideas on what could be causing it?

Cheers

Belén

File 
"/state/partition1/scratch/tartler/yocto-master/poky/bitbake/bin/../lib/toa
ster/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/core/management/__init__.py", line 354, in
execute_from_command_line
    utility.execute()
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/core/management/base.py", line 394, in
run_from_argv
    self.execute(*args, **cmd_options)
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/core/management/commands/migrate.py", line 222, in
handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/db/migrations/executor.py", line 110, in migrate
    self.apply_migration(states[migration], migration, fake=fake,
fake_initial=fake_initial)
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/db/migrations/executor.py", line 148, in
apply_migration
    state = migration.apply(state, schema_editor)
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/db/backends/base/schema.py", line 93, in __exit__
    self.atomic.__exit__(exc_type, exc_value, traceback)
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/db/transaction.py", line 228, in __exit__
    connection.rollback()
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/db/backends/base/base.py", line 183, in rollback
    self._rollback()
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/db/backends/base/base.py", line 147, in _rollback
    return self.connection.rollback()
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File 
"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
.7/site-packages/django/db/backends/base/base.py", line 147, in _rollback
    return self.connection.rollback()
django.db.utils.OperationalError: cannot rollback - no transaction is
active
Failed migrations, aborting system start
Failed start.

Re: Weird migration issue

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 4289 bytes --]

On 20 April 2016 at 10:24, Barros Pena, Belen <belen.barros.pena@intel.com>
wrote:

> I've been coming across a problem with migrations occasionally (and rather
> randomly) when I start Toaster. At the beginning it seemed I was the only
> one seeing this. But, while at ELC, someone came to me because they were
> having the same problem. His trace is below. Restarting the host seems to
> solve the problem for me. In his case, he tried a different host and
> everything worked.
>

Is this with MySQL or SQLite?

Elliot


>
> It doesn't seem distro-related: his was CentOS, mine Ubuntu. That's all I
> know. Michael Wood looked at this briefly, so he might have something to
> add. It was not a problem when it was just me, but if other people are
> seeing this, then I am bit more worried.
>
> Any ideas on what could be causing it?
>
> Cheers
>
> Belén
>
> File
> "/state/partition1/scratch/tartler/yocto-master/poky/bitbake/bin/../lib/toa
> ster/manage.py", line 10, in <module>
>     execute_from_command_line(sys.argv)
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/core/management/__init__.py", line 354, in
> execute_from_command_line
>     utility.execute()
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/core/management/__init__.py", line 346, in execute
>     self.fetch_command(subcommand).run_from_argv(self.argv)
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/core/management/base.py", line 394, in
> run_from_argv
>     self.execute(*args, **cmd_options)
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/core/management/base.py", line 445, in execute
>     output = self.handle(*args, **options)
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/core/management/commands/migrate.py", line 222, in
> handle
>     executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/migrations/executor.py", line 110, in migrate
>     self.apply_migration(states[migration], migration, fake=fake,
> fake_initial=fake_initial)
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/migrations/executor.py", line 148, in
> apply_migration
>     state = migration.apply(state, schema_editor)
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/backends/base/schema.py", line 93, in __exit__
>     self.atomic.__exit__(exc_type, exc_value, traceback)
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/transaction.py", line 228, in __exit__
>     connection.rollback()
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/backends/base/base.py", line 183, in rollback
>     self._rollback()
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/backends/base/base.py", line 147, in _rollback
>     return self.connection.rollback()
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/utils.py", line 98, in __exit__
>     six.reraise(dj_exc_type, dj_exc_value, traceback)
>   File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/backends/base/base.py", line 147, in _rollback
>     return self.connection.rollback()
> django.db.utils.OperationalError: cannot rollback - no transaction is
> active
> Failed migrations, aborting system start
> Failed start.
>
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>



-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 5546 bytes --]

Re: Weird migration issue

[Barros Pena, Belen]

On 20/04/2016 11:03, "Smith, Elliot" <elliot.smith@intel.com> wrote:

>On 20 April 2016 at 10:24, Barros Pena, Belen
><belen.barros.pena@intel.com> wrote:
>
>I've been coming across a problem with migrations occasionally (and rather
>randomly) when I start Toaster. At the beginning it seemed I was the only
>one seeing this. But, while at ELC, someone came to me because they were
>having the same problem. His trace is below. Restarting the host seems to
>solve the problem for me. In his case, he tried a different host and
>everything worked.
>
>
>
>
>Is this with MySQL or SQLite?

SQLite

>
>
>Elliot
>
> 
>
>
>It doesn't seem distro-related: his was CentOS, mine Ubuntu. That's all I
>know. Michael Wood looked at this briefly, so he might have something to
>add. It was not a problem when it was just me, but if other people are
>seeing this, then I am bit more worried.
>
>Any ideas on what could be causing it?
>
>Cheers
>
>Belén
>
>File
>"/state/partition1/scratch/tartler/yocto-master/poky/bitbake/bin/../lib/to
>a
>ster/manage.py", line 10, in <module>
>    execute_from_command_line(sys.argv)
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/core/management/__init__.py", line 354, in
>execute_from_command_line
>    utility.execute()
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/core/management/__init__.py", line 346, in execute
>    self.fetch_command(subcommand).run_from_argv(self.argv)
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/core/management/base.py", line 394, in
>run_from_argv
>    self.execute(*args, **cmd_options)
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/core/management/base.py", line 445, in execute
>    output = self.handle(*args, **options)
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/core/management/commands/migrate.py", line 222, in
>handle
>    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/db/migrations/executor.py", line 110, in migrate
>    self.apply_migration(states[migration], migration, fake=fake,
>fake_initial=fake_initial)
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/db/migrations/executor.py", line 148, in
>apply_migration
>    state = migration.apply(state, schema_editor)
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/db/backends/base/schema.py", line 93, in __exit__
>    self.atomic.__exit__(exc_type, exc_value, traceback)
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/db/transaction.py", line 228, in __exit__
>    connection.rollback()
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/db/backends/base/base.py", line 183, in rollback
>    self._rollback()
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/db/backends/base/base.py", line 147, in _rollback
>    return self.connection.rollback()
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/db/utils.py", line 98, in __exit__
>    six.reraise(dj_exc_type, dj_exc_value, traceback)
>  File
>"/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python
>2
>.7/site-packages/django/db/backends/base/base.py", line 147, in _rollback
>    return self.connection.rollback()
>django.db.utils.OperationalError: cannot rollback - no transaction is
>active
>Failed migrations, aborting system start
>Failed start.
>
>
>--
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster
>
>
>
>
>
>
>
>-- 
>Elliot Smith
>Software Engineer
>Intel Open Source Technology Centre
>
>
>
>

Re: Weird migration issue

[Michael Wood]

Yes I briefly looked into this, and I think it has to do with a memory 
mapping issue. If the database is not closed before trying to run 
database migrate we get into a weird state where there is a transaction 
in memory that can't be completed as the database has been deleted.

This then blocks creating the database until it's gone from memory, the 
only way to do this is seemingly to reboot or use a different location 
on the filesystem for Toaster ...which is kind of worrying, we should 
probably open a django bug about this.

To avoid this make sure that toaster/bitbake is fully stopped before 
running migrate/touching the database file.

Michael

On 20/04/16 10:24, Barros Pena, Belen wrote:
> I've been coming across a problem with migrations occasionally (and rather
> randomly) when I start Toaster. At the beginning it seemed I was the only
> one seeing this. But, while at ELC, someone came to me because they were
> having the same problem. His trace is below. Restarting the host seems to
> solve the problem for me. In his case, he tried a different host and
> everything worked.
>
> It doesn't seem distro-related: his was CentOS, mine Ubuntu. That's all I
> know. Michael Wood looked at this briefly, so he might have something to
> add. It was not a problem when it was just me, but if other people are
> seeing this, then I am bit more worried.
>
> Any ideas on what could be causing it?
>
> Cheers
>
> Belén
>
> File
> "/state/partition1/scratch/tartler/yocto-master/poky/bitbake/bin/../lib/toa
> ster/manage.py", line 10, in <module>
>      execute_from_command_line(sys.argv)
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/core/management/__init__.py", line 354, in
> execute_from_command_line
>      utility.execute()
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/core/management/__init__.py", line 346, in execute
>      self.fetch_command(subcommand).run_from_argv(self.argv)
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/core/management/base.py", line 394, in
> run_from_argv
>      self.execute(*args, **cmd_options)
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/core/management/base.py", line 445, in execute
>      output = self.handle(*args, **options)
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/core/management/commands/migrate.py", line 222, in
> handle
>      executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/migrations/executor.py", line 110, in migrate
>      self.apply_migration(states[migration], migration, fake=fake,
> fake_initial=fake_initial)
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/migrations/executor.py", line 148, in
> apply_migration
>      state = migration.apply(state, schema_editor)
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/backends/base/schema.py", line 93, in __exit__
>      self.atomic.__exit__(exc_type, exc_value, traceback)
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/transaction.py", line 228, in __exit__
>      connection.rollback()
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/backends/base/base.py", line 183, in rollback
>      self._rollback()
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/backends/base/base.py", line 147, in _rollback
>      return self.connection.rollback()
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/utils.py", line 98, in __exit__
>      six.reraise(dj_exc_type, dj_exc_value, traceback)
>    File
> "/state/partition1/scratch/tartler/yocto-master/poky/build/venv/lib/python2
> .7/site-packages/django/db/backends/base/base.py", line 147, in _rollback
>      return self.connection.rollback()
> django.db.utils.OperationalError: cannot rollback - no transaction is
> active
> Failed migrations, aborting system start
> Failed start.
>
>