Leveldb in google-chrome incompatible with btrfs?

Leveldb in google-chrome incompatible with btrfs?

[Marc MERLIN]

I don't know who else uses google-chrome here, but for me, for as long as
I've used btrfs (3+ years now), I've had no end of troubles recovering from
a linux crash, and google-chrome has had problems recovering my tabs and
usually cmoplains about plenty of problems, some are corruption looking.

The way I recover, every time, is to simply rsync over the previous
snapshot's data for ~/.config/google-chrome-dir back over the current
corrupted one, and move on.

I had a bit more look at this with a chrome engineer, and there seems to be
at least a clear problem with leveldb where its file containing the index
name, does not match the files on disk.
Details are in https://bugs.chromium.org/p/chromium/issues/detail?id=738961
but I'll summarize here.


All the files are in a non corrupted state, but I end up with this:
saruman:~/.config/google-chrome-dir/google-chrome-beta/Default/IndexedDB/https_docs.google.com_0.indexeddb.leveldb$ l
(...)
-rw------- 1 merlin merlin      16 Oct  4  2016 CURRENT
-rw------- 1 merlin merlin       0 Apr  9  2016 LOCK
-rw------- 1 merlin merlin       0 Jul  5 22:31 LOG
-rw------- 1 merlin merlin       0 Jul  5 22:31 LOG.old
-rw------- 1 merlin merlin      23 Jul  3 08:31 MANIFEST-000001
-rw------- 1 merlin merlin    8639 May 12 17:36 MANIFEST-022745
saruman:~/.config/google-chrome-dir/google-chrome-beta/Default/IndexedDB/https_docs.google.com_0.indexeddb.leveldb$ cat CURRENT 
MANIFEST-010814

In other words, I think things go like this:
- MANIFEST-010814 is replaced by MANIFEST-022745, 
- MANIFEST-022745 is written into the file CURRENT
- ideally both MANIFEST-010814 and MANIFEST-022745 should be present on disk
  and MANIFEST-010814 deleted only after CURRENT has been written to with
  the new pointer, but I'm not sure if or how that is done.
- my laptop crashes
- after reboot, btrfs has not been able to commit the update to CURRENT to disk
  in a consistent state, therefore discards the COW data to CURRENT and
  leaves it in its prior state, which now holds the old data
- somehow MANIFEST-010814 has however been deleted, so now CURRENT points to
  a non existent file.

I don't know how leveldb works or where its (f)sync statements are. but I'm
guessing it works reliably on ext4 since it's used by many users and I'm the
first one reporting this problem with leveldb.

Does anyone know if it's leveldb relying on non POSIX semantics that just
happen to work out on ext4, or if btrfs COW and atomicity doesn't quite
handle multi file updates in a way that is expected by a spec, or by
application developers?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Leveldb in google-chrome incompatible with btrfs?

[Omar Sandoval]

On Thu, Jul 06, 2017 at 08:00:46AM -0700, Marc MERLIN wrote:
> I don't know who else uses google-chrome here, but for me, for as long as
> I've used btrfs (3+ years now), I've had no end of troubles recovering from
> a linux crash, and google-chrome has had problems recovering my tabs and
> usually cmoplains about plenty of problems, some are corruption looking.

I've also had issues with chrome and Btrfs, not just you.

[snip]

> Does anyone know if it's leveldb relying on non POSIX semantics that just
> happen to work out on ext4, or if btrfs COW and atomicity doesn't quite
> handle multi file updates in a way that is expected by a spec, or by
> application developers?

A quick google search turned this up: https://github.com/google/leveldb/issues/195.
Unless anything has changed since that issue was last updated, it does
sound like LevelDB is making some unsafe assumptions. I'll take a look.

Re: Leveldb in google-chrome incompatible with btrfs?

[Marc MERLIN]

On Thu, Jul 06, 2017 at 02:13:20PM -0700, Omar Sandoval wrote:
> On Thu, Jul 06, 2017 at 08:00:46AM -0700, Marc MERLIN wrote:
> > I don't know who else uses google-chrome here, but for me, for as long as
> > I've used btrfs (3+ years now), I've had no end of troubles recovering from
> > a linux crash, and google-chrome has had problems recovering my tabs and
> > usually cmoplains about plenty of problems, some are corruption looking.
> 
> I've also had issues with chrome and Btrfs, not just you.
> 
> [snip]
> 
> > Does anyone know if it's leveldb relying on non POSIX semantics that just
> > happen to work out on ext4, or if btrfs COW and atomicity doesn't quite
> > handle multi file updates in a way that is expected by a spec, or by
> > application developers?
> 
> A quick google search turned this up: https://github.com/google/leveldb/issues/195.
> Unless anything has changed since that issue was last updated, it does
> sound like LevelDB is making some unsafe assumptions. I'll take a look.

Thanks Omar, this very much looks related indeed.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Leveldb in google-chrome incompatible with btrfs?

[Omar Sandoval]

On Thu, Jul 06, 2017 at 02:24:22PM -0700, Marc MERLIN wrote:
> On Thu, Jul 06, 2017 at 02:13:20PM -0700, Omar Sandoval wrote:
> > On Thu, Jul 06, 2017 at 08:00:46AM -0700, Marc MERLIN wrote:
> > > I don't know who else uses google-chrome here, but for me, for as long as
> > > I've used btrfs (3+ years now), I've had no end of troubles recovering from
> > > a linux crash, and google-chrome has had problems recovering my tabs and
> > > usually cmoplains about plenty of problems, some are corruption looking.
> > 
> > I've also had issues with chrome and Btrfs, not just you.
> > 
> > [snip]
> > 
> > > Does anyone know if it's leveldb relying on non POSIX semantics that just
> > > happen to work out on ext4, or if btrfs COW and atomicity doesn't quite
> > > handle multi file updates in a way that is expected by a spec, or by
> > > application developers?
> > 
> > A quick google search turned this up: https://github.com/google/leveldb/issues/195.
> > Unless anything has changed since that issue was last updated, it does
> > sound like LevelDB is making some unsafe assumptions. I'll take a look.
> 
> Thanks Omar, this very much looks related indeed.
> 
> Marc

Hm, tracing a simple program using LevelDB I put together, it looks like
the relevant sequence of events on open is pretty much

create new MANIFEST-nnnnnn
write out new MANIFEST-nnnnnn
fsync() parent directory
fdatasync() MANIFEST-nnnnnn

create temporary CURRENT
write out temporary CURRENT to point to new MANIFEST-nnnnnn name
fdatasync() temporary CURRENT
rename temporary CURRENT to CURRENT

unlink old MANIFEST-nnnnnn

There's no fsync of the parent directory between renaming CURRENT and
unlinking the old MANIFEST, but that shouldn't be an issue on Btrfs.
Either both of those operations will be committed in the same filesystem
transaction, or the rename will be committed before the unlink. I can't
think of any way that the unlink would end up reordered before the
rename.

What doesn't add up about your bug report is that your CURRENT points to
a MANIFEST-010814 way behind all of the other files in that directory,
which are numbered 022745+. If there were a bug here, I'd expect the
stale MANIFEST file would be one older than the new one. The filenames
seem to be allocated sequentially, so that old MANIFEST file CURRENT
refers to must be really old, which doesn't make sense. I don't see how
Btrfs would screw that up :) I'd be interested to see if you can make
the same condition trigger again.

Re: Leveldb in google-chrome incompatible with btrfs?

[Marc MERLIN]

On Thu, Jul 06, 2017 at 04:01:41PM -0700, Omar Sandoval wrote:
> What doesn't add up about your bug report is that your CURRENT points to
> a MANIFEST-010814 way behind all of the other files in that directory,
> which are numbered 022745+. If there were a bug here, I'd expect the
> stale MANIFEST file would be one older than the new one. The filenames
> seem to be allocated sequentially, so that old MANIFEST file CURRENT
> refers to must be really old, which doesn't make sense. I don't see how
> Btrfs would screw that up :) I'd be interested to see if you can make
> the same condition trigger again.
> 

First, thanks for looking at it.

Second, you are right on the numbers being so far apart that something was
wrong. I checked my snapshots, and I've been carrying that MANIFEST-010814
for a long time.
In other words, it's a old stale manifest that never got deleted.

The new real old one apparently got deleted, the new one was created but
didn't make it to disk, but the pointer in CURRENT did get repointed to the
new one that never made it to actual disk.

So I think what happened is something like this:
MANIFEST-new got created
echo MANIFEST-new > CURRENT
MANIFEST-old got deleted
system crashed

MANIFEST-old was indeed deleted, and MANIFEST-new never made it to disk.

Does that sound more plausible?

As for redoing this at will, apparently I may have been hit by the skylake
hyperthreading CPU bug that I just installed a microcode update for, which
was causing random crashes, which hopefully are now solved.
I can't say if those in turn messed with btrfs writing data, but I'd rather
not recreate this since it's my real filesystem I care about and don't want
to corrupt on purpose :)
That said, the google-chrome on my previous haswell CPU also had routine
problems when restarting chrome, although at this point I don't know if they
were due to leveldb or sqlite or something else.
I'm just mentioning this to say that I'm pretty sure that the haswell HT bug
isn't the sole culprit of this problem, likely just the trigger of some of
my crashes.

Hope this helps
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Leveldb in google-chrome incompatible with btrfs?

[Omar Sandoval]

On Thu, Jul 06, 2017 at 04:31:52PM -0700, Marc MERLIN wrote:
> On Thu, Jul 06, 2017 at 04:01:41PM -0700, Omar Sandoval wrote:
> > What doesn't add up about your bug report is that your CURRENT points to
> > a MANIFEST-010814 way behind all of the other files in that directory,
> > which are numbered 022745+. If there were a bug here, I'd expect the
> > stale MANIFEST file would be one older than the new one. The filenames
> > seem to be allocated sequentially, so that old MANIFEST file CURRENT
> > refers to must be really old, which doesn't make sense. I don't see how
> > Btrfs would screw that up :) I'd be interested to see if you can make
> > the same condition trigger again.
> > 
> 
> First, thanks for looking at it.
> 
> Second, you are right on the numbers being so far apart that something was
> wrong. I checked my snapshots, and I've been carrying that MANIFEST-010814
> for a long time.
> In other words, it's a old stale manifest that never got deleted.
> 
> The new real old one apparently got deleted, the new one was created but
> didn't make it to disk, but the pointer in CURRENT did get repointed to the
> new one that never made it to actual disk.
> 
> So I think what happened is something like this:
> MANIFEST-new got created
> echo MANIFEST-new > CURRENT
> MANIFEST-old got deleted
> system crashed
> 
> MANIFEST-old was indeed deleted, and MANIFEST-new never made it to disk.
> 
> Does that sound more plausible?

In the bug report, you commented that CURRENT contained MANIFEST-010814,
is that indeed the case or was it actually something newer? If it was
the newer one, then it's still tricky how we'd end up that way but not
as outlandish.

> As for redoing this at will, apparently I may have been hit by the skylake
> hyperthreading CPU bug that I just installed a microcode update for, which
> was causing random crashes, which hopefully are now solved.
> I can't say if those in turn messed with btrfs writing data, but I'd rather
> not recreate this since it's my real filesystem I care about and don't want
> to corrupt on purpose :)

Understandable :)

> That said, the google-chrome on my previous haswell CPU also had routine
> problems when restarting chrome, although at this point I don't know if they
> were due to leveldb or sqlite or something else.
> I'm just mentioning this to say that I'm pretty sure that the haswell HT bug
> isn't the sole culprit of this problem, likely just the trigger of some of
> my crashes.
>
> Hope this helps
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/  

Re: Leveldb in google-chrome incompatible with btrfs?

[Marc MERLIN]

On Thu, Jul 06, 2017 at 04:44:51PM -0700, Omar Sandoval wrote:
> In the bug report, you commented that CURRENT contained MANIFEST-010814,
> is that indeed the case or was it actually something newer? If it was
> the newer one, then it's still tricky how we'd end up that way but not
> as outlandish.

You are correct, my bad.
At this point, I'm going to have to assume that something bad happened with
rsync when I rsync'ed an old profile over the one that caused chrome to fail
to restart.
Especially because CURRENT was dated Oct 4th, which does not make sense.

Now that I know what to look for, I'll have a much closer look next time
this happens, with the understanding that it would be a while if I've
successfully fixed the reason why my laptop was crashing too often.

But you said you've also seen issues with google-chrome profile and btrfs.
What did you experience?

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Leveldb in google-chrome incompatible with btrfs?

[Omar Sandoval]

On Thu, Jul 06, 2017 at 04:59:39PM -0700, Marc MERLIN wrote:
> On Thu, Jul 06, 2017 at 04:44:51PM -0700, Omar Sandoval wrote:
> > In the bug report, you commented that CURRENT contained MANIFEST-010814,
> > is that indeed the case or was it actually something newer? If it was
> > the newer one, then it's still tricky how we'd end up that way but not
> > as outlandish.
> 
> You are correct, my bad.
> At this point, I'm going to have to assume that something bad happened with
> rsync when I rsync'ed an old profile over the one that caused chrome to fail
> to restart.
> Especially because CURRENT was dated Oct 4th, which does not make sense.

Okay, this makes sense.

> Now that I know what to look for, I'll have a much closer look next time
> this happens, with the understanding that it would be a while if I've
> successfully fixed the reason why my laptop was crashing too often.

Sounds good.

> But you said you've also seen issues with google-chrome profile and btrfs.
> What did you experience?

I never debugged it, but I had to blow away the profile a couple of times. Then
there's this weird one which looks like a Btrfs bug:

┌[osandov@vader ~/.config]
└$ ls -al google-chrome-busted/**
ls: cannot access 'google-chrome-busted/Local State': No such file or directory
google-chrome-busted/Default:
ls: cannot access 'google-chrome-busted/Default/Preferences': No such file or directory
ls: cannot access 'google-chrome-busted/Default/.com.google.Chrome.VfAUNx': No such file or directory
total 0
drwx------ 1 osandov users 12 Feb  7 16:50 .
drwx------ 1 osandov users 14 Feb  7 16:50 ..
-????????? ? ?       ?      ?            ? .com.google.Chrome.VfAUNx
-????????? ? ?       ?      ?            ? Preferences

Re: Leveldb in google-chrome incompatible with btrfs?

[Cerem Cem ASLAN]

I was also struggling with this issue for quite some time. Today my 2
months old disk is crashed (which really surprises me). It doesn't
even being shown by `fdisk -l`.

After buying this this disk, I installed a Debian on a BTRFS partition
on an LVM partition on a LUKS partition. From that day till today,
while doing my work, there was always a video running on Youtube in
the background. Sometimes my editor (atom, it also shares a lot with
chromium) was slowing down which was making me end up restarting the
Chromium (and it was restarting barely with `killall -9 chromium`.
Another mentionable event is, 3 days ago my laptop suddenly restarted.

I suspect Chromium usage on BTRFS and disk the crash are somehow related.

2017-07-07 8:46 GMT+03:00 Omar Sandoval <osandov@osandov.com>:
> On Thu, Jul 06, 2017 at 04:59:39PM -0700, Marc MERLIN wrote:
>> On Thu, Jul 06, 2017 at 04:44:51PM -0700, Omar Sandoval wrote:
>> > In the bug report, you commented that CURRENT contained MANIFEST-010814,
>> > is that indeed the case or was it actually something newer? If it was
>> > the newer one, then it's still tricky how we'd end up that way but not
>> > as outlandish.
>>
>> You are correct, my bad.
>> At this point, I'm going to have to assume that something bad happened with
>> rsync when I rsync'ed an old profile over the one that caused chrome to fail
>> to restart.
>> Especially because CURRENT was dated Oct 4th, which does not make sense.
>
> Okay, this makes sense.
>
>> Now that I know what to look for, I'll have a much closer look next time
>> this happens, with the understanding that it would be a while if I've
>> successfully fixed the reason why my laptop was crashing too often.
>
> Sounds good.
>
>> But you said you've also seen issues with google-chrome profile and btrfs.
>> What did you experience?
>
> I never debugged it, but I had to blow away the profile a couple of times. Then
> there's this weird one which looks like a Btrfs bug:
>
> ┌[osandov@vader ~/.config]
> └$ ls -al google-chrome-busted/**
> ls: cannot access 'google-chrome-busted/Local State': No such file or directory
> google-chrome-busted/Default:
> ls: cannot access 'google-chrome-busted/Default/Preferences': No such file or directory
> ls: cannot access 'google-chrome-busted/Default/.com.google.Chrome.VfAUNx': No such file or directory
> total 0
> drwx------ 1 osandov users 12 Feb  7 16:50 .
> drwx------ 1 osandov users 14 Feb  7 16:50 ..
> -????????? ? ?       ?      ?            ? .com.google.Chrome.VfAUNx
> -????????? ? ?       ?      ?            ? Preferences
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Leveldb in google-chrome incompatible with btrfs?

[Marc MERLIN]

(removing pwnall at chromium.org to cut spam)

On Thu, Jul 06, 2017 at 10:46:08PM -0700, Omar Sandoval wrote:
> ┌[osandov@vader ~/.config]
> └$ ls -al google-chrome-busted/**
> ls: cannot access 'google-chrome-busted/Local State': No such file or directory
> google-chrome-busted/Default:
> ls: cannot access 'google-chrome-busted/Default/Preferences': No such file or directory
> ls: cannot access 'google-chrome-busted/Default/.com.google.Chrome.VfAUNx': No such file or directory
> total 0
> drwx------ 1 osandov users 12 Feb  7 16:50 .
> drwx------ 1 osandov users 14 Feb  7 16:50 ..
> -????????? ? ?       ?      ?            ? .com.google.Chrome.VfAUNx
> -????????? ? ?       ?      ?            ? Preferences

Yeah, that's definitely not chrome's fault :)

On Fri, Jul 07, 2017 at 06:05:39PM +0300, Cerem Cem ASLAN wrote:
> I was also struggling with this issue for quite some time. Today my 2
> months old disk is crashed (which really surprises me). It doesn't
> even being shown by `fdisk -l`.
> 
> After buying this this disk, I installed a Debian on a BTRFS partition
> on an LVM partition on a LUKS partition. From that day till today,
> while doing my work, there was always a video running on Youtube in
> the background. Sometimes my editor (atom, it also shares a lot with
> chromium) was slowing down which was making me end up restarting the
> Chromium (and it was restarting barely with `killall -9 chromium`.
> Another mentionable event is, 3 days ago my laptop suddenly restarted.

Sorry, I'm not quite making sense out of this.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Leveldb in google-chrome incompatible with btrfs?

[Cerem Cem ASLAN]

The day before I re-checked to see if I could recover any of my files
and it seems that disk is all alive. Nothing missed, nothing deleted.
I run `btrfs scrub`, there are only a few uncorrectable errors (they
were there before the event). I'm using the disk as before for two
days and there is nothing unusual. Quite interesting.

2017-07-07 19:50 GMT+03:00 Marc MERLIN <marc@merlins.org>:
> (removing pwnall at chromium.org to cut spam)
>
> On Thu, Jul 06, 2017 at 10:46:08PM -0700, Omar Sandoval wrote:
>> ┌[osandov@vader ~/.config]
>> └$ ls -al google-chrome-busted/**
>> ls: cannot access 'google-chrome-busted/Local State': No such file or directory
>> google-chrome-busted/Default:
>> ls: cannot access 'google-chrome-busted/Default/Preferences': No such file or directory
>> ls: cannot access 'google-chrome-busted/Default/.com.google.Chrome.VfAUNx': No such file or directory
>> total 0
>> drwx------ 1 osandov users 12 Feb  7 16:50 .
>> drwx------ 1 osandov users 14 Feb  7 16:50 ..
>> -????????? ? ?       ?      ?            ? .com.google.Chrome.VfAUNx
>> -????????? ? ?       ?      ?            ? Preferences
>
> Yeah, that's definitely not chrome's fault :)
>
> On Fri, Jul 07, 2017 at 06:05:39PM +0300, Cerem Cem ASLAN wrote:
>> I was also struggling with this issue for quite some time. Today my 2
>> months old disk is crashed (which really surprises me). It doesn't
>> even being shown by `fdisk -l`.
>>
>> After buying this this disk, I installed a Debian on a BTRFS partition
>> on an LVM partition on a LUKS partition. From that day till today,
>> while doing my work, there was always a video running on Youtube in
>> the background. Sometimes my editor (atom, it also shares a lot with
>> chromium) was slowing down which was making me end up restarting the
>> Chromium (and it was restarting barely with `killall -9 chromium`.
>> Another mentionable event is, 3 days ago my laptop suddenly restarted.
>
> Sorry, I'm not quite making sense out of this.
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/