postgresql activity causes crash, 2.6.36.1

postgresql activity causes crash, 2.6.36.1

[Brian Neu]

After a successful db startup, this crash was reproducible every time with db 
activity, or possibly just time.  The filesystem is a raid1 and even after a 
reformat and restore from backup, the crash remains:

Label: slash  uuid: a9730db3-a475-45bb-af94-a873e235114b
        Total devices 2 FS bytes used 39.04GB
        devid    2 size 97.66GB used 48.51GB path /dev/sdb2
        devid    1 size 97.66GB used 48.53GB path /dev/sda2

mounted as:
# mount |grep sdb2
/dev/sdb2 on / type btrfs (rw,noatime)



This is transcribed from the screen, so forgive type-o's and "<blah>"  .    Let 
me know if I skipped something important.
(every line starts with [27163.079010] )

Call trace:
__btrfs_submit_bio_done+0x1b/0x1d [btrfs]
run_one_async_done+0x94/0x98 [btrfs]
run_ordered_completions++0x78/0xaa [btrfs]
worker_loop+0x199/0x4ec [btrfs]
kthread+0x0d/0xa5
kernel_thread_helper+0x4/0x10
? restore_args+0x0/0x30
? kthread+0x0/0xa5
? kernel_thread_helper+0x0/0x10
Code: 74 04 <blah blah>
RIP [<ffff<blah> 7>] btrfs_map_bio+0x88/0x1c9
RSP <ffb<blah>>
---[ end trace f8<blah> ]---

Re: postgresql activity causes crash, 2.6.36.1

[Chris Mason]

Excerpts from Brian Neu's message of 2010-11-06 21:08:59 -0400:
> After a successful db startup, this crash was reproducible every time with db 
> activity, or possibly just time.  The filesystem is a raid1 and even after a 
> reformat and restore from backup, the crash remains:
> 
> Label: slash  uuid: a9730db3-a475-45bb-af94-a873e235114b
>         Total devices 2 FS bytes used 39.04GB
>         devid    2 size 97.66GB used 48.51GB path /dev/sdb2
>         devid    1 size 97.66GB used 48.53GB path /dev/sda2
> 
> mounted as:
> # mount |grep sdb2
> /dev/sdb2 on / type btrfs (rw,noatime)
> 
> 
> 
> This is transcribed from the screen, so forgive type-o's and "<blah>"  .    Let 
> me know if I skipped something important.

Ok, could you please take a picture of the crash?  We'll need all the
lines.

Which kernel are you on?

-chris

> (every line starts with [27163.079010] )
> 
> Call trace:
> __btrfs_submit_bio_done+0x1b/0x1d [btrfs]
> run_one_async_done+0x94/0x98 [btrfs]
> run_ordered_completions++0x78/0xaa [btrfs]
> worker_loop+0x199/0x4ec [btrfs]
> kthread+0x0d/0xa5
> kernel_thread_helper+0x4/0x10
> ? restore_args+0x0/0x30
> ? kthread+0x0/0xa5
> ? kernel_thread_helper+0x0/0x10
> Code: 74 04 <blah blah>
> RIP [<ffff<blah> 7>] btrfs_map_bio+0x88/0x1c9
> RSP <ffb<blah>>
> ---[ end trace f8<blah> ]---
> 

Re: postgresql activity causes crash, 2.6.36.1

[Chris Mason]

Excerpts from Brian Neu's message of 2010-11-08 12:52:21 -0500:
> 
> > Ok, could you please take a picture of the crash?  We'll need all the lines.
> I really hope you mean "picture" as in photo of the screen, and I'm not 
> committing an egregious list faux pas by attaching this.

No, this is perfect ;) The only problem is that we're missing the stuff
at the very top of the oops.  That's the part that explains why it
oopsed.  The stuff in this png is great, but it only shows where it
oopsed.

-chris

Re: postgresql activity causes crash, 2.6.36.1

[Chris Mason]

Great, this is a known bug in the O_DIRECT code.  I'll fix it up
tomorrow morning.

The problem is that we're allowing bios that are too large for btrfs to
deal with.  It doesn't show any corruption in the FS.

-chris

Excerpts from Brian Neu's message of 2010-11-08 17:50:17 -0500:
> > No, this is perfect ;) The only problem is that we're missing the  stuff
> 
> > at the very top of the oops.  That's the part that explains why  it
> > oopsed.  The stuff in this png is great, but it only shows where  it
> > oopsed.
> > 
> > -chris
> 
> 
> I changed the kernel line in grub.conf for 1280x1024, rebooted, and  
> orchestrated another crash for you.  This time I noted that db activity  was 
> necessary to cause the crash.
> 
> not my steadiest camera work, but still legible

Re: postgresql activity causes crash, 2.6.36.1

[Brian Neu]

----- Original Message ----

> From: Chris Mason <chris.mason@oracle.com>

> Great, this is a known bug in the O_DIRECT code.  I'll fix it  up
> tomorrow morning.


At the risk of sounding like the non-contributing, annoying moocher that I am, I 
thought I'd check on the status of this bug.

Re: postgresql activity causes crash, 2.6.36.1

[Chris Mason]

Excerpts from Brian Neu's message of 2010-11-11 17:05:13 -0500:
> ----- Original Message ----
> 
> > From: Chris Mason <chris.mason@oracle.com>
> 
> > Great, this is a known bug in the O_DIRECT code.  I'll fix it  up
> > tomorrow morning.
> 
> 
> At the risk of sounding like the non-contributing, annoying moocher that I am, I 
> thought I'd check on the status of this bug.

Not at all, sorry for the delay I got caught up in hunting for the
transparent hugepages bug (which wasn't in transparent hugepages in the
end).

I'm working on the O_DIRECT stuff tonight.

-chris