BUG while writing to USB btrfs filesystem

BUG while writing to USB btrfs filesystem

[cwillu]

While untar'ing an image to an sd card via a reader, I got the
following bug.  The system also has a btrfs root, and a whole swath of
processes went into uninterruptable sleep.  I was able to poke around
via ssh and sysrq, and already had netconsole set up to capture the
bug.

Root fs is on /dev/sdi1, and /dev/sdj2 is the card reader which was
the target of the untar.

[29571.448889] sd 11:0:0:0: [sdj] Assuming drive cache: write through
[29571.451153]  sdj: unknown partition table
[29572.047390] sd 11:0:0:0: [sdj] Assuming drive cache: write through
[29572.048149]  sdj: sdj1 sdj2
[29602.642175] device fsid 5648f85292725e76-72190c37da5211b6 devid 1
transid 7 /dev/sdj2
[29602.643256] btrfs: use spread ssd allocation scheme
[36852.550219] sd 11:0:0:0: [sdj] Assuming drive cache: write through
[36852.552958]  sdj: unknown partition table
[36852.571770] ------------[ cut here ]------------
[36852.580008] kernel BUG at fs/btrfs/inode.c:1629!
[36852.580008] invalid opcode: 0000 [#1] SMP [36852.580008] invalid
opcode: 0000 [#1] SMP
[36852.580008] last sysfs file:
/sys/devices/pci0000:00/0000:00:1a.7/usb1/1-2/1-2:1.0/uevent
[36852.580008] CPU 2
[36852.580008] Modules linked in: netconsole configfs nls_cp437 vfat
fat binfmt_misc ppdev configfs ipt_MASQUERADE iptable_nat
nls_iso8859_1 nf_nat nf_conntrack_ipv4 nls_cp437 nf_defrag_ipv4
ipt_REJECT aes_generic snd_rawmidi snd_rawmidi snd_seq_midi_event
snd_seq snd_timer snd_seq_device asus_atk0110 lp snd soundcore
snd_page_alloc shpchp parport raid10 raid456 snd_page_alloc async_pq
async_xor xor async_memcpy shpchp async_raid6_recov parport raid6_pq
async_tx raid10 raid1 raid0 multipath linear raid456 btrfs
zlib_deflate async_pq async_xor xor async_memcpy async_raid6_recov
raid6_pq async_tx raid1 raid0 multipath linear btrfs zlib_deflate
lzo_compress crc32c libcrc32c radeon ttm lzo_compress ohci1394
drm_kms_helper crc32c usb_storage ieee1394 r8169 libcrc32c drm
i2c_algo_bit mii radeon pata_jmicron ahci libahci intel_agp ttm
ohci1394 drm_kms_helper usb_storage ieee1394 r8169 drm i2c_algo_bit
mii pata_jmicron ahci libahci intel_agp

[36852.580008]
[36852.580008] Pid: 437, comm: btrfs-endio-wri Not tainted
2.6.36-btrfs2 #1 P5Q3/System Product Name
[36852.580008] RIP: 0010:[<ffffffffa0224120>] [<ffffffffa0224120>]
T.1224+0x220/0x230 [btrfs]
[36852.580008] RSP: 0000:ffff880231c7dc30  EFLAGS: 00010286
[36852.580008] RAX: 00000000ffffffe4 RBX: ffff88023363a7e0 RCX: ffff88023363ad80
[36852.580008] RDX: 0000000000000000 RSI: ffffea0007b3dcb0 RDI: 0000000000000206
[36852.580008] RBP: ffff880231c7dcc0 R08: 0000000000000005 R09: 00000000ffffffe4
[36852.940017] R10: 0000000000000000 R11: 6db6db6db6db6db7 R12: ffff88022ec8ea80
[36852.940017] R13: 000000000e19a000 R14: ffff88023173c000 R15: ffff8802335fe900
[36852.940017] FS:  0000000000000000(0000) GS:ffff880001f00000(0000)
knlGS:0000000000000000
[36852.940017] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[36852.940017] CR2: 00000000017ed2d8 CR3: 00000001d42ac000 CR4: 00000000000006e0
[36852.940017] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[36852.940017] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[36852.940017] Process btrfs-endio-wri (pid: 437, threadinfo
ffff880231c7c000, task ffff880232738000)
[36852.940017] Stack:
[36852.940017]  000000000e19a000 0000000000001000 000000494189e000
0000000000001000
[36852.940017] <0> ffff880231c7dc70 ffffffff81582fde ffff8802335fe900
ffff88023173c000
[36852.940017] <0> ffff880231c7dcb0 ffffffffa0220e1c 000000000e19a000
ffff88022ec8ea80

[36853.210013] Call Trace:
[36853.210013]  [<ffffffff81582fde>] ? mutex_lock+0x1e/0x50
[36853.210013]  [<ffffffffa0220e1c>] ? start_transaction+0x1cc/0x270 [btrfs]
[36853.210013]  [<ffffffffa022495b>] btrfs_finish_ordered_io+0x2eb/0x330 [btrfs]
[36853.210013]  [<ffffffff81103157>] ? mempool_free_slab+0x17/0x20
[36853.210013]  [<ffffffffa02249ba>]
btrfs_writepage_end_io_hook+0x1a/0x20 [btrfs]
[36853.210013]  [<ffffffffa024217b>]
end_bio_extent_writepage+0x13b/0x180 [btrfs]
[36853.210013]  [<ffffffff815828bb>] ? schedule_timeout+0x19b/0x300
[36853.210013]  [<ffffffff8118212d>] bio_endio+0x1d/0x40
[36853.210013]  [<ffffffffa021a1b4>] end_workqueue_fn+0xf4/0x130 [btrfs]
[36853.210013]  [<ffffffffa024d1cc>] worker_loop+0x15c/0x640 [btrfs]
[36853.210013]  [<ffffffffa024d070>] ? worker_loop+0x0/0x640 [btrfs]
[36853.210013]  [<ffffffff81080bf6>] kthread+0x96/0xa0
[36853.210013]  [<ffffffff8100bea4>] kernel_thread_helper+0x4/0x10
[36853.210013]  [<ffffffff81080b60>] ? kthread+0x0/0xa0
[36853.210013]  [<ffffffff8100bea0>] ? kernel_thread_helper+0x0/0x10
[36853.210013] Code: 68 31 c0 5b 41 5c 41 5d 41 5e 41 5f c9 c3 0f 0b
eb fe 66 0f 1f 84 00 00 00 00 00 0f 0b eb fe 0f 1f 40 00 0f 0b eb fe
0f 1f 40 00 <0f> 0b eb fe 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89
e5 41
[36853.210013] RIP  [<ffffffffa0224120>] T.1224+0x220/0x230 [btrfs]
[36853.690010]  RSP <ffff880231c7dc30>
[36853.877226] ---[ end trace 101a91b31ae9f3ff ]---

Re: BUG while writing to USB btrfs filesystem

[cwillu]

On Tue, Jan 18, 2011 at 4:28 PM, cwillu <cwillu@cwillu.com> wrote:
> While untar'ing an image to an sd card via a reader, I got the
> following bug. =A0The system also has a btrfs root, and a whole swath=
 of
> processes went into uninterruptable sleep. =A0I was able to poke arou=
nd
> via ssh and sysrq, and already had netconsole set up to capture the
> bug.
>
> Root fs is on /dev/sdi1, and /dev/sdj2 is the card reader which was
> the target of the untar.
>
> [29571.448889] sd 11:0:0:0: [sdj] Assuming drive cache: write through
> [29571.451153] =A0sdj: unknown partition table
> [29572.047390] sd 11:0:0:0: [sdj] Assuming drive cache: write through
> [29572.048149] =A0sdj: sdj1 sdj2
> [29602.642175] device fsid 5648f85292725e76-72190c37da5211b6 devid 1
> transid 7 /dev/sdj2
> [29602.643256] btrfs: use spread ssd allocation scheme
> [36852.550219] sd 11:0:0:0: [sdj] Assuming drive cache: write through
> [36852.552958] =A0sdj: unknown partition table
> [36852.571770] ------------[ cut here ]------------
> [36852.580008] kernel BUG at fs/btrfs/inode.c:1629!
> [36852.580008] invalid opcode: 0000 [#1] SMP [36852.580008] invalid
> opcode: 0000 [#1] SMP
> [36852.580008] last sysfs file:
> /sys/devices/pci0000:00/0000:00:1a.7/usb1/1-2/1-2:1.0/uevent
> [36852.580008] CPU 2
> [36852.580008] Modules linked in: netconsole configfs nls_cp437 vfat
> fat binfmt_misc ppdev configfs ipt_MASQUERADE iptable_nat
> nls_iso8859_1 nf_nat nf_conntrack_ipv4 nls_cp437 nf_defrag_ipv4
> ipt_REJECT aes_generic snd_rawmidi snd_rawmidi snd_seq_midi_event
> snd_seq snd_timer snd_seq_device asus_atk0110 lp snd soundcore
> snd_page_alloc shpchp parport raid10 raid456 snd_page_alloc async_pq
> async_xor xor async_memcpy shpchp async_raid6_recov parport raid6_pq
> async_tx raid10 raid1 raid0 multipath linear raid456 btrfs
> zlib_deflate async_pq async_xor xor async_memcpy async_raid6_recov
> raid6_pq async_tx raid1 raid0 multipath linear btrfs zlib_deflate
> lzo_compress crc32c libcrc32c radeon ttm lzo_compress ohci1394
> drm_kms_helper crc32c usb_storage ieee1394 r8169 libcrc32c drm
> i2c_algo_bit mii radeon pata_jmicron ahci libahci intel_agp ttm
> ohci1394 drm_kms_helper usb_storage ieee1394 r8169 drm i2c_algo_bit
> mii pata_jmicron ahci libahci intel_agp
>
> [36852.580008]
> [36852.580008] Pid: 437, comm: btrfs-endio-wri Not tainted
> 2.6.36-btrfs2 #1 P5Q3/System Product Name
> [36852.580008] RIP: 0010:[<ffffffffa0224120>] [<ffffffffa0224120>]
> T.1224+0x220/0x230 [btrfs]
> [36852.580008] RSP: 0000:ffff880231c7dc30 =A0EFLAGS: 00010286
> [36852.580008] RAX: 00000000ffffffe4 RBX: ffff88023363a7e0 RCX: ffff8=
8023363ad80
> [36852.580008] RDX: 0000000000000000 RSI: ffffea0007b3dcb0 RDI: 00000=
00000000206
> [36852.580008] RBP: ffff880231c7dcc0 R08: 0000000000000005 R09: 00000=
000ffffffe4
> [36852.940017] R10: 0000000000000000 R11: 6db6db6db6db6db7 R12: ffff8=
8022ec8ea80
> [36852.940017] R13: 000000000e19a000 R14: ffff88023173c000 R15: ffff8=
802335fe900
> [36852.940017] FS: =A00000000000000000(0000) GS:ffff880001f00000(0000=
)
> knlGS:0000000000000000
> [36852.940017] CS: =A00010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [36852.940017] CR2: 00000000017ed2d8 CR3: 00000001d42ac000 CR4: 00000=
000000006e0
> [36852.940017] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 00000=
00000000000
> [36852.940017] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 00000=
00000000400
> [36852.940017] Process btrfs-endio-wri (pid: 437, threadinfo
> ffff880231c7c000, task ffff880232738000)
> [36852.940017] Stack:
> [36852.940017] =A0000000000e19a000 0000000000001000 000000494189e000
> 0000000000001000
> [36852.940017] <0> ffff880231c7dc70 ffffffff81582fde ffff8802335fe900
> ffff88023173c000
> [36852.940017] <0> ffff880231c7dcb0 ffffffffa0220e1c 000000000e19a000
> ffff88022ec8ea80
>
> [36853.210013] Call Trace:
> [36853.210013] =A0[<ffffffff81582fde>] ? mutex_lock+0x1e/0x50
> [36853.210013] =A0[<ffffffffa0220e1c>] ? start_transaction+0x1cc/0x27=
0 [btrfs]
> [36853.210013] =A0[<ffffffffa022495b>] btrfs_finish_ordered_io+0x2eb/=
0x330 [btrfs]
> [36853.210013] =A0[<ffffffff81103157>] ? mempool_free_slab+0x17/0x20
> [36853.210013] =A0[<ffffffffa02249ba>]
> btrfs_writepage_end_io_hook+0x1a/0x20 [btrfs]
> [36853.210013] =A0[<ffffffffa024217b>]
> end_bio_extent_writepage+0x13b/0x180 [btrfs]
> [36853.210013] =A0[<ffffffff815828bb>] ? schedule_timeout+0x19b/0x300
> [36853.210013] =A0[<ffffffff8118212d>] bio_endio+0x1d/0x40
> [36853.210013] =A0[<ffffffffa021a1b4>] end_workqueue_fn+0xf4/0x130 [b=
trfs]
> [36853.210013] =A0[<ffffffffa024d1cc>] worker_loop+0x15c/0x640 [btrfs=
]
> [36853.210013] =A0[<ffffffffa024d070>] ? worker_loop+0x0/0x640 [btrfs=
]
> [36853.210013] =A0[<ffffffff81080bf6>] kthread+0x96/0xa0
> [36853.210013] =A0[<ffffffff8100bea4>] kernel_thread_helper+0x4/0x10
> [36853.210013] =A0[<ffffffff81080b60>] ? kthread+0x0/0xa0
> [36853.210013] =A0[<ffffffff8100bea0>] ? kernel_thread_helper+0x0/0x1=
0
> [36853.210013] Code: 68 31 c0 5b 41 5c 41 5d 41 5e 41 5f c9 c3 0f 0b
> eb fe 66 0f 1f 84 00 00 00 00 00 0f 0b eb fe 0f 1f 40 00 0f 0b eb fe
> 0f 1f 40 00 <0f> 0b eb fe 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 8=
9
> e5 41
> [36853.210013] RIP =A0[<ffffffffa0224120>] T.1224+0x220/0x230 [btrfs]
> [36853.690010] =A0RSP <ffff880231c7dc30>
> [36853.877226] ---[ end trace 101a91b31ae9f3ff ]---
>

Grumble grumble...  this is a 2.6.36 kernel built from cmason's
for-linus branch last night, commit: acce95
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BUG while writing to USB btrfs filesystem

[cwillu]

And again, without the USB devices.

This time was under a vanilla 2.6.38rc2 build, rootfs is a 4-drive
btrfs raid10 connected internally via sata, and no other drives in the
picture.

The btrfs drives are sda5, sdb1, sdc1 and sdd1;  sda1 is /boot, sda2
is swap, sda3 is a 4gb ext4 with a small install for recovery
purposes.

[33159.480523] ------------[ cut here ]------------
[33159.490003] kernel BUG at /home/kernel-ppa/COD/linux/fs/btrfs/inode.c:1629!
[33159.490003] invalid opcode: 0000 [#1] SMP
[33159.490003] last sysfs file:
/sys/devices/pci0000:00/0000:00:1f.2/host2/target2:0:0/2:0:0:0/block/sda/sda3/alignment_offset
[33159.490003] CPU 2
[33159.490003] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs vfat
msdos fat jfs xfs exportfs reiserfs binfmt_misc ppdev ipt_MASQUERADE
iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables
bridge stp snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep
snd_pcm snd_seq_midi aes_x86_64 snd_rawmidi snd_seq_midi_event
aes_generic snd_seq snd_timer snd_seq_device dm_crypt snd asus_atk0110
soundcore snd_page_alloc lp parport raid10 raid456 async_raid6_recov
async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 raid0
multipath linear btrfs zlib_deflate libcrc32c radeon ttm
drm_kms_helper drm usb_storage usbhid hid uas ahci r8169 i2c_algo_bit
pata_jmicron libahci
[33159.490003]
[33159.490003] Pid: 32327, comm: btrfs-endio-wri Not tainted
2.6.38-020638rc2-generic #201101220905 P5Q3/System Product Name
[33159.490003] RIP: 0010:[<ffffffffa020cab0>]  [<ffffffffa020cab0>]
insert_reserved_file_extent+0x230/0x240 [btrfs]
[33159.490003] RSP: 0000:ffff8801ba199c90  EFLAGS: 00010286
[33159.490003] RAX: 00000000ffffffe4 RBX: ffff880212f51ef0 RCX: ffffffffa01e604f
[33159.490003] RDX: ffff88022c823870 RSI: ffffea00079bc7a8 RDI: 0000000000000206
[33159.490003] RBP: ffff8801ba199d20 R08: ffff8800bfd17cd0 R09: 0000000000000000
[33159.490003] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880212f52058
[33159.490003] R13: ffff88022c823ab0 R14: ffff880212f52058 R15: 0000000000003000
[33159.490003] FS:  0000000000000000(0000) GS:ffff8800bfd00000(0000)
knlGS:0000000000000000
[33159.490003] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[33159.490003] CR2: 00007f32c49bad60 CR3: 00000001ba25a000 CR4: 00000000000006e0
[33159.490003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[33159.490003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[33159.490003] Process btrfs-endio-wri (pid: 32327, threadinfo
ffff8801ba198000, task ffff8802145d5b00)
[33159.490003] Stack:
[33159.490003]  00000000ba199cd0 0000000000001000 0000000000001000
000000266f846000
[33159.490003]  ffff88022cbc5540 ffff88022a7b6800 ffff8801fd8923c0
ffff88022a7b6800
[33159.490003]  ffff8801ba199d10 ffffffffa0207f82 0000000000000000
ffff880212f51ef0
[33159.490003] Call Trace:
[33159.490003]  [<ffffffffa0207f82>] ? start_transaction+0x1c2/0x230 [btrfs]
[33159.490003]  [<ffffffffa020d2b9>] btrfs_finish_ordered_io+0x2b9/0x320 [btrfs]
[33159.490003]  [<ffffffffa020d4ea>]
btrfs_writepage_end_io_hook+0x1a/0x20 [btrfs]
[33159.490003]  [<ffffffffa0225f51>] end_bio_extent_writepage+0x51/0x180 [btrfs]
[33159.490003]  [<ffffffff8118cf2d>] bio_endio+0x1d/0x40
[33159.490003]  [<ffffffffa0201998>] end_workqueue_fn+0x68/0xa0 [btrfs]
[33159.490003]  [<ffffffffa023226c>] worker_loop+0xac/0x260 [btrfs]
[33159.490003]  [<ffffffffa02321c0>] ? worker_loop+0x0/0x260 [btrfs]
[33159.490003]  [<ffffffff81085147>] kthread+0x97/0xa0
[33159.490003]  [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
[33159.490003]  [<ffffffff810850b0>] ? kthread+0x0/0xa0
[33159.490003]  [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
[33159.490003] Code: 89 ef e8 94 95 fd ff 48 83 c4 68 31 c0 5b 41 5c
41 5d 41 5e 41 5f c9 c3 0f 0b eb fe 0f 0b eb fe 0f 0b eb fe 0f 1f 80
00 00 00 00 <0f> 0b eb fe 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89
e5 41
[33159.490003] RIP  [<ffffffffa020cab0>]
insert_reserved_file_extent+0x230/0x240 [btrfs]
[33159.490003]  RSP <ffff8801ba199c90>
[33160.625221] ---[ end trace fefff6d61e4f8a7d ]---

Re: BUG while writing to USB btrfs filesystem

[Chris Samuel]

On 31/01/11 12:33, cwillu wrote:

> [33159.490003] kernel BUG at /home/kernel-ppa/COD/linux/fs/btrfs/inode.c:1629!

It looks like this happens when btrfs_drop_extents() returns
an error, and in the code the only time that seems to happen
without already calling BUG_ON() is if it runs out of memory
at the very start:

        path = btrfs_alloc_path();
        if (!path)
                return -ENOMEM;

Is there anything in the log from before that oops ?

NB: I just did a quick scan of btrfs_drop_extents() so it's
possible I missed something..

cheers,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

Re: BUG while writing to USB btrfs filesystem

[cwillu]

On Sun, Jan 30, 2011 at 8:57 PM, Chris Samuel <chris@csamuel.org> wrote=
:
> On 31/01/11 12:33, cwillu wrote:
>
>> [33159.490003] kernel BUG at /home/kernel-ppa/COD/linux/fs/btrfs/ino=
de.c:1629!
>
> It looks like this happens when btrfs_drop_extents() returns
> an error, and in the code the only time that seems to happen
> without already calling BUG_ON() is if it runs out of memory
> at the very start:
>
> =A0 =A0 =A0 =A0path =3D btrfs_alloc_path();
> =A0 =A0 =A0 =A0if (!path)
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return -ENOMEM;
>
> Is there anything in the log from before that oops ?
>
> NB: I just did a quick scan of btrfs_drop_extents() so it's
> possible I missed something..

There's a bunch of places it could be (any ret =3D foo followed by a
break will exit with that ret rather than oops).

Possibly relevant is that this generally occurs while building a
rootfs in a tmpfs chroot, which gets fairly sizeable (~2gb, on a
machine with 8gb ram).  The previous trace I posted occurred after the
tmpfs was tarred and unmounted, while the tarball was being written
out to a usb card reader, which seems like an odd time for an overly
huge tmpfs to cause trouble.

I've adjusted the job to not use the tmpfs for now, so the chroot is
on btrfs (combined with libeatmydata in order to continue to build
images in 40 minutes rather than 4 hours; dpkg).  We'll see if that
makes a difference.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BUG while writing to USB btrfs filesystem

[Chris Samuel]

On 31/01/11 14:11, cwillu wrote:

> There's a bunch of places it could be (any ret = foo
> followed by a break will exit with that ret rather
> than oops).

Ahh, thanks for that, I plead being a sysadmin rather than
a programmer.. ;-)

-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

Re: BUG while writing to USB btrfs filesystem

[cwillu]

Another one, with the process running on a btrfs chroot instead of tmpfs:


[12350.410412] ------------[ cut here ]------------
[12350.420001] kernel BUG at /home/kernel-ppa/COD/linux/fs/btrfs/inode.c:1759!
[12350.420001] invalid opcode: 0000 [#1] SMP
[12350.420001] last sysfs file:
/sys/devices/pci0000:00/0000:00:1f.2/host2/target2:0:0/2:0:0:0/block/sda/sda3/alignment_offset
[12350.420001] CPU 3
[12350.420001] Modules linked in: binfmt_misc ppdev ipt_MASQUERADE
iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables
bridge stp aes_x86_64 snd_hda_codec_realtek aes_generic dm_crypt
snd_hda_intel snd_hda_codec lp snd_hwdep snd_pcm snd_seq_midi
snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device
asus_atk0110 snd soundcore snd_page_alloc parport raid10 raid456
async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy
async_tx raid1 raid0 multipath linear btrfs zlib_deflate libcrc32c
radeon ttm drm_kms_helper drm usbhid usb_storage hid uas ahci
i2c_algo_bit r8169 libahci pata_jmicron
[12350.420001]
[12350.420001] Pid: 8030, comm: btrfs-endio-wri Not tainted
2.6.38-020638rc2-generic #201101220905 P5Q3/System Product Name
[12350.420001] RIP: 0010:[<ffffffffa021a22d>]  [<ffffffffa021a22d>]
btrfs_finish_ordered_io+0x22d/0x320 [btrfs]
[12350.420001] RSP: 0000:ffff8801fc841d30  EFLAGS: 00010286
[12350.420001] RAX: 00000000ffffffe4 RBX: ffff8801fd4ec030 RCX: ffffffffa01f304f
[12350.420001] RDX: ffff88022ca37c60 RSI: ffffea00079c3c08 RDI: 0000000000000206
[12350.420001] RBP: ffff8801fc841da0 R08: ffff8800bfd97cd0 R09: ffff8801fc841b50
[12350.420001] R10: 0000000000000001 R11: 0000000000000024 R12: ffff8801fd4ec198
[12350.420001] R13: ffff88022b15d800 R14: 0000000000000000 R15: ffff880227c46b00
[12350.420001] FS:  0000000000000000(0000) GS:ffff8800bfd80000(0000)
knlGS:0000000000000000
[12350.420001] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[12350.420001] CR2: 00007f4b565810c8 CR3: 00000001fbdb5000 CR4: 00000000000006e0
[12350.420001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[12350.420001] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[12350.420001] Process btrfs-endio-wri (pid: 8030, threadinfo
ffff8801fc840000, task ffff8802007dc440)
[12350.420001] Stack:
[12350.420001]  0000000000001000 ffffffff00000000 ffffffff00000000
ffff880200000000
[12350.420001]  0000000000000001 ffff8801fc841db0 0000000000570000
0000000000000000
[12350.420001]  ffff880204f13600 ffffea000541e840 ffff8800a85eb7f0
000000000001e000
[12350.420001] Call Trace:
[12350.420001]  [<ffffffffa021a4ea>]
btrfs_writepage_end_io_hook+0x1a/0x20 [btrfs]
[12350.420001]  [<ffffffffa0232f51>] end_bio_extent_writepage+0x51/0x180 [btrfs]
[12350.420001]  [<ffffffff8118cf2d>] bio_endio+0x1d/0x40
[12350.420001]  [<ffffffffa020e998>] end_workqueue_fn+0x68/0xa0 [btrfs]
[12350.420001]  [<ffffffffa023f26c>] worker_loop+0xac/0x260 [btrfs]
[12350.420001]  [<ffffffffa023f1c0>] ? worker_loop+0x0/0x260 [btrfs]
[12350.420001]  [<ffffffff81085147>] kthread+0x97/0xa0
[12350.420001]  [<ffffffff8100ce24>] kernel_thread_helper+0x4/0x10
[12350.420001]  [<ffffffff810850b0>] ? kthread+0x0/0xa0
[12350.420001]  [<ffffffff8100ce20>] ? kernel_thread_helper+0x0/0x10
[12350.420001] Code: 48 40 e8 b7 cf ff ff 48 8b 55 d0 31 f6 4c 89 e7
e8 b9 67 01 00 4c 89 e2 4c 89 ee 4c 89 ff e8 bb ea ff ff 85 c0 0f 84
d5 fe ff ff <0f> 0b eb fe be 01 00 00 00 4c 89 ef e8 d2 ad ff ff 49 89
c7 e9
[12350.420001] RIP  [<ffffffffa021a22d>]
btrfs_finish_ordered_io+0x22d/0x320 [btrfs]
[12350.420001]  RSP <ffff8801fc841d30>
[12351.594605] ---[ end trace f06f9985fbc265cd ]---