* mkfs crash
@ 2012-12-27 12:05 Pim van den Berg
[not found] ` <50DC3977.2080209-IXGSG4U2CCrz+pZb47iToQ@public.gmane.org>
0 siblings, 1 reply; 4+ messages in thread
From: Pim van den Berg @ 2012-12-27 12:05 UTC (permalink / raw)
To: linux-bcache-u79uwXL29TY76Z2rM5mHXA
Hi,
I'm successfully using bcache on 1 partition of my system for a while
now. The SSD is split in 2 partitions to be able to enable bcache on
another partition too.
The first partition is setup like this:
/dev/md3 (mdadm, RAID1, bcache backing device)
- /dev/bcache0
I tried to setup the second partition like this:
/dev/md4 (mdadm, RAID1)
- /dev/dm-0 (luks, bcache backing device)
- /dev/bcache3
All goes well until I try to create an ext4 (or ext3) filesystem on it.
The mkfs command crashes and couple of 1000 lines show up in my syslog
(the full log is over here:
http://pommi.nethuis.nl/storage/software/bcache/log/mkfs-crash.log):
[1631972.332656] bcache: Caching dm-0 as bcache3 on set
6bc79688-a6e0-4c21-8f44-59b0083b8169
[1631983.772362] CPU 0
[1631983.772380] Pid: 28707, comm: mkfs.ext4 Not tainted 3.2.33-kvm
#1 /DH67CF
[1631983.772420] RIP: 0010:[<ffffffff813f91ca>] [<ffffffff813f91ca>]
bch_mark_sectors_bypassed+0x1a/0x35
[1631983.772465] RSP: 0018:ffff880165cadbf8 EFLAGS: 00000a06
[1631983.772486] RAX: ffff8801128e0010 RBX: ffff880165fd0318 RCX:
ffff880165cadc30
[1631983.772521] RDX: 2000000000000000 RSI: 00000000007fffff RDI:
ffff880165fd0318
[1631983.772556] RBP: ffff880165cadbf8 R08: ffff88021e802be0 R09:
00000000ffffff02
[1631983.772592] R10: 00000000ffffff01 R11: ffff880165cadc78 R12:
ffff8801128e0000
[1631983.772627] R13: ffff880118450000 R14: ffff880165cadc68 R15:
0000000000000000
[1631983.772662] FS: 000067d4ad65e760(0000) GS:ffff88021fa00000(0000)
knlGS:0000000000000000
[1631983.772699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1631983.772722] CR2: 0000000001958908 CR3: 0000000163c68000 CR4:
00000000000406f0
[1631983.772757] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[1631983.772792] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[1631983.772827] Process mkfs.ext4 (pid: 28707, threadinfo
ffff880133edacf0, task ffff880133eda800)
[1631983.772880] ffff880165cadc48 ffffffff813f08dc 0000001065cadc48
ffff880163fe0000
[1631983.772919] ffff8801128e0000 ffff880165fd0318 ffff8801128e0000
ffff880165fd0378
[1631983.772957] ffff880165cadc68 ffff880165cadc58 ffff880165cadca8
ffffffff813f1ae0
[1631983.773015] [<ffffffff813f08dc>] check_should_skip+0x31f/0x335
[1631983.773039] [<ffffffff813f1ae0>] request_write+0x7d/0x267
[1631983.773061] [<ffffffff813f1dc8>] cached_dev_make_request+0xfe/0x1ad
[1631983.773087] [<ffffffff8127edff>] generic_make_request+0x17c/0x1d2
[1631983.773110] [<ffffffff8127ef25>] submit_bio+0xd0/0xdb
[1631983.773133] [<ffffffff81284a3d>] blkdev_issue_discard+0x158/0x1a7
[1631983.773156] [<ffffffff812850bb>] blkdev_ioctl+0x2f7/0x69c
[1631983.773180] [<ffffffff811191f8>] block_ioctl+0x32/0x36
[1631983.773203] [<ffffffff810fe7e2>] do_vfs_ioctl+0x5aa/0x5fa
[1631983.773226] [<ffffffff810fe874>] sys_ioctl+0x42/0x65
[1631983.773250] [<ffffffff815657b6>] system_call_fastpath+0x18/0x1d
[1631983.773384] Call Trace:
[1631983.773401] [<ffffffff813f08dc>] check_should_skip+0x31f/0x335
[1631983.773424] [<ffffffff813f1ae0>] request_write+0x7d/0x267
[1631983.773447] [<ffffffff813f1dc8>] cached_dev_make_request+0xfe/0x1ad
[1631983.773470] [<ffffffff8127edff>] generic_make_request+0x17c/0x1d2
[1631983.773499] [<ffffffff8127ef25>] submit_bio+0xd0/0xdb
[1631983.773520] [<ffffffff81284a3d>] blkdev_issue_discard+0x158/0x1a7
[1631983.773544] [<ffffffff812850bb>] blkdev_ioctl+0x2f7/0x69c
[1631983.773567] [<ffffffff811191f8>] block_ioctl+0x32/0x36
[1631983.773590] [<ffffffff810fe7e2>] do_vfs_ioctl+0x5aa/0x5fa
[1631983.773613] [<ffffffff810fe874>] sys_ioctl+0x42/0x65
[1631983.773635] [<ffffffff815657b6>] system_call_fastpath+0x18/0x1d
I'm using the 3.2.33 Linux kernel with
grsecurity-2.9.1-3.2.33-201211072000 and bcache v3.2.28-384-gcafb412.
I've tried to set it up this way multiple times but I hit the same
problem each time. Because I'm successfully running bcache on a mdadm
device, I thought there was an issue with the luks part. So I tested
this part with a USB thumb drive as a backing device:
/dev/sdd
- /dev/dm-0 (luks, bcache backing device)
- /dev/bcache1
In this case the bcache device worked without a problem. As you can see
in the stacktrace, bch_mark_sectors_bypassed (a piece of bcache code)
causes the crash. Do you know what is going wrong here?
--
Regards,
Pim
^ permalink raw reply [flat|nested] 4+ messages in thread[parent not found: <50DC3977.2080209-IXGSG4U2CCrz+pZb47iToQ@public.gmane.org>]
* Re: mkfs crash [not found] ` <50DC3977.2080209-IXGSG4U2CCrz+pZb47iToQ@public.gmane.org> @ 2012-12-28 4:31 ` Kent Overstreet [not found] ` <20121228043144.GC10411-jC9Py7bek1znysI04z7BkA@public.gmane.org> 0 siblings, 1 reply; 4+ messages in thread From: Kent Overstreet @ 2012-12-28 4:31 UTC (permalink / raw) To: Pim van den Berg; +Cc: linux-bcache-u79uwXL29TY76Z2rM5mHXA On Thu, Dec 27, 2012 at 01:05:11PM +0100, Pim van den Berg wrote: > Hi, > > I'm successfully using bcache on 1 partition of my system for a while > now. The SSD is split in 2 partitions to be able to enable bcache on > another partition too. > > The first partition is setup like this: > /dev/md3 (mdadm, RAID1, bcache backing device) > - /dev/bcache0 > > I tried to setup the second partition like this: > /dev/md4 (mdadm, RAID1) > - /dev/dm-0 (luks, bcache backing device) > - /dev/bcache3 > > All goes well until I try to create an ext4 (or ext3) filesystem on it. > The mkfs command crashes and couple of 1000 lines show up in my syslog > (the full log is over here: > http://pommi.nethuis.nl/storage/software/bcache/log/mkfs-crash.log): > > [1631972.332656] bcache: Caching dm-0 as bcache3 on set > 6bc79688-a6e0-4c21-8f44-59b0083b8169 > [1631983.772362] CPU 0 > [1631983.772380] Pid: 28707, comm: mkfs.ext4 Not tainted 3.2.33-kvm > #1 /DH67CF > [1631983.772420] RIP: 0010:[<ffffffff813f91ca>] [<ffffffff813f91ca>] > bch_mark_sectors_bypassed+0x1a/0x35 > [1631983.772465] RSP: 0018:ffff880165cadbf8 EFLAGS: 00000a06 > [1631983.772486] RAX: ffff8801128e0010 RBX: ffff880165fd0318 RCX: > ffff880165cadc30 > [1631983.772521] RDX: 2000000000000000 RSI: 00000000007fffff RDI: > ffff880165fd0318 > [1631983.772556] RBP: ffff880165cadbf8 R08: ffff88021e802be0 R09: > 00000000ffffff02 > [1631983.772592] R10: 00000000ffffff01 R11: ffff880165cadc78 R12: > ffff8801128e0000 > [1631983.772627] R13: ffff880118450000 R14: ffff880165cadc68 R15: > 0000000000000000 > [1631983.772662] FS: 000067d4ad65e760(0000) GS:ffff88021fa00000(0000) > knlGS:0000000000000000 > [1631983.772699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [1631983.772722] CR2: 0000000001958908 CR3: 0000000163c68000 CR4: > 00000000000406f0 > [1631983.772757] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [1631983.772792] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > [1631983.772827] Process mkfs.ext4 (pid: 28707, threadinfo > ffff880133edacf0, task ffff880133eda800) > [1631983.772880] ffff880165cadc48 ffffffff813f08dc 0000001065cadc48 > ffff880163fe0000 > [1631983.772919] ffff8801128e0000 ffff880165fd0318 ffff8801128e0000 > ffff880165fd0378 > [1631983.772957] ffff880165cadc68 ffff880165cadc58 ffff880165cadca8 > ffffffff813f1ae0 > [1631983.773015] [<ffffffff813f08dc>] check_should_skip+0x31f/0x335 > [1631983.773039] [<ffffffff813f1ae0>] request_write+0x7d/0x267 > [1631983.773061] [<ffffffff813f1dc8>] cached_dev_make_request+0xfe/0x1ad > [1631983.773087] [<ffffffff8127edff>] generic_make_request+0x17c/0x1d2 > [1631983.773110] [<ffffffff8127ef25>] submit_bio+0xd0/0xdb > [1631983.773133] [<ffffffff81284a3d>] blkdev_issue_discard+0x158/0x1a7 > [1631983.773156] [<ffffffff812850bb>] blkdev_ioctl+0x2f7/0x69c > [1631983.773180] [<ffffffff811191f8>] block_ioctl+0x32/0x36 > [1631983.773203] [<ffffffff810fe7e2>] do_vfs_ioctl+0x5aa/0x5fa > [1631983.773226] [<ffffffff810fe874>] sys_ioctl+0x42/0x65 > [1631983.773250] [<ffffffff815657b6>] system_call_fastpath+0x18/0x1d > [1631983.773384] Call Trace: > [1631983.773401] [<ffffffff813f08dc>] check_should_skip+0x31f/0x335 > [1631983.773424] [<ffffffff813f1ae0>] request_write+0x7d/0x267 > [1631983.773447] [<ffffffff813f1dc8>] cached_dev_make_request+0xfe/0x1ad > [1631983.773470] [<ffffffff8127edff>] generic_make_request+0x17c/0x1d2 > [1631983.773499] [<ffffffff8127ef25>] submit_bio+0xd0/0xdb > [1631983.773520] [<ffffffff81284a3d>] blkdev_issue_discard+0x158/0x1a7 > [1631983.773544] [<ffffffff812850bb>] blkdev_ioctl+0x2f7/0x69c > [1631983.773567] [<ffffffff811191f8>] block_ioctl+0x32/0x36 > [1631983.773590] [<ffffffff810fe7e2>] do_vfs_ioctl+0x5aa/0x5fa > [1631983.773613] [<ffffffff810fe874>] sys_ioctl+0x42/0x65 > [1631983.773635] [<ffffffff815657b6>] system_call_fastpath+0x18/0x1d > > I'm using the 3.2.33 Linux kernel with > grsecurity-2.9.1-3.2.33-201211072000 and bcache v3.2.28-384-gcafb412. > > I've tried to set it up this way multiple times but I hit the same > problem each time. Because I'm successfully running bcache on a mdadm > device, I thought there was an issue with the luks part. So I tested > this part with a USB thumb drive as a backing device: > > /dev/sdd > - /dev/dm-0 (luks, bcache backing device) > - /dev/bcache1 > > In this case the bcache device worked without a problem. As you can see > in the stacktrace, bch_mark_sectors_bypassed (a piece of bcache code) > causes the crash. Do you know what is going wrong here? That is _odd_. I'm scratching my head over what could possibly have gone wrong _there_. bch_mark_sectors_bypassed() doesn't do much, I think the only thing that _could_ go wrong is derefing a bad pointer but if either of the pointers it derefs are bad things should've exploded earlier. Maybe I'm blind but I'm also not seeing what exactly the kernel is complaining about - no null pointer deref, no BUG(), no oops, just a bunch of backtraces. That's kind of bizzare. Send me your .config, maybe you've got something flipped off. Might be worth building a kernel with a bunch of debug stuff turned on - slab debugging for sure. I may have to try and replicate it on my end. At least it's something that happens reliably... ^ permalink raw reply [flat|nested] 4+ messages in thread
[parent not found: <20121228043144.GC10411-jC9Py7bek1znysI04z7BkA@public.gmane.org>]
* Re: mkfs crash [not found] ` <20121228043144.GC10411-jC9Py7bek1znysI04z7BkA@public.gmane.org> @ 2012-12-29 11:53 ` Pim van den Berg [not found] ` <50DED9A0.40002-IXGSG4U2CCrz+pZb47iToQ@public.gmane.org> 0 siblings, 1 reply; 4+ messages in thread From: Pim van den Berg @ 2012-12-29 11:53 UTC (permalink / raw) To: Kent Overstreet; +Cc: linux-bcache-u79uwXL29TY76Z2rM5mHXA On 12/28/2012 05:31 AM, Kent Overstreet wrote: > That is _odd_. I'm scratching my head over what could possibly have > gone wrong _there_. bch_mark_sectors_bypassed() doesn't do much, I > think the only thing that _could_ go wrong is derefing a bad pointer > but if either of the pointers it derefs are bad things should've > exploded earlier. > > Maybe I'm blind but I'm also not seeing what exactly the kernel is > complaining about - no null pointer deref, no BUG(), no oops, just a > bunch of backtraces. That's kind of bizzare. > > Send me your .config, maybe you've got something flipped off. > > Might be worth building a kernel with a bunch of debug stuff turned > on - slab debugging for sure. > > I may have to try and replicate it on my end. At least it's something > that happens reliably... Yesterday I compiled a new kernel (3.2.35, bcache v3.2.28-384-gcafb412, grsecurity-2.9.1-3.2.35-201212271951) to give it another try. I turned on slab debugging. The same problem again. But when I look at my syslog, I see there is something wrong with the previous logfile. Because syslog was logrotated a while ago, I got my information from /var/log/messages which doesn't contain all of the logging. This is wat I see now (full log: http://pommi.nethuis.nl/storage/software/bcache/log/mkfs-crash2.log): [ 775.832304] PAX: From 127.0.0.6: refcount overflow detected in: mkfs.ext4:3311, uid/euid: 0/0 [ 775.832345] CPU 0 [ 775.832362] Pid: 3311, comm: mkfs.ext4 Not tainted 3.2.35-kvm #3 /DH67CF [ 775.832402] RIP: 0010:[<ffffffff813fa7e2>] [<ffffffff813fa7e2>] bch_mark_sectors_bypassed+0x1a/0x35 [ 775.832446] RSP: 0018:ffff880203f95bf8 EFLAGS: 00000a06 [ 775.832467] RAX: ffff880203888010 RBX: ffff8802038a6278 RCX: 0000000000011200 [ 775.832491] RDX: 2000000000000000 RSI: 00000000007fffff RDI: ffff8802038a6278 [ 775.832515] RBP: ffff880203f95bf8 R08: 000000000000e95e R09: ffff8802038ab560 [ 775.832539] R10: 000000000000e910 R11: ffff880203f95c78 R12: ffff880203888000 [ 775.832563] R13: ffff880202b00000 R14: ffff880203f95c68 R15: 0000000000000000 [ 775.832588] FS: 00006ada56e84760(0000) GS:ffff88021fa00000(0000) knlGS:0000000000000000 [ 775.832624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 775.832646] CR2: 0000000000d0a628 CR3: 0000000202bbf000 CR4: 00000000000406f0 [ 775.832670] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 775.832694] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 775.832718] Process mkfs.ext4 (pid: 3311, threadinfo ffff8802120e5cf0, task ffff8802120e5800) [ 775.832755] Stack: [ 775.832770] ffff880203f95c48 ffffffff813f1ef4 00000010810b5574 ffff8802038af000 [ 775.832809] ffff880203f95c48 ffff8802038a6278 ffff880203888000 ffff8802038a62d8 [ 775.832852] ffff880203f95c68 ffff880203f95c58 ffff880203f95ca8 ffffffff813f30f8 [ 775.834245] Call Trace: [ 775.834265] [<ffffffff813f1ef4>] check_should_skip+0x31f/0x335 [ 775.834288] [<ffffffff813f30f8>] request_write+0x7d/0x267 [ 775.834310] [<ffffffff813f33e0>] cached_dev_make_request+0xfe/0x1ad [ 775.834335] [<ffffffff8128041f>] generic_make_request+0x17c/0x1d2 [ 775.834358] [<ffffffff81280545>] submit_bio+0xd0/0xdb [ 775.834380] [<ffffffff81286061>] blkdev_issue_discard+0x158/0x1a7 [ 775.834403] [<ffffffff812866df>] blkdev_ioctl+0x2f7/0x69c [ 775.834427] [<ffffffff8111a790>] block_ioctl+0x32/0x36 [ 775.834448] [<ffffffff810ffd6e>] do_vfs_ioctl+0x5aa/0x5fa [ 775.834472] [<ffffffff810e1a2d>] ? cache_free_debugcheck+0x7e/0x1ec [ 775.834495] [<ffffffff810ffe00>] sys_ioctl+0x42/0x65 [ 775.834517] [<ffffffff81566fb6>] system_call_fastpath+0x18/0x1d [ 775.834538] Code: 60 01 00 00 71 09 f0 ff 88 60 01 00 00 cd 04 c9 c3 55 48 8b 47 30 48 89 e5 f0 01 b0 64 52 00 00 71 09 f0 29 b0 64 52 00 00 cd 04 <48> 8b 87 10 01 00 00 f0 01 b0 64 01 00 00 71 09 f0 29 b0 64 01 [ 775.834649] Call Trace: [ 775.834666] [<ffffffff813f1ef4>] check_should_skip+0x31f/0x335 [ 775.834689] [<ffffffff813f30f8>] request_write+0x7d/0x267 [ 775.834711] [<ffffffff813f33e0>] cached_dev_make_request+0xfe/0x1ad [ 775.834734] [<ffffffff8128041f>] generic_make_request+0x17c/0x1d2 [ 775.834757] [<ffffffff81280545>] submit_bio+0xd0/0xdb [ 775.834779] [<ffffffff81286061>] blkdev_issue_discard+0x158/0x1a7 [ 775.834801] [<ffffffff812866df>] blkdev_ioctl+0x2f7/0x69c [ 775.834823] [<ffffffff8111a790>] block_ioctl+0x32/0x36 [ 775.834845] [<ffffffff810ffd6e>] do_vfs_ioctl+0x5aa/0x5fa [ 775.834867] [<ffffffff810e1a2d>] ? cache_free_debugcheck+0x7e/0x1ec [ 775.834890] [<ffffffff810ffe00>] sys_ioctl+0x42/0x65 [ 775.834911] [<ffffffff81566fb6>] system_call_fastpath+0x18/0x1d So it starts with PAX, detecting a refcount overflow, and makes mkfs.ext4 crash. The question now is, is it a grsecurity/pax bug, a bcache bug, or is it a combination of things? My .config: http://pommi.nethuis.nl/storage/software/bcache/log/config-3.2.35-kvm I patched the linux kernel in the following order: 1. bcache v3.2.28-384-gcafb412 2. grsecurity-2.9.1-3.2.35-201212271951 3. http://pommi.nethuis.nl/storage/software/bcache/bcache-grsecurity.patch -- Regards, Pim ^ permalink raw reply [flat|nested] 4+ messages in thread
[parent not found: <50DED9A0.40002-IXGSG4U2CCrz+pZb47iToQ@public.gmane.org>]
* Re: mkfs crash [not found] ` <50DED9A0.40002-IXGSG4U2CCrz+pZb47iToQ@public.gmane.org> @ 2012-12-29 20:28 ` Pim van den Berg 0 siblings, 0 replies; 4+ messages in thread From: Pim van den Berg @ 2012-12-29 20:28 UTC (permalink / raw) To: Kent Overstreet; +Cc: linux-bcache-u79uwXL29TY76Z2rM5mHXA On 12/29/2012 12:53 PM, Pim van den Berg wrote: > So it starts with PAX, detecting a refcount overflow, and makes > mkfs.ext4 crash. The question now is, is it a grsecurity/pax bug, a > bcache bug, or is it a combination of things? I got an accidental reply from the PaX Team: On 12/29/2012 07:33 PM, PaX Team wrote: > hi, > > i just accidentally ran across your report on linux-bcache, and > wanted to tell you that this is probably a false positive refcount > overflow detection because the atomic variables in question seem to > be only statistics that only ever get incremented so an eventual > overflow is normal (or at least not a security problem, you may still > want to let Kent know since an overflowing counter means 'false' > stats for the end user). the 'fix' for this problem could be any of > the following (with different benefits/drawbacks): > > 1. disable REFCOUNT in your .config > 2. patch these bcache counters to be of the atomic_unchecked_t type > and also their accessors to use the _unchecked variants > 3. make these counters atomic64_t that will probably not overflow > (but if they still can, there's atomic64_unchecked_t ;) > > cheers and happy new year, > PaX Team ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2012-12-29 20:28 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-12-27 12:05 mkfs crash Pim van den Berg
[not found] ` <50DC3977.2080209-IXGSG4U2CCrz+pZb47iToQ@public.gmane.org>
2012-12-28 4:31 ` Kent Overstreet
[not found] ` <20121228043144.GC10411-jC9Py7bek1znysI04z7BkA@public.gmane.org>
2012-12-29 11:53 ` Pim van den Berg
[not found] ` <50DED9A0.40002-IXGSG4U2CCrz+pZb47iToQ@public.gmane.org>
2012-12-29 20:28 ` Pim van den Berg
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.