Re: [PATCH 03/18] xfs: Do background CIL flushes via a workqueue

Ben Myers
2012-04-17T21:48:11Z

On Tue, Apr 17, 2012 at 04:21:55PM -0500, Ben Myers wrote:
> On Fri, Apr 13, 2012 at 10:10:46PM +1000, Dave Chinner wrote:
> > From: Dave Chinner <dchinner@redhat.com>
> >
> > Doing background CIL flushes adds significant latency to whatever
> > async transaction that triggers it. To avoid blocking async
> > transactions on things like waiting for log buffer IO to complete,
> > move the CIL push off into a workqueue. By moving the push work
> > into a workqueue, we remove all the latency that the commit adds
> > from the foreground transaction commit path. This also means that
> > single threaded workloads won't do the CIL push procssing, leaving
> > them more CPU to do more async transactions.
> >
> > To do this, we need to keep track of the sequence number we have
> > pushed work for. This avoids having many transaction commits
> > attempting to schedule work for the same sequence, and ensures that
> > we only ever have one push (background or forced) in progress at a
> > time. It also means that we don't need to take the CIL lock in write
> > mode to check for potential background push races, which reduces
> > lock contention.
> >
> > To avoid potential issues with "smart" IO schedulers, don't use the
> > workqueue for log force triggered flushes. Instead, do them directly
> > so that the log IO is done directly by the process issuing the log
> > force and so doesn't get stuck on IO elevator queue idling
> > incorrectly delaying the log IO from the workqueue.
> >
> > Signed-off-by: Dave Chinner <dchinner@redhat.com>
> > ---
> > fs/xfs/xfs_log_cil.c | 241 ++++++++++++++++++++++++++++++-------------------
> > fs/xfs/xfs_log_priv.h | 2 +
> > fs/xfs/xfs_mount.h | 1 +
> > fs/xfs/xfs_super.c | 7 ++
> > 4 files changed, 157 insertions(+), 94 deletions(-)
> >
> > diff --git a/fs/xfs/xfs_log_cil.c b/fs/xfs/xfs_log_cil.c
> > index d4fadbe..566a2d5 100644
> > --- a/fs/xfs/xfs_log_cil.c
> > +++ b/fs/xfs/xfs_log_cil.c
> > @@ -32,58 +32,6 @@
> > #include "xfs_discard.h"
> >
> > /*
> > - * Perform initial CIL structure initialisation.
> > - */
> > -int
> > -xlog_cil_init(
> > - struct log *log)
> > -{
> > - struct xfs_cil *cil;
> > - struct xfs_cil_ctx *ctx;
> > -
> > - cil = kmem_zalloc(sizeof(*cil), KM_SLEEP|KM_MAYFAIL);
> > - if (!cil)
> > - return ENOMEM;
> > -
> > - ctx = kmem_zalloc(sizeof(*ctx), KM_SLEEP|KM_MAYFAIL);
> > - if (!ctx) {
> > - kmem_free(cil);
> > - return ENOMEM;
> > - }
> > -
> > - INIT_LIST_HEAD(&cil->xc_cil);
> > - INIT_LIST_HEAD(&cil->xc_committing);
> > - spin_lock_init(&cil->xc_cil_lock);
> > - init_rwsem(&cil->xc_ctx_lock);
> > - init_waitqueue_head(&cil->xc_commit_wait);
> > -
> > - INIT_LIST_HEAD(&ctx->committing);
> > - INIT_LIST_HEAD(&ctx->busy_extents);
> > - ctx->sequence = 1;
> > - ctx->cil = cil;
> > - cil->xc_ctx = ctx;
> > - cil->xc_current_sequence = ctx->sequence;
> > -
> > - cil->xc_log = log;
> > - log->l_cilp = cil;
> > - return 0;
> > -}
> > -
> > -void
> > -xlog_cil_destroy(
> > - struct log *log)
> > -{
> > - if (log->l_cilp->xc_ctx) {
> > - if (log->l_cilp->xc_ctx->ticket)
> > - xfs_log_ticket_put(log->l_cilp->xc_ctx->ticket);
> > - kmem_free(log->l_cilp->xc_ctx);
> > - }
> > -
> > - ASSERT(list_empty(&log->l_cilp->xc_cil));
> > - kmem_free(log->l_cilp);
> > -}
> > -
> > -/*
> > * Allocate a new ticket. Failing to get a new ticket makes it really hard to
> > * recover, so we don't allow failure here. Also, we allocate in a context that
> > * we don't want to be issuing transactions from, so we need to tell the
> > @@ -426,8 +374,7 @@ xlog_cil_committed(
> > */
> > STATIC int
> > xlog_cil_push(
> > - struct log *log,
> > - xfs_lsn_t push_seq)
> > + struct log *log)
> > {
> > struct xfs_cil *cil = log->l_cilp;
> > struct xfs_log_vec *lv;
> > @@ -443,39 +390,35 @@ xlog_cil_push(
> > struct xfs_log_iovec lhdr;
> > struct xfs_log_vec lvhdr = { NULL };
> > xfs_lsn_t commit_lsn;
> > + xfs_lsn_t push_seq;
> >
> > if (!cil)
> > return 0;
> >
> > - ASSERT(!push_seq || push_seq <= cil->xc_ctx->sequence);
> > -
> > new_ctx = kmem_zalloc(sizeof(*new_ctx), KM_SLEEP|KM_NOFS);
> > new_ctx->ticket = xlog_cil_ticket_alloc(log);
> >
> > - /*
> > - * Lock out transaction commit, but don't block for background pushes
> > - * unless we are well over the CIL space limit. See the definition of
> > - * XLOG_CIL_HARD_SPACE_LIMIT() for the full explanation of the logic
> > - * used here.
> > - */
> > - if (!down_write_trylock(&cil->xc_ctx_lock)) {
> > - if (!push_seq &&
> > - cil->xc_ctx->space_used < XLOG_CIL_HARD_SPACE_LIMIT(log))
> > - goto out_free_ticket;
> > - down_write(&cil->xc_ctx_lock);
> > - }
> > + down_write(&cil->xc_ctx_lock);
> > ctx = cil->xc_ctx;
> >
> > - /* check if we've anything to push */
> > - if (list_empty(&cil->xc_cil))
> > - goto out_skip;
> > + spin_lock(&cil->xc_cil_lock);
> > + push_seq = cil->xc_push_seq;
> > + ASSERT(push_seq > 0 && push_seq <= ctx->sequence);
>
> Gah! I just hit this assert.
>
> v3.4-rc2-3-g8a00ebe with:
> Christoph's ilock series
> Christoph's xfsbufd series
> Jan's freeze series
> Dave's queue.
>
> nfs7 login: [ 1175.172406] XFS: Assertion failed: push_seq > 0 && push_seq <= ctx->sequence, file: /root/xfs/fs/xfs/xfs_log_cil.c, line: 406
> [ 1175.183766] ------------[ cut here ]------------
> [ 1175.188010] kernel BUG at /root/xfs/fs/xfs/xfs_message.c:101!
> [ 1175.188010] invalid opcode: 0000 [#1] PREEMPT SMP
> [ 1175.188010] Modules linked in: xfs(O) exportfs af_packet dm_mod floppy iTCO_wdt sg i2c_i801 iTCO_vendor_support e7xxx_edac edac_core sr_mod e100 cdrom e1000 shpchp pci_hotplug button serio_raw pcspkr autofs4 processor thermal_sys ata_generic
> [ 1175.188010]
> [ 1175.188010] Pid: 2760, comm: kworker/3:2 Tainted: G O 3.4.0-rc2-1.2-desktop+ #15 TYAN Computer Corp. S2721-533 Thunder i7501 Pro/S2721-533 Thunder i7501 Pro
> [ 1175.188010] EIP: 0060:[<faa4f966>] EFLAGS: 00010296 CPU: 3
> [ 1175.188010] EIP is at assfail+0x26/0x30 [xfs]
> [ 1175.188010] EAX: 00000087 EBX: f1f10980 ECX: 00000079 EDX: 00000046
> [ 1175.188010] ESI: f1f10780 EDI: 00000000 EBP: f1d93ec4 ESP: f1d93eb0
> [ 1175.188010] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
> [ 1175.188010] CR0: 8005003b CR2: b770ee20 CR3: 2779c000 CR4: 000007f0
> [ 1175.188010] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> [ 1175.188010] DR6: ffff0ff0 DR7: 00000400
> [ 1175.188010] Process kworker/3:2 (pid: 2760, ti=f1d92000 task=f1e671e0 task.ti=f1d92000)
> [ 1175.188010] Stack:
> [ 1175.188010] 00000000 faac7a6c faad0ee4 faad0e28 00000196 f1d93f4c faaacb2c ee24c3b0
> [ 1175.188010] eee1f700 f1d93f00 c0201a8e f1f10994 f1daea00 f1f1098c 00000000 f1e67500
> [ 1175.188010] f1f10c00 00000000 00000000 00000000 00000000 00000000 00000000 c06983de
> [ 1175.188010] Call Trace:
> [ 1175.188010] [<faaacb2c>] xlog_cil_push+0x2cc/0x3e0 [xfs]
> [ 1175.188010] [<c0201a8e>] ? __switch_to+0xde/0x2c0
> [ 1175.188010] [<c06983de>] ? __schedule+0x21e/0x7c0
> [ 1175.188010] [<faaacc4b>] xlog_cil_push_work+0xb/0x10 [xfs]
> [ 1175.188010] [<c024bc07>] process_one_work+0xf7/0x3f0
> [ 1175.188010] [<faaacc40>] ? xlog_cil_push+0x3e0/0x3e0 [xfs]
> [ 1175.188010] [<c024c202>] worker_thread+0x122/0x2d0
> [ 1175.188010] [<c024c0e0>] ? rescuer_thread+0x1b0/0x1b0
> [ 1175.188010] [<c024fd6d>] kthread+0x6d/0x80
> [ 1175.188010] [<c024fd00>] ? kthread_freezable_should_stop+0x50/0x50
> [ 1175.188010] [<c06a02b6>] kernel_thread_helper+0x6/0xd
> [ 1175.188010] Code: bf 00 00 00 00 55 89 e5 83 ec 14 89 4c 24 10 89 54 24 0c 89 44 24 08 c7 44 24 04 6c 7a ac fa c7 04 24 00 00 00 00 e8 da fd ff ff <0f> 0b 90 8d b4 26 00 00 00 00 55 b9 01 00 00 00 89 e5 83 ec 14
> [ 1175.188010] EIP: [<faa4f966>] assfail+0x26/0x30 [xfs] SS:ESP 0068:f1d93eb0
> [ 1175.189377] ---[ end trace b2f84afec1bd6d71 ]---
>
> Unfortunately this machine isn't configured to dump.

Now I've backed off to just Christoph's ilock series and ran 'check -g auto'.

It hung on the first test:

[ 1175.190019] [<faaacc40>] ? xlog_cil_push+0x3e0/0x3e0 [xfs]
[ 1175.190025] [<c024c202>] worker_thread+0x122/0x2d0
nfs7 login: 0] [<c024c0e0>] ? rescuer_thread+0x1b0/0x1b0
Welcome to openSUSE 12.1 "Asparagus" - Kernel 3.4.0-rc2-1.2-desktop+ (ttyS0).
[ 1175.190042] [<c024fd00>] ? kthread_freezable_should_stop+0x50/0x50
[ 1175.190049] [<c06a02b6>] kernel_thread_helper+0x6/0xd
nfs7 login: 2] Code: 89 e0 25 00 e0 ff ff 83 68 14 01 8b 40 08 a8 08 0f 84 42 ff ff ff e8 d6 88 44 00 e9 38 ff ff ff 90 55 8b 80 84 02 00 00 89 e5 5d <8b> 40 fc c3 66 90 3b 05 44 17 9a cWelcome to openSUSE 12.1 "Asparagus" - Kernel 3.4.0-rc2-1.2-desktop+ (ttyS0).
[ 1175.190109] EIP: [<c025013a>] kthread_data+0xa/0x10 SS:ESP 0068:f1d93ca8
[ 1175.190116] CR2: 00000000fffffffc
nfs7 login: [ 390.054357] XFS (�=��P۔�): xlog_recover_inode_pass2: Bad inode magic number, dip = 0xf0ffd800, dino bp = 0xef89c480, ino = 25541592
[ 390.066290] XFS (�=��P۔�): Internal error xlog_recover_inode_pass2(1) at line 2248 of file /root/xfs/fs/xfs/xfs_log_recover.c. Caller 0xfaa08ffa
[ 390.066295] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 2, t=60008 jiffies)
[ 390.082217] XFS: Assertion failed: atomic_read(&pag->pag_ref) == 0, file: /root/xfs/fs/xfs/xfs_mount.c, line: 272
[ 390.092542] ------------[ cut here ]------------ CPUs/tasks: { 3} (detected by 2, t=240013 jiffies)
[ 390.097004] kernel BUG at /root/xfs/fs/xfs/xfs_message.c:101!
[ 390.097004] invalid opcode: 0000 [#1] PREEMPT SMP PUs/tasks: { 3} (detected by 1, t=420012 jiffies)
[ 390.097004] Modules linked in: xfs(O) exportfs af_packet dm_mod e1000 e100 sr_mod shpchp iTCO_wdt cdrom i2c_i801 e7xxx_edac iTCO_vendor_support sg floppy pci_hotplug serio_raw button edac_core pcspkr autofs4 processor thermal_sys ata_genericasks: { 3} (detected by 2, t=600023 jiffies)
[ 390.097004] INFO: Stall ended before state dump start
[ 390.097004] Pid: 5217, comm: mount Tainted: G O 3.4.0-rc2-1.2-desktop+ #15 TYAN Computer Corp. S2721-533 Thunder i7501 Pro/S2721-533 Thunder i7501 Pro
[ 390.097004] EIP: 0060:[<fa9b8d56>] EFLAGS: 00010286 CPU: 3
[ 390.097004] EIP is at assfail+0x26/0x30 [xfs] on CPUs/tasks: { 3} (detected by 2, t=960033 jiffies)
[ 390.097004] EAX: 0000007b EBX: f0f87680 ECX: 000000f3 EDX: 00000046
[ 390.097004] ESI: 00000000 EDI: f1d089a8 EBP: f1d67ddc ESP: f1d67dc8detected by 1, t=1140032 jiffies)
[ 390.097004] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[ 390.097004] CR0: 8005003b CR2: b7735580 CR3: 2ff82000 CR4: 000007f0detected by 1, t=1320037 jiffies)
[ 390.097004] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 390.097004] DR6: ffff0ff0 DR7: 00000400stalls on CPUs/tasks: { 3} (detected by 1, t=1500042 jiffies)
[ 390.097004] Process mount (pid: 5217, ti=f1d66000 task=f0d99060 task.ti=f1d66000)
[ 390.097004] Stack:
[ 390.097004] 00000000 faa31844 faa399d4 faa41a7f 00000110 f1d67df8 faa0b125 f1d0899c
[ 390.097004] f1d08800 00000075 f1d08800 00000014 f1d67e50 faa0dfe9 f1d08800 faa41c34
[ 390.097004] 00005000 ef82b700 fa9aea50 f1d67e24 f1d08acc 00000000 00000002 00000003
[ 390.097004] Call Trace:
[ 390.097004] [<faa0b125>] xfs_free_perag+0x75/0xa0 [xfs]
[ 390.097004] [<faa0dfe9>] xfs_mountfs+0x2d9/0x710 [xfs]
[ 390.097004] [<fa9aea50>] ? _xfs_filestream_pick_ag+0x1b0/0x1b0 [xfs]
[ 390.097004] [<fa9bb5d6>] xfs_fs_fill_super+0x196/0x240 [xfs]
[ 390.097004] [<c031ce72>] mount_bdev+0x172/0x1b0
[ 390.097004] [<fa9b965a>] xfs_fs_mount+0x1a/0x20 [xfs]
[ 390.097004] [<fa9bb440>] ? xfs_finish_flags+0x130/0x130 [xfs]
[ 390.097004] [<c031d9a1>] mount_fs+0x31/0x170
[ 390.097004] [<c02ecbfa>] ? __alloc_percpu+0xa/0x10
[ 390.097004] [<c03336cc>] vfs_kern_mount+0x4c/0xc0
[ 390.097004] [<c0333fd9>] do_kern_mount+0x39/0xd0
[ 390.097004] [<c0335381>] do_mount+0x161/0x710
[ 390.097004] [<c02e7e41>] ? strndup_user+0x41/0x60
[ 390.097004] [<c0335a36>] sys_mount+0x66/0xa0
[ 390.097004] [<c0699d2d>] syscall_call+0x7/0xb
[ 390.097004] Code: bf 00 00 00 00 55 89 e5 83 ec 14 89 4c 24 10 89 54 24 0c 89 44 24 08 c7 44 24 04 44 18 a3 fa c7 04 24 00 00 00 00 e8 da fd ff ff <0f> 0b 90 8d b4 26 00 00 00 00 55 b9 01 00 00 00 89 e5 83 ec 14
[ 390.097004] EIP: [<fa9b8d56>] assfail+0x26/0x30 [xfs] SS:ESP 0068:f1d67dc8
[ 390.097633] ---[ end trace 33a9795b638157b9 ]---

Here it is from the system log

Apr 17 16:35:12 linux kernel: [ 389.637543] XFS (264=211361P۔300): Mounting Filesystem
Apr 17 16:35:12 linux kernel: [ 389.912402] XFS (264=211361P۔300): Starting recovery (logdev: internal)
Apr 17 16:35:12 linux kernel: [ 390.054357] XFS (264=211361P۔300): xlog_recover_inode_pass2: Bad inode magic number, dip = 0xf0ffd800, dino bp = 0xef89c480, ino = 25541592
Apr 17 16:35:12 linux kernel: [ 390.066290] XFS (264=211361P۔300): Internal error xlog_recover_inode_pass2(1) at line 2248 of file /root/xfs/fs/xfs/xfs_log_recover.c. Caller 0xfaa08ffa
Apr 17 16:35:12 linux kernel: [ 390.066295]
Apr 17 16:35:12 linux kernel: [ 390.080932] Pid: 5217, comm: mount Tainted: G O 3.4.0-rc2-1.2-desktop+ #15
Apr 17 16:35:12 linux kernel: [ 390.080938] Call Trace:
Apr 17 16:35:12 linux kernel: [ 390.080998] [<fa9ac826>] xfs_error_report+0x46/0x50 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081063] [<faa08ffa>] ? xlog_recover_commit_pass2+0xea/0x1a0 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081105] [<faa086a1>] xlog_recover_inode_pass2+0x7e1/0x9f0 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081167] [<faa08ffa>] ? xlog_recover_commit_pass2+0xea/0x1a0 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081212] [<faa07d32>] ? xlog_recover_buffer_pass2+0x122/0x2b0 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081255] [<faa08ffa>] xlog_recover_commit_pass2+0xea/0x1a0 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081326] [<faa09105>] xlog_recover_commit_trans+0x55/0xa0 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081371] [<faa0932c>] xlog_recover_process_data+0x1ac/0x2a0 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081444] [<faa0a2d8>] xlog_do_recovery_pass+0x3b8/0x7c0 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081505] [<faa0a793>] xlog_do_log_recovery+0xb3/0x120 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081562] [<faa0a9b4>] xlog_do_recover+0x24/0x1c0 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081612] [<fa9b8c48>] ? xfs_notice+0x28/0x30 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081654] [<faa0abc8>] xlog_recover+0x78/0x90 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081709] [<faa125c2>] xfs_log_mount+0x92/0x180 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081752] [<faa0e073>] xfs_mountfs+0x363/0x710 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081785] [<fa9aea50>] ? _xfs_filestream_pick_ag+0x1b0/0x1b0 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081828] [<fa9bb5d6>] xfs_fs_fill_super+0x196/0x240 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081845] [<c031ce72>] mount_bdev+0x172/0x1b0
Apr 17 16:35:12 linux kernel: [ 390.081888] [<fa9b965a>] xfs_fs_mount+0x1a/0x20 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081932] [<fa9bb440>] ? xfs_finish_flags+0x130/0x130 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.081943] [<c031d9a1>] mount_fs+0x31/0x170
Apr 17 16:35:12 linux kernel: [ 390.081954] [<c02ecbfa>] ? __alloc_percpu+0xa/0x10
Apr 17 16:35:12 linux kernel: [ 390.081965] [<c03336cc>] vfs_kern_mount+0x4c/0xc0
Apr 17 16:35:12 linux kernel: [ 390.081974] [<c0333fd9>] do_kern_mount+0x39/0xd0
Apr 17 16:35:12 linux kernel: [ 390.081984] [<c0335381>] do_mount+0x161/0x710
Apr 17 16:35:12 linux kernel: [ 390.081993] [<c02e7e41>] ? strndup_user+0x41/0x60
Apr 17 16:35:12 linux kernel: [ 390.082035] [<c0335a36>] sys_mount+0x66/0xa0
Apr 17 16:35:12 linux kernel: [ 390.082051] [<c0699d2d>] syscall_call+0x7/0xb
Apr 17 16:35:12 linux kernel: [ 390.082084] XFS (264=211361P۔300): log mount/recovery failed: error 117
Apr 17 16:35:12 linux kernel: [ 390.082209] XFS (264=211361P۔300): log mount failed
Apr 17 16:35:12 linux kernel: [ 390.082217] XFS: Assertion failed: atomic_read(&pag->pag_ref) == 0, file: /root/xfs/fs/xfs/xfs_mount.c, line: 272

Apr 17 16:35:12 linux kernel: [ 390.092542] ------------[ cut here ]------------
Apr 17 16:35:12 linux kernel: [ 390.097004] kernel BUG at /root/xfs/fs/xfs/xfs_message.c:101!
Apr 17 16:35:12 linux kernel: [ 390.097004] invalid opcode: 0000 [#1] PREEMPT SMP
Apr 17 16:35:12 linux kernel: [ 390.097004] Modules linked in: xfs(O) exportfs af_packet dm_mod e1000 e100 sr_mod shpchp iTCO_wdt cdrom i2c_i801 e7xxx_edac iTCO_vendor_support sg floppy pci_hotplug serio_raw button edac_core pcspkr autofs4 processor thermal_sys ata_generic
Apr 17 16:35:12 linux kernel: [ 390.097004]
Apr 17 16:35:12 linux kernel: [ 390.097004] Pid: 5217, comm: mount Tainted: G O 3.4.0-rc2-1.2-desktop+ #15 TYAN Computer Corp. S2721-533 Thunder i7501 Pro/S2721-533 Thunder i7501 Pro
Apr 17 16:35:12 linux kernel: [ 390.097004] EIP: 0060:[<fa9b8d56>] EFLAGS: 00010286 CPU: 3
Apr 17 16:35:12 linux kernel: [ 390.097004] EIP is at assfail+0x26/0x30 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.097004] EAX: 0000007b EBX: f0f87680 ECX: 000000f3 EDX: 00000046
Apr 17 16:35:12 linux kernel: [ 390.097004] ESI: 00000000 EDI: f1d089a8 EBP: f1d67ddc ESP: f1d67dc8
Apr 17 16:35:12 linux kernel: [ 390.097004] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Apr 17 16:35:12 linux kernel: [ 390.097004] CR0: 8005003b CR2: b7735580 CR3: 2ff82000 CR4: 000007f0
Apr 17 16:35:12 linux kernel: [ 390.097004] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Apr 17 16:35:12 linux kernel: [ 390.097004] DR6: ffff0ff0 DR7: 00000400
Apr 17 16:35:12 linux kernel: [ 390.097004] Process mount (pid: 5217, ti=f1d66000 task=f0d99060 task.ti=f1d66000)
Apr 17 16:35:12 linux kernel: [ 390.097004] Stack:
Apr 17 16:35:12 linux kernel: [ 390.097004] 00000000 faa31844 faa399d4 faa41a7f 00000110 f1d67df8 faa0b125 f1d0899c
Apr 17 16:35:12 linux kernel: [ 390.097004] f1d08800 00000075 f1d08800 00000014 f1d67e50 faa0dfe9 f1d08800 faa41c34
Apr 17 16:35:12 linux kernel: [ 390.097004] 00005000 ef82b700 fa9aea50 f1d67e24 f1d08acc 00000000 00000002 00000003
Apr 17 16:35:12 linux kernel: [ 390.097004] Call Trace:
Apr 17 16:35:12 linux kernel: [ 390.097004] [<faa0b125>] xfs_free_perag+0x75/0xa0 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.097004] [<faa0dfe9>] xfs_mountfs+0x2d9/0x710 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.097004] [<fa9aea50>] ? _xfs_filestream_pick_ag+0x1b0/0x1b0 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.097004] [<fa9bb5d6>] xfs_fs_fill_super+0x196/0x240 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.097004] [<c031ce72>] mount_bdev+0x172/0x1b0
Apr 17 16:35:12 linux kernel: [ 390.097004] [<fa9b965a>] xfs_fs_mount+0x1a/0x20 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.097004] [<fa9bb440>] ? xfs_finish_flags+0x130/0x130 [xfs]
Apr 17 16:35:12 linux kernel: [ 390.097004] [<c031d9a1>] mount_fs+0x31/0x170
Apr 17 16:35:12 linux kernel: [ 390.097004] [<c02ecbfa>] ? __alloc_percpu+0xa/0x10
Apr 17 16:35:12 linux kernel: [ 390.097004] [<c03336cc>] vfs_kern_mount+0x4c/0xc0
Apr 17 16:35:12 linux kernel: [ 390.097004] [<c0333fd9>] do_kern_mount+0x39/0xd0
Apr 17 16:35:12 linux kernel: [ 390.097004] [<c0335381>] do_mount+0x161/0x710
Apr 17 16:35:12 linux kernel: [ 390.097004] [<c02e7e41>] ? strndup_user+0x41/0x60
Apr 17 16:35:12 linux kernel: [ 390.097004] [<c0335a36>] sys_mount+0x66/0xa0
Apr 17 16:35:12 linux kernel: [ 390.097004] [<c0699d2d>] syscall_call+0x7/0xb
Apr 17 16:35:12 linux kernel: [ 390.097004] Code: bf 00 00 00 00 55 89 e5 83 ec 14 89 4c 24 10 89 54 24 0c 89 44 24 08 c7 44 24 04 44 18 a3 fa c7 04 24 00 00 00 00 e8 da fd ff ff <0f> 0b 90 8d b4 26 00 00 00 00 55 b9 01 00 00 00 89 e5 83 ec 14
Apr 17 16:35:12 linux kernel: [ 390.097004] EIP: [<fa9b8d56>] assfail+0x26/0x30 [xfs] SS:ESP 0068:f1d67dc8
Apr 17 16:35:12 linux kernel: [ 390.097633] ---[ end trace 33a9795b638157b9 ]---

-Ben