Re: New CentOS4/RHEL4-compatible xfs module rpms

Re: New CentOS4/RHEL4-compatible xfs module rpms

[James Pearson]

>> That's odd.  You have the module on the server, exporting an xfs
>> filesystem, and you're getting permission denied on the client?
> 
> Yep. And rmmod'ing the updated XFS module and insmod'ing the older  
> module makes it work again.

The 'stock' RHEL4/CentOS4 kernels don't have xfs modules - so, I guess 
you have rebuilt your kernel with the XFS code that is there by default?

If this is the case, then this _may be_ the cause of the problem ... the 
updated xfs module code uses any existing XFS configs in the kernel you 
are building against - the Makefile states:

# Set up our config.
#
# If the kernel already has an XFS config, use it.
# Else if config.xfs is here, use it for our config.  Otherwise,
# Else default to only CONFIG_XFS_FS=m (simplest config)

The problem is that the 'stock' 2.6.9 kernel doesn't define (or use) 
CONFIG_XFS_EXPORT - but the updated xfs module code requires this to 
allow NFS exports of a XFS file system ...

So my guess is that your re-built updated xfs modules don't use
CONFIG_XFS_EXPORT

I guess with a bit of hacking to the Makefile, you could force 
'CONFIG_XFS_EXPORT=y' to be added - you might even be able to do this 
via the rpmbuild command line ... although I don't know how.

James Pearson

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Eric Sandeen]

James Pearson wrote:
>>> That's odd.  You have the module on the server, exporting an xfs
>>> filesystem, and you're getting permission denied on the client?
>> Yep. And rmmod'ing the updated XFS module and insmod'ing the older  
>> module makes it work again.
> 
> The 'stock' RHEL4/CentOS4 kernels don't have xfs modules - so, I guess 
> you have rebuilt your kernel with the XFS code that is there by default?
> 
> If this is the case, then this _may be_ the cause of the problem ... the 
> updated xfs module code uses any existing XFS configs in the kernel you 
> are building against - the Makefile states:
> 
> # Set up our config.
> #
> # If the kernel already has an XFS config, use it.
> # Else if config.xfs is here, use it for our config.  Otherwise,
> # Else default to only CONFIG_XFS_FS=m (simplest config)
> 
> The problem is that the 'stock' 2.6.9 kernel doesn't define (or use) 
> CONFIG_XFS_EXPORT - but the updated xfs module code requires this to 
> allow NFS exports of a XFS file system ...
> 
> So my guess is that your re-built updated xfs modules don't use
> CONFIG_XFS_EXPORT
> 
> I guess with a bit of hacking to the Makefile, you could force 
> 'CONFIG_XFS_EXPORT=y' to be added - you might even be able to do this 
> via the rpmbuild command line ... although I don't know how.

Ah, that may well be it.  yeah, the rpm needs to set its own config
options since the centos kernel has no xfs config.  (Sorry, I just
haven't had time to look into it yet) - but I bet you're right that
setting the config option in the makefile in the rpm will fix things
right up!

-Eric

New CentOS4/RHEL4-compatible xfs module rpms

[Eric Sandeen]

http://sandeen.net/rhel4_xfs/kernel-module-xfs-2.6.9-42.0.2.EL-0.2-1.src.rpm

rpmbuild --rebuild --target i686 
kernel-module-xfs-2.6.9-42.0.2.EL-0.2-1.src.rpm

will build against the currently running kernel, or

rpmbuild --rebuild --target i686 --define "kernel_topdir 
/lib/modules/2.6.9-42.0.2.EL/build" 
kernel-module-xfs-2.6.9-42.0.2.EL-0.2-1.src.rpm

will build against what is defined in kernel_topdir

you need matching kernel & kernel-devel rpms installed to build.

Changelog: mostly pulling in fixes sgi sent for sles9, plus specfile 
cleanups.

[root@sandeen rhel4_xfs]# rpm -qp --changelog 
kernel-module-xfs-2.6.9-42.0.2.EL-0.2-1.src.rpm
* Mon Nov 13 2006 - sandeen-centos@sandeen.net

- removed xfs_direct_io_locking.patch, RHEL4U4 has the fix now.
- Update to xfs codebase from SLES9_SP3_BRANCH_20061107171129
- xfs-kern-26347a-fix-race-on-link: fix race on link
   (191713, SGI:PV953287).
- xfs-kern-26040a-do-not-dirty-inode-being-freed:
   Don't dirty the inode if it being freed in xfs_iunpin (179117,
   SGI:PV952967).
- xfs-kern-25687a-sles9sp3-iunpin-reclaim-fix: Fix
   an inode use-after-free during an unpin (SGI:PV946321, 142533).
- xfs-kern-930841-default_acl_enospc_fix:
   Default acl ENOSPC fix (133990, SGI:PV930841).
- xfs-kern-25238a-quota-trans-diag: quota trans diag
   (131262, SGI:PV931456).
- xfs-ftruncate-stale-data: XFS ftruncate() bug
   could expose stale data (151055).
- xfs-log-diag: log_runout_diagnostics (131262,
   SGI:PV947110).
- xfs-kern-202363a-fix-xfs_finish_reclaim_all-umount-deadlock:
   fix xfs_finish_reclaim_all umount deadlock (132358,
   SGI:PV943821).
- clean up specfile:
   default to building against running kernel
   try to match fedora conventions slightly better
   re-enable debuginfo pkg building, module stripping
   fix recognition of root on xfs, (grep -i XFS)
   remove unused -source subpackage

-Eric

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Geir A. Myrestrand]

Eric Sandeen wrote:
> http://sandeen.net/rhel4_xfs/kernel-module-xfs-2.6.9-42.0.2.EL-0.2-1.src.rpm 
> 
> 
> rpmbuild --rebuild --target i686 
> kernel-module-xfs-2.6.9-42.0.2.EL-0.2-1.src.rpm
> 
> will build against the currently running kernel, or
> 
> rpmbuild --rebuild --target i686 --define "kernel_topdir 
> /lib/modules/2.6.9-42.0.2.EL/build" 
> kernel-module-xfs-2.6.9-42.0.2.EL-0.2-1.src.rpm
> 
> will build against what is defined in kernel_topdir
> 
> you need matching kernel & kernel-devel rpms installed to build.
> 

I built this for RHELAS4U2 x86_64 with the 2.6.9-22.ELsmp kernel.

I did enable the xfs_direct_io_locking.patch (see inside 
kernel-module-xfs.spec for details).

However, I run into issues with xfs_freeze as it often locks up when I 
try to freeze a file system where there is I/O activity. Sometimes it 
happen on the first xfs_freeze invocation to freeze the file system, 
other times I have to unfreeze and then it happens on the second time I 
freeze. xfs_freeze never returns when this happens.

Looks like xfs_io get stuck --see partial output from `ps auxf`:

strace -ff -o freeze.txt xfs_freeze -f /mnt/xfs
  \_ /bin/sh -f /usr/sbin/xfs_freeze -f /mnt/xfs
      \_ /usr/sbin/xfs_io -r -p xfs_freeze -x -c freeze /mnt/xfs

Anyone else encountering this issue?

-- 

Geir A. Myrestrand

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Eric Sandeen]

Geir A. Myrestrand wrote:

> However, I run into issues with xfs_freeze as it often locks up when I 
> try to freeze a file system where there is I/O activity. Sometimes it 
> happen on the first xfs_freeze invocation to freeze the file system, 
> other times I have to unfreeze and then it happens on the second time I 
> freeze. xfs_freeze never returns when this happens.
> 
> Looks like xfs_io get stuck --see partial output from `ps auxf`:
> 
> strace -ff -o freeze.txt xfs_freeze -f /mnt/xfs
>   \_ /bin/sh -f /usr/sbin/xfs_freeze -f /mnt/xfs
>       \_ /usr/sbin/xfs_io -r -p xfs_freeze -x -c freeze /mnt/xfs
> 
> Anyone else encountering this issue?
> 

Did you have this problem on the previous version?

You might do sysrq-t (echo t > /proc/sysrq-trigger) and see where the
thread is stuck.

-Eric

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Geir A. Myrestrand]

Eric Sandeen wrote:
> Geir A. Myrestrand wrote:
> 
>> However, I run into issues with xfs_freeze as it often locks up when I 
>> try to freeze a file system where there is I/O activity. Sometimes it 
>> happen on the first xfs_freeze invocation to freeze the file system, 
>> other times I have to unfreeze and then it happens on the second time I 
>> freeze. xfs_freeze never returns when this happens.
>>
>> Looks like xfs_io get stuck --see partial output from `ps auxf`:
>>
>> strace -ff -o freeze.txt xfs_freeze -f /mnt/xfs
>>   \_ /bin/sh -f /usr/sbin/xfs_freeze -f /mnt/xfs
>>       \_ /usr/sbin/xfs_io -r -p xfs_freeze -x -c freeze /mnt/xfs
>>
>> Anyone else encountering this issue?
>>
> 
> Did you have this problem on the previous version?
> 
> You might do sysrq-t (echo t > /proc/sysrq-trigger) and see where the
> thread is stuck.

I reproduced it and dumped the task list.

The user-mode XFS processes were:

root      2816  0.0  0.1  2812  628 pts/3    S+   12:58   0:00  | 
\_ strace -ff -o freeze.txt xfs_freeze -f /mnt/xfs
root      2847  0.0  0.2 52752 1048 pts/3    T+   12:58   0:00  | 
     \_ /bin/sh -f /usr/sbin/xfs_freeze -f /mnt/xfs
root      2891  0.0  0.1 49976  600 pts/3    D+   12:58   0:00  | 
         \_ /usr/sbin/xfs_io -r -p xfs_freeze -x -c freeze /mnt/xfs

My I/O generating processes were:

root      3832  1.9  0.2 59644 1376 pts/1    S+   12:57   0:17  | 
\_ /usr/bin/perl ./test.pl
root      2992  0.0  0.0 49912  404 pts/1    D+   12:58   0:00  | 
     \_ touch 857

Here are some relevant information from /var/log/messages (I stuck in a 
divider when I pasted in sections that weren't just right after the 
previous section):


pdflush       D ffffffff8014a190     0    48      6            50    47 
(L-TLB)
000001001f99dcb8 0000000000000046 0000000080134722 ffffffffa0054a51
        000000001f99dc30 0000000000000000 000001001fa2e310 0000000000000000
        000001001f9737f0 00000000000000d9
Call Trace:<ffffffffa0054a51>{:ext3:ext3_ordered_writepage+0}
        <ffffffffa01d96b5>{:xfs:xlog_state_sync_all+456} 
<ffffffffa01f4b9c>{:xfs:pagebuf_rele+54}
        <ffffffff8014a190>{keventd_create_kthread+0} 
<ffffffff80302637>{__down+147}
        <ffffffff80132e8d>{default_wake_function+0} 
<ffffffff80303c13>{__down_failed+53}
        <ffffffffa01e59da>{:xfs:xfs_sync+0} 
<ffffffffa01f64ad>{:xfs:.text.lock.xfs_buf+15}
        <ffffffffa01df1fa>{:xfs:xfs_getsb+37} 
<ffffffffa01e5817>{:xfs:xfs_syncsub+2507}
        <ffffffff8014a190>{keventd_create_kthread+0} 
<ffffffff8010f459>{__down_trylock+68}
        <ffffffff8014a190>{keventd_create_kthread+0} 
<ffffffffa01f347a>{:xfs:linvfs_write_super+33}
        <ffffffff8017ca9e>{sync_supers+167} 
<ffffffff8015d8e8>{wb_kupdate+36}
        <ffffffff8015e38c>{pdflush+323} <ffffffff8015d8c4>{wb_kupdate+0}
        <ffffffff8015e249>{pdflush+0} <ffffffff8014a167>{kthread+200}
        <ffffffff80110ca3>{child_rip+8} 
<ffffffff8014a190>{keventd_create_kthread+0}
        <ffffffff8014a09f>{kthread+0} <ffffffff80110c9b>{child_rip+0}

----------------------------------------------------------------------------

xfs           S 0000000000000006     0  2723      1          2742  2697 
(NOTLB)
00000100156e9d78 0000000000000002 00000100156e9d98 0000010000000074
        000001001eb87880 000001001eb878f0 000000d000000000 0000000100000246
        00000100154cc030 0000000000005f88
Call Trace:<ffffffff8013ea74>{__mod_timer+293} 
<ffffffff803039b3>{schedule_timeout+244}
        <ffffffff8013f49e>{process_timeout+0} 
<ffffffff8018902b>{do_select+939}
        <ffffffff80188bc5>{__pollwait+0} <ffffffff801893aa>{sys_select+820}
        <ffffffff801908fc>{dnotify_parent+34} 
<ffffffff80110052>{system_call+126}

----------------------------------------------------------------------------

xfslogd/0     S ffffffffa01f54b0     0  3818      6          3819  1981 
(L-TLB)
00000100104e5e68 0000000000000046 0000000000000283 0000010011a268e0
        ffffffffa01f54b0 0000000000000246 ffffffff80303cc1 00000000a01bb7a8
        0000010011d377f0 000000000000023e
Call Trace:<ffffffffa01f54b0>{:xfs:pagebuf_iodone_work+0} 
<ffffffff80303cc1>{__up_wakeup+53}
        <ffffffffa01f54b0>{:xfs:pagebuf_iodone_work+0} 
<ffffffff80146431>{worker_thread+226}
        <ffffffff80132e8d>{default_wake_function+0} 
<ffffffff80132e8d>{default_wake_function+0}
        <ffffffff8014a190>{keventd_create_kthread+0} 
<ffffffff8014634f>{worker_thread+0}
        <ffffffff8014a190>{keventd_create_kthread+0} 
<ffffffff8014a167>{kthread+200}
        <ffffffff80110ca3>{child_rip+8} 
<ffffffff8014a190>{keventd_create_kthread+0}
        <ffffffff8014a09f>{kthread+0} <ffffffff80110c9b>{child_rip+0}

xfslogd/1     S ffffffffa01f54b0     0  3819      6          3820  3818 
(L-TLB)
00000100105d7e68 0000000000000046 0000000000000287 ffffffffa01e2960
        000001000e48eb20 000001000e48eb20 0000010012d7e260 0000000100000246
        000001001053f7f0 000000000000022f
Call Trace:<ffffffffa01e2960>{:xfs:xfs_trans_delete_ail+45} 
<ffffffffa01f54b0>{:xfs:pagebuf_iodone_work+0}
        <ffffffff80146431>{worker_thread+226} 
<ffffffff80132e8d>{default_wake_function+0}
        <ffffffff80132e8d>{default_wake_function+0} 
<ffffffff8014a190>{keventd_create_kthread+0}
        <ffffffff8014634f>{worker_thread+0} 
<ffffffff8014a190>{keventd_create_kthread+0}
        <ffffffff8014a167>{kthread+200} <ffffffff80110ca3>{child_rip+8}
        <ffffffff8014a190>{keventd_create_kthread+0} 
<ffffffff8014a09f>{kthread+0}
        <ffffffff80110c9b>{child_rip+0}
xfsdatad/0    S 00000100164aa9c0     0  3820      6          3821  3819 
(L-TLB)
00000100104e1e68 0000000000000046 000000000032a7bd 000000190000006a
        000001001053e7f0 000000000000006a 0000010001707840 0000000000187397
        000001001053f030 0000000000000d8a
Call Trace:<ffffffff8014a190>{keventd_create_kthread+0} 
<ffffffff8014634f>{worker_thread+0}
        <ffffffff80146431>{worker_thread+226} 
<ffffffff80132e8d>{default_wake_function+0}
        <ffffffff80132e8d>{default_wake_function+0} 
<ffffffff8014a190>{keventd_create_kthread+0}
        <ffffffff8014634f>{worker_thread+0} 
<ffffffff8014a190>{keventd_create_kthread+0}
        <ffffffff8014a167>{kthread+200} <ffffffff80110ca3>{child_rip+8}
        <ffffffff8014a190>{keventd_create_kthread+0} 
<ffffffff8014a09f>{kthread+0}
        <ffffffff80110c9b>{child_rip+0}
xfsdatad/1    S 00000100164aaa40     0  3821      6                3820 
(L-TLB)
00000100104e3e68 0000000000000046 000000260000000a 0000001900000074
        000001001f9bd030 0000000000000074 000001000170f840 000000010032ba4b
        000001001053e7f0 0000000000000b24
Call Trace:<ffffffff8014a190>{keventd_create_kthread+0} 
<ffffffff8014634f>{worker_thread+0}
        <ffffffff80146431>{worker_thread+226} 
<ffffffff80132e8d>{default_wake_function+0}
        <ffffffff80132e8d>{default_wake_function+0} 
<ffffffff8014a190>{keventd_create_kthread+0}
        <ffffffff8014634f>{worker_thread+0} 
<ffffffff8014a190>{keventd_create_kthread+0}
        <ffffffff8014a167>{kthread+200} <ffffffff80110ca3>{child_rip+8}
        <ffffffff8014a190>{keventd_create_kthread+0} 
<ffffffff8014a09f>{kthread+0}
        <ffffffff80110c9b>{child_rip+0}
xfsbufd       S 000000010027d0c0     0  3822      1          3827  3815 
(L-TLB)
00000100111fdea8 0000000000000046 ffffffff803d9920 0000001900000073
        000001001f9bd030 0000000000000073 0000010001707840 0000000080138348
        000001001053e030 000000000000019b
Call Trace:<ffffffff8013ea74>{__mod_timer+293} 
<ffffffff803039b3>{schedule_timeout+244}
        <ffffffff8013f49e>{process_timeout+0} 
<ffffffffa01f62db>{:xfs:xfsbufd+172}
        <ffffffff80110ca3>{child_rip+8} 
<ffffffff8011de14>{flat_send_IPI_mask+0}
        <ffffffffa01f622f>{:xfs:xfsbufd+0} <ffffffff80110c9b>{child_rip+0}

xfssyncd      S ffffffffa01f36b9     0  3827      1                3822 
(L-TLB)
0000010011f0fea8 0000000000000046 ffffffffa01f36b9 0000000000000202
        ffffffff8010f459 00000100126e2d40 000001001d5e3980 0000000100000000
        000001001e4dd7f0 00000000000025eb
Call Trace:<ffffffffa01f36b9>{:xfs:linvfs_fill_super+0} 
<ffffffff8010f459>{__down_trylock+68}
        <ffffffff8013ea74>{__mod_timer+293} 
<ffffffffa01f36b9>{:xfs:linvfs_fill_super+0}
        <ffffffff803039b3>{schedule_timeout+244} 
<ffffffff8013f49e>{process_timeout+0}
        <ffffffffa01f323c>{:xfs:xfssyncd+120} 
<ffffffff80110ca3>{child_rip+8}
        <ffffffffa01f36b9>{:xfs:linvfs_fill_super+0} 
<ffffffff801cc053>{dummy_d_instantiate+0}
        <ffffffffa01f31c4>{:xfs:xfssyncd+0} <ffffffff80110c9b>{child_rip+0}

----------------------------------------------------------------------------

xfs_freeze    t 00000000006c51e0     0  2847   2816          2891 
(NOTLB)
00000100074d7e78 0000000000000002 0000010015654030 00000100111fec40
        0000000000000206 ffffffff801410a6 0000000000000011 0000000000040004
        0000010015654030 0000000000001a97
Call Trace:<ffffffff801410a6>{do_notify_parent_cldstop+243} 
<ffffffff801418d8>{ptrace_stop+315}
        <ffffffff801419c4>{ptrace_notify+139} 
<ffffffff8011419b>{syscall_trace+37}
        <ffffffff8011420f>{syscall_trace_enter+43} 
<ffffffff80110156>{tracesys+113}

xfs_io        D 00000100146af380     0  2891   2816                2847 
(NOTLB)
00000100075c9af8 0000000000000002 000001000a415b80 0000000000000064
        000000000007d000 ffffffff802499ca ffffffff80134722 00000001075c9a90
        000001000e83f7f0 0000000001a7ca67
Call Trace:<ffffffff802499ca>{generic_make_request+355} 
<ffffffff80134722>{autoremove_wake_function+0}
        <ffffffff80302637>{__down+147} 
<ffffffff80132e8d>{default_wake_function+0}
        <ffffffff80303c13>{__down_failed+53} 
<ffffffffa01f2a45>{:xfs:xfs_bdstrat_cb+0}
        <ffffffffa01f64ad>{:xfs:.text.lock.xfs_buf+15} 
<ffffffffa01f60f6>{:xfs:xfs_flush_buftarg+213}
        <ffffffffa01e59af>{:xfs:xfs_quiesce_fs+69} 
<ffffffffa01f34c1>{:xfs:linvfs_sync_super+65}
        <ffffffff80178367>{freeze_bdev+215} 
<ffffffffa01f0cd1>{:xfs:xfs_ioctl+5161}
        <ffffffff801588f0>{find_get_page+65} 
<ffffffff801593d2>{filemap_nopage+378}
        <ffffffff80131c6d>{finish_task_switch+55} 
<ffffffff80303358>{thread_return+42}
        <ffffffff8014191f>{ptrace_stop+386} 
<ffffffff801419e0>{ptrace_notify+167}
        <ffffffffa01ef380>{:xfs:linvfs_ioctl+112} 
<ffffffff80188631>{sys_ioctl+853}
        <ffffffff801101b6>{tracesys+209}
touch         D 0000000000239000     0  2992   3832 
(NOTLB)
0000010008dd76e8 0000000000000006 0000000000000001 0000000000000001
        0000000000000016 ffffffff80131931 0000000100000000 0000000000000003
        000001001ed54030 00000000000078de
Call Trace:<ffffffff80131931>{try_to_wake_up+734} 
<ffffffffa01f4dc2>{:xfs:_pagebuf_find+327}
        <ffffffff80302637>{__down+147} 
<ffffffff80132e8d>{default_wake_function+0}
        <ffffffff80303c13>{__down_failed+53} 
<ffffffffa01f64ad>{:xfs:.text.lock.xfs_buf+15}
        <ffffffffa01f4de1>{:xfs:_pagebuf_find+358} 
<ffffffffa01f5065>{:xfs:xfs_buf_get_flags+100}
        <ffffffffa01b1be0>{:xfs:xfs_bmap_search_extents+92}
        <ffffffffa01f5b01>{:xfs:xfs_buf_read_flags+16} 
<ffffffffa01e30b3>{:xfs:xfs_trans_read_buf+428}
        <ffffffffa01bc37c>{:xfs:xfs_da_do_buf+984} 
<ffffffffa01bc605>{:xfs:xfs_da_read_buf+21}
        <ffffffffa01bc87b>{:xfs:xfs_da_node_lookup_int+145}
        <ffffffffa01bc87b>{:xfs:xfs_da_node_lookup_int+145}
        <ffffffffa01c58a2>{:xfs:xfs_dir2_node_addname+83} 
<ffffffffa01d26bc>{:xfs:xfs_ichgtime+93}
        <ffffffff8018f975>{wake_up_inode+6} 
<ffffffffa01b286b>{:xfs:xfs_bmap_last_offset+179}
        <ffffffffa01c056e>{:xfs:xfs_dir2_createname+283} 
<ffffffffa01e8818>{:xfs:xfs_create+992}
        <ffffffff801cbecc>{dummy_inode_permission+0} 
<ffffffffa01f0fb7>{:xfs:linvfs_mknod+453}
        <ffffffffa01bbf7c>{:xfs:xfs_da_brelse+116} 
<ffffffffa01c6167>{:xfs:xfs_dir2_node_lookup+170}
        <ffffffffa01c042d>{:xfs:xfs_dir2_lookup+248} 
<ffffffff80184d4b>{vfs_create+214}
        <ffffffff80185179>{open_namei+430} <ffffffff80176558>{filp_open+39}
        <ffffffff801ea045>{strncpy_from_user+74} 
<ffffffff80176661>{get_unused_fd+230}
        <ffffffff80176750>{sys_open+57} <ffffffff80110052>{system_call+126}
-- 

Geir A. Myrestrand

Re: New CentOS4/RHEL4-compatible xfs module rpms

[David Chinner]

On Thu, Dec 07, 2006 at 01:17:32PM -0500, Geir A. Myrestrand wrote:
> Eric Sandeen wrote:
> >Geir A. Myrestrand wrote:
> >
> >>However, I run into issues with xfs_freeze as it often locks up when I 
> >>try to freeze a file system where there is I/O activity. Sometimes it 
> >>happen on the first xfs_freeze invocation to freeze the file system, 
> >>other times I have to unfreeze and then it happens on the second time I 
> >>freeze. xfs_freeze never returns when this happens.
> >>
> >>Looks like xfs_io get stuck --see partial output from `ps auxf`:
> >>
> >>strace -ff -o freeze.txt xfs_freeze -f /mnt/xfs
> >>  \_ /bin/sh -f /usr/sbin/xfs_freeze -f /mnt/xfs
> >>      \_ /usr/sbin/xfs_io -r -p xfs_freeze -x -c freeze /mnt/xfs
> >>
> >>Anyone else encountering this issue?

Yes, and I fixed it about a 2 weeks ago. It's an ABBA deadlock between
lookup of multiple, already dirty, metadata buffers and synchronous buftarg
flushing (that occurs when trying to freeze a filesystem)

<sigh>

I just went looking for the Take message in the archive, and it is not
there. I cc all my takes to xfs@oss.sgi.com, so I'm not sure why it
isn't in the archive....

http://oss.sgi.com/archives/xfs/2006-11/msg00291.html

Was a followup cleanup of a problem found during review of the
fix for the freeze problem.

The text of the take message fo rthe fix is:

Fix a synchronous buftarg flush deadlock when freezing.

At the last stage of a freeze, we flush the buftarg synchronously
over and over again until it succeeds twice without skipping
any buffers.

The delwri list flush skips pinned buffers, but tries to flush
all others. It removes the buffers from the delwri list, then tries
to lock them one at a time as it traverses the list to issue
the I/O. It holds them locked until we issue all of the I/O
and then unlocks them once we've waited for it to complete.

The problem is that during a freeze, the filesystem may
still be doing stuff - like flushing delalloc data buffers -
in the background and hence we can be trying to lock buffers
that were on the delwri list at the same time.  Hence we can
get ABBA deadlocks between threads doing allocation and the
buftarg flush (freeze) thread.

Fix it by skipping locked (and pinned) buffers as we traverse the
delwri buffer list.

----

And the diff was:

http://oss.sgi.com/cgi-bin/cvsweb.cgi/linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.c.diff?r1=1.229;r2=1.230

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Geir A. Myrestrand]

[-- Attachment #1: Type: text/plain, Size: 1906 bytes --]

David Chinner wrote:
> On Thu, Dec 07, 2006 at 01:17:32PM -0500, Geir A. Myrestrand wrote:
>>> Geir A. Myrestrand wrote:
>>>
>>>> However, I run into issues with xfs_freeze as it often locks up when I 
>>>> try to freeze a file system where there is I/O activity. Sometimes it 
>>>> happen on the first xfs_freeze invocation to freeze the file system, 
>>>> other times I have to unfreeze and then it happens on the second time I 
>>>> freeze. xfs_freeze never returns when this happens.
>>>>
>>>> Looks like xfs_io get stuck --see partial output from `ps auxf`:
>>>>
>>>> strace -ff -o freeze.txt xfs_freeze -f /mnt/xfs
>>>>  \_ /bin/sh -f /usr/sbin/xfs_freeze -f /mnt/xfs
>>>>      \_ /usr/sbin/xfs_io -r -p xfs_freeze -x -c freeze /mnt/xfs
>>>>
>>>> Anyone else encountering this issue?
> 
> Yes, and I fixed it about a 2 weeks ago. It's an ABBA deadlock between
> lookup of multiple, already dirty, metadata buffers and synchronous buftarg
> flushing (that occurs when trying to freeze a filesystem)
> 
> <sigh>

That is awesome news, mate!  :-)

> The problem is that during a freeze, the filesystem may
> still be doing stuff - like flushing delalloc data buffers -
> in the background and hence we can be trying to lock buffers
> that were on the delwri list at the same time.  Hence we can
> get ABBA deadlocks between threads doing allocation and the
> buftarg flush (freeze) thread.

That sounds like an accurate description of my test environment. I bet 
this is the issue...

> Fix it by skipping locked (and pinned) buffers as we traverse the
> delwri buffer list.

Good to know that you're one step ahead!

> And the diff was:
> 
> http://oss.sgi.com/cgi-bin/cvsweb.cgi/linux-2.6-xfs/fs/xfs/linux-2.6/xfs_buf.c.diff?r1=1.229;r2=1.230

Excellent. I will try this tomorrow (it's late in the evening here in 
New York now). I'll let you know how it works out.

Thanks!


-- 

Geir A. Myrestrand

[-- Attachment #2: geir.myrestrand.vcf --]
[-- Type: text/x-vcard, Size: 446 bytes --]

begin:vcard
fn:Geir A. Myrestrand
n:Myrestrand;Geir A.
org:FalconStor Software, Inc.
adr:Suite 2S01;;2 Huntington Quadrangle;Melville;NY;11747;U.S.A.
email;internet:geir.myrestrand@falconstor.com
title:Senior Software Engineer
tel;work:(631) 773-5842
tel;cell:(631) 747-5049
note;quoted-printable:Skype: gmyrestrand=0D=0A=
	MSN Messenger: gmyrestrand@hotmail.com=0D=0A=
	
x-mozilla-html:FALSE
url:http://www.falconstor.com
version:2.1
end:vcard

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Geir A. Myrestrand]

Eric Sandeen wrote:
> Did you have this problem on the previous version?

Not sure if this occur with the older XFS modules, because I haven't 
tested it.

-- 

Geir A. Myrestrand

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Geir A. Myrestrand]

Eric Sandeen wrote:
> Geir A. Myrestrand wrote:
> 
>> However, I run into issues with xfs_freeze as it often locks up when I 
>> try to freeze a file system where there is I/O activity. Sometimes it 
>> happen on the first xfs_freeze invocation to freeze the file system, 
>> other times I have to unfreeze and then it happens on the second time I 
>> freeze. xfs_freeze never returns when this happens.
>>
> 
> Did you have this problem on the previous version?

I just tested with the old version, and I run into the same issue.

-- 

Geir A. Myrestrand

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Nathan Scott]

On Thu, 2006-12-07 at 15:51 -0500, Geir A. Myrestrand wrote:
> Eric Sandeen wrote:
> > Geir A. Myrestrand wrote:
> > 
> >> However, I run into issues with xfs_freeze as it often locks up when I 
> >> try to freeze a file system where there is I/O activity. Sometimes it 
> >> happen on the first xfs_freeze invocation to freeze the file system, 
> >> other times I have to unfreeze and then it happens on the second time I 
> >> freeze. xfs_freeze never returns when this happens.
> >>
> > 
> > Did you have this problem on the previous version?
> 
> I just tested with the old version, and I run into the same issue.
> 

Does it happen with mainline (current) kernels?  This is ringing some
distant bells, something to do with an atomic_inc being in the wrong
spot in [_]xfs_trans_alloc ... or maybe a wrong call to xfs_trans_alloc
instead of the '_' prefixed version - details are murky now though.  I
do remember fixing ... something in here, a few months back, that fix
may not be in Eric's codebase yet.

cheers.

-- 
Nathan

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Geir A. Myrestrand]

Nathan Scott wrote:
> Does it happen with mainline (current) kernels?  This is ringing some
> distant bells, something to do with an atomic_inc being in the wrong
> spot in [_]xfs_trans_alloc ... or maybe a wrong call to xfs_trans_alloc
> instead of the '_' prefixed version - details are murky now though.  I
> do remember fixing ... something in here, a few months back, that fix
> may not be in Eric's codebase yet.
> 
> cheers.

I've only tried this on RHELAS4 with the kernel shipped with Update 2. I 
have not tried to update the kernel, or install Update 4 or whatever is 
current now. My kernel is 2.6.9-22.ELsmp.

It wouldn't be easy for me to switch to a newer kernel, because it is 
not just a matter of my machine --we have a product built for this 
particular configuration. Switching to a new kernel would reset our QA 
efforts.

If this may be fixed in the XFS modules on the other hand, then it would 
be awesome if we could get it resolved with an updated XFS package for 
RHEL4. I would be more than willing to put in an extra effort and work 
with anyone on this.

-- 

Geir A. Myrestrand

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Nathan Scott]

On Thu, 2006-12-07 at 16:35 -0500, Geir A. Myrestrand wrote:
> 
> It wouldn't be easy for me to switch to a newer kernel, because it is 
> not just a matter of my machine --we have a product built for this 
> particular configuration. Switching to a new kernel would reset our 
> QA efforts.

You misunderstood me I think - I didn't suggest switching kernels, just
that you test out the latest.  If its OK, then its relatively easy to
search for a change that fixed it.  If its not OK, then theres a bug in
mainline which should get some attention too.

cheers.

-- 
Nathan

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Geir A. Myrestrand]

Nathan Scott wrote:
> On Thu, 2006-12-07 at 16:35 -0500, Geir A. Myrestrand wrote:
>> It wouldn't be easy for me to switch to a newer kernel, because it is 
>> not just a matter of my machine --we have a product built for this 
>> particular configuration. Switching to a new kernel would reset our 
>> QA efforts.
> 
> You misunderstood me I think - I didn't suggest switching kernels, just
> that you test out the latest.  If its OK, then its relatively easy to
> search for a change that fixed it.  If its not OK, then theres a bug in
> mainline which should get some attention too.

Sorry Nathan, but I'm not sure I understand what you refer to with "test 
out the latest" --are you referring to the kernel or XFS?

-- 

Geir A. Myrestrand

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Nathan Scott]

On Thu, 2006-12-07 at 16:51 -0500, Geir A. Myrestrand wrote:
> Sorry Nathan, but I'm not sure I understand what you refer to with
> "test out the latest" --are you referring to the kernel or XFS?

Yes. :)  Current mainline kernels, or CVS from oss.sgi.com would
have this fix I'm thinking of.

cheers.

-- 
Nathan

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Geir A. Myrestrand]

[-- Attachment #1: Type: text/plain, Size: 594 bytes --]

Nathan Scott wrote:
> On Thu, 2006-12-07 at 16:51 -0500, Geir A. Myrestrand wrote:
>> Sorry Nathan, but I'm not sure I understand what you refer to with
>> "test out the latest" --are you referring to the kernel or XFS?
> 
> Yes. :)  Current mainline kernels, or CVS from oss.sgi.com would
> have this fix I'm thinking of.

Thanks for the "clarification"... ;-)

I'll try to update the XFS code first, hopefully it won't break too many 
of the patches Eric had in his source RPM...but I will try to correct 
them if they fail to apply cleanly...

I'll keep you posted.

-- 

Geir A. Myrestrand

[-- Attachment #2: geir.myrestrand.vcf --]
[-- Type: text/x-vcard, Size: 446 bytes --]

begin:vcard
fn:Geir A. Myrestrand
n:Myrestrand;Geir A.
org:FalconStor Software, Inc.
adr:Suite 2S01;;2 Huntington Quadrangle;Melville;NY;11747;U.S.A.
email;internet:geir.myrestrand@falconstor.com
title:Senior Software Engineer
tel;work:(631) 773-5842
tel;cell:(631) 747-5049
note;quoted-printable:Skype: gmyrestrand=0D=0A=
	MSN Messenger: gmyrestrand@hotmail.com=0D=0A=
	
x-mozilla-html:FALSE
url:http://www.falconstor.com
version:2.1
end:vcard

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Eric Sandeen]

Geir A. Myrestrand wrote:
> Nathan Scott wrote:
>> On Thu, 2006-12-07 at 16:51 -0500, Geir A. Myrestrand wrote:
>>> Sorry Nathan, but I'm not sure I understand what you refer to with
>>> "test out the latest" --are you referring to the kernel or XFS?
>> Yes. :)  Current mainline kernels, or CVS from oss.sgi.com would
>> have this fix I'm thinking of.
> 
> Thanks for the "clarification"... ;-)
> 
> I'll try to update the XFS code first, hopefully it won't break too many 
> of the patches Eric had in his source RPM...but I will try to correct 
> them if they fail to apply cleanly...
> 
> I'll keep you posted.
> 

I would just get kernel-2.6.19 and run the test, see if it's fixed
upstream, don't worry about merging & munging xfs code from one place o
another...

-Eric

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Geir A. Myrestrand]

Eric Sandeen wrote:
> 
> I would just get kernel-2.6.19 and run the test, see if it's fixed
> upstream, don't worry about merging & munging xfs code from one place o
> another...
> 
> -Eric

I have SLES 10 (SUSE Linux Enterprise Server) installed on the same box, 
  the kernel is 2.6.16.21-0.8-smp. Would it be worth checking that one 
first?

-- 

Geir A. Myrestrand

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Eric Sandeen]

Geir A. Myrestrand wrote:
> Eric Sandeen wrote:
>> I would just get kernel-2.6.19 and run the test, see if it's fixed
>> upstream, don't worry about merging & munging xfs code from one place o
>> another...
>>
>> -Eric
> 
> I have SLES 10 (SUSE Linux Enterprise Server) installed on the same box, 
>   the kernel is 2.6.16.21-0.8-smp. Would it be worth checking that one 
> first?
> 

Sure.

-Eric

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Geir A. Myrestrand]

Eric Sandeen wrote:
> Geir A. Myrestrand wrote:
>> Eric Sandeen wrote:
>>> I would just get kernel-2.6.19 and run the test, see if it's fixed
>>> upstream, don't worry about merging & munging xfs code from one place o
>>> another...
>>>
>>> -Eric
>> I have SLES 10 (SUSE Linux Enterprise Server) installed on the same box, 
>>   the kernel is 2.6.16.21-0.8-smp. Would it be worth checking that one 
>> first?
>>
> 
> Sure.
> 
> -Eric

O'boy, looks like it happens on SLES10 as well. It froze on the first 
XFS freeze call, it does not return.

strace -ff -o freeze-1.txt xfs_freeze -f /mnt/xfs
  \_ /bin/sh -f /usr/sbin/xfs_freeze -f /mnt/xfs
      \_ /usr/sbin/xfs_io -r -p xfs_freeze -x -c freeze /mnt/xfs

A sync call hangs too. Looks like I need to test an even newer kernel...


-- 

Geir A. Myrestrand

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Eric Sandeen]

Geir A. Myrestrand wrote:

> O'boy, looks like it happens on SLES10 as well. It froze on the first 
> XFS freeze call, it does not return.
> 
> strace -ff -o freeze-1.txt xfs_freeze -f /mnt/xfs
>   \_ /bin/sh -f /usr/sbin/xfs_freeze -f /mnt/xfs
>       \_ /usr/sbin/xfs_io -r -p xfs_freeze -x -c freeze /mnt/xfs
> 
> A sync call hangs too. Looks like I need to test an even newer kernel...

You don't have multiple concurrent freezes happening do you....

-Eric

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Geir A. Myrestrand]

Eric Sandeen wrote:
> 
> You don't have multiple concurrent freezes happening do you....
> 
> -Eric
> 

Nope, I don't need to. Now it happens on the first call to xfs_freeze. 
Sometimes in the past I did freeze, unfreeze, and then freeze before it 
happened, but this was still serialized from the same terminal window.

Oh, could strace be the reason as to why it now always happens on the 
first invocation of the freeze command? Maybe it affects the wait 
calls... Well, without strace it would probably just happen on the 
second invocation of freeze (after unfreeze) like it did on RHELAS4U2...

My test case is very simple, the only thing special is that I simplified 
my test case so much that I end up using a file as my disk (setup as a 
loopback device). However, the problem was initially discovered using a 
real disk, so I don't think that is the problem.

If you're interested I can forward the test case involving probably less 
than 10 commands and a simple Perl script of maybe 20 lines.

-- 

Geir A. Myrestrand

Re: New CentOS4/RHEL4-compatible xfs module rpms

[James Braid]

> http://sandeen.net/rhel4_xfs/kernel-module-xfs-2.6.9-42.0.2.EL-0.2-1.src.rpm
>

Has anyone had any luck with getting NFS exports to work with these
updated modules? I'm not having any luck, and just getting "Permission
denied" errors when trying to mount an NFS exported filesystem using these
modules. Works fine with the stock modules from a rebuilt RHEL4 kernel.

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Eric Sandeen]

James Braid wrote:
>> http://sandeen.net/rhel4_xfs/kernel-module-xfs-2.6.9-42.0.2.EL-0.2-1.src.rpm
>>
> 
> Has anyone had any luck with getting NFS exports to work with these
> updated modules? I'm not having any luck, and just getting "Permission
> denied" errors when trying to mount an NFS exported filesystem using these
> modules. Works fine with the stock modules from a rebuilt RHEL4 kernel.

That's odd.  You have the module on the server, exporting an xfs
filesystem, and you're getting permission denied on the client?

-Eric

Re: New CentOS4/RHEL4-compatible xfs module rpms

[James Braid]

On 19 Jul 2007, at 16:15, Eric Sandeen wrote:
> That's odd.  You have the module on the server, exporting an xfs
> filesystem, and you're getting permission denied on the client?

Yep. And rmmod'ing the updated XFS module and insmod'ing the older  
module makes it work again.

(yeah, in an ideal world we'd just run the latest kernel, but we're  
stuck on RHEL4 for a bunch of reasons and XFS is so much better than  
ext3 for our purposes, even with the old crufty version in the RHEL4  
kernel)

Re: New CentOS4/RHEL4-compatible xfs module rpms

[Eric Sandeen]

James Braid wrote:
> On 19 Jul 2007, at 16:15, Eric Sandeen wrote:
>> That's odd.  You have the module on the server, exporting an xfs
>> filesystem, and you're getting permission denied on the client?
> 
> Yep. And rmmod'ing the updated XFS module and insmod'ing the older  
> module makes it work again.

Interesting - what are the exact versions of "updated" and "older" that
you are using?  I'll look at the diff...

Thanks,
-Eric

> (yeah, in an ideal world we'd just run the latest kernel, but we're  
> stuck on RHEL4 for a bunch of reasons and XFS is so much better than  
> ext3 for our purposes, even with the old crufty version in the RHEL4  
> kernel)
> 

Re: New CentOS4/RHEL4-compatible xfs module rpms

[James Braid]

On 21 Jul 2007, at 15:58, Eric Sandeen wrote:
> Interesting - what are the exact versions of "updated" and "older"  
> that
> you are using?  I'll look at the diff...

The updated one is the one from here http://sandeen.net/rhel4_xfs/ 
kernel-module-xfs-2.6.9-42.0.2.EL-0.2-1.src.rpm

The older one is the stock RHEL4U4 code (2.6.9-42.0.3 from memory)