Re: ext4+quota+nfs issue

Re: ext4+quota+nfs issue

[Jiri Kosina]

[ adding relevant CCs ]

On Wed, 9 Sep 2009, Pavol Cvengros wrote:

> Hi,
> 
> can somebody who is aware of ext4 and quota have a look on this one?
> 
> 
> Thanks,
> 
> P.
> 
> On 8. 9. 2009 7:04, Pavol Cvengros wrote:
> > Hello,
> > 
> > recently we have build and started to use raid storage with formatted
> > capacity of 4.5T (ext4 formatted, default params).
> > FS has quota turned on and is exported via NFS to nodes.
> > If we turn qouta on on this FS and are trying to use it over NFS we get the
> > following:
> > 
> > ------------[ cut here ]------------
> > WARNING: at fs/quota/dquot.c:964 dquot_claim_space+0x181/0x190()
> > Hardware name: S3210SH
> > Modules linked in: nfs fscache nfsd lockd auth_rpcgss exportfs sunrpc
> > coretemp hwmon ipmi_si ipmi_msghandler ehci_hcd sr_mod cdrom uhci_hcd floppy
> > usbcore i2c_i801 i2c_core processor 3w_9xxx button thermal
> > Pid: 268, comm: pdflush Tainted: G        W  2.6.30-gentoo-r3_host #1
> > Call Trace:
> >  [<ffffffff803151e1>] ? dquot_claim_space+0x181/0x190
> >  [<ffffffff80245c59>] ? warn_slowpath_common+0x89/0x100
> >  [<ffffffff803151e1>] ? dquot_claim_space+0x181/0x190
> >  [<ffffffff80367e83>] ? ext4_mb_mark_diskspace_used+0x423/0x440
> >  [<ffffffff8036c05f>] ? ext4_mb_new_blocks+0x2cf/0x460
> >  [<ffffffff80360a17>] ? ext4_ext_find_extent+0x307/0x330
> >  [<ffffffff80362508>] ? ext4_ext_get_blocks+0x578/0xfc0
> >  [<ffffffff8028e828>] ? __pagevec_free+0x48/0x70
> >  [<ffffffff803a1c65>] ? blk_rq_bio_prep+0x35/0x130
> >  [<ffffffff8034d310>] ? ext4_get_blocks_wrap+0x210/0x380
> >  [<ffffffff8034d8d8>] ? mpage_da_map_blocks+0xe8/0x750
> >  [<ffffffff80292cee>] ? pagevec_lookup_tag+0x2e/0x50
> >  [<ffffffff8029084c>] ? write_cache_pages+0x11c/0x400
> >  [<ffffffff8034e500>] ? __mpage_da_writepage+0x0/0x190
> >  [<ffffffff8034e269>] ? ext4_da_writepages+0x329/0x4b0
> >  [<ffffffff80290bd2>] ? do_writepages+0x32/0x70
> >  [<ffffffff802e4140>] ? __writeback_single_inode+0xb0/0x490
> >  [<ffffffff8023c753>] ? dequeue_entity+0x23/0x1c0
> >  [<ffffffff802e4b16>] ? generic_sync_sb_inodes+0x316/0x4f0
> >  [<ffffffff802e4f4e>] ? writeback_inodes+0x5e/0x110
> >  [<ffffffff80290e56>] ? wb_kupdate+0xc6/0x160
> >  [<ffffffff80292110>] ? pdflush+0x120/0x230
> >  [<ffffffff80290d90>] ? wb_kupdate+0x0/0x160
> >  [<ffffffff80291ff0>] ? pdflush+0x0/0x230
> >  [<ffffffff80261154>] ? kthread+0x64/0xc0
> >  [<ffffffff8020d13a>] ? child_rip+0xa/0x20
> >  [<ffffffff802610f0>] ? kthread+0x0/0xc0
> >  [<ffffffff8020d130>] ? child_rip+0x0/0x20
> > ---[ end trace cb54e6523e9ab60d ]---
> > 
> > fstab entry:
> > /dev/sdb1              /mnt/storage    ext4
> > noatime,nodiratime,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 
> > 
> > qith quotaoff on tihs FS, warnings stop.
> > 
> > Question is if it's safe to use quotas with this problem (warning) or not.
> > Can't afford data damage.
> > 
> > Thanks,
> > 
> > Pavol Cvengros
> > -- 
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
Jiri Kosina
SUSE Labs, Novell Inc.

Re: ext4+quota+nfs issue

[Jan Kara]

On Wed 09-09-09 11:42:03, Jiri Kosina wrote:
> [ adding relevant CCs ]
  Thanks.

> On Wed, 9 Sep 2009, Pavol Cvengros wrote:
> > > recently we have build and started to use raid storage with formatted
> > > capacity of 4.5T (ext4 formatted, default params).
> > > FS has quota turned on and is exported via NFS to nodes.
> > > If we turn qouta on on this FS and are trying to use it over NFS we get the
> > > following:
> > > 
> > > ------------[ cut here ]------------
> > > WARNING: at fs/quota/dquot.c:964 dquot_claim_space+0x181/0x190()
> > > Hardware name: S3210SH
> > > Modules linked in: nfs fscache nfsd lockd auth_rpcgss exportfs sunrpc
> > > coretemp hwmon ipmi_si ipmi_msghandler ehci_hcd sr_mod cdrom uhci_hcd floppy
> > > usbcore i2c_i801 i2c_core processor 3w_9xxx button thermal
> > > Pid: 268, comm: pdflush Tainted: G        W  2.6.30-gentoo-r3_host #1
> > > Call Trace:
> > >  [<ffffffff803151e1>] ? dquot_claim_space+0x181/0x190
> > >  [<ffffffff80245c59>] ? warn_slowpath_common+0x89/0x100
> > >  [<ffffffff803151e1>] ? dquot_claim_space+0x181/0x190
> > >  [<ffffffff80367e83>] ? ext4_mb_mark_diskspace_used+0x423/0x440
> > >  [<ffffffff8036c05f>] ? ext4_mb_new_blocks+0x2cf/0x460
> > >  [<ffffffff80360a17>] ? ext4_ext_find_extent+0x307/0x330
> > >  [<ffffffff80362508>] ? ext4_ext_get_blocks+0x578/0xfc0
> > >  [<ffffffff8028e828>] ? __pagevec_free+0x48/0x70
> > >  [<ffffffff803a1c65>] ? blk_rq_bio_prep+0x35/0x130
> > >  [<ffffffff8034d310>] ? ext4_get_blocks_wrap+0x210/0x380
> > >  [<ffffffff8034d8d8>] ? mpage_da_map_blocks+0xe8/0x750
> > >  [<ffffffff80292cee>] ? pagevec_lookup_tag+0x2e/0x50
> > >  [<ffffffff8029084c>] ? write_cache_pages+0x11c/0x400
> > >  [<ffffffff8034e500>] ? __mpage_da_writepage+0x0/0x190
> > >  [<ffffffff8034e269>] ? ext4_da_writepages+0x329/0x4b0
> > >  [<ffffffff80290bd2>] ? do_writepages+0x32/0x70
> > >  [<ffffffff802e4140>] ? __writeback_single_inode+0xb0/0x490
> > >  [<ffffffff8023c753>] ? dequeue_entity+0x23/0x1c0
> > >  [<ffffffff802e4b16>] ? generic_sync_sb_inodes+0x316/0x4f0
> > >  [<ffffffff802e4f4e>] ? writeback_inodes+0x5e/0x110
> > >  [<ffffffff80290e56>] ? wb_kupdate+0xc6/0x160
> > >  [<ffffffff80292110>] ? pdflush+0x120/0x230
> > >  [<ffffffff80290d90>] ? wb_kupdate+0x0/0x160
> > >  [<ffffffff80291ff0>] ? pdflush+0x0/0x230
> > >  [<ffffffff80261154>] ? kthread+0x64/0xc0
> > >  [<ffffffff8020d13a>] ? child_rip+0xa/0x20
> > >  [<ffffffff802610f0>] ? kthread+0x0/0xc0
> > >  [<ffffffff8020d130>] ? child_rip+0x0/0x20
> > > ---[ end trace cb54e6523e9ab60d ]---
> > > 
> > > fstab entry:
> > > /dev/sdb1              /mnt/storage    ext4
> > > noatime,nodiratime,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 
> > > 
> > > qith quotaoff on tihs FS, warnings stop.
> > > 
> > > Question is if it's safe to use quotas with this problem (warning) or not.
> > > Can't afford data damage.
  The warning definitely doesn't directly indicate a danger of data loss.
But there *is* some bug causing that ext4 asks for more quota space than
it has reserved.
  From your report it seems it is easily reproducible, right? Do you have
an easy way to trigger this warning - by some test script or so?

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: ext4+quota+nfs issue

[Pavol Cvengros]

On 9/9/2009 4:46 PM, Jan Kara wrote:
> On Wed 09-09-09 11:42:03, Jiri Kosina wrote:
>    
>> [ adding relevant CCs ]
>>      
>    Thanks.
>
>    
>> On Wed, 9 Sep 2009, Pavol Cvengros wrote:
>>      
>>>> recently we have build and started to use raid storage with formatted
>>>> capacity of 4.5T (ext4 formatted, default params).
>>>> FS has quota turned on and is exported via NFS to nodes.
>>>> If we turn qouta on on this FS and are trying to use it over NFS we get the
>>>> following:
>>>>
>>>> ------------[ cut here ]------------
>>>> WARNING: at fs/quota/dquot.c:964 dquot_claim_space+0x181/0x190()
>>>> Hardware name: S3210SH
>>>> Modules linked in: nfs fscache nfsd lockd auth_rpcgss exportfs sunrpc
>>>> coretemp hwmon ipmi_si ipmi_msghandler ehci_hcd sr_mod cdrom uhci_hcd floppy
>>>> usbcore i2c_i801 i2c_core processor 3w_9xxx button thermal
>>>> Pid: 268, comm: pdflush Tainted: G        W  2.6.30-gentoo-r3_host #1
>>>> Call Trace:
>>>>   [<ffffffff803151e1>] ? dquot_claim_space+0x181/0x190
>>>>   [<ffffffff80245c59>] ? warn_slowpath_common+0x89/0x100
>>>>   [<ffffffff803151e1>] ? dquot_claim_space+0x181/0x190
>>>>   [<ffffffff80367e83>] ? ext4_mb_mark_diskspace_used+0x423/0x440
>>>>   [<ffffffff8036c05f>] ? ext4_mb_new_blocks+0x2cf/0x460
>>>>   [<ffffffff80360a17>] ? ext4_ext_find_extent+0x307/0x330
>>>>   [<ffffffff80362508>] ? ext4_ext_get_blocks+0x578/0xfc0
>>>>   [<ffffffff8028e828>] ? __pagevec_free+0x48/0x70
>>>>   [<ffffffff803a1c65>] ? blk_rq_bio_prep+0x35/0x130
>>>>   [<ffffffff8034d310>] ? ext4_get_blocks_wrap+0x210/0x380
>>>>   [<ffffffff8034d8d8>] ? mpage_da_map_blocks+0xe8/0x750
>>>>   [<ffffffff80292cee>] ? pagevec_lookup_tag+0x2e/0x50
>>>>   [<ffffffff8029084c>] ? write_cache_pages+0x11c/0x400
>>>>   [<ffffffff8034e500>] ? __mpage_da_writepage+0x0/0x190
>>>>   [<ffffffff8034e269>] ? ext4_da_writepages+0x329/0x4b0
>>>>   [<ffffffff80290bd2>] ? do_writepages+0x32/0x70
>>>>   [<ffffffff802e4140>] ? __writeback_single_inode+0xb0/0x490
>>>>   [<ffffffff8023c753>] ? dequeue_entity+0x23/0x1c0
>>>>   [<ffffffff802e4b16>] ? generic_sync_sb_inodes+0x316/0x4f0
>>>>   [<ffffffff802e4f4e>] ? writeback_inodes+0x5e/0x110
>>>>   [<ffffffff80290e56>] ? wb_kupdate+0xc6/0x160
>>>>   [<ffffffff80292110>] ? pdflush+0x120/0x230
>>>>   [<ffffffff80290d90>] ? wb_kupdate+0x0/0x160
>>>>   [<ffffffff80291ff0>] ? pdflush+0x0/0x230
>>>>   [<ffffffff80261154>] ? kthread+0x64/0xc0
>>>>   [<ffffffff8020d13a>] ? child_rip+0xa/0x20
>>>>   [<ffffffff802610f0>] ? kthread+0x0/0xc0
>>>>   [<ffffffff8020d130>] ? child_rip+0x0/0x20
>>>> ---[ end trace cb54e6523e9ab60d ]---
>>>>
>>>> fstab entry:
>>>> /dev/sdb1              /mnt/storage    ext4
>>>> noatime,nodiratime,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0
>>>>
>>>> qith quotaoff on tihs FS, warnings stop.
>>>>
>>>> Question is if it's safe to use quotas with this problem (warning) or not.
>>>> Can't afford data damage.
>>>>          
>    The warning definitely doesn't directly indicate a danger of data loss.
> But there *is* some bug causing that ext4 asks for more quota space than
> it has reserved.
>    From your report it seems it is easily reproducible, right? Do you have
> an easy way to trigger this warning - by some test script or so?
>
> 								Honza
>    
really easy to reproduce, our storage is on live production so I can't 
experiment with kernel but it's enough to do quotaon and system will get 
"spammed" with these warning on every file access..

P.

Re: ext4+quota+nfs issue

[Eric Sandeen]

Jiri Kosina wrote:
> [ adding relevant CCs ]
> 
> On Wed, 9 Sep 2009, Pavol Cvengros wrote:
> 
>> Hi,
>>
>> can somebody who is aware of ext4 and quota have a look on this one?
>>

This was also just reported at:

https://bugzilla.redhat.com/show_bug.cgi?id=521914

-Eric

>> Thanks,
>>
>> P.
>>
>> On 8. 9. 2009 7:04, Pavol Cvengros wrote:
>>> Hello,
>>>
>>> recently we have build and started to use raid storage with formatted
>>> capacity of 4.5T (ext4 formatted, default params).
>>> FS has quota turned on and is exported via NFS to nodes.
>>> If we turn qouta on on this FS and are trying to use it over NFS we get the
>>> following:
>>>
>>> ------------[ cut here ]------------
>>> WARNING: at fs/quota/dquot.c:964 dquot_claim_space+0x181/0x190()
>>> Hardware name: S3210SH
>>> Modules linked in: nfs fscache nfsd lockd auth_rpcgss exportfs sunrpc
>>> coretemp hwmon ipmi_si ipmi_msghandler ehci_hcd sr_mod cdrom uhci_hcd floppy
>>> usbcore i2c_i801 i2c_core processor 3w_9xxx button thermal
>>> Pid: 268, comm: pdflush Tainted: G        W  2.6.30-gentoo-r3_host #1
>>> Call Trace:
>>>  [<ffffffff803151e1>] ? dquot_claim_space+0x181/0x190
>>>  [<ffffffff80245c59>] ? warn_slowpath_common+0x89/0x100
>>>  [<ffffffff803151e1>] ? dquot_claim_space+0x181/0x190
>>>  [<ffffffff80367e83>] ? ext4_mb_mark_diskspace_used+0x423/0x440
>>>  [<ffffffff8036c05f>] ? ext4_mb_new_blocks+0x2cf/0x460
>>>  [<ffffffff80360a17>] ? ext4_ext_find_extent+0x307/0x330
>>>  [<ffffffff80362508>] ? ext4_ext_get_blocks+0x578/0xfc0
>>>  [<ffffffff8028e828>] ? __pagevec_free+0x48/0x70
>>>  [<ffffffff803a1c65>] ? blk_rq_bio_prep+0x35/0x130
>>>  [<ffffffff8034d310>] ? ext4_get_blocks_wrap+0x210/0x380
>>>  [<ffffffff8034d8d8>] ? mpage_da_map_blocks+0xe8/0x750
>>>  [<ffffffff80292cee>] ? pagevec_lookup_tag+0x2e/0x50
>>>  [<ffffffff8029084c>] ? write_cache_pages+0x11c/0x400
>>>  [<ffffffff8034e500>] ? __mpage_da_writepage+0x0/0x190
>>>  [<ffffffff8034e269>] ? ext4_da_writepages+0x329/0x4b0
>>>  [<ffffffff80290bd2>] ? do_writepages+0x32/0x70
>>>  [<ffffffff802e4140>] ? __writeback_single_inode+0xb0/0x490
>>>  [<ffffffff8023c753>] ? dequeue_entity+0x23/0x1c0
>>>  [<ffffffff802e4b16>] ? generic_sync_sb_inodes+0x316/0x4f0
>>>  [<ffffffff802e4f4e>] ? writeback_inodes+0x5e/0x110
>>>  [<ffffffff80290e56>] ? wb_kupdate+0xc6/0x160
>>>  [<ffffffff80292110>] ? pdflush+0x120/0x230
>>>  [<ffffffff80290d90>] ? wb_kupdate+0x0/0x160
>>>  [<ffffffff80291ff0>] ? pdflush+0x0/0x230
>>>  [<ffffffff80261154>] ? kthread+0x64/0xc0
>>>  [<ffffffff8020d13a>] ? child_rip+0xa/0x20
>>>  [<ffffffff802610f0>] ? kthread+0x0/0xc0
>>>  [<ffffffff8020d130>] ? child_rip+0x0/0x20
>>> ---[ end trace cb54e6523e9ab60d ]---
>>>
>>> fstab entry:
>>> /dev/sdb1              /mnt/storage    ext4
>>> noatime,nodiratime,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 
>>>
>>> qith quotaoff on tihs FS, warnings stop.
>>>
>>> Question is if it's safe to use quotas with this problem (warning) or not.
>>> Can't afford data damage.
>>>
>>> Thanks,
>>>
>>> Pavol Cvengros

Re: ext4+quota+nfs issue

[Justin Maggard]

On Wed, Sep 9, 2009 at 8:02 AM, Eric Sandeen<sandeen@redhat.com> wrote:
>> On Wed, 9 Sep 2009, Pavol Cvengros wrote:
>>
>>> Hi,
>>>
>>> can somebody who is aware of ext4 and quota have a look on this one?
>>>
>
> This was also just reported at:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=521914
>
> -Eric
>

I've seen exactly the same thing myself as well, but on local I/O.
The only difference I was able to find between filesystems I saw this
on, versus filesystems that I didn't see this on, was how it was
created.  The filesystems without this issue were made using
mkfs.ext4, and the ones that _did_ have the issue were created with
mkfs.ext3, and then mounted -t ext4.  Pavol, can you check your
filesystem features from "dumpe2fs -h [your_device]"?

-Justin

Re: ext4+quota+nfs issue

[Pavol Cvengros]

On 9/9/2009 7:45 PM, Justin Maggard wrote:
> On Wed, Sep 9, 2009 at 8:02 AM, Eric Sandeen<sandeen@redhat.com>  wrote:
>    
>>> On Wed, 9 Sep 2009, Pavol Cvengros wrote:
>>>
>>>        
>>>> Hi,
>>>>
>>>> can somebody who is aware of ext4 and quota have a look on this one?
>>>>
>>>>          
>> This was also just reported at:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=521914
>>
>> -Eric
>>
>>      
> I've seen exactly the same thing myself as well, but on local I/O.
> The only difference I was able to find between filesystems I saw this
> on, versus filesystems that I didn't see this on, was how it was
> created.  The filesystems without this issue were made using
> mkfs.ext4, and the ones that _did_ have the issue were created with
> mkfs.ext3, and then mounted -t ext4.  Pavol, can you check your
> filesystem features from "dumpe2fs -h [your_device]"?
>
> -Justin
> --
>    

here is the dump....

host_stor0 ~ # dumpe2fs -h /dev/sdb1
dumpe2fs 1.41.9 (22-Aug-2009)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID:          f8aef49b-1903-4e25-9a7b-a3f5557107fb
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index 
filetype needs_recovery extent flex_bg sparse_super large_file huge_file 
uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              305176576
Block count:              1220689911
Reserved block count:     12206899
Free blocks:              977820919
Free inodes:              250981592
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      732
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Tue Jun 30 20:04:20 2009
Last mount time:          Tue Aug 18 12:21:18 2009
Last write time:          Tue Aug 18 12:21:18 2009
Mount count:              10
Maximum mount count:      -1
Last checked:             Tue Jun 30 20:04:20 2009
Check interval:           0 (<none>)
Lifetime writes:          73 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      317c2fc4-9c86-42ca-a3c3-0d6c632dcb46
Journal backup:           inode blocks
Journal size:             128M

P.

Re: ext4+quota+nfs issue

[Pavol Cvengros]

Hi,

did this dump helped ?
Should I try something?

P.

On 9/9/2009 9:02 PM, Pavol Cvengros wrote:
> On 9/9/2009 7:45 PM, Justin Maggard wrote:
>> On Wed, Sep 9, 2009 at 8:02 AM, Eric Sandeen<sandeen@redhat.com>  wrote:
>>>> On Wed, 9 Sep 2009, Pavol Cvengros wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> can somebody who is aware of ext4 and quota have a look on this one?
>>>>>
>>> This was also just reported at:
>>>
>>> https://bugzilla.redhat.com/show_bug.cgi?id=521914
>>>
>>> -Eric
>>>
>> I've seen exactly the same thing myself as well, but on local I/O.
>> The only difference I was able to find between filesystems I saw this
>> on, versus filesystems that I didn't see this on, was how it was
>> created.  The filesystems without this issue were made using
>> mkfs.ext4, and the ones that _did_ have the issue were created with
>> mkfs.ext3, and then mounted -t ext4.  Pavol, can you check your
>> filesystem features from "dumpe2fs -h [your_device]"?
>>
>> -Justin
>> -- 
>
> here is the dump....
>
> host_stor0 ~ # dumpe2fs -h /dev/sdb1
> dumpe2fs 1.41.9 (22-Aug-2009)
> Filesystem volume name: <none>
> Last mounted on: <not available>
> Filesystem UUID:          f8aef49b-1903-4e25-9a7b-a3f5557107fb
> Filesystem magic number:  0xEF53
> Filesystem revision #:    1 (dynamic)
> Filesystem features:      has_journal ext_attr resize_inode dir_index 
> filetype needs_recovery extent flex_bg sparse_super large_file 
> huge_file uninit_bg dir_nlink extra_isize
> Filesystem flags:         signed_directory_hash
> Default mount options:    (none)
> Filesystem state:         clean
> Errors behavior:          Continue
> Filesystem OS type:       Linux
> Inode count:              305176576
> Block count:              1220689911
> Reserved block count:     12206899
> Free blocks:              977820919
> Free inodes:              250981592
> First block:              0
> Block size:               4096
> Fragment size:            4096
> Reserved GDT blocks:      732
> Blocks per group:         32768
> Fragments per group:      32768
> Inodes per group:         8192
> Inode blocks per group:   512
> Flex block group size:    16
> Filesystem created:       Tue Jun 30 20:04:20 2009
> Last mount time:          Tue Aug 18 12:21:18 2009
> Last write time:          Tue Aug 18 12:21:18 2009
> Mount count:              10
> Maximum mount count:      -1
> Last checked:             Tue Jun 30 20:04:20 2009
> Check interval:           0 (<none>)
> Lifetime writes:          73 GB
> Reserved blocks uid:      0 (user root)
> Reserved blocks gid:      0 (group root)
> First inode:              11
> Inode size:               256
> Required extra isize:     28
> Desired extra isize:      28
> Journal inode:            8
> Default directory hash:   half_md4
> Directory Hash Seed:      317c2fc4-9c86-42ca-a3c3-0d6c632dcb46
> Journal backup:           inode blocks
> Journal size:             128M
>
> P.

Re: ext4+quota+nfs issue

[Jan Kara]

  Hello,

On Fri 11-09-09 16:33:48, Pavol Cvengros wrote:
> On 9/9/2009 9:02 PM, Pavol Cvengros wrote:
>> On 9/9/2009 7:45 PM, Justin Maggard wrote:
>>> On Wed, Sep 9, 2009 at 8:02 AM, Eric Sandeen<sandeen@redhat.com>  wrote:
>>>>> On Wed, 9 Sep 2009, Pavol Cvengros wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> can somebody who is aware of ext4 and quota have a look on this one?
>>>>>>
>>>> This was also just reported at:
>>>>
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=521914
>>>>
>>>> -Eric
>>>>
>>> I've seen exactly the same thing myself as well, but on local I/O.
>>> The only difference I was able to find between filesystems I saw this
>>> on, versus filesystems that I didn't see this on, was how it was
>>> created.  The filesystems without this issue were made using
>>> mkfs.ext4, and the ones that _did_ have the issue were created with
>>> mkfs.ext3, and then mounted -t ext4.  Pavol, can you check your
>>> filesystem features from "dumpe2fs -h [your_device]"?
>>>
>>> -Justin
>>> -- 
>>
>> here is the dump....
>>
>> host_stor0 ~ # dumpe2fs -h /dev/sdb1
>> dumpe2fs 1.41.9 (22-Aug-2009)
>> Filesystem volume name: <none>
>> Last mounted on: <not available>
>> Filesystem UUID:          f8aef49b-1903-4e25-9a7b-a3f5557107fb
>> Filesystem magic number:  0xEF53
>> Filesystem revision #:    1 (dynamic)
>> Filesystem features:      has_journal ext_attr resize_inode dir_index 
>> filetype needs_recovery extent flex_bg sparse_super large_file huge_file 
>> uninit_bg dir_nlink extra_isize
>> Filesystem flags:         signed_directory_hash
>> Default mount options:    (none)
>> Filesystem state:         clean
>> Errors behavior:          Continue
>> Filesystem OS type:       Linux
>> Inode count:              305176576
>> Block count:              1220689911
>> Reserved block count:     12206899
>> Free blocks:              977820919
>> Free inodes:              250981592
>> First block:              0
>> Block size:               4096
>> Fragment size:            4096
>> Reserved GDT blocks:      732
>> Blocks per group:         32768
>> Fragments per group:      32768
>> Inodes per group:         8192
>> Inode blocks per group:   512
>> Flex block group size:    16
>> Filesystem created:       Tue Jun 30 20:04:20 2009
>> Last mount time:          Tue Aug 18 12:21:18 2009
>> Last write time:          Tue Aug 18 12:21:18 2009
>> Mount count:              10
>> Maximum mount count:      -1
>> Last checked:             Tue Jun 30 20:04:20 2009
>> Check interval:           0 (<none>)
>> Lifetime writes:          73 GB
>> Reserved blocks uid:      0 (user root)
>> Reserved blocks gid:      0 (group root)
>> First inode:              11
>> Inode size:               256
>> Required extra isize:     28
>> Desired extra isize:      28
>> Journal inode:            8
>> Default directory hash:   half_md4
>> Directory Hash Seed:      317c2fc4-9c86-42ca-a3c3-0d6c632dcb46
>> Journal backup:           inode blocks
>> Journal size:             128M
  I've found some time to look into this and I can see a few problems in
the code. Firstly, what may cause your problems:
  vfs_dq_claim_blocks() is called in ext4_mb_mark_diskspace_used(). But
as far as I can understand the code, ext4_mb_normalize_request() can
increase the amount of space we really allocate and thus we try to
allocate more blocks than we have actually reserved in quota. Aneesh, is
that right?
  Secondly, ext4_da_reserve_space() seems to have a bug that it can reserve
quota blocks multiple times if ext4_claim_free_blocks() fail and we retry
the allocation. We should release the quota reservation before restarting.
Actually, when we find out we cannot reserve quota space, we could force
some delayed allocated writes to disk (thus possibly release some quota
in case we have overestimated the amount of blocks needed). But that's
a different issue.
  Thirdly, ext4_indirect_calc_metadata_amount() is wrong for sparse files.
The worst case is 3 metadata blocks per data block if we make the file
sufficiently sparse and there's no easy way around that...

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: ext4+quota+nfs issue

[Aneesh Kumar K.V]

On Mon, Sep 14, 2009 at 07:50:56PM +0200, Jan Kara wrote:
>   I've found some time to look into this and I can see a few problems in
> the code. Firstly, what may cause your problems:
>   vfs_dq_claim_blocks() is called in ext4_mb_mark_diskspace_used(). But
> as far as I can understand the code, ext4_mb_normalize_request() can
> increase the amount of space we really allocate and thus we try to
> allocate more blocks than we have actually reserved in quota. Aneesh, is
> that right?

ext4_mb_mark_diskspace_used use ac->ac_b_ex.fe_len which is NOT the normalized
request len. it is min(allocated_len, original_len). So i guess that code
should be safe
 

>   Secondly, ext4_da_reserve_space() seems to have a bug that it can reserve
> quota blocks multiple times if ext4_claim_free_blocks() fail and we retry
> the allocation. We should release the quota reservation before restarting.
> Actually, when we find out we cannot reserve quota space, we could force
> some delayed allocated writes to disk (thus possibly release some quota
> in case we have overestimated the amount of blocks needed). But that's
> a different issue.

That would imply the file system was full. But the dumpe2fs ouput list
large number of free blocks. But yes the code should have released the
quota reservation before trying block reservation again.


>   Thirdly, ext4_indirect_calc_metadata_amount() is wrong for sparse files.
> The worst case is 3 metadata blocks per data block if we make the file
> sufficiently sparse and there's no easy way around that...
> 

-aneesh

Re: ext4+quota+nfs issue

[Pavol Cvengros]

On 9/14/2009 8:52 PM, Aneesh Kumar K.V wrote:
> On Mon, Sep 14, 2009 at 07:50:56PM +0200, Jan Kara wrote:
>    
>>    I've found some time to look into this and I can see a few problems in
>> the code. Firstly, what may cause your problems:
>>    vfs_dq_claim_blocks() is called in ext4_mb_mark_diskspace_used(). But
>> as far as I can understand the code, ext4_mb_normalize_request() can
>> increase the amount of space we really allocate and thus we try to
>> allocate more blocks than we have actually reserved in quota. Aneesh, is
>> that right?
>>      
> ext4_mb_mark_diskspace_used use ac->ac_b_ex.fe_len which is NOT the normalized
> request len. it is min(allocated_len, original_len). So i guess that code
> should be safe
>
>
>    
>>    Secondly, ext4_da_reserve_space() seems to have a bug that it can reserve
>> quota blocks multiple times if ext4_claim_free_blocks() fail and we retry
>> the allocation. We should release the quota reservation before restarting.
>> Actually, when we find out we cannot reserve quota space, we could force
>> some delayed allocated writes to disk (thus possibly release some quota
>> in case we have overestimated the amount of blocks needed). But that's
>> a different issue.
>>      
> That would imply the file system was full. But the dumpe2fs ouput list
> large number of free blocks. But yes the code should have released the
> quota reservation before trying block reservation again.
>
>
>    

file system is fresh but unfortunately already used and I can't 
"re-format" it. Is there a way around this? I think that FS was created 
by "mkfs.ext4 -j /dev....."

>>    Thirdly, ext4_indirect_calc_metadata_amount() is wrong for sparse files.
>> The worst case is 3 metadata blocks per data block if we make the file
>> sufficiently sparse and there's no easy way around that...
>>
>>      
> -aneesh
>    

Re: ext4+quota+nfs issue

[Mingming]

On Tue, 2009-09-15 at 00:22 +0530, Aneesh Kumar K.V wrote:
> On Mon, Sep 14, 2009 at 07:50:56PM +0200, Jan Kara wrote:
> >   I've found some time to look into this and I can see a few problems in
> > the code. Firstly, what may cause your problems:
> >   vfs_dq_claim_blocks() is called in ext4_mb_mark_diskspace_used(). But
> > as far as I can understand the code, ext4_mb_normalize_request() can
> > increase the amount of space we really allocate and thus we try to
> > allocate more blocks than we have actually reserved in quota. Aneesh, is
> > that right?
> 
> ext4_mb_mark_diskspace_used use ac->ac_b_ex.fe_len which is NOT the normalized
> request len. it is min(allocated_len, original_len). So i guess that code
> should be safe
> 

That's right. ac->ac_b_ex.fe_len is the number of blocks that account
for the original request. It is used for update the per-fs free blocks
counter and the per-fs dirty(delayed) blocks counter too. Quota counters
are updated based on this value.

> 
> >   Secondly, ext4_da_reserve_space() seems to have a bug that it can reserve
> > quota blocks multiple times if ext4_claim_free_blocks() fail and we retry
> > the allocation. We should release the quota reservation before restarting.
> > Actually, when we find out we cannot reserve quota space, we could force
> > some delayed allocated writes to disk (thus possibly release some quota
> > in case we have overestimated the amount of blocks needed). But that's
> > a different issue.
> 
> That would imply the file system was full. But the dumpe2fs ouput list
> large number of free blocks. But yes the code should have released the
> quota reservation before trying block reservation again.
> 
> 

I'll send out a patch shortly.

> >   Thirdly, ext4_indirect_calc_metadata_amount() is wrong for sparse files.
> > The worst case is 3 metadata blocks per data block if we make the file
> > sufficiently sparse and there's no easy way around that...
> > 
> 

If this is a real concern, I am all for fix it.  Just we have consider
the worse case before, at that time it seems a little overprotect that
we have always account for the worse case. In regularly cases, we pretty
much always reserve more metadata(indirect) blocks than needed...

> -aneesh
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ext4+quota+nfs issue

[Mingming]

On Wed, 2009-09-09 at 10:02 -0500, Eric Sandeen wrote:
> Jiri Kosina wrote:
> > [ adding relevant CCs ]
> > 
> > On Wed, 9 Sep 2009, Pavol Cvengros wrote:
> > 
> >> Hi,
> >>
> >> can somebody who is aware of ext4 and quota have a look on this one?
> >>
> 
> This was also just reported at:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=521914
> 

Checked the bugzilla, it seems the fs enabled quota, but no user quota
limit is specified. Is this the case with the ext4 quota+ nfs issue too?

If no user quota is specified, then IS_NOQUOTA() should avoid doing
quota reservation/claim at all. Not sure what is missing... I will check
if I could reproduce this on local filesystem...

Thanks,
Mingming
> -Eric
> 
> >> Thanks,
> >>
> >> P.
> >>
> >> On 8. 9. 2009 7:04, Pavol Cvengros wrote:
> >>> Hello,
> >>>
> >>> recently we have build and started to use raid storage with formatted
> >>> capacity of 4.5T (ext4 formatted, default params).
> >>> FS has quota turned on and is exported via NFS to nodes.
> >>> If we turn qouta on on this FS and are trying to use it over NFS we get the
> >>> following:
> >>>
> >>> ------------[ cut here ]------------
> >>> WARNING: at fs/quota/dquot.c:964 dquot_claim_space+0x181/0x190()
> >>> Hardware name: S3210SH
> >>> Modules linked in: nfs fscache nfsd lockd auth_rpcgss exportfs sunrpc
> >>> coretemp hwmon ipmi_si ipmi_msghandler ehci_hcd sr_mod cdrom uhci_hcd floppy
> >>> usbcore i2c_i801 i2c_core processor 3w_9xxx button thermal
> >>> Pid: 268, comm: pdflush Tainted: G        W  2.6.30-gentoo-r3_host #1
> >>> Call Trace:
> >>>  [<ffffffff803151e1>] ? dquot_claim_space+0x181/0x190
> >>>  [<ffffffff80245c59>] ? warn_slowpath_common+0x89/0x100
> >>>  [<ffffffff803151e1>] ? dquot_claim_space+0x181/0x190
> >>>  [<ffffffff80367e83>] ? ext4_mb_mark_diskspace_used+0x423/0x440
> >>>  [<ffffffff8036c05f>] ? ext4_mb_new_blocks+0x2cf/0x460
> >>>  [<ffffffff80360a17>] ? ext4_ext_find_extent+0x307/0x330
> >>>  [<ffffffff80362508>] ? ext4_ext_get_blocks+0x578/0xfc0
> >>>  [<ffffffff8028e828>] ? __pagevec_free+0x48/0x70
> >>>  [<ffffffff803a1c65>] ? blk_rq_bio_prep+0x35/0x130
> >>>  [<ffffffff8034d310>] ? ext4_get_blocks_wrap+0x210/0x380
> >>>  [<ffffffff8034d8d8>] ? mpage_da_map_blocks+0xe8/0x750
> >>>  [<ffffffff80292cee>] ? pagevec_lookup_tag+0x2e/0x50
> >>>  [<ffffffff8029084c>] ? write_cache_pages+0x11c/0x400
> >>>  [<ffffffff8034e500>] ? __mpage_da_writepage+0x0/0x190
> >>>  [<ffffffff8034e269>] ? ext4_da_writepages+0x329/0x4b0
> >>>  [<ffffffff80290bd2>] ? do_writepages+0x32/0x70
> >>>  [<ffffffff802e4140>] ? __writeback_single_inode+0xb0/0x490
> >>>  [<ffffffff8023c753>] ? dequeue_entity+0x23/0x1c0
> >>>  [<ffffffff802e4b16>] ? generic_sync_sb_inodes+0x316/0x4f0
> >>>  [<ffffffff802e4f4e>] ? writeback_inodes+0x5e/0x110
> >>>  [<ffffffff80290e56>] ? wb_kupdate+0xc6/0x160
> >>>  [<ffffffff80292110>] ? pdflush+0x120/0x230
> >>>  [<ffffffff80290d90>] ? wb_kupdate+0x0/0x160
> >>>  [<ffffffff80291ff0>] ? pdflush+0x0/0x230
> >>>  [<ffffffff80261154>] ? kthread+0x64/0xc0
> >>>  [<ffffffff8020d13a>] ? child_rip+0xa/0x20
> >>>  [<ffffffff802610f0>] ? kthread+0x0/0xc0
> >>>  [<ffffffff8020d130>] ? child_rip+0x0/0x20
> >>> ---[ end trace cb54e6523e9ab60d ]---
> >>>
> >>> fstab entry:
> >>> /dev/sdb1              /mnt/storage    ext4
> >>> noatime,nodiratime,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 
> >>>
> >>> qith quotaoff on tihs FS, warnings stop.
> >>>
> >>> Question is if it's safe to use quotas with this problem (warning) or not.
> >>> Can't afford data damage.
> >>>
> >>> Thanks,
> >>>
> >>> Pavol Cvengros
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ext4+quota+nfs issue

[Pavol Cvengros]

On 9. 9. 2009 23:32, Mingming wrote:
> On Wed, 2009-09-09 at 10:02 -0500, Eric Sandeen wrote:
>    
>> Jiri Kosina wrote:
>>      
>>> [ adding relevant CCs ]
>>>
>>> On Wed, 9 Sep 2009, Pavol Cvengros wrote:
>>>
>>>        
>>>> Hi,
>>>>
>>>> can somebody who is aware of ext4 and quota have a look on this one?
>>>>
>>>>          
>> This was also just reported at:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=521914
>>
>>      
> Checked the bugzilla, it seems the fs enabled quota, but no user quota
> limit is specified. Is this the case with the ext4 quota+ nfs issue too?
>
> If no user quota is specified, then IS_NOQUOTA() should avoid doing
> quota reservation/claim at all. Not sure what is missing... I will check
> if I could reproduce this on local filesystem...
>
> Thanks,
> Mingming
>    
yes, FS has quota just enabled and only initial quotacheck was done

P.

>> -Eric
>>
>>      
>>>> Thanks,
>>>>
>>>> P.
>>>>
>>>> On 8. 9. 2009 7:04, Pavol Cvengros wrote:
>>>>          
>>>>> Hello,
>>>>>
>>>>> recently we have build and started to use raid storage with formatted
>>>>> capacity of 4.5T (ext4 formatted, default params).
>>>>> FS has quota turned on and is exported via NFS to nodes.
>>>>> If we turn qouta on on this FS and are trying to use it over NFS we get the
>>>>> following:
>>>>>
>>>>> ------------[ cut here ]------------
>>>>> WARNING: at fs/quota/dquot.c:964 dquot_claim_space+0x181/0x190()
>>>>> Hardware name: S3210SH
>>>>> Modules linked in: nfs fscache nfsd lockd auth_rpcgss exportfs sunrpc
>>>>> coretemp hwmon ipmi_si ipmi_msghandler ehci_hcd sr_mod cdrom uhci_hcd floppy
>>>>> usbcore i2c_i801 i2c_core processor 3w_9xxx button thermal
>>>>> Pid: 268, comm: pdflush Tainted: G        W  2.6.30-gentoo-r3_host #1
>>>>> Call Trace:
>>>>>   [<ffffffff803151e1>] ? dquot_claim_space+0x181/0x190
>>>>>   [<ffffffff80245c59>] ? warn_slowpath_common+0x89/0x100
>>>>>   [<ffffffff803151e1>] ? dquot_claim_space+0x181/0x190
>>>>>   [<ffffffff80367e83>] ? ext4_mb_mark_diskspace_used+0x423/0x440
>>>>>   [<ffffffff8036c05f>] ? ext4_mb_new_blocks+0x2cf/0x460
>>>>>   [<ffffffff80360a17>] ? ext4_ext_find_extent+0x307/0x330
>>>>>   [<ffffffff80362508>] ? ext4_ext_get_blocks+0x578/0xfc0
>>>>>   [<ffffffff8028e828>] ? __pagevec_free+0x48/0x70
>>>>>   [<ffffffff803a1c65>] ? blk_rq_bio_prep+0x35/0x130
>>>>>   [<ffffffff8034d310>] ? ext4_get_blocks_wrap+0x210/0x380
>>>>>   [<ffffffff8034d8d8>] ? mpage_da_map_blocks+0xe8/0x750
>>>>>   [<ffffffff80292cee>] ? pagevec_lookup_tag+0x2e/0x50
>>>>>   [<ffffffff8029084c>] ? write_cache_pages+0x11c/0x400
>>>>>   [<ffffffff8034e500>] ? __mpage_da_writepage+0x0/0x190
>>>>>   [<ffffffff8034e269>] ? ext4_da_writepages+0x329/0x4b0
>>>>>   [<ffffffff80290bd2>] ? do_writepages+0x32/0x70
>>>>>   [<ffffffff802e4140>] ? __writeback_single_inode+0xb0/0x490
>>>>>   [<ffffffff8023c753>] ? dequeue_entity+0x23/0x1c0
>>>>>   [<ffffffff802e4b16>] ? generic_sync_sb_inodes+0x316/0x4f0
>>>>>   [<ffffffff802e4f4e>] ? writeback_inodes+0x5e/0x110
>>>>>   [<ffffffff80290e56>] ? wb_kupdate+0xc6/0x160
>>>>>   [<ffffffff80292110>] ? pdflush+0x120/0x230
>>>>>   [<ffffffff80290d90>] ? wb_kupdate+0x0/0x160
>>>>>   [<ffffffff80291ff0>] ? pdflush+0x0/0x230
>>>>>   [<ffffffff80261154>] ? kthread+0x64/0xc0
>>>>>   [<ffffffff8020d13a>] ? child_rip+0xa/0x20
>>>>>   [<ffffffff802610f0>] ? kthread+0x0/0xc0
>>>>>   [<ffffffff8020d130>] ? child_rip+0x0/0x20
>>>>> ---[ end trace cb54e6523e9ab60d ]---
>>>>>
>>>>> fstab entry:
>>>>> /dev/sdb1              /mnt/storage    ext4
>>>>> noatime,nodiratime,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0
>>>>>
>>>>> qith quotaoff on tihs FS, warnings stop.
>>>>>
>>>>> Question is if it's safe to use quotas with this problem (warning) or not.
>>>>> Can't afford data damage.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Pavol Cvengros
>>>>>            
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>      
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: ext4+quota+nfs issue

[Jan Kara]

On Wed 09-09-09 14:32:09, Mingming wrote:
> On Wed, 2009-09-09 at 10:02 -0500, Eric Sandeen wrote:
> > Jiri Kosina wrote:
> > > [ adding relevant CCs ]
> > > 
> > > On Wed, 9 Sep 2009, Pavol Cvengros wrote:
> > > 
> > >> Hi,
> > >>
> > >> can somebody who is aware of ext4 and quota have a look on this one?
> > >>
> > 
> > This was also just reported at:
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=521914
> > 
> 
> Checked the bugzilla, it seems the fs enabled quota, but no user quota
> limit is specified. Is this the case with the ext4 quota+ nfs issue too?
> 
> If no user quota is specified, then IS_NOQUOTA() should avoid doing
> quota reservation/claim at all. Not sure what is missing... I will check
> if I could reproduce this on local filesystem...
  Mingming, IS_NOQUOTA() check is true only for quota-files. For all other
files it is false since we have to know how much space each user uses even
if it does not have a quota limit set (for the case that someone sets the
limit in future).

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR