2.6.33.1: ext4 disk free weirdness

2.6.33.1: ext4 disk free weirdness

[CaT]

Using kernel 2.6.33.1. Am rsyncing from /data/mirror (a raid0 mount) to
/dev/sda1 (a usb HD). Both filesystems formatted as ext3 but mounted as
ext4. /mnt filesystem is only being added to and currently large files
(iso files 700MB-4.4GB) are being copied. When I remounted /mnt as ext3
I could not duplicate the issue and the amount of disk used stabilised
at 182G. The df results, whilst /mnt is mounted as ext4, are below:

1 [29/03 15:05:58] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  317G  555G  37% /mnt
1 [29/03 15:11:05] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  224G  647G  26% /mnt
1 [29/03 15:41:38] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  670G  201G  77% /mnt 
1 [29/03 15:48:32] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  246G  626G  29% /mnt 
1 [29/03 15:48:36] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  313G  559G  36% /mnt 
1 [29/03 15:48:37] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  396G  475G  46% /mnt 
1 [29/03 15:48:37] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  615G  256G  71% /mnt 
1 [29/03 15:48:39] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  671G  201G  77% /mnt 
1 [29/03 15:48:40] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  671G  201G  77% /mnt 
1 [29/03 15:48:40] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  215G  656G  25% /mnt 
1 [29/03 15:48:41] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  334G  538G  39% /mnt 
1 [29/03 15:48:42] >> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              1.3T  1.2T  113G  92% /data/mirror
/dev/sda1             917G  522G  349G  60% /mnt 
1 [29/03 15:48:43] >> 

-- 
  "A search of his car uncovered pornography, a homemade sex aid, women's 
  stockings and a Jack Russell terrier."
    - http://www.news.com.au/story/0%2C27574%2C24675808-421%2C00.html

Re: 2.6.33.1: ext4 disk free weirdness

[Eric Sandeen]

CaT wrote:
> Using kernel 2.6.33.1. Am rsyncing from /data/mirror (a raid0 mount) to
> /dev/sda1 (a usb HD). Both filesystems formatted as ext3 but mounted as
> ext4. /mnt filesystem is only being added to and currently large files
> (iso files 700MB-4.4GB) are being copied. When I remounted /mnt as ext3
> I could not duplicate the issue and the amount of disk used stabilised
> at 182G. The df results, whilst /mnt is mounted as ext4, are below:

I would say this is likely speculative allocation due to delalloc
(ext4 has to reserve worst-case metadata amounts to prepare for
delalloc writeback).  When the file data actually gets written out,
the worst-case reservation is freed up again.  We also flush and
switch to nodelalloc when the filesystem is pretty close to full.
see ext4_nonda_switch() for example.

This looks pretty severe though, more than I would expect
from that behavior.

Some interesting tests would be to try it again -o nodelalloc
(I expect the behavior to go away) and maybe to copy a single
large file and see how bad the overshoot is before and after
a "sync".

-Eric


> 1 [29/03 15:05:58] >> df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md5              1.3T  1.2T  113G  92% /data/mirror
> /dev/sda1             917G  317G  555G  37% /mnt
> 1 [29/03 15:11:05] >> df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md5              1.3T  1.2T  113G  92% /data/mirror
> /dev/sda1             917G  224G  647G  26% /mnt
> 1 [29/03 15:41:38] >> df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md5              1.3T  1.2T  113G  92% /data/mirror
> /dev/sda1             917G  670G  201G  77% /mnt 
> 1 [29/03 15:48:32] >> df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md5              1.3T  1.2T  113G  92% /data/mirror
> /dev/sda1             917G  246G  626G  29% /mnt 
> 1 [29/03 15:48:36] >> df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md5              1.3T  1.2T  113G  92% /data/mirror
> /dev/sda1             917G  313G  559G  36% /mnt 
> 1 [29/03 15:48:37] >> df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md5              1.3T  1.2T  113G  92% /data/mirror
> /dev/sda1             917G  396G  475G  46% /mnt 
> 1 [29/03 15:48:37] >> df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md5              1.3T  1.2T  113G  92% /data/mirror
> /dev/sda1             917G  615G  256G  71% /mnt 
> 1 [29/03 15:48:39] >> df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md5              1.3T  1.2T  113G  92% /data/mirror
> /dev/sda1             917G  671G  201G  77% /mnt 
> 1 [29/03 15:48:40] >> df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md5              1.3T  1.2T  113G  92% /data/mirror
> /dev/sda1             917G  671G  201G  77% /mnt 
> 1 [29/03 15:48:40] >> df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md5              1.3T  1.2T  113G  92% /data/mirror
> /dev/sda1             917G  215G  656G  25% /mnt 
> 1 [29/03 15:48:41] >> df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md5              1.3T  1.2T  113G  92% /data/mirror
> /dev/sda1             917G  334G  538G  39% /mnt 
> 1 [29/03 15:48:42] >> df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/md5              1.3T  1.2T  113G  92% /data/mirror
> /dev/sda1             917G  522G  349G  60% /mnt 
> 1 [29/03 15:48:43] >> 
> 

Re: 2.6.33.1: ext4 disk free weirdness

[tytso]

On Mon, Mar 29, 2010 at 09:48:50AM -0500, Eric Sandeen wrote:
> CaT wrote:
> > Using kernel 2.6.33.1. Am rsyncing from /data/mirror (a raid0 mount) to
> > /dev/sda1 (a usb HD). Both filesystems formatted as ext3 but mounted as
> > ext4. /mnt filesystem is only being added to and currently large files
> > (iso files 700MB-4.4GB) are being copied. When I remounted /mnt as ext3
> > I could not duplicate the issue and the amount of disk used stabilised
> > at 182G. The df results, whilst /mnt is mounted as ext4, are below:
> 
> I would say this is likely speculative allocation due to delalloc
> (ext4 has to reserve worst-case metadata amounts to prepare for
> delalloc writeback).  When the file data actually gets written out,
> the worst-case reservation is freed up again.  We also flush and
> switch to nodelalloc when the filesystem is pretty close to full.
> see ext4_nonda_switch() for example.
> 
> This looks pretty severe though, more than I would expect
> from that behavior.

This is fixed in commit d330a5bef, which which got merged post
2.6.34-rc2, and which we need to get into 2.6.33 stable.  If you
cherry-pick that commit, I think you'll see that it fixes this
problem.

						- Ted

Re: 2.6.33.1: ext4 disk free weirdness

[Eric Sandeen]

tytso@mit.edu wrote:
> On Mon, Mar 29, 2010 at 09:48:50AM -0500, Eric Sandeen wrote:
>> CaT wrote:
>>> Using kernel 2.6.33.1. Am rsyncing from /data/mirror (a raid0 mount) to
>>> /dev/sda1 (a usb HD). Both filesystems formatted as ext3 but mounted as
>>> ext4. /mnt filesystem is only being added to and currently large files
>>> (iso files 700MB-4.4GB) are being copied. When I remounted /mnt as ext3
>>> I could not duplicate the issue and the amount of disk used stabilised
>>> at 182G. The df results, whilst /mnt is mounted as ext4, are below:
>> I would say this is likely speculative allocation due to delalloc
>> (ext4 has to reserve worst-case metadata amounts to prepare for
>> delalloc writeback).  When the file data actually gets written out,
>> the worst-case reservation is freed up again.  We also flush and
>> switch to nodelalloc when the filesystem is pretty close to full.
>> see ext4_nonda_switch() for example.
>>
>> This looks pretty severe though, more than I would expect
>> from that behavior.
> 
> This is fixed in commit d330a5bef, which which got merged post
> 2.6.34-rc2, and which we need to get into 2.6.33 stable.  If you
> cherry-pick that commit, I think you'll see that it fixes this
> problem.

Oh, right, spaced that one.  Thanks for the reminder Ted.  :)

And from now on ext3-mounted-as-ext4 is special-cased so you won't get
delalloc, and you won't get this behavior at all (nor will you get
any delalloc benefits, FWIW).

Thanks,
-Eric
 
> 						- Ted

Re: 2.6.33.1: ext4 disk free weirdness

[Justin Maggard]

On Mon, Mar 29, 2010 at 8:01 AM, Eric Sandeen <sandeen@redhat.com> wrote:
> tytso@mit.edu wrote:
>> On Mon, Mar 29, 2010 at 09:48:50AM -0500, Eric Sandeen wrote:
>>> CaT wrote:
>>>> Using kernel 2.6.33.1. Am rsyncing from /data/mirror (a raid0 mount) to
>>>> /dev/sda1 (a usb HD). Both filesystems formatted as ext3 but mounted as
>>>> ext4. /mnt filesystem is only being added to and currently large files
>>>> (iso files 700MB-4.4GB) are being copied. When I remounted /mnt as ext3
>>>> I could not duplicate the issue and the amount of disk used stabilised
>>>> at 182G. The df results, whilst /mnt is mounted as ext4, are below:
>>> I would say this is likely speculative allocation due to delalloc
>>> (ext4 has to reserve worst-case metadata amounts to prepare for
>>> delalloc writeback).  When the file data actually gets written out,
>>> the worst-case reservation is freed up again.  We also flush and
>>> switch to nodelalloc when the filesystem is pretty close to full.
>>> see ext4_nonda_switch() for example.
>>>
>>> This looks pretty severe though, more than I would expect
>>> from that behavior.
>>
>> This is fixed in commit d330a5bef, which which got merged post
>> 2.6.34-rc2, and which we need to get into 2.6.33 stable.  If you
>> cherry-pick that commit, I think you'll see that it fixes this
>> problem.
>
> Oh, right, spaced that one.  Thanks for the reminder Ted.  :)
>
> And from now on ext3-mounted-as-ext4 is special-cased so you won't get
> delalloc, and you won't get this behavior at all (nor will you get
> any delalloc benefits, FWIW).
>

Out of curiosity, when you say ext3-mounted-as-ext4, are you referring
to filesystems without ext4 features mounted -t ext4, or filesystems
mounted -t ext3 when CONFIG_EXT4_USE_FOR_EXT23 is defined?

-Justin
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2.6.33.1: ext4 disk free weirdness

[Eric Sandeen]

Justin Maggard wrote:
> On Mon, Mar 29, 2010 at 8:01 AM, Eric Sandeen <sandeen@redhat.com> wrote:
>> tytso@mit.edu wrote:
>>> On Mon, Mar 29, 2010 at 09:48:50AM -0500, Eric Sandeen wrote:
>>>> CaT wrote:
>>>>> Using kernel 2.6.33.1. Am rsyncing from /data/mirror (a raid0 mount) to
>>>>> /dev/sda1 (a usb HD). Both filesystems formatted as ext3 but mounted as
>>>>> ext4. /mnt filesystem is only being added to and currently large files
>>>>> (iso files 700MB-4.4GB) are being copied. When I remounted /mnt as ext3
>>>>> I could not duplicate the issue and the amount of disk used stabilised
>>>>> at 182G. The df results, whilst /mnt is mounted as ext4, are below:
>>>> I would say this is likely speculative allocation due to delalloc
>>>> (ext4 has to reserve worst-case metadata amounts to prepare for
>>>> delalloc writeback).  When the file data actually gets written out,
>>>> the worst-case reservation is freed up again.  We also flush and
>>>> switch to nodelalloc when the filesystem is pretty close to full.
>>>> see ext4_nonda_switch() for example.
>>>>
>>>> This looks pretty severe though, more than I would expect
>>>> from that behavior.
>>> This is fixed in commit d330a5bef, which which got merged post
>>> 2.6.34-rc2, and which we need to get into 2.6.33 stable.  If you
>>> cherry-pick that commit, I think you'll see that it fixes this
>>> problem.
>> Oh, right, spaced that one.  Thanks for the reminder Ted.  :)
>>
>> And from now on ext3-mounted-as-ext4 is special-cased so you won't get
>> delalloc, and you won't get this behavior at all (nor will you get
>> any delalloc benefits, FWIW).
>>
> 
> Out of curiosity, when you say ext3-mounted-as-ext4, are you referring
> to filesystems without ext4 features mounted -t ext4, or filesystems
> mounted -t ext3 when CONFIG_EXT4_USE_FOR_EXT23 is defined?

hmmm to be honest i've lost track a little ;)

I was referring to:

commit ba69f9ab7df844125898104e854e063b47c26637
Author: Jan Kara <jack@suse.cz>
Date:   Wed Mar 24 20:18:37 2010 -0400

    ext4: Don't use delayed allocation by default when used instead of ext3
    
    When ext4 driver is used to mount a filesystem instead of the ext3 file
    system driver (through CONFIG_EXT4_USE_FOR_EXT23), do not enable delayed
    allocation by default since some ext3 users and application writers have
    developed unfortunate expectations about the safety of writing files on
    systems subject to sudden and violent death without using fsync().
    
    Signed-off-by: Jan Kara <jack@suse.cz>
    Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>

looking at that commit I guess it's -only- when CONFIG_EXT4_USE_FOR_EXT23
is defined.  So in answer to your question above, it's your latter case.

See previous complaint about this all becoming rather confusing....

-Eric


> -Justin

Re: 2.6.33.1: ext4 disk free weirdness

[CaT]

On Mon, Mar 29, 2010 at 10:53:11AM -0400, tytso@mit.edu wrote:
> This is fixed in commit d330a5bef, which which got merged post
> 2.6.34-rc2, and which we need to get into 2.6.33 stable.  If you
> cherry-pick that commit, I think you'll see that it fixes this
> problem.

As it turns out, that USB drive died soon after so, as soon as I
get a warranty replacement I'll either cherry-pick this or install
the 33 point release with it in. It's kinda wild and wooly with its
speculation. All else fails, I'll use the mount option.

Thanks to both you and Eric for your replies. :)

-- 
  "A search of his car uncovered pornography, a homemade sex aid, women's 
  stockings and a Jack Russell terrier."
    - http://www.news.com.au/story/0%2C27574%2C24675808-421%2C00.html