Incorrect user quota handling in fallocate

Incorrect user quota handling in fallocate

[Martin Svec]

Hi all,

I've found a bug in XFS user quota handling in two subsequent fallocate() calls. This bug can be
easily reproduced by the following script:

# assume empty XFS mounted on /mnt/testxfs with -o usrquota, grpquota

FILE="/mnt/testxfs/test.file"
USER="testuser"

setquota -u $USER $QUOTA $QUOTA 0 0 -a
touch $FILE
chown $USER:users $FILE
fallocate --keep-size -o 0 -l $FILESIZE $FILE
fallocate -o 0 -l $FILESIZE $FILE

That is, we create an empty file, preallocate requested size while keeping zero file size and then
call fallocate again to set the file size. Assume that there's enaugh free quota to fit the
requested file size. In this case, both fallocate calls should succeed because the second one just
increases the file size but does not change the allocated space. However, I observed that the second
fallocate fails with EDQUOT if the free quota is less than _two times_ of the requested file size. I
guess that the second fallocate ignores the fact that the space was already preallocated and
accounts the requested size for the second time. For example, if QUOTA=2GiB, file size FILESIZE=800
MiB succeeds but FILESIZE=1600 MiB triggers EDQUOT in second fallocate. The same test performed on
EXT4 always succeeds.

I've found this issue while investigating why Samba (ver. 4.9.5) returns disk full error although
there's still enaugh room for the copied file. Indeed, when Samba's "strict allocate" options is
turned on Samba uses the above described sequence of two fallocate() syscalls to create a new file.

We noticed this behavior on Debian Buster 4.19 kernel and confirmed on stable 5.10.15 vanilla kernel.

Best regards,

Martin

Re: Incorrect user quota handling in fallocate

[Dave Chinner]

On Fri, Mar 05, 2021 at 11:14:32AM +0100, Martin Svec wrote:
> Hi all,
> 
> I've found a bug in XFS user quota handling in two subsequent fallocate() calls. This bug can be
> easily reproduced by the following script:
> 
> # assume empty XFS mounted on /mnt/testxfs with -o usrquota, grpquota
> 
> FILE="/mnt/testxfs/test.file"
> USER="testuser"
> 
> setquota -u $USER $QUOTA $QUOTA 0 0 -a
> touch $FILE
> chown $USER:users $FILE
> fallocate --keep-size -o 0 -l $FILESIZE $FILE
> fallocate -o 0 -l $FILESIZE $FILE
> 
> That is, we create an empty file, preallocate requested size while keeping zero file size and then
> call fallocate again to set the file size. Assume that there's enaugh free quota to fit the
> requested file size. In this case, both fallocate calls should succeed because the second one just
> increases the file size but does not change the allocated space. However, I observed that the second
> fallocate fails with EDQUOT if the free quota is less than _two times_ of the requested file size. I

I'd call that expected behaviour. We enforce space restrictions
(ENOSPC and EDQUOT) on the size being requested before the operation
takes place and return the unused space reservation after the
fallocate() completes and has allocated space.

We cannot overcommit space or quota in XFS - that way leads to
deadlocks and data loss at ENOSPC because we can end up in
situations where we cannot write back data that users have written.
Hence we check up front, and if the worst case space requirement for
a given operation cannot fit under the ENOSPC or EDQUOT limits, it
is rejected with EDQUOT/ENOSPC.

Yes, that means we get corner cases when near EDQUOT/ENOSPC where
stuff like this happens, but that tends to be exceedingly rare in
the rare world as few applications use the entire filesystem space
or quota allowance in just 1-2 allocations.

> guess that the second fallocate ignores the fact that the space was already preallocated and
> accounts the requested size for the second time. For example, if QUOTA=2GiB, file size FILESIZE=800
> MiB succeeds but FILESIZE=1600 MiB triggers EDQUOT in second fallocate. The same test performed on
> EXT4 always succeeds.

Yup, filesystems behave differently at edge cases.

> I've found this issue while investigating why Samba (ver. 4.9.5) returns disk full error although
> there's still enaugh room for the copied file. Indeed, when Samba's "strict allocate" options is
> turned on Samba uses the above described sequence of two fallocate() syscalls to create a new file.

I'd say using fallocate to extend the file like this is .... not a
very good idea. All you actually need is a ftruncate() to change the
file size after the first fallocate (or use the first fallocate to
extend the file, too). The second fallocate() is largely useless on
most filesystems - internally they turn it into an extending
truncate because no allocation is required, so you may as well just
call ftruncate() and that way you will not trip over this space
reservation issue at all..

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Incorrect user quota handling in fallocate

[Martin Svec]

Dne 05.03.2021 v 22:45 Dave Chinner napsal(a):
> On Fri, Mar 05, 2021 at 11:14:32AM +0100, Martin Svec wrote:
>> Hi all,
>>
>> I've found a bug in XFS user quota handling in two subsequent fallocate() calls. This bug can be
>> easily reproduced by the following script:
>>
>> # assume empty XFS mounted on /mnt/testxfs with -o usrquota, grpquota
>>
>> FILE="/mnt/testxfs/test.file"
>> USER="testuser"
>>
>> setquota -u $USER $QUOTA $QUOTA 0 0 -a
>> touch $FILE
>> chown $USER:users $FILE
>> fallocate --keep-size -o 0 -l $FILESIZE $FILE
>> fallocate -o 0 -l $FILESIZE $FILE
>>
>> That is, we create an empty file, preallocate requested size while keeping zero file size and then
>> call fallocate again to set the file size. Assume that there's enaugh free quota to fit the
>> requested file size. In this case, both fallocate calls should succeed because the second one just
>> increases the file size but does not change the allocated space. However, I observed that the second
>> fallocate fails with EDQUOT if the free quota is less than _two times_ of the requested file size. I
> I'd call that expected behaviour. We enforce space restrictions
> (ENOSPC and EDQUOT) on the size being requested before the operation
> takes place and return the unused space reservation after the
> fallocate() completes and has allocated space.

Well, I would expect that allocation of an already allocated space won't account that space again so
it's a clear bug for me. But I'm not aware of XFS internals and understand that it can follow from
overall space allocation logic.

> We cannot overcommit space or quota in XFS - that way leads to
> deadlocks and data loss at ENOSPC because we can end up in
> situations where we cannot write back data that users have written.
> Hence we check up front, and if the worst case space requirement for
> a given operation cannot fit under the ENOSPC or EDQUOT limits, it
> is rejected with EDQUOT/ENOSPC.
>
> Yes, that means we get corner cases when near EDQUOT/ENOSPC where
> stuff like this happens, but that tends to be exceedingly rare in
> the rare world as few applications use the entire filesystem space
> or quota allowance in just 1-2 allocations.

If our customer pays for a disk quota and cannot upload it's favorite 16 GiB movie to Samba share
although he sees more than 30 GiB of space free, it's a bit hard to explain him that this is an
expected and exceedingly rare behavior. :-)

>
>> guess that the second fallocate ignores the fact that the space was already preallocated and
>> accounts the requested size for the second time. For example, if QUOTA=2GiB, file size FILESIZE=800
>> MiB succeeds but FILESIZE=1600 MiB triggers EDQUOT in second fallocate. The same test performed on
>> EXT4 always succeeds.
> Yup, filesystems behave differently at edge cases.
>
>> I've found this issue while investigating why Samba (ver. 4.9.5) returns disk full error although
>> there's still enaugh room for the copied file. Indeed, when Samba's "strict allocate" options is
>> turned on Samba uses the above described sequence of two fallocate() syscalls to create a new file.
> I'd say using fallocate to extend the file like this is .... not a
> very good idea. All you actually need is a ftruncate() to change the
> file size after the first fallocate (or use the first fallocate to
> extend the file, too). The second fallocate() is largely useless on
> most filesystems - internally they turn it into an extending
> truncate because no allocation is required, so you may as well just
> call ftruncate() and that way you will not trip over this space
> reservation issue at all..

I definitely agree that this is ... ehm, stupid way of file space reservation. I'll try to discuss
it on Samba mailing list too. Fortunately, turning off Samba's "string allocate" feature solves the
issue with minimal drawbacks.

> Cheers,
>
> Dave.

Thank you for your response,

Martin