btrfs loses 32-bit application compatibility after a while

btrfs loses 32-bit application compatibility after a while

[Florian Weimer]

As far as I can tell, btrfs assigns inode numbers sequentially using
this function:

int btrfs_get_free_objectid(struct btrfs_root *root, u64 *objectid)
{
	int ret;
	mutex_lock(&root->objectid_mutex);

	if (unlikely(root->free_objectid >= BTRFS_LAST_FREE_OBJECTID)) {
		btrfs_warn(root->fs_info,
			   "the objectid of root %llu reaches its highest value",
			   root->root_key.objectid);
		ret = -ENOSPC;
		goto out;
	}

	*objectid = root->free_objectid++;
	ret = 0;
out:
	mutex_unlock(&root->objectid_mutex);
	return ret;
}

Even after deletion of the object, inode numbers are never reused.

This is a problem because after creating and deleting two billion files,
the 31-bit inode number space is exhausted.  (Not sure if negative inode
numbers are a thing, so there could be one extra bit.)  From that point
onwards, future files will end up with an inode number that cannot be
represented with the non-LFS interfaces (stat, getdents, etc.), causing
system calls to fail with EOVERFLOW.

For ABI reasons, we can't switch everything to 64-bit ino_t and LFS
interfaces.  (If we could recompile, we wouldn't still be using 32-bit
software.)

So far, we have seen this on our 32-bit Koji builders, which create and
delete many, many files.  But I suspect end users will encounter it
eventually, too.

It seems to me that the on-disk format already allows range searches on
inode numbers (see btrfs_init_root_free_objectid), so maybe this could
be used to find a sufficiently large unused range to allocate from.

Thanks,
Florian

Re: btrfs loses 32-bit application compatibility after a while

[Goffredo Baroncelli]

On 13/07/2023 10.09, Florian Weimer wrote:
> As far as I can tell, btrfs assigns inode numbers sequentially using
> this function:
> 
> int btrfs_get_free_objectid(struct btrfs_root *root, u64 *objectid)
> {
> 	int ret;
> 	mutex_lock(&root->objectid_mutex);
> 
> 	if (unlikely(root->free_objectid >= BTRFS_LAST_FREE_OBJECTID)) {
> 		btrfs_warn(root->fs_info,
> 			   "the objectid of root %llu reaches its highest value",
> 			   root->root_key.objectid);
> 		ret = -ENOSPC;
> 		goto out;
> 	}
> 
> 	*objectid = root->free_objectid++;
> 	ret = 0;
> out:
> 	mutex_unlock(&root->objectid_mutex);
> 	return ret;
> }
> 
> Even after deletion of the object, inode numbers are never reused.
> 
> This is a problem because after creating and deleting two billion files,
> the 31-bit inode number space is exhausted.  (Not sure if negative inode
> numbers are a thing, so there could be one extra bit.)  From that point
> onwards, future files will end up with an inode number that cannot be
> represented with the non-LFS interfaces (stat, getdents, etc.), causing
> system calls to fail with EOVERFLOW.
> 
It is a know btrfs issue. On 32 bit some workloads may cause the
exhaustion of the inode number.

There was the inode_cache feature that it was supposed to permit to
reuse of the inode. It was removed, below some more details

https://patchwork.kernel.org/project/linux-btrfs/patch/20200623185032.14983-1-dsterba@suse.com/

It was deprecated in 5.9 and removed in 5.11 (see commit
f5297199a8bca12b8b96afcbf2341605efb6005de)

Despite the technical details, I am curious about how many 32 bit
systems still uses btrfs.

> For ABI reasons, we can't switch everything to 64-bit ino_t and LFS
> interfaces.  (If we could recompile, we wouldn't still be using 32-bit
> software.)
> 
> So far, we have seen this on our 32-bit Koji builders, which create and
> delete many, many files.  But I suspect end users will encounter it
> eventually, too.
> 
> It seems to me that the on-disk format already allows range searches on
> inode numbers (see btrfs_init_root_free_objectid), so maybe this could
> be used to find a sufficiently large unused range to allocate from.
> 
> Thanks,
> Florian
> 

-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: btrfs loses 32-bit application compatibility after a while

[Neal Gompa]

On Sat, Jul 15, 2023 at 4:32 AM Goffredo Baroncelli <kreijack@libero.it> wrote:
>
> On 13/07/2023 10.09, Florian Weimer wrote:
> > As far as I can tell, btrfs assigns inode numbers sequentially using
> > this function:
> >
> > int btrfs_get_free_objectid(struct btrfs_root *root, u64 *objectid)
> > {
> >       int ret;
> >       mutex_lock(&root->objectid_mutex);
> >
> >       if (unlikely(root->free_objectid >= BTRFS_LAST_FREE_OBJECTID)) {
> >               btrfs_warn(root->fs_info,
> >                          "the objectid of root %llu reaches its highest value",
> >                          root->root_key.objectid);
> >               ret = -ENOSPC;
> >               goto out;
> >       }
> >
> >       *objectid = root->free_objectid++;
> >       ret = 0;
> > out:
> >       mutex_unlock(&root->objectid_mutex);
> >       return ret;
> > }
> >
> > Even after deletion of the object, inode numbers are never reused.
> >
> > This is a problem because after creating and deleting two billion files,
> > the 31-bit inode number space is exhausted.  (Not sure if negative inode
> > numbers are a thing, so there could be one extra bit.)  From that point
> > onwards, future files will end up with an inode number that cannot be
> > represented with the non-LFS interfaces (stat, getdents, etc.), causing
> > system calls to fail with EOVERFLOW.
> >
> It is a know btrfs issue. On 32 bit some workloads may cause the
> exhaustion of the inode number.
>
> There was the inode_cache feature that it was supposed to permit to
> reuse of the inode. It was removed, below some more details
>
> https://patchwork.kernel.org/project/linux-btrfs/patch/20200623185032.14983-1-dsterba@suse.com/
>
> It was deprecated in 5.9 and removed in 5.11 (see commit
> f5297199a8bca12b8b96afcbf2341605efb6005de)
>
> Despite the technical details, I am curious about how many 32 bit
> systems still uses btrfs.
>

It was somewhat common for Fedora users. A number of Fedora 32-bit ARM
variants used Btrfs until 32-bit ARM was discontinued in Fedora 37[1].
openSUSE still has 32-bit ARM and x86 support in Tumbleweed.

This issue is also possible on 64-bit x86 systems where 32-bit x86
applications run on it. That's what this report is about. We're
hitting it in Fedora because our 32-bit x86 builds in Fedora
infrastructure run on 64-bit x86 environments and triggered this[2].

[1]: https://fedoraproject.org/wiki/Changes/RetireARMv7
[2]: https://pagure.io/releng/issue/11531


-- 
真実はいつも一つ！/ Always, there's only one truth!

Re: btrfs loses 32-bit application compatibility after a while

[Goffredo Baroncelli]

On 15/07/2023 11.09, Neal Gompa wrote:
> On Sat, Jul 15, 2023 at 4:32 AM Goffredo Baroncelli <kreijack@libero.it> wrote:
>>
[..]
> 
> It was somewhat common for Fedora users. A number of Fedora 32-bit ARM
> variants used Btrfs until 32-bit ARM was discontinued in Fedora 37[1].
> openSUSE still has 32-bit ARM and x86 support in Tumbleweed.
> 
> This issue is also possible on 64-bit x86 systems where 32-bit x86
> applications run on it. That's what this report is about. We're
> hitting it in Fedora because our 32-bit x86 builds in Fedora
> infrastructure run on 64-bit x86 environments and triggered this[2].
> 

 From what you wrote, it is seems more "it is technically supported" but not
big users. Otherwise I expected that a lot of bugs or complaints happened
when it was deprecated from 5.9 and removed in 5.11.

Despite that, I am curious about what could happen when a 32 bit
application tries to access a 64 bit inode: does the kernel return only
the lower part of the inode number ? How this is handled in
other FS: what happens when an fs hosts more than 2^32 files ?
Unlikely but this may happen. BTRFS makes this more easy to happen.

> [1]: https://fedoraproject.org/wiki/Changes/RetireARMv7
> [2]: https://pagure.io/releng/issue/11531
> 
> 

-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: btrfs loses 32-bit application compatibility after a while

[Florian Weimer]

* Goffredo Baroncelli:

> On 15/07/2023 11.09, Neal Gompa wrote:
>> On Sat, Jul 15, 2023 at 4:32 AM Goffredo Baroncelli <kreijack@libero.it> wrote:
>>>
> [..]
>> It was somewhat common for Fedora users. A number of Fedora 32-bit
>> ARM
>> variants used Btrfs until 32-bit ARM was discontinued in Fedora 37[1].
>> openSUSE still has 32-bit ARM and x86 support in Tumbleweed.
>> This issue is also possible on 64-bit x86 systems where 32-bit x86
>> applications run on it. That's what this report is about. We're
>> hitting it in Fedora because our 32-bit x86 builds in Fedora
>> infrastructure run on 64-bit x86 environments and triggered this[2].
>> 
>
> From what you wrote, it is seems more "it is technically supported" but not
> big users. Otherwise I expected that a lot of bugs or complaints happened
> when it was deprecated from 5.9 and removed in 5.11.

I think that for most users, it will take some time (years?) until they
hit this issue.  The builders are a bit of a special case.

This is something where telemetry really could help.

> Despite that, I am curious about what could happen when a 32 bit
> application tries to access a 64 bit inode: does the kernel return only
> the lower part of the inode number ? How this is handled in
> other FS: what happens when an fs hosts more than 2^32 files ?
> Unlikely but this may happen. BTRFS makes this more easy to happen.

The expectation is that the kernel interfaces return EOVERFLOW, the same
error code that is used for file sizes that cannot be represented using
the 32-bit interfaces.

Thanks,
Florian