Is EXT4 the right FS for > 16TB?

Is EXT4 the right FS for > 16TB?

[Justin Piszcz]

Hi,

I've read a lot of posts regarding people who setup RAID volumes of and up 
to around 16TB and EXT4 is typically used.

However, in various forums, people still ask what is the correct 
filesystem for > 16TB?  I did read one post somewhere that stated the ext4 
developers did not recommend using ext4 for very large volumes, is this 
still true?

I am looking at creating a 43TB volume possibly in the near future and I 
have used XFS in the past, which works well and would probably not have 
any problem with it; however, I have bitten quite a number of times by XFS 
bugs in the past several years, so I was curious, how does EXT4 perform on 
larger volumes, e.g., 20TB?

Are there any caveats / problems?

Justin.

Re: Is EXT4 the right FS for > 16TB?

[Sandon Van Ness]

Was it me (houkouonchi) on hard forum? I asked if > 16 TiB support was
considered stable on here a while back:

Is >16TB support considered stable?

This was 6 months ago so maybe things have changed. The thread:

http://kerneltrap.org/mailarchive/linux-ext4/2010/5/28/6884603/thread

Luckily JFS fixed there userland utilities bug of not being able to
handle > 32TiB very shortly after this and I ended up going that route
and I have yet to have any data loss or problems on my JFS volume:

root@dekabutsu: 08:32 AM :~# df -H /data
Filesystem             Size   Used  Avail Use% Mounted on
/dev/sdd1               36T    22T    15T  61% /data
root@dekabutsu: 08:32 AM :~#      

At work with our hundreds/thousands of servers we will likely be going
ext4 as we wont be using it on >16 TiB. I think its a huge improvement
over ext3 but for my use JFS ended up being a better fit. I
refuse/refused to go XFS.

On 12/19/2010 03:52 AM, Justin Piszcz wrote:
> Hi,
>
> I've read a lot of posts regarding people who setup RAID volumes of
> and up to around 16TB and EXT4 is typically used.
>
> However, in various forums, people still ask what is the correct
> filesystem for > 16TB?  I did read one post somewhere that stated the
> ext4 developers did not recommend using ext4 for very large volumes,
> is this still true?
>
> I am looking at creating a 43TB volume possibly in the near future and
> I have used XFS in the past, which works well and would probably not
> have any problem with it; however, I have bitten quite a number of
> times by XFS bugs in the past several years, so I was curious, how
> does EXT4 perform on larger volumes, e.g., 20TB?
>
> Are there any caveats / problems?
>
> Justin.
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Is EXT4 the right FS for > 16TB?

[Justin Piszcz]

Hi,

Wow, there were no updates though after Eric's last comment..
Eric, have there been any improvements in the past 6 months?

Or should one still steer clear from EXT4 > 16TB?

Justin.

On Sun, 19 Dec 2010, Sandon Van Ness wrote:

> Was it me (houkouonchi) on hard forum? I asked if > 16 TiB support was
> considered stable on here a while back:
>
> Is >16TB support considered stable?
>
> This was 6 months ago so maybe things have changed. The thread:
>
> http://kerneltrap.org/mailarchive/linux-ext4/2010/5/28/6884603/thread
>
> Luckily JFS fixed there userland utilities bug of not being able to
> handle > 32TiB very shortly after this and I ended up going that route
> and I have yet to have any data loss or problems on my JFS volume:
>
> root@dekabutsu: 08:32 AM :~# df -H /data
> Filesystem             Size   Used  Avail Use% Mounted on
> /dev/sdd1               36T    22T    15T  61% /data
> root@dekabutsu: 08:32 AM :~#
>
> At work with our hundreds/thousands of servers we will likely be going
> ext4 as we wont be using it on >16 TiB. I think its a huge improvement
> over ext3 but for my use JFS ended up being a better fit. I
> refuse/refused to go XFS.
>
> On 12/19/2010 03:52 AM, Justin Piszcz wrote:
>> Hi,
>>
>> I've read a lot of posts regarding people who setup RAID volumes of
>> and up to around 16TB and EXT4 is typically used.
>>
>> However, in various forums, people still ask what is the correct
>> filesystem for > 16TB?  I did read one post somewhere that stated the
>> ext4 developers did not recommend using ext4 for very large volumes,
>> is this still true?
>>
>> I am looking at creating a 43TB volume possibly in the near future and
>> I have used XFS in the past, which works well and would probably not
>> have any problem with it; however, I have bitten quite a number of
>> times by XFS bugs in the past several years, so I was curious, how
>> does EXT4 perform on larger volumes, e.g., 20TB?
>>
>> Are there any caveats / problems?
>>
>> Justin.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Is EXT4 the right FS for > 16TB?

[Eric Sandeen]

On 12/19/10 10:53 AM, Justin Piszcz wrote:
> Hi,
> 
> Wow, there were no updates though after Eric's last comment..
> Eric, have there been any improvements in the past 6 months?
> 
> Or should one still steer clear from EXT4 > 16TB?

There is still no released e2fsprogs which supports > 16T for
ext4, but testing of the not-released bits is welcomed...
Ted says a 16T-capable version is coming soon.  There's still
work to be done there, though.

-Eric

> Justin.
> 
> On Sun, 19 Dec 2010, Sandon Van Ness wrote:
> 
>> Was it me (houkouonchi) on hard forum? I asked if > 16 TiB support was
>> considered stable on here a while back:
>>
>> Is >16TB support considered stable?
>>
>> This was 6 months ago so maybe things have changed. The thread:
>>
>> http://kerneltrap.org/mailarchive/linux-ext4/2010/5/28/6884603/thread
>>
>> Luckily JFS fixed there userland utilities bug of not being able to
>> handle > 32TiB very shortly after this and I ended up going that route
>> and I have yet to have any data loss or problems on my JFS volume:
>>
>> root@dekabutsu: 08:32 AM :~# df -H /data
>> Filesystem             Size   Used  Avail Use% Mounted on
>> /dev/sdd1               36T    22T    15T  61% /data
>> root@dekabutsu: 08:32 AM :~#
>>
>> At work with our hundreds/thousands of servers we will likely be going
>> ext4 as we wont be using it on >16 TiB. I think its a huge improvement
>> over ext3 but for my use JFS ended up being a better fit. I
>> refuse/refused to go XFS.
>>
>> On 12/19/2010 03:52 AM, Justin Piszcz wrote:
>>> Hi,
>>>
>>> I've read a lot of posts regarding people who setup RAID volumes of
>>> and up to around 16TB and EXT4 is typically used.
>>>
>>> However, in various forums, people still ask what is the correct
>>> filesystem for > 16TB?  I did read one post somewhere that stated the
>>> ext4 developers did not recommend using ext4 for very large volumes,
>>> is this still true?
>>>
>>> I am looking at creating a 43TB volume possibly in the near future and
>>> I have used XFS in the past, which works well and would probably not
>>> have any problem with it; however, I have bitten quite a number of
>>> times by XFS bugs in the past several years, so I was curious, how
>>> does EXT4 perform on larger volumes, e.g., 20TB?
>>>
>>> Are there any caveats / problems?
>>>
>>> Justin.

Re: Is EXT4 the right FS for > 16TB?

[Ric Wheeler]

On 12/19/2010 12:01 PM, Eric Sandeen wrote:
> On 12/19/10 10:53 AM, Justin Piszcz wrote:
>> Hi,
>>
>> Wow, there were no updates though after Eric's last comment..
>> Eric, have there been any improvements in the past 6 months?
>>
>> Or should one still steer clear from EXT4>  16TB?
> There is still no released e2fsprogs which supports>  16T for
> ext4, but testing of the not-released bits is welcomed...
> Ted says a 16T-capable version is coming soon.  There's still
> work to be done there, though.
>
> -Eric

I usually tend to point people towards XFS when you need something at greater 
than 16TB in size....

Good luck,

Ric

>> Justin.
>>
>> On Sun, 19 Dec 2010, Sandon Van Ness wrote:
>>
>>> Was it me (houkouonchi) on hard forum? I asked if>  16 TiB support was
>>> considered stable on here a while back:
>>>
>>> Is>16TB support considered stable?
>>>
>>> This was 6 months ago so maybe things have changed. The thread:
>>>
>>> http://kerneltrap.org/mailarchive/linux-ext4/2010/5/28/6884603/thread
>>>
>>> Luckily JFS fixed there userland utilities bug of not being able to
>>> handle>  32TiB very shortly after this and I ended up going that route
>>> and I have yet to have any data loss or problems on my JFS volume:
>>>
>>> root@dekabutsu: 08:32 AM :~# df -H /data
>>> Filesystem             Size   Used  Avail Use% Mounted on
>>> /dev/sdd1               36T    22T    15T  61% /data
>>> root@dekabutsu: 08:32 AM :~#
>>>
>>> At work with our hundreds/thousands of servers we will likely be going
>>> ext4 as we wont be using it on>16 TiB. I think its a huge improvement
>>> over ext3 but for my use JFS ended up being a better fit. I
>>> refuse/refused to go XFS.
>>>
>>> On 12/19/2010 03:52 AM, Justin Piszcz wrote:
>>>> Hi,
>>>>
>>>> I've read a lot of posts regarding people who setup RAID volumes of
>>>> and up to around 16TB and EXT4 is typically used.
>>>>
>>>> However, in various forums, people still ask what is the correct
>>>> filesystem for>  16TB?  I did read one post somewhere that stated the
>>>> ext4 developers did not recommend using ext4 for very large volumes,
>>>> is this still true?
>>>>
>>>> I am looking at creating a 43TB volume possibly in the near future and
>>>> I have used XFS in the past, which works well and would probably not
>>>> have any problem with it; however, I have bitten quite a number of
>>>> times by XFS bugs in the past several years, so I was curious, how
>>>> does EXT4 perform on larger volumes, e.g., 20TB?
>>>>
>>>> Are there any caveats / problems?
>>>>
>>>> Justin.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is EXT4 the right FS for > 16TB?

[Justin Piszcz]

On Sun, 19 Dec 2010, Eric Sandeen wrote:

> On 12/19/10 10:53 AM, Justin Piszcz wrote:
>> Hi,
>>
>> Wow, there were no updates though after Eric's last comment..
>> Eric, have there been any improvements in the past 6 months?
>>
>> Or should one still steer clear from EXT4 > 16TB?
>
> There is still no released e2fsprogs which supports > 16T for
> ext4, but testing of the not-released bits is welcomed...
> Ted says a 16T-capable version is coming soon.  There's still
> work to be done there, though.
>
> -Eric
>

Thanks Eric for confirming.

With 7 x 3TB HDD its now possible to breach 16TB (16.38TB) in RAID-5 so I
suppose more people may start asking about this.

Justin.

Re: Is EXT4 the right FS for > 16TB?

[Eric Sandeen]

On 12/19/10 1:14 PM, Justin Piszcz wrote:
> 
> 
> On Sun, 19 Dec 2010, Eric Sandeen wrote:
> 
>> On 12/19/10 10:53 AM, Justin Piszcz wrote:
>>> Hi,
>>>
>>> Wow, there were no updates though after Eric's last comment..
>>> Eric, have there been any improvements in the past 6 months?
>>>
>>> Or should one still steer clear from EXT4 > 16TB?
>>
>> There is still no released e2fsprogs which supports > 16T for
>> ext4, but testing of the not-released bits is welcomed...
>> Ted says a 16T-capable version is coming soon.  There's still
>> work to be done there, though.
>>
>> -Eric
>>
> 
> Thanks Eric for confirming.
> 
> With 7 x 3TB HDD its now possible to breach 16TB (16.38TB) in RAID-5 so I
> suppose more people may start asking about this.

Agreed, 16T is not that much these days.

As Ric said, XFS will handle it without problem, though.

-Eric

> Justin.
> 

Re: Is EXT4 the right FS for > 16TB?

[Ric Wheeler]

On 12/19/2010 02:30 PM, Eric Sandeen wrote:
> On 12/19/10 1:14 PM, Justin Piszcz wrote:
>>
>> On Sun, 19 Dec 2010, Eric Sandeen wrote:
>>
>>> On 12/19/10 10:53 AM, Justin Piszcz wrote:
>>>> Hi,
>>>>
>>>> Wow, there were no updates though after Eric's last comment..
>>>> Eric, have there been any improvements in the past 6 months?
>>>>
>>>> Or should one still steer clear from EXT4>  16TB?
>>> There is still no released e2fsprogs which supports>  16T for
>>> ext4, but testing of the not-released bits is welcomed...
>>> Ted says a 16T-capable version is coming soon.  There's still
>>> work to be done there, though.
>>>
>>> -Eric
>>>
>> Thanks Eric for confirming.
>>
>> With 7 x 3TB HDD its now possible to breach 16TB (16.38TB) in RAID-5 so I
>> suppose more people may start asking about this.
> Agreed, 16T is not that much these days.
>
> As Ric said, XFS will handle it without problem, though.
>
> -Eric
>

XFS as a base file system is in fact very popular with commercial storage 
vendors just because of the size limitations.

Also note that will really large drives, people are also really encouraged to 
use RAID-6 (larger drives take longer to rebuild, so you have more exposure to a 
double failure that would lose data in RAID-5)

Ric