[PATCH] add support for larger files in minix filesystem

[PATCH] add support for larger files in minix filesystem

[Joshua Hudson]

From: Joshua Hudson <joshudson@gmail.com>

The Minix v3 filesystem and kernel driver have no actual dependency on files
being less than 2GB in size; however the kernel does not allow creating a
file of 2GB or larger on a Minix v3 filesystem. I was able to remove the pseudo-
dependency easily by changing one line of code (filesystems need to tell VFS
how big of files they allow).

This code won't do anything useful unless the filesystem superblock is
patched at offset 1040 to a larger value. The largest safe value is
0,252,255,255.

Signed-off-by Joshua Hudson <joshudson@gmail.com>
---
I'm not on the list anymore so if you don't CC me I won't see it.

Background: I've been playing around with work on a new embedded device.
where the hardware simply isn't powerful enough to sustain ext2 filesystem.
This resulted in me choosing the Minix filesystem as the operational filesystem
for reasons including find-next-free-block is sixteen times faster than FAT.
I discovered the need to collect about 3gb of sensor data in one tape-like run.
Naturally, this leads to creating a 3gb file; but Linux Kernel can't
handle that.

Please note I'm pretty far down branches and I don't think a stock kernel
will run on my system (non-free hardware drivers). Thankfully the Minix fs
code changes so slowly this should apply all the same. I have tested that
the patch applies and builds a kernel that fixes the problem.

The fact that the superblock has to be edited for the patch to do anything
interesting (else it replaces s_maxbytes with the same value) makes this
quite safe to apply even in the presence of a bug I don't know about.

--- linux-4.2.3/fs/minix/inode.c.orig   2015-11-01 17:13:57.227148723 -0800
+++ linux-4.2.3/fs/minix/inode.c        2015-11-01 17:21:49.785390753 -0800
@@ -232,6 +232,7 @@
                s->s_max_links = MINIX2_LINK_MAX;
        } else
                goto out_no_fs;
+       s->s_maxbytes = (unsigned)sbi->s_max_size; /* s_max_size
cannot be >=4GB and MAX_LFS_SIZE must be >= 4GB */

        /*
         * Allocate the buffer map to keep the superblock small.

Re: [PATCH] add support for larger files in minix filesystem

[Joshua Hudson]

No response. I suppose I must be doing something wrong ...

On 11/1/15, Joshua Hudson <joshudson@gmail.com> wrote:
> From: Joshua Hudson <joshudson@gmail.com>
>
> The Minix v3 filesystem and kernel driver have no actual dependency on
> files
> being less than 2GB in size; however the kernel does not allow creating a
> file of 2GB or larger on a Minix v3 filesystem. I was able to remove the
> pseudo-
> dependency easily by changing one line of code (filesystems need to tell
> VFS
> how big of files they allow).
>
> This code won't do anything useful unless the filesystem superblock is
> patched at offset 1040 to a larger value. The largest safe value is
> 0,252,255,255.
>
> Signed-off-by Joshua Hudson <joshudson@gmail.com>
> ---
> I'm not on the list anymore so if you don't CC me I won't see it.
>
> Background: I've been playing around with work on a new embedded device.
> where the hardware simply isn't powerful enough to sustain ext2 filesystem.
> This resulted in me choosing the Minix filesystem as the operational
> filesystem
> for reasons including find-next-free-block is sixteen times faster than
> FAT.
> I discovered the need to collect about 3gb of sensor data in one tape-like
> run.
> Naturally, this leads to creating a 3gb file; but Linux Kernel can't
> handle that.
>
> Please note I'm pretty far down branches and I don't think a stock kernel
> will run on my system (non-free hardware drivers). Thankfully the Minix fs
> code changes so slowly this should apply all the same. I have tested that
> the patch applies and builds a kernel that fixes the problem.
>
> The fact that the superblock has to be edited for the patch to do anything
> interesting (else it replaces s_maxbytes with the same value) makes this
> quite safe to apply even in the presence of a bug I don't know about.
>
> --- linux-4.2.3/fs/minix/inode.c.orig   2015-11-01 17:13:57.227148723 -0800
> +++ linux-4.2.3/fs/minix/inode.c        2015-11-01 17:21:49.785390753 -0800
> @@ -232,6 +232,7 @@
>                 s->s_max_links = MINIX2_LINK_MAX;
>         } else
>                 goto out_no_fs;
> +       s->s_maxbytes = (unsigned)sbi->s_max_size; /* s_max_size
> cannot be >=4GB and MAX_LFS_SIZE must be >= 4GB */
>
>         /*
>          * Allocate the buffer map to keep the superblock small.
>

Re: [PATCH] add support for larger files in minix filesystem

[Al Viro]

On Sun, Nov 01, 2015 at 07:56:36PM -0800, Joshua Hudson wrote:
> From: Joshua Hudson <joshudson@gmail.com>
> 
> The Minix v3 filesystem and kernel driver have no actual dependency on files
> being less than 2GB in size; however the kernel does not allow creating a
> file of 2GB or larger on a Minix v3 filesystem. I was able to remove the pseudo-
> dependency easily by changing one line of code (filesystems need to tell VFS
> how big of files they allow).

Umm...  AFAICS, native Minix v3 fsck will throw a fit if it sees anything
between 2Gb and 4Gb and truncate the value.  Seeing that it's their
format _and_ silent changes like that (especially hidden by something like
"you need to binary-patch the field at this offset in superblock first")
are generally considered rude.

I'd suggest you to talk to Minix folks and convince them to raise that limit;
if it's merely a matter of unhappy fsck, it shouldn't be hard, but if their
minix/mfs/*.c code would get unhappy on files longer than 2Gb, it would be
harder and in that case we _really_ shouldn't run around creating such files
there.

Seriously, talk to Minix folks first.

Re: [PATCH] add support for larger files in minix filesystem

[Joshua Hudson]

> "you need to binary-patch the field at this offset in superblock first"
> are generally considered rude.

That is in fact the only reason why it's safe. I'm not using Minix at all
and don't care what its limit is. I'm using the Minix fs because it's the
only filesystem lightweight enough and fast enough for the embedded
hardware I am developing for (32K total RAM), and I want to be able
to mount the SD card in Linux.

The only alternative is developing a completely new filesystem, and
that's a lot more pain in the behind for everybody.

I accessed the minix code at
http://users.sosdg.org/~qiyong/mxr/source/minix/fs/mfs/read.c
and they use unsigned for the block variables so the kernel would be
fine with it; except for super.c truncates the cap at 2GB, so they
simply won't be able to open large files.

Maybe we'd be happier if I limited this to a new superblock magic value;
and their code won't even mount it.

Re: [PATCH] add support for larger files in minix filesystem

[One Thousand Gnomes]

> I accessed the minix code at
> http://users.sosdg.org/~qiyong/mxr/source/minix/fs/mfs/read.c
> and they use unsigned for the block variables so the kernel would be
> fine with it; except for super.c truncates the cap at 2GB, so they
> simply won't be able to open large files.

The question is what users in general (notably Minix) do with that. As I
read it both Minix and ELKS will get mightily upset by files over 2GB
long. It would be bad if Linux allowed a user to corrupt their minixfs
images as seen by the normal users of Minix fs.

> Maybe we'd be happier if I limited this to a new superblock magic value;
> and their code won't even mount it.

Yep. That would be sensible. You might also want to pick a fixed
endianness and also decide the bit-endianness of the file system. Minixfs
proper made rather a mess of that.

If your physical media does not suffer from needing to keep blocks on the
same disk cylinder (ie rotating rust) then the V7 file system is even
smaller than the Minix one but also has the 2GB limit.

Alan

Re: [PATCH] add support for larger files in minix filesystem

[Al Viro]

On Sat, Jan 09, 2016 at 09:37:20PM +0000, One Thousand Gnomes wrote:

> If your physical media does not suffer from needing to keep blocks on the
> same disk cylinder (ie rotating rust) then the V7 file system is even
> smaller than the Minix one but also has the 2GB limit.

Keep in mind that v7 support comes along with sysvfs one, so the module
is quite likely bigger than in case of minixfs...

Re: [PATCH] add support for larger files in minix filesystem

[Joshua Hudson]

On 1/9/16, One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> wrote:
>> I accessed the minix code at
>> http://users.sosdg.org/~qiyong/mxr/source/minix/fs/mfs/read.c
>> and they use unsigned for the block variables so the kernel would be
>> fine with it; except for super.c truncates the cap at 2GB, so they
>> simply won't be able to open large files.
>
> The question is what users in general (notably Minix) do with that. As I
> read it both Minix and ELKS will get mightily upset by files over 2GB
> long. It would be bad if Linux allowed a user to corrupt their minixfs
> images as seen by the normal users of Minix fs.
>
>> Maybe we'd be happier if I limited this to a new superblock magic value;
>> and their code won't even mount it.
>
> Yep. That would be sensible. You might also want to pick a fixed
> endianness and also decide the bit-endianness of the file system. Minixfs
> proper made rather a mess of that.

Already did. If I get a big-endian filesystem the embedded code rejects.
I also have to reject shift > 0 because then the buffers don't fit in RAM.

>
> If your physical media does not suffer from needing to keep blocks on the
> same disk cylinder (ie rotating rust) then the V7 file system is even
> smaller than the Minix one but also has the 2GB limit.
>
> Alan
>

Doesn't -- SD card. And I can't meet my hard realtime requirement w/o
a free block bitmap so the only usable sysv variant is AFS and it's
readonly.

I was hoping to get a dirt-simple patch merged so I didn't have to
worry about getting the kernel to open the tape file the embedded
device collects its data; however it doesn't look like it's going to
happen. It would make sense to break the tape file up, but record
chunks
aren't powers of two in size at all so the break would fall in the
middle of a record.
It also places more strain on the embedded CPU than I'd like (no MUL
or DIV instruction).

(I call it a tape file because its internal structure looks like a
multi-track tape.)

I will certainly consider using another superblock magic value.