* Re: [PATCH v2] create_inode: fix copying large files
2019-02-01 18:59 ` Darrick J. Wong
@ 2019-02-01 19:16 ` Burton, Ross
2019-02-06 11:01 ` Burton, Ross
2019-02-07 16:05 ` Theodore Y. Ts'o
2 siblings, 0 replies; 5+ messages in thread
From: Burton, Ross @ 2019-02-01 19:16 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: linux-ext4
Obviously this was a problem that the test suite should have spotted.
Would it be simple enough to write a test suite to make a 4.1GB file,
put in in a ext with -d, and the md5 the file in the file system? I
was using sudo and loopback mounts which isn't really ideal for the
test suite.
Ross
On Fri, 1 Feb 2019 at 18:59, Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> On Fri, Feb 01, 2019 at 06:32:47PM +0000, Ross Burton wrote:
> > When copying large files into a ext filesystem at mkfs time the copy fails at
> > 2^31 bytes in. There are two problems:
> >
> > copy_file_chunk() passes an offset (off_t, 64-bit typically) to
> > ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value
> > is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset
> > instead.
> >
> > try_lseek_copy() rounds the data and hole offsets as found by lseek() to block
> > boundaries, but the calculation gets truncated to 32-bits. Solve by casting the
> > 32-bit blocksize to off_t to ensure this doesn't happen.
> >
> > Signed-off-by: Ross Burton <ross.burton@intel.com>
>
> Looks ok,
> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
>
> --D
>
> > ---
> > misc/create_inode.c | 6 +++---
> > 1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/misc/create_inode.c b/misc/create_inode.c
> > index 05aa6363..1b35c769 100644
> > --- a/misc/create_inode.c
> > +++ b/misc/create_inode.c
> > @@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file,
> > ptr += blen;
> > continue;
> > }
> > - err = ext2fs_file_lseek(e2_file, off + bpos,
> > + err = ext2fs_file_llseek(e2_file, off + bpos,
> > EXT2_SEEK_SET, NULL);
> > if (err)
> > goto fail;
> > @@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,
> > if (hole < 0)
> > return EXT2_ET_UNIMPLEMENTED;
> >
> > - data_blk = data & ~(fs->blocksize - 1);
> > - hole_blk = (hole + (fs->blocksize - 1)) & ~(fs->blocksize - 1);
> > + data_blk = data & ~(off_t)(fs->blocksize - 1);
> > + hole_blk = (hole + (off_t)(fs->blocksize - 1)) & ~(off_t)(fs->blocksize - 1);
> > err = copy_file_chunk(fs, fd, e2_file, data_blk, hole_blk, buf,
> > zerobuf);
> > if (err)
> > --
> > 2.11.0
> >
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2] create_inode: fix copying large files
2019-02-01 18:59 ` Darrick J. Wong
2019-02-01 19:16 ` Burton, Ross
@ 2019-02-06 11:01 ` Burton, Ross
2019-02-07 16:05 ` Theodore Y. Ts'o
2 siblings, 0 replies; 5+ messages in thread
From: Burton, Ross @ 2019-02-06 11:01 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: linux-ext4
Gentle reminder to the maintainers that this patch hasn't been merged yet.
Ross
On Fri, 1 Feb 2019 at 18:59, Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> On Fri, Feb 01, 2019 at 06:32:47PM +0000, Ross Burton wrote:
> > When copying large files into a ext filesystem at mkfs time the copy fails at
> > 2^31 bytes in. There are two problems:
> >
> > copy_file_chunk() passes an offset (off_t, 64-bit typically) to
> > ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value
> > is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset
> > instead.
> >
> > try_lseek_copy() rounds the data and hole offsets as found by lseek() to block
> > boundaries, but the calculation gets truncated to 32-bits. Solve by casting the
> > 32-bit blocksize to off_t to ensure this doesn't happen.
> >
> > Signed-off-by: Ross Burton <ross.burton@intel.com>
>
> Looks ok,
> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
>
> --D
>
> > ---
> > misc/create_inode.c | 6 +++---
> > 1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/misc/create_inode.c b/misc/create_inode.c
> > index 05aa6363..1b35c769 100644
> > --- a/misc/create_inode.c
> > +++ b/misc/create_inode.c
> > @@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file,
> > ptr += blen;
> > continue;
> > }
> > - err = ext2fs_file_lseek(e2_file, off + bpos,
> > + err = ext2fs_file_llseek(e2_file, off + bpos,
> > EXT2_SEEK_SET, NULL);
> > if (err)
> > goto fail;
> > @@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,
> > if (hole < 0)
> > return EXT2_ET_UNIMPLEMENTED;
> >
> > - data_blk = data & ~(fs->blocksize - 1);
> > - hole_blk = (hole + (fs->blocksize - 1)) & ~(fs->blocksize - 1);
> > + data_blk = data & ~(off_t)(fs->blocksize - 1);
> > + hole_blk = (hole + (off_t)(fs->blocksize - 1)) & ~(off_t)(fs->blocksize - 1);
> > err = copy_file_chunk(fs, fd, e2_file, data_blk, hole_blk, buf,
> > zerobuf);
> > if (err)
> > --
> > 2.11.0
> >
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v2] create_inode: fix copying large files
2019-02-01 18:59 ` Darrick J. Wong
2019-02-01 19:16 ` Burton, Ross
2019-02-06 11:01 ` Burton, Ross
@ 2019-02-07 16:05 ` Theodore Y. Ts'o
2 siblings, 0 replies; 5+ messages in thread
From: Theodore Y. Ts'o @ 2019-02-07 16:05 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: Ross Burton, linux-ext4
On Fri, Feb 01, 2019 at 10:59:50AM -0800, Darrick J. Wong wrote:
> On Fri, Feb 01, 2019 at 06:32:47PM +0000, Ross Burton wrote:
> > When copying large files into a ext filesystem at mkfs time the copy fails at
> > 2^31 bytes in. There are two problems:
> >
> > copy_file_chunk() passes an offset (off_t, 64-bit typically) to
> > ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value
> > is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset
> > instead.
> >
> > try_lseek_copy() rounds the data and hole offsets as found by lseek() to block
> > boundaries, but the calculation gets truncated to 32-bits. Solve by casting the
> > 32-bit blocksize to off_t to ensure this doesn't happen.
> >
> > Signed-off-by: Ross Burton <ross.burton@intel.com>
>
> Looks ok,
> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Thanks, applied.
- Ted
^ permalink raw reply [flat|nested] 5+ messages in thread