Bug: Buffer I/O error with an ext4 filesystem inside a file

Bug: Buffer I/O error with an ext4 filesystem inside a file

[Etienne Schmidt OSS]

Hello everyone!

I have tried to allocate disk space for a service. To do this I created a file with fixed disk usage and created an ext4 file system in it. When I mount this file the mount point should be reserved space but something went wrong. With full memory I get a buffer I/O error.

Observation
==========
I created an ext4 filesystem inside a file (ext4 partition below) and mounted it (e.g. in /var/persistent/reserved). Then I fill up the underlying file system. There is enough space inside the file but when I write a file to it the journal gives a "buffer I/O error".  

Steps to Reproduce
================
The following steps reproduce these bug. I execute them as root user.

Preparation:
1.	Create a file with fix disk usage:
	`fallocate -l 32M /var/reserved.ext4`
2.	Create a ext4 filesystem inside it:
	`mkfs.ext4 /var/reserved.ext4`
3.	Create the mountpoint:
	`mkdir /var/reserved/`
4.	Mount the file:
	`mount /var/reserved.ext4 /var/reserved/`
5.	(Optional) Check the filesystem with fsck.

Now the reserved storage works fine!

The Bug:
1.	Fill up the underlying filesystem:
	`fallocate -l 100G /var/very_big_file`
2.	Write into the reserved storage:
	`echo "Test" > /var/reserved/test_file_1`
	or anything else.
3.	The file is written but the journal shows the following error:

	```
	May 03 08:31:42 ucm kernel: loop: Write error at byte offset 8913920, length 1024.
	May 03 08:31:42 ucm kernel: blk_update_request: I/O error, dev loop0, sector 17410 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
	May 03 08:31:42 ucm kernel: EXT4-fs warning (device loop0): ext4_end_bio:344: I/O error 10 writing to inode 12 starting block 8706)
	May 03 08:31:42 ucm kernel: Buffer I/O error on device loop0, logical block 8705
	May 03 08:31:42 ucm kernel: JBD2: Detected IO errors while flushing file data on loop0-8
	```

Quick fix
=======

Allocate a larger space to the file AFTER creating the ext4 file system.
E.g after step 2. within the reserved space preparation: `fallocate -l 33M /var/reserved.ext4`


Is this a bug or is my use wrong?

Best regards

Etienne Schmidt

Bug: Buffer I/O error with an ext4 filesystem inside a file

[Etienne Schmidt OSS]

Hello everyone!

I have tried to allocate disk space for a service. To do this I created a file with fixed disk usage and created an ext4 file system in it. When I mount this file the mount point should be reserved space but something went wrong. With full memory I get a buffer I/O error.

Observation
==========
I created an ext4 filesystem inside a file (ext4 partition below) and mounted it (e.g. in /var/persistent/reserved). Then I fill up the underlying file system. There is enough space inside the file but when I write a file to it the journal gives a "buffer I/O error".  

Steps to Reproduce
================
The following steps reproduce these bug. I execute them as root user.

Preparation:
1.	Create a file with fix disk usage:
	`fallocate -l 32M /var/reserved.ext4`
2.	Create a ext4 filesystem inside it:
	`mkfs.ext4 /var/reserved.ext4`
3.	Create the mountpoint:
	`mkdir /var/reserved/`
4.	Mount the file:
	`mount /var/reserved.ext4 /var/reserved/`
5.	(Optional) Check the filesystem with fsck.

Now the reserved storage works fine!

The Bug:
1.	Fill up the underlying filesystem:
	`fallocate -l 100G /var/very_big_file`
2.	Write into the reserved storage:
	`echo "Test" > /var/reserved/test_file_1`
	or anything else.
3.	The file is written but the journal shows the following error:

	```
	May 03 08:31:42 ucm kernel: loop: Write error at byte offset 8913920, length 1024.
	May 03 08:31:42 ucm kernel: blk_update_request: I/O error, dev loop0, sector 17410 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
	May 03 08:31:42 ucm kernel: EXT4-fs warning (device loop0): ext4_end_bio:344: I/O error 10 writing to inode 12 starting block 8706)
	May 03 08:31:42 ucm kernel: Buffer I/O error on device loop0, logical block 8705
	May 03 08:31:42 ucm kernel: JBD2: Detected IO errors while flushing file data on loop0-8
	```

Quick fix
=======

Allocate a larger space to the file AFTER creating the ext4 file system.
E.g after step 2. within the reserved space preparation: `fallocate -l 33M /var/reserved.ext4`


Is this a bug or is my usage wrong?

Best regards

Etienne Schmidt

Re: Bug: Buffer I/O error with an ext4 filesystem inside a file

[Theodore Ts'o]

On Tue, Feb 21, 2023 at 11:57:36AM +0000, Etienne Schmidt OSS wrote:
> Hello everyone!
> 
> I have tried to allocate disk space for a service. To do this I created a file with fixed disk usage and created an ext4 file system in it. When I mount this file the mount point should be reserved space but something went wrong. With full memory I get a buffer I/O error.
> 
> Steps to Reproduce
> ================
> The following steps reproduce these bug. I execute them as root user.
> 
> Preparation:
> 1.	Create a file with fix disk usage:
> 	`fallocate -l 32M /var/reserved.ext4`
> 2.	Create a ext4 filesystem inside it:
> 	`mkfs.ext4 /var/reserved.ext4`
> 3.	Create the mountpoint:
> 	`mkdir /var/reserved/`
> 4.	Mount the file:
> 	`mount /var/reserved.ext4 /var/reserved/`
> 5.	(Optional) Check the filesystem with fsck.
> 
> Now the reserved storage works fine!
> 
> The Bug:
> 1.	Fill up the underlying filesystem:
> 	`fallocate -l 100G /var/very_big_file`
> 2.	Write into the reserved storage:
> 	`echo "Test" > /var/reserved/test_file_1`
> 	or anything else.
> 3.	The file is written but the journal shows the following error:
> 
> 	```

Fallocate reserves the data blocks, but all of the extents are marked
as "uninitialized".  So when you write into /var/reserved.ext4, either
through the loop device or by directly writing into the file ---
especially if you are writing in a number of different locations in
the file, a large extent marked "uinitiailized" might need to be split
into 3 extents:

     |--------------------- uninitialized ----------------------------|
     
                  			becomes

     |---------- uninitialized --|--- written --|-- unitialized-------|


With enough writes, ext4 will need to allocate a block in the
underlying file system to expand the extent tree.  But since you've
filled up the underlying file system, there is no space for the
expanded extent tree, and so the attempted write by the loop device
fails:

> 	May 03 08:31:42 ucm kernel: loop: Write error at byte offset 8913920, length 1024.
> 	May 03 08:31:42 ucm kernel: blk_update_request: I/O error, dev loop0, sector 17410 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0

So there are a couple of different solutions to avoid this.

1) Force the entire /var/reserved.ext4 to be initialized, by using:

	dd if=/dev/zero of=/var/reserved.ext4 bs=1M count=32

... instead of using the fallocate command.

2) Don't fill the underlying file system.  "Doctor, doctor it hurts
when I do that!"  "Well, then don't do that!"

Best regards,

						- Ted