FIBMAP unsupported

FIBMAP unsupported

[Marc Dietrich]

Hi,

I have a large (25G) virtual disk on a btrfs fs. Yes, I know this is not 
optimial. So I try to defrag it from time to time. However, using "btrfs fi 
defrag -c vm.vdi" results in even more fragments than before (reported by 
filefrag). So I wrote my own pseudo defragger,

-----
#!/bin/sh

test -f $1 || exit 2

echo "defrag $1"
/usr/sbin/filefrag $1 || exit

fallocate -n -l `filesize $1` $1.new || exit
chattr +C $1.new
dd if=$1 of=$1.new conv=notrunc oflag=append status=none
chmod --reference $1 $1.new
chown --reference $1 $1.new
mv $1.new $1
/usr/sbin/filefrag $1

-----

which produces much better results (ok, the file must not be in use). 
Somewhere in the 3.17 cycle the resulting image got corrupted using the script 
above. 

Running filefrag on it returns "FIBMAP unsupported".

Virtualbox returns  "AHCI#0P0: Read at offset 606236672 (49152 bytes left) 
returned rc=VERR_DEV_IO_ERROR". No errors in the kernel log.

Trying "cp vm.vdi /dev/null" returns: cp: Error reading „vm.vdi“: IO-Error

kernel 3.17-rc7
btrfs 3.17.x
mount options: rw,nodiratime,relatime,compress=lzo,space_cache,autodefrag

Marc

Re: FIBMAP unsupported

[David Sterba]

On Thu, Oct 02, 2014 at 05:13:22PM +0200, Marc Dietrich wrote:
> I have a large (25G) virtual disk on a btrfs fs. Yes, I know this is not 
> optimial. So I try to defrag it from time to time. However, using "btrfs fi 
> defrag -c vm.vdi" results in even more fragments than before (reported by 
> filefrag). So I wrote my own pseudo defragger,

Unfortunatelly the default target fragment size is 256k. Try
'btrfs filesystem defrag -t 32m ...' or higher numbers and see if it
helps.

> which produces much better results (ok, the file must not be in use). 
> Somewhere in the 3.17 cycle the resulting image got corrupted using the script 
> above. 
> 
> Running filefrag on it returns "FIBMAP unsupported".

This message doe not mean it is a corruption, but filefrag tries to use
the FIBMAP ioctl that is not implemented on btrfs, instead FIEMAP is
used.

filefrag on a nocow file works for me here (3.16.x kernel), I can see
that filefrag on a directory prints the FIBMAP message.

> Virtualbox returns  "AHCI#0P0: Read at offset 606236672 (49152 bytes left) 
> returned rc=VERR_DEV_IO_ERROR". No errors in the kernel log.
> 
> Trying "cp vm.vdi /dev/null" returns: cp: Error reading „vm.vdi“: IO-Error

This could be caused by the virtualization layer. Try to run scrub and
fsck in the non-destru^Wchecking mode if it finds problems.

As you're using compression and autodefrag, a quick skim of the 3.17
patches points to e9512d72e8e61c750c90efacd720abe3c4569822 "fix
autodefrag with compression", but that's just "keyword" match.

There's another report about nocow corruption and VirtualBox in a 3.16 +
for-linus version (which is almost 3.17-rc)
http://article.gmane.org/gmane.comp.file-systems.btrfs/38701/

But according to the attached messages, the underlying device is
unreliable and logs a lot of IO errors.

For now it looks like VirutalBox is not writing the data or there is a
bug introduced post 3.16 killing nocow files.

Re: FIBMAP unsupported

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2816 bytes --]

On Thu, Oct 02, 2014 at 07:25:49PM +0200, David Sterba wrote:
> On Thu, Oct 02, 2014 at 05:13:22PM +0200, Marc Dietrich wrote:
> > I have a large (25G) virtual disk on a btrfs fs. Yes, I know this is not 
> > optimial. So I try to defrag it from time to time. However, using "btrfs fi 
> > defrag -c vm.vdi" results in even more fragments than before (reported by 
> > filefrag). So I wrote my own pseudo defragger,
> 
> Unfortunatelly the default target fragment size is 256k. Try
> 'btrfs filesystem defrag -t 32m ...' or higher numbers and see if it
> helps.

   Note also that a compressed file will have fragments on the scale
of about 128k reported by filefrag, because of the way that the
compression works. The file may actually be contiguous, but filefrag
won't know about it. (At least, that's historically been the case. I
don't know if filefrag has recently grown some extra knowledge of
compressed extents.)

   Hugo.

> > which produces much better results (ok, the file must not be in use). 
> > Somewhere in the 3.17 cycle the resulting image got corrupted using the script 
> > above. 
> > 
> > Running filefrag on it returns "FIBMAP unsupported".
> 
> This message doe not mean it is a corruption, but filefrag tries to use
> the FIBMAP ioctl that is not implemented on btrfs, instead FIEMAP is
> used.
> 
> filefrag on a nocow file works for me here (3.16.x kernel), I can see
> that filefrag on a directory prints the FIBMAP message.
> 
> > Virtualbox returns  "AHCI#0P0: Read at offset 606236672 (49152 bytes left) 
> > returned rc=VERR_DEV_IO_ERROR". No errors in the kernel log.
> > 
> > Trying "cp vm.vdi /dev/null" returns: cp: Error reading „vm.vdi“: IO-Error
> 
> This could be caused by the virtualization layer. Try to run scrub and
> fsck in the non-destru^Wchecking mode if it finds problems.
> 
> As you're using compression and autodefrag, a quick skim of the 3.17
> patches points to e9512d72e8e61c750c90efacd720abe3c4569822 "fix
> autodefrag with compression", but that's just "keyword" match.
> 
> There's another report about nocow corruption and VirtualBox in a 3.16 +
> for-linus version (which is almost 3.17-rc)
> http://article.gmane.org/gmane.comp.file-systems.btrfs/38701/
> 
> But according to the attached messages, the underlying device is
> unreliable and logs a lot of IO errors.
> 
> For now it looks like VirutalBox is not writing the data or there is a
> bug introduced post 3.16 killing nocow files.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
          --- ...  one ping(1) to rule them all, and in the ---          
                         darkness bind(2) them.                          

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: FIBMAP unsupported

[Marc Dietrich]

Am Donnerstag 02 Oktober 2014, 19:25:49 schrieb David Sterba:
> On Thu, Oct 02, 2014 at 05:13:22PM +0200, Marc Dietrich wrote:
> > I have a large (25G) virtual disk on a btrfs fs. Yes, I know this is not
> > optimial. So I try to defrag it from time to time. However, using "btrfs
> > fi
> > defrag -c vm.vdi" results in even more fragments than before (reported by
> > filefrag). So I wrote my own pseudo defragger,
> 
> Unfortunatelly the default target fragment size is 256k. Try
> 'btrfs filesystem defrag -t 32m ...' or higher numbers and see if it
> helps.

ok, need to try if I can ever recover from this error ...

> > which produces much better results (ok, the file must not be in use).
> > Somewhere in the 3.17 cycle the resulting image got corrupted using the
> > script above.
> > 
> > Running filefrag on it returns "FIBMAP unsupported".
> 
> This message doe not mean it is a corruption, but filefrag tries to use
> the FIBMAP ioctl that is not implemented on btrfs, instead FIEMAP is
> used.
> 
> filefrag on a nocow file works for me here (3.16.x kernel), I can see
> that filefrag on a directory prints the FIBMAP message.

ah, for some reason FIBMAP is used on a file:

# strace filefrag vm.vdi
open("vm.vdi", O_RDONLY)           = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=26681016320, ...}) = 0
fstatfs(3, {f_type=0x9123683e, f_bsize=4096, f_blocks=106856459, 
f_bfree=35124686, f_bavail=33785307, f_files=0, f_ffree=0, f_fsid={741604101, 
-890814488}, f_namelen=255, f_frsize=4096}) = 0
ioctl(3, FIGETBSZ, 0x603150)            = 0
ioctl(3, FS_IOC_FIEMAP, 0x7fff8e0b65a0) = -1 EEXIST (File exists)
ioctl(3, FIBMAP, 0x7fff8e0ba65c)        = -1 EINVAL (Invalid argument)
write(2, "vm.vdi: FIBMAP unsupported\n", 32vm.vdi: FIBMAP unsupported
) = 32
close(3)                                = 0
exit_group(22)                          = ?

So it tries first FIEMAP and fails and then it tries FIBMAP which also fails.

> > Virtualbox returns  "AHCI#0P0: Read at offset 606236672 (49152 bytes left)
> > returned rc=VERR_DEV_IO_ERROR". No errors in the kernel log.
> > 
> > Trying "cp vm.vdi /dev/null" returns: cp: Error reading „vm.vdi“: IO-Error
> 
> This could be caused by the virtualization layer. Try to run scrub and
> fsck in the non-destru^Wchecking mode if it finds problems.

the first error comes from the virtual machine log, maybe we can ignore it.

the second error is on the bare metal (no virtual machine). The disk is ok, so 
the I/O error comes from btrfs itself.

> As you're using compression and autodefrag, a quick skim of the 3.17
> patches points to e9512d72e8e61c750c90efacd720abe3c4569822 "fix
> autodefrag with compression", but that's just "keyword" match.

I have this in my kernel already...

> There's another report about nocow corruption and VirtualBox in a 3.16 +
> for-linus version (which is almost 3.17-rc)
> http://article.gmane.org/gmane.comp.file-systems.btrfs/38701/
> 
> But according to the attached messages, the underlying device is
> unreliable and logs a lot of IO errors.
> 
> For now it looks like VirutalBox is not writing the data or there is a
> bug introduced post 3.16 killing nocow files.

as said above, this is not related to virtualbox (problem exists on bare 
metal).

Will try to restore the file using btrfs restore....

Marc

Re: FIBMAP unsupported

[Marc Dietrich]

Am Donnerstag 02 Oktober 2014, 21:55:55 schrieb Marc Dietrich:
> Will try to restore the file using btrfs restore....

ok, restore worked. I did some more tests. This is unrelated to CoW. It seems 
that the "fallocate -n" in combination with "dd conv=notrunc" using large 
files (>10G) triggers it. Maybe this rings some bells.

Marc

Re: FIBMAP unsupported

[Filipe Manana]

On 10/02/2014 11:11 PM, Marc Dietrich wrote:
> Am Donnerstag 02 Oktober 2014, 21:55:55 schrieb Marc Dietrich:
>> Will try to restore the file using btrfs restore....
> 
> ok, restore worked. I did some more tests. This is unrelated to CoW. It seems 
> that the "fallocate -n" in combination with "dd conv=notrunc" using large 
> files (>10G) triggers it. Maybe this rings some bells.

Just tried it and I confirm filefrag's call to ioctl FS_IOC_FIEMAP fails
with -EEXIST.

It's actually a known issue affecting any of the 3.17 RCs (except RC1).
The extent map manipulation/merging is broken for some cases. Try with
this 2 patches on top of 3.17-rcX:

https://patchwork.kernel.org/patch/4929981/
https://patchwork.kernel.org/patch/4945191/

Or, alternatively, reverting this patch:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=51f395ad4058883e4273b02fdebe98072dbdc0d2

Someone else reported on this list a write/pwrite/writev failure with
errno EEXIST too (and apparently caused by the same reason).

This broken extent map handling is serious IMHO, it can make fsync log
bogus extent items for example, amongst other possible bad and weird things.

> 
> Marc
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: FIBMAP unsupported

[Marc Dietrich]

On Fri, 03 Oct 2014 14:15:11 +0100
Filipe Manana <fdmanana@suse.com> wrote:
> Just tried it and I confirm filefrag's call to ioctl FS_IOC_FIEMAP fails
> with -EEXIST.
> 
> It's actually a known issue affecting any of the 3.17 RCs (except RC1).
> The extent map manipulation/merging is broken for some cases. Try with
> this 2 patches on top of 3.17-rcX:
> 
> https://patchwork.kernel.org/patch/4929981/
> https://patchwork.kernel.org/patch/4945191/

ok, 2nd patch did not apply cleanly, so I just replaced the < with <=.

Otherwise, I can confirm the patches are fixing the issue here.

> Or, alternatively, reverting this patch:
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=51f395ad4058883e4273b02fdebe98072dbdc0d2
> 
> Someone else reported on this list a write/pwrite/writev failure with
> errno EEXIST too (and apparently caused by the same reason).
> 
> This broken extent map handling is serious IMHO, it can make fsync log
> bogus extent items for example, amongst other possible bad and weird things.

Yes, it clearly corrupted by files in a strange way. Please try to get
it into final, or at least -stable if too late.

Marc